If you enable captcha they must be able to post to enter the captcha as that is where it is shown
You can not delete an avatar once set that I see. Not really a bug thats just how it is coded now since it is a relatively new feature I am sure it will grow in time