I would make 2nd usergroup with upload rights to the gallery and disallow the default usergroup "Registered Users". Auto move users to 2nd usergroup after 5 posts.
1) Add new Usergroup that can upload.
AdminCP => Usergroups => Add New Usergroup
- this usergroup will be where users are moved after 5 posts.
- set setting on the usergroup
2) Make a Promotion to move users
AdminCP => Usergroups => Promotions
- Add New Promotion
Usergroup: "Registered Users"
Promotion Strategy: Posts or Reputation or Date
Move User to Usergroup: select your new usergroup. (from step 1)
3) Disallow uploads from default Registered Users
AdminCP => vBGallery => Usergroup Permissions
- "Registered Users"
-- Uploading Options
--- Can upload images [no]