Disk space used = 112 GB, that's not right [PP 6.2]

Swanny · January 25th, 2009, 07:41 PM

OK, so I did the upgrade today. From 5.21 to 6.2 (latest build). The upgrade went very smooth considering there's over 30,000 photos (and just under 100,000 members).

There is a problem though... when I (Webmaster) click "Upload Photo" the page takes literally 2-5 minutes to load. And when it does load I notice this interesting thing:

Quote:

Disk space allowed on your account: Unlimited
Disk space used on your account: 112,482,772kb (112482772 bytes)
Disk space remaining for your account: Unlimited

Obviously I'm not really using 112 GB of data (112 MB, that's more reasonable and possible). In fact, the entire data folder is maybe 1.0 - 1.5 GB. In the admin panel, it shows space used in the data folder as 1.43 GB (which is correct).

Now, I am assuming the extremely long page load time is due to it calculating how much space I've used (is that right?). And it calculates it WAY higher than it actually is. The odd thing is when I log in as a regular member with no existing photos, the uploadphoto.php page loads quickly (presumably because the calculation of disk space used is 0).

So I suspect this is a bug. Please investigate and advise. I can give you login details if you need. If Chuck sees this the login details are username photopost and the password is your last name (all lowercase). That's a regular user account BTW. http://www.fordf150.net/photos/

Thanks.

Swanny · January 25th, 2009, 08:12 PM

Oh, I also wanted to add... what I've done and other notes:
- Re-uploaded a fresh copy of upload.php and uploadphoto.php (current 1/8/09 build)
- Verified permissions on upload folder (777)
- Category cache is disabled
- Flash uploader is turned off (I read in other posts here that it's a little buggy)
- My site is running PHP 4.4.2 (on a VPS, not shared hosting), connecting to a phpBB user db
- Uploads *do* work, but the entire process takes upwards of 10 minutes!! (for me only)
- No other speed/timing issues on any other page on the site, this only happened after the upgrade

Swanny · January 25th, 2009, 09:04 PM

OK, well now add to that...

I cannot edit any photo (editphoto.php). That just times out after "hanging" for a few minutes. Help!

Chuck S · January 26th, 2009, 10:11 AM

well I will check out the upload number probally a simple match calculation issue.

As far as hanging etc have you tried this?

HTTP 500 internal Server Error

Swanny · January 26th, 2009, 11:42 AM

Hi Chuck,

I added this in the pp-inc.php:
ini_set("memory_limit", "48M");
ini_set("max_execution_time", 0);

It made no difference. There are no 500 errors, I checked the logs. When I login as a regular user the site is speedy, it's just on my account with 2,400 photos that it's slow. So far no one else has complained.

So, here's where I'm at today. I can upload photos, it just takes a few minutes between each step. I *cannot* however edit any photos. When I'm in a photo and I click edit, that does seem to timeout.

Perhaps the only way you'll see what I see is if I give you my own admin login details. If it comes down to it I'll do that Chuck.

Chuck S · January 26th, 2009, 11:47 AM

I am not seeing any issues with your site when logged in editing uploading etc.

Ford F150 Photos - 0231

I see things are pretty modified on the site not sure if any overhead is added by those ads.

You can try upping that memory statement to 60. It could possibly be a timeout generating your cats list if its real long.

Swanny · January 26th, 2009, 11:51 AM

Well that's kind of the problem. It's only happening on my Webmaster account, I think it's due to having so many photos... so I can't replicate the issue with the generic photopost account...

You tell me. When loading uploadphoto.php or editphoto.php (the two files that seem to be causing the problems), what process is in place for checking permissions / file usage / etc. What could be hanging /slowing down those two files? Obviously uploadphoto.php does some checking on how much space I'm using, but I can't figure out editphoto.php...

I'll up that limit right now and try again. I'll post back in a few minutes.

Swanny · January 26th, 2009, 11:54 AM

Quote:

Originally Posted by Chuck S

I see things are pretty modified on the site not sure if any overhead is added by those ads.

I'm not sure what you mean by pretty modified. I changed some colors and added one ad on the showphoto.php page beside the photo. Otherwise it's stock. I made no edits to the uploadphoto.php or upload.php files. I guess it's a little modified ;-)

Quote:

Originally Posted by Chuck S

You can try upping that memory statement to 60. It could possibly be a timeout generating your cats list if its real long.

AHA! I wonder if that's it? There are lots of categories, and oodles of personal galleries as well. I'll also create another admin account and see if I can duplicate it there. Because if another admin account bombs out on the editphoto.php page too it could be due to the long cats list.

I'll keep you posted.

Chuck S · January 26th, 2009, 12:05 PM

I mean the ads on the index and showproduct yeah they should not cause any issue but from a support standpoint I need to meantion them as some things where modified from the default code. I have seen code edits cause many issues so just a note.

I highly suspect your issue is grabbing all the cats especially if lots of personal albums since admin sees everything. If only admin accounts have an issue we can change the cat pull to maybe only show the admins direct albums which would help you.

Swanny · January 26th, 2009, 12:06 PM

Quote:

Originally Posted by Swanny

...AHA! I wonder if that's it? There are lots of categories, and oodles of personal galleries as well. I'll also create another admin account and see if I can duplicate it there. Because if another admin account bombs out on the editphoto.php page too it could be due to the long cats list...

But wait! SUCCESS!!! Well, sort of.... I gave another account admin privileges and now when I click Edit Photo it hangs too. It didn't hang when that account had regular permissions. So it looks like that problem is narrowed down. I'm actually able to reproduce that problem!

So where do we go from here? I guess the fix would be to decrease the number of galleries somehow, or adjust a setting somewhere in the scripts... FYI, I have 1,167 rows in the pp_categories table. That means there are 1,167 categories, and I guess they're all being generated into that dropdown list box for "moving" a photo... Is that right? About 1,000 of those are for personal member galleries. I would be willing to disable personal galleries if that helps... should I try that next or would that not affect existing galleries?

Progress!

p.s. I didn't mention this earlier but it takes a minute or two for the main administration page to load too (presumably because it's doing some size calculations as well).

p.p.s. Updating from 48M to 60M had no effect.

Chuck S · January 26th, 2009, 12:13 PM

In Uploadphoto.php linesd 246-248 need to be changed to simply this

Code:

Content visible to verified customers only.

Chuck S · January 26th, 2009, 12:20 PM

As far as limited the cat display on editphoto in that script find this

Code:

Content visible to verified customers only.

replace twice I beleive with this

Code:

Content visible to verified customers only.

Swanny · January 26th, 2009, 12:30 PM

I made the change. I timed it. From the time I clicked "Upload Photos" until the page loaded was almost exactly 4 minutes.

Well that helped:

Quote:

Disk space allowed on your account: Unlimited
Disk space used on your account: 109,846kb (112482772 bytes)
Disk space remaining for your account: Unlimited

One of the issues is now resolved. Will this be updated in the next release of PP or do I have to remember to edit this?

I also noticed that the number of member galleries on the Upload page is about about 1,000. So it does appear that generating that big list is potentially causing the problem. I'd like to do one of two things:
1. Delete all personal galleries so the script loads faster; or
2. Remove the personal galleries listing from the editphoto.php and uploadphoto.php pages

Honestly I don't want to remove the personal galleries simply because people use them. So is there a way to temporarily remove them to see if that's causing the slowdown?

Is it possible that the calculation of my disk space is slowing down the page as well? I'm not sure exactly how that works. Is PP actually scanning the file system to see how much space I'm physically using or is it just adding up the file sizes from all my photos (as entered in the db)?

Chuck S · January 26th, 2009, 12:33 PM

Well I gave you an edit for editphoto to catmoveopt you can use you can also apply that to the catmoveopt lines in uploadphoto.php script as well to only show your albums and the main categories.

Swanny · January 26th, 2009, 12:33 PM

Quote:

Originally Posted by Chuck S

As far as limited the cat display on editphoto in that script find this

Code:

Content visible to verified customers only.

replace twice I beleive with this

Code:

Content visible to verified customers only.

That did it. The editphoto page is fast now!

Chuck S · January 26th, 2009, 12:34 PM

As noted do the same to uploadphoto only difference now would be you cant move photos to other users personal albums.

Swanny · January 26th, 2009, 12:35 PM

I was typing that my earlier reply while you posted, I didn't see it until after I post it.

So in uploadphoto.php, lines 176-179 are where the catmoveopt lines are. What should I change them to? This is what it is right now:

Code:

Content visible to verified customers only.

Swanny · January 26th, 2009, 12:45 PM

Quote:

Originally Posted by Chuck S

As noted do the same to uploadphoto only difference now would be you cant move photos to other users personal albums.

Chuck,

Now that it seems the issue is resolved this begs the question. Is this something that could be coded as a setting in the admin panel? You know how there are currently the following "tweaks" for large PP installs:
- Use photo view caching to reduce view update queries on high load sites? Set this to YES if you want to enable the view caching code which caches photo view updates
- Load limit for server? If running *nix and you want to disable your forums when load goes over a certain limit; place the high limit here.
- Use category cache? On some larger sites, using the category cache can actually reduce performance.

Well... obviously the issue I've identified is an issue for some larger sites. And as we can see other accommodations have been made for larger sites. I would like to see an additional option:
- Disable admin move photos to personal categories On some larger sites uploadphoto and editphoto can take a long time to load if there are lots of personal categories. Turn this off to speed up those pages (for admin only)

Something to that effect would benefit PP customers in two ways:
1. I wouldn't have to "hack" every upgrade I do in the future
2. As other customers' PP installs grow, this is something that can be easily toggled on/off as necessary

What do you think? How hard would that be to code?

Chuck S · January 26th, 2009, 12:47 PM

It is something I will discuss with Michael about adding as an option.

Swanny · January 26th, 2009, 12:49 PM

Quote:

Originally Posted by Chuck S

It is something I will discuss with Michael about adding as an option.

Thanks. In a way I hope others have the same problem to help the answer to that be "yes, we need it". LOL. Now about that code update for uploadphoto.... see my post a couple posts up. Please...