Data directory cleanup

daxter · August 9th, 2006, 11:56 AM

We have a number of images from deleted ads stored in our data folders.
Did a test and images files are not deleted when deleting an image from an ad, or when the ad itself is deleted.

As far as I can tell there are no tools to delete these orphaned image files?

Chuck S · August 9th, 2006, 04:28 PM

what version of the software you using? also remember an ad is not deleted just cause it is expired you must delete it from the expired que

daxter · August 9th, 2006, 04:45 PM

I'm using 2.5 downloaded 3.july.
Expired que is empty, but the status display shows that several ads have been deleted and some have been automatically removed after being marked as sold.

Chuck S · August 9th, 2006, 10:11 PM

Our program does not delete ads. It is marked expired which includes sold etc and the admin deletes them. You can also set that users can delete there ads.

The only issue I do see looking at this would be the large images are not being deleted. In pp-inc.php find this and change medium to large as there is not a medium size in classifieds

Code:

Content visible to verified customers only.

daxter · August 10th, 2006, 04:40 AM

I'll corrected pp-inc in my 08.08 download and uploaded all files to a test setup.
Placed an ad with an image and then deleted the ad.
All images were deleted instantly(normal, large and thumb).
So it seems like some corrections have been made since our build from beginning of july. Can you tell which files are involved?
But even with the 08.08 build nothing in the expired queue.

And our data directory contains lots of orphaned images that probably needs to be removed manually, there should be a cleanup routine for this.

Chuck S · August 10th, 2006, 07:53 AM

expired que would not be populated unless you run the cron task sendnotice.php

daxter · August 12th, 2006, 04:30 AM

Ok, which other functions rely on this being run?

I see that the medium error is still present in todays build. Which I guess means that all users are stuck with the "large" files "forever"...?
Did a test (with todays build) what happens if a user delete his ad:
Only the first image uploaded is deleted (main/thumbs/large), the other remains.
A cleanup utility is necessary...

Chuck S · August 12th, 2006, 07:37 AM

You setup your cron task as noted here

Cron Job: sendnotice.php

The bug was just a couple days ago so no nothing is updated yet

daxter · August 13th, 2006, 08:19 AM

The "medium" problem is solved, but what about the problem I just described with user deleted ads, is this a known problem or is it something with my installation?

I also see that my root and large folders contains lots of uncompressed BIG image files uploaded by users.
How can this happen?

Chuck S · August 13th, 2006, 10:19 AM

If your large settings and larger than the large images uploaded the images will be uncompressed not touched by the image processor since that is how you set it.

I am not sure what your saying by a problem with user deleted ads. I dont see anywhere in this thread where your reporting a secondary issue.

daxter · August 14th, 2006, 02:19 PM

Quote:

Originally Posted by Chuck S

If your large settings and larger than the large images uploaded the images will be uncompressed not touched by the image processor since that is how you set it.

No, these images are LARGE, up to 3MB. They must be the original, uncompressed jpg's. (A typcal "large" image is up to 100kb).
And they are present both in the main and large folder. And there are no "large" og "thumb" created from them.

Quote:

Originally Posted by Chuck S

I am not sure what your saying by a problem with user deleted ads. I dont see anywhere in this thread where your reporting a secondary issue.

See post 7:
Did a test (with todays build) what happens if a user delete his ad:
Only the first image uploaded is deleted (main/thumbs/large), the other remains.

Chuck S · August 14th, 2006, 02:34 PM

A typical large image is anysize depending on you settings as this can easily be a non compresed touched image if your supersize setting is quite large.

Maximium side for a supersized image? (Larger images will be resized down)

See this setting if this is large and the file dimension is lower than this it is not touched

You might want to refresh the cache on your ftp etc as in my testing if I delete an ad all images are deleted both the main thumbs and large are you talking about bigimage2 3 4 5 6?

redownload the build and upload adm-product.php

daxter · August 15th, 2006, 06:44 AM

Quote:

Originally Posted by Chuck S

A typical large image is anysize depending on you settings as this can easily be a non compresed touched image if your supersize setting is quite large.

Sorry for the confusion, I meant to say that 100k is typical in our setup. (800pix, 70 quality)
And these files are MUCH larger than that, up tp 3MB.

Quote:

Originally Posted by Chuck S

You might want to refresh the cache on your ftp etc as in my testing if I delete an ad all images are deleted both the main thumbs and large are you talking about bigimage2 3 4 5 6?
redownload the build and upload adm-product.php

Thanks, uploading latest adm-product.php fixed the problem, all images are now removed if a user delete his ad.

Back to these large, orphaned files. I've had a suspicion that they might be results of failed/aborted uploads.
Which I got confirmed by testing today.

I places an ad with 3 images at around 900kb.
After a while the upload failed with:
Fatal error: Maximum execution time of 30 seconds exceeded in /public_html/p/image-inc.php on line 138.
(This is on our test server, the real server doesn't have any execution time limit. But still things can go wrong..)

The ad was visible, but with no images. And the image folders contained whats shown on the attachment.
Deleting the ad didnt delete any of the image files.

So obviously failed/aborted uploads leads to orphaned images that currently can only be removed manually.
As said, I really think a cleanup utility is necessary....

Chuck S · August 15th, 2006, 08:14 AM

well uploads where PHP shuts down its prevents the script from deleting failed uploads since PHP is haulted the process is interupted. Nothing we can do about this.

I can not say whether we as a company would release or not release some server cleanup scripts based on server errors such as this.

daxter · August 15th, 2006, 02:41 PM

I will also happen if an upload is aborted by the user, lost connection etc.

I'm surprised that you seem to take this so lightly, without a cleanup utility your customers will have their data folders filled up with orphaned images.
We already have over 1GB of data to backup and I dont like the idea of keeping more than necessary...

Chuck S · August 15th, 2006, 03:10 PM

You can always place a suggestion for something like that in the suggestions forum for future versions when we go through a beta cycle.

I can tell you though I am not going to create something like this cause you see a need for it. I can say also a script like this is kind of outside the software in a way since it is dealing with server failures during a PHP process and not something in our software specifically

daxter · August 20th, 2006, 07:45 AM

I'll be sursprised if I'm the only one who see a need for it.

First, customers who have been using PPC up to now will have "large" images that they will never get rid of unless deleting the manually.
Second, it's not primarely a server issue, as I tried to explain it can also happen during normal use.

A typical scenario will be posting of an ad with several big images, and the user don't wait long enough for the upload.
Or call another page by mistake.

Chuck S · August 20th, 2006, 10:32 AM

Yes but what you describe is server issue or user error both of which we can not combat in any normal situation.

If the server fails or the user switches pages in the middle of a process I could see this happen. Same can be said for any application which allows uploads like vbulletin etc. If the file upload is processed but the process is aborted before the mysql insert is done you will see this

daxter · August 20th, 2006, 04:55 PM

Sure, but other applications have a cleanup utility, at least Invision has.

Chuck S · August 20th, 2006, 05:39 PM

I did not say this was not something we might not add in the future.