High volume photopost sites: how do you backup?

[dontom]

Hi, we are backing up our pp db every night , this is no problem.

But our data-directory is something around 30 GB, we tar it and scp it to a different server (both connected directly). This takes long (full backup) cron starts it every second night for us, but the whole process (tar, scp) takes around 260 minutes.

How do you do this? Anyone uses some kind of incremental backup?
Thomas

[pengrus]

We are using sftp download them to local harddrive using syncronizing function. It worked very well before. But now because of the emtpy index.html files in each of the folder within data/, the syncronizing does not work any more.

We also have server incremental backup. But I'd like to have a copy at my local computer too.

[dontom]

How does your incremental server backup work? Which software does this?

[pengrus]

cpanel has this setup.

[dontom]

How much data do you backup? (GB)

[pengrus]

25 gb

[WB]

dontom:

Assuming your servers are Unix based, rsync has worked well for us (there may be a Win version, just haven't used it since we are Linux/Unix based). We use it to keep a mirror copy on another server of our PhotoPost install.

It only syncs what has changed so we don't have to copy over everything each time.

[dontom]

WB: yes, we are using debian - how much volume do you backup with rsync? how long does it take?

[WB]

Our PhotoPost install is relatively small at around 4 GB but our largest mirror rsync is about 90 GB total.

Off the top of my head I don't know the exact times since all are run in the wee hours of the morning through cron.

When I've had to manually run one though, I don't ever recall it being over 20 minutes. Keep in mind though that in our instance we are only copying over what has changed (and removing any deleted files). Could definitely take longer depending on the amount of data that changes on a daily basis.

The available 'pipe' size between the two machines would also be a big factor. You can limit the amount of the pipe that rsync uses but in our case we have a decently large limit on some due to the connection in use between the machines.

[dontom]

WB - Thank you for the information - we will look into this
Thomas

[WB]

Sure, glad to help.

I dug up a URL to a site that shows a sample of automated snapshot style backups with rsync:

http://www.mikerubel.org/computers/rsync_snapshots/

We don't use snapshots in all instances (just a straight nightly rsync for most) but it has worked well where we've used it.

[Michael P]

I do MySQL backups on a daily, weekly and monthly setup (using a cron). I also tar up my data directory and store it on another volume on my server from time to time.