PhotoPost Photo Gallery Sales PhotoPost Sales Toll Free Phone Number
Mon-Fri 9am-4pm EST
  PhotoPost Photo Sharing Photo Gallery    Visualize community tm
| | | | | | | | |

Go Back   PhotoPost Community > General Forums > General Discussion

General Discussion General use discussion forum for PhotoPost products.

Reply
 
Thread Tools Rate Thread Display Modes
Old October 26th, 2007, 01:21 PM   #1
jdelasko
Member
Verified Customer
 
Join Date: Jan 2007
Posts: 61
How to block offline browsers A.K.A site rippers

There's a lot of offline browsers that will sit on your site and download every single file. You don't need one of these on your site sucking up your bandwith and slowing your site down for legitimate users. Here's how to deal with them:

Put the following lines in a .htaccess file in your sites root directory. It will give ANY IP address using one of these ofline browsers an error 403. Included below are most currently known offline browsers:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} Indy\ Library [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus
RewriteRule ^.* - [F,L]



Replace the last line above with this optional Rewrite Rule and any ofline browser will be redirected to the site you specify. This is handy if you simply want them off your site immediately. You can redirect them to that site that site that emails you all that presciption drug spam for instance:


RewriteRule /*$ http://www.yourdomain.com [L,R]


Just replace the 'www.yourdomain.com' with whatever web site you choose and the site ripper will be redirected there


If you really want to fight back, below is a great site to redirect to. The site below will redirect email harvesting bots to trap sites that will feed them with an almost infinite loop of dynamically generated fake email addresses, mostly on known spammer owned domains! This will render their harvested lists practically useless and of no commercial value.



RewriteRule /*$ http://english-111745432354.spampoison.com [L,R]

Last edited by jdelasko; October 26th, 2007 at 11:07 PM.
jdelasko is offline   Reply With Quote
Old October 26th, 2007, 01:30 PM   #2
Chuck S
Photopost Developer
Verified Customer
 
Chuck S's Avatar
 
Join Date: Jun 2002
Location: Abingdon,MD
Posts: 75,041
Nice one
__________________
Photopost Developer and Support Engineer

Please do not PM me for support or sales questions. Thank you for your understanding.
Chuck S is online now   Reply With Quote
Old October 30th, 2007, 05:17 PM   #3
Johnny Doomo
Senior Member
Verified Customer
 
Johnny Doomo's Avatar
 
Join Date: Mar 2003
Posts: 877
1. I've always been told to upload that htaccess file as a htaccess.txt file first, then rename it on the server. Sometimes uploading it as a .htaccess file garbles the contents of the file.

2. Do you know if this would work on a video site if you're trying to prevent videos from being ripped? Or would that be a whole batch of different ripper names?

Thanks for the heads up.
Johnny Doomo is offline   Reply With Quote
Old October 30th, 2007, 06:12 PM   #4
jdelasko
Member
Verified Customer
 
Join Date: Jan 2007
Posts: 61
The .htacces file described in this post prevents any and all files from being accesed by any of the user agents in the file. That doesn't mean someone still can't rip files some other way.

Not sure about that uploading .htaccess as txt. I use CuteFTP Pro and have never had any problems.
jdelasko is offline   Reply With Quote
Old October 30th, 2007, 09:06 PM   #5
oldengine
Member
Verified Customer
 
Join Date: Oct 2004
Location: Florida
Posts: 317
Thanks for that! I put it right to work because the other evening I had load levels as follows:

22:15:17 up 23 days, 16:09, 1 user, load average: 223.64, 107.13, 42.64

I was shocked that the server didn't go all the way down and this is a Dell Dual Xeon with 4 Gig of RAM and I'm the ONLY user. I suspected that some robot was having its way with my site because it cleared back down to its normal 1.5 not long afterward.
oldengine is offline   Reply With Quote
Old December 1st, 2007, 09:06 PM   #6
PossumX
Junior Member
Verified Customer
 
Join Date: Oct 2007
Posts: 25
I'll be adding the ones I don't have in my .htaccess file tonight. Those site rippers can be a serious pain for your bandwidth.
PossumX is offline   Reply With Quote
Old December 19th, 2007, 12:14 AM   #7
Alien515
Member
Verified Customer
 
Join Date: Jul 2003
Posts: 41
If I have a subdomain like forums.domain.com, do I put these changes in a .htaccess in the root (domain.com) or put it in the root of my subdomain (/forums) or both?

Thanks!
Alien515 is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 01:42 PM.

Powered by vBulletin® Version 3.8.1
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.