Htaccess Deny

Note that malware or email harvesting bots will ignore the directives of the robots.txt file, you must create an entry in your .htaccess file.

The following example was provided by Gustave Dahl on User2 list 11/12/2009 11:05 PM

BrowserMatchNoCase PiplBot bad_bot
BrowserMatchNoCase MJ12bod bad_bot
Order Deny, Allow
Deny from env=bad_bot

ICDSoft support also provided the following as a another way to ban bots

SetEnvIfNoCase User-Agent "Yahoo" badBot
SetEnvIfNoCase User-Agent "bingbot" badBot
SetEnvIfNoCase User-Agent "MJ12bot" badBot
SetEnvIfNoCase User-Agent "Yandex" badBot
SetEnvIfNoCase User-Agent "BaiDuSpider" badBot
SetEnvIfNoCase User-Agent "Mail.ru" badBot
Deny from env=badBot

References

Provided by Gustave Dahl on User2 list on 13 Nov 2009

• User Agent String Explained
• List of User Agent Strings

See also:

• How To Block Bots, Ban IP Addresses With .htaccess

• Block unwanted users from your site using .htaccess

Related links

The following provide additional security measures:

Controlling Site Access

• Using tngrobots.php
• Robots txt
• How to setup a robots.txt file
• Bot Trap
• Htaccess
• Htaccess Deny
• Htaccess Rewrite

Protecting Resources

• Permissions Explained
• Database User
• Move your configuration files
• Move your backup files
• Move your gedcom files

• Overlaid Subroot how to recover from subroot.php overlay
• Prevent Directory Listing
• Protecting access log

• Turn off global register and other PHP overrides

Checking your site for Malware

• Sucuri SiteCheck