Note that malware or email harvesting bots will ignore the directives of the robots.txt file, you must create an entry in your .htaccess file.
The following example was provided by Gustave Dahl on User2 list 11/12/2009 11:05 PM
BrowserMatchNoCase PiplBot bad_bot BrowserMatchNoCase MJ12bod bad_bot Order Deny, Allow Deny from env=bad_bot
References on User Agent
Provided by Gustave Dahl on User2 list on 13 Nov 2009