Reading User Agents to Identify Bot Traffic

by demtron on Monday, February 02, 2009 08:41 AM

Sometimes, it's a real pain to tell the difference between a human visitor and a bot when reviewing a Web server traffic log.  I recently had an acquiantance ask me for information on this topic.  I found a great resource at http://www.botsvsbrowsers.com/category/1/index.html that catalogs nearly 3000 known user agents that are associated with bots.

In many cases, there are identifiers such as GoogleBot, msnbot and Slurp that are easy to spot as these are common bot user agent signatures.  Unfortunately, there's no common identifier among all of them.  I figure that this list could be pulled into a lookup table and used for matching against a server log.  What I wan't able to identify is how frequently this list is updated for new signatures.


Comments

Comments are closed

Powered by BlogEngine.NET 1.5.1.18
Theme by Mads Kristensen · Adapted by Demtron

Bookmark and Share

Calendar

<<  May 2017  >>
MoTuWeThFrSaSu
24252627282930
1234567
891011121314
15161718192021
22232425262728
2930311234

View posts in large calendar
Log in

Milwaukee SEO Company

Milwaukee Access Programmer/Developer

Milwaukee Website Designer and Developer



Marketing / SEO

Blog Directory
blogarama - the blog directory
TopOfBlogs
Milwaukee area SEO, SEM, ASP.Net