Reading User Agents to Identify Bot Traffic

by demtron on Monday, February 02, 2009 08:41 AM

Sometimes, it's a real pain to tell the difference between a human visitor and a bot when reviewing a Web server traffic log. I recently had an acquiantance ask me for information on this topic. I found a great resource at http://www.botsvsbrowsers.com/category/1/index.html that catalogs nearly 3000 known user agents that are associated with bots.

In many cases, there are identifiers such as GoogleBot, msnbot and Slurp that are easy to spot as these are common bot user agent signatures. Unfortunately, there's no common identifier among all of them. I figure that this list could be pulled into a lookup table and used for matching against a server log. What I wan't able to identify is how frequently this list is updated for new signatures.

Tags: web design, analytics, user agents, bots

Killing Your Search Engine Ranking in 7 Easy Steps

by demtron on Wednesday, January 14, 2009 07:40 PM

Have you ever wanted to completely destroy your search engine ranking or do it for someone else? Maybe you never want your site to get found again? Believe it or not, I've helped clients in the last six months with each of these problems that were torching their search engine profile and strangling their organic traffic.

1) Domain masking: I took over one site where the entire site was using domain masking. In seven years, the client had absolutely no idea that the previous designer was doing this to save himself a buck on a hosting plan for the site. Only the home page was found in search engines. It turns out the designer did exactly the same thing with all the rest of his clients' sites that have been live for years. They, too, only have the home page to show for it in search engine results.

2) Use only a JavaScript menu for linking pages: Sure, JavaScript menus are cool. They can drop down, slide across, have pictures and generally spice up a site. But they can't be crawled by search engines. What's more, the links in them don't contribute anchor text, either. One recent client had over a hundred pages in a JavaScript menu and practically no linking using anchor tags. Most of the 200 pages of the site had not been crawled at all

3) Use a JavaScript page strip: ASP.Net is famous of offering page navigation strips using is doPostBack JavaScript methods. Another client I acquired had over 6 thousand pages on a site, but only 32 pages actually crawled. The remaining pages were all accessible through paginated tables. Another great waste of code that a search engine ignores.

4) All pictures and Flash with little text: Some designers with a flair for graphic design take sites a little overboard. If you're a famous pop star and have zillions of fans finding your site every day by just typing yourname.com, then who cares? In the real world, most sites are not wildly popular and are only found through search engine results. SEs love text, especially keyword-rich, backlinked text. Pictures and Flash sure are pretty, but they basic tell nothing to a crawler.

5) Renaming pages without redirects: One site I redesigned earlier this year had tons of links from other sites pointing to a page that was non-existent. What a complete waste of free traffic and promotion! Both search engines AND human visitors wouldn't find the site. Oh, what a little bit of 301-redirect action did to help out that one.

6) Leaving title tags blank: One of the aforementioned sites had about 60 of it's 200 pages with blank titles. How is anyone going to find those pages, and why would anyone click on them? Here, let's write a book, then tear the front cover off and leave one of those "this page intentionally left blank" pages as the new front cover. Real slick.

And last, but not least...

7) user agent: * disallow: / in the ROBOTS.TXT file: This one didn't actually happen, although it was close. The site had the disallow all set for a user agent of Google. So, they kissed 81% of their traffic away just by a simple screw-up by the former designer.

And there you have it. If you implement these seven key steps, your success with annihilating your search engine exposure and traffic is pretty much guaranteed. Good luck and happy destroying!

Tags: google, traffic, web design, search engine optimization, seo

Google | Internet Marketing | SEO | Web Design

Centering a DIV Window with Cross-Browser JavaScript

by demtron on Wednesday, January 14, 2009 02:28 PM

Years ago, I wrote a little bit of JavaScript to position a DIV control in the center of a browser window space by calculating the browser area, the window size, then determine the coordinates of the place where the DIV would need to be placed. Since I forgot where I put it and it brobably wouldn't be cross-browser compatible anyway (orginally written for IE 5/6), I searched the web and ran across this JavaScript centering code that accomplishes exactly that. Here are the excerpts of the code needed to do this.


window.size = function()
{
var w = 0;
var h = 0;
//IE
if(!window.innerWidth)
{
//strict mode
if(!(document.documentElement.clientWidth == 0))
{
w = document.documentElement.clientWidth;
h = document.documentElement.clientHeight;
}
//quirks mode
else
{
w = document.body.clientWidth;
h = document.body.clientHeight;
}
}
//w3c
else
{
w = window.innerWidth;
h = window.innerHeight;
}
return {width:w,height:h};
}
window.center = function()
{
var hWnd = (arguments[0] != null) ? arguments[0] : {width:0,height:0};
var _x = 0;
var _y = 0;
var offsetX = 0;
var offsetY = 0;
//IE
if(!window.pageYOffset)
{
//strict mode
if(!(document.documentElement.scrollTop == 0))
{
offsetY = document.documentElement.scrollTop;
offsetX = document.documentElement.scrollLeft;
}
//quirks mode
else
{
offsetY = document.body.scrollTop;
offsetX = document.body.scrollLeft;
}
}
//w3c
else
{
offsetX = window.pageXOffset;
offsetY = window.pageYOffset;
}
_x = ((this.size().width-hWnd.width)/2)+offsetX;
_y = ((this.size().height-hWnd.height)/2)+offsetY;
return{x:_x,y:_y};
}

Here's some example code to show how it's supposed to be used:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<title></title>
<script type="text/javascript" src="Window.Size.js"></script>
<script type="text/javascript">
function showCenter(point)
{
var div = document.createElement("div");
div.style.background = "#dedede";
div.style.position = "absolute";
div.style.top = point.y + "px";
div.style.left = point.x + "px";
div.style.width = "100px";
div.style.height = "100px";
document.body.appendChild(div);
}
</script>
</head>
<body>
<div style="height:1200px"></div>
<input type="button" value="Get Center" onclick="showCenter(window.center({width:100,height:100}))"/>
</body>
</html>

Tags: javascript, ajax, web design

Ajax | Web Design

Control Focus, AJAX UpdatePanel Postback and Cursor Position

by demtron on Thursday, December 04, 2008 05:19 PM

When using ASP.Net AJAX to perform a postback from within a textbox input control, I found that the textbox loses focus after the postback. To overcome this, I added the following code:

ScriptManager.GetCurrent(Me.Page).SetFocus(MyTextBox)

However, upon setting focus back to the textbox, the cursor lands at the beginning of the text in the control. To solve this, just add the following JavaScript on the page or within window.setTimeout:

var textInput = $get('MyTextBox');
if (textInput.createTextRange)
{
    var range = textInput.createTextRange();
        range.moveStart('character', (textInput.value.length));
        range.collapse();
        range.select();
    }

Tags: javascript, asp.net, web design

ASP.Net | Web Design

Mo	Tu	We	Th	Fr	Sa	Su
25	26	27	28	29	30	31
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30	1	2	3	4	5

Milwaukee area SEO, SEM, ASP.Net, and eCommerce consulting firm

Reading User Agents to Identify Bot Traffic

Killing Your Search Engine Ranking in 7 Easy Steps

Centering a DIV Window with Cross-Browser JavaScript

Control Focus, AJAX UpdatePanel Postback and Cursor Position

RecentPosts

Calendar

Mo	Tu	We	Th	Fr	Sa	Su
25	26	27	28	29	30	31
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30	1	2	3	4	5

Mo	Tu	We	Th	Fr	Sa	Su
25	26	27	28	29	30	31
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30	1	2	3	4	5

Milwaukee area SEO, SEM, ASP.Net, and eCommerce consulting firm

Reading User Agents to Identify Bot Traffic

Killing Your Search Engine Ranking in 7 Easy Steps

Centering a DIV Window with Cross-Browser JavaScript

Control Focus, AJAX UpdatePanel Postback and Cursor Position

Category list

RecentPosts

Calendar

Mo	Tu	We	Th	Fr	Sa	Su
25	26	27	28	29	30	31
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30	1	2	3	4	5