March 13, 2012

Don't let robots.txt muck up your AdSense ads

By Donald Ritchie

If you are an AdSense publisher, take a moment to review your robots.txt file. It could be hurting your revenue.

Your site probably has a number of admin pages - things like a contact page or a privacy policy - that you don't want to show up in search results. Webmasters typically use robots.txt to block the search engines' crawlers from accessing those pages. This ensures that the pages in question don't outrank your more important pages within the search results.

But what if the admin pages also carry AdSense ads? After all, that's a perfectly legitimate thing for them to do.

The problem is that the robots.txt file will also prevent the AdSense crawler from analyzing the text on the page. The ads will still show up, but the chances are that they will no longer be related to the subject of the page. That in turn will lead to lower earnings.

Fortunately, the problem is easy to fix. Just add the following lines to your robots.txt file:

User-agent: Mediapartners-Google

This will give blanket access to the AdSense bot without affecting any other crawlers.

By the way, if you want to see whether the AdSense crawler can access a given page on your site, log into AdSense, click on the "Account settings" link (on the Home tab), scroll down to "Access and Authorisation", and click the "View errors" link next to "Crawler access".

See also:

No comments:

Post a Comment