Comments on: Creating a Robots.txt Page

By: Sean Rasmussen

Sean Rasmussen — Wed, 14 Jul 2010 03:13:46 +0000

You are on the right track there Jill. Another example would be if you have a private members area of your site that you don’t want to be listed in search results.

By: Jill Brown

Jill Brown — Wed, 14 Jul 2010 01:20:06 +0000

Hi Sean,
I hope my understanding is correct, Sean. I see a benefit by using the robot.txt file in that, I can direct search engine spiders to bypass certain pages for indexing.

I may want this because I have duplicate content or weak content that could bring my page ranking down. There could be other reasons why I want a page to be private and not indexed for optimization by search engines. (?)

By: Jody Chambers

Jody Chambers — Sun, 11 Jul 2010 07:12:04 +0000

As I am new to SEO is this what this article means….’By having control over which website pages the search engine robot can crawl will give you more control over your search engine ranking’ ?

By: Sean Rasmussen

Sean Rasmussen — Mon, 22 Mar 2010 05:16:11 +0000

It’s hard to go into detail there Jazz. In a nutshell, the best way to use a robots.txt is to block pages that you do not want to be ranked for and those that provide no SEO value to your website. There is more to it than that, but I will leave that up to your own research

By: Jazz Salinger

Jazz Salinger — Mon, 22 Mar 2010 04:26:35 +0000

Hi Sean,

I’m with Gee. What is the point of blocking certain pages from being crawled. I see where you said that if you only allow access to pages that are optimized, your rankings should increase. Shouldn’t they all be optimized?

Or are you referring to the Contact, About Us pages etc?