Stop robots from crossing into subsites

[Reed92]

I have two domains. One is pointed to a folder inside of my main site.

If I make a robots.txt that stops robots from crawling that folder, can they still access it by using the secondary domain? Or will they never crawl that site again?

 

[PixelPusher]

FYI, robots.txt is merely a request to the web crawler(s). Most bots/crawlers will adhere to your requests in the txt file, but there are those bots that will ignore it.

In regards to your question, what you are saying is you have two domains like so:

1. example.com
2. coolsites.com (which actually points to example.com/websites)

Correct?

If this is the case, the robots.txt file that you have for the 1st domain will effect all sub directories (including example.com/websites).

 

[Reed92]

FYI, robots.txt is merely a request to the web crawler(s). Most bots/crawlers will adhere to your requests in the txt file, but there are those bots that will ignore it.

In regards to your question, what you are saying is you have two domains like so:

1. example.com
2. coolsites.com (which actually points to example.com/websites)

Correct?

If this is the case, the robots.txt file that you have for the 1st domain will effect all sub directories (including example.com/websites).

I know it's just a request, but it should stop most big search engines from crawling the wrong folders.

That is exactly the scenario I have. If I Disallow example.com/websites, it will stop crawling that from the example.com domain.

But, it will still be able to crawl coolsites.com (which is example.com/websites) correct?

 

[PixelPusher]

If you disallow "/websites" that will effect "coolsites.com", (it will not be crawled). All that second url does is redirect to to a sub directory of the main domain.