Robots.txt Question

[Seraphim]

When using my own domain at various free blog hosts, I occasionally have issues with two versions of the same site getting indexed by Google [example.com + example.bloghost.com]

Is there a robots.txt command that I can use to keep Google from indexing the free blog host sub domain, while obviously keeping the main domain available for indexing?

[copper]

You mean something like this?

If you want to insert for each page...

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

If you want to block entire site for all se robots...

User-agent: *
Disallow: /

If you want to block specific folder only...

User-Agent: *
Disallow: /specific-folder/

If you want to block specific file only and block just google...

User-Agent: Googlebot
Disallow: /private/privatefile.htm

[maregee]

If I understand your question, robots.txt isn't what you want ...

You're trying to solve a canonical issue, right? Where two (or more) filenames reference the same file?

Ideally, you'd use .htaccess to rewrite the url's, but since you're on a free host they probably don't let you do that. The other option is to put a "rel=canonical" link in the head section of your pages, pointing each to the name you want to use. For example :

<link rel="canonical" href="http://www.example.com/yourpagenamehere.htm">

The blogging software probably has some token that can be used to fill in the page name.

[Seraphim]

Thanks for the feedback. I'm going to test the canonical tag [cross domain], as Google now recognizes it. It might take a couple of months to confirm whether it works or not, but I'll report back.