Welcome to Welcome to DNF.com™ - Domain Sales, Domain Forum, Domain Appraisals, Domain Registrars

If you are new to domains and looking to buy, sell and learn about domains then you have come to the right place. DNForum is the largest domain name community on the internet and continues to grow every day. There are over 105,000 domainers on DNForum doing everything from buying domains, selling domains, learning about domains and discussing domains. Take a minute and Register.

Register Today on DNForum IT'S FREE!

Results 1 to 4 of 4
  1. #1
    Platinum Lifetime Member
    Seraphim's Avatar
    Join Date
    Jan 2006
    Location
    Hillsboro, OR
    Posts
    3,609
    DNF$
    18,198
    Bank
    0
    Total DNF$
    18,198
    Donate  

    Robots.txt Question

    When using my own domain at various free blog hosts, I occasionally have issues with two versions of the same site getting indexed by Google [example.com + example.bloghost.com]

    Is there a robots.txt command that I can use to keep Google from indexing the free blog host sub domain, while obviously keeping the main domain available for indexing?
    Poland City Portfolio For Sale: Bialystok.com | Gdynia.com | Sosnowiec.com | Torun.com | Zabrze.com | Olsztyn.com | Rybnik.com | Tychy.com | Elblag.com | Opole.com | ZielonaGora.com | Wloclawek.com

  2. #2
    CrossLogix.com
    copper's Avatar
    Join Date
    Mar 2006
    Location
    Matthews, NC. U
    Posts
    2,560
    DNF$
    3,876
    Bank
    0
    Total DNF$
    3,876
    Donate  
    You mean something like this?

    If you want to insert for each page...
    Code:
    <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
    If you want to block entire site for all se robots...
    Code:
    User-agent: *
    Disallow: /
    If you want to block specific folder only...
    Code:
    User-Agent: *
    Disallow: /specific-folder/
    If you want to block specific file only and block just google...
    Code:
    User-Agent: Googlebot
    Disallow: /private/privatefile.htm
    Last edited by copper; 03-19-2010 at 11:29 AM.

  3. #3
    Platinum Lifetime Member
    maregee's Avatar
    Join Date
    Jan 2006
    Location
    New York Metro
    Posts
    17
    DNF$
    823
    Bank
    0
    Total DNF$
    823
    Donate  
    If I understand your question, robots.txt isn't what you want ...

    You're trying to solve a canonical issue, right? Where two (or more) filenames reference the same file?

    Ideally, you'd use .htaccess to rewrite the url's, but since you're on a free host they probably don't let you do that. The other option is to put a "rel=canonical" link in the head section of your pages, pointing each to the name you want to use. For example :

    <link rel="canonical" href="http://www.example.com/yourpagenamehere.htm">

    The blogging software probably has some token that can be used to fill in the page name.

  4. #4
    Platinum Lifetime Member
    Seraphim's Avatar
    Join Date
    Jan 2006
    Location
    Hillsboro, OR
    Posts
    3,609
    DNF$
    18,198
    Bank
    0
    Total DNF$
    18,198
    Donate  
    Thanks for the feedback. I'm going to test the canonical tag [cross domain], as Google now recognizes it. It might take a couple of months to confirm whether it works or not, but I'll report back.
    Poland City Portfolio For Sale: Bialystok.com | Gdynia.com | Sosnowiec.com | Torun.com | Zabrze.com | Olsztyn.com | Rybnik.com | Tychy.com | Elblag.com | Opole.com | ZielonaGora.com | Wloclawek.com

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

Domain name forum recommended by Domaining.com