SEO - google boot and nofollow question - DNForum - Domain Sales, Domain Forum, Domain Appraisals, Domain Registrars

**nikniknik** · 03-04-2007, 04:31 PM

Hi , I have an intresting question for all the great SEO guys (and gils) here?

Lately I noticed an issue whice few webmasters talked about:

========================
Spiders:
Many webmasters have noticed that there are now two different Google spiders that index their web pages. At least one of them is performing a complete site scan:
* The normal Google spider: 66.249.64.47 - “GET /robots.txt HTTP/1.0″ 404 1227 “-” “Googlebot/2.1 (www.google.com/bot.html)”

* The additional Google spider: 66.249.66.129 - “GET / HTTP/1.1″ 200 38358 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; www.google.com/bot.html)”

Difference between these two Google spiders
The new Google spider uses a slightly different user agent: “Mozilla/5.0″. This means that Googlebot now also accepts the HTTP 1.1 protocol. The new spider might be able to understand more content formats, including compressed HTML.
======================

does any of you guys knows or think that the new google boot neglect the Nofollow or the nofollow external value of the property rel under the a tags?

If some one has a proff or analyzed his server reports , I would realy appriciate it.

what do you think about other search engines in that issue?

**nikniknik** · 03-07-2007, 10:32 AM

surprisingly , no one in DNforum have an answer for me .....
No one ever researched this?

**eurodomain** · 03-14-2007, 08:30 PM

Quote:

Originally Posted by nikniknik

does any of you guys knows or think that the new google boot neglect the Nofollow or the nofollow external value of the property rel under the a tags?

Why would you think that they would neglect nofollow that they themselves introduced and promoted to battle comment spam? Doesn't make any sense.

Different user agents - sure have seen them - a nice tool on Google's part for catching webmasters with sneaky redirects and cloaking that Googel penalizes. I would NOT be surprised (and pretty sure it happens) that many "regular browsers" are in fact GoogleBot rechecking if the same content is served for everyone!

**nikniknik** · 03-15-2007, 07:54 AM

In one hand the nofollow does tell the boot: "listen, dont go there it is not my business what is out there behind that link" on the other hand , the boots do want to bring the user the best content for his search.
lets take for example somthing extremly dangaures:
kids related site , directs to a porn images pages with a nofollow attribute.
will google ever realize this site should be baned.

I assume they do crawel the location , look inside but do not rank it.

My question is , do they even check whats behind the nofollow?
what if this is content that should be baned ?
what if this is caontent that is opposed to the source link content related ? will it infloance the relevance of the source page ? of the entire site?

**eurodomain** · 03-15-2007, 10:24 AM

lets look at what Google told us when they implemented NOFOLLOW

http://googleblog.blogspot.com/2005/...ment-spam.html

Quote:

From now on, when Google sees the attribute (rel="nofollow") on hyperlinks, those links won't get any credit when we rank websites in our search results. This isn't a negative vote for the site where the comment was posted; it's just a way to make sure that spammers get no benefit from abusing public areas like blog comments, trackbacks, and referrer lists.

So their IDEA is to make it so that LINK with nofollow would not BENFIT the spammer in any way. I would count SPIDERING a link a big benefit to it so logic tells me that - NO Google and other engines will NOT spider a nofollow link and will ignore it completely. Oterviese it would still make sense to SPAM high PR blogs to get your site spidered more quickly, cause high PageRank sites do get spidered more often.

tell me if it makes sense?

**nikniknik** · 03-15-2007, 10:39 AM

yes , but still you dont answer my question regarding the "kid" / "porn" site sample.

google claims for not spidering "nofollow" but can he realy do that ?
can he allow people hide from his "eyes" what ever you want?
that doesnt make sense!!!

I am looking for some one that researched this issue , checked google boot path on the site and saw if they ever entered no follow sections.

I am sure google is not that stupid and will not blind him self with his own rules.

**eurodomain** · 03-15-2007, 12:21 PM

Its a BOT that is constantly changing so you will never get a 100% correct / reliable answer... Google Bot could get the link that you have NoFollowed from some other page.

Besides there are so may ways to hide where you are linking to or have the page that holds links blocked in ROBOTS.TXT

I hope you would agree that GoogleBot does obey ROBOTS.TXT ???
So there you have an example where Google would "blind him self with his own rules". Many legit reasons to limit BOT access. Duplicate content being the first one that comes to my mind.
(Like hiding the Print friendly copy of the page you don't want indexed!)

And how abut my argument that SPIDERNG / INDEXING the spammers page benefits him?

Google will try to "do no evil" but there are so many gray/gray choices they have to make.

===

Could you tell me the real-life situation you need this applied to?
You are not linking to a P0rn site from a kids site?

**nikniknik** · 03-15-2007, 12:34 PM

Could you tell me the real-life situation you need this applied to?
You are not linking to a P0rn site from a kids site?

but I have a lot of ads onmy company sites.
we use alot of nofolow , and lately came up to my mind that it all dosent make sense.

How do you think google deal with that issue?
I realy try to find a proff for for your claim . web server reports can probably prove it. dont you think ?

**JuniperPark** · 03-15-2007, 02:16 PM

Quote:

Originally Posted by eurodomain

lets look at what Google told us when they implemented NOFOLLOW

http://googleblog.blogspot.com/2005/...ment-spam.html

So their IDEA is to make it so that LINK with nofollow would not BENFIT the spammer in any way. I would count SPIDERING a link a big benefit to it so logic tells me that - NO Google and other engines will NOT spider a nofollow link and will ignore it completely. Oterviese it would still make sense to SPAM high PR blogs to get your site spidered more quickly, cause high PageRank sites do get spidered more often.

tell me if it makes sense?

It makes sense, but you're wrong.

At the past Pubcon, both Google engineers and Yahoo said the nofollow is badly names, they DO follow but don't count the link in pagerank.

**eurodomain** · 03-15-2007, 05:51 PM

OK - So in this case what you do is make a PHP (/out/site1.php) file for redirecting to Advertises Sites (also can use it to count stats of outgoing users) AND place this file in a directory on your site that is invisible for search engines - use robots.txt to do it:

---
User-agent: *
Disallow: /out/
---

Nice an clean and links will not count and spider will not follow....

www.mysite.com/out/partnerwebstore.php

**JuniperPark** · 03-15-2007, 07:03 PM

Quote:

Originally Posted by eurodomain

OK - So in this case what you do is make a PHP (/out/site1.php) file for redirecting to Advertises Sites (also can use it to count stats of outgoing users) AND place this file in a directory on your site that is invisible for search engines - use robots.txt to do it:

---
User-agent: *
Disallow: /out/
---

Nice an clean and links will not count and spider will not follow....

www.mysite.com/out/partnerwebstore.php

Also called 'cloaking', which will get your entire site banned from Google's index.

**eurodomain** · 03-16-2007, 08:37 AM

Quote:

Originally Posted by JuniperPark

Also called 'cloaking', which will get your entire site banned from Google's index.

Are you sure you got it right?

Cloaking is a black hat search engine optimization (SEO) technique in which the content presented to the search engine spider is different from that presented to the users' browser. This is done by delivering content based on the IP addresses or the User-Agent HTTP header of the user requesting the page. When a user is identified as a search engine spider, a server-side script delivers a different version of the web page, one that contains content not present on the visible page. The purpose of cloaking is to deceive search engines so they display the page when it would not otherwise be displayed.

In the example I suggested (and I use it on multiple sites) the SPIDER BOT will be given NO content, and the user would get a PHP file that redirects it to another site - affiliate store.

I personally have been switching to using "nofollow" lately as I feel it is more simple and so better for me and better for my users.