robots.txt file?

[GT Web]

Will there be any difference if you have no file as opposed to having one that says:

User-agent: *
Disallow:

Will search engines treat you better if you have a simple robots.txt file like I posted above?

 

[007]

I've been doing some research on this subject too. I'd be interested to hear the best way also. Good question.

 

[cwsteam.com]

I would suggest having a robots.txt is a good sign in itself, 2nd, try to have a robots.txt like www.seomasters.com/robots.txt

 

[eggbilly]

I've been doing some research on this subject too. I'd be interested to hear the best way also. Good question.

I've updated many of my sites in the past week to include a simple robots.txt file, since reading somewhere that a simple robots.txt file is better than none at all.

It could all be rumour, but seeing that all of my 404 errors in the logs were down to missing robots.txt files, I wanted to tidy things up a bit.

The rumour was that Google won't rank you as highly without the file - it could be rubbish, but I didn't want to take the risk.

My file is simple:

User-agent: *
Disallow: /images/

I try and change the disallowed directories on each site, just in case the s/e's think I'm spamming.

 

[alien]

I don't include the robots.txt unless I want to disallow certain directories.

Also disallow certain user-agents like Teleport or Wget are pretty useless since they tend not to follow robots rules or they can simply change the user-agent.

 

[007]

I have a site that is #1 on google for a certain keyword. No robots.txt file either.

 

[BLazeD]

If you want them to spider your whole site it doesnt matter if you have one or not

 

[udt89]

i checked my site stats and i have a lot of visits to robots.txt which doesnt exist on my site. It comes up with 848 "not found" hits. So i'm adding one right now. I assume i add it to the main www folder on ftp?

 

[SirSpider]

I have a site that is #1 on google for a certain keyword. No robots.txt file either.

Me too - and for a very competitive travel term. I guess I have been lucky with that one. it is optimized and has solid content, but it's not even a dot-com and has no robots.txt.

 

[WebPhil]

If you don't have a robots.txt, then instead of the search engines finding it and knowing to either ignore stuff or not ignore anything, it'll give you a 404 error.

I know that some people don't care about this, but for me, I'd rather not have the 404 errors on the sites.

We've had sites which have been top 10 in Google for certain keywords without robots.txt, but we can do without the error reports in the logs. :approve:

 

[cwsteam.com]

yes, just keep a robots.txt with the user agents you absolutely want to disallow and with directories which u do not want crawled.

Also, create a favicon.ico file and upload in the root. Netspace uses it very well. And I see a lot of requests for that in my logs.

 

[flexiwebhost.com]

be careful when using robots.txt you could inadvertantly make peope aware of directories you would prefer them not to know about.

 

[ForumDomains]

yes, just keep a robots.txt with the user agents you absolutely want to disallow and with directories which u do not want crawled.

Also, create a favicon.ico file and upload in the root. Netspace uses it very well. And I see a lot of requests for that in my logs.

Do not create favicon.ico file if you run a PPC campaign through FindWhat or PPC engines based on FW listings. For time being, their click fraud prevention system can not filter out fraudulent click practices that exploit favicon.ico file. It makes difference if favicon.ico request is 404 or 200 in that case.