Enjoy unlimited access to all forum features for FREE! Optional upgrade available for extra perks.
Domain summit 2024

Design A Spider Friendly Site

Status
Not open for further replies.

A D

Level 14
Legacy Exclusive Member
Joined
Feb 20, 2003
Messages
15,040
Reaction score
1,189
To be successful in the search engines it's important to design your web site with the spiders in mind. Using the latest in web page design is not generally the best way to go. Spiders don't view web pages like humans do, they must read the HTML in the page to see what it's about. Below you will find tips on how to best design your web site with search engines in mind.

Do not use frames at all. Some search engines cannot spider web pages with frames at all. For the other search engines that can, they can have problems spidering it and sometimes they too can't index the web page. Do not only use image's to link out. You should always use text links to link out to important content on your web site. Spiders can follow image links, but like text links more though.

Use external JavaScript files instead of using Java Script code in the HTML document, using Java Script in the HTML document will make the page size much larger. Using an external Java Script file to do the job will reduce page size and make it easier for both spiders and browsers to download the page. Using Cascading Style Sheets can reduce page size and making the download time much faster in most cases. It will allow the spider to index your web page faster and can help your ranking.

Avoid using web page creators such as FrontPage, Dreamweaver or a WYSIWYG editor. Software such as that will often times add scripting code that is not needed, making the page larger than it needed to be and making it harder to crawl. It will also add code that can't be read by the search engines, causing the spider not to index the page or not index the whole web page. It is better to use standard HTML. Adding code that they can't read or have a hard time to read can lead to major problems with your ranking.

Try not to use Flash when possible. Flash cannot be read by the search engines to date and will cause download time to slow a bit. If you do decide to use Flash anyway, make sure you add text to the web page, so the search engines have something to read and find out what your web page is about. It will also allow your visitors to have something to read while the Flash file loads. Also don't use Flash as a way of navigation, as I said before spiders cannot read Flash.

It's important to add a site map to your web site. Not only will this make it easier for internet surfers to get around your web site, but it will also allow spiders to find your site's content easier and index your web page sooner. The site map should contain text links and not image links.

I highly suggest that you look at your web page with a Lynx browser because this is similar to how search engines will view your web page. There are other tools on the internet that will allow you to view your web page without a Lynx browser, but see a web page just like it, so you may want to check those out as well.
 

ibuddy

Level 2
Legacy Platinum Member
Joined
Jan 17, 2006
Messages
33
Reaction score
0
This was pretty good to know because I was wondering about FLASH and FRAMES.

Thx for the info
 

katherine

Country hopper
Legacy Exclusive Member
Joined
Jul 9, 2005
Messages
8,428
Reaction score
1,290
Possibly. Some people actually prefer Notepad !
If you use tools like frontpage then be prepared to clear the mess, as it puts a lot of proprietary tags in your code etc. You have to know HTML a little bit to do the cleaning and if you know HTML then you don't need tools like this.

I would also advise to design using CSS and learn compliant HTML coding - HTML 4.01 or XTML 1.0 - see www.w3.org and http://validator.w3.org/
 

denny007

Level 9
Legacy Exclusive Member
Joined
Oct 5, 2004
Messages
3,298
Reaction score
24
Also try to give true "last modified" (with proper response "not modified" in case it has not been modified since) in the header for scripted pages. It can be easily done even for pages generated from sql databases. Robots will take you more seriosuly, as you save them time and bandwidth, because they will spider only if new content there.

Also do not use SESSIONS in URL !

BTW not long time ago when I dared to mention here on DNF, that WYSIWYG editors are for amateurs only. Some of the "wanna_be_a_pro" page creators here was very angry with me. Some of them told "I use Dreamweaver for 11 years, so I am an amateur or what" - so again - yes, you are an amateur for 11 years already ;)
 

Jacksplat

No time to chit-chat
Legacy Exclusive Member
Joined
Jul 25, 2005
Messages
1,785
Reaction score
2
Excellent info DCG, thanks.

I came across this thread searching for "sql" and although I didnt find what I was looking for, this was a valuable read for me as I'm just beginning to develop. You probably saved me alot of greif.

Back to my search.. If anyone knows how I can take my 387,000 lyrics from my sql db and make them into html pages, please, please point me in the right direction. I've been searching the net all night.


denny007 said:
BTW not long time ago when I dared to mention here on DNF, that WYSIWYG editors are for amateurs only. Some of the "wanna_be_a_pro" page creators here was very angry with me. Some of them told "I use Dreamweaver for 11 years, so I am an amateur or what" - so again - yes, you are an amateur for 11 years already ;)


I remember that denny. It got messy too.
 

denny007

Level 9
Legacy Exclusive Member
Joined
Oct 5, 2004
Messages
3,298
Reaction score
24
If anyone knows how I can take my 387,000 lyrics from my sql db and make them into html pages, please, please point me in the right direction.

Install it (preferably locally because of speed) with php generated pages. Take a mass downloader and let it download entire server. Mass downloders save all pagesas html with structure and correct linking.
 

Jacksplat

No time to chit-chat
Legacy Exclusive Member
Joined
Jul 25, 2005
Messages
1,785
Reaction score
2
denny007 said:
If anyone knows how I can take my 387,000 lyrics from my sql db and make them into html pages, please, please point me in the right direction.

Install it (preferably locally because of speed) with php generated pages. Take a mass downloader and let it download entire server. Mass downloders save all pagesas html with structure and correct linking.

Sweet... It's being done now. thanks Denny. Now I see why you suggested doing it locally. At close to 400,000 pages being generated, I should be done by the end of next week
 

denny007

Level 9
Legacy Exclusive Member
Joined
Oct 5, 2004
Messages
3,298
Reaction score
24
I should be done by the end of next week

Yeah and if you do not want to spend another week uploading it, upload tar.gz and untar-zip it on the server. Not only because of the huge size (if each page would be only 10KB, you are at 4GB, for 20KB per page 8GB uncompressed). But also ftp client would be communicating 400,000 times with the server. Plaint text should compress to less than 5% of original size.

But frankly - i do not understand why you are generating html...
 

prototypexx

New Member
Legacy Platinum Member
Joined
May 6, 2006
Messages
4
Reaction score
0
Wow, i will definately try some of these tips, seems pretty good!
 

MAllie

Level 8
Legacy Exclusive Member
Joined
Mar 2, 2008
Messages
1,838
Reaction score
38
The original post is very interesting, but I'm wondering if there's an update due. Is XHTML the way to go now? I've been reading that HTML has a lot of stuff that's being phased out. Will the crawlers have difficulty with these or will they still work?
 
Status
Not open for further replies.

The Rule #1

Do not insult any other member. Be polite and do business. Thank you!

Sedo - it.com Premiums

IT.com

Premium Members

AucDom
UKBackorder
Be a Squirrel
MariaBuy

Our Mods' Businesses

UrlPick.com
URL Shortener

*the exceptional businesses of our esteemed moderators

Top Bottom