Search Engine Crawler

[Scott.Mc]

I am selling my script.

It works like a search engine and will gather urls for you to run your own search engine. Crawler will run when ever you want it to run.

Price - $30
PM me or say in this thread.

 

[stoned]

Can you show me an example please?

 

[Scott.Mc]

There is not really an example because it crawls and inserts urls to the database what is to show.

 

[stevey]

what info does it gather, just the urls? or does it read metta tags etc aswell?

 

[Scott.Mc]

At the moment just urls which was my experiment.

Version 1.4 which will cost $35 includes page cache.

SMCrawler With Meta And Cache Support Now Available For $35 At
http://www.scottmcintyre.net/scripts/

 

[cwsteam.com]

can u show a demo please ?

 

[Scott.Mc]

Hello i would show a demo but there is nothing to show.

The script runs from the sites listed in the database and will build up the urls.

I am working on a script that will use that information if you are unable to show it yourself.

SMCrawler is only a crawler script which will gather urls to the specified settings in the config.

 

[stevey]

how many sites can it crawl a day? does it use alot of cpu/ram?

 

[Scott.Mc]

On my server it had 0.00
As load and i had it running every 30 minutes.

I crawled over 30,000 domains in 2 days.

If you give me 10 minutes i will recompile apache on one of my servers and i will make the example crawler output what site its crawling and then what urls it found.

-Scott