Website crawler.

[prose]

Does anyone know of a program that will crawl and download webpages.

It can do a straight download of one domain when I command it or it could be more advanced and check for updates nightly.

It can be windows or linux, it just has to have the ability to log into web forums and download all content. (phpbb, vbulletin, ibp). This is the top priority.

Cost is not an issue.

 

[RenderStream1]

Send me a PM with your AIM/MSN

 

[Ranked Up]

Send me a PM with your AIM/MSN

"Lynx" in Linux.

Just type it in on the command line.
It's made me a fair few bob when combined with awk scripts.
Check the manpage.

Regards,

bonnag