.com Zone File Query - http://www.ZFbot.com

**kengreenwood** · 06-27-2009, 09:07 PM

--- Moved the sort into the initial file clean up on the Unix side of things. Had to split the zone file into 5 different files of around 20 million records a piece 'cause I was exceeding disk quota in my tmp directory due to the sort... then I merge them all back together, resorting the entire file and bulkload it into the database. Thus removing the need for the ORDER BY in the query code. That was the culprit for the lengthy queries... Even crazy wide open queries are coming back in a reasonable amount of time... So I bumped up the max return records to 250,000. That's mainly due to the limitation of the Flex Datagrid.

**Dale Hubbard** · 06-28-2009, 03:56 AM

Ken, it may be worth getting a 60Gb drive attached. You can then tell the unix sort to use that drive for the temporary files created during the sort routine. The extra drive is a better solution than a 'bigger' drive if you ever decide to upgrade.

**kengreenwood** · 06-28-2009, 10:59 AM

Originally Posted by aZooZa

Ken, it may be worth getting a 60Gb drive attached. You can then tell the unix sort to use that drive for the temporary files created during the sort routine. The extra drive is a better solution than a 'bigger' drive if you ever decide to upgrade.

thanks for the tip Dale. i should have moved the sort to the Unix side of things a long time ago. Unix does it much more efficiently and it only needs to be done once daily as the data is loaded instead of on each query request.

**kengreenwood** · 06-29-2009, 07:49 PM

NEW: ZFBot MINI ... for those of you who don't want or need the fluff that exists on ZFBot.com (webmaster radio, charts, dnjournal rss feed, etc). Click on the link below the ZFBot.com application that says "NEW: ZFBot MINI". Make sure you have pop ups enabled... it's a down and dirty search. Has the same filters and export functionality.

http://www.zfbot.com

Added search for dictionary word domains that are dropping... I'm comparing the dropping domains to a dictionary file of just over 300,000 words.

Here's an example of a search for dictionary .com's starting with the letter T that less than or equal to 13 letters in length and are dropping...

http://www.zfbot.com/#/search=t;drop...2;ext=1;part=0

**kengreenwood** · 07-02-2009, 10:54 AM

For easy access to top dropping domains (from your blackberry or IPHONE for example), I created the following links... no need for flash player:

http://www.zfbot.com/dd.php (all top dropping domains by category... dictionary, cvcv, etc)

http://www.zfbot.com/dd_com.php (same as dd.php but just .com)
http://www.zfbot.com/dd_net.php (same as dd.php but just .net)
http://www.zfbot.com/dd_org.php (same as dd.php but just .org)

I'm going to build a lite version of the entire zfbot app that can be accessed from mobile devices without flash as well...

**Dale Hubbard** · 07-02-2009, 10:56 AM

Great to see this evolve Ken and kudos to you for all the hard work.

**kengreenwood** · 07-02-2009, 09:18 PM

Thanks Dale. Another nice bit of info:

I just sent requests in for access to the following zone files:
.info
.biz
.us
.mobi
.name

Assuming I get approval, which shouldn't be an issue, I will be adding those domains to zfbot shortly...

**Acro** · 07-02-2009, 09:52 PM

"Awesome" is an understatement

Love the new features.

**kengreenwood** · 07-02-2009, 10:40 PM

Originally Posted by Acro

"Awesome" is an understatement

Love the new features.

Welcome back Theo... there has been much added to ZFBot... and you have a personal "test" account! Have you tried it?

**Dale Hubbard** · 07-03-2009, 01:41 AM

Ken, there are no problems with those zones. I have access to them. Not sure about .name though... But you can build the bulk of any zone if you know how

PM me if any problems.

**copper** · 07-03-2009, 08:51 AM

I see big speed improvements and I like "ZFBot MINI".
This is turning into monster script

**kengreenwood** · 07-03-2009, 11:49 AM

Originally Posted by copper

I see big speed improvements and I like "ZFBot MINI".
This is turning into monster script

Thanks - yeah, once I moved the sort out of the queries and in to the unix cleanup/load, I was able to pull the ORDER BY out of the queries. That really sped things up significantly.

Stay tuned for ZFBot Mobile (non-flash based)!

UPDATE - .us domains are now included in ZFBot! For today, the .us domains will appear at the end of any search... starting tomorrow, they'll be sorted within the rest of the domains.

Oooops...spoke too soon... looks like i have some invalid .us domains in there...

Fixed... there were some namerserver records that looked like domains... had to get rid of them.

**kengreenwood** · 07-07-2009, 11:06 AM

.biz domains are now included in zfbot

**kengreenwood** · 07-09-2009, 12:26 PM

Check this out... I tested it out on my Blackberry and an IPhone. Still a work in progress but it functions pretty much like the main ZFBot application. No flash on this version and it's a really cool and quick way to see what's reg'd and what's dropping... I haven't yet added paging so be careful of what you request - it could result in a very large dataset being sent back to your phone. I'll add the paging, sending results to email, whois script, etc...

Access: http://www.zfbot.com/mobile from your mobile device.

**Dale Hubbard** · 07-09-2009, 01:54 PM

You never cease to amaze Ken. Great work.

**kengreenwood** · 07-09-2009, 05:19 PM

Originally Posted by aZooZa

You never cease to amaze Ken. Great work.

Thanks Dale - I'm sure the formatting is going to need to be tweaked for different devices... I just noticed something odd going on with how my 8703 BB is rendering the site... looks like the IPhone version works well though...

Fixed the issue on my 8703... if anyone tests this out on other devices and wants to let me know of issues, I'd greatly appreciate it.

.MOBI now included in ZFBOT

**kengreenwood** · 07-29-2009, 10:08 PM

I need some help -

I have recently (still in test) added whether a domain is parked or not in the search results and within the filter. But I'm filtering on the basic nameservers that the top parking sites use ... there are A LOT of other types of "parking" ...ie GoDaddy... and after racking my brain, it's very difficult to determine if there is a clear definition of what defines a "parked" domain. not sure if there is one definition. Anyone have any experience identifying a parked domain, please contact me...

**Albert Tai** · 07-29-2009, 10:19 PM

Originally Posted by kengreenwood

I need some help -

I have recently (still in test) added whether a domain is parked or not in the search results and within the filter. But I'm filtering on the basic nameservers that the top parking sites use ... there are A LOT of other types of "parking" ...ie GoDaddy... and after racking my brain, it's very difficult to determine if there is a clear definition of what defines a "parked" domain. not sure if there is one definition. Anyone have any experience identifying a parked domain, please contact me...

Well I think the nameserver is the best approach.
As for adsense is not possible to have one a page with no content (like a parked page) so I think the best is to filter everything out with nameservers of all the parking companies:
parked.com, sedo.com, etc.

**jmcc** · 07-29-2009, 10:22 PM

Originally Posted by kengreenwood

I need some help -

I have recently (still in test) added whether a domain is parked or not in the search results and within the filter. But I'm filtering on the basic nameservers that the top parking sites use ... there are A LOT of other types of "parking" ...ie GoDaddy... and after racking my brain, it's very difficult to determine if there is a clear definition of what defines a "parked" domain. not sure if there is one definition. Anyone have any experience identifying a parked domain, please contact me...

It is a fairly complex task as it involves a lot of regexping of the raw HTML of the webpage. Grouping by parking company nameserver is the easy way but the evolution of parking and the reaction of search engines to the easy kill of penalising domains on known PPC nameservers forced the PPC companies into using javascript and IFRAMEs to load their adverts. The domains remain on the users' own nameservers and hence escape simple DNS based filitering. The danger is that too wide a regexp would automatically flag any website with Adsense as being parked. Then there are the non-English language parked sites. This is where it stops being about simple domain and DNS problems and enters the realm of search engine algorithm development. The rule of thumb that I use in TLD surveys is if there is no genuine content on the page other than adverts. However that does involve a multipass process to analyse each page and it is done offline rather than live.

Regards...jmcc

**south** · 07-29-2009, 10:22 PM

Originally Posted by kengreenwood

I need some help -

I have recently (still in test) added whether a domain is parked or not in the search results and within the filter. But I'm filtering on the basic nameservers that the top parking sites use ... there are A LOT of other types of "parking" ...ie GoDaddy... and after racking my brain, it's very difficult to determine if there is a clear definition of what defines a "parked" domain. not sure if there is one definition. Anyone have any experience identifying a parked domain, please contact me...

I would think the only way to tell would be by known nameservers. There would be no way to tell by content, as there are plenty of crappy sites out there that are worse than even parked names.

Another thought would be to use ivegotafang dot com.