Need Good tool for Zone queries

[Adam DickerAdam Dicker is verified member.]

Example, I want to download the whole .com zone file and then easily search for all registered names starting with the keyword florida?

PM me info if you have any, will be much appreaciated.

-=DCG=-

 

[Theo]

Adam, given the size of the zone files I suggest that you ask a programmer to build a web interface on top of a linux box. Much faster than anything under Windows.

 

[kengreenwood]

FYI - I just loaded the Verisign zone file for all of the .net domains and that alone was nearly 28 million records. The .com file is about 7 times the size of the .net file - so it probably has around 210 million records in it...

 

[Dale Hubbard]

Adam, this is my speciality. PM me of you have no solution, or ask Luc to do it. He has downloadable zone files with the NS crap removed. It's a one-liner in Linux.

 

[Theo]

Just Grep it (tm)

 

[Dale Hubbard]

Grep what? Post your Linux one-line parsers hee

 

[Theo]

Off the top of my head:

cat zonefile | [FONT=Verdana, Arial, Helvetica, sans-serif][SIZE=-1]grep '\<florida' > florida-results.txt[/SIZE][/FONT]

 

[Dale Hubbard]

No need to 'cat' Just the 'grep'

 

[Theo]

Let's see how Windows would do the same in a DOS prompt

 

[audubon]

Let's see how Windows would do the same in a DOS prompt

findstr "^florida" zonefile > florida-results.txt

 

[Dale Hubbard]

I can do it in Windows too

 

[kengreenwood]

Why not load it in to a database ? I loaded over 200 million .com's into a table and I'll be able to build a php script that can rip through finding matches on whatever you could possible want using REGEXP. That way, you can run queries over the web easily.

 

[Dale Hubbard]

Win? http://www.freepascal.org/docs-html/rtl/sysutils/leftstr.html

 

[Theo]

findstr "^florida" zonefile > florida-results.txt

Can DOS process files a few gigabytes in size?

 

[south]

Can DOS process files a few gigabytes in size?

The filesystem wouldn't support a file over 2 gb if I remember correctly. DOS itself shouldn't care. The language also might have issues.

 

[kengreenwood]

The best way to find strings in a 6 gigabyte + file is to load it into a database table and to use SQL to find what you need. By indexing the domain field you can retrieve hits on a string in a couple seconds.

 

[audubon]

Can DOS process files a few gigabytes in size?

NTFS supports file sizes of around 16 Terabytes. On computers running Windows NT-based operating systems (2000, XP, Vista), we don't have DOS per se, but instead there is a DOS interpreter (cmd.exe), which is a native program. I don't recall the file-size limitations of MS-DOS, but I have used the DOS-like commands of the command-line interpreter to process .com zone files on computers running Windows 2000 and XP.

Another option if you're running Windows is to install ActivePerl.

 

[Dale Hubbard]

Yes, DOS can process multi Gb sequentially. Any length.

In >process
Out > processed

No limit.

 

[kengreenwood]

Ok - I loaded 180 million of the .com zone file records into a database table on my server and even without an index (which will take some time to build with that amount of data), it only took about 10 seconds to find the 112,744 records that start with "FLORIDA". I wasn't expecting that many!

- correction: I needed to do a DISTINCT due to the fact that domains have multiple name servers... the actual count is 49,796 and it took about 30 seconds...need to get that index on there.

 

[Theo]

Good stuff