Enjoy unlimited access to all forum features for FREE! Optional upgrade available for extra perks.
Domain summit 2024

We need to monitor hundreds of thousands of domains per month - Looking for paid help

Status
Not open for further replies.

DomainEmpire.com

Level 9
Legacy Platinum Member
Joined
Aug 13, 2002
Messages
2,538
Reaction score
58
I see there're companies able to monitor the status of all domains in zone files (at least for com/net/org) and they're even able to save every few days the whole whois database, how are they doing that ?

Well, we need a solution for a smaller problem: we need a server side software/script or any other way to monitor the registration status of some hundreds of thousands of domains.
Basically, a fulltime automated monitoring solution, to get a daily list of all entered domains which changed their registration status in the latest 24hours.

We're obviously looking for a "paid help"; for more details and a more clear idea of our needs and expectations, email us at: domainsales (at) domainshowcase.com if you think to have the right solution in mind, thank you.
 
Domain summit 2024

amplify

Level 5
Legacy Exclusive Member
Joined
Sep 15, 2009
Messages
3,464
Reaction score
1,171

DomainEmpire.com

Level 9
Legacy Platinum Member
Joined
Aug 13, 2002
Messages
2,538
Reaction score
58
I've some concerns at this point, just curious to understand more:

1. Companies like domaintools.com offers an archive of past whois for all com/net/org domains, for each domain they save new records every few weeks, we're members there, we queries multiple domains and yes, they've multiple records for each of them, no exception. How can they do this ? Does this mean they've a way to check/save whois records for all names available in zone files ?
Yes, that's correct, they save complete whois records (including registrant details, exp dates, domain status and dns) ...

2. In the near past there was a spammer with the technology to check daily whois for all newly registered domains and to email to any registrant (independently by the registered domain) his link to invite them to check their website, to find better available domains related to the topic of the registered one so this's another example of "someone" able to get whois records of dozens of thousands of domains per day.

Briefly, if in a far future we should become a registrar, should we have the necessary tools to perform this kind of activity ?
I'm just trying to learn more about this topic, thanks for helping to clarify my doubts.
 

aclass

Level 5
Legacy Platinum Member
Joined
Jul 15, 2006
Messages
283
Reaction score
11
1. It might be done by using many many IPs so you don't get your IP blocked for too many whois requests. The programming part is easy. Just the IP part for such a big number of domains might get a bit pricey.

2. There are still such spammers as I still get an email a couple of days after I register a domain.
 

amplify

Level 5
Legacy Exclusive Member
Joined
Sep 15, 2009
Messages
3,464
Reaction score
1,171
IPs won't help when you get a GD domain, though won't flag you for spam, if you're using a human verified captcha service (a couple bucks per 1000).
 

katherine

Country hopper
Legacy Exclusive Member
Joined
Jul 9, 2005
Messages
8,428
Reaction score
1,290
1. Companies like domaintools.com offers an archive of past whois for all com/net/org domains, for each domain they save new records every few weeks, we're members there, we queries multiple domains and yes, they've multiple records for each of them, no exception. How can they do this ? Does this mean they've a way to check/save whois records for all names available in zone files ?
In the zone files, you will only get the list of domain names with associated name servers. Nothing more.
So just by comparing zone files on a daily basis for example, you can see which domain names have changed name servers. You can also see names added/removed (typically new registrations/dropping domains).

For .com/.net the full whois record (with registrant information) is maintained at the registrars.
For .org the information is available at registry level.

Scraping whois data en masse is difficult because access is capped so your IP address will be quickly banned after more than n queries per minutes. To get around this you typically need a lot of proxies.
How domaintools.com does that ?
Well the registrars often have mutual arrangements to whitelist designated IP addresses, and domaintools.com is a registrar.
 

DomainEmpire.com

Level 9
Legacy Platinum Member
Joined
Aug 13, 2002
Messages
2,538
Reaction score
58
Thanks for these detailed info, I'm continuing to learn.

Well, basically what we really need is much simple to do I think.
We need a way to monitor on a daily basis "domain status & dns" of a list of about 400'000 domains (mainly com/net, around 20% of .org).

Briefly, we'd use a remote server and we'd pay for the necessary resources, we'd need a "stable software" which should automatically perform this daily task and save in a txt file the list of all domains which added any of the "given keywords" (entered by us, of course) in the just changed dns or status.

For example, all domains where the keyword "renew" has been added in their dns or the keyword "hold" in their status.
We're collecting few quotes, hoping to get some professional help in realizing this project.

Feel free to PM or email if you think to be able to help, thanks.
 
Status
Not open for further replies.

The Rule #1

Do not insult any other member. Be polite and do business. Thank you!

Sedo - it.com Premiums

IT.com

Premium Members

AucDom
UKBackorder
Be a Squirrel
MariaBuy

New Threads

Our Mods' Businesses

UrlPick.com
Free QR Code Generator by MerchArts

*the exceptional businesses of our esteemed moderators

Top Bottom