- Joined
- Mar 29, 2014
- Messages
- 2,619
- Reaction score
- 1,014
I was excited by the comparisons in Biraj's SLD Checker update thread.
This morning, I also wanted to check whether our tool really provides better results for its core task: finding the same domain keyword across different extensions.
And, a selfish one > wanted to brag about it a little on our newsletter
So, I asked my Hermes AI assistant to run a fresh independent comparison work, preserve the results and investigate discrepancies. The instruction was to test both tools fairly, not to make DNForum win.
For transparency: I own DNForum, and this test was commissioned by me. It is an AI-assisted comparison with supporting evidence, not an independent third-party certification.
The findings include genuine DNForum wins, results where dotDB leads, and issues in our own results that deserve attention. I am publishing all of them.
What was completed
The test used a fixed sample of 20 keywords. All 20 were checked on DNForum, but only 10 valid paired comparisons were completed before dotDB's daily search limit prevented further results.
Attempts to resume did not remove the limit in the test browser session. The remaining comparisons are therefore incomplete.
The limited dotDB pages displayed dummy rows and zero counts. These were excluded, not treated as genuine zero-result searches. This access limitation is not evidence that DNForum has better data.
How the comparison worked
The comparison focused on the exact keyword row across all extensions and all domain statuses.
For example, the search for "cloud" compared cloud.com, cloud.net, cloud.co.uk and other exact-name results. It did not compare the total number of domains containing "cloud" somewhere in a longer name.
A few important details:
The 10 completed comparisons
DNForum's counts changed between the initial searches and later repeat searches. Both observations are included below.
On the repeat observations:
These are reported coverage counts, not independently validated registration totals.
The changing counts are important
On the initial observations, DNForum led on only 2 of the 10 paired searches. On the repeat observations, it led on 6.
Some changes were substantial:
This behaviour is consistent with the live fallback and enrichment described in the development thread, but this test did not establish the precise cause.
The repeat figures are a second snapshot. They are not proof that every background check had finished.
For a stronger comparison, we need to distinguish initial stored results from later enriched results and make it clear when a search is still being updated.
Genuine DNForum wins: registered domains that dotDB missed
DNForum returned helmuts.co.uk and helmuts.online. Neither appeared in dotDB's exact "helmuts" row.
Registry checks supported both results:
Both names returned NXDOMAIN in the DNS A-record checks performed during the test.
That is a useful reminder: a domain can be registered without resolving in DNS. An unsuccessful DNS lookup does not mean a domain is available to register.
My earlier "helmuts" example therefore has substance beyond a higher displayed number.
An accuracy issue in our results: liene.fr
DNForum returned liene.fr, but AFNIC's authoritative RDAP returned HTTP 404 with this message:
That is evidence of a stale or false-positive result at the time of testing. It deserves investigation.
I do not want us celebrating additional results while ignoring whether those results are correct.
A wildcard-DNS warning: helmuts.org.de
DNForum also included helmuts.org.de.
It resolved to 64.190.63.222. However, a freshly generated random control name, dnf-audit-b0611553912b40968b.org.de, resolved to exactly the same address.
This indicates wildcard DNS under .org.de.
It means that DNS resolution alone cannot establish that helmuts.org.de is a separately registered name. It does not, by itself, prove that the particular name has no separately provisioned record.
This is another area where our discovery and validation logic needs careful checking.
Similar totals can hide very different results
Across the ten completed comparisons, the extension lists contained:
For "cloud", the overall count differed by just eight, yet there were 85 DNForum-only results and 77 dotDB-only results.
For "liene", both tools returned 21 extensions on the repeat comparison, but each had three extensions missing from the other.
A matching total does not mean matching coverage. A slightly higher total does not mean one tool contains everything the other tool found.
The remaining ten keywords
These DNForum observations are included for completeness. They are not completed head-to-head comparisons.
What this test does not establish
This is a small, partial comparison. It does not establish:
The DNS discrepancy checks were diagnostic spot checks, not a representative accuracy measurement. DNS success alone does not prove registration, NXDOMAIN does not prove availability, and SERVFAIL is inconclusive.
My conclusion
I am pleased to see DNForum competing this closely and finding additional registered domains. Biraj deserves credit for the work behind that.
However, I would not use this test to claim that we have generally beaten dotDB.
The conclusion I am comfortable publishing is:
The next useful work is to investigate liene.fr, check wildcard handling and registration-suffix boundaries, make the enrichment process clearer to users, and complete the remaining paired searches with sufficient dotDB access.
I would rather publish a result that helps us improve than a flattering comparison that members cannot trust.
Attached files
I am including all three files so members can inspect the findings:
The files preserve the observations from the test. Live search, DNS and RDAP results can change after publication.
Reference sources
Original development update:
https://www.dnforum.com/threads/update-on-sld-checker.630679/page-4#post-2434341
DNForum SLD Checker:
https://www.dnforum.com/sld
DNForum exact "helmuts" search:
https://www.dnforum.com/sld?keyword=helmuts
dotDB exact "helmuts" row within its search results:
https://dotdb.com/search?keyword=helmuts&position=any
Nominet RDAP, helmuts.co.uk:
https://rdap.nominet.uk/uk/domain/helmuts.co.uk
Radix RDAP, helmuts.online:
https://rdap.radix.host/rdap/domain/helmuts.online
AFNIC RDAP, liene.fr:
https://rdap.nic.fr/domain/liene.fr
Google DNS, helmuts.org.de:
https://dns.google/resolve?name=helmuts.org.de&type=A
Google DNS, random .org.de control:
https://dns.google/resolve?name=dnf-audit-b0611553912b40968b.org.de&type=A
Happy Tuesday!!
Helmuts
Your thoughts?
This morning, I also wanted to check whether our tool really provides better results for its core task: finding the same domain keyword across different extensions.
And, a selfish one > wanted to brag about it a little on our newsletter
So, I asked my Hermes AI assistant to run a fresh independent comparison work, preserve the results and investigate discrepancies. The instruction was to test both tools fairly, not to make DNForum win.
For transparency: I own DNForum, and this test was commissioned by me. It is an AI-assisted comparison with supporting evidence, not an independent third-party certification.
The findings include genuine DNForum wins, results where dotDB leads, and issues in our own results that deserve attention. I am publishing all of them.
What was completed
The test used a fixed sample of 20 keywords. All 20 were checked on DNForum, but only 10 valid paired comparisons were completed before dotDB's daily search limit prevented further results.
Attempts to resume did not remove the limit in the test browser session. The remaining comparisons are therefore incomplete.
The limited dotDB pages displayed dummy rows and zero counts. These were excluded, not treated as genuine zero-result searches. This access limitation is not evidence that DNForum has better data.
How the comparison worked
The comparison focused on the exact keyword row across all extensions and all domain statuses.
For example, the search for "cloud" compared cloud.com, cloud.net, cloud.co.uk and other exact-name results. It did not compare the total number of domains containing "cloud" somewhere in a longer name.
A few important details:
- The sample was deliberately varied, not random or statistically representative.
- "helmuts", "liene" and "api" reproduced earlier examples from the update thread. The other 17 keywords broadened the test.
- Extension counts include registration suffixes such as .co.uk, not only top-level domains.
- DNForum results were collected from its public server-rendered /sld?keyword= pages. The "cloud" result was also checked in the rendered browser interface.
- dotDB results were collected from its live Chrome-rendered search pages.
- The extracted extension lists were deduplicated and checked against the displayed counts.
- This was not a speed test or a fully equivalent test of both interactive search workflows.
The 10 completed comparisons
DNForum's counts changed between the initial searches and later repeat searches. Both observations are included below.
| Keyword | DNForum initial | DNForum repeat | dotDB |
|---|---|---|---|
| helmuts | 12 | 12 | 8 |
| liene | 19 | 21 | 21 |
| api | 843 | 844 | 791 |
| cloud | 706 | 814 | 806 |
| hosting | 259 | 530 | 535 |
| domain | 307 | 570 | 566 |
| bitcoin | 500 | 825 | 830 |
| wallet | 538 | 557 | 553 |
| solar | 447 | 515 | 507 |
| coffee | 439 | 477 | 481 |
On the repeat observations:
- DNForum returned more extensions for 6 keywords.
- dotDB returned more for 3 keywords.
- 1 keyword was tied.
- The combined counts were 5,165 for DNForum and 5,098 for dotDB.
- DNForum's net count advantage was 67, approximately 1.31%.
These are reported coverage counts, not independently validated registration totals.
The changing counts are important
On the initial observations, DNForum led on only 2 of the 10 paired searches. On the repeat observations, it led on 6.
Some changes were substantial:
- hosting: 259 to 530
- domain: 307 to 570
- bitcoin: 500 to 825
- cloud: 706 to 814
This behaviour is consistent with the live fallback and enrichment described in the development thread, but this test did not establish the precise cause.
The repeat figures are a second snapshot. They are not proof that every background check had finished.
For a stronger comparison, we need to distinguish initial stored results from later enriched results and make it clear when a search is still being updated.
Genuine DNForum wins: registered domains that dotDB missed
DNForum returned helmuts.co.uk and helmuts.online. Neither appeared in dotDB's exact "helmuts" row.
Registry checks supported both results:
- Nominet RDAP returned a domain record for helmuts.co.uk.
- Radix RDAP returned a domain record for helmuts.online.
Both names returned NXDOMAIN in the DNS A-record checks performed during the test.
That is a useful reminder: a domain can be registered without resolving in DNS. An unsuccessful DNS lookup does not mean a domain is available to register.
My earlier "helmuts" example therefore has substance beyond a higher displayed number.
An accuracy issue in our results: liene.fr
DNForum returned liene.fr, but AFNIC's authoritative RDAP returned HTTP 404 with this message:
That is evidence of a stale or false-positive result at the time of testing. It deserves investigation.
I do not want us celebrating additional results while ignoring whether those results are correct.
A wildcard-DNS warning: helmuts.org.de
DNForum also included helmuts.org.de.
It resolved to 64.190.63.222. However, a freshly generated random control name, dnf-audit-b0611553912b40968b.org.de, resolved to exactly the same address.
This indicates wildcard DNS under .org.de.
It means that DNS resolution alone cannot establish that helmuts.org.de is a separately registered name. It does not, by itself, prove that the particular name has no separately provisioned record.
This is another area where our discovery and validation logic needs careful checking.
Similar totals can hide very different results
Across the ten completed comparisons, the extension lists contained:
- 492 names found only in DNForum.
- 425 names found only in dotDB.
For "cloud", the overall count differed by just eight, yet there were 85 DNForum-only results and 77 dotDB-only results.
For "liene", both tools returned 21 extensions on the repeat comparison, but each had three extensions missing from the other.
A matching total does not mean matching coverage. A slightly higher total does not mean one tool contains everything the other tool found.
The remaining ten keywords
These DNForum observations are included for completeness. They are not completed head-to-head comparisons.
| Keyword | DNForum count | dotDB test status |
|---|---|---|
| garden | 350 | Daily limit |
| insurance | 499 | Daily limit |
| riga | 144 | Daily limit |
| kempten | 42 | Daily limit |
| nairobi | 102 | Daily limit |
| bluebird | 197 | Not submitted after persistent limit |
| northstar | 263 | Not submitted after persistent limit |
| cleverfox | 31 | Not submitted after persistent limit |
| pixelnest | 49 | Not submitted after persistent limit |
| green-harbor | 3 | Not submitted after persistent limit |
What this test does not establish
This is a small, partial comparison. It does not establish:
- That DNForum is more accurate overall.
- That either tool has complete registration coverage.
- That DNForum's related-keyword results are better.
- That one tool is faster.
- That Active, Parked and Inactive classifications are correct.
- How the tools compare across their paid features.
- How much either tool has improved historically.
The DNS discrepancy checks were diagnostic spot checks, not a representative accuracy measurement. DNS success alone does not prove registration, NXDOMAIN does not prove availability, and SERVFAIL is inconclusive.
My conclusion
I am pleased to see DNForum competing this closely and finding additional registered domains. Biraj deserves credit for the work behind that.
However, I would not use this test to claim that we have generally beaten dotDB.
The conclusion I am comfortable publishing is:
The next useful work is to investigate liene.fr, check wildcard handling and registration-suffix boundaries, make the enrichment process clearer to users, and complete the remaining paired searches with sufficient dotDB access.
I would rather publish a result that helps us improve than a flattering comparison that members cannot trust.
Attached files
I am including all three files so members can inspect the findings:
- comparison.csv: the 20-keyword comparison, including missing dotDB results.
- report.md: the fuller methodology, findings and limitations.
- benchmark-evidence.zip: saved result data, extension differences, DNS checks, RDAP responses and supporting files.
The files preserve the observations from the test. Live search, DNS and RDAP results can change after publication.
Reference sources
Original development update:
https://www.dnforum.com/threads/update-on-sld-checker.630679/page-4#post-2434341
DNForum SLD Checker:
https://www.dnforum.com/sld
DNForum exact "helmuts" search:
https://www.dnforum.com/sld?keyword=helmuts
dotDB exact "helmuts" row within its search results:
https://dotdb.com/search?keyword=helmuts&position=any
Nominet RDAP, helmuts.co.uk:
https://rdap.nominet.uk/uk/domain/helmuts.co.uk
Radix RDAP, helmuts.online:
https://rdap.radix.host/rdap/domain/helmuts.online
AFNIC RDAP, liene.fr:
https://rdap.nic.fr/domain/liene.fr
Google DNS, helmuts.org.de:
https://dns.google/resolve?name=helmuts.org.de&type=A
Google DNS, random .org.de control:
https://dns.google/resolve?name=dnf-audit-b0611553912b40968b.org.de&type=A
Happy Tuesday!!
Helmuts
Your thoughts?
Full domain searches should work across extensions properly.
