Mark your calendar, on 11 October 2018, your resolvers will (should) do DNSSEC validation using the new KSK-2017 instead of the KSK-2010, otherwise you or your customers might run into some trouble. Exactly a year before, on 11 October 2017, the same Root Zone Key Signing Key (KSK) roll was scheduled, but it was suspended a few weeks before the set date - as it was found that there was still a considerable amount of validating resolvers without the latest root key. After extensive community review, the new flag day has been set for 11 October 2018.
As mentioned in our previous blog post, it is important for DNS Resolver operators to make sure that they have the latest KSK by either manually updating the configurations of the resolver software with the new key or automatically through RFC 5011, which instructs resolvers to download the latest trust anchor.
Why was the initial rollover suspended?
In order to understand how many resolvers currently have the new trust anchor configured, RFC 8145 (Signaling Trust Anchor Knowledge in DNS Security Extensions) was introduced. This protocol allows validating resolvers to signal their trust anchor material to an authoritative server. This feature has allowed root server operators to gather interesting insights about the prevalence of the new KSK in the wild. Even if there are no deterministic ways to get the status of the whole population of resolvers on the Internet, ICANN claims that data gathered through RFC 8145 is the closest we could get - however as explained below...with a few caveats. The initial dataset as of September 2017, showed that about 4% of the approximately 12,000 DNSSEC-validating resolvers were configured with only the KSK-2010. Even if this number seems to be relatively small, ICANN was not convinced why a certain number of resolvers did not have the new KSK configured, hence the decision to postpone the key rollover.
Some statistics
ICANN publishes a daily snapshot of the RFC 8145 dataset by gathering data from 11 root servers they have been given access to. The data is obtained from resolvers reporting their trust anchor keytags to the root servers, through RFC 8145.
Figure 1. RFC 8145 Trust anchor reports from all root servers (Source: ICANN)
The graph in figure 1, shows the trends since 01 September 2017. We can see a constant increase in the number of resolvers reporting trust anchor data (GREEN LINE). The RED LINE corresponds to the number of sources only reporting KSK-2010, in which, we do see a slight increase from January 2018 till March 2018 (from zero to ~25,000). This could be explained by an increasing number of resolvers reporting their keys as opposed to before the release of RFC 8145 in April 2017. The BOLD LINE represents the percentage value of the number of sources reporting KSK-2010 only.
The spikes in the plots can be attributed to multiple events for e.g. release of a resolver software upgrade, deployment and instant removal of dynamic/temporary (sometimes VM-based) resolvers. In January 2018, there was a release of Unbound 1.6.8 which also contained a “fix for CVE-2017-15105: vulnerability in the processing of wildcard synthesized NSEC records”. It is supposed that the upgrade also contained RFC 5011 and RFC 8145 code.
Insights into DNS Resolvers in Africa
We took a slice of the RFC 8145 dataset by selecting only AFRINIC allocated ASN by merging it to the AFRINIC’s delegated file. We found 102 ASN (6.1%) in 28 countries in Africa that are operating at least a resolver reporting only KSK-2010. The total number of unique IP addresses reported is 1,776, including 7 IPv6 addresses, corresponding to DNS resolvers in those networks.
Figure 2. Number of ASNs per country operating RFC 8145-enabled DNS Resolvers
As seen in figure 2, South Africa has the most number of ASNs reporting trust anchor details to root servers, followed by Nigeria, Ghana, Kenya, Egypt and Sudan. With regards to the number of unique resolvers identified, Nigeria has 810, Ghana has 311 and Morocco has 143. This follows more or less the distribution of IP addresses in Africa.
Figure 3. Number of DNS resolvers reporting their trust anchors by country
Out of those 1,776 unique IP addresses report their trust anchor configurations, 1,628 (92%) are only reporting the KSK-2010 and 148 (8%) report both KSK-2010 and KSK-2017.
However, resolvers are very volatile, meaning that they can come and go in short spans of time. The reasons are multiple, for e.g. resolvers can be software-based i.e. embedded in a software component which means they are not connected all the time. To understand which IPs are consistently sending their trust anchor signals, ICANN observed the address over a span of 28 days. Figure 4 shows the distribution of the number of times a unique host has been seen online. Most of them are short-lived (95.2%), with the remaining 2.87% seen in last 10-20 days and 1.91% seen in the last 21-28 days. The CDF shows that 80% of resolvers have been seen in the last 1 or 2 days only, confirming the phenomenon of the volatility of DNS resolvers on the Internet.
|
Days |
|
Figure 4. Seen in the last 28 days
ASNs most likely to be affected
APNIC provides country and ASN-level data on DNSSEC validation through their Ad-based measurement platform. The data is classified by ASNs in the country of operation. One ASN can effectively appear in multiple locations, the same amount of samples collected through the Ad-based system will differ country by country. By merging this list with the RFC 8145 dataset from ICANN with APNIC DNSSEC Validation dataset, we sort out the list of the ‘most important’ resolver by taking into consideration (i) % of DNSSEC validation, (ii) number of samples collected by the measurement platform. Below is a list of ASNs with their netname and countries that appeared more than 10 times in the last 28 days and have resolvers that have reported KSK-2010 only.
Table 1. List of ASNs that are most likely to be affected by the KSK roll
|
ASN |
Country |
Netname |
DNSSEC Validates |
Samples |
|
37473 |
SO |
TELESOM |
0.9268 |
37174 |
|
37315 |
ZA |
CipherWave |
0.799 |
587 |
|
37153 |
ZA |
HETZNER |
0.4125 |
160 |
|
37473 |
DJ |
TELESOM |
0.3247 |
77 |
|
36937 |
ZA |
Neotel-AS |
0.2886 |
7357 |
|
36874 |
ZA |
Cybersmart |
0.205 |
2317 |
|
36994 |
ZA |
Vodacom-VB |
0.1384 |
3641 |
|
33788 |
SD |
KANARTEL |
0.1343 |
6285 |
|
37123 |
ZW |
TELCO- |
0.1283 |
1598 |
|
37680 |
ZA |
COOL-IDEAS |
0.0853 |
1969 |
|
36907 |
AO |
TVCaboAngola |
0.0744 |
25509 |
|
37054 |
MG |
TGN |
0.0368 |
30083 |
|
327712 |
DZ |
ATM |
0.0323 |
748481 |
|
37341 |
GH |
GLOMOBILE |
0.0179 |
11566 |
|
37148 |
NG |
globacom-as |
0.0115 |
126620 |
Caveats
We need to handle the RFC 8145 data with a pinch of salt. Further research has shown that the signals obtained from RFC 8145, are biased by multiple factors and therefore it might not be 100% representative of the actual situation. For e.g. it may happen that a set of DNS resolvers are configured to talk to a single DNS forwarder, which in turn takes care of querying the authoritative servers. RFC 8145 does not distinguish whether a query is coming from a DNS Forwarder, potentially masking the behaviours that end-users would exhibit. Notwithstanding the fact that queries and responses from DNS resolvers to the forwarders can also be cached.
Another issue noticed is that some non-validating resolvers are also sending RFC 8145 signals, while they are not supposed to. They are therefore sending false signals which are being added to the dataset.
Finally, the complexity of the DNS resolver ecosystem makes it difficult to have a complete view from the outside, mostly because the DNS was not designed with telemetry principles in mind. RFC 8145 does not allow to distinguish the population of users to the population of resolvers and evaluating the direct impact on end-users in a bit tricky. APNIC provides a slightly better view by merging the RFC 8145 dataset with their Ad-based DNSSEC measurements and add a “weight by use” metric. In the same vein, Geoff Huston et al. recently proposed “A Sentinel for Detecting Trusted Keys in DNSSEC” which offers an alternative and more accurate solution to understand the deployment of root keys in DNS resolvers.
APNIC concludes[1] that ultimately it’s likely that the impact on the Internet population would be 0.05%, who will be without DNS resolution in the event of a KSK roll to KSK-2017.
Recommendations
If you or one of your clients are running a DNS resolver, make sure you are aware of whether your DNS resolvers are performing DNSSEC validation either over IPv4 or IPv6[2]. Verify which DNS software you are operating and make sure you have also surveyed all embedded DNS resolvers that are sometimes hidden and operating in the background. Normally, newer versions are of your resolver software are recommended as they would contain the latest standard most importantly RFC 5011, which allow automatic update of the trusted key. In some cases where you are dealing with legacy configurations files (over newly updated software) make sure your configuration files have the required options to enable automatic download of the new trusted key.
Before, during and after the KSK roll, you should keep a close eye on your DNS monitoring platform. Check for DNSSEC validation failures in the DNS log, this may point you to different failure cases such as “too large DNS messages”, “wrong trust anchor”, etc.
If your resolvers have automatic updates enabled, check your configuration file to see if the KSK-2017 has been downloaded, if not, then you should retrieve the KSK manually.
How to get the new KSK?
Automatically:
Enable Automatic Updates (RFC 5011) to trust KSK-2017. Your resolver will get updated with the KSK-2017 key at the appropriate time.
Manually:
Download the trust anchor XML file from:
- IANA: https:// data.iana.org/root-anchors/root-anchors.xml
- DNS: Query a root server for “./IN/DNSKEY”
- Other means: for e.g. using https://github.com/kirei/get_trust_anchor
Resources
ICANN Draft plan for KSK Roll
https://www.icann.org/news/blog/announcing-draft-plan-for-continuing-with-the-ksk-roll
Checking the Current Trust Anchors in DNS Validating Resolvers
https://www.icann.org/dns-resolvers-checking-current-trust-anchors
RFC 8145: Signaling Trust Anchor Knowledge in DNS Security Extensions (DNSSEC)
https://tools.ietf.org/html/rfc8145
Measuring Root Zone KSK Trust
http://www.potaroo.net/ispcol/2018-04/ksk.pdf
RFC 8145 Root Trust Anchor Reports
