Monitoring your DANE deployment Thumbnail
Deploy360 18 December 2017

Monitoring your DANE deployment

By Jan ŽoržFormer Operational Engagement Programme Manager

In a previous series of articles (part 1, part 2) we described how to install and use DANE for verifying your email and web server certificates through the DNS. In this article, I discuss how to monitor whether your TLSA records still match the certificates used for your services.

The aim is help people doing something similar, as this monitoring system saved the DANE deployment in the Go6lab from being broken many times, especially at the very beginning of deployment when the automated systems we built didn’t work completely correctly all the time 🙂

If we’re using self-signed certificates for our services and we manually change the underlying key, then we’ll probably also change the values of the TLSA records. If we forget to change them or are using, for example, Let’s Encrypt that automatically renews the certificate every 90 days, then we need to have some automatic DANE monitoring system that informs us of misalignment between the certificate being used and the TLSA record.

For this reason there are more and more online tools to check individual services by entering the URL and port, and with a press of a button tell if things are still working fine. A few examples of such tools include:

These tools allow you to quickly check your services, but you need to manually enter their details and check them one-by-one. In the Go6lab we use TLSA records to secure many services, and since we’re using Let’e Encrypt certificates that are automatically renewed, we had to build a tool that checks for DANE validity and even dynamically change the TLSA records in our DNS to match the new certificate.

Using the default mode of certbot, one of the Let’s Encrypt applications to fetch and renew certificates, the underlying key of certificate is also changed whenever we renew. In this case, the proper process is to renew the key, use a tool to generate a TLSA record from our certificate using the Command Line Interface (e.g. tlsagen), use the dynamic DNS updater (nsupdate) to add the new TLSA record, wait for the DNSSEC process to finish, sign and distribute the zone, then restart the service to commence using the new certificate and then remove the old TLSA record from our zone.

Let’s focus here on the DANE monitoring tools that we are using for remote DANE monitoring in the Go6lab:

  • For checking HTTPS we are using ldns-dane, part of LDNS toolkit from NLNETLabs.
  • For checking SMTP we are using a tool named “danecheck“, built by Viktor Dukhovni.

LDNS toolkit from NLNETLabs is pretty straightforward, and you can install it with the apt or yum package installers if you happen to use a Linux system (just search for “ldns”). However, if you have an SNI-based https domain service (multiple virtual hosts using the same IP address), then please note that Server Name Indication (SNI) was introduced in ldns-dane in version 1.6.17. At the time of writing, the majority of apt and yum repositories offer version 1.6.16 and with that version checking of the SNI domains will fail despite the fact that your TLSA might be correct.

The tool for testing DANE is called ldns-dane and usage is pretty simple:

[root@mx dane-monitor]# /usr/local/bin/ldns-dane verify www.go6lab.si 443
 91.239.96.52 dane-validated successfully
 2001:67c:27e4::52 dane-validated successfully

The tool will send an DNS query for the TLSA record for, in our case, ‘_443._tcp.www.go6lab.si.’, and will also query for A and AAAA records associated with ‘www.go6lab.si’. It will then communicate with the IP addresses, ask for the certificate and compare the two of them.

If your output does not look similar to the above, something is wrong with your TLSA records or DNSSEC signatures.

If you decide to download and build LDNS toolkit from the NLNETLabs website, please note that version 1.17 requires OpenSSL 1.1. If you use version 1.0, then stick with LDNS version 1.16.17.

Viktor Dukhovni’s danecheck tool is built a little bit differently as it requires a working Haskell GHC plus Stack toolchain. These can be downloaded from the Haskell platform website and are also available as packages for various operating systems.

Using the tool is quite simple, you specify just a domain name that you wish to test (in most cases):

root@openNMS:/# danecheck go6lab.si
go6lab.si. IN DS 15361 8 1 104e2a57cd968ec0bb283872c7f546d9b174
1839 ; AD=1 NoError
go6lab.si. IN DS 15361 8 2 f2cf0d4b46d3be6ccd25516cafc33fb5105f
28c3a9a82d782afe2ff1017bb208 ; AD=1 NoError
go6lab.si. IN DNSKEY 256 3 8 AwEAAc48Qs/Ew8nipd9vYGlk+NzXqE/llT
FU0IePzbEnH8sGbCGziDyo86aNIZQ3RuVjc30TUCR3cLZ6wrrOynPBeNGc20WpV
IvidQeWOWGieXkVYdQi6U+Cgk5LInq5aXg6IHriRC80mv7K1cofLVLRTTPlKXPf
tUbTStVzBX85k28cw7KfA92CbSbnAYl023eRaIE6lzKyzpuuel1OLUWxwmAGHH5
bTc6aa6A+jfcelkzUahCFZrQh+cU303LU8jVhNZGr6AgqiE6m2AE3iFdjzm6AMk
lqWNmYJeuQ0WwbSwu04Kuco9H5E0tdx26jbzaRffsxRPB/Kr0ydVqTU84fJnc= ; 
AD=1 NoError
go6lab.si. IN DNSKEY 257 3 8 AwEAAcgqIy9CjcyqWY+sJpNk9hC7CAUozO
xSpJddgwjIdvn6uZaEL7mHeAEGzHtubaBQBjVfiT7FwWL3tF8T/pirKuQ3tgVYS
4CS9T9uNFyYaue0P8lGX1s8vxtLNcnVfmoDRHJp/ne/6w4CqD4olOoExECPkXk5
LYAFG1fP+dPJKblWHE5jIlG6sxRp2dDxr6o7iEoL8VYEqWKe1sXZciF5P2N9/jo
NKe1QY4WNYoV/Hdr5ZqEeleArDa9UjWZ8sKQ6pXbJNdr1vDj23ANo4zmw/TS3Oz
O2VBU2Y16bHKOQsck4WyPovc0J7zdUX7QqIQod0DzB2qbdf1sXo4+T24yJzvU= ;
AD=1 NoError
go6lab.si. IN DNSKEY 257 3 8 AwEAAcmW5udATR8Gmpeqyv4WYVGvr3gE5s
nQZUSKgAU0mua39pbihh9EA8mxHNybbczYPlzS8JTTUZxcTNhuQzCkD1ELTA51K
WtZpnMqQWv3Rbvse8Eh0T7mvTGU6z6WVW0bh2uauKKR8obIyHTo3ol6Em7Jp4tJ
vbvdkTAi+3XFxalZlK0b94P2xzNmlfnttgJXPfNrKMukZqUljoZ5D/2ewEhY2K9
K++QkLLYO6kuwD0dPVKrJ2f3HUdbv+JGM0SOnXSiskdziOMJU2QrOd6g5Wku4VP
juQgFotDTaxX3yR/y7hPCOq02EtQD2kMqgsOHK4nyqx79l1sME1CZ8aDSFBgc= ; 
AD=1 NoError
go6lab.si. IN MX 10 mail.go6lab.si. ; AD=1 NoError
go6lab.si. IN MX 20 mail.go6.si. ; AD=1 NoError
mail.go6lab.si. IN A 91.239.96.17 ; AD=1 NoError
mail.go6lab.si. IN AAAA 2001:67c:27e4::beee ; AD=1 NoError
_25._tcp.mail.go6lab.si. IN TLSA 3 1 1 39726a2fe2bb052cf00e6b95
a8385f7446c3ea76d015fd0644085d6e9f3e8853 ; AD=1 NoError
 mail.go6lab.si[91.239.96.17]: pass: TLSA match: depth = 0, 
name = mail.go6lab.si
 TLS = TLS12 with ECDHE-RSA-AES128GCM-SHA256
 name = mail.go6lab.si
 depth = 0
 Issuer CommonName = Let's Encrypt Authority X3
 Issuer Organization = Let's Encrypt
 notBefore = 2017-10-29T12:33:26Z
 notAfter = 2018-01-27T12:33:26Z
 Subject CommonName = mail.go6lab.si
 pkey sha256 [matched] <- 3 1 1 39726a2fe2bb052cf00e6b95a8385f7
446c3ea76d015fd0644085d6e9f3e8853
 depth = 1
 Issuer CommonName = DST Root CA X3
 Issuer Organization = Digital Signature Trust Co.
 notBefore = 2016-03-17T16:40:46Z
 notAfter = 2021-03-17T16:40:46Z
 Subject CommonName = Let's Encrypt Authority X3
 Subject Organization = Let's Encrypt
 pkey sha256 [nomatch] <- 2 1 1 60b87575447dcba2a36b7d11ac09fb2
4a9db406fee12d2cc90180517616e8a18
mail.go6.si. IN A 91.239.96.61 ; AD=1 NoError
mail.go6.si. IN AAAA 2001:67c:27e4::61 ; AD=1 NoError
_25._tcp.mail.go6.si. IN TLSA 3 1 1 43fc43f31135d9542658b87356d
129ccfdd654f996499ac97cd6d34b2b8f9668 ; AD=1 NoError
 mail.go6.si[91.239.96.61]: pass: TLSA match: depth = 0, 
name = mail.go6.si
 TLS = TLS12 with ECDHE-RSA-AES256GCM-SHA384
 name = mail.go6.si
 depth = 0
 Issuer CommonName = Let's Encrypt Authority X3
 Issuer Organization = Let's Encrypt
 notBefore = 2017-11-18T22:45:21Z
 notAfter = 2018-02-16T22:45:21Z
 Subject CommonName = mail.go6.si
 pkey sha256 [matched] <- 3 1 1 43fc43f31135d9542658b87356d129c
cfdd654f996499ac97cd6d34b2b8f9668
 depth = 1
 Issuer CommonName = DST Root CA X3
 Issuer Organization = Digital Signature Trust Co.
 notBefore = 2016-03-17T16:40:46Z
 notAfter = 2021-03-17T16:40:46Z
 Subject CommonName = Let's Encrypt Authority X3
 Subject Organization = Let's Encrypt
 pkey sha256 [nomatch] <- 2 1 1 60b87575447dcba2a36b7d11ac09fb2
4a9db406fee12d2cc90180517616e8a18

If everything works as it should, then you should see similar output to the above.

Note the lines with a string of “AD=1 NoError” in it, that can be can be used in a script to automatically determine whether anything is wrong. For Go6lab purposes, we test each domain and “grep” for string “AD=” to extract messages from the tool’s output, and “grep -v” string “AD=1 NoError” to eliminate everything that seems OK and valid. If there is anything after the “grep” processing, omething is wrong with the domain and an e-mail can be sent flagging that.

Here’s an example how we monitor our DANE SMTP environment using a simple bash script:

file="/root/dane-smtp-monitor/domains.txt"
domains=`cat $file`
for domain in $domains; do
  echo "Checking SMTP DANE for $domain"
  /usr/local/bin/danecheck "$domain" | grep "AD=" | grep -v NoError \ 
  > /root/dane-smtp-monitor/"$domain"
    if [ -s /root/dane-smtp-monitor/"$domain" ]
    then
      echo "DANE for $domain is broken"
      mail -s "DANE TLSA config for $domain is broken!!!" [email protected] \ 
      < /root/dane-smtp-monitor/"$domain"
    else
      echo "All good for $domain ..."
    fi
done

This shows how you can monitor all your domains that are entered line-by-line in the specified domains.txt file. You can also use the same script to monitor your https services, but of course it will need to be modified as the output of the tool is different, and you would probably want to use “grep -v “dane-validated successfully”” as an indicator that all is well and functioning.

DANE and DNSSEC are very nice add-ons for security and provide better verification of certificates and services. But if they’re not working properly then you’re not doing yourselves a favour, hence the importance of monitoring to enable action to be taken if something fails.

Jan Žorž

 

 

Disclaimer: Viewpoints expressed in this post are those of the author and may or may not reflect official Internet Society positions.

Related Posts

Improving Technical Security 15 March 2019

DNS Privacy Frequently Asked Questions (FAQ)

We previously posted about how the DNS does not inherently employ any mechanisms to provide confidentiality for DNS transactions,...

Improving Technical Security 14 March 2019

Introduction to DNS Privacy

Almost every time we use an Internet application, it starts with a DNS (Domain Name System) transaction to map...

Improving Technical Security 13 March 2019

IPv6 Security for IPv4 Engineers

It is often argued that IPv4 practices should be forgotten when deploying IPv6, as after all IPv6 is a...