Route53 is Amazon’s highly scaleable Domain Name Service that tightly integrates with the rest of Amazon’s platform services. Route53 has two categories of Domain Zones. A Public Zone works the same way you would expect other Domain Name Service’s to work. You buy a domain name from a domain name registrar and point it at your AWS Route53 name servers. Then everyone on the internet has access to the DNS records you have set up in your Public Zone.

In addition to a Public Zone, Route53 also has a Private Zone feature. This allows you to setup a set of DNS records and attach them to a VPC. After doing this it allows you to route traffic within one or more VPCs with domain and subdomain records that are only available within your VPC and without exposing them to the internet.

In this post we’re going to talk a bit more about Private Zones and how they can be useful with local development.

I recently started using Private Zones in Route53 more frequently in order to lower the burden on my configuration management system to maintain direct links to other dependent machines with IP addresses and to help with service discovery. It’s a lot easier to find an API node when they are all behind an Elastic Load Balancer that has an api.internal-loftylabs.com DNS Record pointing at it. I don’t have to know all of the IP addresses of all of the API nodes. Nor do I have to worry about the overhead of a service discovery system like Consul (although, Consul is really rad). Even if I have to replace a node, AWS’s Autoscaling Groups automatically put the new node back into the load balancer with the internal domain name. No need to rerun our Ansible playbooks to update configurations for the new machine.

So I had a lot of convenient internal domain names in Route53 for machines in my VPC that are not internet accessible, and they work great while I’m logged into an EC2 instance in that VPC. I also have a publicly accessible instance on that VPC running OpenVPN so that I can connect to my private network and get access to all of my not-publicly accessible instances.

However, after connecting to the VPN you do not automatically have access to the DNS records on your Route53 Private Zone. So even if you can connect the VPN and ssh into a machine on the VPC and access api.internal-loftylabs.com, you can not connect to the VPN and access api.internal-loftylabs.com from your local development machine. You have the convenience of api.internal-loftylabs.com within your VPC but have to resort to 10.12.352.123 on the machine where you actually do all of your work. Not that convenient!

There is a solution: Setup DNS forwarding on an instance on your VPC and add that instance as a DNS server to your local machine.

TL;DR:

• Setup a BIND Open Source DNS Server on an instance on your VPC.
• Configure the BIND server to forward DNS queries to the internal AWS DNS servers. The same ones your EC2 Instances use automatically. AWS DNS servers are always accessible at the base of the VPC IPv4 network range plus two. For example, the DNS Server on a 10.0.0.0/16 network is located at 10.0.0.2.
• Configure your VPN Server to push a DNS Server configuration to your development machine on connect.
• While connected to your VPN, your machine routes all DNS inquiries to your BIND DNS server on your VPC, which forwards them to the AWS internal DNS service. Allowing you to resolve Route53 Private Zone records on your local development machine.

Assumptions:

• You have DNS support and DNS Hostnames enabled on your VPC.
• You already have a Private Zone set up on Route53 and connected to your VPC.
• You have a VPN set up that you can connect to and successfully reach all of your private EC2 Instances on your VPC.
• Your machines are Ubuntu/Debian based. Sorry CentHOSERs.

Step 1: Setup BIND

This is pretty simple thanks to apt

sudo apt-get install bind9 bind9utils bind9-doc

Step 2: Configure BIND

BIND is an open source DNS server. We’re not going to setup our own records in it but we’re going to use it as a DNS Forwarder. It’ll take our DNS queries and forward them to another server and return the response.

We will need to modify the following file:

/etc/bind/named.conf.options

Make sure the file looks like this example below, but be sure to setup your own IP address for the AWS internal DNS of your VPC.

options {            directory "/var/cache/bind";            recursion yes;            allow-query { any; };            dnssec-enable yes;            dnssec-validation yes;            auth-nxdomain no;    # conform to RFC1035            listen-on-v6 { any; };            forwarders {              10.0.0.2; ## Note A              8.8.8.8; ## Note B            };            forward only;};

Note A: AWS DNS servers are always accessible at the base of the VPC IPv4 network range plus two. For example, the DNS Server on a 10.0.0.0/16 network is located at 10.0.0.2.

Note B: Your DNS queries will hit the AWS DNS server in the first line first, if it returns nothing, it’ll fall back to the standard Google DNS servers. This allows you to query your local DNS names as well as anything on the internet.

Be sure to restart BIND after changing the configuration!

sudo service bind9 restart

Essentially we’re done with our main goal now. You can now use this machine as a DNS server (forwarder) to get access to all of your private AWS Route53 records as well as public internet records. If you’re on a mac, you can go to the Network panel in System Preferences, and manually add this server as a new DNS server to your internet connection. That’s not really convenient though. Also, it will only work if you are connected to the VPN and have access to that machine.

Luckily OpenVPN has a shortcut for us.

Step 3: Configure OpenVPN to push a DNS Server configuration to your machine on connect.

OpenVPN has the ability to change our network configuration when we connect successfully to the VPN and change it back to the original settings when we disconnect. This will allow us to configure our local development machine to use our BIND DNS server as a DNS server but only when we are connected to the VPN. And it’s all done automatically behind the scenes.

We will need to modify the following file:

/etc/openvpn/server.conf

Add the following line to the bottom of the file:

dhcp-option DNS <IP address of machine you set up BIND on>

Restart OpenVPN

sudo service openvpn restart

If you reconnect to your VPN you should see the following in the logs:

Retrieved from OpenVPN: name server(s) [ 10.20.255.190 ], search domain(s) [  ] and SMB server(s) [  ] and using default domain name [ openvpn ]Saved the DNS and SMB configurations so they can be restoredChanged DNS ServerAddresses setting from '192.168.1.1' to '10.20.255.190'Changed DNS SearchDomains setting from '' to 'openvpn'Changed DNS DomainName setting from '' to 'openvpn'DNS servers '10.20.255.190' will be used for DNS queries when the VPN is active

The IP address will be different for you, but if you see these lines, OpenVPN has temporarily configured your local development machine to use your new BIND DNS server as your DNS.

Now on your local development machine you can hit all of the standard internal AWS domain names like ip-10-12-125-254.ec2.internal as well as any you might have set up on your Route53 Private Zone. This is really useful for when you have trouble remembering 10.32.23.52 but db.int-lofty.com is much easier to remember.

Happy Hacking!

Bonus Trivia: Do you know why Amazon’s DNS web service is called Route53? DNS runs on port 53.

‍