Extended Version: How the VA communicates with Umbrella and local DNS

Introduction

This article is a continuation example from this article: How the VA communicates with Umbrella and local DNS. Please begin at the article above before continuing here. Just looking for an overview, click above.  

Detailed Example

Once the forwarder has determined if a query needs to go to the local DNS or to Umbrella, it then starts the process of sending the queries to the upstream servers. Using an increasing timeout value, it will query each server in random order at the current timeout value until it either gets a response or has queried all the servers. Then it tries the next timeout value, and so on. 

The exact steps taken are as follows: 

1. Randomly order the servers.
2. Set the timeout value to 2.0
3. For each server in order, do the following:
4. Check if the server/qname/qtype combination exists in the SERVFAIL cache (from the simple article). If so, skip this server. The SERVFAIL cache will expire, therefore domains are not permanently in this state. 
5. Check if server's RTT is in the cache. If so, and if it is greater than the current timeout, skip this server.

Where does the timeout value come into play? In step 5, the server is not used if it does not reply quicker than the timeout time. Servers that don't respond in 4.5 seconds are never queried to avoid the slowdown potential. To avoid going through the process multiple times in succession for a domain/nameserver, we cache this SERVFAIL for a period of time. 

 iii. Query the server.

1.  If the query responds within the timeout, cached the response time as the RTT for that server, return the response and then stop.
2.   Else, cache the timeout as the RTT for that server.
3. Set the timeout value to 2.5, then repeat step #3
4. Set the timeout value to 4.0, then repeat step #3
5. Set the timeout value to 4.5, then repeat step #3
6. If we still don't have a response at this point, we try step 3 for one final server. If that server fails to respond, we will return a SERVFAIL response and update the SERVFAIL cache with the server and number of tries.

Exceptionally Detailed Example

Let's say that I have two internal servers configured in my VA, one locally where the average RTT is ~20 msec ("localdns1"), and one on the other side of the planet where the average RTT is ~2.1 seconds ("localdns2"). 

The first query to hit the VA might look like this:

1. Randomly ordering the servers results in localdns2 first, then localdns1
2. The timeout is set to 2.0 seconds.
3. localdns2 isn't in the SERVFAIL cache for any combinations, so we continue.
4. localdns2 isn't in the RTT cache, so we continue.
5. Query is sent to localdns2.
6. After 2 seconds, the query hasn't responded, so we set the RTT to 2.0 seconds for that server.
7. localdns1 isn't in the SERVFAIL cache for any combinations, so we continue.
8. localdns1 isn't in the RTT cache, so we continue.
9. Query is sent to localdns1.
10. localdns1 responds 20 msec later. We set the RTT to 0.02 seconds for that server, and then return the response.

Note that the query time here would be slightly above 2.02 seconds in total.

Then, the next query comes in, which would look like this: 

1. Randomly ordering the servers results in localdns2 first, then localdns1
2. The timeout is set to 2.0 seconds.
3. localdns2 isn't in the SERVFAIL cache for any combinations, so we continue.
4. localdns2 is in the RTT cache, but it's value is equal to 2.0 seconds, so we skip this server.
5. localdns1 isn't in the SERVFAIL cache for any combinations, so we continue.
6. localdns1 is in the RTT cache, but it's value is lower than 2.0 seconds, so we continue.
7. Query is sent to localdns1.
8. localdns1 responds 20 msec later. We refresh the RTT to 0.02 seconds for that server, and then return the response.

Note the query time here would be slightly above 0.02 seconds in total. Thus, we've now correctly started to prefer the closer server.

That pattern would be repeated for the next hour until the RTT cache entry for localdns2 expires (other than the random ordering of the servers may change each time). At that point, the first scenario would repeat once, refreshing the RTT cache for localdns2. 

Now let's say, at some point, localdns1 goes down and completely stops responding to queries. Assuming localdns2's RTT hasn't expired, the next query to come in might look like this:

1. Randomly ordering the servers results in localdns2 first, then localdns1
2. The timeout is set to 2.0 seconds.
3. localdns2 isn't in the SERVFAIL cache for any combinations, so we continue.
4. localdns2 is in the RTT cache, but it's value is equal to 2.0 seconds, so we skip this server.
5. localdns1 isn't in the SERVFAIL cache for any combinations, so we continue.
6. localdns1 is in the RTT cache, but it's value is lower than 2.0 seconds, so we continue.
7. Query is sent to localdns1.
8. After 2 seconds, the query hasn't responded, so we set the RTT to 2.0 seconds for that server.
9. The timeout is increased to 2.5
10. localdns2 isn't in the SERVFAIL cache for any combinations, so we continue.
11. localdns2 is in the RTT cache, but it's value is lower than 2.5 seconds, so we continue.
12. Query is sent to localdns2
13. localdns2 responds 2.1 seconds later. We set the RTT to 2.1 seconds for that server, and then return the response. 

Note that the query time here would have been slightly over 4.1 seconds in total. This may result in a DNS timeout return to the requesting client on the first try. 

The next query to come in might look like this:

1. Randomly ordering the servers results in localdns1 first, then localdns2
2. The timeout is set to 2.0 seconds.
3. localdns1 isn't in the SERVFAIL cache for any combinations, so we continue.
4. localdns1 is in the RTT cache, but it's value is higher than 2.0 seconds, so we skip this server.
5. localdns2 isn't in the SERVFAIL cache for any combinations, so we continue.
6. localdns2 is in the RTT cache, but it's value is equal to 2.0 seconds, so we skip this server.
7. The timeout is increased to 2.5
8. localdns1 isn't in the SERVFAIL cache for any combinations, so we continue.
9. localdns1 is in the RTT cache, but it's value is lower than 2.5 seconds, so we continue.
10. Query is sent to localdns1.
11. After 2.5 seconds, the query hasn't responded, so we set the RTT to 2.5 seconds for that server.
12. localdns2 isn't in the SERVFAIL cache for any combinations, so we continue.
13. localdns2 is in the RTT cache, but it's value is lower than 2.5 seconds, so we continue.
14. Query is sent to localdns2
15. localdns2 responds 2.1 seconds later. We refresh the RTT to 2.1 seconds for that server, and then return the response.

Note that the query time here would have been slightly over 4.6 seconds.

The next query to come in might look like this:

1. Randomly ordering the servers results in localdns1 first, then localdns2
2. The timeout is set to 2.0 seconds.
3. localdns1 isn't in the SERVFAIL cache for any combinations, so we continue.
4. localdns1 is in the RTT cache, but it's value is higher than 2.0 seconds, so we skip this server.
5. localdns2 isn't in the SERVFAIL cache for any combinations, so we continue.
6. localdns2 is in the RTT cache, but it's value is higher than 2.0 seconds, so we skip this server.
7. The timeout is increased to 2.5
8. localdns1 isn't in the SERVFAIL cache for any combinations, so we continue.
9. localdns1 is in the RTT cache, but it's value is equal to 2.5 seconds, so we skip this server.
10. localdns2 isn't in the SERVFAIL cache for any combinations, so we continue.
11. localdns2 is in the RTT cache, but it's value is lower than 2.5 seconds, so we continue.
12. Query is sent to localdns2
13. localdns2 responds 2.1 seconds later. We refresh the RTT to 2.1 seconds for that server, and then return the response.

Note that the query time here would be slightly over 2.1 seconds, so we've now successfully switched preference to localdns2. localdns1 wouldn't be queried again until it's RTT cache entry expired 1 hour later.

Now, hypothetically, let's say localdns2 stops responding to queries as well. The next query might look like this:

1. Randomly ordering the servers results in localdns1 first, then localdns2
2. The timeout is set to 2.0 seconds.
3. localdns1 isn't in the SERVFAIL cache for any combinations, so we continue.
4. localdns1 is in the RTT cache, but it's value is higher than 2.0 seconds, so we skip this server.
5. localdns2 isn't in the SERVFAIL cache for any combinations, so we continue.
6. localdns2 is in the RTT cache, but it's value is higher than 2.0 seconds, so we skip this server.
7. The timeout is increased to 2.5
8. localdns1 isn't in the SERVFAIL cache for any combinations, so we continue.
9. localdns1 is in the RTT cache, but it's value is equal to 2.5 seconds, so we skip this server.
10. localdns2 isn't in the SERVFAIL cache for any combinations, so we continue.
11. localdns2 is in the RTT cache, but it's value is lower than 2.5 seconds, so we continue.
12. Query is sent to localdns2
13. After 2.5 seconds, the query hasn't responded, so we set the RTT to 2.5 seconds for that server.
14. The timeout is increased to 4.0
15. localdns1 isn't in the SERVFAIL cache for any combinations, so we continue.
16. localdns1 is in the RTT cache, but it's value is lower than 4.0 seconds, so we continue.
17. Query is sent to localdns1.
18. After 4.0 seconds, the query hasn't responded, so we set the RTT to 4.0 seconds for that server.
19. localdns2 isn't in the SERVFAIL cache for any combinations, so we continue.
20. localdns2 is in the RTT cache, but it's value is lower than 4.0 seconds, so we continue.
21. Query is sent to localdns2
22. After 4.0 seconds, the query hasn't responded, so we set the RTT to 4.0 seconds for that server.
23. The timeout is increased to 4.5
24. localdns1 isn't in the SERVFAIL cache for any combinations, so we continue.
25. localdns1 is in the RTT cache, but it's value is lower than 4.5 seconds, so we continue.
26. Query is sent to localdns1.
27. After 4.5 seconds, the query hasn't responded, so we set the RTT to 4.5 seconds for that server.
28. localdns2 isn't in the SERVFAIL cache for any combinations, so we continue.
29. localdns2 is in the RTT cache, but it's value is lower than 4.5 seconds, so we continue.
30. Query is sent to localdns2
31. After 4.5 seconds, the query hasn't responded, so we set the RTT to 4.5 seconds for that server.
32. We still don't have a response, and we've reached the end of our timeout values. So, we send one final query to localdns1.
33. After 4.5 seconds, the query hasn't responded, so we increment the SERVFAIL tries for that server.
34. A SERVFAIL response is returned.

Ouch. Note that this took a whopping 24 seconds to happen. However, the RTT value for both servers is now at 4.5, so for every subsequent query, we should only have to wait 4.5 seconds to get a SERVFAIL.

Assuming both servers continue to fail to respond, they will eventually hit 20 SERVFAIL tries, at which point we put the server/qname/qtype combination for each subsequent query into the SERVFAIL cache. This means that queries matching those entries will not be sent to that server at all until the SERVFAIL cache entry expires. If both servers have that combination in the SERVFAIL cache, we will immediately SERVFAIL. When the SERVFAIL cache entries expire, we'll send another query to the server, and then put it back in the SERVFAIL cache if it doesn't respond. That way, we'll notice when the servers recover, but we won't be constantly sending out queries that we know will just fail.