The Virtual Appliance will talk to both the Umbrella resolvers as well as local DNS, depending on the DNS query and user configuration. Unlike simpler DNS clients, the VA does not prioritize one server over the other, or do a simple round robin. Instead, the VA uses the process outlined here.
This ensures, after an initial query, that the best DNS server is used. This also explains why a DNS query may be slow for one query, but speed up significantly after the first in select scenarios.
This article specifically refers to the Umbrella Virtual Appliances (VAs). The forwarder will query servers in a random order using an increasing set of timeout values until it gets a response. How it deals with servers that don't respond within the given timeouts will be discussed in the rest of this article.
The VA forwarder maintains an RTT (round trip time) cache for use when deciding if a query should be sent to a server.
The RTT which is a measure in seconds of how long it took for us to get a response from a server. Each time the forwarder sends a query to a server, it caches the RTT for 15 minutes. Once that expires, the RTT is effectively 0 for that server which resets it to the default state of "use this server".
If a server ultimately fails to respond at the highest timeout level, we will try it once more and then reply to the client with a SERVFAIL if it fails to respond. Any subsequent queries of this nature will be retried against the servers in question according to the current timeout level.
Note, DNS responses are not cached on the VA. The data that is cached is how long an authoritative nameserver takes to respond for a given domain.
This process determines which local DNS server and which of our public resolvers respond the quickest, and ensures that it is used rather than doing a round robin every time. This allows for avoiding sending DNS to a local DNS server that went down, for example.