browse
Overview
This article is intended for users seeing issues with Akamai-served content and domains after switching to Cisco Umbrella.
What is Akamai and how is it used?
Akamai is a Content Delivery Network (CDN) which stores and serves content for many content providers across the web. This includes streaming video, images, site content, scripts, advertising and more. Akamai is one of many CDNs, such as Amazon CloudFront or Limelight Networks, which provide the backbone of many popular sites.
A CDN is used by a website because it has very high traffic, including major traffic spikes-- for instance, when there is new content that is going 'viral', or a breaking news story. Rather than pay for their own hosting infrastructure to host their site (which needs to be powerful enough to serve peak times but leaving it partly idle a large portion of the time), webmasters use CDNs so they only pay for the resources they actually use. During slow times, content flows out using fewer resources and during peak times the CDN scales up their resources to serve everyone without missing a beat. The end result is a site which has much better uptime and saves money for the webmaster. Akamai is one of the very largest of this type of CDN.
To figure out if a site is using Akamai, look for www.sitename.com pointing to subdomains as a CNAME such as edgesuite.net, akamai.net, edgekey.net, amakaiedge.net (generally all .net). Be sure to look up www.domain.com rather than domain.com to check if there is an Akamai CNAME.
Note: some sites only use Akamai for images or video content within the site and only images reference a domain served by Akamai via CNAME.
Does using Umbrella affect Akamai content?
Why am I getting a different IP returned from Umbrella compared with my ISP?
Using Umbrella will not affect Akamai content on its own unless any Akamai domains are blocked by one of your domain lists. Remember, any domain added is a wildcard automatically for all subdomains (eg: *.domain) so entering "akamai.net" into your block list will break many sites across the web.
You may have come across this article because you've noticed that there are differences between the way your ISP's DNS servers and Umbrella hdandle Akamai traffic, specifically. It may even be the case that the ISP's DNS results in different content being loaded. It's fairly common for the IP address returned by an ISP's DNS server differs from the one received when querying Cisco - but that doesn't mean Cisco's results are incorrect. Quite the opposite! While the results do differ, this is because Cisco/OpenDNS and Akamai participate in the Global Internet Speedup Project using EDNS Client Subnet (ECS). How does this affect the IP returned from the DNS request? Read on to learn more.
Does this just affect Akamai?
No! This may affect any CDN provider who utilizes ECS. Akamai is the most common occurrence.
What is ECS and how does it make a difference?
ECS stands for EDNS Client Subnet and it is part of the Global Internet Speedup Project, designed to improve the functionality and speed of distributed websites - notably CDNs - across the globe.
Without ECS: DNS is requested from your current DNS server - which queries the Authoritative DNS server of the domain and returns back an IP address of the server to connect to. This server is close to that of the DNS server queried and may be very far from your current location. The DNS server queried determines which region's content server will be used to provide an answer. The end user may be very far from the server providing content resulting in slow speeds. For example:
User A in Brazil queries the local DNS server set which happens to be a DNS server in Miami. The DNS server in Miami returns a reply IP address close to Miami for the domain. User A is frustrated that the download is so slow and coming from the USA.
With ECS: DNS to a recursive DNS server has the source subnet (typically a /24) appended to it's DNS request to the authoritative DNS servers. The authoritative server for the domain responds back with a custom answer to what it believes is the closest, best server to serve the end user's request. The user can query a DNS server anywhere and still get a local server to serve up the content. For example:
User A in Brazil queries the local DNS server set which happens to be a DNS server in Miami. The DNS server has ECS enabled in Miami and passes onto the authoritative DNS server for the domain that the user is coming from a subnet in Brazil. It then returns a reply IP address for a datacenter in Sao Paulo, Brazil for the domain. User A is happy that the DNS server in Miami supports ECS and has a fast download from the content provider.
What issues can that cause when using Umbrella?
Cisco Umbrella's global DNS network and Akamai are ECS enabled and therefore reply back with the best server IP for your current egress subnet of DNS. This can become an issue when load balancing with certain ISPs or when ISPs have routing issues on their end. For example, one ISP may have an internal Akamai server and directs anyone using their ISP DNS servers to this local server. Being clever, they then block the IPs of other Akamai servers to force everyone to use their local Akamai server to save on transit costs.
When switching to Cisco, we ask Akamai directly what the best server to use is for the end user subnet, and Akamai responds back directly. Especially on load balanced connections, Akamai may return a valid IP that the ISP does not correctly route to.
It is in this scenario where, after switching to Umbrella, Akamai-served content intermittently does not load. The most common manifestation are with pages such as "www.foxnews.com" loading as a wireframe of HTML without the rich content.
The most commonly seen issues are with the ISP Time Warner Cable (RoadRunner).
It is important to note that the IP returned by Umbrella for Akamai content is valid in these cases and the ISP is timing out as it is not making the connection. To confirm that it is indeed this type of issue, test the exact same IP on a device on a different type of network connection such as a cell phone on mobile data.
If you are not getting an IP address at all in response to a DNS request to Umbrella, this is a different issue. Contact support@umbrella.com for help! This case applies only to when pages are not loading when Umbrella is responding with a valid Akamai IP address.
Who can help if this is happening?
Your ISP is the only party who will be able to assist in resolving this issue. This is because the DNS request returns an IP address to which the ISP times out to while the rest of the world is able to connect to that very same IP. Umbrella has completed it's part in returning a valid IP address in response to the DNS request. The ISP has not completed it's part in routing data to this IP address. Your subscription to the ISP is to allow access to Internet resources and this is a failure to do so on their side.
The ISP is the only one able to troubleshoot routing between their network and it's peers; Umbrella is has no visibility and cannot remediate this routing issue. We can only show we are returning a valid IP and it cannot be routed to.
Example: domain.com is a CNAME for a1234.a.akamaiedge.net which points to the IP 1.2.3.4. Connections to 1.2.3.4 time out. When running a traceroute, we get 8 steps in, past the ISP but then cease getting returns.
Example issue and solution: After speaking with the ISP, the issue was that the ISP's IP range does not have a return route from Akamai back to the user's IP within the ISP's space. The ISP presented two options: 1) Wait for Akamai to add a route back to this IP 2) Change IPs within the ISP's ranges to an IP that does have a return route from Akamai that works.
Need assistance in working with your ISP? We can help explain and provide evidence with confirmation that the IP returned from the Akamai A-record is valid from Akamai through our resolvers.
Troubleshooting Tips
Do you believe this issue affects you but you're not sure? Here some helpful hints for troubleshooting. These steps would be also required for our support team to be able to assist further. Once this info is gathered, contact support@umbrella for assistance.
1) nslookup queries: for the domain(s) which is being problematic. For this example. we'll use www.foxnews.com, which is an Akamai-powered site at the time of writing.
- nslookup www.foxnews.com
- nslookup www.foxnews.com 8.8.8.8
- nslookup www.foxnews.com 208.67.222.222
- nslookup www.foxnews.com 208.67.220.220
2) traceroutes: OS dependent, this may be tracert (Windows) or traceroute (OS X). In this case, you're trying to trace to the FQDN as well as the IP addresses you got in step 1 above.
- tracert www.foxnews.com
- tracert <resultant IP of nslookup www.foxnews.com>
- tracert <resultant IP of nslookup www.foxnews.com 8.8.8.8>
- tracert <resultant IP of nslookup www.foxnews.com 208.67.222.222>
3) Gather a list of the domains and IPs you are not able to reach, they may share common Akamai host infrastructure being blocked by your ISP.