Troubleshooting SWG Website access issues

Overview

This KB article is a one-stop troubleshooting guide for different types of website access issues seen with Umbrella Secure Web gateway (SWG) Proxy. Let's assume the website www.xyz.com is not accessible via SWG proxy and when users try to access the internet directly (without Umbrella SWG being in the picture), it works fine.  Let's review various symptoms and different types of error messages reported when website is inaccessible via SWG. Most common ones are 502 bad gateway, 502 could not relay message upstream error, upstream certificate revoked, access denied 403 forbidden, upstream ciphers mismatch, website just timed out after spinning for sometime etc.

"Access Denied 403" error due to upstream block

Webserver or upstream side is blocking or throttling our SWG proxy egress IP ranges. For example, Akamai WAF has blacklisted couple of SWG egress IP ranges. To resolve this issue, only option is that reach out to website admins and have them unblock our IP ranges. Until then, bypass SWG using external domains management list for Anyconnect SWG and PAC file deployments. In short, this type of issue is not because of proxy itself, rather its due to incompatibility between proxy and Webservers. Here is the link to refer the KB specifically for "Access Denied 403" error due to Egress IP's block.

In addition, here is the link covering few possible reasons why Akamai is blacklisting IP addresses.

"Access Denied 403" error due to Java Issue

Website is not accessible and throwing "Access Denied or 403 Forbidden - Umbrella cloud security gateway error" when the request is sent through SWG MPS proxy with the file inspection setting enabled. But if File Inspection is disabled, websites loads successfully. Or if we put the website in bypass decryption, websites loads successfully.

Root Cause of issue at very high level

What is Java related issue with MPS?

The site or web server in question returns a TLS warning regarding SNI or SSL alert back to the proxy after proxy tries to connect to the server. Basically, this happens after the client hello is sent. MPS proxy (which is based on Java and as such) by design, it treats any TLS alerts with "Unrecognized Name" in the description field as an error during SNI parsing and it terminates the transaction. More details found here

Please be aware that this is not SWG or MPS proxy issue. This is one of the incompatibilities with SWG or any other proxies due to misconfiguration on the server side. Browsers usually ignore this warning but SWG or other content security filter treats the SSL warning as a fatal error and terminates the session, which results in 403 forbidden error pages to the users. It may also report 502 Bad Gateway error, but with most of the examples what we have seen is 403 forbidden error, as shown below.

As MPS works at application layer, it has little to no control how the TLS layer handles the transaction based on the alerts produced in TLS protocol. It is the responsibility of the server to ensure their TLS endpoint/certificates are configured correctly. Please refer the link

To narrow down or troubleshoot the issue, it can be easily pointed out from SSL lab. 

When the website is accessed without SWG proxy in the middle or bypass HTTPS inspection from SWG, the website will work because browser is ignoring the SNI Unrecognized name alert and continues communicating with the web server.

At the time of writing this article, the workaround recommended below is the best mitigation we can suggest to our customers. In near future, with the new proxy architecture, we should be able to handle these issues more gracefully.

Resolution

1. Disable Decryption for the affected domains - OR
2. Add the domain to a destination list and associate an allow rule (if customer trusts the site)

What is 502 Bad gateway

A 502 Bad Gateway Error means that the server was acting as a gateway or proxy and received an invalid response from the upstream server. When the user tries to access website via SWG Proxy, there are two flows of communication happen.

a) Client --> Proxy connection (Downstream)
b) Proxy--> End web server connection. (Upstream )

502 Bad Gateway error occurs between SWG Proxy (MPS, Nginx) to end server connection.

COMMON FACTORS for 502 Bad Gateway and RESOLUTION

1. Unsupported SWG Cipher Suites

2. Client certificate authentication request

3. Headers added or removed by SWG Proxy

Unsupported SWG Cipher suites

Let's assume a web server reporting unsupported SWG cipher suites during TLS negotiation. Please note that SWG MPS (Modular Proxy Service) Proxy doesn't support the TLS_CHACHA20_POLY1305_SHA256 cipher suite. Please be aware that there is a separate article to cover SWG-supported cipher suites and TLS. We can easily pinpoint this issue by reviewing ther packets captured during cipher suites exchange in client hello and server hello. As a troubleshooting step, utilize the CURL command enforcing usage of specific ciphers to narrow down the issue and to confirm it is due to cipher suites as shown in example 1 and 2 below.

Example of Curl commands:

1. curl -vvv "<Destination>" --ciphers TLS_DHE_DSS_WITH_AES_256_GCM_SHA384 >> /dev/null
2. curl -vvv "<Destination>" --ciphers ECDHE-RSA-AES256-GCM-SHA384 >> /dev/null
3. Testing website With Proxy: - curl -x  proxy.sig.umbrella.com:80 -v  xyz.com:80
4. curl -x swg-url-proxy-https.sigproxy.qq.opendns.com:443 -vvv -k "https://www.cnn.com" >> null
5. Testing website without Proxy: - curl -v www.xyz.com:80

6. Mac/Linux:  - curl -vvv -o /dev/null -k -L www.cnn.com

7. Windows: - curl -vvv -o null -k -L www.cnn.com

Resolution: To resolve the issue, skip the inspection for the problematic website using selective decryption list.

Client certificate authentication request

During the TLS handshake between SWG Proxy and upstream, the upstream webserver expects Client certificate authentication. As client certificate authentication is not supported, we need to bypass those domains from proxy using external domains management list, and bypassing just https inspection is not enough. For ex: https://valuedoor2.smbc.co.jp 

Headers added by Proxy

The web server is reporting 502 bad gateway error due to X-Forward-For header (XFF) added by SWG Proxy when https inspection is enabled. We can easily narrow down most of the 502 bad gateway issues by first troubleshooting the issue with or without https inspection, and with or without file inspection to rule out file scanning issue with MPS Proxy.

We use XFF header when the HTTPS inspection is turned on, so that the upstream server can provide optimal geo-location content based on client IP (which provides the user's physical location).

When HTTPS inspection is not enabled, this header is not added by the proxy, so there will not be a 502 Bad Gateway error. This is not a SWG proxy issue . This error is due to the upstream web server which is misconfigured to not support standard XFF header.

Resolution: To resolve the issue, bypass HTTPS inspection for specific domain(s) using selective decryption lists.

517 Upstream Certificate Revoked

Certificate and TLS Protocol Errors

Select SWG DC manually for Internal testing