After configuring your Cisco Umbrella Log Management in Amazon's AWS S3, the logs you'll download will be a gzip'd CSV format. Unzipping the logs and opening the CSV files will show the columns of information extracted from your Umbrella logging. Although some of this information may be already familiar to you, there are additional data fields that are exposed in these logs that are worth understanding when you're consuming the data into a SIEM or other log management software.
Depending on the type of customer you are, and depending on the type of bucket you configure, there are different versions of the log formats. In future, we will continue to increase the types of data in these logs. There are two current versions.
Version 1 is for customers who have configured their own S3 buckets previous to November 2017. Version 1 has a single sub-folder in the bucket and contains only DNS traffic logs.
Version 2 is for customers who have configured their own S3 buckets after November 2017, or are using the Cisco-managed buckets or are using the Centralized Log Management for MSP, MSSP or Multi-org console. Version 2 is inclusive of everything in version 1. In version 2, there are three subfolders, one for DNS traffic logs, one for proxy traffic logs and one for IP traffic logs.
If you are on version 1 and would like to upgrade, removing your existing S3 bucket, disabling the integration, then creating a new bucket from scratch will result in the new bucket being on version 2. We are planning to migrate existing buckets to version 2 in the near future but will do so after notifying customers of the planned change.
Log Management Export Columns - Version 1
A typical snippet of DNS logs will look like this:
"2015-01-16 17:48:41","ActiveDirectoryUserName","ActiveDirectoryUserName,ADSite,Network","10.10.1.100","220.127.116.11","Allowed","1 (A)","NOERROR","domain-visited.com.","Chat,Photo Sharing,Social Networking,Allow List"
|When this request was made in UTC.
Note: This is different than the Umbrella dashboard, which converts the time to your specified time zone.
Most Granular Identity
|The first identity matched with this request in order of granularity. The order of granularity is roaming computers first, then AD Users, AD Computers, and Sites and Networks. The most granular identity to match this request is listed here.|
|All identities associated with this request in order of the first matched against your policy hierarchy.|
|The internal IP address that made the request.|
|The external IP address that made the request.|
|Whether the request was allowed or blocked.|
|The type of DNS request that was made. For more information, see here: https://support.umbrella.com/hc/en-us/articles/232252848|
|The DNS return code for this request. For more information, see here: https://support.umbrella.com/hc/en-us/articles/232254248|
|The domain that was requested.|
|Information about the security category or content category of the domain requested, e.g. "Search Engines", "Malware", "Blogs", etc. The category may also match your custom destination lists. The complete list of categories can be found here: https://api.opendns.com/v3/categories
Note: In order to view the information in that link, you must click it in an authenticated browser session with the Umbrella dashboard. The output from the link is in JSON, a tool like JsonView for Chrome can help parse it, and JSON to CSV convert can be found here: https://konklone.io/json/
Log Management Export Columns - Version 2
Version 2 contains all of the DNS log information from version 1 and the following two new log types—Proxy Logs and IP Logs.
Proxy logs show traffic that's been passed through the Umbrella intelligent proxy, and include files that have been inspected with the file inspection feature. A typical snippet of proxy logging will look like this:
"2017-10-02 23:52:53","TheComputerName","ActiveDirectoryUserName,ADSite,Network","18.104.22.168","22.214.171.124","","ALLOWED","http://google.com/the.js","www.google.com","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36","200","562","1489","","","","","","","","Networks"
Note: not all fields below are actually used in most or all requests and are included for future enhancement.
|Timestamp||The timestamp of the request transaction in UTC (2015-01-16 17:48:41).|
|Identities||Which identities, in order of granularity, made the request via the intelligent proxy.|
|Internal IP||The internal IP address of the computer making the request, if available.|
|External IP||The external IP address of the network egress.|
|Destination IP||The destination IP address if available.|
|Content Type||The type of web content, typically (text/html).|
|Verdict||Whether this destination allowed or blocked after being proxied|
|URL||The URL to which the request was made, if available. A request made simply be made the top level domain.|
|Referer||The referring domain or URL, if available|
|userAgent||The browser agent that made the request.|
|statusCode||The HTTP status code; should always be 200 or 201.|
|requestSize (bytes)||Request size in bytes.|
|responseSize (bytes)||Response size in bytes.|
|responseBodySize (bytes)||Response body size in bytes.|
|SHA||SHA256 hex digest of the response content.|
|Categories||The Umbrella category for this request, such as Malware|
|AVDetections||The detection name according to the antivirus engine used in the file inspection.|
|PUAs||A list of all potentially unwanted application (PUA) results for the proxied file as returned by the antivirus scanner).|
|AMP Disposition||The status of the files proxied and scanned byCisco Advanced Malware Protection (AMP) as part of the Umbrella File Inspection feature - can be Clean, Malicious or Unknown.|
|AMP Malware Name||If Malicious, the name of the malware according to Cisco Advanced Malware Protection (AMP).|
|AMP Score||The score of the malware from AMP. Note, this field is not currently used and will be blank (AMP).|
|Identity Type||The type of identity that made the request-- Roaming Computer, Network and so on.|
IP logs show traffic that's been handled by our IP layer enforcement feature. A typical snippet of IP logging will look like this:
"2017-10-02 19:58:12","TheComputerName","126.96.36.199","55605","188.8.131.52","443","Unauthorized IP Tunnel Access"
When this request was made in UTC.
|The first identity matched with this request in order of granularity. The order of granularity is Roaming Computers first, then AD Users, AD Computers, and Sites and Networks. The most granular identity to match this request is listed here.|
|Source IP||The IP of the computer making the request. This can be obfuscated by NAT and only show the external IP of the network from which the request was made.|
|Source Port||The port the request was made on|
|Destination IP||The destination IP the request was made to|
|Destination Port||The destination port the request was made on.|
|Categories||Which Umbrella security categories, if any, matched against the destination IP address/port requested.|