Log Export Format and Versioning

After configuring Log Management with AWS S3, the logs you download will be gzipped CSVs. Unzipping the logs and opening the CSV files will show the columns of information extracted from your Umbrella logs.

This information is intended to be helpful when consuming the data into a SIEM system or other log management software.

There are additional fields that are exposed in these logs that are not normally shown through the Reporting section of the dashboard. For more information on reporting, see Getting Started with Reports.

Versioning

Depending on the Umbrella subscription you have, and depending on the type of bucket you configure, there are different versions of the log formats. Currently, there are 4 versions.

Version 1 is for customers who have configured their own S3 bucket before November 2017. This version has a single sub-folder in the bucket and contains only DNS traffic logs.

Version 2 is for customers who have configured their own S3 bucket after November 2017, are using a Cisco-managed bucket, or are using Centralized Log Management for MSP/MSSP/Multi-org consoles. This version is inclusive of everything in version 1, and adds two new log types: Proxy traffic logs and IP traffic logs. Each log type has its own sub-folder.

Version 3 is the same as version 2, but adds two new fields: Most Granular Identity Type and Identity Types for DNS logs.

Version 4 is the same as version 3, but adds the Blocked Categories field for DNS and Proxy logs.

File Name Format

Logs are uploaded to S3 buckets in the appropriate subfolder with the following naming format.

<subfolder>/<YYYY>-<MM>-<DD>/<YYYY>-<MM>-<DD>-<hh>-<mm>-<xxxx>.csv.gz

<subfolder> will either be dnslogs, proxylogs, or iplogs, depending on the types of logs within. <xxxx> is a random string of four alphanumeric characters, which prevents duplicate file names from being overwritten.

Example:

dnslogs/2019-01-01/2019-01-01-00-00-e4e1.csv.gz

DNS Logs

DNS logs show traffic that has reached our DNS resolvers. A typical snippet of DNS logs will look like this:

"2015-01-16 17:48:41","ActiveDirectoryUserName",
"ActiveDirectoryUserName,ADSite,Network",
"10.10.1.100","24.123.132.133","Allowed","1 (A)",
"NOERROR","domain-visited.com.",
"Chat,Photo Sharing,Social Networking,Allow List"

• Timestamp—When this request was made in UTC. This is different than the Umbrella dashboard, which converts the time to your specified time zone.
• Most Granular Identity—The first identity matched with this request in order of granularity.
• Identities—All identities associated with this request.
• InternalIp—The internal IP address that made the request.
• ExternalIp—The external IP address that made the request.
• Action—Whether the request was allowed or blocked.
• QueryType—The type of DNS request that was made. For more information, see Common DNS Request Types.
• ResponseCode—The DNS return code for this request. For more information, see Common DNS return codes for any DNS service (and Umbrella).
• Domain—The domain that was requested.
• Categories—The security or content categories that the destination matches. For category definitions, see Understanding Security Categories and Understanding Content Categories.
• Most Granular Identity Type—The first identity type matched with this request in order of granularity. Available in version 3 and above.
• Identity Types—The type of identity that made the request. For example, Roaming Computer, Network, and so on. Available in version 3 and above.
• Blocked Categories—The categories that resulted in the destination being blocked. Available in version 4 and above.

Proxy Logs

Proxy logs show traffic that has passed through the Intelligent Proxy, and include files that have been inspected with the File Inspection feature. A typical snippet of proxy logs will look like this:

"2017-10-02 23:52:53","TheComputerName","ActiveDirectoryUserName,
ADSite,Network","192.192.192.135","1.1.1.91","","ALLOWED",
"http://google.com/the.js","www.google.com",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36
 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36","200",
"562","1489","","","","","","","","Networks"

• Timestamp—The timestamp of the request transaction in UTC (2015-01-16 17:48:41).
• Identities—Which identities, in order of granularity, made the request through the intelligent proxy.
• Internal IP—The internal IP address of the computer making the request.
• External IP—The egress IP address of the network where the request originated.
• Destination IP—The destination IP address of the request.
• Content Type—The type of web content, typically text/html.
• Verdict—Whether the destination was blocked or allowed.
• URL—The URL requested.
• Referer—The referring domain or URL.
• userAgent—The browser agent that made the request.
• statusCode—The HTTP status code; should always be 200 or 201.
• requestSize (bytes)—Request size in bytes.
• responseSize (bytes)—Response size in bytes.
• responseBodySize (bytes)—Response body size in bytes.
• SHA—SHA256 hex digest of the response content.
• Categories—The security categories for this request, such as Malware.
• AVDetections—The detection name according to the antivirus engine used in file inspection.
• PUAs—A list of all potentially unwanted application (PUA) results for the proxied file as returned by the antivirus scanner.
• AMP Disposition—The status of the files proxied and scanned by Cisco Advanced Malware Protection (AMP) as part of the Umbrella File Inspection feature; can be Clean, Malicious or Unknown.
• AMP Malware Name—If Malicious, the name of the malware according to AMP.
• AMP Score—The score of the malware from AMP. This field is not currently used and will be blank.
• Identity Type—The type of identity that made the request. For example, Roaming Computer, Network, and so on.
• Blocked Categories—The category that resulted in the destination being blocked. Available in version 4 and above.

IP Logs

IP logs show traffic that has been handled by the IP Layer Enforcement feature. A typical snippet of IP logs will look like this:

"2017-10-02 19:58:12","TheComputerName","198.198.198.1",
"55605","107.152.24.219","443","Unauthorized IP Tunnel Access"

• Timestamp—When this request was made in UTC.
• Identity—The first identity matched with this request in order of granularity.
• Source IP—The IP of the computer making the request.
• Source Port—The port the request was made on.
• Destination IP—The destination IP requested.
• Destination Port—The destination port the request was made on.
• Categories—Which security categories, if any, matched against the destination IP address/port requested.

For more information on the IP Layer Enforcement feature, see Add IP Layer Enforcement.

Cloud Firewall Logs

Cloud Firewall logs show traffic that has been handled by network tunnels. A typical snippet of Cloud Firewall logs will look like this:

"2019-01-14 18:03:46","[211039844]","Passive Monitor",
"CDFW Tunnel Device","OUTBOUND","1","84","172.17.3.4","","146.112.255.129",
"","ams1.edc","12","ALLOW"

• Timestamp—The timestamp of the request transaction in UTC.
• originId—The unique identity of the network tunnel.
• Identity—The name of the network tunnel.
• Identity Type—The type of identity that made the request. Should always be "CDFW Tunnel Device".
• Direction—The direction of the packet. It is destined either towards the internet or to the customer's network.
• ipProtocol—The actual protocol of the traffic. It could be TCP, UDP, ICMP.
• packetSize—The size of the packet that Umbrella CDFW received.
• sourceIp—The internal IP address of the user generated traffic towards the CDFW. If the traffic goes through NAT before it comes to CDFW, it will be the NAT IP address.
• sourcePort—The internal port number of the user generated traffic towards the CDFW.
• destinationIp—The destination IP address of the user generated traffic towards the CDFW.
• destinationPort—The destination port number of the user generated traffic towards the CDFW.
• dataCenter—The name of the Umbrella Data Center that processed the user generated traffic.
• ruleId—The ID of the rule that processed the user traffic.
• verdict—The final verdict whether to allow or block the traffic based on the rule.