Umbrella is improving the experience of using Amazon's S3 service with a new service: a managed S3 bucket that we (Cisco Umbrella) take care of in the Amazon cloud. Each customer's bucket in S3 is totally unique to that customer and only you have the credentials for your own bucket.
Umbrella has the ability to upload, store and archive the traffic activity logs from your organization in the cloud. The archiving of logs is done using the Amazon AWS S3 service. S3 is Amazon's Simple Storage Service (hence, the three S's). This feature is sometimes referred to as 'offline storage' or 'log retention.'
The logs are stored in a compressed (gzip) archive in CSV format. Logs are uploaded every ten minutes so there's a minimum of delay between network traffic coming from your network, being logged by Umbrella and then being available to download from S3.
The Cisco Log Management feature can be found in the Umbrella dashboard at Settings > Log Management.
The benefits are the same as managing your own S3 bucket. By having your logs uploaded 'offline' to an S3 instance, you can then download the logs from Amazon automatically to keep in perpetuity in backup storage. Or ingest the logs via your SIEM or other security tool to determine if any security events in your Umbrella logs coincide with events in other security tools.
Pros to having Umbrella manage your S3 instance:
- Extremely easy to setup-- it only takes a couple of minutes -- and afterward it's extremely easy to manage.
- Included in your license cost with Umbrella, effectively making it free. Although having your own bucket is very inexpensive, the overhead of having to manage another bill to pay can be prohibitive.
Cons to having Cisco manage your S3 instance:
- Limitations on how long data can be stored offline-- 30 days is the maximum.
- You cannot add anything to your bucket besides log files from Umbrella and the bucket cannot be used by another application.
- You cannot get support from directly Amazon for advanced configuration assistance, such as automation or help with command line.
NOTE: Existing Umbrella Insights and Umbrella Platform customers can access Log Management with Amazon S3 via the dashboard. Log Management is not available in all packages. If you are interested in this feature, please contact your account manager or email our account management team at email@example.com.
Configuring a Cisco-managed S3 Bucket
Navigate to Settings > Log Management in your Umbrella dashboard. There are two options: use your company-managed Amazon S3 bucket, or use a Cisco-managed Amazon S3 bucket.
Pick "Use a Cisco-managed Amazon S3 bucket" and you'll be given two new options: "Select a Region" and "Select a Retention Duration".
Select a Region
Regional endpoints are important to minimize latency when downloading logs to your servers. The regions match those available in Amazon S3, however not all regions are available-- for instance, China is not listed.
Pick the region that's closest to you from the dropdown. If you wish to change your region in the future, you will need to delete your current settings and start over.
Select a Retention Duration
The retention duration is simply 7, 14 or 30 days. Beyond the selected time period, all data will be purged and cannot be retrieved no matter what the situation is. We recommend a smaller time period if your ingestion cycle is regular. The retention duration can be changed at a later time.
Once you've made your selections, click Next and you'll be asked to confirm your region and duration:
Once you agree to continue, you'll get an activation notification:
Then you'll get your access key and secret key. You must accept (click "Got it!") because this the only time you'll get to see either of the keys ever. The access and secret keys are required in order to access your bucket and download your logs.
The last step is the summary screen showing the configuration and most importantly, your bucket name:
You can turn off and on logging at your convenience. However, logs will continue to be purged based on your retention duration, whether or not you are continuing to log new data.
Log Upload Failures
Bucket structure and log format
Logs are uploaded in ten-minute intervals from the Umbrella log queue to the S3 bucket. Within the first two hours after a completed configuration, you should receive your first log upload to your S3 bucket, but it's usually much faster than that-- often, nearly immediately. However, you must have newly generated log data in order for anything to be uploaded so if you're trying this on a test environment, ensure network data is being logged in the Activity Search.
To check to see if everything is working, the Last Sync time in the Umbrella dashboard should update and logs should begin to appear in your S3 bucket.
Amazon S3 > bucket-name > dnslogs
The logs will appear in a GZIP format with the following file name format. The files will also be sorted into date stamped folders:
for DNS traffic
proxied traffic (the intelligent proxy in the Advanced Settings)
for IP traffic generated from the IP Layer enforcement feature (a subfeature of the intelligent proxy).
The log format of each type of log is outlined here:
If you do not see logs in your bucket within 10 minutes, please contact support outlining the steps you've taken thus far.
Once logs do appear, we recommend reviewing the data by unzipping the contents of the first few log uploads that are received to ensure the data is viewable in a text editor (or even Microsoft Excel, often the default for .CSV). For information on which each field in the log represents read here: https://support.umbrella.com/hc/en-us/articles/231248508-Log-Management-Export-Format
In the case of a failure to upload logs from Cisco Umbrella to your S3 bucket, alerts trigger with our teams and we will begin investigation.
There are a few UI-based tools out there such as Cyberduck (https://cyberduck.io/) that allow you to look through your bucket and download chunks of data.
Most people find using a command line (CLI) to be much easier for scripting the automated download of the files to a secondary location for consumption.
To do this:
- Download the Amazon Command Line Interface (CLI): AWS Command Line Interface
- Configure the CLI, instructions can be found on Amazon's website here: Configuring the AWS CLI - AWS Command Line Interface
- Use the CLI to list the contents of an S3 bucket or download the contents: Using Amazon S3 with the AWS Command Line Interface - AWS Command Line Interface