The MSP, MSSP, and Multi-org consoles have the ability to store the DNS, URL and IP logs of your customers offline in cloud storage. The storage is in Amazon S3 and after the logs have been uploaded, they can be downloaded and kept for compliance reasons or security analysis.

This documentation will help you understand this feature, set it up in both your Umbrella dashboard and your Amazon S3 console, and run through several options for configuration, including the duration of time you'd like the logs to be kept in S3.

Table of Contents

• Overview
  • Two Types of Umbrella Log Management
  • Getting Started

• Configuring a Self-managed S3 Bucket
  
  • Prerequisites
  • Setting up Your Amazon S3 Bucket
  • Verifying Your Amazon S3 Bucket
  • Managing the Log Lifecycle

• Configuring a Cisco-managed S3 Bucket

• Post Configuration Options
  
  • Log Upload Failures
  • Checking Uploaded Logs and Format
  • Enable Logging on a Per-customer Basis
  • Downloading Logs, Understanding the Format and Splunk / QRadar Integration 
  • How Large Will S3 Logs Be?

Overview

Umbrella for MSP, MSSP, and Multi-Org all have the ability to upload the traffic activity logs from the child organizations of the console and store those logs in the cloud. The archiving of logs is done with Amazon’s AWS S3 service. S3 is Amazon's Simple Storage Service (hence, the three S's).  This feature is sometimes referred to as 'offline storage' or 'log retention.'

Archiving of logs can be useful for several reasons, depending on your need. For some people, the exported and archived logs can then be imported into data analysis or security forensic tools, such as SIEMs. For others, having an archive of activity logs can be useful for data forensics in case of a security incident, or human resources records.

The logs are stored in a compressed (gzip) archive in CSV format. Logs are uploaded every ten minutes so there's a minimum of delay between network traffic coming from your network, being logged by Umbrella and then being available to download from S3.  

Each customer organization has their logs uploaded individually, using the orgID number from the Console to map each customer to a folder. The feature can also be enabled or disabled on a per-customer / per-organization basis.

Two Types of Umbrella Log Management

Log Management is done by uploading logs to what is called a 'bucket' (essentially a folder within AWS's S3 environment).   A bucket for your Umbrella logs can be hosted in one of two ways:

• Administered, managed and paid for by you, the company administrator.
• Administered, managed and paid for by Cisco Umbrella. 

There are pros and cons to having Cisco manage your S3 bucket.  With a Cisco-managed bucket:

• Extremely easy to setup-- it only takes a couple of minutes -- and afterward it's extremely easy to manage.
• Included in your license cost with Umbrella, effectively making it free.  Although having your own bucket is very inexpensive, the overhead of having to manage another bill to pay can be prohibitive.

By managing an S3 instance yourself:

• No limitation on how long data can be stored offline-- 30 days is the maximum when managed by Cisco.
• You can add anything to your bucket, including log files from Umbrella so the bucket can be used by another application as well.
• You can get support from directly Amazon for advanced configuration assistance, such as automation or help with command line.

For most customers the cost of maintaining a bucket is very inexpensive, but can prove to be an additional thing to manage.

Getting Started

The Log Management feature can be found in the Console under Settings > Log Management (you may need to hit the dropdown arrow):

Configuring a Self-managed S3 Bucket

Prerequisites

In order to archive logs, you must meet the following requirements:

• Full administrative access to the Cisco Umbrella MSP, MSSP, or Multi-org Console.
• A login to Amazon AWS service (http://aws.amazon.com/console/). If you don't have an account, Amazon provides free signup for S3. They do require a credit card in case your usage exceeds free plan usage.
• A bucket configured in Amazon S3 to be used for storing logs. Instructions for configuring and setting up the Amazon S3 bucket are below.

Setting up Your Amazon S3 Bucket

1. Start by signing into the AWS Console, and selecting "S3" from the list of options under Storage:

2. You should see an introduction screen welcoming you to the Amazon Simple Storage System.

3. Next, if you don't already have a bucket, you'll want to create one. Click Create Bucket.
4. Start by entering a Bucket Name.
   The bucket name must be universally unique—not just to your AWS or your Umbrella, but to all of Amazon AWS. Using something personal, such as "my-organization-name-log-bucket" can help you bypass the requirement for universally unique bucket name. The bucket name must only use lowercase letters and cannot contain spaces or periods, and must comply with DNS naming conventions. For more information on name restrictions, read: here: http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html.
   
   For more information on bucket creation, including naming, read here: https://docs.aws.amazon.com/AmazonS3/latest/UG/CreatingaBucket.html
5. Select whichever Region works best for your location and click Create. Do not copy the settings from another bucket.
6. In the next step, "Set properties", just click Next.  These can be adjusted later.
7. In the next step, "Set permissions", just click Next.  Again, these can be adjusted after the fact and we'll revisit the permissions to set up the bucket for uploading later.
8. Finalize the review process and click Create bucket:
9. Next, you will need to configure the bucket to accept uploads from the Umbrella Service. In S3, this is referred to as a bucket policy.  Click the name of your newly configured bucket and then pick the Permissions tab across the top of the interface:
10. In the UI, pick "Bucket Policy" and then you'll be asked to paste in the bucket policy.  
11. Copy and paste the JSON string below, which contains the bucket policy, to a text editor or simply paste it into the window. Substitute your exact bucket name where bucketname is specified below. Failure to do this will result in an error message.

{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::568526795995:user/logs"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::bucketname/*"
},
{
"Sid": "",
"Effect": "Deny",
"Principal": {
"AWS": "arn:aws:iam::568526795995:user/logs"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::bucketname/*"
},

{
"Sid": "",
"Effect": "Allow",
"Principal":

{ "AWS": "arn:aws:iam::568526795995:user/logs" }

,
"Action": "s3:GetBucketLocation",
"Resource": "arn:aws:s3:::bucketname"
},

{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::568526795995:user/logs"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::bucketname"
}
]
}

12. Click Save to confirm this change.

Verifying Your Amazon S3 Bucket

Step 1

1. Go back to your Umbrella Console and navigate to Settings > Log Management. 
2. Click "Amazon S3" to expand the window.
3. In the Bucket Name field, type or paste the exact bucket name you created in S3 and click Verify.
   You should receive a confirmation message in your dashboard indicating that the bucket was successfully verified.
   

If you receive an error indicating that your bucket could not be verified, recheck the syntax of the bucket name and review the configuration. If problems persist, please open a case with our support department. 

Step 2

As a secondary precaution to ensure the correct bucket was specified, Umbrella will request that you enter a unique activation token. The activation token can be obtained by revisiting your S3 bucket. As part of the verification process, a file named README_FROM_UMBRELLA.txt was uploaded from Umbrella to your Amazon S3 bucket and should appear there.

1. Download the readme file by double-clicking on it and then open it in a text editor. Within the file, there will be a unique token tying your S3 bucket to your Umbrella dashboard.
   
   
   

    

   NOTE:

   You may need to refresh your S3 bucket in the browser in order to see the README file after it's been uploaded.

    
2. Return to the Umbrella dashboard, and paste the token into the field labeled "Unique token" and click Save. At this point, the configuration is complete. To review your configuration, just click the Amazon S3 name in the Log Management section: 

Managing the Log Lifecycle

When you're using S3, you can manage the lifecycle of the data within the bucket to extend the duration of time you'd like to retain logs for. Depending on the reason you're using the external log management, the duration could be very short or very long. For instance, you may wish to simply download the logs from the S3 bucket after 24 hours and store them offline, or retain the logs indefinitely in the cloud.

By default, Amazon stores the data in a bucket indefinitely but unlimited storage does raise the cost of maintaining the bucket. For more information on S3 lifecycles, please read: https://docs.aws.amazon.com/AmazonS3/latest/UG/LifecycleConfiguration.html

To configure the lifecycle of your bucket:

1. Select Management and then click Lifecycle: 
2. Click Add a Rule, then Apply the Rule to the whole bucket (or a subfolder if you've configured it as such).
3. Select an Action on Objects, such as Delete or Archive, then select the time period and whether you'd like to use Glacier storage to help reduce your Amazon costs. (Glacier is 'cold' off-line storage, which while slower to access, is less expensive.)

4. If you'd prefer to manage logs in another method—for example on your internal backup solution—you can simply download the logs from S3 and preserve them in another way, then set your retention time to a few days.

Configuring a Cisco-managed S3 Bucket

Navigate to Settings > Log Management in your Umbrella dashboard.  There are two options: use your company-managed Amazon S3 bucket, or use a Cisco-managed Amazon S3 bucket.

Pick "Use a Cisco-managed Amazon S3 bucket" and you'll be given two new options: "Select a Region" and "Select a Retention Duration".

Select a Region

Regional endpoints are important to minimize latency when downloading logs to your servers. The regions match those available in Amazon S3, however not all regions are available-- for instance, China is not listed.

Pick the region that's closest to you from the dropdown. If you wish to change your region in the future, you will need to delete your current settings and start over.

Select a Retention Duration

The retention duration is simply 7, 14 or 30 days.  Beyond the selected time period, all data will be purged and cannot be retrieved no matter what the situation is.  We recommend a smaller time period if your ingestion cycle is regular.  The retention duration can be changed at a later time.

Once you've made your selections, click Next and you'll be asked to confirm your region and duration:

Once you agree to continue, you'll get an activation notification:

Then you'll get your access key and secret key. You must accept (click "Got it!") because this the only time you'll get to see either of the keys ever. The access and secret keys are required in order to access your bucket and download your logs.

The last step is the summary screen showing the configuration and most importantly, your bucket name:

You can turn off and on logging at your convenience. However, logs will continue to be purged based on your retention duration, whether or not you are continuing to log new data.

Post Configuration Options

Log Upload Failures

Checking Uploaded Logs and Format

Logs are uploaded in ten-minute intervals from the Umbrella log queue to the S3 buckets. Within the first two hours after a completed configuration, you should receive your first log upload to your S3 bucket, but it's usually much faster than that-- often, nearly immediately. However, you must have newly generated log data in order for anything to be uploaded so if you're trying this on a test environment, ensure network data is being logged in the Activity Search.

To check to see if everything is working, the Last Sync time in the Umbrella dashboard should update and logs should begin to appear in your S3 bucket.

Within your bucket, each customer or organization will be labelled with their org ID, so the folder structure will be:

Amazon S3/<bucket-name>/<orgID>/<subfolder>

<bucket-name> is your bucket name, <orgID> is your organization's ID, and <subfolder> will either be dnslogs, proxylogs, or iplogs, depending on the types of logs within.

For MSP and MSSP customers, the orgID will match the one in Customer Settings under each customer detail in the deployment parameters section.  Multi-org customers can gather the orgID by logging into each individual sub-org and noting the orgID in the browser url (https://dashboard.umbrella.com/o/#####/ )

The logs will appear in a GZIP format with the following format. Currently, the log format version for the MSP, MSSP, and Multi-org customers is version 1.1.

Logs are uploaded to S3 buckets in the appropriate subfolder with the following naming format.

<subfolder>/<YYYY>-<MM>-<DD>/<YYYY>-<MM>-<DD>-<hh>-<mm>-<xxxx>.csv.gz

<subfolder> will either be dnslogs, proxylogs, or iplogs, depending on the types of logs within. <xxxx> is a random string of four alphanumeric characters, which prevents duplicate file names from being overwritten.

Example:

dnslogs/2019-01-01/2019-01-01-00-00-e4e1.csv.gz

If you do not see logs in your bucket within 10 minutes, please contact support outlining the steps you've taken thus far.

Once logs do appear, we recommend reviewing the data by unzipping the contents of the first few log uploads that are received to ensure the data is viewable in a text editor (or even Microsoft Excel, often the default for .CSV). For information on which each field in the log represents read here

In the case of a failure to upload logs from Cisco Umbrella to your S3 bucket, there is a grace period of four hours during which the service will retry every 20 minutes. After four hours, a case will be opened with our Support team, who will begin an investigation as to the cause of the issue and will proactively reach out to you to let you know about the problem.

Enable Logging on a Per-customer Basis 

Out of the box, this feature is enabled for all customers unless otherwise specified. The feature can be turned off for individual customers; this is helpful if you have different service levels for customers who do have the feature.  This is under each customer's settings in the Console.   The screenshot in the previous section shows the toggle to turn it off.

It is also possible to create IAM users in Amazon and assign those IAM users to individual org's subfolders of the bucket.  By doing so, you can allow an end user access to their logs, but *only* their logs.

Downloading Logs, Understanding the Format and Splunk / QRadar Integration

In order to download the logs for retention or consumption, there are a few approaches to downloading the DNS logs from S3. We've created an article outlining a few approaches to this problem here: https://support.umbrella.com/hc/en-us/articles/231248468-How-to-Downloading-logs-from-Cisco-Umbrella-Log-Management-in-AWS-S3

You may also have a few questions about the log format and how it differs slightly from the logs that are displayed in the Umbrella dashboard. For more information about the exported log format, read this article: https://support.umbrella.com/hc/en-us/articles/231248508-Log-Management-Export-Format

Lastly, one of the primary uses of exporting DNS logs is integration with SIEM tools. Although configuration for a SIEM when dealing with logs like this can often come down to an administrator's personal preferences, we have some guidance for the most popular SIEMs.

For more information on setting up the Splunk plug-in for Amazon AWS S3 and Umbrella, read here: https://support.umbrella.com/hc/en-us/articles/230650987-Configuring-Splunk-for-use-with-Cisco-Umbrella-Log-Management-in-AWS-S3

For information about configuring IBM QRadar to pull logs from Amazon S3 and digest them, read here: https://support.umbrella.com/hc/en-us/articles/231248488-Configuring-QRadar-for-use-with-Cisco-Umbrella-Log-Management-in-AWS-S3

How Large Will S3 Logs Be?

The size of your S3 logs depends on the number of events that occur which is dependent on the volume of your DNS traffic.

You'll find the log format for the S3 Logging here:

https://support.umbrella.com/hc/en-us/articles/231248508-Log-Management-Export-Format

The example entry is 220 bytes, but the size of each log line varies based on a number of items (length of domain name, number of categories, etc). Assuming each log line is 220 bytes, a million requests would be 220 MB.

To get an estimate of how many DNS queries will be seen each day:

1. In the Umbrella dashboard, navigate to Reporting > Activity Search.
2. Under Filters, run a report for the last 24 hours and then click the Export CSV icon.
3. Open the downloaded .csv file. The number of rows (minus one for the header) is the number of DNS queries per day; multiply that by 220 bytes to get the estimate for one day.

In terms of cost, although it is variable, we find that even our most voluminous customers spend only a few dollars a month on the service.  One cost is tied to storage time and another is tied to data download from S3 to your environment.  Check with Amazon for more details.

As with any of our features, we'd love to know what you think, especially around SIEM integrations or any additional questions that aren't cover in this documentation. If you have any feedback, please let us know!