Setting up a self-managed Amazon bucket in S3

 

Prerequisites

In order to archive DNS logs, you must meet the following requirements:

• Full administrative access to the Cisco Umbrella dashboard.
• A login to Amazon AWS service (http://aws.amazon.com/console/). If you don't have an account, Amazon provides free signup for S3. They do require a credit card in case your usage exceeds free plan usage.
• A bucket configured in Amazon S3 to be used for storing logs. Instructions for configuring and setting up the Amazon S3 bucket are below.

 

Setting up Your Amazon S3 Bucket

1. Start by signing into the AWS Console, and selecting "S3 - Scalable Storage in the Cloud" from the list of options. It's in the upper left, under Storage & Content Delivery.
   
   
   
   

   You should see an introduction screen welcoming you to the Amazon Simple Storage System.

2. Next, if you don't already have a bucket, you'll want to create one. Click Create Bucket  .
3. Start by entering a Bucket Name.
   The bucket name must be universally unique—not just to your AWS or your Umbrella, but to all of Amazon AWS. Using something personal, such as "my-organization-name-log-bucket" can help you bypass the requirement for universally unique bucket name. The bucket name must only use lowercase letters and cannot contain spaces or periods, and must comply with DNS naming conventions.
   
   For more information on name restrictions, read here: http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html.
   
   For more information on bucket creation, including naming, read here: https://docs.aws.amazon.com/AmazonS3/latest/UG/CreatingaBucket.html
4. After this is complete, select whichever Region works best for your location.
   
   
5. Click Create. Next, you will need to configure the bucket to accept uploads from the Umbrella Service. In S3, this is referred to as a bucket policy.
6. Click the newly created bucket to open it.  
   
   
7. Then select Properties in the upper right-hand corner.
   
     
8. In the Properties drop-down list, select and expand Permissions. Within the Permissions drop-down, click Add bucket policy.
   
   
   
   A modal window appears.
9. At this point, you'll want to upload the preconfigured bucket policy provided in this article.
10. Copy and paste the JSON string below, which contains the bucket policy, to a text editor or simply paste it into the window. Substitute your exact bucket name where bucketname is specified below.
    
    
    

     

    IMPORTANT:

    The bucketname must be exact or the service will not accept the bucket policy and you will receive the error message "Policy has invalid resource - arn:aws:s3:::bucketname/*"

     

{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::568526795995:user/logs"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::bucketname/*"
},
{
"Sid": "",
"Effect": "Deny",
"Principal": {
"AWS": "arn:aws:iam::568526795995:user/logs"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::bucketname/*"
},

{
"Sid": "",
"Effect": "Allow",
"Principal":

{ "AWS": "arn:aws:iam::568526795995:user/logs" }

,
"Action": "s3:GetBucketLocation",
"Resource": "arn:aws:s3:::bucketname"
},

{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::568526795995:user/logs"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::bucketname"
}
]
}

11. Click Save to confirm this change.

      

 

Verifying Your Amazon S3 Bucket

Step 1

1. Go back to your Umbrella dashboard and navigate to Settings > Log Management. 
2. Click "Amazon S3" to expand the window.
3. In the Bucket Name field, type or paste the exact bucket name you created in S3 and click Verify.
   You should receive a confirmation message in your dashboard indicating that the bucket was successfully verified.
   
   

If you receive an error indicating that your bucket could not be verified, recheck the syntax of the bucket name and review the configuration. If problems persist, please open a case with our support department. 

Step 2

As a secondary precaution to ensure the correct bucket was specified, Umbrella will request that you enter a unique activation token. The activation token can be obtained by revisiting your S3 bucket. As part of the verification process, a file named README_FROM_UMBRELLA.txt was uploaded from Umbrella to your Amazon S3 bucket and should appear there.  

1. Download the readme file by double-clicking on it and then open it in a text editor. Within the file, there will be a unique token tying your S3 bucket to your Umbrella dashboard. 
   
   
   

    

   NOTE:

   You may need to refresh your S3 bucket in the browser in order to see the README file after it's been uploaded.

    
2. Return to the Umbrella dashboard, and paste the token into the field labeled "Please enter your unique token" and click Save. At this point, the configuration is complete. To review your configuration, just click the Amazon S3 name in the Log Management section of the Umbrella dashboard.
   
   
   
   

Managing the Log Lifecycle

When you're using S3, you can manage the lifecycle of the data within the bucket to extend the duration of time you'd like to retain logs for. Depending on the reason you're using the external log management, the duration could be very short or very long. For instance, you may wish to simply download the logs from the S3 bucket after 24 hours and store them offline, or retain the logs indefinitely in the cloud.

By default, Amazon stores the data in a bucket indefinitely but unlimited storage does raise the cost of maintaining the bucket. For more information on S3 lifecycles, please read: https://docs.aws.amazon.com/AmazonS3/latest/UG/LifecycleConfiguration.html

To configure the lifecycle of your bucket:

1. Select Properties and then click Lifecycle.
   
    
2. Click Add a Rule, then Apply the Rule to the whole bucket (or a subfolder if you've configured it as such).
3. Select an Action on Objects, such as Delete or Archive, then select the time period and whether you'd like to use Glacier storage to help reduce your Amazon costs. (Glacier is 'cold' off-line storage, which while slower to access, is much less expensive.)

If you'd prefer to manage logs in another method—for example on your internal backup solution—you can simply download the logs from S3 and preserve them in another way.

 

Ensuring logs are being uploaded

Logs are uploaded in ten-minute intervals from the Umbrella log queue to the S3 buckets. Within the first two hours after a completed configuration, you should receive your first log upload to your S3 bucket. To check to see if everything is working, the Last Sync time in the Umbrella dashboard should update and logs should begin to appear in your S3 bucket.

The logs will appear in a GZIP format with the following file name format. The files will also be sorted into date stamped folders:

dnslogs/<year>-<month>-<day>/<year>-<month>-<day>-<hour>-<minute>.csv.gz

for DNS traffic

proxylogs/<year>-<month>-<day>/<year>-<month>-<day>-<hour>-<minute>.csv.gz

proxied traffic (the intelligent proxy in the Advanced Settings)

iplogs/<year>-<month>-<day>/<year>-<month>-<day>-<hour>-<minute>.csv.gz

for IP traffic generated from the IP Layer enforcement feature (a subfeature of the intelligent proxy).

If you do not see logs in your bucket within 10 minutes, please contact support outlining the steps you've taken thus far. 

Once logs do appear, we recommend reviewing the data by unzipping the contents of the first few log uploads that are received to ensure the data is viewable in a text editor (or even Microsoft Excel, often the default for .CSV). For information on which each field in the log represents read here: https://support.umbrella.com/hc/en-us/articles/231248508-Log-Management-Export-Format

  

Log Upload Failures

In the case of a failure to upload logs from Cisco Umbrella to your S3 bucket, the service will retry every 10 minutes and failures will be reported in the dashboard.

 

Downloading the Logs, understanding the Format and Splunk / QRadar Integration

In order to download the logs for retention or consumption, there are a few approaches to downloading the DNS logs from S3. We've created an article outlining a few approaches to this problem here: https://support.umbrella.com/hc/en-us/articles/231248468-How-to-Downloading-logs-from-Cisco-Umbrella-Log-Management-in-AWS-S3

You may also have a few questions about the log format and how it differs slightly from the logs that are displayed in the Umbrella dashboard. For more information about the exported log format, read this article: https://support.umbrella.com/hc/en-us/articles/231248508-Log-Management-Export-Format 

Lastly, one of the primary uses of exporting DNS logs is integration with SIEM tools. Although configuration for a SIEM when dealing with logs like this can often come down to an administrator's personal preferences, we have some guidance for the most popular SIEMs.

For more information on setting up the Splunk plug-in for Amazon AWS S3 and Umbrella, read here: https://support.umbrella.com/hc/en-us/articles/230650987-Configuring-Splunk-for-use-with-Cisco-Umbrella-Log-Management-in-AWS-S3

For information about configuring IBM QRadar to pull logs from Amazon S3 and digest them, read here: https://support.umbrella.com/hc/en-us/articles/231248488-Configuring-QRadar-for-use-with-Cisco-Umbrella-Log-Management-in-AWS-S3

As with any of our features, we'd love to know what you think, especially around SIEM integrations or any additional questions that aren't cover in this documentation. If you have any feedback, please let us know!

 

How big are my S3 logs going to be?

The size of your S3 logs depends on the number of events that occur which is dependent on the volume of your DNS traffic.

You'll find the log format for the S3 Logging here:

https://support.umbrella.com/hc/en-us/articles/231248508-Log-Management-Export-Format

The example entry is 220 bytes, but the size of each log line varies based on a number of items (length of domain name, number of categories, etc). Assuming each log line is 220 bytes, a million requests would be 220 MB.

To get an estimate of how many DNS queries will be seen each day:

1. In the Umbrella dashboard, navigate to Reporting > Activity Search.
2. Under Filters, run a report for the last 24 hours and then click the Export CSV icon. 
3. Open the downloaded .csv file. The number of rows (minus one for the header) is the number of DNS queries per day; multiply that by 220 bytes to get the estimate for one day.