Guide to Active Directory Connector Performance

Overview

The Umbrella Connector service is used to monitor User/Computer login events as part of our Active Directory integration. The OpenDNS Connector service reads login information from the Security Event Log of each AD Domain Controller in it's Site.

In environments with a high frequency of user login events it is important to review these performance guidelines.  For accuate user identification the Connector service must be able to retrieve login information quickly.

Maximum Events/second

There is no hard limit on the number of events that can be processed.   The Umbrella Connector service is tested to support a continuous 850 events per second across all Domain Controllers in a "Site". This is based on a dedicated lab environment with no third-party software running.  Real world results may differ based on network latency and other bottlenecks.

Customers can determine an approximate number of events/s by reading Appendix A.

Performance Recommendations

Connector Sizing

The server running the Active Directory Connector service should have CPU and Memory resources as specified in our Sizing Guide.

Dedicated Connector

Although the Connector service can be installed on a Domain Controller directly, Cisco Umbrella recommend that the Connector is installed on a member server dedicated to the Connector service.  This member server should have no other third-party software installed.  The installation process is described here:  https://docs.umbrella.com/deployment-umbrella/docs/5-connect-active-directory-to-umbrella#section-install-the-connector

Umbrella Sites

Where possible, Umbrella deployments should be segregated into "Sites" that restrict which components communicate across the network.  The Connector service will only communicate with components in the same Umbrella site.  This feature should always be used when users have a deployment distributed over large geographical areas.

Typically an Umbrella site is created for each physical location.  Umbrella sites should follow the rules in this article:  https://docs.umbrella.com/deployment-umbrella/docs/appx-b-multiple-ad-sites-and-sites-for-umbrella

Proper use of Umbrella sites can greatly improve the deployment and prevent components communicating over the Wide Area Network. 

Network Latency

Login events will be transferred to the Connector across the network.  It is important that there is high-speed connection between the Connector and each Domain Controller to reduce network-related delays.   The Connector should be positioned as close possible to the Domain Controller(s) and Virtual Appliance(s). 

Number of Connectors

One Connector is required for each Umbrella site.  Having multiple Connectors in an Umbrella site is possible, but is only required for redundancy purposes. Having additional Connectors places extra load on the Domain Controllers as they are duplicating the same function as the first Connector.  We recommend a maximum of 2 Connectors for each Umbrella site.

Event Log Size

Large Windows Security Event logs can have an adverse impact on the performance of this WMI operation.  We recommend to limit the event log size.  The best performance is found with a log file < 512MB, however, this should be adjusted in line with your log retention requirements.  The log file size can be tuned as follows:

• Open the Event Viewer application (eventvwr.msc)
• Go to Windows Logs > System
• Right-click on the System log and select 'Properties'.
• Tune the maximum log file size as desired and click OK

Third Party Software

A number of other software products also utilize WMI which can create a bottleneck in WMI on the Domain Controller.  This can include:

• Third-Party Security / Analytic software which monitors event logs
• Windows Event Log Forwarding
• SIEM integration and other software which monitors event logs

If any of this software is no longer required we recommend to disable it.  Alternatively this issue can be mitigated using the 'Direct Event Log Reader Connection' method described in the Appendix.

Anti-Virus Software

Exclude the following folder and executables from Anti-Virus scanning:

C:\Program Files (x86)\OpenDNS\OpenDNS Connector
C:\Program Files (x86)\OpenDNS\OpenDNS Connector\OpenDNSAuditService.exe
C:\Program Files (x86)\OpenDNS\OpenDNS Connector\<VERSION>OpenDNSAuditClient.exe

Additional Domain Controllers

The WMI notification system on the Domain Controller queues and processes each Event Log entry, and sends them to WMI subscribers.  This is effectively a push mechanism where the events are sent by the DC.  As such there can be a performance bottleneck on the Domain Controller itself raffecting how quickly events are sent.

This bottleneck can be mitigated by adding additional Domain Controllers to your AD environment.  Umbrella has tested a single Domain Controller up to 850 events/s.

Service Account Exceptions

Reduce the number of AD logins detected by Umbrella by excluding Service accounts - These accounts must be excluded anyway for correct policy application.  You can also exclude servers and other devices which are not using AD User policies but might have a high volume of user logons.

WMI Patches

Please ensure the Domain Controller and connector server are up to date with the latest Microsoft patches.  Examples of hotfixes which resolve known WMI performance issues are here:
https://support.umbrella.com/hc/en-us/articles/115006117508-What-Microsoft-patches-are-required-for-reliable-WMI-connections-

WMI Memory and Handle Limits

WMI contains it's own internal limits which may create a bottleneck.  This is particularly true when other software is also performing intensive WMI operations. An example of how to increase these limits is found here:
https://blogs.technet.microsoft.com/bulentozkir/2014/01/14/increase-wmi-quota-properties-to-maximum-values/

Umbrella support are unable to advise the correct limits for your environment. Please contact Microsoft for assistance.

*NEW* DC Load Balancing

Umbrella now support a load-balancing feature which is useful when a site has multiple domain controllers and a large number of logon events.  In this scenario additional Connectors are installed and Domain Controllers are then assigned to a Connector via a Load-Balancing group.

In a simple environment, Load Balancing would work as follows:

• DC_A and DC_B are assigned to load-balancing Group_1 which is handled by Connector_1.
• DC_C and DC_D are assigned to load-balancing Group_2 which is handled by Connector_2.
• Virtual Appliances still receive events from both Connectors so are still aware of all logon events.
• If redundancy is required an additional Connector can be installed in each Load Balancing Group. 

This feature has the following benefits:

• The workload of each Connector is greatly reduced.   Each Connector is handling a smaller number of Domain Controllers.  
• This typically helps in scenarios where there is a high delay receiving events from a DC.

Load Balancing can scale up to be used in complex multi-site environments with many Domain Controllers.  There is no drawback to using Load Balancing beyond the installation of additional Connectors.

At this time the Load Balancing feature must be enabled by Umbrella support.  Please contact us to discuss your requirements.

*NEW* Virtual Appliance Parallel Communication

The Connector is now able to send login events to multiple Virtual Appliances in parallel, rather than using the default serial method.  This is useful when a site has multiple virtual appliances and a large number of logon events.

This feature has the following benefits:

• Minimizes any delay in sending login information when there are multiple appliances.  An event can be sent to all appliances at once.
• Prevents a communication issue or outage with one appliance having a knock-on effect for other appliances.  A separate event queue is maintained for each.

This feature is now enabled automatically, but only when the server meets the CPU and memory recommendations.

*NEW* Accelerated Transmission of User Login Events

The Connector is now able to transmit User Login Events in Batches which significantly increases the number of events per second which can be sent to the Virtual Appliance (per-second).  This is particularly important for connectors communicating with virtual appliances at remote locations.

This feature will now be enabled automatically but has the following requirements:

• Parallel Communication (above) must be enabled.  The server must meet the CPU and memory recommendations.
• ADC Version 1.8+ Required
• Connector Version 3.2.0+ Required

*NEW* Direct Event Log Reader Connection 

 Version 1.4+ of the Active Directory connector supports a new method to connect directly to the Security Event Log of the Domain Controller(s) without using a WMI query.  This cuts out WMI as a 'middle man' and significantly improves performance in cases where WMI is a bottleneck.    This is particularly useful in scenarios where individual domain controllers are processing a large number of login events.

This feature works using a pull mechanism where the Connector pulls new events every 5 seconds, as such there is a short (eg. 5 second) delay in the correct user being identified.  

This optimization is now enabled by default.  For more information on this feature please contact Umbrella support.

Appendix A:  Events Per Second

It is possible to count the number of recent events on a Domain Controller to estimate the events per second.  This should be done at peak time:

• Open the Event Viewer application (eventvwr.msc)
• Go to Windows Logs > System
• Select 'Filter current log' and choose events logged in Last hour
• Click OK
• Once the filter has loaded, the Event log will show the number of events in the last hour
• This value can be divided by 3600 to estimate the events per second