Part 3: NBAD Behavioral Detection

Part 1: History of NBAD
Part 2: NBAD Signature Detection

Where signature based detection is focused on a single communication (flow), behavioral detection in NBAD  monitors multiple flows originating from a single network host that when observed depict a known bad behavior. Since observation must occur over multiple flows several components must exist in a NBAD solution that are not necessary in a signature based IDS.

Calculation Components

The first data object that must exist is a host object. This object serves as a record of collective behavior and intelligence on a network host. Every monitored host on the network must have a host object to allow for proper behavioral analysis. The host object will have several counters that will be incremented by behavioral algorithms.

As an example a host that sends a single ping packet to a single destination will increase counters on “number of pings” and “number of pinged hosts” from 0 to 1. There is normally no reason to alarm at this point but when each of those counters start to get high it will hit the maximum allowable value for each of those counters and will alarm.

The strength of behavioral detection in NBAD and Network Behavior Analysis (NBA) is the number of useful behavioral detection algorithms. As the industry leader in NBAD detection, the StealthWatch System leads with more than 150+ unique behavioral algorithms.

Reconnaissance Algorithms

One area that behavioral detection excels is in reconnaissance detection. As noted in the previous example, ping scanning can be detected via NBAD behavioral algorithms.  Additional checks can include firewall denies, ICMP unreachable, TCP resets, SYN scan, and illegal flag sequences. Below note a sampling of recon algorithms included the StealthWatch System 6.5.

Botnet Algorithms

In the previous installment, signature detection of known BotNet C&C connections was discussed. In addition to catching the known botnet infections, behavioral analysis can also provide intelligence on unknown botnet infections.

When a communication is expected to last a short period of time and a lengthy flow is observed instead, an alarm for “Suspect Long Flow” can be triggered.

These flows can indicate communications to a custom, targeted C&C server.

Additionally, regular small packets going out to the Internet can flag alarms for “Beaconing Host.” This type of conversation can be connected to the “muster command” associated with botnet traffic.

DDoS Algorithms

When multiple hosts across the Internet participate in a distributed denial of service (DDoS), NBAD behavioral counters can be used to reveal the behavior.

When focusing on individual hosts in the context of DDoS, there are three notable behavioral algorithms. The first is Target Index (TI.) TI focuses on the suspicious behavior hitting a target server. As a server received more “hits” TI on that server increases. When TI exceeds the acceptable threshold, alarms on DDoS will occur.

When one or more hosts are participating in a TCP based DDoS (also see: When DDoS Happens to Good Networks) the attacking hosts will exceed counters on acceptable number of TCP SYN packets received. This will trigger an alarm for a SYN Flood.

In occasions where attacking hosts are using application DDoS such as Solaris, TCP connections with low packets per second (PPS) are counted to yield a Slow Connection Flood alarm (also see: Application DDoS Detection)

Wrap Up

NBAD monitoring of host behaviors on a network requires the use of sophisticated counters and indices. This method of advanced detection allows mature NBAD solutions to catch recon, targeted botnet infection and DDoS attacks.