The growing complexity and dynamism of modern networks make anomaly detection a challenging problem, especially in large-scale environments with non-stationary traffic. Many existing methods rely on high-resolution packet-level data, limiting their use in real-world deployments that primarily generate aggregated data like NetFlow records. To address this, we introduce a scalable, lightweight approach for detecting behavioral shifts in network traffic using NetFlow-based monitoring data. Our method integrates principal component analysis (PCA) for dimensionality reduction, time series decomposition to isolate trends, and the Pruned Exact Linear Time (PELT) algorithm for detecting mean shifts. We evaluate its effectiveness in a large-scale nationwide enterprise network in Germany, demonstrating its ability to identify change points in traffic. Additionally, we assess the expressiveness of various features in detecting these shifts. This repository provides a detailed description of our features.
We compute temporal features for each 5-minute interval, resulting in a d-dimensional feature vector for every interval. By aggregating data across all relevant subnets, we shift from a per-subnet perspective to an application-level view. In total, we extract 116 features, which are organized into eight categories. It’s important to note that a record, in this context, represents communication between two subnets within a 5-minute window, using a specific protocol. Consequently, for each 5-minute interval, we may have thousands of records, from which we derive our features.
| ID | Feature Name | Description |
|---|---|---|
| 1 | n | Number of records |
| 2 | flows | Number of flows |
| 3 | bytes | Number of bytes |
| 4 | packets | Number of packets |
| 5 | in_n | Number of ingress records (from any subnet to a VDS subnet) |
| 6 | in_flows | Number of ingress flows |
| 7 | in_packets | Number of ingress packets |
| 8 | in_bytes | Number of ingress bytes |
| 9 | out_n | Number of egress records (from a VDS subnet to any subnet) |
| 10 | out_flows | Number of egress flows |
| 11 | out_packets | Number of egress packets |
| 12 | out_bytes | Number of egress bytes |
| ID | Feature Name | Description |
|---|---|---|
| 13 | n_protocols | Number of protocols |
| 14 | n_protocols_ingress | Number of ingress protocols |
| 15 | n_protocols_egress | Number of egress protocols |
| 16 | dom_prot_share_in | Share of ingress flows of the dominating protocol in total ingress flows |
| 17 | dom_prot_share_out | Share of egress flows of the dominating protocol in total egress flows |
| 18 | dom_prot_share_total | Share of flows of the dominating protocol in total flows |
| ID | Feature Name | Description |
|---|---|---|
| 19 | n_receiving_subnets | Number of subnets receiving data from VDS |
| 20 | n_sending_subnets | Number of subnets sending data to VDS |
| ID | Feature Name | Description |
|---|---|---|
| 21 | TCP_n_in | Number of TCP ingress records |
| 22 | TCP_bytes_in | Number of TCP ingress bytes |
| 23 | TCP_packets_in | Number of TCP ingress packets |
| 24 | TCP_flows_in | Number of TCP ingress flows |
| 25 | TCP_n_share_in | Share of TCP ingress records in total ingress records |
| 26 | TCP_bytes_share_in | Share of TCP ingress bytes in total ingress bytes |
| 27 | TCP_packets_share_in | Share of TCP ingress packets in total ingress packets |
| 28 | TCP_flows_share_in | Share of TCP ingress flows in total ingress flows |
| 29 | TCP_n_out | Number of TCP egress records |
| 30 | TCP_bytes_out | Number of TCP egress bytes |
| 31 | TCP_packets_out | Number of TCP egress packets |
| 32 | TCP_flows_out | Number of TCP egress flows |
| 33 | TCP_n_share_out | Share of TCP egress records in total egress records |
| 34 | TCP_bytes_share_out | Share of TCP egress bytes in total egress bytes |
| 35 | TCP_packets_share_out | Share of TCP egress packets in total egress packets |
| 36 | TCP_flows_share_out | Share of TCP egress flows in total egress flows |
| ID | Feature Name | Description |
|---|---|---|
| 37 | UDP_n_in | Number of UDP ingress records |
| 38 | UDP_bytes_in | Number of UDP ingress bytes |
| 39 | UDP_packets_in | Number of UDP ingress packets |
| 40 | UDP_flows_in | Number of UDP ingress flows |
| 41 | UDP_n_share_in | Share of UDP ingress records in total ingress records |
| 42 | UDP_bytes_share_in | Share of UDP ingress bytes in total ingress bytes |
| 43 | UDP_packets_share_in | Share of UDP ingress packets in total ingress packets |
| 44 | UDP_flows_share_in | Share of UDP ingress flows in total ingress flows |
| 45 | UDP_n_out | Number of UDP egress records |
| 46 | UDP_bytes_out | Number of UDP egress bytes |
| 47 | UDP_packets_out | Number of UDP egress packets |
| 48 | UDP_flows_out | Number of UDP egress flows |
| 49 | UDP_n_share_out | Share of UDP egress records in total egress records |
| 50 | UDP_bytes_share_out | Share of UDP egress bytes in total egress bytes |
| 51 | UDP_packets_share_out | Share of UDP egress packets in total egress packets |
| 52 | UDP_flows_share_out | Share of UDP egress flows in total egress flows |
| ID | Feature Name | Description |
|---|---|---|
| 53 | Other_n_in | Number of Other ingress records |
| 54 | Other_bytes_in | Number of Other ingress bytes |
| 55 | Other_packets_in | Number of Other ingress packets |
| 56 | Other_flows_in | Number of Other ingress flows |
| 57 | Other_n_share_in | Share of Other ingress records in total ingress records |
| 58 | Other_bytes_share_in | Share of Other ingress bytes in total ingress bytes |
| 59 | Other_packets_share_in | Share of Other ingress packets in total ingress packets |
| 60 | Other_flows_share_in | Share of Other ingress flows in total ingress flows |
| 61 | Other_n_out | Number of Other egress records |
| 62 | Other_bytes_out | Number of Other egress bytes |
| 63 | Other_packets_out | Number of Other egress packets |
| 64 | Other_flows_out | Number of Other egress flows |
| 65 | Other_n_share_out | Share of Other egress records in total egress records |
| 66 | Other_bytes_share_out | Share of Other egress bytes in total egress bytes |
| 67 | Other_packets_share_out | Share of Other egress packets in total egress packets |
| 68 | Other_flows_share_out | Share of Other egress flows in total egress flows |
| ID | Feature Name | Description |
|---|---|---|
| 69 | mean_flows | Mean number of flows per record |
| 70 | mean_bytes | Mean number of bytes per record |
| 71 | mean_packets | Mean number of packets per record |
| 72 | sd_flows | Standard deviation of flows per record |
| 73 | sd_bytes | Standard deviation of bytes per record |
| 74 | sd_packets | Standard deviation of packets per record |
| 75 | mean_in_flows | Mean number of flows per ingress record |
| 76 | mean_in_bytes | Mean number of bytes per ingress record |
| 77 | mean_in_packets | Mean number of packets per ingress record |
| 78 | sd_in_flows | Standard deviation of flows per ingress record |
| 79 | sd_in_bytes | Standard deviation of bytes per ingress record |
| 80 | sd_in_packets | Standard deviation of packets per ingress record |
| 81 | mean_out_flows | Mean number of flows per egress record |
| 82 | mean_out_bytes | Mean number of bytes per egress record |
| 83 | mean_out_packets | Mean number of packets per egress record |
| 84 | sd_out_flows | Standard deviation of flows per egress record |
| 85 | sd_out_bytes | Standard deviation of bytes per egress record |
| 86 | sd_out_packets | Standard deviation of packets per egress record |
| 87 | mean_packets_per_in_flows | Mean packets per ingress flows (Calculated by taking the mean of the ratio of packets to flows for each record) |
| 88 | sd_packets_per_in_flows | Standard deviation of packets per ingress flows |
| 89 | mean_packets_per_out_flows | Mean packets per egress flows (Calculated by taking the mean of the ratio of packets to flows for each record) |
| 90 | sd_packets_per_out_flows | Standard deviation of packets per egress flows |
| 91 | mean_bytes_per_in_packets | Mean bytes per ingress packets (Calculated by taking the mean of the ratio of bytes to packets for each record) |
| 92 | sd_bytes_per_in_packets | Standard deviation of bytes per ingress packets |
| 93 | mean_bytes_per_out_packets | Mean bytes per egress packets (Calculated by taking the mean of the ratio of bytes to packets for each record) |
| 94 | sd_bytes_per_out_packets | Standard deviation of bytes per egress packets |
| 95 | Other_mean_flows_in | Mean of Other ingress flows per Other protocol |
| 96 | Other_mean_bytes_in | Mean of Other ingress bytes per Other protocol |
| 97 | Other_mean_packets_in | Mean of Other ingress packets per Other protocol |
| 98 | Other_mean_flows_out | Mean of Other egress flows per Other protocol |
| 99 | Other_mean_bytes_out | Mean of Other egress bytes per Other protocol |
| 100 | Other_mean_packets_out | Mean of Other egress packets per Other protocol |
| 101 | TCP_mean_flows_in | Mean of TCP ingress flows per TCP protocol |
| 102 | TCP_mean_bytes_in | Mean of TCP ingress bytes per TCP protocol |
| 103 | TCP_mean_packets_in | Mean of TCP ingress packets per TCP protocol |
| 104 | TCP_mean_flows_out | Mean of TCP egress flows per TCP protocol |
| 105 | TCP_mean_bytes_out | Mean of TCP egress bytes per TCP protocol |
| 106 | TCP_mean_packets_out | Mean of TCP egress packets per TCP protocol |
| 107 | UDP_mean_flows_in | Mean of UDP ingress flows per UDP protocol |
| 108 | UDP_mean_bytes_in | Mean of UDP ingress bytes per UDP protocol |
| 109 | UDP_mean_packets_in | Mean of UDP ingress packets per UDP protocol |
| 110 | UDP_mean_flows_out | Mean of UDP egress flows per UDP protocol |
| 111 | UDP_mean_bytes_out | Mean of UDP egress bytes per UDP protocol |
| 112 | UDP_mean_packets_out | Mean of UDP egress packets per UDP protocol |
| ID | Feature Name | Description |
|---|---|---|
| 113 | sym_coeff | Symmetry coefficient of records: Specifies the ratio between ingress records and egress records. Value between -1 and 1, where 0 represents a perfectly symmetrical ratio. |
| 114 | sym_coeff_flows | Symmetry coefficient of flows: Specifies the ratio between ingress flows and egress flows. Value between -1 and 1, where 0 represents a perfectly symmetrical ratio. |
| 115 | sym_coeff_bytes | Symmetry coefficient of bytes: Specifies the ratio between ingress bytes and egress bytes. Value between -1 and 1, where 0 represents a perfectly symmetrical ratio. |
| 116 | sym_coeff_packets | Symmetry coefficient of packets: Specifies the ratio between ingress packets and egress packets. Value between -1 and 1, where 0 represents a perfectly symmetrical ratio. |