add DNS correlation to connect tracking

Author: nabilschear

Dockerfile.tests

@@ -50,8 +50,8 @@ RUN apt-get update && apt-get install -y \
       libtinfo5 \
       libtinfo-dev
 
-RUN pip3 install pyroute2 netaddr
-RUN pip install pyroute2 netaddr
+RUN pip3 install pyroute2 netaddr dnslib cachetools
+RUN pip install pyroute2 netaddr dnslib cachetools
 
 # FIXME this is faster than building from source, but it seems there is a bug
 # in probing libruby.so rather than ruby binary

Dockerfile.ubuntu

@@ -23,5 +23,5 @@ COPY --from=builder /root/bcc/*.deb /root/bcc/
 
 RUN \
   apt-get update -y && \
-  DEBIAN_FRONTEND=noninteractive apt-get install -y python python3 binutils libelf1 kmod && \
+  DEBIAN_FRONTEND=noninteractive apt-get install -y python python3 binutils libelf1 kmod python-dnslib python-cachetools python3-dnslib python3-cachetools && \
   dpkg -i /root/bcc/*.deb

INSTALL.md

@@ -325,7 +325,7 @@ apt-get -t jessie-backports install linux-base linux-image-4.9.0-0.bpo.2-amd64 l
 apt-get install debhelper cmake libllvm3.8 llvm-3.8-dev libclang-3.8-dev \
   libelf-dev bison flex libedit-dev clang-format-3.8 python python-netaddr \
   python-pyroute2 luajit libluajit-5.1-dev arping iperf netperf ethtool \
-  devscripts zlib1g-dev libfl-dev
+  devscripts zlib1g-dev libfl-dev python-dnslib python-cachetools
 ```
 
 #### Sudo
@@ -419,7 +419,7 @@ popd
 ```
 sudo dnf install -y bison cmake ethtool flex git iperf libstdc++-static \
   python-netaddr python-pip gcc gcc-c++ make zlib-devel \
-  elfutils-libelf-devel
+  elfutils-libelf-devel python-cachetools
 sudo dnf install -y luajit luajit-devel  # for Lua support
 sudo dnf install -y \
   http://repo.iovisor.org/yum/extra/mageia/cauldron/x86_64/netperf-2.7.0-1.mga6.x86_64.rpm
@@ -548,7 +548,7 @@ Tested on Amazon Linux AMI release 2018.03 (kernel 4.14.47-56.37.amzn1.x86_64)
 # enable epel to get iperf, luajit, luajit-devel, cmake3 (cmake3 is required to support c++11)
 sudo yum-config-manager --enable epel
 
-sudo yum install -y bison cmake3 ethtool flex git iperf libstdc++-static python-netaddr gcc gcc-c++ make zlib-devel elfutils-libelf-devel
+sudo yum install -y bison cmake3 ethtool flex git iperf libstdc++-static python-netaddr python-cachetools gcc gcc-c++ make zlib-devel elfutils-libelf-devel
 sudo yum install -y luajit luajit-devel
 sudo yum install -y http://repo.iovisor.org/yum/extra/mageia/cauldron/x86_64/netperf-2.7.0-1.mga6.x86_64.rpm
 sudo pip install pyroute2
@@ -590,7 +590,7 @@ sudo /usr/share/bcc/tools/execsnoop
 # enable epel to get iperf, luajit, luajit-devel, cmake3 (cmake3 is required to support c++11)
 sudo yum-config-manager --enable epel
 
-sudo yum install -y bison cmake3 ethtool flex git iperf libstdc++-static python-netaddr gcc gcc-c++ make zlib-devel elfutils-libelf-devel
+sudo yum install -y bison cmake3 ethtool flex git iperf libstdc++-static python-netaddr python-cachetools gcc gcc-c++ make zlib-devel elfutils-libelf-devel
 sudo yum install -y luajit luajit-devel
 sudo yum install -y http://repo.iovisor.org/yum/extra/mageia/cauldron/x86_64/netperf-2.7.0-1.mga6.x86_64.rpm
 sudo pip install pyroute2

man/man8/tcpconnect.8

@@ -2,7 +2,7 @@
 .SH NAME
 tcpconnect \- Trace TCP active connections (connect()). Uses Linux eBPF/bcc.
 .SH SYNOPSIS
-.B tcpconnect [\-h] [\-c] [\-t] [\-p PID] [-P PORT] [-u UID] [-U] [\-\-cgroupmap MAPPATH] [\-\-mntnsmap MAPPATH]
+.B tcpconnect [\-h] [\-c] [\-t] [\-p PID] [-P PORT] [-u UID] [-U] [\-\-cgroupmap MAPPATH] [\-\-mntnsmap MAPPATH] [\-d]
 .SH DESCRIPTION
 This tool traces active TCP connections (eg, via a connect() syscall;
 accept() are passive connections). This can be useful for general
@@ -14,9 +14,18 @@ This works by tracing the kernel tcp_v4_connect() and tcp_v6_connect() functions
 using dynamic tracing, and will need updating to match any changes to these
 functions.
 
+When provided with the \-d or \-\-dns option, this tool will also correlate
+connect calls with the most recent DNS query that matches the IP connected.
+This feature works by tracing the kernel udp_recvmsg() function to collect DNS
+responses.
+
 Since this uses BPF, only the root user can use this tool.
 .SH REQUIREMENTS
 CONFIG_BPF and bcc.
+
+If using the \-d or \-\-dns option, you must have the
+dnslib and cachetools python packages installed.  You can install them with pip3 or with
+apt on Ubuntu 18.04+ using the python3\-dnslib and python3\-cachetools packages.
 .SH OPTIONS
 .TP
 \-h
@@ -45,6 +54,26 @@ Trace cgroups in this BPF map only (filtered in-kernel).
 .TP
 \--mntnsmap  MAPPATH
 Trace mount namespaces in this BPF map only (filtered in-kernel).
+.TP
+\-d
+Shows the most recent DNS query for the IP address in the connect call.
+This is likely related to the TCP connection details in the other columns, but is not guaranteed.
+This
+feature works by tracing the udp_recvmsg kernel function and tracking DNS
+responses received by the server.  It only supports UDP DNS packets up to 512 bytes
+in length.  The python code keeps a cache of 10k DNS responses in memory
+for up 24 hours.
+
+If the time difference in milliseconds
+between when the system received a DNS response and when a
+connect syscall was traced using an IP in that DNS response is greater than 100ms,
+this tool will report this delta after the query.
+These deltas should be relatively short for most applications.  A
+long delay between the response and connect could be either anomalous activity
+or indicate a misattribution between the DNS name requested and the IP that
+the connect syscall is using.
+
+The \-d option may not be used with the count feature (option \-c)
 .SH EXAMPLES
 .TP
 Trace all active TCP connections:
@@ -55,6 +84,10 @@ Trace all TCP connects, and include timestamps:
 #
 .B tcpconnect \-t
 .TP
+Trace all TCP connects, and include most recent matching DNS query for each connected IP
+#
+.B tcpconnect \-d
+.TP
 Trace PID 181 only:
 #
 .B tcpconnect \-p 181
@@ -110,12 +143,27 @@ Destination port
 .TP
 CONNECTS
 Accumulated active connections since start.
+.TP
+QUERY
+Shows the most recent DNS query for the IP address in the connect call.
+This is likely related to the TCP connection details in the other columns, but is not guaranteed.
 .SH OVERHEAD
 This traces the kernel tcp_v[46]_connect functions and prints output for each
 event. As the rate of this is generally expected to be low (< 1000/s), the
 overhead is also expected to be negligible. If you have an application that
 is calling a high rate of connect()s, such as a proxy server, then test and
 understand this overhead before use.
+
+If you are using the \-d option to track DNS requests, this tool will trace the
+udp_recvmsg function and generate an event for any packets from UDP port 53.
+This event contains up to 512 bytes of the UDP packet payload.
+Typical applications do not extensively use UDP, so the performance overhead of
+tracing udp_recvmsg is
+expected to be negligible,   However, if you have an application that receives
+many UDP packets, then you should test and understand the overhead of tracing
+every received UDP message.  Furthermore, performance overhead of running
+this tool on a DNS server is expected to be higher than average because all
+DNS response packets will be copied to userspace.
 .SH SOURCE
 This is from bcc.
 .IP

scripts/bpf_demo.ks.erb

@@ -35,6 +35,7 @@ kexec-tools
 cmake
 libstdc++-static
 python-netaddr
+python-cachetools
 python-futures
 %end
 

scripts/build-rpm.sh

@@ -10,6 +10,17 @@ function cleanup() {
 }
 trap cleanup EXIT
 
+# install python dependencies for test
+if [ -f os-release ]; then
+    . os-release
+fi
+if [[ $VERSION_ID -lt 30 ]]; then
+  PKGS="python3-cachetools python-cachetools"
+else
+  PKGS="python3-cachetools"
+fi
+sudo dnf install -y $PKGS
+
 mkdir $TMP/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
 
 llvmver=3.7.1