Skip to content

Commit 4cbcd9a

Browse files
pizhenweipizhenwei
authored
tcprtt: add --byladdr/--byraddr (iovisor#3106)
On a server side, a server listens a local port and accepts lots of connections. To seperate histogram into views by each remote address, so add --byraddr to support this. Suggested-by: Brendan Gregg <[email protected]> Signed-off-by: zhenwei pi <[email protected]>
1 parent 561a71e commit 4cbcd9a

File tree

4 files changed

+179
-79
lines changed

4 files changed

+179
-79
lines changed

‎README.md

+1
Original file line numberDiff line numberDiff line change
@@ -158,6 +158,7 @@ pair of .c and .py files, and some are directories of files.
158158
- tools/[tcpdrop](tools/tcpdrop.py): Trace kernel-based TCP packet drops with details. [Examples](tools/tcpdrop_example.txt).
159159
- tools/[tcplife](tools/tcplife.py): Trace TCP sessions and summarize lifespan. [Examples](tools/tcplife_example.txt).
160160
- tools/[tcpretrans](tools/tcpretrans.py): Trace TCP retransmits and TLPs. [Examples](tools/tcpretrans_example.txt).
161+
- tools/[tcprtt](tools/tcprtt.py): Trace TCP round trip time. [Examples](tools/tcprtt_example.txt).
161162
- tools/[tcpstates](tools/tcpstates.py): Trace TCP session state changes with durations. [Examples](tools/tcpstates_example.txt).
162163
- tools/[tcpsubnet](tools/tcpsubnet.py): Summarize and aggregate TCP send by subnet. [Examples](tools/tcpsubnet_example.txt).
163164
- tools/[tcpsynbl](tools/tcpsynbl.py): Show TCP SYN backlog. [Examples](tools/tcpsynbl_example.txt).

‎man/man8/tcprtt.8

+21-11
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
.SH NAME
33
tcprtt \- Trace TCP RTT of established connections. Uses Linux eBPF/bcc.
44
.SH SYNOPSIS
5-
.B tcprtt [\-h] [\-T] [\-D] [\-m] [\-i INTERVAL] [\-d DURATION]
5+
.B tcprtt [\-h] [\-T] [\-D] [\-m] [\-i INTERVAL] [\-d DURATION] [\-b] [\-B]
66
.SH DESCRIPTION
77
This tool traces established connections RTT(round-trip time) to analyze the
88
quality of network. This can be useful for general troubleshooting to
@@ -31,17 +31,23 @@ Print output every interval seconds.
3131
\-d DURATION
3232
Total duration of trace in seconds.
3333
.TP
34-
\-p SPORT
35-
Filter for source port.
34+
\-p LPORT
35+
Filter for local port.
3636
.TP
37-
\-P DPORT
38-
Filter for destination port.
37+
\-P RPORT
38+
Filter for remote port.
3939
.TP
40-
\-a SADDR
41-
Filter for source address.
40+
\-a LADDR
41+
Filter for local address.
4242
.TP
43-
\-A DADDR
44-
Filter for destination address.
43+
\-A RADDR
44+
Filter for remote address.
45+
.TP
46+
\-b
47+
Show sockets histogram by local address.
48+
.TP
49+
\-B
50+
Show sockets histogram by remote address.
4551
.SH EXAMPLES
4652
.TP
4753
Trace TCP RTT and print 1 second summaries, 10 times:
@@ -52,9 +58,13 @@ Summarize in millisecond, and timestamps:
5258
#
5359
.B tcprtt \-m \-T
5460
.TP
55-
Only trace TCP RTT for destination address 192.168.1.100 and destination port 80:
61+
Only trace TCP RTT for remote address 192.168.1.100 and remote port 80:
62+
#
63+
.B tcprtt \-i 1 \-d 10 \-A 192.168.1.100 \-P 80
64+
.TP
65+
Trace local port and show a breakdown of remote hosts RTT:
5666
#
57-
.B tcprtt \-i 1 \-d 10 -A 192.168.1.100 -P 80
67+
.B tcprtt \-i 3 --lport 80 --byraddr
5868
.SH OVERHEAD
5969
This traces the kernel tcp_rcv_established function and collects TCP RTT. The
6070
rate of this depends on your server application. If it is a web or proxy server

‎tools/tcprtt.py

+96-52
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
# tcprtt Summarize TCP RTT as a histogram. For Linux, uses BCC, eBPF.
55
#
66
# USAGE: tcprtt [-h] [-T] [-D] [-m] [-i INTERVAL] [-d DURATION]
7-
# [-p SPORT] [-P DPORT] [-a SADDR] [-A DADDR]
7+
# [-p LPORT] [-P RPORT] [-a LADDR] [-A RADDR] [-b] [-B]
88
#
99
# Copyright (c) 2020 zhenwei pi
1010
# Licensed under the Apache License, Version 2.0 (the "License")
@@ -14,6 +14,7 @@
1414
from __future__ import print_function
1515
from bcc import BPF
1616
from time import sleep, strftime
17+
from socket import inet_ntop, AF_INET
1718
import socket, struct
1819
import argparse
1920

@@ -22,10 +23,12 @@
2223
./tcprtt # summarize TCP RTT
2324
./tcprtt -i 1 -d 10 # print 1 second summaries, 10 times
2425
./tcprtt -m -T # summarize in millisecond, and timestamps
25-
./tcprtt -p # filter for source port
26-
./tcprtt -P # filter for destination port
27-
./tcprtt -a # filter for source address
28-
./tcprtt -A # filter for destination address
26+
./tcprtt -p # filter for local port
27+
./tcprtt -P # filter for remote port
28+
./tcprtt -a # filter for local address
29+
./tcprtt -A # filter for remote address
30+
./tcprtt -b # show sockets histogram by local address
31+
./tcprtt -B # show sockets histogram by remote address
2932
./tcprtt -D # show debug bpf text
3033
"""
3134
parser = argparse.ArgumentParser(
@@ -40,14 +43,18 @@
4043
help="include timestamp on output")
4144
parser.add_argument("-m", "--milliseconds", action="store_true",
4245
help="millisecond histogram")
43-
parser.add_argument("-p", "--sport",
44-
help="source port")
45-
parser.add_argument("-P", "--dport",
46-
help="destination port")
47-
parser.add_argument("-a", "--saddr",
48-
help="source address")
49-
parser.add_argument("-A", "--daddr",
50-
help="destination address")
46+
parser.add_argument("-p", "--lport",
47+
help="filter for local port")
48+
parser.add_argument("-P", "--rport",
49+
help="filter for remote port")
50+
parser.add_argument("-a", "--laddr",
51+
help="filter for local address")
52+
parser.add_argument("-A", "--raddr",
53+
help="filter for remote address")
54+
parser.add_argument("-b", "--byladdr", action="store_true",
55+
help="show sockets histogram by local address")
56+
parser.add_argument("-B", "--byraddr", action="store_true",
57+
help="show sockets histogram by remote address")
5158
parser.add_argument("-D", "--debug", action="store_true",
5259
help="print BPF program before starting (for debugging purposes)")
5360
parser.add_argument("--ebpf", action="store_true",
@@ -67,65 +74,72 @@
6774
#include <net/inet_sock.h>
6875
#include <bcc/proto.h>
6976
70-
BPF_HISTOGRAM(hist_srtt);
77+
typedef struct sock_key {
78+
u64 addr;
79+
u64 slot;
80+
} sock_key_t;
81+
82+
STORAGE
7183
7284
int trace_tcp_rcv(struct pt_regs *ctx, struct sock *sk, struct sk_buff *skb)
7385
{
7486
struct tcp_sock *ts = tcp_sk(sk);
7587
u32 srtt = ts->srtt_us >> 3;
7688
const struct inet_sock *inet = inet_sk(sk);
89+
u16 sport = 0;
90+
u16 dport = 0;
91+
u32 saddr = 0;
92+
u32 daddr = 0;
93+
94+
bpf_probe_read_kernel(&sport, sizeof(sport), (void *)&inet->inet_sport);
95+
bpf_probe_read_kernel(&dport, sizeof(dport), (void *)&inet->inet_dport);
96+
bpf_probe_read_kernel(&saddr, sizeof(saddr), (void *)&inet->inet_saddr);
97+
bpf_probe_read_kernel(&daddr, sizeof(daddr), (void *)&inet->inet_daddr);
98+
99+
LPORTFILTER
100+
RPORTFILTER
101+
LADDRFILTER
102+
RADDRFILTER
77103
78-
SPORTFILTER
79-
DPORTFILTER
80-
SADDRFILTER
81-
DADDRFILTER
82104
FACTOR
83105
84-
hist_srtt.increment(bpf_log2l(srtt));
106+
STORE
85107
86108
return 0;
87109
}
88110
"""
89111

90-
# filter for source port
91-
if args.sport:
92-
bpf_text = bpf_text.replace(b'SPORTFILTER',
93-
b"""u16 sport = 0;
94-
bpf_probe_read_kernel(&sport, sizeof(sport), (void *)&inet->inet_sport);
95-
if (ntohs(sport) != %d)
96-
return 0;""" % int(args.sport))
112+
# filter for local port
113+
if args.lport:
114+
bpf_text = bpf_text.replace(b'LPORTFILTER',
115+
b"""if (ntohs(sport) != %d)
116+
return 0;""" % int(args.lport))
97117
else:
98-
bpf_text = bpf_text.replace(b'SPORTFILTER', b'')
118+
bpf_text = bpf_text.replace(b'LPORTFILTER', b'')
99119

100-
# filter for dest port
101-
if args.dport:
102-
bpf_text = bpf_text.replace(b'DPORTFILTER',
103-
b"""u16 dport = 0;
104-
bpf_probe_read_kernel(&dport, sizeof(dport), (void *)&inet->inet_dport);
105-
if (ntohs(dport) != %d)
106-
return 0;""" % int(args.dport))
120+
# filter for remote port
121+
if args.rport:
122+
bpf_text = bpf_text.replace(b'RPORTFILTER',
123+
b"""if (ntohs(dport) != %d)
124+
return 0;""" % int(args.rport))
107125
else:
108-
bpf_text = bpf_text.replace(b'DPORTFILTER', b'')
126+
bpf_text = bpf_text.replace(b'RPORTFILTER', b'')
109127

110-
# filter for source address
111-
if args.saddr:
112-
bpf_text = bpf_text.replace(b'SADDRFILTER',
113-
b"""u32 saddr = 0;
114-
bpf_probe_read_kernel(&saddr, sizeof(saddr), (void *)&inet->inet_saddr);
115-
if (saddr != %d)
116-
return 0;""" % struct.unpack("=I", socket.inet_aton(args.saddr))[0])
128+
# filter for local address
129+
if args.laddr:
130+
bpf_text = bpf_text.replace(b'LADDRFILTER',
131+
b"""if (saddr != %d)
132+
return 0;""" % struct.unpack("=I", socket.inet_aton(args.laddr))[0])
117133
else:
118-
bpf_text = bpf_text.replace(b'SADDRFILTER', b'')
134+
bpf_text = bpf_text.replace(b'LADDRFILTER', b'')
119135

120-
# filter for source address
121-
if args.daddr:
122-
bpf_text = bpf_text.replace(b'DADDRFILTER',
123-
b"""u32 daddr = 0;
124-
bpf_probe_read_kernel(&daddr, sizeof(daddr), (void *)&inet->inet_daddr);
125-
if (daddr != %d)
126-
return 0;""" % struct.unpack("=I", socket.inet_aton(args.daddr))[0])
136+
# filter for remote address
137+
if args.raddr:
138+
bpf_text = bpf_text.replace(b'RADDRFILTER',
139+
b"""if (daddr != %d)
140+
return 0;""" % struct.unpack("=I", socket.inet_aton(args.raddr))[0])
127141
else:
128-
bpf_text = bpf_text.replace(b'DADDRFILTER', b'')
142+
bpf_text = bpf_text.replace(b'RADDRFILTER', b'')
129143

130144
# show msecs or usecs[default]
131145
if args.milliseconds:
@@ -135,6 +149,30 @@
135149
bpf_text = bpf_text.replace('FACTOR', '')
136150
label = "usecs"
137151

152+
print_header = "srtt"
153+
# show byladdr/byraddr histogram
154+
if args.byladdr:
155+
bpf_text = bpf_text.replace('STORAGE',
156+
'BPF_HISTOGRAM(hist_srtt, sock_key_t);')
157+
bpf_text = bpf_text.replace('STORE',
158+
b"""sock_key_t key;
159+
key.addr = saddr;
160+
key.slot = bpf_log2l(srtt);
161+
hist_srtt.increment(key);""")
162+
print_header = "Local Address: "
163+
elif args.byraddr:
164+
bpf_text = bpf_text.replace('STORAGE',
165+
'BPF_HISTOGRAM(hist_srtt, sock_key_t);')
166+
bpf_text = bpf_text.replace('STORE',
167+
b"""sock_key_t key;
168+
key.addr = daddr;
169+
key.slot = bpf_log2l(srtt);
170+
hist_srtt.increment(key);""")
171+
print_header = "Remote Address: "
172+
else:
173+
bpf_text = bpf_text.replace('STORAGE', 'BPF_HISTOGRAM(hist_srtt);')
174+
bpf_text = bpf_text.replace('STORE', 'hist_srtt.increment(bpf_log2l(srtt));')
175+
138176
# debug/dump ebpf enable or not
139177
if args.debug or args.ebpf:
140178
print(bpf_text)
@@ -147,6 +185,12 @@
147185

148186
print("Tracing TCP RTT... Hit Ctrl-C to end.")
149187

188+
def print_section(addr):
189+
if args.byladdr:
190+
return inet_ntop(AF_INET, struct.pack("I", addr)).encode()
191+
elif args.byraddr:
192+
return inet_ntop(AF_INET, struct.pack("I", addr)).encode()
193+
150194
# output
151195
exiting = 0 if args.interval else 1
152196
dist = b.get_table("hist_srtt")
@@ -162,7 +206,7 @@
162206
if args.timestamp:
163207
print("%-8s\n" % strftime("%H:%M:%S"), end="")
164208

165-
dist.print_log2_hist(label, "srtt")
209+
dist.print_log2_hist(label, section_header=print_header, section_print_fn=print_section)
166210
dist.clear()
167211

168212
if exiting or seconds >= args.duration:

‎tools/tcprtt_example.txt

+61-16
Original file line numberDiff line numberDiff line change
@@ -40,16 +40,57 @@ also shows unstable TCP RTT. So in this situation, we need to make sure the
4040
quality of network is good or not firstly.
4141

4242

43-
Use filter for address and(or) port. Ex, only collect source address 192.168.122.200
44-
and destination address 192.168.122.100 and destination port 80.
43+
Use filter for address and(or) port. Ex, only collect local address 192.168.122.200
44+
and remote address 192.168.122.100 and remote port 80.
4545
# ./tcprtt -i 1 -d 10 -m -a 192.168.122.200 -A 192.168.122.100 -P 80
4646

4747

48+
Tracing at server side, show each clients with its own histogram.
49+
For example, run tcprtt on a storage node to show initiators' rtt histogram:
50+
# ./tcprtt -i 1 -m --lport 3260 --byraddr
51+
Tracing TCP RTT... Hit Ctrl-C to end.
52+
53+
Remote Address: = 10.131.90.16
54+
msecs : count distribution
55+
0 -> 1 : 0 | |
56+
2 -> 3 : 0 | |
57+
4 -> 7 : 0 | |
58+
8 -> 15 : 2 |****************************************|
59+
60+
Remote Address: = 10.131.90.13
61+
msecs : count distribution
62+
0 -> 1 : 0 | |
63+
2 -> 3 : 0 | |
64+
4 -> 7 : 4 |************************** |
65+
8 -> 15 : 6 |****************************************|
66+
67+
Remote Address: = 10.131.89.153
68+
msecs : count distribution
69+
0 -> 1 : 120 |****************************************|
70+
2 -> 3 : 31 |********** |
71+
4 -> 7 : 32 |********** |
72+
73+
Remote Address: = 10.131.89.150
74+
msecs : count distribution
75+
0 -> 1 : 12 |****************************************|
76+
2 -> 3 : 12 |****************************************|
77+
4 -> 7 : 9 |****************************** |
78+
8 -> 15 : 3 |********** |
79+
80+
Remote Address: = 10.131.89.148
81+
msecs : count distribution
82+
0 -> 1 : 0 | |
83+
2 -> 3 : 0 | |
84+
4 -> 7 : 4 |****************************************|
85+
86+
....
87+
88+
4889
Full USAGE:
4990

5091
# ./tcprtt -h
51-
usage: tcprtt [-h] [-i INTERVAL] [-d DURATION] [-T] [-m] [-p SPORT]
52-
[-P DPORT] [-a SADDR] [-A DADDR] [-D]
92+
usage: tcprtt.py [-h] [-i INTERVAL] [-d DURATION] [-T] [-m] [-p LPORT]
93+
[-P RPORT] [-a LADDR] [-A RADDR] [-b] [-B] [-D]
5394

5495
Summarize TCP RTT as a histogram
5596

@@ -61,23 +102,27 @@ optional arguments:
61102
total duration of trace, seconds
62103
-T, --timestamp include timestamp on output
63104
-m, --milliseconds millisecond histogram
64-
-p SPORT, --sport SPORT
65-
source port
66-
-P DPORT, --dport DPORT
67-
destination port
68-
-a SADDR, --saddr SADDR
69-
source address
70-
-A DADDR, --daddr DADDR
71-
destination address
105+
-p LPORT, --lport LPORT
106+
filter for local port
107+
-P RPORT, --rport RPORT
108+
filter for remote port
109+
-a LADDR, --laddr LADDR
110+
filter for local address
111+
-A RADDR, --raddr RADDR
112+
filter for remote address
113+
-b, --byladdr show sockets histogram by local address
114+
-B, --byraddr show sockets histogram by remote address
72115
-D, --debug print BPF program before starting (for debugging
73116
purposes)
74117

75118
examples:
76119
./tcprtt # summarize TCP RTT
77120
./tcprtt -i 1 -d 10 # print 1 second summaries, 10 times
78121
./tcprtt -m -T # summarize in millisecond, and timestamps
79-
./tcprtt -p # filter for source port
80-
./tcprtt -P # filter for destination port
81-
./tcprtt -a # filter for source address
82-
./tcprtt -A # filter for destination address
122+
./tcprtt -p # filter for local port
123+
./tcprtt -P # filter for remote port
124+
./tcprtt -a # filter for local address
125+
./tcprtt -A # filter for remote address
126+
./tcprtt -b # show sockets histogram by local address
127+
./tcprtt -B # show sockets histogram by remote address
83128
./tcprtt -D # show debug bpf text

0 commit comments

Comments
 (0)