Date: Mon, 28 Sep 2009 17:46:02 +0300 From: Oleg <agile.quad@gmail.com> To: freebsd-net@freebsd.org Subject: tap device poor performance Message-ID: <d0fcb8ea0909280746u7b6acd5dhd0633395e8fdb5be@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello,
I wrote a simple program to test tap device performance.
It open 2 tap devices, configure them (with aliases):
tap64: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 00:13:ec:00:10:00
inet 192.168.64.1 netmask 0xffffff00 broadcast 192.168.64.255
inet 192.168.64.64 netmask 0xffffffff broadcast 192.168.64.64
Opened by PID 1353
tap32: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 00:13:ec:00:20:00
inet 192.168.32.1 netmask 0xffffff00 broadcast 192.168.32.255
inet 192.168.32.32 netmask 0xffffffff broadcast 192.168.32.32
Opened by PID 1353
Then read packet from one tap device, change mask to opposite
(192.168.64.0/24 <--> 192.168.32.0/24)
in the IP headers (src_addr, dst_addr), recalculate checksum and push
to opposite tap. (tap64 <--> tap32).
How to reproduce:
run tapbridge:
#./tapbridge
Then, 2 choise:
1) with help of TTCP utilitty.
TTCP (need patched version, look ports/139112,
http://www.freebsd.org/cgi/query-pr.cgi?pr=139112)
Run ttcp in listen mode:
# ttcp -r -s -n 131000
(output skipped)
On second console run ttcp in sender mode:
# ttcp -t -s -n 131000 -S 192.168.64.64 192.168.64.32
ttcp-t: buflen=8192, nbuf=131000, align=16384/0, port=5001 tcp ->
192.168.64.32
ttcp-t: socket
ttcp-t: connect
ttcp-t: 1073152000 bytes in 25.70 real seconds = 40778.27 KB/sec +++
ttcp-t: 131000 I/O calls, msec/call = 0.20, calls/sec = 5097.28
ttcp-t: 0.0user 4.5sys 0:25real 17% 26i+214d 494maxrss 0+2pf 493201+173csw
SPEED is appr. 40 MB/sec.
Now lets run ttcp in sender mode via lo0:
# ttcp -t -s -n 131000 192.168.64.64
ttcp-t: buflen=8192, nbuf=131000, align=16384/0, port=5001 tcp ->
192.168.64.64
ttcp-t: socket
ttcp-t: connect
ttcp-t: 1073152000 bytes in 2.06 real seconds = 508079.09 KB/sec +++
ttcp-t: 131000 I/O calls, msec/call = 0.02, calls/sec = 63509.89
ttcp-t: 0.0user 1.3sys 0:02real 64% 21i+180d 494maxrss 0+2pf 33559+41csw
SPEED is appr. 500 MB/sec.
2) with httpd, wget
configure httpd and run it.
Download some file:
# wget --bind-address=192.168.64.64 -O /dev/null http://192.168.64.32:81/8G
--2009-09-28 18:00:45-- http://192.168.64.32:81/8G
Connecting to 192.168.64.32:81... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8589934592 (8.0G) [text/plain]
Saving to: `/dev/null'
100%[=========================>] 8,589,934,592 40.5M/s in 3m 8s
2009-09-28 18:03:53 (43.6 MB/s) - `/dev/null' saved [8589934592/8589934592]
SPEED is appr. 43 MB/sec.
The test is gone on Intel Xeon 8 core machine with 16 GB of memory
running FreeBSD 7.0-RELEASE-p12 amd64.
The SAME SPEED results measuring on virtual machines
(FreeBSD 7.0 amd64, FreeBSD 7.2 amd64, FreeBSD 8.0 RC1 amd64).
So, where is bottleneck?
Also, I make a LOCK_PROFILING measure.
# sysctl debug.lock.prof.stats | grep if_tap
max total wait_total count avg wait_avg
cnt_hold cnt_lock name
252 1163161 64 317424 3 0 28
10 /usr/src/sys/modules/if_tap/../../net/if_tap.c:681 (sleep
mutex:tap_mtx)
109 783634 761 317426 2 0 67
74 /usr/src/sys/modules/if_tap/../../net/if_tap.c:848 (sleep
mutex:tap_mtx)
103 173087 0 69617 2 0
1 0 /usr/src/sys/modules/if_tap/../../net/if_tap.c:872
(sleep mutex:tap_mtx)
204 460785 303 190427 2 0 24
67 /usr/src/sys/modules/if_tap/../../net/if_tap.c:865 (sleep
mutex:tap32)
81 487945 0 196614 2 0
1 0 /usr/src/sys/modules/if_tap/../../net/if_tap.c:865
(sleep mutex:tap64)
1 3 0 2 1 0
0 0
/usr/src/sys/modules/if_tap/../../net/if_tap.c:493 (sleep
mutex:tap_mtx)
319 471162 509 317428 1 0 34
122 /usr/src/sys/modules/if_tap/../../net/if_tap.c:653 (sleep
mutex:tap_mtx)
Seems, that this is not a locking problem...
Who can help me?
p.s. if your downloading will stuck - you need to patch your tap
device with kern/123892
(http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/123892)
p.p.s. btw on Linux similar test give me about 70 MB/sec...
Regards,
Oleg Dolgov.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?d0fcb8ea0909280746u7b6acd5dhd0633395e8fdb5be>