From owner-freebsd-net@FreeBSD.ORG Mon Sep 28 15:14:39 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22E731065672 for ; Mon, 28 Sep 2009 15:14:39 +0000 (UTC) (envelope-from agile.quad@gmail.com) Received: from mail-bw0-f227.google.com (mail-bw0-f227.google.com [209.85.218.227]) by mx1.freebsd.org (Postfix) with ESMTP id 9850E8FC12 for ; Mon, 28 Sep 2009 15:14:38 +0000 (UTC) Received: by bwz27 with SMTP id 27so3236912bwz.43 for ; Mon, 28 Sep 2009 08:14:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=VU9Ski5H7ePOgYjkPS3SeSWtkmUxh2rvRCOcoVPtmkI=; b=EppyzkTTskUIRvApNI+FTPopi1FpI6+5BI3K3lAwf8ygoD2OZ7G1Khd/ATjCw/fioi M6P5g19IgpLF1qk+iX+SNWl9SHEez1l7r8ryQ1Jls1r+IA7TY2tWo9tj7dYmPAFUIA5r FJOPgXYuSRfOqvfY8js9EUjt65MDMpxfj/Q5c= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=c3oFGh7eMIsa2lLElcrUkVtxQn1U7vQMhnOvDkavg/f0K4toIPxFjI0GS8t93njM7x 3GbUJ5V7xDBzKgdigc0SouCF5Kavlwx6V0VrYigCLNh4IzceL+fzP2AhtBkYusbxS73J LHQC1HWtfkPnVQnDDtKVtfbAdXHL0jUrptGug= MIME-Version: 1.0 Received: by 10.204.155.79 with SMTP id r15mr3139858bkw.142.1254149162092; Mon, 28 Sep 2009 07:46:02 -0700 (PDT) Date: Mon, 28 Sep 2009 17:46:02 +0300 Message-ID: From: Oleg To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: tap device poor performance X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Sep 2009 15:14:39 -0000 Hello, I wrote a simple program to test tap device performance. It open 2 tap devices, configure them (with aliases): tap64: flags=8843 metric 0 mtu 1500 ether 00:13:ec:00:10:00 inet 192.168.64.1 netmask 0xffffff00 broadcast 192.168.64.255 inet 192.168.64.64 netmask 0xffffffff broadcast 192.168.64.64 Opened by PID 1353 tap32: flags=8843 metric 0 mtu 1500 ether 00:13:ec:00:20:00 inet 192.168.32.1 netmask 0xffffff00 broadcast 192.168.32.255 inet 192.168.32.32 netmask 0xffffffff broadcast 192.168.32.32 Opened by PID 1353 Then read packet from one tap device, change mask to opposite (192.168.64.0/24 <--> 192.168.32.0/24) in the IP headers (src_addr, dst_addr), recalculate checksum and push to opposite tap. (tap64 <--> tap32). How to reproduce: run tapbridge: #./tapbridge Then, 2 choise: 1) with help of TTCP utilitty. TTCP (need patched version, look ports/139112, http://www.freebsd.org/cgi/query-pr.cgi?pr=139112) Run ttcp in listen mode: # ttcp -r -s -n 131000 (output skipped) On second console run ttcp in sender mode: # ttcp -t -s -n 131000 -S 192.168.64.64 192.168.64.32 ttcp-t: buflen=8192, nbuf=131000, align=16384/0, port=5001 tcp -> 192.168.64.32 ttcp-t: socket ttcp-t: connect ttcp-t: 1073152000 bytes in 25.70 real seconds = 40778.27 KB/sec +++ ttcp-t: 131000 I/O calls, msec/call = 0.20, calls/sec = 5097.28 ttcp-t: 0.0user 4.5sys 0:25real 17% 26i+214d 494maxrss 0+2pf 493201+173csw SPEED is appr. 40 MB/sec. Now lets run ttcp in sender mode via lo0: # ttcp -t -s -n 131000 192.168.64.64 ttcp-t: buflen=8192, nbuf=131000, align=16384/0, port=5001 tcp -> 192.168.64.64 ttcp-t: socket ttcp-t: connect ttcp-t: 1073152000 bytes in 2.06 real seconds = 508079.09 KB/sec +++ ttcp-t: 131000 I/O calls, msec/call = 0.02, calls/sec = 63509.89 ttcp-t: 0.0user 1.3sys 0:02real 64% 21i+180d 494maxrss 0+2pf 33559+41csw SPEED is appr. 500 MB/sec. 2) with httpd, wget configure httpd and run it. Download some file: # wget --bind-address=192.168.64.64 -O /dev/null http://192.168.64.32:81/8G --2009-09-28 18:00:45-- http://192.168.64.32:81/8G Connecting to 192.168.64.32:81... connected. HTTP request sent, awaiting response... 200 OK Length: 8589934592 (8.0G) [text/plain] Saving to: `/dev/null' 100%[=========================>] 8,589,934,592 40.5M/s in 3m 8s 2009-09-28 18:03:53 (43.6 MB/s) - `/dev/null' saved [8589934592/8589934592] SPEED is appr. 43 MB/sec. The test is gone on Intel Xeon 8 core machine with 16 GB of memory running FreeBSD 7.0-RELEASE-p12 amd64. The SAME SPEED results measuring on virtual machines (FreeBSD 7.0 amd64, FreeBSD 7.2 amd64, FreeBSD 8.0 RC1 amd64). So, where is bottleneck? Also, I make a LOCK_PROFILING measure. # sysctl debug.lock.prof.stats | grep if_tap max total wait_total count avg wait_avg cnt_hold cnt_lock name 252 1163161 64 317424 3 0 28 10 /usr/src/sys/modules/if_tap/../../net/if_tap.c:681 (sleep mutex:tap_mtx) 109 783634 761 317426 2 0 67 74 /usr/src/sys/modules/if_tap/../../net/if_tap.c:848 (sleep mutex:tap_mtx) 103 173087 0 69617 2 0 1 0 /usr/src/sys/modules/if_tap/../../net/if_tap.c:872 (sleep mutex:tap_mtx) 204 460785 303 190427 2 0 24 67 /usr/src/sys/modules/if_tap/../../net/if_tap.c:865 (sleep mutex:tap32) 81 487945 0 196614 2 0 1 0 /usr/src/sys/modules/if_tap/../../net/if_tap.c:865 (sleep mutex:tap64) 1 3 0 2 1 0 0 0 /usr/src/sys/modules/if_tap/../../net/if_tap.c:493 (sleep mutex:tap_mtx) 319 471162 509 317428 1 0 34 122 /usr/src/sys/modules/if_tap/../../net/if_tap.c:653 (sleep mutex:tap_mtx) Seems, that this is not a locking problem... Who can help me? p.s. if your downloading will stuck - you need to patch your tap device with kern/123892 (http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/123892) p.p.s. btw on Linux similar test give me about 70 MB/sec... Regards, Oleg Dolgov.