From owner-freebsd-current@FreeBSD.ORG Mon Mar 5 18:23:16 2007 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0499C16A402; Mon, 5 Mar 2007 18:23:16 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.freebsd.org (Postfix) with ESMTP id CA40013C4A6; Mon, 5 Mar 2007 18:23:15 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.14.0/8.14.0) with ESMTP id l25IN8c9009002 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 5 Mar 2007 13:23:08 -0500 (EST) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id l25IMsrj056048; Mon, 5 Mar 2007 13:22:54 -0500 (EST) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17900.24574.751134.397740@grasshopper.cs.duke.edu> Date: Mon, 5 Mar 2007 13:22:54 -0500 (EST) To: Andre Oppermann In-Reply-To: <45E8276D.60105@freebsd.org> References: <45E8276D.60105@freebsd.org> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Cc: freebsd-net@freebsd.org, freebsd-current@freebsd.org, rwatson@freebsd.org, kmacy@freebsd.org Subject: Re: New optimized soreceive_stream() for TCP sockets, proof of concept X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Mar 2007 18:23:16 -0000 Andre Oppermann writes: > The patch is here: > > http://people.freebsd.org/~andre/soreceive_stream-20070302.diff > > Any testing, especially on 10Gig cards, and feedback appreciated. I just tested with my standard mxge setup (details and data below). This is *awesome*. Before your patch, the peak bandwidth was 7.5Gb/s with jumbo frames, and 2.47Gb/s with standard frames. After the patch, it is 9.0Gb/s and 2.92Gb/s. Before the patch, the bandwidth "collapses" above a certain receive socket buffer size. Eg, it is 7.5Gb/s with a 128KB socket buffer, but 6.2Gb/s with 192KB or more. Now the peak is with a 192KB socket buffer, and there is no sudden collapse above the peak! With the patch, we finally seem to be performance competative on the receive side with Linux x86_64 and Solaris/amd64 on this same hardware. Both of those OSes do much better (saturate the link with jumbos) when CPU affinity is used to bind the interrupt handler and netserver process to different cores on the same socket. I imagine FreeBSD may be able to do even better if it ever grows CPU affinity support for both interrupt handlers and processes. With the patch, it performs at least as well, if not better than, Solaris and Linux do without CPU affinity. Venice: AMD Athlon(tm) 64 Processor 3000+, 1.8GHz, 512KB L2 Venice: Linux 2.6.19, UP Rome: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+, 2.0GHz, 512KB L2 Rome: -current, SMP Motherboard (both machines): DFI Corp, LP NF4 Series Chipset (both machines): Nvidia CK804 NIC (both machines): Myri 10G-PCIE-8A-C (CX4) The tests were run from venice -> rome using netperf242 -Hrome-my -tTCP_SENDFILE -F /boot/vmlinuz-2.6.9-11.EL -C -c -- -S $SOCKETBUFFER_SIZE I took the median of 5 runs at each socket buffer size. I really need to write some scripts.. MTU 1500 Pre: 65536 65536 65536 10.00 2472.08 31.83 90.38 1.055 5.990 98304 65536 65536 10.00 2280.62 29.61 85.77 1.064 6.162 131072 65536 65536 10.00 2190.13 28.23 84.67 1.056 6.334 196608 65536 65536 10.01 2111.03 23.75 81.54 0.921 6.328 262144 65536 65536 10.01 1843.48 15.87 73.46 0.705 6.529 524288 65536 65536 10.01 1808.00 17.28 76.54 0.783 6.936 1048576 65536 65536 10.01 1919.01 15.72 81.54 0.671 6.962 Post: 65536 65536 65536 10.00 2839.09 37.42 86.92 1.080 5.016 98304 65536 65536 10.00 2923.07 36.42 86.92 1.021 4.872 131072 65536 65536 10.01 2694.91 30.74 86.92 0.934 5.285 196608 65536 65536 10.00 2640.54 27.62 86.15 0.857 5.346 262144 65536 65536 10.00 2422.32 22.23 82.69 0.752 5.593 524288 65536 65536 10.01 2168.59 16.96 80.77 0.641 6.102 1048576 65536 65536 10.01 2138.10 16.39 85.77 0.628 6.572 MTU 9000 Pre: 65536 65536 65536 10.00 4908.35 34.61 52.31 0.578 1.746 98304 65536 65536 10.00 6623.84 44.71 68.46 0.553 1.693 131072 65536 65536 10.00 7504.46 41.83 77.69 0.457 1.696 196608 65536 65536 10.00 6248.29 36.91 77.69 0.484 2.037 262144 65536 65536 10.00 6230.94 35.51 76.92 0.467 2.023 524288 65536 65536 10.00 6242.76 36.03 80.38 0.473 2.110 1048576 65536 65536 10.00 6162.54 37.03 74.62 0.492 1.984 Post: 65536 65536 65536 10.00 4957.98 36.72 49.62 0.607 1.640 98304 65536 65536 10.00 6799.78 44.61 68.08 0.537 1.640 131072 65536 65536 10.00 7998.88 46.92 73.08 0.481 1.497 196608 65536 65536 10.00 9017.95 48.92 82.69 0.444 1.502 262144 65536 65536 10.00 8874.62 49.52 81.15 0.457 1.498 524288 65536 65536 10.00 8787.09 48.12 81.92 0.449 1.527 1048576 65536 65536 10.00 8631.88 46.02 82.31 0.437 1.562 Drew PS: It didn't want to compile at first. I needed to comment out the offending 2 lines: cc -c -O2 -frename-registers -pipe -fno-strict-aliasing -std=c99 -g -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -fformat-extensions -nostdinc -I- -I. -I../../.. -I../../../contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-omit-frame-pointer -mcmodel=kernel -mno-red-zone -mfpmath=387 -mno-sse -mno-sse2 -mno-mmx -mno-3dnow -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -Werror ../../../kern/uipc_sockbuf.c ../../../kern/uipc_sockbuf.c: In function `sbpull_locked': ../../../kern/uipc_sockbuf.c:841: error: structure has no member named `sb_sndptroff' ../../../kern/uipc_sockbuf.c:842: error: structure has no member named `sb_sndptroff' *** Error code 1 Stop in /usr/var/tmp/sys/amd64/compile/ROME.