From owner-freebsd-net@FreeBSD.ORG Wed Sep 17 18:18:45 2014 Return-Path: <owner-freebsd-net@FreeBSD.ORG> Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C1E11A70; Wed, 17 Sep 2014 18:18:45 +0000 (UTC) Received: from mail-qg0-x233.google.com (mail-qg0-x233.google.com [IPv6:2607:f8b0:400d:c04::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5BA4D2E3; Wed, 17 Sep 2014 18:18:45 +0000 (UTC) Received: by mail-qg0-f51.google.com with SMTP id e89so2475738qgf.24 for <multiple recipients>; Wed, 17 Sep 2014 11:18:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=tW86HSWxNkMFHRlYJxWkTNooCln+St7+VoNWM8Difu0=; b=nKxisU3PEeTUB01C1+BtJ9upkKS+cNHMVpdKpAkNIRoyAxL2zMWWyOppfb9rlyNsC8 pEmqzhCijGfMbdKVbUI5B3G1uJ9C2b7nDvMci/SREh+IeIP5fUE36TrZJcyGIJkzxAhR Q7QW+fL+WApAUWivnqJIQCBAG4KhICceH/Xr9x1A1sKIq6a/+BTvYW6gA8H61WhiMS6p qk2+JO3Gv0MCjNpLHekw4E/ojGKqduZKFIaw7FZ6iRh02qDLXpHyeFH4Bjdoliq3/h4l fdyVH6g0z2R1TPKy4oAJk3DOCDtHhkKL64dc3RQ/kFDaZAaTGv74OzFfx261blKjZAXM /R0g== MIME-Version: 1.0 X-Received: by 10.224.86.5 with SMTP id q5mr60394777qal.36.1410977924449; Wed, 17 Sep 2014 11:18:44 -0700 (PDT) Received: by 10.140.104.241 with HTTP; Wed, 17 Sep 2014 11:18:44 -0700 (PDT) In-Reply-To: <CAJ-VmomY7M2h2Bbuu5BXRUYFRMnKXezZfCNbNWGFHACuFwNDmg@mail.gmail.com> References: <CAO0mX5bDoCe8oRNmd+UBbr4bJcgQEzhAQKyTsyxEQciyvGTdgQ@mail.gmail.com> <CAJ-VmomY7M2h2Bbuu5BXRUYFRMnKXezZfCNbNWGFHACuFwNDmg@mail.gmail.com> Date: Wed, 17 Sep 2014 20:18:44 +0200 Message-ID: <CAO0mX5Zdt487=entnjrMjxpt+LhKTNkS=sUHsnyVDYr_PXyS7Q@mail.gmail.com> Subject: Re: [RFC] Patch to add Software/Generic Segmentation Offload (GSO) support in FreeBSD From: Stefano Garzarella <stefanogarzarella@gmail.com> To: Adrian Chadd <adrian@freebsd.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, freebsd-current <freebsd-current@freebsd.org>, Luigi Rizzo <rizzo@iet.unipi.it> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/> List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Wed, 17 Sep 2014 18:18:45 -0000 Hi Adrian, the results that I sent, regard just one flow, but I can try with two simultaneous flows and I'll send you the results. Thanks, Stefano 2014-09-17 19:27 GMT+02:00 Adrian Chadd <adrian@freebsd.org>: > Hi! > > Cool! > > How many flows were you testing with? Just one or two? > > It's for outbound, so it's not _as_ big a deal as it is for inbound, > but it'd still be nice to know. > > > -a > > > On 17 September 2014 01:27, Stefano Garzarella > <stefanogarzarella@gmail.com> wrote: > > Hi all, > > I have recently worked, during my master=E2=80=99s thesis with the supe= rvision > > of Prof. Luigi Rizzo, on a project to add GSO (Generic Segmentation > > Offload) support in FreeBSD. I will present this project at EuroBSDcon > > 2014, in Sofia (Bulgaria) on September 28, 2014. > > > > Following is a brief description of our project: > > > > The use of large frames makes network communication much less > > demanding for the CPU. Yet, backward compatibility and slow links > > requires the use of 1500 byte or smaller frames. Modern NICs with > > hardware TCP segmentation offloading (TSO) address this problem. > > However, a generic software version (GSO) provided by the OS has > > reason to exist, for use on paths with no suitable hardware, such > > as between virtual machines or with older or buggy NICs. > > > > Much of the advantage of TSO comes from crossing the network stack only > > once per (large) segment instead of once per 1500-byte frame. > > GSO does the same both for segmentation (TCP) and fragmentation (UDP) > > by doing these operations as late as possible. Ideally, this could be > done > > within the device driver, but that would require modifications to all > > drivers. > > A more convenient, similarly effective approach is to segment > > just before the packet is passed to the driver (in ether_output()). > > > > Our preliminary implementation supports TCP and UDP on IPv4/IPv6; > > it only intercepts packets large than the MTU (others are left > unchanged), > > and only when GSO is marked as enabled for the interface. > > > > Segments larger than the MTU are not split in tcp_output(), > > udp_output(), or ip_output(), but marked with a flag (contained in > > m_pkthdr.csum_flags), which is processed by ether_output() just > > before calling the device driver. > > > > ether_output(), through gso_dispatch(), splits the large frame as neede= d, > > creating headers and possibly doing checksums if not supported by > > the hardware. > > > > In experiments agains an LRO-enabled receiver (otherwise TSO/GSO > > are ineffective) we have seen the following performance, > > taken at different clock speeds (because at top speeds the > > 10G link becomes the bottleneck): > > > > > > Testing enviroment (all with Intel 10Gbit NIC) > > Sender: FreeBSD 11-CURRENT - CPU i7-870 at 2.93 GHz + Turboboost > > Receiver: Linux 3.12.8 - CPU i7-3770K at 3.50GHz + Turboboost > > Benchmark tool: netperf 2.6.0 > > > > --- TCP/IPv4 packets (checksum offloading enabled) --- > > Freq. TSO GSO none Speedup > > [GHz] [Gbps] [Gbps] [Gbps] GSO-none > > 2.93 9347 9298 8308 12 % > > 2.53 9266 9401 6771 39 % > > 2.00 9408 9294 5499 69 % > > 1.46 9408 8087 4075 98 % > > 1.05 9408 5673 2884 97 % > > 0.45 6760 2206 1244 77 % > > > > > > --- TCP/IPv6 packets (checksum offloading enabled) --- > > Freq. TSO GSO none Speedup > > [GHz] [Gbps] [Gbps] [Gbps] GSO-none > > 2.93 7530 6939 4966 40 % > > 2.53 5133 7145 4008 78 % > > 2.00 5965 6331 3152 101 % > > 1.46 5565 5180 2348 121 % > > 1.05 8501 3607 1732 108 % > > 0.45 3665 1505 651 131 % > > > > > > --- UDP/IPv4 packets (9K) --- > > Freq. GSO none Speedup > > [GHz] [Gbps] [Gbps] GSO-none > > 2.93 9440 8084 17 % > > 2.53 7772 6649 17 % > > 2.00 6336 5338 19 % > > 1.46 4748 4014 18 % > > 1.05 3359 2831 19 % > > 0.45 1312 1120 17 % > > > > > > --- UDP/IPv6 packets (9K) --- > > Freq. GSO none Speedup > > [GHz] [Gbps] [Gbps] GSO-none > > 2.93 7281 6197 18 % > > 2.53 5953 5020 19 % > > 2.00 4804 4048 19 % > > 1.46 3582 3004 19 % > > 1.05 2512 2092 20 % > > 0.45 998 826 21 % > > > > We tried to change as little as possible the network stack to add > > GSO support. To avoid changing API/ABI, we temporarily used spare > > fields in struct tcpcb (TCP Control Block) and struct ifnet to store > > some information related to GSO (enabled, max burst size, etc.). > > The code that performs the segmentation/fragmentation is contained > > in the file gso.[h|c] in sys/net. We used 4 bit in m_pkthdr.csum_flags > > (CSUM_GSO_MASK) to encode the packet type (TCP/IPv4, TCP/IPv6, etc) > > to prevent access to the TCP/IP/Ethernet headers of each packet. > > In ether_output_frame(), if the packet requires the GSO > > ((m->m_pkthdr.csum_flags & CSUM_GSO_MASK) !=3D 0), it is segmented > > or fragmented, and then they are sent to the device driver. > > > > At https://github.com/stefano-garzarella/freebsd-gso > > you can find the kernel patches for FreeBSD-current, FreeBSD > > 10-stable, FreeBSD 9-stable, a simple application (gso-stats.c) > > that prints the GSO statistics and picobsd images with GSO support. > > > > At https://github.com/stefano-garzarella/freebsd-gso-src > > you can get the FreeBSD source with GSO patch (various branch for > > FreeBSD current, 10-stable, 9-stable). > > > > Any feedbacks, comments, questions are welcome. > > > > Thank you very much, > > Stefano Garzarella > > > > > -------------------------------------------------------------------------= ----------------------------------- > > How to use GSO: > > > > - Apply the right kernel patch. > > > > - To compile the GSO support add =E2=80=98 options GSO ' to your kernel= config > file > > and > > rebuild a kernel. > > > > - To manage the GSO parameters there are some sysctls: > > - net.inet.tcp.gso - GSO enable on TCP communications (!=3D0) > > - net.inet.udp.gso - GSO enable on UDP communications (!=3D0) > > > > - for each interface: > > - net.gso.dev."ifname=E2=80=9D.max_burst - GSO burst length l= imit > > [default: IP_MAXPACKET=3D65535] > > - net.gso.dev."ifname=E2=80=9D.enable_gso - GSO enable on =E2= =80=9Cifname=E2=80=9D > > interface (!=3D0) > > > > - To show statistics: > > - make sure that the GSO_STATS macro is defined in sys/net/gso.h > > - use the simple gso-stats.c application to access the sysctl > > net.gso.stats > > that contains the address of the gsostats structure (defined in > > gso.h) > > which records the statistics. (compile with > > -I/path/to/kernel/src/patched/) > > > -------------------------------------------------------------------------= ----------------------------------- > > > > -- > > *Stefano Garzarella* > > stefano.garzarella@gmail.com > > _______________________________________________ > > freebsd-current@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to " > freebsd-current-unsubscribe@freebsd.org" > --=20 *Stefano Garzarella* stefano.garzarella@gmail.com