From owner-freebsd-hpc@freebsd.org Fri Dec 2 08:55:35 2016 Return-Path: Delivered-To: freebsd-hpc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 98D51C62B60 for ; Fri, 2 Dec 2016 08:55:35 +0000 (UTC) (envelope-from jpa@semihalf.com) Received: from mail-qt0-x22f.google.com (mail-qt0-x22f.google.com [IPv6:2607:f8b0:400d:c0d::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 673D91849 for ; Fri, 2 Dec 2016 08:55:35 +0000 (UTC) (envelope-from jpa@semihalf.com) Received: by mail-qt0-x22f.google.com with SMTP id c47so245866787qtc.2 for ; Fri, 02 Dec 2016 00:55:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=KUl3vjzHYvhF7XhxZQNMQ7k6orHwQ/2vuHIUD5DWSWQ=; b=BYwWzBo4oYO4EVXkSL1j74QYZqRu43/DxfDLxGhG8TbJeWDmT59+xi6tvN8DFL3ZMM nio+T1rJub6bvlaWOqQ5MuzR1WEtM9hMfeGCCuPrl8iIAM1L9L+lfSEnwUQ3NCMeyUfa QLnexIbQv54gOrFJi6Nb82oP8Es4CvQuSJ5wCBHr088QzCe0tZE1SF5ccxlBb4Z1m1ia 7Gkoo4FJK3Al43gFCl+TyG9cbvPD0d44kZts/ZldmSofYIeklnFz+1WPMD4dMbwfIUMH Ah+Y4ubDTjbMMO5n7SlaTSB/WhCD+kmuBqdYb2EHwnm74DKaQWoABbv64lmckF7IOBF7 dV0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=KUl3vjzHYvhF7XhxZQNMQ7k6orHwQ/2vuHIUD5DWSWQ=; b=CUrCpWsdEM8kkxhg61FM31dW0xZWDHOM5A+YdWrSfZjKc5brkjnkU++WxVnnR6+WI+ a4eH0lccI+TtrDHkzDsuj4TSmCBe/QBby+DgbOZua3+paEVmx7OZZF4rELkYnpjsuc6r uHNdRYuZqCak3aiJetjGO5TDcqjUJtnpYO0+L3MF6CFKliNpP6xrseNlwR+VKKx7ovLq PupzbnryNeZ5+B2cPj99usmepTkkeanXMYDCC/3bw/iuOh2P0N74cV5jY2+AbqPuZ0bM cTXJ9WIPzA6tt/EhGnK/32yyqWXbB7qBdy5Ajs3S88S3qIqyCpZJwx47OFxrSc6LVpkD TwAw== X-Gm-Message-State: AKaTC03ByS2ktRubXNLxdOzFFtE8e6zNe/8cbr+SWU8LSHJ7OEdzxqD8n84Cf/U7lb36irB4qUWxi+do9VzIKA== X-Received: by 10.200.56.20 with SMTP id q20mr37023654qtb.178.1480668934251; Fri, 02 Dec 2016 00:55:34 -0800 (PST) MIME-Version: 1.0 Received: by 10.55.38.15 with HTTP; Fri, 2 Dec 2016 00:55:03 -0800 (PST) In-Reply-To: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> References: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> From: Jakub Palider Date: Fri, 2 Dec 2016 09:55:03 +0100 Message-ID: Subject: Re: netmap, netmap-fwd, and how many M packets-per-second? To: Jordan Caraballo Cc: freebsd-net@freebsd.org, freebsd-hpc@freebsd.org, freebsd-performance@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hpc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD in High Performance Computing environments." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2016 08:55:35 -0000 Hi, I have run some performance tests but not as router, which is your case, so some information might not be relevant: 0. validate your hw stats with what you get from the output 1. Try using FLOWTABLE option in your kernel config - this may kill your machine in -CURRENT, but it's worth checking 2. MTU set to some reasonably small value (~120) shall prevent from consolidating small packets into big frame 3. disable all kernel tracing (INVARIANTS, WITNESS, etc) 4. I read you tuned the system, but make sure there are enough resources 5. Check that context switches are not killing it Regardless on the above you might want profile the system with dtrace and export results to flamegraph, this will give you an idea what going on in the system and possibly locate the cause of the problem. It's definitely worth the effort. Jakub On Fri, Dec 2, 2016 at 12:55 AM, Jordan Caraballo < jordancaraballo87@gmail.com> wrote: > Feedback and/or tips and tricks more than welcome. > > We are trying to process huge amounts of small (64 bytes) pps through a > router. So far results have not been as we expected. We have tested > FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without > netmap. Based on netmap documentation we were expecting about 5.0M pps; > alongside with the routing improvements from the freebsd routing > proposal a total of 12.0M. > > Server Description: > > Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5=C2=AD2695 CPU's, 18 cores = per > cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. > > BIOS tweaks: > > Hyperthreading (or Logical Processors) is turned off. > > Current results are shown below. Additional configurations can be given > upon request. > > Test Environment: > 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one > equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2). > > Script that execute the following on each host. > > #!/usr/local/bin/bash > # Iterate through ports and start tests > for ((i=3D1;i<=3DPORTS;i++)); do > PORT=3D$(($PORT+1)) > iperf3 -c 172.16.2.$IP -u -b10000m -i 0 -N -l$PKT -t$TIME -P$STREAMS > -p$PORT & > #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT & > done > > # FreeBSD 10.3 - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.9M 0 1.3M 194M 540k 0 57M 0 0 > 2.1M 0 1.5M 216M 556k 0 58M 0 0 > 1.8M 0 1.3M 192M 553k 0 58M 0 0 > 1.7M 0 1.1M 174M 542k 0 57M 0 0 > 1.9M 0 1.4M 204M 537k 0 56M 0 0 > 1.6M 0 1.1M 171M 550k 0 58M 0 0 > 1.6M 0 1.1M 173M 546k 0 57M 0 0 > 1.7M 0 1.1M 176M 564k 0 59M 0 0 > 2.0M 0 1.5M 212M 543k 0 57M 0 0 > 2.1M 0 1.5M 219M 557k 0 58M 0 0 > 1.9M 0 1.4M 205M 547k 0 57M 0 0 > 1.7M 0 1.2M 179M 553k 0 58M 0 0 > > # FreeBSD 11.0 - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 3.1M 0 1.8M 326M 1.3M 0 134M 0 0 > 2.6M 0 1.5M 269M 1.1M 0 116M 0 0 > 2.7M 0 1.5M 285M 1.2M 0 127M 0 0 > 2.4M 0 1.3M 257M 1.1M 0 119M 0 0 > 2.7M 0 1.5M 287M 1.3M 0 134M 0 0 > 2.5M 0 1.3M 262M 1.2M 0 127M 0 0 > 2.1M 0 1.1M 224M 1.0M 0 108M 0 0 > 2.7M 0 1.4M 285M 1.4M 0 143M 0 0 > 2.6M 0 1.3M 272M 1.3M 0 136M 0 0 > 2.5M 0 1.4M 265M 1.1M 0 120M 0 0 > # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.9M 0 849k 195M 1.0M 0 107M 0 0 > 1.9M 0 854k 196M 1.0M 0 106M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 852k 196M 1.0M 0 107M 0 0 > 1.9M 0 847k 195M 1.0M 0 107M 0 0 > 1.9M 0 836k 195M 1.0M 0 107M 0 0 > 1.9M 0 843k 195M 1.0M 0 107M 0 0 > # FreeBSD 12.0-CURRENT - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.1M 259 0 115M 1.1M 0 115M 0 0 > 1.2M 273 0 124M 1.2M 0 124M 0 0 > 1.1M 200 0 112M 1.1M 0 112M 0 0 > 1.2M 290 0 122M 1.2M 0 122M 0 0 > 1.0M 132 0 107M 1.0M 0 107M 0 0 > 1.1M 303 0 118M 1.1M 0 118M 0 0 > 1.1M 278 0 112M 1.1M 0 112M 0 0 > 1.2M 243 0 122M 1.2M 0 122M 0 0 > 1.1M 168 0 112M 1.1M 0 112M 0 0 > 1.1M 161 0 112M 1.1M 0 112M 0 0 > # FreeBSD 12.0-CURRENT + Netmap - 4 streams to 80 ports from each client > (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.4M 10 0 144M 1.4M 0 144M 0 0 > 1.5M 6 0 159M 1.5M 0 159M 0 0 > 1.4M 5 0 144M 1.4M 0 144M 0 0 > 1.5M 14 0 158M 1.5M 0 158M 0 0 > 1.4M 5 0 151M 1.4M 0 151M 0 0 > 1.4M 10 0 152M 1.4M 0 152M 0 0 > 1.4M 12 0 148M 1.4M 0 148M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.4M 23 0 151M 1.4M 0 151M 0 0 > 1.4M 11 0 151M 1.4M 0 151M 0 0 > # FreeBSD 12.0-CURRENT + Netmap + Tuning - 4 streams to 80 ports from > each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.4M 15 0 145M 1.4M 0 145M 0 0 > 1.5M 18 0 157M 1.5M 0 157M 0 0 > 1.5M 10 0 156M 1.5M 0 156M 0 0 > 1.5M 15 0 154M 1.5M 0 154M 0 0 > 1.4M 13 0 146M 1.4M 0 146M 0 0 > 1.5M 15 0 156M 1.5M 0 156M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.5M 13 0 153M 1.5M 0 153M 0 0 > 1.4M 14 0 145M 1.4M 0 145M 0 0 > 1.4M 17 0 151M 1.4M 0 151M 0 0 > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"