From owner-freebsd-performance@freebsd.org Thu Dec 1 23:56:00 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BBE92C6246E; Thu, 1 Dec 2016 23:56:00 +0000 (UTC) (envelope-from jordancaraballo87@gmail.com) Received: from mail-qk0-x241.google.com (mail-qk0-x241.google.com [IPv6:2607:f8b0:400d:c09::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6E23C175B; Thu, 1 Dec 2016 23:55:57 +0000 (UTC) (envelope-from jordancaraballo87@gmail.com) Received: by mail-qk0-x241.google.com with SMTP id x190so28498601qkb.0; Thu, 01 Dec 2016 15:55:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=to:from:subject:message-id:date:user-agent:mime-version :content-transfer-encoding; bh=qjaNJl1tNUlvHHPfgb6DniaPFb0SeNfUaHLVG4mocFM=; b=W7MVe7/JGx3+Ww+vElPNIOBEa+1q4XsNRUkfBUtMYKI2AybK9sKlYCwiG2q5m8+Nnq 6HrVYfXd1TcMtrr9cZLWHX8LY2doKmH+55FqAYeswqdsiwlk29GSeB0VOd0f0Dp6KFuB qcYF4Xa19L+ZD8hfjc0zEMK6tPXNtKSpksTFytTm6qGU0f4cpebYpaYwNkY9kCFep6le iqEzlXa7P0bUyRGVPqJd9/ajSGf6qp9J/U3kbIgGnpSIXiOz6bjpDVxdxSWpvzM3caN9 Ujth4gBevxtGBZsvDJpirmFrohsY4td3BXECRxRysP1yAay52EW5EecIkNO7KPvTzzaX BZGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-transfer-encoding; bh=qjaNJl1tNUlvHHPfgb6DniaPFb0SeNfUaHLVG4mocFM=; b=NeFm0S2xBNw/4DcIxLgdXzJbH0rzt/A1yxzSIskJQ3fMjp+byiIS3QiB0ogncwMi38 abgxjaX5+yGfUYSIZbWM7PBUA2/xB83JT7hkiiGK5BirxUusxq8oEox+ZghfjYn5pB+G 8IVtdn7qeezuI2IlCtzvU+RiU1fqs4CmX1XPaO32zwBvFe823nwVlcJHrCjSD/Umsmdv MCcgot0BqxbnirzY+EMn1q9UTJfGEJ1G00v5VZWiSGyAC/hehJJZ/ZIS0rdS8yk5cI8f 8dDLkURGaoi36PHKAF+ozusFTKRb0WkDj0DVJlC5XJ0JIhJXoNxi8URWLktBRUkJm+aZ bkYg== X-Gm-Message-State: AKaTC00Q3yjMss+vj364rtdOalsl4KR26aC5cWc16UaiHO/pmNSZiTDhNEADrDfqM+C32Q== X-Received: by 10.55.116.135 with SMTP id p129mr40035180qkc.54.1480636556636; Thu, 01 Dec 2016 15:55:56 -0800 (PST) Received: from [192.168.1.5] (pool-96-241-162-208.washdc.fios.verizon.net. [96.241.162.208]) by smtp.gmail.com with ESMTPSA id 30sm1323246qth.14.2016.12.01.15.55.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 01 Dec 2016 15:55:55 -0800 (PST) To: freebsd-net@freebsd.org, freebsd-hpc@freebsd.org, freebsd-performance@freebsd.org From: Jordan Caraballo Subject: netmap, netmap-fwd, and how many M packets-per-second? Message-ID: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> Date: Thu, 1 Dec 2016 19:55:55 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Fri, 02 Dec 2016 00:26:38 +0000 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2016 23:56:00 -0000 Feedback and/or tips and tricks more than welcome. We are trying to process huge amounts of small (64 bytes) pps through a router. So far results have not been as we expected. We have tested FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without netmap. Based on netmap documentation we were expecting about 5.0M pps; alongside with the routing improvements from the freebsd routing proposal a total of 12.0M. Server Description: Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5­2695 CPU's, 18 cores per cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. BIOS tweaks: Hyperthreading (or Logical Processors) is turned off. Current results are shown below. Additional configurations can be given upon request. Test Environment: 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2). Script that execute the following on each host. #!/usr/local/bin/bash # Iterate through ports and start tests for ((i=1;i<=PORTS;i++)); do PORT=$(($PORT+1)) iperf3 -c 172.16.2.$IP -u -b10000m -i 0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT & #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT & done # FreeBSD 10.3 - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 1.9M 0 1.3M 194M 540k 0 57M 0 0 2.1M 0 1.5M 216M 556k 0 58M 0 0 1.8M 0 1.3M 192M 553k 0 58M 0 0 1.7M 0 1.1M 174M 542k 0 57M 0 0 1.9M 0 1.4M 204M 537k 0 56M 0 0 1.6M 0 1.1M 171M 550k 0 58M 0 0 1.6M 0 1.1M 173M 546k 0 57M 0 0 1.7M 0 1.1M 176M 564k 0 59M 0 0 2.0M 0 1.5M 212M 543k 0 57M 0 0 2.1M 0 1.5M 219M 557k 0 58M 0 0 1.9M 0 1.4M 205M 547k 0 57M 0 0 1.7M 0 1.2M 179M 553k 0 58M 0 0 # FreeBSD 11.0 - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 3.1M 0 1.8M 326M 1.3M 0 134M 0 0 2.6M 0 1.5M 269M 1.1M 0 116M 0 0 2.7M 0 1.5M 285M 1.2M 0 127M 0 0 2.4M 0 1.3M 257M 1.1M 0 119M 0 0 2.7M 0 1.5M 287M 1.3M 0 134M 0 0 2.5M 0 1.3M 262M 1.2M 0 127M 0 0 2.1M 0 1.1M 224M 1.0M 0 108M 0 0 2.7M 0 1.4M 285M 1.4M 0 143M 0 0 2.6M 0 1.3M 272M 1.3M 0 136M 0 0 2.5M 0 1.4M 265M 1.1M 0 120M 0 0 # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 1.9M 0 849k 195M 1.0M 0 107M 0 0 1.9M 0 854k 196M 1.0M 0 106M 0 0 1.9M 0 851k 196M 1.0M 0 107M 0 0 1.9M 0 851k 196M 1.0M 0 107M 0 0 1.9M 0 851k 196M 1.0M 0 107M 0 0 1.9M 0 852k 196M 1.0M 0 107M 0 0 1.9M 0 847k 195M 1.0M 0 107M 0 0 1.9M 0 836k 195M 1.0M 0 107M 0 0 1.9M 0 843k 195M 1.0M 0 107M 0 0 # FreeBSD 12.0-CURRENT - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 1.1M 259 0 115M 1.1M 0 115M 0 0 1.2M 273 0 124M 1.2M 0 124M 0 0 1.1M 200 0 112M 1.1M 0 112M 0 0 1.2M 290 0 122M 1.2M 0 122M 0 0 1.0M 132 0 107M 1.0M 0 107M 0 0 1.1M 303 0 118M 1.1M 0 118M 0 0 1.1M 278 0 112M 1.1M 0 112M 0 0 1.2M 243 0 122M 1.2M 0 122M 0 0 1.1M 168 0 112M 1.1M 0 112M 0 0 1.1M 161 0 112M 1.1M 0 112M 0 0 # FreeBSD 12.0-CURRENT + Netmap - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 1.4M 10 0 144M 1.4M 0 144M 0 0 1.5M 6 0 159M 1.5M 0 159M 0 0 1.4M 5 0 144M 1.4M 0 144M 0 0 1.5M 14 0 158M 1.5M 0 158M 0 0 1.4M 5 0 151M 1.4M 0 151M 0 0 1.4M 10 0 152M 1.4M 0 152M 0 0 1.4M 12 0 148M 1.4M 0 148M 0 0 1.5M 9 0 155M 1.5M 0 155M 0 0 1.4M 23 0 151M 1.4M 0 151M 0 0 1.4M 11 0 151M 1.4M 0 151M 0 0 # FreeBSD 12.0-CURRENT + Netmap + Tuning - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 1.4M 15 0 145M 1.4M 0 145M 0 0 1.5M 18 0 157M 1.5M 0 157M 0 0 1.5M 10 0 156M 1.5M 0 156M 0 0 1.5M 15 0 154M 1.5M 0 154M 0 0 1.4M 13 0 146M 1.4M 0 146M 0 0 1.5M 15 0 156M 1.5M 0 156M 0 0 1.5M 9 0 155M 1.5M 0 155M 0 0 1.5M 13 0 153M 1.5M 0 153M 0 0 1.4M 14 0 145M 1.4M 0 145M 0 0 1.4M 17 0 151M 1.4M 0 151M 0 0 From owner-freebsd-performance@freebsd.org Fri Dec 2 01:00:08 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6D9DC607E8; Fri, 2 Dec 2016 01:00:08 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-lf0-x243.google.com (mail-lf0-x243.google.com [IPv6:2a00:1450:4010:c07::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1CF651649; Fri, 2 Dec 2016 01:00:08 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-lf0-x243.google.com with SMTP id p100so20624877lfg.2; Thu, 01 Dec 2016 17:00:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=MDxyQiSYQFdmFFoNBZfre7b611ahJL++XZefXqGXG5A=; b=DGkYmLGy5yWW+mBF1+M6g4UB2dfrVCloRXfdWSLws26fVfGJjjW1x2c96I3Ub5nqw9 Hh5YdFzBjZZ7y2u+amwlvlS6KrscvP2KUlfo8+gkTHLv+oYM5pzFbJjyJYTKehsq+5yV reHVsdvlbIkXc/1wFkRp5+aUErX1rzLprwbhrHACStLjm2xqo23uh+wI5L1o9HEUFV+y 3jnDmktUSKQw6a7Jcd9enBcnu3HICebB4ci9xc+cHHAJ1zCjyC3DMm67UQPx/NuQu+ti w5dGdUcIcKDqkL0YPt7qHZy5AWkDQCYnfubNkDLY+9ijFLUzAGITXzT8YFfHyQZ/e4U7 7ZXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=MDxyQiSYQFdmFFoNBZfre7b611ahJL++XZefXqGXG5A=; b=Io8ravib8V2naRdjb3j+FP99uuthJI68oiBYNWQudV1Tr1lgn7upiiTHUdq1/H6uSX A7tVX1Vz6OLsETW57KMD4BJpWeI1PytocwKkVbhKmCIcgxgxsUtCeuwFbxD2BhnT3OFa a8pQgudAJvKOHcOMPVCfpyFf7qsLxvNJuqwDoPs6n7AWdauA6ylcRC4A2zHETeIvLGo1 Bth2jYLtuQdf4cM+qtci9uTFBR/54G5WBsmvE4RQUBZViPQSnZgTXbNCSNUu5TFdE3v4 qnxXUeivPgEOUf5u9ECjJtahX3HsPdIiEYhduS7mLTVA3jY7OSclF50WLgWA2QsrJTo6 Uq5w== X-Gm-Message-State: AKaTC03t1eSeBex/QyXI5/CIkxIBnsYLzhYwyea7vdz2gyUwzQ761hH/ms/x8Tp3pdN/4OucV49/v1eOmtCmow== X-Received: by 10.25.130.149 with SMTP id e143mr15864736lfd.87.1480640406254; Thu, 01 Dec 2016 17:00:06 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.208.141 with HTTP; Thu, 1 Dec 2016 17:00:05 -0800 (PST) In-Reply-To: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> References: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> From: Navdeep Parhar Date: Thu, 1 Dec 2016 17:00:05 -0800 Message-ID: Subject: Re: netmap, netmap-fwd, and how many M packets-per-second? To: Jordan Caraballo Cc: "freebsd-net@freebsd.org" , freebsd-hpc@freebsd.org, freebsd-performance@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2016 01:00:08 -0000 How have you configured netmap-fwd? If you provide details on how the router or firewall is setup I can try similar experiments here. On Thu, Dec 1, 2016 at 3:55 PM, Jordan Caraballo wrote: > Feedback and/or tips and tricks more than welcome. > > We are trying to process huge amounts of small (64 bytes) pps through a > router. So far results have not been as we expected. We have tested > FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without > netmap. Based on netmap documentation we were expecting about 5.0M pps; > alongside with the routing improvements from the freebsd routing > proposal a total of 12.0M. > > Server Description: > > Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5=C2=AD2695 CPU's, 18 cores = per > cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. > > BIOS tweaks: > > Hyperthreading (or Logical Processors) is turned off. > > Current results are shown below. Additional configurations can be given > upon request. > > Test Environment: > 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one > equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2). > > Script that execute the following on each host. > > #!/usr/local/bin/bash > # Iterate through ports and start tests > for ((i=3D1;i<=3DPORTS;i++)); do > PORT=3D$(($PORT+1)) > iperf3 -c 172.16.2.$IP -u -b10000m -i 0 -N -l$PKT -t$TIME -P$STREAMS > -p$PORT & > #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT & > done > > # FreeBSD 10.3 - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.9M 0 1.3M 194M 540k 0 57M 0 0 > 2.1M 0 1.5M 216M 556k 0 58M 0 0 > 1.8M 0 1.3M 192M 553k 0 58M 0 0 > 1.7M 0 1.1M 174M 542k 0 57M 0 0 > 1.9M 0 1.4M 204M 537k 0 56M 0 0 > 1.6M 0 1.1M 171M 550k 0 58M 0 0 > 1.6M 0 1.1M 173M 546k 0 57M 0 0 > 1.7M 0 1.1M 176M 564k 0 59M 0 0 > 2.0M 0 1.5M 212M 543k 0 57M 0 0 > 2.1M 0 1.5M 219M 557k 0 58M 0 0 > 1.9M 0 1.4M 205M 547k 0 57M 0 0 > 1.7M 0 1.2M 179M 553k 0 58M 0 0 > > # FreeBSD 11.0 - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 3.1M 0 1.8M 326M 1.3M 0 134M 0 0 > 2.6M 0 1.5M 269M 1.1M 0 116M 0 0 > 2.7M 0 1.5M 285M 1.2M 0 127M 0 0 > 2.4M 0 1.3M 257M 1.1M 0 119M 0 0 > 2.7M 0 1.5M 287M 1.3M 0 134M 0 0 > 2.5M 0 1.3M 262M 1.2M 0 127M 0 0 > 2.1M 0 1.1M 224M 1.0M 0 108M 0 0 > 2.7M 0 1.4M 285M 1.4M 0 143M 0 0 > 2.6M 0 1.3M 272M 1.3M 0 136M 0 0 > 2.5M 0 1.4M 265M 1.1M 0 120M 0 0 > # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.9M 0 849k 195M 1.0M 0 107M 0 0 > 1.9M 0 854k 196M 1.0M 0 106M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 852k 196M 1.0M 0 107M 0 0 > 1.9M 0 847k 195M 1.0M 0 107M 0 0 > 1.9M 0 836k 195M 1.0M 0 107M 0 0 > 1.9M 0 843k 195M 1.0M 0 107M 0 0 > # FreeBSD 12.0-CURRENT - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.1M 259 0 115M 1.1M 0 115M 0 0 > 1.2M 273 0 124M 1.2M 0 124M 0 0 > 1.1M 200 0 112M 1.1M 0 112M 0 0 > 1.2M 290 0 122M 1.2M 0 122M 0 0 > 1.0M 132 0 107M 1.0M 0 107M 0 0 > 1.1M 303 0 118M 1.1M 0 118M 0 0 > 1.1M 278 0 112M 1.1M 0 112M 0 0 > 1.2M 243 0 122M 1.2M 0 122M 0 0 > 1.1M 168 0 112M 1.1M 0 112M 0 0 > 1.1M 161 0 112M 1.1M 0 112M 0 0 > # FreeBSD 12.0-CURRENT + Netmap - 4 streams to 80 ports from each client = (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.4M 10 0 144M 1.4M 0 144M 0 0 > 1.5M 6 0 159M 1.5M 0 159M 0 0 > 1.4M 5 0 144M 1.4M 0 144M 0 0 > 1.5M 14 0 158M 1.5M 0 158M 0 0 > 1.4M 5 0 151M 1.4M 0 151M 0 0 > 1.4M 10 0 152M 1.4M 0 152M 0 0 > 1.4M 12 0 148M 1.4M 0 148M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.4M 23 0 151M 1.4M 0 151M 0 0 > 1.4M 11 0 151M 1.4M 0 151M 0 0 > # FreeBSD 12.0-CURRENT + Netmap + Tuning - 4 streams to 80 ports from > each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.4M 15 0 145M 1.4M 0 145M 0 0 > 1.5M 18 0 157M 1.5M 0 157M 0 0 > 1.5M 10 0 156M 1.5M 0 156M 0 0 > 1.5M 15 0 154M 1.5M 0 154M 0 0 > 1.4M 13 0 146M 1.4M 0 146M 0 0 > 1.5M 15 0 156M 1.5M 0 156M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.5M 13 0 153M 1.5M 0 153M 0 0 > 1.4M 14 0 145M 1.4M 0 145M 0 0 > 1.4M 17 0 151M 1.4M 0 151M 0 0 > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-performance@freebsd.org Fri Dec 2 01:57:57 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9D31C61F06 for ; Fri, 2 Dec 2016 01:57:57 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 96CC015C4 for ; Fri, 2 Dec 2016 01:57:57 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (unknown [10.1.1.2]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 4415ED699 for ; Fri, 2 Dec 2016 01:57:50 +0000 (UTC) Subject: Re: netmap, netmap-fwd, and how many M packets-per-second? To: freebsd-performance@freebsd.org References: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> From: Allan Jude Message-ID: Date: Thu, 1 Dec 2016 20:57:44 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="A5NBumTpSC0H133TEWGpOh5wh2R2wnvC3" X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2016 01:57:57 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --A5NBumTpSC0H133TEWGpOh5wh2R2wnvC3 Content-Type: multipart/mixed; boundary="dlN5p3kIwoAhIs69TnRdLn95HeM226HHo"; protected-headers="v1" From: Allan Jude To: freebsd-performance@freebsd.org Message-ID: Subject: Re: netmap, netmap-fwd, and how many M packets-per-second? References: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> In-Reply-To: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> --dlN5p3kIwoAhIs69TnRdLn95HeM226HHo Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 2016-12-01 18:55, Jordan Caraballo wrote: > Feedback and/or tips and tricks more than welcome. >=20 > We are trying to process huge amounts of small (64 bytes) pps through a= > router. So far results have not been as we expected. We have tested > FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without > netmap. Based on netmap documentation we were expecting about 5.0M pps;= > alongside with the routing improvements from the freebsd routing > proposal a total of 12.0M. >=20 > Server Description: >=20 > Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5=C2=AD2695 CPU's, 18 core= s per > cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. >=20 > BIOS tweaks: >=20 > Hyperthreading (or Logical Processors) is turned off. >=20 > Current results are shown below. Additional configurations can be given= > upon request. >=20 > Test Environment: > 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one > equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2). >=20 > Script that execute the following on each host. >=20 > #!/usr/local/bin/bash > # Iterate through ports and start tests > for ((i=3D1;i<=3DPORTS;i++)); do > PORT=3D$(($PORT+1)) > iperf3 -c 172.16.2.$IP -u -b10000m -i 0 -N -l$PKT -t$TIME -P$STREAM= S > -p$PORT & > #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT & > done >=20 > # FreeBSD 10.3 - 4 streams to 80 ports from each client (5) >=20 > input (Total) output > packets errs idrops bytes packets errs bytes colls dr= ops > 1.9M 0 1.3M 194M 540k 0 57M 0 0 > 2.1M 0 1.5M 216M 556k 0 58M 0 0 > 1.8M 0 1.3M 192M 553k 0 58M 0 0 > 1.7M 0 1.1M 174M 542k 0 57M 0 0 > 1.9M 0 1.4M 204M 537k 0 56M 0 0 > 1.6M 0 1.1M 171M 550k 0 58M 0 0 > 1.6M 0 1.1M 173M 546k 0 57M 0 0 > 1.7M 0 1.1M 176M 564k 0 59M 0 0 > 2.0M 0 1.5M 212M 543k 0 57M 0 0 > 2.1M 0 1.5M 219M 557k 0 58M 0 0 > 1.9M 0 1.4M 205M 547k 0 57M 0 0 > 1.7M 0 1.2M 179M 553k 0 58M 0 0 >=20 > # FreeBSD 11.0 - 4 streams to 80 ports from each client (5) >=20 > input (Total) output > packets errs idrops bytes packets errs bytes colls dr= ops > 3.1M 0 1.8M 326M 1.3M 0 134M 0 0 > 2.6M 0 1.5M 269M 1.1M 0 116M 0 0 > 2.7M 0 1.5M 285M 1.2M 0 127M 0 0 > 2.4M 0 1.3M 257M 1.1M 0 119M 0 0 > 2.7M 0 1.5M 287M 1.3M 0 134M 0 0 > 2.5M 0 1.3M 262M 1.2M 0 127M 0 0 > 2.1M 0 1.1M 224M 1.0M 0 108M 0 0 > 2.7M 0 1.4M 285M 1.4M 0 143M 0 0 > 2.6M 0 1.3M 272M 1.3M 0 136M 0 0 > 2.5M 0 1.4M 265M 1.1M 0 120M 0 0 > # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5) >=20 > input (Total) output > packets errs idrops bytes packets errs bytes colls dr= ops > 1.9M 0 849k 195M 1.0M 0 107M 0 0 > 1.9M 0 854k 196M 1.0M 0 106M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 852k 196M 1.0M 0 107M 0 0 > 1.9M 0 847k 195M 1.0M 0 107M 0 0 > 1.9M 0 836k 195M 1.0M 0 107M 0 0 > 1.9M 0 843k 195M 1.0M 0 107M 0 0 > # FreeBSD 12.0-CURRENT - 4 streams to 80 ports from each client (5) >=20 > input (Total) output > packets errs idrops bytes packets errs bytes colls dr= ops > 1.1M 259 0 115M 1.1M 0 115M 0 0 > 1.2M 273 0 124M 1.2M 0 124M 0 0 > 1.1M 200 0 112M 1.1M 0 112M 0 0 > 1.2M 290 0 122M 1.2M 0 122M 0 0 > 1.0M 132 0 107M 1.0M 0 107M 0 0 > 1.1M 303 0 118M 1.1M 0 118M 0 0 > 1.1M 278 0 112M 1.1M 0 112M 0 0 > 1.2M 243 0 122M 1.2M 0 122M 0 0 > 1.1M 168 0 112M 1.1M 0 112M 0 0 > 1.1M 161 0 112M 1.1M 0 112M 0 0 > # FreeBSD 12.0-CURRENT + Netmap - 4 streams to 80 ports from each clien= t > (5) >=20 > input (Total) output > packets errs idrops bytes packets errs bytes colls dr= ops > 1.4M 10 0 144M 1.4M 0 144M 0 0 > 1.5M 6 0 159M 1.5M 0 159M 0 0 > 1.4M 5 0 144M 1.4M 0 144M 0 0 > 1.5M 14 0 158M 1.5M 0 158M 0 0 > 1.4M 5 0 151M 1.4M 0 151M 0 0 > 1.4M 10 0 152M 1.4M 0 152M 0 0 > 1.4M 12 0 148M 1.4M 0 148M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.4M 23 0 151M 1.4M 0 151M 0 0 > 1.4M 11 0 151M 1.4M 0 151M 0 0 > # FreeBSD 12.0-CURRENT + Netmap + Tuning - 4 streams to 80 ports from > each client (5) >=20 > input (Total) output > packets errs idrops bytes packets errs bytes colls dr= ops > 1.4M 15 0 145M 1.4M 0 145M 0 0 > 1.5M 18 0 157M 1.5M 0 157M 0 0 > 1.5M 10 0 156M 1.5M 0 156M 0 0 > 1.5M 15 0 154M 1.5M 0 154M 0 0 > 1.4M 13 0 146M 1.4M 0 146M 0 0 > 1.5M 15 0 156M 1.5M 0 156M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.5M 13 0 153M 1.5M 0 153M 0 0 > 1.4M 14 0 145M 1.4M 0 145M 0 0 > 1.4M 17 0 151M 1.4M 0 151M 0 0 >=20 > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" You might try pktgen, the netmap tool for generating packets, rather than iperf, as your source of traffic. Olivier Cochard-Labb=C3=A9 has a series of benchmarks on github in his 'netbenches' project https://github.com/ocochard/netbenches/blob/master/Xeon_E5-2650-8Cores-Ch= elsio_T540-CR/fastforwarding-pf-ipfw/results/fbsd12-head.r299288.D8526/RE= ADME.md This shows FreeBSD 12 @r299288 doing ~ 2.5 M pps of forwarding (so recv and then resend) in a stock configuration, and reaching almost 5 million pps after a patch that has since been committed to -CURRENT. --=20 Allan Jude --dlN5p3kIwoAhIs69TnRdLn95HeM226HHo-- --A5NBumTpSC0H133TEWGpOh5wh2R2wnvC3 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJYQNUdAAoJEBmVNT4SmAt+FUAQAMuDmX1M0z5Z+s3EgYRBMsEt Tygqe4BvlzYVNJTnreIDZR3gJt28dMhdxoT1EHvpEi5L/1bcbMM5f0CDQhf4E/XF M1mpSgXlx9CsNrNDZlVEKOLDwZapRBoAn8qxw46vfVrZlwO8DYqW+vQBPBCNiPQb 1LcjBE4H+cAamx9Q2PE8CjZOhXunkbiqh1jAAho6kSbmg0jG7sxkytcGWsYn5s/f dAUVvb5XgJtU6In8s74E3PG2qsi7c6EDUu5zWIbczkf+ecEjxQ4uXlEsB0K0yRYp oD/KI/SMMJ7P9yu0f5v1DQeDuKm38k6hetxkbNIW2nDfF1/KzdYsRMnytSjqt1GT pbjR1vWCFahUE2XpG2KUHBw3fxdXBk1BvOAV7Z3kupSXRHPXPuiS1bFEV+aMYdIZ FW8JzhhyCOMOpPfLkmlcA5qKpfL4+mkytaRiWNudJFrx0gbErTNVDgv5Hnx/Z9Tp h+OnX1qNimxhylYSXyYgp63RGy/lPFQ1mxoJZe/GT19j4aaOZKG/LakPDdakwccy 25zSy3lGekDLKgdIJX3/Wisqm9Fo5VHlplqWvHt7uLV38WS3hNS9G87FqKzZdfw+ vJB8IbG+ZxR/y93pStHzG3gKbwZ/sqL+FR0zAZI+0IRc4PhfHb5oqsp29NZc1b1R OiF0v8e964tNkWoBuySx =IiJE -----END PGP SIGNATURE----- --A5NBumTpSC0H133TEWGpOh5wh2R2wnvC3-- From owner-freebsd-performance@freebsd.org Fri Dec 2 08:55:35 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A4483C62B62 for ; Fri, 2 Dec 2016 08:55:35 +0000 (UTC) (envelope-from jpa@semihalf.com) Received: from mail-qt0-x22e.google.com (mail-qt0-x22e.google.com [IPv6:2607:f8b0:400d:c0d::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 673641847 for ; Fri, 2 Dec 2016 08:55:35 +0000 (UTC) (envelope-from jpa@semihalf.com) Received: by mail-qt0-x22e.google.com with SMTP id c47so245866786qtc.2 for ; Fri, 02 Dec 2016 00:55:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=KUl3vjzHYvhF7XhxZQNMQ7k6orHwQ/2vuHIUD5DWSWQ=; b=BYwWzBo4oYO4EVXkSL1j74QYZqRu43/DxfDLxGhG8TbJeWDmT59+xi6tvN8DFL3ZMM nio+T1rJub6bvlaWOqQ5MuzR1WEtM9hMfeGCCuPrl8iIAM1L9L+lfSEnwUQ3NCMeyUfa QLnexIbQv54gOrFJi6Nb82oP8Es4CvQuSJ5wCBHr088QzCe0tZE1SF5ccxlBb4Z1m1ia 7Gkoo4FJK3Al43gFCl+TyG9cbvPD0d44kZts/ZldmSofYIeklnFz+1WPMD4dMbwfIUMH Ah+Y4ubDTjbMMO5n7SlaTSB/WhCD+kmuBqdYb2EHwnm74DKaQWoABbv64lmckF7IOBF7 dV0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=KUl3vjzHYvhF7XhxZQNMQ7k6orHwQ/2vuHIUD5DWSWQ=; b=MBG0Bt6+Uj5TBrC2DOtuISqkMRX66yEbtDi/YFcZRm/kgvtA0W3d4CfM5e3v66424O jWTF3sJ9i/AZ1CPl89jcTgz3uz8ZsEqtjxftTPS0z3cCrRA8ybnv2DAgYVhF03QBmPRG g/nHJUh2vQho0yl0bMueVS/XPEhsJJfvuI2MGWLWpiw3w5c911vxsZN4SFT4d14y+sqZ yctAxG6HonH9bhgUAcdgmOTqFPlbT/2yWW9Nok0PRGY3kugeP9hQLwDMwOJTa29qqIxs 3HE2sz4vW4uR5k4+Twf2l1OoGj2eJJ3HcOmZ/ymPfhu/L1ynUE5faZZvLeyI1e7boov2 +xaA== X-Gm-Message-State: AKaTC00F2yyb/ep3UNGv0OXTZaqoxbIP/mmynfXfalbhLexdMGP72z+ODEDtWpyr57pTCMy/U1oAL2q3HYBSeA== X-Received: by 10.200.56.20 with SMTP id q20mr37023654qtb.178.1480668934251; Fri, 02 Dec 2016 00:55:34 -0800 (PST) MIME-Version: 1.0 Received: by 10.55.38.15 with HTTP; Fri, 2 Dec 2016 00:55:03 -0800 (PST) In-Reply-To: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> References: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> From: Jakub Palider Date: Fri, 2 Dec 2016 09:55:03 +0100 Message-ID: Subject: Re: netmap, netmap-fwd, and how many M packets-per-second? To: Jordan Caraballo Cc: freebsd-net@freebsd.org, freebsd-hpc@freebsd.org, freebsd-performance@freebsd.org X-Mailman-Approved-At: Fri, 02 Dec 2016 12:25:56 +0000 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2016 08:55:35 -0000 Hi, I have run some performance tests but not as router, which is your case, so some information might not be relevant: 0. validate your hw stats with what you get from the output 1. Try using FLOWTABLE option in your kernel config - this may kill your machine in -CURRENT, but it's worth checking 2. MTU set to some reasonably small value (~120) shall prevent from consolidating small packets into big frame 3. disable all kernel tracing (INVARIANTS, WITNESS, etc) 4. I read you tuned the system, but make sure there are enough resources 5. Check that context switches are not killing it Regardless on the above you might want profile the system with dtrace and export results to flamegraph, this will give you an idea what going on in the system and possibly locate the cause of the problem. It's definitely worth the effort. Jakub On Fri, Dec 2, 2016 at 12:55 AM, Jordan Caraballo < jordancaraballo87@gmail.com> wrote: > Feedback and/or tips and tricks more than welcome. > > We are trying to process huge amounts of small (64 bytes) pps through a > router. So far results have not been as we expected. We have tested > FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without > netmap. Based on netmap documentation we were expecting about 5.0M pps; > alongside with the routing improvements from the freebsd routing > proposal a total of 12.0M. > > Server Description: > > Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5=C2=AD2695 CPU's, 18 cores = per > cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. > > BIOS tweaks: > > Hyperthreading (or Logical Processors) is turned off. > > Current results are shown below. Additional configurations can be given > upon request. > > Test Environment: > 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one > equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2). > > Script that execute the following on each host. > > #!/usr/local/bin/bash > # Iterate through ports and start tests > for ((i=3D1;i<=3DPORTS;i++)); do > PORT=3D$(($PORT+1)) > iperf3 -c 172.16.2.$IP -u -b10000m -i 0 -N -l$PKT -t$TIME -P$STREAMS > -p$PORT & > #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT & > done > > # FreeBSD 10.3 - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.9M 0 1.3M 194M 540k 0 57M 0 0 > 2.1M 0 1.5M 216M 556k 0 58M 0 0 > 1.8M 0 1.3M 192M 553k 0 58M 0 0 > 1.7M 0 1.1M 174M 542k 0 57M 0 0 > 1.9M 0 1.4M 204M 537k 0 56M 0 0 > 1.6M 0 1.1M 171M 550k 0 58M 0 0 > 1.6M 0 1.1M 173M 546k 0 57M 0 0 > 1.7M 0 1.1M 176M 564k 0 59M 0 0 > 2.0M 0 1.5M 212M 543k 0 57M 0 0 > 2.1M 0 1.5M 219M 557k 0 58M 0 0 > 1.9M 0 1.4M 205M 547k 0 57M 0 0 > 1.7M 0 1.2M 179M 553k 0 58M 0 0 > > # FreeBSD 11.0 - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 3.1M 0 1.8M 326M 1.3M 0 134M 0 0 > 2.6M 0 1.5M 269M 1.1M 0 116M 0 0 > 2.7M 0 1.5M 285M 1.2M 0 127M 0 0 > 2.4M 0 1.3M 257M 1.1M 0 119M 0 0 > 2.7M 0 1.5M 287M 1.3M 0 134M 0 0 > 2.5M 0 1.3M 262M 1.2M 0 127M 0 0 > 2.1M 0 1.1M 224M 1.0M 0 108M 0 0 > 2.7M 0 1.4M 285M 1.4M 0 143M 0 0 > 2.6M 0 1.3M 272M 1.3M 0 136M 0 0 > 2.5M 0 1.4M 265M 1.1M 0 120M 0 0 > # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.9M 0 849k 195M 1.0M 0 107M 0 0 > 1.9M 0 854k 196M 1.0M 0 106M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 852k 196M 1.0M 0 107M 0 0 > 1.9M 0 847k 195M 1.0M 0 107M 0 0 > 1.9M 0 836k 195M 1.0M 0 107M 0 0 > 1.9M 0 843k 195M 1.0M 0 107M 0 0 > # FreeBSD 12.0-CURRENT - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.1M 259 0 115M 1.1M 0 115M 0 0 > 1.2M 273 0 124M 1.2M 0 124M 0 0 > 1.1M 200 0 112M 1.1M 0 112M 0 0 > 1.2M 290 0 122M 1.2M 0 122M 0 0 > 1.0M 132 0 107M 1.0M 0 107M 0 0 > 1.1M 303 0 118M 1.1M 0 118M 0 0 > 1.1M 278 0 112M 1.1M 0 112M 0 0 > 1.2M 243 0 122M 1.2M 0 122M 0 0 > 1.1M 168 0 112M 1.1M 0 112M 0 0 > 1.1M 161 0 112M 1.1M 0 112M 0 0 > # FreeBSD 12.0-CURRENT + Netmap - 4 streams to 80 ports from each client > (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.4M 10 0 144M 1.4M 0 144M 0 0 > 1.5M 6 0 159M 1.5M 0 159M 0 0 > 1.4M 5 0 144M 1.4M 0 144M 0 0 > 1.5M 14 0 158M 1.5M 0 158M 0 0 > 1.4M 5 0 151M 1.4M 0 151M 0 0 > 1.4M 10 0 152M 1.4M 0 152M 0 0 > 1.4M 12 0 148M 1.4M 0 148M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.4M 23 0 151M 1.4M 0 151M 0 0 > 1.4M 11 0 151M 1.4M 0 151M 0 0 > # FreeBSD 12.0-CURRENT + Netmap + Tuning - 4 streams to 80 ports from > each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.4M 15 0 145M 1.4M 0 145M 0 0 > 1.5M 18 0 157M 1.5M 0 157M 0 0 > 1.5M 10 0 156M 1.5M 0 156M 0 0 > 1.5M 15 0 154M 1.5M 0 154M 0 0 > 1.4M 13 0 146M 1.4M 0 146M 0 0 > 1.5M 15 0 156M 1.5M 0 156M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.5M 13 0 153M 1.5M 0 153M 0 0 > 1.4M 14 0 145M 1.4M 0 145M 0 0 > 1.4M 17 0 151M 1.4M 0 151M 0 0 > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-performance@freebsd.org Fri Dec 2 12:16:14 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 168F3C6237B; Fri, 2 Dec 2016 12:16:14 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: from mail-oi0-x241.google.com (mail-oi0-x241.google.com [IPv6:2607:f8b0:4003:c06::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CE33A15BD; Fri, 2 Dec 2016 12:16:13 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: by mail-oi0-x241.google.com with SMTP id f201so27070665oib.0; Fri, 02 Dec 2016 04:16:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=JS3Jt6b7QflLiHAOqQRJjf+Ov20cCEY4J0XCR7cTdYI=; b=sgS9gIj5owIAjuYj6XtvoEpZQ9dKYpQS7kUngpl18uzSe2TWI1K1np3oWgyp6yzzCf aFGpGDJRp/cxu1kQDeI9kG1u9wmrFoTDUzBt0ZHJoD2JZhQvXEiK+Ca4kd0GFs3F+cPi 0EWIISyvHmJJD8YkHy4dU5F9paImRfulTT1ghBd4QhwPyEyE94XmJIbpWcdVaFTpCW15 TvpOfcwG1/3K2j5oXiFgyPPRZ8Lg03HCSGTjhh1T9TASythhuQudHLDnMpQ897iYiKgK 2/w/5u5RKdd9jUJgprifck1yHVhxyX45h0Pwej5Z255y4JaOtmPPNxkerKWGi/zdz2MU YmuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=JS3Jt6b7QflLiHAOqQRJjf+Ov20cCEY4J0XCR7cTdYI=; b=BRXagHNKQWSmWMpR84dClwevX06ylbjs5qSZNUjZKSG2BmSPeK8jlzb3meL6BTwLUi XUxmRd1hqrtK8kP8OOftpGBqiTkH69+QPIbU9rm/Rv7QNoGoSwgtKrs+y5Z2+ynTFyFK zAn8mW3myTXcrdqvdSsZxJW27LPaw2eHtz5IrnO9eFcToDFNVTgZ+kQibxfzlTyVi5ki t/rxAdQEb7lYdCASJqdcZ20ID4POWON4ThhCCuHsdFr1iFbLOgElRFAzBga/WQGhzb8m G0WoouPjqXecl8JlbY2CehHlDCjvsVqvx1uwWNqVZDzcHlWSi0sOH3dUwfDp8fJJpxt9 2gCA== X-Gm-Message-State: AKaTC00IPK3QDtAX1wAJg+vOe1K/RYiyEGj+OTLkTcl2ASitBGEKIYrjLu4o1FBb/zp2eV09nnoFd05iUF5AuA== X-Received: by 10.202.235.78 with SMTP id j75mr22518632oih.74.1480680973110; Fri, 02 Dec 2016 04:16:13 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.45.200 with HTTP; Fri, 2 Dec 2016 04:16:12 -0800 (PST) In-Reply-To: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> References: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> From: Vincenzo Maffione Date: Fri, 2 Dec 2016 13:16:12 +0100 Message-ID: Subject: Re: netmap, netmap-fwd, and how many M packets-per-second? To: Jordan Caraballo Cc: "freebsd-net@freebsd.org" , freebsd-hpc@freebsd.org, freebsd-performance@freebsd.org X-Mailman-Approved-At: Fri, 02 Dec 2016 12:39:51 +0000 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2016 12:16:14 -0000 Hi, If I'm getting it right, you are doing a test with 10 machines directly connected on the same LAN, using iperf to test the performance of the TCP/IP stack with your 10Gbit cards. But as far as I can see, netmap is not involved in your setup, you are not using it, aren't you? IOW , I don't understand what does that "+ Netmap" mean you our test description banner. Cheers, Vincenzo 2016-12-02 0:55 GMT+01:00 Jordan Caraballo : > Feedback and/or tips and tricks more than welcome. > > We are trying to process huge amounts of small (64 bytes) pps through a > router. So far results have not been as we expected. We have tested > FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without > netmap. Based on netmap documentation we were expecting about 5.0M pps; > alongside with the routing improvements from the freebsd routing > proposal a total of 12.0M. > > Server Description: > > Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5=C2=AD2695 CPU's, 18 cores = per > cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. > > BIOS tweaks: > > Hyperthreading (or Logical Processors) is turned off. > > Current results are shown below. Additional configurations can be given > upon request. > > Test Environment: > 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one > equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2). > > Script that execute the following on each host. > > #!/usr/local/bin/bash > # Iterate through ports and start tests > for ((i=3D1;i<=3DPORTS;i++)); do > PORT=3D$(($PORT+1)) > iperf3 -c 172.16.2.$IP -u -b10000m -i 0 -N -l$PKT -t$TIME -P$STREAMS > -p$PORT & > #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT & > done > > # FreeBSD 10.3 - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.9M 0 1.3M 194M 540k 0 57M 0 0 > 2.1M 0 1.5M 216M 556k 0 58M 0 0 > 1.8M 0 1.3M 192M 553k 0 58M 0 0 > 1.7M 0 1.1M 174M 542k 0 57M 0 0 > 1.9M 0 1.4M 204M 537k 0 56M 0 0 > 1.6M 0 1.1M 171M 550k 0 58M 0 0 > 1.6M 0 1.1M 173M 546k 0 57M 0 0 > 1.7M 0 1.1M 176M 564k 0 59M 0 0 > 2.0M 0 1.5M 212M 543k 0 57M 0 0 > 2.1M 0 1.5M 219M 557k 0 58M 0 0 > 1.9M 0 1.4M 205M 547k 0 57M 0 0 > 1.7M 0 1.2M 179M 553k 0 58M 0 0 > > # FreeBSD 11.0 - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 3.1M 0 1.8M 326M 1.3M 0 134M 0 0 > 2.6M 0 1.5M 269M 1.1M 0 116M 0 0 > 2.7M 0 1.5M 285M 1.2M 0 127M 0 0 > 2.4M 0 1.3M 257M 1.1M 0 119M 0 0 > 2.7M 0 1.5M 287M 1.3M 0 134M 0 0 > 2.5M 0 1.3M 262M 1.2M 0 127M 0 0 > 2.1M 0 1.1M 224M 1.0M 0 108M 0 0 > 2.7M 0 1.4M 285M 1.4M 0 143M 0 0 > 2.6M 0 1.3M 272M 1.3M 0 136M 0 0 > 2.5M 0 1.4M 265M 1.1M 0 120M 0 0 > # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.9M 0 849k 195M 1.0M 0 107M 0 0 > 1.9M 0 854k 196M 1.0M 0 106M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 851k 196M 1.0M 0 107M 0 0 > 1.9M 0 852k 196M 1.0M 0 107M 0 0 > 1.9M 0 847k 195M 1.0M 0 107M 0 0 > 1.9M 0 836k 195M 1.0M 0 107M 0 0 > 1.9M 0 843k 195M 1.0M 0 107M 0 0 > # FreeBSD 12.0-CURRENT - 4 streams to 80 ports from each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.1M 259 0 115M 1.1M 0 115M 0 0 > 1.2M 273 0 124M 1.2M 0 124M 0 0 > 1.1M 200 0 112M 1.1M 0 112M 0 0 > 1.2M 290 0 122M 1.2M 0 122M 0 0 > 1.0M 132 0 107M 1.0M 0 107M 0 0 > 1.1M 303 0 118M 1.1M 0 118M 0 0 > 1.1M 278 0 112M 1.1M 0 112M 0 0 > 1.2M 243 0 122M 1.2M 0 122M 0 0 > 1.1M 168 0 112M 1.1M 0 112M 0 0 > 1.1M 161 0 112M 1.1M 0 112M 0 0 > # FreeBSD 12.0-CURRENT + Netmap - 4 streams to 80 ports from each client > (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.4M 10 0 144M 1.4M 0 144M 0 0 > 1.5M 6 0 159M 1.5M 0 159M 0 0 > 1.4M 5 0 144M 1.4M 0 144M 0 0 > 1.5M 14 0 158M 1.5M 0 158M 0 0 > 1.4M 5 0 151M 1.4M 0 151M 0 0 > 1.4M 10 0 152M 1.4M 0 152M 0 0 > 1.4M 12 0 148M 1.4M 0 148M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.4M 23 0 151M 1.4M 0 151M 0 0 > 1.4M 11 0 151M 1.4M 0 151M 0 0 > # FreeBSD 12.0-CURRENT + Netmap + Tuning - 4 streams to 80 ports from > each client (5) > > input (Total) output > packets errs idrops bytes packets errs bytes colls drop= s > 1.4M 15 0 145M 1.4M 0 145M 0 0 > 1.5M 18 0 157M 1.5M 0 157M 0 0 > 1.5M 10 0 156M 1.5M 0 156M 0 0 > 1.5M 15 0 154M 1.5M 0 154M 0 0 > 1.4M 13 0 146M 1.4M 0 146M 0 0 > 1.5M 15 0 156M 1.5M 0 156M 0 0 > 1.5M 9 0 155M 1.5M 0 155M 0 0 > 1.5M 13 0 153M 1.5M 0 153M 0 0 > 1.4M 14 0 145M 1.4M 0 145M 0 0 > 1.4M 17 0 151M 1.4M 0 151M 0 0 > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" --=20 Vincenzo Maffione