From owner-freebsd-hpc@freebsd.org Thu Dec 1 23:56:00 2016 Return-Path: Delivered-To: freebsd-hpc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BBE92C6246E; Thu, 1 Dec 2016 23:56:00 +0000 (UTC) (envelope-from jordancaraballo87@gmail.com) Received: from mail-qk0-x241.google.com (mail-qk0-x241.google.com [IPv6:2607:f8b0:400d:c09::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6E23C175B; Thu, 1 Dec 2016 23:55:57 +0000 (UTC) (envelope-from jordancaraballo87@gmail.com) Received: by mail-qk0-x241.google.com with SMTP id x190so28498601qkb.0; Thu, 01 Dec 2016 15:55:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=to:from:subject:message-id:date:user-agent:mime-version :content-transfer-encoding; bh=qjaNJl1tNUlvHHPfgb6DniaPFb0SeNfUaHLVG4mocFM=; b=W7MVe7/JGx3+Ww+vElPNIOBEa+1q4XsNRUkfBUtMYKI2AybK9sKlYCwiG2q5m8+Nnq 6HrVYfXd1TcMtrr9cZLWHX8LY2doKmH+55FqAYeswqdsiwlk29GSeB0VOd0f0Dp6KFuB qcYF4Xa19L+ZD8hfjc0zEMK6tPXNtKSpksTFytTm6qGU0f4cpebYpaYwNkY9kCFep6le iqEzlXa7P0bUyRGVPqJd9/ajSGf6qp9J/U3kbIgGnpSIXiOz6bjpDVxdxSWpvzM3caN9 Ujth4gBevxtGBZsvDJpirmFrohsY4td3BXECRxRysP1yAay52EW5EecIkNO7KPvTzzaX BZGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-transfer-encoding; bh=qjaNJl1tNUlvHHPfgb6DniaPFb0SeNfUaHLVG4mocFM=; b=NeFm0S2xBNw/4DcIxLgdXzJbH0rzt/A1yxzSIskJQ3fMjp+byiIS3QiB0ogncwMi38 abgxjaX5+yGfUYSIZbWM7PBUA2/xB83JT7hkiiGK5BirxUusxq8oEox+ZghfjYn5pB+G 8IVtdn7qeezuI2IlCtzvU+RiU1fqs4CmX1XPaO32zwBvFe823nwVlcJHrCjSD/Umsmdv MCcgot0BqxbnirzY+EMn1q9UTJfGEJ1G00v5VZWiSGyAC/hehJJZ/ZIS0rdS8yk5cI8f 8dDLkURGaoi36PHKAF+ozusFTKRb0WkDj0DVJlC5XJ0JIhJXoNxi8URWLktBRUkJm+aZ bkYg== X-Gm-Message-State: AKaTC00Q3yjMss+vj364rtdOalsl4KR26aC5cWc16UaiHO/pmNSZiTDhNEADrDfqM+C32Q== X-Received: by 10.55.116.135 with SMTP id p129mr40035180qkc.54.1480636556636; Thu, 01 Dec 2016 15:55:56 -0800 (PST) Received: from [192.168.1.5] (pool-96-241-162-208.washdc.fios.verizon.net. [96.241.162.208]) by smtp.gmail.com with ESMTPSA id 30sm1323246qth.14.2016.12.01.15.55.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 01 Dec 2016 15:55:55 -0800 (PST) To: freebsd-net@freebsd.org, freebsd-hpc@freebsd.org, freebsd-performance@freebsd.org From: Jordan Caraballo Subject: netmap, netmap-fwd, and how many M packets-per-second? Message-ID: <596e5c15-1a85-1fdc-ca88-11a0ccd5e125@gmail.com> Date: Thu, 1 Dec 2016 19:55:55 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hpc@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "FreeBSD in High Performance Computing environments." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2016 23:56:00 -0000 Feedback and/or tips and tricks more than welcome. We are trying to process huge amounts of small (64 bytes) pps through a router. So far results have not been as we expected. We have tested FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without netmap. Based on netmap documentation we were expecting about 5.0M pps; alongside with the routing improvements from the freebsd routing proposal a total of 12.0M. Server Description: Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5­2695 CPU's, 18 cores per cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. BIOS tweaks: Hyperthreading (or Logical Processors) is turned off. Current results are shown below. Additional configurations can be given upon request. Test Environment: 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2). Script that execute the following on each host. #!/usr/local/bin/bash # Iterate through ports and start tests for ((i=1;i<=PORTS;i++)); do PORT=$(($PORT+1)) iperf3 -c 172.16.2.$IP -u -b10000m -i 0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT & #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT & done # FreeBSD 10.3 - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 1.9M 0 1.3M 194M 540k 0 57M 0 0 2.1M 0 1.5M 216M 556k 0 58M 0 0 1.8M 0 1.3M 192M 553k 0 58M 0 0 1.7M 0 1.1M 174M 542k 0 57M 0 0 1.9M 0 1.4M 204M 537k 0 56M 0 0 1.6M 0 1.1M 171M 550k 0 58M 0 0 1.6M 0 1.1M 173M 546k 0 57M 0 0 1.7M 0 1.1M 176M 564k 0 59M 0 0 2.0M 0 1.5M 212M 543k 0 57M 0 0 2.1M 0 1.5M 219M 557k 0 58M 0 0 1.9M 0 1.4M 205M 547k 0 57M 0 0 1.7M 0 1.2M 179M 553k 0 58M 0 0 # FreeBSD 11.0 - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 3.1M 0 1.8M 326M 1.3M 0 134M 0 0 2.6M 0 1.5M 269M 1.1M 0 116M 0 0 2.7M 0 1.5M 285M 1.2M 0 127M 0 0 2.4M 0 1.3M 257M 1.1M 0 119M 0 0 2.7M 0 1.5M 287M 1.3M 0 134M 0 0 2.5M 0 1.3M 262M 1.2M 0 127M 0 0 2.1M 0 1.1M 224M 1.0M 0 108M 0 0 2.7M 0 1.4M 285M 1.4M 0 143M 0 0 2.6M 0 1.3M 272M 1.3M 0 136M 0 0 2.5M 0 1.4M 265M 1.1M 0 120M 0 0 # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 1.9M 0 849k 195M 1.0M 0 107M 0 0 1.9M 0 854k 196M 1.0M 0 106M 0 0 1.9M 0 851k 196M 1.0M 0 107M 0 0 1.9M 0 851k 196M 1.0M 0 107M 0 0 1.9M 0 851k 196M 1.0M 0 107M 0 0 1.9M 0 852k 196M 1.0M 0 107M 0 0 1.9M 0 847k 195M 1.0M 0 107M 0 0 1.9M 0 836k 195M 1.0M 0 107M 0 0 1.9M 0 843k 195M 1.0M 0 107M 0 0 # FreeBSD 12.0-CURRENT - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 1.1M 259 0 115M 1.1M 0 115M 0 0 1.2M 273 0 124M 1.2M 0 124M 0 0 1.1M 200 0 112M 1.1M 0 112M 0 0 1.2M 290 0 122M 1.2M 0 122M 0 0 1.0M 132 0 107M 1.0M 0 107M 0 0 1.1M 303 0 118M 1.1M 0 118M 0 0 1.1M 278 0 112M 1.1M 0 112M 0 0 1.2M 243 0 122M 1.2M 0 122M 0 0 1.1M 168 0 112M 1.1M 0 112M 0 0 1.1M 161 0 112M 1.1M 0 112M 0 0 # FreeBSD 12.0-CURRENT + Netmap - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 1.4M 10 0 144M 1.4M 0 144M 0 0 1.5M 6 0 159M 1.5M 0 159M 0 0 1.4M 5 0 144M 1.4M 0 144M 0 0 1.5M 14 0 158M 1.5M 0 158M 0 0 1.4M 5 0 151M 1.4M 0 151M 0 0 1.4M 10 0 152M 1.4M 0 152M 0 0 1.4M 12 0 148M 1.4M 0 148M 0 0 1.5M 9 0 155M 1.5M 0 155M 0 0 1.4M 23 0 151M 1.4M 0 151M 0 0 1.4M 11 0 151M 1.4M 0 151M 0 0 # FreeBSD 12.0-CURRENT + Netmap + Tuning - 4 streams to 80 ports from each client (5) input (Total) output packets errs idrops bytes packets errs bytes colls drops 1.4M 15 0 145M 1.4M 0 145M 0 0 1.5M 18 0 157M 1.5M 0 157M 0 0 1.5M 10 0 156M 1.5M 0 156M 0 0 1.5M 15 0 154M 1.5M 0 154M 0 0 1.4M 13 0 146M 1.4M 0 146M 0 0 1.5M 15 0 156M 1.5M 0 156M 0 0 1.5M 9 0 155M 1.5M 0 155M 0 0 1.5M 13 0 153M 1.5M 0 153M 0 0 1.4M 14 0 145M 1.4M 0 145M 0 0 1.4M 17 0 151M 1.4M 0 151M 0 0