From owner-freebsd-net@freebsd.org Tue Jan 31 17:53:19 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5C25CCAD6A for ; Tue, 31 Jan 2017 17:53:19 +0000 (UTC) (envelope-from jordancaraballo87@gmail.com) Received: from mail-ua0-x244.google.com (mail-ua0-x244.google.com [IPv6:2607:f8b0:400c:c08::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 693431AF9 for ; Tue, 31 Jan 2017 17:53:19 +0000 (UTC) (envelope-from jordancaraballo87@gmail.com) Received: by mail-ua0-x244.google.com with SMTP id 96so33703014uaq.2 for ; Tue, 31 Jan 2017 09:53:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=csBLw4/AMIclKVk8dH/syPV4fpgtwm7aITy7M+J5U2E=; b=RmRu9pK/i8MKnrvJ40iUpV1UCI44cJ0O7FSm2DuYO+g9sD7GLcR1McTEgIUyXCsKgW hHFbqJ17od7O/puuoUrGc5Mhygb3JlPh4qMnO0k2eLlWeAn09Mglv8qzeklklRoPFvAo +S0/IEARIGeBupXy9w55etTxijv5uTmGM7GQ59etPousqJzQJV3evd0CD8bW37VUqPrz d0gu9L8BsoK12GaPhM+hvMDPGqedLjotnskpOslWWtWObNIaS2wt2B3bIt526G/ae0h4 x7qZKThHPzl9lD03/7HCwCp2rAMbCneIt5MXklLyDDCd8aDwtx0weQHAI8Lo+f6c+1hV uZLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=csBLw4/AMIclKVk8dH/syPV4fpgtwm7aITy7M+J5U2E=; b=MVbcTcMQuYIzpIUtsvOXbb3w2V8n+iZ2w78WtGwJCnFzbSrMyJaROEbBVjGecfYaLl P06gjV0JXdu8O1Uol6OLO5X3IqZ4m8h/CrtGgYd3BTuPygm1RCGvwitBgAX7SDAEzjIC t6JU9YZgiKqyC5AyIvX6S816lBJyH3c12uX+xz3X2BXk0suozfMRTP+IOqaYNXCr3c7P mZfzPoQYKcEgI8JfZHL8y/VffWSG2DR6Wxj1u6XSS5mo9YcjQpo+FsaMiAgTarR3jNII P31+/cSHpuCVjQVUyKYz0v+hSdV/5Ni3R/IA0Mc3ON6Lv03siN/s40G3xyNcMAk7V8W5 NDfA== X-Gm-Message-State: AIkVDXKGS9IScjVxqcQgl0V+G1/1tLjqbWYs5tVZy6SoJBM8pT8SD+zuC6S0aYb6sCFEhQ== X-Received: by 10.176.85.158 with SMTP id v30mr15471500uaa.82.1485885198195; Tue, 31 Jan 2017 09:53:18 -0800 (PST) Received: from [136.145.55.111] ([136.145.55.111]) by smtp.gmail.com with ESMTPSA id q78sm6287269vkb.17.2017.01.31.09.53.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 31 Jan 2017 09:53:17 -0800 (PST) Subject: Re: Disappointing packets-per-second performance results on a Dell, PE R530 To: Slawa Olhovchenkov References: <8f637e2e-cd59-dc65-8476-30989bea516b@gmail.com> <20170103174627.GW37118@zxy.spb.ru> Cc: freebsd-net@freebsd.org From: Jordan Caraballo Message-ID: Date: Tue, 31 Jan 2017 13:53:15 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <20170103174627.GW37118@zxy.spb.ru> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jan 2017 17:53:19 -0000 This are the most recent stats. No advances so far. The system has -Current right now. Any help or feedback would be appreciated. Hardware Configuration: Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5­2695 CPU's, 18 cores per cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. BIOS tweaks: Hyperthreading (or Logical Processors) is turned off. loader.conf # Chelsio Modules t4fw_cfg_load="YES" t5fw_cfg_load="YES" if_cxgbe_load="YES" rc.conf # Gateway Configuration ifconfig_cxl0="inet 172.16.1.1/24" ifconfig_cxl1="inet 172.16.2.1/24" gateway_enable="YES" Last Results: packets errs idrops bytes packets errs bytes colls drops 2.7M 0 2.0M 1.4G 696k 0 368M 0 0 2.7M 0 2.0M 1.4G 686k 0 363M 0 0 2.6M 0 2.0M 1.4G 668k 0 353M 0 0 2.7M 0 2.0M 1.4G 661k 0 350M 0 0 2.8M 0 2.1M 1.5G 697k 0 369M 0 0 2.8M 0 2.1M 1.4G 684k 0 361M 0 0 2.7M 0 2.1M 1.4G 674k 0 356M 0 0 root@router1:~ # vmstat -i interrupt total rate irq9: acpi0 73 0 irq18: ehci0 ehci1 1155973 3 cpu0:timer 3551157 10 cpu29:timer 9303048 27 cpu9:timer 71693455 207 cpu16:timer 9798380 28 cpu18:timer 9287094 27 cpu26:timer 9342495 27 cpu20:timer 9145888 26 cpu8:timer 9791228 28 cpu22:timer 9288116 27 cpu35:timer 9376578 27 cpu30:timer 9396294 27 cpu23:timer 9248760 27 cpu10:timer 9756455 28 cpu25:timer 9300202 27 cpu27:timer 9227291 27 cpu14:timer 10083548 29 cpu28:timer 9325684 27 cpu11:timer 9906405 29 cpu34:timer 9419170 27 cpu31:timer 9392089 27 cpu33:timer 9350540 27 cpu15:timer 9804551 28 cpu32:timer 9413182 27 cpu19:timer 9231505 27 cpu12:timer 9813506 28 cpu13:timer 10872130 31 cpu4:timer 9920237 29 cpu2:timer 9786498 28 cpu3:timer 9896011 29 cpu5:timer 9890207 29 cpu6:timer 9737869 28 cpu7:timer 9790119 28 cpu1:timer 9847913 28 cpu21:timer 9192561 27 cpu24:timer 9300259 27 cpu17:timer 9786186 28 irq264: mfi0 151818 0 irq266: bge0 30466 0 irq272: t5nex0:evt 4 0 Total 402604945 1161 top -PHS last pid: 18557; load averages: 2.58, 1.90, 0.95 up 4+00:39:54 18:30:46 231 processes: 40 running, 126 sleeping, 65 waiting CPU 0: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 2: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 4: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 5: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 6: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle CPU 7: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 8: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 9: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 10: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 11: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 12: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 13: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 14: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 15: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 16: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 17: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 18: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 19: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 20: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 21: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 22: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 23: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 24: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 25: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 26: 0.0% user, 0.0% nice, 0.0% system, 59.6% interrupt, 40.4% idle CPU 27: 0.0% user, 0.0% nice, 0.0% system, 96.3% interrupt, 3.7% idle CPU 28: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 29: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 30: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 31: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 32: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 33: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 34: 0.0% user, 0.0% nice, 0.0% system, 100% interrupt, 0.0% idle CPU 35: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 15M Active, 224M Inact, 1544M Wired, 393M Buf, 29G Free Swap: 3881M Total, 3881M Free pmcstat -R sample.out -G - | head @ CPU_CLK_UNHALTED_CORE [159 samples] 39.62% [63] acpi_cpu_idle_mwait @ /boot/kernel/kernel 100.0% [63] acpi_cpu_idle 100.0% [63] cpu_idle_acpi 100.0% [63] cpu_idle 100.0% [63] sched_idletd 100.0% [63] fork_exit 17.61% [28] cpu_idle @ /boot/kernel/kernel root@router1:~ # pmcstat -R sample0.out -G - | head @ CPU_CLK_UNHALTED_CORE [750 samples] 31.60% [237] acpi_cpu_idle_mwait @ /boot/kernel/kernel 100.0% [237] acpi_cpu_idle 100.0% [237] cpu_idle_acpi 100.0% [237] cpu_idle 100.0% [237] sched_idletd 100.0% [237] fork_exit 10.67% [80] cpu_idle @ /boot/kernel/kernel On 03/01/17 13:46, Slawa Olhovchenkov wrote: > On Tue, Jan 03, 2017 at 12:35:42PM -0400, Jordan Caraballo wrote: > >> We recently tested a Dell R530 with a Chelsio T580 card, under FreeBSD 10.3, 11.0, -STABLE and -CURRENT, and Centos 7. >> >> Based on our research, including netmap-fwd and with the routing improvements project (https://wiki.freebsd.org/ProjectsRoutingProposal), >> we hoped for packets-per-second (pps) in the 5+ million range, or even higher. >> >> Based on prior testing (http://marc.info/?t=140604252400002&r=1&w=2), we expected 3-4 Million to be easily obtainable. >> >> Unfortunately, our current results top out at no more than 1.5 M (64 bytes length packets) with FreeBSD, and >> surprisingly around 3.2 M (128 bytes length packets) with Centos 7, and we are at a loss as to why. >> >> Server Description: >> Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5­2695 CPU's, 18 cores per >> cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. >> >> ** Can this be a lack in support issue related to the R530's hardware? ** >> >> Any help appreciated! > What hardware configuration? > What BIOS setting? > What loader.conf/sysctl.conf setting? > What `vmstat -i`? > What `top -PHS`? > what > ==== > pmcstat -S CPU_CLK_UNHALTED_CORE -l 10 -O sample.out > pmcstat -R sample.out -G out.txt > pmcstat -c 0 -S CPU_CLK_UNHALTED_CORE -l 10 -O sample0.out > pmcstat -R sample0.out -G out0.txt > ====