From owner-freebsd-current@FreeBSD.ORG Tue Dec 6 11:28:46 2011 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 05F1F106566C for ; Tue, 6 Dec 2011 11:28:46 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 4B7F28FC16 for ; Tue, 6 Dec 2011 11:28:44 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [192.92.129.5]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.4/8.14.4) with ESMTP id pB6BSXRa043914 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 6 Dec 2011 13:28:39 +0200 (EET) (envelope-from daniel@digsys.bg) Message-ID: <4EDDFC61.5050400@digsys.bg> Date: Tue, 06 Dec 2011 13:28:33 +0200 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Luigi Rizzo References: <20111205192703.GA49118@onelab2.iet.unipi.it> <2D87D847-A2B7-4E77-B6C1-61D73C9F582F@digsys.bg> <20111205222834.GA50285@onelab2.iet.unipi.it> <4EDDF9F4.9070508@digsys.bg> In-Reply-To: <4EDDF9F4.9070508@digsys.bg> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: current@freebsd.org Subject: Re: datapoints on 10G throughput with TCP ? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2011 11:28:46 -0000 On 06.12.11 13:18, Daniel Kalchev wrote: > [...] > second blade: > > # nuttcp -t -T 5 -w 128 -v 10.2.101.13 > nuttcp-t: v6.1.2: socket > nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.13 > nuttcp-t: time limit = 5.00 seconds > nuttcp-t: connect to 10.2.101.13 with mss=1448, RTT=0.164 ms > nuttcp-t: send window size = 131768, receive window size = 66608 > nuttcp-t: 1290.3750 MB in 5.00 real seconds = 264173.96 KB/sec = > 2164.1131 Mbps > nuttcp-t: host-retrans = 0 > nuttcp-t: 20646 I/O calls, msec/call = 0.25, calls/sec = 4127.72 > nuttcp-t: 0.0user 3.8sys 0:05real 77% 96i+1299d 616maxrss 0+3pf > 27389+0csw > > nuttcp-r: v6.1.2: socket > nuttcp-r: buflen=65536, nstream=1, port=5001 tcp > nuttcp-r: accept from 10.2.101.14 > nuttcp-r: send window size = 33304, receive window size = 131768 > nuttcp-r: 1290.3750 MB in 5.14 real seconds = 256835.92 KB/sec = > 2103.9998 Mbps > nuttcp-r: 85668 I/O calls, msec/call = 0.06, calls/sec = 16651.70 > nuttcp-r: 0.0user 4.8sys 0:05real 94% 107i+1437d 624maxrss 0+15pf > 11848+0csw > > > Not impresive... I am rebuilding now to -stable. > > Daniel I also noticed interrupt storms happening while this was running on the second pair of blades: interrupt storm detected on "irq272:"; throttling interrupt source interrupt storm detected on "irq272:"; throttling interrupt source interrupt storm detected on "irq272:"; throttling interrupt source interrupt storm detected on "irq270:"; throttling interrupt source interrupt storm detected on "irq270:"; throttling interrupt source interrupt storm detected on "irq270:"; throttling interrupt source some stats # sysctl -a dev.ix.1 dev.ix.1.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.10 dev.ix.1.%driver: ix dev.ix.1.%location: slot=0 function=1 dev.ix.1.%pnpinfo: vendor=0x8086 device=0x10fc subvendor=0xffff subdevice=0xffff class=0x020000 dev.ix.1.%parent: pci3 dev.ix.1.flow_control: 3 dev.ix.1.advertise_gig: 0 dev.ix.1.enable_aim: 1 dev.ix.1.rx_processing_limit: 128 dev.ix.1.dropped: 0 dev.ix.1.mbuf_defrag_failed: 0 dev.ix.1.no_tx_dma_setup: 0 dev.ix.1.watchdog_events: 0 dev.ix.1.tso_tx: 1193460 dev.ix.1.link_irq: 1 dev.ix.1.queue0.interrupt_rate: 1000000 dev.ix.1.queue0.txd_head: 45 dev.ix.1.queue0.txd_tail: 45 dev.ix.1.queue0.no_desc_avail: 0 dev.ix.1.queue0.tx_packets: 23 dev.ix.1.queue0.rxd_head: 16 dev.ix.1.queue0.rxd_tail: 15 dev.ix.1.queue0.rx_packets: 16 dev.ix.1.queue0.rx_bytes: 2029 dev.ix.1.queue0.lro_queued: 0 dev.ix.1.queue0.lro_flushed: 0 dev.ix.1.queue1.interrupt_rate: 62500 dev.ix.1.queue1.txd_head: 0 dev.ix.1.queue1.txd_tail: 0 dev.ix.1.queue1.no_desc_avail: 0 dev.ix.1.queue1.tx_packets: 0 dev.ix.1.queue1.rxd_head: 0 dev.ix.1.queue1.rxd_tail: 2047 dev.ix.1.queue1.rx_packets: 0 dev.ix.1.queue1.rx_bytes: 0 dev.ix.1.queue1.lro_queued: 0 dev.ix.1.queue1.lro_flushed: 0 dev.ix.1.queue2.interrupt_rate: 200000 dev.ix.1.queue2.txd_head: 545 dev.ix.1.queue2.txd_tail: 545 dev.ix.1.queue2.no_desc_avail: 0 dev.ix.1.queue2.tx_packets: 331690 dev.ix.1.queue2.rxd_head: 1099 dev.ix.1.queue2.rxd_tail: 1098 dev.ix.1.queue2.rx_packets: 498763 dev.ix.1.queue2.rx_bytes: 32954702 dev.ix.1.queue2.lro_queued: 0 dev.ix.1.queue2.lro_flushed: 0 dev.ix.1.queue3.interrupt_rate: 62500 dev.ix.1.queue3.txd_head: 0 dev.ix.1.queue3.txd_tail: 0 dev.ix.1.queue3.no_desc_avail: 0 dev.ix.1.queue3.tx_packets: 0 dev.ix.1.queue3.rxd_head: 0 dev.ix.1.queue3.rxd_tail: 2047 dev.ix.1.queue3.rx_packets: 0 dev.ix.1.queue3.rx_bytes: 0 dev.ix.1.queue3.lro_queued: 0 dev.ix.1.queue3.lro_flushed: 0 dev.ix.1.queue4.interrupt_rate: 1000000 dev.ix.1.queue4.txd_head: 13 dev.ix.1.queue4.txd_tail: 13 dev.ix.1.queue4.no_desc_avail: 0 dev.ix.1.queue4.tx_packets: 6 dev.ix.1.queue4.rxd_head: 6 dev.ix.1.queue4.rxd_tail: 5 dev.ix.1.queue4.rx_packets: 6 dev.ix.1.queue4.rx_bytes: 899 dev.ix.1.queue4.lro_queued: 0 dev.ix.1.queue4.lro_flushed: 0 dev.ix.1.queue5.interrupt_rate: 200000 dev.ix.1.queue5.txd_head: 982 dev.ix.1.queue5.txd_tail: 982 dev.ix.1.queue5.no_desc_avail: 0 dev.ix.1.queue5.tx_packets: 302592 dev.ix.1.queue5.rxd_head: 956 dev.ix.1.queue5.rxd_tail: 955 dev.ix.1.queue5.rx_packets: 474044 dev.ix.1.queue5.rx_bytes: 31319840 dev.ix.1.queue5.lro_queued: 0 dev.ix.1.queue5.lro_flushed: 0 dev.ix.1.queue6.interrupt_rate: 200000 dev.ix.1.queue6.txd_head: 1902 dev.ix.1.queue6.txd_tail: 1902 dev.ix.1.queue6.no_desc_avail: 0 dev.ix.1.queue6.tx_packets: 184922 dev.ix.1.queue6.rxd_head: 1410 dev.ix.1.queue6.rxd_tail: 1409 dev.ix.1.queue6.rx_packets: 402818 dev.ix.1.queue6.rx_bytes: 27759640 dev.ix.1.queue6.lro_queued: 0 dev.ix.1.queue6.lro_flushed: 0 dev.ix.1.queue7.interrupt_rate: 200000 dev.ix.1.queue7.txd_head: 660 dev.ix.1.queue7.txd_tail: 660 dev.ix.1.queue7.no_desc_avail: 0 dev.ix.1.queue7.tx_packets: 378078 dev.ix.1.queue7.rxd_head: 885 dev.ix.1.queue7.rxd_tail: 884 dev.ix.1.queue7.rx_packets: 705397 dev.ix.1.queue7.rx_bytes: 46572290 dev.ix.1.queue7.lro_queued: 0 dev.ix.1.queue7.lro_flushed: 0 dev.ix.1.mac_stats.crc_errs: 0 dev.ix.1.mac_stats.ill_errs: 0 dev.ix.1.mac_stats.byte_errs: 0 dev.ix.1.mac_stats.short_discards: 0 dev.ix.1.mac_stats.local_faults: 3 dev.ix.1.mac_stats.remote_faults: 1 dev.ix.1.mac_stats.rec_len_errs: 0 dev.ix.1.mac_stats.link_xon_txd: 0 dev.ix.1.mac_stats.link_xon_rcvd: 0 dev.ix.1.mac_stats.link_xoff_txd: 0 dev.ix.1.mac_stats.link_xoff_rcvd: 0 dev.ix.1.mac_stats.total_octets_rcvd: 146983578 dev.ix.1.mac_stats.good_octets_rcvd: 146933576 dev.ix.1.mac_stats.total_pkts_rcvd: 2081451 dev.ix.1.mac_stats.good_pkts_rcvd: 2081044 dev.ix.1.mac_stats.mcast_pkts_rcvd: 0 dev.ix.1.mac_stats.bcast_pkts_rcvd: 2 dev.ix.1.mac_stats.rx_frames_64: 4 dev.ix.1.mac_stats.rx_frames_65_127: 2081036 dev.ix.1.mac_stats.rx_frames_128_255: 0 dev.ix.1.mac_stats.rx_frames_256_511: 0 dev.ix.1.mac_stats.rx_frames_512_1023: 4 dev.ix.1.mac_stats.rx_frames_1024_1522: 0 dev.ix.1.mac_stats.recv_undersized: 0 dev.ix.1.mac_stats.recv_fragmented: 0 dev.ix.1.mac_stats.recv_oversized: 0 dev.ix.1.mac_stats.recv_jabberd: 0 dev.ix.1.mac_stats.management_pkts_rcvd: 0 dev.ix.1.mac_stats.management_pkts_drpd: 0 dev.ix.1.mac_stats.checksum_errs: 0 dev.ix.1.mac_stats.good_octets_txd: 4752616248 dev.ix.1.mac_stats.total_pkts_txd: 3302683 dev.ix.1.mac_stats.good_pkts_txd: 3302683 dev.ix.1.mac_stats.bcast_pkts_txd: 3 dev.ix.1.mac_stats.mcast_pkts_txd: 0 dev.ix.1.mac_stats.management_pkts_txd: 0 dev.ix.1.mac_stats.tx_frames_64: 3 dev.ix.1.mac_stats.tx_frames_65_127: 9203 dev.ix.1.mac_stats.tx_frames_128_255: 31588 dev.ix.1.mac_stats.tx_frames_256_511: 50179 dev.ix.1.mac_stats.tx_frames_512_1023: 153285 dev.ix.1.mac_stats.tx_frames_1024_1522: 3058425 dev.ix.1.mac_stats.fc_crc: 0 dev.ix.1.mac_stats.fc_last: 0 dev.ix.1.mac_stats.fc_drpd: 0 dev.ix.1.mac_stats.fc_pkts_rcvd: 0 dev.ix.1.mac_stats.fc_pkts_txd: 0 dev.ix.1.mac_stats.fc_dword_rcvd: 0 dev.ix.1.mac_stats.fc_dword_txd: 0 Maybe this controller needs some tuning? Daniel