From owner-freebsd-stable@FreeBSD.ORG Mon Aug 26 16:06:56 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 3DD8D81D for ; Mon, 26 Aug 2013 16:06:56 +0000 (UTC) (envelope-from se@freebsd.org) Received: from nm10-vm1.bullet.mail.ird.yahoo.com (nm10-vm1.bullet.mail.ird.yahoo.com [77.238.189.93]) by mx1.freebsd.org (Postfix) with SMTP id 8BF792DC5 for ; Mon, 26 Aug 2013 16:06:55 +0000 (UTC) Received: from [77.238.189.48] by nm10.bullet.mail.ird.yahoo.com with NNFMP; 26 Aug 2013 16:06:48 -0000 Received: from [46.228.39.78] by tm1.bullet.mail.ird.yahoo.com with NNFMP; 26 Aug 2013 16:06:47 -0000 Received: from [127.0.0.1] by smtp115.mail.ir2.yahoo.com with NNFMP; 26 Aug 2013 16:06:47 -0000 X-Yahoo-Newman-Id: 798681.51677.bm@smtp115.mail.ir2.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: NMnCut0VM1k7U6BqB7IVb04ASWIfYvPQNsON86VuEmh.mF2 DwZMvLF3jJ43JaWs2cTF9fed2.hpeQK.431qRqqm0Tc0hrmbQEw2EKt._LT3 dWkZSnwaSJdOlqyb_ZL5J71RQBtrxqQRD69q1kOVsdQcL.UYET0UH650xpGE 4Lc7zYIYXwj23qS9AwRBMp2.80.TKarEkoA8PdRzhDgrV9mUd0yIE25wXSWE OMhv3egvx0lcu_oZ3lB5GOQFqV7B6MD4lGqTRSQeFINOnJb6xSctJNqUI6Rt O1_Y3s81ZanUHnSTgfEqWr4Ng0bOhY5VECVk_S7QcP.xoeSAbdHOJ0wwgCtY KER.X.WXlml.27t_HYGCb88z1JG4y95k1BXKybwIHPrZKBfQ8JsdXH_GmM_c OrxXw9ZLRSIICUPDDmfpYAP.2d6uZP8_ubo2n4l4FVJKFjR9.kfryaWkIW1y mDt1UiBy_rn01x74EL9G2abvTha5HFbx_qHq6ZT1EcDNge_d.sWo0dNu_Qh6 LdBijOEGXUCakTuYYDVXKtz14U.pQLw-- X-Yahoo-SMTP: iDf2N9.swBDAhYEh7VHfpgq0lnq. X-Rocket-Received: from [192.168.119.11] (se@84.154.99.225 with ) by smtp115.mail.ir2.yahoo.com with SMTP; 26 Aug 2013 16:06:47 +0000 UTC Message-ID: <521B7D0C.2010109@freebsd.org> Date: Mon, 26 Aug 2013 18:06:36 +0200 From: Stefan Esser User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: Harald Schmalzbauer Subject: Re: if_em, legacy nic and GbE saturation References: <521AFE7E.2040705@omnilan.de> <521B1FD4.4050702@omnilan.de> In-Reply-To: <521B1FD4.4050702@omnilan.de> X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit Cc: FreeBSD Stable Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Aug 2013 16:06:56 -0000 Am 26.08.2013 11:28, schrieb Harald Schmalzbauer: > Bezüglich Adrian Chadd's Nachricht vom 26.08.2013 10:34 > (localtime): >> Hi, >> >> There's bus limits on how much data you can push over a PCI bus. >> You can look around online to see what 32/64 bit, 33/66MHz PCI >> throughput estimates are. >> >> It changes massively if you use small versus large frames as >> well. >> >> The last time I tried it i couldn't hit gige on PCI; I only >> managed to get to around 350mbit doing TCP tests. > > Thanks, I'm roughly aware about the PCI bus limit, but I guess it > should be good for almost GbE: 33*10^6*32=1056, so if one considers > overhead and other bus-blocking things (nothing of significance is > active on the PCI bus in this case), I'd expect at least 800Mbis/s, > which is what I get with jumbo frames. But PCI bus throughput might be much lower than expected: - The arbitration overhead is quite high, in the order of 0.2 to 0.3us. - Depending on device capabilities and chip-set configuration and features there may be many more arbitration phases than one might expect. - A cache line flush is requested for data held in the CPU, unless the bus-master uses special transfer commands to indicate that the full cache line will be invalidated within the requested transfer. These overheads combined may reduce the effective PCI throughput to a fraction of the nominal performance (1/3 to 1/4 for bursts of 16 bytes). The "minimum grant" value is the minimum burst length the device wants (to avoid a buffer underrun/overrun due to too low effective bandwidth) the "maximum latency" corresponds to the number of PCI clocks the device is willing to wait for the bus to be granted (to avoid buffer underrun/overrun while waiting to get access to the bus granted). The maximum latency value is useful to calculate the maximum arbitration unit for which no device is stalled longer than allowed by MAXLAT. MINGNT and MAXLAT of a device can be displayed with pciconf: # pciconf -r -b pci0:1:0 0x3e:0x3f (e.g., for bus 0 device 1 function 0) The PCI bus will be "lost" whenever another device gets access to the bus, whether CPU or another PCI (or PCIe) device. Especially when simultanously sending and receiving packets with two Ethernet controllers, bus arbitration will occur for every 16 to 32 transfers (depending on bus arbiter settings and programmed MINGNT). Regards, STefan