From owner-freebsd-net@FreeBSD.ORG Wed Aug 19 16:27:03 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 04292106568B for ; Wed, 19 Aug 2009 16:27:03 +0000 (UTC) (envelope-from gigabyte.tmn@gmail.com) Received: from mail-bw0-f206.google.com (mail-bw0-f206.google.com [209.85.218.206]) by mx1.freebsd.org (Postfix) with ESMTP id 6F9CF8FC57 for ; Wed, 19 Aug 2009 16:27:02 +0000 (UTC) Received: by bwz2 with SMTP id 2so3564177bwz.43 for ; Wed, 19 Aug 2009 09:27:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:reply-to:from:to :cc:references:subject:date:mime-version:content-type :content-transfer-encoding:x-priority:x-msmail-priority:x-mailer :x-mimeole; bh=89tqJjg/nAcwlKTtc8RxidFsL4Zyo2sclaihu3qOHYg=; b=KeZBDKck5s4B462YSkviWYfnjdqIOfJZgRrJf3Zt7o3r1zeDOFEVXUu1SRrcalm0A5 gt45fEUayT+DvpOoh4sN4Q5aeoDTarVnjM3bYcvSJs1C06hFs4ExAh+B5LlFsjOSDoq+ vDYzpgmqCBTSkE3vA41ajm+e5g1ue28JOlr7c= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:reply-to:from:to:cc:references:subject:date:mime-version :content-type:content-transfer-encoding:x-priority:x-msmail-priority :x-mailer:x-mimeole; b=rUfbgsWwelwE6Ob6HLQGFJWImgESQoeMpD2unC5IfocNPwyRkzyES/4XNFEGqt84ED M2fIMX8X+h23kk3yOJt8vtlZr9d588/ANTMVX5jTfRGWNuTeAcSo1cQnEtCtR4X/i0y4 d8/OcerTFdL6hEnyt0jt3vHrpYUwYu4MdXgVI= Received: by 10.204.155.73 with SMTP id r9mr5092113bkw.101.1250699221348; Wed, 19 Aug 2009 09:27:01 -0700 (PDT) Received: from dm (152.dynamic-n192.r72.info [91.211.192.152]) by mx.google.com with ESMTPS id p9sm435474fkb.7.2009.08.19.09.27.00 (version=SSLv3 cipher=RC4-MD5); Wed, 19 Aug 2009 09:27:00 -0700 (PDT) Message-ID: <000e01ca20e9$e19caa10$1e010a0a@in72.ru> From: =?UTF-8?B?0JTQvNC40YLRgNC40Lkg0JfQsNC80YPRgNCw0LXQsg==?= To: References: <24727.68667.qm@web56404.mail.re3.yahoo.com> Date: Wed, 19 Aug 2009 22:27:00 +0600 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="UTF-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3138 X-Mailman-Approved-At: Wed, 19 Aug 2009 16:34:46 +0000 Cc: freebsd-net@freebsd.org Subject: Re: em driver input errors X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: =?UTF-8?B?0JTQvNC40YLRgNC40Lkg0JfQsNC80YPRgNCw0LXQsg==?= List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Aug 2009 16:27:03 -0000 Hello Alex. What sheduler are you using? ULE or 4BSD Have you NIC IRQ sharing with other hardware? What HZ value? 1000? >Thanks for the suggestion. >From a "clean" box: >dev.em.0.rx_int_delay: 0 >dev.em.0.tx_int_delay: 66 >dev.em.0.rx_abs_int_delay: 66 >dev.em.0.tx_abs_int_delay: 66 >I reset all the values (errors still appearing), then tried your suggestion >(rx_int_delay=600, rx_abs_int_delay=1000). This has reduced the number of > >interrupts for em0 (from about 7200/sec to around 6500/sec). After some >time, I started getting errors again. mmm, try the maximum value 67108, what hapens... > But that has made me try this also: >dev.em.0.tx_int_delay=600 >dev.em.0.tx_abs_int_delay=1000 I think it's a bad idea, but don't know because: >Meaning using your suggested values for tx too. Now em0 is seeing about >1800 interrupts/second, which is way better, but after some time I saw >errors >again... >From the output of "netstat -nI em0 -w 5": maybe mistake, did you meen "netstat -w5 em0" ? I have PPPoE concenrator based on S3000AHV motherboard with Core2Quad 6600 and four (to load all cores in CPU) Intel PCI-E x1 and PCI-E x4 NIC's My load: bras1 [/usr/home/dm]# netstat -w5 em0 input (Total) output packets errs bytes packets errs bytes colls 943831 0 803741196 932221 0 766771487 0 ^C bras1 [/usr/home/dm]# netstat -w1 -Iem0 input (em0) output packets errs bytes packets errs bytes colls 24067 0 20593033 17152 0 17361755 0 ^C bras1 [/usr/home/dm]# netstat -w1 -Ilagg0 input (lagg0) output packets errs bytes packets errs bytes colls 47085 0 38454150 46708 0 38128482 0 44888 0 36087138 44714 0 35985529 0 49607 0 40467232 49326 0 40227456 0 ^C bras1 [/usr/home/dm]# netstat -w5 -Ilagg0 input (lagg0) output packets errs bytes packets errs bytes colls 230260 0 187650240 228911 0 186485136 0 238023 0 194650670 236648 0 193471650 0 218424 0 175576014 216860 0 174282762 0 ^C The lagg0 interface includes em0, em1, em2, em3 for lacp protocol, and comunicates with cisco 2960G switch. vmstat -i says: interrupt total rate irq4: sio0 95234 0 irq19: atapci1 8430157 1 cpu0: timer 1275549106 258 irq256: em0 2329917460 472 irq257: em1 645070135 130 irq258: em2 3527395550 715 irq259: em3 3923746474 795 cpu1: timer 1275548822 258 cpu3: timer 1275548798 258 cpu2: timer 1275548865 258 Total 15536850601 3149 And i have't any problems. I think i select the good hardware. > input (em0) output > packets errs bytes packets errs bytes colls > 87267 0 50372599 106931 0 81598993 0 > 86496 0 50990332 105467 0 80064657 0 > 81726 3056 49876613 99080 0 73273640 0 > 90425 0 59172531 105299 0 77110096 0 > 120292 0 70369292 109597 0 78626248 0 >... a few minutes pass with zero errors ... > 89646 0 56951878 111240 0 86493393 0 > 86031 0 53549721 108695 0 83592747 0 > 77760 3054 48505562 96912 0 73185576 0 > 87508 0 56116394 106094 0 79130608 0 > 89031 0 56490982 103039 0 77398567 0 >What's interesting is that I'm seeing errors in a 80k packets/5 sec (so >around 16k packets/s) zone, but no errors at 120k packets/5sec (24kpps). Yes, it's not normaly. >Interrupts total (as reported by systat): around 13500/second. I would >estimate the old IRQ load at around 30000-35000/second, which doesn't seem >too >much to me, for a dual xeon machine. I think it depends by motherbord, what full hardware specification are you using? with chips names >Speaking of which, I did compile the kernel with "options DEVICE_POLLING", >but enabling polling only made the errors appear more often, and in greater > >numbers. I don't use polling on FBSD 7.x, it's usable on FBSD older versions > - 1 x dual-port gigabit interface, PCI-X Maybe I have this card. And it works unstable, i don't remember what happens, but i seen by tcpdump "truncated IP, missing XX bytes" Good luck.