From owner-freebsd-current@FreeBSD.ORG Thu Oct 28 10:13:39 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 487F716A4CE; Thu, 28 Oct 2004 10:13:39 +0000 (GMT) Received: from juniper.fornext.org (53.35.138.210.xn.2iij.net [210.138.35.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5B49943D5E; Thu, 28 Oct 2004 10:13:38 +0000 (GMT) (envelope-from shino@fornext.org) Received: from [127.0.0.1] (ariel.net.ss.titech.ac.jp [131.112.21.25]) by juniper.fornext.org (Postfix) with ESMTP id 958452A; Thu, 28 Oct 2004 19:13:36 +0900 (JST) Date: Thu, 28 Oct 2004 19:13:36 +0900 From: Shunsuke SHINOMIYA To: John Baldwin In-Reply-To: <200410271352.04069.jhb@FreeBSD.org> References: <20041022171924.73B3.SHINO@fornext.org> <200410271352.04069.jhb@FreeBSD.org> Message-Id: <20041028165604.DCF8.SHINO@fornext.org> MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Mailer: Becky! ver. 2.11.02 [ja] cc: freebsd-current@FreeBSD.org cc: obrien@FreeBSD.org Subject: Re[2]: disabling interrupt storm protection X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Oct 2004 10:13:39 -0000 Thank you, John. > Can you try this patch: > > --- //depot/vendor/freebsd/src/sys/kern/kern_intr.c 2004/09/05 02:10:52 > +++ //depot/projects/smpng/sys/kern/kern_intr.c 2004/10/27 17:41:06 snip > First, can you test that your system works with the default threshold, and > second, can you test that setting the threshold to zero disables the storm > detection? I applied your patch to kern_intr.c, and then I transmitted the high rate(148kpps) IP traffic to the box with the default threshold. But storm protection was not invoked. It seem that sensitivity of the detector is fallen. I tried some patterns of the packet loss test(transmitting the packets of the wire rate in 100Mbps to the box for 10 seconds. The frame size of each packets is 64 octet). These results are as follows. Each `before' and `after' are outputs of `vmstat -i | grep em'. Please observe the first numerical value(number of interrupts since system startup). kern.hz is 100 at all trials. And `Interrupt Moderation', one of the feature of em was disabled. ========== o kern_intr.c applied your patch. o hw.intr_storm_threshold=500(default) remark o The test results of both 1st and 2nd trial are good, low packet loss rate. A difference of both trials may be measurement error. o Interrupt storm protections is not invoked. o The interrupts rate of em0(input side) is about 25000 interrupts per second during a trial. o When hw.intr_storm_threshold=200, Interrupt storm protections is not invoked. ---------- 1st trial before irq54: em0 4 0 irq55: em1 10 0 after irq54: em0 255247 2092 irq55: em1 168688 1382 test result Plr 0.875% ---------- 2nd trial before irq54: em0 255247 2092 irq55: em1 168688 1382 after irq54: em0 510332 2883 irq55: em1 337599 1907 result Plr 0.957% ========== o kern_intr.c applied your patch and compiled with -DHACK flags. (it cancel the your hack. is it right?) o hw.intr_storm_threshold=500(default) remark o The test results of 1st to 3rd trials are bad, high packet loss rate. During these tirals, em0 may be in the warming up phase and interrupt storm detection was invoked(the console output `Interrupt storm detected'). Because the number of interrupts since system startup is less than initial `warming' value(= threshold * 10). o I want to avoid this phenomenon. o The result of 4th trial is higher plr than 5th trial. My guess is that em0 was in the warming phase at the forepart of 4th trial. ---------- 1st trial before irq54: em0 3 0 irq55: em1 8 0 after irq54: em0 1577 4 irq55: em1 9302 25 result Plr 84.037% ---------- 2nd trial before irq54: em0 1577 4 irq55: em1 9302 25 after irq54: em0 3150 6 irq55: em1 19584 39 result Plr 84.428% ---------- 3rd trial before irq54: em0 3150 6 irq55: em1 19584 39 after irq54: em0 4721 8 irq55: em1 30127 53 result Plr 83.855% ---------- 4th trial before irq54: em0 4721 8 irq55: em1 30127 53 after irq54: em0 269893 420 irq55: em1 196292 305 result Plr 0.492% ---------- 5th trial before irq54: em0 269893 420 irq55: em1 196292 305 after irq54: em0 532196 719 irq55: em1 358797 484 result Plr 0.361% ========== o kern_intr.c applied your patch and compiled with -DHACK flags. o hw.intr_storm_threshold=0 remark o The test result of 1st trial is worse than 2nd one. I think the cause is em0 in warming up phase at the forpart of the trial. ---------- 1st trial before irq54: em0 3 0 irq55: em1 8 0 after irq54: em0 275850 1319 irq55: em1 180104 861 result Plr 1.133% ---------- 2nd trial before irq54: em0 275850 1319 irq55: em1 180104 861 after irq54: em0 548886 1932 irq55: em1 355397 1251 result Plr 0.937% ---------- 3rd trial before irq54: em0 548886 1932 irq55: em1 355397 1251 after irq54: em0 821117 2280 irq55: em1 531581 1476 result Plr 0.902% -- Shunsuke SHINOMIYA