From owner-freebsd-stable@FreeBSD.ORG Wed Feb 27 11:01:26 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 19CE91065673 for ; Wed, 27 Feb 2008 11:01:26 +0000 (UTC) (envelope-from hk@alogis.com) Received: from alogis.com (firewall.solit-ag.de [212.184.102.1]) by mx1.freebsd.org (Postfix) with ESMTP id AD5E38FC17 for ; Wed, 27 Feb 2008 11:01:25 +0000 (UTC) (envelope-from hk@alogis.com) Received: from alogis.com (localhost [127.0.0.1]) by alogis.com (8.13.4/8.13.1) with ESMTP id m1RAnh2Q015330; Wed, 27 Feb 2008 11:49:43 +0100 (CET) (envelope-from hk@alogis.com) Received: (from hk@localhost) by alogis.com (8.13.4/8.13.1/Submit) id m1RAngaf015329; Wed, 27 Feb 2008 11:49:42 +0100 (CET) (envelope-from hk) Date: Wed, 27 Feb 2008 11:49:42 +0100 From: Holger Kipp To: freebsd-stable@freebsd.org Message-ID: <20080227104942.GA14530@intserv.int1.b.intern> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.1i Subject: em very slow, shared irq... on 6.3p8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Feb 2008 11:01:26 -0000 Hello, I updated a system with 12 dc-interfaces to a new hardware with 14 em-interfaces. Yes, it is a firewall. New System is 6.2-RELEASE-p8. What I now experience between two internal networks (100MBit/s each) is the following: 1318 packets transmitted, 1317 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.387/246.153/2441.392/324.142 ms tcpdump on the firewall shows similar delays (on the outgoing interface). tcpdump on the system I ping however shows very quick responses for incoming packages (ie usually less than a millisecond). I therefore assume that the problem is between receiving the irq from em and getting the data from the interface on the firewall itself. My first option would be to activate polling on em-interfaces - but as I did not experience this sort of notieceable slowdown with the old dc-based firewall (without polling), maybe someone can shed some light on this strange behaviour or has other suggestions as well? More details and dmesg below. Help and suggestions welcome :-) Best regards, Holger Kipp PS: Regarding the MPSAFE-issue - I need to use ipsec, so don't have a choice here. On the firewall I see many irqs (sometimes more than 5000) very often on one of the em-interfaces using sysctl/vmstat: -------------------------------------------------------------------------------- 2 users Load 0.00 0.00 0.04 Feb 27 12:20 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 14804 3724 32800 3800 1327728 count All 722324 5248 76723364 5696 pages Interrupts Proc:r p d s w Csw Trp Sys Int Sof Flt cow 7606 total 27 7862 1 462 4105 158840 wire 1: atkb 14988 act 6: fdc0 0.2%Sys 3.0%Intr 0.0%User 0.0%Nice 96.8%Idl 547868 inact 12: psm | | | | | | | | | | 1244 cache 14: ata ++ 1326484 free 359 16: em4 daefr 2761 17: em5 Namei Name-cache Dir-cache prcfr 28: twe Calls hits % hits % react 306 52: em0 pdwak 1 53: em1 zfod pdpgs 177 55: em3 Disks twed0 ozfod intrn 2001 cpu0: time KB/t 21.33 %slo-z 114464 buf 2001 cpu1: time tps 1 1 tfree 10 dirtybuf MB/s 0.01 100000 desiredvnodes % busy 0 66594 numvnodes 19758 freevnodes -------------------------------------------------------------------------------- dmesg Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RELEASE-p8 #1: Thu Oct 4 16:07:31 CEST 2007 WARNING: debug.mpsafenet forced to 0 as ipsec requires Giant WARNING: MPSAFE network stack disabled, expect reduced performance. MPTable: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 CPU 6420 @ 2.13GHz (2128.01-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6 Features=0xbfebfbff Features2=0xe3bd,CX16,,> AMD Features=0x20100000 AMD Features2=0x1 Cores per package: 2 real memory = 2146349056 (2046 MB) avail memory = 2095296512 (1998 MB) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0: Assuming intbase of 0 ioapic1: Assuming intbase of 24 ioapic2: Assuming intbase of 48 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard kbd1 at kbdmux0 cpu0 on motherboard cpu1 on motherboard pcib0: pcibus 0 on motherboard pci0: on pcib0 pcib0: unable to route slot 28 INTB pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 pcib2: at device 0.0 on pci1 pci2: on pcib2 twe0: <3ware Storage Controller. Driver version 1.50.01.002> port 0x4000-0x400f mem 0xe0100000-0xe010000f,0xe0800000-0xe0ffffff irq 28 at device 2.0 on pci2 twe0: [GIANT-LOCKED] twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048 pci1: at device 0.1 (no driver attached) pcib3: at device 0.2 on pci1 pci3: on pcib3 pcib4: at device 3.0 on pci3 pci4: on pcib4 em0: port 0x5000-0x503f mem 0xe1000000-0xe101ffff irq 52 at device 4.0 on pci4 em0: Ethernet address: 00:0e:0c:c6:cc:54 em0: [GIANT-LOCKED] em1: port 0x5040-0x507f mem 0xe1020000-0xe103ffff irq 53 at device 4.1 on pci4 em1: Ethernet address: 00:0e:0c:c6:cc:55 em1: [GIANT-LOCKED] em2: port 0x5080-0x50bf mem 0xe1040000-0xe105ffff irq 54 at device 6.0 on pci4 em2: Ethernet address: 00:0e:0c:c6:cc:56 em2: [GIANT-LOCKED] em3: port 0x50c0-0x50ff mem 0xe1060000-0xe107ffff irq 55 at device 6.1 on pci4 em3: Ethernet address: 00:0e:0c:c6:cc:57 em3: [GIANT-LOCKED] pci1: at device 0.3 (no driver attached) pcib5: irq 16 at device 3.0 on pci0 pci10: on pcib5 pcib6: at device 0.0 on pci10 pci11: on pcib6 pcib7: at device 0.0 on pci11 pci12: on pcib7 em4: port 0x6000-0x601f mem 0xe1120000-0xe113ffff,0xe1100000-0xe111ffff irq 16 at device 0.0 on pci12 em4: Memory Access and/or Bus Master bits were not set! em4: Ethernet address: 00:15:17:25:48:88 em4: [GIANT-LOCKED] em5: port 0x6020-0x603f mem 0xe1160000-0xe117ffff,0xe1140000-0xe115ffff irq 17 at device 0.1 on pci12 em5: Memory Access and/or Bus Master bits were not set! em5: Ethernet address: 00:15:17:25:48:89 em5: [GIANT-LOCKED] pcib8: at device 1.0 on pci11 pci13: on pcib8 em6: port 0x7000-0x701f mem 0xe1220000-0xe123ffff,0xe1200000-0xe121ffff irq 16 at device 0.0 on pci13 em6: Memory Access and/or Bus Master bits were not set! em6: Ethernet address: 00:15:17:25:48:8a em6: [GIANT-LOCKED] em7: port 0x7020-0x703f mem 0xe1260000-0xe127ffff,0xe1240000-0xe125ffff irq 17 at device 0.1 on pci13 em7: Memory Access and/or Bus Master bits were not set! em7: Ethernet address: 00:15:17:25:48:8b em7: [GIANT-LOCKED] pcib9: irq 17 at device 28.0 on pci0 pci14: on pcib9 pcib10: at device 0.0 on pci14 pci15: on pcib10 pcib11: at device 0.0 on pci15 pci16: on pcib11 em8: port 0x8000-0x801f mem 0xe1320000-0xe133ffff,0xe1300000-0xe131ffff irq 16 at device 0.0 on pci16 em8: Memory Access and/or Bus Master bits were not set! em8: Ethernet address: 00:15:17:25:48:64 em8: [GIANT-LOCKED] em9: port 0x8020-0x803f mem 0xe1360000-0xe137ffff,0xe1340000-0xe135ffff irq 17 at device 0.1 on pci16 em9: Memory Access and/or Bus Master bits were not set! em9: Ethernet address: 00:15:17:25:48:65 em9: [GIANT-LOCKED] pcib12: at device 1.0 on pci15 pci17: on pcib12 em10: port 0x9000-0x901f mem 0xe1420000-0xe143ffff,0xe1400000-0xe141ffff irq 16 at device 0.0 on pci17 em10: Memory Access and/or Bus Master bits were not set! em10: Ethernet address: 00:15:17:25:48:66 em10: [GIANT-LOCKED] em11: port 0x9020-0x903f mem 0xe1460000-0xe147ffff,0xe1440000-0xe145ffff irq 17 at device 0.1 on pci17 em11: Memory Access and/or Bus Master bits were not set! em11: Ethernet address: 00:15:17:25:48:67 em11: [GIANT-LOCKED] pcib13: irq 17 at device 28.4 on pci0 pci18: on pcib13 em12: port 0xa000-0xa01f mem 0xe1500000-0xe151ffff irq 16 at device 0.0 on pci18 em12: Ethernet address: 00:30:48:90:86:7a em12: [GIANT-LOCKED] pcib14: irq 7 at device 28.5 on pci0 pci19: on pcib14 em13: port 0xb000-0xb01f mem 0xe1600000-0xe161ffff irq 17 at device 0.0 on pci19 em13: Ethernet address: 00:30:48:90:86:7b em13: [GIANT-LOCKED] pcib15: at device 30.0 on pci0 pci20: on pcib15 pci20: at device 4.0 (no driver attached) isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x3020-0x302f at device 31.1 on pci0 ata0: on atapci0 ata1: on atapci0 pci0: at device 31.3 (no driver attached) pmtimer0 on isa0 orm0: at iomem 0xc0000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model MouseMan+, device ID 0 fdc0: at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0 fdc0: [FAST] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 ppc0: parallel port not found. sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 sio0: type 16550A sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 unknown: can't assign resources (port) unknown: can't assign resources (memory) unknown: can't assign resources (memory) unknown: can't assign resources (irq) unknown: can't assign resources (port) unknown: can't assign resources (port) Timecounters tick every 1.000 msec IPsec: Initialized Security Association Processing. IP Filter: v4.1.13 initialized. Default = pass all, Logging = enabled acd0: DVDROM at ata0-slave UDMA33 twed0: on twe0 twed0: 76318MB (156299440 sectors) SMP: AP CPU #1 Launched! Trying to mount root from ufs:/dev/twed0s1a em9: link state changed to UP em8: link state changed to UP em10: link state changed to UP em0: link state changed to UP em3: link state changed to UP [...]