From owner-freebsd-smp Wed Jan 2 3:37:36 2002 Delivered-To: freebsd-smp@freebsd.org Received: from elvis.mu.org (elvis.mu.org [216.33.66.196]) by hub.freebsd.org (Postfix) with ESMTP id AFD1637B426 for ; Wed, 2 Jan 2002 03:37:24 -0800 (PST) Received: by elvis.mu.org (Postfix, from userid 1192) id 4044881E0A; Wed, 2 Jan 2002 05:37:19 -0600 (CST) Date: Wed, 2 Jan 2002 05:37:19 -0600 From: Alfred Perlstein To: smp@freebsd.org Subject: smp hang on -current? Message-ID: <20020102053719.V16101@elvis.mu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org I have a box, I've enabled SMP, and APIC, disabled WITNESS and INVARIANTS. It hangs after probing scsi right before mountroot. It looks like something may be botched with interrupts: APIC_IO: Testing 8254 interrupt delivery APIC_IO: Broken MP table detected: 8254 is not connected to IOAPIC #0 intpin 2 APIC_IO: routing 8254 via 8259 and IOAPIC #0 intpin 0 Waiting 15 seconds for scsi devices to settle (noperiph:sym0:0:-1:-1): SCSI BUS reset delivered. (noperiph:sym1:0:-1:-1): SCSI BUS reset delivered. (probe0:sym0:0:0:1): Autosense Failed (probe0:sym0:0:0:2): Autosense Failed (probe0:sym0:0:0:3): Autosense Failed (probe0:sym0:0:0:4): Autosense Failed (probe0:sym0:0:0:5): Autosense Failed (probe0:sym0:0:0:6): Autosense Failed (probe0:sym0:0:0:7): Autosense Failed -- -Alfred Perlstein [alfred@freebsd.org] 'Instead of asking why a piece of software is using "1970s technology," start asking why software is ignoring 30 years of accumulated wisdom.' Tax deductable donations for FreeBSD: http://www.freebsdfoundation.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Jan 2 3:43:26 2002 Delivered-To: freebsd-smp@freebsd.org Received: from elvis.mu.org (elvis.mu.org [216.33.66.196]) by hub.freebsd.org (Postfix) with ESMTP id 70B6B37B405 for ; Wed, 2 Jan 2002 03:43:24 -0800 (PST) Received: by elvis.mu.org (Postfix, from userid 1192) id 1F9E481D03; Wed, 2 Jan 2002 05:43:24 -0600 (CST) Date: Wed, 2 Jan 2002 05:43:24 -0600 From: Alfred Perlstein To: smp@freebsd.org Subject: Re: smp hang on -current? Message-ID: <20020102054324.W16101@elvis.mu.org> References: <20020102053719.V16101@elvis.mu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20020102053719.V16101@elvis.mu.org>; from bright@mu.org on Wed, Jan 02, 2002 at 05:37:19AM -0600 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org * Alfred Perlstein [020102 05:37] wrote: > I have a box, I've enabled SMP, and APIC, disabled WITNESS and > INVARIANTS. It hangs after probing scsi right before > mountroot. It looks like something may be botched with interrupts: I'll have info tomorrow and any suggestions for providing that info would be followed. thanks, -Alfred To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Jan 2 9:53:23 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mail5.speakeasy.net (mail5.speakeasy.net [216.254.0.205]) by hub.freebsd.org (Postfix) with ESMTP id 7DC5437B41F for ; Wed, 2 Jan 2002 09:53:21 -0800 (PST) Received: (qmail 16582 invoked from network); 2 Jan 2002 17:53:20 -0000 Received: from unknown (HELO laptop.baldwin.cx) ([64.81.54.73]) (envelope-sender ) by mail5.speakeasy.net (qmail-ldap-1.03) with SMTP for ; 2 Jan 2002 17:53:20 -0000 Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <20020102054324.W16101@elvis.mu.org> Date: Wed, 02 Jan 2002 09:53:07 -0800 (PST) From: John Baldwin To: Alfred Perlstein Subject: Re: smp hang on -current? Cc: smp@freebsd.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On 02-Jan-02 Alfred Perlstein wrote: > * Alfred Perlstein [020102 05:37] wrote: >> I have a box, I've enabled SMP, and APIC, disabled WITNESS and >> INVARIANTS. It hangs after probing scsi right before >> mountroot. It looks like something may be botched with interrupts: > > I'll have info tomorrow and any suggestions for providing that > info would be followed. Rest of the dmesg might be nice. :) -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Jan 2 11: 3:11 2002 Delivered-To: freebsd-smp@freebsd.org Received: from tao.org.uk (genius.tao.org.uk [212.135.162.51]) by hub.freebsd.org (Postfix) with ESMTP id B331237B41A; Wed, 2 Jan 2002 11:03:00 -0800 (PST) Received: by tao.org.uk (Postfix, from userid 100) id 7DC86558; Wed, 2 Jan 2002 19:02:53 +0000 (GMT) Date: Wed, 2 Jan 2002 19:02:53 +0000 From: Josef Karthauser To: Bosko Milekic Cc: freebsd-arch@freebsd.org, freebsd-smp@freebsd.org Subject: Re: SMPng: Interrupt Latency Issues Message-ID: <20020102190253.B47550@tao.org.uk> References: <20011226021800.A14608@technokratis.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-md5; protocol="application/pgp-signature"; boundary="tsOsTdHNUZQcU9Ye" Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20011226021800.A14608@technokratis.com>; from bmilekic@technokratis.com on Wed, Dec 26, 2001 at 02:18:00AM -0500 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org --tsOsTdHNUZQcU9Ye Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Dec 26, 2001 at 02:18:00AM -0500, Bosko Milekic wrote: >=20 > Hi, >=20 > It has become obvious, recently in particular, that some important > improvements are required in the way we take interrupts in -CURRENT > SMPng. As previously mentionned, we are experiencing lousy interrupt > latency in -CURRENT. This comes as no surprise. I'm interested in helping with with, although my experience in this area is zero ;(. I was waiting with bated breath for answers to this email and am a bit disappointed that no-one's replied. I hope that doesn't mean that it's not important to anyone. (I was hoping to learn something :) My problem is that my -current box regularly dies, usually when resuming from ACPI, which I now avoid because of it. I'm sure that it's interrupt related. A noticeable characteristic is that a console bell (beep) once started never ends, suggesting that a timer has stopped working, also things like top hang and never refresh. Joe > The purpose of this Email is to get the ball rolling on a discussion > pertaining not only to the approach that we're going to take to properly > remedy the latency issues, but also on the overall ways that we will be > handeling interrupts in SMPng. I know that there are already several > ideas floating around, and briefly talking to Jake and some others over > IRC, I see that there are a couple that are more common than others. Now > I know that this is a sensitive topic but in discussing it, I would > appreciate it if all points are strictly technical and deal with the > implementation, that we not over-generalize (in other words, that we > stay `on topic' [*]), and that after the discussion is done, that we set = up > some milestones and get the work done _soon_ (I'm prepared to take up > some of the work, yes - however, I do not feel that *I* am the right > person to oversee _all_ of the technical aspects of this heavily > burdened task alone). >=20 > [*] This means please don't start bashing anything and everything in our > system and stating useless things like "well, yeah, but our VM system=20 > does not do X, Y, and Z" if it is of no _direct_ relevance to > the way we take interrupts. >=20 > With that out of the way, here's our situation: >=20 > 1. We presently take an interrupt, and in the general case, proceed to > schedule an interrupt thread to run. While placing the thread on the > run queue and because we need to check for the "SWAIT" process flag > we must acquire the sched_lock. This is where the jist of the problem > lies. We bottleneck here because it means that only one CPU can be > scheduling the interrupt thread at a time, and the rest have to spin > waiting for sched_lock. To make matters worse, interrupts are > disabled on the local CPU and we cannot take any other interrupts > either. This is where we stand.=20 >=20 > 2. BSD/OS does things differently. An interrupt comes in and in the > general case, the interrupted thread's VM address space is borrowed > and the handler is immediately executed after interrupts are > re-enabled. Then the handler(s) run and only if it happens to hit a > mutex lock for which it must wait, a clean-up is done to provide a > thread that can block on the mutex for the interrupt, and to allow > the interrupted thread to continue doing its thing. The tradeoff is > that the interrupted thread _cannot_ run anywhere else, not even on > another CPU, while the interrupt that pre-empted it is not finished > executing or has not hit a mutex and could not aquire it quickly. > This may result in some kernel thread priority rules not being 100% > obeyed but I guess that this is part of the tradeoff. >=20 > 3. I believe that some others have alternative suggestions. I encourage > them to present them here clearly, assuming that they are realistic > and implementable approaches, so that we are sure to make the right > decision before we setup milestones. >=20 > In particular, I know that, after briefly speaking to Jake, there is > the idea of per-CPU "interrupt-only run queues" floating around. The > jist of this method would be to keep per-CPU run queues to which each > CPU can schedule their own interrupt threads without having to acquire > any locks (i.e., only have interrupts disabled). The un-balancing of the > queues as well as special priority cases - should they arise - could be > handeled by issuing IPIs. >=20 > I also know that some others (notably Rik van Riel - I hope I spelled > that correctly :-) ) have mentionned realistic ideas that are quite > logical in light of what we presently have and the points above. To me, > following a very brief overview, some of them shared some of the > qualities of the BSD/OS way of doing it so I'd like to invite him (and > others) to lay those methods out for us now, so that we have a greater > picture of various alternatives. >=20 > That's all for now. I hope that we can agree on something worthwhile > soon so that we can establish clear milestones and get this thing done. > It's been long overdue. >=20 > Best regards and Seasons Greetings to you all, > --=20 > Bosko Milekic > bmilekic@FreeBSD.org >=20 >=20 > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-arch" in the body of the message --tsOsTdHNUZQcU9Ye Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (FreeBSD) Comment: For info see http://www.gnupg.org iEYEARECAAYFAjwzWVwACgkQXVIcjOaxUBbqfwCfUiOchBvdRxYpj2+r6jD8bJ5D w4oAoKSs7R/XPJxsxYml+bUV7F1Kh/fJ =MOw6 -----END PGP SIGNATURE----- --tsOsTdHNUZQcU9Ye-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Jan 3 3:43:37 2002 Delivered-To: freebsd-smp@freebsd.org Received: from critter.freebsd.dk (esplanaden.cybercity.dk [212.242.40.114]) by hub.freebsd.org (Postfix) with ESMTP id 600EB37B405; Thu, 3 Jan 2002 03:43:28 -0800 (PST) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.11.6/8.11.6) with ESMTP id g039Gr600685; Thu, 3 Jan 2002 10:16:53 +0100 (CET) (envelope-from phk@critter.freebsd.dk) To: Josef Karthauser Cc: Bosko Milekic , freebsd-arch@FreeBSD.ORG, freebsd-smp@FreeBSD.ORG Subject: Re: SMPng: Interrupt Latency Issues In-Reply-To: Your message of "Wed, 02 Jan 2002 19:02:53 GMT." <20020102190253.B47550@tao.org.uk> Date: Thu, 03 Jan 2002 10:16:53 +0100 Message-ID: <683.1010049413@critter.freebsd.dk> From: Poul-Henning Kamp Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org In message <20020102190253.B47550@tao.org.uk>, Josef Karthauser writes: >> It has become obvious, recently in particular, that some important >> improvements are required in the way we take interrupts in -CURRENT >> SMPng. As previously mentionned, we are experiencing lousy interrupt >> latency in -CURRENT. This comes as no surprise. > >I'm interested in helping with with, although my experience in this >area is zero ;(. I was waiting with bated breath for answers to >this email and am a bit disappointed that no-one's replied. I hope >that doesn't mean that it's not important to anyone. (I was hoping >to learn something :) I think it takes more than a few days to answer meaningfully to something that abstract (I'm sure our local superhero will jump in now and explain how he solved all of this back in the mid eighties on his Amiga :-) There are basically two things to getting good interrupt latency: 1. An architecture making it possible. 2. Care in coding all over the place. I think we are on track to #1. My main contribution will probably be in trying to establish a metrology to document the result, but to the extent time allows I will participate and help where I can. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Jan 3 9:23:24 2002 Delivered-To: freebsd-smp@freebsd.org Received: from elvis.mu.org (elvis.mu.org [216.33.66.196]) by hub.freebsd.org (Postfix) with ESMTP id A27C837B405; Thu, 3 Jan 2002 09:23:11 -0800 (PST) Received: by elvis.mu.org (Postfix, from userid 1192) id 2F93B81D03; Thu, 3 Jan 2002 11:23:06 -0600 (CST) Date: Thu, 3 Jan 2002 11:23:06 -0600 From: Alfred Perlstein To: John Baldwin Cc: smp@freebsd.org Subject: Re: smp hang on -current? Message-ID: <20020103112306.D82406@elvis.mu.org> References: <20020102054324.W16101@elvis.mu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from jhb@FreeBSD.org on Wed, Jan 02, 2002 at 09:53:07AM -0800 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org * John Baldwin [020102 11:53] wrote: > > On 02-Jan-02 Alfred Perlstein wrote: > > * Alfred Perlstein [020102 05:37] wrote: > >> I have a box, I've enabled SMP, and APIC, disabled WITNESS and > >> INVARIANTS. It hangs after probing scsi right before > >> mountroot. It looks like something may be botched with interrupts: > > > > I'll have info tomorrow and any suggestions for providing that > > info would be followed. > > Rest of the dmesg might be nice. :) This is mptable+dmesg from the kernel that boots. =============================================================================== MPTable, version 2.0.15 ------------------------------------------------------------------------------- MP Floating Pointer Structure: location: BIOS physical address: 0x000ff780 signature: '_MP_' length: 16 bytes version: 1.4 checksum: 0xd4 mode: Virtual Wire ------------------------------------------------------------------------------- MP Config Table Header: physical address: 0x000f0db0 signature: 'PCMP' base table length: 276 version: 1.4 checksum: 0x8f OEM ID: 'AMI ' Product ID: 'CNB30LE ' OEM table pointer: 0x00000000 OEM table size: 0 entry count: 26 local APIC address: 0xfee00000 extended table length: 200 extended table checksum: 43 ------------------------------------------------------------------------------- MP Config Base Table Entries: -- Processors: APIC ID Version State Family Model Step Flags 0 0x11 BSP, usable 6 8 6 0x387fbff 1 0x11 AP, usable 6 8 6 0x387fbff -- Bus: Bus ID Type 0 PCI 1 PCI 2 ISA -- I/O APICs: APIC ID Version State Address 4 0x11 usable 0xfec00000 5 0x11 usable 0xfec01000 -- I/O Ints: Type Polarity Trigger Bus ID IRQ APIC ID PIN# INT active-lo level 0 1:A 5 6 INT active-lo level 1 6:B 5 12 INT active-lo level 0 4:A 5 4 INT active-lo level 0 5:A 5 5 INT active-lo level 0 15:A 4 10 INT active-lo level 1 6:A 5 13 ExtINT active-hi edge 2 0 4 0 INT active-hi edge 2 1 4 1 INT active-hi edge 2 0 4 2 INT active-hi edge 2 3 4 3 INT active-hi edge 2 4 4 4 INT active-hi edge 2 6 4 6 INT active-hi edge 2 8 4 8 INT active-hi edge 2 12 4 12 INT active-hi edge 2 13 4 13 INT active-hi edge 2 14 4 14 INT active-hi edge 2 15 4 15 -- Local Ints: Type Polarity Trigger Bus ID IRQ APIC ID PIN# ExtINT active-hi edge 2 0 255 0 NMI active-hi edge 0 0:A 255 1 ------------------------------------------------------------------------------- MP Config Extended Table Entries: -- System Address Space bus ID: 0 address type: I/O address address base: 0xc000 address range: 0x2000 -- System Address Space bus ID: 0 address type: I/O address address base: 0x0 address range: 0x100 -- System Address Space bus ID: 0 address type: memory address address base: 0xfc200000 address range: 0x2900000 -- System Address Space bus ID: 0 address type: prefetch address address base: 0xfc000000 address range: 0x100000 -- System Address Space bus ID: 1 address type: I/O address address base: 0xe000 address range: 0x1000 -- System Address Space bus ID: 1 address type: memory address address base: 0xfeb00000 address range: 0x100000 -- System Address Space bus ID: 1 address type: prefetch address address base: 0xfc100000 address range: 0x100000 -- Bus Heirarchy bus ID: 2 bus info: 0x01 parent bus ID: 0 -- Compatibility Bus Address bus ID: 0 address modifier: add predefined range: 0x00000000 -- Compatibility Bus Address bus ID: 1 address modifier: subtract predefined range: 0x00000000 -- System Address Space bus ID: 0 address type: memory address address base: 0xa0000 address range: 0x20000 -- Compatibility Bus Address bus ID: 0 address modifier: add predefined range: 0x00000001 -- Compatibility Bus Address bus ID: 1 address modifier: subtract predefined range: 0x00000001 =============================================================================== Copyright (c) 1992-2001 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.0-20011222-CURRENT #0: Sat Dec 22 11:54:13 GMT 2001 root@usw2.freebsd.org:/usr/src/sys/i386/compile/GENERIC Preloaded elf kernel "/boot/kernel.ok/kernel" at 0xc04f9000. Timecounter "i8254" frequency 1193182 Hz Timecounter "TSC" frequency 847182446 Hz CPU: Pentium III/Pentium III Xeon/Celeron (847.18-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x686 Stepping = 6 Features=0x387fbff real memory = 536870912 (524288K bytes) avail memory = 516562944 (504456K bytes) Pentium Pro MTRR support enabled Using $PIR table, 10 entries at 0xc00f51e0 npx0: on motherboard npx0: INT 16 interface pcib0: at pcibus 0 on motherboard pci0: on pcib0 pci0: at device 1.0 (no driver attached) fxp0: port 0xd400-0xd43f mem 0xfe900000-0xfe9fffff,0xfeafe000-0xfeafefff irq 9 at device 4.0 on pci0 fxp0: Ethernet address 00:e0:81:02:50:2a inphy0: on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp1: port 0xd000-0xd03f mem 0xfe700000-0xfe7fffff,0xfeafd000-0xfeafdfff irq 5 at device 5.0 on pci0 fxp1: Ethernet address 00:e0:81:02:50:2b inphy1: on miibus1 inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto isab0: port 0x580-0x58f at device 15.0 on pci0 isa0: on isab0 atapci0: port 0xffa0-0xffaf at device 15.1 on pci0 ata0: at 0x1f0 irq 14 on atapci0 ata1: at 0x170 irq 15 on atapci0 ohci0: mem 0xfeafc000-0xfeafcfff irq 10 at device 15.2 on pci0 usb0: OHCI version 1.0, legacy support usb0: on ohci0 usb0: USB revision 1.0 uhub0: (unknown) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 4 ports with 4 removable, self powered pcib1: at pcibus 1 on motherboard pci1: on pcib1 sym0: <1010-66> port 0xe400-0xe4ff mem 0xfebd8000-0xfebd9fff,0xfebe0000-0xfebe03ff irq 7 at device 6.0 on pci1 sym0: Symbios NVRAM, ID 7, Fast-80, LVD, parity checking sym0: open drain IRQ line driver, using on-chip SRAM sym0: using LOAD/STORE-based firmware. sym0: handling phase mismatch from SCRIPTS. sym1: <1010-66> port 0xe800-0xe8ff mem 0xfebf0000-0xfebf1fff,0xfebf8000-0xfebf83ff irq 10 at device 6.1 on pci1 sym1: Symbios NVRAM, ID 7, Fast-80, LVD, parity checking sym1: open drain IRQ line driver, using on-chip SRAM sym1: using LOAD/STORE-based firmware. sym1: handling phase mismatch from SCRIPTS. ata-: ata0 already exists, skipping it ata-: ata1 already exists, skipping it sc-: sc0 already exists, skipping it vga-: vga0 already exists, skipping it orm0: