From owner-freebsd-smp Sun Apr 23 11:43:28 2000 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 95F3A37B8E4 for ; Sun, 23 Apr 2000 11:43:26 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id LAA63088; Sun, 23 Apr 2000 11:43:24 -0700 (PDT) (envelope-from dillon) Date: Sun, 23 Apr 2000 11:43:24 -0700 (PDT) From: Matthew Dillon Message-Id: <200004231843.LAA63088@apollo.backplane.com> To: Bernhard Beck Cc: smp@FreeBSD.ORG, luoqi@watermarkgroup.com Subject: Re: Patch works with Dell 2450 (Re: Multiple APIC support) References: <39023622.8777F483@mindmaker.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org :I just confirmed that Luoqi's patch works against 4.0-RELEASE on a Dell :PowerEdge 2450. :The system boots fine. I ran a quick check by re-compiling the kernel. :So far no problems. :FBSD 4.0, RCC chipset, AIC 7899 and SMP all working at the same time. : :Now waiting for the commit of the patch to 4.0-STABLE. : :Thanks a lot guys, you saved my day! : :Bernhard :bbeck@mindmaker.com It looks like Luoqi has committed & MFC'd the patch! I would like to take this opportunity to congratulate Luoqi (and those that provided the excellent bug reports) on locating and fixing this problem. The issue with SMP breakage on the Dell boxes has been with us for a while and it's nice to see progress! While there are certainly going to be continuing issues with SMP compatibility (for example, the NEC box, which is still an open issue), getting it working on Dell boxes was a major hurdle due to the number of people using Dell SMP boxes. -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Apr 23 20: 4:44 2000 Delivered-To: freebsd-smp@freebsd.org Received: from TYO202.gate.nec.co.jp (TYO202.gate.nec.co.jp [202.247.6.41]) by hub.freebsd.org (Postfix) with ESMTP id 35C9137B9AB for ; Sun, 23 Apr 2000 20:04:35 -0700 (PDT) (envelope-from mihara@prd.fc.nec.co.jp) Received: from mailsv4.nec.co.jp (mailsv4-le1 [192.168.1.93]) by TYO202.gate.nec.co.jp (8.9.3/3.7W00031314) with ESMTP id MAA17754; Mon, 24 Apr 2000 12:04:32 +0900 (JST) Received: from elmer.prd.fc.nec.co.jp (root@elmer.prd.fc.nec.co.jp [10.32.193.1]) by mailsv4.nec.co.jp (8.9.3/3.7W-MAILSV4-NEC) with ESMTP id MAA02522; Mon, 24 Apr 2000 12:04:30 +0900 (JST) Received: from oz.prd.fc.nec.co.jp (oz.prd.fc.nec.co.jp [10.32.193.3]) by elmer.prd.fc.nec.co.jp (8.8.8/3.6W-00032121) with ESMTP id MAA03874; Mon, 24 Apr 2000 12:04:29 +0900 (JST) Date: Mon, 24 Apr 2000 12:03:57 +0900 Message-ID: <861z3w42ua.wl@oz.prd.fc.nec.co.jp> From: Osamu MIHARA To: dillon@apollo.backplane.com, luoqi@watermarkgroup.com Cc: bbeck@mindmaker.com, smp@FreeBSD.ORG Subject: Re: Patch works with Dell 2450 (Re: Multiple APIC support) In-Reply-To: In your message of "Sun, 23 Apr 2000 11:43:24 -0700 (PDT)" <200004231843.LAA63088@apollo.backplane.com> References: <39023622.8777F483@mindmaker.com> <200004231843.LAA63088@apollo.backplane.com> User-Agent: Wanderlust/1.1.0 (Overjoyed) SEMI/1.13.7 (Awazu) FLIM/1.13.2 (Kasanui) MULE XEmacs/21.1 (patch 9) (Canyonlands) (i386-unknown-freebsd3.4) Organization: NEC Corporation, Tokyo Japan MIME-Version: 1.0 (generated by SEMI 1.13.7 - "Awazu") Content-Type: text/plain; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org At Sun, 23 Apr 2000 11:43:24 -0700 (PDT), Matthew Dillon wrote: > It looks like Luoqi has committed & MFC'd the patch! I would like to > take this opportunity to congratulate Luoqi (and those that provided > the excellent bug reports) on locating and fixing this problem. The > issue with SMP breakage on the Dell boxes has been with us for a while > and it's nice to see progress! > > While there are certainly going to be continuing issues with SMP > compatibility (for example, the NEC box, which is still an open issue), > getting it working on Dell boxes was a major hurdle due to the number > of people using Dell SMP boxes. Banzai! Luoqi's patch also works for my NEC box with RCC LE chipset. Thank you, Luoqi and other guys! BTW, there still be a panic problm in the other NEC box (with Intel chipset, second IO-APIC enabled). As long as the second IO-APIC is disabled, this machine works fine with SMP kernel, so the problem is not critical for my case. However, I put some information for this issue. The panic seems to be caused in bus_dmamem_free() around ../../i386/i386/busdma_machdep.c:358 (5.0-CURRENT). Here is the list from symbol list. (kgdb) list *0xc02cb368 0xc02cb368 is in bus_dmamem_free (../../i386/i386/busdma_machdep.c:358). 353 * NULL 354 */ 355 if (map != NULL) 356 panic("bus_dmamem_free: Invalid map freed\n"); 357 /* XXX There is no "contigfree" and "free" doesn't work */ 358 if ((dmat->maxsize <= PAGE_SIZE) && dmat->lowaddr >= ptoa(Maxmem )) 359 free(vaddr, M_DEVBUF); 360 } 361 362 #define BUS_DMAMAP_NSEGS ((BUS_SPACE_MAXSIZE / PAGE_SIZE) + 1) And here is the boot message, 'trace' and other info from DDB. bus_dmamem_free() is called from ahc_free(). Copyright (c) 1992-2000 The FreeBSD Project. Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. FreeBSD 5.0-CURRENT #6: Mon Apr 24 10:37:29 JST 2000 root@swd-poseidon.prd.fc.nec.co.jp:/usr/src/sys/compile/POSEIDON Timecounter "i8254" frequency 1193182 Hz CPU: Pentium/P54C (98.20-MHz 586-class CPU) Origin = "GenuineIntel" Id = 0x525 Stepping = 5 Features=0x3bf real memory = 100663296 (98304K bytes) avail memory = 93732864 (91536K bytes) Programming 16 pins in IOAPIC #0 IOAPIC #0 intpin 2 -> irq 0 Programming 16 pins in IOAPIC #1 IOAPIC #1 intpin 0 -> irq 2 IOAPIC #1 intpin 1 -> irq 16 IOAPIC #1 intpin 2 -> irq 17 IOAPIC #1 intpin 3 -> irq 18 IOAPIC #1 intpin 4 -> irq 19 IOAPIC #1 intpin 5 -> irq 20 IOAPIC #1 intpin 6 -> irq 21 IOAPIC #1 intpin 7 -> irq 22 IOAPIC #1 intpin 12 -> irq 23 FreeBSD/SMP: Multiprocessor motherboard cpu0 (BSP): apic id: 0, version: 0x00030010, at 0xfee00000 cpu1 (AP): apic id: 3, version: 0x00030010, at 0xfee00000 io0 (APIC): apic id: 14, version: 0x000f0011, at 0xfec00000 io1 (APIC): apic id: 13, version: 0x000f0011, at 0xfff7ec00 Intel Pentium detected, installing workaround for F00F bug md0: Malloc disk npx0: on motherboard npx0: INT 16 interface pcib0: on motherboard pci0: on pcib0 isab0: at device 14.0 on pci0 eisa0: on isab0 mainboard0: on eisa0 slot 0 ep0: <3Com 3C579-TP EISA Network Adapter> at 0x2000-0x200f, 0x2c80-0x2c89 ep0: irq 5 (level) on eisa0 slot 2 ep0: Ethernet address 00:20:af:ee:b5:04 ep0: supplying EUI64: 00:20:af:ff:fe:ee:b5:04 isa0: on isab0 chip1: <> mem 0xfff7ec00-0xfff7efff at device 15.0 on pci0 pcib1: on motherboard pci1: on pcib1 ahc0: port 0xfc00-0xfcff mem 0xff9fe000-0xff9feff f irq 23 at device 13.0 on pci1 ahc0: Using left over BIOS settings ahc0: aic7870 Wide Channel A, SCSI Id=7, 16/255 SCBs ahc1: port 0xf800-0xf8ff mem 0xff9ff000-0xff9ffff f at device 14.0 on pci1 Fatal trap 12: page fault while in kernel mode mp_lock = 00000002; cpuid = 0; lapic.id = 00000000 fault virtual address = 0x1c fault code = supervisor read, page not present instruction pointer = 0x8:0xc02cb368 stack pointer = 0x10:0xc0411e54 frame pointer = 0x10:0xc0411e54 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (swapper) interrupt mask = net tty bio cam <- SMP: XXX kernel: type 12 trap, code=0 Stopped at bus_dmamem_free+0x18: cmpl $0x1000,0x1c(%edx) db> x/I bus_dmamem_free+0x18: cmpl $0x1000,0x1c(%edx) db> trace bus_dmamem_free(0,0,0,c0df3200,7) at bus_dmamem_free+0x18 ahc_free(c0df3200) at ahc_free+0x44b ahc_free(c0df3200,c0df1048,c0dfcf00,c0dfcc00,4) at ahc_free+0xd read_seeprom(c0dfcc00,c0dfcc00,c0df4680,c0dfcf00,0) at read_seeprom+0x136c device_probe_and_attach(c0dfcc00) at device_probe_and_attach+0x9e bus_generic_attach(c0dfcf00,c0dfcf00,c0df4780,c0df4680,0) at bus_generic_attach+ 0x16 device_probe_and_attach(c0dfcf00) at device_probe_and_attach+0x9e bus_generic_attach(c0df4680,c0df4680,c0a7f800,c0df4780,0) at bus_generic_attach+ 0x16 device_probe_and_attach(c0df4680) at device_probe_and_attach+0x9e bus_generic_attach(c0df4780,c0d95048,c0411f70,c01abcea,c0df4780) at bus_generic_ attach+0x16 rel_mplock(c0df4780,c0df4780,c0368770,419000,0) at rel_mplock+0x1ad device_probe_and_attach(c0df4780) at device_probe_and_attach+0x9e root_bus_configure(c0a7f800,c033d72c,0) at root_bus_configure+0x16 atomic_subtract_long(0,40fc00,419000,0,c0122136) at atomic_subtract_long+0x3e mi_startup(c0411fb4,0,0,a04,0) at mi_startup+0x70 btext() at btext+0xb6 db> show all procs pid proc addr uid ppid pgrp flag stat wmesg wchan cmd 0 c0390720 c0410000 0 0 0 000204 2 swapper db> show registers cs 0x8 gd_npxproc ds 0x10 gd_switchtime es 0x10 gd_switchtime fs 0x18 gd_common_tss ss 0x10 gd_switchtime eax 0 ecx 0x1 edx 0 ebx 0 esp 0xc0411e54 ebp 0xc0411e54 esi 0xc0df32fc edi 0x12 gd_switchtime+0x2 eip 0xc02cb368 bus_dmamem_free+0x18 efl 0x90246 bus_dmamem_free+0x18: cmpl $0x1000,0x1c(%edx) db> show map/f Task map 0xc02cb368: pmap=0x458b0000, nentries=60485, version=205666568 Fatal trap 12: page fault while in kernel mode mp_lock = 00000003; cpuid = 0; lapic.id = 00000000 fault virtual address = 0x7700001c fault code = supervisor read, page not present instruction pointer = 0x8:0xc029fca4 stack pointer = 0x10:0xc0411c7c frame pointer = 0x10:0xc0411c88 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (swapper) interrupt mask = net tty bio cam <- SMP: XXX kernel: type 12 trap, code=0 Stopped at bus_dmamem_free+0x18: cmpl $0x1000,0x1c(%edx) db> show object/f Object 0xc02cb368: type=139, size=0x768dc3, res=-1983633149, ref=-2082109099, flags=0xc120 Fatal trap 12: page fault while in kernel mode mp_lock = 00000004; cpuid = 0; lapic.id = 00000000 fault virtual address = 0x7d83f88d fault code = supervisor read, page not present instruction pointer = 0x8:0xc02a2638 stack pointer = 0x10:0xc0411c5c frame pointer = 0x10:0xc0411c88 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (swapper) interrupt mask = net tty bio cam <- SMP: XXX kernel: type 12 trap, code=0 Stopped at bus_dmamem_free+0x18: cmpl $0x1000,0x1c(%edx) db> -- Osamu MIHARA // NEC Printers Division To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 25 4:55: 7 2000 Delivered-To: freebsd-smp@freebsd.org Received: from lor.watermarkgroup.com (lor.watermarkgroup.com [207.202.73.33]) by hub.freebsd.org (Postfix) with ESMTP id B58F837BCC0 for ; Tue, 25 Apr 2000 04:55:04 -0700 (PDT) (envelope-from luoqi@watermarkgroup.com) Received: (from luoqi@localhost) by lor.watermarkgroup.com (8.10.1/8.10.1) id e3PBt0D19565; Tue, 25 Apr 2000 07:55:00 -0400 (EDT) Date: Tue, 25 Apr 2000 07:55:00 -0400 (EDT) From: Luoqi Chen Message-Id: <200004251155.e3PBt0D19565@lor.watermarkgroup.com> To: sthaug@nethelp.no Subject: Re: Multiple APIC support (RE: Issues with Spankin' new Poweredge 4400 w/ dual 866's and 133MHz FSB?) Cc: freebsd-smp@FreeBSD.ORG, mihara@prd.fc.nec.co.jp, smp@csn.net Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > The pmap.c code looks somewhat different in 3.4-STABLE, and it wasn't > obvious to me what the complete patch for 3.4-STABLE would be. > The fix for non-page-aligned ioapic is to add the offset within the page to get the mapped ioapic address. For 3.4-STABLE, it would be ioapic[i] = (ioapic_t *)&SMP_ioapic[j * PAGE_SIZE + (io_apic_addres[0] & PAGE_MASK)]; I don't have any machine myself to test it with, so if you can confirm it won't break anything on your machine, I would MFC the fix to 3.4-STABLE. -lq To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 25 8:10:22 2000 Delivered-To: freebsd-smp@freebsd.org Received: from Ilsa.StevesCafe.com (Ilsa.StevesCafe.com [206.168.13.65]) by hub.freebsd.org (Postfix) with ESMTP id 138FB37BD3B for ; Tue, 25 Apr 2000 08:10:20 -0700 (PDT) (envelope-from smp@Ilsa.StevesCafe.com) Received: from Ilsa.StevesCafe.com (localhost [127.0.0.1]) by Ilsa.StevesCafe.com (8.9.3/8.9.3) with ESMTP id JAA10136; Tue, 25 Apr 2000 09:09:59 -0600 (MDT) (envelope-from smp@Ilsa.StevesCafe.com) Message-Id: <200004251509.JAA10136@Ilsa.StevesCafe.com> X-Mailer: exmh version 2.0.2 2/24/98 From: Steve Passe To: Luoqi Chen Cc: sthaug@nethelp.no, freebsd-smp@FreeBSD.ORG, mihara@prd.fc.nec.co.jp Subject: Re: Multiple APIC support (RE: Issues with Spankin' new Poweredge 4400 w/ dual 866's and 133MHz FSB?) In-reply-to: Your message of "Tue, 25 Apr 2000 07:55:00 EDT." <200004251155.e3PBt0D19565@lor.watermarkgroup.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 25 Apr 2000 09:09:59 -0600 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > The pmap.c code looks somewhat different in 3.4-STABLE, and it wasn't > > obvious to me what the complete patch for 3.4-STABLE would be. > > > The fix for non-page-aligned ioapic is to add the offset within the page > to get the mapped ioapic address. For 3.4-STABLE, it would be > ioapic[i] = (ioapic_t *)&SMP_ioapic[j * PAGE_SIZE > + (io_apic_addres[0] & PAGE_MASK)]; ^^^ I don't have 3.4 code handy to see the outer loop, but didn't the const 0 go away here? > I don't have any machine myself to test it with, so if you can confirm > it won't break anything on your machine, I would MFC the fix to 3.4-STABLE. -- Steve Passe | powered by smp@csn.net | Symmetric MultiProcessor FreeBSD To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 25 8:29:56 2000 Delivered-To: freebsd-smp@freebsd.org Received: from lor.watermarkgroup.com (lor.watermarkgroup.com [207.202.73.33]) by hub.freebsd.org (Postfix) with ESMTP id 89EDE37BDC0 for ; Tue, 25 Apr 2000 08:29:49 -0700 (PDT) (envelope-from luoqi@watermarkgroup.com) Received: (from luoqi@localhost) by lor.watermarkgroup.com (8.10.1/8.10.1) id e3PFTgp22603; Tue, 25 Apr 2000 11:29:42 -0400 (EDT) Date: Tue, 25 Apr 2000 11:29:42 -0400 (EDT) From: Luoqi Chen Message-Id: <200004251529.e3PFTgp22603@lor.watermarkgroup.com> To: smp@csn.net Subject: Re: Multiple APIC support (RE: Issues with Spankin' new Poweredge 4400 w/ dual 866's and 133MHz FSB?) Cc: freebsd-smp@FreeBSD.ORG, mihara@prd.fc.nec.co.jp, sthaug@nethelp.no Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > > The pmap.c code looks somewhat different in 3.4-STABLE, and it wasn't > > > obvious to me what the complete patch for 3.4-STABLE would be. > > > > > The fix for non-page-aligned ioapic is to add the offset within the page > > to get the mapped ioapic address. For 3.4-STABLE, it would be > > ioapic[i] = (ioapic_t *)&SMP_ioapic[j * PAGE_SIZE > > + (io_apic_addres[0] & PAGE_MASK)]; > ^^^ > I don't have 3.4 code handy to see the outer loop, but didn't the const 0 go > away here? > > > I don't have any machine myself to test it with, so if you can confirm > > it won't break anything on your machine, I would MFC the fix to 3.4-STABLE. > > -- > Steve Passe | powered by > smp@csn.net | Symmetric MultiProcessor FreeBSD > > Sorry, the 0 should be i (cut-n-paste error). -luoqi To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 25 12:10:37 2000 Delivered-To: freebsd-smp@freebsd.org Received: from smtp03.primenet.com (smtp03.primenet.com [206.165.6.133]) by hub.freebsd.org (Postfix) with ESMTP id 02A8437B810 for ; Tue, 25 Apr 2000 12:10:34 -0700 (PDT) (envelope-from tlambert@usr05.primenet.com) Received: (from daemon@localhost) by smtp03.primenet.com (8.9.3/8.9.3) id MAA10014; Tue, 25 Apr 2000 12:10:25 -0700 (MST) Received: from usr05.primenet.com(206.165.6.205) via SMTP by smtp03.primenet.com, id smtpdAAAf3a4Gt; Tue Apr 25 12:10:17 2000 Received: (from tlambert@localhost) by usr05.primenet.com (8.8.5/8.8.5) id MAA05939; Tue, 25 Apr 2000 12:10:19 -0700 (MST) From: Terry Lambert Message-Id: <200004251910.MAA05939@usr05.primenet.com> Subject: Re: I/O APIC To: mihara@prd.fc.nec.co.jp (Osamu MIHARA) Date: Tue, 25 Apr 2000 19:10:19 +0000 (GMT) Cc: gallatin@cs.duke.edu, freebsd-smp@FreeBSD.ORG In-Reply-To: <868zy9v9cs.wl@oz.prd.fc.nec.co.jp> from "Osamu MIHARA" at Apr 20, 2000 10:39:47 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Not to chime in late here... > > When installing Solaris/x86 on one of our PowerEdge 2400s (SMP > > capable, 2 I/O APICs, 1 CPU), I noticed that it uses the I/O APICs for > > interrupts rather than the normal isa irq x. Is there an advantage to > > this? > > One advantage is that multiple I/O-APICs can handle more interrupts > than traditional PIC. If you use PIC, it can handles only 16 > interrupts at most, and you may need share a interrupt for some > devices. With multiple IO-APICs, you don't need to share interrupts, > and it does not pay for overheads of interrupt sharing, resulting in > better I/O throughput, even with single CPU. Another advantage, according to the Intel MultiProcessing spec., version 1.4, is that it allows you to run in "virtual wire mode". In "virtual wire mode", interrupt processing is symmetric between processors. In practice, you really want to split interrupt processing in virtual wire mode, but non-symmetrically, using IPI's to signal the "correct" processor, with the net effect that you "lock" interrupt handling for each of the 4 cards to each one of 4 processors. This is the method NT used on its 4 ethernet card, 4 processor machine in the infamous NetCraft and Ziff-Davis tests, where NT was able to handily beat Linux, running with the same hardware configuration. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 25 13:10:55 2000 Delivered-To: freebsd-smp@freebsd.org Received: from caladan.tdx.co.uk (caladan.tdx.co.uk [195.188.177.4]) by hub.freebsd.org (Postfix) with ESMTP id D478637B8B0 for ; Tue, 25 Apr 2000 13:10:27 -0700 (PDT) (envelope-from kpielorz@tdx.co.uk) Received: from tdx.co.uk (lorca-tx.tdx.co.uk [195.188.177.242]) by caladan.tdx.co.uk (8.10.0/8.10.0/Kp) with ESMTP id e3PKFGG42777 for ; Tue, 25 Apr 2000 21:15:16 +0100 (BST) Message-ID: <3905FB83.E6431D88@tdx.co.uk> Date: Tue, 25 Apr 2000 21:09:39 +0100 From: Karl Pielorz Organization: TDX - The Digital eXchange X-Mailer: Mozilla 4.7 [en-gb] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: freebsd-smp@freebsd.org Subject: apic / ahc workarounds in 4.0-Stable? Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, Anyone know if the APIC workarounds have been 're-worked' into the ahc drivers in 4.0-Stable yet? Apparently these were left out (my machine used to display such delights as "Broken APIC routing on xxxxxx registering second handler...." [can't you tell I'm not at the machine?]). This has the net affect of making all my SCSI drives inaccessible under SMP / 4.0-Stable (as the additional cards didn't see any interrupts)... I believe someone was putting them back in, or re-working them etc? Sorry if this is so vague - whilst not being totally out of my depth, it's sufficiently past my sell-by date - I wouldn't even know which file to watch for changes [Hints appreciated?] Cheers, -Karl To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 25 15:17:34 2000 Delivered-To: freebsd-smp@freebsd.org Received: from verdi.nethelp.no (verdi.nethelp.no [158.36.41.162]) by hub.freebsd.org (Postfix) with SMTP id 91FC837B56F for ; Tue, 25 Apr 2000 15:17:14 -0700 (PDT) (envelope-from sthaug@nethelp.no) Received: (qmail 99426 invoked by uid 1001); 25 Apr 2000 22:17:12 +0000 (GMT) To: luoqi@watermarkgroup.com Cc: smp@csn.net, freebsd-smp@FreeBSD.ORG, mihara@prd.fc.nec.co.jp Subject: Re: Multiple APIC support (RE: Issues with Spankin' new Poweredge 4400 w/ dual 866's and 133MHz FSB?) From: sthaug@nethelp.no In-Reply-To: Your message of "Tue, 25 Apr 2000 11:29:42 -0400 (EDT)" References: <200004251529.e3PFTgp22603@lor.watermarkgroup.com> X-Mailer: Mew version 1.05+ on Emacs 19.34.2 Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Date: Wed, 26 Apr 2000 00:17:12 +0200 Message-ID: <99424.956701032@verdi.nethelp.no> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > > > The pmap.c code looks somewhat different in 3.4-STABLE, and it wasn't > > > > obvious to me what the complete patch for 3.4-STABLE would be. > > > > > > > The fix for non-page-aligned ioapic is to add the offset within the page > > > to get the mapped ioapic address. For 3.4-STABLE, it would be > > > ioapic[i] = (ioapic_t *)&SMP_ioapic[j * PAGE_SIZE > > > + (io_apic_addres[0] & PAGE_MASK)]; > > ^^^ > > I don't have 3.4 code handy to see the outer loop, but didn't the const 0 go > > away here? > > > > > I don't have any machine myself to test it with, so if you can confirm > > > it won't break anything on your machine, I would MFC the fix to 3.4-STABLE. > > > > -- > > Steve Passe | powered by > > smp@csn.net | Symmetric MultiProcessor FreeBSD > > > > > Sorry, the 0 should be i (cut-n-paste error). OK, I ended up with the following patch for 3.4-STABLE. This runs SMP nicely on the IBM Netfinity 5600 with the RCC chipset. Verbose boot messages included at the end. *** pmap.c.orig Sat Apr 1 23:58:17 2000 --- pmap.c Tue Apr 25 23:59:29 2000 *************** *** 434,441 **** for (j = 0; j < 16; j++) { /* same page frame as a previous IO apic? */ if (((vm_offset_t)SMP_prvpt[j + 16] & PG_FRAME) == ! (io_apic_address[0] & PG_FRAME)) { ! ioapic[i] = (ioapic_t *)&SMP_ioapic[j * PAGE_SIZE]; break; } /* use this slot if available */ --- 434,442 ---- for (j = 0; j < 16; j++) { /* same page frame as a previous IO apic? */ if (((vm_offset_t)SMP_prvpt[j + 16] & PG_FRAME) == ! (io_apic_address[i] & PG_FRAME)) { ! ioapic[i] = (ioapic_t *)&SMP_ioapic[j * PAGE_SIZE ! + (io_apic_address[i] & PAGE_MASK)]; break; } /* use this slot if available */ Steinar Haug, Nethelp consulting, sthaug@nethelp.no ---------------------------------------------------------------------- Copyright (c) 1992-1999 FreeBSD Inc. Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. FreeBSD 3.4-STABLE #2: Wed Apr 26 00:05:33 CEST 2000 sthaug@mail1.enitel.no:/local/freebsd/stable3/src/sys/compile/NF5600_SMP Calibrating clock(s) ... TSC clock: 599613673 Hz, i8254 clock: 1192987 Hz CLK_USE_I8254_CALIBRATION not specified - using default frequency Timecounter "i8254" frequency 1193182 Hz CLK_USE_TSC_CALIBRATION not specified - using old calibration method CPU: Pentium III/Pentium III Xeon (599.71-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x681 Stepping = 1 Features=0x387fbff,MMX,FXSR,> real memory = 939524096 (917504K bytes) Physical memory chunk(s): 0x00001000 - 0x0009cfff, 638976 bytes (156 pages) 0x0030d000 - 0x37ff7fff, 936292352 bytes (228587 pages) avail memory = 911634432 (890268K bytes) Programming 16 pins in IOAPIC #0 Programming 16 pins in IOAPIC #1 IOAPIC #1 intpint 0 -> irq 16 IOAPIC #1 intpint 2 -> irq 17 IOAPIC #1 intpint 3 -> irq 18 IOAPIC #1 intpint 4 -> irq 21 IOAPIC #1 intpint 5 -> irq 22 IOAPIC #1 intpint 6 -> irq 23 IOAPIC #1 intpint 7 -> irq -1 IOAPIC #1 intpint 8 -> irq -1 IOAPIC #1 intpint 9 -> irq -1 IOAPIC #1 intpint 10 -> irq -1 IOAPIC #1 intpint 11 -> irq 5 IOAPIC #1 intpint 12 -> irq 19 IOAPIC #1 intpint 13 -> irq 20 SMP: CPU0 apic_initialize(): lint0: 0x00000700 lint1: 0x00010400 TPR: 0x00000010 SVR: 0x000001ff FreeBSD/SMP: Multiprocessor motherboard cpu0 (BSP): apic id: 1, version: 0x00040011, at 0xfee00000 cpu1 (AP): apic id: 0, version: 0x00040011, at 0xfee00000 io0 (APIC): apic id: 14, version: 0x000f0011, at 0xfec00000 io1 (APIC): apic id: 15, version: 0x000f0011, at 0xfec01000 Found BIOS32 Service Directory header at 0xc00fd300 Entry = 0xfd311 (0xc00fd311) Rev = 0 Len = 1 PCI BIOS entry at 0xd34c DMI header at 0xc00fdfd0 Version 2.1 Table at 0xf45da, 62 entries, 2294 bytes Other BIOS signatures found: ACPI: 00000000 $PnP: 000fdca0 Preloaded elf kernel "kernel" at 0xc02f5000. Pentium Pro MTRR support enabled Math emulator present SMP: CPU0 bsp_apic_configure(): lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000010 SVR: 0x000001ff pci_open(1): mode 1 addr port (0x0cf8) is 0x00000070 pci_open(1a): mode1res=0x80000000 (0x80000000) pci_cfgcheck: device 0 [class=060000] [hdr=80] is there (id=00091166) Probing for devices on PCI bus 0: found-> vendor=0x1166, dev=0x0009, revid=0x05 class=06-00-00, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 chip0: rev 0x05 on pci0.0.0 found-> vendor=0x1166, dev=0x0009, revid=0x05 class=06-00-00, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 chip1: rev 0x05 on pci0.0.1 found-> vendor=0x5333, dev=0x8904, revid=0x01 class=03-00-00, hdrtype=0x00, mfdev=0 subordinatebus=0 secondarybus=0 intpin=a, irq=255 map[0]: type 1, range 32, base f8000000, size 26 vga0: rev 0x01 int a irq 255 on pci0.1.0 Freeing (NOT implemented) redirected PCI irq 11. found-> vendor=0x1022, dev=0x2000, revid=0x43 class=02-00-00, hdrtype=0x00, mfdev=0 subordinatebus=0 secondarybus=0 intpin=a, irq=5 map[0]: type 4, range 32, base 00002000, size 5 map[1]: type 1, range 32, base febffc00, size 5 lnc1: rev 0x43 int a irq 5 on pci0.2.0 Freeing (NOT implemented) redirected PCI irq 10. found-> vendor=0x9005, dev=0x0010, revid=0x00 class=01-00-00, hdrtype=0x00, mfdev=0 subordinatebus=0 secondarybus=0 intpin=a, irq=16 map[0]: type 4, range 32, base 00002200, size 8 map[1]: type 1, range 64, base febfe000, size 12 map[2]: type 0, range 0, base 00000000, size 0 ahc0: rev 0x00 int a irq 16 on pci0.9.0 ahc0: Reading SEEPROM...done. ahc0: BIOS eeprom is present ahc0: Secondary High byte termination Enabled ahc0: Secondary Low byte termination Enabled ahc0: Primary Low Byte termination Enabled ahc0: Primary High Byte termination Enabled ahc0: aic7890/91 Wide Channel A, SCSI Id=7, 16/255 SCBs ahc0: Downloading Sequencer Program... 399 instructions downloaded found-> vendor=0x1011, dev=0x0024, revid=0x03 class=06-04-00, hdrtype=0x01, mfdev=0 subordinatebus=1 secondarybus=1 chip2: rev 0x03 on pci0.10.0 found-> vendor=0x1166, dev=0x0200, revid=0x4f class=06-01-00, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 chip3: rev 0x4f on pci0.15.0 found-> vendor=0x1166, dev=0x0211, revid=0x00 class=01-01-8a, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 map[0]: type 4, range 32, base 00000840, size 4 found-> vendor=0x1166, dev=0x0220, revid=0x04 class=0c-03-10, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 intpin=a, irq=9 map[0]: type 1, range 32, base ff700000, size 12 Probing for devices on PCI bus 1: Freeing (NOT implemented) redirected PCI irq 15. found-> vendor=0x8086, dev=0x1229, revid=0x05 class=02-00-00, hdrtype=0x00, mfdev=0 subordinatebus=0 secondarybus=0 intpin=a, irq=17 map[0]: type 3, range 32, base fea00000, size 12 map[1]: type 4, range 32, base 00003100, size 5 map[2]: type 1, range 32, base fe800000, size 20 fxp0: rev 0x05 int a irq 17 on pci1.4.0 fxp0: Ethernet address 00:d0:b7:5f:b7:a8 bpf: fxp0 attached Freeing (NOT implemented) redirected PCI irq 11. found-> vendor=0x8086, dev=0x1229, revid=0x05 class=02-00-00, hdrtype=0x00, mfdev=0 subordinatebus=0 secondarybus=0 intpin=a, irq=18 map[0]: type 3, range 32, base fea01000, size 12 map[1]: type 4, range 32, base 00003120, size 5 map[2]: type 1, range 32, base fe900000, size 20 fxp1: rev 0x05 int a irq 18 on pci1.5.0 fxp1: Ethernet address 00:d0:b7:5f:b7:a9 bpf: fxp1 attached Probing for devices on PCI bus 2: Freeing (NOT implemented) redirected PCI irq 10. found-> vendor=0x9005, dev=0x005f, revid=0x00 class=01-00-00, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 intpin=a, irq=19 map[0]: type 4, range 32, base 00004b00, size 8 map[1]: type 1, range 64, base f6fff000, size 12 map[2]: type 0, range 0, base 00000000, size 0 ahc1: rev 0x00 int a irq 19 on pci2.3.0 bogus MP table, 2 IO APIC pins connected to the same PCI device or ISA/EISA interrupt Registered extra interrupt handler for int 20 (in addition to int 19) ahc1: Reading SEEPROM...done. ahc1: Manual LVD Termination ahc1: BIOS eeprom is present ahc1: Secondary High byte termination Enabled ahc1: Secondary Low byte termination Enabled ahc1: Primary Low Byte termination Enabled ahc1: Primary High Byte termination Enabled ahc1: aic7896/97 Wide Channel A, SCSI Id=7, 16/255 SCBs ahc1: Downloading Sequencer Program... 399 instructions downloaded Freeing (NOT implemented) redirected PCI irq 10. found-> vendor=0x9005, dev=0x005f, revid=0x00 class=01-00-00, hdrtype=0x00, mfdev=1 subordinatebus=0 secondarybus=0 intpin=a, irq=19 map[0]: type 4, range 32, base 00004c00, size 8 map[1]: type 1, range 64, base f6ffe000, size 12 map[2]: type 0, range 0, base 00000000, size 0 ahc2: rev 0x00 int a irq 19 on pci2.3.1 using shared irq19. bogus MP table, 2 IO APIC pins connected to the same PCI device or ISA/EISA interrupt using shared irq20. Registered extra interrupt handler for int 20 (in addition to int 19) ahc2: Reading SEEPROM...done. ahc2: Manual LVD Termination ahc2: BIOS eeprom is present ahc2: Secondary High byte termination Enabled ahc2: Secondary Low byte termination Enabled ahc2: Primary Low Byte termination Enabled ahc2: Primary High Byte termination Enabled ahc2: aic7896/97 Wide Channel B, SCSI Id=7, 16/255 SCBs ahc2: Downloading Sequencer Program... 399 instructions downloaded Freeing (NOT implemented) redirected PCI irq 15. found-> vendor=0x1385, dev=0x620a, revid=0x01 class=02-00-00, hdrtype=0x00, mfdev=0 subordinatebus=0 secondarybus=0 intpin=a, irq=21 map[0]: type 1, range 32, base f6ff8000, size 14 ti0: rev 0x01 int a irq 21 on pci2.5.0 ti0: Ethernet address: 00:a0:cc:73:32:dd bpf: ti0 attached Probing for devices on PCI bus 3: Probing for devices on the ISA bus: sc0 on isa sc0: fb0 kbd0 sc0: VGA color <16 virtual consoles, flags=0x0> atkbdc0 at 0x60-0x6f on motherboard atkbd0 irq 1 on isa kbd0: atkbd0, generic (3), config:0x0, flags:0x3f0000 psm0: current command byte:0065 kbdc: TEST_AUX_PORT status:0000 kbdc: RESET_AUX return code:00fe kbdc: RESET_AUX return code:00fe kbdc: RESET_AUX return code:00fe kbdc: DIAGNOSE status:0055 kbdc: TEST_KBD_PORT status:0000 psm0: failed to reset the aux device. psm0 not found sio0: irq maps: 0x1 0x11 0x1 0x1 sio0 at 0x3f8-0x3ff irq 4 flags 0x10 on isa sio0: type 16550A sio1: irq maps: 0x1 0x9 0x1 0x1 sio1 at 0x2f8-0x2ff irq 3 on isa sio1: type 16550A fdc0 at 0x3f0-0x3f7 irq 6 drq 2 on isa fdc0: FIFO enabled, 8 bytes threshold fd0: 1.44MB 3.5in lnc0 not found at 0x280 vga0 at 0x3b0-0x3df maddr 0xa0000 msize 131072 on isa fb0: vga0, vga, type:VGA (5), flags:0x7007f fb0: port:0x3b0-0x3df, crtc:0x3d4, mem:0xa0000 0x20000 fb0: init mode:24, bios mode:3, current mode:24 fb0: window:0xc00b8000 size:32k gran:32k, buf:0x0 size:0k VGA parameters upon power-up 50 18 10 00 00 00 03 00 02 67 5f 4f 50 82 55 81 bf 1f 00 4f 0d 0e 00 00 07 30 9c 8e 8f 28 1f 96 b9 a3 ff 00 01 02 03 04 05 14 07 38 39 3a 3b 3c 3d 3e 3f 0c 00 0f 08 00 00 00 00 00 10 0e 00 ff VGA parameters in BIOS for mode 24 50 18 10 00 10 00 03 00 02 67 5f 4f 50 82 55 81 bf 1f 00 4f 0d 0e 00 00 00 00 9c 8e 8f 28 1f 96 b9 a3 ff 00 01 02 03 04 05 14 07 38 39 3a 3b 3c 3d 3e 3f 0c 00 0f 08 00 00 00 00 00 10 0e 00 ff EGA/VGA parameters to be used for mode 24 50 18 10 00 10 00 03 00 02 67 5f 4f 50 82 55 81 bf 1f 00 4f 0d 0e 00 00 00 00 9c 8e 8f 28 1f 96 b9 a3 ff 00 01 02 03 04 05 14 07 38 39 3a 3b 3c 3d 3e 3f 0c 00 0f 08 00 00 00 00 00 10 0e 00 ff npx0 on motherboard npx0: INT 16 interface imasks: bio c8000040, tty c300001a, net c6260000 SMP: enabled INTs: 1, 3, 4, 6, 16, 17, 18, 19, 20, 21, apic_imen: 0x00c0ffa5 BIOS Geometries: 0:03fefe3f 0..1022=1023 cylinders, 0..254=255 heads, 1..63=63 sectors 1:03fefe3f 0..1022=1023 cylinders, 0..254=255 heads, 1..63=63 sectors 2:03fefe3f 0..1022=1023 cylinders, 0..254=255 heads, 1..63=63 sectors 3:03fefe3f 0..1022=1023 cylinders, 0..254=255 heads, 1..63=63 sectors 4:03fefe3f 0..1022=1023 cylinders, 0..254=255 heads, 1..63=63 sectors 0 accounted for Device configuration finished. APIC_IO: Testing 8254 interrupt delivery APIC_IO: Broken MP table detected: 8254 is not connected to IO APIC int pin 2 APIC_IO: routing 8254 via 8259 on pin 0 bpf: tun0 attached bpf: ppp0 attached new masks: bio c8000040, tty c300001a, net c726001a bpf: lo0 attached Waiting 3 seconds for SCSI devices to settle (noperiph:ahc0:0:-1:-1): SCSI bus reset delivered. 0 SCBs aborted. (noperiph:ahc1:0:-1:-1): SCSI bus reset delivered. 0 SCBs aborted. (noperiph:ahc2:0:-1:-1): SCSI bus reset delivered. 0 SCBs aborted. SMP: AP CPU #1 Launched! SMP: CPU1 apic_initialize(): lint0: 0x00010700 lint1: 0x00010400 TPR: 0x00000010 SVR: 0x000001ff Sending WDTR! ahc1: target 0 using 16bit transfers ahc1: target 0 synchronous at 40.0MHz, offset = 0xf Sending SDTR! Sending WDTR! ahc1: target 1 using 16bit transfers ahc1: target 1 synchronous at 40.0MHz, offset = 0xf Sending SDTR! ahc1: target 0 using asynchronous transfers ahc1: target 0 synchronous at 40.0MHz, offset = 0xf ahc1: target 1 using asynchronous transfers ahc1: target 1 synchronous at 40.0MHz, offset = 0xf ahc1: target 1 using asynchronous transfers ahc1: target 1 synchronous at 40.0MHz, offset = 0xf ahc1: target 0 using asynchronous transfers ahc1: target 0 synchronous at 40.0MHz, offset = 0xf ahc2: target 0 using 16bit transfers ahc2: target 0 synchronous at 20.0MHz, offset = 0xf ahc2: target 0 using asynchronous transfers ahc2: target 0 synchronous at 20.0MHz, offset = 0xf ahc0: target 0 using 16bit transfers ahc0: target 0 synchronous at 20.0MHz, offset = 0xf ahc2: target 0 using asynchronous transfers ahc2: target 0 synchronous at 20.0MHz, offset = 0xf ahc0: target 0 using asynchronous transfers ahc0: target 0 synchronous at 20.0MHz, offset = 0xf ahc0: target 0 using asynchronous transfers ahc0: target 0 synchronous at 20.0MHz, offset = 0xf sa0 at ahc0 bus 0 target 0 lun 0 sa0: Removable Sequential Access SCSI-2 device sa0: Serial Number 0009280111 sa0: 40.000MB/s transfers (20.000MHz, offset 15, 16bit) pass0 at ahc0 bus 0 target 0 lun 0 pass0: Removable Sequential Access SCSI-2 device pass0: Serial Number 0009280111 pass0: 40.000MB/s transfers (20.000MHz, offset 15, 16bit) pass1 at ahc0 bus 0 target 0 lun 1 pass1: Removable Changer SCSI-2 device pass1: Serial Number 3761633968 pass1: 40.000MB/s transfers (20.000MHz, offset 15, 16bit) pass2 at ahc1 bus 0 target 0 lun 0 pass2: Fixed Direct Access SCSI-2 device pass2: Serial Number 3AL11X6M000070213A84 pass2: 80.000MB/s transfers (40.000MHz, offset 15, 16bit), Tagged Queueing Enabled pass3 at ahc1 bus 0 target 1 lun 0 pass3: Fixed Direct Access SCSI-2 device pass3: Serial Number 3AL0Q3QW00007013J51Y pass3: 80.000MB/s transfers (40.000MHz, offset 15, 16bit), Tagged Queueing Enabled pass4 at ahc1 bus 0 target 15 lun 0 pass4: Fixed Processor SCSI-2 device pass4: Serial Number 1 pass4: 3.300MB/s transfers pass5 at ahc2 bus 0 target 0 lun 0 pass5: Fixed Direct Access SCSI-2 device pass5: Serial Number 00000040038804170017180d5c620300 pass5: 40.000MB/s transfers (20.000MHz, offset 15, 16bit), Tagged Queueing Enabled pass6 at ahc2 bus 0 target 0 lun 1 pass6: Fixed Direct Access SCSI-2 device pass6: Serial Number 00000040076504170017183091d90201 pass6: 40.000MB/s transfers (20.000MHz, offset 15, 16bit), Tagged Queueing Enabled pass7 at ahc2 bus 0 target 0 lun 2 pass7: Fixed Direct Access SCSI-2 device pass7: Serial Number 000000400765041700171a3514ab0102 pass7: 40.000MB/s transfers (20.000MHz, offset 15, 16bit), Tagged Queueing Enabled pass1 at ahc0 bus 0 target 0 lun 1 pass1: Removable Changer SCSI-2 device pass1: Serial Number 3761633968 pass1: 40.000MB/s transfers (20.000MHz, offset 15, 16bit) pass4 at ahc1 bus 0 target 15 lun 0 pass4: Fixed Processor SCSI-2 device pass4: Serial Number 1 pass4: 3.300MB/s transfers da2 at ahc2 bus 0 target 0 lun 0 da2: Fixed Direct Access SCSI-2 device da2: Serial Number 00000040038804170017180d5c620300 da2: 40.000MB/s transfers (20.000MHz, offset 15, 16bit), Tagged Queueing Enabled da2: 140014MB (286748672 512 byte sectors: 255H 63S/T 17849C) da3 at ahc2 bus 0 target 0 lun 1 da3: Fixed Direct Access SCSI-2 device da3: Serial Number 00000040076504170017183091d90201 da3: 40.000MB/s transfers (20.000MHz, offset 15, 16bit), Tagged Queueing Enabled da3: 140014MB (286748672 512 byte sectors: 255H 63S/T 17849C) da4 at ahc2 bus 0 target 0 lun 2 da4: Fixed Direct Access SCSI-2 device da4: Serial Number 000000400765041700171a3514ab0102 da4: 40.000MB/s transfers (20.000MHz, offset 15, 16bit), Tagged Queueing Enabled da4: 140014MB (286748672 512 byte sectors: 255H 63S/T 17849C) da0 at ahc1 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device da0: Serial Number 3AL11X6M000070213A84 da0: 80.000MB/s transfers (40.000MHz, offset 15, 16bit), Tagged Queueing Enabled da0: 8678MB (17774160 512 byte sectors: 255H 63S/T 1106C) da1 at ahc1 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-2 device da1: Serial Number 3AL0Q3QW00007013J51Y da1: 80.000MB/s transfers (40.000MHz, offset 15, 16bit), Tagged Queueing Enabled da1: 8678MB (17774160 512 byte sectors: 255H 63S/T 1106C) Considering FFS root f/s. changing root device to da0s1a da0s1: type 0xa5, start 63, end = 17767889, size 17767827 : OK start_init: trying /sbin/init da1s1: type 0xa5, start 63, end = 17767889, size 17767827 : OK da1s1: type 0xa5, start 63, end = 17767889, size 17767827 : OK da1s1: type 0xa5, start 63, end = 17767889, size 17767827 : OK da1s1: type 0xa5, start 63, end = 17767889, size 17767827 : OK da1s1: type 0xa5, start 63, end = 17767889, size 17767827 : OK cmd ntpd pid 132 tried to use non-present sched_get_priority_max cmd ntpd pid 132 tried to use non-present sched_setscheduler To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 5:26:15 2000 Delivered-To: freebsd-smp@freebsd.org Received: from relativity.student.utwente.nl (wit389306.student.utwente.nl [130.89.234.166]) by hub.freebsd.org (Postfix) with ESMTP id 9B1B537B53F for ; Thu, 27 Apr 2000 05:26:12 -0700 (PDT) (envelope-from djb@wit389306.student.utwente.nl) Received: by relativity.student.utwente.nl (Postfix, from userid 1000) id 55A4B5DEE; Thu, 27 Apr 2000 14:26:13 +0200 (CEST) Date: Thu, 27 Apr 2000 14:26:13 +0200 From: Dave Boers To: freebsd-smp@freebsd.org Subject: hlt instructions and temperature issues Message-ID: <20000427142613.A7795@relativity.student.utwente.nl> Reply-To: djb@ifa.au.dk Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Not being a kernel programmer, I have the following question. If it is a dumb question, please excuse me, but I still need an answer :-) I have noticed that the difference in processor temperatures between idle and full load on Abit BP6 dual celeron systems is much less for FreeBSD (2 degrees celcius) than it is for Linux (13 degrees Celcius). Also overall system temperature seems to be much higher using FreeBSD than Linux. I have two Abit BP6 boxes sitting here, one running FreeBSD, the other Linux. The Linux box is 30 degrees celcius while the FreeBSD one is 50 degrees Celcius. So my question is, is FreeBSD issuing hlt instructions when the cpu's are idle or not? Might this explain the difference between Linux and FreeBSD? Regards, Dave Boers. -- djb@ifa.au.dk d.j.boers@tn.utwente.nl PGP key: ftp://relativity.student.utwente.nl:/pub/pgpkeys/djb.asc To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 7: 3:22 2000 Delivered-To: freebsd-smp@freebsd.org Received: from fw.wintelcom.net (ns1.wintelcom.net [209.1.153.20]) by hub.freebsd.org (Postfix) with ESMTP id 14C4E37B734 for ; Thu, 27 Apr 2000 07:03:20 -0700 (PDT) (envelope-from bright@fw.wintelcom.net) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id e3REWkm27894; Thu, 27 Apr 2000 07:32:46 -0700 (PDT) Date: Thu, 27 Apr 2000 07:32:46 -0700 From: Alfred Perlstein To: Dave Boers Cc: freebsd-smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues Message-ID: <20000427073245.J489@fw.wintelcom.net> References: <20000427142613.A7795@relativity.student.utwente.nl> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i In-Reply-To: <20000427142613.A7795@relativity.student.utwente.nl>; from djb@ifa.au.dk on Thu, Apr 27, 2000 at 02:26:13PM +0200 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org * Dave Boers [000427 05:57] wrote: > Not being a kernel programmer, I have the following question. If it is a > dumb question, please excuse me, but I still need an answer :-) > > I have noticed that the difference in processor temperatures between idle > and full load on Abit BP6 dual celeron systems is much less for FreeBSD (2 > degrees celcius) than it is for Linux (13 degrees Celcius). Also overall > system temperature seems to be much higher using FreeBSD than Linux. I have > two Abit BP6 boxes sitting here, one running FreeBSD, the other Linux. The > Linux box is 30 degrees celcius while the FreeBSD one is 50 degrees > Celcius. > > So my question is, is FreeBSD issuing hlt instructions when the cpu's are > idle or not? Might this explain the difference between Linux and FreeBSD? http://www.freebsd.org/FAQ/misc.html#AEN3892 -Alfred To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 7:38:51 2000 Delivered-To: freebsd-smp@freebsd.org Received: from relativity.student.utwente.nl (wit389306.student.utwente.nl [130.89.234.166]) by hub.freebsd.org (Postfix) with ESMTP id B3CC337B7D1 for ; Thu, 27 Apr 2000 07:38:49 -0700 (PDT) (envelope-from djb@wit389306.student.utwente.nl) Received: by relativity.student.utwente.nl (Postfix, from userid 1000) id 27BE05DEE; Thu, 27 Apr 2000 16:38:51 +0200 (CEST) Date: Thu, 27 Apr 2000 16:38:51 +0200 From: Dave Boers To: Alfred Perlstein Cc: freebsd-smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues Message-ID: <20000427163851.A8149@relativity.student.utwente.nl> Reply-To: djb@ifa.au.dk References: <20000427142613.A7795@relativity.student.utwente.nl> <20000427073245.J489@fw.wintelcom.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i In-Reply-To: <20000427073245.J489@fw.wintelcom.net>; from bright@wintelcom.net on Thu, Apr 27, 2000 at 07:32:46AM -0700 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Thu, Apr 27, 2000 at 07:32:46AM -0700, Alfred Perlstein wrote: > http://www.freebsd.org/FAQ/misc.html#AEN3892 I was not asking about UP FreeBSD. I know that linux and FreeBSD behave more or less the same in that case. I was specifically referring to the SMP case. I am also not into OS advocacy AT ALL! I am simply concerned about the temperature in my FreeBSD box and a similar Linux box is the only thing sensible I can compare to. I am sorry if my question looked like an advocacy issue. I also don't need to consult the faq if I want to know how hot or cool a freebsd UP box is. I have one here (single celeron, same clock frequency as the SMP boxes) which is only 29 degrees Celcius. I did search my own archive of freebsd-smp and I also searched using the freebsd.org search engine. The only relevant mail I found was from Remy Nonnenmacher, asking about the hlt instruction in the SMP case. As far as I found, Remy never got an answer. Regards, Dave Boers. -- djb@ifa.au.dk d.j.boers@tn.utwente.nl PGP key: ftp://relativity.student.utwente.nl:/pub/pgpkeys/djb.asc To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 8:26:18 2000 Delivered-To: freebsd-smp@freebsd.org Received: from Ilsa.StevesCafe.com (Ilsa.StevesCafe.com [206.168.13.65]) by hub.freebsd.org (Postfix) with ESMTP id 15A1C37B594 for ; Thu, 27 Apr 2000 08:26:16 -0700 (PDT) (envelope-from fbsd@Ilsa.StevesCafe.com) Received: from Ilsa.StevesCafe.com (localhost [127.0.0.1]) by Ilsa.StevesCafe.com (8.9.3/8.9.3) with ESMTP id JAA24484; Thu, 27 Apr 2000 09:26:05 -0600 (MDT) (envelope-from fbsd@Ilsa.StevesCafe.com) Message-Id: <200004271526.JAA24484@Ilsa.StevesCafe.com> X-Mailer: exmh version 2.0.2 2/24/98 From: Steve Passe To: djb@ifa.au.dk Cc: Alfred Perlstein , freebsd-smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues In-reply-to: Your message of "Thu, 27 Apr 2000 16:38:51 +0200." <20000427163851.A8149@relativity.student.utwente.nl> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 27 Apr 2000 09:26:05 -0600 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, switch.s: ENTRY(default_halt) sti #ifndef SMP hlt /* XXX: until a wakeup IPI */ #endif ret I think the reason we don't hlt is that it is problematic to get the CPU running again without an INT to wake it up. So the scheduler has a process for it, but its not given a chance to run it as the CPU is hlt'd. As the comment suggests a wakeup IPI (inter-processor INT) could wake it, but the more complicated code is that which hands off the process from the signaling CPU to the recently woken CPU. -- Steve Passe | powered by smp@csn.net | Symmetric MultiProcessor FreeBSD To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 8:34:36 2000 Delivered-To: freebsd-smp@freebsd.org Received: from lor.watermarkgroup.com (lor.watermarkgroup.com [207.202.73.33]) by hub.freebsd.org (Postfix) with ESMTP id E071B37B62E for ; Thu, 27 Apr 2000 08:34:32 -0700 (PDT) (envelope-from luoqi@watermarkgroup.com) Received: (from luoqi@localhost) by lor.watermarkgroup.com (8.10.1/8.10.1) id e3RFXoI01828; Thu, 27 Apr 2000 11:33:50 -0400 (EDT) Date: Thu, 27 Apr 2000 11:33:50 -0400 (EDT) From: Luoqi Chen Message-Id: <200004271533.e3RFXoI01828@lor.watermarkgroup.com> To: bright@wintelcom.net, djb@ifa.au.dk Subject: Re: hlt instructions and temperature issues Cc: freebsd-smp@FreeBSD.ORG Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > On Thu, Apr 27, 2000 at 07:32:46AM -0700, Alfred Perlstein wrote: > > http://www.freebsd.org/FAQ/misc.html#AEN3892 > > I was not asking about UP FreeBSD. I know that linux and FreeBSD behave > more or less the same in that case. I was specifically referring to the SMP > case. I am also not into OS advocacy AT ALL! I am simply concerned about > the temperature in my FreeBSD box and a similar Linux box is the only thing > sensible I can compare to. I am sorry if my question looked like an > advocacy issue. > > I also don't need to consult the faq if I want to know how hot or cool a > freebsd UP box is. I have one here (single celeron, same clock frequency as > the SMP boxes) which is only 29 degrees Celcius. > > I did search my own archive of freebsd-smp and I also searched using > the freebsd.org search engine. The only relevant mail I found was from Remy > Nonnenmacher, asking about the hlt instruction in the SMP case. > > As far as I found, Remy never got an answer. > > Regards, > > Dave Boers. > > > -- > djb@ifa.au.dk d.j.boers@tn.utwente.nl > PGP key: ftp://relativity.student.utwente.nl:/pub/pgpkeys/djb.asc > No, we don't use hlt instruction on SMP. We could, but I guess nobody bothered to work on it. It should be fairly straight forward: prior to hlt, lower TPR priority so that this processor is first in line to receive an interrupt. If you are interested, you may give the follwing patch a try. -lq Index: swtch.s =================================================================== RCS file: /home/ncvs/src/sys/i386/i386/swtch.s,v retrieving revision 1.91 diff -u -r1.91 swtch.s --- swtch.s 2000/03/29 06:15:38 1.91 +++ swtch.s 2000/04/27 15:31:11 @@ -258,9 +258,14 @@ ENTRY(default_halt) sti -#ifndef SMP - hlt /* XXX: until a wakeup IPI */ +#ifdef SMP +#ifdef CHEAP_TPR + movl $0, lapic_tpr +#else + andl $~APIC_TPR_PRIO, lapic_tpr +#endif /** CHEAP_TPR */ #endif + hlt ret /* To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 8:37:56 2000 Delivered-To: freebsd-smp@freebsd.org Received: from musse.tninet.se (musse.tninet.se [195.100.94.12]) by hub.freebsd.org (Postfix) with SMTP id 2624A37B5CE for ; Thu, 27 Apr 2000 08:37:53 -0700 (PDT) (envelope-from mal@algonet.se) Received: (qmail 18413 invoked from network); 27 Apr 2000 17:37:51 +0200 Received: from sinclair.tninet.se (HELO algonet.se) (195.100.94.101) by musse.tninet.se with SMTP; 27 Apr 2000 17:37:51 +0200 Received: from kairos.algonet.se (kairos.algonet.se [194.213.74.201]) by algonet.se (BLUETAIL Mail Robustifier 2.0.2) with ESMTP id sinclair-s0-956.849871.97648 ; Thu, 27 Apr 2000 17:37:51 +0200 Received: (mal@localhost) by kairos.algonet.se (8.8.8+Sun/8.6.12) id RAA10284; Thu, 27 Apr 2000 17:37:50 +0200 (MET DST) Date: Thu, 27 Apr 2000 17:37:50 +0200 (MET DST) Message-Id: <200004271537.RAA10284@kairos.algonet.se> X-Authentication-Warning: kairos.algonet.se: mal set sender to mal@kairos.algonet.se using -f From: Mats Lofkvist To: djb@ifa.au.dk In-reply-to: <20000427163851.A8149@relativity.student.utwente.nl> (message from Dave Boers on Thu, 27 Apr 2000 16:38:51 +0200) Subject: Re: hlt instructions and temperature issues Cc: freebsd-smp@FreeBSD.ORG References: <20000427142613.A7795@relativity.student.utwente.nl> <20000427073245.J489@fw.wintelcom.net> <20000427163851.A8149@relativity.student.utwente.nl> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org If I remember correctly someone has said on the mailing lists that FreeBSD SMP does _not_ use the HLT instruction. I agree with the argument that a SMP box should be built with good enough cooling not to have any problem with this, but on the other hand I wouldn't mind if my dual ppro box warmed the room its in (my kitchen) a bit less... _ Mats Lofkvist mal@algonet.se To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 8:57:54 2000 Delivered-To: freebsd-smp@freebsd.org Received: from thehousleys.net (frenchknot.ne.mediaone.net [24.147.224.201]) by hub.freebsd.org (Postfix) with ESMTP id B015137B8D3 for ; Thu, 27 Apr 2000 08:57:51 -0700 (PDT) (envelope-from jim@thehousleys.net) Received: from thehousleys.net (baby.int.thehousleys.net [192.168.0.24]) by thehousleys.net (8.9.3/8.9.3) with ESMTP id LAA00543 for ; Thu, 27 Apr 2000 11:57:49 -0400 (EDT) Message-ID: <3908637D.FA57D436@thehousleys.net> Date: Thu, 27 Apr 2000 11:57:49 -0400 From: James Housley Organization: The Housleys dot Net X-Mailer: Mozilla 4.72 [en] (X11; U; FreeBSD 3.4-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 Cc: freebsd-smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues References: <200004271533.e3RFXoI01828@lor.watermarkgroup.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Luoqi Chen wrote: > > Index: swtch.s > =================================================================== > RCS file: /home/ncvs/src/sys/i386/i386/swtch.s,v > retrieving revision 1.91 > diff -u -r1.91 swtch.s > --- swtch.s 2000/03/29 06:15:38 1.91 > +++ swtch.s 2000/04/27 15:31:11 > @@ -258,9 +258,14 @@ > > ENTRY(default_halt) > sti > -#ifndef SMP > - hlt /* XXX: until a wakeup IPI */ > +#ifdef SMP > +#ifdef CHEAP_TPR > + movl $0, lapic_tpr > +#else > + andl $~APIC_TPR_PRIO, lapic_tpr > +#endif /** CHEAP_TPR */ > #endif > + hlt > ret > I applied this patch to my 3.4-STABLE SMP cvs's yesterday afternoon. According to healthd my temps were 44.5 and 38.5 for the CPUs. After rebooting they cooled down to 32.0 and 30.0. I guess the next question is, "Is this seriously impacting performance?" Personaly a small hit is okay. Jim -- Do not meddle in the affairs of dragons, for you are crunchy and taste good with ketchup. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 9: 1:22 2000 Delivered-To: freebsd-smp@freebsd.org Received: from Ilsa.StevesCafe.com (Ilsa.StevesCafe.com [206.168.13.65]) by hub.freebsd.org (Postfix) with ESMTP id 459EF37B573 for ; Thu, 27 Apr 2000 09:01:18 -0700 (PDT) (envelope-from fbsd@Ilsa.StevesCafe.com) Received: from Ilsa.StevesCafe.com (localhost [127.0.0.1]) by Ilsa.StevesCafe.com (8.9.3/8.9.3) with ESMTP id KAA24638; Thu, 27 Apr 2000 10:01:08 -0600 (MDT) (envelope-from fbsd@Ilsa.StevesCafe.com) Message-Id: <200004271601.KAA24638@Ilsa.StevesCafe.com> X-Mailer: exmh version 2.0.2 2/24/98 From: Steve Passe To: Luoqi Chen Cc: bright@wintelcom.net, djb@ifa.au.dk, freebsd-smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues In-reply-to: Your message of "Thu, 27 Apr 2000 11:33:50 EDT." <200004271533.e3RFXoI01828@lor.watermarkgroup.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 27 Apr 2000 10:01:08 -0600 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, > No, we don't use hlt instruction on SMP. We could, but I guess nobody > bothered to work on it. It should be fairly straight forward: prior to > hlt, lower TPR priority so that this processor is first in line to > receive an interrupt. If you are interested, you may give the follwing > patch a try. > > -lq > > Index: swtch.s > =================================================================== > RCS file: /home/ncvs/src/sys/i386/i386/swtch.s,v > retrieving revision 1.91 > diff -u -r1.91 swtch.s > --- swtch.s 2000/03/29 06:15:38 1.91 > +++ swtch.s 2000/04/27 15:31:11 > @@ -258,9 +258,14 @@ > > ENTRY(default_halt) > sti > -#ifndef SMP > - hlt /* XXX: until a wakeup IPI */ > +#ifdef SMP > +#ifdef CHEAP_TPR > + movl $0, lapic_tpr > +#else > + andl $~APIC_TPR_PRIO, lapic_tpr > +#endif /** CHEAP_TPR */ > #endif > + hlt > ret That won't help, we already make it first candidate for an INT in idle(): 2: /* enable intrs for a halt */ movl $0, lapic_tpr /* 1st candidate for an INT */ call *_hlt_vector /* wait for interrupt */ cli jmp idle_loop The problem is that if a CPU is in the hlt'd state it will never re-enter cpu_switch() to be able to be assigned a new process, at least until it happens to catch an INT that wakes it up and leaves it in cpu_switch at a point where there is another ready process. It isn't guaranteed that the first INT will be sent to the halted CPU, especially in a >2 CPU machine. As I stated before, code is needed that sends a wakeup IPI from cpu_switch() when the running CPU detects both a hlt'de CPU and more than 1 READY process in the que. -- Steve Passe | powered by smp@csn.net | Symmetric MultiProcessor FreeBSD To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 9:10: 6 2000 Delivered-To: freebsd-smp@freebsd.org Received: from Ilsa.StevesCafe.com (Ilsa.StevesCafe.com [206.168.13.65]) by hub.freebsd.org (Postfix) with ESMTP id DF26537B62E for ; Thu, 27 Apr 2000 09:10:02 -0700 (PDT) (envelope-from fbsd@Ilsa.StevesCafe.com) Received: from Ilsa.StevesCafe.com (localhost [127.0.0.1]) by Ilsa.StevesCafe.com (8.9.3/8.9.3) with ESMTP id KAA24711; Thu, 27 Apr 2000 10:09:47 -0600 (MDT) (envelope-from fbsd@Ilsa.StevesCafe.com) Message-Id: <200004271609.KAA24711@Ilsa.StevesCafe.com> X-Mailer: exmh version 2.0.2 2/24/98 From: Steve Passe To: James Housley Cc: freebsd-smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues In-reply-to: Your message of "Thu, 27 Apr 2000 11:57:49 EDT." <3908637D.FA57D436@thehousleys.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 27 Apr 2000 10:09:47 -0600 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, > Luoqi Chen wrote: > > > > Index: swtch.s > > =================================================================== > > RCS file: /home/ncvs/src/sys/i386/i386/swtch.s,v > > retrieving revision 1.91 > > diff -u -r1.91 swtch.s > > --- swtch.s 2000/03/29 06:15:38 1.91 > > +++ swtch.s 2000/04/27 15:31:11 > > @@ -258,9 +258,14 @@ > > > > ENTRY(default_halt) > > sti > > -#ifndef SMP > > - hlt /* XXX: until a wakeup IPI */ > > +#ifdef SMP > > +#ifdef CHEAP_TPR > > + movl $0, lapic_tpr > > +#else > > + andl $~APIC_TPR_PRIO, lapic_tpr > > +#endif /** CHEAP_TPR */ > > #endif > > + hlt > > ret > > > I applied this patch to my 3.4-STABLE SMP cvs's yesterday afternoon. > According to healthd my temps were 44.5 and 38.5 for the CPUs. After > rebooting they cooled down to 32.0 and 30.0. > > I guess the next question is, "Is this seriously impacting > performance?" Personaly a small hit is okay. My experience is yes, we used to have the hlt instruction, but added the #ifdef and XXX comment after we noticed bizarre behaviour where CPU's remained stalled when there were processes to run. My 2 previous mails describe the reasons so I won't repeat them again. (We're playing dueling mailings here, the thread is a little scewed...). Try a buildworld both ways to tell the tale. If you could send me a few lines about how to install healthd on my 4way I'll try to find time to do the same. (last time I tried to use the MB monitors under SMP I got nowhere...) -- Steve Passe | powered by smp@csn.net | Symmetric MultiProcessor FreeBSD To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 9:23:15 2000 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 8243937B65B for ; Thu, 27 Apr 2000 09:23:11 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id JAA05184; Thu, 27 Apr 2000 09:23:07 -0700 (PDT) (envelope-from dillon) Date: Thu, 27 Apr 2000 09:23:07 -0700 (PDT) From: Matthew Dillon Message-Id: <200004271623.JAA05184@apollo.backplane.com> To: Luoqi Chen Cc: bright@wintelcom.net, djb@ifa.au.dk, freebsd-smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues References: <200004271533.e3RFXoI01828@lor.watermarkgroup.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Slight problem: This patch isn't hlt'ing in a legal way. The hlt instruction must be right after the sti instruction. Intel guarentees that an sti + hlt combination will halt without taking an interrupt in the middle. If you put any code whatsoever inbetween the sti and the hlt then it is possible for the machine to take an interrupt prior to halting, wakeup a task, and then halt and get no further wakeups. This will cause a lockup. -Matt :No, we don't use hlt instruction on SMP. We could, but I guess nobody :bothered to work on it. It should be fairly straight forward: prior to :hlt, lower TPR priority so that this processor is first in line to :receive an interrupt. If you are interested, you may give the follwing :patch a try. : :-lq : :Index: swtch.s :=================================================================== :RCS file: /home/ncvs/src/sys/i386/i386/swtch.s,v :retrieving revision 1.91 :diff -u -r1.91 swtch.s :--- swtch.s 2000/03/29 06:15:38 1.91 :+++ swtch.s 2000/04/27 15:31:11 :@@ -258,9 +258,14 @@ : : ENTRY(default_halt) : sti :-#ifndef SMP :- hlt /* XXX: until a wakeup IPI */ :+#ifdef SMP :+#ifdef CHEAP_TPR :+ movl $0, lapic_tpr :+#else :+ andl $~APIC_TPR_PRIO, lapic_tpr :+#endif /** CHEAP_TPR */ : #endif :+ hlt : ret : To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 9:24:24 2000 Delivered-To: freebsd-smp@freebsd.org Received: from lor.watermarkgroup.com (lor.watermarkgroup.com [207.202.73.33]) by hub.freebsd.org (Postfix) with ESMTP id 0C2C137BCFC for ; Thu, 27 Apr 2000 09:24:19 -0700 (PDT) (envelope-from luoqi@watermarkgroup.com) Received: (from luoqi@localhost) by lor.watermarkgroup.com (8.10.1/8.10.1) id e3RGNXX02617; Thu, 27 Apr 2000 12:23:33 -0400 (EDT) Date: Thu, 27 Apr 2000 12:23:33 -0400 (EDT) From: Luoqi Chen Message-Id: <200004271623.e3RGNXX02617@lor.watermarkgroup.com> To: smp@csn.net Subject: Re: hlt instructions and temperature issues Cc: bright@wintelcom.net, djb@ifa.au.dk, freebsd-smp@FreeBSD.ORG Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > The problem is that if a CPU is in the hlt'd state it will never re-enter > cpu_switch() to be able to be assigned a new process, at least until it > happens to catch an INT that wakes it up and leaves it in cpu_switch at a point > where there is another ready process. It isn't guaranteed that the first INT > will be sent to the halted CPU, especially in a >2 CPU machine. As I stated > before, code is needed that sends a wakeup IPI from cpu_switch() when the > running CPU detects both a hlt'de CPU and more than 1 READY process in the que. > > -- > Steve Passe | powered by > smp@csn.net | Symmetric MultiProcessor FreeBSD > The halted CPU may not receive the interrupt only if there's another CPU also have 0 TPR value, which means the second CPU either o is halted as well. It doesn't matter which CPU got first waken up. o has the kernel lock. The interrupt should be handled by the second CPU any way. So not too much is lost, if the first CPU doesn't wake up. I can see there's still benefit to have the first CPU waken up in the second case. Why don't we change the TPR for the kernel lock holder to 1? So that an idle processor would receive interrupts first, if it couldn't get the kernel lock, it would IPI the lock holder. No cpu time is wasted, the only downside is a little higher latency for the interrupt. But you will gain when it is one of the fast interrupts, the idle process will handle the interrupt itself, without bothering the lock holder which is supposedly busy with something else. -lq To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 9:36: 8 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mass.cdrom.com (adsl-63-202-176-132.dsl.snfc21.pacbell.net [63.202.176.132]) by hub.freebsd.org (Postfix) with ESMTP id 61DF937BC82 for ; Thu, 27 Apr 2000 09:36:03 -0700 (PDT) (envelope-from msmith@mass.cdrom.com) Received: from mass.cdrom.com (localhost [127.0.0.1]) by mass.cdrom.com (8.9.3/8.9.3) with ESMTP id JAA08277; Thu, 27 Apr 2000 09:43:46 -0700 (PDT) (envelope-from msmith@mass.cdrom.com) Message-Id: <200004271643.JAA08277@mass.cdrom.com> X-Mailer: exmh version 2.1.1 10/15/1999 To: djb@ifa.au.dk Cc: Alfred Perlstein , freebsd-smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues In-reply-to: Your message of "Thu, 27 Apr 2000 16:38:51 +0200." <20000427163851.A8149@relativity.student.utwente.nl> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 27 Apr 2000 09:43:46 -0700 From: Mike Smith Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > On Thu, Apr 27, 2000 at 07:32:46AM -0700, Alfred Perlstein wrote: > > http://www.freebsd.org/FAQ/misc.html#AEN3892 > > I was not asking about UP FreeBSD. I know that linux and FreeBSD behave > more or less the same in that case. I was specifically referring to the SMP > case. I am also not into OS advocacy AT ALL! I am simply concerned about > the temperature in my FreeBSD box and a similar Linux box is the only thing > sensible I can compare to. I am sorry if my question looked like an > advocacy issue. > > I also don't need to consult the faq if I want to know how hot or cool a > freebsd UP box is. I have one here (single celeron, same clock frequency as > the SMP boxes) which is only 29 degrees Celcius. > > I did search my own archive of freebsd-smp and I also searched using > the freebsd.org search engine. The only relevant mail I found was from Remy > Nonnenmacher, asking about the hlt instruction in the SMP case. > > As far as I found, Remy never got an answer. He did, and we don't. You can read the code in sys/i386/i386/swtch.s for yourself. -- \\ Give a man a fish, and you feed him for a day. \\ Mike Smith \\ Tell him he should learn how to fish himself, \\ msmith@freebsd.org \\ and he'll hate you for a lifetime. \\ msmith@cdrom.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 9:40: 8 2000 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 2021337BAD2 for ; Thu, 27 Apr 2000 09:40:06 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id JAA05314; Thu, 27 Apr 2000 09:40:03 -0700 (PDT) (envelope-from dillon) Date: Thu, 27 Apr 2000 09:40:03 -0700 (PDT) From: Matthew Dillon Message-Id: <200004271640.JAA05314@apollo.backplane.com> To: Luoqi Chen Cc: smp@csn.net, bright@wintelcom.net, djb@ifa.au.dk, freebsd-smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues References: <200004271623.e3RGNXX02617@lor.watermarkgroup.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org :> running CPU detects both a hlt'de CPU and more than 1 READY process in the que. :> :> -- :> Steve Passe | powered by :> smp@csn.net | Symmetric MultiProcessor FreeBSD :> :The halted CPU may not receive the interrupt only if there's another CPU :also have 0 TPR value, which means the second CPU either : o is halted as well. It doesn't matter which CPU got first waken up. : o has the kernel lock. The interrupt should be handled by the second CPU : any way. So not too much is lost, if the first CPU doesn't wake up. : :I can see there's still benefit to have the first CPU waken up in the :second case. Why don't we change the TPR for the kernel lock holder to 1? :... When you are doing SMP work, you have to assume that an infinite number of instructions can run on cpu B between any two instructions running on cpu A. So think it through... if you have sti + BLAH + hlt and the 'nop' stalls for a long, long time, is it possible for an interrupt to occur? The answer I think is yes. You can wind up in a situation where both cpu's go idle but one takes an interrupt (which, say, wakes up a process) before having a chance to HLT, then HLT's and never wakes up again even though there is a runnable process. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 9:55:43 2000 Delivered-To: freebsd-smp@freebsd.org Received: from lor.watermarkgroup.com (lor.watermarkgroup.com [207.202.73.33]) by hub.freebsd.org (Postfix) with ESMTP id DB42A37BA83 for ; Thu, 27 Apr 2000 09:55:39 -0700 (PDT) (envelope-from luoqi@watermarkgroup.com) Received: (from luoqi@localhost) by lor.watermarkgroup.com (8.10.1/8.10.1) id e3RGsOv03195; Thu, 27 Apr 2000 12:54:24 -0400 (EDT) Date: Thu, 27 Apr 2000 12:54:24 -0400 (EDT) From: Luoqi Chen Message-Id: <200004271654.e3RGsOv03195@lor.watermarkgroup.com> To: dillon@apollo.backplane.com Subject: Re: hlt instructions and temperature issues Cc: bright@wintelcom.net, djb@ifa.au.dk, freebsd-smp@FreeBSD.ORG, smp@csn.net Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > :> running CPU detects both a hlt'de CPU and more than 1 READY process in the que. > :> > :> -- > :> Steve Passe | powered by > :> smp@csn.net | Symmetric MultiProcessor FreeBSD > :> > :The halted CPU may not receive the interrupt only if there's another CPU > :also have 0 TPR value, which means the second CPU either > : o is halted as well. It doesn't matter which CPU got first waken up. > : o has the kernel lock. The interrupt should be handled by the second CPU > : any way. So not too much is lost, if the first CPU doesn't wake up. > : > :I can see there's still benefit to have the first CPU waken up in the > :second case. Why don't we change the TPR for the kernel lock holder to 1? > :... > > When you are doing SMP work, you have to assume that an infinite number > of instructions can run on cpu B between any two instructions running > on cpu A. So think it through... if you have sti + BLAH + hlt and > the 'nop' stalls for a long, long time, is it possible for an interrupt > to occur? > I don't quite get the question... Are you talking about we can't have BLAH between sti and hlt? I think you are right. But we don't really need the BLAH, as Steve has said, the idle cpu already has the lowest TPR. > The answer I think is yes. You can wind up in a situation where both > cpu's go idle but one takes an interrupt (which, say, wakes up a > process) before having a chance to HLT, then HLT's and never wakes > up again even though there is a runnable process. > What's the difference here between one cpu and multiple cpus? > -Matt > -lq To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 9:57:32 2000 Delivered-To: freebsd-smp@freebsd.org Received: from Ilsa.StevesCafe.com (Ilsa.StevesCafe.com [206.168.13.65]) by hub.freebsd.org (Postfix) with ESMTP id 2314037BB42 for ; Thu, 27 Apr 2000 09:57:28 -0700 (PDT) (envelope-from smp@Ilsa.StevesCafe.com) Received: from Ilsa.StevesCafe.com (localhost [127.0.0.1]) by Ilsa.StevesCafe.com (8.9.3/8.9.3) with ESMTP id KAA25062; Thu, 27 Apr 2000 10:56:58 -0600 (MDT) (envelope-from smp@Ilsa.StevesCafe.com) Message-Id: <200004271656.KAA25062@Ilsa.StevesCafe.com> X-Mailer: exmh version 2.0.2 2/24/98 From: Steve Passe To: Luoqi Chen Cc: bright@wintelcom.net, djb@ifa.au.dk, freebsd-smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues In-reply-to: Your message of "Thu, 27 Apr 2000 12:23:33 EDT." <200004271623.e3RGNXX02617@lor.watermarkgroup.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 27 Apr 2000 10:56:58 -0600 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > The problem is that if a CPU is in the hlt'd state it will never re-enter > > cpu_switch() to be able to be assigned a new process, at least until it > > happens to catch an INT that wakes it up and leaves it in cpu_switch at a point > > where there is another ready process. It isn't guaranteed that the first INT > > will be sent to the halted CPU, especially in a >2 CPU machine. As I stated > > before, code is needed that sends a wakeup IPI from cpu_switch() when the > > running CPU detects both a hlt'de CPU and more than 1 READY process in the que. > > > > -- > > Steve Passe | powered by > > smp@csn.net | Symmetric MultiProcessor FreeBSD > > > The halted CPU may not receive the interrupt only if there's another CPU > also have 0 TPR value, Its been several years since I coded the APIC stuff, but I seem to remember that there is also a mechanism where CPUs adjust their level depending on other "factors" (which I would need to refer to the docs to remeber the specifics of). which means the second CPU either > o is halted as well. It doesn't matter which CPU got first waken up. > o has the kernel lock. The interrupt should be handled by the second CPU > any way. So not too much is lost, if the first CPU doesn't wake up. There are several scenarios that I remember, including: CPU0 running at spl level CPU1 hlt'd INT occurs, CPU1 is woken and starts to process the ISR. first level ISR routine is locked out by spl, marks INT pending, CPU1 hlts again. CPU0 gets to splx(), sees the pending INT, processes it, processing of INT causes another process to become ready. CPU0 continues running it current process, CPU1 still hlt'd, "newly readied" process remains on que. CPU0 hits end of its quantum, reschedules, runs "newly readied" process, CPU1 remains hlt'd. > > I can see there's still benefit to have the first CPU waken up in the > second case. Why don't we change the TPR for the kernel lock holder to 1? > So that an idle processor would receive interrupts first, if it couldn't > get the kernel lock, it would IPI the lock holder. No cpu time is wasted, > the only downside is a little higher latency for the interrupt. But you > will gain when it is one of the fast interrupts, the idle process will > handle the interrupt itself, without bothering the lock holder which is > supposedly busy with something else. apic_vector.s: We already do forward INTs: 0: ; \ APIC_ITRACE(apic_itrace_tryisrlock, irq_num, APIC_ITRACE_TRYISRLOCK) ;\ MP_TRYLOCK ; /* XXX this is going away... */ \ testl %eax, %eax ; /* did we get it? */ \ jz 3f ; /* no */ \ ... 3: ; /* other cpu has isr lock */ \ APIC_ITRACE(apic_itrace_noisrlock, irq_num, APIC_ITRACE_NOISRLOCK) ;\ lock ; \ orl $IRQ_BIT(irq_num), _ipending ; \ testl $IRQ_BIT(irq_num), _cpl ; \ jne 4f ; /* this INT masked */ \ call forward_irq ; /* forward irq to lock holder */ \ -- you might have a point about setting the BGL holder to apic prio 1 instead of 0, in the case of fast INTs this might be a win, but for regular INTs its cheaper for the BGL holder to get them directly from the hardware. Testing is the only clear way to tell. My guess is that for most situations the extra INT latency isn't worth the gain for fast INTs (sio and ???). -- Steve Passe | powered by smp@csn.net | Symmetric MultiProcessor FreeBSD To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 10: 1:52 2000 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id BA13C37B798 for ; Thu, 27 Apr 2000 10:01:49 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id KAA05490; Thu, 27 Apr 2000 10:01:44 -0700 (PDT) (envelope-from dillon) Date: Thu, 27 Apr 2000 10:01:44 -0700 (PDT) From: Matthew Dillon Message-Id: <200004271701.KAA05490@apollo.backplane.com> To: Luoqi Chen Cc: bright@wintelcom.net, djb@ifa.au.dk, freebsd-smp@FreeBSD.ORG, smp@csn.net Subject: Re: hlt instructions and temperature issues References: <200004271654.e3RGsOv03195@lor.watermarkgroup.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org :> of instructions can run on cpu B between any two instructions running :> on cpu A. So think it through... if you have sti + BLAH + hlt and :> the 'nop' stalls for a long, long time, is it possible for an interrupt :> to occur? :> :I don't quite get the question... Are you talking about we can't have BLAH :between sti and hlt? I think you are right. But we don't really need the :BLAH, as Steve has said, the idle cpu already has the lowest TPR. :-lq What happens when both cpu's go idle? As far as I can tell, when both cpu's go idle it is possible for an interrupt to occur on one or the other just at the point after the STI instruction when you are messing with the APIC, before the HLT instruction. Intel guarentees that an STI+HLT combination will not interrupt in the middle as long as the HLT instruction immediately follows the STI instruction. If you have a single instruction inserted inbetween them, that guarentee goes out the window. -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 10: 2:37 2000 Delivered-To: freebsd-smp@freebsd.org Received: from Ilsa.StevesCafe.com (Ilsa.StevesCafe.com [206.168.13.65]) by hub.freebsd.org (Postfix) with ESMTP id CBAF237BE24 for ; Thu, 27 Apr 2000 10:02:33 -0700 (PDT) (envelope-from smp@Ilsa.StevesCafe.com) Received: from Ilsa.StevesCafe.com (localhost [127.0.0.1]) by Ilsa.StevesCafe.com (8.9.3/8.9.3) with ESMTP id LAA25127; Thu, 27 Apr 2000 11:02:02 -0600 (MDT) (envelope-from smp@Ilsa.StevesCafe.com) Message-Id: <200004271702.LAA25127@Ilsa.StevesCafe.com> X-Mailer: exmh version 2.0.2 2/24/98 From: Steve Passe To: Luoqi Chen Cc: dillon@apollo.backplane.com, bright@wintelcom.net, djb@ifa.au.dk, freebsd-smp@FreeBSD.ORG, smp@timing.net Subject: Re: hlt instructions and temperature issues In-reply-to: Your message of "Thu, 27 Apr 2000 12:54:24 EDT." <200004271654.e3RGsOv03195@lor.watermarkgroup.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 27 Apr 2000 11:02:02 -0600 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, to keep me in this conversation the remainder of day, send mail to: smp@timing.com -- Steve Passe | powered by smp@csn.net | Symmetric MultiProcessor FreeBSD To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 10:31:59 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mail.rdc1.sdca.home.com (ha2.rdc1.sdca.home.com [24.0.3.67]) by hub.freebsd.org (Postfix) with ESMTP id 96A2637BF88 for ; Thu, 27 Apr 2000 10:31:52 -0700 (PDT) (envelope-from jgowdy@home.com) Received: from cx443070a ([24.4.93.90]) by mail.rdc1.sdca.home.com (InterMail vM.4.01.02.00 201-229-116) with SMTP id <20000427173151.FTHM20195.mail.rdc1.sdca.home.com@cx443070a>; Thu, 27 Apr 2000 10:31:51 -0700 Message-ID: <001d01bfb070$12236b00$5a5d0418@vista1.sdca.home.com> From: "Jeremiah Gowdy" To: "Steve Passe" , "James Housley" Cc: References: <200004271609.KAA24711@Ilsa.StevesCafe.com> Subject: Re: hlt instructions and temperature issues Date: Thu, 27 Apr 2000 10:43:17 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4132.1800 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4132.1800 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org In this piece of code: ------------------------------------ ENTRY(default_halt) sti #ifndef SMP hlt /* XXX: until a wakeup IPI */ #ifdef SMP #ifdef CHEAP_TPR movl $0, lapic_tpr #else andl $~APIC_TPR_PRIO, lapic_tpr #endif /** CHEAP_TPR */ #endif hlt ret ------------------------------------ Seems to me, if SMP is not defined, hlt gets executed twice. In other words: ENTRY(default_halt) sti #ifndef SMP hlt /* XXX: until a wakeup IPI */ hlt ret Is this not the case ? Am I missing something in the behavior of hlt ? Does it return to the next line of code EIP is pointing to after it recieves an INT and wakes up ? If that's the case, it seems this code would break the nonSMP version (it would take two INTs to wake, because the next instruction after the first wake would be another hlt. If I'm way off base so be it. I just thought maybe I'd point that out, and if I'm wrong, learn something :) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 13:38:13 2000 Delivered-To: freebsd-smp@freebsd.org Received: from gemini.bnc.net (gemini.bnc.net [195.247.233.33]) by hub.freebsd.org (Postfix) with ESMTP id B50CF37B818 for ; Thu, 27 Apr 2000 13:38:09 -0700 (PDT) (envelope-from ap@bnc.net) Received: from entity (entity.mobile.bnc.net [195.247.233.29]) by gemini.bnc.net (8.9.3/8.9.3) with SMTP id WAA83589; Thu, 27 Apr 2000 22:38:01 +0200 (CEST) (envelope-from ap@bnc.net) Message-ID: <003301bfb088$5ca71d80$1de9f7c3@bnc.net> From: "Achim Patzner" To: "Steve Passe" , "Luoqi Chen" Cc: , , References: <200004271601.KAA24638@Ilsa.StevesCafe.com> Subject: Re: hlt instructions and temperature issues Date: Thu, 27 Apr 2000 22:37:09 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org From: "Steve Passe" > where there is another ready process. It isn't guaranteed that the first INT > will be sent to the halted CPU, especially in a >2 CPU machine. Urgs. Just tried it and it turned an "8 cylinder turbo" machine (HP LH8500) into a snail. Not quite what I'd expect... Achim To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 18:42:57 2000 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 24D9837B8D6 for ; Thu, 27 Apr 2000 18:42:54 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id SAA07744; Thu, 27 Apr 2000 18:42:48 -0700 (PDT) (envelope-from dillon) Date: Thu, 27 Apr 2000 18:42:48 -0700 (PDT) From: Matthew Dillon Message-Id: <200004280142.SAA07744@apollo.backplane.com> To: "Jeremiah Gowdy" Cc: "Steve Passe" , "James Housley" , Subject: Re: hlt instructions and temperature issues References: <200004271609.KAA24711@Ilsa.StevesCafe.com> <001d01bfb070$12236b00$5a5d0418@vista1.sdca.home.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org :In this piece of code: :------------------------------------ :ENTRY(default_halt) :sti :#ifndef SMP :hlt /* XXX: until a wakeup IPI */ :#ifdef SMP :#ifdef CHEAP_TPR :movl $0, lapic_tpr :#else :andl $~APIC_TPR_PRIO, lapic_tpr :#endif /** CHEAP_TPR */ :#endif :hlt :ret Umm... where'd you get the above code? This is not the current halt code for 3.x, 4.x, or 5.x. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 20:23: 4 2000 Delivered-To: freebsd-smp@freebsd.org Received: from mail.rdc1.sdca.home.com (ha1.rdc1.sdca.home.com [24.0.3.66]) by hub.freebsd.org (Postfix) with ESMTP id 27E2E37B7EB for ; Thu, 27 Apr 2000 20:23:02 -0700 (PDT) (envelope-from jgowdy@home.com) Received: from cx443070a ([24.4.93.90]) by mail.rdc1.sdca.home.com (InterMail vM.4.01.02.00 201-229-116) with SMTP id <20000428032258.MBQX13130.mail.rdc1.sdca.home.com@cx443070a>; Thu, 27 Apr 2000 20:22:58 -0700 Message-ID: <007101bfb0c2$a6fce9c0$5a5d0418@vista1.sdca.home.com> From: "Jeremiah Gowdy" To: "Matthew Dillon" Cc: "Steve Passe" , "James Housley" , References: <200004271609.KAA24711@Ilsa.StevesCafe.com> <001d01bfb070$12236b00$5a5d0418@vista1.sdca.home.com> <200004280142.SAA07744@apollo.backplane.com> Subject: Re: hlt instructions and temperature issues Date: Thu, 27 Apr 2000 20:34:25 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4132.1800 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4132.1800 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org You said: > Umm... where'd you get the above code? This is not the current > halt code for 3.x, 4.x, or 5.x. That code was in : > Luoqi Chen wrote: > > > > Index: swtch.s > > =================================================================== > > RCS file: /home/ncvs/src/sys/i386/i386/swtch.s,v > > retrieving revision 1.91 > > diff -u -r1.91 swtch.s > > --- swtch.s 2000/03/29 06:15:38 1.91 > > +++ swtch.s 2000/04/27 15:31:11 > > @@ -258,9 +258,14 @@ > > > > ENTRY(default_halt) > > sti > > -#ifndef SMP > > - hlt /* XXX: until a wakeup IPI */ > > +#ifdef SMP > > +#ifdef CHEAP_TPR > > + movl $0, lapic_tpr > > +#else > > + andl $~APIC_TPR_PRIO, lapic_tpr > > +#endif /** CHEAP_TPR */ > > #endif > > + hlt > > ret > > > I applied this patch to my 3.4-STABLE SMP cvs's yesterday afternoon. > According to healthd my temps were 44.5 and 38.5 for the CPUs. After > rebooting they cooled down to 32.0 and 30.0. > > I guess the next question is, "Is this seriously impacting > performance?" Personaly a small hit is okay. I was just saying that although he applied that patch to his SMP setup, and it should work (I'm assuming), if he were to distribute that code or something to others without SMP, it would double hlt on non-SMP setups. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 27 23: 1:38 2000 Delivered-To: freebsd-smp@freebsd.org Received: from ns.pan-asia.net (ns.pan-asia.net [210.250.10.76]) by hub.freebsd.org (Postfix) with ESMTP id E3BF037B770 for ; Thu, 27 Apr 2000 23:01:35 -0700 (PDT) (envelope-from togura@catura.com) Received: from mobile.catura.com (mobile.mirai.org [210.250.10.43]) by ns.pan-asia.net (8.10.1/8.10.0.Beta6) with SMTP id e3S61nB03891 for ; Fri, 28 Apr 2000 15:01:49 +0900 (JST) Message-Id: <10004280601.AA00464@mobile.catura.com> From: Tadayuki Ogura Date: Fri, 28 Apr 2000 15:01:33 +0900 To: freebsd-smp@freebsd.org Subject: Quad PenPro page-fault MIME-Version: 1.0 X-Mailer: AL-Mail 1.32 Content-Type: text/plain; charset=iso-2022-jp Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, All My name is Tadayuki Ogura, a new commer. I fourtunately get IBM Netfinity 7000 Quad Pentium Pro SMP system. And configure as a FreeBSD 4.0 SMP system. But, it fault at boot with the info, listed below. Fatal trap 12: Page fault while in kernel mode mp_lock = 00000002; cpuid=0 ; lapic=00000000 fault virtual address = 0x1c fault code = supervisor read, page not present instruction pointer = 0x8:0xc02ac734 stack pointer = 0x10:0xc03e9e34 frame pointer = 0x10:oxc03e9e34 code segement = base 0x0, limit 0xfffff, type 0x1b = DPL0, pres1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL=0 current process = 0(swapper) interrupt mask = net tty bio cam<-SMP:xxx trap number = 12 panic : page fault SMP kernel configuration parameter is options SMP options APIC_IO options NCPU=4 options NAPIC=2 options NINTR=43 If someone have the suggestion or patch informations, please let me know. Thank you -- Tadayuki Ogura(小倉 忠行） togura@catura.com togura@catura.co.jp To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 28 7: 2:16 2000 Delivered-To: freebsd-smp@freebsd.org Received: from lor.watermarkgroup.com (lor.watermarkgroup.com [207.202.73.33]) by hub.freebsd.org (Postfix) with ESMTP id 31F8737BEC7 for ; Fri, 28 Apr 2000 07:02:14 -0700 (PDT) (envelope-from luoqi@watermarkgroup.com) Received: (from luoqi@localhost) by lor.watermarkgroup.com (8.10.1/8.10.1) id e3SE1Wa18614; Fri, 28 Apr 2000 10:01:32 -0400 (EDT) Date: Fri, 28 Apr 2000 10:01:32 -0400 (EDT) From: Luoqi Chen Message-Id: <200004281401.e3SE1Wa18614@lor.watermarkgroup.com> To: smp@csn.net Subject: Re: hlt instructions and temperature issues Cc: bright@wintelcom.net, djb@ifa.au.dk, freebsd-smp@FreeBSD.ORG Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > There are several scenarios that I remember, including: > > CPU0 running at spl level > CPU1 hlt'd > > INT occurs, CPU1 is woken and starts to process the ISR. > first level ISR routine is locked out by spl, marks INT pending, > CPU1 hlts again. > > CPU0 gets to splx(), sees the pending INT, processes it, > processing of INT causes another process to become ready. > > CPU0 continues running it current process, CPU1 still hlt'd, > "newly readied" process remains on que. > > CPU0 hits end of its quantum, reschedules, runs "newly readied" > process, CPU1 remains hlt'd. > You're right, it seems that this could only be solved with an IPI. Maybe this should be part of a per-cpu run queues implementation... -lq To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 28 11:50: 9 2000 Delivered-To: freebsd-smp@freebsd.org Received: from verdi.nethelp.no (verdi.nethelp.no [158.36.41.162]) by hub.freebsd.org (Postfix) with SMTP id DEA8737C08D for ; Fri, 28 Apr 2000 11:50:03 -0700 (PDT) (envelope-from sthaug@nethelp.no) Received: (qmail 66054 invoked by uid 1001); 28 Apr 2000 18:50:01 +0000 (GMT) To: togura@catura.com Cc: freebsd-smp@freebsd.org Subject: Re: Quad PenPro page-fault From: sthaug@nethelp.no In-Reply-To: Your message of "Fri, 28 Apr 2000 15:01:33 +0900" References: <10004280601.AA00464@mobile.catura.com> X-Mailer: Mew version 1.05+ on Emacs 19.34.2 Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Date: Fri, 28 Apr 2000 20:50:01 +0200 Message-ID: <66051.956947801@verdi.nethelp.no> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > My name is Tadayuki Ogura, a new commer. I fourtunately get > IBM Netfinity 7000 Quad Pentium Pro SMP system. And configure > as a FreeBSD 4.0 SMP system. But, it fault at boot with the > info, listed below. If you're trying to use 4.0-RELEASE, you'll probably need at least the following patch to handle the two IOAPICs. If the 7000 has the same RCC chipset as the 5600, you may also need a separate patch to recognize this chipset and handle all the PCI buses. Steinar Haug, Nethelp consulting, sthaug@nethelp.no ---------------------------------------------------------------------- Date: Wed, 19 Apr 2000 23:19:01 -0400 (EDT) From: Luoqi Chen Message-Id: <200004200319.XAA26909@lor.watermarkgroup.com> To: mihara@prd.fc.nec.co.jp Subject: Re: Multiple APIC support (RE: Issues with Spankin' new Poweredge 4400 w/ dual 866's and 133MHz FSB?) Cc: freebsd-smp@FreeBSD.ORG, smp@csn.net > Thank you for the patch, but applying your patch results in panic > while booting, just after showing avail memory. Here is the boot log. > (Even if second IO-APIC is disabled, the phenomenon is same.) > My fault, I made a mistake in the patch. Here's the correct one (the offset should be added before typecasting), Index: pmap.c =================================================================== RCS file: /home/ncvs/src/sys/i386/i386/pmap.c,v retrieving revision 1.252 diff -u -r1.252 pmap.c --- pmap.c 2000/03/16 08:51:49 1.252 +++ pmap.c 2000/04/20 03:14:48 @@ -426,9 +426,10 @@ for (j = 0; j < mp_napics; j++) { /* same page frame as a previous IO apic? */ if (((vm_offset_t)SMPpt[NPTEPG-2-j] & PG_FRAME) == - (io_apic_address[0] & PG_FRAME)) { + (io_apic_address[i] & PG_FRAME)) { ioapic[i] = (ioapic_t *)((u_int)SMP_prvspace - + (NPTEPG-2-j)*PAGE_SIZE); + + (NPTEPG-2-j)*PAGE_SIZE + + (io_apic_address[i] & PAGE_MASK)); break; } /* use this slot if available */ @@ -436,7 +437,8 @@ SMPpt[NPTEPG-2-j] = (pt_entry_t)(PG_V | PG_RW | pgeflag | (io_apic_address[i] & PG_FRAME)); ioapic[i] = (ioapic_t *)((u_int)SMP_prvspace - + (NPTEPG-2-j)*PAGE_SIZE); + + (NPTEPG-2-j)*PAGE_SIZE + + (io_apic_address[i] & PAGE_MASK)); break; } } -lq To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 28 14: 8:35 2000 Delivered-To: freebsd-smp@freebsd.org Received: from smtp03.primenet.com (smtp03.primenet.com [206.165.6.133]) by hub.freebsd.org (Postfix) with ESMTP id 8F73337BA15 for ; Fri, 28 Apr 2000 14:08:25 -0700 (PDT) (envelope-from tlambert@usr08.primenet.com) Received: (from daemon@localhost) by smtp03.primenet.com (8.9.3/8.9.3) id OAA08637; Fri, 28 Apr 2000 14:08:13 -0700 (MST) Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp03.primenet.com, id smtpdAAA9SaOVq; Fri Apr 28 14:08:05 2000 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id OAA01313; Fri, 28 Apr 2000 14:08:06 -0700 (MST) From: Terry Lambert Message-Id: <200004282108.OAA01313@usr08.primenet.com> Subject: Re: hlt instructions and temperature issues To: dillon@apollo.backplane.com (Matthew Dillon) Date: Fri, 28 Apr 2000 21:08:06 +0000 (GMT) Cc: jgowdy@home.com (Jeremiah Gowdy), smp@csn.net (Steve Passe), jim@thehousleys.net (James Housley), freebsd-smp@FreeBSD.ORG In-Reply-To: <200004280142.SAA07744@apollo.backplane.com> from "Matthew Dillon" at Apr 27, 2000 06:42:48 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > :In this piece of code: > :------------------------------------ > :ENTRY(default_halt) > :sti > :#ifndef SMP > :hlt /* XXX: until a wakeup IPI */ > :#ifdef SMP > :#ifdef CHEAP_TPR > :movl $0, lapic_tpr > :#else > :andl $‾APIC_TPR_PRIO, lapic_tpr > :#endif /** CHEAP_TPR */ > :#endif > :hlt > :ret > > Umm... where'd you get the above code? This is not the current > halt code for 3.x, 4.x, or 5.x. This was Loqui's patch; in it he suggested replacing (in swtch.s): ENTRY(default_halt) sti #ifndef SMP hlt /* XXX: until a wakeup IPI */ #endif ret With: ENTRY(default_halt) sti #ifdef SMP #ifdef CHEAP_TPR movl $0, lapic_tpr #else andl $‾APIC_TPR_PRIO, lapic_tpr #endif /** CHEAP_TPR */ #endif hlt ret Some people have (correctly) pointed out that this would slow down SMP operations, since it reduces halted CPU's to "wake on int". This is correct. Some people have also pointed out that the TPR is already 0 when the "hlt" would have been executed. I'm not positive about this in the "just finished handling a fastintr" case. Others have complained about the "air gap" between the "sti" and the "hlt". I think that this is not really an issue, but it's very easy to rectify this, if it were. It's clearly not an issue if the TPR claims are correct, and the new code merely removes the "#ifdef SMP/#endif" directives. The comment "until a wakeup IPI" applies to the case: when releasing the BGL while leaving the scheduler, with a process still on the ready-to-run queue, and a CPU that could take it having been halted ..at least in the scheduling code as it currently sits. So it's pretty trivial to fix the "slows to a crawl" problem, and for the person with the 8 processor system to verify that it is fixed for us (having seen the "slows to a crawl" problem, in person). The comment about the TPR level for the lock holder vs. the "hlt"'ed processor is a valid point. I think that there is, however, on an NCPU > 2 machine, a new "thundering herd" problem, if all halted CPU's have a TPR of 0, and the IPI is a broadcast IPI that wakes them all. I would be very tempted to have broadcast IPIs of a high level, with the lock holder at a higher level (2 * NCPU + 1), and an unblocked processor at yet a higher level, and then an entry count for the TPR for processors, as they "get in line" for the "hlt". Then you could IPI with the min of the number of processes waiting in the ready-to-run state plus the number of processors. Each CPU would subtract the IPI level from their TPR, and, if zero or less, "go live" on one of the ready-to-run processes after setting the highest ("running") TPR. Otherwise, the CPU would decrement their TPR by the remainder, and go back to sleep. This would provide a generic "wakeone/waken/wakeup" mechanism, which should be the most efficient for a single, system wide ready to run queue. We don't care that we wakeup the CPUs with no work to do and send them right back to sleep, since they weren't doing anything valuable anyway. This scheme would not provide completely optimal "hlt"-ness, but it would provide the largest amount of "hlt"-ness which would not unduly slow the system relative to no "hlt" at all. It seems to me the best trade-off between running temperature, vs. the optimal amount of work you can squeeze out of the system. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 28 15:40:38 2000 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 1635B37BA4A for ; Fri, 28 Apr 2000 15:40:36 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id PAA14200; Fri, 28 Apr 2000 15:40:33 -0700 (PDT) (envelope-from dillon) Date: Fri, 28 Apr 2000 15:40:33 -0700 (PDT) From: Matthew Dillon Message-Id: <200004282240.PAA14200@apollo.backplane.com> To: Terry Lambert Cc: jgowdy@home.com (Jeremiah Gowdy), smp@csn.net (Steve Passe), jim@thehousleys.net (James Housley), freebsd-smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues References: <200004282108.OAA01313@usr08.primenet.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org :Others have complained about the "air gap" between the "sti" and :the "hlt". I think that this is not really an issue, but it's :very easy to rectify this, if it were. It's clearly not an issue :if the TPR claims are correct, and the new code merely removes :the "#ifdef SMP/#endif" directives. This is definitely an issue. If both cpu's go idle the interrupt that's supposed to wake one of them up (e.g. some event that causes a process to be woken up) can get lost in the air-gap. It's trivial to rectify, so we should just do it :-). Besides, you can't mess with the APIC stuff with interrupts enabled anyway because, again, an interrupt might occur that alters the state of the system just before or just after you modify the APIC priority. The sequence of events should be: (1) mess with apic (2) sti (3) hlt. These windows are small, but as we have seen an ample number of times even one-instruction windows can get hit when the code in question is being run thousands of times a second. I like the HLT + IPI idea, but none of the patches to date really cover the bases and switching performance is not going to be as good as when you don't have the HLT due to the overhead of sending the IPIs and having to keep track of which cpu's are in a HLT'd state and which are not (so you don't send IPI's to all cpu's gratuitously). This is not a trivial problem because we cannot afford to have N cpu's all trying to do locked bitset instructions on the same memory location in order to go idle -- that alone will create big latencies. We should consider testing other possible solutions, such as having a really tight idle loop that stays in the same cacheline and thus does not greatly exercise the cpu's circuitry, resulting in less heat without having to HLT. For example, if we can remove *ALL* memory writes from the best-case idle loop it should make a huge difference in heat dissipation without having to resort to HLT! Right now we make a number of subroutine calls (such as to procrunnable()) which will result in external bus cycles. If those can be inlined it should have a noticeable effect. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 28 18:20:29 2000 Delivered-To: freebsd-smp@freebsd.org Received: from smtp05.primenet.com (smtp05.primenet.com [206.165.6.135]) by hub.freebsd.org (Postfix) with ESMTP id 8525437B824 for ; Fri, 28 Apr 2000 18:20:24 -0700 (PDT) (envelope-from tlambert@usr08.primenet.com) Received: (from daemon@localhost) by smtp05.primenet.com (8.9.3/8.9.3) id SAA19968; Fri, 28 Apr 2000 18:20:19 -0700 (MST) Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp05.primenet.com, id smtpdAAARcai_M; Fri Apr 28 18:20:12 2000 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id SAA11745; Fri, 28 Apr 2000 18:20:14 -0700 (MST) From: Terry Lambert Message-Id: <200004290120.SAA11745@usr08.primenet.com> Subject: Re: hlt instructions and temperature issues To: dillon@apollo.backplane.com (Matthew Dillon) Date: Sat, 29 Apr 2000 01:20:13 +0000 (GMT) Cc: tlambert@primenet.com (Terry Lambert), jgowdy@home.com (Jeremiah Gowdy), smp@csn.net (Steve Passe), jim@thehousleys.net (James Housley), freebsd-smp@FreeBSD.ORG In-Reply-To: <200004282240.PAA14200@apollo.backplane.com> from "Matthew Dillon" at Apr 28, 2000 03:40:33 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > :Others have complained about the "air gap" between the "sti" and > :the "hlt". I think that this is not really an issue, but it's > :very easy to rectify this, if it were. It's clearly not an issue > :if the TPR claims are correct, and the new code merely removes > :the "#ifdef SMP/#endif" directives. > > This is definitely an issue. If both cpu's go idle the interrupt > that's supposed to wake one of them up (e.g. some event that causes > a process to be woken up) can get lost in the air-gap. The reason I said that is that I think that access to that code might be serialized, until the priority is dropped. > It's trivial to rectify, so we should just do it :-). Besides, you > can't mess with the APIC stuff with interrupts enabled anyway because, > again, an interrupt might occur that alters the state of the system > just before or just after you modify the APIC priority. The sequence > of events should be: (1) mess with apic (2) sti (3) hlt. Right, very easy to rectify. > These windows are small, but as we have seen an ample number of times > even one-instruction windows can get hit when the code in > question is being run thousands of times a second. Ouch! Still smarting from the lock stuff. 8-) 8-). > I like the HLT + IPI idea, but none of the patches to date > really cover the bases and switching performance is not going > to be as good as when you don't have the HLT due to the > overhead of sending the IPIs This is a non-issue, I think. The IPIs will be sent at a time that the sending processor would otherwise be going idle. The need to do this is no more of a hit, I think, than the hit FreeBSD normally takes from "hlt" in the non-SMP case. > and having to keep track of which cpu's are in a HLT'd state > and which are not (so you don't send IPI's to all cpu's > gratuitously). A trivial gross approximation here would be to have a 32 bit bitmap, one bit per processor, which did an XOR with memory of only its own personal bit. The only danger here is a window in which someone (holding the BGL) leaving the scheduler send a spurious IPI between the wakeup and the XOR operation. You could fix this by having the bit set when it is going to sleep, and unset based on the IPI about to be sent. > This is not a trivial problem because we cannot afford to > have N cpu's all trying to do locked bitset instructions on > the same memory location in order to go idle -- that alone > will create big latencies. There is a lot of current SMP code that assumes MESI cache coherency. Adding to this will not be an issue. The XOR instructions will not need to be locked, I believe, since the cache coherency notifications should handle synchronization. As I said, the bit will only ever be being cleared in the BGL case. If you want to get gross, you can set the bit in the scheduler with the BGL held on the processor that's about to go idle, which would take care of your objection: the bitmap is only ever manipulated with the BGL held, and the manipulation is done opportunistically, so there is not additional locking overhead. You would, of course, have to undo this hack when you went to per CPU ready-to-run queues. But realize that per CPU ready-to-run queues already magically have an IPI call location reserved in the code which migrates processes from one CPU ready-to-run queue to another. 8-). > We should consider testing other possible solutions, such as having a > really tight idle loop that stays in the same cacheline and thus does > not greatly exercise the cpu's circuitry, resulting in less heat without > having to HLT. I think that going outside is the least of the heat dissipation workies; it strikes me that line drivers are not where the heat is coming from, and that running over the same cache line would be a very bad thing. The other problem with this idea is that you rely on a shootdown notification for a data change in order to exit the loop, and that is, defacto, an IPI in all but name. > For example, if we can remove *ALL* memory writes from the > best-case idle loop it should make a huge difference in heat > dissipation without having to resort to HLT! Right now we > make a number of subroutine calls (such as to procrunnable()) > which will result in external bus cycles. If those can be > inlined it should have a noticeable effect. You can give it a try, but I don't think it will have the effect you think that it will. I think the numbers for a 2 CPU system with Loqui's patch were extremely exagerated by the CPU stalling-until-interrupt issue, and that the heat numbers will not be nearly so good, even on a totally "correct" solotuion because of this. I expect your approach would result in temeperatures nearly as high, if not downright indistinguishable from, the measured numbers for an unmodified system. This is really not an issue, anyway, except for power consumption and heat dissipation critical environments, but that said, if it's for an SMP box going into a colocation server room rack somewhere in a 1U case, this could be significant for some percentage of users, so maybe it's worth still talking about. 8-). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 28 18:31:57 2000 Delivered-To: freebsd-smp@freebsd.org Received: from smtp05.primenet.com (smtp05.primenet.com [206.165.6.135]) by hub.freebsd.org (Postfix) with ESMTP id 77A4337B9CC for ; Fri, 28 Apr 2000 18:31:53 -0700 (PDT) (envelope-from tlambert@usr08.primenet.com) Received: (from daemon@localhost) by smtp05.primenet.com (8.9.3/8.9.3) id SAA21999; Fri, 28 Apr 2000 18:31:46 -0700 (MST) Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp05.primenet.com, id smtpdAAAcQaO6Q; Fri Apr 28 18:31:39 2000 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id SAA12025; Fri, 28 Apr 2000 18:31:41 -0700 (MST) From: Terry Lambert Message-Id: <200004290131.SAA12025@usr08.primenet.com> Subject: Re: hlt instructions and temperature issues To: BHechinger@half.com (Brian Hechinger) Date: Sat, 29 Apr 2000 01:31:41 +0000 (GMT) Cc: dillon@apollo.backplane.com ('Matthew Dillon'), tlambert@primenet.com (Terry Lambert), jgowdy@home.com, smp@csn.net, jim@thehousleys.net, freebsd-smp@FreeBSD.ORG In-Reply-To: from "Brian Hechinger" at Apr 28, 2000 08:23:55 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > ok, bear with me here. i'm a little new to how this all works. don't know > a whole lot about programming this deep. so i will ask questions until i > understand. i read everything i can get my hands on, but i have a long way > to go, and i'm sure i have questions that won't be answered there. > > so the whole point here is that if we continuously run instructions then we > will burn out the CPU (or rather, run the possibility of burning out the > CPU)? No. The point is that the CPUs run hotter, which means shorter batter life, more power consumption, higher cooling requirements, and limitations on installtion in close space, as a result. > so, is this an issue purely for intel CPUs? would something > like this have to be done if doing this on SPARC, motorola, > Alpha, etc? It's a small win for most applications, and a big win for some. The win applies to all processors, where you can do a halt in an idle loop and IPI or interrupt back to life. > aprox how much time (percentage) is being spent cooling the > CPU? It's in "not heating the CPU". If done correctly, there's about a six instruction latency overall, counting the code in the scheduler and the halt and wakeup overhead. This will vary from processor to processor, based on the voodoo necessary to make it work. > would there be a significant increase in speed if we could > avoid this? Hotter processors run fractionally slower. All in all, it's about a wash, in terms of processor effectiveness. The real wins are heat dissipation and power consumption. > also, is heat measured and we act apropriately, or do we just > cool enough to ensure that we don't have a chance to heat up > too far? It's not an active "cooling". I guess if you had chip temeperature detection code and ran the cpu only so long as it was below a certain temperature, and when it hit it, you didn't run it until it was some ways below that temeperature, we could get the overclocking weenies machines to work stablely. I suspect their overall performance would end up being less than what it would have been, had they not been overclocking... Just think... no more "my machine is doing wierd behaviour X!" and "Are you overclocking your CPU?" type exchanges, ever again. 8-) 8-) 8-). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 28 19:26:57 2000 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id EB39C37B723 for ; Fri, 28 Apr 2000 19:26:55 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id TAA20985; Fri, 28 Apr 2000 19:26:53 -0700 (PDT) (envelope-from dillon) Date: Fri, 28 Apr 2000 19:26:53 -0700 (PDT) From: Matthew Dillon Message-Id: <200004290226.TAA20985@apollo.backplane.com> To: Terry Lambert Cc: tlambert@primenet.com (Terry Lambert), jgowdy@home.com (Jeremiah Gowdy), smp@csn.net (Steve Passe), jim@thehousleys.net (James Housley), freebsd-smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues References: <200004290120.SAA11745@usr08.primenet.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Terry, MESI caching is the *problem* here, not the thing that makes the issue go away. Having multiple cpu's XORing to the same memory location, whether or not they are only messing with their own bits, MUST be a locked instruction outside the BGL or you *will* get data corruption. Intel does not use cache-coherent read-modify-write instructions without the lock prefix, except for a special case in one of the cmpexg instructions. And we are outside the BGL at this point in the scheduler code. It is precisely attempting to do something like that which will cause the MESI caching to go haywire. XOR latency on an 8-cpu box with all cpu's trying to mess with the bits will border on 1.6 uS. We are talking about creating hundreds of cycles of overhead in the scheduling code for no good reason here. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 28 19:37: 5 2000 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 0E73837B723 for ; Fri, 28 Apr 2000 19:37:02 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id TAA21020; Fri, 28 Apr 2000 19:37:00 -0700 (PDT) (envelope-from dillon) Date: Fri, 28 Apr 2000 19:37:00 -0700 (PDT) From: Matthew Dillon Message-Id: <200004290237.TAA21020@apollo.backplane.com> To: Terry Lambert Cc: tlambert@primenet.com (Terry Lambert), jgowdy@home.com (Jeremiah Gowdy), smp@csn.net (Steve Passe), jim@thehousleys.net (James Housley), freebsd-smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues References: <200004290120.SAA11745@usr08.primenet.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org :> We should consider testing other possible solutions, such as having a :> really tight idle loop that stays in the same cacheline and thus does :> not greatly exercise the cpu's circuitry, resulting in less heat without :> having to HLT. : :I think that going outside is the least of the heat dissipation :workies; it strikes me that line drivers are not where the heat :is coming from, and that running over the same cache line would :be a very bad thing. Heat dissipation is caused mainly by gates on the chip changing state. You can reduce the amount of gates that change state in a good portion of the chip with careful coding. We aren't just talking about external accesses here, though they are a big part of it. We are talking about, for example, isolating the idle loop within a single code cache line and the data accesses within a single data cache line. By doing so the cache memory addressing will not change and since the cache memory is essentially static ram using combinational decoding, this will save a considerable amount of power and thus a considerable amount of heat. I'm not sure if there are any tricks we can play to reduce dissipation in the pipeline. Sometimes running very long-running instructions, like DIV (try -1/-1 for example) in the middle of the loop works. :The other problem with this idea is that you rely on a shootdown :notification for a data change in order to exit the loop, and :that is, defacto, an IPI in all but name. We would rely on a data-change. The point is that the data change is generated by an outside source (such as an interrupt or one of the cpu's that IS running). A data change is very different from an IPI. The overhead is much, much lower. :I think the numbers for a 2 CPU system with Loqui's patch were :extremely exagerated by the CPU stalling-until-interrupt issue, :and that the heat numbers will not be nearly so good, even on a :totally "correct" solotuion because of this. I expect your :approach would result in temeperatures nearly as high, if not :downright indistinguishable from, the measured numbers for an :unmodified system. Possibly, but possibly not. I don't know the intel architecture well enough to say for sure. I've done a lot of low-power design so I do know that these sorts of tricks work - often quite well. But Intel has never been particularly good at doing low-power cpu designs. -Matt :This is really not an issue, anyway, except for power consumption :and heat dissipation critical environments, but that said, if it's :for an SMP box going into a colocation server room rack somewhere :in a 1U case, this could be significant for some percentage of :users, so maybe it's worth still talking about. 8-). : : Terry Lambert : terry@lambert.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 28 19:43:55 2000 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id AA3DF37BFE8 for ; Fri, 28 Apr 2000 19:43:52 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id TAA21072; Fri, 28 Apr 2000 19:43:49 -0700 (PDT) (envelope-from dillon) Date: Fri, 28 Apr 2000 19:43:49 -0700 (PDT) From: Matthew Dillon Message-Id: <200004290243.TAA21072@apollo.backplane.com> To: Brian Hechinger Cc: Terry Lambert , jgowdy@home.com, smp@csn.net, jim@thehousleys.net, freebsd-smp@FreeBSD.ORG Subject: Re: RE: hlt instructions and temperature issues References: Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org :ok, bear with me here. i'm a little new to how this all works. don't know :a whole lot about programming this deep. so i will ask questions until i :understand. i read everything i can get my hands on, but i have a long way :to go, and i'm sure i have questions that won't be answered there. : :so the whole point here is that if we continuously run instructions then we :will burn out the CPU (or rather, run the possibility of burning out the :CPU)? : :so, is this an issue purely for intel CPUs? would something like this have :to :be done if doing this on SPARC, motorola, Alpha, etc? This is mainly just an issue with Intel cpu's, because intel cpu's tend to be thrown into boxes with insufficient cooling. Heat dissipation is a direct consequence of bit changes occuring in logic gates in the cpu. Such bit changes are usually related to the clock speed in one way or another, but it is often possible with careful coding and a knowledge of how the chip was designed to construct a sequence of instructions which yields a noticeable decrease in the heat dissipation. It's also possible with careful coding and a knowledge of the chip design to construct a sequence of instructions which yields maximum heat. Now wouldn't that be an interesting virus! :-) :-) Personally speaking I don't think it's a legitimate issue. I certainly would oppose anybody trying to commit complex code to make HLT work for SMP at least not before trying out simpler solutions. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 28 20: 2:43 2000 Delivered-To: freebsd-smp@freebsd.org Received: from thehousleys.net (frenchknot.ne.mediaone.net [24.147.224.201]) by hub.freebsd.org (Postfix) with ESMTP id 8E24437B9AE for ; Fri, 28 Apr 2000 20:02:39 -0700 (PDT) (envelope-from jim@thehousleys.net) Received: from thehousleys.net (baby.int.thehousleys.net [192.168.0.24]) by thehousleys.net (8.9.3/8.9.3) with ESMTP id XAA06707; Fri, 28 Apr 2000 23:01:52 -0400 (EDT) Message-ID: <390A50A0.554B1D1A@thehousleys.net> Date: Fri, 28 Apr 2000 23:01:52 -0400 From: James Housley Organization: The Housleys dot Net X-Mailer: Mozilla 4.72 [en] (X11; U; FreeBSD 3.4-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 To: Matthew Dillon Cc: Terry Lambert , Jeremiah Gowdy , Steve Passe , freebsd-smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues References: <200004282108.OAA01313@usr08.primenet.com> <200004282240.PAA14200@apollo.backplane.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Matthew Dillon wrote: > > > For example, if we can remove *ALL* memory writes from the best-case > idle loop it should make a huge difference in heat dissipation without > having to resort to HLT! Right now we make a number of subroutine > calls (such as to procrunnable()) which will result in external > bus cycles. If those can be inlined it should have a noticeable effect. > I haven't looked the the Pentuim instruction set in a long while, but is there low-power mode that could be used instead of hlt? Slow down, but not sleep the CPU until it is needed? Jim -- "Eagles may soar, but weasels don't get sucked into jet engines" -- Anon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 28 22:19:23 2000 Delivered-To: freebsd-smp@freebsd.org Received: from overcee.netplex.com.au (peter1.yahoo.com [208.48.107.4]) by hub.freebsd.org (Postfix) with ESMTP id C5A1437B735 for ; Fri, 28 Apr 2000 22:19:19 -0700 (PDT) (envelope-from peter@netplex.com.au) Received: from netplex.com.au (localhost [127.0.0.1]) by overcee.netplex.com.au (Postfix) with ESMTP id B52881CD7; Fri, 28 Apr 2000 22:19:18 -0700 (PDT) (envelope-from peter@netplex.com.au) X-Mailer: exmh version 2.1.1 10/15/1999 To: Matthew Dillon Cc: Luoqi Chen , bright@wintelcom.net, djb@ifa.au.dk, freebsd-smp@FreeBSD.ORG, smp@csn.net Subject: Re: hlt instructions and temperature issues In-Reply-To: Message from Matthew Dillon of "Thu, 27 Apr 2000 10:01:44 PDT." <200004271701.KAA05490@apollo.backplane.com> Date: Fri, 28 Apr 2000 22:19:18 -0700 From: Peter Wemm Message-Id: <20000429051918.B52881CD7@overcee.netplex.com.au> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Matthew Dillon wrote: > :> of instructions can run on cpu B between any two instructions running > :> on cpu A. So think it through... if you have sti + BLAH + hlt and > :> the 'nop' stalls for a long, long time, is it possible for an interrup t > :> to occur? > :> > :I don't quite get the question... Are you talking about we can't have BLAH > :between sti and hlt? I think you are right. But we don't really need the > :BLAH, as Steve has said, the idle cpu already has the lowest TPR. > :-lq > > What happens when both cpu's go idle? As far as I can tell, when both > cpu's go idle it is possible for an interrupt to occur on one or the > other just at the point after the STI instruction when you are > messing with the APIC, before the HLT instruction. > > Intel guarentees that an STI+HLT combination will not interrupt in the > middle as long as the HLT instruction immediately follows the STI > instruction. If you have a single instruction inserted inbetween them, > that guarentee goes out the window. I haven't read the full thread yet, but the reason why we turned HLT back off last time was because of this scenario: CPU1 - sees empty run queue, halts. CPU2 - wakes up a new process making it runnable. CPU1 - remains halted, we didn't generate an IPI on runqueue changes. This was the situation last time we activated HALT. Things have changed since then. For starters, there is an easy place to add an IPI send (C code not asm), so it should be trivial to fix that particular part of the problem now. Other things have changed in -current's low level SMP code, I'm not sure if it's a simple thing to reactivate it any more. Cheers, -Peter To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Apr 29 4:54: 1 2000 Delivered-To: freebsd-smp@freebsd.org Received: from relativity.student.utwente.nl (wit389306.student.utwente.nl [130.89.234.166]) by hub.freebsd.org (Postfix) with ESMTP id DF1B937B7BB for ; Sat, 29 Apr 2000 04:53:56 -0700 (PDT) (envelope-from djb@wit389306.student.utwente.nl) Received: by relativity.student.utwente.nl (Postfix, from userid 1000) id 60CAF5DEE; Sat, 29 Apr 2000 13:53:50 +0200 (CEST) Date: Sat, 29 Apr 2000 13:53:50 +0200 From: Dave Boers To: Terry Lambert Cc: Matthew Dillon , Jeremiah Gowdy , Steve Passe , James Housley , freebsd-smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues Message-ID: <20000429135350.A3919@relativity.student.utwente.nl> Reply-To: djb@ifa.au.dk References: <200004282240.PAA14200@apollo.backplane.com> <200004290120.SAA11745@usr08.primenet.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i In-Reply-To: <200004290120.SAA11745@usr08.primenet.com>; from tlambert@primenet.com on Sat, Apr 29, 2000 at 01:20:13AM +0000 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Sat, Apr 29, 2000 at 01:20:13AM +0000, Terry Lambert wrote: > I think the numbers for a 2 CPU system with Loqui's patch were > extremely exagerated by the CPU stalling-until-interrupt issue, > and that the heat numbers will not be nearly so good, even on a > totally "correct" solotuion because of this. I expect your > approach would result in temeperatures nearly as high, if not > downright indistinguishable from, the measured numbers for an > unmodified system. To further input some experimental facts into this discussion, I have applied Luoqi Chen's patch and I'm testing it now. The system (Abit BP6 dual Celeron) has been up for nearly 14 hours now, mostly being idle. The temperature has dropped dramatically: from 50 degrees Celcius to 27 degrees Celcius. Look at that: 23 degrees temperature difference! Besides, 27 degrees is almost the same temperature as my single Celeron unit (which has only 3 fans while my dual box has 6 of them. Also, diagnostics report that the core voltage has risen by 0.05 Volt as a consequence of the reduced power consumption of the CPU's. The kernel looks stable, even though from the thread I get the impression that many problems might arise. To stress the system somewhat, I'll do a make -j 12 buildworld now. See if it crashes. > This is really not an issue, anyway, except for power consumption > and heat dissipation critical environments, but that said, if it's > for an SMP box going into a colocation server room rack somewhere > in a 1U case, this could be significant for some percentage of > users, so maybe it's worth still talking about. 8-). Excuse me, but I think this *is* an issue worth talking about. My SMP box is in a full tower server housing. Each CPU has a (normal size) fan, two fans are cooling the harddrives (which are in the top of the case), one fan (in the PS) cools the mainboard and processor area and finally there is one fan (in the bottom of the case) blowing cool air into the case and directly at the hottest PCI cards. Not even all this is enough to cool the processors to a decent temperature if I'm running the unmodified kernel. Though I have made a special order for some globalwin king size cpu fans, I'd much rather have my sustem produce less heat by running hlt instructions. For extra clarity, let me point out that I my system is *not* overclocked and environmental temperatures for all measurements are between 21 and 24 degrees Celcius. Every bit of heat my system produces will have to be pumped away by the airconditioning which is having a hard time on hot days. And also consider the overall waste of electrical energy by all the SMP systems running FreeBSD around the world. I think this is a big issue. Regards, Dave Boers. P.S. Let me add that for me being a theoretical (computational) physicist it's kind of funny to be on the experimental side for a change! -- djb@ifa.au.dk d.j.boers@tn.utwente.nl PGP key: ftp://relativity.student.utwente.nl:/pub/pgpkeys/djb.asc To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Apr 29 7:17:14 2000 Delivered-To: freebsd-smp@freebsd.org Received: from relativity.student.utwente.nl (wit389306.student.utwente.nl [130.89.234.166]) by hub.freebsd.org (Postfix) with ESMTP id 4B54C37B6D0 for ; Sat, 29 Apr 2000 07:17:12 -0700 (PDT) (envelope-from djb@wit389306.student.utwente.nl) Received: by relativity.student.utwente.nl (Postfix, from userid 1000) id C505C5DEF; Sat, 29 Apr 2000 16:17:13 +0200 (CEST) Date: Sat, 29 Apr 2000 16:17:13 +0200 From: Dave Boers To: freebsd-smp@freebsd.org Cc: Terry Lambert , Matthew Dillon , Jeremiah Gowdy , Steve Passe , James Housley Subject: Re: hlt instructions and temperature issues Message-ID: <20000429161713.A49155@relativity.student.utwente.nl> Reply-To: djb@ifa.au.dk References: <200004282240.PAA14200@apollo.backplane.com> <200004290120.SAA11745@usr08.primenet.com> <20000429135350.A3919@relativity.student.utwente.nl> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i In-Reply-To: <20000429135350.A3919@relativity.student.utwente.nl>; from djb@ifa.au.dk on Sat, Apr 29, 2000 at 01:53:50PM +0200 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Sat, Apr 29, 2000 at 01:53:50PM +0200, Dave Boers wrote: > The kernel looks stable, even though from the thread I get the impression > that many problems might arise. To stress the system somewhat, I'll do a > make -j 12 buildworld now. See if it crashes. It didn't crash. During the process I followed processor utilization using top and everything seems just normal to me. Also, the system does not appear slower than normal at all. During the buildworld the processor temperature never reached above 40 degrees celcius. So I guess, comparing to the 50 degrees Celcius, that whatever an unmodified SMP FreeBSD-system is doing during idle time, it sure makes the processor very hot; even hotter than during full load in buildworld -j 12. Regards, Dave Boers. -- djb@ifa.au.dk d.j.boers@tn.utwente.nl PGP key: ftp://relativity.student.utwente.nl:/pub/pgpkeys/djb.asc To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Apr 29 7:46: 2 2000 Delivered-To: freebsd-smp@freebsd.org Received: from Ilsa.StevesCafe.com (Ilsa.StevesCafe.com [206.168.13.65]) by hub.freebsd.org (Postfix) with ESMTP id 71C8B37B568 for ; Sat, 29 Apr 2000 07:45:59 -0700 (PDT) (envelope-from fbsd@Ilsa.StevesCafe.com) Received: from Ilsa.StevesCafe.com (localhost [127.0.0.1]) by Ilsa.StevesCafe.com (8.9.3/8.9.3) with ESMTP id IAA09070; Sat, 29 Apr 2000 08:45:46 -0600 (MDT) (envelope-from fbsd@Ilsa.StevesCafe.com) Message-Id: <200004291445.IAA09070@Ilsa.StevesCafe.com> X-Mailer: exmh version 2.0.2 2/24/98 From: Steve Passe To: djb@ifa.au.dk Cc: freebsd-smp@FreeBSD.ORG, Terry Lambert , Matthew Dillon , Jeremiah Gowdy , James Housley Subject: Re: hlt instructions and temperature issues In-reply-to: Your message of "Sat, 29 Apr 2000 16:17:13 +0200." <20000429161713.A49155@relativity.student.utwente.nl> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 29 Apr 2000 08:45:46 -0600 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, > On Sat, Apr 29, 2000 at 01:53:50PM +0200, Dave Boers wrote: > > The kernel looks stable, even though from the thread I get the impression > > that many problems might arise. To stress the system somewhat, I'll do a > > make -j 12 buildworld now. See if it crashes. > > It didn't crash. During the process I followed processor utilization using > top and everything seems just normal to me. Also, the system does not > appear slower than normal at all. > > During the buildworld the processor temperature never reached above 40 > degrees celcius. So I guess, comparing to the 50 degrees Celcius, that > whatever an unmodified SMP FreeBSD-system is doing during idle time, it > sure makes the processor very hot; even hotter than during full load in > buildworld -j 12. Did you use the version that had the explicit instruction between the sti and hlt: ENTRY(default_halt) sti movl $0, lapic_tpr hlt ret or one of the other variations we discussed? I would like to repeat your experiment. -- Steve Passe | powered by smp@csn.net | Symmetric MultiProcessor FreeBSD To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Apr 29 8:29: 5 2000 Delivered-To: freebsd-smp@freebsd.org Received: from wint.itfs.nsk.su (wint.itfs.nsk.su [212.20.32.43]) by hub.freebsd.org (Postfix) with ESMTP id 469B337BC6B for ; Sat, 29 Apr 2000 08:28:58 -0700 (PDT) (envelope-from nnd@wint.itfs.nsk.su) Received: (from nnd@localhost) by wint.itfs.nsk.su (8.9.3/8.9.3) id WAA00735; Sat, 29 Apr 2000 22:28:54 +0700 (NOVST) (envelope-from nnd) Date: Sat, 29 Apr 2000 22:28:54 +0700 (NOVST) Message-Id: <200004291528.WAA00735@wint.itfs.nsk.su> From: Nickolay Dudorov To: smp@freebsd.org Subject: Re: hlt instructions and temperature issues In-Reply-To: <200004291445.IAA09070@Ilsa.StevesCafe.com> User-Agent: tin/1.4.2-20000123 ("Polish") (UNIX) (FreeBSD/5.0-CURRENT (i386)) Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org In <200004291445.IAA09070@Ilsa.StevesCafe.com> Steve Passe wrote: > > Did you use the version that had the explicit instruction between the sti > and hlt: > ENTRY(default_halt) > sti > movl $0, lapic_tpr > hlt > ret > or one of the other variations we discussed? > > I would like to repeat your experiment. I've made three 'make -j32 buildworlds' using shown version of the swtch.s patch. (It was on the ABIT BP6 based system with two Celerons 366 overclocked to 550 - I know, I know ;-). All three was successfull and the total time (52-54 minutes) was in the same range as on the same system without the patch. But now I use the kernel with the 'sti' and the next instruction swapped. The temperature of idle system changed from ‾55 deg. Celsius to ‾35 deg. And due the 'make buildworld's it was about the same as without the patch (may be ‾5 deg. lower). N.Dudorov To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Apr 29 10:34:27 2000 Delivered-To: freebsd-smp@freebsd.org Received: from relativity.student.utwente.nl (wit389306.student.utwente.nl [130.89.234.166]) by hub.freebsd.org (Postfix) with ESMTP id D6E8537B752 for ; Sat, 29 Apr 2000 10:34:24 -0700 (PDT) (envelope-from djb@wit389306.student.utwente.nl) Received: by relativity.student.utwente.nl (Postfix, from userid 1000) id B7A905DEE; Sat, 29 Apr 2000 19:34:25 +0200 (CEST) Date: Sat, 29 Apr 2000 19:34:25 +0200 From: Dave Boers To: Steve Passe Cc: djb@ifa.au.dk, freebsd-smp@FreeBSD.ORG, Terry Lambert , Matthew Dillon , Jeremiah Gowdy , James Housley Subject: Re: hlt instructions and temperature issues Message-ID: <20000429193425.A49505@relativity.student.utwente.nl> Reply-To: djb@ifa.au.dk References: <20000429161713.A49155@relativity.student.utwente.nl> <200004291445.IAA09070@Ilsa.StevesCafe.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i In-Reply-To: <200004291445.IAA09070@Ilsa.StevesCafe.com>; from smp@csn.net on Sat, Apr 29, 2000 at 08:45:46AM -0600 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Sat, Apr 29, 2000 at 08:45:46AM -0600, Steve Passe wrote: > Did you use the version that had the explicit instruction between the sti > and hlt: Yes, I used Luoqi Chen's original patch, no modifications. I'm still using it, even to write this email. Uptime is now: 7:33PM up 20:02, 5 users, load averages: 0.25, 0.14, 0.09 Regards, Dave. -- djb@ifa.au.dk d.j.boers@tn.utwente.nl PGP key: ftp://relativity.student.utwente.nl:/pub/pgpkeys/djb.asc To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Apr 29 20:55:20 2000 Delivered-To: freebsd-smp@freebsd.org Received: from Ilsa.StevesCafe.com (Ilsa.StevesCafe.com [206.168.13.65]) by hub.freebsd.org (Postfix) with ESMTP id 8C77F37B71A for ; Sat, 29 Apr 2000 20:55:16 -0700 (PDT) (envelope-from fbsd@Ilsa.StevesCafe.com) Received: from Ilsa.StevesCafe.com (localhost [127.0.0.1]) by Ilsa.StevesCafe.com (8.9.3/8.9.3) with ESMTP id VAA13194; Sat, 29 Apr 2000 21:50:48 -0600 (MDT) (envelope-from fbsd@Ilsa.StevesCafe.com) Message-Id: <200004300350.VAA13194@Ilsa.StevesCafe.com> X-Mailer: exmh version 2.0.2 2/24/98 From: Steve Passe To: Dave Boers Cc: smp@FreeBSD.ORG Subject: Re: hlt instructions and temperature issues Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 29 Apr 2000 21:50:47 -0600 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, > In <200004291445.IAA09070@Ilsa.StevesCafe.com> Steve Passe wrote: > > > > Did you use the version that had the explicit instruction between the sti > > and hlt: > > ENTRY(default_halt) > > sti > > movl $0, lapic_tpr > > hlt > > ret > > or one of the other variations we discussed? > > > > I would like to repeat your experiment. > Yes, I used Luoqi Chen's original patch, no modifications. This is very weird, my times improved with the hlt instruction in place: http://people.FreeBSD.org/‾fsmp/SMP/hltbenches.html -- Steve Passe | powered by smp@csn.net | Symmetric MultiProcessor FreeBSD To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message