From owner-freebsd-bugs Mon Sep 9 17:50:05 1996 Return-Path: owner-bugs Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id RAA02168 for bugs-outgoing; Mon, 9 Sep 1996 17:50:05 -0700 (PDT) Received: (from gnats@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id RAA02151; Mon, 9 Sep 1996 17:50:02 -0700 (PDT) Resent-Date: Mon, 9 Sep 1996 17:50:02 -0700 (PDT) Resent-Message-Id: <199609100050.RAA02151@freefall.freebsd.org> Resent-From: gnats (GNATS Management) Resent-To: freebsd-bugs Resent-Reply-To: FreeBSD-gnats@freefall.FreeBSD.org, smp@csn.net Received: from rick.systemsix.com (rick.systemsix.com [198.99.86.136]) by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id RAA01667 for ; Mon, 9 Sep 1996 17:42:33 -0700 (PDT) Received: (from smp@localhost) by rick.systemsix.com (8.7.5/8.7.3) id SAA04080; Mon, 9 Sep 1996 18:43:16 -0600 (MDT) Message-Id: <199609100043.SAA04080@rick.systemsix.com> Date: Mon, 9 Sep 1996 18:43:16 -0600 (MDT) From: smp@csn.net Reply-To: smp@csn.net To: FreeBSD-gnats-submit@freebsd.org X-Send-Pr-Version: 3.2 Subject: kern/1594: SMP kernel fix 960909.4 Sender: owner-bugs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >Number: 1594 >Category: kern >Synopsis: apic_startup() needs work >Confidential: no >Severity: serious >Priority: high >Responsible: freebsd-bugs >State: open >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Sep 9 17:50:01 PDT 1996 >Last-Modified: >Originator: Steve Passe >Organization: New Ideas >Release: FreeBSD 2.2-CURRENT i386 >Environment: smp-sys as of August 27, 1996 >Description: The current version of apic_startup() does not properly start the 2nd CPU on an Intel XXPRESS box. Experimentation shows that it can be re-written in such a way as to be more robust and support a wider range of motherboards. The current version appears to follow the pseudo code in example B-1 of the MP spec v1.4, page B3. The example appears to be incorrect and part of the problem. Specifically it shows an APIC STARTUP IPI twice in a row, while databooks and the MP spec claim this should only be done ONCE after a RESET IPI or hard reset. Secondly, some hardware (eg the XXPRESS) depends on the INIT/RESET IPI to actually start the 2nd CPU via the BIOS warm-start vector mechanism. The current code ignores the fact that the INIT/RESET IPI depends on a valid warm-start vector. This code needs further refinement to deal with motherboards that depend on discrete APICs, as they don't support the STARTUP IPI. I believe this includes 80486 class mbs. apic_probe() would have to be extended to determine such facts and leave info around for apic_startup(). To deal with the XXPRESS's need for the funky 'WARMBOOT_TO_BOOTSTRAP' one could parse the "PRODUCT_ID_STRING" in the mpfps for "XXPRESS" and act accordingly. (what a kludge) The basic algorithm: if ( AP's local APIC == discrete ) /* eg. 80486 with 80489 */ { INIT_RESET_IPI( toBootMP ); } else if ( rogueHardware ) /* eg. Intel XXPRESS */ { INIT_RESET_IPI( toBootMP ); } else /* APIC == embedded */ /* eg. 586/686 */ { INIT_RESET_IPI( toHlt ); } if ( AP's local APIC == embedded ) /* eg. 586/686 */ { STARTUP_IPI( bootMPVector ); } >How-To-Repeat: run current code on Intel XXPRESS box. >Fix: [ I created several defines in smpasm.h to conditionalize code: ] /* * defining this will cause a warm-start vector to the bootMP * code to be built and installed, NOT the preferred method! * * Rogue hardware known to need this define: * Intel XXPRESS motherboard. * possibly any hardware with discrete APIC support (80486?) * #define WARMBOOT_TO_BOOTSTRAP */ /* * defines needed for the code: */ #define WARMBOOT_OFF 0xf0000467 #define WARMBOOT_SEG 0xf0000469 #define CMOS_REG 0x70 #define CMOS_DATA 0x71 #define BIOS_RESET 0x0f #define BIOS_WARM 0x0a [ it is necessary to restore the warm-boot vector to its original value after starting the 2nd CPU. modify machdep.s, at top and bottom of cpu_startup(), add: ] cpu_startup(dummy) void *dummy; { extern int mpbioswarmvec, mpbiosreason; ... /* restore the original warm-boot information */ *(u_long*)(KERNBASE+0x0467) = mpbioswarmvec; outb( 0x70, 0x0f ); outb( 0x71, (u_char)(mpbiosreason & 0xff) ); } [ create entry point in i386/i386/mpboot.s for the warmstart @ line 139: ] NON_GPROF_ENTRY(haltpoint) dead: hlt /* We should never get here */ [ the new code for apic_startup in i386/i386/mpcore.s ] NON_GPROF_ENTRY(apic_startup) /* * this version of apic_startup (more or less) complies to the MP spec v1.4 */ movl _mpenabled, %eax /* See if the probe succeeded */ cmpl $1, %eax jz 1f ret /* SMP not enabled */ 1: pushl %esi pushl %ebx call _get_mplock /* create a BIOS warm-start vector to a 'HLT' instruction */ movl WARMBOOT_OFF,%eax movl %eax,_mpbioswarmvec #if defined( WARMBOOT_TO_BOOTSTRAP ) /* this is for rogue hardware */ movw $0,WARMBOOT_OFF /* store offset */ #else lea _haltpoint,%eax subl $_bootMP,%eax /* calculate offset */ movw %ax,WARMBOOT_OFF /* store offset */ #endif /* WARMBOOT_TO_BOOTSTRAP */ movl $MP_INITADDR,%eax /* get 32bit address */ shrl $4,%eax /* convert to segment */ movw %ax,WARMBOOT_SEG /* store segment */ /* place "warm-start" value in CMOS ram for BIOS */ cli /* disable INTs */ movb $BIOS_RESET,%al /* address of "reason" */ outb %al,$CMOS_REG inb $CMOS_DATA,%al /* read current value */ movl %eax,_mpbiosreason /* save it for later */ movb $BIOS_RESET,%al /* address of "reason" */ outb %al,$CMOS_REG movb $BIOS_WARM,%al /* "warm" thru (40:67) */ outb %al,$CMOS_DATA /* store it */ sti /* re-enable INTs */ movl _apic_base, %esi /* Step 1 - Do a INIT/Reset seqeuence */ movl APIC_ICR_HI(%esi),%eax andl $0xf0ffffff,%eax /* mask out ID bits */ orl $CPUNBR,%eax movl %eax,APIC_ICR_HI(%esi) movl APIC_ICR_LOW(%esi),%eax andl $0xfff00000,%eax orl $0x0000c500,%eax /* do a INIT IPI - ASSERT/RESET */ movl %eax,APIC_ICR_LOW(%esi) 1: movl APIC_ICR_LOW(%esi),%eax /* wait for pending status end */ andl $0x00001000,%eax jnz 1b xorl %ebx, %ebx movl $0x10000, %eax /* Delay a bit */ 2: decl %eax cmpl %eax, %ebx jnz 2b movl APIC_ICR_HI(%esi),%eax andl $0xf0ffffff,%eax /* mask out ID bits */ orl $CPUNBR,%eax movl %eax,APIC_ICR_HI(%esi) movl APIC_ICR_LOW(%esi),%eax andl $0xfff00000,%eax orl $0x00008500,%eax /* do a INIT IPI - DEASSERT/RESET */ movl %eax,APIC_ICR_LOW(%esi) 3: movl APIC_ICR_LOW(%esi),%eax /* wait for pending status end */ andl $0x00001000,%eax jnz 3b /* Step 2 - Do the STARTUP IPI sequence */ xorl %ebx, %ebx movl $0x10000, %eax /* Delay a bit */ 1: decl %eax cmpl %eax, %ebx jnz 1b 2: movl APIC_ICR_LOW(%esi),%eax /* wait for pending status end */ andl $0x00001000,%eax jnz 2b movl APIC_ICR_HI(%esi),%eax andl $0xf0ffffff,%eax /* mask out ID bits */ orl $CPUNBR,%eax movl %eax,APIC_ICR_HI(%esi) movl APIC_ICR_LOW(%esi),%eax andl $0xfff00000,%eax orl $0x0000069f,%eax /* do a STARTUP IPI */ movl %eax,APIC_ICR_LOW(%esi) xorl %ebx, %ebx movl $0x1000000, %eax /* Delay a bit */ 3: decl %eax cmpl %eax, %ebx jnz 3b 4: movl APIC_ICR_LOW(%esi),%eax /* wait for pending status end */ andl $0x00001000,%eax jnz 4b popl %ebx popl %esi ret >Audit-Trail: >Unformatted: