From owner-freebsd-mips@FreeBSD.ORG Sun Apr 18 17:38:08 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA17A1065672; Sun, 18 Apr 2010 17:38:07 +0000 (UTC) (envelope-from rrs@lakerest.net) Received: from lakerest.net (unknown [IPv6:2001:240:585:2:213:d4ff:fef3:2d8d]) by mx1.freebsd.org (Postfix) with ESMTP id 6E4868FC14; Sun, 18 Apr 2010 17:38:07 +0000 (UTC) Received: from [192.168.2.178] (pool-173-67-91-86.snfcca.dsl-w.verizon.net [173.67.91.86]) (authenticated bits=0) by lakerest.net (8.14.3/8.14.3) with ESMTP id o3IHbw3x041531 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sun, 18 Apr 2010 13:38:04 -0400 (EDT) (envelope-from rrs@lakerest.net) Message-Id: From: Randall Stewart To: Rui Paulo In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Sun, 18 Apr 2010 10:37:53 -0700 References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> X-Mailer: Apple Mail (2.936) Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Apr 2010 17:38:08 -0000 OK, it appears in my early morning fog as I started I missed a patch ;-) Anyway... I know better than to work before my first cup of coffee ... I am rebuilding and will try again with all the patches ;-) I suspect I will see your panic.. my guess is the lockup JC saw was a LOR and he ran with NO WITNESS/INVARIANT in his build. We should be able to fix this pretty easily ;-) Give me a bit of time... R On Apr 18, 2010, at 10:17 AM, Rui Paulo wrote: > On 18 Apr 2010, at 13:09, Rui Paulo wrote: > >> On 17 Apr 2010, at 23:40, C. Jayachandran wrote: >> >>> I've a set of initial patches to enable SMP for RMI processors. It >>> comes up in multi-user with 32 CPUs. I could do buildworld before I >>> updated to HEAD - with head there is a hang during buildworld which >>> I'm looking at, but I think the initial work can be checked in. >>> >>> Neel, can you have a look at the first two patches - one is to >>> enable >>> ULE scheduler and the second one is to move platform_init_ap to >>> slightly later in the initialization sequence. >>> >>> The patches are : >>> 1. mips-ule-support.patch >>> - Enable ULE scheduler for MIPS >>> >>> 2. mips-smp-move-platform.patch >>> - We need a hook to setup message ring and its interrupts, we use >>> platform_init_ap now, and move it be called later for XLR >>> >>> 3. rmi-prid.patch >>> Add RMI processor ID prints - right now it prints unknown processor. >>> >>> 4. rmi-pcib-fix.patch >>> XLR pci bridge should be off the IODI bus, currently it is off the >>> nexus bus which will cause issue when bus_setup_intr on nexus is >>> called >>> >>> 5. cleanup-reorg-cpuid-rge-kx.patch >>> This set has a many changes rolled up: >>> - clean up rge driver, remove unused code paths and commented code >>> (long way to go still) >>> - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage >>> updated >>> - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses >>> here, I have made changes for using XKPHYS to do the same process. >>> - the KX bit is enabled and interrupts are disabled before the >>> physical memory is read, to avoid the KX bit setting from affecting >>> other code. >>> - move message ring code to on_chip.c from xlr_machdep.c, now all >>> message ring code is in on_chip.c >>> >>> 6. rmi-xlr-smp.patch >>> SMP Support for XLR >>> - mpwait.S added to get the other CPUs out of bootloader code at >>> startup. >>> - SMP platform related functions. >>> - disable shared TLB code for SMP - since threads have different >>> mappings. >>> >>> The patches are also available at >>> http://sites.google.com/site/cjayachandran/files >>> >>> Enabling all 32 threads will need a minor fix in the SMP code, see >>> the >>> patch subr_smp-fix.patch at the same place. >> >> On the netperf XLR, this panics with SCHED_ULE activated: >> >> Copyright (c) 1992-2010 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >> 1994 >> The Regents of the University of California. All rights reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010 >> rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/ >> rpaulo/freebsd/head/sys/XLR mips >> real memory = 3488608256 (3406844K bytes) >> avail memory = 3395207168 (3237MB) >> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs >> nexus0: >> iodi0 on nexus0 >> uart0: <16550 or compatible> on iodi0 >> uart0: [FILTER] >> uart0: console (38551,n,8,1) >> pcib0: on iodi0 >> pci0: on pcib0 >> pci0: at device 1.0 (no driver attached) >> pci0: at device 3.0 (no driver attached) >> rge0 on iodi0 >> rge0: [ITHREAD] >> rge0: Ethernet address: >> rge0: [10Mbps] >> Total Active Core 8 >> rge1 on iodi0 >> rge1: [ITHREAD] >> rge1: Ethernet address: >> rge1: [1000Mbps] >> rge2 on iodi0 >> rge2: [ITHREAD] >> rge2: Ethernet address: >> rge2: [10Mbps] >> rge3 on iodi0 >> rge3: [ITHREAD] >> rge3: Ethernet address: >> rge3: [10Mbps] >> Enabling MDIO interrupts >> Timecounter "MIPS32" frequency 66000000 Hz quality 800 >> Timecounters tick every 1.000 msec >> bootpc_init: wired to interface 'rge1' >> Sending DHCP Discover packet from interface rge1 >> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no >> root path) >> Sending DHCP Request packet from interface rge1 >> Sending DHCP Discover packet from interface rge1 >> DHCP/BOOTP timeout for server 255.255.255.255 >> DHCP/BOOTP timeout for server 255.255.255.255 >> DHCP/BOOTP timeout for server 255.255.255.255 >> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no >> root path) >> Sending DHCP Request packet from interface rge1 >> DHCP/BOOTP timeout for server 255.255.255.255 >> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got >> root path) >> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org >> boot file /zoo/rpaulo/lama1/boot/kernel/kernel >> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/ >> rpaulo/lama1 rootopts nolockd >> Adjusted interface rge1 >> panic: mutex sched lock 4 not owned at /usr/home/rpaulo/freebsd/ >> head/sys/kern/sched_ule.c:2054 >> cpuid = 4 >> KDB: enter: panic > > If I use SCHED_4BSD this happens: (am I missing something?) > > Copyright (c) 1992-2010 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, > 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 9.0-CURRENT #4 r206791M: Sun Apr 18 17:10:55 UTC 2010 > rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/ > rpaulo/freebsd/head/sys/XLR mips > real memory = 3488608256 (3406844K bytes) > avail memory = 3395252224 (3237MB) > FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs > nexus0: > iodi0 on nexus0 > uart0: <16550 or compatible> on iodi0 > uart0: [FILTER] > uart0: console (38551,n,8,1) > pcib0: on iodi0 > pci0: on pcib0 > pci0: at device 1.0 (no driver attached) > pci0: at device 3.0 (no driver attached) > rge0 on iodi0 > rge0: [ITHREAD] > rge0: Ethernet address: 00:0f:30:00:20:4e > rge0: [10Mbps] > Total Active Core 8 > rge1 on iodi0 > rge1: [ITHREAD] > rge1: Ethernet address: 00:0f:30:00:20:4f > rge1: [1000Mbps] > rge2 on iodi0 > rge2: [ITHREAD] > rge2: Ethernet address: 00:0f:30:00:20:50 > rge2: [10Mbps] > rge3 on iodi0 > rge3: [ITHREAD] > rge3: Ethernet address: 00:0f:30:00:20:51 > rge3: [10Mbps] > Enabling MDIO interrupts > Timecounter "MIPS32" frequency 66000000 Hz quality 800 > Timecounters tick every 1.000 msec > bootpc_init: wired to interface 'rge1' > Sending DHCP Discover packet from interface rge1 (00:0f:30:00:20:4f) > Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no > root path) > Sending DHCP Request packet from interface rge1 (00:0f:30:00:20:4f) > Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got > root path) > rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org > boot file /zoo/rpaulo/lama1/boot/kernel/kernel > subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/rpaulo/ > lama1 rootopts nolockd > Adjusted interface rge1 > spin lock 0x804b8ea4 (sched lock) held by 0xc98c0260 (tid 100027) > too long > panic: spin lock held too long > cpuid = 12 > KDB: enter: panic > > > Regards, > -- > Rui Paulo > > ------------------------------ Randall Stewart 803-317-4952 (cell) 803-345-0391(direct)