From owner-freebsd-mips@FreeBSD.ORG Sun Apr 18 18:30:42 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3A1DF106566B; Sun, 18 Apr 2010 18:30:42 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-pz0-f201.google.com (mail-pz0-f201.google.com [209.85.222.201]) by mx1.freebsd.org (Postfix) with ESMTP id 051E48FC0C; Sun, 18 Apr 2010 18:30:41 +0000 (UTC) Received: by pzk39 with SMTP id 39so2857337pzk.7 for ; Sun, 18 Apr 2010 11:30:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=h2vums1kLbzic7YP9GPDfSN1UPHiY/dUcruBf9XvJ3w=; b=DyovbqqByYe/t/+feYbfrdWLKP8LCPJXBJxubBZywKB4bSu5eS75xMQvGU1nmLqgVr uR1PuUTxRm7hViBhErdz8JffKRoJ87xlOd30VS7JQLEua+kC31iNz6kiPBafXs+4P9ES 02FiFE2oqJ1POZMATahT9xLLjvdIZ4ITeh6EI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=TFfAdmhpH6DWOd28Z2uQYYGFxJlwMp8YCuEfRJH9ntzpED/l8upgx4azm3SOWG2bjf kBTB8gqdN9zzB6V/DRvBINWVxPaRvNeDXWp8lJiXgATCAcptH+alkOeeUB0U+w7Y1tJU JOwBK3Naw4sCTsNxCrha8pIRhAefWUNQM0Yyg= MIME-Version: 1.0 Received: by 10.141.29.15 with HTTP; Sun, 18 Apr 2010 11:30:41 -0700 (PDT) In-Reply-To: References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> Date: Mon, 19 Apr 2010 00:00:41 +0530 Received: by 10.140.247.20 with SMTP id u20mr1032940rvh.122.1271615441350; Sun, 18 Apr 2010 11:30:41 -0700 (PDT) Message-ID: From: "C. Jayachandran" To: Randall Stewart Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Apr 2010 18:30:42 -0000 I did not have invariants enabled when I moved to HEAD, and I got to multiuser login prompt. The panic is easily seen with INVARIANTS enabled with all 32 CPUs. But it does not happen if I cut down the cpus to 16 [ userapp_mask_cpus 0xffff on boot prompt]. Looks like there is some race I'm hitting with more CPUs. Will look at this... JC. On Sun, Apr 18, 2010 at 11:07 PM, Randall Stewart wrote: > OK, > > it appears in my early morning fog as I started I missed a patch ;-) > Anyway... I know better than to work before my first cup of coffee ... > > I am rebuilding and will try again with all the patches ;-) > > I suspect I will see your panic.. my guess is the lockup JC saw > was a LOR and he ran with NO WITNESS/INVARIANT in his build. > > We should be able to fix this pretty easily ;-) > > Give me a bit of time... > > R > > On Apr 18, 2010, at 10:17 AM, Rui Paulo wrote: > >> On 18 Apr 2010, at 13:09, Rui Paulo wrote: >> >>> On 17 Apr 2010, at 23:40, C. Jayachandran wrote: >>> >>>> I've a set of initial patches to enable SMP for RMI processors. It >>>> comes up in multi-user with 32 CPUs. =A0I could do buildworld before I >>>> updated to HEAD - with head there is a hang during buildworld which >>>> I'm looking at, but I think the initial work can be checked in. >>>> >>>> Neel, can you have a look at the first two patches - one is to enable >>>> ULE scheduler and the second one is to move platform_init_ap to >>>> slightly later in the initialization sequence. >>>> >>>> The patches are : >>>> 1. mips-ule-support.patch >>>> - Enable ULE scheduler for MIPS >>>> >>>> 2. mips-smp-move-platform.patch >>>> - We need a hook to setup message ring and its interrupts, we use >>>> platform_init_ap now, and move it be called later for XLR >>>> >>>> 3. rmi-prid.patch >>>> Add RMI processor ID prints - right now it prints unknown processor. >>>> >>>> 4. rmi-pcib-fix.patch >>>> XLR pci bridge should be off the IODI bus, currently it is off the >>>> nexus bus which will cause issue when bus_setup_intr on nexus is >>>> called >>>> >>>> 5. cleanup-reorg-cpuid-rge-kx.patch >>>> This set has a many changes rolled up: >>>> - clean up rge driver, remove unused code paths and commented code >>>> (long way to go still) >>>> - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage updat= ed >>>> - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses >>>> here, I have made changes for using XKPHYS to do the same process. >>>> - the KX bit is enabled and interrupts are disabled before the >>>> physical memory is read, to avoid the KX bit setting from affecting >>>> other code. >>>> - move message ring code to on_chip.c from xlr_machdep.c, now all >>>> message ring code is in on_chip.c >>>> >>>> 6. rmi-xlr-smp.patch >>>> SMP Support for XLR >>>> - mpwait.S added to get the other CPUs out of bootloader code at >>>> startup. >>>> - SMP platform related functions. >>>> - disable shared TLB code for SMP - since threads have different >>>> mappings. >>>> >>>> The patches are also available at >>>> http://sites.google.com/site/cjayachandran/files >>>> >>>> Enabling all 32 threads will need a minor fix in the SMP code, see the >>>> patch subr_smp-fix.patch at the same place. >>> >>> On the netperf XLR, this panics with SCHED_ULE activated: >>> >>> Copyright (c) 1992-2010 The FreeBSD Project. >>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 199= 4 >>> =A0 =A0 =A0 =A0The Regents of the University of California. All rights = reserved. >>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>> FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010 >>> >>> =A0rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpa= ulo/freebsd/head/sys/XLR >>> mips >>> real memory =A0=3D 3488608256 (3406844K bytes) >>> avail memory =3D 3395207168 (3237MB) >>> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs >>> nexus0: >>> iodi0 on nexus0 >>> uart0: <16550 or compatible> on iodi0 >>> uart0: [FILTER] >>> uart0: console (38551,n,8,1) >>> pcib0: on iodi0 >>> pci0: on pcib0 >>> pci0: at device 1.0 (no driver attached) >>> pci0: at device 3.0 (no driver attached) >>> rge0 on iodi0 >>> rge0: [ITHREAD] >>> rge0: Ethernet address: >>> rge0: [10Mbps] >>> Total Active Core 8 >>> rge1 on iodi0 >>> rge1: [ITHREAD] >>> rge1: Ethernet address: >>> rge1: [1000Mbps] >>> rge2 on iodi0 >>> rge2: [ITHREAD] >>> rge2: Ethernet address: >>> rge2: [10Mbps] >>> rge3 on iodi0 >>> rge3: [ITHREAD] >>> rge3: Ethernet address: >>> rge3: [10Mbps] >>> Enabling MDIO interrupts >>> Timecounter "MIPS32" frequency 66000000 Hz quality 800 >>> Timecounters tick every 1.000 msec >>> bootpc_init: wired to interface 'rge1' >>> Sending DHCP Discover packet from interface rge1 >>> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root >>> path) >>> Sending DHCP Request packet from interface rge1 >>> Sending DHCP Discover packet from interface rge1 >>> DHCP/BOOTP timeout for server 255.255.255.255 >>> DHCP/BOOTP timeout for server 255.255.255.255 >>> DHCP/BOOTP timeout for server 255.255.255.255 >>> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root >>> path) >>> Sending DHCP Request packet from interface rge1 >>> DHCP/BOOTP timeout for server 255.255.255.255 >>> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root >>> path) >>> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org boo= t >>> file /zoo/rpaulo/lama1/boot/kernel/kernel >>> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs >>> /zoo/rpaulo/lama1 rootopts nolockd >>> Adjusted interface rge1 >>> panic: mutex sched lock 4 not owned at >>> /usr/home/rpaulo/freebsd/head/sys/kern/sched_ule.c:2054 >>> cpuid =3D 4 >>> KDB: enter: panic >> >> If I use SCHED_4BSD this happens: (am I missing something?) >> >> Copyright (c) 1992-2010 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >> =A0 =A0 =A0 =A0The Regents of the University of California. All rights r= eserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 9.0-CURRENT #4 r206791M: Sun Apr 18 17:10:55 UTC 2010 >> >> rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpaulo/= freebsd/head/sys/XLR >> mips >> real memory =A0=3D 3488608256 (3406844K bytes) >> avail memory =3D 3395252224 (3237MB) >> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs >> nexus0: >> iodi0 on nexus0 >> uart0: <16550 or compatible> on iodi0 >> uart0: [FILTER] >> uart0: console (38551,n,8,1) >> pcib0: on iodi0 >> pci0: on pcib0 >> pci0: at device 1.0 (no driver attached) >> pci0: at device 3.0 (no driver attached) >> rge0 on iodi0 >> rge0: [ITHREAD] >> rge0: Ethernet address: 00:0f:30:00:20:4e >> rge0: [10Mbps] >> Total Active Core 8 >> rge1 on iodi0 >> rge1: [ITHREAD] >> rge1: Ethernet address: 00:0f:30:00:20:4f >> rge1: [1000Mbps] >> rge2 on iodi0 >> rge2: [ITHREAD] >> rge2: Ethernet address: 00:0f:30:00:20:50 >> rge2: [10Mbps] >> rge3 on iodi0 >> rge3: [ITHREAD] >> rge3: Ethernet address: 00:0f:30:00:20:51 >> rge3: [10Mbps] >> Enabling MDIO interrupts >> Timecounter "MIPS32" frequency 66000000 Hz quality 800 >> Timecounters tick every 1.000 msec >> bootpc_init: wired to interface 'rge1' >> Sending DHCP Discover packet from interface rge1 (00:0f:30:00:20:4f) >> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root >> path) >> Sending DHCP Request packet from interface rge1 (00:0f:30:00:20:4f) >> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root >> path) >> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org boot >> file /zoo/rpaulo/lama1/boot/kernel/kernel >> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/rpaulo/lam= a1 >> rootopts nolockd >> Adjusted interface rge1 >> spin lock 0x804b8ea4 (sched lock) held by 0xc98c0260 (tid 100027) too lo= ng >> panic: spin lock held too long >> cpuid =3D 12 >> KDB: enter: panic >> >> >> Regards, >> -- >> Rui Paulo >> >> > > ------------------------------ > Randall Stewart > 803-317-4952 (cell) > 803-345-0391(direct) > > --=20 C. Jayachandran c.jayachandran@gmail.com