From owner-freebsd-mips@FreeBSD.ORG Sun Apr 18 12:36:43 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5CDCD1065672 for ; Sun, 18 Apr 2010 12:36:43 +0000 (UTC) (envelope-from rpaulo@freebsd.org) Received: from karen.lavabit.com (karen.lavabit.com [72.249.41.33]) by mx1.freebsd.org (Postfix) with ESMTP id 2FA558FC13 for ; Sun, 18 Apr 2010 12:36:42 +0000 (UTC) Received: from e.earth.lavabit.com (e.earth.lavabit.com [192.168.111.14]) by karen.lavabit.com (Postfix) with ESMTP id A283711B9CC; Sun, 18 Apr 2010 07:10:02 -0500 (CDT) Received: from 10.0.10.2 (54.81.54.77.rev.vodafone.pt [77.54.81.54]) by lavabit.com with ESMTP id VQD7HREK6007; Sun, 18 Apr 2010 07:10:02 -0500 Mime-Version: 1.0 (Apple Message framework v1078) Content-Type: text/plain; charset=us-ascii From: Rui Paulo In-Reply-To: Date: Sun, 18 Apr 2010 13:09:59 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> References: To: C. Jayachandran X-Mailer: Apple Mail (2.1078) Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Apr 2010 12:36:43 -0000 On 17 Apr 2010, at 23:40, C. Jayachandran wrote: > I've a set of initial patches to enable SMP for RMI processors. It > comes up in multi-user with 32 CPUs. I could do buildworld before I > updated to HEAD - with head there is a hang during buildworld which > I'm looking at, but I think the initial work can be checked in. >=20 > Neel, can you have a look at the first two patches - one is to enable > ULE scheduler and the second one is to move platform_init_ap to > slightly later in the initialization sequence. >=20 > The patches are : > 1. mips-ule-support.patch > - Enable ULE scheduler for MIPS >=20 > 2. mips-smp-move-platform.patch > - We need a hook to setup message ring and its interrupts, we use > platform_init_ap now, and move it be called later for XLR >=20 > 3. rmi-prid.patch > Add RMI processor ID prints - right now it prints unknown processor. >=20 > 4. rmi-pcib-fix.patch > XLR pci bridge should be off the IODI bus, currently it is off the > nexus bus which will cause issue when bus_setup_intr on nexus is > called >=20 > 5. cleanup-reorg-cpuid-rge-kx.patch > This set has a many changes rolled up: > - clean up rge driver, remove unused code paths and commented code > (long way to go still) > - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage = updated > - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses > here, I have made changes for using XKPHYS to do the same process. > - the KX bit is enabled and interrupts are disabled before the > physical memory is read, to avoid the KX bit setting from affecting > other code. > - move message ring code to on_chip.c from xlr_machdep.c, now all > message ring code is in on_chip.c >=20 > 6. rmi-xlr-smp.patch > SMP Support for XLR > - mpwait.S added to get the other CPUs out of bootloader code at = startup. > - SMP platform related functions. > - disable shared TLB code for SMP - since threads have different = mappings. >=20 > The patches are also available at > http://sites.google.com/site/cjayachandran/files >=20 > Enabling all 32 threads will need a minor fix in the SMP code, see the > patch subr_smp-fix.patch at the same place. On the netperf XLR, this panics with SCHED_ULE activated: Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights = reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010 = rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpaulo/fr= eebsd/head/sys/XLR mips real memory =3D 3488608256 (3406844K bytes) avail memory =3D 3395207168 (3237MB) FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs nexus0: iodi0 on nexus0 uart0: <16550 or compatible> on iodi0 uart0: [FILTER] uart0: console (38551,n,8,1) pcib0: on iodi0 pci0: on pcib0 pci0: at device 1.0 (no driver attached) pci0: at device 3.0 (no driver attached) rge0 on iodi0 rge0: [ITHREAD] rge0: Ethernet address:=20 rge0: [10Mbps] Total Active Core 8 rge1 on iodi0 rge1: [ITHREAD] rge1: Ethernet address: rge1: [1000Mbps] rge2 on iodi0 rge2: [ITHREAD] rge2: Ethernet address:=20 rge2: [10Mbps] rge3 on iodi0 rge3: [ITHREAD] rge3: Ethernet address:=20 rge3: [10Mbps] Enabling MDIO interrupts Timecounter "MIPS32" frequency 66000000 Hz quality 800 Timecounters tick every 1.000 msec bootpc_init: wired to interface 'rge1' Sending DHCP Discover packet from interface rge1 Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root = path) Sending DHCP Request packet from interface rge1 Sending DHCP Discover packet from interface rge1 DHCP/BOOTP timeout for server 255.255.255.255 DHCP/BOOTP timeout for server 255.255.255.255 DHCP/BOOTP timeout for server 255.255.255.255 Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root = path) Sending DHCP Request packet from interface rge1 DHCP/BOOTP timeout for server 255.255.255.255 Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root = path) rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org boot = file /zoo/rpaulo/lama1/boot/kernel/kernel subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs = /zoo/rpaulo/lama1 rootopts nolockd=20 Adjusted interface rge1 panic: mutex sched lock 4 not owned at = /usr/home/rpaulo/freebsd/head/sys/kern/sched_ule.c:2054 cpuid =3D 4 KDB: enter: panic Regards, -- Rui Paulo From owner-freebsd-mips@FreeBSD.ORG Sun Apr 18 17:17:47 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A6F0E106566B for ; Sun, 18 Apr 2010 17:17:47 +0000 (UTC) (envelope-from rpaulo@freebsd.org) Received: from karen.lavabit.com (karen.lavabit.com [72.249.41.33]) by mx1.freebsd.org (Postfix) with ESMTP id 79DF68FC15 for ; Sun, 18 Apr 2010 17:17:47 +0000 (UTC) Received: from e.earth.lavabit.com (e.earth.lavabit.com [192.168.111.14]) by karen.lavabit.com (Postfix) with ESMTP id F01D3157544; Sun, 18 Apr 2010 12:17:46 -0500 (CDT) Received: from 10.0.10.2 (54.81.54.77.rev.vodafone.pt [77.54.81.54]) by lavabit.com with ESMTP id W5GDDAWKCJ7U; Sun, 18 Apr 2010 12:17:46 -0500 Mime-Version: 1.0 (Apple Message framework v1078) Content-Type: text/plain; charset=us-ascii From: Rui Paulo In-Reply-To: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> Date: Sun, 18 Apr 2010 18:17:43 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> To: "C. Jayachandran" X-Mailer: Apple Mail (2.1078) Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Apr 2010 17:17:47 -0000 On 18 Apr 2010, at 13:09, Rui Paulo wrote: > On 17 Apr 2010, at 23:40, C. Jayachandran wrote: >=20 >> I've a set of initial patches to enable SMP for RMI processors. It >> comes up in multi-user with 32 CPUs. I could do buildworld before I >> updated to HEAD - with head there is a hang during buildworld which >> I'm looking at, but I think the initial work can be checked in. >>=20 >> Neel, can you have a look at the first two patches - one is to enable >> ULE scheduler and the second one is to move platform_init_ap to >> slightly later in the initialization sequence. >>=20 >> The patches are : >> 1. mips-ule-support.patch >> - Enable ULE scheduler for MIPS >>=20 >> 2. mips-smp-move-platform.patch >> - We need a hook to setup message ring and its interrupts, we use >> platform_init_ap now, and move it be called later for XLR >>=20 >> 3. rmi-prid.patch >> Add RMI processor ID prints - right now it prints unknown processor. >>=20 >> 4. rmi-pcib-fix.patch >> XLR pci bridge should be off the IODI bus, currently it is off the >> nexus bus which will cause issue when bus_setup_intr on nexus is >> called >>=20 >> 5. cleanup-reorg-cpuid-rge-kx.patch >> This set has a many changes rolled up: >> - clean up rge driver, remove unused code paths and commented code >> (long way to go still) >> - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage = updated >> - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses >> here, I have made changes for using XKPHYS to do the same process. >> - the KX bit is enabled and interrupts are disabled before the >> physical memory is read, to avoid the KX bit setting from affecting >> other code. >> - move message ring code to on_chip.c from xlr_machdep.c, now all >> message ring code is in on_chip.c >>=20 >> 6. rmi-xlr-smp.patch >> SMP Support for XLR >> - mpwait.S added to get the other CPUs out of bootloader code at = startup. >> - SMP platform related functions. >> - disable shared TLB code for SMP - since threads have different = mappings. >>=20 >> The patches are also available at >> http://sites.google.com/site/cjayachandran/files >>=20 >> Enabling all 32 threads will need a minor fix in the SMP code, see = the >> patch subr_smp-fix.patch at the same place. >=20 > On the netperf XLR, this panics with SCHED_ULE activated: >=20 > Copyright (c) 1992-2010 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, = 1994 > The Regents of the University of California. All rights = reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010 > = rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpaulo/fr= eebsd/head/sys/XLR mips > real memory =3D 3488608256 (3406844K bytes) > avail memory =3D 3395207168 (3237MB) > FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs > nexus0: > iodi0 on nexus0 > uart0: <16550 or compatible> on iodi0 > uart0: [FILTER] > uart0: console (38551,n,8,1) > pcib0: on iodi0 > pci0: on pcib0 > pci0: at device 1.0 (no driver attached) > pci0: at device 3.0 (no driver attached) > rge0 on iodi0 > rge0: [ITHREAD] > rge0: Ethernet address:=20 > rge0: [10Mbps] > Total Active Core 8 > rge1 on iodi0 > rge1: [ITHREAD] > rge1: Ethernet address: > rge1: [1000Mbps] > rge2 on iodi0 > rge2: [ITHREAD] > rge2: Ethernet address:=20 > rge2: [10Mbps] > rge3 on iodi0 > rge3: [ITHREAD] > rge3: Ethernet address:=20 > rge3: [10Mbps] > Enabling MDIO interrupts > Timecounter "MIPS32" frequency 66000000 Hz quality 800 > Timecounters tick every 1.000 msec > bootpc_init: wired to interface 'rge1' > Sending DHCP Discover packet from interface rge1 > Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no = root path) > Sending DHCP Request packet from interface rge1 > Sending DHCP Discover packet from interface rge1 > DHCP/BOOTP timeout for server 255.255.255.255 > DHCP/BOOTP timeout for server 255.255.255.255 > DHCP/BOOTP timeout for server 255.255.255.255 > Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no = root path) > Sending DHCP Request packet from interface rge1 > DHCP/BOOTP timeout for server 255.255.255.255 > Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root = path) > rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org = boot file /zoo/rpaulo/lama1/boot/kernel/kernel > subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs = /zoo/rpaulo/lama1 rootopts nolockd=20 > Adjusted interface rge1 > panic: mutex sched lock 4 not owned at = /usr/home/rpaulo/freebsd/head/sys/kern/sched_ule.c:2054 > cpuid =3D 4 > KDB: enter: panic If I use SCHED_4BSD this happens: (am I missing something?) Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights = reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-CURRENT #4 r206791M: Sun Apr 18 17:10:55 UTC 2010 = rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpaulo/fr= eebsd/head/sys/XLR mips real memory =3D 3488608256 (3406844K bytes) avail memory =3D 3395252224 (3237MB) FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs nexus0: iodi0 on nexus0 uart0: <16550 or compatible> on iodi0 uart0: [FILTER] uart0: console (38551,n,8,1) pcib0: on iodi0 pci0: on pcib0 pci0: at device 1.0 (no driver attached) pci0: at device 3.0 (no driver attached) rge0 on iodi0 rge0: [ITHREAD] rge0: Ethernet address: 00:0f:30:00:20:4e rge0: [10Mbps] Total Active Core 8 rge1 on iodi0 rge1: [ITHREAD] rge1: Ethernet address: 00:0f:30:00:20:4f rge1: [1000Mbps] rge2 on iodi0 rge2: [ITHREAD] rge2: Ethernet address: 00:0f:30:00:20:50 rge2: [10Mbps] rge3 on iodi0 rge3: [ITHREAD] rge3: Ethernet address: 00:0f:30:00:20:51 rge3: [10Mbps] Enabling MDIO interrupts Timecounter "MIPS32" frequency 66000000 Hz quality 800 Timecounters tick every 1.000 msec bootpc_init: wired to interface 'rge1' Sending DHCP Discover packet from interface rge1 (00:0f:30:00:20:4f) Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root = path) Sending DHCP Request packet from interface rge1 (00:0f:30:00:20:4f) Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root = path) rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org boot = file /zoo/rpaulo/lama1/boot/kernel/kernel subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs = /zoo/rpaulo/lama1 rootopts nolockd=20 Adjusted interface rge1 spin lock 0x804b8ea4 (sched lock) held by 0xc98c0260 (tid 100027) too = long panic: spin lock held too long cpuid =3D 12 KDB: enter: panic Regards, -- Rui Paulo From owner-freebsd-mips@FreeBSD.ORG Sun Apr 18 17:25:13 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 98DE0106566B; Sun, 18 Apr 2010 17:25:13 +0000 (UTC) (envelope-from rrs@lakerest.net) Received: from lakerest.net (unknown [IPv6:2001:240:585:2:213:d4ff:fef3:2d8d]) by mx1.freebsd.org (Postfix) with ESMTP id 164008FC1A; Sun, 18 Apr 2010 17:25:12 +0000 (UTC) Received: from [192.168.2.178] (pool-173-67-91-86.snfcca.dsl-w.verizon.net [173.67.91.86]) (authenticated bits=0) by lakerest.net (8.14.3/8.14.3) with ESMTP id o3IHP4MY040942 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sun, 18 Apr 2010 13:25:11 -0400 (EDT) (envelope-from rrs@lakerest.net) Message-Id: <6C232389-B51B-406A-BA28-1B5F77977A2F@lakerest.net> From: Randall Stewart To: Rui Paulo In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Sun, 18 Apr 2010 10:24:57 -0700 References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> X-Mailer: Apple Mail (2.936) Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Apr 2010 17:25:13 -0000 Rui: Not sure.. I have applied the patches to head as of this am rebuilt everything and am not seeing the same crash.. I may have missed a patch since I could not even config ULE.. and my smp build is now not handling the DHCP offer.. I think I will revert and retry all the patches R On Apr 18, 2010, at 10:17 AM, Rui Paulo wrote: > On 18 Apr 2010, at 13:09, Rui Paulo wrote: > >> On 17 Apr 2010, at 23:40, C. Jayachandran wrote: >> >>> I've a set of initial patches to enable SMP for RMI processors. It >>> comes up in multi-user with 32 CPUs. I could do buildworld before I >>> updated to HEAD - with head there is a hang during buildworld which >>> I'm looking at, but I think the initial work can be checked in. >>> >>> Neel, can you have a look at the first two patches - one is to >>> enable >>> ULE scheduler and the second one is to move platform_init_ap to >>> slightly later in the initialization sequence. >>> >>> The patches are : >>> 1. mips-ule-support.patch >>> - Enable ULE scheduler for MIPS >>> >>> 2. mips-smp-move-platform.patch >>> - We need a hook to setup message ring and its interrupts, we use >>> platform_init_ap now, and move it be called later for XLR >>> >>> 3. rmi-prid.patch >>> Add RMI processor ID prints - right now it prints unknown processor. >>> >>> 4. rmi-pcib-fix.patch >>> XLR pci bridge should be off the IODI bus, currently it is off the >>> nexus bus which will cause issue when bus_setup_intr on nexus is >>> called >>> >>> 5. cleanup-reorg-cpuid-rge-kx.patch >>> This set has a many changes rolled up: >>> - clean up rge driver, remove unused code paths and commented code >>> (long way to go still) >>> - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage >>> updated >>> - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses >>> here, I have made changes for using XKPHYS to do the same process. >>> - the KX bit is enabled and interrupts are disabled before the >>> physical memory is read, to avoid the KX bit setting from affecting >>> other code. >>> - move message ring code to on_chip.c from xlr_machdep.c, now all >>> message ring code is in on_chip.c >>> >>> 6. rmi-xlr-smp.patch >>> SMP Support for XLR >>> - mpwait.S added to get the other CPUs out of bootloader code at >>> startup. >>> - SMP platform related functions. >>> - disable shared TLB code for SMP - since threads have different >>> mappings. >>> >>> The patches are also available at >>> http://sites.google.com/site/cjayachandran/files >>> >>> Enabling all 32 threads will need a minor fix in the SMP code, see >>> the >>> patch subr_smp-fix.patch at the same place. >> >> On the netperf XLR, this panics with SCHED_ULE activated: >> >> Copyright (c) 1992-2010 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >> 1994 >> The Regents of the University of California. All rights reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010 >> rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/ >> rpaulo/freebsd/head/sys/XLR mips >> real memory = 3488608256 (3406844K bytes) >> avail memory = 3395207168 (3237MB) >> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs >> nexus0: >> iodi0 on nexus0 >> uart0: <16550 or compatible> on iodi0 >> uart0: [FILTER] >> uart0: console (38551,n,8,1) >> pcib0: on iodi0 >> pci0: on pcib0 >> pci0: at device 1.0 (no driver attached) >> pci0: at device 3.0 (no driver attached) >> rge0 on iodi0 >> rge0: [ITHREAD] >> rge0: Ethernet address: >> rge0: [10Mbps] >> Total Active Core 8 >> rge1 on iodi0 >> rge1: [ITHREAD] >> rge1: Ethernet address: >> rge1: [1000Mbps] >> rge2 on iodi0 >> rge2: [ITHREAD] >> rge2: Ethernet address: >> rge2: [10Mbps] >> rge3 on iodi0 >> rge3: [ITHREAD] >> rge3: Ethernet address: >> rge3: [10Mbps] >> Enabling MDIO interrupts >> Timecounter "MIPS32" frequency 66000000 Hz quality 800 >> Timecounters tick every 1.000 msec >> bootpc_init: wired to interface 'rge1' >> Sending DHCP Discover packet from interface rge1 >> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no >> root path) >> Sending DHCP Request packet from interface rge1 >> Sending DHCP Discover packet from interface rge1 >> DHCP/BOOTP timeout for server 255.255.255.255 >> DHCP/BOOTP timeout for server 255.255.255.255 >> DHCP/BOOTP timeout for server 255.255.255.255 >> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no >> root path) >> Sending DHCP Request packet from interface rge1 >> DHCP/BOOTP timeout for server 255.255.255.255 >> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got >> root path) >> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org >> boot file /zoo/rpaulo/lama1/boot/kernel/kernel >> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/ >> rpaulo/lama1 rootopts nolockd >> Adjusted interface rge1 >> panic: mutex sched lock 4 not owned at /usr/home/rpaulo/freebsd/ >> head/sys/kern/sched_ule.c:2054 >> cpuid = 4 >> KDB: enter: panic > > If I use SCHED_4BSD this happens: (am I missing something?) > > Copyright (c) 1992-2010 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, > 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 9.0-CURRENT #4 r206791M: Sun Apr 18 17:10:55 UTC 2010 > rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/ > rpaulo/freebsd/head/sys/XLR mips > real memory = 3488608256 (3406844K bytes) > avail memory = 3395252224 (3237MB) > FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs > nexus0: > iodi0 on nexus0 > uart0: <16550 or compatible> on iodi0 > uart0: [FILTER] > uart0: console (38551,n,8,1) > pcib0: on iodi0 > pci0: on pcib0 > pci0: at device 1.0 (no driver attached) > pci0: at device 3.0 (no driver attached) > rge0 on iodi0 > rge0: [ITHREAD] > rge0: Ethernet address: 00:0f:30:00:20:4e > rge0: [10Mbps] > Total Active Core 8 > rge1 on iodi0 > rge1: [ITHREAD] > rge1: Ethernet address: 00:0f:30:00:20:4f > rge1: [1000Mbps] > rge2 on iodi0 > rge2: [ITHREAD] > rge2: Ethernet address: 00:0f:30:00:20:50 > rge2: [10Mbps] > rge3 on iodi0 > rge3: [ITHREAD] > rge3: Ethernet address: 00:0f:30:00:20:51 > rge3: [10Mbps] > Enabling MDIO interrupts > Timecounter "MIPS32" frequency 66000000 Hz quality 800 > Timecounters tick every 1.000 msec > bootpc_init: wired to interface 'rge1' > Sending DHCP Discover packet from interface rge1 (00:0f:30:00:20:4f) > Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no > root path) > Sending DHCP Request packet from interface rge1 (00:0f:30:00:20:4f) > Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got > root path) > rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org > boot file /zoo/rpaulo/lama1/boot/kernel/kernel > subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/rpaulo/ > lama1 rootopts nolockd > Adjusted interface rge1 > spin lock 0x804b8ea4 (sched lock) held by 0xc98c0260 (tid 100027) > too long > panic: spin lock held too long > cpuid = 12 > KDB: enter: panic > > > Regards, > -- > Rui Paulo > > ------------------------------ Randall Stewart 803-317-4952 (cell) 803-345-0391(direct) From owner-freebsd-mips@FreeBSD.ORG Sun Apr 18 17:38:08 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA17A1065672; Sun, 18 Apr 2010 17:38:07 +0000 (UTC) (envelope-from rrs@lakerest.net) Received: from lakerest.net (unknown [IPv6:2001:240:585:2:213:d4ff:fef3:2d8d]) by mx1.freebsd.org (Postfix) with ESMTP id 6E4868FC14; Sun, 18 Apr 2010 17:38:07 +0000 (UTC) Received: from [192.168.2.178] (pool-173-67-91-86.snfcca.dsl-w.verizon.net [173.67.91.86]) (authenticated bits=0) by lakerest.net (8.14.3/8.14.3) with ESMTP id o3IHbw3x041531 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sun, 18 Apr 2010 13:38:04 -0400 (EDT) (envelope-from rrs@lakerest.net) Message-Id: From: Randall Stewart To: Rui Paulo In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Sun, 18 Apr 2010 10:37:53 -0700 References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> X-Mailer: Apple Mail (2.936) Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Apr 2010 17:38:08 -0000 OK, it appears in my early morning fog as I started I missed a patch ;-) Anyway... I know better than to work before my first cup of coffee ... I am rebuilding and will try again with all the patches ;-) I suspect I will see your panic.. my guess is the lockup JC saw was a LOR and he ran with NO WITNESS/INVARIANT in his build. We should be able to fix this pretty easily ;-) Give me a bit of time... R On Apr 18, 2010, at 10:17 AM, Rui Paulo wrote: > On 18 Apr 2010, at 13:09, Rui Paulo wrote: > >> On 17 Apr 2010, at 23:40, C. Jayachandran wrote: >> >>> I've a set of initial patches to enable SMP for RMI processors. It >>> comes up in multi-user with 32 CPUs. I could do buildworld before I >>> updated to HEAD - with head there is a hang during buildworld which >>> I'm looking at, but I think the initial work can be checked in. >>> >>> Neel, can you have a look at the first two patches - one is to >>> enable >>> ULE scheduler and the second one is to move platform_init_ap to >>> slightly later in the initialization sequence. >>> >>> The patches are : >>> 1. mips-ule-support.patch >>> - Enable ULE scheduler for MIPS >>> >>> 2. mips-smp-move-platform.patch >>> - We need a hook to setup message ring and its interrupts, we use >>> platform_init_ap now, and move it be called later for XLR >>> >>> 3. rmi-prid.patch >>> Add RMI processor ID prints - right now it prints unknown processor. >>> >>> 4. rmi-pcib-fix.patch >>> XLR pci bridge should be off the IODI bus, currently it is off the >>> nexus bus which will cause issue when bus_setup_intr on nexus is >>> called >>> >>> 5. cleanup-reorg-cpuid-rge-kx.patch >>> This set has a many changes rolled up: >>> - clean up rge driver, remove unused code paths and commented code >>> (long way to go still) >>> - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage >>> updated >>> - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses >>> here, I have made changes for using XKPHYS to do the same process. >>> - the KX bit is enabled and interrupts are disabled before the >>> physical memory is read, to avoid the KX bit setting from affecting >>> other code. >>> - move message ring code to on_chip.c from xlr_machdep.c, now all >>> message ring code is in on_chip.c >>> >>> 6. rmi-xlr-smp.patch >>> SMP Support for XLR >>> - mpwait.S added to get the other CPUs out of bootloader code at >>> startup. >>> - SMP platform related functions. >>> - disable shared TLB code for SMP - since threads have different >>> mappings. >>> >>> The patches are also available at >>> http://sites.google.com/site/cjayachandran/files >>> >>> Enabling all 32 threads will need a minor fix in the SMP code, see >>> the >>> patch subr_smp-fix.patch at the same place. >> >> On the netperf XLR, this panics with SCHED_ULE activated: >> >> Copyright (c) 1992-2010 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >> 1994 >> The Regents of the University of California. All rights reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010 >> rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/ >> rpaulo/freebsd/head/sys/XLR mips >> real memory = 3488608256 (3406844K bytes) >> avail memory = 3395207168 (3237MB) >> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs >> nexus0: >> iodi0 on nexus0 >> uart0: <16550 or compatible> on iodi0 >> uart0: [FILTER] >> uart0: console (38551,n,8,1) >> pcib0: on iodi0 >> pci0: on pcib0 >> pci0: at device 1.0 (no driver attached) >> pci0: at device 3.0 (no driver attached) >> rge0 on iodi0 >> rge0: [ITHREAD] >> rge0: Ethernet address: >> rge0: [10Mbps] >> Total Active Core 8 >> rge1 on iodi0 >> rge1: [ITHREAD] >> rge1: Ethernet address: >> rge1: [1000Mbps] >> rge2 on iodi0 >> rge2: [ITHREAD] >> rge2: Ethernet address: >> rge2: [10Mbps] >> rge3 on iodi0 >> rge3: [ITHREAD] >> rge3: Ethernet address: >> rge3: [10Mbps] >> Enabling MDIO interrupts >> Timecounter "MIPS32" frequency 66000000 Hz quality 800 >> Timecounters tick every 1.000 msec >> bootpc_init: wired to interface 'rge1' >> Sending DHCP Discover packet from interface rge1 >> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no >> root path) >> Sending DHCP Request packet from interface rge1 >> Sending DHCP Discover packet from interface rge1 >> DHCP/BOOTP timeout for server 255.255.255.255 >> DHCP/BOOTP timeout for server 255.255.255.255 >> DHCP/BOOTP timeout for server 255.255.255.255 >> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no >> root path) >> Sending DHCP Request packet from interface rge1 >> DHCP/BOOTP timeout for server 255.255.255.255 >> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got >> root path) >> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org >> boot file /zoo/rpaulo/lama1/boot/kernel/kernel >> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/ >> rpaulo/lama1 rootopts nolockd >> Adjusted interface rge1 >> panic: mutex sched lock 4 not owned at /usr/home/rpaulo/freebsd/ >> head/sys/kern/sched_ule.c:2054 >> cpuid = 4 >> KDB: enter: panic > > If I use SCHED_4BSD this happens: (am I missing something?) > > Copyright (c) 1992-2010 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, > 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 9.0-CURRENT #4 r206791M: Sun Apr 18 17:10:55 UTC 2010 > rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/ > rpaulo/freebsd/head/sys/XLR mips > real memory = 3488608256 (3406844K bytes) > avail memory = 3395252224 (3237MB) > FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs > nexus0: > iodi0 on nexus0 > uart0: <16550 or compatible> on iodi0 > uart0: [FILTER] > uart0: console (38551,n,8,1) > pcib0: on iodi0 > pci0: on pcib0 > pci0: at device 1.0 (no driver attached) > pci0: at device 3.0 (no driver attached) > rge0 on iodi0 > rge0: [ITHREAD] > rge0: Ethernet address: 00:0f:30:00:20:4e > rge0: [10Mbps] > Total Active Core 8 > rge1 on iodi0 > rge1: [ITHREAD] > rge1: Ethernet address: 00:0f:30:00:20:4f > rge1: [1000Mbps] > rge2 on iodi0 > rge2: [ITHREAD] > rge2: Ethernet address: 00:0f:30:00:20:50 > rge2: [10Mbps] > rge3 on iodi0 > rge3: [ITHREAD] > rge3: Ethernet address: 00:0f:30:00:20:51 > rge3: [10Mbps] > Enabling MDIO interrupts > Timecounter "MIPS32" frequency 66000000 Hz quality 800 > Timecounters tick every 1.000 msec > bootpc_init: wired to interface 'rge1' > Sending DHCP Discover packet from interface rge1 (00:0f:30:00:20:4f) > Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no > root path) > Sending DHCP Request packet from interface rge1 (00:0f:30:00:20:4f) > Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got > root path) > rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org > boot file /zoo/rpaulo/lama1/boot/kernel/kernel > subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/rpaulo/ > lama1 rootopts nolockd > Adjusted interface rge1 > spin lock 0x804b8ea4 (sched lock) held by 0xc98c0260 (tid 100027) > too long > panic: spin lock held too long > cpuid = 12 > KDB: enter: panic > > > Regards, > -- > Rui Paulo > > ------------------------------ Randall Stewart 803-317-4952 (cell) 803-345-0391(direct) From owner-freebsd-mips@FreeBSD.ORG Sun Apr 18 17:47:46 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C7181065672; Sun, 18 Apr 2010 17:47:46 +0000 (UTC) (envelope-from rrs@lakerest.net) Received: from lakerest.net (unknown [IPv6:2001:240:585:2:213:d4ff:fef3:2d8d]) by mx1.freebsd.org (Postfix) with ESMTP id 414738FC12; Sun, 18 Apr 2010 17:47:45 +0000 (UTC) Received: from [192.168.2.178] (pool-173-67-91-86.snfcca.dsl-w.verizon.net [173.67.91.86]) (authenticated bits=0) by lakerest.net (8.14.3/8.14.3) with ESMTP id o3IHldBp041942 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sun, 18 Apr 2010 13:47:42 -0400 (EDT) (envelope-from rrs@lakerest.net) Message-Id: From: Randall Stewart To: Rui Paulo In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Sun, 18 Apr 2010 10:47:34 -0700 References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> X-Mailer: Apple Mail (2.936) Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Apr 2010 17:47:46 -0000 Rui: What type of load are you doing.. NFS or MFS fs? I am using a root NFS mounted and with all the patchs applied the machine no longer recognizes the DHCP responses from my daemon. I can clearly see these in wireshark... I will revert all patches out and see what happens without the rge-kx cleanup.. R On Apr 18, 2010, at 10:17 AM, Rui Paulo wrote: > On 18 Apr 2010, at 13:09, Rui Paulo wrote: > >> On 17 Apr 2010, at 23:40, C. Jayachandran wrote: >> >>> I've a set of initial patches to enable SMP for RMI processors. It >>> comes up in multi-user with 32 CPUs. I could do buildworld before I >>> updated to HEAD - with head there is a hang during buildworld which >>> I'm looking at, but I think the initial work can be checked in. >>> >>> Neel, can you have a look at the first two patches - one is to >>> enable >>> ULE scheduler and the second one is to move platform_init_ap to >>> slightly later in the initialization sequence. >>> >>> The patches are : >>> 1. mips-ule-support.patch >>> - Enable ULE scheduler for MIPS >>> >>> 2. mips-smp-move-platform.patch >>> - We need a hook to setup message ring and its interrupts, we use >>> platform_init_ap now, and move it be called later for XLR >>> >>> 3. rmi-prid.patch >>> Add RMI processor ID prints - right now it prints unknown processor. >>> >>> 4. rmi-pcib-fix.patch >>> XLR pci bridge should be off the IODI bus, currently it is off the >>> nexus bus which will cause issue when bus_setup_intr on nexus is >>> called >>> >>> 5. cleanup-reorg-cpuid-rge-kx.patch >>> This set has a many changes rolled up: >>> - clean up rge driver, remove unused code paths and commented code >>> (long way to go still) >>> - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage >>> updated >>> - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses >>> here, I have made changes for using XKPHYS to do the same process. >>> - the KX bit is enabled and interrupts are disabled before the >>> physical memory is read, to avoid the KX bit setting from affecting >>> other code. >>> - move message ring code to on_chip.c from xlr_machdep.c, now all >>> message ring code is in on_chip.c >>> >>> 6. rmi-xlr-smp.patch >>> SMP Support for XLR >>> - mpwait.S added to get the other CPUs out of bootloader code at >>> startup. >>> - SMP platform related functions. >>> - disable shared TLB code for SMP - since threads have different >>> mappings. >>> >>> The patches are also available at >>> http://sites.google.com/site/cjayachandran/files >>> >>> Enabling all 32 threads will need a minor fix in the SMP code, see >>> the >>> patch subr_smp-fix.patch at the same place. >> >> On the netperf XLR, this panics with SCHED_ULE activated: >> >> Copyright (c) 1992-2010 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >> 1994 >> The Regents of the University of California. All rights reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010 >> rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/ >> rpaulo/freebsd/head/sys/XLR mips >> real memory = 3488608256 (3406844K bytes) >> avail memory = 3395207168 (3237MB) >> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs >> nexus0: >> iodi0 on nexus0 >> uart0: <16550 or compatible> on iodi0 >> uart0: [FILTER] >> uart0: console (38551,n,8,1) >> pcib0: on iodi0 >> pci0: on pcib0 >> pci0: at device 1.0 (no driver attached) >> pci0: at device 3.0 (no driver attached) >> rge0 on iodi0 >> rge0: [ITHREAD] >> rge0: Ethernet address: >> rge0: [10Mbps] >> Total Active Core 8 >> rge1 on iodi0 >> rge1: [ITHREAD] >> rge1: Ethernet address: >> rge1: [1000Mbps] >> rge2 on iodi0 >> rge2: [ITHREAD] >> rge2: Ethernet address: >> rge2: [10Mbps] >> rge3 on iodi0 >> rge3: [ITHREAD] >> rge3: Ethernet address: >> rge3: [10Mbps] >> Enabling MDIO interrupts >> Timecounter "MIPS32" frequency 66000000 Hz quality 800 >> Timecounters tick every 1.000 msec >> bootpc_init: wired to interface 'rge1' >> Sending DHCP Discover packet from interface rge1 >> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no >> root path) >> Sending DHCP Request packet from interface rge1 >> Sending DHCP Discover packet from interface rge1 >> DHCP/BOOTP timeout for server 255.255.255.255 >> DHCP/BOOTP timeout for server 255.255.255.255 >> DHCP/BOOTP timeout for server 255.255.255.255 >> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no >> root path) >> Sending DHCP Request packet from interface rge1 >> DHCP/BOOTP timeout for server 255.255.255.255 >> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got >> root path) >> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org >> boot file /zoo/rpaulo/lama1/boot/kernel/kernel >> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/ >> rpaulo/lama1 rootopts nolockd >> Adjusted interface rge1 >> panic: mutex sched lock 4 not owned at /usr/home/rpaulo/freebsd/ >> head/sys/kern/sched_ule.c:2054 >> cpuid = 4 >> KDB: enter: panic > > If I use SCHED_4BSD this happens: (am I missing something?) > > Copyright (c) 1992-2010 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, > 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 9.0-CURRENT #4 r206791M: Sun Apr 18 17:10:55 UTC 2010 > rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/ > rpaulo/freebsd/head/sys/XLR mips > real memory = 3488608256 (3406844K bytes) > avail memory = 3395252224 (3237MB) > FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs > nexus0: > iodi0 on nexus0 > uart0: <16550 or compatible> on iodi0 > uart0: [FILTER] > uart0: console (38551,n,8,1) > pcib0: on iodi0 > pci0: on pcib0 > pci0: at device 1.0 (no driver attached) > pci0: at device 3.0 (no driver attached) > rge0 on iodi0 > rge0: [ITHREAD] > rge0: Ethernet address: 00:0f:30:00:20:4e > rge0: [10Mbps] > Total Active Core 8 > rge1 on iodi0 > rge1: [ITHREAD] > rge1: Ethernet address: 00:0f:30:00:20:4f > rge1: [1000Mbps] > rge2 on iodi0 > rge2: [ITHREAD] > rge2: Ethernet address: 00:0f:30:00:20:50 > rge2: [10Mbps] > rge3 on iodi0 > rge3: [ITHREAD] > rge3: Ethernet address: 00:0f:30:00:20:51 > rge3: [10Mbps] > Enabling MDIO interrupts > Timecounter "MIPS32" frequency 66000000 Hz quality 800 > Timecounters tick every 1.000 msec > bootpc_init: wired to interface 'rge1' > Sending DHCP Discover packet from interface rge1 (00:0f:30:00:20:4f) > Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no > root path) > Sending DHCP Request packet from interface rge1 (00:0f:30:00:20:4f) > Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got > root path) > rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org > boot file /zoo/rpaulo/lama1/boot/kernel/kernel > subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/rpaulo/ > lama1 rootopts nolockd > Adjusted interface rge1 > spin lock 0x804b8ea4 (sched lock) held by 0xc98c0260 (tid 100027) > too long > panic: spin lock held too long > cpuid = 12 > KDB: enter: panic > > > Regards, > -- > Rui Paulo > > ------------------------------ Randall Stewart 803-317-4952 (cell) 803-345-0391(direct) From owner-freebsd-mips@FreeBSD.ORG Sun Apr 18 18:22:53 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 28E14106564A for ; Sun, 18 Apr 2010 18:22:53 +0000 (UTC) (envelope-from rpaulo@freebsd.org) Received: from karen.lavabit.com (karen.lavabit.com [72.249.41.33]) by mx1.freebsd.org (Postfix) with ESMTP id 071F08FC0A for ; Sun, 18 Apr 2010 18:22:52 +0000 (UTC) Received: from e.earth.lavabit.com (e.earth.lavabit.com [192.168.111.14]) by karen.lavabit.com (Postfix) with ESMTP id 4F11215754E; Sun, 18 Apr 2010 13:22:52 -0500 (CDT) Received: from 10.0.10.2 (54.81.54.77.rev.vodafone.pt [77.54.81.54]) by lavabit.com with ESMTP id OTTV59M2Y56D; Sun, 18 Apr 2010 13:22:52 -0500 Mime-Version: 1.0 (Apple Message framework v1078) Content-Type: text/plain; charset=us-ascii From: Rui Paulo In-Reply-To: Date: Sun, 18 Apr 2010 19:22:49 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <3BCD65EB-B997-449D-864C-CA24C7B19026@freebsd.org> References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> To: Randall Stewart X-Mailer: Apple Mail (2.1078) Cc: freebsd-mips@freebsd.org Subject: Re: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Apr 2010 18:22:53 -0000 On 18 Apr 2010, at 18:47, Randall Stewart wrote: > Rui: >=20 > What type of load are you doing.. NFS or MFS fs? >=20 > I am using a root NFS mounted and with all the patchs applied > the machine no longer recognizes the DHCP responses from my daemon. >=20 > I can clearly see these in wireshark... I will revert all patches > out and see what happens without the rge-kx cleanup.. I'm using NFS. It takes a bit more time to recognize the DHCP offer than = before, not sure why. Regards, -- Rui Paulo From owner-freebsd-mips@FreeBSD.ORG Sun Apr 18 18:26:25 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 689F31065679; Sun, 18 Apr 2010 18:26:25 +0000 (UTC) (envelope-from rrs@lakerest.net) Received: from lakerest.net (unknown [IPv6:2001:240:585:2:213:d4ff:fef3:2d8d]) by mx1.freebsd.org (Postfix) with ESMTP id E802E8FC0A; Sun, 18 Apr 2010 18:26:24 +0000 (UTC) Received: from [192.168.2.178] (pool-173-67-91-86.snfcca.dsl-w.verizon.net [173.67.91.86]) (authenticated bits=0) by lakerest.net (8.14.3/8.14.3) with ESMTP id o3IIQKjd043522 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sun, 18 Apr 2010 14:26:23 -0400 (EDT) (envelope-from rrs@lakerest.net) Message-Id: From: Randall Stewart To: Rui Paulo In-Reply-To: <3BCD65EB-B997-449D-864C-CA24C7B19026@freebsd.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Sun, 18 Apr 2010 11:26:14 -0700 References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> <3BCD65EB-B997-449D-864C-CA24C7B19026@freebsd.org> X-Mailer: Apple Mail (2.936) Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Apr 2010 18:26:25 -0000 I am backing out the rge part of the patch and things get normal again.. Still working on getting one that will work in SMP though.. On Apr 18, 2010, at 11:22 AM, Rui Paulo wrote: > On 18 Apr 2010, at 18:47, Randall Stewart wrote: > >> Rui: >> >> What type of load are you doing.. NFS or MFS fs? >> >> I am using a root NFS mounted and with all the patchs applied >> the machine no longer recognizes the DHCP responses from my daemon. >> >> I can clearly see these in wireshark... I will revert all patches >> out and see what happens without the rge-kx cleanup.. > > I'm using NFS. It takes a bit more time to recognize the DHCP offer > than before, not sure why. > > Regards, > -- > Rui Paulo > > ------------------------------ Randall Stewart 803-317-4952 (cell) 803-345-0391(direct) From owner-freebsd-mips@FreeBSD.ORG Sun Apr 18 18:30:42 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3A1DF106566B; Sun, 18 Apr 2010 18:30:42 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-pz0-f201.google.com (mail-pz0-f201.google.com [209.85.222.201]) by mx1.freebsd.org (Postfix) with ESMTP id 051E48FC0C; Sun, 18 Apr 2010 18:30:41 +0000 (UTC) Received: by pzk39 with SMTP id 39so2857337pzk.7 for ; Sun, 18 Apr 2010 11:30:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=h2vums1kLbzic7YP9GPDfSN1UPHiY/dUcruBf9XvJ3w=; b=DyovbqqByYe/t/+feYbfrdWLKP8LCPJXBJxubBZywKB4bSu5eS75xMQvGU1nmLqgVr uR1PuUTxRm7hViBhErdz8JffKRoJ87xlOd30VS7JQLEua+kC31iNz6kiPBafXs+4P9ES 02FiFE2oqJ1POZMATahT9xLLjvdIZ4ITeh6EI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=TFfAdmhpH6DWOd28Z2uQYYGFxJlwMp8YCuEfRJH9ntzpED/l8upgx4azm3SOWG2bjf kBTB8gqdN9zzB6V/DRvBINWVxPaRvNeDXWp8lJiXgATCAcptH+alkOeeUB0U+w7Y1tJU JOwBK3Naw4sCTsNxCrha8pIRhAefWUNQM0Yyg= MIME-Version: 1.0 Received: by 10.141.29.15 with HTTP; Sun, 18 Apr 2010 11:30:41 -0700 (PDT) In-Reply-To: References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> Date: Mon, 19 Apr 2010 00:00:41 +0530 Received: by 10.140.247.20 with SMTP id u20mr1032940rvh.122.1271615441350; Sun, 18 Apr 2010 11:30:41 -0700 (PDT) Message-ID: From: "C. Jayachandran" To: Randall Stewart Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Apr 2010 18:30:42 -0000 I did not have invariants enabled when I moved to HEAD, and I got to multiuser login prompt. The panic is easily seen with INVARIANTS enabled with all 32 CPUs. But it does not happen if I cut down the cpus to 16 [ userapp_mask_cpus 0xffff on boot prompt]. Looks like there is some race I'm hitting with more CPUs. Will look at this... JC. On Sun, Apr 18, 2010 at 11:07 PM, Randall Stewart wrote: > OK, > > it appears in my early morning fog as I started I missed a patch ;-) > Anyway... I know better than to work before my first cup of coffee ... > > I am rebuilding and will try again with all the patches ;-) > > I suspect I will see your panic.. my guess is the lockup JC saw > was a LOR and he ran with NO WITNESS/INVARIANT in his build. > > We should be able to fix this pretty easily ;-) > > Give me a bit of time... > > R > > On Apr 18, 2010, at 10:17 AM, Rui Paulo wrote: > >> On 18 Apr 2010, at 13:09, Rui Paulo wrote: >> >>> On 17 Apr 2010, at 23:40, C. Jayachandran wrote: >>> >>>> I've a set of initial patches to enable SMP for RMI processors. It >>>> comes up in multi-user with 32 CPUs. =A0I could do buildworld before I >>>> updated to HEAD - with head there is a hang during buildworld which >>>> I'm looking at, but I think the initial work can be checked in. >>>> >>>> Neel, can you have a look at the first two patches - one is to enable >>>> ULE scheduler and the second one is to move platform_init_ap to >>>> slightly later in the initialization sequence. >>>> >>>> The patches are : >>>> 1. mips-ule-support.patch >>>> - Enable ULE scheduler for MIPS >>>> >>>> 2. mips-smp-move-platform.patch >>>> - We need a hook to setup message ring and its interrupts, we use >>>> platform_init_ap now, and move it be called later for XLR >>>> >>>> 3. rmi-prid.patch >>>> Add RMI processor ID prints - right now it prints unknown processor. >>>> >>>> 4. rmi-pcib-fix.patch >>>> XLR pci bridge should be off the IODI bus, currently it is off the >>>> nexus bus which will cause issue when bus_setup_intr on nexus is >>>> called >>>> >>>> 5. cleanup-reorg-cpuid-rge-kx.patch >>>> This set has a many changes rolled up: >>>> - clean up rge driver, remove unused code paths and commented code >>>> (long way to go still) >>>> - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage updat= ed >>>> - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses >>>> here, I have made changes for using XKPHYS to do the same process. >>>> - the KX bit is enabled and interrupts are disabled before the >>>> physical memory is read, to avoid the KX bit setting from affecting >>>> other code. >>>> - move message ring code to on_chip.c from xlr_machdep.c, now all >>>> message ring code is in on_chip.c >>>> >>>> 6. rmi-xlr-smp.patch >>>> SMP Support for XLR >>>> - mpwait.S added to get the other CPUs out of bootloader code at >>>> startup. >>>> - SMP platform related functions. >>>> - disable shared TLB code for SMP - since threads have different >>>> mappings. >>>> >>>> The patches are also available at >>>> http://sites.google.com/site/cjayachandran/files >>>> >>>> Enabling all 32 threads will need a minor fix in the SMP code, see the >>>> patch subr_smp-fix.patch at the same place. >>> >>> On the netperf XLR, this panics with SCHED_ULE activated: >>> >>> Copyright (c) 1992-2010 The FreeBSD Project. >>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 199= 4 >>> =A0 =A0 =A0 =A0The Regents of the University of California. All rights = reserved. >>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>> FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010 >>> >>> =A0rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpa= ulo/freebsd/head/sys/XLR >>> mips >>> real memory =A0=3D 3488608256 (3406844K bytes) >>> avail memory =3D 3395207168 (3237MB) >>> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs >>> nexus0: >>> iodi0 on nexus0 >>> uart0: <16550 or compatible> on iodi0 >>> uart0: [FILTER] >>> uart0: console (38551,n,8,1) >>> pcib0: on iodi0 >>> pci0: on pcib0 >>> pci0: at device 1.0 (no driver attached) >>> pci0: at device 3.0 (no driver attached) >>> rge0 on iodi0 >>> rge0: [ITHREAD] >>> rge0: Ethernet address: >>> rge0: [10Mbps] >>> Total Active Core 8 >>> rge1 on iodi0 >>> rge1: [ITHREAD] >>> rge1: Ethernet address: >>> rge1: [1000Mbps] >>> rge2 on iodi0 >>> rge2: [ITHREAD] >>> rge2: Ethernet address: >>> rge2: [10Mbps] >>> rge3 on iodi0 >>> rge3: [ITHREAD] >>> rge3: Ethernet address: >>> rge3: [10Mbps] >>> Enabling MDIO interrupts >>> Timecounter "MIPS32" frequency 66000000 Hz quality 800 >>> Timecounters tick every 1.000 msec >>> bootpc_init: wired to interface 'rge1' >>> Sending DHCP Discover packet from interface rge1 >>> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root >>> path) >>> Sending DHCP Request packet from interface rge1 >>> Sending DHCP Discover packet from interface rge1 >>> DHCP/BOOTP timeout for server 255.255.255.255 >>> DHCP/BOOTP timeout for server 255.255.255.255 >>> DHCP/BOOTP timeout for server 255.255.255.255 >>> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root >>> path) >>> Sending DHCP Request packet from interface rge1 >>> DHCP/BOOTP timeout for server 255.255.255.255 >>> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root >>> path) >>> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org boo= t >>> file /zoo/rpaulo/lama1/boot/kernel/kernel >>> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs >>> /zoo/rpaulo/lama1 rootopts nolockd >>> Adjusted interface rge1 >>> panic: mutex sched lock 4 not owned at >>> /usr/home/rpaulo/freebsd/head/sys/kern/sched_ule.c:2054 >>> cpuid =3D 4 >>> KDB: enter: panic >> >> If I use SCHED_4BSD this happens: (am I missing something?) >> >> Copyright (c) 1992-2010 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >> =A0 =A0 =A0 =A0The Regents of the University of California. All rights r= eserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 9.0-CURRENT #4 r206791M: Sun Apr 18 17:10:55 UTC 2010 >> >> rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpaulo/= freebsd/head/sys/XLR >> mips >> real memory =A0=3D 3488608256 (3406844K bytes) >> avail memory =3D 3395252224 (3237MB) >> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs >> nexus0: >> iodi0 on nexus0 >> uart0: <16550 or compatible> on iodi0 >> uart0: [FILTER] >> uart0: console (38551,n,8,1) >> pcib0: on iodi0 >> pci0: on pcib0 >> pci0: at device 1.0 (no driver attached) >> pci0: at device 3.0 (no driver attached) >> rge0 on iodi0 >> rge0: [ITHREAD] >> rge0: Ethernet address: 00:0f:30:00:20:4e >> rge0: [10Mbps] >> Total Active Core 8 >> rge1 on iodi0 >> rge1: [ITHREAD] >> rge1: Ethernet address: 00:0f:30:00:20:4f >> rge1: [1000Mbps] >> rge2 on iodi0 >> rge2: [ITHREAD] >> rge2: Ethernet address: 00:0f:30:00:20:50 >> rge2: [10Mbps] >> rge3 on iodi0 >> rge3: [ITHREAD] >> rge3: Ethernet address: 00:0f:30:00:20:51 >> rge3: [10Mbps] >> Enabling MDIO interrupts >> Timecounter "MIPS32" frequency 66000000 Hz quality 800 >> Timecounters tick every 1.000 msec >> bootpc_init: wired to interface 'rge1' >> Sending DHCP Discover packet from interface rge1 (00:0f:30:00:20:4f) >> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root >> path) >> Sending DHCP Request packet from interface rge1 (00:0f:30:00:20:4f) >> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root >> path) >> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org boot >> file /zoo/rpaulo/lama1/boot/kernel/kernel >> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/rpaulo/lam= a1 >> rootopts nolockd >> Adjusted interface rge1 >> spin lock 0x804b8ea4 (sched lock) held by 0xc98c0260 (tid 100027) too lo= ng >> panic: spin lock held too long >> cpuid =3D 12 >> KDB: enter: panic >> >> >> Regards, >> -- >> Rui Paulo >> >> > > ------------------------------ > Randall Stewart > 803-317-4952 (cell) > 803-345-0391(direct) > > --=20 C. Jayachandran c.jayachandran@gmail.com From owner-freebsd-mips@FreeBSD.ORG Mon Apr 19 11:11:41 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3427A1065674; Mon, 19 Apr 2010 11:11:41 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-pz0-f175.google.com (mail-pz0-f175.google.com [209.85.222.175]) by mx1.freebsd.org (Postfix) with ESMTP id EC9B38FC0A; Mon, 19 Apr 2010 11:11:40 +0000 (UTC) Received: by pzk5 with SMTP id 5so2913818pzk.14 for ; Mon, 19 Apr 2010 04:11:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type; bh=hht/NKIqOLRs8WxnJu2+pRcxrhNE9KFY7f0gwC5xkO4=; b=jzfuA6gBTKptWlVQZmKIerZMQhv53PdPkM61c0nO/1xZTCP+Ft49VkCUeZ3ZIvOtHX nR99R6itmgO1s1ss0dNAh8eV78bl4fC6taSb9BSGCJ6lCompqkxE16jihu+uA4U98Dd6 5WatwdcsBBZ368DYlbjxrDZXf8Ta+ettW65iU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=iOz7rdv0/3XndvDzLCSn6M+tghSA19NKe9MC7H6d+iTefXeIjEapnVkvyjaKe7FgVZ xUPDA71Rx4g7GWz8IKfQ1qU8XJo8UfL17MY0iok4iBZz99zeM1Ig11ho6mDhZ1+1aEGz UidO/fYsDRB1DYuQq2DFL2XeWfTPZ5fH9Qv/E= MIME-Version: 1.0 Received: by 10.141.29.15 with HTTP; Mon, 19 Apr 2010 04:11:40 -0700 (PDT) In-Reply-To: References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> <3BCD65EB-B997-449D-864C-CA24C7B19026@freebsd.org> Date: Mon, 19 Apr 2010 16:41:40 +0530 Received: by 10.140.248.13 with SMTP id v13mr3929284rvh.25.1271675500306; Mon, 19 Apr 2010 04:11:40 -0700 (PDT) Message-ID: From: "C. Jayachandran" To: Randall Stewart Content-Type: multipart/mixed; boundary=000e0cd0ea8677238004849506a4 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Apr 2010 11:11:41 -0000 --000e0cd0ea8677238004849506a4 Content-Type: text/plain; charset=ISO-8859-1 On Sun, Apr 18, 2010 at 11:56 PM, Randall Stewart wrote: > I am backing out the rge part of the patch and things > get normal again.. > > Still working on getting one that will work in SMP though.. Can you try the attached patch for this - this will defer the packet distribution to different cores until SMP is started. This should apply on top of the ones I send earlier. Patch also at http://sites.google.com/site/cjayachandran/files Regards, JC. --000e0cd0ea8677238004849506a4-- From owner-freebsd-mips@FreeBSD.ORG Mon Apr 19 13:13:31 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF2861065672; Mon, 19 Apr 2010 13:13:31 +0000 (UTC) (envelope-from rrs@lakerest.net) Received: from lakerest.net (unknown [IPv6:2001:240:585:2:213:d4ff:fef3:2d8d]) by mx1.freebsd.org (Postfix) with ESMTP id 6CFE18FC08; Mon, 19 Apr 2010 13:13:31 +0000 (UTC) Received: from [192.168.2.178] (pool-173-67-91-86.snfcca.dsl-w.verizon.net [173.67.91.86]) (authenticated bits=0) by lakerest.net (8.14.3/8.14.3) with ESMTP id o3JDDNT8089700 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Mon, 19 Apr 2010 09:13:29 -0400 (EDT) (envelope-from rrs@lakerest.net) Message-Id: <6BDB3874-D779-45A6-ABAE-4C331D78A189@lakerest.net> From: Randall Stewart To: "C. Jayachandran" In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Mon, 19 Apr 2010 06:13:15 -0700 References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> <3BCD65EB-B997-449D-864C-CA24C7B19026@freebsd.org> X-Mailer: Apple Mail (2.936) Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Apr 2010 13:13:32 -0000 JC I will give it a whirl.. There are other issues though... I tried with only 16 processors and a LOT of erratic things occur. I will apply your patch and then I am going to start working on being able to bring up only 1 thread on each core... I think some of the problems I am seeing are coming from threads getting a bit cross with each other.. but I can't prove that unless I can set the mask up 0x11111111 and right now when I try that we panic :-) R On Apr 19, 2010, at 4:11 AM, C. Jayachandran wrote: > On Sun, Apr 18, 2010 at 11:56 PM, Randall Stewart > wrote: >> I am backing out the rge part of the patch and things >> get normal again.. >> >> Still working on getting one that will work in SMP though.. > > Can you try the attached patch for this - this will defer the packet > distribution to different cores until SMP is started. This should > apply on top of the ones I send earlier. > > Patch also at http://sites.google.com/site/cjayachandran/files > > Regards, > JC. > ------------------------------ Randall Stewart 803-317-4952 (cell) 803-345-0391(direct) From owner-freebsd-mips@FreeBSD.ORG Mon Apr 19 13:57:56 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B7A11065670; Mon, 19 Apr 2010 13:57:56 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 249048FC1D; Mon, 19 Apr 2010 13:57:55 +0000 (UTC) Received: by pwi9 with SMTP id 9so3449190pwi.13 for ; Mon, 19 Apr 2010 06:57:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type; bh=X0+Nktfte2nGnutTBr1AJ2Y9uF945Q8ycUrzKJa3+yg=; b=YRWsiYJnsWmbPPV5wxoqs7ODVg+6ncLQ8dfX/KBWlUGOpM5/BdJQt2Rhe54NWjEn2y AuDZYfBUB/CODMrtB2WuDhj2DTKjECqHbdzP0Xljt7qr5TILSBJRXcDRL0/tEwPPbqD/ hLMsJ/Iku3FX+rLqELQmTPFodl7OLzQFbZr6U= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=hOSZkItKmq3rdvVa19xNhpGldyczQjrjLwKvhHq/va4dVRSw3kVOwQeyMvdxazh60u /dYmD5j+ytiFgyek/IqFLeun7ijP96wtX8Pw/bsE/6UasR6HoNtMumbW4v+Wcq1RqekS FHwk8rj3t5eLs4G4jsJKbGBxdoerrYvttJ3nQ= MIME-Version: 1.0 Received: by 10.141.29.15 with HTTP; Mon, 19 Apr 2010 06:57:54 -0700 (PDT) In-Reply-To: <6BDB3874-D779-45A6-ABAE-4C331D78A189@lakerest.net> References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> <3BCD65EB-B997-449D-864C-CA24C7B19026@freebsd.org> <6BDB3874-D779-45A6-ABAE-4C331D78A189@lakerest.net> Date: Mon, 19 Apr 2010 19:27:54 +0530 Received: by 10.140.56.1 with SMTP id e1mr3987472rva.136.1271685474794; Mon, 19 Apr 2010 06:57:54 -0700 (PDT) Message-ID: From: "C. Jayachandran" To: Randall Stewart Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Apr 2010 13:57:56 -0000 On Mon, Apr 19, 2010 at 6:43 PM, Randall Stewart wrote: > JC I will give it a whirl.. > > There are other issues though... > > I tried with only 16 processors and a LOT of erratic things occur. > > I will apply your patch and then I am going to start > working on being able to bring up only 1 thread on each > core... > > I think some of the problems I am seeing are coming from threads > getting a bit cross with each other.. but I can't prove that > unless I can set the mask up > > 0x11111111 > > and right now when I try that we panic :-) I have a possible cause for the panic with invariants - we should not schedule the msgring threads unless the smp is completely up. I guess we start getting message ring interrupts on before the message ring threads can be scheduled. I am trying out some changes for this - will send you a patch if this fixes it. Thanks, JC. From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 09:52:55 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 86AC8106564A; Tue, 20 Apr 2010 09:52:55 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-pv0-f182.google.com (mail-pv0-f182.google.com [74.125.83.182]) by mx1.freebsd.org (Postfix) with ESMTP id 48E4D8FC08; Tue, 20 Apr 2010 09:52:55 +0000 (UTC) Received: by pvc7 with SMTP id 7so3877754pvc.13 for ; Tue, 20 Apr 2010 02:52:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type; bh=UVmI6xVbJVT/JygC9m3Pi5DDS7em7rL1f7KKyUVbFQ4=; b=W50mvN7vcZzajs/GceYCt+rdTHpEX6ZX02A6vra3C7+ZdtXuBWGIEJ2UmH+qgch+CR QdP07m1hsLozup58HuXQl8nmRsT1t9eaDojfwr+np+XC9i3yabZsVG6+XRHAOwNNATwB Y4sLPz6DNhrercT9mEjxft7FaZO67+B3SjfEA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=SCXmvUlN/1DcX8vSwDd6IOKCO4+4nN8MLgu5kiLdbX4aPP5dov1hDuIFgKAadOU/4s hiUlP7t3XtIB4QiVQtDSr309wrU1zq0J8RQGrsbd7m9AO18WhM2r7fgTggnFtCYLVSwg olR4YdvaNiJ9jQ2Ef3HipP5Lss5l5szJma9yw= MIME-Version: 1.0 Received: by 10.141.29.15 with HTTP; Tue, 20 Apr 2010 02:52:52 -0700 (PDT) In-Reply-To: References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> <3BCD65EB-B997-449D-864C-CA24C7B19026@freebsd.org> <6BDB3874-D779-45A6-ABAE-4C331D78A189@lakerest.net> Date: Tue, 20 Apr 2010 15:22:52 +0530 Received: by 10.141.4.10 with SMTP id g10mr5369144rvi.179.1271757173151; Tue, 20 Apr 2010 02:52:53 -0700 (PDT) Message-ID: From: "C. Jayachandran" To: Randall Stewart Content-Type: multipart/mixed; boundary=000e0cd0ebd08be60c0484a80a7b X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 09:52:55 -0000 --000e0cd0ebd08be60c0484a80a7b Content-Type: text/plain; charset=ISO-8859-1 On Mon, Apr 19, 2010 at 7:27 PM, C. Jayachandran wrote: > I have a possible cause for the panic with invariants - we should not > schedule the msgring threads unless the smp is completely up. I guess > we start getting message ring interrupts on before the message ring > threads can be scheduled. I am trying out some changes for this - > will send you a patch if this fixes it. I've attached a patch that should fix the issue. The cause was the way message ring threads are started on individual cores and the way interrupts are enabled in the core. I've moved starting message ring threads on other cpus to be a SYSINIT after SMP is started. I'd thought originally that it was due to some clash with the changes in HEAD - but looks like I was completely off-track there. Please let me know if you don't get multi-user with 32 cpus with this patch. There is still the original hang in buildworld, but that should be a bug elsewhere I have a copy at http://sites.google.com/site/cjayachandran/files too Thanks, JC. --000e0cd0ebd08be60c0484a80a7b-- From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 10:05:05 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63581106566B for ; Tue, 20 Apr 2010 10:05:05 +0000 (UTC) (envelope-from rpaulo@freebsd.org) Received: from karen.lavabit.com (karen.lavabit.com [72.249.41.33]) by mx1.freebsd.org (Postfix) with ESMTP id 3C0A38FC08 for ; Tue, 20 Apr 2010 10:05:05 +0000 (UTC) Received: from e.earth.lavabit.com (e.earth.lavabit.com [192.168.111.14]) by karen.lavabit.com (Postfix) with ESMTP id A823124EDEA; Tue, 20 Apr 2010 05:05:04 -0500 (CDT) Received: from 10.0.10.3 (54.81.54.77.rev.vodafone.pt [77.54.81.54]) by lavabit.com with ESMTP id 0O1SEQ2FBBEQ; Tue, 20 Apr 2010 05:05:04 -0500 Mime-Version: 1.0 (Apple Message framework v1078) Content-Type: text/plain; charset=us-ascii From: Rui Paulo In-Reply-To: Date: Tue, 20 Apr 2010 11:05:01 +0100 Content-Transfer-Encoding: 7bit Message-Id: <7BEFA3F5-97AE-477C-9DD3-EF1C4B7DCEB0@freebsd.org> References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> <3BCD65EB-B997-449D-864C-CA24C7B19026@freebsd.org> <6BDB3874-D779-45A6-ABAE-4C331D78A189@lakerest.net> To: "C. Jayachandran" X-Mailer: Apple Mail (2.1078) Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 10:05:05 -0000 On 20 Apr 2010, at 10:52, C. Jayachandran wrote: > On Mon, Apr 19, 2010 at 7:27 PM, C. Jayachandran > wrote: >> I have a possible cause for the panic with invariants - we should not >> schedule the msgring threads unless the smp is completely up. I guess >> we start getting message ring interrupts on before the message ring >> threads can be scheduled. I am trying out some changes for this - >> will send you a patch if this fixes it. > > I've attached a patch that should fix the issue. The cause was the way > message ring threads are started on individual cores and the way > interrupts are enabled in the core. I've moved starting message ring > threads on other cpus to be a SYSINIT after SMP is started. I'd > thought originally that it was due to some clash with the changes in > HEAD - but looks like I was completely off-track there. > > Please let me know if you don't get multi-user with 32 cpus with this > patch. There is still the original hang in buildworld, but that should > be a bug elsewhere > > I have a copy at http://sites.google.com/site/cjayachandran/files too This works perfectly, thanks! Regards, -- Rui Paulo From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 10:33:53 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2B0271065673 for ; Tue, 20 Apr 2010 10:33:53 +0000 (UTC) (envelope-from rpaulo@freebsd.org) Received: from karen.lavabit.com (karen.lavabit.com [72.249.41.33]) by mx1.freebsd.org (Postfix) with ESMTP id EB8368FC13 for ; Tue, 20 Apr 2010 10:33:52 +0000 (UTC) Received: from e.earth.lavabit.com (e.earth.lavabit.com [192.168.111.14]) by karen.lavabit.com (Postfix) with ESMTP id E8A6915755D; Tue, 20 Apr 2010 05:33:51 -0500 (CDT) Received: from 10.0.10.3 (54.81.54.77.rev.vodafone.pt [77.54.81.54]) by lavabit.com with ESMTP id 5HIF9WPFG4KG; Tue, 20 Apr 2010 05:33:51 -0500 Mime-Version: 1.0 (Apple Message framework v1078) Content-Type: text/plain; charset=us-ascii From: Rui Paulo In-Reply-To: <7BEFA3F5-97AE-477C-9DD3-EF1C4B7DCEB0@freebsd.org> Date: Tue, 20 Apr 2010 11:33:48 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> <3BCD65EB-B997-449D-864C-CA24C7B19026@freebsd.org> <6BDB3874-D779-45A6-ABAE-4C331D78A189@lakerest.net> <7BEFA3F5-97AE-477C-9DD3-EF1C4B7DCEB0@freebsd.org> To: "C. Jayachandran" X-Mailer: Apple Mail (2.1078) Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 10:33:53 -0000 On 20 Apr 2010, at 11:05, Rui Paulo wrote: > On 20 Apr 2010, at 10:52, C. Jayachandran wrote: >=20 >> On Mon, Apr 19, 2010 at 7:27 PM, C. Jayachandran >> wrote: >>> I have a possible cause for the panic with invariants - we should = not >>> schedule the msgring threads unless the smp is completely up. I = guess >>> we start getting message ring interrupts on before the message ring >>> threads can be scheduled. I am trying out some changes for this - >>> will send you a patch if this fixes it. >>=20 >> I've attached a patch that should fix the issue. The cause was the = way >> message ring threads are started on individual cores and the way >> interrupts are enabled in the core. I've moved starting message ring >> threads on other cpus to be a SYSINIT after SMP is started. I'd >> thought originally that it was due to some clash with the changes in >> HEAD - but looks like I was completely off-track there. >>=20 >> Please let me know if you don't get multi-user with 32 cpus with this >> patch. There is still the original hang in buildworld, but that = should >> be a bug elsewhere >>=20 >> I have a copy at http://sites.google.com/site/cjayachandran/files too >=20 > This works perfectly, thanks! On further inspection, I noticed that the load avg is now 7. last pid: 1613; load averages: 6.99, 6.97, 6.08 up 0+00:30:11 = 10:32:48 108 processes: 40 running, 24 sleeping, 44 waiting CPU: 0.0% user, 0.0% nice, 21.9% system, 0.0% interrupt, 78.1% idle Mem: 8444K Active, 6028K Inact, 37M Wired, 308K Cache, 6800K Buf, 3190M = Free Swap:=20 PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU = COMMAND 10 root 32 171 ki31 0G 0G CPU0 0 263:26 2500.00% = idle 17 root 1 -16 - 0K 0G CPU12 2 0:00 100.00% = msg_intr12 15 root 1 -16 - 0K 0G CPU4 2 0:00 100.00% = msg_intr4 16 root 1 -16 - 0K 0G CPU8 2 0:00 100.00% = msg_intr8 20 root 1 -16 - 0K 0G CPU24 1 0:00 100.00% = msg_intr24 19 root 1 -16 - 0K 0G CPU20 1 0:00 100.00% = msg_intr20 21 root 1 -16 - 0K 0G CPU28 1 0:00 100.00% = msg_intr28 18 root 1 -16 - 0K 0G CPU16 1 0:00 100.00% = msg_intr16 What are these msg_intrXX kprocs doing? Regards, -- Rui Paulo From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 10:49:43 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC4C01065672; Tue, 20 Apr 2010 10:49:43 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-pv0-f182.google.com (mail-pv0-f182.google.com [74.125.83.182]) by mx1.freebsd.org (Postfix) with ESMTP id 716F08FC20; Tue, 20 Apr 2010 10:49:43 +0000 (UTC) Received: by pvc7 with SMTP id 7so3896918pvc.13 for ; Tue, 20 Apr 2010 03:49:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=eJm9g/bEd5lWIogG415DHYk7XQPWF+6o1yvEigmnU1I=; b=RX+JEacD4I1WvOw+pFyhux+3AUC1KmciGgKhfWDGLjV+/J7EPJR3bv86eCJDN+3W91 ItGoUrjjlSZLLbZPs2bYtYSc6lT7gfIh5pELOziWw2ZvozJyGucWlvDrTDJKtMtKxfsh 1gU2y0jSubMd+IeteOts7/KXYtc767vBE0vMw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=iw18uzZy5SGpo/Cd0PY5eqFnCKPjmbGRk+eCmN0d3P00bW7aPf49NXYFAvpLkblRUJ ID0Y3fWF3zvEaAYvyb3DnhDkKY3/D+dYYpcCA//HCzWnP2ir/DuSbq/8xzTaAFW4XSQN XcLRtQre90bS7G1RtV3i1CLq7vd+AGdcfx1pw= MIME-Version: 1.0 Received: by 10.141.29.15 with HTTP; Tue, 20 Apr 2010 03:49:42 -0700 (PDT) In-Reply-To: References: <3BCD65EB-B997-449D-864C-CA24C7B19026@freebsd.org> <6BDB3874-D779-45A6-ABAE-4C331D78A189@lakerest.net> <7BEFA3F5-97AE-477C-9DD3-EF1C4B7DCEB0@freebsd.org> Date: Tue, 20 Apr 2010 16:19:42 +0530 Received: by 10.140.251.8 with SMTP id y8mr297638rvh.231.1271760582896; Tue, 20 Apr 2010 03:49:42 -0700 (PDT) Message-ID: From: "C. Jayachandran" To: Rui Paulo Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 10:49:43 -0000 On Tue, Apr 20, 2010 at 4:03 PM, Rui Paulo wrote: > On 20 Apr 2010, at 11:05, Rui Paulo wrote: > >> On 20 Apr 2010, at 10:52, C. Jayachandran wrote: >> >>> On Mon, Apr 19, 2010 at 7:27 PM, C. Jayachandran >>> wrote: >>>> I have a possible cause for the panic with invariants - we should not >>>> schedule the msgring threads unless the smp is completely up. I guess >>>> we start getting message ring interrupts on before the message ring >>>> threads can be scheduled. =A0I am trying out some changes for this - >>>> will send you a patch if this fixes it. >>> >>> I've attached a patch that should fix the issue. The cause was the way >>> message ring threads are started on individual cores and the way >>> interrupts are enabled in the core. =A0I've moved starting message ring >>> threads on other cpus to be a SYSINIT after SMP is started. =A0I'd >>> thought originally that it was due to some clash with the changes in >>> HEAD - but looks like I was completely off-track there. >>> >>> Please let me know if you don't get multi-user with 32 cpus with this >>> patch. There is still the original hang in buildworld, but that should >>> be a bug elsewhere >>> >>> I have a copy at http://sites.google.com/site/cjayachandran/files too >> >> This works perfectly, thanks! > > On further inspection, I noticed that the load avg is now 7. > > last pid: =A01613; =A0load averages: =A06.99, =A06.97, =A06.08 =A0 =A0up = 0+00:30:11 =A010:32:48 > 108 processes: 40 running, 24 sleeping, 44 waiting > CPU: =A00.0% user, =A00.0% nice, 21.9% system, =A00.0% interrupt, 78.1% i= dle > Mem: 8444K Active, 6028K Inact, 37M Wired, 308K Cache, 6800K Buf, 3190M F= ree > Swap: > > =A0PID USERNAME =A0THR PRI NICE =A0 SIZE =A0 =A0RES STATE =A0 C =A0 TIME = =A0 WCPU COMMAND > =A0 10 root =A0 =A0 =A0 32 171 ki31 =A0 =A0 0G =A0 =A0 0G CPU0 =A0 =A00 2= 63:26 2500.00% idle > =A0 17 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU12 =A0= 2 =A0 0:00 100.00% msg_intr12 > =A0 15 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU4 =A0 = =A02 =A0 0:00 100.00% msg_intr4 > =A0 16 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU8 =A0 = =A02 =A0 0:00 100.00% msg_intr8 > =A0 20 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU24 =A0= 1 =A0 0:00 100.00% msg_intr24 > =A0 19 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU20 =A0= 1 =A0 0:00 100.00% msg_intr20 > =A0 21 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU28 =A0= 1 =A0 0:00 100.00% msg_intr28 > =A0 18 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU16 =A0= 1 =A0 0:00 100.00% msg_intr16 > > What are these msg_intrXX kprocs doing? They should really be sleeping unless there is a lot of network traffic :) The msg_intr threads are interrupt handlers which we run one per core, in the first thread of each core. They were modelled after interrupt threads (in FreeBSD 6). This should be sleeping until there is a message ring interrupt (which tells us that an IO has send data to our core over the message ring). Thanks for the report - I will look at the sleep logic. JC. From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 10:56:40 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D11F4106566B for ; Tue, 20 Apr 2010 10:56:40 +0000 (UTC) (envelope-from rpaulo@freebsd.org) Received: from karen.lavabit.com (karen.lavabit.com [72.249.41.33]) by mx1.freebsd.org (Postfix) with ESMTP id 9C9E18FC1A for ; Tue, 20 Apr 2010 10:56:40 +0000 (UTC) Received: from e.earth.lavabit.com (e.earth.lavabit.com [192.168.111.14]) by karen.lavabit.com (Postfix) with ESMTP id 0E0D424EDE1; Tue, 20 Apr 2010 05:56:40 -0500 (CDT) Received: from 10.0.10.3 (54.81.54.77.rev.vodafone.pt [77.54.81.54]) by lavabit.com with ESMTP id N7QPQN4M8FYC; Tue, 20 Apr 2010 05:56:40 -0500 Mime-Version: 1.0 (Apple Message framework v1078) Content-Type: text/plain; charset=us-ascii From: Rui Paulo In-Reply-To: Date: Tue, 20 Apr 2010 11:56:36 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <3BCD65EB-B997-449D-864C-CA24C7B19026@freebsd.org> <6BDB3874-D779-45A6-ABAE-4C331D78A189@lakerest.net> <7BEFA3F5-97AE-477C-9DD3-EF1C4B7DCEB0@freebsd.org> To: "C. Jayachandran" X-Mailer: Apple Mail (2.1078) Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 10:56:41 -0000 On 20 Apr 2010, at 11:49, C. Jayachandran wrote: > On Tue, Apr 20, 2010 at 4:03 PM, Rui Paulo wrote: >> On 20 Apr 2010, at 11:05, Rui Paulo wrote: >>=20 >>> On 20 Apr 2010, at 10:52, C. Jayachandran wrote: >>>=20 >>>> On Mon, Apr 19, 2010 at 7:27 PM, C. Jayachandran >>>> wrote: >>>>> I have a possible cause for the panic with invariants - we should = not >>>>> schedule the msgring threads unless the smp is completely up. I = guess >>>>> we start getting message ring interrupts on before the message = ring >>>>> threads can be scheduled. I am trying out some changes for this - >>>>> will send you a patch if this fixes it. >>>>=20 >>>> I've attached a patch that should fix the issue. The cause was the = way >>>> message ring threads are started on individual cores and the way >>>> interrupts are enabled in the core. I've moved starting message = ring >>>> threads on other cpus to be a SYSINIT after SMP is started. I'd >>>> thought originally that it was due to some clash with the changes = in >>>> HEAD - but looks like I was completely off-track there. >>>>=20 >>>> Please let me know if you don't get multi-user with 32 cpus with = this >>>> patch. There is still the original hang in buildworld, but that = should >>>> be a bug elsewhere >>>>=20 >>>> I have a copy at http://sites.google.com/site/cjayachandran/files = too >>>=20 >>> This works perfectly, thanks! >>=20 >> On further inspection, I noticed that the load avg is now 7. >>=20 >> last pid: 1613; load averages: 6.99, 6.97, 6.08 up 0+00:30:11 = 10:32:48 >> 108 processes: 40 running, 24 sleeping, 44 waiting >> CPU: 0.0% user, 0.0% nice, 21.9% system, 0.0% interrupt, 78.1% = idle >> Mem: 8444K Active, 6028K Inact, 37M Wired, 308K Cache, 6800K Buf, = 3190M Free >> Swap: >>=20 >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU = COMMAND >> 10 root 32 171 ki31 0G 0G CPU0 0 263:26 2500.00% = idle >> 17 root 1 -16 - 0K 0G CPU12 2 0:00 100.00% = msg_intr12 >> 15 root 1 -16 - 0K 0G CPU4 2 0:00 100.00% = msg_intr4 >> 16 root 1 -16 - 0K 0G CPU8 2 0:00 100.00% = msg_intr8 >> 20 root 1 -16 - 0K 0G CPU24 1 0:00 100.00% = msg_intr24 >> 19 root 1 -16 - 0K 0G CPU20 1 0:00 100.00% = msg_intr20 >> 21 root 1 -16 - 0K 0G CPU28 1 0:00 100.00% = msg_intr28 >> 18 root 1 -16 - 0K 0G CPU16 1 0:00 100.00% = msg_intr16 >>=20 >> What are these msg_intrXX kprocs doing? >=20 > They should really be sleeping unless there is a lot of network > traffic :) The msg_intr threads are interrupt handlers which we run > one per core, in the first thread of each core. They were modelled > after interrupt threads (in FreeBSD 6). This should be sleeping until > there is a message ring interrupt (which tells us that an IO has send > data to our core over the message ring). >=20 > Thanks for the report - I will look at the sleep logic. There's almost no network traffic and only one rge (rge1) is connected. = BTW, I'm not using the rge patch you sent yesterday. Regards, -- Rui Paulo From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 11:06:15 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D34B81065676; Tue, 20 Apr 2010 11:06:15 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-pv0-f182.google.com (mail-pv0-f182.google.com [74.125.83.182]) by mx1.freebsd.org (Postfix) with ESMTP id 991CD8FC14; Tue, 20 Apr 2010 11:06:15 +0000 (UTC) Received: by pvc7 with SMTP id 7so3903075pvc.13 for ; Tue, 20 Apr 2010 04:06:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=2wJaRQ91rRtcC6lmxkDERx0cdoV8+VM2hgHtT2w/5ug=; b=sucXktgffiKEn0uFhECJQOuBIFMXGeuc7PKpdIVbf9VYy7CMcen3SRQKoW1A1qXpQn 4kKZZwmUxYbEwYN7vpIPxI5ttUixKrzBBdsxv+Lngtjz7Lcm1bZolYn9kRZkm2h+ZvyZ G4WX6IG+V5+smJ0hWg9pWweM+xNLDDboiLwuk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=oyMoM0mm6HJSqgJKxL413ueCjaAjv0crlEkKygsAQ9OuB9MrjOPvPVGRrxzrwSx8N0 C7XD9NJn4dZ3ucIn8mWCasc+dwOhs1FlR2XbSYaLFAAP/UFXb+lZ9sPEHSbCTd4lyN8i Ab2CeiLVlRklPY+JkA4XwCSjjLNm71FS9j8Gw= MIME-Version: 1.0 Received: by 10.141.29.15 with HTTP; Tue, 20 Apr 2010 04:06:14 -0700 (PDT) In-Reply-To: References: <6BDB3874-D779-45A6-ABAE-4C331D78A189@lakerest.net> <7BEFA3F5-97AE-477C-9DD3-EF1C4B7DCEB0@freebsd.org> Date: Tue, 20 Apr 2010 16:36:14 +0530 Received: by 10.141.187.14 with SMTP id o14mr2227301rvp.244.1271761574863; Tue, 20 Apr 2010 04:06:14 -0700 (PDT) Message-ID: From: "C. Jayachandran" To: Rui Paulo Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 11:06:15 -0000 On Tue, Apr 20, 2010 at 4:26 PM, Rui Paulo wrote: > On 20 Apr 2010, at 11:49, C. Jayachandran wrote: > >> On Tue, Apr 20, 2010 at 4:03 PM, Rui Paulo wrote: >>> On 20 Apr 2010, at 11:05, Rui Paulo wrote: >>> >>>> On 20 Apr 2010, at 10:52, C. Jayachandran wrote: >>>> >>>>> On Mon, Apr 19, 2010 at 7:27 PM, C. Jayachandran >>>>> wrote: >>>>>> I have a possible cause for the panic with invariants - we should no= t >>>>>> schedule the msgring threads unless the smp is completely up. I gues= s >>>>>> we start getting message ring interrupts on before the message ring >>>>>> threads can be scheduled. =A0I am trying out some changes for this - >>>>>> will send you a patch if this fixes it. >>>>> >>>>> I've attached a patch that should fix the issue. The cause was the wa= y >>>>> message ring threads are started on individual cores and the way >>>>> interrupts are enabled in the core. =A0I've moved starting message ri= ng >>>>> threads on other cpus to be a SYSINIT after SMP is started. =A0I'd >>>>> thought originally that it was due to some clash with the changes in >>>>> HEAD - but looks like I was completely off-track there. >>>>> >>>>> Please let me know if you don't get multi-user with 32 cpus with this >>>>> patch. There is still the original hang in buildworld, but that shoul= d >>>>> be a bug elsewhere >>>>> >>>>> I have a copy at http://sites.google.com/site/cjayachandran/files too >>>> >>>> This works perfectly, thanks! >>> >>> On further inspection, I noticed that the load avg is now 7. >>> >>> last pid: =A01613; =A0load averages: =A06.99, =A06.97, =A06.08 =A0 =A0u= p 0+00:30:11 =A010:32:48 >>> 108 processes: 40 running, 24 sleeping, 44 waiting >>> CPU: =A00.0% user, =A00.0% nice, 21.9% system, =A00.0% interrupt, 78.1%= idle >>> Mem: 8444K Active, 6028K Inact, 37M Wired, 308K Cache, 6800K Buf, 3190M= Free >>> Swap: >>> >>> =A0PID USERNAME =A0THR PRI NICE =A0 SIZE =A0 =A0RES STATE =A0 C =A0 TIM= E =A0 WCPU COMMAND >>> =A0 10 root =A0 =A0 =A0 32 171 ki31 =A0 =A0 0G =A0 =A0 0G CPU0 =A0 =A00= 263:26 2500.00% idle >>> =A0 17 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU12 = =A0 2 =A0 0:00 100.00% msg_intr12 >>> =A0 15 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU4 = =A0 =A02 =A0 0:00 100.00% msg_intr4 >>> =A0 16 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU8 = =A0 =A02 =A0 0:00 100.00% msg_intr8 >>> =A0 20 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU24 = =A0 1 =A0 0:00 100.00% msg_intr24 >>> =A0 19 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU20 = =A0 1 =A0 0:00 100.00% msg_intr20 >>> =A0 21 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU28 = =A0 1 =A0 0:00 100.00% msg_intr28 >>> =A0 18 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU16 = =A0 1 =A0 0:00 100.00% msg_intr16 >>> >>> What are these msg_intrXX kprocs doing? >> >> They should really be sleeping unless there is a lot of network >> traffic :) =A0The msg_intr threads are interrupt handlers which we run >> one per core, in the first thread of each core. =A0They were modelled >> after interrupt threads (in FreeBSD 6). This should be sleeping until >> there is a message ring interrupt (which tells us that an IO has send >> data to our core over the message ring). >> >> Thanks for the report - I will look at the sleep logic. I'm not seeing the issue here(my output for ref below). The rge patch should not really make a difference - but it will be good to try with that. The only other difference I can think of between our configs is MFS root/NFS root and rge0/rge1 - but none of these should affect the message ring threads. Can you send me the config you use? regards, JC. 114 processes: 33 running, 29 sleeping, 52 waiting CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 7380K Active, 9252K Inact, 38M Wired, 8K Cache, 9200K Buf, 3160M Free Swap: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 10 root 32 171 ki31 0G 0G CPU0 0 61:46 3199.46% idle 1802 rmi 1 44 0 0G 0G CPU1 1 0:00 0.29% top 11 root 44 -68 - 0K 0G WAIT 0 0:00 0.00% intr 0 root 3 8 0 0G 0G - 0 0:00 0.00% kernel 1789 root 1 55 0 0G 0G sbwait 0 0:00 0.00% sshd 1784 root 1 70 0 0G 0G ttyin 1 0:00 0.00% csh 1 root 1 44 0 0G 0G wait 26 0:00 0.00% init 1783 root 1 50 0 0G 0G wait 1 0:00 0.00% login 6 root 1 -8 - 0K 0G mdwait 3 0:00 0.00% md0 1792 rmi 1 44 0 0G 0G select 0 0:00 0.00% sshd 13 root 1 -16 - 0K 0G WAIT 0 0:00 0.00% msg_intr= 0 22 root 1 -16 - 0K 0G WAIT 28 0:00 0.00% msg_intr= 28 12 root 1 -16 - 0K 0G - 2 0:00 0.00% yarrow 1424 root 1 44 0 0G 0G select 3 0:00 0.00% syslogd 19 root 1 -16 - 0K 0G WAIT 16 0:00 0.00% msg_intr= 16 18 root 1 -16 - 0K 0G WAIT 12 0:00 0.00% msg_intr= 12 21 root 1 -16 - 0K 0G WAIT 24 0:00 0.00% msg_intr= 24 17 root 1 -16 - 0K 0G WAIT 8 0:00 0.00% msg_intr= 8 20 root 1 -16 - 0K 0G WAIT 20 0:00 0.00% msg_intr= 20 1793 rmi 1 44 0 0G 0G wait 0 0:00 0.00% sh 2 root 1 -8 - 0K 0G - 4 0:00 0.00% g_event 3 root 1 -8 - 0K 0G - 4 0:00 0.00% g_up 16 root 1 -16 - 0K 0G WAIT 4 0:00 0.00% msg_intr= 4 From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 11:12:42 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C656B1065680 for ; Tue, 20 Apr 2010 11:12:42 +0000 (UTC) (envelope-from rpaulo@freebsd.org) Received: from karen.lavabit.com (karen.lavabit.com [72.249.41.33]) by mx1.freebsd.org (Postfix) with ESMTP id 92E838FC1A for ; Tue, 20 Apr 2010 11:12:42 +0000 (UTC) Received: from e.earth.lavabit.com (e.earth.lavabit.com [192.168.111.14]) by karen.lavabit.com (Postfix) with ESMTP id E4A0A11BA21; Tue, 20 Apr 2010 06:12:41 -0500 (CDT) Received: from 10.0.10.3 (54.81.54.77.rev.vodafone.pt [77.54.81.54]) by lavabit.com with ESMTP id XGM5AMGKUIS2; Tue, 20 Apr 2010 06:12:41 -0500 Mime-Version: 1.0 (Apple Message framework v1078) Content-Type: text/plain; charset=us-ascii From: Rui Paulo In-Reply-To: Date: Tue, 20 Apr 2010 12:12:38 +0100 Content-Transfer-Encoding: 7bit Message-Id: <3820374E-010D-40EA-A596-4AF1694B4119@freebsd.org> References: <6BDB3874-D779-45A6-ABAE-4C331D78A189@lakerest.net> <7BEFA3F5-97AE-477C-9DD3-EF1C4B7DCEB0@freebsd.org> To: "C. Jayachandran" X-Mailer: Apple Mail (2.1078) Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 11:12:42 -0000 On 20 Apr 2010, at 12:06, C. Jayachandran wrote: > I'm not seeing the issue here(my output for ref below). The rge patch > should not really make a difference - but it will be good to try with > that. The only other difference I can think of between our configs is > MFS root/NFS root and rge0/rge1 - but none of these should affect the > message ring threads. Can you send me the config you use? I use the basic XLR config with some mods: Index: XLR =================================================================== --- XLR (revision 206791) +++ XLR (working copy) @@ -59,10 +59,10 @@ makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols #profile 2 -#options SCHED_ULE # ULE scheduler +options SCHED_ULE # ULE scheduler #options VERBOSE_SYSINIT -options SCHED_4BSD # 4BSD scheduler -#options PREEMPTION # Enable kernel thread preemption +#options SCHED_4BSD # 4BSD scheduler +options PREEMPTION # Enable kernel thread preemption #options FULL_PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options INET6 # IPv6 communications protocols @@ -72,13 +72,14 @@ options UFS_DIRHASH # Improve performance on big directories options NFSCLIENT options NFS_ROOT +options SMP # options BOOTP options BOOTP_NFSROOT options BOOTP_NFSV3 -options BOOTP_WIRED_TO=rge0 +options BOOTP_WIRED_TO=rge1 options BOOTP_COMPAT -options ROOTDEVNAME=\"nfs:10.1.1.8:/usr/extra/nfsroot\" +#options ROOTDEVNAME=\"nfs:10.1.1.8:/usr/extra/nfsroot\" # #options MD_ROOT # MD is a potential root device #options MD_ROOT_SIZE=27000 Regards, -- Rui Paulo From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 11:36:34 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E1131106564A for ; Tue, 20 Apr 2010 11:36:34 +0000 (UTC) (envelope-from rpaulo@freebsd.org) Received: from karen.lavabit.com (karen.lavabit.com [72.249.41.33]) by mx1.freebsd.org (Postfix) with ESMTP id 8E3988FC25 for ; Tue, 20 Apr 2010 11:36:34 +0000 (UTC) Received: from e.earth.lavabit.com (e.earth.lavabit.com [192.168.111.14]) by karen.lavabit.com (Postfix) with ESMTP id 06B3111BA4C for ; Tue, 20 Apr 2010 06:36:34 -0500 (CDT) Received: from 10.0.10.3 (54.81.54.77.rev.vodafone.pt [77.54.81.54]) by lavabit.com with ESMTP id 0K8GFKKXANLV for ; Tue, 20 Apr 2010 06:36:34 -0500 From: Rui Paulo Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Tue, 20 Apr 2010 12:36:31 +0100 Message-Id: To: freebsd-mips@freebsd.org Mime-Version: 1.0 (Apple Message framework v1078) X-Mailer: Apple Mail (2.1078) Subject: [PATCH] Account RMI XLR interrupts X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 11:36:35 -0000 Hi, The following patch accounts interrupts on the XLR and makes vmstat -i = and systat work display useful data. I think the commented out atomic = inc can go away, but I'm not sure. Here's an example: $ vmstat -i interrupt total rate int16: 56118 984 int17: 214 3 Total 56332 988 int16 is the timer and int17 is the network interface (rge1). Index: intr_machdep.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- intr_machdep.c (revision 206791) +++ intr_machdep.c (working copy) @@ -157,28 +157,17 @@ for (i =3D sizeof(eirr) * 8 - 1; i >=3D 0; i--) { if ((eirr & (1ULL << i)) =3D=3D 0) continue; ie =3D mips_intr_events[i]; /* atomic_add_long(mih->cntp, 1); */ + /* Don't account special IRQs */ + switch (i) { + case IRQ_IPI: + case IRQ_MSGRING: + break; + default: + mips_intrcnt_inc(mips_intr_counters[i]); + } =20 write_c0_eirr64(1ULL << i); pic_ack(i, 0); Regards, -- Rui Paulo From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 11:39:58 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B29DC1065678 for ; Tue, 20 Apr 2010 11:39:58 +0000 (UTC) (envelope-from rpaulo@freebsd.org) Received: from karen.lavabit.com (karen.lavabit.com [72.249.41.33]) by mx1.freebsd.org (Postfix) with ESMTP id 59ACC8FC14 for ; Tue, 20 Apr 2010 11:39:58 +0000 (UTC) Received: from e.earth.lavabit.com (e.earth.lavabit.com [192.168.111.14]) by karen.lavabit.com (Postfix) with ESMTP id CCB1D11BA50 for ; Tue, 20 Apr 2010 06:39:57 -0500 (CDT) Received: from 10.0.10.3 (54.81.54.77.rev.vodafone.pt [77.54.81.54]) by lavabit.com with ESMTP id TNC3U5B7SQKP for ; Tue, 20 Apr 2010 06:39:57 -0500 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Apple Message framework v1078) From: Rui Paulo In-Reply-To: Date: Tue, 20 Apr 2010 12:39:55 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <4C314E9E-E947-4572-8ABC-F2C415EEBFF9@freebsd.org> References: To: freebsd-mips@freebsd.org X-Mailer: Apple Mail (2.1078) Subject: Re: [PATCH] Account RMI XLR interrupts X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 11:39:58 -0000 On 20 Apr 2010, at 12:36, Rui Paulo wrote: > Hi, > The following patch accounts interrupts on the XLR and makes vmstat -i = and systat work display useful data. I think the commented out atomic = inc can go away, but I'm not sure. >=20 > Here's an example: >=20 > $ vmstat -i > interrupt total rate > int16: 56118 984 > int17: 214 3 > Total 56332 988 >=20 > int16 is the timer and int17 is the network interface (rge1). Actually, int17 is the serial console. Not sure why the network = interface doesn't show up. Regards, -- Rui Paulo From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 13:10:08 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BDC431065670; Tue, 20 Apr 2010 13:10:08 +0000 (UTC) (envelope-from rrs@lakerest.net) Received: from lakerest.net (unknown [IPv6:2001:240:585:2:213:d4ff:fef3:2d8d]) by mx1.freebsd.org (Postfix) with ESMTP id 5380D8FC16; Tue, 20 Apr 2010 13:10:08 +0000 (UTC) Received: from [192.168.2.112] (pool-173-67-91-86.snfcca.dsl-w.verizon.net [173.67.91.86]) (authenticated bits=0) by lakerest.net (8.14.3/8.14.3) with ESMTP id o3KDA2S5051035 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Tue, 20 Apr 2010 09:10:05 -0400 (EDT) (envelope-from rrs@lakerest.net) Message-Id: From: Randall Stewart To: Rui Paulo In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Tue, 20 Apr 2010 06:09:57 -0700 References: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> <3BCD65EB-B997-449D-864C-CA24C7B19026@freebsd.org> <6BDB3874-D779-45A6-ABAE-4C331D78A189@lakerest.net> <7BEFA3F5-97AE-477C-9DD3-EF1C4B7DCEB0@freebsd.org> X-Mailer: Apple Mail (2.936) Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 13:10:08 -0000 Rui: I don't see this at all either... hmm I wonder why.. I am getting up and running (thanks JC).. I will launch a buildworld -j16 before heading to work ;-) R On Apr 20, 2010, at 3:33 AM, Rui Paulo wrote: > On 20 Apr 2010, at 11:05, Rui Paulo wrote: > >> On 20 Apr 2010, at 10:52, C. Jayachandran wrote: >> >>> On Mon, Apr 19, 2010 at 7:27 PM, C. Jayachandran >>> wrote: >>>> I have a possible cause for the panic with invariants - we should >>>> not >>>> schedule the msgring threads unless the smp is completely up. I >>>> guess >>>> we start getting message ring interrupts on before the message ring >>>> threads can be scheduled. I am trying out some changes for this - >>>> will send you a patch if this fixes it. >>> >>> I've attached a patch that should fix the issue. The cause was the >>> way >>> message ring threads are started on individual cores and the way >>> interrupts are enabled in the core. I've moved starting message >>> ring >>> threads on other cpus to be a SYSINIT after SMP is started. I'd >>> thought originally that it was due to some clash with the changes in >>> HEAD - but looks like I was completely off-track there. >>> >>> Please let me know if you don't get multi-user with 32 cpus with >>> this >>> patch. There is still the original hang in buildworld, but that >>> should >>> be a bug elsewhere >>> >>> I have a copy at http://sites.google.com/site/cjayachandran/files >>> too >> >> This works perfectly, thanks! > > On further inspection, I noticed that the load avg is now 7. > > last pid: 1613; load averages: 6.99, 6.97, 6.08 up > 0+00:30:11 10:32:48 > 108 processes: 40 running, 24 sleeping, 44 waiting > CPU: 0.0% user, 0.0% nice, 21.9% system, 0.0% interrupt, 78.1% idle > Mem: 8444K Active, 6028K Inact, 37M Wired, 308K Cache, 6800K Buf, > 3190M Free > Swap: > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 10 root 32 171 ki31 0G 0G CPU0 0 263:26 2500.00% > idle > 17 root 1 -16 - 0K 0G CPU12 2 0:00 100.00% > msg_intr12 > 15 root 1 -16 - 0K 0G CPU4 2 0:00 100.00% > msg_intr4 > 16 root 1 -16 - 0K 0G CPU8 2 0:00 100.00% > msg_intr8 > 20 root 1 -16 - 0K 0G CPU24 1 0:00 100.00% > msg_intr24 > 19 root 1 -16 - 0K 0G CPU20 1 0:00 100.00% > msg_intr20 > 21 root 1 -16 - 0K 0G CPU28 1 0:00 100.00% > msg_intr28 > 18 root 1 -16 - 0K 0G CPU16 1 0:00 100.00% > msg_intr16 > > What are these msg_intrXX kprocs doing? > > Regards, > -- > Rui Paulo > > ------------------------------ Randall Stewart 803-317-4952 (cell) 803-345-0391(direct) From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 13:11:23 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 72F87106566B; Tue, 20 Apr 2010 13:11:23 +0000 (UTC) (envelope-from rrs@lakerest.net) Received: from lakerest.net (unknown [IPv6:2001:240:585:2:213:d4ff:fef3:2d8d]) by mx1.freebsd.org (Postfix) with ESMTP id E8B2A8FC23; Tue, 20 Apr 2010 13:11:22 +0000 (UTC) Received: from [192.168.2.112] (pool-173-67-91-86.snfcca.dsl-w.verizon.net [173.67.91.86]) (authenticated bits=0) by lakerest.net (8.14.3/8.14.3) with ESMTP id o3KDBJPk051103 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Tue, 20 Apr 2010 09:11:21 -0400 (EDT) (envelope-from rrs@lakerest.net) Message-Id: From: Randall Stewart To: Rui Paulo In-Reply-To: <3820374E-010D-40EA-A596-4AF1694B4119@freebsd.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Tue, 20 Apr 2010 06:11:13 -0700 References: <6BDB3874-D779-45A6-ABAE-4C331D78A189@lakerest.net> <7BEFA3F5-97AE-477C-9DD3-EF1C4B7DCEB0@freebsd.org> <3820374E-010D-40EA-A596-4AF1694B4119@freebsd.org> X-Mailer: Apple Mail (2.936) Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 13:11:23 -0000 Hmmm I don't have PREMPTION on.. let me try that and see if I can get the same thing as you R On Apr 20, 2010, at 4:12 AM, Rui Paulo wrote: > On 20 Apr 2010, at 12:06, C. Jayachandran wrote: >> I'm not seeing the issue here(my output for ref below). The rge >> patch >> should not really make a difference - but it will be good to try with >> that. The only other difference I can think of between our configs >> is >> MFS root/NFS root and rge0/rge1 - but none of these should affect the >> message ring threads. Can you send me the config you use? > > I use the basic XLR config with some mods: > > Index: XLR > =================================================================== > --- XLR (revision 206791) > +++ XLR (working copy) > @@ -59,10 +59,10 @@ > makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols > #profile 2 > > -#options SCHED_ULE # ULE scheduler > +options SCHED_ULE # ULE scheduler > #options VERBOSE_SYSINIT > -options SCHED_4BSD # 4BSD scheduler > -#options PREEMPTION # Enable kernel thread preemption > +#options SCHED_4BSD # 4BSD scheduler > +options PREEMPTION # Enable kernel thread preemption > #options FULL_PREEMPTION # Enable kernel thread preemption > options INET # InterNETworking > options INET6 # IPv6 communications protocols > @@ -72,13 +72,14 @@ > options UFS_DIRHASH # Improve performance on big directories > options NFSCLIENT > options NFS_ROOT > +options SMP > # > options BOOTP > options BOOTP_NFSROOT > options BOOTP_NFSV3 > -options BOOTP_WIRED_TO=rge0 > +options BOOTP_WIRED_TO=rge1 > options BOOTP_COMPAT > -options ROOTDEVNAME=\"nfs:10.1.1.8:/usr/extra/nfsroot\" > +#options ROOTDEVNAME=\"nfs:10.1.1.8:/usr/extra/nfsroot\" > # > #options MD_ROOT # MD is a potential root device > #options MD_ROOT_SIZE=27000 > > > Regards, > -- > Rui Paulo > > ------------------------------ Randall Stewart 803-317-4952 (cell) 803-345-0391(direct) From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 13:29:37 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 47E7A1065674; Tue, 20 Apr 2010 13:29:36 +0000 (UTC) (envelope-from rrs@lakerest.net) Received: from lakerest.net (unknown [IPv6:2001:240:585:2:213:d4ff:fef3:2d8d]) by mx1.freebsd.org (Postfix) with ESMTP id 76DD48FC1A; Tue, 20 Apr 2010 13:29:36 +0000 (UTC) Received: from [192.168.2.112] (pool-173-67-91-86.snfcca.dsl-w.verizon.net [173.67.91.86]) (authenticated bits=0) by lakerest.net (8.14.3/8.14.3) with ESMTP id o3KDTWZl051911 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Tue, 20 Apr 2010 09:29:35 -0400 (EDT) (envelope-from rrs@lakerest.net) Message-Id: <0DD79652-5568-4D36-B199-C5CF109D85A8@lakerest.net> From: Randall Stewart To: Rui Paulo , "C. Jayachandran" In-Reply-To: <4C314E9E-E947-4572-8ABC-F2C415EEBFF9@freebsd.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Tue, 20 Apr 2010 06:29:27 -0700 References: <4C314E9E-E947-4572-8ABC-F2C415EEBFF9@freebsd.org> X-Mailer: Apple Mail (2.936) Cc: freebsd-mips@freebsd.org Subject: Re: [PATCH] Account RMI XLR interrupts X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 13:29:37 -0000 Ok With all patches in I have a make buildworld -j30 .. that should prove interesting ;-) I have never seen Rui's issue with 100% cpu... seems pretty good. we will see how it is at dark when I come home from work ;-) R On Apr 20, 2010, at 4:39 AM, Rui Paulo wrote: > On 20 Apr 2010, at 12:36, Rui Paulo wrote: > >> Hi, >> The following patch accounts interrupts on the XLR and makes vmstat >> -i and systat work display useful data. I think the commented out >> atomic inc can go away, but I'm not sure. >> >> Here's an example: >> >> $ vmstat -i >> interrupt total rate >> int16: 56118 984 >> int17: 214 3 >> Total 56332 988 >> >> int16 is the timer and int17 is the network interface (rge1). > > Actually, int17 is the serial console. Not sure why the network > interface doesn't show up. > > Regards, > -- > Rui Paulo > > > _______________________________________________ > freebsd-mips@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-mips > To unsubscribe, send any mail to "freebsd-mips- > unsubscribe@freebsd.org" > ------------------------------ Randall Stewart 803-317-4952 (cell) 803-345-0391(direct) From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 14:18:46 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 81BE6106567A; Tue, 20 Apr 2010 14:18:46 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-pz0-f197.google.com (mail-pz0-f197.google.com [209.85.222.197]) by mx1.freebsd.org (Postfix) with ESMTP id 4F8B28FC14; Tue, 20 Apr 2010 14:18:45 +0000 (UTC) Received: by pzk35 with SMTP id 35so211190pzk.3 for ; Tue, 20 Apr 2010 07:18:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=cBjTNJqKhqVC1GDmlMzLEc2XvKg6B68+LMYyFbOPEqk=; b=XCVSRBfU+DIPWh584Zuu9E//AgI4eQ4sVhrHIu8u/norpqrEXeY0zVgdgMC2wvtA6j Z2zo4V9dtG1htv4BC7a4aXWubEDKVb/VUbU9MQ/tHdJK6jr347SZeV5X6qJLtDeIsSPv XrKQa8vcpg751/vrLw0WPV2JjOa8+gn+5+/UA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=iV7/FtK7JcP47ZNgcjfdjFzrlFrZ/CZKWn6xebQgEyepZhX4HlkzjrsrJUD6wCZRmf V1h6rDuiw+jgHG+5wHE50iu8K6Hrdle8ymykh0odgrH4GKI+lctQHS3rH1cQ8Fm90Ag0 KJjFNkUv9QQMliCAhxof7/hTRiPf97j2Y4SXk= MIME-Version: 1.0 Received: by 10.141.29.15 with HTTP; Tue, 20 Apr 2010 07:18:43 -0700 (PDT) In-Reply-To: References: Date: Tue, 20 Apr 2010 19:48:43 +0530 Received: by 10.141.4.10 with SMTP id g10mr5748840rvi.179.1271773125330; Tue, 20 Apr 2010 07:18:45 -0700 (PDT) Message-ID: From: "C. Jayachandran" To: Rui Paulo Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org Subject: Re: [PATCH] Account RMI XLR interrupts X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 14:18:46 -0000 On Tue, Apr 20, 2010 at 5:06 PM, Rui Paulo wrote: > Hi, > The following patch accounts interrupts on the XLR and makes vmstat -i an= d systat work display useful data. I think the commented out atomic inc can= go away, but I'm not sure. > > Here's an example: > > $ vmstat -i > interrupt =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0total =A0 = =A0 =A0 rate > int16: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 56118 =A0 = =A0 =A0 =A0984 > int17: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 214 = =A0 =A0 =A0 =A0 =A03 > Total =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A056332 = =A0 =A0 =A0 =A0988 > > int16 is the timer and int17 is the network interface (rge1). Are you sure that the mips_intr_events array has the size to hold all the XLR interrupts? The XLR has a slightly different scheme (with COP0 EIRR EIMR registers) and can have upto 64 interrupts. A mechanism to unify the interrupt, and handling the XLR PIC may be a good = idea. The network packets come on MSGRING interrupt - only link status etc will come on the GMAC interrupt. JC. > Index: intr_machdep.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- intr_machdep.c =A0 =A0 =A0(revision 206791) > +++ intr_machdep.c =A0 =A0 =A0(working copy) > @@ -157,28 +157,17 @@ > =A0 =A0 =A0 =A0for (i =3D sizeof(eirr) * 8 - 1; i >=3D 0; i--) { > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if ((eirr & (1ULL << i)) =3D=3D 0) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0continue; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0ie =3D mips_intr_events[i]; > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0/* atomic_add_long(mih->cntp, 1); */ > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* Don't account special IRQs */ > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 switch (i) { > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 case IRQ_IPI: > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 case IRQ_MSGRING: > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 break; > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 default: > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 mips_intrcnt_inc(mips_intr_= counters[i]); > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0write_c0_eirr64(1ULL << i); > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0pic_ack(i, 0); > > Regards, > -- > Rui Paulo From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 14:23:25 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C61451065675; Tue, 20 Apr 2010 14:23:25 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9313B8FC13; Tue, 20 Apr 2010 14:23:25 +0000 (UTC) Received: by pwi9 with SMTP id 9so4367139pwi.13 for ; Tue, 20 Apr 2010 07:23:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=TWP4BLXlv5+eOEcWUpH346u8hCxusS9CqaH+w2OBoyc=; b=RoaiIDKN4wbGFssl2EzuX4lDJfvO7uMi5IWCbeZJ/ZdXNBWH9UR5HcsmJ5quGu9I1H 2GQWvrnrAiHGjiAmAHPLQ6R5hEyDDq9sHAYhMHIRdlq+6L4HOrA4eAFSGrINom0iJj1B GydTKtxKori0ZVDKdE25C+UfaGGgGShPjIsxI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=QS2s8cOutvsTl5H360H/jesCKXfPfHe2uNSFsG3AAFJujbYp+NSnfx/aQs9+lSrMuC kFHavY8BKz6PIuhPBndCGurkkOJEmRP3/gY2dfIQ1hYeE62KErRswOw07VpAjhLIogsn Uxb4/jc2W8ue0RAOxwnJtWsCPl2EL4mHWLku8= MIME-Version: 1.0 Received: by 10.141.29.15 with HTTP; Tue, 20 Apr 2010 07:23:23 -0700 (PDT) In-Reply-To: References: Date: Tue, 20 Apr 2010 19:53:23 +0530 Received: by 10.141.88.3 with SMTP id q3mr5753849rvl.162.1271773403829; Tue, 20 Apr 2010 07:23:23 -0700 (PDT) Message-ID: From: "C. Jayachandran" To: Rui Paulo Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org Subject: Re: [PATCH] Account RMI XLR interrupts X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 14:23:25 -0000 Please ignore the first part of the mail about array size, I seem to have missed the definition in rmi files. JC. On Tue, Apr 20, 2010 at 7:48 PM, C. Jayachandran wrote: > On Tue, Apr 20, 2010 at 5:06 PM, Rui Paulo wrote: >> Hi, >> The following patch accounts interrupts on the XLR and makes vmstat -i a= nd systat work display useful data. I think the commented out atomic inc ca= n go away, but I'm not sure. >> >> Here's an example: >> >> $ vmstat -i >> interrupt =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0total =A0 = =A0 =A0 rate >> int16: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 56118 =A0= =A0 =A0 =A0984 >> int17: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 214 = =A0 =A0 =A0 =A0 =A03 >> Total =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A056332 = =A0 =A0 =A0 =A0988 >> >> int16 is the timer and int17 is the network interface (rge1). > > Are you sure that the mips_intr_events array has the size to hold all > the XLR interrupts? =A0The XLR has a slightly different scheme (with > COP0 EIRR EIMR registers) and can have upto 64 interrupts. > > A mechanism to unify the interrupt, and handling the XLR PIC may be a goo= d idea. > > The network packets come on MSGRING interrupt - only link status etc > will come on the GMAC interrupt. > > JC. > > >> Index: intr_machdep.c >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> --- intr_machdep.c =A0 =A0 =A0(revision 206791) >> +++ intr_machdep.c =A0 =A0 =A0(working copy) >> @@ -157,28 +157,17 @@ >> =A0 =A0 =A0 =A0for (i =3D sizeof(eirr) * 8 - 1; i >=3D 0; i--) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if ((eirr & (1ULL << i)) =3D=3D 0) >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0continue; >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0ie =3D mips_intr_events[i]; >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0/* atomic_add_long(mih->cntp, 1); */ >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* Don't account special IRQs */ >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 switch (i) { >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 case IRQ_IPI: >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 case IRQ_MSGRING: >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 break; >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 default: >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 mips_intrcnt_inc(mips_intr= _counters[i]); >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } >> >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0write_c0_eirr64(1ULL << i); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0pic_ack(i, 0); >> >> Regards, >> -- >> Rui Paulo > --=20 C. Jayachandran c.jayachandran@gmail.com From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 14:48:13 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 829EA1065670; Tue, 20 Apr 2010 14:48:13 +0000 (UTC) (envelope-from rrs@lakerest.net) Received: from lakerest.net (unknown [IPv6:2001:240:585:2:213:d4ff:fef3:2d8d]) by mx1.freebsd.org (Postfix) with ESMTP id E4D0D8FC1D; Tue, 20 Apr 2010 14:48:12 +0000 (UTC) Received: from [32.177.69.229] ([32.177.69.229]) (authenticated bits=0) by lakerest.net (8.14.3/8.14.3) with ESMTP id o3KElxMT055258 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Tue, 20 Apr 2010 10:48:08 -0400 (EDT) (envelope-from rrs@lakerest.net) Message-Id: From: Randall Stewart To: "C. Jayachandran" In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Tue, 20 Apr 2010 07:47:53 -0700 References: X-Mailer: Apple Mail (2.936) Cc: freebsd-mips@freebsd.org Subject: Re: [PATCH] Account RMI XLR interrupts X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 14:48:13 -0000 Yeah.. I made sure that it was big enough ;-) R On Apr 20, 2010, at 7:23 AM, C. Jayachandran wrote: > Please ignore the first part of the mail about array size, I seem to > have missed the definition in rmi files. > > JC. > > On Tue, Apr 20, 2010 at 7:48 PM, C. Jayachandran > wrote: >> On Tue, Apr 20, 2010 at 5:06 PM, Rui Paulo >> wrote: >>> Hi, >>> The following patch accounts interrupts on the XLR and makes >>> vmstat -i and systat work display useful data. I think the >>> commented out atomic inc can go away, but I'm not sure. >>> >>> Here's an example: >>> >>> $ vmstat -i >>> interrupt total rate >>> int16: 56118 984 >>> int17: 214 3 >>> Total 56332 988 >>> >>> int16 is the timer and int17 is the network interface (rge1). >> >> Are you sure that the mips_intr_events array has the size to hold all >> the XLR interrupts? The XLR has a slightly different scheme (with >> COP0 EIRR EIMR registers) and can have upto 64 interrupts. >> >> A mechanism to unify the interrupt, and handling the XLR PIC may be >> a good idea. >> >> The network packets come on MSGRING interrupt - only link status etc >> will come on the GMAC interrupt. >> >> JC. >> >> >>> Index: intr_machdep.c >>> =================================================================== >>> --- intr_machdep.c (revision 206791) >>> +++ intr_machdep.c (working copy) >>> @@ -157,28 +157,17 @@ >>> for (i = sizeof(eirr) * 8 - 1; i >= 0; i--) { >>> if ((eirr & (1ULL << i)) == 0) >>> continue; >>> ie = mips_intr_events[i]; >>> /* atomic_add_long(mih->cntp, 1); */ >>> + /* Don't account special IRQs */ >>> + switch (i) { >>> + case IRQ_IPI: >>> + case IRQ_MSGRING: >>> + break; >>> + default: >>> + mips_intrcnt_inc(mips_intr_counters[i]); >>> + } >>> >>> write_c0_eirr64(1ULL << i); >>> pic_ack(i, 0); >>> >>> Regards, >>> -- >>> Rui Paulo >> > > > > -- > C. Jayachandran c.jayachandran@gmail.com > _______________________________________________ > freebsd-mips@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-mips > To unsubscribe, send any mail to "freebsd-mips- > unsubscribe@freebsd.org" > ------------------------------ Randall Stewart 803-317-4952 (cell) 803-345-0391(direct) From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 15:02:58 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 45B2E1065670 for ; Tue, 20 Apr 2010 15:02:58 +0000 (UTC) (envelope-from rpaulo@freebsd.org) Received: from karen.lavabit.com (karen.lavabit.com [72.249.41.33]) by mx1.freebsd.org (Postfix) with ESMTP id 262B18FC13 for ; Tue, 20 Apr 2010 15:02:57 +0000 (UTC) Received: from e.earth.lavabit.com (e.earth.lavabit.com [192.168.111.14]) by karen.lavabit.com (Postfix) with ESMTP id 83E3B11B7F8; Tue, 20 Apr 2010 10:02:57 -0500 (CDT) Received: from 10.0.10.3 (54.81.54.77.rev.vodafone.pt [77.54.81.54]) by lavabit.com with ESMTP id E300JD4AWBZR; Tue, 20 Apr 2010 10:02:57 -0500 Mime-Version: 1.0 (Apple Message framework v1078) Content-Type: text/plain; charset=us-ascii From: Rui Paulo In-Reply-To: Date: Tue, 20 Apr 2010 16:02:54 +0100 Content-Transfer-Encoding: 7bit Message-Id: References: To: C. Jayachandran X-Mailer: Apple Mail (2.1078) Cc: freebsd-mips@freebsd.org Subject: Re: [PATCH] Account RMI XLR interrupts X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 15:02:58 -0000 On 20 Apr 2010, at 15:18, C. Jayachandran wrote: > The network packets come on MSGRING interrupt - only link status etc > will come on the GMAC interrupt. IC, that makes sense. Regards, -- Rui Paulo From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 21:36:01 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A2A2D106567C for ; Tue, 20 Apr 2010 21:36:01 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 761218FC12 for ; Tue, 20 Apr 2010 21:36:01 +0000 (UTC) Received: by pwi9 with SMTP id 9so4798423pwi.13 for ; Tue, 20 Apr 2010 14:36:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=wdl3CApJSQmY3foy6e/yMvmKJ9uMmRureLAot5azcJk=; b=vOnk16SEoE2/sjFSgGNv35pddNjeN7trvKUnUOUhYRNoGtzupQR47bLwz8tjZLC1GE kPThdm90YftgVjVFDI2G/bLQA9BWbGSJqDaCtfwlDpjNP60K+ApEJnjiksuGP4hLtDdh 5dwfDzSZI4fO/Vm2q75xwS4WVNujB5H1+6+d0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=kkSAdKicoGMgsnt8CTBWji1H1xkuq/7xElfXhZ3DuOp4qBI7Zn6FkHisdzRwNzmupA 9524vqX4o9HeR/1F4hdAGX8z7+tFoDjTNm3KDSHgzFkw6Dse7jg9wFF70GD3LQLlmPQL i74srp5mxLAP0uXiOtgCYgMU3nQMKvWOWobPA= MIME-Version: 1.0 Received: by 10.142.49.20 with HTTP; Tue, 20 Apr 2010 14:36:00 -0700 (PDT) In-Reply-To: References: Date: Tue, 20 Apr 2010 14:36:00 -0700 Received: by 10.142.122.24 with SMTP id u24mr1562235wfc.264.1271799360670; Tue, 20 Apr 2010 14:36:00 -0700 (PDT) Message-ID: From: Neel Natu To: "C. Jayachandran" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 21:36:01 -0000 Hi JC, On Sat, Apr 17, 2010 at 3:40 PM, C. Jayachandran wrote: > I've a set of initial patches to enable SMP for RMI processors. It > comes up in multi-user with 32 CPUs. =A0I could do buildworld before I > updated to HEAD - with head there is a hang during buildworld which > I'm looking at, but I think the initial work can be checked in. > > Neel, can you have a look at the first two patches - one is to enable > ULE scheduler and the second one is to move platform_init_ap to > slightly later in the initialization sequence. > > The patches are : > 1. mips-ule-support.patch > - Enable ULE scheduler for MIPS > > 2. mips-smp-move-platform.patch > - We need a hook to setup message ring and its interrupts, we use > platform_init_ap now, and move it be called later for XLR > I would like to see us move away from #ifdef TARGET_FOO in files under mips/mips as much as possible. I think that is achieved easily in this instance. How about we create a function platform_ap_enable_interrupts() that is called from smp_init_secondary()? This function will be defined for each platform. In the common case this function will simply call mips_ap_enable_interrupts() that encapsulates the logic that we currently have to setup interrupt masks for clock and ipi interrupts in the status register. In the XLR case however it can do something different. Ditto about the #ifdef TARGET_XLR_XLS in mpboot.S. You can simply have an empty platform_init_ap() for XLR. best Neel > 3. rmi-prid.patch > Add RMI processor ID prints - right now it prints unknown processor. > > 4. rmi-pcib-fix.patch > XLR pci bridge should be off the IODI bus, currently it is off the > nexus bus which will cause issue when bus_setup_intr on nexus is > called > > 5. cleanup-reorg-cpuid-rge-kx.patch > This set has a many changes rolled up: > =A0- clean up rge driver, remove unused code paths and commented code > (long way to go still) > =A0- xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage updat= ed > =A0- fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses > here, I have made changes for using XKPHYS to do the same process. > =A0- the KX bit is enabled and interrupts are disabled before the > physical memory is read, to avoid the KX bit setting from affecting > other code. > - move message ring code to on_chip.c from xlr_machdep.c, now all > message ring code is in on_chip.c > > 6. rmi-xlr-smp.patch > SMP Support for XLR > - mpwait.S added to get the other CPUs out of bootloader code at startup. > - SMP platform related functions. > - disable shared TLB code for SMP - since threads have different mappings= . > > The patches are also available at > http://sites.google.com/site/cjayachandran/files > > Enabling all 32 threads will need a minor fix in the SMP code, see the > patch subr_smp-fix.patch at the same place. > > Regards, > JC. > From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 22:04:40 2010 Return-Path: Delivered-To: freebsd-mips@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1704F1065676 for ; Tue, 20 Apr 2010 22:04:40 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id BA7308FC29 for ; Tue, 20 Apr 2010 22:04:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.14.3/8.14.1) with ESMTP id o3KLuKwK072868; Tue, 20 Apr 2010 15:56:20 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Tue, 20 Apr 2010 15:56:33 -0600 (MDT) Message-Id: <20100420.155633.1075901050443800973.imp@bsdimp.com> To: neelnatu@gmail.com From: "M. Warner Losh" In-Reply-To: References: X-Mailer: Mew version 6.3 on Emacs 22.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@FreeBSD.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 22:04:40 -0000 In message: Neel Natu writes: : Hi JC, : = : On Sat, Apr 17, 2010 at 3:40 PM, C. Jayachandran : wrote: : > I've a set of initial patches to enable SMP for RMI processors. It : > comes up in multi-user with 32 CPUs. =A0I could do buildworld befor= e I : > updated to HEAD - with head there is a hang during buildworld which= : > I'm looking at, but I think the initial work can be checked in. : > : > Neel, can you have a look at the first two patches - one is to enab= le : > ULE scheduler and the second one is to move platform_init_ap to : > slightly later in the initialization sequence. : > : > The patches are : : > 1. mips-ule-support.patch : > - Enable ULE scheduler for MIPS : > : > 2. mips-smp-move-platform.patch : > - We need a hook to setup message ring and its interrupts, we use : > platform_init_ap now, and move it be called later for XLR : > : = : I would like to see us move away from #ifdef TARGET_FOO in files unde= r : mips/mips as much as possible. I think that is achieved easily in thi= s : instance. : = : How about we create a function platform_ap_enable_interrupts() that i= s : called from smp_init_secondary()? This function will be defined for : each platform. : = : In the common case this function will simply call : mips_ap_enable_interrupts() that encapsulates the logic that we : currently have to setup interrupt masks for clock and ipi interrupts : in the status register. In the XLR case however it can do something : different. : = : Ditto about the #ifdef TARGET_XLR_XLS in mpboot.S. You can simply hav= e : an empty platform_init_ap() for XLR. In general, I'd love to see the TARGET_xxx stuff die entirely in the mips tree. That's a hack inherited from the Octeon port. There's other mechanisms that are better suited for this stuff... Warner : best : Neel : = : > 3. rmi-prid.patch : > Add RMI processor ID prints - right now it prints unknown processor= .= : > : > 4. rmi-pcib-fix.patch : > XLR pci bridge should be off the IODI bus, currently it is off the : > nexus bus which will cause issue when bus_setup_intr on nexus is : > called : > : > 5. cleanup-reorg-cpuid-rge-kx.patch : > This set has a many changes rolled up: : > =A0- clean up rge driver, remove unused code paths and commented co= de : > (long way to go still) : > =A0- xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage= updated : > =A0- fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addre= sses : > here, I have made changes for using XKPHYS to do the same process. : > =A0- the KX bit is enabled and interrupts are disabled before the : > physical memory is read, to avoid the KX bit setting from affecting= : > other code. : > - move message ring code to on_chip.c from xlr_machdep.c, now all : > message ring code is in on_chip.c : > : > 6. rmi-xlr-smp.patch : > SMP Support for XLR : > - mpwait.S added to get the other CPUs out of bootloader code at st= artup. : > - SMP platform related functions. : > - disable shared TLB code for SMP - since threads have different ma= ppings. : > : > The patches are also available at : > http://sites.google.com/site/cjayachandran/files : > : > Enabling all 32 threads will need a minor fix in the SMP code, see = the : > patch subr_smp-fix.patch at the same place. : > : > Regards, : > JC. : > : _______________________________________________ : freebsd-mips@freebsd.org mailing list : http://lists.freebsd.org/mailman/listinfo/freebsd-mips : To unsubscribe, send any mail to "freebsd-mips-unsubscribe@freebsd.or= g" : = : = From owner-freebsd-mips@FreeBSD.ORG Wed Apr 21 04:54:11 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF261106564A for ; Wed, 21 Apr 2010 04:54:10 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-pz0-f191.google.com (mail-pz0-f191.google.com [209.85.222.191]) by mx1.freebsd.org (Postfix) with ESMTP id B26468FC16 for ; Wed, 21 Apr 2010 04:54:10 +0000 (UTC) Received: by pzk29 with SMTP id 29so5771179pzk.29 for ; Tue, 20 Apr 2010 21:54:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=mLNHXK9f2hhBTH9qpDCoLSa3M/FOYa42maRKMnufPCA=; b=qKKc+9V3nVGseY9hwUWxMdVodoK/k2oFxGaDAmdzQ5DfOR/HWsppq+yAxF/jAFQEJt KCmFO7PItoptlpi2+yJ5toWT5BdQAaj2+nOg5POngJDF1Oivn9ymHCnLQhGZcAyjIYAR ryJdAmgKWGoPjZEWCFo1ItKvaHQV1gNYBWJ1g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=u+uSmh6B1o55a0q7qVWAws0eCHotkk9Yn4yxlY3bNstL4bzuxBNwj3946JLSIZtp2K 5VCMPMKq73ulZtURDgh7hODe4KBIiBYWVTQ6Lhl0zdFsW6amSSxo71cswTAVerWsuaEK KNNouEq5z7P3iquCbEA9+uKbe/3fvVBtTE0No= MIME-Version: 1.0 Received: by 10.141.29.15 with HTTP; Tue, 20 Apr 2010 21:54:09 -0700 (PDT) In-Reply-To: <20100420.155633.1075901050443800973.imp@bsdimp.com> References: <20100420.155633.1075901050443800973.imp@bsdimp.com> Date: Wed, 21 Apr 2010 10:24:09 +0530 Received: by 10.141.214.26 with SMTP id r26mr2186320rvq.225.1271825650126; Tue, 20 Apr 2010 21:54:10 -0700 (PDT) Message-ID: From: "C. Jayachandran" To: "M. Warner Losh" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Apr 2010 04:54:11 -0000 On Wed, Apr 21, 2010 at 3:26 AM, M. Warner Losh wrote: > In message: > =A0 =A0 =A0 =A0 =A0 =A0Neel Natu writes: > : Hi JC, > : > : On Sat, Apr 17, 2010 at 3:40 PM, C. Jayachandran > : wrote: > : > I've a set of initial patches to enable SMP for RMI processors. It > : > comes up in multi-user with 32 CPUs. =A0I could do buildworld before = I > : > updated to HEAD - with head there is a hang during buildworld which > : > I'm looking at, but I think the initial work can be checked in. > : > > : > Neel, can you have a look at the first two patches - one is to enable > : > ULE scheduler and the second one is to move platform_init_ap to > : > slightly later in the initialization sequence. > : > > : > The patches are : > : > 1. mips-ule-support.patch > : > - Enable ULE scheduler for MIPS > : > > : > 2. mips-smp-move-platform.patch > : > - We need a hook to setup message ring and its interrupts, we use > : > platform_init_ap now, and move it be called later for XLR > : > > : > : I would like to see us move away from #ifdef TARGET_FOO in files under > : mips/mips as much as possible. I think that is achieved easily in this > : instance. > : > : How about we create a function platform_ap_enable_interrupts() that is > : called from smp_init_secondary()? This function will be defined for > : each platform. > : > : In the common case this function will simply call > : mips_ap_enable_interrupts() that encapsulates the logic that we > : currently have to setup interrupt masks for clock and ipi interrupts > : in the status register. In the XLR case however it can do something > : different. > : > : Ditto about the #ifdef TARGET_XLR_XLS in mpboot.S. You can simply have > : an empty platform_init_ap() for XLR. > > In general, I'd love to see the TARGET_xxx stuff die entirely in the > mips tree. =A0That's a hack inherited from the Octeon port. =A0There's > other mechanisms that are better suited for this stuff... Agree with you and Neel on this here. That explains why status register handling for Octeon is the most common ifdef :) JC. From owner-freebsd-mips@FreeBSD.ORG Wed Apr 21 04:54:18 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 075401065670 for ; Wed, 21 Apr 2010 04:54:18 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id CF1CE8FC08 for ; Wed, 21 Apr 2010 04:54:17 +0000 (UTC) Received: by pwi9 with SMTP id 9so4973360pwi.13 for ; Tue, 20 Apr 2010 21:54:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=HdpwhG5aKOJF84urktL54veWzmtGgEzmqg+1fLUDLjM=; b=JT9T14yZl9DK/89LsVZmeLbOqKTj4zHNUZaIBQ89uvdu5zS00Tnm+3UKbpYoqMbSiX Kf0Ta/ghhOATo01wDwRXPDwIKMetqSOM/jYuqKdMTM8Y50k/nD4dMaaFAUHxC99lZxub JjGYvV8FpW1pkG4uhcSs5kMiT9v2OaPCFOii4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=NDSsNCqe4d9sdOU8aGZrCecoRRaUp399Tnj8/PY35/OCjRs55YELaieNOAza86Q+Rg l15jixTpDyxwXLRsxE/AhAglWV6+RJTHYyQKDT+S5SIbUlR3PXQ2T2zMUjg1NSyEJAHr u11AC+z3QCYDH1HeprxFad9oiloPWodg93AFc= MIME-Version: 1.0 Received: by 10.141.29.15 with HTTP; Tue, 20 Apr 2010 21:54:16 -0700 (PDT) In-Reply-To: References: Date: Wed, 21 Apr 2010 10:24:16 +0530 Received: by 10.141.90.11 with SMTP id s11mr2459424rvl.152.1271825657150; Tue, 20 Apr 2010 21:54:17 -0700 (PDT) Message-ID: From: "C. Jayachandran" To: Neel Natu Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Apr 2010 04:54:18 -0000 On Wed, Apr 21, 2010 at 3:06 AM, Neel Natu wrote: > Hi JC, > > On Sat, Apr 17, 2010 at 3:40 PM, C. Jayachandran > wrote: >> I've a set of initial patches to enable SMP for RMI processors. It >> comes up in multi-user with 32 CPUs. =A0I could do buildworld before I >> updated to HEAD - with head there is a hang during buildworld which >> I'm looking at, but I think the initial work can be checked in. >> >> Neel, can you have a look at the first two patches - one is to enable >> ULE scheduler and the second one is to move platform_init_ap to >> slightly later in the initialization sequence. >> >> The patches are : >> 1. mips-ule-support.patch >> - Enable ULE scheduler for MIPS >> >> 2. mips-smp-move-platform.patch >> - We need a hook to setup message ring and its interrupts, we use >> platform_init_ap now, and move it be called later for XLR >> > > I would like to see us move away from #ifdef TARGET_FOO in files under > mips/mips as much as possible. I think that is achieved easily in this > instance. > > How about we create a function platform_ap_enable_interrupts() that is > called from smp_init_secondary()? This function will be defined for > each platform. > > In the common case this function will simply call > mips_ap_enable_interrupts() that encapsulates the logic that we > currently have to setup interrupt masks for clock and ipi interrupts > in the status register. In the XLR case however it can do something > different. My initial thinking was you can just move the platform_init_ap here, The ifdef was added not to break other platforms while this is discussed (and to show what is needed for XLR). Another hook at the same place is fine too - as you said, XLR now has a different intr_machdep.c so even just having mips_ap_enable_interrupts would be fine. I can send you a patch for this. > Ditto about the #ifdef TARGET_XLR_XLS in mpboot.S. You can simply have > an empty platform_init_ap() for XLR. Yes, I think we can go with the existing platform_init_ap() and another interrupt enable hook.. JC. From owner-freebsd-mips@FreeBSD.ORG Fri Apr 23 13:38:18 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D8BA61065670; Fri, 23 Apr 2010 13:38:18 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-pv0-f182.google.com (mail-pv0-f182.google.com [74.125.83.182]) by mx1.freebsd.org (Postfix) with ESMTP id AD4C88FC1E; Fri, 23 Apr 2010 13:38:18 +0000 (UTC) Received: by pva18 with SMTP id 18so392067pva.13 for ; Fri, 23 Apr 2010 06:38:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:cc:content-type; bh=J2L/xxIZjgK5xbS1E83tOhpXDZRs7o9kqBXd0Zz98+Q=; b=buvlsvVa2PZ3LvRwpoGoacrjDohz7BSGJ1tgcdrZaR2CAFqxhfKSLve9JUJ7n1flnk fm05WJieHpupjWkhBU9S+69M6H/uZSVHyRxhc8C4pheBv2AAea4ZJL+6wCw81gxDorWM YvV/90HXwC3HgOGQRsWrr49JqXzZyPkKs1Sks= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:cc:content-type; b=wYcEvi1bmuHtsauVEEQu6KzR1KX26Z0fl9XfkZKy595YVdseSFNIYkMAXCvRHvbVhy Pi7lwEQhAb23CbyOuwhvy0An4m8Z7bzRu29MffQ4rfxEEpUJgJu+58/e5wbveMsFHrNN XBSP7xpnCi5hD9E99pCtIScugmYZ1ffIvMKk8= MIME-Version: 1.0 Received: by 10.141.214.6 with SMTP id r6mr40352rvq.138.1272029898023; Fri, 23 Apr 2010 06:38:18 -0700 (PDT) Received: by 10.141.29.15 with HTTP; Fri, 23 Apr 2010 06:38:17 -0700 (PDT) Date: Fri, 23 Apr 2010 19:08:17 +0530 Message-ID: From: "C. Jayachandran" To: freebsd-mips@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: Crashes with the current -HEAD X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Apr 2010 13:38:18 -0000 I'm seeing crashes like the one below when I do a 'make -j16 buildworld' in the current HEAD. This is without SMP enabled and without any of the XLR smp changes. buildworld used to work on XLR with earlier kernels - the last good kernel I have is around r206712. Any idea what can be causing this? Thanks, JC. xlrboard# Mar 18 14:04:43 xlrboard kernel: BAD_PAGE_FAULT: pid 12596 (cc), uid 1001: pc 0 got a read fault at 0 Mar 18 14:04:43 xlrboard kernel: Trapframe Register Dump: Mar 18 14:04:43 xlrboard kernel: zero: 0 at: 0x1000a1f4 v0: 0x7fffcf28 v1: 0 Mar 18 14:04:43 xlrboard kernel: a0: 0x7fffd638 a1: 0x470bf8 a2: 0x7fffcf28 a3: 0x470c24 Mar 18 14:04:43 xlrboard kernel: t0: 0 t1: 0x4000000 t2: 0 t3: 0 Mar 18 14:04:43 xlrboard kernel: t4: 0 t5: 0 t6: 0 t7: 0 Mar 18 14:04:43 xlrboard kernel: t8: 0 t9: 0 s0: 0x7fffd638 s1: 0 Mar 18 14:04:43 xlrboard kernel: s2: 0x1b s3: 0 s4: 0 s5: 0x7fffcf31 Mar 18 14:04:43 xlrboard kernel: s6: 0x471398 s7: 0x3 k0: 0 k1: 0 Mar 18 14:04:43 xlrboard kernel: gp: 0x1000f630 sp: 0x7fffcec8 s8: 0x7fffcf31 ra: 0x43a200 Mar 18 14:04:43 xlrboard kernel: sr: 0xc013 mullo: 0 mulhi: 0 badvaddr: 0 Mar 18 14:04:43 xlrboard kernel: cause: 0x8 pc: 0 Mar 18 14:04:43 xlrboard kernel: pc address 0 is inaccessible, pde = 0x0, pte = 0x0 Mar 18 14:04:43 xlrboard kernel: BAD_PAGE_FAULT: pid 12601 (as), uid 1001: pc 0x524818 got a read fault at 0x4b90 -- C. Jayachandran c.jayachandran@gmail.com From owner-freebsd-mips@FreeBSD.ORG Fri Apr 23 15:18:27 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EC85E106566C; Fri, 23 Apr 2010 15:18:27 +0000 (UTC) (envelope-from rrs@lakerest.net) Received: from lakerest.net (unknown [IPv6:2001:240:585:2:213:d4ff:fef3:2d8d]) by mx1.freebsd.org (Postfix) with ESMTP id 65F748FC0C; Fri, 23 Apr 2010 15:18:27 +0000 (UTC) Received: from 68.56.242.10.in-addr.arpa (md60f36d0.tmodns.net [208.54.15.214]) (authenticated bits=0) by lakerest.net (8.14.3/8.14.3) with ESMTP id o3NFINai038752 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Fri, 23 Apr 2010 11:18:25 -0400 (EDT) (envelope-from rrs@lakerest.net) Message-Id: <79B7AA80-4D2D-4A5C-973D-4D0FB22E2D85@lakerest.net> From: Randall Stewart To: "C. Jayachandran" In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Fri, 23 Apr 2010 08:18:17 -0700 References: X-Mailer: Apple Mail (2.936) Cc: freebsd-mips@freebsd.org Subject: Re: Crashes with the current -HEAD X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Apr 2010 15:18:28 -0000 JC: My crash is different... hmmm I have not tried one without SMP... need to do that. What I am seeing is a spin lock held to long.. sh /usr/src/lib/ncurses/ncurses/../../../contrib/ncurses/include/ edit_cfg.sh /usr/src/lib/ncurses/ncurses/ncurses_cfg.h term.h.new ** edit: HAVE_TCGETATTR 1 spin lock 0x804c91f4 (smp rendezvous) held by 0xcb943270 (tid 100205) too long panic: spin lock held too long cpuid = 2 KDB: enter: panic ~~ I think this is probably a different issue... It smells like a recursive call to the spin lock... possibly.. but I have not had time to dig too deep into it... and for some reason we don't enter DDB... even though I manually entered it before starting the buildworld to make sure it would work ;-o I will have to find the specific panic and put an enter DDB in there and see if that will work ;-) R On Apr 23, 2010, at 6:38 AM, C. Jayachandran wrote: > I'm seeing crashes like the one below when I do a 'make -j16 > buildworld' in the current HEAD. This is without SMP enabled and > without any of the XLR smp changes. > > buildworld used to work on XLR with earlier kernels - the last good > kernel I have is around r206712. Any idea what can be causing this? > > Thanks, > JC. > > xlrboard# Mar 18 14:04:43 xlrboard kernel: BAD_PAGE_FAULT: pid > 12596 (cc), uid 1001: pc 0 got a read fault at 0 > Mar 18 14:04:43 xlrboard kernel: Trapframe Register Dump: > Mar 18 14:04:43 xlrboard kernel: zero: 0 at: 0x1000a1f4 v0: > 0x7fffcf28 v1: 0 > Mar 18 14:04:43 xlrboard kernel: a0: 0x7fffd638 a1: 0x470bf8 a2: > 0x7fffcf28 a3: 0x470c24 > Mar 18 14:04:43 xlrboard kernel: t0: 0 t1: 0x4000000 t2: 0 t3: 0 > Mar 18 14:04:43 xlrboard kernel: t4: 0 t5: 0 t6: 0 t7: 0 > Mar 18 14:04:43 xlrboard kernel: t8: 0 t9: 0 s0: 0x7fffd638 s1: 0 > Mar 18 14:04:43 xlrboard kernel: s2: 0x1b s3: 0 s4: 0 s5: > 0x7fffcf31 > Mar 18 14:04:43 xlrboard kernel: s6: 0x471398 s7: 0x3 k0: 0 k1: 0 > Mar 18 14:04:43 xlrboard kernel: gp: 0x1000f630 sp: 0x7fffcec8 s8: > 0x7fffcf31 ra: 0x43a200 > Mar 18 14:04:43 xlrboard kernel: sr: 0xc013 mullo: 0 mulhi: > 0 badvaddr: 0 > Mar 18 14:04:43 xlrboard kernel: cause: 0x8 pc: 0 > Mar 18 14:04:43 xlrboard kernel: pc address 0 is inaccessible, pde = > 0x0, pte = 0x0 > Mar 18 14:04:43 xlrboard kernel: BAD_PAGE_FAULT: pid 12601 (as), uid > 1001: pc 0x524818 got a read fault at 0x4b90 > > -- > C. Jayachandran c.jayachandran@gmail.com > ------------------------------ Randall Stewart 803-317-4952 (cell) 803-345-0391(direct) From owner-freebsd-mips@FreeBSD.ORG Fri Apr 23 15:38:16 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 682E7106564A; Fri, 23 Apr 2010 15:38:16 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 331628FC13; Fri, 23 Apr 2010 15:38:15 +0000 (UTC) Received: by pwi9 with SMTP id 9so7129581pwi.13 for ; Fri, 23 Apr 2010 08:38:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=k1jyhzWYkUFVHaZjbNI/4ltXtx3cmiK19WfhsEVmSVA=; b=kCD4JIr+5E4kBdYHNBLgi6+HmMgE3MV5q4eeOCl5ecjqIOqVMrQRa9xtAMg/OzKHVa UxIFn+wEULm9hVTDF0wm1AsPgXApG2xU3ruNosfQvfQCt1CW27CFbkrrnR2Jhq2Qm0jp pANdOWN968PcQdphWofpM0HGc17y/ABUi4vs0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=pALlwyKVCNwKQEh7lzjFJUB3E5cyytWhlO37YaS0PoS1GtmOzGB3GvoXPM3MspxpGn 43VaZnzY6JkSC1rvdt8hgUi9Zwe9jwADQhW14Uxhvvp/6lDqzLLebtsoW8rIfDLHCGSC V0l8gKG+ZpMFJg5PBDs5LxLFBFNE5O5bu0CVo= MIME-Version: 1.0 Received: by 10.141.23.19 with SMTP id a19mr290571rvj.63.1272037095628; Fri, 23 Apr 2010 08:38:15 -0700 (PDT) Received: by 10.140.194.17 with HTTP; Fri, 23 Apr 2010 08:38:15 -0700 (PDT) In-Reply-To: <79B7AA80-4D2D-4A5C-973D-4D0FB22E2D85@lakerest.net> References: <79B7AA80-4D2D-4A5C-973D-4D0FB22E2D85@lakerest.net> Date: Fri, 23 Apr 2010 21:08:15 +0530 Message-ID: From: "C. Jayachandran" To: Randall Stewart Content-Type: multipart/mixed; boundary=000e0cd1b72c39e05f0484e937ab X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-mips@freebsd.org Subject: Re: Crashes with the current -HEAD X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Apr 2010 15:38:16 -0000 --000e0cd1b72c39e05f0484e937ab Content-Type: text/plain; charset=ISO-8859-1 On Fri, Apr 23, 2010 at 8:48 PM, Randall Stewart wrote: > JC: > > My crash is different... hmmm I have not tried one without SMP... need > to do that. > > What I am seeing is a spin lock held to long.. I could fix this, but I hit the crash I reported immediately after that. So I reverted my changes to the HEAD and the crash was still there without the changes. I've attached a fix for the smp_rendezvous hang - in the LMEM macros, the pmap_invalidate_page will call smp_rendezvous with interrupts disabled. If two threads do this it hangs beacuse smp_rendezvous needs interrupts enabled so that it can do IPI. (patch also at http://sites.google.com/site/cjayachandran/files). > sh /usr/src/lib/ncurses/ncurses/../../../contrib/ncurses/include/edit_cfg.sh > /usr/src/lib/ncurses/ncurses/ncurses_cfg.h term.h.new > ** edit: HAVE_TCGETATTR 1 > spin lock 0x804c91f4 (smp rendezvous) held by 0xcb943270 (tid 100205) too > long > panic: spin lock held too long > cpuid = 2 > KDB: enter: panic > ~~ > > I think this is probably a different issue... It smells like a recursive > call > to the spin lock... possibly.. but I have not had time to dig too deep into > it... and for some reason we don't enter DDB... even though I manually > entered it before starting the buildworld to make sure it would > work ;-o > > I will have to find the specific panic and put an enter DDB in there and > see if that will work ;-) JC. --000e0cd1b72c39e05f0484e937ab-- From owner-freebsd-mips@FreeBSD.ORG Fri Apr 23 18:25:46 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A68CF1065673 for ; Fri, 23 Apr 2010 18:25:46 +0000 (UTC) (envelope-from juli@clockworksquid.com) Received: from mail-yw0-f193.google.com (mail-yw0-f193.google.com [209.85.211.193]) by mx1.freebsd.org (Postfix) with ESMTP id 6C8F28FC08 for ; Fri, 23 Apr 2010 18:25:46 +0000 (UTC) Received: by ywh31 with SMTP id 31so4973598ywh.3 for ; Fri, 23 Apr 2010 11:25:45 -0700 (PDT) Received: by 10.101.202.17 with SMTP id e17mr600860anq.73.1272047145419; Fri, 23 Apr 2010 11:25:45 -0700 (PDT) MIME-Version: 1.0 Sender: juli@clockworksquid.com Received: by 10.100.131.16 with HTTP; Fri, 23 Apr 2010 11:25:22 -0700 (PDT) In-Reply-To: References: From: Juli Mallett Date: Fri, 23 Apr 2010 11:25:22 -0700 X-Google-Sender-Auth: 8e4e738a3437e4b7 Message-ID: To: "C. Jayachandran" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org Subject: Re: Crashes with the current -HEAD X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Apr 2010 18:25:46 -0000 JC, It's probably my fault if that's when your last good kernel was. Do you mind trying updating to r206713 then r206714 then r206717 then r206819 and r206834? Or start with r206717 and go earlier or later based on whether it works. If you can figure out what commit was at fault, I'll be glad to work with you to isolate whatever's responsible. In the meantime, I'll try to reproduce and fix this locally. Thanks, Juli. On Fri, Apr 23, 2010 at 06:38, C. Jayachandran w= rote: > I'm seeing crashes like the one below when I do a 'make -j16 > buildworld' in the current HEAD. =A0This is without SMP enabled and > without any of the XLR smp changes. > > buildworld used to work on XLR with earlier kernels - the last good > kernel I have is around r206712. =A0Any idea what can be causing this? > > Thanks, > JC. > > xlrboard# =A0 =A0Mar 18 14:04:43 xlrboard kernel: BAD_PAGE_FAULT: pid > 12596 (cc), uid 1001: pc 0 got a read fault at 0 > Mar 18 14:04:43 xlrboard kernel: Trapframe Register Dump: > Mar 18 14:04:43 xlrboard kernel: zero: 0 =A0 =A0 =A0 =A0at: 0x1000a1f4 = =A0v0: > 0x7fffcf28 =A0v1: 0 > Mar 18 14:04:43 xlrboard kernel: a0: 0x7fffd638 a1: 0x470bf8 =A0 =A0a2: > 0x7fffcf28 =A0a3: 0x470c24 > Mar 18 14:04:43 xlrboard kernel: t0: 0 =A0t1: 0x4000000 =A0 t2: 0 =A0 t3:= 0 > Mar 18 14:04:43 xlrboard kernel: t4: 0 =A0t5: 0 =A0 t6: 0 =A0 t7: 0 > Mar 18 14:04:43 xlrboard kernel: t8: 0 =A0t9: 0 =A0 s0: 0x7fffd638 =A0s1:= 0 > Mar 18 14:04:43 xlrboard kernel: s2: 0x1b =A0 =A0 =A0 s3: 0 =A0 s4: 0 =A0= s5: > 0x7fffcf31 > Mar 18 14:04:43 xlrboard kernel: s6: 0x471398 =A0 s7: 0x3 k0: 0 =A0 k1: 0 > Mar 18 14:04:43 xlrboard kernel: gp: 0x1000f630 sp: 0x7fffcec8 =A0s8: > 0x7fffcf31 =A0ra: 0x43a200 > Mar 18 14:04:43 xlrboard kernel: sr: 0xc013 =A0 =A0 mullo: 0 =A0 =A0 =A0 = =A0mulhi: > 0 =A0 =A0 =A0 =A0badvaddr: 0 > Mar 18 14:04:43 xlrboard kernel: cause: 0x8 =A0 =A0 pc: 0 > Mar 18 14:04:43 xlrboard kernel: pc address 0 is inaccessible, pde =3D > 0x0, pte =3D 0x0 > Mar 18 14:04:43 xlrboard kernel: BAD_PAGE_FAULT: pid 12601 (as), uid > 1001: pc 0x524818 got a read fault at 0x4b90 > > -- > C. Jayachandran =A0 =A0c.jayachandran@gmail.com >