From owner-freebsd-mips@FreeBSD.ORG Sun Apr 18 12:36:43 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5CDCD1065672 for ; Sun, 18 Apr 2010 12:36:43 +0000 (UTC) (envelope-from rpaulo@freebsd.org) Received: from karen.lavabit.com (karen.lavabit.com [72.249.41.33]) by mx1.freebsd.org (Postfix) with ESMTP id 2FA558FC13 for ; Sun, 18 Apr 2010 12:36:42 +0000 (UTC) Received: from e.earth.lavabit.com (e.earth.lavabit.com [192.168.111.14]) by karen.lavabit.com (Postfix) with ESMTP id A283711B9CC; Sun, 18 Apr 2010 07:10:02 -0500 (CDT) Received: from 10.0.10.2 (54.81.54.77.rev.vodafone.pt [77.54.81.54]) by lavabit.com with ESMTP id VQD7HREK6007; Sun, 18 Apr 2010 07:10:02 -0500 Mime-Version: 1.0 (Apple Message framework v1078) Content-Type: text/plain; charset=us-ascii From: Rui Paulo In-Reply-To: Date: Sun, 18 Apr 2010 13:09:59 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> References: To: C. Jayachandran X-Mailer: Apple Mail (2.1078) Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Apr 2010 12:36:43 -0000 On 17 Apr 2010, at 23:40, C. Jayachandran wrote: > I've a set of initial patches to enable SMP for RMI processors. It > comes up in multi-user with 32 CPUs. I could do buildworld before I > updated to HEAD - with head there is a hang during buildworld which > I'm looking at, but I think the initial work can be checked in. >=20 > Neel, can you have a look at the first two patches - one is to enable > ULE scheduler and the second one is to move platform_init_ap to > slightly later in the initialization sequence. >=20 > The patches are : > 1. mips-ule-support.patch > - Enable ULE scheduler for MIPS >=20 > 2. mips-smp-move-platform.patch > - We need a hook to setup message ring and its interrupts, we use > platform_init_ap now, and move it be called later for XLR >=20 > 3. rmi-prid.patch > Add RMI processor ID prints - right now it prints unknown processor. >=20 > 4. rmi-pcib-fix.patch > XLR pci bridge should be off the IODI bus, currently it is off the > nexus bus which will cause issue when bus_setup_intr on nexus is > called >=20 > 5. cleanup-reorg-cpuid-rge-kx.patch > This set has a many changes rolled up: > - clean up rge driver, remove unused code paths and commented code > (long way to go still) > - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage = updated > - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses > here, I have made changes for using XKPHYS to do the same process. > - the KX bit is enabled and interrupts are disabled before the > physical memory is read, to avoid the KX bit setting from affecting > other code. > - move message ring code to on_chip.c from xlr_machdep.c, now all > message ring code is in on_chip.c >=20 > 6. rmi-xlr-smp.patch > SMP Support for XLR > - mpwait.S added to get the other CPUs out of bootloader code at = startup. > - SMP platform related functions. > - disable shared TLB code for SMP - since threads have different = mappings. >=20 > The patches are also available at > http://sites.google.com/site/cjayachandran/files >=20 > Enabling all 32 threads will need a minor fix in the SMP code, see the > patch subr_smp-fix.patch at the same place. On the netperf XLR, this panics with SCHED_ULE activated: Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights = reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010 = rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpaulo/fr= eebsd/head/sys/XLR mips real memory =3D 3488608256 (3406844K bytes) avail memory =3D 3395207168 (3237MB) FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs nexus0: iodi0 on nexus0 uart0: <16550 or compatible> on iodi0 uart0: [FILTER] uart0: console (38551,n,8,1) pcib0: on iodi0 pci0: on pcib0 pci0: at device 1.0 (no driver attached) pci0: at device 3.0 (no driver attached) rge0 on iodi0 rge0: [ITHREAD] rge0: Ethernet address:=20 rge0: [10Mbps] Total Active Core 8 rge1 on iodi0 rge1: [ITHREAD] rge1: Ethernet address: rge1: [1000Mbps] rge2 on iodi0 rge2: [ITHREAD] rge2: Ethernet address:=20 rge2: [10Mbps] rge3 on iodi0 rge3: [ITHREAD] rge3: Ethernet address:=20 rge3: [10Mbps] Enabling MDIO interrupts Timecounter "MIPS32" frequency 66000000 Hz quality 800 Timecounters tick every 1.000 msec bootpc_init: wired to interface 'rge1' Sending DHCP Discover packet from interface rge1 Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root = path) Sending DHCP Request packet from interface rge1 Sending DHCP Discover packet from interface rge1 DHCP/BOOTP timeout for server 255.255.255.255 DHCP/BOOTP timeout for server 255.255.255.255 DHCP/BOOTP timeout for server 255.255.255.255 Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root = path) Sending DHCP Request packet from interface rge1 DHCP/BOOTP timeout for server 255.255.255.255 Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root = path) rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org boot = file /zoo/rpaulo/lama1/boot/kernel/kernel subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs = /zoo/rpaulo/lama1 rootopts nolockd=20 Adjusted interface rge1 panic: mutex sched lock 4 not owned at = /usr/home/rpaulo/freebsd/head/sys/kern/sched_ule.c:2054 cpuid =3D 4 KDB: enter: panic Regards, -- Rui Paulo