From owner-freebsd-mips@FreeBSD.ORG Tue Apr 20 10:49:43 2010 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC4C01065672; Tue, 20 Apr 2010 10:49:43 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-pv0-f182.google.com (mail-pv0-f182.google.com [74.125.83.182]) by mx1.freebsd.org (Postfix) with ESMTP id 716F08FC20; Tue, 20 Apr 2010 10:49:43 +0000 (UTC) Received: by pvc7 with SMTP id 7so3896918pvc.13 for ; Tue, 20 Apr 2010 03:49:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=eJm9g/bEd5lWIogG415DHYk7XQPWF+6o1yvEigmnU1I=; b=RX+JEacD4I1WvOw+pFyhux+3AUC1KmciGgKhfWDGLjV+/J7EPJR3bv86eCJDN+3W91 ItGoUrjjlSZLLbZPs2bYtYSc6lT7gfIh5pELOziWw2ZvozJyGucWlvDrTDJKtMtKxfsh 1gU2y0jSubMd+IeteOts7/KXYtc767vBE0vMw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=iw18uzZy5SGpo/Cd0PY5eqFnCKPjmbGRk+eCmN0d3P00bW7aPf49NXYFAvpLkblRUJ ID0Y3fWF3zvEaAYvyb3DnhDkKY3/D+dYYpcCA//HCzWnP2ir/DuSbq/8xzTaAFW4XSQN XcLRtQre90bS7G1RtV3i1CLq7vd+AGdcfx1pw= MIME-Version: 1.0 Received: by 10.141.29.15 with HTTP; Tue, 20 Apr 2010 03:49:42 -0700 (PDT) In-Reply-To: References: <3BCD65EB-B997-449D-864C-CA24C7B19026@freebsd.org> <6BDB3874-D779-45A6-ABAE-4C331D78A189@lakerest.net> <7BEFA3F5-97AE-477C-9DD3-EF1C4B7DCEB0@freebsd.org> Date: Tue, 20 Apr 2010 16:19:42 +0530 Received: by 10.140.251.8 with SMTP id y8mr297638rvh.231.1271760582896; Tue, 20 Apr 2010 03:49:42 -0700 (PDT) Message-ID: From: "C. Jayachandran" To: Rui Paulo Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2010 10:49:43 -0000 On Tue, Apr 20, 2010 at 4:03 PM, Rui Paulo wrote: > On 20 Apr 2010, at 11:05, Rui Paulo wrote: > >> On 20 Apr 2010, at 10:52, C. Jayachandran wrote: >> >>> On Mon, Apr 19, 2010 at 7:27 PM, C. Jayachandran >>> wrote: >>>> I have a possible cause for the panic with invariants - we should not >>>> schedule the msgring threads unless the smp is completely up. I guess >>>> we start getting message ring interrupts on before the message ring >>>> threads can be scheduled. =A0I am trying out some changes for this - >>>> will send you a patch if this fixes it. >>> >>> I've attached a patch that should fix the issue. The cause was the way >>> message ring threads are started on individual cores and the way >>> interrupts are enabled in the core. =A0I've moved starting message ring >>> threads on other cpus to be a SYSINIT after SMP is started. =A0I'd >>> thought originally that it was due to some clash with the changes in >>> HEAD - but looks like I was completely off-track there. >>> >>> Please let me know if you don't get multi-user with 32 cpus with this >>> patch. There is still the original hang in buildworld, but that should >>> be a bug elsewhere >>> >>> I have a copy at http://sites.google.com/site/cjayachandran/files too >> >> This works perfectly, thanks! > > On further inspection, I noticed that the load avg is now 7. > > last pid: =A01613; =A0load averages: =A06.99, =A06.97, =A06.08 =A0 =A0up = 0+00:30:11 =A010:32:48 > 108 processes: 40 running, 24 sleeping, 44 waiting > CPU: =A00.0% user, =A00.0% nice, 21.9% system, =A00.0% interrupt, 78.1% i= dle > Mem: 8444K Active, 6028K Inact, 37M Wired, 308K Cache, 6800K Buf, 3190M F= ree > Swap: > > =A0PID USERNAME =A0THR PRI NICE =A0 SIZE =A0 =A0RES STATE =A0 C =A0 TIME = =A0 WCPU COMMAND > =A0 10 root =A0 =A0 =A0 32 171 ki31 =A0 =A0 0G =A0 =A0 0G CPU0 =A0 =A00 2= 63:26 2500.00% idle > =A0 17 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU12 =A0= 2 =A0 0:00 100.00% msg_intr12 > =A0 15 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU4 =A0 = =A02 =A0 0:00 100.00% msg_intr4 > =A0 16 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU8 =A0 = =A02 =A0 0:00 100.00% msg_intr8 > =A0 20 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU24 =A0= 1 =A0 0:00 100.00% msg_intr24 > =A0 19 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU20 =A0= 1 =A0 0:00 100.00% msg_intr20 > =A0 21 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU28 =A0= 1 =A0 0:00 100.00% msg_intr28 > =A0 18 root =A0 =A0 =A0 =A01 -16 =A0 =A0- =A0 =A0 0K =A0 =A0 0G CPU16 =A0= 1 =A0 0:00 100.00% msg_intr16 > > What are these msg_intrXX kprocs doing? They should really be sleeping unless there is a lot of network traffic :) The msg_intr threads are interrupt handlers which we run one per core, in the first thread of each core. They were modelled after interrupt threads (in FreeBSD 6). This should be sleeping until there is a message ring interrupt (which tells us that an IO has send data to our core over the message ring). Thanks for the report - I will look at the sleep logic. JC.