From owner-svn-src-head@freebsd.org Thu Oct 29 20:26:05 2015 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34996A217C8; Thu, 29 Oct 2015 20:26:05 +0000 (UTC) (envelope-from tijl@freebsd.org) Received: from mailrelay105.isp.belgacom.be (mailrelay105.isp.belgacom.be [195.238.20.132]) by mx1.freebsd.org (Postfix) with ESMTP id EC69C1399; Thu, 29 Oct 2015 20:26:03 +0000 (UTC) (envelope-from tijl@freebsd.org) X-Belgacom-Dynamic: yes X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CXAgAqgDJW/zhosVtegzZTYA+/LAENgVohhXgCgTM5FAEBAQEBAQGBCoQ2AQEEOhwjEAsUBAklDyoeBhOINAEIxUYBAQEBAQEBAwEBAQEBAQEYBIt1hRAHhC0FlkOFHYJwhRBjm1sfAQFCghEdgVc9NIV+AQEB Received: from 56.104-177-91.adsl-dyn.isp.belgacom.be (HELO kalimero.tijl.coosemans.org) ([91.177.104.56]) by relay.skynet.be with ESMTP; 29 Oct 2015 21:25:55 +0100 Received: from kalimero.tijl.coosemans.org (kalimero.tijl.coosemans.org [127.0.0.1]) by kalimero.tijl.coosemans.org (8.15.2/8.15.2) with ESMTP id t9TKPtdb001413; Thu, 29 Oct 2015 21:25:55 +0100 (CET) (envelope-from tijl@FreeBSD.org) Date: Thu, 29 Oct 2015 21:25:54 +0100 From: Tijl Coosemans To: Jeff Roberson Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org, kib@FreeBSD.org Subject: Re: svn commit: r289279 - in head/sys: kern vm Message-ID: <20151029212554.799f76eb@kalimero.tijl.coosemans.org> In-Reply-To: <201510140210.t9E2A79H056595@repo.freebsd.org> References: <201510140210.t9E2A79H056595@repo.freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2015 20:26:05 -0000 On Wed, 14 Oct 2015 02:10:07 +0000 (UTC) Jeff Roberson wrote: > Author: jeff > Date: Wed Oct 14 02:10:07 2015 > New Revision: 289279 > URL: https://svnweb.freebsd.org/changeset/base/289279 > > Log: > Parallelize the buffer cache and rewrite getnewbuf(). This results in a > 8x performance improvement in a micro benchmark on a 4 socket machine. > > - Get buffer headers from a per-cpu uma cache that sits in from of the > free queue. > - Use a per-cpu quantum cache in vmem to eliminate contention for kva. > - Use multiple clean queues according to buffer cache size to eliminate > clean queue lock contention. > - Introduce a bufspace daemon that attempts to prevent getnewbuf() callers > from blocking or doing direct recycling. > - Close some bufspace allocation races that could lead to endless > recycling. > - Further the transition to a more modern style of small functions grouped > by prefix in order to improve growing complexity. I have an i386 system that locks up easily after this commit. Booting into single user and running make installkernel triggers it consistently. I haven't been able to reproduce it on amd64. Examining threads with DDB shows that they are all at sched_switch (a few at fork_trampoline). The only lock being held is Giant (by the interrupt handler for ctrl+alt+esc I think). So it doesn't look like a dead lock. It's more a sleeping beauty situation. All threads in the castle are sleeping and there's no prince to wake them up. (kgdb) info thread Id Target Id Frame These are from make installkernel: 72 Thread 100071 (PID=107: install) sched_switch (td=0xc667d000, newtd=0xc6407000, flags=) at /usr/src/sys/kern/sched_ule.c:1969 71 Thread 100070 (PID=81: make) sched_switch (td=0xc667d340, newtd=0xc667d000, flags=) at /usr/src/sys/kern/sched_ule.c:1969 70 Thread 100067 (PID=30: make) sched_switch (td=0xc667e000, newtd=0xc667d340, flags=) at /usr/src/sys/kern/sched_ule.c:1969 69 Thread 100066 (PID=25: make) sched_switch (td=0xc667e340, newtd=0xc667e000, flags=) at /usr/src/sys/kern/sched_ule.c:1969 Single user shell: 68 Thread 100065 (PID=17: sh) sched_switch (td=0xc6406000, newtd=0xc667e340, flags=) at /usr/src/sys/kern/sched_ule.c:1969 Kernel threads: 67 Thread 100063 (PID=16: vnlru) sched_switch (td=0xc6406680, newtd=0xc6407340, flags=) at /usr/src/sys/kern/sched_ule.c:1969 66 Thread 100062 (PID=9: syncer) sched_switch (td=0xc64069c0, newtd=0xc667d000, flags=) at /usr/src/sys/kern/sched_ule.c:1969 65 Thread 100061 (PID=8: bufspacedaemon) sched_switch (td=0xc6407000, newtd=0xc62dc000, flags=) at /usr/src/sys/kern/sched_ule.c:1969 64 Thread 100060 (PID=7: bufdaemon) sched_switch (td=0xc6407340, newtd=0xc6408000, flags=) at /usr/src/sys/kern/sched_ule.c:1969 63 Thread 100068 (PID=7: bufdaemon//var worker) sched_switch ( td=0xc667d9c0, newtd=0xc6407000, flags=) at /usr/src/sys/kern/sched_ule.c:1969 62 Thread 100069 (PID=7: bufdaemon//usr worker) sched_switch ( td=0xc667d680, newtd=0xc667d000, flags=) at /usr/src/sys/kern/sched_ule.c:1969 61 Thread 100059 (PID=6: pagezero) sched_switch (td=0xc6407680, newtd=0xc55ba680, flags=) at /usr/src/sys/kern/sched_ule.c:1969 60 Thread 100058 (PID=5: vmdaemon) sched_switch (td=0xc64079c0, newtd=0xc6407340, flags=) at /usr/src/sys/kern/sched_ule.c:1969 59 Thread 100057 (PID=4: pagedaemon) sched_switch (td=0xc6408000, newtd=0xc6407000, flags=) at /usr/src/sys/kern/sched_ule.c:1969 58 Thread 100064 (PID=4: pagedaemon/uma) sched_switch (td=0xc6406340, newtd=0xc55b9340, flags=) at /usr/src/sys/kern/sched_ule.c:1969 57 Thread 100050 (PID=15: acpi_cooling0) sched_switch (td=0xc62dc340, newtd=0xc6407000, flags=) at /usr/src/sys/kern/sched_ule.c:1969 .... Anything else you need to debug this?