From owner-svn-src-head@freebsd.org Fri Oct 30 21:45:29 2015 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A93E0A22B2C; Fri, 30 Oct 2015 21:45:29 +0000 (UTC) (envelope-from tijl@freebsd.org) Received: from mailrelay112.isp.belgacom.be (mailrelay112.isp.belgacom.be [195.238.20.139]) by mx1.freebsd.org (Postfix) with ESMTP id 66DDE11F0; Fri, 30 Oct 2015 21:45:27 +0000 (UTC) (envelope-from tijl@freebsd.org) X-Belgacom-Dynamic: yes X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2B7AgDA4zNW/zhosVteFoMlU2+/MgENgVojhXYCgTU5FAEBAQEBAQGBCoQ2AQEEOhwjEAsOBgQJJQ8SGB4GE4gbAxYBCL9DDYRCAQEBAQEBAQMBAQEBAQEZBIt1glOGbQWWQ4UdgnCDIoFuY5QOh1EfAQFCghEdgVc9NAEBhXwBAQE Received: from 56.104-177-91.adsl-dyn.isp.belgacom.be (HELO kalimero.tijl.coosemans.org) ([91.177.104.56]) by relay.skynet.be with ESMTP; 30 Oct 2015 22:44:17 +0100 Received: from kalimero.tijl.coosemans.org (kalimero.tijl.coosemans.org [127.0.0.1]) by kalimero.tijl.coosemans.org (8.15.2/8.15.2) with ESMTP id t9ULiGdx001139; Fri, 30 Oct 2015 22:44:17 +0100 (CET) (envelope-from tijl@FreeBSD.org) Date: Fri, 30 Oct 2015 22:44:16 +0100 From: Tijl Coosemans To: Konstantin Belousov Cc: Jeff Roberson , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: Re: svn commit: r289279 - in head/sys: kern vm Message-ID: <20151030224416.51580e0f@kalimero.tijl.coosemans.org> In-Reply-To: <20151029203334.GA2257@kib.kiev.ua> References: <201510140210.t9E2A79H056595@repo.freebsd.org> <20151029212554.799f76eb@kalimero.tijl.coosemans.org> <20151029203334.GA2257@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Oct 2015 21:45:29 -0000 On Thu, 29 Oct 2015 22:33:34 +0200 Konstantin Belousov wrote: > On Thu, Oct 29, 2015 at 09:25:54PM +0100, Tijl Coosemans wrote: >> On Wed, 14 Oct 2015 02:10:07 +0000 (UTC) Jeff Roberson wrote: >>> Author: jeff >>> Date: Wed Oct 14 02:10:07 2015 >>> New Revision: 289279 >>> URL: https://svnweb.freebsd.org/changeset/base/289279 >>> >>> Log: >>> Parallelize the buffer cache and rewrite getnewbuf(). This results in a >>> 8x performance improvement in a micro benchmark on a 4 socket machine. >>> >>> - Get buffer headers from a per-cpu uma cache that sits in from of the >>> free queue. >>> - Use a per-cpu quantum cache in vmem to eliminate contention for kva. >>> - Use multiple clean queues according to buffer cache size to eliminate >>> clean queue lock contention. >>> - Introduce a bufspace daemon that attempts to prevent getnewbuf() callers >>> from blocking or doing direct recycling. >>> - Close some bufspace allocation races that could lead to endless >>> recycling. >>> - Further the transition to a more modern style of small functions grouped >>> by prefix in order to improve growing complexity. >> >> I have an i386 system that locks up easily after this commit. Booting >> into single user and running make installkernel triggers it consistently. >> I haven't been able to reproduce it on amd64. Examining threads with >> DDB shows that they are all at sched_switch (a few at fork_trampoline). >> The only lock being held is Giant (by the interrupt handler for >> ctrl+alt+esc I think). So it doesn't look like a dead lock. It's more >> a sleeping beauty situation. All threads in the castle are sleeping and >> there's no prince to wake them up. >> >> (kgdb) info thread >> Id Target Id Frame >> >> These are from make installkernel: >> >> 72 Thread 100071 (PID=107: install) sched_switch (td=0xc667d000, >> newtd=0xc6407000, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> 71 Thread 100070 (PID=81: make) sched_switch (td=0xc667d340, >> newtd=0xc667d000, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> 70 Thread 100067 (PID=30: make) sched_switch (td=0xc667e000, >> newtd=0xc667d340, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> 69 Thread 100066 (PID=25: make) sched_switch (td=0xc667e340, >> newtd=0xc667e000, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> >> Single user shell: >> >> 68 Thread 100065 (PID=17: sh) sched_switch (td=0xc6406000, >> newtd=0xc667e340, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> >> Kernel threads: >> >> 67 Thread 100063 (PID=16: vnlru) sched_switch (td=0xc6406680, >> newtd=0xc6407340, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> 66 Thread 100062 (PID=9: syncer) sched_switch (td=0xc64069c0, >> newtd=0xc667d000, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> 65 Thread 100061 (PID=8: bufspacedaemon) sched_switch (td=0xc6407000, >> newtd=0xc62dc000, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> 64 Thread 100060 (PID=7: bufdaemon) sched_switch (td=0xc6407340, >> newtd=0xc6408000, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> 63 Thread 100068 (PID=7: bufdaemon//var worker) sched_switch ( >> td=0xc667d9c0, newtd=0xc6407000, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> 62 Thread 100069 (PID=7: bufdaemon//usr worker) sched_switch ( >> td=0xc667d680, newtd=0xc667d000, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> 61 Thread 100059 (PID=6: pagezero) sched_switch (td=0xc6407680, >> newtd=0xc55ba680, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> 60 Thread 100058 (PID=5: vmdaemon) sched_switch (td=0xc64079c0, >> newtd=0xc6407340, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> 59 Thread 100057 (PID=4: pagedaemon) sched_switch (td=0xc6408000, >> newtd=0xc6407000, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> 58 Thread 100064 (PID=4: pagedaemon/uma) sched_switch (td=0xc6406340, >> newtd=0xc55b9340, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> 57 Thread 100050 (PID=15: acpi_cooling0) sched_switch (td=0xc62dc340, >> newtd=0xc6407000, flags=) >> at /usr/src/sys/kern/sched_ule.c:1969 >> .... >> >> Anything else you need to debug this? > > Start with gathering the information listed in > https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html https://people.freebsd.org/~tijl/r289279-dead.txt r290155 doesn't fix it by the way.