From owner-freebsd-current@FreeBSD.ORG Fri Mar 13 05:32:10 2015 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 52368382; Fri, 13 Mar 2015 05:32:10 +0000 (UTC) Received: from mail-we0-x22e.google.com (mail-we0-x22e.google.com [IPv6:2a00:1450:400c:c03::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D2ACDBF2; Fri, 13 Mar 2015 05:32:09 +0000 (UTC) Received: by wesk11 with SMTP id k11so20889276wes.13; Thu, 12 Mar 2015 22:32:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=8KLLYj6iwg99PUFsvSpbEn7hQe/g4f2e/sQd8PgO+k8=; b=NZOtSHkKyDE+rTQlLEwN0BhJNVdU/lQ0r6kf9GjLbOd55FFx2BI8iv7Z82qQwMKMeP 5LckIrlaEJgNUJKes63A0eI/qj+yyUip3dfaYe7Me9jIbreihJDQpvSwL275XBau0fAt YKKmTA86EMGu8wy1ydFVyYVaAhUlA+0w9KK/XLXG40MonEvJpojZPgbRPBNjPB4lIWJ4 mfGMQzNWTBOoo+bvO1TX4x+KjreVySsupR+gdQvLSQ97uSVLrhyUB+bcjEZvOKsbPGXq vlf7rI/ZZA+6gaZ8SKlZzV1OcUXPNtxec0cd7WPst28Q8gSYza+g6CRfvepJvbVcTYqu ns8A== X-Received: by 10.194.177.132 with SMTP id cq4mr89779969wjc.99.1426224727900; Thu, 12 Mar 2015 22:32:07 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id ax10sm1262674wjc.26.2015.03.12.22.32.05 (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Thu, 12 Mar 2015 22:32:06 -0700 (PDT) Date: Fri, 13 Mar 2015 06:32:03 +0100 From: Mateusz Guzik To: alc@freebsd.org Subject: Re: [PATCH] Convert the VFS cache lock to an rmlock Message-ID: <20150313053203.GC9153@dft-labs.eu> Mail-Followup-To: Mateusz Guzik , alc@freebsd.org, Ryan Stone , FreeBSD Current References: <20150312173635.GB9153@dft-labs.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD Current , Ryan Stone X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Mar 2015 05:32:10 -0000 On Thu, Mar 12, 2015 at 06:13:00PM -0500, Alan Cox wrote: > Below is partial results from a profile of a parallel (-j7) "buildworld" on > a 6-core machine that I did after the introduction of pmap_advise, so this > is not a new profile. The results are sorted by total waiting time and > only the top 20 entries are listed. > Well, I ran stuff on lynx2 in the zoo on fresh -head with debugging disabled (MALLOC_PRODUCTION included) and got quite different results. The machine is Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz 2 package(s) x 10 core(s) x 2 SMT threads 32GB of ram Stuff was built in a chroot with world hosted on zfs. > max wait_max total wait_total count avg wait_avg > cnt_hold cnt_lock name > > 1027 208500 16292932 1658585700 5297163 3 313 0 > 3313855 kern/vfs_cache.c:629 (rw:Name Cache) > > 208564 186514 19080891106 1129189627 355575930 53 3 0 > 1323051 kern/vfs_subr.c:2099 (lockmgr:ufs) > > 169241 148057 193721142 419075449 13819553 14 30 0 > 110089 kern/vfs_subr.c:2210 (lockmgr:ufs) > > 187092 191775 1923061952 257319238 328416784 5 0 0 > 5106537 kern/vfs_cache.c:488 (rw:Name Cache) > make -j 12 buildworld on freshly booted system (i.e. the most namecache insertions): 32 292 3042019 33400306 8419725 0 3 0 2578026 kern/sys_pipe.c:1438 (sleep mutex:pipe mutex) 170608 152572 642385744 27054977 202605015 3 0 0 1306662 kern/vfs_subr.c:2176 (lockmgr:zfs) 66 198 45170221 22523597 161976016 0 0 0 19988525 vm/vm_page.c:1502 (sleep mutex:vm page free queue) 45 413 17804028 20770896 160786478 0 0 0 19368394 vm/vm_page.c:2294 (sleep mutex:vm page free queue) 32 625 19406548 8414459 142554059 0 0 0 11198547 vm/vm_page.c:2053 (sleep mutex:vm active pagequeue) 35 1704 19560396 7867435 142655646 0 0 0 9641161 vm/vm_page.c:2097 (sleep mutex:vm active pagequeue) 14 6675994 27451 6677152 53550 0 124 0 2394 kern/sched_ule.c:2630 (spin mutex:sched lock 23) 2121 879 19982319 4157880 7753007 2 0 0 235477 vm/vm_fault.c:785 (rw:vm object) 27715 1104 9922805 3339829 12614622 0 0 0 83840 vm/vm_map.c:2883 (rw:vm object) 6 2240335 26594 2833529 55057 0 51 0 2643 kern/sched_ule.c:2630 (spin mutex:sched lock 17) 31 22617 1424889 2768798 368786 3 7 0 11555 kern/kern_exec.c:590 (lockmgr:zfs) 7 2027019 26247 2218701 53980 0 41 0 2432 kern/sched_ule.c:2630 (spin mutex:sched lock 5) 57942 153184 41616375 2120917 368786 112 5 0 9438 kern/imgact_elf.c:829 (lockmgr:zfs) 184 557 65168745 1715154 214930217 0 0 0 2104013 kern/vfs_cache.c:487 (rw:Name Cache) ^^^^ name cache only here 3 1695608 26302 1696719 56150 0 30 0 2377 kern/sched_ule.c:2630 (spin mutex:sched lock 18) 52 176 49658348 1606545 199234071 0 0 0 2212598 kern/vfs_cache.c:668 (sleep mutex:vnode interlock) 6 1497134 26337 1583199 55416 0 28 0 2096 kern/sched_ule.c:2630 (spin mutex:sched lock 13) 1705 2155 55312677 1519894 142655701 0 0 0 435090 vm/vm_fault.c:997 (sleep mutex:vm page) 14 721 187832 1449400 2126043 0 0 0 28314 vm/vm_object.c:646 (rw:vm object) 74 62 31785614 1342727 268853124 0 0 0 2235545 kern/vfs_subr.c:2254 (sleep mutex:vnode interlock) So even despite the need for a lot of insertions name cache contention was not a big concern. Here is another buildworld after clearing /usr/obj and resetting stats (no reboot, so the cache was populated): 31 378 3827573 40116363 8544224 0 4 0 2602464 kern/sys_pipe.c:1438 (sleep mutex:pipe mutex) 53 680 45790806 26978449 161004693 0 0 0 21077331 vm/vm_page.c:1502 (sleep mutex:vm page free queue) 39 210 18513457 25286194 160721062 0 0 0 20946513 vm/vm_page.c:2294 (sleep mutex:vm page free queue) 19806 19377 596036045 19086474 202605527 2 0 0 1361883 kern/vfs_subr.c:2176 (lockmgr:zfs) 40 810 19593696 9458254 142544401 0 0 0 11659059 vm/vm_page.c:2053 (sleep mutex:vm active pagequeue) 45 1713 19955926 8883884 142638570 0 0 0 10061154 vm/vm_page.c:2097 (sleep mutex:vm active pagequeue) 15 4702161 28765 4715991 59859 0 78 0 2659 kern/sched_ule.c:2630 (spin mutex:sched lock 6) 1838 1213 20189264 4246013 7751227 2 0 0 243511 vm/vm_fault.c:785 (rw:vm object) 34942 782 10815453 3461181 12611561 0 0 0 87884 vm/vm_map.c:2883 (rw:vm object) 7 2111512 27390 3164572 55775 0 56 0 2239 kern/sched_ule.c:2630 (spin mutex:sched lock 7) 18 2503 1417189 2849233 368785 3 7 0 12099 kern/kern_exec.c:590 (lockmgr:zfs) 52 898 66378192 1861837 214861582 0 0 0 2221482 kern/vfs_cache.c:487 (rw:Name Cache) 16 52 49359798 1685568 199202365 0 0 0 2288836 kern/vfs_cache.c:668 (sleep mutex:vnode interlock) 13 811 190617 1527468 2125719 0 0 0 30154 vm/vm_object.c:646 (rw:vm object) 38 39 31672997 1393102 268812945 0 0 0 2304916 kern/vfs_subr.c:2254 (sleep mutex:vnode interlock) 1714 2111 56782095 1303511 142638594 0 0 0 199781 vm/vm_fault.c:997 (sleep mutex:vm page) 15 765633 28820 1220541 59670 0 20 0 2805 kern/sched_ule.c:2630 (spin mutex:sched lock 8) 177 143 59407392 1213817 58267983 1 0 0 377555 amd64/amd64/pmap.c:5362 (rw:pmap pv list) 37 21 28518097 1199499 291370530 0 0 0 1372317 kern/subr_sleepqueue.c:251 (spin mutex:sleepq chain) 15 809971 29102 1103478 59356 0 18 0 2737 kern/sched_ule.c:2630 (spin mutex:sched lock 19) So, it may be somewthing is wrong with my test environment, but as it is I do not expect namecache lock contention to have significant impact on buildworld/kernel. -- Mateusz Guzik