Date: Wed, 18 Mar 2015 18:58:13 +0100 From: Mateusz Guzik <mjguzik@gmail.com> To: John Baldwin <jhb@freebsd.org> Cc: alc@freebsd.org, freebsd-current@freebsd.org, Ryan Stone <rysto32@gmail.com> Subject: Re: [PATCH] Convert the VFS cache lock to an rmlock Message-ID: <20150318175812.GA3512@dft-labs.eu> In-Reply-To: <6376695.VOvhinOncy@ralph.baldwin.cx> References: <CAFMmRNysnUezX9ozGrCpivPCTMYRJtoxm9ijR0yQO03LpXnwBQ@mail.gmail.com> <CAJUyCcMoRu7JMCWfYb3acBF=fNopKAV4Ge8-mhApjuJ7ujOqFg@mail.gmail.com> <20150313053203.GC9153@dft-labs.eu> <6376695.VOvhinOncy@ralph.baldwin.cx>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Mar 18, 2015 at 10:17:22AM -0400, John Baldwin wrote:
> On Friday, March 13, 2015 06:32:03 AM Mateusz Guzik wrote:
> > On Thu, Mar 12, 2015 at 06:13:00PM -0500, Alan Cox wrote:
> > > Below is partial results from a profile of a parallel (-j7) "buildworld" on
> > > a 6-core machine that I did after the introduction of pmap_advise, so this
> > > is not a new profile. The results are sorted by total waiting time and
> > > only the top 20 entries are listed.
> > >
> >
> > Well, I ran stuff on lynx2 in the zoo on fresh -head with debugging
> > disabled (MALLOC_PRODUCTION included) and got quite different results.
> >
> > The machine is Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
> > 2 package(s) x 10 core(s) x 2 SMT threads
> >
> > 32GB of ram
> >
> > Stuff was built in a chroot with world hosted on zfs.
> >
> > > max wait_max total wait_total count avg wait_avg
> > > cnt_hold cnt_lock name
> > >
> > > 1027 208500 16292932 1658585700 5297163 3 313 0
> > > 3313855 kern/vfs_cache.c:629 (rw:Name Cache)
> > >
> > > 208564 186514 19080891106 1129189627 355575930 53 3 0
> > > 1323051 kern/vfs_subr.c:2099 (lockmgr:ufs)
> > >
> > > 169241 148057 193721142 419075449 13819553 14 30 0
> > > 110089 kern/vfs_subr.c:2210 (lockmgr:ufs)
> > >
> > > 187092 191775 1923061952 257319238 328416784 5 0 0
> > > 5106537 kern/vfs_cache.c:488 (rw:Name Cache)
> > >
> >
> > make -j 12 buildworld on freshly booted system (i.e. the most namecache insertions):
> >
> > 32 292 3042019 33400306 8419725 0 3 0 2578026 kern/sys_pipe.c:1438 (sleep mutex:pipe mutex)
> > 170608 152572 642385744 27054977 202605015 3 0 0 1306662 kern/vfs_subr.c:2176 (lockmgr:zfs)
>
> You are using ZFS, Alan was using UFS. It would not surprise me that those
> would perform quite differently, and it would not surprise me that UFS is
> more efficient in terms of its interactions with the VM.
>
This is lots of forks + execs and page queue has only one lock.
Fwiw, ufs world backed by dedidacted parition, other conditions unchanged
First build:
52 411 3327572 36528102 8508545 0 4 0 2587337 kern/sys_pipe.c:1438 (sleep mutex:pipe mutex)
308102 308099 430348158 36163333 203246241 2 0 0 459200 kern/vfs_subr.c:2176 (lockmgr:ufs)
78 818 44308162 29968440 165009282 0 0 0 22053854 vm/vm_page.c:1502 (sleep mutex:vm page free queue)
48 238 18872405 28456578 164327020 0 0 0 22512151 vm/vm_page.c:2294 (sleep mutex:vm page free queue)
208 1486 69780863 17484511 144970085 0 0 0 134429 amd64/amd64/pmap.c:4204 (rw:pmap pv global)
52 1263 19390169 8186840 142392234 0 0 0 11151398 vm/vm_page.c:2053 (sleep mutex:vm active pagequeue)
27 1801 19754927 8092312 142403798 0 0 0 9993164 vm/vm_page.c:2097 (sleep mutex:vm active pagequeue)
1145 1300 19872450 7158951 7690094 2 0 0 269930 vm/vm_fault.c:785 (rw:vm object)
25579 1636 10955717 5962336 12620413 0 0 0 96691 vm/vm_map.c:2883 (rw:vm object)
39 54994 1428266 5141221 368787 3 13 0 13391 kern/kern_exec.c:590 (lockmgr:ufs)
311470 311513 67241105 3971357 30997102 2 0 0 5094 kern/vfs_lookup.c:509 (lockmgr:ufs)
55 568 64300400 3821394 214921016 0 0 0 2699822 kern/vfs_cache.c:487 (rw:Name Cache)
14 2224 31486 3784823 266762 0 14 0 43516 amd64/amd64/pmap.c:3767 (rw:pmap pv global)
15 1179 184521 2651974 2126398 0 1 0 31955 vm/vm_object.c:646 (rw:vm object)
43267 54635 56190811 2228815 368787 152 6 0 10399 kern/imgact_elf.c:829 (lockmgr:ufs)
1802 3276 55104622 2042552 142404165 0 0 0 543434 vm/vm_fault.c:997 (sleep mutex:vm page)
29 54 50897307 1792095 199350363 0 0 0 2305326 kern/vfs_cache.c:668 (sleep mutex:vnode interlock)
1051 1252 3640803 1792074 1030592 3 1 0 18897 vm/vm_object.c:1703 (rw:vm object)
17 1389 269455 1778764 2828515 0 0 0 106843 vm/vm_object.c:1222 (rw:vm object)
472 1444 14742782 1731247 6851167 2 0 0 20011 amd64/amd64/pmap.c:3620 (rw:pmap pv global)
reset + rm -r /usr/obj/*
230791 100799 405140208 51092446 203312369 1 0 0 472299 kern/vfs_subr.c:2176 (lockmgr:ufs)
188 8046 69993327 41591950 144943809 0 0 0 175226 amd64/amd64/pmap.c:4204 (rw:pmap pv global)
76 647 45324624 35204587 164788099 0 0 0 22749884 vm/vm_page.c:1502 (sleep mutex:vm page free queue)
40 237 19428445 33209934 164266257 0 0 0 22973810 vm/vm_page.c:2294 (sleep mutex:vm page free queue)
52 298 3038837 32813929 8418613 0 3 0 2587957 kern/sys_pipe.c:1438 (sleep mutex:pipe mutex)
1939 3216 19524902 14623470 7688013 2 1 0 274392 vm/vm_fault.c:785 (rw:vm object)
21696 2861 11773227 13246477 12617163 0 1 0 106342 vm/vm_map.c:2883 (rw:vm object)
54 7664 1445280 10079322 368785 3 27 0 15598 kern/kern_exec.c:590 (lockmgr:ufs)
22 2736 19935005 9087069 142381462 0 0 0 10214061 vm/vm_page.c:2097 (sleep mutex:vm active pagequeue)
36 885 19540659 8825819 142378309 0 0 0 11305041 vm/vm_page.c:2053 (sleep mutex:vm active pagequeue)
15 3045 33404 7922947 267066 0 29 0 48667 amd64/amd64/pmap.c:3767 (rw:pmap pv global)
13 2605 194977 6294921 2126083 0 2 0 36096 vm/vm_object.c:646 (rw:vm object)
82 2269 269861 4506215 2827953 0 1 0 107885 vm/vm_object.c:1222 (rw:vm object)
2179 6501 82846723 4123049 368785 224 11 0 13145 kern/imgact_elf.c:829 (lockmgr:ufs)
81 722 64614191 4099711 214861181 0 0 0 2763484 kern/vfs_cache.c:487 (rw:Name Cache)
3925 2851 3900231 3909417 1031608 3 3 0 22052 vm/vm_object.c:1703 (rw:vm object)
483 3335 14875079 3554944 6850151 2 0 0 24182 amd64/amd64/pmap.c:3620 (rw:pmap pv global)
187 129 103680324 2146404 250760755 0 0 0 1630451 amd64/amd64/pmap.c:5362 (rw:pmap pv list)
30468 2402 126229130 2042062 158088931 0 0 0 71708 vm/vm_object.c:517 (rw:vm object)
28 76 51334143 1892757 199327513 0 0 0 2384837 kern/vfs_cache.c:668 (sleep mutex:vnode interlock)
--
Mateusz Guzik <mjguzik gmail.com>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20150318175812.GA3512>