Date: Sat, 19 Jul 2008 16:12:36 +0200 From: Lars Engels <lme@FreeBSD.org> To: Jeff Roberson <jroberson@jroberson.net> Cc: arch@freebsd.org Subject: Re: witness performance improvements Message-ID: <20080719141236.GF56464@e.0x20.net> In-Reply-To: <20080718163231.B954@desktop> References: <20080718163231.B954@desktop>
next in thread | previous in thread | raw e-mail | index | archive | help
--7iMSBzlTiPOCCT2k Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 18, 2008 at 04:41:58PM -1000, Jeff Roberson wrote: > Hello, >=20 > I have a patch that improves witness performance available at: >=20 > http://people.freebsd.org/~jeff/witness.diff >=20 > This improvement comes at the cost of some significant space overhead. > It changes the witness graph from a linked tree to a matrix based > approach. Relationships can be quickly resolved with a table lookup. > The table size is WITNESS_COUNT^2, or 1MB with the current count of > 1024. >=20 > This patch also makes struct witness objects persistent even after the > last lock using this name has been removed. This is helpful for short > lived objects which may be created frequently. >=20 > To reduce lock contention on SMP witness_checkorder() now runs without > the w_mtx when there are no lock violations. I also cache a > lock_list_entry in each thread as allocating these requires the w_mtx. > The entry is disposed of at thread_exit(). >=20 > There is also a new sysctl that produces dot output which graphs lock > order relationships with the graphviz program. >=20 > Most of this work was done by Ilya Maykov while he was at Isilon > systems. The locking work and some cleanup/porting/refinement was done > by me on behalf of Nokia. >=20 > The performance improvement can be significant. It is only on the > order of 10-20% for buildkernel but on a packet forwarding test at > nokia it sped things up by 5x putting a witness enabled kernel within > about 50% of the performance of a kernel without. I believe > buildworld isn't helped as much because forking and exiting a lot > would then contend on the witness lock. >=20 > I'm mostly interested in hearing what people have to say about the > space bloat. I believe it is in a commit ready state. >=20 > Thanks, > Jeff The speed improvement is significant here (Core Duo Machine). The kernel build time went from ~8:30 min to 5:30 min.=20 But when I run sysctl -a the kernel panics. kgdb output: Unread portion of the kernel message buffer: panic: Assertion mcount =3D=3D fcount failed at /usr/src/sys/kern/subr_witness.c:2882 cpuid =3D 1 KDB: enter: panic panic: from debugger cpuid =3D 1 KDB: stack backtrace: Physical memory: 1002 MB Dumping 68 MB: 53 37 21 5 (kgdb) bt #0 doadump () at pcpu.h:196 #1 0xc0476c69 in db_fncall (dummy1=3D-1061959744, dummy2=3D0, dummy3=3D3, dummy4=3D0xe54ce8ec "") at /usr/src/sys/ddb/db_command.c:516 #2 0xc0477218 in db_command (last_cmdp=3D0xc099a9b0, cmd_table=3D0x0, dopager=3D1) at /usr/src/sys/ddb/db_command.c:413 #3 0xc047734a in db_command_loop () at /usr/src/sys/ddb/db_command.c:466 #4 0xc0478b3d in db_trap (type=3D3, code=3D0) at /usr/src/sys/ddb/db_main.c:228 #5 0xc0618d56 in kdb_trap (type=3D3, code=3D0, tf=3D0xe54cea94) at /usr/src/sys/kern/subr_kdb.c:534 #6 0xc087ec36 in trap (frame=3D0xe54cea94) at /usr/src/sys/i386/i386/trap.c:683 #7 0xc08634bb in calltrap () at /usr/src/sys/i386/i386/exception.s:165 #8 0xc0618eda in kdb_enter (why=3D0xc08dc49c "panic", msg=3D0xc08dc49c "panic") at cpufunc.h:60 #9 0xc05ebe9c in panic (fmt=3D0xc08d72ed "Assertion %s failed at %s:%d") at /usr/src/sys/kern/kern_shutdown.c:556 #10 0xc062bf62 in sysctl_debug_witness_cyclegraph (oidp=3D0xc096bec0, arg1=3D0x0, arg2=3D0, req=3D0xe54ceba4) at /usr/src/sys/kern/subr_witness.c:2882 #11 0xc05f59d7 in sysctl_root (oidp=3DVariable "oidp" is not available. ) at /usr/src/sys/kern/kern_sysctl.c:1325 #12 0xc05f5b05 in userland_sysctl (td=3D0xc49c08c0, name=3D0xe54cec10, namelen=3D3, old=3D0x0, oldlenp=3D0xbfbfd9b0, inkernel=3D0, new=3D0x0, newl= en=3D0, retval=3D0xe54cec70, flags=3D0) at /usr/src/sys/kern/kern_sysctl.c:1420 #13 0xc05f5f4c in __sysctl (td=3D0xc49c08c0, uap=3D0xe54cecf8) at /usr/src/sys/kern/kern_sysctl.c:1355 #14 0xc087e3b3 in syscall (frame=3D0xe54ced38) at /usr/src/sys/i386/i386/trap.c:1081 #15 0xc0863520 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:261 #16 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) The kernel sources are one week old. --7iMSBzlTiPOCCT2k Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkiB9lQACgkQKc512sD3afh1XgCfUuP0qvBLuE+GYN1lF3iEdkxU wmYAoIP2aIDjccTcVViLbY09WUWZsiaL =kD/c -----END PGP SIGNATURE----- --7iMSBzlTiPOCCT2k--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080719141236.GF56464>