From owner-freebsd-arch@FreeBSD.ORG Sat Jul 19 14:12:38 2008
Return-Path: <owner-freebsd-arch@FreeBSD.ORG>
Delivered-To: arch@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
by hub.freebsd.org (Postfix) with ESMTP id 3D17A106566B
for <arch@freebsd.org>; Sat, 19 Jul 2008 14:12:38 +0000 (UTC)
(envelope-from lars@e.0x20.net)
Received: from mail.0x20.net (mail.ipv6.0x20.net [IPv6:2001:aa8:fffb::3])
by mx1.freebsd.org (Postfix) with ESMTP id 987AD8FC19
for <arch@freebsd.org>; Sat, 19 Jul 2008 14:12:37 +0000 (UTC)
(envelope-from lars@e.0x20.net)
Received: by mail.0x20.net (Postfix, from userid 1002)
id 2ACD53A682; Sat, 19 Jul 2008 16:12:36 +0200 (CEST)
Date: Sat, 19 Jul 2008 16:12:36 +0200
From: Lars Engels <lme@FreeBSD.org>
To: Jeff Roberson <jroberson@jroberson.net>
Message-ID: <20080719141236.GF56464@e.0x20.net>
References: <20080718163231.B954@desktop>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
protocol="application/pgp-signature"; boundary="7iMSBzlTiPOCCT2k"
Content-Disposition: inline
In-Reply-To: <20080718163231.B954@desktop>
X-Editor: VIM - Vi IMproved 7.1
X-Operation-System: FreeBSD 5.5-RELEASE-p19
User-Agent: mutt-ng/devel-r804 (FreeBSD)
Cc: arch@freebsd.org
Subject: Re: witness performance improvements
X-BeenThere: freebsd-arch@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
Reply-To: Lars Engels <lme@FreeBSD.org>
List-Id: Discussion related to FreeBSD architecture <freebsd-arch.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-arch>,
<mailto:freebsd-arch-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-arch>
List-Post: <mailto:freebsd-arch@freebsd.org>
List-Help: <mailto:freebsd-arch-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-arch>,
<mailto:freebsd-arch-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 19 Jul 2008 14:12:38 -0000
--7iMSBzlTiPOCCT2k
Content-Type: text/plain; charset=iso-8859-15
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Fri, Jul 18, 2008 at 04:41:58PM -1000, Jeff Roberson wrote:
> Hello,
>=20
> I have a patch that improves witness performance available at:
>=20
> http://people.freebsd.org/~jeff/witness.diff
>=20
> This improvement comes at the cost of some significant space overhead.
> It changes the witness graph from a linked tree to a matrix based
> approach. Relationships can be quickly resolved with a table lookup.
> The table size is WITNESS_COUNT^2, or 1MB with the current count of
> 1024.
>=20
> This patch also makes struct witness objects persistent even after the
> last lock using this name has been removed. This is helpful for short
> lived objects which may be created frequently.
>=20
> To reduce lock contention on SMP witness_checkorder() now runs without
> the w_mtx when there are no lock violations. I also cache a
> lock_list_entry in each thread as allocating these requires the w_mtx.
> The entry is disposed of at thread_exit().
>=20
> There is also a new sysctl that produces dot output which graphs lock
> order relationships with the graphviz program.
>=20
> Most of this work was done by Ilya Maykov while he was at Isilon
> systems. The locking work and some cleanup/porting/refinement was done
> by me on behalf of Nokia.
>=20
> The performance improvement can be significant. It is only on the
> order of 10-20% for buildkernel but on a packet forwarding test at
> nokia it sped things up by 5x putting a witness enabled kernel within
> about 50% of the performance of a kernel without. I believe
> buildworld isn't helped as much because forking and exiting a lot
> would then contend on the witness lock.
>=20
> I'm mostly interested in hearing what people have to say about the
> space bloat. I believe it is in a commit ready state.
>=20
> Thanks,
> Jeff
The speed improvement is significant here (Core Duo Machine). The kernel
build time went from ~8:30 min to 5:30 min.=20
But when I run sysctl -a the kernel panics.
kgdb output:
Unread portion of the kernel message buffer:
panic: Assertion mcount =3D=3D fcount failed at
/usr/src/sys/kern/subr_witness.c:2882
cpuid =3D 1
KDB: enter: panic
panic: from debugger
cpuid =3D 1
KDB: stack backtrace:
Physical memory: 1002 MB
Dumping 68 MB: 53 37 21 5
(kgdb) bt
#0 doadump () at pcpu.h:196
#1 0xc0476c69 in db_fncall (dummy1=3D-1061959744, dummy2=3D0, dummy3=3D3,
dummy4=3D0xe54ce8ec "") at /usr/src/sys/ddb/db_command.c:516
#2 0xc0477218 in db_command (last_cmdp=3D0xc099a9b0, cmd_table=3D0x0,
dopager=3D1) at /usr/src/sys/ddb/db_command.c:413
#3 0xc047734a in db_command_loop () at
/usr/src/sys/ddb/db_command.c:466
#4 0xc0478b3d in db_trap (type=3D3, code=3D0) at
/usr/src/sys/ddb/db_main.c:228
#5 0xc0618d56 in kdb_trap (type=3D3, code=3D0, tf=3D0xe54cea94) at
/usr/src/sys/kern/subr_kdb.c:534
#6 0xc087ec36 in trap (frame=3D0xe54cea94) at
/usr/src/sys/i386/i386/trap.c:683
#7 0xc08634bb in calltrap () at /usr/src/sys/i386/i386/exception.s:165
#8 0xc0618eda in kdb_enter (why=3D0xc08dc49c "panic", msg=3D0xc08dc49c
"panic") at cpufunc.h:60
#9 0xc05ebe9c in panic (fmt=3D0xc08d72ed "Assertion %s failed at %s:%d")
at /usr/src/sys/kern/kern_shutdown.c:556
#10 0xc062bf62 in sysctl_debug_witness_cyclegraph (oidp=3D0xc096bec0,
arg1=3D0x0, arg2=3D0, req=3D0xe54ceba4) at
/usr/src/sys/kern/subr_witness.c:2882
#11 0xc05f59d7 in sysctl_root (oidp=3DVariable "oidp" is not available.
) at /usr/src/sys/kern/kern_sysctl.c:1325
#12 0xc05f5b05 in userland_sysctl (td=3D0xc49c08c0, name=3D0xe54cec10,
namelen=3D3, old=3D0x0, oldlenp=3D0xbfbfd9b0, inkernel=3D0, new=3D0x0, newl=
en=3D0,
retval=3D0xe54cec70, flags=3D0)
at /usr/src/sys/kern/kern_sysctl.c:1420
#13 0xc05f5f4c in __sysctl (td=3D0xc49c08c0, uap=3D0xe54cecf8) at
/usr/src/sys/kern/kern_sysctl.c:1355
#14 0xc087e3b3 in syscall (frame=3D0xe54ced38) at
/usr/src/sys/i386/i386/trap.c:1081
#15 0xc0863520 in Xint0x80_syscall () at
/usr/src/sys/i386/i386/exception.s:261
#16 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
The kernel sources are one week old.
--7iMSBzlTiPOCCT2k
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (FreeBSD)
iEYEARECAAYFAkiB9lQACgkQKc512sD3afh1XgCfUuP0qvBLuE+GYN1lF3iEdkxU
wmYAoIP2aIDjccTcVViLbY09WUWZsiaL
=kD/c
-----END PGP SIGNATURE-----
--7iMSBzlTiPOCCT2k--