Date: Wed, 4 Apr 2007 05:17:13 -0400 (EDT) From: Robert Watson <rwatson@FreeBSD.org> To: current@FreeBSD.org Cc: performance@FreeBSD.org Subject: HEADS UP: filedesc_sx patch in CVS HEAD Message-ID: <20070404051355.P25236@fledge.watson.org>
next in thread | raw e-mail | index | archive | help
I've committed the below to the tree; Kris has performed quite a lot of
performance and stability testing, but since he tends to run with specific
workloads, I wouldn't be surprised if there are minor (and hopefully quickly
corrected) issues reported. If you experience hangs or other problems, please
make sure to run with INVARIANTS and WITNESS, which will help with debugging.
This patch represents a significant part of the performance improvements for
improved scalability on 7-CURRENT with respect to threaded databases, and is
only possible because of the long hours of work Attilio, Kris, John, and
others have put in preparing the sxlock optimizations this patch depends on,
as well as reviewing and testing the patch.
Please let me know if you experience any problems.
Thanks,
Robert N M Watson
Computer Laboratory
University of Cambridge
---------- Forwarded message ----------
Date: Wed, 4 Apr 2007 09:11:34 +0000 (UTC)
From: Robert Watson <rwatson@FreeBSD.org>
To: src-committers@FreeBSD.org, cvs-src@FreeBSD.org, cvs-all@FreeBSD.org
Subject: cvs commit: src/sys/compat/linux linux_file.c src/sys/compat/svr4
svr4_filio.c src/sys/dev/streams streams.c src/sys/fs/devfs
devfs_vnops.c src/sys/fs/fdescfs fdesc_vfsops.c fdesc_vnops.c
src/sys/fs/fifofs fifo_vnops.c src/sys/fs/unionfs union_subr.c ...
rwatson 2007-04-04 09:11:34 UTC
FreeBSD src repository
Modified files:
sys/compat/linux linux_file.c
sys/compat/svr4 svr4_filio.c
sys/dev/streams streams.c
sys/fs/devfs devfs_vnops.c
sys/fs/fdescfs fdesc_vfsops.c fdesc_vnops.c
sys/fs/fifofs fifo_vnops.c
sys/fs/unionfs union_subr.c
sys/kern kern_descrip.c kern_event.c kern_fork.c
subr_witness.c sys_generic.c
uipc_mqueue.c uipc_syscalls.c
uipc_usrreq.c vfs_cache.c vfs_lookup.c
vfs_mount.c vfs_syscalls.c
sys/netsmb smb_dev.c
sys/opencrypto cryptodev.c
sys/security/audit audit_bsm_klib.c
sys/sys filedesc.h
Log:
Replace custom file descriptor array sleep lock constructed using a mutex
and flags with an sxlock. This leads to a significant and measurable
performance improvement as a result of access to shared locking for
frequent lookup operations, reduced general overhead, and reduced overhead
in the event of contention. All of these are imported for threaded
applications where simultaneous access to a shared file descriptor array
occurs frequently. Kris has reported 2x-4x transaction rate improvements
on 8-core MySQL benchmarks; smaller improvements can be expected for many
workloads as a result of reduced overhead.
- Generally eliminate the distinction between "fast" and regular
acquisisition of the filedesc lock; the plan is that they will now all
be fast. Change all locking instances to either shared or exclusive
locks.
- Correct a bug (pointed out by kib) in fdfree() where previously msleep()
was called without the mutex held; sx_sleep() is now always called with
the sxlock held exclusively.
- Universally hold the struct file lock over changes to struct file,
rather than the filedesc lock or no lock. Always update the f_ops
field last. A further memory barrier is required here in the future
(discussed with jhb).
- Improve locking and reference management in linux_at(), which fails to
properly acquire vnode references before using vnode pointers. Annotate
improper use of vn_fullpath(), which will be replaced at a future date.
In fcntl(), we conservatively acquire an exclusive lock, even though in
some cases a shared lock may be sufficient, which should be revisited.
The dropping of the filedesc lock in fdgrowtable() is no longer required
as the sxlock can be held over the sleep operation; we should consider
removing that (pointed out by attilio).
Tested by: kris
Discussed with: jhb, kris, attilio, jeff
Revision Changes Path
1.103 +17 -4 src/sys/compat/linux/linux_file.c
1.35 +4 -4 src/sys/compat/svr4/svr4_filio.c
1.55 +2 -2 src/sys/dev/streams/streams.c
1.143 +3 -1 src/sys/fs/devfs/devfs_vnops.c
1.56 +2 -2 src/sys/fs/fdescfs/fdesc_vfsops.c
1.104 +5 -5 src/sys/fs/fdescfs/fdesc_vnops.c
1.136 +3 -1 src/sys/fs/fifofs/fifo_vnops.c
1.91 +2 -2 src/sys/fs/unionfs/union_subr.c
1.307 +174 -170 src/sys/kern/kern_descrip.c
1.109 +9 -9 src/sys/kern/kern_event.c
1.270 +2 -2 src/sys/kern/kern_fork.c
1.228 +0 -2 src/sys/kern/subr_witness.c
1.155 +11 -12 src/sys/kern/sys_generic.c
1.21 +10 -11 src/sys/kern/uipc_mqueue.c
1.250 +14 -9 src/sys/kern/uipc_syscalls.c
1.201 +10 -9 src/sys/kern/uipc_usrreq.c
1.108 +4 -4 src/sys/kern/vfs_cache.c
1.100 +2 -2 src/sys/kern/vfs_lookup.c
1.252 +2 -2 src/sys/kern/vfs_mount.c
1.436 +26 -25 src/sys/kern/vfs_syscalls.c
1.32 +3 -3 src/sys/netsmb/smb_dev.c
1.33 +3 -1 src/sys/opencrypto/cryptodev.c
1.6 +2 -2 src/sys/security/audit/audit_bsm_klib.c
1.76 +15 -61 src/sys/sys/filedesc.h
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070404051355.P25236>