Date: Thu, 7 Dec 2006 19:09:32 -0500 From: Kris Kennaway <kris@obsecurity.org> To: Dieter <freebsd@sopwith.solgatos.com> Cc: freebsd-questions@freebsd.org, Kris Kennaway <kris@obsecurity.org> Subject: Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output ) Message-ID: <20061208000932.GA35387@xor.obsecurity.org> In-Reply-To: <200612072321.XAA23385@sopwith.solgatos.com> References: <20061123192011.GA81772@xor.obsecurity.org> <200612072321.XAA23385@sopwith.solgatos.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--NzB8fVQJ5HfG6fxh Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Dec 07, 2006 at 03:21:46PM +0000, Dieter wrote: > > > > > > > hw.ata.wc=3D3D3D3D0 > > > > > > ^^^^^^^^^^^ > > > > > > "Make my hard drive go reeeeally slow please (just in case I cr= ash)=3D > > " :) > > > > >=3D3D20 > > > > > Slower, yes, but not *that* slow. > > > > >=3D3D20 > > > > > Normal ls : 0.032 second. Two processes using same disk, multipl= y by=3D > > two, > > > > > so 0.064 second. Maybe the multiplier is more than 2, call it 10= x, so > > > > > 0.32 second. But I'm seeing a factor of over 9100x. > > > >=3D20 > > > > Humour me and turn it back on, then see what happens. > > >=3D20 > > > Where is the knob to turn the write cache on/off on a per-drive basis > > > in FreeBSD? I can do this in NetBSD, but the only knob I can find in > > > FreeBSD affects all drives, and requires a reboot. > >=20 > > Yes, I think you need to do it globally at boot time. > >=20 > > > Humour me and read the Subject line. The ls does not get its fair sh= are > > > of disk I/O. > > >=3D20 > > > Both times are with the disk's write cache in write-through mode. > > > I'm not comparing times with the write cache in different modes. > > > I'm comparing ls by itself against ls competing with cp. > >=20 > > Your cp is going to be running synchronously, i.e. spend a lot of time > > waiting on the disk to perform the writes. This may well be the cause > > of your problem. Once we have established whether or not it is the > > cause, we can proceed to whether this behaviour can be improved. >=20 > I submitted PR 106340 asking for a way to control the disk write cache on > a per disk basis like NetBSD can. Meanwhile, I added a PATA via USB disk, > which judging from the write speed, appears to be immune from hw.ata.wc= =3D0. >=20 > So I now have a disk which has the write cache on, is connected via a dif= ferent > controller, and thus uses a different device driver. >=20 > I still see the same problems. Writing to one disk *significantly* slows= down > writing to another disk. Even if one process is at normal default priori= ty > and the other is running at rtprio 5. Regardless of which process uses t= he > USB disk and which uses the direct-to-chipset disk. Even if the rtprio 5 > process only needs a very small fraction of the disk bandwidth, it still = gets > slowed down to the point that data is lost. >=20 > My current SWAG is that writing to a disk requires some spl/mutex/lock th= at > is global across all disks on the system. And this spl/mutex/lock is a > bottleneck. In the case of USB devices, yes - all USB accesses require Giant so all USB I/O is serialized. This isn't true in general though, unless you have debug.mpsafevfs=3D0 set (or forced because of something else, e.g. quotas). If this is set then all filesystem I/O is serialized (and maybe it's even worse, if there are also device drivers in the I/O path that also require Giant, like USB). However, I don't know what you mean by "data is lost". Data should never be lost from the filesystem regardless of how slow the I/O is happening, unless there's something else going wrong (e.g. driver bug). Also, rtprio should not be used in general - see the manpage. Were you using rtprio in your original scenario? It can easily cause resource starvation. Kris --NzB8fVQJ5HfG6fxh Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (FreeBSD) iD8DBQFFeK08Wry0BWjoQKURAkA2AJsE9ZrmVJN05LZWo2glkLm8z1zUAACg3lbU jr8v3uVFEYGOsPeBpWC+Eos= =hxh2 -----END PGP SIGNATURE----- --NzB8fVQJ5HfG6fxh--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20061208000932.GA35387>