Date: Wed, 23 Nov 2005 20:02:29 -0500 From: Kris Kennaway <kris@obsecurity.org> To: Sam Lawrance <boris@brooknet.com.au> Cc: current@freebsd.org, Kris Kennaway <kris@obsecurity.org> Subject: Re: 4BSD process starvation during I/O Message-ID: <20051124010229.GA11953@xor.obsecurity.org> In-Reply-To: <1686B643-388A-4848-9683-AEDF3B7F60FF@brooknet.com.au> References: <20051123201837.GA4822@xor.obsecurity.org> <1686B643-388A-4848-9683-AEDF3B7F60FF@brooknet.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
--SLDf9lqlvOQaIe6s
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Thu, Nov 24, 2005 at 11:54:05AM +1100, Sam Lawrance wrote:
>=20
> On 24/11/2005, at 7:18 AM, Kris Kennaway wrote:
>=20
> >I have noticed that when multiple identical processes (e.g. gtar, or
> >dd) are run on 4BSD, when there are N CPUs on a machine there will be
> >N processes that run with a higher CPU share than all the others. As
> >a result, these N processes finish first, then another N, and so on.
> >
> >This is true under both 4.11 and 6.0 (so in that sense it's not so
> >surprising), but the effect is much more pronounced on 6.0 (which may
> >be possible to fix).
> >
> >Here are the exit times for 6 identical gtar processes (and same 4.11
> >gtar binary) started together on a 2-CPU machine:
> >
> >6.0:
> >
> >1132776233
> >1132776235
> >1132776264
> >1132776265
> >1132776279
> >1132776279
> > 238.86 real 10.87 user 166.00 sys
> >
> >You can see they finish in pairs, and there's a spread of 46 seconds
> >from first to last.
> >
> >On 4.11:
> >
> >1132775426
> >1132775429
> >1132775431
> >1132775432
> >1132775448
> >1132775449
> > 275.56 real 0.43 user 336.26 sys
> >
> >They also finish in pairs, but the spread is half, at 23 seconds.
> >
> >This seems to be correlated to the rate at which the processes perform
> >I/O. On a quad amd64 machine running 6.0 when I run multiple dd
> >processes at different offsets in a md device:
> >
> >268435456 bytes transferred in 1.734285 secs (154781618 bytes/sec)
> >268435456 bytes transferred in 1.737857 secs (154463501 bytes/sec)
> >268435456 bytes transferred in 1.751760 secs (153237575 bytes/sec)
> >268435456 bytes transferred in 3.263460 secs (82254865 bytes/sec)
> >268435456 bytes transferred in 3.295294 secs (81460244 bytes/sec)
> >268435456 bytes transferred in 3.349770 secs (80135487 bytes/sec)
> >268435456 bytes transferred in 4.716637 secs (56912467 bytes/sec)
> >268435456 bytes transferred in 4.850927 secs (55336941 bytes/sec)
> >268435456 bytes transferred in 4.953528 secs (54190760 bytes/sec)
> >
> >They finish in groups of 3 here since the 4th CPU is being used to
> >drive the md worker thread (which takes up most of the CPU). In this
> >case the first 3 dd processes get essentially 100% of the CPU, and the
> >rest get close to 0% until those 3 processes finish.
> >
> >Perhaps this can be tweaked.
> >
>=20
> I tried this on a dual Xeon, with 12 processes like
>=20
> mdconfig -a -t swap -s 320m
> dd if=3D/dev/md0 of=3D1 bs=3D1m skip=3D0 count=3D40 &
> dd if=3D/dev/md0 of=3D2 bs=3D1m skip=3D40 count=3D40 &
You're reading from the md, not writing to it. Sorry if that wasn't
clear.
My test is:
#!/bin/sh
mdconfig -d -u 0
mdconfig -a -t swap -s 16g
for i in `jot $1 1`; do
dd if=3D/dev/zero of=3D/dev/md0 seek=3D$(($i*16384)) count=3D16384 =
bs=3D16k > /dev/null &
done
wait
Kris
--SLDf9lqlvOQaIe6s
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (FreeBSD)
iD8DBQFDhRElWry0BWjoQKURAvZTAJ4ujEdmqrD+cBVhAJfFlfb+zurHTQCgjkMF
I0nwZfPgQGR9zs1vbGip7jQ=
=mcvC
-----END PGP SIGNATURE-----
--SLDf9lqlvOQaIe6s--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20051124010229.GA11953>