Date: Wed, 28 Jan 2004 21:15:47 -0800 From: Kris Kennaway <kris@obsecurity.org> To: Kris Kennaway <kris@obsecurity.org> Cc: Kent Stewart <kstewart@owt.com> Subject: Data corruption with 'portsdb -U' Message-ID: <20040129051547.GA38430@xor.obsecurity.org> In-Reply-To: <20040127234305.GA70166@xor.obsecurity.org> References: <004401c3e4cf$647d91e0$c10133ce@dilkie.com> <200401271202.17222.kstewart@owt.com> <20040127213718.GC68505@xor.obsecurity.org> <200401271437.57571.kstewart@owt.com> <20040127234305.GA70166@xor.obsecurity.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--SLDf9lqlvOQaIe6s Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jan 27, 2004 at 03:43:05PM -0800, Kris Kennaway wrote: > > That hasn't always been true. Make index would find 1 or 2 more ports= =20 > > than -U would. There are also times when make index falls flat and -U= =20 > > would still produce a useful INDEX. In addition, portsdb -U used to run= =20 > > quite a bit faster than make index did. >=20 > I'm seeing what look like data corruption from portsdb -U: >=20 > Updating the ports index ... p5-DateTime-Format-HTTP-0.35:"" non-existent= -- dependency list incomplete > ind/Rule.pm:/usr/ports/devel/p5-File-Find-Rule /foobar/lib/perl5/site_per= l/5.6.1/File/Spec.pm:/usr/ports/devel/p5-File-Spec /foobar/bin/perl5.6.1:/u= sr/ports/lang/perl5: malformed entry: ind/Rule.pm:/usr/ports/devel/p5-File-= Find-Rule /foobar/lib/perl5/site_perl/5.6.1/File/Spec.pm:/usr/ports/devel/p= 5-File-Spec /foobar/bin/perl5.6.1:/usr/ports/lang/perl5| >=20 > This may be because make_describe_pass1 is using 'make -j3' to try and > optimize the build, and when I tried a similar trick with 'make index' > on the weekend I also got this kind of corruption. I think it's > happening because the child makes are occasionally overlapping their > I/O, so the parent ends up with a corrupted list where two > neighbouring entries are intertwined. Removing '-j3' from /usr/local/sbin/make_describe_pass1 allows the INDEX build to finish without data corruption (and associated 'errors'). I don't know why portsdb uses -j, because the problems are easily reproduced in my testing (on both 5.x and 4.x). Kris --SLDf9lqlvOQaIe6s Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQFAGJcCWry0BWjoQKURAvD9AJ9OiHYmkvO0RMOS4yujsRMwsA8edgCg5XbM ZbmLzsrwB16lWyUGIbOVshs= =VV65 -----END PGP SIGNATURE----- --SLDf9lqlvOQaIe6s--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20040129051547.GA38430>