Date: Thu, 1 Jan 2009 10:51:52 +0100 From: "Simon L. Nielsen" <simon@FreeBSD.org> To: Tim Kientzle <tim@kientzle.com> Cc: Max Laier <max@love2party.net>, Tim Kientzle <kientzle@freebsd.org>, cperciva@freebsd.org, freebsd-current@freebsd.org Subject: Re: tar/libarchive gzip problem [was: portsnap corrupted] Message-ID: <20090101095151.GA1117@arthur.nitro.dk> In-Reply-To: <495C2AF5.6030703@kientzle.com> References: <200812301616.11132.max@love2party.net> <200812302213.07155.max@love2party.net> <20081231094159.GA964@zaphod.nitro.dk> <200812311109.57955.max@love2party.net> <20081231105952.GB964@zaphod.nitro.dk> <495C017F.9000408@freebsd.org> <495C2AF5.6030703@kientzle.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2008.12.31 18:31:17 -0800, Tim Kientzle wrote: > I just committed a fix for this (r186670). Please let > me know if you see any other problems. This seems to fix it, thanks! Max, could you see if you have portsnap problems with the latest libarchive? > Tim Kientzle wrote: > > I think I know what this is. I tried recently to add > > support for concatenated gzip files. This involves looking > > ahead for another GZip header after the end of the compressed > > data. The current version screws this up and ends up trying > > to match the file CRC as a Gzip header. About 1 in 256 files > > will match the first byte, which triggers the subsequent meltdown. > > This also explains why neither of us saw it in testing. > > (I knew the code didn't actually work for concatenated gzip > > files but didn't realize it would break decode of some > > regular non-concatenated files.) > > > > The attached patch simply disables this additional > > header check, which should fix the immediate problem. > > Please try it and let me know. > > > > Apologies, > > > > Tim > > > > Simon L. Nielsen wrote: > > > >> Hey Tim, > >> > >> I think one of the recent changes to tar or libarchive broke gzip > >> handling in some cases. See more below. > >> > >> [portsnap extract fails with gzip error] > >> > >> I'm not sure why I didn't run into it in my tests, but I think the > >> problem is in tar / libarchive's handling of gzip files. Taking one > >> random "broken" file [1] it fails with tar's build in decompression, > >> but works using external zcat. > >> > >> [1] > >> http://portsnap1.freebsd.org/f/19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz > >> > >> http://people.freebsd.org/~simon/tmp/19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz > >> > >> > >> > >> Broken system: > >> > >> [simon@eddie:/tmp] tar --version > >> bsdtar 2.5.903a - libarchive 2.5.903a > >> [simon@eddie:/tmp] uname -a > >> FreeBSD eddie.nitro.dk 8.0-CURRENT FreeBSD 8.0-CURRENT #1: Tue Dec 30 > >> 22:28:33 CET 2008 > >> simon@eddie.nitro.dk:/FreeBSD/obj/FreeBSD/system-CURRENT/sys/EDDIE i386 > >> [simon@eddie:/tmp] tar tvf > >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz > > >> /dev/null > >> tar: Error opening archive: Invalid GZip header (saw 99 at offset 1) > >> [simon@eddie:/tmp] zcat > >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | > >> tar tf - > /dev/null [simon@eddie:/tmp] zcat > >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | > >> sha256 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d > >> > >> > >> OK system: > >> > >> [simon@benji:/tmp] tar --version > >> bsdtar 2.5.5 - libarchive 2.5.5 > >> [simon@benji:/tmp] uname -a > >> FreeBSD benji.s 7.1-RC2 FreeBSD 7.1-RC2 #0: Tue Dec 23 15:18:30 UTC > >> 2008 root@logan.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386 > >> [simon@benji:/tmp] tar tvf > >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz > > >> /dev/null [simon@benji:/tmp] zcat > >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | > >> sha256 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d > >> > >> > >> Another OK system: > >> > >> [simon@ref8-amd64:files] tar --version > >> bsdtar 2.5.5 - libarchive 2.5.5 > >> [simon@ref8-amd64:files] uname -a > >> FreeBSD ref8-amd64.freebsd.org 8.0-CURRENT FreeBSD 8.0-CURRENT #2 > >> r184542:185402: Fri Nov 28 19:14:40 UTC 2008 > >> peter@ref8-amd64.freebsd.org:/scratch/src/sys/amd64/compile/REF8-AMD64 > >> amd64 > >> [simon@ref8-amd64:files] tar tvf > >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz > > >> /dev/null > >> [simon@ref8-amd64:files] zcat > >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | > >> sha256 > >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d > >> > > > > ------------------------------------------------------------------------ > > > > Index: archive_read_support_compression_gzip.c > > =================================================================== > > --- archive_read_support_compression_gzip.c (revision 185679) > > +++ archive_read_support_compression_gzip.c (working copy) > > @@ -428,8 +428,7 @@ > > "Failed to clean up gzip decompressor"); > > return (ARCHIVE_FATAL); > > } > > - /* Restart header parser with the next block. */ > > - state->header_state = state->header_done = 0; > > + state->eof = 1; > > /* FALL THROUGH */ > > case Z_OK: /* Decompressor made some progress. */ > > /* If we filled our buffer, update stats and return. */ > -- Simon L. Nielsen
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090101095151.GA1117>