From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 20 00:40:11 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A6073E6C for ; Thu, 20 Mar 2014 00:40:11 +0000 (UTC) Received: from mx11.netapp.com (mx11.netapp.com [216.240.18.76]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 889D46B3 for ; Thu, 20 Mar 2014 00:40:11 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.97,690,1389772800"; d="scan'208,217";a="110728859" Received: from vmwexceht05-prd.hq.netapp.com ([10.106.77.35]) by mx11-out.netapp.com with ESMTP; 19 Mar 2014 17:40:10 -0700 Received: from SACEXCMBX04-PRD.hq.netapp.com ([169.254.6.187]) by vmwexceht05-prd.hq.netapp.com ([10.106.77.35]) with mapi id 14.03.0123.003; Wed, 19 Mar 2014 17:40:10 -0700 From: "Gumpula, Suresh" To: "freebsd-hackers@freebsd.org" Subject: Use after free in sys/net/zlib.c code Thread-Topic: Use after free in sys/net/zlib.c code Thread-Index: Ac9D0pGtfoVDvgwwQPC/7fto2jHQ8A== Date: Thu, 20 Mar 2014 00:40:09 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.106.53.53] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.17 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Mar 2014 00:40:11 -0000 Hi Zlib experts, I am trying to debug a corruption in zlib code. I have enabled the memguard= (9) for 'geom_uzip' malloc type and is faulting all the time while booting as shown below. I have changed the memguard fre= e() code to save the PC info of who freed, and it shows that a buffer was freed from inflate_blocks --> inflate_codes_free()->ZFREE(). And it seems we are u= sing in inflate_blocks after it was already freed. Please see my debug analysis. Can someone who has better understanding of z= lib code throw some lights on this ? Is this a known issue and fixed recen= tly ? (kgdb-amd64-7.4-08) bt #0 breakpoint () at ./machine/cpufunc.h:64 #1 0xffffffff803e6572 in kdb_enter (why=3D0xffffffff806fd681 "panic", msg= =3D0xffffffff806fd681 "panic") at ../../../../sys/kern/subr_kdb.c:367 #2 0xffffffff803a3eb4 in panic (fmt=3D0xffffffff8075d580 "page fault (%s %= s %s, %s) on VA %#lx cs:rip %#lx:%#lx rflags %#lx") at ../../../../sys/kern= /kern_shutdown.c:1010 #3 0xffffffff8060edf0 in trap_fatal (frame=3D0xffffff802094f890, eva=3D184= 46743523955920960) at ../../../../sys/amd64/amd64/trap.c:999 #4 0xffffffff8060e4a7 in trap_pfault (frame=3D0xffffff802094f890, usermode= =3D0) at ../../../../sys/amd64/amd64/trap.c:824 #5 0xffffffff8060df4a in trap (frame=3D0xffffff802094f890) at ../../../../= sys/amd64/amd64/trap.c:595 #6 0xffffffff805e6009 in () at ../../../../sys/amd= 64/amd64/exception.S:253 #7 0xffffffff804adcb7 in inflate_blocks (s=3D0xffffff8000211000, z=3D0xfff= fff802094fa80, r=3D0) at ../../../../sys/net/zlib.c:3856 #8 0xffffffff804ac816 in inflate_ppp (z=3D0xffffff802094fa80, f=3D5) at ..= /../../../sys/net/zlib.c:3263 #9 0xffffffff80330eb3 in g_uzip_done (bp=3D0xffffff00034f2400) at ../../..= /../sys/geom/uzip/g_uzip.c:177 #10 0xffffffff8044b1e5 in biodone (bp=3D0xffffff00034f2400) at ../../../../= sys/kern/vfs_bio.c:3137 #11 0xffffffff8032204a in g_io_schedule_up (tp=3D0xffffff000317a820) at ../= ../../../sys/geom/geom_io.c:676 #12 0xffffffff8032264a in g_up_procbody () at ../../../../sys/geom/geom_ker= n.c:95 #13 0xffffffff803663f5 in fork_exit (callout=3D0xffffffff803225c0 , arg=3D0x0, frame=3D0xffffff802094fc80) at ../../../../sys/kern/kern= _fork.c:1063 (kgdb-amd64-7.4-08) f 7 #7 0xffffffff804adcb7 in inflate_blocks (s=3D0xffffff8000211000, z=3D0xfff= fff802094fa80, r=3D0) at ../../../../sys/net/zlib.c:3856 3856 s->sub.trees.blens[border[s->sub.trees.index++]] =3D (uInt)= b & 7; (kgdb-amd64-7.4-08) p s->sub.trees.blens $10 =3D (uIntf *) 0xffffff8000215000 (kgdb-amd64-7.4-08) p panicstr $11 =3D 0xffffffff80a8e560 "page fault (supervisor write data, protection v= iolation) on VA 0xffffff8000215040 cs:rip 0x20:0xffffffff804adcb7 rflags 0x= 10206" (kgdb-amd64-7.4-08) x/100 0xffffff8000215040 0xffffff8000215040: 0x804b0b06 0x804b0b06 0x804ae542 0xf= fffffff 0xffffff8000215050: 0x804b0b06 0x804b0b06 0x804ae542 0xf= fffffff ## stack trace of last free() 0xffffff8000215060: 0x804b0b06 0x804b0b06 0x804ae542 0xf= fffffff (kgdb-amd64-7.4-08) l *0xffffffff804ae542 0xffffffff804ae542 is in inflate_blocks (../../../../sys/net/zlib.c:3969). 3964 UPDATE 3965 if ((r =3D inflate_codes(s, z, r)) !=3D Z_STREAM_END) 3966 return inflate_flush(s, z, r); 3967 r =3D Z_OK; 3968 inflate_codes_free(s->sub.decode.codes, z); 3969 inflate_trees_free(s->sub.decode.td, z); 3970 inflate_trees_free(s->sub.decode.tl, z); (kgdb-amd64-7.4-08) l *0xffffffff804b0b06 0xffffffff804b0b06 is in inflate_codes_free (../../../../sys/net/zlib.c:484= 6). 4841 inflate_codes_statef *c; 4842 z_streamp z; 4843 { 4844 ZFREE(z, c); 4845 Tracev((stderr, "inflate: codes free\n")); 4846 } And I see sometimes its faulting in huft_build() and it shows it was freed = from inflate_trees_free(). All the time, I see either of these two back traces faulting. kgdb-amd64-7.4-08) bt #0 breakpoint () at ./machine/cpufunc.h:64 #1 0xffffffff803e6572 in kdb_enter (why=3D0xffffffff806fd681 "panic", msg= =3D0xffffffff806fd681 "panic") at ../../../../sys/kern/subr_kdb.c:367 #2 0xffffffff803a3eb4 in panic (fmt=3D0xffffffff8075d580 "page fault (%s %= s %s, %s) on VA %#lx cs:rip %#lx:%#lx rflags %#lx") at ../../../../sys/kern= /kern_shutdown.c:1010 #3 0xffffffff8060edf0 in trap_fatal (frame=3D0xffffff802094f200, eva=3D184= 46743523955908616) at ../../../../sys/amd64/amd64/trap.c:999 #4 0xffffffff8060e4a7 in trap_pfault (frame=3D0xffffff802094f200, usermode= =3D0) at ../../../../sys/amd64/amd64/trap.c:824 #5 0xffffffff8060df4a in trap (frame=3D0xffffff802094f200) at ../../../../= sys/amd64/amd64/trap.c:595 #6 0xffffffff805e6009 in () at ../../../../sys/amd= 64/amd64/exception.S:253 #7 0xffffffff804af203 in huft_build (b=3D0xffffff8002f0f000, n=3D276, s=3D= 257, d=3D0xffffffff807255c0, e=3D0xffffffff80725640, t=3D0xffffff8000212008= , m=3D0xffffff802094f9f8, zs=3D0xffffff802094fa80) at ../../../../sys/net/z= lib.c:4346 #8 0xffffffff804af5c5 in inflate_trees_dynamic (nl=3D284, nd=3D28, c=3D0xf= fffff8002f0f000, bl=3D0xffffff802094f9f8, bd=3D0xffffff802094f9f4, tl=3D0xf= fffff802094f9c8, td=3D0xffffff802094f9c0, z=3D0xffffff802094fa80) at ../../= ../../sys/net/zlib.c:4435 #9 0xffffffff804ae2d4 in inflate_blocks (s=3D0xffffff8002f06000, z=3D0xfff= fff802094fa80, r=3D0) at ../../../../sys/net/zlib.c:3933 #10 0xffffffff804ac816 in inflate_ppp (z=3D0xffffff802094fa80, f=3D5) at ..= /../../../sys/net/zlib.c:3263 #11 0xffffffff80330eb3 in g_uzip_done (bp=3D0xffffff0003492700) at ../../..= /../sys/geom/uzip/g_uzip.c:177 #12 0xffffffff8044b1e5 in biodone (bp=3D0xffffff0003492700) at ../../../../= sys/kern/vfs_bio.c:3137 #13 0xffffffff8032204a in g_io_schedule_up (tp=3D0xffffff000317a820) at ../= ../../../sys/geom/geom_io.c:676 #14 0xffffffff8032264a in g_up_procbody () at ../../../../sys/geom/geom_ker= n.c:95 #15 0xffffffff803663f5 in fork_exit (callout=3D0xffffffff803225c0 , arg=3D0x0, frame=3D0xffffff802094fc80) at ../../../../sys/kern/kern= _fork.c:1063 (kgdb-amd64-7.4-08) p panicstr $2 =3D 0xffffffff80a8e560 "page fault (supervisor write data, protection vi= olation) on VA 0xffffff8000212008 cs:rip 0x20:0xffffffff804af203 rflags 0x1= 0282" (kgdb-amd64-7.4-08) x/100 0xffffff8000212008 0xffffff8000212008: 0x804ae25a 0xffffffff 0x804af9aa 0x8= 04af9aa ## Stack trace of last free() 0xffffff8000212018: 0x804ae25a 0xffffffff 0x804af9aa 0x8= 04af9aa (kgdb-amd64-7.4-08) l *0xffffffff804ae25a 0xffffffff804ae25a is in inflate_blocks (../../../../sys/net/zlib.c:3921). 3916 } while (--j); 3917 s->sub.trees.index =3D i; 3918 } 3919 } 3920 inflate_trees_free(s->sub.trees.tb, z); 3921 s->sub.trees.tb =3D Z_NULL; 3922 { (kgdb-amd64-7.4-08) l *0xffffffff804af9aa 0xffffffff804af9aa is in inflate_trees_free (../../../../sys/net/zlib.c:457= 4). 4569 /* Go through linked list, freeing from the malloced (t[-1]) addr= ess. */ 4570 while (p !=3D Z_NULL) 4571 { 4572 q =3D (--p)->next; 4573 ZFREE(z,p); Thank you!