From owner-freebsd-current@freebsd.org Sat Aug 25 19:12:16 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D394F1093C4A for ; Sat, 25 Aug 2018 19:12:15 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 837197E58B; Sat, 25 Aug 2018 19:12:15 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-io0-f170.google.com (mail-io0-f170.google.com [209.85.223.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id 3F8C6121B7; Sat, 25 Aug 2018 19:12:15 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-io0-f170.google.com with SMTP id l7-v6so9725014iok.6; Sat, 25 Aug 2018 12:12:15 -0700 (PDT) X-Gm-Message-State: APzg51CtaHMdUEt6azWxAxgz3liL1NrXgECkEqNToZ2o2waEl/xD5bKN 6GrU7jfswWYUp48AvuFCcjkCtyo9YvO83P9iHU4= X-Google-Smtp-Source: ANB0VdbQPXOqRj4gMtAsRyTqve7MlRBeTJQjhLc7E1balkOfhcY0ZgtDn5TJTufZZHm9Jk2S5g3m8Lqiq2IRL3XdPPQ= X-Received: by 2002:a5e:dd4b:: with SMTP id u11-v6mr5250297iop.237.1535224334595; Sat, 25 Aug 2018 12:12:14 -0700 (PDT) MIME-Version: 1.0 References: <20180824221955.7hkftov25otk6bjc@mutt-hbsd> <7A724399-B264-41A9-B85F-A49D3B0B4730@FreeBSD.org> <2287F455-EDFD-4065-BB83-33DDDF74D027@FreeBSD.org> <3B8F3112-6DD3-4461-9256-ECB044FA6D44@FreeBSD.org> In-Reply-To: <3B8F3112-6DD3-4461-9256-ECB044FA6D44@FreeBSD.org> From: Matthew Macy Date: Sat, 25 Aug 2018 12:12:03 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: ifnet use after free To: Kristof Provost Cc: Shawn Webb , freebsd-current Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2018 19:12:16 -0000 On Sat, Aug 25, 2018 at 12:10 PM Kristof Provost wrote: > You may be right about that. With memguard (on ifnet) it implicates pf: > > pfi_cleanup_vnet() at pfi_cleanup_vnet+0xa4/frame 0xfffffe084f775320 > vnet_pf_uninit() at vnet_pf_uninit+0x85f/frame 0xfffffe084f7757c0 > vnet_destroy() at vnet_destroy+0x12c/frame 0xfffffe084f7757f0 > prison_deref() at prison_deref+0x29d/frame 0xfffffe084f775830 > sys_jail_remove() at sys_jail_remove+0x28a/frame 0xfffffe084f775880 > amd64_syscall() at amd64_syscall+0x28c/frame 0xfffffe084f7759b0 > fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe084f7759= b0 > --- syscall (508, FreeBSD ELF64, sys_jail_remove), rip =3D 0x8003130da, r= sp > =3D 0x7fffffffe848, rbp =3D 0x7fffffffe8d0 --- > > I=E2=80=99ll investigate that. Sorry for the noise. > Thanks for the pointer to memguard. Very useful. > > And thank you for updating me. -M > Kristof > > On 25 Aug 2018, at 19:44, Matthew Macy wrote: > > I'll take a look. But it's likely to not be the OP's issue. For future > reference memguard on the memory type in question is extremely useful in > catching use after free. > > -M > > On Sat, Aug 25, 2018 at 05:51 Kristof Provost wrote: > >> On 25 Aug 2018, at 0:47, Kristof Provost wrote: >> >> On 25 Aug 2018, at 0:26, Matthew Macy wrote: >> >> On Fri, Aug 24, 2018 at 15:25 Shawn Webb >> wrote: >> >> Hey All, >> >> Somewhere in the last month or so, a use after free was introduced. I >> don't have the time right now to bisect the commits and figure out >> which commit introduced the breakage. Attached is the core.txt (which >> seems nonsensical because the dump is reporting on a different >> thread). If the core.txt gets scrubbed, I've posted it here: >> https://gist.github.com/796ea88cec19a1fd2a85f4913482286a >> >> Do you have any guidance on how to reproduce? The hardenedbsd rev isn=E2= =80=99t >> useful - the svn commit that it=E2=80=99s based against is what is neede= d. >> >> For what it=E2=80=99s worth, it=E2=80=99s not a hardenedbsd thing. I=E2= =80=99ve been chasing the >> same one (same offset, same allocation size, same most recent user). >> Something gets set to zero/NULL. 8 bytes on amd64, so presumably a point= er. >> >> I currently only trigger it on a development branch, but I=E2=80=99ll se= e if I >> can clean that up into something I can share tomorrow. >> >> In my test scenario it happens after shutdown of a vnet jail with a few >> interfaces in it (including a pfsync interface which will disappear with >> the jail), and new jails are started. It=E2=80=99s pretty reliable. >> >> At a guess something=E2=80=99s wrong with the delayed cleanup of ifnets = and vnet >> shutdown. >> >> I see this: >> >> Memory modified after free 0xfffff800623ab000(2040) val=3D0 @ 0xfffff800= 623ab398 >> panic: Most recently used by ifnet >> >> cpuid =3D 7 >> time =3D 1535199812 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe008c= 8e13c0 >> vpanic() at vpanic+0x1a3/frame 0xfffffe008c8e1420 >> panic() at panic+0x43/frame 0xfffffe008c8e1480 >> mtrash_ctor() at mtrash_ctor+0x81/frame 0xfffffe008c8e14a0 >> uma_zalloc_arg() at uma_zalloc_arg+0x72c/frame 0xfffffe008c8e1510 >> malloc() at malloc+0x9a/frame 0xfffffe008c8e1560 >> if_alloc() at if_alloc+0x23/frame 0xfffffe008c8e1590 >> epair_clone_create() at epair_clone_create+0x239/frame 0xfffffe008c8e161= 0 >> if_clone_createif() at if_clone_createif+0x4a/frame 0xfffffe008c8e1660 >> ifioctl() at ifioctl+0x852/frame 0xfffffe008c8e1750 >> kern_ioctl() at kern_ioctl+0x2ba/frame 0xfffffe008c8e17b0 >> sys_ioctl() at sys_ioctl+0x15e/frame 0xfffffe008c8e1880 >> amd64_syscall() at amd64_syscall+0x28c/frame 0xfffffe008c8e19b0 >> fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe008c8e1= 9b0 >> --- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x80047b74a, rsp =3D= 0x7fffffffe208, rbp =3D 0x7fffffffe250 --- >> KDB: enter: panic >> [ thread pid 1426 tid 100466 ] >> Stopped at kdb_enter+0x3b: movq $0,kdb_why >> db> >> >> It does require a couple of bug fixes in pfsync to trigger. You can get >> them from the pfsync_vnet branch in >> https://github.com/kprovost/freebsd/tree/pfsync_vnet >> >> After that: >> kldload pfsync >> pkg install scapy >> cd /usr/tests/sys/netpfil/pf >> kyua test >> >> It should panic reliably. >> >> Regards, >> Kristof >> >