From owner-freebsd-current@freebsd.org Sat Aug 25 17:44:27 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BEE1810916A5 for ; Sat, 25 Aug 2018 17:44:27 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6F7E57B059; Sat, 25 Aug 2018 17:44:27 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-io0-f181.google.com (mail-io0-f181.google.com [209.85.223.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id 315C9118A9; Sat, 25 Aug 2018 17:44:27 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-io0-f181.google.com with SMTP id r196-v6so9635199iod.0; Sat, 25 Aug 2018 10:44:27 -0700 (PDT) X-Gm-Message-State: APzg51DqVPOS641yYXoFvtQKUOsB6RT3VvM/Wj2+kbK2t0BzFp5L4avO pxqkYsY2J3gdP6yk6wM6xytbTVVn0xJz2Bcx7Uk= X-Google-Smtp-Source: ANB0VdZC6kGV4edQpJ1eMJfJqSEnv8HxXM96iHKoGx8D/cKhPUF3vY2zdY9OXjW6xEz/BMrU9e3BGYWNhGW5lT3oKc8= X-Received: by 2002:a5e:dd4b:: with SMTP id u11-v6mr5080251iop.237.1535219066460; Sat, 25 Aug 2018 10:44:26 -0700 (PDT) MIME-Version: 1.0 References: <20180824221955.7hkftov25otk6bjc@mutt-hbsd> <7A724399-B264-41A9-B85F-A49D3B0B4730@FreeBSD.org> <2287F455-EDFD-4065-BB83-33DDDF74D027@FreeBSD.org> In-Reply-To: <2287F455-EDFD-4065-BB83-33DDDF74D027@FreeBSD.org> From: Matthew Macy Date: Sat, 25 Aug 2018 10:44:15 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: ifnet use after free To: Kristof Provost Cc: Shawn Webb , freebsd-current@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2018 17:44:28 -0000 I'll take a look. But it's likely to not be the OP's issue. For future reference memguard on the memory type in question is extremely useful in catching use after free. -M On Sat, Aug 25, 2018 at 05:51 Kristof Provost wrote: > On 25 Aug 2018, at 0:47, Kristof Provost wrote: > > On 25 Aug 2018, at 0:26, Matthew Macy wrote: > > On Fri, Aug 24, 2018 at 15:25 Shawn Webb > wrote: > > Hey All, > > Somewhere in the last month or so, a use after free was introduced. I > don't have the time right now to bisect the commits and figure out > which commit introduced the breakage. Attached is the core.txt (which > seems nonsensical because the dump is reporting on a different > thread). If the core.txt gets scrubbed, I've posted it here: > https://gist.github.com/796ea88cec19a1fd2a85f4913482286a > > Do you have any guidance on how to reproduce? The hardenedbsd rev isn=E2= =80=99t > useful - the svn commit that it=E2=80=99s based against is what is needed= . > > For what it=E2=80=99s worth, it=E2=80=99s not a hardenedbsd thing. I=E2= =80=99ve been chasing the > same one (same offset, same allocation size, same most recent user). > Something gets set to zero/NULL. 8 bytes on amd64, so presumably a pointe= r. > > I currently only trigger it on a development branch, but I=E2=80=99ll see= if I can > clean that up into something I can share tomorrow. > > In my test scenario it happens after shutdown of a vnet jail with a few > interfaces in it (including a pfsync interface which will disappear with > the jail), and new jails are started. It=E2=80=99s pretty reliable. > > At a guess something=E2=80=99s wrong with the delayed cleanup of ifnets a= nd vnet > shutdown. > > I see this: > > Memory modified after free 0xfffff800623ab000(2040) val=3D0 @ 0xfffff8006= 23ab398 > panic: Most recently used by ifnet > > cpuid =3D 7 > time =3D 1535199812 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe008c8= e13c0 > vpanic() at vpanic+0x1a3/frame 0xfffffe008c8e1420 > panic() at panic+0x43/frame 0xfffffe008c8e1480 > mtrash_ctor() at mtrash_ctor+0x81/frame 0xfffffe008c8e14a0 > uma_zalloc_arg() at uma_zalloc_arg+0x72c/frame 0xfffffe008c8e1510 > malloc() at malloc+0x9a/frame 0xfffffe008c8e1560 > if_alloc() at if_alloc+0x23/frame 0xfffffe008c8e1590 > epair_clone_create() at epair_clone_create+0x239/frame 0xfffffe008c8e1610 > if_clone_createif() at if_clone_createif+0x4a/frame 0xfffffe008c8e1660 > ifioctl() at ifioctl+0x852/frame 0xfffffe008c8e1750 > kern_ioctl() at kern_ioctl+0x2ba/frame 0xfffffe008c8e17b0 > sys_ioctl() at sys_ioctl+0x15e/frame 0xfffffe008c8e1880 > amd64_syscall() at amd64_syscall+0x28c/frame 0xfffffe008c8e19b0 > fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe008c8e19= b0 > --- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x80047b74a, rsp =3D = 0x7fffffffe208, rbp =3D 0x7fffffffe250 --- > KDB: enter: panic > [ thread pid 1426 tid 100466 ] > Stopped at kdb_enter+0x3b: movq $0,kdb_why > db> > > It does require a couple of bug fixes in pfsync to trigger. You can get > them from the pfsync_vnet branch in > https://github.com/kprovost/freebsd/tree/pfsync_vnet > > After that: > kldload pfsync > pkg install scapy > cd /usr/tests/sys/netpfil/pf > kyua test > > It should panic reliably. > > Regards, > Kristof >