Date: Sat, 25 Aug 2018 21:10:45 +0200 From: "Kristof Provost" <kp@FreeBSD.org> To: "Matthew Macy" <mmacy@freebsd.org> Cc: "Shawn Webb" <shawn.webb@hardenedbsd.org>, freebsd-current@freebsd.org Subject: Re: ifnet use after free Message-ID: <3B8F3112-6DD3-4461-9256-ECB044FA6D44@FreeBSD.org> In-Reply-To: <CAPrugNrT3qcy%2BE5LUVYMBKze8Cc=454ergJbbQs7GuzfFNOt3Q@mail.gmail.com> References: <20180824221955.7hkftov25otk6bjc@mutt-hbsd> <CAPrugNqiX5udzOchu=yBAEEqnkK-LAZZhTW4poen13Gguc1Xng@mail.gmail.com> <7A724399-B264-41A9-B85F-A49D3B0B4730@FreeBSD.org> <2287F455-EDFD-4065-BB83-33DDDF74D027@FreeBSD.org> <CAPrugNrT3qcy%2BE5LUVYMBKze8Cc=454ergJbbQs7GuzfFNOt3Q@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
You may be right about that. With memguard (on ifnet) it implicates pf: pfi_cleanup_vnet() at pfi_cleanup_vnet+0xa4/frame 0xfffffe084f775320 vnet_pf_uninit() at vnet_pf_uninit+0x85f/frame 0xfffffe084f7757c0 vnet_destroy() at vnet_destroy+0x12c/frame 0xfffffe084f7757f0 prison_deref() at prison_deref+0x29d/frame 0xfffffe084f775830 sys_jail_remove() at sys_jail_remove+0x28a/frame 0xfffffe084f775880 amd64_syscall() at amd64_syscall+0x28c/frame 0xfffffe084f7759b0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe084f7759b0 --- syscall (508, FreeBSD ELF64, sys_jail_remove), rip = 0x8003130da, rsp = 0x7fffffffe848, rbp = 0x7fffffffe8d0 --- I’ll investigate that. Sorry for the noise. Thanks for the pointer to memguard. Very useful. Kristof On 25 Aug 2018, at 19:44, Matthew Macy wrote: > I'll take a look. But it's likely to not be the OP's issue. For future > reference memguard on the memory type in question is extremely useful > in > catching use after free. > > -M > > On Sat, Aug 25, 2018 at 05:51 Kristof Provost <kp@freebsd.org> wrote: > >> On 25 Aug 2018, at 0:47, Kristof Provost wrote: >> >> On 25 Aug 2018, at 0:26, Matthew Macy wrote: >> >> On Fri, Aug 24, 2018 at 15:25 Shawn Webb <shawn.webb@hardenedbsd.org> >> wrote: >> >> Hey All, >> >> Somewhere in the last month or so, a use after free was introduced. I >> don't have the time right now to bisect the commits and figure out >> which commit introduced the breakage. Attached is the core.txt (which >> seems nonsensical because the dump is reporting on a different >> thread). If the core.txt gets scrubbed, I've posted it here: >> https://gist.github.com/796ea88cec19a1fd2a85f4913482286a >> >> Do you have any guidance on how to reproduce? The hardenedbsd rev >> isn’t >> useful - the svn commit that it’s based against is what is needed. >> >> For what it’s worth, it’s not a hardenedbsd thing. I’ve been >> chasing the >> same one (same offset, same allocation size, same most recent user). >> Something gets set to zero/NULL. 8 bytes on amd64, so presumably a >> pointer. >> >> I currently only trigger it on a development branch, but I’ll see >> if I can >> clean that up into something I can share tomorrow. >> >> In my test scenario it happens after shutdown of a vnet jail with a >> few >> interfaces in it (including a pfsync interface which will disappear >> with >> the jail), and new jails are started. It’s pretty reliable. >> >> At a guess something’s wrong with the delayed cleanup of ifnets and >> vnet >> shutdown. >> >> I see this: >> >> Memory modified after free 0xfffff800623ab000(2040) val=0 @ >> 0xfffff800623ab398 >> panic: Most recently used by ifnet >> >> cpuid = 7 >> time = 1535199812 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe008c8e13c0 >> vpanic() at vpanic+0x1a3/frame 0xfffffe008c8e1420 >> panic() at panic+0x43/frame 0xfffffe008c8e1480 >> mtrash_ctor() at mtrash_ctor+0x81/frame 0xfffffe008c8e14a0 >> uma_zalloc_arg() at uma_zalloc_arg+0x72c/frame 0xfffffe008c8e1510 >> malloc() at malloc+0x9a/frame 0xfffffe008c8e1560 >> if_alloc() at if_alloc+0x23/frame 0xfffffe008c8e1590 >> epair_clone_create() at epair_clone_create+0x239/frame >> 0xfffffe008c8e1610 >> if_clone_createif() at if_clone_createif+0x4a/frame >> 0xfffffe008c8e1660 >> ifioctl() at ifioctl+0x852/frame 0xfffffe008c8e1750 >> kern_ioctl() at kern_ioctl+0x2ba/frame 0xfffffe008c8e17b0 >> sys_ioctl() at sys_ioctl+0x15e/frame 0xfffffe008c8e1880 >> amd64_syscall() at amd64_syscall+0x28c/frame 0xfffffe008c8e19b0 >> fast_syscall_common() at fast_syscall_common+0x101/frame >> 0xfffffe008c8e19b0 >> --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x80047b74a, rsp = >> 0x7fffffffe208, rbp = 0x7fffffffe250 --- >> KDB: enter: panic >> [ thread pid 1426 tid 100466 ] >> Stopped at kdb_enter+0x3b: movq $0,kdb_why >> db> >> >> It does require a couple of bug fixes in pfsync to trigger. You can >> get >> them from the pfsync_vnet branch in >> https://github.com/kprovost/freebsd/tree/pfsync_vnet >> >> After that: >> kldload pfsync >> pkg install scapy >> cd /usr/tests/sys/netpfil/pf >> kyua test >> >> It should panic reliably. >> >> Regards, >> Kristof >>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3B8F3112-6DD3-4461-9256-ECB044FA6D44>