Date: Sun, 14 Feb 2016 16:27:49 -0600 From: Alan Cox <alan.l.cox@gmail.com> To: Andriy Gapon <avg@freebsd.org> Cc: FreeBSD Current <freebsd-current@freebsd.org> Subject: Re: Memory modified after free in "MAP ENTRY" zone (vm_map_entry_t->read_ahead) Message-ID: <CAJUyCcNy19rArmgjzpPZvsDE7Ln4OoMWU%2B75Q%2BTHdp0T7%2BDxPg@mail.gmail.com> In-Reply-To: <56C08AAA.5050206@FreeBSD.org> References: <56BBAB6E.5050601@FreeBSD.org> <56C08AAA.5050206@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Feb 14, 2016 at 8:09 AM, Andriy Gapon <avg@freebsd.org> wrote:
> On 10/02/2016 23:28, Andriy Gapon wrote:
> >
> > Over a span of approximately 3 weeks I have got two slightly different
> panics of
> > the same kind. The affected system is a several months old amd64 head.
>
> I added a small assertion and it got tripped:
> --- a/sys/vm/vm_fault.c
> +++ b/sys/vm/vm_fault.c
> @@ -608,8 +608,11 @@ readrest:
> cluster_offset;
> }
> ahead = ulmin(ahead, atop(fs.entry->end - vaddr) -
> 1);
> - if (era != nera)
> + if (era != nera) {
> + KASSERT(fs.lookup_still_valid,
> + ("modifying read_ahead without map
> lock"));
> fs.entry->read_ahead = nera;
> + }
>
> /*
> * Call the pager to retrieve the data, if any,
> after
>
> So, it seems that the code allows modification of read_ahead field of an
> entry
> without holding a map lock. I'd guess that that should be mostly harmless
> as
> read_ahead value is not critical, but it is still not nice.
>
>
Not intentionally. The nearby code to the read_ahead assignment expects
the map to be locked, so I wouldn't categorize this as mostly harmless.
In general, you shouldn't get to the read_ahead assignment without the map
being locked, and almost all of the code paths that unlock the map jump to
RetryFault shortly thereafter, so it's hard to imagine how the map lock
wouldn't be reacquired. I speculate that the root cause of your panic is a
case where vm_pager_get_pages() fails, and in such a case we might loop
around, descending to the next backing object without reacquiring the map
lock. In other words, your above assertion failure is happening on the
next iteration after an initial vm_pager_get_pages() failure, and we're
about to do another vm_pager_get_pages() call on a different object.
In summary, I have no trouble believing that the code that handles a
failure by vm_pager_get_pages() in vm_fault() is not quite right, and I
think that's what's biting you.
> Some details from the panic caused by the assertion can be found here:
> http://dpaste.com/39BYV7S.txt
>
>
> > ======================== 1 ===========================================
> > Unread portion of the kernel message buffer:
> > panic: Memory modified after free 0xfffff8008c15ac80(128) val=adc0de @
> > 0xfffff8008c15acdc
> >
> > KDB: stack backtrace:
> > db_trace_self_wrapper() at 0xffffffff8041e90b =
> db_trace_self_wrapper+0x2b/frame
> > 0xfffffe04f5349530
> > kdb_backtrace() at 0xffffffff80669a09 = kdb_backtrace+0x39/frame
> 0xfffffe04f53495e0
> > vpanic() at 0xffffffff80634dec = vpanic+0x14c/frame 0xfffffe04f5349620
> > panic() at 0xffffffff80634b33 = panic+0x43/frame 0xfffffe04f5349680
> > trash_ctor() at 0xffffffff807de9c8 = trash_ctor+0x48/frame
> 0xfffffe04f5349690
> > uma_zalloc_arg() at 0xffffffff807da785 = uma_zalloc_arg+0x475/frame
> > 0xfffffe04f5349720
> > uma_zalloc() at 0xffffffff807e44af = uma_zalloc+0xf/frame
> 0xfffffe04f5349730
> > vm_map_entry_create() at 0xffffffff807e5a2e =
> vm_map_entry_create+0x2e/frame
> > 0xfffffe04f5349740
> > _vm_map_clip_start() at 0xffffffff807e69e3 =
> _vm_map_clip_start+0x123/frame
> > 0xfffffe04f5349770
> > vm_map_wire() at 0xffffffff807e797d = vm_map_wire+0x11d/frame
> 0xfffffe04f53497f0
> > vslock() at 0xffffffff807e1a2b = vslock+0x6b/frame 0xfffffe04f5349810
> > sysctl_wire_old_buffer() at 0xffffffff8064148a =
> > sysctl_wire_old_buffer+0x4a/frame 0xfffffe04f5349830
> > sysctl_kern_proc() at 0xffffffff8062259b = sysctl_kern_proc+0x8b/frame
> > 0xfffffe04f5349880
> > sysctl_root_handler_locked() at 0xffffffff80641a8e =
> > sysctl_root_handler_locked+0x8e/frame 0xfffffe04f53498c0
> > sysctl_root() at 0xffffffff806412fe = sysctl_root+0x13e/frame
> 0xfffffe04f5349940
> > userland_sysctl() at 0xffffffff8064183d = userland_sysctl+0x16d/frame
> > 0xfffffe04f53499e0
> > sys___sysctl() at 0xffffffff80641694 = sys___sysctl+0x74/frame
> 0xfffffe04f5349a90
> > syscallenter() at 0xffffffff8081fa20 = syscallenter+0x320/frame
> 0xfffffe04f5349b00
> > amd64_syscall() at 0xffffffff8081f5ef = amd64_syscall+0x1f/frame
> 0xfffffe04f5349bf0
> > Xfast_syscall() at 0xffffffff80807c5b = Xfast_syscall+0xfb/frame
> 0xfffffe04f5349bf0
> > --- syscall (202, FreeBSD ELF64, sys___sysctl), rip = 0x8042225ea, rsp =
> > 0x7fffffffcef8, rbp = 0x7fffffffcf30 ---
> > Uptime: 14d22h38m17s
> > ======================================================================
> >
> > ======================== 2 ===========================================
> > Unread portion of the kernel message buffer:
> > panic: Memory modified after free 0xfffff80176692680(128) val=adc0de @
> > 0xfffff801766926dc
> >
> > KDB: stack backtrace:
> > db_trace_self_wrapper() at 0xffffffff8041e90b =
> db_trace_self_wrapper+0x2b/frame
> > 0xfffffe04f507c5a0
> > kdb_backtrace() at 0xffffffff80669a09 = kdb_backtrace+0x39/frame
> 0xfffffe04f507c650
> > vpanic() at 0xffffffff80634dec = vpanic+0x14c/frame 0xfffffe04f507c690
> > panic() at 0xffffffff80634b33 = panic+0x43/frame 0xfffffe04f507c6f0
> > trash_ctor() at 0xffffffff807de9c8 = trash_ctor+0x48/frame
> 0xfffffe04f507c700
> > uma_zalloc_arg() at 0xffffffff807da785 = uma_zalloc_arg+0x475/frame
> > 0xfffffe04f507c790
> > uma_zalloc() at 0xffffffff807e44af = uma_zalloc+0xf/frame
> 0xfffffe04f507c7a0
> > vm_map_entry_create() at 0xffffffff807e5a2e =
> vm_map_entry_create+0x2e/frame
> > 0xfffffe04f507c7b0
> > vm_map_insert() at 0xffffffff807e558a = vm_map_insert+0x2fa/frame
> 0xfffffe04f507c850
> > vm_map_stack_locked() at 0xffffffff807e63cb =
> vm_map_stack_locked+0x13b/frame
> > 0xfffffe04f507c8b0
> > vm_map_find() at 0xffffffff807e65d3 = vm_map_find+0x183/frame
> 0xfffffe04f507c950
> > vm_mmap_object() at 0xffffffff807eac99 = vm_mmap_object+0x329/frame
> > 0xfffffe04f507c9d0
> > sys_mmap() at 0xffffffff807ea8ec = sys_mmap+0x41c/frame
> 0xfffffe04f507ca90
> > syscallenter() at 0xffffffff8081fa20 = syscallenter+0x320/frame
> 0xfffffe04f507cb00
> > amd64_syscall() at 0xffffffff8081f5ef = amd64_syscall+0x1f/frame
> 0xfffffe04f507cbf0
> > Xfast_syscall() at 0xffffffff80807c5b = Xfast_syscall+0xfb/frame
> 0xfffffe04f507cbf0
> > --- syscall (477, FreeBSD ELF64, sys_mmap), rip = 0x80418776a, rsp =
> > 0x7fffffffb808, rbp = 0x7fffffffb840 ---
> > Uptime: 4d4h36m36s
> > ======================================================================
> >
> > I have crash dumps from both panics.
> > It seems that it was read_ahead field that was reset to zero in both
> cases.
> > Perhaps the issue has been fixed already?
> > I would appreciate any suggestions. Thanks!
> >
>
>
> --
> Andriy Gapon
> _______________________________________________
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJUyCcNy19rArmgjzpPZvsDE7Ln4OoMWU%2B75Q%2BTHdp0T7%2BDxPg>