From owner-freebsd-current@freebsd.org Sun Feb 14 22:27:50 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 20208AA9F88 for ; Sun, 14 Feb 2016 22:27:50 +0000 (UTC) (envelope-from alan.l.cox@gmail.com) Received: from mail-io0-x22e.google.com (mail-io0-x22e.google.com [IPv6:2607:f8b0:4001:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E19891E9B; Sun, 14 Feb 2016 22:27:49 +0000 (UTC) (envelope-from alan.l.cox@gmail.com) Received: by mail-io0-x22e.google.com with SMTP id z135so72076955iof.0; Sun, 14 Feb 2016 14:27:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=apdbFvovvUcAbjP7c6pcHZwieYwpz0RwamCDJ829ZLM=; b=L9e8kEt5FiKH0afFLI/PIuXJcNhxyhytP8ocUr7DwIZe9t7/72ay6TwHW0ALm7rjMm pK1VXS56cr3VTWM3VuG2Ng5+PS/8YgkJnimeKraY2SsaQ4c2Zlad9VAp7nHALv6sWshc DquTDPZbyRjBYQ0qfCwCU/5TCn9qlvR1CmgR1lmECloYyZ16YIbL2j/m+0nSxnL4sp6s UzzgziW+fZPOVFcoL0d1ObqMT9A7ASWwXrnf/eovHFpEim3Vgcfbw3DE/P3hZs8cGQ/w vpnoUpAppqeo2C4RfyZ0Q7C+huIqTrC631ywtgXpCY/II5HaZM9M1gIS3ofw2OjNwo8l C5tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=apdbFvovvUcAbjP7c6pcHZwieYwpz0RwamCDJ829ZLM=; b=CsYg+gashB/Eh6ELspf6tRMzC2dHRcHlUbnkT7x9xf2eo79qLtG/sXmFsQo1ky9N6u xEMaJKN4+obKgNm9Q0ZtqBNIcgEyYOA4prbyofmcviy9YFwu9MpPHVdXLIgwrsR0oh64 8ENtR9R823dn1OGHW0hJO8/IAceihrlv1ohaqTUuxl91b6jpfpA9E+LWxLDz4E4iAeyM 2YuRaMjBkwzWfBPog867WqoDkfrtNA/aNkTie4GdZU1w9ov6Nqd+C0O6kA4n6vyQwTuR tRYmCuyUcoGyI3f9kIg3H5JqF1Bf9OV8LERBvs8nRIceSvgdV8CPzQZJBk3ZOlWuG/M7 hMJQ== X-Gm-Message-State: AG10YOSXGdrcordZ9KV1cOlVYShR5/6AC2VdjB//0hdZAwZeUkZUT43qcqQXSxvnK04YaWRZsGsZG1og8/g12Q== MIME-Version: 1.0 X-Received: by 10.107.170.79 with SMTP id t76mr17307424ioe.71.1455488869325; Sun, 14 Feb 2016 14:27:49 -0800 (PST) Received: by 10.79.84.71 with HTTP; Sun, 14 Feb 2016 14:27:49 -0800 (PST) Reply-To: alc@freebsd.org In-Reply-To: <56C08AAA.5050206@FreeBSD.org> References: <56BBAB6E.5050601@FreeBSD.org> <56C08AAA.5050206@FreeBSD.org> Date: Sun, 14 Feb 2016 16:27:49 -0600 Message-ID: Subject: Re: Memory modified after free in "MAP ENTRY" zone (vm_map_entry_t->read_ahead) From: Alan Cox To: Andriy Gapon Cc: FreeBSD Current Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Feb 2016 22:27:50 -0000 On Sun, Feb 14, 2016 at 8:09 AM, Andriy Gapon wrote: > On 10/02/2016 23:28, Andriy Gapon wrote: > > > > Over a span of approximately 3 weeks I have got two slightly different > panics of > > the same kind. The affected system is a several months old amd64 head. > > I added a small assertion and it got tripped: > --- a/sys/vm/vm_fault.c > +++ b/sys/vm/vm_fault.c > @@ -608,8 +608,11 @@ readrest: > cluster_offset; > } > ahead = ulmin(ahead, atop(fs.entry->end - vaddr) - > 1); > - if (era != nera) > + if (era != nera) { > + KASSERT(fs.lookup_still_valid, > + ("modifying read_ahead without map > lock")); > fs.entry->read_ahead = nera; > + } > > /* > * Call the pager to retrieve the data, if any, > after > > So, it seems that the code allows modification of read_ahead field of an > entry > without holding a map lock. I'd guess that that should be mostly harmless > as > read_ahead value is not critical, but it is still not nice. > > Not intentionally. The nearby code to the read_ahead assignment expects the map to be locked, so I wouldn't categorize this as mostly harmless. In general, you shouldn't get to the read_ahead assignment without the map being locked, and almost all of the code paths that unlock the map jump to RetryFault shortly thereafter, so it's hard to imagine how the map lock wouldn't be reacquired. I speculate that the root cause of your panic is a case where vm_pager_get_pages() fails, and in such a case we might loop around, descending to the next backing object without reacquiring the map lock. In other words, your above assertion failure is happening on the next iteration after an initial vm_pager_get_pages() failure, and we're about to do another vm_pager_get_pages() call on a different object. In summary, I have no trouble believing that the code that handles a failure by vm_pager_get_pages() in vm_fault() is not quite right, and I think that's what's biting you. > Some details from the panic caused by the assertion can be found here: > http://dpaste.com/39BYV7S.txt > > > > ======================== 1 =========================================== > > Unread portion of the kernel message buffer: > > panic: Memory modified after free 0xfffff8008c15ac80(128) val=adc0de @ > > 0xfffff8008c15acdc > > > > KDB: stack backtrace: > > db_trace_self_wrapper() at 0xffffffff8041e90b = > db_trace_self_wrapper+0x2b/frame > > 0xfffffe04f5349530 > > kdb_backtrace() at 0xffffffff80669a09 = kdb_backtrace+0x39/frame > 0xfffffe04f53495e0 > > vpanic() at 0xffffffff80634dec = vpanic+0x14c/frame 0xfffffe04f5349620 > > panic() at 0xffffffff80634b33 = panic+0x43/frame 0xfffffe04f5349680 > > trash_ctor() at 0xffffffff807de9c8 = trash_ctor+0x48/frame > 0xfffffe04f5349690 > > uma_zalloc_arg() at 0xffffffff807da785 = uma_zalloc_arg+0x475/frame > > 0xfffffe04f5349720 > > uma_zalloc() at 0xffffffff807e44af = uma_zalloc+0xf/frame > 0xfffffe04f5349730 > > vm_map_entry_create() at 0xffffffff807e5a2e = > vm_map_entry_create+0x2e/frame > > 0xfffffe04f5349740 > > _vm_map_clip_start() at 0xffffffff807e69e3 = > _vm_map_clip_start+0x123/frame > > 0xfffffe04f5349770 > > vm_map_wire() at 0xffffffff807e797d = vm_map_wire+0x11d/frame > 0xfffffe04f53497f0 > > vslock() at 0xffffffff807e1a2b = vslock+0x6b/frame 0xfffffe04f5349810 > > sysctl_wire_old_buffer() at 0xffffffff8064148a = > > sysctl_wire_old_buffer+0x4a/frame 0xfffffe04f5349830 > > sysctl_kern_proc() at 0xffffffff8062259b = sysctl_kern_proc+0x8b/frame > > 0xfffffe04f5349880 > > sysctl_root_handler_locked() at 0xffffffff80641a8e = > > sysctl_root_handler_locked+0x8e/frame 0xfffffe04f53498c0 > > sysctl_root() at 0xffffffff806412fe = sysctl_root+0x13e/frame > 0xfffffe04f5349940 > > userland_sysctl() at 0xffffffff8064183d = userland_sysctl+0x16d/frame > > 0xfffffe04f53499e0 > > sys___sysctl() at 0xffffffff80641694 = sys___sysctl+0x74/frame > 0xfffffe04f5349a90 > > syscallenter() at 0xffffffff8081fa20 = syscallenter+0x320/frame > 0xfffffe04f5349b00 > > amd64_syscall() at 0xffffffff8081f5ef = amd64_syscall+0x1f/frame > 0xfffffe04f5349bf0 > > Xfast_syscall() at 0xffffffff80807c5b = Xfast_syscall+0xfb/frame > 0xfffffe04f5349bf0 > > --- syscall (202, FreeBSD ELF64, sys___sysctl), rip = 0x8042225ea, rsp = > > 0x7fffffffcef8, rbp = 0x7fffffffcf30 --- > > Uptime: 14d22h38m17s > > ====================================================================== > > > > ======================== 2 =========================================== > > Unread portion of the kernel message buffer: > > panic: Memory modified after free 0xfffff80176692680(128) val=adc0de @ > > 0xfffff801766926dc > > > > KDB: stack backtrace: > > db_trace_self_wrapper() at 0xffffffff8041e90b = > db_trace_self_wrapper+0x2b/frame > > 0xfffffe04f507c5a0 > > kdb_backtrace() at 0xffffffff80669a09 = kdb_backtrace+0x39/frame > 0xfffffe04f507c650 > > vpanic() at 0xffffffff80634dec = vpanic+0x14c/frame 0xfffffe04f507c690 > > panic() at 0xffffffff80634b33 = panic+0x43/frame 0xfffffe04f507c6f0 > > trash_ctor() at 0xffffffff807de9c8 = trash_ctor+0x48/frame > 0xfffffe04f507c700 > > uma_zalloc_arg() at 0xffffffff807da785 = uma_zalloc_arg+0x475/frame > > 0xfffffe04f507c790 > > uma_zalloc() at 0xffffffff807e44af = uma_zalloc+0xf/frame > 0xfffffe04f507c7a0 > > vm_map_entry_create() at 0xffffffff807e5a2e = > vm_map_entry_create+0x2e/frame > > 0xfffffe04f507c7b0 > > vm_map_insert() at 0xffffffff807e558a = vm_map_insert+0x2fa/frame > 0xfffffe04f507c850 > > vm_map_stack_locked() at 0xffffffff807e63cb = > vm_map_stack_locked+0x13b/frame > > 0xfffffe04f507c8b0 > > vm_map_find() at 0xffffffff807e65d3 = vm_map_find+0x183/frame > 0xfffffe04f507c950 > > vm_mmap_object() at 0xffffffff807eac99 = vm_mmap_object+0x329/frame > > 0xfffffe04f507c9d0 > > sys_mmap() at 0xffffffff807ea8ec = sys_mmap+0x41c/frame > 0xfffffe04f507ca90 > > syscallenter() at 0xffffffff8081fa20 = syscallenter+0x320/frame > 0xfffffe04f507cb00 > > amd64_syscall() at 0xffffffff8081f5ef = amd64_syscall+0x1f/frame > 0xfffffe04f507cbf0 > > Xfast_syscall() at 0xffffffff80807c5b = Xfast_syscall+0xfb/frame > 0xfffffe04f507cbf0 > > --- syscall (477, FreeBSD ELF64, sys_mmap), rip = 0x80418776a, rsp = > > 0x7fffffffb808, rbp = 0x7fffffffb840 --- > > Uptime: 4d4h36m36s > > ====================================================================== > > > > I have crash dumps from both panics. > > It seems that it was read_ahead field that was reset to zero in both > cases. > > Perhaps the issue has been fixed already? > > I would appreciate any suggestions. Thanks! > > > > > -- > Andriy Gapon > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >