From owner-freebsd-current@freebsd.org Thu Aug 6 02:27:23 2015 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1DD079B2DB3 for ; Thu, 6 Aug 2015 02:27:23 +0000 (UTC) (envelope-from freebsd@grem.de) Received: from mail.grem.de (outcast.grem.de [213.239.217.27]) by mx1.freebsd.org (Postfix) with SMTP id 5DD0198D for ; Thu, 6 Aug 2015 02:27:21 +0000 (UTC) (envelope-from freebsd@grem.de) Received: (qmail 32984 invoked by uid 89); 6 Aug 2015 02:20:40 -0000 Received: from unknown (HELO bsd64.grem.de) (mg@grem.de@185.17.207.40) by mail.grem.de with ESMTPA; 6 Aug 2015 02:20:40 -0000 Date: Thu, 6 Aug 2015 04:20:39 +0200 From: Michael Gmelin To: Pawel Jakub Dawidek Cc: Ed Maste , FreeBSD Current Subject: Re: Memory modified after free, seemingly geli related Message-ID: <20150806042039.78aa4ad3@bsd64.grem.de> In-Reply-To: <20150806020639.GA72832@garage.freebsd.pl> References: <20150806020639.GA72832@garage.freebsd.pl> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.25; amd64-portbld-freebsd10.0) X-Face: $wrgCtfdVw_H9WAY?S&9+/F"!41z'L$uo*WzT8miX?kZ~W~Lr5W7v?j0Sde\mwB&/ypo^}> +a'4xMc^^KroE~+v^&^#[B">soBo1y6(TW6#UZiC]o>C6`ej+i Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWJBwe5BQDl LASZU0/LTEWEfHbyj0Txi32+sKrp1Mv944X8/fm1rS+cAAAACXBIWXMAAAsTAAAL EwEAmpwYAAAAB3RJTUUH3wESCxwC7OBhbgAAACFpVFh0Q29tbWVudAAAAAAAQ3Jl YXRlZCB3aXRoIFRoZSBHSU1QbbCXAAAAAghJREFUOMu11DFvEzEUAGCfEhBVFzuq AKkLd0O6VrIQsLXVSZXoWE5N1K3DobBBA9fQpRWc8OkWouaIjedWKiyREOKs+3PY fvalCNjgLVHeF7/3bMtBzV8C/VsQ8tecEgCcDgrzjekwKZ7TwsJZd/ywEKwwP+ZM 8P3drTsAwWn2mpWuDDuYiK1bFs6De0KUUFw0tWxm+D4AIhuuvZqtyWYeO7jQ4Aea 7jUqI+ixhQoHex4WshEvSXdood7stlv4oSuFOC4tqGcr0NjEqXgV4mMJO38nld4+ xKNxRDon7khyKVqY7YR4d+Cg0OMrkWXZOM7YDkEfKiilCn1qYv4mighZiynuHHOA Wq9QJq+BIES7lMFUtcikMnkDGHUoncA+uHgrP0ctIEqfwLHzeSo+eUA66AqzwN6n 2ZHJhw6Qh/PoyC/QENyEyC/AyNjq74Bs+3UH0xYwzDUC4B97HgLocg1QLYgDDO1v f3UX9Y307Ew4AHh67YAFFsxEpkXwpXY3eIgMhAAE3R19L919nNnuD2wlPcDE3UeT L2ytEICQib9BXgS2fU8PrD82ToYO1OEmMSnYTjSqSv9wdC0tPYC+rQRQD9ESnldF CyqfmiYW+tlALt8gH2xrMdC/youbjzPXEun+/ReXsMCDyve3dZc09fn2Oas8oXGc Jj6/fOeK5UmSMPmf/jL+GD8BEj0k/Fn6IO4AAAAASUVORK5CYII= MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Aug 2015 02:27:23 -0000 On Thu, 6 Aug 2015 04:06:40 +0200 Pawel Jakub Dawidek wrote: > On Wed, Aug 05, 2015 at 03:24:26AM +0000, Ed Maste wrote: > > I've encountered a few memory modified after free panics recently, > > which seem to be from geli. I don't yet have any debugging to > > completely confirm it's geli, but it has not happened on my other > > test laptop which configured similarly but without geli. > > > > This has a few local patches from my to-commit-to-HEAD queue. > > FreeBSD volta 11.0-CURRENT FreeBSD 11.0-CURRENT #10 > > r284409+6a002d9(staging): Tue Jul 7 17:57:01 EDT 2015 > > > > panic: Memory modified after free 0xfffff80009d504d8(248) val=0 @ > > 0xfffff80009d50518 > > I'm seeing it too. I tracked it down to ZFS. The bio was last owned by > the ZFS::VDEV GEOM class, which is modyfing bio_error on freed bio. > I'm investigating further and will let you know here once I find the > cause. > > > cpuid = 1 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > > 0xfffffe011414a880 vpanic() at vpanic+0x189/frame 0xfffffe011414a900 > > panic() at panic+0x43/frame 0xfffffe011414a960 > > trash_ctor() at trash_ctor+0x48/frame 0xfffffe011414a970 > > uma_zalloc_arg() at uma_zalloc_arg+0x573/frame 0xfffffe011414a9e0 > > g_clone_bio() at g_clone_bio+0x1d/frame 0xfffffe011414aa00 > > g_eli_start() at g_eli_start+0xbd/frame 0xfffffe011414aa30 > > g_io_schedule_down() at g_io_schedule_down+0xe6/frame > > 0xfffffe011414aa60 g_down_procbody() at g_down_procbody+0x7d/frame > > 0xfffffe011414aa70 fork_exit() at fork_exit+0x84/frame > > 0xfffffe011414aab0 fork_trampoline() at fork_trampoline+0xe/frame > > 0xfffffe011414aab0 --- trap 0, rip = 0, rsp = 0xfffffe011414ab70, > > rbp = 0 --- > I've seen those as well while destroying 190.000 zfs snapshots (caused by an lpreserver runaway). Got four panics in the process, also running on top of geli. I'll mail you screenshots off-list. - Michael -- Michael Gmelin