From owner-freebsd-geom@FreeBSD.ORG Sat Jun 23 07:00:34 2007 Return-Path: X-Original-To: freebsd-geom@freebsd.org Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1DACB16A400 for ; Sat, 23 Jun 2007 07:00:34 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from mgw-ext14.nokia.com (smtp.nokia.com [131.228.20.173]) by mx1.freebsd.org (Postfix) with ESMTP id A05E413C45B for ; Sat, 23 Jun 2007 07:00:33 +0000 (UTC) (envelope-from david.cecil@nokia.com) Received: from esebh107.NOE.Nokia.com (esebh107.ntc.nokia.com [172.21.143.143]) by mgw-ext14.nokia.com (Switch-3.2.5/Switch-3.2.5) with ESMTP id l5N70SZD000454 for ; Sat, 23 Jun 2007 10:00:31 +0300 Received: from siebh101.NOE.Nokia.com ([172.30.195.27]) by esebh107.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Sat, 23 Jun 2007 10:00:30 +0300 Received: from syebe101.NOE.Nokia.com ([172.30.128.65]) by siebh101.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Sat, 23 Jun 2007 15:00:28 +0800 Received: from [172.30.67.199] ([172.30.67.199]) by syebe101.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Sat, 23 Jun 2007 17:00:26 +1000 Message-ID: <467CC50A.3080900@nokia.com> Date: Sat, 23 Jun 2007 17:00:26 +1000 From: David Cecil User-Agent: Thunderbird 1.5.0.12 (Windows/20070509) MIME-Version: 1.0 To: freebsd-geom@freebsd.org References: <467CA5B0.4090508@nokia.com> In-Reply-To: <467CA5B0.4090508@nokia.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 23 Jun 2007 07:00:26.0913 (UTC) FILETIME=[2D903110:01C7B564] X-Nokia-AV: Clean Cc: ext David Cecil Subject: Re: Duplicate free X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jun 2007 07:00:34 -0000 A little more information that I hope might trigger some thoughts. I have also seen a trace very similar to the one in my original mail, but instead of bio_done calling g_disk_done, it calls g_mirror_done. This time it panics because it tries to dereference the bio_from field. The common thread is that in both cases, the bio is freed prior to the done handler referencing it (well at least the bio_from field is NULL in the g_mirror_case, so I'm assuming it was freed). I wonder if these handlers are not playing nicely together? Some locking required between them that is missing maybe? Unread portion of the kernel message buffer: ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=46231338 Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0x0 fault code = supervisor read, page not present instruction pointer = 0x20:0x8053318b stack pointer = 0x28:0xe1466c4c frame pointer = 0x28:0xe1466c54 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 19 (swi6: task queue) trap number = 12 panic: page fault cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper(8075f7be) at db_trace_self_wrapper+0x25 kdb_backtrace(100,863ba190,28,e1466c0c,c,...) at kdb_backtrace+0x29 panic(807403d4,8077f4a5,0,fffff,863be29b,...) at panic+0x124 trap_fatal(e1466c0c,0,863ba190,0,c,...) at trap_fatal+0x2ce trap_pfault(e1466c0c,0,0) at trap_pfault+0x1e7 trap(e1460008,28,28,8a0cb840,89b1a6b4,...) at trap+0x36d calltrap() at calltrap+0x5 --- trap 0xc, eip = 0x8053318b, esp = 0xe1466c4c, ebp = 0xe1466c54 --- g_mirror_done(89b1a6b4) at g_mirror_done+0xb biodone(89b1a6b4) at biodone+0x8b ad_done(8a0cb840) at ad_done+0x2a ata_completed(8a0cb840,0) at ata_completed+0x534 taskqueue_run(8644bac0,e1466cec,80566279,0,0,...) at taskqueue_run+0xbd taskqueue_swi_run(0) at taskqueue_swi_run+0xe ithread_execute_handlers(863be200,86433000) at ithread_execute_handlers+0x139 ithread_loop(86468460,e1466d38) at ithread_loop+0x64 fork_exit(805662ec,86468460,e1466d38) at fork_exit+0x71 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe1466d6c, ebp = 0 --- ext David Cecil wrote: > Hi, > > I've encountered a duplicate free in 6.1-RELEASE-based code. I have > noticed that the same stack trace was reported about two years ago on > a number of occasions, and some code was added to try and help debug > the situation. However, I don't see any resolution. Does anyone have > any more information on this before I try and debug it further? > > Any hints for trying to find who freed it first? Maybe I should add > the KTR debug that went into 1.66 of geom_io.c. > > db> bt > Tracing pid 17 tid 100016 td 0x86badaf0 > kdb_enter(80750631) at kdb_enter+0x2b > panic(807786cd,8a778d80,81856780,80747d25,807786b1,...) at panic+0x137 > uma_dbg_free(81856780,0,8a778d80) at uma_dbg_free+0x110 > uma_zfree_arg(81856780,8a778d80,0) at uma_zfree_arg+0x66 > g_destroy_bio(8a778d80,805319b4,8a778d80,e1ca3c60,805c9620,...) at > g_destroy_bio+0x13 > g_disk_done(8a778d80) at g_disk_done+0x62 > biodone(8a778d80) at biodone+0x58 > ad_done(88042840) at ad_done+0x2a > ata_completed(88042840,0,86c38cdc,0,80753b6d,...) at ata_completed+0x504 > taskqueue_run(86c38cc0,e1ca3cec,8056999a,0,0,...) at taskqueue_run+0x86 > taskqueue_swi_run(0) at taskqueue_swi_run+0xe > ithread_execute_handlers(86c14000,86c29280) at > ithread_execute_handlers+0xfa > ithread_loop(86c534f0,e1ca3d38,86c534f0,80569a10,0,...) at > ithread_loop+0x76 > fork_exit(80569a10,86c534f0,e1ca3d38) at fork_exit+0xa0 > fork_trampoline() at fork_trampoline+0x8 > --- trap 0x1, eip = 0, esp = 0xe1ca3d6c, ebp = 0 --- > > The panic string is: > Duplicate free of item 0x8a778d80 from zone 0x81856780(g_bio) > > Thanks, > Dave > _______________________________________________ > freebsd-geom@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-geom > To unsubscribe, send any mail to "freebsd-geom-unsubscribe@freebsd.org" -- Software Engineer Secure and Mobile Connectivity Enterprise Solutions Nokia +61 7 5553 8307 (office) +61 412 728 222 (cell)