From owner-freebsd-stable@FreeBSD.ORG Thu May 28 14:38:15 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E19A6106564A; Thu, 28 May 2009 14:38:15 +0000 (UTC) (envelope-from hlh@restart.be) Received: from tignes.restart.be (tignes.restart.be [IPv6:2001:41d0:2:2d29:0:1::]) by mx1.freebsd.org (Postfix) with ESMTP id 68E4E8FC15; Thu, 28 May 2009 14:38:15 +0000 (UTC) (envelope-from hlh@restart.be) Received: from restart.be (avoriaz.tunnel.bel [IPv6:2001:41d0:2:2d29:1:ffff::]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "avoriaz.restart.be", Issuer "CA master" (verified OK)) by tignes.restart.be (Postfix) with ESMTPS id 92CCE4C42; Thu, 28 May 2009 16:38:14 +0200 (CEST) Received: from avoriaz.restart.bel (avoriaz.restart.be [IPv6:2001:41d0:2:2d29:1:1::]) (authenticated bits=0) by restart.be (8.14.3/8.14.3) with ESMTP id n4SEcBIA070898; Thu, 28 May 2009 16:38:11 +0200 (CEST) (envelope-from hlh@restart.be) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=restart.be; s=avoriaz; t=1243521494; bh=pW11BcVyUMQtByVgSXhdJAI9gtGG5zyPLHkVRax9jEg=; h=Message-ID:Date:From:MIME-Version:To:CC:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=vl4E5NeTAuH5XxJEWp60o/pKv5j9wvG7TszeThL84H1DRa8cgHg8RiaVswjwQGK+Q pfZj3Yqqaw98JmPLBYE9g== DomainKey-Signature: a=rsa-sha1; s=avoriaz; d=restart.be; c=nofws; q=dns; h=message-id:date:from:organization:user-agent:mime-version:to:cc: subject:references:in-reply-to:content-type: content-transfer-encoding:x-scanned-by; b=Q+mvpryMGd1eXM6VclAxHfoBn6JjIjxWX8OTuq4L4ByGOFon/jMQGyPN3WSDMRSHs S0CRWsvFgVFKlOL7uU1dg== Message-ID: <4A1EA1D3.405@restart.be> Date: Thu, 28 May 2009 16:38:11 +0200 From: Henri Hennebert Organization: RestartSoft User-Agent: Thunderbird 2.0.0.21 (X11/20090412) MIME-Version: 1.0 To: Andriy Gapon References: <3c1674c90905201643m540c8b1v8a8bd88f071c233d@mail.gmail.com> <4A1D0F2B.4030006@restart.be> <3c1674c90905280052q281f6172j2409fe2d64db6914@mail.gmail.com> <4A1E90F7.2000000@restart.be> <4A1E97D8.4080901@icyb.net.ua> In-Reply-To: <4A1E97D8.4080901@icyb.net.ua> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.64 on IPv6:2001:41d0:2:2d29:1:1:: Cc: freebsd-stable@freebsd.org, Kip Macy , Artem Belevich Subject: Re: ZFS MFC heads down X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 May 2009 14:38:16 -0000 Andriy Gapon wrote: > on 28/05/2009 16:26 Henri Hennebert said the following: >> (gdb) bt >> #0 0x00000008012a6f22 in strlen () from /lib/libc.so.7 >> #1 0x00000008012a0feb in open () from /lib/libc.so.7 >> #2 0x000000080129ea59 in open () from /lib/libc.so.7 >> #3 0x00000008012a1f2e in vfprintf () from /lib/libc.so.7 >> #4 0x0000000801291158 in fprintf () from /lib/libc.so.7 >> #5 0x0000000801290fb0 in __assert () from /lib/libc.so.7 > > I find the above part interesting. > Could this be because of the following discrepancy: > > 1) > cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h: > extern void __assert(const char *, const char *, int); > 2) > lib/libc/gen/assert.c: > void > __assert(func, file, line, failedexpr) > const char *func, *file; > int line; > const char *failedexpr; > >> #6 0x0000000800fef120 in zmutex_destroy () from /lib/libzpool.so.1 >> #7 0x000000080102e1a0 in dsl_dataset_fast_stat () from /lib/libzpool.so.1 >> #8 0x0000000801045ffa in dbuf_find () from /lib/libzpool.so.1 >> #9 0x0000000801047bf3 in dmu_buf_rele () from /lib/libzpool.so.1 >> #10 0x0000000801027546 in dsl_pool_open () from /lib/libzpool.so.1 >> #11 0x000000080101bcec in spa_create () from /lib/libzpool.so.1 >> #12 0x000000080101c820 in spa_tryimport () from /lib/libzpool.so.1 > > But back to the problem - without an additional printf we still can not what was > the value in m_owner. Only that it was not null. > Probably it's better to build with debugging symbols and examine with gdb. Firt try: [root@avoriaz libzpool]# gdb zdb GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)... (gdb) r pool1 Starting program: /usr/sbin/zdb pool1 (no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...[New LWP 100299] (no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...[New Thread 0x8018020b0 (LWP 100299)] [New Thread 0x801802240 (LWP 100354)] version=13 name='pool1' state=0 txg=4 pool_guid=9156958376606789 hostid=1133576597 hostname='unset' vdev_tree type='root' id=0 guid=9156958376606789 children[0] type='raidz' id=0 guid=8214939615613279020 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=500108886016 is_log=0 children[0] type='disk' id=0 guid=7001907692988243779 path='/dev/ad8p2' whole_disk=0 children[1] type='disk' id=1 guid=1909032920962573263 path='/dev/ad10p2' whole_disk=0 [New Thread 0x8018023d0 (LWP 100369)] [New Thread 0x801802560 (LWP 100370)] [New Thread 0x8018026f0 (LWP 100371)] [New Thread 0x801802880 (LWP 100372)] [New Thread 0x801802a10 (LWP 100376)] [New Thread 0x801802ba0 (LWP 100382)] [New Thread 0x801802d30 (LWP 100383)] [New Thread 0x801802ec0 (LWP 100384)] [New Thread 0x801803050 (LWP 100385)] [New Thread 0x8018031e0 (LWP 100386)] [New Thread 0x801803370 (LWP 100387)] [New Thread 0x801803500 (LWP 100388)] [New Thread 0x801803690 (LWP 100389)] [New Thread 0x801803820 (LWP 100390)] [New Thread 0x8018039b0 (LWP 100391)] [New Thread 0x801803b40 (LWP 100392)] [New Thread 0x801803cd0 (LWP 100393)] [New Thread 0x801803e60 (LWP 100394)] [New Thread 0x801803ff0 (LWP 100395)] [New Thread 0x801804180 (LWP 100396)] [New Thread 0x801804310 (LWP 100397)] [New Thread 0x8018044a0 (LWP 100398)] [New Thread 0x801804630 (LWP 100399)] [New Thread 0x8018047c0 (LWP 100400)] [New Thread 0x801804950 (LWP 100401)] [New Thread 0x801804ae0 (LWP 100402)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x8018020b0 (LWP 100299)] 0x00000008012a6f22 in strlen () from /lib/libc.so.7 (gdb) bt #0 0x00000008012a6f22 in strlen () from /lib/libc.so.7 #1 0x00000008012a0feb in open () from /lib/libc.so.7 #2 0x000000080129ea59 in open () from /lib/libc.so.7 #3 0x00000008012a1f2e in vfprintf () from /lib/libc.so.7 #4 0x0000000801291158 in fprintf () from /lib/libc.so.7 #5 0x0000000801290fb0 in __assert () from /lib/libc.so.7 #6 0x0000000800fef230 in zmutex_destroy (mp=0x8018b2cc0) at /usr/src/cddl/lib/libzpool/../../../cddl/contrib/opensolaris/lib/libzpool/common/kernel.c:112 #7 0x000000080102e2b0 in dsl_dataset_evict (db=Variable "db" is not available. ) at /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:266 #8 0x000000080104610a in dbuf_evict_user (db=0x8018ca960) at /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:215 #9 0x0000000801047d03 in dbuf_rele (db=0x8018ca960, tag=Variable "tag" is not available. ) at /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1739 #10 0x0000000801027656 in dsl_pool_open (spa=0x8018a2000, txg=Variable "txg" is not available. ) at /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c:139 #11 0x000000080101bdfc in spa_load (spa=0x8018a2000, config=Variable "config" is not available. ) at /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:1134 #12 0x000000080101c930 in spa_open_common (pool=0x7fffffffee06 "pool1", spapp=0x7fffffffebb8, tag=0x40b790, config=0x0) at /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:1474 #13 0x0000000000408b41 in ?? () #14 0x00000000004036de in ?? () #15 0x0000000800534000 in ?? () #16 0x0000000000000000 in ?? () #17 0x0000000000000000 in ?? () #18 0x0000000000000002 in ?? () #19 0x00007fffffffedf8 in ?? () #20 0x00007fffffffee06 in ?? () #21 0x0000000000000000 in ?? () #22 0x00007fffffffee0c in ?? () #23 0x00007fffffffee22 in ?? () #24 0x00007fffffffee2d in ?? () #25 0x00007fffffffee47 in ?? () #26 0x00007fffffffee64 in ?? () #27 0x00007fffffffee6e in ?? () #28 0x00007fffffffee79 in ?? () #29 0x00007fffffffee85 in ?? () #30 0x00007fffffffee9a in ?? () #31 0x00007fffffffeeae in ?? () #32 0x00007fffffffef08 in ?? () #33 0x00007fffffffef17 in ?? () #34 0x00007fffffffef23 in ?? () #35 0x00007fffffffef42 in ?? () #36 0x00007fffffffef4f in ?? () #37 0x00007fffffffef64 in ?? () #38 0x00007fffffffef76 in ?? () #39 0x00007fffffffef7f in ?? () #40 0x00007fffffffef8f in ?? () #41 0x00007fffffffef97 in ?? () #42 0x0000000000000000 in ?? () #43 0x0000000000000003 in ?? () #44 0x0000000000400040 in ?? () #45 0x0000000000000004 in ?? () #46 0x0000000000000038 in ?? () #47 0x0000000000000005 in ?? () ---Type to continue, or q to quit---q Quit (gdb) frame 6 #6 0x0000000800fef230 in zmutex_destroy (mp=0x8018b2cc0) at /usr/src/cddl/lib/libzpool/../../../cddl/contrib/opensolaris/lib/libzpool/common/kernel.c:112 112 ASSERT(mp->m_owner == NULL); (gdb) print mp->m_owner $1 = (void *) 0x8018f2060 As I understand mp->m_owner is a thread ref ... Feel free to ask for more Henri >