From owner-freebsd-fs@FreeBSD.ORG Sun May 17 09:17:43 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 95C02BF0 for ; Sun, 17 May 2015 09:17:43 +0000 (UTC) Received: from smtp.free.de (smtp.free.de [91.204.6.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 019EA17D4 for ; Sun, 17 May 2015 09:17:42 +0000 (UTC) Received: (qmail 76948 invoked from network); 17 May 2015 11:17:32 +0200 Received: from smtp.free.de (HELO [192.168.178.21]) (k@free.de@[91.204.4.103]) (envelope-sender ) by smtp.free.de (qmail-ldap-1.03) with AES128-SHA encrypted SMTP for ; 17 May 2015 11:17:32 +0200 Message-ID: <55585CAA.6080105@free.de> Date: Sun, 17 May 2015 11:17:30 +0200 From: Kai Gallasch Organization: FREE! User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: ZFS RAID 10 capacity expansion and uneven data distribution References: In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="mLJCukBQA4ftFBfQCgHnoluGGbgsecphj" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 May 2015 09:17:43 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --mLJCukBQA4ftFBfQCgHnoluGGbgsecphj Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 14.05.2015 15:59 Daniel Kalchev wrote: > Not a total bs, but.. it could be made simpler/safer. >=20 > skip 2,3,4 and 5 > 7a. zfs snapshot -r zpool.old@send > 7b. zfs send -R zpool.old@send | zfs receive -F zpool > do not skip 8 :) > 11. zpool attach zpool da1 da2 && zpool attach zpool da3 da4 Somehow nifty. I tried this on a test server and found out, that after the zpool split it is safer to do the import with e.g."-o altroot=3D/mnt"= , because if you are doing this on a root pool the import will be mounted over the existing root fs and the zfs installation becomes unusable at this point. (-> reboot) Also the zfs receive should not mount the received Filesystem. > After this operation, you should have the exact same zpool, with evenly= redistributed data. You could use the chance to change ashift etc. Sadly= , this works only for mirrors. In my case this is not true. After completion, data is still not evenly distributed across the mirror pairs and each pair has a differing FRAG value. (Before doing the zfs send I destroyed the old z filesystems on the receiving side..) Although when accessing the data afterwards the situation with one mirror pair being overused and the other almost idle has become better - so this method fixes the problem a bit.. When I recreate the pool and restore the data the picture looks different: Data then is really equal size across the mirrors and they all have the same FRAG value - as expected. My conclusion: Expanding a RAID 10 zpool by adding mirrored vdevs is not really an option if you also want to benefit from the gained IOPS of the new devices. In this case recreating the pool is the cleanest solution. If you cannot recreate the pool you can think about this zpool split hack to reditribute data across all vdevs - althoug you temporarly loose your pool redundancy between zpool split and the end of the resilvering process (risky) Kai. --mLJCukBQA4ftFBfQCgHnoluGGbgsecphj Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCgAGBQJVWFyqAAoJEHBlTXxPsfWIJEcQAIn1U7KXsfPHVnJIXHr/QrY7 dgegqx5zLdDGYZYbgxVvU2N4yxs7YktNcMrdXeXqT/y4kf59/5Q7ByRjldE3d9VI fZSO74F9TETjajH87OjFP7opSScfv/FSxAZtveHXtrGsw6KyQUu2H4i+EUWnvpMA C1RgEnDixyX6JyDPiesgdtAUxa/l7fRINNWoKr+o6GpZJ3Btvtibs/JTDYd+Zyv6 +f7H3NCN79OrAyIhmbR+1S59RPPRg5OgZaGI5EGnzWs9df/0uq/pEVAzDn0qF4OI 4tgxPpYDybHtbVF31WZsii6JiVDOVueoTwoDNhGcumDAt5NFDznrnC7pC+FmNkro e8qjBTonbSI+YQo/RvN+PL4pESxK/lV0aPeLguZpUEgQPKoj4vSWS2GdLOfd2eIX b247G56KDRnMjPtxragaP9AWkNh/AuJdLMRyP+Y4hvpQN2N+zVlmkD5OeffK2cQy J2+jvhH9mWPpNtZFYsV8G72gybFyUre6EMQQnTrkmr4miDgmGIkrRzSFASvmJMwr LtsizhH6vmtLVClI5jbaB+ZSHhwTyclxyJBRZT9zGht2xm2xkdGdM1FhAL86jkTl QglskgFo6bhihBxG9gV5RuHDsGEcgbl8I/Whne4r4qPrLYjIPvewUBB8NbHLuw/a 87NM1iquyGb5kJ3JsZN2 =FQwl -----END PGP SIGNATURE----- --mLJCukBQA4ftFBfQCgHnoluGGbgsecphj-- From owner-freebsd-fs@FreeBSD.ORG Sun May 17 18:03:35 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5357C1FA for ; Sun, 17 May 2015 18:03:35 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3D01F1C1F for ; Sun, 17 May 2015 18:03:35 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4HI3ZIF078669 for ; Sun, 17 May 2015 18:03:35 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 197789] (zfs+i386 No PAE) panic: kmem_malloc(36864): kmem_map too small: 431976448 total allocated Date: Sun, 17 May 2015 18:03:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: michelle@sorbs.net X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 May 2015 18:03:35 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197789 --- Comment #4 from Michelle Sullivan --- Ok got it down to something a little more clearer... No changing of vm.kmem_size_max has any effect except delaying the issue... just as vm.kmem_size doesn't. What does stop the panic (so far I've build 800+ packages using poudriere - with 'svn update' prior - which before the update didn't even complete and after going through several boots it manages around 20 packages before panicing - if it even gets past the checking sanity phase).... Reduce the CPU count to one cpu .... Interestingly setting max ARC size to 40M and within minutes of multiple CPUs it's over 400M ... before it panics... With one CPU it grows to 80(ish)M and doesn't panic... some 40 or 50 cores publicly available at the link i post earlier... both 9.2 and 9.3 are affected... lets hope someone will see to patch it before 9.4.. I wouldn't have a clue where to look for the issue or I'd take a shot at it myself. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Sun May 17 18:56:32 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B1688E0C for ; Sun, 17 May 2015 18:56:32 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9B5231146 for ; Sun, 17 May 2015 18:56:32 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4HIuWUS037165 for ; Sun, 17 May 2015 18:56:32 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 197789] (zfs+i386 No PAE) panic: kmem_malloc(36864): kmem_map too small: 431976448 total allocated Date: Sun, 17 May 2015 18:56:32 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mmoll@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 May 2015 18:56:32 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197789 Michael Moll changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mmoll@freebsd.org --- Comment #5 from Michael Moll --- Michelle, did you compile that kernel with raised KVA_PAGES as described in https://wiki.freebsd.org/ZFSTuningGuide#i386 ? -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Sun May 17 19:14:31 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D283E1B4 for ; Sun, 17 May 2015 19:14:31 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BC77B1313 for ; Sun, 17 May 2015 19:14:31 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4HJEVZk086766 for ; Sun, 17 May 2015 19:14:31 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 197789] (zfs+i386 No PAE) panic: kmem_malloc(36864): kmem_map too small: 431976448 total allocated Date: Sun, 17 May 2015 19:14:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: michelle@sorbs.net X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 May 2015 19:14:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197789 --- Comment #6 from Michelle Sullivan --- No I haven't - and quite deliberately (mainly because I saw that later and the systems are set to update using freebsd-update and I don't want the kernel to bugger up the patching)... and I wanted to see if I could get to the bottom of the cause or at least make it reproducible... I see no reason why zfs on i386 shouldn't work without the need to recompile the kernel (or zfs should be removed so that anyone wishing to use it should have to compile)... and I've finally got progress, single CPU and no panic, multiple CPUs and reproducible panic.. can it be looked at to resolve it..? (because in reality it has to be some form of bug - ARC should not exhaust the memory, and it should be constrained by the limits in the loader conf(ig)) -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Sun May 17 20:05:45 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E1D09A33 for ; Sun, 17 May 2015 20:05:45 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CBE231814 for ; Sun, 17 May 2015 20:05:45 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4HK5ja5063858 for ; Sun, 17 May 2015 20:05:45 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 197789] (zfs+i386 No PAE) panic: kmem_malloc(36864): kmem_map too small: 431976448 total allocated Date: Sun, 17 May 2015 20:05:45 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mmoll@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 May 2015 20:05:46 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197789 --- Comment #7 from Michael Moll --- It has been quite a while I used ZFS on i386, but from what I remember: - Default kernels can allocate 512MB max. as kmem (ALL kmem, not only ARC!) o That means ARC should be limited to 256MB or so, to still have room for other kernel tasks and some safety buffer. - Limiting the memory down to such values will make ZFS _very_ slow. - In general ZFS was not really designed for 32 bit systems anyway. - I used ZFS on i386 successfully with 4GB of RAM by setting: o options KVA_PAGES=512 in the custom kernel o vm.kmem_size and vm.kmem_size_max to 1536MB in loader.conf IMHO, at the end of the day the only advise here can be to move on to amd64 or if that's not possible to use a custom kernel with increased KVA_PAGES. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Sun May 17 21:00:51 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 07806573 for ; Sun, 17 May 2015 21:00:51 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D4E951DCA for ; Sun, 17 May 2015 21:00:50 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4HL0ojc046317 for ; Sun, 17 May 2015 21:00:50 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201505172100.t4HL0ojc046317@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 Date: Sun, 17 May 2015 21:00:50 +0000 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 May 2015 21:00:51 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f 3 problems total for which you should take action. From owner-freebsd-fs@FreeBSD.ORG Sun May 17 21:07:05 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EED3E9E9 for ; Sun, 17 May 2015 21:07:05 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C049C1ECF for ; Sun, 17 May 2015 21:07:05 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4HL751s013689 for ; Sun, 17 May 2015 21:07:05 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 197789] (zfs+i386 No PAE) panic: kmem_malloc(36864): kmem_map too small: 431976448 total allocated Date: Sun, 17 May 2015 21:07:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: michelle@sorbs.net X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 May 2015 21:07:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197789 --- Comment #8 from Michelle Sullivan --- 100% right with the 512MB limit. Setting arc_max to 40M is ignored... I have that now with a single CPU and it's showing (in "top"): Total: 52M 1858K MFU, 24M MRU, 400K Anon, 1975K Header, 24M Other If I set more than one CPU in the VM - the 'Total' will get to around 467M and then panic (same loader.conf settings) This I believe is the bug/a bug... Running ZFS on i386 should not be recommended, 100% with you... however: 1/ It is available in default kernels. 2/ KVA Pages needs to be set for default kernels. 3/ I'm not using ZFS in production anywhere - it's used for poudriere otherwise I'd disable it completely. We have a couple of things/statements here: 1/ Its available for use by default, recommended or not. 2/ It seems to work (all be it slowly) on i386 with the correct tuning. 3/ More than one CPU and it doesn't work 4/ KVA pages is not set by default. At least some should be resolved.... and this is how I see it: 1/ It either should be enabled by default and KVA_PAGES=512 set by default. 2/ It should be disabled by default with the warning that KVA_PAGES needs to be set if enabling. and: A/ someone should look into and resolve (if possible) the fact that arc_max is not respected when multiple CPUs are present. B/ documentation to indicating (1) and/or (2) should be updated (currently there is the link that you indicated, it probably should be expanded to indicate that ZFS is disabled and KVA_PAGES should be added if enabling, or that zfs is enabled and so it KVA_PAGES by default and what ever risk that may entail - it should also probably add that ZFS is really not for i386 because it will be really slow as it wasn't designed for 32 bit... [does that make sense?]) Thoughts? Not trying to be a pain here - but the default should work (even if that's "Disable ZFS in the default kernel") ... and I really think that 1 CPU working, 2 CPU = panic is a bug, and possibly an important one that may even be there on amd64 just it is not noticed because of the platform difference. Regards, Michelle -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Mon May 18 07:41:32 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C7CF85E5 for ; Mon, 18 May 2015 07:41:32 +0000 (UTC) Received: from mail-wi0-x235.google.com (mail-wi0-x235.google.com [IPv6:2a00:1450:400c:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 529721D6D for ; Mon, 18 May 2015 07:41:32 +0000 (UTC) Received: by wicnf17 with SMTP id nf17so59050662wic.1 for ; Mon, 18 May 2015 00:41:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=pQQiD0Rv0wA9DEwHewUyxtV3YAHm6mkmSgHDRxw5+aU=; b=wLefrjkFH8eqBPpGykeeP2d4uHJBt3pjftCURtPaGYSh25ipNHgLXDGX5o0qCgfEJ8 nTCDnV1KCU76vbo3xkMFnBCCZWPJsUVo/+LGbBjH2ivQB9pwRwODRtqiQjk1uORS+UQ2 fQ54fn8WCcYpJluHPLacx2cl77kWpdau834ZVObxRliU55AjxTJKqxrSyRqBV3fQw0cu RHnV4muhfN+njlinl9k7fhBjdGbB9uTn3HAqy3Xn7aGm+V5waKGyIbbhhzyehlKvdC1M fMBtO+6FCkzvK+sJaSQca+x+ScBpG/5Kq2zH4v+QClUmXR4sywoqY27Hcmr60Stmuk1t hnIQ== MIME-Version: 1.0 X-Received: by 10.180.188.100 with SMTP id fz4mr17473281wic.91.1431934890578; Mon, 18 May 2015 00:41:30 -0700 (PDT) Received: by 10.28.210.149 with HTTP; Mon, 18 May 2015 00:41:30 -0700 (PDT) In-Reply-To: References: Date: Mon, 18 May 2015 08:41:30 +0100 Message-ID: Subject: Re: ZFS RAID 10 capacity expansion and uneven data distribution From: krad To: Daniel Kalchev Cc: Gabor Radnai , FreeBSD FS Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 May 2015 07:41:32 -0000 depending on your dataset and you could also break it down into the file level rather than mess around with zfs send etc eg cp some_file some_file.new rm some_file mv some_file.new some_file just be careful with permissions etc (you might need a flag or to extra) On 14 May 2015 at 14:59, Daniel Kalchev wrote: > Not a total bs, but.. it could be made simpler/safer. > > skip 2,3,4 and 5 > 7a. zfs snapshot -r zpool.old@send > 7b. zfs send -R zpool.old@send | zfs receive -F zpool > do not skip 8 :) > 11. zpool attach zpool da1 da2 && zpool attach zpool da3 da4 > > Everywhere in the instruction where it says daX replace with gpt/zpool-da= X > as in the original config. > > After this operation, you should have the exact same zpool, with evenly > redistributed data. You could use the chance to change ashift etc. Sadly, > this works only for mirrors. > > Important to understand that since the first step you have an > non-redundant pool. It=E2=80=99s very reasonable to do a scrub before sta= rting this > process and of course have usable backup. > > Daniel > > > On 14.05.2015 =D0=B3., at 16:42, Gabor Radnai = wrote: > > > > Hi Kai, > > > > As others pointed out the cleanest way is to destroy / recreate your po= ol > > from backup. > > > > Though if you have no backup a hackish, in-place recreation process can > be > > the following. > > But please be *WARNED* it is your data, the recommended solution is to > use > > backup, > > if you follow below process it is your call - it may work but I cannot > > guarantee. You can > > have power outage, disk outage, sky falling down, whatever and you may > lose > > your data. > > And this may not even work - more skilled readers could bit me on head > how > > stupid this is. > > > > So, again be warned. > > > > If you are still interested: > > > >> On one server I am currently using a four disk RAID 10 zpool: > >> > >> zpool ONLINE 0 0 0 > >> mirror-0 ONLINE 0 0 0 > >> gpt/zpool-da2 ONLINE 0 0 0 > >> gpt/zpool-da3 ONLINE 0 0 0 > >> mirror-1 ONLINE 0 0 0 > >> gpt/zpool-da4 ONLINE 0 0 0 > >> gpt/zpool-da5 ONLINE 0 0 0 > > > > > > 1. zpool split zpool zpool.old > > this will leave your current zpool composed from slice of da2 and da4, > and > > create a new pool from da3 and da5. > > 2. zpool destroy zpool > > 3. truncate -s /tmp/dummy.1 && truncate -s > > /tmp/dummy.2 > > 4. zpool create zpool mirror da2 /tmp/dummy.1 mirror da4 > > /tmp/dummy.2 > > 5. zpool zpool offline /tmp/dummy.1 & zpool offline /tmp/dummy.2 > > 6. zpool import zpool.old > > 7. (zfs create ... on zpool as needed) copy your stuff from zpool.old t= o > > zpool > > 8. cross your fingers, *no* return from here !! > > 9. zpool destroy zpool.old > > 10. zpool labelclear da3 && zpool labelclear da5 # just to be on clear > side > > 11. zpool replace zpool /tmp/dummy.1 da3 && zpool replace zpool > > /tmp/dummy.2 da5 > > 12. wait for resilver ... > > > > If this is total sh*t please ignore, i tried it in VM seemed to work. > > > > Thanks. > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Mon May 18 13:39:44 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B05F22EB for ; Mon, 18 May 2015 13:39:44 +0000 (UTC) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [IPv6:2001:470:1f0f:3ad:223:7dff:fe9e:6e8a]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "thebighonker.lerctr.org", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 867611958 for ; Mon, 18 May 2015 13:39:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=qs84DiwQ+8ULBATYJ2ePBeBM0ZXlH85Tr0id5lL2870=; b=D/HZolNQSlaZiLrfT1noDGPSsP0VUfgFjFx1XV3+HmNqaJITltwdnLufc0xjHYu4xOQ2zhU0ZngSl6ttHUpwd3M3G0DFWbyGppJBZlhDsbFKg6A3g642FmGBzLUaXB1DW3R1zRJGVlkFdHylGzGf59tBkfyfOnxuKr8hbdPy/iY=; Received: from thebighonker.lerctr.org ([2001:470:1f0f:3ad:223:7dff:fe9e:6e8a]:16350 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLSv1:DHE-RSA-AES128-SHA:128) (Exim 4.85 (FreeBSD)) (envelope-from ) id 1YuLGh-000A7h-GV for freebsd-fs@freebsd.org; Mon, 18 May 2015 08:39:43 -0500 Received: from proxy.lucent.com ([135.245.49.13]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Mon, 18 May 2015 08:39:43 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Mon, 18 May 2015 08:39:43 -0500 From: Larry Rosenman To: Freebsd fs Subject: Fwd: Re: use after free panic ZFS In-Reply-To: <20150518125600.GA1274@borg.lerctr.org> References: <20150518124247.GA6220@borg.lerctr.org> <20150518125600.GA1274@borg.lerctr.org> Message-ID: X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/1.1.1 X-Spam-Score: -1.0 (-) X-LERCTR-Spam-Score: -1.0 (-) X-Spam-Report: SpamScore (-1.0/5.0) ALL_TRUSTED=-1,SHORTCIRCUIT=-0.0001 X-LERCTR-Spam-Report: SpamScore (-1.0/5.0) ALL_TRUSTED=-1, SHORTCIRCUIT=-0.0001 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 May 2015 13:39:44 -0000 Forwarding to -fs as well..... -------- Original Message -------- Subject: Re: use after free panic ZFS Date: 2015-05-18 07:56 From: Larry Rosenman To: freebsd-current@freebsd.org On Mon, May 18, 2015 at 07:42:47AM -0500, Larry Rosenman wrote: > found the following panic this am: > > borg.lerctr.org dumped core - see /var/crash/vmcore.5 > > Sun May 17 23:47:48 CDT 2015 > > FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #40 r283007: > Sat May 16 07:23:43 CDT 2015 > root@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64 > > panic: Most recently used by solaris > > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and > you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for > details. > This GDB was configured as "amd64-marcel-freebsd"... > > Unread portion of the kernel message buffer: > Memory modified after free 0xfffff808535ea000(120) val=deadc0dd @ > 0xfffff808535ea050 > panic: Most recently used by solaris > > cpuid = 5 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfffffe100bfb7660 > vpanic() at vpanic+0x189/frame 0xfffffe100bfb76e0 > panic() at panic+0x43/frame 0xfffffe100bfb7740 > mtrash_dtor() at mtrash_dtor/frame 0xfffffe100bfb7760 > uma_zalloc_arg() at uma_zalloc_arg+0x4c2/frame 0xfffffe100bfb77d0 > malloc() at malloc+0x198/frame 0xfffffe100bfb7820 > zfs_range_lock() at zfs_range_lock+0x4a/frame 0xfffffe100bfb7880 > zfs_get_data() at zfs_get_data+0x14c/frame 0xfffffe100bfb78f0 > zil_commit() at zil_commit+0x94c/frame 0xfffffe100bfb7a10 > zfs_freebsd_fsync() at zfs_freebsd_fsync+0xc8/frame 0xfffffe100bfb7a40 > VOP_FSYNC_APV() at VOP_FSYNC_APV+0xf7/frame 0xfffffe100bfb7a70 > sys_fsync() at sys_fsync+0x173/frame 0xfffffe100bfb7ae0 > amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe100bfb7bf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe100bfb7bf0 > --- syscall (95, FreeBSD ELF64, sys_fsync), rip = 0x801eb5daa, rsp = > 0x7fffffffd598, rbp = 0x7fffffffd5b0 --- > Uptime: 1d14h25m26s > Dumping 12469 out of 64457 > MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > > Reading symbols from /boot/kernel/linux.ko.symbols...done. > Loaded symbols for /boot/kernel/linux.ko.symbols > Reading symbols from /boot/kernel/if_lagg.ko.symbols...done. > Loaded symbols for /boot/kernel/if_lagg.ko.symbols > Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done. > Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols > Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done. > Loaded symbols for /boot/kernel/snd_spicds.ko.symbols > Reading symbols from /boot/kernel/coretemp.ko.symbols...done. > Loaded symbols for /boot/kernel/coretemp.ko.symbols > Reading symbols from /boot/kernel/ichsmb.ko.symbols...done. > Loaded symbols for /boot/kernel/ichsmb.ko.symbols > Reading symbols from /boot/kernel/smbus.ko.symbols...done. > Loaded symbols for /boot/kernel/smbus.ko.symbols > Reading symbols from /boot/kernel/ichwd.ko.symbols...done. > Loaded symbols for /boot/kernel/ichwd.ko.symbols > Reading symbols from /boot/kernel/cpuctl.ko.symbols...done. > Loaded symbols for /boot/kernel/cpuctl.ko.symbols > Reading symbols from /boot/kernel/crypto.ko.symbols...done. > Loaded symbols for /boot/kernel/crypto.ko.symbols > Reading symbols from /boot/kernel/cryptodev.ko.symbols...done. > Loaded symbols for /boot/kernel/cryptodev.ko.symbols > Reading symbols from /boot/kernel/dtraceall.ko.symbols...done. > Loaded symbols for /boot/kernel/dtraceall.ko.symbols > Reading symbols from /boot/kernel/profile.ko.symbols...done. > Loaded symbols for /boot/kernel/profile.ko.symbols > Reading symbols from /boot/kernel/dtrace.ko.symbols...done. > Loaded symbols for /boot/kernel/dtrace.ko.symbols > Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done. > Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols > Reading symbols from /boot/kernel/systrace.ko.symbols...done. > Loaded symbols for /boot/kernel/systrace.ko.symbols > Reading symbols from /boot/kernel/sdt.ko.symbols...done. > Loaded symbols for /boot/kernel/sdt.ko.symbols > Reading symbols from /boot/kernel/lockstat.ko.symbols...done. > Loaded symbols for /boot/kernel/lockstat.ko.symbols > Reading symbols from /boot/kernel/fasttrap.ko.symbols...done. > Loaded symbols for /boot/kernel/fasttrap.ko.symbols > Reading symbols from /boot/kernel/fbt.ko.symbols...done. > Loaded symbols for /boot/kernel/fbt.ko.symbols > Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done. > Loaded symbols for /boot/kernel/dtnfscl.ko.symbols > Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done. > Loaded symbols for /boot/kernel/dtmalloc.ko.symbols > Reading symbols from /boot/modules/vboxdrv.ko...done. > Loaded symbols for /boot/modules/vboxdrv.ko > Reading symbols from /boot/modules/nvidia.ko...done. > Loaded symbols for /boot/modules/nvidia.ko > Reading symbols from /boot/kernel/ipmi.ko.symbols...done. > Loaded symbols for /boot/kernel/ipmi.ko.symbols > Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done. > Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols > Reading symbols from /boot/kernel/radeonkms.ko.symbols...done. > Loaded symbols for /boot/kernel/radeonkms.ko.symbols > Reading symbols from /boot/kernel/iicbb.ko.symbols...done. > Loaded symbols for /boot/kernel/iicbb.ko.symbols > Reading symbols from /boot/kernel/iicbus.ko.symbols...done. > Loaded symbols for /boot/kernel/iicbus.ko.symbols > Reading symbols from /boot/kernel/iic.ko.symbols...done. > Loaded symbols for /boot/kernel/iic.ko.symbols > Reading symbols from /boot/kernel/drm2.ko.symbols...done. > Loaded symbols for /boot/kernel/drm2.ko.symbols > Reading symbols from > /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done. > Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols > Reading symbols from /boot/kernel/uhid.ko.symbols...done. > Loaded symbols for /boot/kernel/uhid.ko.symbols > Reading symbols from /boot/kernel/ums.ko.symbols...done. > Loaded symbols for /boot/kernel/ums.ko.symbols > Reading symbols from /boot/modules/vboxnetflt.ko...done. > Loaded symbols for /boot/modules/vboxnetflt.ko > Reading symbols from /boot/kernel/netgraph.ko.symbols...done. > Loaded symbols for /boot/kernel/netgraph.ko.symbols > Reading symbols from /boot/kernel/ng_ether.ko.symbols...done. > Loaded symbols for /boot/kernel/ng_ether.ko.symbols > Reading symbols from /boot/modules/vboxnetadp.ko...done. > Loaded symbols for /boot/modules/vboxnetadp.ko > #0 doadump (textdump=Unhandled dwarf expression opcode 0x93 > ) at pcpu.h:221 > 221 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) #0 doadump (textdump=Unhandled dwarf expression opcode 0x93 > ) at pcpu.h:221 > #1 0xffffffff80a839b5 in kern_reboot (howto=Unhandled dwarf expression > opcode 0x93 > ) > at /usr/src/sys/kern/kern_shutdown.c:447 > #2 0xffffffff80a83fa8 in vpanic (fmt=, > ap=) at /usr/src/sys/kern/kern_shutdown.c:744 > #3 0xffffffff80a83ff3 in panic (fmt=0x0) > at /usr/src/sys/kern/kern_shutdown.c:675 > #4 0xffffffff80d13750 in mtrash_ctor (mem=, > size=, arg=, > flags=) at /usr/src/sys/vm/uma_dbg.c:138 > #5 0xffffffff80d0f6d2 in uma_zalloc_arg (zone=0xfffff80ffffc9680, > udata=0x0, > flags=2) at /usr/src/sys/vm/uma_core.c:2197 > #6 0xffffffff80a64158 in malloc (size=, > mtp=0xffffffff815e16e0, flags=) at uma.h:336 > #7 0xffffffff80402b4a in zfs_range_lock (zp=0xfffff8075e835730, > off=9158656, > len=8192, type=Unhandled dwarf expression opcode 0x93 > ) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c:432 > #8 0xffffffff8040886c in zfs_get_data (arg=, > lr=, > buf=0xfffffe0662be8178
, > zio=0xfffff80d78b89ac8) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1250 > #9 0xffffffff8041c71c in zil_commit (zilog=0xfffff800185c1400, > foid=) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c:1108 > #10 0xffffffff80410168 in zfs_freebsd_fsync (ap=) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:2747 > #11 0xffffffff80fdfcd7 in VOP_FSYNC_APV (vop=, > a=) at vnode_if.c:1328 > #12 0xffffffff80b40883 in sys_fsync (td=0xfffff8011b253940, > uap=) at vnode_if.h:549 > #13 0xffffffff80e968da in amd64_syscall (td=0xfffff8011b253940, > traced=0) > at subr_syscall.c:133 > #14 0xffffffff80e767bb in Xfast_syscall () > at /usr/src/sys/amd64/amd64/exception.S:395 > #15 0x0000000801eb5daa in ?? () > Previous frame inner to this frame (corrupt stack?) > Current language: auto; currently minimal > (kgdb) > > I have the core. And, trying to re-compile to pick up the latest, got another one: borg.lerctr.org dumped core - see /var/crash/vmcore.6 Mon May 18 07:51:57 CDT 2015 FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #40 r283007: Sat May 16 07:23:43 CDT 2015 root@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64 panic: Most recently used by solaris GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Memory modified after free 0xfffff80c88f1f980(120) val=deadc0dd @ 0xfffff80c88f1f9c0 panic: Most recently used by solaris cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe100c459600 vpanic() at vpanic+0x189/frame 0xfffffe100c459680 panic() at panic+0x43/frame 0xfffffe100c4596e0 mtrash_dtor() at mtrash_dtor/frame 0xfffffe100c459700 uma_zalloc_arg() at uma_zalloc_arg+0x4c2/frame 0xfffffe100c459770 malloc() at malloc+0x198/frame 0xfffffe100c4597c0 zfs_range_lock() at zfs_range_lock+0x4a/frame 0xfffffe100c459820 zfs_freebsd_read() at zfs_freebsd_read+0x1c7/frame 0xfffffe100c4598c0 VOP_READ_APV() at VOP_READ_APV+0xf1/frame 0xfffffe100c4598f0 vn_read() at vn_read+0x237/frame 0xfffffe100c459970 vn_io_fault() at vn_io_fault+0x10a/frame 0xfffffe100c4599f0 dofileread() at dofileread+0x95/frame 0xfffffe100c459a40 kern_readv() at kern_readv+0x68/frame 0xfffffe100c459a90 sys_read() at sys_read+0x63/frame 0xfffffe100c459ae0 amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe100c459bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe100c459bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8009638fa, rsp = 0x7fffffffe968, rbp = 0x7fffffffe980 --- Uptime: 7h59m25s Dumping 14815 out of 64457 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/linux.ko.symbols...done. Loaded symbols for /boot/kernel/linux.ko.symbols Reading symbols from /boot/kernel/if_lagg.ko.symbols...done. Loaded symbols for /boot/kernel/if_lagg.ko.symbols Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done. Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done. Loaded symbols for /boot/kernel/snd_spicds.ko.symbols Reading symbols from /boot/kernel/coretemp.ko.symbols...done. Loaded symbols for /boot/kernel/coretemp.ko.symbols Reading symbols from /boot/kernel/ichsmb.ko.symbols...done. Loaded symbols for /boot/kernel/ichsmb.ko.symbols Reading symbols from /boot/kernel/smbus.ko.symbols...done. Loaded symbols for /boot/kernel/smbus.ko.symbols Reading symbols from /boot/kernel/ichwd.ko.symbols...done. Loaded symbols for /boot/kernel/ichwd.ko.symbols Reading symbols from /boot/kernel/cpuctl.ko.symbols...done. Loaded symbols for /boot/kernel/cpuctl.ko.symbols Reading symbols from /boot/kernel/crypto.ko.symbols...done. Loaded symbols for /boot/kernel/crypto.ko.symbols Reading symbols from /boot/kernel/cryptodev.ko.symbols...done. Loaded symbols for /boot/kernel/cryptodev.ko.symbols Reading symbols from /boot/kernel/dtraceall.ko.symbols...done. Loaded symbols for /boot/kernel/dtraceall.ko.symbols Reading symbols from /boot/kernel/profile.ko.symbols...done. Loaded symbols for /boot/kernel/profile.ko.symbols Reading symbols from /boot/kernel/dtrace.ko.symbols...done. Loaded symbols for /boot/kernel/dtrace.ko.symbols Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done. Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols Reading symbols from /boot/kernel/systrace.ko.symbols...done. Loaded symbols for /boot/kernel/systrace.ko.symbols Reading symbols from /boot/kernel/sdt.ko.symbols...done. Loaded symbols for /boot/kernel/sdt.ko.symbols Reading symbols from /boot/kernel/lockstat.ko.symbols...done. Loaded symbols for /boot/kernel/lockstat.ko.symbols Reading symbols from /boot/kernel/fasttrap.ko.symbols...done. Loaded symbols for /boot/kernel/fasttrap.ko.symbols Reading symbols from /boot/kernel/fbt.ko.symbols...done. Loaded symbols for /boot/kernel/fbt.ko.symbols Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done. Loaded symbols for /boot/kernel/dtnfscl.ko.symbols Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done. Loaded symbols for /boot/kernel/dtmalloc.ko.symbols Reading symbols from /boot/modules/vboxdrv.ko...done. Loaded symbols for /boot/modules/vboxdrv.ko Reading symbols from /boot/modules/nvidia.ko...done. Loaded symbols for /boot/modules/nvidia.ko Reading symbols from /boot/kernel/ipmi.ko.symbols...done. Loaded symbols for /boot/kernel/ipmi.ko.symbols Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done. Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols Reading symbols from /boot/kernel/radeonkms.ko.symbols...done. Loaded symbols for /boot/kernel/radeonkms.ko.symbols Reading symbols from /boot/kernel/iicbb.ko.symbols...done. Loaded symbols for /boot/kernel/iicbb.ko.symbols Reading symbols from /boot/kernel/iicbus.ko.symbols...done. Loaded symbols for /boot/kernel/iicbus.ko.symbols Reading symbols from /boot/kernel/iic.ko.symbols...done. Loaded symbols for /boot/kernel/iic.ko.symbols Reading symbols from /boot/kernel/drm2.ko.symbols...done. Loaded symbols for /boot/kernel/drm2.ko.symbols Reading symbols from /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done. Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols Reading symbols from /boot/kernel/uhid.ko.symbols...done. Loaded symbols for /boot/kernel/uhid.ko.symbols Reading symbols from /boot/kernel/ums.ko.symbols...done. Loaded symbols for /boot/kernel/ums.ko.symbols Reading symbols from /boot/modules/vboxnetflt.ko...done. Loaded symbols for /boot/modules/vboxnetflt.ko Reading symbols from /boot/kernel/netgraph.ko.symbols...done. Loaded symbols for /boot/kernel/netgraph.ko.symbols Reading symbols from /boot/kernel/ng_ether.ko.symbols...done. Loaded symbols for /boot/kernel/ng_ether.ko.symbols Reading symbols from /boot/modules/vboxnetadp.ko...done. Loaded symbols for /boot/modules/vboxnetadp.ko #0 doadump (textdump=Unhandled dwarf expression opcode 0x93 ) at pcpu.h:221 221 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=Unhandled dwarf expression opcode 0x93 ) at pcpu.h:221 #1 0xffffffff80a839b5 in kern_reboot (howto=Unhandled dwarf expression opcode 0x93 ) at /usr/src/sys/kern/kern_shutdown.c:447 #2 0xffffffff80a83fa8 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:744 #3 0xffffffff80a83ff3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:675 #4 0xffffffff80d13750 in mtrash_ctor (mem=, size=, arg=, flags=) at /usr/src/sys/vm/uma_dbg.c:138 #5 0xffffffff80d0f6d2 in uma_zalloc_arg (zone=0xfffff80ffffc9680, udata=0x0, flags=2) at /usr/src/sys/vm/uma_core.c:2197 #6 0xffffffff80a64158 in malloc (size=, mtp=0xffffffff815e16e0, flags=) at uma.h:336 #7 0xffffffff80402b4a in zfs_range_lock (zp=0xfffff806afc8d170, off=75316383, len=131072, type=Unhandled dwarf expression opcode 0x93 ) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c:432 #8 0xffffffff8040e517 in zfs_freebsd_read (ap=) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:703 #9 0xffffffff80fdf3b1 in VOP_READ_APV (vop=, a=) at vnode_if.c:930 #10 0xffffffff80b461b7 in vn_read (fp=0xfffff8002e1389b0, uio=0xfffffe100c459ab0, active_cred=, flags=, td=0x0) at vnode_if.h:384 #11 0xffffffff80b425ea in vn_io_fault (fp=0xfffff8002e1389b0, uio=0xfffffe100c459ab0, active_cred=0x0, flags=0, td=0x0) at /usr/src/sys/kern/vfs_vnops.c:1167 #12 0xffffffff80ae1525 in dofileread (td=0xfffff804bb5f8940, fd=3, fp=0xfffff8002e1389b0, auio=0xfffffe100c459ab0, offset=, flags=Unhandled dwarf expression opcode 0x93 ) at file.h:296 #13 0xffffffff80ae1228 in kern_readv (td=0xfffff804bb5f8940, fd=Unhandled dwarf expression opcode 0x93 ) at /usr/src/sys/kern/sys_generic.c:272 #14 0xffffffff80ae11b3 in sys_read (td=0x0, uap=) at /usr/src/sys/kern/sys_generic.c:185 #15 0xffffffff80e968da in amd64_syscall (td=0xfffff804bb5f8940, traced=0) at subr_syscall.c:133 #16 0xffffffff80e767bb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:395 #17 0x00000008009638fa in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) I have BOTH cores. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 From owner-freebsd-fs@FreeBSD.ORG Mon May 18 18:07:05 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 19B98AA7; Mon, 18 May 2015 18:07:05 +0000 (UTC) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [IPv6:2001:470:1f0f:3ad:223:7dff:fe9e:6e8a]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "thebighonker.lerctr.org", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CC96B1D01; Mon, 18 May 2015 18:07:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=bsBhB4/x+C8WQMlqM5/AdzUTtf2ue1RY2Ib/RR4l2P8=; b=m26UL7DaKCpiu1JZuddX/ULdifxI4tiW5WN2mOIM2IsT1hRDSHZY0e4p7GfS8iisNLNfElqhwHhTWhD3UZAtnbsLTh1QNAuFoNqRtdQApJQhbnAhubQht5784IF3/PnCHYwH8nBPtMZPKPNuKxTIa2ldwbWPPTsc0F0hUJoxO38=; Received: from thebighonker.lerctr.org ([2001:470:1f0f:3ad:223:7dff:fe9e:6e8a]:62298 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLSv1:DHE-RSA-AES128-SHA:128) (Exim 4.85 (FreeBSD)) (envelope-from ) id 1YuPRP-000D2D-A3; Mon, 18 May 2015 13:07:03 -0500 Received: from proxy.lucent.com ([135.245.49.13]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Mon, 18 May 2015 13:07:03 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Mon, 18 May 2015 13:07:03 -0500 From: Larry Rosenman To: freebsd-current@freebsd.org Cc: Freebsd fs Subject: Re: use after free panic ZFS In-Reply-To: <20150518125600.GA1274@borg.lerctr.org> References: <20150518124247.GA6220@borg.lerctr.org> <20150518125600.GA1274@borg.lerctr.org> Message-ID: <2b200914e9c3473c30be97590f02ef49@thebighonker.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/1.1.1 X-Spam-Score: -1.0 (-) X-LERCTR-Spam-Score: -1.0 (-) X-Spam-Report: SpamScore (-1.0/5.0) ALL_TRUSTED=-1,SHORTCIRCUIT=-0.0001 X-LERCTR-Spam-Report: SpamScore (-1.0/5.0) ALL_TRUSTED=-1, SHORTCIRCUIT=-0.0001 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 May 2015 18:07:05 -0000 I've filed: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 for this. I'd appreciate someone looking at it. I'm STUCK here. On 2015-05-18 07:56, Larry Rosenman wrote: > On Mon, May 18, 2015 at 07:42:47AM -0500, Larry Rosenman wrote: >> found the following panic this am: >> >> borg.lerctr.org dumped core - see /var/crash/vmcore.5 >> >> Sun May 17 23:47:48 CDT 2015 >> >> FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #40 r283007: >> Sat May 16 07:23:43 CDT 2015 >> root@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64 >> >> panic: Most recently used by solaris >> >> GNU gdb 6.1.1 [FreeBSD] >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and >> you are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for >> details. >> This GDB was configured as "amd64-marcel-freebsd"... >> >> Unread portion of the kernel message buffer: >> Memory modified after free 0xfffff808535ea000(120) val=deadc0dd @ >> 0xfffff808535ea050 >> panic: Most recently used by solaris >> >> cpuid = 5 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe100bfb7660 >> vpanic() at vpanic+0x189/frame 0xfffffe100bfb76e0 >> panic() at panic+0x43/frame 0xfffffe100bfb7740 >> mtrash_dtor() at mtrash_dtor/frame 0xfffffe100bfb7760 >> uma_zalloc_arg() at uma_zalloc_arg+0x4c2/frame 0xfffffe100bfb77d0 >> malloc() at malloc+0x198/frame 0xfffffe100bfb7820 >> zfs_range_lock() at zfs_range_lock+0x4a/frame 0xfffffe100bfb7880 >> zfs_get_data() at zfs_get_data+0x14c/frame 0xfffffe100bfb78f0 >> zil_commit() at zil_commit+0x94c/frame 0xfffffe100bfb7a10 >> zfs_freebsd_fsync() at zfs_freebsd_fsync+0xc8/frame 0xfffffe100bfb7a40 >> VOP_FSYNC_APV() at VOP_FSYNC_APV+0xf7/frame 0xfffffe100bfb7a70 >> sys_fsync() at sys_fsync+0x173/frame 0xfffffe100bfb7ae0 >> amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe100bfb7bf0 >> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe100bfb7bf0 >> --- syscall (95, FreeBSD ELF64, sys_fsync), rip = 0x801eb5daa, rsp = >> 0x7fffffffd598, rbp = 0x7fffffffd5b0 --- >> Uptime: 1d14h25m26s >> Dumping 12469 out of 64457 >> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% >> >> Reading symbols from /boot/kernel/linux.ko.symbols...done. >> Loaded symbols for /boot/kernel/linux.ko.symbols >> Reading symbols from /boot/kernel/if_lagg.ko.symbols...done. >> Loaded symbols for /boot/kernel/if_lagg.ko.symbols >> Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done. >> Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols >> Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done. >> Loaded symbols for /boot/kernel/snd_spicds.ko.symbols >> Reading symbols from /boot/kernel/coretemp.ko.symbols...done. >> Loaded symbols for /boot/kernel/coretemp.ko.symbols >> Reading symbols from /boot/kernel/ichsmb.ko.symbols...done. >> Loaded symbols for /boot/kernel/ichsmb.ko.symbols >> Reading symbols from /boot/kernel/smbus.ko.symbols...done. >> Loaded symbols for /boot/kernel/smbus.ko.symbols >> Reading symbols from /boot/kernel/ichwd.ko.symbols...done. >> Loaded symbols for /boot/kernel/ichwd.ko.symbols >> Reading symbols from /boot/kernel/cpuctl.ko.symbols...done. >> Loaded symbols for /boot/kernel/cpuctl.ko.symbols >> Reading symbols from /boot/kernel/crypto.ko.symbols...done. >> Loaded symbols for /boot/kernel/crypto.ko.symbols >> Reading symbols from /boot/kernel/cryptodev.ko.symbols...done. >> Loaded symbols for /boot/kernel/cryptodev.ko.symbols >> Reading symbols from /boot/kernel/dtraceall.ko.symbols...done. >> Loaded symbols for /boot/kernel/dtraceall.ko.symbols >> Reading symbols from /boot/kernel/profile.ko.symbols...done. >> Loaded symbols for /boot/kernel/profile.ko.symbols >> Reading symbols from /boot/kernel/dtrace.ko.symbols...done. >> Loaded symbols for /boot/kernel/dtrace.ko.symbols >> Reading symbols from >> /boot/kernel/systrace_freebsd32.ko.symbols...done. >> Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols >> Reading symbols from /boot/kernel/systrace.ko.symbols...done. >> Loaded symbols for /boot/kernel/systrace.ko.symbols >> Reading symbols from /boot/kernel/sdt.ko.symbols...done. >> Loaded symbols for /boot/kernel/sdt.ko.symbols >> Reading symbols from /boot/kernel/lockstat.ko.symbols...done. >> Loaded symbols for /boot/kernel/lockstat.ko.symbols >> Reading symbols from /boot/kernel/fasttrap.ko.symbols...done. >> Loaded symbols for /boot/kernel/fasttrap.ko.symbols >> Reading symbols from /boot/kernel/fbt.ko.symbols...done. >> Loaded symbols for /boot/kernel/fbt.ko.symbols >> Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done. >> Loaded symbols for /boot/kernel/dtnfscl.ko.symbols >> Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done. >> Loaded symbols for /boot/kernel/dtmalloc.ko.symbols >> Reading symbols from /boot/modules/vboxdrv.ko...done. >> Loaded symbols for /boot/modules/vboxdrv.ko >> Reading symbols from /boot/modules/nvidia.ko...done. >> Loaded symbols for /boot/modules/nvidia.ko >> Reading symbols from /boot/kernel/ipmi.ko.symbols...done. >> Loaded symbols for /boot/kernel/ipmi.ko.symbols >> Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done. >> Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols >> Reading symbols from /boot/kernel/radeonkms.ko.symbols...done. >> Loaded symbols for /boot/kernel/radeonkms.ko.symbols >> Reading symbols from /boot/kernel/iicbb.ko.symbols...done. >> Loaded symbols for /boot/kernel/iicbb.ko.symbols >> Reading symbols from /boot/kernel/iicbus.ko.symbols...done. >> Loaded symbols for /boot/kernel/iicbus.ko.symbols >> Reading symbols from /boot/kernel/iic.ko.symbols...done. >> Loaded symbols for /boot/kernel/iic.ko.symbols >> Reading symbols from /boot/kernel/drm2.ko.symbols...done. >> Loaded symbols for /boot/kernel/drm2.ko.symbols >> Reading symbols from >> /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done. >> Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols >> Reading symbols from /boot/kernel/uhid.ko.symbols...done. >> Loaded symbols for /boot/kernel/uhid.ko.symbols >> Reading symbols from /boot/kernel/ums.ko.symbols...done. >> Loaded symbols for /boot/kernel/ums.ko.symbols >> Reading symbols from /boot/modules/vboxnetflt.ko...done. >> Loaded symbols for /boot/modules/vboxnetflt.ko >> Reading symbols from /boot/kernel/netgraph.ko.symbols...done. >> Loaded symbols for /boot/kernel/netgraph.ko.symbols >> Reading symbols from /boot/kernel/ng_ether.ko.symbols...done. >> Loaded symbols for /boot/kernel/ng_ether.ko.symbols >> Reading symbols from /boot/modules/vboxnetadp.ko...done. >> Loaded symbols for /boot/modules/vboxnetadp.ko >> #0 doadump (textdump=Unhandled dwarf expression opcode 0x93 >> ) at pcpu.h:221 >> 221 pcpu.h: No such file or directory. >> in pcpu.h >> (kgdb) #0 doadump (textdump=Unhandled dwarf expression opcode 0x93 >> ) at pcpu.h:221 >> #1 0xffffffff80a839b5 in kern_reboot (howto=Unhandled dwarf >> expression opcode 0x93 >> ) >> at /usr/src/sys/kern/kern_shutdown.c:447 >> #2 0xffffffff80a83fa8 in vpanic (fmt=, >> ap=) at /usr/src/sys/kern/kern_shutdown.c:744 >> #3 0xffffffff80a83ff3 in panic (fmt=0x0) >> at /usr/src/sys/kern/kern_shutdown.c:675 >> #4 0xffffffff80d13750 in mtrash_ctor (mem=, >> size=, arg=, >> flags=) at /usr/src/sys/vm/uma_dbg.c:138 >> #5 0xffffffff80d0f6d2 in uma_zalloc_arg (zone=0xfffff80ffffc9680, >> udata=0x0, >> flags=2) at /usr/src/sys/vm/uma_core.c:2197 >> #6 0xffffffff80a64158 in malloc (size=, >> mtp=0xffffffff815e16e0, flags=) at uma.h:336 >> #7 0xffffffff80402b4a in zfs_range_lock (zp=0xfffff8075e835730, >> off=9158656, >> len=8192, type=Unhandled dwarf expression opcode 0x93 >> ) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c:432 >> #8 0xffffffff8040886c in zfs_get_data (arg=, >> lr=, >> buf=0xfffffe0662be8178
, >> zio=0xfffff80d78b89ac8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1250 >> #9 0xffffffff8041c71c in zil_commit (zilog=0xfffff800185c1400, >> foid=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c:1108 >> #10 0xffffffff80410168 in zfs_freebsd_fsync (ap=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:2747 >> #11 0xffffffff80fdfcd7 in VOP_FSYNC_APV (vop=, >> a=) at vnode_if.c:1328 >> #12 0xffffffff80b40883 in sys_fsync (td=0xfffff8011b253940, >> uap=) at vnode_if.h:549 >> #13 0xffffffff80e968da in amd64_syscall (td=0xfffff8011b253940, >> traced=0) >> at subr_syscall.c:133 >> #14 0xffffffff80e767bb in Xfast_syscall () >> at /usr/src/sys/amd64/amd64/exception.S:395 >> #15 0x0000000801eb5daa in ?? () >> Previous frame inner to this frame (corrupt stack?) >> Current language: auto; currently minimal >> (kgdb) >> >> I have the core. > And, trying to re-compile to pick up the latest, got another one: > > borg.lerctr.org dumped core - see /var/crash/vmcore.6 > > Mon May 18 07:51:57 CDT 2015 > > FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #40 r283007: > Sat May 16 07:23:43 CDT 2015 > root@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64 > > panic: Most recently used by solaris > > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and > you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for > details. > This GDB was configured as "amd64-marcel-freebsd"... > > Unread portion of the kernel message buffer: > Memory modified after free 0xfffff80c88f1f980(120) val=deadc0dd @ > 0xfffff80c88f1f9c0 > panic: Most recently used by solaris > > cpuid = 0 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfffffe100c459600 > vpanic() at vpanic+0x189/frame 0xfffffe100c459680 > panic() at panic+0x43/frame 0xfffffe100c4596e0 > mtrash_dtor() at mtrash_dtor/frame 0xfffffe100c459700 > uma_zalloc_arg() at uma_zalloc_arg+0x4c2/frame 0xfffffe100c459770 > malloc() at malloc+0x198/frame 0xfffffe100c4597c0 > zfs_range_lock() at zfs_range_lock+0x4a/frame 0xfffffe100c459820 > zfs_freebsd_read() at zfs_freebsd_read+0x1c7/frame 0xfffffe100c4598c0 > VOP_READ_APV() at VOP_READ_APV+0xf1/frame 0xfffffe100c4598f0 > vn_read() at vn_read+0x237/frame 0xfffffe100c459970 > vn_io_fault() at vn_io_fault+0x10a/frame 0xfffffe100c4599f0 > dofileread() at dofileread+0x95/frame 0xfffffe100c459a40 > kern_readv() at kern_readv+0x68/frame 0xfffffe100c459a90 > sys_read() at sys_read+0x63/frame 0xfffffe100c459ae0 > amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe100c459bf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe100c459bf0 > --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8009638fa, rsp = > 0x7fffffffe968, rbp = 0x7fffffffe980 --- > Uptime: 7h59m25s > Dumping 14815 out of 64457 > MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > > Reading symbols from /boot/kernel/linux.ko.symbols...done. > Loaded symbols for /boot/kernel/linux.ko.symbols > Reading symbols from /boot/kernel/if_lagg.ko.symbols...done. > Loaded symbols for /boot/kernel/if_lagg.ko.symbols > Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done. > Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols > Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done. > Loaded symbols for /boot/kernel/snd_spicds.ko.symbols > Reading symbols from /boot/kernel/coretemp.ko.symbols...done. > Loaded symbols for /boot/kernel/coretemp.ko.symbols > Reading symbols from /boot/kernel/ichsmb.ko.symbols...done. > Loaded symbols for /boot/kernel/ichsmb.ko.symbols > Reading symbols from /boot/kernel/smbus.ko.symbols...done. > Loaded symbols for /boot/kernel/smbus.ko.symbols > Reading symbols from /boot/kernel/ichwd.ko.symbols...done. > Loaded symbols for /boot/kernel/ichwd.ko.symbols > Reading symbols from /boot/kernel/cpuctl.ko.symbols...done. > Loaded symbols for /boot/kernel/cpuctl.ko.symbols > Reading symbols from /boot/kernel/crypto.ko.symbols...done. > Loaded symbols for /boot/kernel/crypto.ko.symbols > Reading symbols from /boot/kernel/cryptodev.ko.symbols...done. > Loaded symbols for /boot/kernel/cryptodev.ko.symbols > Reading symbols from /boot/kernel/dtraceall.ko.symbols...done. > Loaded symbols for /boot/kernel/dtraceall.ko.symbols > Reading symbols from /boot/kernel/profile.ko.symbols...done. > Loaded symbols for /boot/kernel/profile.ko.symbols > Reading symbols from /boot/kernel/dtrace.ko.symbols...done. > Loaded symbols for /boot/kernel/dtrace.ko.symbols > Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done. > Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols > Reading symbols from /boot/kernel/systrace.ko.symbols...done. > Loaded symbols for /boot/kernel/systrace.ko.symbols > Reading symbols from /boot/kernel/sdt.ko.symbols...done. > Loaded symbols for /boot/kernel/sdt.ko.symbols > Reading symbols from /boot/kernel/lockstat.ko.symbols...done. > Loaded symbols for /boot/kernel/lockstat.ko.symbols > Reading symbols from /boot/kernel/fasttrap.ko.symbols...done. > Loaded symbols for /boot/kernel/fasttrap.ko.symbols > Reading symbols from /boot/kernel/fbt.ko.symbols...done. > Loaded symbols for /boot/kernel/fbt.ko.symbols > Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done. > Loaded symbols for /boot/kernel/dtnfscl.ko.symbols > Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done. > Loaded symbols for /boot/kernel/dtmalloc.ko.symbols > Reading symbols from /boot/modules/vboxdrv.ko...done. > Loaded symbols for /boot/modules/vboxdrv.ko > Reading symbols from /boot/modules/nvidia.ko...done. > Loaded symbols for /boot/modules/nvidia.ko > Reading symbols from /boot/kernel/ipmi.ko.symbols...done. > Loaded symbols for /boot/kernel/ipmi.ko.symbols > Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done. > Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols > Reading symbols from /boot/kernel/radeonkms.ko.symbols...done. > Loaded symbols for /boot/kernel/radeonkms.ko.symbols > Reading symbols from /boot/kernel/iicbb.ko.symbols...done. > Loaded symbols for /boot/kernel/iicbb.ko.symbols > Reading symbols from /boot/kernel/iicbus.ko.symbols...done. > Loaded symbols for /boot/kernel/iicbus.ko.symbols > Reading symbols from /boot/kernel/iic.ko.symbols...done. > Loaded symbols for /boot/kernel/iic.ko.symbols > Reading symbols from /boot/kernel/drm2.ko.symbols...done. > Loaded symbols for /boot/kernel/drm2.ko.symbols > Reading symbols from > /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done. > Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols > Reading symbols from /boot/kernel/uhid.ko.symbols...done. > Loaded symbols for /boot/kernel/uhid.ko.symbols > Reading symbols from /boot/kernel/ums.ko.symbols...done. > Loaded symbols for /boot/kernel/ums.ko.symbols > Reading symbols from /boot/modules/vboxnetflt.ko...done. > Loaded symbols for /boot/modules/vboxnetflt.ko > Reading symbols from /boot/kernel/netgraph.ko.symbols...done. > Loaded symbols for /boot/kernel/netgraph.ko.symbols > Reading symbols from /boot/kernel/ng_ether.ko.symbols...done. > Loaded symbols for /boot/kernel/ng_ether.ko.symbols > Reading symbols from /boot/modules/vboxnetadp.ko...done. > Loaded symbols for /boot/modules/vboxnetadp.ko > #0 doadump (textdump=Unhandled dwarf expression opcode 0x93 > ) at pcpu.h:221 > 221 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) #0 doadump (textdump=Unhandled dwarf expression opcode 0x93 > ) at pcpu.h:221 > #1 0xffffffff80a839b5 in kern_reboot (howto=Unhandled dwarf > expression opcode 0x93 > ) > at /usr/src/sys/kern/kern_shutdown.c:447 > #2 0xffffffff80a83fa8 in vpanic (fmt=, > ap=) at /usr/src/sys/kern/kern_shutdown.c:744 > #3 0xffffffff80a83ff3 in panic (fmt=0x0) > at /usr/src/sys/kern/kern_shutdown.c:675 > #4 0xffffffff80d13750 in mtrash_ctor (mem=, > size=, arg=, > flags=) at /usr/src/sys/vm/uma_dbg.c:138 > #5 0xffffffff80d0f6d2 in uma_zalloc_arg (zone=0xfffff80ffffc9680, > udata=0x0, > flags=2) at /usr/src/sys/vm/uma_core.c:2197 > #6 0xffffffff80a64158 in malloc (size=, > mtp=0xffffffff815e16e0, flags=) at uma.h:336 > #7 0xffffffff80402b4a in zfs_range_lock (zp=0xfffff806afc8d170, > off=75316383, len=131072, type=Unhandled dwarf expression opcode > 0x93 > ) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c:432 > #8 0xffffffff8040e517 in zfs_freebsd_read (ap=) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:703 > #9 0xffffffff80fdf3b1 in VOP_READ_APV (vop=, > a=) at vnode_if.c:930 > #10 0xffffffff80b461b7 in vn_read (fp=0xfffff8002e1389b0, > uio=0xfffffe100c459ab0, active_cred=, > flags=, td=0x0) at vnode_if.h:384 > #11 0xffffffff80b425ea in vn_io_fault (fp=0xfffff8002e1389b0, > uio=0xfffffe100c459ab0, active_cred=0x0, flags=0, td=0x0) > at /usr/src/sys/kern/vfs_vnops.c:1167 > #12 0xffffffff80ae1525 in dofileread (td=0xfffff804bb5f8940, fd=3, > fp=0xfffff8002e1389b0, auio=0xfffffe100c459ab0, > offset=, flags=Unhandled dwarf expression > opcode 0x93 > ) at file.h:296 > #13 0xffffffff80ae1228 in kern_readv (td=0xfffff804bb5f8940, > fd=Unhandled dwarf expression opcode 0x93 > ) > at /usr/src/sys/kern/sys_generic.c:272 > #14 0xffffffff80ae11b3 in sys_read (td=0x0, uap=) > at /usr/src/sys/kern/sys_generic.c:185 > #15 0xffffffff80e968da in amd64_syscall (td=0xfffff804bb5f8940, > traced=0) > at subr_syscall.c:133 > #16 0xffffffff80e767bb in Xfast_syscall () > at /usr/src/sys/amd64/amd64/exception.S:395 > #17 0x00000008009638fa in ?? () > Previous frame inner to this frame (corrupt stack?) > Current language: auto; currently minimal > (kgdb) > I have BOTH cores. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 From owner-freebsd-fs@FreeBSD.ORG Mon May 18 19:06:12 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 793DC5D2 for ; Mon, 18 May 2015 19:06:12 +0000 (UTC) Received: from ns512915.ip-198-27-66.net (unknown [IPv6:2607:5300:60:1b41::1]) by mx1.freebsd.org (Postfix) with ESMTP id 5897915B8 for ; Mon, 18 May 2015 19:06:12 +0000 (UTC) Received: from support-team (localhost.localdomain [127.0.0.1]) by ns512915.ip-198-27-66.net (Postfix) with SMTP id 416B4942B0 for ; Mon, 18 May 2015 14:58:53 -0400 (EDT) X-GEPFBL: ZnJlZWJzZC1mc0BmcmVlYnNkLm9yZw Message-ID: <4fbf44a228e003025203977f13897941@apple.com> From: "Apple" To: Subject: =?WINDOWS-1252?Q?Your_Apple_ID_(freebsd-fs@freebsd.org)_has_been_disabled.?= Date: Mon, 18 May 2015 20:58:58 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="WINDOWS-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 May 2015 19:06:12 -0000 Hello freebsd-fs@freebsd.org=2E For your protection, your Apple ID is automatically disabled, it happens = when your account password is incorrectly entered numerous times. This affects = some services you may access with your Apple ID, such as your accounts with: = Apple Print Products, iCloud, Apple Store and iTunes Store=2E ---------------------------------------------------------------------------= ----- Once we've confirmed your personal details, we'll check your records by = looking at: security questions, telephone numbers, billing profile and other = account data=2E We may ask you to confirm any of this information so we can clearly = determine which activity is yours, and which are those from another person using your account=2E Use the link below to reset your account. This will restore your access to = the disabled services=2E To reset your Apple ID, visit the Apple ID Support site =2E Failure to fix this process or submission of incorrect data will lead to = account termination=2E Thanks, Apple Support My Apple ID | Support | Privacy Policy Copyright =A9 2015 Apple Inc. 1 Infinite Loop, Cupertino, CA 95014, United States?All Rights Reserved. From owner-freebsd-fs@FreeBSD.ORG Mon May 18 23:36:26 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9F997900 for ; Mon, 18 May 2015 23:36:26 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8A6EC15D7 for ; Mon, 18 May 2015 23:36:26 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4INaQFv003477 for ; Mon, 18 May 2015 23:36:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Mon, 18 May 2015 23:36:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc bug_severity assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 May 2015 23:36:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 Garrett Cooper,425-314-3911 changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ngie@FreeBSD.org Severity|Affects Only Me |Affects Some People Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --- Comment #6 from Garrett Cooper,425-314-3911 --- Please recompile the kernel with "option DEBUG_MEMGUARD" if possible so this can be tracked down a bit further. When was the last time your system functioned for you without this panic? -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Mon May 18 23:59:38 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 21DA4E51 for ; Mon, 18 May 2015 23:59:38 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0CCE917EE for ; Mon, 18 May 2015 23:59:38 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4INxb6j022165 for ; Mon, 18 May 2015 23:59:37 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Mon, 18 May 2015 23:59:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ler@lerctr.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 May 2015 23:59:38 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #7 from Larry Rosenman --- It had been running fine for a few months. I did replace a disk in the array last week but that was uneventful. I'll TRY and compile a kernel with DEBUG_MEMGUARD but so far building even WORLD has provoked the panic. I **DO** have all 3 vmcore's available if someone wants to traipse through them. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Tue May 19 00:23:40 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 94DA21EF for ; Tue, 19 May 2015 00:23:40 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7FBF41AE9 for ; Tue, 19 May 2015 00:23:40 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4J0New8023374 for ; Tue, 19 May 2015 00:23:40 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Tue, 19 May 2015 00:23:40 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ler@lerctr.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 May 2015 00:23:40 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #8 from Larry Rosenman --- was able to get a kernel compiled/booted with DEBUG_MEMGUARD. I'll attach that core.txt. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Tue May 19 00:24:14 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 76E6A25F for ; Tue, 19 May 2015 00:24:14 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 614B81AF0 for ; Tue, 19 May 2015 00:24:14 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4J0OEwk023594 for ; Tue, 19 May 2015 00:24:14 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Tue, 19 May 2015 00:24:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ler@lerctr.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 May 2015 00:24:14 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #9 from Larry Rosenman --- Created attachment 156917 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=156917&action=edit PANIC with DEBUG_MEMGUARD -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Tue May 19 10:18:26 2015 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4BDC61D5 for ; Tue, 19 May 2015 10:18:26 +0000 (UTC) Received: from mail-wi0-x22c.google.com (mail-wi0-x22c.google.com [IPv6:2a00:1450:400c:c05::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D4A241310 for ; Tue, 19 May 2015 10:18:25 +0000 (UTC) Received: by wicmx19 with SMTP id mx19so111088675wic.0 for ; Tue, 19 May 2015 03:18:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=nbLCOn2lsfNTQI2kkjW43J7zUcSQmdFa9U+Q0AQ6EQc=; b=SflQwhhzDqINqs+guEWiPijRPDwK27nFw/fwUZpgAXsyagLgkTc5fjAgh9cPYDHfwy z+F3KaTH083vwJjo7VXzBqU4FobVM8TCqUjRlfvcoYVDoFIV4x7FrF5FQlP0ZMPaqd3i TxxM9cb+c5C437egC33YnWT5qOGYoEAG6BnvVtEQOmhoByyLLGt0Sw5Pxa+RKvaIxTt3 JLZ1syK49u1cYbFpDkw2QfVMDOE3gKSAiKBfBsOkkuok5mAWQln4PASeBWhVrvEUb+Li p1xWPzEmGkyNy49GslMid+fXh57B++dcU2fP0XnK5XVbOFQ5FPx7iSIs691vw/Mtv6js Zfyg== X-Received: by 10.180.99.166 with SMTP id er6mr30428113wib.58.1432030704331; Tue, 19 May 2015 03:18:24 -0700 (PDT) Received: from brick.home (acyt45.neoplus.adsl.tpnet.pl. [83.11.203.45]) by mx.google.com with ESMTPSA id d14sm21069030wjn.30.2015.05.19.03.18.23 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 May 2015 03:18:23 -0700 (PDT) Sender: =?UTF-8?Q?Edward_Tomasz_Napiera=C5=82a?= Date: Tue, 19 May 2015 12:18:21 +0200 From: Edward Tomasz Napierala To: Willem Jan Withagen Cc: fs@freebsd.org Subject: Re: Unexpected reboot after ctld run into trouble. Message-ID: <20150519101821.GC3292@brick.home> References: <55573756.9070503@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <55573756.9070503@digiware.nl> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 May 2015 10:18:26 -0000 On 0516T1425, Willem Jan Withagen wrote: > Hi, > > Found the following in my logs: > Losts of > ---- > (0:4:0/0): Task Action: LUN Reset > (0:4:0/0): CTL Status: Command Completed Successfully > sonewconn: pcb 0xfffff8004e69e930: Listen queue overflow: 8 already in > queue awaiting acceptance (740688 occurrences) > (0:4:0/0): Task Action: LUN Reset > (0:4:0/0): CTL Status: Command Completed Successfully > sonewconn: pcb 0xfffff8004e69e930: Listen queue overflow: 8 already in > queue awaiting acceptance (713721 occurrences) > (0:4:0/0): Task Action: LUN Reset > (0:4:0/0): CTL Status: Command Completed Successfully > sonewconn: pcb 0xfffff8004e69e930: Listen queue overflow: 8 already in > queue awaiting acceptance (691776 occurrences) > ---- > > Which then ends in: > ---- > panic: deadlkres: possible deadlock detected for 0xfffff8001ee94920, > blocked for 1801009 ticks > > > cpuid = 1 > Uptime: 14d13h13m47s > Dumping 7557 out of 8175 > MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%Table 'FACP' at > 0xcfedbcf8 > ---- > > The system is running ZFS with ZFS-on-root: > FreeBSD zfs.digiware.nl 10.1-STABLE FreeBSD 10.1-STABLE #221 r282282: > Fri May 1 06:51:41 CEST 2015 > > This could stem from the fact that I woke up my Win8 PC which has a > iscsi volume mounted. It is used to store security cam captures on and > does have somewhat bigger traffic on it. > > Suggestions or question to look at are welcom. > > --WjW > > > I do have a core in /var/crash, but will need some guidance to retreive > stuff from it. Could you send me the /var/crash/core.txt.N for that crashdump? From owner-freebsd-fs@FreeBSD.ORG Tue May 19 14:28:49 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D597E230 for ; Tue, 19 May 2015 14:28:49 +0000 (UTC) Received: from mail.worldserver.net (mail.worldserver.net [217.13.200.36]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.worldserver.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 76E64130B for ; Tue, 19 May 2015 14:28:49 +0000 (UTC) Received: from mailer.rman (unknown [2.239.24.102]) (Authenticated sender: mailer@campese.de) by mail.worldserver.net (Postfix) with ESMTPSA id CD8C63003DB; Tue, 19 May 2015 16:20:47 +0200 (CEST) From: Simon Campese To: freebsd-fs@freebsd.org Cc: Subject: hardware fault during ZFS send/receive blocks /dev/zfs indefinitely Date: Tue, 19 May 2015 16:20:46 +0200 Message-ID: <86wq048x8h.fsf@emacs.campese.org> MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 May 2015 14:28:49 -0000 Hello, I tried to send/receive a ZFS filesystem from a raidz2-pool to another pool with just a single disk, when this disk failed. As a result, now both, the zfs send and zfs receive processes are in uninterruptible sleep state and all new zpool and zfs commands which I issue immediately enter uninterruptible sleep. Is this just bad luck (i.e. my disk failed in the wrong moment) or might this be a bug? Anyway, my only solution is to schedule a reboot soon as the machine is a file server and the operational status of zfs is critical. I'm not very experienced with zfs or the FreeBSD kernel, so I just try to supply as much relevant information as possible. Please tell me if there is more I can do. The system I run is FreeBSD 10.1-RELEASE-p6, the machine is a small intel file server (eight core Atom, 64G Ram, Supermicro board, two raidz2 pools connected via reflashed IBM M1015 controllers). Here are the relevant lines from "ps ax" (with anonymized pool/filesystem names): 79776 1- DE 0:05.27 zfs send -R zpool0/fs0 79777 1- D+ 0:09.80 zfs receive -e zpool1/fs1 a "lsof -p" for the send process yields no output, for the receive process I get: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME zfs 79777 root cwd VDIR 37,1520762947 29 4 / zfs 79777 root rtd VDIR 37,1520762947 29 4 / zfs 79777 root txt VREG 37,1520762947 92952 608 /sbin/zfs zfs 79777 root txt VREG 37,1520762947 118520 137 /libexec/ld-elf.so.1 zfs 79777 root txt VREG 37,1520762947 20768 41 /lib/libgeom.so.5 zfs 79777 root txt VREG 37,1520762947 20272 75 /lib/libjail.so.1 zfs 79777 root txt VREG 37,1520762947 82888 51 /lib/libnvpair.so.2 zfs 79777 root txt VREG 37,1520762947 5880 57 /lib/libumem.so.2 zfs 79777 root txt VREG 37,1520762947 70816 69 /lib/libutil.so.9 zfs 79777 root txt VREG 37,1520762947 37984 52 /lib/libuutil.so.2 zfs 79777 root txt VREG 37,1520762947 15840 49 /lib/libzfs_core.so.2 zfs 79777 root txt VREG 37,1520762947 279968 76 /lib/libzfs.so.2 zfs 79777 root txt VREG 37,1520762947 1631216 556 /lib/libc.so.7 zfs 79777 root txt VREG 37,1520762947 157664 77 /lib/libbsdxml.so.4 zfs 79777 root txt VREG 37,1520762947 10976 34 /lib/libsbuf.so.6 zfs 79777 root txt VREG 37,1520762947 66200 56 /lib/libmd.so.6 zfs 79777 root txt VREG 37,1520762947 105248 71 /lib/libthr.so.3 zfs 79777 root txt VREG 37,1520762947 181840 64 /lib/libm.so.5 zfs 79777 root txt VREG 37,1520762947 8784 45 /lib/libavl.so.2 zfs 79777 root 0u PIPE 0xfffff800462db000 65536 ->0xfffff800462db160, cnt=65224, out=312 zfs 79777 root 1u VBAD (revoked) zfs 79777 root 2u VBAD (revoked) zfs 79777 root 3u VCHR 0,72 0t0 72 /dev/zfs zfs 79777 root 4r VCHR 0,8 0t0 8 /dev/zero zfs 79777 root 5r VREG 37,1520762947 4505 954 /etc/zfs/exports zfs 79777 root 6u VCHR 0,72 0t0 72 /dev/zfs zfs 79777 root 7r VCHR 0,8 0t0 8 /dev/zero zfs 79777 root 8u VCHR 0,72 0t0 72 /dev/zfs Here are some more lines of "ps ax" showing some zpool and zfs commands which immediately entered uninterruptible sleep: 81000 19- D+ 0:00.00 zpool list 80828 15- D+ 0:00.01 zpool destroy zpool1 80753 14- D+ 0:00.09 zpool export zpool1 The errors showing up in /var/log/messages when my harddisk went west are (excerpt): May 19 14:57:47 srv0 kernel: ahcich7: Timeout on slot 2 port 0 May 19 14:57:47 srv0 kernel: ahcich7: is 00000000 cs 000007f0 ss 000007fc rs 000007fc tfd 40 serr 00000000 cmd 0004c317 May 19 14:57:47 srv0 kernel: (ada7:ahcich7:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 8c bf 6a 40 00 00 00 01 00 00 May 19 14:57:47 srv0 kernel: (ada7:ahcich7:0:0:0): CAM status: Command timeout May 19 14:57:47 srv0 kernel: (ada7:ahcich7:0:0:0): Retrying command May 19 14:58:17 srv0 kernel: ahcich7: Timeout on slot 18 port 0 May 19 14:58:17 srv0 kernel: ahcich7: is 00000000 cs 00040000 ss 00000000 rs 00040000 tfd c0 serr 00000000 cmd 0004d217 May 19 14:58:17 srv0 kernel: (ada7:ahcich7:0:0:0): WRITE_DMA. ACB: ca 00 8c c8 6a 40 00 00 00 00 00 00 May 19 14:58:17 srv0 kernel: (ada7:ahcich7:0:0:0): CAM status: Command timeout May 19 14:58:17 srv0 kernel: (ada7:ahcich7:0:0:0): Retrying command May 19 14:58:47 srv0 kernel: ahcich7: Timeout on slot 19 port 0 May 19 14:58:47 srv0 kernel: ahcich7: is 00000000 cs 00000000 ss 00080000 rs 00080000 tfd 40 serr 00000000 cmd 0004d317 May 19 14:58:47 srv0 kernel: (ada7:ahcich7:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 8c ca 6a 40 00 00 00 01 00 00 May 19 14:58:47 srv0 kernel: (ada7:ahcich7:0:0:0): CAM status: Command timeout May 19 14:58:47 srv0 kernel: (ada7:ahcich7:0:0:0): Retrying command May 19 14:59:17 srv0 kernel: ahcich7: Timeout on slot 26 port 0 May 19 14:59:17 srv0 kernel: ahcich7: is 00000000 cs 04000000 ss 00000000 rs 04000000 tfd c0 serr 00000000 cmd 0004da17 May 19 14:59:17 srv0 kernel: (ada7:ahcich7:0:0:0): WRITE_DMA. ACB: ca 00 8c d4 6a 40 00 00 00 00 00 00 May 19 14:59:17 srv0 kernel: (ada7:ahcich7:0:0:0): CAM status: Command timeout May 19 14:59:17 srv0 kernel: (ada7:ahcich7:0:0:0): Retrying command May 19 14:59:47 srv0 kernel: ahcich7: Timeout on slot 26 port 0 May 19 14:59:47 srv0 kernel: ahcich7: is 00000000 cs f0000007 ss fc000007 rs fc000007 tfd 40 serr 00000000 cmd 0004db17 May 19 14:59:47 srv0 kernel: (ada7:ahcich7:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 8c d5 6a 40 00 00 00 01 00 00 May 19 14:59:47 srv0 kernel: (ada7:ahcich7:0:0:0): CAM status: Command timeout May 19 14:59:47 srv0 kernel: (ada7:ahcich7:0:0:0): Retrying command May 19 15:00:17 srv0 kernel: ahcich7: Timeout on slot 10 port 0 May 19 15:00:17 srv0 kernel: ahcich7: is 00000000 cs 0007f000 ss 0007fc00 rs 0007fc00 tfd 40 serr 00000000 cmd 0004cb17 May 19 15:00:17 srv0 kernel: (ada7:ahcich7:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 8c e0 6a 40 00 00 00 01 00 00 May 19 15:00:17 srv0 kernel: (ada7:ahcich7:0:0:0): CAM status: Command timeout May 19 15:00:17 srv0 kernel: (ada7:ahcich7:0:0:0): Retrying command May 19 15:00:48 srv0 kernel: ahcich7: Timeout on slot 0 port 0 May 19 15:00:48 srv0 kernel: ahcich7: is 00000000 cs c000001f ss f800001f rs f800001f tfd 40 serr 00000000 cmd 0004dd17 May 19 15:00:48 srv0 kernel: (ada7:ahcich7:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 0b 8c f3 6a 40 00 00 00 00 00 00 May 19 15:00:48 srv0 kernel: (ada7:ahcich7:0:0:0): CAM status: Command timeout May 19 15:00:48 srv0 kernel: (ada7:ahcich7:0:0:0): Retrying command Lines of this form continued for some minutes and after a while, my geli volume on this hdd began complaining as well: May 19 15:03:08 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3593285120, length=131072)] May 19 15:03:08 srv0 kernel: May 19 15:03:08 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3593416192, length=131072)] May 19 15:03:08 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3593547264, length=131072)] May 19 15:03:08 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3593678336, length=131072)] May 19 15:03:08 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3593809408, length=131072)] May 19 15:03:08 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3593940480, length=131072)] May 19 15:03:08 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3594071552, length=131072)] May 19 15:03:08 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3594202624, length=131072)] May 19 15:03:08 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3594333696, length=131072)] May 19 15:03:08 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3594464768, length=131072)] May 19 15:03:08 srv0 kernel: GEOM_ELI: g_eli_read_done() failed label/bkp101.eli[READ(offset=270336, length=8192)] May 19 15:03:08 srv0 kernel: GEOM_ELI: g_eli_read_done() failed label/bkp101.eli[READ(offset=1500301238272, length=8192)] May 19 15:03:08 srv0 kernel: GEOM_ELI: g_eli_read_done() failed label/bkp101.eli[READ(offset=1500301500416, length=8192)] May 19 15:03:08 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3594595840, length=131072)] May 19 15:03:08 srv0 kernel: GEOM_ELI May 19 15:03:08 srv0 kernel: : Crypto WRITE request failed (error=6). May 19 15:03:08 srv0 kernel: May 19 15:03:08 srv0 kernel: label/bkp101.eli[WRITE(offset=3595251200, length=131072)] May 19 15:03:09 srv0 kernel: May 19 15:03:09 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3594857984, length=131072)] May 19 15:03:09 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3595513344, length=131072)] May 19 15:03:09 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3595382272, length=131072)] May 19 15:03:09 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3595120128, length=131072)] May 19 15:03:09 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3594989056, length=131072)] May 19 15:03:09 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3594726912, length=131072)] May 19 15:03:09 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3595644416, length=131072)] May 19 15:03:09 srv0 kernel: GEOM_ELI: Crypto WRITE request failed (error=6). label/bkp101.eli[WRITE(offset=3595775488, length=131072)] Is there any hope for me to resolve this issue without a reboot? Thanks for your help, Simon From owner-freebsd-fs@FreeBSD.ORG Tue May 19 16:03:10 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D193D8D4 for ; Tue, 19 May 2015 16:03:10 +0000 (UTC) Received: from mail.egr.msu.edu (hill.egr.msu.edu [35.9.37.162]) by mx1.freebsd.org (Postfix) with ESMTP id AC78B10F5 for ; Tue, 19 May 2015 16:03:10 +0000 (UTC) Received: from hill (localhost [127.0.0.1]) by mail.egr.msu.edu (Postfix) with ESMTP id E9A6E2A0E1 for ; Tue, 19 May 2015 12:03:08 -0400 (EDT) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mail.egr.msu.edu ([127.0.0.1]) by hill (hill.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hfYgl_5uUU6f for ; Tue, 19 May 2015 12:03:08 -0400 (EDT) Received: from EGR authenticated sender mcdouga9 Message-ID: <555B5EBB.20306@egr.msu.edu> Date: Tue, 19 May 2015 12:03:07 -0400 From: Adam McDougall User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: hardware fault during ZFS send/receive blocks /dev/zfs indefinitely References: <86wq048x8h.fsf@emacs.campese.org> In-Reply-To: <86wq048x8h.fsf@emacs.campese.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 May 2015 16:03:10 -0000 (trimmed) On 05/19/2015 10:20, Simon Campese wrote: > Hello, > > I tried to send/receive a ZFS filesystem from a raidz2-pool to another > pool with just a single disk, when this disk failed. As a result, now > both, the zfs send and zfs receive processes are in uninterruptible > sleep state and all new zpool and zfs commands which I issue immediately > enter uninterruptible sleep. Is this just bad luck (i.e. my disk failed > in the wrong moment) or might this be a bug? > > Anyway, my only solution is to schedule a reboot soon as the machine is > a file server and the operational status of zfs is critical. > > I'm not very experienced with zfs or the FreeBSD kernel, so I just try > to supply as much relevant information as possible. Please tell me if > there is more I can do. > > The system I run is FreeBSD 10.1-RELEASE-p6, the machine is a small intel > file server (eight core Atom, 64G Ram, Supermicro board, two raidz2 > pools connected via reflashed IBM M1015 controllers). Here are the > relevant lines from "ps ax" (with anonymized pool/filesystem names): > > The errors showing up in /var/log/messages when my harddisk went west > are (excerpt): > > May 19 15:00:48 srv0 kernel: ahcich7: Timeout on slot 0 port 0 > May 19 15:00:48 srv0 kernel: ahcich7: is 00000000 cs c000001f ss > f800001f rs f800001f tfd 40 serr 00000000 cmd 0004dd17 > May 19 15:00:48 srv0 kernel: (ada7:ahcich7:0:0:0): > WRITE_FPDMA_QUEUED. ACB: 61 0b 8c f3 6a 40 00 00 00 00 00 00 > May 19 15:00:48 srv0 kernel: (ada7:ahcich7:0:0:0): CAM status: Command > timeout > May 19 15:00:48 srv0 kernel: (ada7:ahcich7:0:0:0): Retrying command > > Lines of this form continued for some minutes and after a while, my geli > volume on this hdd began complaining as well: > > May 19 15:03:09 srv0 kernel: GEOM_ELI: Crypto WRITE request failed > (error=6). label/bkp101.eli[WRITE(offset=3595775488, length=131072)] > > Is there any hope for me to resolve this issue without a reboot? > > Thanks for your help, > > Simon Can you try using the geli and/or glabel command to force detach label/bkp101.eli so zfs treats it as a failure? Also I'm not sure how geli and glabel will treat it but you could try sysctl kern.cam.ada.retry_count=0 to make the kernel give up on the disk quicker and the "failure" might cascade up to zfs where it should hopefully give up on the disk. I think the problem here is ZFS does not know about the incomplete failures on the lower layers. From owner-freebsd-fs@FreeBSD.ORG Tue May 19 23:35:43 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 32966AF3 for ; Tue, 19 May 2015 23:35:43 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id F16B41616 for ; Tue, 19 May 2015 23:35:42 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2D5AwAEyFtV/95baINcg2RkgxjBcQmBVoV6ggAUAQEBAQEBAYEKhEyBCwINGQJfiD+ZJY9dpCoBAQEHAQEBAQEBGASBIYoZhDoXgyOBRQWXHYd0g2mSACOCCQEcgW4igXeBAQEBAQ X-IronPort-AV: E=Sophos;i="5.13,460,1427774400"; d="scan'208";a="211556600" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 19 May 2015 19:35:36 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 21747B41F1 for ; Tue, 19 May 2015 19:35:36 -0400 (EDT) Date: Tue, 19 May 2015 19:35:36 -0400 (EDT) From: Rick Macklem To: FreeBSD Filesystems Message-ID: <1556767526.41012297.1432078536125.JavaMail.root@uoguelph.ca> Subject: Review of patch that makes NFSv4 server hash table sizing tunable MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 May 2015 23:35:43 -0000 I think the subject line says it. There is a patch.. https://reviews.freebsd.org/D2596 which makes the hash table sizes for the NFSv4 server tunable. It does not change the default sizes of the hash tables and, since these tables are only used by NFSv4, should have no effect on NFSv3. I do not believe there is sufficient experience with the NFSv4 server under heavy client load to autotune these hash tables, but hopefully tunable hash table sizing will provide further experience/testing of different sizes. Testing of a single client did not show that larger tables had any significant effect on performance, but a server handling many NFSv4 clients may need larger tables. Please review/comment on this patch, rick From owner-freebsd-fs@FreeBSD.ORG Wed May 20 00:04:23 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4062EF69 for ; Wed, 20 May 2015 00:04:23 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2BBCE198E for ; Wed, 20 May 2015 00:04:23 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K04NOG038424 for ; Wed, 20 May 2015 00:04:23 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 00:04:23 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ler@lerctr.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 00:04:23 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #10 from Larry Rosenman --- Created attachment 156962 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=156962&action=edit 2nd Panic with DEBUG_MEMGUARD on buildworld -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 00:04:52 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 55BF7FDA for ; Wed, 20 May 2015 00:04:52 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 418FE1996 for ; Wed, 20 May 2015 00:04:52 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K04quu059890 for ; Wed, 20 May 2015 00:04:52 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 00:04:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ler@lerctr.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 00:04:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #11 from Larry Rosenman --- What else can I supply? I have all the vmcore's -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 00:25:53 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5653D52F; Wed, 20 May 2015 00:25:53 +0000 (UTC) Received: from mail-qg0-x232.google.com (mail-qg0-x232.google.com [IPv6:2607:f8b0:400d:c04::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0F6F11C6E; Wed, 20 May 2015 00:25:53 +0000 (UTC) Received: by qgew3 with SMTP id w3so16351604qge.2; Tue, 19 May 2015 17:25:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type; bh=TAM4u2ekNliyvMzJuUDz9L+AKnhMeDMC/6gmrgT0+sU=; b=Y7d9Xszn+59GLPIjS4MmeXZOFbj6pfqxh/Yr0b6XoPbSj/KnS0Z859gpbkZQmzC98i UUUaez00KFwnTxmBV4BkzG33iHSURD4r45OTNEuP0wUFspBpJMPiMT538Sohr72PEqoT VdkkkyaLt7ayHPLIXkv0fo5h9BO6wE21UeuYiJxauqj2eQusrI3cM8e5JH6U6h9X3DKi RPyBEaaodoj9SJkctC01Euzd2FqaoM9c5UI+I1Rlhz3zE4wKcrYmTYPH+mFT4kX8Q6H6 4wEKFOvFFMFpy4zdhMsDRhTSvQjxbl3qWhx6DrsaFfzgWEgZZ1fijAPI7ONUPxATBLJj 40jA== X-Received: by 10.55.24.170 with SMTP id 42mr10303267qky.48.1432081552021; Tue, 19 May 2015 17:25:52 -0700 (PDT) Received: from kan ([2601:6:6780:7e0:226:18ff:fe00:232e]) by mx.google.com with ESMTPSA id 187sm10062104qhc.39.2015.05.19.17.25.51 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 May 2015 17:25:51 -0700 (PDT) Date: Tue, 19 May 2015 20:25:44 -0400 From: Alexander Kabaev To: bugzilla-noreply@freebsd.org Cc: freebsd-fs@FreeBSD.org Subject: Re: [Bug 200288] Modify after Free: ZFS Message-ID: <20150519202544.21df1532@kan> In-Reply-To: References: X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.27; amd64-portbld-freebsd11.0) MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/oM=bSAaMYaxFLSVeI=MDg./"; protocol="application/pgp-signature" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 00:25:53 -0000 --Sig_/oM=bSAaMYaxFLSVeI=MDg./ Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Wed, 20 May 2015 00:04:52 +0000 bugzilla-noreply@freebsd.org wrote: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D200288 >=20 > --- Comment #11 from Larry Rosenman --- > What else can I supply?=20 >=20 > I have all the vmcore's >=20 > --=20 > You are receiving this mail because: > You are the assignee for the bug. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" DEBUG_MEMGUARD is not useful by itself, unfortunately, so two panicsyou posted last add nothing to previous reports You need to enable memguard protection for 'solaris' malloc type which might quickly become impractical as this is an umbrella for all allocations done by OpenSolaris code (read: most of XFS and dtrace). See memguard(9) for details on how to enable memguard for specific malloc type.=20 --=20 Alexander Kabaev --Sig_/oM=bSAaMYaxFLSVeI=MDg./ Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlVb1IoACgkQQ6z1jMm+XZZqDACfeK/vAMYYlOyOLHT2dV5zA9m4 9RYAnjvbTf2PsmKUp+2u/2j/YqLqzxaP =pkoB -----END PGP SIGNATURE----- --Sig_/oM=bSAaMYaxFLSVeI=MDg./-- From owner-freebsd-fs@FreeBSD.ORG Wed May 20 00:32:41 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0679E616 for ; Wed, 20 May 2015 00:32:41 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E5AC21D57 for ; Wed, 20 May 2015 00:32:40 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K0Wect058992 for ; Wed, 20 May 2015 00:32:40 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 00:32:40 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: delphij@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 00:32:41 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 Xin LI changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |delphij@FreeBSD.org --- Comment #12 from Xin LI --- (In reply to Larry Rosenman from comment #10) I have looked at all these backtraces and they are almost identical, and unfortunately errors like this are relatively hard to trace down, it's like looking at dead body (assertion, which we already know where the deadly wound is) but not a smoking gun (the code that touches freed memory). Debugging this can be really time consuming. I don't know why memguard didn't catch the issue (which should), did you added the loader.conf option: vm.memguard.desc="solaris" ? I *think* that the issue is probably a decreasing counter, but a quick glance at zfs_rlock.c didn't reveal one, and the code have been there for quite a while. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 00:39:28 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CFA01739 for ; Wed, 20 May 2015 00:39:28 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BACED1DA4 for ; Wed, 20 May 2015 00:39:28 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K0dSDG061199 for ; Wed, 20 May 2015 00:39:28 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 00:39:28 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ler@lerctr.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 00:39:28 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #13 from Larry Rosenman --- had NOT added vm.memguard.desc="solaris" bug have now -- will get a new crash -- it's easy to repro :( -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 00:45:33 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6392F854 for ; Wed, 20 May 2015 00:45:33 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F1981E93 for ; Wed, 20 May 2015 00:45:33 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K0jXJa069062 for ; Wed, 20 May 2015 00:45:33 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 00:45:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kan@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 00:45:33 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 Alexander Kabaev changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kan@FreeBSD.org --- Comment #14 from Alexander Kabaev --- Can you tell me what you doing to get the crash? The box crashed twice for me, but stopped before I can get an useful core from it. I wonder if I can speed things up by mimicking your workload. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 00:48:11 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E1AB78FA for ; Wed, 20 May 2015 00:48:11 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CD3A71EBB for ; Wed, 20 May 2015 00:48:11 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K0mB38074041 for ; Wed, 20 May 2015 00:48:11 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 00:48:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ler@lerctr.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 00:48:12 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #15 from Larry Rosenman --- (In reply to Alexander Kabaev from comment #14) I have boinc running, and then do a buildworld. and my "Normal" stuff. I have a core being savecore'd now with solaris in the memguard list. I'll post the core.txt in a couple of minutes. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 01:23:10 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 19D68974 for ; Wed, 20 May 2015 01:23:10 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 04EA01376 for ; Wed, 20 May 2015 01:23:10 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K1N9Ka042561 for ; Wed, 20 May 2015 01:23:09 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 01:23:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kan@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 01:23:10 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #16 from Alexander Kabaev --- Looking at the allocation path, I think it tries to allocate structure of 96 bytes in size, which should put the allocation into '128' malloc zone. You might be better off watching that instead of 'solaris'. 'solaris' is an umbrella type that covers all of the allocations of the code imported from OpenSolaris. On my machine, vmstat -z gives 123470 active allocations in said bucket, while vmstat -m reports 1614027 active allocations for the whole solaris type, about 13x more. Due to the way memguard is implemented, it might be more practical to use selector with less active entries. In reality, I think one will need to hack memguard_cmp_mtp so match 'solaris' _and_ size to match sizeof(struct rl). -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 01:32:53 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3C87EDE2 for ; Wed, 20 May 2015 01:32:53 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 278671507 for ; Wed, 20 May 2015 01:32:53 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K1WrFA052259 for ; Wed, 20 May 2015 01:32:53 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 01:32:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ler@lerctr.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 01:32:53 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #17 from Larry Rosenman --- Created attachment 156969 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=156969&action=edit core.txt with solaris in the memguard list -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 01:42:17 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EF67FF33 for ; Wed, 20 May 2015 01:42:17 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DA7251633 for ; Wed, 20 May 2015 01:42:17 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K1gHTH061863 for ; Wed, 20 May 2015 01:42:17 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 01:42:18 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ler@lerctr.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 01:42:18 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #18 from Larry Rosenman --- I'll have a 2nd core.txt here in a bit -- the system panic'd whilst it was just doing it's background boinc stuff. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 01:55:49 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 58D47292 for ; Wed, 20 May 2015 01:55:49 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 43E01174F for ; Wed, 20 May 2015 01:55:49 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K1tnfV074589 for ; Wed, 20 May 2015 01:55:49 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 01:55:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kan@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 01:55:49 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #19 from Alexander Kabaev --- (In reply to Larry Rosenman from comment #18) Any chance you coulg go to frame 10 and run 'x/i $rip; in kgdb? I was successful in crashing my machine, but it turns out I now neew to update my boot code, which also means I will need a bootable memstick with recent image on it, so my capabilities are reduced to rempote debug only:( If nobody picks this up, I'll continue once my dev box is up again. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 01:59:19 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 23C053C4 for ; Wed, 20 May 2015 01:59:19 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0EC62177D for ; Wed, 20 May 2015 01:59:19 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K1xIgb076153 for ; Wed, 20 May 2015 01:59:18 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 01:59:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ler@lerctr.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 01:59:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #20 from Larry Rosenman --- as soon as the box is finished savecore'ing. (It's slow with memguard). I can also give you ssh access if that would help -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 02:09:31 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6657F49E for ; Wed, 20 May 2015 02:09:31 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 50DD0187D for ; Wed, 20 May 2015 02:09:31 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K29VDR018018 for ; Wed, 20 May 2015 02:09:31 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 02:09:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kan@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 02:09:31 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #21 from Alexander Kabaev --- (In reply to Larry Rosenman from comment #20) I have all I need for a moment, but will take you up on that promise tomorrow, assuming nobody beats me to it. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 02:32:08 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 42EF0761 for ; Wed, 20 May 2015 02:32:08 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2E05A1BA8 for ; Wed, 20 May 2015 02:32:08 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K2W8d0048979 for ; Wed, 20 May 2015 02:32:08 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 02:32:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kan@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 02:32:08 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #22 from Alexander Kabaev --- So, by looking at the zfs range code, I see these constructs: if (rl->r_write_wanted) { cv_broadcast(&rl->r_wr_cv); cv_destroy(&rl->r_wr_cv); } if (rl->r_read_wanted) { cv_broadcast(&rl->r_rd_cv); cv_destroy(&rl->r_rd_cv); } kmem_free(rl, sizeof (rl_t)); The code appears to wakeup waiters running on cv and then immediately proceeds to destroy the container structure. Looking at the recent change in kern_condvar.c, it does appear that thread that awakens does attempt to decrement the cv_waiters counter in what other thread has already freed. Hilarity ensures. Please try to back out the revision 282971 by jhb and see it that brings your stability back. FWIW, I think code in zfs is hilarious. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 03:07:21 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 72E32DF2 for ; Wed, 20 May 2015 03:07:21 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5D9DE101F for ; Wed, 20 May 2015 03:07:21 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K37LD6085421 for ; Wed, 20 May 2015 03:07:21 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 03:07:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ler@lerctr.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 03:07:21 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #23 from Larry Rosenman --- Backed out the revision -- Building a kernel. Have 2 more core.txt's if you want them. We'll see what happens after the kernel boot. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 03:12:35 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 82099EB5 for ; Wed, 20 May 2015 03:12:35 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6D0051122 for ; Wed, 20 May 2015 03:12:35 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K3CZ1H076867 for ; Wed, 20 May 2015 03:12:35 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 03:12:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ler@lerctr.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 03:12:35 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #24 from Larry Rosenman --- added jhb to the CC list since it's his rev that seems(!) to be causing the grief -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 03:43:09 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8C2E31CB for ; Wed, 20 May 2015 03:43:09 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 76DA813E6 for ; Wed, 20 May 2015 03:43:09 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K3h9Jg009070 for ; Wed, 20 May 2015 03:43:09 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 03:43:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ler@lerctr.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 03:43:09 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #25 from Larry Rosenman --- booted the new kernel, and we've made it WAY past where it was crashing on a make buildworld. I think Alexander has nailed the issue. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 03:47:26 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ABA15271 for ; Wed, 20 May 2015 03:47:26 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 96128144A for ; Wed, 20 May 2015 03:47:26 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4K3lQU7012962 for ; Wed, 20 May 2015 03:47:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Wed, 20 May 2015 03:47:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ler@lerctr.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 03:47:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 --- Comment #26 from Larry Rosenman --- for completeness: (kgdb) x/i $rip 0xffffffff80402d39 : mov %r15,%rsi (kgdb) -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 06:29:36 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A8EC7DBC for ; Wed, 20 May 2015 06:29:36 +0000 (UTC) Received: from mail.worldserver.net (mail.worldserver.net [217.13.200.36]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.worldserver.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6CE0A1516 for ; Wed, 20 May 2015 06:29:35 +0000 (UTC) Received: from mailer.rman (unknown [2.239.24.102]) (Authenticated sender: mailer@campese.de) by mail.worldserver.net (Postfix) with ESMTPSA id 8F631300390; Wed, 20 May 2015 08:29:27 +0200 (CEST) From: Simon Campese To: freebsd-fs@freebsd.org Subject: Re: hardware fault during ZFS send/receive blocks /dev/zfs indefinitely In-Reply-To: <86wq048x8h.fsf@emacs.campese.org> References: <86wq048x8h.fsf@emacs.campese.org> Date: Wed, 20 May 2015 08:29:25 +0200 Message-ID: <867fs3danu.fsf@emacs.campese.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 06:29:36 -0000 Hello, >Can you try using the geli and=C3=B8r glabel command to force detach >label/bkp101.eli so zfs treats it as a failure? Also I'm not sure how >geli and glabel will treat it but you could try sysctl >kern.cam.ada.retry_count=3D0 to make the kernel give up on the disk >quicker and the "failure"might cascade up to zfs where it should >hopefully give up on the disk. I think the problem here is ZFS does not >know about the incomplete failures on the lower layers. I've tried that already (forcefully closing the geli device and removing the label) but it doesn't change anything. In fact, this happened automatically as the drive stopped reacting after some time. So in the end it had to be a reboot. I can replicate the situation on a test machine with the same hardware but, strangely enough, it turns out that the drive seems to be in perfectly fine condition (and it should be, used it as cold storage, it is 5 years old but was powered up <30 times for just a couple of hours).=20= =20=20 I will open another thread for this but for the moment, let me tell you the situation: I've created a label on the naked drive and put a geli volume on it. On this geli volume, I can read and write with dd just fine (tested data up to 10G), but if I put a zfs pool on it and either write directly to it, or via send/receive, the drive stops reacting after some seconds (the "write" hdd light stays on indefinitely) and I start getting the CAM errors.=20 To rule out a FreeBSD specific issue, the disk is tested in a linux machine right now (where it has been working before). It already passed an extended SMART test without any errors and right now is in the middle of a badblocks check (also without errors so far). If no errors are found, I will put it back into my FreeBSD test machine and try to put a plain ufs filesystem on it. Best, Simon From owner-freebsd-fs@FreeBSD.ORG Wed May 20 15:43:26 2015 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E35EBDFC; Wed, 20 May 2015 15:43:25 +0000 (UTC) Received: from smtp.digiware.nl (unknown [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A73F61A87; Wed, 20 May 2015 15:43:25 +0000 (UTC) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 2926D16A406; Wed, 20 May 2015 17:43:22 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id B8Dvz59DQ2Gf; Wed, 20 May 2015 17:43:11 +0200 (CEST) Received: from [192.168.101.176] (vpn.ecoracks.nl [31.223.170.173]) by smtp.digiware.nl (Postfix) with ESMTPA id 8658116A404; Wed, 20 May 2015 17:43:11 +0200 (CEST) Message-ID: <555CAB90.7070506@digiware.nl> Date: Wed, 20 May 2015 17:43:12 +0200 From: Willem Jan Withagen User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: fs@freebsd.org, Edward Tomasz Napierala Subject: ZFS / NFS deadlock??? (Was: Re: Unexpected reboot after ctld run into trouble.) References: <55573756.9070503@digiware.nl> In-Reply-To: <55573756.9070503@digiware.nl> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 15:43:26 -0000 On 16/05/2015 14:25, Willem Jan Withagen wrote: > Hi, > > Found the following in my logs: > Losts of > ---- > (0:4:0/0): Task Action: LUN Reset > (0:4:0/0): CTL Status: Command Completed Successfully > sonewconn: pcb 0xfffff8004e69e930: Listen queue overflow: 8 already in > queue awaiting acceptance (740688 occurrences) > (0:4:0/0): Task Action: LUN Reset > (0:4:0/0): CTL Status: Command Completed Successfully > sonewconn: pcb 0xfffff8004e69e930: Listen queue overflow: 8 already in > queue awaiting acceptance (713721 occurrences) > (0:4:0/0): Task Action: LUN Reset > (0:4:0/0): CTL Status: Command Completed Successfully > sonewconn: pcb 0xfffff8004e69e930: Listen queue overflow: 8 already in > queue awaiting acceptance (691776 occurrences) > ---- > > Which then ends in: > ---- > panic: deadlkres: possible deadlock detected for 0xfffff8001ee94920, > blocked for 1801009 ticks > > > cpuid = 1 > Uptime: 14d13h13m47s > Dumping 7557 out of 8175 > MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%Table 'FACP' at > 0xcfedbcf8 > ---- > The system is running ZFS with ZFS-on-root: FreeBSD zfs.digiware.nl > 10.1-STABLE FreeBSD 10.1-STABLE #221 r282282: Fri May 1 06:51:41 > CEST 2015 > > This could stem from the fact that I woke up my Win8 PC which has a > iscsi volume mounted. It is used to store security cam captures on > and does have somewhat bigger traffic on it. > > Suggestions or question to look at are welcome. > I do have a core in /var/crash, but will need some guidance to > retrieve stuff from it. Followup to this story, after some discussion with/debugging by Edward (trasz@): >> Now, the bad news: I don't think I'll be able to help you with this >> one. It looks like the problem is actually NFS-related. Using the >> hex address from the deadlock message in dmesg: >> >> % kgdb boot/kernel/kernel vmcore.3 >> >> (kgdb) p ((struct thread *)0xfffff8001ee94920)->td_proc->p_comm $6 >> = "nfsd", '\0' (kgdb) p ((struct thread >> *)0xfffff8001ee94920)->td_wmesg $7 = 0xffffffff80edcfc3 "zfs" >> >> So it might actually be a ZFS deadlock the nfsd thread tripped on. > The panic was triggered by deadlkres; it noticed that there was a > thread that spent way too much time waiting for something - so, > presumably, it become "hung" due to a deadlock. > The 0xfffff8001ee94920 in dmesg is the address of "struct thread" of the > problematic thread. > The first print shows the "command name" (p_comm) of the process the > thread belongs to. The second print shows the "wait channel", on > which the thread sleeped. So now the questions are: 1) Is this indeed a ZFS / NFS deadlock problem? 2) Who can/wil help to get this worked out? Thanx, --WjW From owner-freebsd-fs@FreeBSD.ORG Wed May 20 16:17:47 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F2398B66; Wed, 20 May 2015 16:17:46 +0000 (UTC) Received: from mail-wg0-x22a.google.com (mail-wg0-x22a.google.com [IPv6:2a00:1450:400c:c00::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A31F31EA5; Wed, 20 May 2015 16:17:46 +0000 (UTC) Received: by wghq2 with SMTP id q2so58331626wgh.1; Wed, 20 May 2015 09:17:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=5IOMShMm8+dArp26rdC6HlVSD9yyeToRQxv5F6nypjA=; b=rlGXV6zxwzpsuqugADBCLJ+gDNOVIfrZrzfy4p4skG0y2/TkHnnJprgdl4N+nSvcW9 xa4WohbKHgqm1ayu6LPGl0EjbZOnxEuwPNOU4QsXL7wLolmE2aRCZo7HiaVpEHulS982 5x/q5ZJGJpltDbzDCUsrohwrif/Rx2V3QFXqAB9QTTDWh2RQ3t81fPwCoi7/8hAtzsLt nDToD5vsxYjdNGQkqJK5bAqdreLZuQuXzq58aqET2TdS4GpzWWfCm2z6niPGvMrBUek7 g8X/AM2Y6BdHgIC6adVj0EDS636a/BJiylBtwI6oxnmp4NwdrJxwvVxURt6cV/OqvN5a NUAw== MIME-Version: 1.0 X-Received: by 10.194.60.43 with SMTP id e11mr30617936wjr.36.1432138664942; Wed, 20 May 2015 09:17:44 -0700 (PDT) Received: by 10.28.30.22 with HTTP; Wed, 20 May 2015 09:17:44 -0700 (PDT) Date: Wed, 20 May 2015 19:17:44 +0300 Message-ID: Subject: L2ARC degraded. Checksum errors, I/O errors From: George Kontostanos To: freebsd-stable , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 16:17:47 -0000 Hello, Appologies if that has been asked before. I was wondering if the following bug has been addressed so far: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198242 Thanks -- George Kontostanos --- From owner-freebsd-fs@FreeBSD.ORG Wed May 20 17:50:43 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9D687394; Wed, 20 May 2015 17:50:43 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 583821B01; Wed, 20 May 2015 17:50:43 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1Yv88X-0004vI-4Q; Wed, 20 May 2015 20:50:33 +0300 Date: Wed, 20 May 2015 20:50:33 +0300 From: Slawa Olhovchenkov To: George Kontostanos Cc: freebsd-stable , "freebsd-fs@freebsd.org" Subject: Re: L2ARC degraded. Checksum errors, I/O errors Message-ID: <20150520175033.GK1394@zxy.spb.ru> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 17:50:43 -0000 On Wed, May 20, 2015 at 07:17:44PM +0300, George Kontostanos wrote: > Hello, > > Appologies if that has been asked before. I was wondering if the following > bug has been addressed so far: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198242 As I know kstat.zfs.misc.arcstats.l2_cksum_bad: 14495 kstat.zfs.misc.arcstats.l2_io_error: 12877 caused by hardware failure on L2ARC device. try replace it. From owner-freebsd-fs@FreeBSD.ORG Wed May 20 19:08:26 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A0F02933; Wed, 20 May 2015 19:08:26 +0000 (UTC) Received: from EXCH2-1.slu.se (webmail.slu.se [77.235.224.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "webmail.slu.se", Issuer "TERENA SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 32BF317ED; Wed, 20 May 2015 19:08:25 +0000 (UTC) Received: from exch2-4.slu.se (77.235.224.124) by EXCH2-1.slu.se (77.235.224.121) with Microsoft SMTP Server (TLS) id 15.0.1076.9; Wed, 20 May 2015 20:53:05 +0200 Received: from exch2-4.slu.se ([fe80::3117:818f:aa48:9d9b]) by exch2-4.slu.se ([fe80::3117:818f:aa48:9d9b%22]) with mapi id 15.00.1076.000; Wed, 20 May 2015 20:53:05 +0200 From: =?utf-8?B?S2FybGkgU2rDtmJlcmc=?= To: Slawa Olhovchenkov CC: George Kontostanos , "freebsd-fs@freebsd.org" , freebsd-stable Subject: Re: L2ARC degraded. Checksum errors, I/O errors Thread-Topic: L2ARC degraded. Checksum errors, I/O errors Thread-Index: AQHQky400/SoOpaDyEOxzwCf8lvG/Q== Date: Wed, 20 May 2015 18:53:05 +0000 Message-ID: <68854ba9-1852-4a2c-b329-f3fecee176b6@email.android.com> Accept-Language: sv-SE, en-US Content-Language: sv-SE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 19:08:26 -0000 DQpEZW4gMjAgbWFqIDIwMTUgNzo1MCBlbSBza3JldiBTbGF3YSBPbGhvdmNoZW5rb3YgPHNsd0B6 eHkuc3BiLnJ1PjoNCj4NCj4gT24gV2VkLCBNYXkgMjAsIDIwMTUgYXQgMDc6MTc6NDRQTSArMDMw MCwgR2VvcmdlIEtvbnRvc3Rhbm9zIHdyb3RlOg0KPg0KPiA+IEhlbGxvLA0KPiA+DQo+ID4gQXBw b2xvZ2llcyBpZiB0aGF0IGhhcyBiZWVuIGFza2VkIGJlZm9yZS4gSSB3YXMgd29uZGVyaW5nIGlm IHRoZSBmb2xsb3dpbmcNCj4gPiBidWcgaGFzIGJlZW4gYWRkcmVzc2VkIHNvIGZhcjoNCj4gPiBo dHRwczovL2J1Z3MuZnJlZWJzZC5vcmcvYnVnemlsbGEvc2hvd19idWcuY2dpP2lkPTE5ODI0Mg0K Pg0KPiBBcyBJIGtub3cNCj4NCj4ga3N0YXQuemZzLm1pc2MuYXJjc3RhdHMubDJfY2tzdW1fYmFk OiAxNDQ5NQ0KPiBrc3RhdC56ZnMubWlzYy5hcmNzdGF0cy5sMl9pb19lcnJvcjogMTI4NzcNCj4N Cj4gY2F1c2VkIGJ5IGhhcmR3YXJlIGZhaWx1cmUgb24gTDJBUkMgZGV2aWNlLg0KPiB0cnkgcmVw bGFjZSBpdC4NCg0KSGFyZGx5LiBXZSBhcmUgc2VlaW5nIHRoaXMgcGhlbm9tZW5vbiBvbiBhbGwg b2YgeW91ciBaRlMgc3RvcmFnZSdzLCB3aGVyZSBzb21lIG9mIHRoZSBMMkFSQydzIGFyZSBuZXcs IHNvIEknZCBiZSB2ZXJ5IHN1cnByaXNlZCBpZiB0aGVzZSBhcmUgcmVhbCBoYXJkd2FyZSBmYWls dXJlcy4gVGhhdCwgYXMgd2VsbCBhcyBiZWluZyByZXBvcnRlZCBieSBzbyBtYW55LCBteSB2b3Rl IGlzIG9uIGJ1Zy4NCg0KQGdrb250b3MNClRoYW5rcyBmb3IgYnJpbmdpbmcgdGhpcyB0byBhdHRl bnRpb24sIEknbSBhbHNvIGludGVyZXN0ZWQgaW4gaGVhcmluZyBpZiBhbnlvbmUncyB3b3JraW5n IG9uIHRoaXMuDQoNCi9LDQoNCj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX18NCj4gZnJlZWJzZC1mc0BmcmVlYnNkLm9yZyBtYWlsaW5nIGxpc3QNCj4gaHR0 cDovL2xpc3RzLmZyZWVic2Qub3JnL21haWxtYW4vbGlzdGluZm8vZnJlZWJzZC1mcw0KPiBUbyB1 bnN1YnNjcmliZSwgc2VuZCBhbnkgbWFpbCB0byAiZnJlZWJzZC1mcy11bnN1YnNjcmliZUBmcmVl YnNkLm9yZyINCg== From owner-freebsd-fs@FreeBSD.ORG Wed May 20 19:40:22 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 04C73A1A; Wed, 20 May 2015 19:40:22 +0000 (UTC) Received: from mail-wi0-x230.google.com (mail-wi0-x230.google.com [IPv6:2a00:1450:400c:c05::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9FFE61C4E; Wed, 20 May 2015 19:40:21 +0000 (UTC) Received: by wicmx19 with SMTP id mx19so166598187wic.0; Wed, 20 May 2015 12:40:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=v89E/u2S2gkhXndX+AkJZ5nRkFyy0ZsWHruq8NUN708=; b=GE3tn0qWp4P2RePbdPfg+J4uoQfpF3BKP/vGw/8hdrpywi85LXGqtAuc/ILvM+SkJU UQLpyPElfbuW+S5SVf422AFGrNONx1bU6JxecVYf9hnbm8xY/xjNpGlx1TwrHgA6FYlP nZT84lrv9ZEMteFAf7CIBZL1jbdZNaOzci6WqVc8bGfiuALBo56a6vlDjSjqGmEV7+Pc FmMT0EHw2wmB8j5FyuRbpx9fFO7aAJmYC/mLqPWMk1dyyMMS73Qbp471h0B93cOU8/NZ eF1Lfjet7Sfvyy1kLjaVuHsbN0GCd2jQUV3U6t7lT9wAGzsOm26qyOBozhSxe2uvpDgm a+Vw== MIME-Version: 1.0 X-Received: by 10.194.82.38 with SMTP id f6mr64894131wjy.16.1432150820175; Wed, 20 May 2015 12:40:20 -0700 (PDT) Received: by 10.28.30.22 with HTTP; Wed, 20 May 2015 12:40:20 -0700 (PDT) In-Reply-To: <68854ba9-1852-4a2c-b329-f3fecee176b6@email.android.com> References: <68854ba9-1852-4a2c-b329-f3fecee176b6@email.android.com> Date: Wed, 20 May 2015 22:40:20 +0300 Message-ID: Subject: Re: L2ARC degraded. Checksum errors, I/O errors From: George Kontostanos To: =?UTF-8?Q?Karli_Sj=C3=B6berg?= Cc: "freebsd-fs@freebsd.org" , freebsd-stable Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 19:40:22 -0000 On Wed, May 20, 2015 at 9:53 PM, Karli Sj=C3=B6berg = wrote: > > Den 20 maj 2015 7:50 em skrev Slawa Olhovchenkov : > > > > On Wed, May 20, 2015 at 07:17:44PM +0300, George Kontostanos wrote: > > > > > Hello, > > > > > > Appologies if that has been asked before. I was wondering if the > following > > > bug has been addressed so far: > > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D198242 > > > > As I know > > > > kstat.zfs.misc.arcstats.l2_cksum_bad: 14495 > > kstat.zfs.misc.arcstats.l2_io_error: 12877 > > > > caused by hardware failure on L2ARC device. > > try replace it. > > Hardly. We are seeing this phenomenon on all of your ZFS storage's, where > some of the L2ARC's are new, so I'd be very surprised if these are real > hardware failures. That, as well as being reported by so many, my vote is > on bug. > > @gkontos > Thanks for bringing this to attention, I'm also interested in hearing if > anyone's working on this. > > /K > Thanks, I am ignoring the previous comment as it makes no real sense. > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > --=20 George Kontostanos --- From owner-freebsd-fs@FreeBSD.ORG Wed May 20 20:16:22 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 04285639; Wed, 20 May 2015 20:16:22 +0000 (UTC) Received: from EXCH2-1.slu.se (pop.slu.se [77.235.224.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "webmail.slu.se", Issuer "TERENA SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 618791116; Wed, 20 May 2015 20:16:20 +0000 (UTC) Received: from exch2-4.slu.se (77.235.224.124) by EXCH2-1.slu.se (77.235.224.121) with Microsoft SMTP Server (TLS) id 15.0.1076.9; Wed, 20 May 2015 22:16:10 +0200 Received: from exch2-4.slu.se ([fe80::3117:818f:aa48:9d9b]) by exch2-4.slu.se ([fe80::3117:818f:aa48:9d9b%22]) with mapi id 15.00.1076.000; Wed, 20 May 2015 22:16:10 +0200 From: =?utf-8?B?S2FybGkgU2rDtmJlcmc=?= To: George Kontostanos CC: "freebsd-fs@freebsd.org" , freebsd-stable Subject: Re: L2ARC degraded. Checksum errors, I/O errors Thread-Topic: L2ARC degraded. Checksum errors, I/O errors Thread-Index: AQHQkznP0/SoOpaDyEOxzwCf8lvG/Q== Date: Wed, 20 May 2015 20:16:09 +0000 Message-ID: Accept-Language: sv-SE, en-US Content-Language: sv-SE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 May 2015 20:16:22 -0000 DQpEZW4gMjAgbWFqIDIwMTUgOTo0MCBlbSBza3JldiBHZW9yZ2UgS29udG9zdGFub3MgPGdrb250 b3MubWFpbEBnbWFpbC5jb20+Og0KPg0KPg0KPg0KPiBPbiBXZWQsIE1heSAyMCwgMjAxNSBhdCA5 OjUzIFBNLCBLYXJsaSBTasO2YmVyZyA8a2FybGkuc2pvYmVyZ0BzbHUuc2U+IHdyb3RlOg0KPj4N Cj4+DQo+PiBEZW4gMjAgbWFqIDIwMTUgNzo1MCBlbSBza3JldiBTbGF3YSBPbGhvdmNoZW5rb3Yg PHNsd0B6eHkuc3BiLnJ1PjoNCj4+ID4NCj4+ID4gT24gV2VkLCBNYXkgMjAsIDIwMTUgYXQgMDc6 MTc6NDRQTSArMDMwMCwgR2VvcmdlIEtvbnRvc3Rhbm9zIHdyb3RlOg0KPj4gPg0KPj4gPiA+IEhl bGxvLA0KPj4gPiA+DQo+PiA+ID4gQXBwb2xvZ2llcyBpZiB0aGF0IGhhcyBiZWVuIGFza2VkIGJl Zm9yZS4gSSB3YXMgd29uZGVyaW5nIGlmIHRoZSBmb2xsb3dpbmcNCj4+ID4gPiBidWcgaGFzIGJl ZW4gYWRkcmVzc2VkIHNvIGZhcjoNCj4+ID4gPiBodHRwczovL2J1Z3MuZnJlZWJzZC5vcmcvYnVn emlsbGEvc2hvd19idWcuY2dpP2lkPTE5ODI0Mg0KPj4gPg0KPj4gPiBBcyBJIGtub3cNCj4+ID4N Cj4+ID4ga3N0YXQuemZzLm1pc2MuYXJjc3RhdHMubDJfY2tzdW1fYmFkOiAxNDQ5NQ0KPj4gPiBr c3RhdC56ZnMubWlzYy5hcmNzdGF0cy5sMl9pb19lcnJvcjogMTI4NzcNCj4+ID4NCj4+ID4gY2F1 c2VkIGJ5IGhhcmR3YXJlIGZhaWx1cmUgb24gTDJBUkMgZGV2aWNlLg0KPj4gPiB0cnkgcmVwbGFj ZSBpdC4NCj4+DQo+PiBIYXJkbHkuIFdlIGFyZSBzZWVpbmcgdGhpcyBwaGVub21lbm9uIG9uIGFs bCBvZiB5b3VyIFpGUyBzdG9yYSBoZWxhLCB3aGVyZSBzb21lIG9mIHRoZSBMMkFSQydzIGFyZSBu ZXcsIHNvIEknZCBiZSB2ZXJ5IHN1cnByaXNlZCBpZiB0aGVzZSBhcmUgcmVhbCBoYXJkd2FyZSBm YWlsdXJlcy4gVGhhdCwgYXMgd2VsbCBhcyBiZWluZyByZXBvcnRlZCBieSBzbyBtYW55LCBteSB2 b3RlIGlzIG9uIGJ1Zy4NCj4+DQo+PiBAZ2tvbnRvcw0KPj4gVGhhbmtzIGZvciBicmluZ2luZyB0 aGlzIHRvIGF0dGVudGlvbiwgSSdtIGFsc28gaW50ZXJlc3RlZCBpbiBoZWFyaW5nIGlmIGFueW9u ZSdzIHdvcmtpbmcgb24gdGhpcy4NCj4+DQo+PiAvSw0KPg0KPg0KPiBUaGFua3MsIEkgYW0gaWdu b3JpbmcgdGhlIHByZXZpb3VzIGNvbW1lbnQgYXMgaXQgbWFrZXMgbm8gcmVhbCBzZW5zZS4NCg0K V2hvb3BzIQ0KDQoncy95b3VyIFpGUyBzdG9yYWdlJ3Mvb3VyIFpGUyBzdG9yYWdlJ3MvJyA6KQ0K DQovSw0KDQo+DQo+Pg0KPj4gPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fXw0KPj4gPiBmcmVlYnNkLWZzQGZyZWVic2Qub3JnIG1haWxpbmcgbGlzdA0KPj4g PiBodHRwOi8vbGlzdHMuZnJlZWJzZC5vcmcvbWFpbG1hbi9saXN0aW5mby9mcmVlYnNkLWZzDQo+ PiA+IFRvIHVuc3Vic2NyaWJlLCBzZW5kIGFueSBtYWlsIHRvICJmcmVlYnNkLWZzLXVuc3Vic2Ny aWJlQGZyZWVic2Qub3JnIg0KPg0KPg0KPg0KPg0KPiAtLQ0KPiBHZW9yZ2UgS29udG9zdGFub3MN Cj4gLS0tDQo= From owner-freebsd-fs@FreeBSD.ORG Thu May 21 01:54:41 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 50352128 for ; Thu, 21 May 2015 01:54:41 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3AFEA1764 for ; Thu, 21 May 2015 01:54:41 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4L1sf8g068447 for ; Thu, 21 May 2015 01:54:41 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Thu, 21 May 2015 01:54:41 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: grehan@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 May 2015 01:54:41 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 Peter Grehan changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |grehan@FreeBSD.org --- Comment #27 from Peter Grehan --- I can insta-repro this by running fio on a zvol with mixed r/w e.g. fio --filename=/dev/zvol/xxxxxx --direct=1 --rw=rw --refill_buffers \ --norandommap --randrepeat=0 --bs=8k \ --rwmixread=70 --iodepth=8 --numjobs=24 --runtime=60 \ --ioengine=pvsync \ --group_reporting --name=crashtest I can also confirm that reverting the condvar change stopped the crashes. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Thu May 21 11:07:32 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C20C8DA4; Thu, 21 May 2015 11:07:32 +0000 (UTC) Received: from mail-wi0-x22b.google.com (mail-wi0-x22b.google.com [IPv6:2a00:1450:400c:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F86C1AF8; Thu, 21 May 2015 11:07:32 +0000 (UTC) Received: by wizk4 with SMTP id k4so10621178wiz.1; Thu, 21 May 2015 04:07:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=OUe2uac6aVgrKI/xGrXJzTJ6KnbFlXBD8kByNq3gXbQ=; b=lX22Yiu7aV7YueNU6OlgPnr9rK6fXDv6UUJ01tcHiVRS/AV8CDaR8Y5LmBd6rSzxQ+ u/KyuASq4l61iyV+j6eVWt5+XUuIh690ARoAY9TuSD63JtuaiiwC7BumNVMgFXqeV4GT /szAsacfRBHo8bZ3Oc5OLRSmzdX/cLFo44SgfvgXUYf1KNUM9Slu8ajLyrtn8FeZE8Nq JCj8rDVlDCBVpG6N8eoplrN2oaP1yOF0gBgXc2ZIXMOWvGjsPfZFQEuWVCv5eDpmRYH3 SM/ZAkoRSFMYshX+O79Q760JvOzMhXaaOFvD5WrxlGEA0fTKHtPZohOYBL6bwg0d6YOs YRaA== MIME-Version: 1.0 X-Received: by 10.180.101.138 with SMTP id fg10mr5238792wib.46.1432206450700; Thu, 21 May 2015 04:07:30 -0700 (PDT) Received: by 10.28.30.22 with HTTP; Thu, 21 May 2015 04:07:30 -0700 (PDT) In-Reply-To: References: Date: Thu, 21 May 2015 14:07:30 +0300 Message-ID: Subject: Re: L2ARC degraded. Checksum errors, I/O errors From: George Kontostanos To: =?UTF-8?Q?Karli_Sj=C3=B6berg?= Cc: "freebsd-fs@freebsd.org" , freebsd-stable Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 May 2015 11:07:32 -0000 On Wed, May 20, 2015 at 11:16 PM, Karli Sj=C3=B6berg wrote: > > Den 20 maj 2015 9:40 em skrev George Kontostanos = : > > > > > > > > On Wed, May 20, 2015 at 9:53 PM, Karli Sj=C3=B6berg > wrote: > >> > >> > >> Den 20 maj 2015 7:50 em skrev Slawa Olhovchenkov : > >> > > >> > On Wed, May 20, 2015 at 07:17:44PM +0300, George Kontostanos wrote: > >> > > >> > > Hello, > >> > > > >> > > Appologies if that has been asked before. I was wondering if the > following > >> > > bug has been addressed so far: > >> > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D198242 > >> > > >> > As I know > >> > > >> > kstat.zfs.misc.arcstats.l2_cksum_bad: 14495 > >> > kstat.zfs.misc.arcstats.l2_io_error: 12877 > >> > > >> > caused by hardware failure on L2ARC device. > >> > try replace it. > >> > >> Hardly. We are seeing this phenomenon on all of your ZFS stora hela, > where some of the L2ARC's are new, so I'd be very surprised if these are > real hardware failures. That, as well as being reported by so many, my vo= te > is on bug. > >> > >> @gkontos > >> Thanks for bringing this to attention, I'm also interested in hearing > if anyone's working on this. > >> > >> /K > > > > > > Thanks, I am ignoring the previous comment as it makes no real sense. > > Whoops! > > 's/your ZFS storage's/our ZFS storage's/' :) > > /K > > That too :) But I was referring to the comment made by @slw --=20 George Kontostanos --- From owner-freebsd-fs@FreeBSD.ORG Thu May 21 17:01:53 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 76A6A2CB for ; Thu, 21 May 2015 17:01:53 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6201318F3 for ; Thu, 21 May 2015 17:01:53 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4LH1rM6088997 for ; Thu, 21 May 2015 17:01:53 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200288] Modify after Free: ZFS Date: Thu, 21 May 2015 17:01:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: allanjude@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 May 2015 17:01:53 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 Allan Jude changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |allanjude@FreeBSD.org --- Comment #28 from Allan Jude --- jhb@ just committed r283250 which reverts the commit that caused the problem: https://svnweb.freebsd.org/changeset/base/r283250 -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Fri May 22 05:25:26 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5D62ED15 for ; Fri, 22 May 2015 05:25:26 +0000 (UTC) Received: from mail-la0-x232.google.com (mail-la0-x232.google.com [IPv6:2a00:1450:4010:c03::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C2BA61BCB for ; Fri, 22 May 2015 05:25:25 +0000 (UTC) Received: by lami4 with SMTP id i4so5199612lam.0 for ; Thu, 21 May 2015 22:25:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=s88SXTdc5vg+hkpSlRQur960MMDhHt69VbernXzVGeI=; b=J4X4m+2RMfx3k/fVwKwxpFn3NNUKZgdwYMOvwxm0LpVSGbfU0g/4C+Gk3wj5CYnCCz 5dGh4298jV977eTZe7P2NKXJ3VDS6jym8SpaNpnNqblrcq8jdc6lk0kJuhrXblALc6vr k8SJYf9hbELbCpLEaruPHyvEfLUGwhy5voTyl50VRKbVq+Zv3Q4eVg0Xb7UjeWL+soR1 uBMnwNY0V7GNIz5RA8iMbgT1x/E1qlokURtDtmIkcw7gfXayZWiI/6uZrUNLIe7hT3gx 3akIym5Q1xlLL9xmVE4JAJW8By7q4nysy9O8HyDTxUxr3o7q6OX3itUBUHuxukFDj3zq rd7g== MIME-Version: 1.0 X-Received: by 10.152.43.37 with SMTP id t5mr4869878lal.96.1432272322708; Thu, 21 May 2015 22:25:22 -0700 (PDT) Received: by 10.114.96.8 with HTTP; Thu, 21 May 2015 22:25:22 -0700 (PDT) Date: Thu, 21 May 2015 22:25:22 -0700 Message-ID: Subject: Odd member label in zpool From: javocado To: FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2015 05:25:26 -0000 What is going on with the first device label (should be da0): # zpool status pool: data1 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM data1 ONLINE 0 0 0 raidz3-0 ONLINE 0 0 0 diskid/DISK-%20%20%20%20%20%20PK1301PBJDWK8S ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 da8 ONLINE 0 0 0 da9 ONLINE 0 0 0 da10 ONLINE 0 0 0 da11 ONLINE 0 0 0 da12 ONLINE 0 0 0 da13 ONLINE 0 0 0 da14 ONLINE 0 0 0 da15 ONLINE 0 0 0 errors: No known data errors I made this pool as follows: gnop create -S 4096 /dev/da0 zpool create -O devices=off -O atime=off -O setuid=off -O exec=off \ -O compression=on -m /mnt/data1 data1 raidz3 da0.nop \ da{1,2,3,4,5,6,7,8,9,10,11,12,13,14} zpool export data1 gnop destroy /dev/da0.nop zpool import data1 From owner-freebsd-fs@FreeBSD.ORG Fri May 22 09:22:46 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B2ACE9F4 for ; Fri, 22 May 2015 09:22:46 +0000 (UTC) Received: from Exch2-3.slu.se (exch2-3.slu.se [77.235.224.123]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "webmail.slu.se", Issuer "TERENA SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46EA81838 for ; Fri, 22 May 2015 09:22:45 +0000 (UTC) Received: from exch2-4.slu.se (77.235.224.124) by Exch2-3.slu.se (77.235.224.123) with Microsoft SMTP Server (TLS) id 15.0.1076.9; Fri, 22 May 2015 11:07:25 +0200 Received: from exch2-4.slu.se ([fe80::3117:818f:aa48:9d9b]) by exch2-4.slu.se ([fe80::3117:818f:aa48:9d9b%22]) with mapi id 15.00.1076.000; Fri, 22 May 2015 11:07:25 +0200 From: =?utf-8?B?S2FybGkgU2rDtmJlcmc=?= To: javocado CC: FreeBSD Filesystems Subject: Re: Odd member label in zpool Thread-Topic: Odd member label in zpool Thread-Index: AQHQlE+2z4Z59JoILU6mLoW9i1llZZ2Hk1CA Date: Fri, 22 May 2015 09:07:24 +0000 Message-ID: <1432285651.6471.43.camel@data-b104.adm.slu.se> References: In-Reply-To: Accept-Language: sv-SE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [77.235.228.32] Content-Type: text/plain; charset="utf-8" Content-ID: <9ECA9255390ECE42A081B98C5B84944D@ad.slu.se> Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2015 09:22:46 -0000 dG9yIDIwMTUtMDUtMjEga2xvY2thbiAyMjoyNSAtMDcwMCBza3JldiBqYXZvY2FkbzoNCj4gV2hh dCBpcyBnb2luZyBvbiB3aXRoIHRoZSBmaXJzdCBkZXZpY2UgbGFiZWwgKHNob3VsZCBiZSBkYTAp Og0KPiANCj4gIyB6cG9vbCBzdGF0dXMNCj4gICBwb29sOiBkYXRhMQ0KPiAgc3RhdGU6IE9OTElO RQ0KPiAgIHNjYW46IG5vbmUgcmVxdWVzdGVkDQo+IGNvbmZpZzoNCj4gDQo+ICAgICAgICAgTkFN RSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBTVEFURSAgICAg UkVBRA0KPiBXUklURSBDS1NVTQ0KPiAgICAgICAgIGRhdGExICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgT05MSU5FICAgICAgIDAgMA0KPiAgICAgMA0KPiAgICAg ICAgICAgcmFpZHozLTAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgT05M SU5FICAgICAgIDAgMA0KPiAgICAgMA0KPiAgICAgICAgICAgICBkaXNraWQvRElTSy0lMjAlMjAl MjAlMjAlMjAlMjBQSzEzMDFQQkpEV0s4UyAgT05MSU5FICAgICAgIDAgMA0KPiAgICAgMA0KPiAg ICAgICAgICAgICBkYTEgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg T05MSU5FICAgICAgIDAgMA0KPiAgICAgMA0KPiAgICAgICAgICAgICBkYTIgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgT05MSU5FICAgICAgIDAgMA0KPiAgICAgMA0K PiAgICAgICAgICAgICBkYTMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgT05MSU5FICAgICAgIDAgMA0KPiAgICAgMA0KPiAgICAgICAgICAgICBkYTQgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgT05MSU5FICAgICAgIDAgMA0KPiAgICAg MA0KPiAgICAgICAgICAgICBkYTUgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgT05MSU5FICAgICAgIDAgMA0KPiAgICAgMA0KPiAgICAgICAgICAgICBkYTYgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgT05MSU5FICAgICAgIDAgMA0KPiAg ICAgMA0KPiAgICAgICAgICAgICBkYTcgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgT05MSU5FICAgICAgIDAgMA0KPiAgICAgMA0KPiAgICAgICAgICAgICBkYTggICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgT05MSU5FICAgICAgIDAgMA0K PiAgICAgMA0KPiAgICAgICAgICAgICBkYTkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgT05MSU5FICAgICAgIDAgMA0KPiAgICAgMA0KPiAgICAgICAgICAgICBkYTEw ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgT05MSU5FICAgICAgIDAg MA0KPiAgICAgMA0KPiAgICAgICAgICAgICBkYTExICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgT05MSU5FICAgICAgIDAgMA0KPiAgICAgMA0KPiAgICAgICAgICAgICBk YTEyICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgT05MSU5FICAgICAg IDAgMA0KPiAgICAgMA0KPiAgICAgICAgICAgICBkYTEzICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgT05MSU5FICAgICAgIDAgMA0KPiAgICAgMA0KPiAgICAgICAgICAg ICBkYTE0ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgT05MSU5FICAg ICAgIDAgMA0KPiAgICAgMA0KPiAgICAgICAgICAgICBkYTE1ICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgT05MSU5FICAgICAgIDAgMA0KPiAgICAgMA0KPiANCj4gZXJy b3JzOiBObyBrbm93biBkYXRhIGVycm9ycw0KPiANCj4gSSBtYWRlIHRoaXMgcG9vbCBhcyBmb2xs b3dzOg0KPiANCj4gZ25vcCBjcmVhdGUgLVMgNDA5NiAvZGV2L2RhMA0KPiANCj4genBvb2wgY3Jl YXRlIC1PIGRldmljZXM9b2ZmIC1PIGF0aW1lPW9mZiAtTyBzZXR1aWQ9b2ZmIC1PIGV4ZWM9b2Zm IFwNCj4gLU8gY29tcHJlc3Npb249b24gLW0gL21udC9kYXRhMSBkYXRhMSByYWlkejMgZGEwLm5v cCBcDQo+IGRhezEsMiwzLDQsNSw2LDcsOCw5LDEwLDExLDEyLDEzLDE0fQ0KPiANCj4genBvb2wg ZXhwb3J0IGRhdGExDQo+IGdub3AgZGVzdHJveSAvZGV2L2RhMC5ub3ANCj4genBvb2wgaW1wb3J0 IGRhdGExDQoNCiMgenBvb2wgaW1wb3J0IC1kIC9kZXYgZGF0YTENCg0KL0sNCg0KPiBfX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiBmcmVlYnNkLWZzQGZy ZWVic2Qub3JnIG1haWxpbmcgbGlzdA0KPiBodHRwOi8vbGlzdHMuZnJlZWJzZC5vcmcvbWFpbG1h bi9saXN0aW5mby9mcmVlYnNkLWZzDQo+IFRvIHVuc3Vic2NyaWJlLCBzZW5kIGFueSBtYWlsIHRv ICJmcmVlYnNkLWZzLXVuc3Vic2NyaWJlQGZyZWVic2Qub3JnIg0KDQo= From owner-freebsd-fs@FreeBSD.ORG Fri May 22 15:16:01 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 21C7C435 for ; Fri, 22 May 2015 15:16:01 +0000 (UTC) Received: from mail-lb0-x229.google.com (mail-lb0-x229.google.com [IPv6:2a00:1450:4010:c04::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8121E123A for ; Fri, 22 May 2015 15:16:00 +0000 (UTC) Received: by lbcmx3 with SMTP id mx3so15158077lbc.1 for ; Fri, 22 May 2015 08:15:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=4TLSMfJfSE5KUeMv+XGNyxAuvRqfML56jotz3toTnRQ=; b=w6znQi4mfXrALuOy6+DGotCiEimOZMmRU6bsdpcroCiRuT4slSqXKA6B5Cf83oTfZp Af95ctk+09tIHHyNriw/rvh54SWLXebDU/uSC1o6WAc2agYnIrm2R+XdVhBYcY5KsBiB uDj2iubn/wE3wl1slkpd76oCTL7aeGFEa5tA0hbvXUS4p2nHaeILiOYcfmbvalAlbHgS qCK+oM/SEy+TWXgLXQttLz7KJ8nECrYQg/M1O4KnR9gNem7cf2dRfkUnpPzStIwFSIWe bEJwWA1W0Exhuh4dZFLUQSf29i0DO7P+YrCwFbsMwhrtLgAzjbNm9UV/P5VQ++7gbFhw rd7g== MIME-Version: 1.0 X-Received: by 10.152.5.98 with SMTP id r2mr6879884lar.101.1432307758487; Fri, 22 May 2015 08:15:58 -0700 (PDT) Received: by 10.114.96.8 with HTTP; Fri, 22 May 2015 08:15:58 -0700 (PDT) In-Reply-To: <1432285651.6471.43.camel@data-b104.adm.slu.se> References: <1432285651.6471.43.camel@data-b104.adm.slu.se> Date: Fri, 22 May 2015 08:15:58 -0700 Message-ID: Subject: Re: Odd member label in zpool From: javocado To: =?UTF-8?Q?Karli_Sj=C3=B6berg?= Cc: FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2015 15:16:01 -0000 Thank you! No change, however: # zpool export data1 # zpool import -d /dev data1 # zpool status pool: data1 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM data1 ONLINE 0 0 0 raidz3-0 ONLINE 0 0 0 diskid/DISK-%20%20%20%20%20%20PK1301PBJDWK8S ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 da8 ONLINE 0 0 0 da9 ONLINE 0 0 0 da10 ONLINE 0 0 0 da11 ONLINE 0 0 0 da12 ONLINE 0 0 0 da13 ONLINE 0 0 0 da14 ONLINE 0 0 0 da15 ONLINE 0 0 0 errors: No known data errors On Fri, May 22, 2015 at 2:07 AM, Karli Sj=C3=B6berg = wrote: > tor 2015-05-21 klockan 22:25 -0700 skrev javocado: > > What is going on with the first device label (should be da0): > > > > # zpool status > > pool: data1 > > state: ONLINE > > scan: none requested > > config: > > > > NAME STATE REA= D > > WRITE CKSUM > > data1 ONLINE = 0 > 0 > > 0 > > raidz3-0 ONLINE = 0 > 0 > > 0 > > diskid/DISK-%20%20%20%20%20%20PK1301PBJDWK8S ONLINE = 0 > 0 > > 0 > > da1 ONLINE = 0 > 0 > > 0 > > da2 ONLINE = 0 > 0 > > 0 > > da3 ONLINE = 0 > 0 > > 0 > > da4 ONLINE = 0 > 0 > > 0 > > da5 ONLINE = 0 > 0 > > 0 > > da6 ONLINE = 0 > 0 > > 0 > > da7 ONLINE = 0 > 0 > > 0 > > da8 ONLINE = 0 > 0 > > 0 > > da9 ONLINE = 0 > 0 > > 0 > > da10 ONLINE = 0 > 0 > > 0 > > da11 ONLINE = 0 > 0 > > 0 > > da12 ONLINE = 0 > 0 > > 0 > > da13 ONLINE = 0 > 0 > > 0 > > da14 ONLINE = 0 > 0 > > 0 > > da15 ONLINE = 0 > 0 > > 0 > > > > errors: No known data errors > > > > I made this pool as follows: > > > > gnop create -S 4096 /dev/da0 > > > > zpool create -O devices=3Doff -O atime=3Doff -O setuid=3Doff -O exec=3D= off \ > > -O compression=3Don -m /mnt/data1 data1 raidz3 da0.nop \ > > da{1,2,3,4,5,6,7,8,9,10,11,12,13,14} > > > > zpool export data1 > > gnop destroy /dev/da0.nop > > zpool import data1 > > # zpool import -d /dev data1 > > /K > > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > From owner-freebsd-fs@FreeBSD.ORG Fri May 22 19:10:17 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A72BDBF3 for ; Fri, 22 May 2015 19:10:17 +0000 (UTC) Received: from mail-lb0-x22c.google.com (mail-lb0-x22c.google.com [IPv6:2a00:1450:4010:c04::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1438E1FC0 for ; Fri, 22 May 2015 19:10:17 +0000 (UTC) Received: by lbcmx3 with SMTP id mx3so19204439lbc.1 for ; Fri, 22 May 2015 12:10:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=OYrmMQI+LOyF/WZ3XhD17Kxa2VFJWhaFq1r/iQCMF2Q=; b=wL6MnhWCdnEv/TY9n76fq0LY5i25aScvLLk4aplJ6hurFUc9qjo7QFxpjbpZ2t0wUt FRNxDPOr/tO4eD79YIqYuhM7VbhJG1MnTmvtNDTdOpKgv+UyOL9n83WvcY+U6hxZa0C7 SlwcjNOjgY86BjwpsB7VP6wQDxzbrl7NRAyaX6TiFeYz+TqFbsyUMKNdlZtpYnzTJTJ+ dCbiR/SxyCBQwoi7ZlNJbdQ4f6b/popXipxvktqr8bpnrE3iu2bMpRKeCypjYUsSn6Xi A8GPqFboaFLRItWwIrN/EjyAS0QbQjC0+S31A1GysPAafQDc/eqlGNMv2Nex8Cuwty0z BP/w== MIME-Version: 1.0 X-Received: by 10.152.87.13 with SMTP id t13mr7779445laz.66.1432321815161; Fri, 22 May 2015 12:10:15 -0700 (PDT) Received: by 10.25.210.79 with HTTP; Fri, 22 May 2015 12:10:15 -0700 (PDT) Date: Fri, 22 May 2015 14:10:15 -0500 Message-ID: Subject: zpool on Dell MD3000 causes frequent hangs From: Thomas Johnson To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2015 19:10:17 -0000 Hello, I am trying to track down an ongoing issue that I've been having, and looking for any suggestions on a possible cause, or suggestions on how I might troubleshoot further. The issue seems to be related to a Dell MD3000 storage array, which contains a zpool. It seems that the host attached to the array will occasionally hang, usually during periods of high disk activity (annoyingly, usually about 0300). When the system hangs, I can ping the host, and switch between virtual consoles (but not interact with them). The system is otherwise unresponsive; with no errors reported on the console or logs. The only remedy I have found is to hard-reset the host. I believe this issue is tied to the MD3000. I have tried swapping out SAS cables, HBAs, the controller on the MD3000, and the host itself. I have updated all the firmware I can find. Before I upgraded the host OS to FreeBSD 10.1 (from 10.0) last month, I experienced hangs about once a month. Since the upgrade, I have seen several events per week. In addition to the MD3000, I have a set of USB drives that are used in a rotation as offsite backups for the zpool. I have seen a number of hang events during zfs send/receive transfers to the USB disk. After the most recent hang, I removed two [consumer] SSDs from the pool that were being used as cache devices. It is too early to tell if this change had any impact. Here is some of the pertinent output from the host. I can provide any other information that would be helpful. root@leopard:/home/tom-> uname -a FreeBSD leopard 10.1-RELEASE-p9 FreeBSD 10.1-RELEASE-p9 #0 r281232: Tue Apr 7 17:38:04 CDT 2015 root@cheshire-b:/pkg/base/obj_10.1-RELEASE-p9/pkg/base/src_10.1-RELEASE-p9/sys/GENERIC amd64 root@leopard:/home/tom-> zpool list NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT backup 5.31T 3.61T 1.70T 22% - 68% 1.00x ONLINE - jumpdrive_f 2.72T 2.04T 693G 30% - 75% 1.00x ONLINE - root@leopard:/home/tom-> zpool status backup pool: backup state: ONLINE scan: scrub repaired 0 in 13h15m with 0 errors on Wed May 13 16:17:29 2015 config: NAME STATE READ WRITE CKSUM backup ONLINE 0 0 0 da0 ONLINE 0 0 0 errors: No known data errors root@leopard:/home/tom-> zpool get all backup NAME PROPERTY VALUE SOURCE backup size 5.31T - backup capacity 68% - backup altroot - default backup health ONLINE - backup guid 12638712474922952450 default backup version - default backup bootfs - default backup delegation on default backup autoreplace off default backup cachefile - default backup failmode wait default backup listsnapshots off default backup autoexpand off default backup dedupditto 0 default backup dedupratio 1.00x - backup free 1.70T - backup allocated 3.61T - backup readonly off - backup comment - default backup expandsize 0 - backup freeing 0 default backup fragmentation 22% - backup leaked 0 default backup feature@async_destroy enabled local backup feature@empty_bpobj active local backup feature@lz4_compress active local backup feature@multi_vdev_crash_dump enabled local backup feature@spacemap_histogram active local backup feature@enabled_txg active local backup feature@hole_birth active local backup feature@extensible_dataset enabled local backup feature@embedded_data active local backup feature@bookmarks enabled local backup feature@filesystem_limits enabled local From owner-freebsd-fs@FreeBSD.ORG Fri May 22 19:36:39 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 349A6F1B for ; Fri, 22 May 2015 19:36:39 +0000 (UTC) Received: from smtp102-5.vfemail.net (eightfive.vfemail.net [96.30.253.185]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EC84F123D for ; Fri, 22 May 2015 19:36:38 +0000 (UTC) Received: (qmail 32470 invoked by uid 89); 22 May 2015 19:29:55 -0000 Received: by simscan 1.4.0 ppid: 32452, pid: 32466, t: 0.3254s scanners:none Received: from unknown (HELO d3d3MTExQDE0MzIzMjI5OTU=) (cmlja0BoYXZva21vbi5jb21AMTQzMjMyMjk5NQ==@MTcyLjE2LjEwMC45M0AxNDMyMzIyOTk1) by 172.16.100.62 with ESMTPA; 22 May 2015 19:29:55 -0000 Date: Fri, 22 May 2015 14:29:53 -0500 Message-ID: <20150522142953.Horde.0xvlvqTOT_zCOpzSulV9wA1@www.vfemail.net> From: Rick Romero To: freebsd-fs@freebsd.org Subject: Re: zpool on Dell MD3000 causes frequent hangs In-Reply-To: User-Agent: Internet Messaging Program (IMP) H5 (6.2.2) X-VFEmail-Originating-IP: MTIuMzEuMTAwLjE0Ng== X-VFEmail-AntiSpam: Notify admin@vfemail.net of any spam, and include VFEmail headers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes Content-Transfer-Encoding: 8bit Content-Disposition: inline Content-Description: Plaintext Message X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2015 19:36:39 -0000 Quoting Thomas Johnson : > Hello, > > I am trying to track down an ongoing issue that I've been having, and > looking for any suggestions on a possible cause, or suggestions on how I > might troubleshoot further. > > The issue seems to be related to a Dell MD3000 storage array, which > contains a zpool. It seems that the host attached to the array will > occasionally hang, usually during periods of high disk activity > (annoyingly, usually about 0300). Hi Thomas, You might want to try downgrading the firmware.  The latest firmware on those LSI controllers really sucks. Rick From owner-freebsd-fs@FreeBSD.ORG Fri May 22 20:16:43 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 33462A03 for ; Fri, 22 May 2015 20:16:43 +0000 (UTC) Received: from mail-lb0-x22e.google.com (mail-lb0-x22e.google.com [IPv6:2a00:1450:4010:c04::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ACBF31722 for ; Fri, 22 May 2015 20:16:42 +0000 (UTC) Received: by lbcmx3 with SMTP id mx3so20150540lbc.1 for ; Fri, 22 May 2015 13:16:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=APHideq0Dclyj7ENVl7ITUrI6hnWUe04aspqBkEVYsQ=; b=tK44PwkO4o1+sx8yRHPEB9/AJ8lBzyHUjrBe8s1/GrXNcfxTn9hVZjJy2t+CnGc6ld iCqRwVV1um/BDWgtwgVZ8th41hNBT6HsXzzZEVMlyCn6WwqYZJl1sjeX6zqmAr8BFzkx hg4d62VBNW9ZwK4HSeyrQv1FPVLeQDfVKZp5DZjCUswvpWWmrZYQeoImdPWvM9YCJWBO LdbkPhGIYZwz7APfA0wR/a64fp3AkxFkxJMcJLibO5AkjTZ0oQV0G95RaHE8h2Ddg4tG T6VHqQG1c3KSg+ytEq3tGZC68ykJS04gYvyAAuaQ81w2QDCJDafO4Xz0a0vpOGD4pL6g M1xQ== MIME-Version: 1.0 X-Received: by 10.112.163.168 with SMTP id yj8mr7815973lbb.36.1432325800812; Fri, 22 May 2015 13:16:40 -0700 (PDT) Received: by 10.25.210.79 with HTTP; Fri, 22 May 2015 13:16:40 -0700 (PDT) In-Reply-To: <20150522142953.Horde.0xvlvqTOT_zCOpzSulV9wA1@www.vfemail.net> References: <20150522142953.Horde.0xvlvqTOT_zCOpzSulV9wA1@www.vfemail.net> Date: Fri, 22 May 2015 15:16:40 -0500 Message-ID: Subject: Re: zpool on Dell MD3000 causes frequent hangs From: Thomas Johnson To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2015 20:16:43 -0000 Thanks for the suggestion. Do you have any knowledge of known-good versions? The HBAs are SAS 5/e cards, that identify as SAS1068. I believe the current firmware version is 0.10.51.00-IT. The reported firmware version of the MD3000 controllers is 07.35.39.64 mpt0: port 0xec00-0xecff mem 0xdf2ec000-0xdf2effff,0xdf2f0000-0xdf2fffff irq 40 at device 8.0 on pci6 mpt0: MPI Version=1.5.13.0 Thanks! On Fri, May 22, 2015 at 2:29 PM, Rick Romero wrote: > Quoting Thomas Johnson : > > Hello, >> >> I am trying to track down an ongoing issue that I've been having, and >> looking for any suggestions on a possible cause, or suggestions on how I >> might troubleshoot further. >> >> The issue seems to be related to a Dell MD3000 storage array, which >> contains a zpool. It seems that the host attached to the array will >> occasionally hang, usually during periods of high disk activity >> (annoyingly, usually about 0300). >> > > Hi Thomas, > > You might want to try downgrading the firmware. The latest firmware on > those LSI controllers really sucks. > > Rick > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Fri May 22 20:42:33 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A43D050D for ; Fri, 22 May 2015 20:42:33 +0000 (UTC) Received: from Exch2-2.slu.se (imap.slu.se [77.235.224.122]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "webmail.slu.se", Issuer "TERENA SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0D1951AA3 for ; Fri, 22 May 2015 20:42:32 +0000 (UTC) Received: from exch2-4.slu.se (77.235.224.124) by Exch2-2.slu.se (77.235.224.122) with Microsoft SMTP Server (TLS) id 15.0.1076.9; Fri, 22 May 2015 22:27:12 +0200 Received: from exch2-4.slu.se ([fe80::3117:818f:aa48:9d9b]) by exch2-4.slu.se ([fe80::3117:818f:aa48:9d9b%22]) with mapi id 15.00.1076.000; Fri, 22 May 2015 22:27:12 +0200 From: =?utf-8?B?S2FybGkgU2rDtmJlcmc=?= To: Thomas Johnson CC: "freebsd-fs@freebsd.org" Subject: Re: zpool on Dell MD3000 causes frequent hangs Thread-Topic: zpool on Dell MD3000 causes frequent hangs Thread-Index: AQHQlM2vNisyLu49rkuJyBP333N0lQ== Date: Fri, 22 May 2015 20:27:12 +0000 Message-ID: Accept-Language: sv-SE, en-US Content-Language: sv-SE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2015 20:42:33 -0000 DQpEZW4gMjIgbWFqIDIwMTUgOToxMCBlbSBza3JldiBUaG9tYXMgSm9obnNvbiA8dG9tbXlqMjdA Z21haWwuY29tPjoNCj4NCj4gSGVsbG8sDQo+DQo+IEkgYW0gdHJ5aW5nIHRvIHRyYWNrIGRvd24g YW4gb25nb2luZyBpc3N1ZSB0aGF0IEkndmUgYmVlbiBoYXZpbmcsIGFuZA0KPiBsb29raW5nIGZv ciBhbnkgc3VnZ2VzdGlvbnMgb24gYSBwb3NzaWJsZSBjYXVzZSwgb3Igc3VnZ2VzdGlvbnMgb24g aG93IEkNCj4gbWlnaHQgdHJvdWJsZXNob290IGZ1cnRoZXIuDQo+DQo+IFRoZSBpc3N1ZSBzZWVt cyB0byBiZSByZWxhdGVkIHRvIGEgRGVsbCBNRDMwMDAgc3RvcmFnZSBhcnJheSwgd2hpY2gNCj4g Y29udGFpbnMgYSB6cG9vbC4gSXQgc2VlbXMgdGhhdCB0aGUgaG9zdCBhdHRhY2hlZCB0byB0aGUg YXJyYXkgd2lsbA0KPiBvY2Nhc2lvbmFsbHkgaGFuZywgdXN1YWxseSBkdXJpbmcgcGVyaW9kcyBv ZiBoaWdoIGRpc2sgYWN0aXZpdHkNCj4gKGFubm95aW5nbHksIHVzdWFsbHkgYWJvdXQgMDMwMCku DQo+DQo+IFdoZW4gdGhlIHN5c3RlbSBoYW5ncywgSSBjYW4gcGluZyB0aGUgaG9zdCwgYW5kIHN3 aXRjaCBiZXR3ZWVuIHZpcnR1YWwNCj4gY29uc29sZXMgKGJ1dCBub3QgaW50ZXJhY3Qgd2l0aCB0 aGVtKS4gVGhlIHN5c3RlbSBpcyBvdGhlcndpc2UNCj4gdW5yZXNwb25zaXZlOyB3aXRoIG5vIGVy cm9ycyByZXBvcnRlZCBvbiB0aGUgY29uc29sZSBvciBsb2dzLiBUaGUgb25seQ0KPiByZW1lZHkg SSBoYXZlIGZvdW5kIGlzIHRvIGhhcmQtcmVzZXQgdGhlIGhvc3QuDQo+DQo+IEkgYmVsaWV2ZSB0 aGlzIGlzc3VlIGlzIHRpZWQgdG8gdGhlIE1EMzAwMC4gSSBoYXZlIHRyaWVkIHN3YXBwaW5nIG91 dCBTQVMNCj4gY2FibGVzLCBIQkFzLCB0aGUgY29udHJvbGxlciBvbiB0aGUgTUQzMDAwLCBhbmQg dGhlIGhvc3QgaXRzZWxmLiBJIGhhdmUNCj4gdXBkYXRlZCBhbGwgdGhlIGZpcm13YXJlIEkgY2Fu IGZpbmQuIEJlZm9yZSBJIHVwZ3JhZGVkIHRoZSBob3N0IE9TIHRvDQo+IEZyZWVCU0QgMTAuMSAo ZnJvbSAxMC4wKSBsYXN0IG1vbnRoLCBJIGV4cGVyaWVuY2VkIGhhbmdzIGFib3V0IG9uY2UgYQ0K PiBtb250aC4gU2luY2UgdGhlIHVwZ3JhZGUsIEkgaGF2ZSBzZWVuIHNldmVyYWwgZXZlbnRzIHBl ciB3ZWVrLg0KDQpNeSBiZXQgaXMgb24gdGhpczoNCmh0dHBzOi8vYnVncy5mcmVlYnNkLm9yZy9i dWd6aWxsYS9zaG93X2J1Zy5jZ2k/aWQ9MTk3MTY0DQoNCi9LDQoNCj4NCj4gSW4gYWRkaXRpb24g dG8gdGhlIE1EMzAwMCwgSSBoYXZlIGEgc2V0IG9mIFVTQiBkcml2ZXMgdGhhdCBhcmUgdXNlZCBp biBhDQo+IHJvdGF0aW9uIGFzIG9mZnNpdGUgYmFja3VwcyBmb3IgdGhlIHpwb29sLiBJIGhhdmUg c2VlbiBhIG51bWJlciBvZiBoYW5nDQo+IGV2ZW50cyBkdXJpbmcgemZzIHNlbmQvcmVjZWl2ZSB0 cmFuc2ZlcnMgdG8gdGhlIFVTQiBkaXNrLg0KPg0KPiBBZnRlciB0aGUgbW9zdCByZWNlbnQgaGFu ZywgSSByZW1vdmVkIHR3byBbY29uc3VtZXJdIFNTRHMgZnJvbSB0aGUgcG9vbA0KPiB0aGF0IHdl cmUgYmVpbmcgdXNlZCBhcyBjYWNoZSBkZXZpY2VzLiBJdCBpcyB0b28gZWFybHkgdG8gdGVsbCBp ZiB0aGlzDQo+IGNoYW5nZSBoYWQgYW55IGltcGFjdC4NCj4NCj4gSGVyZSBpcyBzb21lIG9mIHRo ZSBwZXJ0aW5lbnQgb3V0cHV0IGZyb20gdGhlIGhvc3QuIEkgY2FuIHByb3ZpZGUgYW55IG90aGVy DQo+IGluZm9ybWF0aW9uIHRoYXQgd291bGQgYmUgaGVscGZ1bC4NCj4NCj4gcm9vdEBsZW9wYXJk Oi9ob21lL3RvbS0+IHVuYW1lIC1hDQo+IEZyZWVCU0QgbGVvcGFyZCAxMC4xLVJFTEVBU0UtcDkg RnJlZUJTRCAxMC4xLVJFTEVBU0UtcDkgIzAgcjI4MTIzMjogVHVlDQo+IEFwciAgNyAxNzozODow NCBDRFQgMjAxNQ0KPiByb290QGNoZXNoaXJlLWI6L3BrZy9iYXNlL29ial8xMC4xLVJFTEVBU0Ut cDkvcGtnL2Jhc2Uvc3JjXzEwLjEtUkVMRUFTRS1wOS9zeXMvR0VORVJJQw0KPiBhbWQ2NA0KPiBy b290QGxlb3BhcmQ6L2hvbWUvdG9tLT4genBvb2wgbGlzdA0KPiBOQU1FICAgICAgICAgIFNJWkUg IEFMTE9DICAgRlJFRSAgIEZSQUcgIEVYUEFORFNaICAgIENBUCAgREVEVVAgIEhFQUxUSA0KPiBB TFRST09UDQo+IGJhY2t1cCAgICAgICA1LjMxVCAgMy42MVQgIDEuNzBUICAgIDIyJSAgICAgICAg IC0gICAgNjglICAxLjAweCAgT05MSU5FICAtDQo+IGp1bXBkcml2ZV9mICAyLjcyVCAgMi4wNFQg ICA2OTNHICAgIDMwJSAgICAgICAgIC0gICAgNzUlICAxLjAweCAgT05MSU5FICAtDQo+IHJvb3RA bGVvcGFyZDovaG9tZS90b20tPiB6cG9vbCBzdGF0dXMgYmFja3VwDQo+ICAgcG9vbDogYmFja3Vw DQo+ICBzdGF0ZTogT05MSU5FDQo+ICAgc2Nhbjogc2NydWIgcmVwYWlyZWQgMCBpbiAxM2gxNW0g d2l0aCAwIGVycm9ycyBvbiBXZWQgTWF5IDEzIDE2OjE3OjI5IDIwMTUNCj4gY29uZmlnOg0KPg0K PiAgICAgTkFNRSAgICAgICAgU1RBVEUgICAgIFJFQUQgV1JJVEUgQ0tTVU0NCj4gICAgIGJhY2t1 cCAgICAgIE9OTElORSAgICAgICAwICAgICAwICAgICAwDQo+ICAgICAgIGRhMCAgICAgICBPTkxJ TkUgICAgICAgMCAgICAgMCAgICAgMA0KPg0KPiBlcnJvcnM6IE5vIGtub3duIGRhdGEgZXJyb3Jz DQo+IHJvb3RAbGVvcGFyZDovaG9tZS90b20tPiB6cG9vbCBnZXQgYWxsIGJhY2t1cA0KPiBOQU1F ICAgIFBST1BFUlRZICAgICAgICAgICAgICAgICAgICAgICBWQUxVRSAgICAgICAgICAgICAgICAg ICAgICAgICAgU09VUkNFDQo+IGJhY2t1cCAgc2l6ZSAgICAgICAgICAgICAgICAgICAgICAgICAg IDUuMzFUICAgICAgICAgICAgICAgICAgICAgICAgICAtDQo+IGJhY2t1cCAgY2FwYWNpdHkgICAg ICAgICAgICAgICAgICAgICAgIDY4JSAgICAgICAgICAgICAgICAgICAgICAgICAgICAtDQo+IGJh Y2t1cCAgYWx0cm9vdCAgICAgICAgICAgICAgICAgICAgICAgIC0NCj4gZGVmYXVsdA0KPiBiYWNr dXAgIGhlYWx0aCAgICAgICAgICAgICAgICAgICAgICAgICBPTkxJTkUgICAgICAgICAgICAgICAg ICAgICAgICAgLQ0KPiBiYWNrdXAgIGd1aWQgICAgICAgICAgICAgICAgICAgICAgICAgICAxMjYz ODcxMjQ3NDkyMjk1MjQ1MA0KPiBkZWZhdWx0DQo+IGJhY2t1cCAgdmVyc2lvbiAgICAgICAgICAg ICAgICAgICAgICAgIC0NCj4gZGVmYXVsdA0KPiBiYWNrdXAgIGJvb3RmcyAgICAgICAgICAgICAg ICAgICAgICAgICAtDQo+IGRlZmF1bHQNCj4gYmFja3VwICBkZWxlZ2F0aW9uICAgICAgICAgICAg ICAgICAgICAgb24NCj4gZGVmYXVsdA0KPiBiYWNrdXAgIGF1dG9yZXBsYWNlICAgICAgICAgICAg ICAgICAgICBvZmYNCj4gZGVmYXVsdA0KPiBiYWNrdXAgIGNhY2hlZmlsZSAgICAgICAgICAgICAg ICAgICAgICAtDQo+IGRlZmF1bHQNCj4gYmFja3VwICBmYWlsbW9kZSAgICAgICAgICAgICAgICAg ICAgICAgd2FpdA0KPiBkZWZhdWx0DQo+IGJhY2t1cCAgbGlzdHNuYXBzaG90cyAgICAgICAgICAg ICAgICAgIG9mZg0KPiBkZWZhdWx0DQo+IGJhY2t1cCAgYXV0b2V4cGFuZCAgICAgICAgICAgICAg ICAgICAgIG9mZg0KPiBkZWZhdWx0DQo+IGJhY2t1cCAgZGVkdXBkaXR0byAgICAgICAgICAgICAg ICAgICAgIDANCj4gZGVmYXVsdA0KPiBiYWNrdXAgIGRlZHVwcmF0aW8gICAgICAgICAgICAgICAg ICAgICAxLjAweCAgICAgICAgICAgICAgICAgICAgICAgICAgLQ0KPiBiYWNrdXAgIGZyZWUgICAg ICAgICAgICAgICAgICAgICAgICAgICAxLjcwVCAgICAgICAgICAgICAgICAgICAgICAgICAgLQ0K PiBiYWNrdXAgIGFsbG9jYXRlZCAgICAgICAgICAgICAgICAgICAgICAzLjYxVCAgICAgICAgICAg ICAgICAgICAgICAgICAgLQ0KPiBiYWNrdXAgIHJlYWRvbmx5ICAgICAgICAgICAgICAgICAgICAg ICBvZmYgICAgICAgICAgICAgICAgICAgICAgICAgICAgLQ0KPiBiYWNrdXAgIGNvbW1lbnQgICAg ICAgICAgICAgICAgICAgICAgICAtDQo+IGRlZmF1bHQNCj4gYmFja3VwICBleHBhbmRzaXplICAg ICAgICAgICAgICAgICAgICAgMCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC0NCj4gYmFj a3VwICBmcmVlaW5nICAgICAgICAgICAgICAgICAgICAgICAgMA0KPiBkZWZhdWx0DQo+IGJhY2t1 cCAgZnJhZ21lbnRhdGlvbiAgICAgICAgICAgICAgICAgIDIyJSAgICAgICAgICAgICAgICAgICAg ICAgICAgICAtDQo+IGJhY2t1cCAgbGVha2VkICAgICAgICAgICAgICAgICAgICAgICAgIDANCj4g ZGVmYXVsdA0KPiBiYWNrdXAgIGZlYXR1cmVAYXN5bmNfZGVzdHJveSAgICAgICAgICBlbmFibGVk ICAgICAgICAgICAgICAgICAgICAgICAgbG9jYWwNCj4gYmFja3VwICBmZWF0dXJlQGVtcHR5X2Jw b2JqICAgICAgICAgICAgYWN0aXZlICAgICAgICAgICAgICAgICAgICAgICAgIGxvY2FsDQo+IGJh Y2t1cCAgZmVhdHVyZUBsejRfY29tcHJlc3MgICAgICAgICAgIGFjdGl2ZSAgICAgICAgICAgICAg ICAgICAgICAgICBsb2NhbA0KPiBiYWNrdXAgIGZlYXR1cmVAbXVsdGlfdmRldl9jcmFzaF9kdW1w ICBlbmFibGVkICAgICAgICAgICAgICAgICAgICAgICAgbG9jYWwNCj4gYmFja3VwICBmZWF0dXJl QHNwYWNlbWFwX2hpc3RvZ3JhbSAgICAgYWN0aXZlICAgICAgICAgICAgICAgICAgICAgICAgIGxv Y2FsDQo+IGJhY2t1cCAgZmVhdHVyZUBlbmFibGVkX3R4ZyAgICAgICAgICAgIGFjdGl2ZSAgICAg ICAgICAgICAgICAgICAgICAgICBsb2NhbA0KPiBiYWNrdXAgIGZlYXR1cmVAaG9sZV9iaXJ0aCAg ICAgICAgICAgICBhY3RpdmUgICAgICAgICAgICAgICAgICAgICAgICAgbG9jYWwNCj4gYmFja3Vw ICBmZWF0dXJlQGV4dGVuc2libGVfZGF0YXNldCAgICAgZW5hYmxlZCAgICAgICAgICAgICAgICAg ICAgICAgIGxvY2FsDQo+IGJhY2t1cCAgZmVhdHVyZUBlbWJlZGRlZF9kYXRhICAgICAgICAgIGFj dGl2ZSAgICAgICAgICAgICAgICAgICAgICAgICBsb2NhbA0KPiBiYWNrdXAgIGZlYXR1cmVAYm9v a21hcmtzICAgICAgICAgICAgICBlbmFibGVkICAgICAgICAgICAgICAgICAgICAgICAgbG9jYWwN Cj4gYmFja3VwICBmZWF0dXJlQGZpbGVzeXN0ZW1fbGltaXRzICAgICAgZW5hYmxlZCAgICAgICAg ICAgICAgICAgICAgICAgIGxvY2FsDQo+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fDQo+IGZyZWVic2QtZnNAZnJlZWJzZC5vcmcgbWFpbGluZyBsaXN0DQo+ IGh0dHA6Ly9saXN0cy5mcmVlYnNkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2ZyZWVic2QtZnMNCj4g VG8gdW5zdWJzY3JpYmUsIHNlbmQgYW55IG1haWwgdG8gImZyZWVic2QtZnMtdW5zdWJzY3JpYmVA ZnJlZWJzZC5vcmciDQo= From owner-freebsd-fs@FreeBSD.ORG Fri May 22 21:05:06 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9B2FDB46 for ; Fri, 22 May 2015 21:05:06 +0000 (UTC) Received: from mail-lb0-x22a.google.com (mail-lb0-x22a.google.com [IPv6:2a00:1450:4010:c04::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 055B81D12 for ; Fri, 22 May 2015 21:05:06 +0000 (UTC) Received: by lbbuc2 with SMTP id uc2so20775402lbb.2 for ; Fri, 22 May 2015 14:05:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=JCBUtPmcip3LPts2GJRQxHBwyulJywod3X8ilJLKOIM=; b=ysYZNN/v/o7UP8NKs4VVelgqkpaQTxD3wv91ERDl4N9Ejzo5JKgsggLPtZdRcTKbao Eb2BYZOE9ENWXB71Be5V4t3nMwhbBgRpCzTWNq2QkvALx6hnPsarZERrRzuHNiH0Q/eW 06vstVePQXIWLxs8tqcyeRMfQrMAFZzZA/VzlDGMgIJWjngQKZXykF7VqClzdpqDs4o1 0n6VaywwFQERBXmaoTmtN/JIcsXyqZITpzqV7TOmSndAdvab5d0y/jOop1qoSlaMjKiM XvE+zYIJDSzUhZqCdlFn9ImNwQ+HrpbXqRM1a8ylKsFno3VqypVshx8ngMo8vd/lpT87 Wgcw== MIME-Version: 1.0 X-Received: by 10.112.139.1 with SMTP id qu1mr7955041lbb.8.1432328703631; Fri, 22 May 2015 14:05:03 -0700 (PDT) Received: by 10.25.210.79 with HTTP; Fri, 22 May 2015 14:05:03 -0700 (PDT) In-Reply-To: References: Date: Fri, 22 May 2015 16:05:03 -0500 Message-ID: Subject: Re: zpool on Dell MD3000 causes frequent hangs From: Thomas Johnson To: =?UTF-8?Q?Karli_Sj=C3=B6berg?= Cc: "freebsd-fs@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2015 21:05:06 -0000 That looks like a match. I'll the "abuse" knob up to 11, and see if I can break it. Thanks! On Fri, May 22, 2015 at 3:27 PM, Karli Sj=C3=B6berg = wrote: > > Den 22 maj 2015 9:10 em skrev Thomas Johnson : > > > > Hello, > > > > I am trying to track down an ongoing issue that I've been having, and > > looking for any suggestions on a possible cause, or suggestions on how = I > > might troubleshoot further. > > > > The issue seems to be related to a Dell MD3000 storage array, which > > contains a zpool. It seems that the host attached to the array will > > occasionally hang, usually during periods of high disk activity > > (annoyingly, usually about 0300). > > > > When the system hangs, I can ping the host, and switch between virtual > > consoles (but not interact with them). The system is otherwise > > unresponsive; with no errors reported on the console or logs. The only > > remedy I have found is to hard-reset the host. > > > > I believe this issue is tied to the MD3000. I have tried swapping out S= AS > > cables, HBAs, the controller on the MD3000, and the host itself. I have > > updated all the firmware I can find. Before I upgraded the host OS to > > FreeBSD 10.1 (from 10.0) last month, I experienced hangs about once a > > month. Since the upgrade, I have seen several events per week. > > My bet is on this: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D197164 > > /K > > > > > In addition to the MD3000, I have a set of USB drives that are used in = a > > rotation as offsite backups for the zpool. I have seen a number of hang > > events during zfs send/receive transfers to the USB disk. > > > > After the most recent hang, I removed two [consumer] SSDs from the pool > > that were being used as cache devices. It is too early to tell if this > > change had any impact. > > > > Here is some of the pertinent output from the host. I can provide any > other > > information that would be helpful. > > > > root@leopard:/home/tom-> uname -a > > FreeBSD leopard 10.1-RELEASE-p9 FreeBSD 10.1-RELEASE-p9 #0 r281232: Tue > > Apr 7 17:38:04 CDT 2015 > > root@cheshire-b > :/pkg/base/obj_10.1-RELEASE-p9/pkg/base/src_10.1-RELEASE-p9/sys/GENERIC > > amd64 > > root@leopard:/home/tom-> zpool list > > NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH > > ALTROOT > > backup 5.31T 3.61T 1.70T 22% - 68% 1.00x ONLINE > - > > jumpdrive_f 2.72T 2.04T 693G 30% - 75% 1.00x ONLINE > - > > root@leopard:/home/tom-> zpool status backup > > pool: backup > > state: ONLINE > > scan: scrub repaired 0 in 13h15m with 0 errors on Wed May 13 16:17:29 > 2015 > > config: > > > > NAME STATE READ WRITE CKSUM > > backup ONLINE 0 0 0 > > da0 ONLINE 0 0 0 > > > > errors: No known data errors > > root@leopard:/home/tom-> zpool get all backup > > NAME PROPERTY VALUE > SOURCE > > backup size 5.31T - > > backup capacity 68% - > > backup altroot - > > default > > backup health ONLINE - > > backup guid 12638712474922952450 > > default > > backup version - > > default > > backup bootfs - > > default > > backup delegation on > > default > > backup autoreplace off > > default > > backup cachefile - > > default > > backup failmode wait > > default > > backup listsnapshots off > > default > > backup autoexpand off > > default > > backup dedupditto 0 > > default > > backup dedupratio 1.00x - > > backup free 1.70T - > > backup allocated 3.61T - > > backup readonly off - > > backup comment - > > default > > backup expandsize 0 - > > backup freeing 0 > > default > > backup fragmentation 22% - > > backup leaked 0 > > default > > backup feature@async_destroy enabled > local > > backup feature@empty_bpobj active > local > > backup feature@lz4_compress active > local > > backup feature@multi_vdev_crash_dump enabled > local > > backup feature@spacemap_histogram active > local > > backup feature@enabled_txg active > local > > backup feature@hole_birth active > local > > backup feature@extensible_dataset enabled > local > > backup feature@embedded_data active > local > > backup feature@bookmarks enabled > local > > backup feature@filesystem_limits enabled > local > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > From owner-freebsd-fs@FreeBSD.ORG Sat May 23 14:04:30 2015 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 705CF1C8 for ; Sat, 23 May 2015 14:04:30 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5B43B12EA for ; Sat, 23 May 2015 14:04:30 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t4NE4Urc015332 for ; Sat, 23 May 2015 14:04:30 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 200383] newnfs server pid884@svl-junos-j010:/jails/r71j8i386/volume: not responding Date: Sat, 23 May 2015 14:04:30 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 9.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 May 2015 14:04:30 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200383 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Sat May 23 19:24:00 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 65055EE6; Sat, 23 May 2015 19:24:00 +0000 (UTC) Received: from mail-wi0-x234.google.com (mail-wi0-x234.google.com [IPv6:2a00:1450:400c:c05::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F1B0B139E; Sat, 23 May 2015 19:23:59 +0000 (UTC) Received: by wicmx19 with SMTP id mx19so16516866wic.0; Sat, 23 May 2015 12:23:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=cAWN18orTAh33CPffJehDjpRVDARtpGTw11vQyxWrm4=; b=JXBjW5oDBB063k+nzC2GKyCou+Vt6aEcC1DRI1FDH18ScJw7ea0FRylvgENlDOM1km b9pTff/qKS7VEiEXThaYWXJfte19zTrYv6bUSO2DzFKnZQ7knpTSl4ij82s9vHcIIl2x R5V7rDvJIe6JGd4za+pNWoMbluqvNKisM2KC7Ry48eiy3vcrto/mzSzPD/MSGl+XQJ8A FAWVVbsUzcBdcG8ixsy2ngD0SeyM68HiszEAc5i59x7LP8loVLlWofHbEaBrW3+DNgb+ 4iWAUO6l/rtQJ1KbLrRjrttrdtHwlt9wyZkPBxVvzrtv+kTR4H9wMn71JlLfhL2hhUWk lWOw== MIME-Version: 1.0 X-Received: by 10.194.60.43 with SMTP id e11mr26503166wjr.36.1432409037121; Sat, 23 May 2015 12:23:57 -0700 (PDT) Received: by 10.28.30.22 with HTTP; Sat, 23 May 2015 12:23:57 -0700 (PDT) In-Reply-To: References: Date: Sat, 23 May 2015 22:23:57 +0300 Message-ID: Subject: Re: L2ARC degraded. Checksum errors, I/O errors From: George Kontostanos To: "freebsd-fs@freebsd.org" , freebsd-stable Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 May 2015 19:24:00 -0000 Once again, do we know if this https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198242 has been committed yet? -- George Kontostanos ---