From owner-freebsd-fs@FreeBSD.ORG Sun Nov 25 10:19:23 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7CB25257 for ; Sun, 25 Nov 2012 10:19:23 +0000 (UTC) (envelope-from nvass@gmx.com) Received: from mailout-eu.gmx.com (mailout-eu.gmx.com [213.165.64.44]) by mx1.freebsd.org (Postfix) with SMTP id C6A328FC12 for ; Sun, 25 Nov 2012 10:19:22 +0000 (UTC) Received: (qmail invoked by alias); 25 Nov 2012 10:19:15 -0000 Received: from 178.128.120.133.dsl.dyn.forthnet.gr (EHLO [192.168.1.77]) [178.128.120.133] by mail.gmx.com (mp-eu006) with SMTP; 25 Nov 2012 11:19:15 +0100 X-Authenticated: #46156728 X-Provags-ID: V01U2FsdGVkX18h/XJCHv8hYVyConDKM4XOaKAJdKRpBnbSpyeuWF rEU0ZgzlEZuYLH Message-ID: <50B1F09D.60803@gmx.com> Date: Sun, 25 Nov 2012 12:19:09 +0200 From: Nikos Vassiliadis User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20120604 Thunderbird/13.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: gjournal and UFS snapshots Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Y-GMX-Trusted: 0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Nov 2012 10:19:23 -0000 Hi, I have a fileserver which uses geom journaling. I'd like to add some kind of snapshot solution. I'd like to take a weekly snapshot and have them RO mounted so users can access them. The number of snapshots I am after is two and the speed of snapshoting is not very important given that it will be taken at night. FS activity never ceases, the system is also a low traffic mail server. Migrating to ZFS is not an option, the system is an old x86 with 1GB of RAM(It is an option but you know... that's not up to me). I am bit hesitant to do it because I have seen reports about panics regarding gjounal and ufs snapshots. I currently use the 8 branch. So, is someone using UFS with geom journaling and a few FS snapshots? My numbers are like this: > %df -ih /storage/ > Filesystem Size Used Avail Capacity iused ifree %iused Mounted on > /dev/mirror/s2a.journal 1.8T 712G 974G 42% 688k 19M 3% /storage Thanks, Nikos From owner-freebsd-fs@FreeBSD.ORG Sun Nov 25 18:54:06 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1A1B480F for ; Sun, 25 Nov 2012 18:54:06 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id A5E508FC13 for ; Sun, 25 Nov 2012 18:54:04 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id CCC82EE230A; Sun, 25 Nov 2012 19:46:56 +0100 (CET) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 11.0207] X-CRM114-CacheID: sfid-20121125_19465_904A01CB X-CRM114-Status: Good ( pR: 11.0207 ) X-DSPAM-Result: Whitelisted X-DSPAM-Processed: Sun Nov 25 19:46:56 2012 X-DSPAM-Confidence: 0.7607 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 50b267a02501889419329 X-DSPAM-Factors: 27, From*Attila Nagy , 0.00010, Subject*to+use, 0.00552, ZFS, 0.00613, Subject*ZFS, 0.00690, do+something, 0.00690, it's+not, 0.00719, Is+it, 0.00870, I'm+wondering, 0.00917, Date*Sun+25, 0.99000, buf, 0.01000, Received*Sun+25, 0.99000, it+fails, 0.01000, User-Agent*i686, 0.01156, User-Agent*Mozilla/5.0+(X11, 0.01156, fails, 0.01290, User-Agent*Linux+i686, 0.01290, will+it, 0.01370, I+guess, 0.01370, guess, 0.01520, Received*fs, 0.01562, User-Agent*i686+en, 0.01681, Received*fs+freebsd.org>, 0.01818, User-Agent*Linux, 0.01818, From*Attila, 0.01818, Subject*use, 0.01818, Received*; Sun, 25 Nov 2012 19:46:55 +0100 (CET) Message-ID: <50B2679E.3030600@fsn.hu> Date: Sun, 25 Nov 2012 19:46:54 +0100 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Is it possible to use ZFS transactions from user space? Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Nov 2012 18:54:06 -0000 Hi, I'm wondering, will it be possible to use ZFS transactions from user space? For example, I would like to do something similar: tx=zfs_tx_init(); write(fd,buf,strlen(buf)); unlink(file); zfs_tx_commit(tx); The write and unlink should both happen or fail together (it's fine with me if it fails only at tx_commit). I can see a lot of problems here, so I guess it's not an easy topic. Is it doable at all? Anyone working on it? From owner-freebsd-fs@FreeBSD.ORG Sun Nov 25 19:56:19 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 56F4F401 for ; Sun, 25 Nov 2012 19:56:19 +0000 (UTC) (envelope-from trent@snakebite.org) Received: from exchange.liveoffice.com (exchla3.liveoffice.com [64.70.67.188]) by mx1.freebsd.org (Postfix) with ESMTP id 350198FC12 for ; Sun, 25 Nov 2012 19:56:18 +0000 (UTC) Received: from exhub13.exchhosting.com (192.168.11.122) by exhub06.exchhosting.com (192.168.11.102) with Microsoft SMTP Server (TLS) id 8.3.213.0; Sun, 25 Nov 2012 11:55:11 -0800 Received: from localhost (35.8.247.10) by exchange.liveoffice.com (192.168.11.122) with Microsoft SMTP Server id 8.3.213.0; Sun, 25 Nov 2012 11:55:10 -0800 Date: Sun, 25 Nov 2012 14:55:09 -0500 From: Trent Nelson To: Subject: Solved persistent stalls in 'zio->io_cv)' Message-ID: <20121125195509.GB52355@snakebite.org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Nov 2012 19:56:19 -0000 [ For the archives... ] Over the course of about a month I noticed progressively worse performance on my main FreeBSD ZFS box. It's a simple box; 1U, root-on-zfs, 2TB WD EARS zpool mirror. Almost everything started stalling towards the end. A simple ls would take 15-20 seconds to return, with ^T always indicating the wait state was 'zio->io_cv)'. smartctl/smartmond had detected two offline uncorrectable sectors on one of the drives, but subsequent full scans revealed no problems. I vaguely recall reading an old mailing list post that mentioned zio->io_cv was essentially the zfs equivalent to biord, i.e. "the system is waiting for the disk". Coupled with knowledge that a zpool mirror will equally split disk load between physical devices regardless of underlying device performance (i.e. it doesn't grok "disk 1 takes 600% longer to return my reads() than disk 0"), so I decided to run `diskinfo -tv` on each disk, not being able to think of anything else to run that would be useful, diagnostically. The output was pretty telling (pasted in full at the end of this e-mail): ada1: Seek Times: Full stroke: 250 iter in 38.386551 sec = 153.546 msec Half stroke: 250 iter in 32.733690 sec = 130.935 msec ... Transfer Rates: outside: 102400 kbytes in 10.554800 sec = 9702 kbytes/sec middle: 102400 kbytes in 2.299292 sec = 44535 kbytes/sec ada0: Seek Times: Full stroke: 250 iter in 13.585292 sec = 54.341 msec Half stroke: 250 iter in 6.156059 sec = 24.624 msec ... Transfer Rates: outside: 102400 kbytes in 2.234584 sec = 45825 kbytes/sec middle: 102400 kbytes in 4.405934 sec = 23241 kbytes/sec So, I concluded (or rather, hoped) that the abysmal performance was due to ada1 responding exponentially slower to requests than its zpool mirror counterpart ada0. I didn't have a replacement available, but performance was so bad I decided to break the zpool mirror and try simply limping along on a single disk until a replacement arrived (drive is still under WD warranty, so sent it in for an RMA replacement). Performance has been fine ever since (been about a week now on the single disk). Just posting for the benefit of the archives. There are lots of hits for 'zio->io_cv)', but nothing related to the particulars in this case. Trent. Full diskinfo output just prior to splitting the mirror: [root@hydrogen/ttypts/3(~)#] diskinfo -tv ada1 ada1 512 # sectorsize 2000398934016 # mediasize in bytes (1.8T) 3907029168 # mediasize in sectors 4096 # stripesize 0 # stripeoffset 3876021 # Cylinders according to firmware. 16 # Heads according to firmware. 63 # Sectors according to firmware. WD-WMAZA5463028 # Disk ident. Seek times: Full stroke: 250 iter in 38.386551 sec = 153.546 msec Half stroke: 250 iter in 32.733690 sec = 130.935 msec Quarter stroke: 500 iter in 30.632886 sec = 61.266 msec Short forward: 400 iter in 53.067122 sec = 132.668 msec Short backward: 400 iter in 6.971919 sec = 17.430 msec Seq outer: 2048 iter in 0.332570 sec = 0.162 msec Seq inner: 2048 iter in 0.921388 sec = 0.450 msec Transfer rates: outside: 102400 kbytes in 10.554800 sec = 9702 kbytes/sec middle: 102400 kbytes in 2.299292 sec = 44535 kbytes/sec inside: 102400 kbytes in 7.301463 sec = 14025 kbytes/sec [root@hydrogen/ttypts/3(~)#] diskinfo -tv ada0 ada0 512 # sectorsize 2000398934016 # mediasize in bytes (1.8T) 3907029168 # mediasize in sectors 4096 # stripesize 0 # stripeoffset 3876021 # Cylinders according to firmware. 16 # Heads according to firmware. 63 # Sectors according to firmware. WD-WMAZA5467587 # Disk ident. Seek times: Full stroke: 250 iter in 13.585292 sec = 54.341 msec Half stroke: 250 iter in 6.156059 sec = 24.624 msec Quarter stroke: 500 iter in 8.901942 sec = 17.804 msec Short forward: 400 iter in 5.421932 sec = 13.555 msec Short backward: 400 iter in 3.237963 sec = 8.095 msec Seq outer: 2048 iter in 0.636089 sec = 0.311 msec Seq inner: 2048 iter in 0.231709 sec = 0.113 msec Transfer rates: outside: 102400 kbytes in 2.234584 sec = 45825 kbytes/sec middle: 102400 kbytes in 4.405934 sec = 23241 kbytes/sec inside: 102400 kbytes in 3.050623 sec = 33567 kbytes/sec Interestingly enough, I just re-ran diskinfo again then on the remaining disk, and results are significantly better than those above: [root@hydrogen/ttypts/0(~)#] diskinfo -tv ada0 ada0 512 # sectorsize 2000398934016 # mediasize in bytes (1.8T) 3907029168 # mediasize in sectors 4096 # stripesize 0 # stripeoffset 3876021 # Cylinders according to firmware. 16 # Heads according to firmware. 63 # Sectors according to firmware. WD-WMAZA5467587 # Disk ident. Seek times: Full stroke: 250 iter in 7.037776 sec = 28.151 msec Half stroke: 250 iter in 5.805128 sec = 23.221 msec Quarter stroke: 500 iter in 7.452755 sec = 14.906 msec Short forward: 400 iter in 3.966540 sec = 9.916 msec Short backward: 400 iter in 1.255183 sec = 3.138 msec Seq outer: 2048 iter in 0.245077 sec = 0.120 msec Seq inner: 2048 iter in 0.212632 sec = 0.104 msec Transfer rates: outside: 102400 kbytes in 1.286970 sec = 79567 kbytes/sec middle: 102400 kbytes in 1.062096 sec = 96413 kbytes/sec inside: 102400 kbytes in 1.996499 sec = 51290 kbytes/sec -- http://www.snakebite.net | @trentnelson From owner-freebsd-fs@FreeBSD.ORG Mon Nov 26 11:00:03 2012 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 13D97266; Mon, 26 Nov 2012 11:00:03 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id D80088FC08; Mon, 26 Nov 2012 11:00:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id qAQB02gf018161; Mon, 26 Nov 2012 11:00:02 GMT (envelope-from avg@freefall.freebsd.org) Received: (from avg@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id qAQB0233018157; Mon, 26 Nov 2012 11:00:02 GMT (envelope-from avg) Date: Mon, 26 Nov 2012 11:00:02 GMT Message-Id: <201211261100.qAQB0233018157@freefall.freebsd.org> To: avg@FreeBSD.org, freebsd-fs@FreeBSD.org, avg@FreeBSD.org From: avg@FreeBSD.org Subject: Re: kern/151111: [zfs] vnodes leakage during zfs unmount X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2012 11:00:03 -0000 Synopsis: [zfs] vnodes leakage during zfs unmount Responsible-Changed-From-To: freebsd-fs->avg Responsible-Changed-By: avg Responsible-Changed-When: Mon Nov 26 10:59:47 UTC 2012 Responsible-Changed-Why: I am interested in this one. http://www.freebsd.org/cgi/query-pr.cgi?pr=151111 From owner-freebsd-fs@FreeBSD.ORG Mon Nov 26 11:06:43 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BE2DC536 for ; Mon, 26 Nov 2012 11:06:43 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id A251A8FC26 for ; Mon, 26 Nov 2012 11:06:43 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id qAQB6hS5019373 for ; Mon, 26 Nov 2012 11:06:43 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id qAQB6h1S019371 for freebsd-fs@FreeBSD.org; Mon, 26 Nov 2012 11:06:43 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 26 Nov 2012 11:06:43 GMT Message-Id: <201211261106.qAQB6h1S019371@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2012 11:06:43 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/173657 fs [nfs] strange UID map with nfsuserd o kern/173363 fs [zfs] [panic] Panic on 'zpool replace' on readonly poo o kern/173234 fs [zfs] [patch] Allow filtering of properties on zfs rec o kern/173136 fs [unionfs] mounting above the NFS read-only share panic o kern/172348 fs [unionfs] umount -f of filesystem in use with readonly o kern/172334 fs [unionfs] unionfs permits recursive union mounts; caus o kern/172259 fs [zfs] [patch] ZFS fails to receive valid snapshots (pa o kern/171626 fs [tmpfs] tmpfs should be noisier when the requested siz o kern/171415 fs [zfs] zfs recv fails with "cannot receive incremental o kern/170945 fs [gpt] disk layout not portable between direct connect o kern/170914 fs [zfs] [patch] Import patchs related with issues 3090 a o kern/170912 fs [zfs] [patch] unnecessarily setting DS_FLAG_INCONSISTE o bin/170778 fs [zfs] [panic] FreeBSD panics randomly o kern/170680 fs [nfs] Multiple NFS Client bug in the FreeBSD 7.4-RELEA o kern/170497 fs [xfs][panic] kernel will panic whenever I ls a mounted o kern/170238 fs [zfs] [panic] Panic when deleting data o kern/169945 fs [zfs] [panic] Kernel panic while importing zpool (afte o kern/169480 fs [zfs] ZFS stalls on heavy I/O o kern/169398 fs [zfs] Can't remove file with permanent error o kern/169339 fs panic while " : > /etc/123" o kern/169319 fs [zfs] zfs resilver can't complete o kern/168947 fs [nfs] [zfs] .zfs/snapshot directory is messed up when o kern/168942 fs [nfs] [hang] nfsd hangs after being restarted (not -HU o kern/168158 fs [zfs] incorrect parsing of sharenfs options in zfs (fs o kern/167979 fs [ufs] DIOCGDINFO ioctl does not work on 8.2 file syste o kern/167977 fs [smbfs] mount_smbfs results are differ when utf-8 or U o kern/167688 fs [fusefs] Incorrect signal handling with direct_io o kern/167685 fs [zfs] ZFS on USB drive prevents shutdown / reboot o kern/167612 fs [portalfs] The portal file system gets stuck inside po o kern/167272 fs [zfs] ZFS Disks reordering causes ZFS to pick the wron o kern/167260 fs [msdosfs] msdosfs disk was mounted the second time whe o kern/167109 fs [zfs] [panic] zfs diff kernel panic Fatal trap 9: gene o kern/167105 fs [nfs] mount_nfs can not handle source exports wiht mor o kern/167067 fs [zfs] [panic] ZFS panics the server o kern/167065 fs [zfs] boot fails when a spare is the boot disk o kern/167048 fs [nfs] [patch] RELEASE-9 crash when using ZFS+NULLFS+NF o kern/166912 fs [ufs] [panic] Panic after converting Softupdates to jo o kern/166851 fs [zfs] [hang] Copying directory from the mounted UFS di o kern/166477 fs [nfs] NFS data corruption. o kern/165950 fs [ffs] SU+J and fsck problem o kern/165923 fs [nfs] Writing to NFS-backed mmapped files fails if flu o kern/165521 fs [zfs] [hang] livelock on 1 Gig of RAM with zfs when 31 o kern/165392 fs Multiple mkdir/rmdir fails with errno 31 o kern/165087 fs [unionfs] lock violation in unionfs o kern/164472 fs [ufs] fsck -B panics on particular data inconsistency o kern/164370 fs [zfs] zfs destroy for snapshot fails on i386 and sparc o kern/164261 fs [nullfs] [patch] fix panic with NFS served from NULLFS o kern/164256 fs [zfs] device entry for volume is not created after zfs o kern/164184 fs [ufs] [panic] Kernel panic with ufs_makeinode o kern/163801 fs [md] [request] allow mfsBSD legacy installed in 'swap' o kern/163770 fs [zfs] [hang] LOR between zfs&syncer + vnlru leading to o kern/163501 fs [nfs] NFS exporting a dir and a subdir in that dir to o kern/162944 fs [coda] Coda file system module looks broken in 9.0 o kern/162860 fs [zfs] Cannot share ZFS filesystem to hosts with a hyph o kern/162751 fs [zfs] [panic] kernel panics during file operations o kern/162591 fs [nullfs] cross-filesystem nullfs does not work as expe o kern/162519 fs [zfs] "zpool import" relies on buggy realpath() behavi o kern/162362 fs [snapshots] [panic] ufs with snapshot(s) panics when g o kern/161968 fs [zfs] [hang] renaming snapshot with -r including a zvo o kern/161864 fs [ufs] removing journaling from UFS partition fails on o bin/161807 fs [patch] add option for explicitly specifying metadata o kern/161579 fs [smbfs] FreeBSD sometimes panics when an smb share is o kern/161533 fs [zfs] [panic] zfs receive panic: system ioctl returnin o kern/161438 fs [zfs] [panic] recursed on non-recursive spa_namespace_ o kern/161424 fs [nullfs] __getcwd() calls fail when used on nullfs mou o kern/161280 fs [zfs] Stack overflow in gptzfsboot o kern/161205 fs [nfs] [pfsync] [regression] [build] Bug report freebsd o kern/161169 fs [zfs] [panic] ZFS causes kernel panic in dbuf_dirty o kern/161112 fs [ufs] [lor] filesystem LOR in FreeBSD 9.0-BETA3 o kern/160893 fs [zfs] [panic] 9.0-BETA2 kernel panic o kern/160860 fs [ufs] Random UFS root filesystem corruption with SU+J o kern/160801 fs [zfs] zfsboot on 8.2-RELEASE fails to boot from root-o o kern/160790 fs [fusefs] [panic] VPUTX: negative ref count with FUSE o kern/160777 fs [zfs] [hang] RAID-Z3 causes fatal hang upon scrub/impo o kern/160706 fs [zfs] zfs bootloader fails when a non-root vdev exists o kern/160591 fs [zfs] Fail to boot on zfs root with degraded raidz2 [r o kern/160410 fs [smbfs] [hang] smbfs hangs when transferring large fil o kern/160283 fs [zfs] [patch] 'zfs list' does abort in make_dataset_ha o kern/159930 fs [ufs] [panic] kernel core o kern/159402 fs [zfs][loader] symlinks cause I/O errors o kern/159357 fs [zfs] ZFS MAXNAMELEN macro has confusing name (off-by- o kern/159356 fs [zfs] [patch] ZFS NAME_ERR_DISKLIKE check is Solaris-s o kern/159351 fs [nfs] [patch] - divide by zero in mountnfs() o kern/159251 fs [zfs] [request]: add FLETCHER4 as DEDUP hash option o kern/159077 fs [zfs] Can't cd .. with latest zfs version o kern/159048 fs [smbfs] smb mount corrupts large files o kern/159045 fs [zfs] [hang] ZFS scrub freezes system o kern/158839 fs [zfs] ZFS Bootloader Fails if there is a Dead Disk o kern/158802 fs amd(8) ICMP storm and unkillable process. o kern/158231 fs [nullfs] panic on unmounting nullfs mounted over ufs o f kern/157929 fs [nfs] NFS slow read o kern/157399 fs [zfs] trouble with: mdconfig force delete && zfs strip o kern/157179 fs [zfs] zfs/dbuf.c: panic: solaris assert: arc_buf_remov o kern/156797 fs [zfs] [panic] Double panic with FreeBSD 9-CURRENT and o kern/156781 fs [zfs] zfs is losing the snapshot directory, p kern/156545 fs [ufs] mv could break UFS on SMP systems o kern/156193 fs [ufs] [hang] UFS snapshot hangs && deadlocks processes o kern/156039 fs [nullfs] [unionfs] nullfs + unionfs do not compose, re o kern/155615 fs [zfs] zfs v28 broken on sparc64 -current o kern/155587 fs [zfs] [panic] kernel panic with zfs p kern/155411 fs [regression] [8.2-release] [tmpfs]: mount: tmpfs : No o kern/155199 fs [ext2fs] ext3fs mounted as ext2fs gives I/O errors o bin/155104 fs [zfs][patch] use /dev prefix by default when importing o kern/154930 fs [zfs] cannot delete/unlink file from full volume -> EN o kern/154828 fs [msdosfs] Unable to create directories on external USB o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 p kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o bin/153142 fs [zfs] ls -l outputs `ls: ./.zfs: Operation not support o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an f bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o conf/144213 fs [rc.d] [patch] Disappearing zvols on reboot o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o kern/118318 fs [nfs] NFS server hangs under special circumstances o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118126 fs [nfs] [patch] Poor NFS server write performance o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116583 fs [ffs] [hang] System freezes for short time when using o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 293 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Nov 26 17:17:11 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4FEF8AC2; Mon, 26 Nov 2012 17:17:11 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id AD3548FC0C; Mon, 26 Nov 2012 17:17:10 +0000 (UTC) Received: from tom.home (localhost [127.0.0.1]) by kib.kiev.ua (8.14.5/8.14.5) with ESMTP id qAQHGwbN049728; Mon, 26 Nov 2012 19:16:58 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.1 kib.kiev.ua qAQHGwbN049728 Received: (from kostik@localhost) by tom.home (8.14.5/8.14.5/Submit) id qAQHGwq6049727; Mon, 26 Nov 2012 19:16:58 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 26 Nov 2012 19:16:58 +0200 From: Konstantin Belousov To: sig6247 Subject: Re: clang compiled kernel panic when mounting zfs root on i386 Message-ID: <20121126171658.GD3013@kib.kiev.ua> References: <50b37d46.8584440a.735c.ffffb4e6@mx.google.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="maH1Gajj2nflutpK" Content-Disposition: inline In-Reply-To: <50b37d46.8584440a.735c.ffffb4e6@mx.google.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=0.2 required=5.0 tests=ALL_TRUSTED, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-current@freebsd.org, fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2012 17:17:11 -0000 --maH1Gajj2nflutpK Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Nov 26, 2012 at 06:31:34AM -0800, sig6247 wrote: >=20 > Hi, >=20 > Just checked out r243529, this only happens when the kernel is compiled > by clang, and only on i386, either recompiling the kernel with gcc or > booting from a UFS root works fine. Is it a known problem? It looks like that clang uses more stack than gcc, and zfs makes quite deep call chains. It would be a waste, generally, to increase the init process kernel stack size only to pacify zfs. And I suspect that it would not help in the similar situations when the same procedure initiated for non-root mounts. >=20 > Thanks, >=20 > ---------------------------------------------------------------------- > WARNING: WITNESS option enabled, expect reduced performance. > Trying to mount root from zfs:zroot []... >=20 > Fatal double fault: > eip =3D 0xc0adc37d > esp =3D 0xc86bffc8 > ebp =3D 0xc86c003c > cpuid =3D 1; apic id =3D 01 > panic: double fault > cpuid =3D 1 > KDB: enter: panic > [ thread pid 1 tid 100002 ] > Stopped at kdb_enter+0x3d: movl $0,kdb_why > db> bt > Tracing pid 1 tid 100002 td 0xc89efbc0 > kdb_enter(c1064aa4,c1064aa4,c10b806f,c139e3b8,f5eacada,...) at kdb_enter+= 0x3d > panic(c10b806f,1,1,1,c86c003c,...) at panic+0x14b > dblfault_handler() at dblfault_handler+0xab > --- trap 0x17, eip =3D 0xc0adc37d, esp =3D 0xc86bffc8, ebp =3D 0xc86c003c= --- > witness_checkorder(c1fd7508,9,c109df18,7fa,0,...) at witness_checkorder+0= x37d > __mtx_lock_flags(c1fd7518,0,c109df18,7fa,c135d918,...) at __mtx_lock_flag= s+0x87 > uma_zalloc_arg(c1fd66c0,0,1,4d3,c86c0110,...) at uma_zalloc_arg+0x605 > vm_map_insert(c1fd508c,c13dfc10,bb1f000,0,cba1e000,...) at vm_map_insert+= 0x499 > kmem_back(c1fd508c,cba1e000,1000,3,c86c01d4,...) at kmem_back+0x76 > kmem_malloc(c1fd508c,1000,3) at kmem_malloc+0x250 > page_alloc(c1fd1d80,1000,c86c020b,3,c1fd1d80,...) at page_alloc+0x27 > keg_alloc_slab(103,4,c109df18,870,cb99ef6c,...) at keg_alloc_slab+0xc3 > keg_fetch_slab(103,c1fd1d80,cb99ef6c,c1fc8230,c86c02c0,...) at keg_fetch_= slab+0xe2 > zone_fetch_slab(c1fd1d80,c1fd0480,103,826,0,...) at zone_fetch_slab+0x43 > uma_zalloc_arg(c1fd1d80,0,102,3,2,...) at uma_zalloc_arg+0x3f2 > malloc(4c,c1686100,102,c86c0388,c173d09a,...) at malloc+0xe9 > zfs_kmem_alloc(4c,102,cb618820,c89efbc0,cb618820,...) at zfs_kmem_alloc+0= x20 > vdev_mirror_io_start(cb8218a0,10,cb8218a0,1,0,...) at vdev_mirror_io_star= t+0x14a > zio_vdev_io_start(cb8218a0,c89efbc0,0,cb8218a0,c86c0600,...) at zio_vdev_= io_start+0x228 > zio_execute(cb8218a0,cb618000,cba1b640,cb900000,400,...) at zio_execute+0= x106 > spa_load_verify_cb(cb618000,0,cba1b640,cb884b40,c86c0600,...) at spa_load= _verify_cb+0x89 > traverse_visitbp(cb884b40,cba1b640,c86c0600,c86c0ba0,0,...) at traverse_v= isitbp+0x29f > traverse_dnode(cb884b40,0,0,8b,0,...) at traverse_dnode+0x92 > traverse_visitbp(cb884bb8,cba07200,c86c0890,cb884bf4,c16ce7e0,...) at tra= verse_visitbp+0xe47 > traverse_visitbp(cb884bf4,cb9bf840,c86c0968,c86c0ba0,0,...) at traverse_v= isitbp+0xf32 > traverse_dnode(cb884bf4,0,0,0,0,...) at traverse_dnode+0x92 > traverse_visitbp(0,cb618398,c86c0b50,2,cb9f1c78,...) at traverse_visitbp+= 0x96d > traverse_impl(0,0,cb618398,3e1,0,...) at traverse_impl+0x268 > traverse_pool(cb618000,3e1,0,d,c1727830,...) at traverse_pool+0x79 > spa_load(0,1,c86c0ec4,1e,0,...) at spa_load+0x1dde > spa_load(0,0,c13d8c94,1,3,...) at spa_load+0x11a5 > spa_load_best(0,ffffffff,ffffffff,1,c0adc395,...) at spa_load_best+0x71 > spa_open_common(c17e0e1e,0,0,c86c1190,c16f5a1c,...) at spa_open_common+0x= 11a > spa_open(c86c1078,c86c1074,c17e0e1e,c135d918,c1fd7798,...) at spa_open+0x= 27 > dsl_dir_open_spa(0,cb770030,c17e11b1,c86c11f8,c86c11f4,...) at dsl_dir_op= en_spa+0x6c > dsl_dataset_hold(cb770030,cb613800,c86c1240,cb613800,cb613800,...) at dsl= _dataset_hold+0x3a > dsl_dataset_own(cb770030,0,cb613800,c86c1240,c1684e30,...) at dsl_dataset= _own+0x21 > dmu_objset_own(cb770030,2,1,cb613800,c86c1290,...) at dmu_objset_own+0x2a > zfsvfs_create(cb770030,c86c13ac,c17ee05d,681,0,...) at zfsvfs_create+0x4c > zfs_mount(cb78ed20,c17f411c,c9ff4600,c89cae80,0,...) at zfs_mount+0x42c > vfs_donmount(c89efbc0,4000,0,c86c1790,cb6c0800,...) at vfs_donmount+0xc6d > kernel_mount(cb7700b0,4000,0,0,1,...) at kernel_mount+0x6b > parse_mount(cb7700e0,c1194498,0,1,0,...) at parse_mount+0x606 > vfs_mountroot(c13d95b0,4,c105c042,2bb,0,...) at vfs_mountroot+0x6cf > start_init(0,c86c1d08,c105e94c,3db,0,...) at start_init+0x6a > fork_exit(c0a42090,0,c86c1d08) at fork_exit+0x7f > fork_trampoline() at fork_trampoline+0x8 > --- trap 0, eip =3D 0, esp =3D 0xc86c1d40, ebp =3D 0 --- > db>=20 > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" --maH1Gajj2nflutpK Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlCzpAoACgkQC3+MBN1Mb4j9FQCfX/BKJvG1317xh1X9RQQaCRmb FG0AoJTD3ObL+l6uWhI4ORT8GkSsGtIJ =/qn9 -----END PGP SIGNATURE----- --maH1Gajj2nflutpK-- From owner-freebsd-fs@FreeBSD.ORG Mon Nov 26 21:21:15 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A5472583; Mon, 26 Nov 2012 21:21:15 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail13.syd.optusnet.com.au (mail13.syd.optusnet.com.au [211.29.132.194]) by mx1.freebsd.org (Postfix) with ESMTP id 263408FC1C; Mon, 26 Nov 2012 21:21:14 +0000 (UTC) Received: from c122-106-175-26.carlnfd1.nsw.optusnet.com.au (c122-106-175-26.carlnfd1.nsw.optusnet.com.au [122.106.175.26]) by mail13.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id qAQLL5T2008212 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 27 Nov 2012 08:21:06 +1100 Date: Tue, 27 Nov 2012 08:21:05 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Konstantin Belousov Subject: Re: clang compiled kernel panic when mounting zfs root on i386 In-Reply-To: <20121126171658.GD3013@kib.kiev.ua> Message-ID: <20121127071243.D1255@besplex.bde.org> References: <50b37d46.8584440a.735c.ffffb4e6@mx.google.com> <20121126171658.GD3013@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-Cloudmark-Score: 0 X-Optus-Cloudmark-Analysis: v=2.0 cv=TL4d0CZa c=1 sm=1 a=Dqyo4nusdH8A:10 a=kj9zAlcOel0A:10 a=PO7r1zJSAAAA:8 a=JzwRw_2MAAAA:8 a=PZxpgLn_-10A:10 a=IMFgP6cxdJxPq-vExgcA:9 a=CjuIK1q_8ugA:10 a=yiplnose_AnHQ6Q_:21 a=7WiQoUoJrVuut8JH:21 a=bxQHXO5Py4tHmhUgaywp5w==:117 Cc: freebsd-current@freebsd.org, fs@freebsd.org, sig6247 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2012 21:21:15 -0000 On Mon, 26 Nov 2012, Konstantin Belousov wrote: > On Mon, Nov 26, 2012 at 06:31:34AM -0800, sig6247 wrote: >> >> Just checked out r243529, this only happens when the kernel is compiled >> by clang, and only on i386, either recompiling the kernel with gcc or >> booting from a UFS root works fine. Is it a known problem? > It looks like that clang uses more stack than gcc, and zfs makes quite > deep call chains. > > It would be a waste, generally, to increase the init process kernel > stack size only to pacify zfs. And I suspect that it would not help > in the similar situations when the same procedure initiated for non-root > mounts. Or to pacify clang... >> ---------------------------------------------------------------------- >> WARNING: WITNESS option enabled, expect reduced performance. >> Trying to mount root from zfs:zroot []... >> >> Fatal double fault: >> eip = 0xc0adc37d >> esp = 0xc86bffc8 >> ebp = 0xc86c003c >> cpuid = 1; apic id = 01 >> panic: double fault >> cpuid = 1 >> KDB: enter: panic >> [ thread pid 1 tid 100002 ] >> Stopped at kdb_enter+0x3d: movl $0,kdb_why >> db> bt >> Tracing pid 1 tid 100002 td 0xc89efbc0 >> kdb_enter(c1064aa4,c1064aa4,c10b806f,c139e3b8,f5eacada,...) at kdb_enter+0x3d >> panic(c10b806f,1,1,1,c86c003c,...) at panic+0x14b >> dblfault_handler() at dblfault_handler+0xab >> --- trap 0x17, eip = 0xc0adc37d, esp = 0xc86bffc8, ebp = 0xc86c003c --- >> witness_checkorder(c1fd7508,9,c109df18,7fa,0,...) at witness_checkorder+0x37d >> __mtx_lock_flags(c1fd7518,0,c109df18,7fa,c135d918,...) at __mtx_lock_flags+0x87 >> uma_zalloc_arg(c1fd66c0,0,1,4d3,c86c0110,...) at uma_zalloc_arg+0x605 >> vm_map_insert(c1fd508c,c13dfc10,bb1f000,0,cba1e000,...) at vm_map_insert+0x499 >> kmem_back(c1fd508c,cba1e000,1000,3,c86c01d4,...) at kmem_back+0x76 >> kmem_malloc(c1fd508c,1000,3) at kmem_malloc+0x250 >> page_alloc(c1fd1d80,1000,c86c020b,3,c1fd1d80,...) at page_alloc+0x27 >> keg_alloc_slab(103,4,c109df18,870,cb99ef6c,...) at keg_alloc_slab+0xc3 >> keg_fetch_slab(103,c1fd1d80,cb99ef6c,c1fc8230,c86c02c0,...) at keg_fetch_slab+0xe2 >> zone_fetch_slab(c1fd1d80,c1fd0480,103,826,0,...) at zone_fetch_slab+0x43 >> uma_zalloc_arg(c1fd1d80,0,102,3,2,...) at uma_zalloc_arg+0x3f2 >> malloc(4c,c1686100,102,c86c0388,c173d09a,...) at malloc+0xe9 >> zfs_kmem_alloc(4c,102,cb618820,c89efbc0,cb618820,...) at zfs_kmem_alloc+0x20 >> vdev_mirror_io_start(cb8218a0,10,cb8218a0,1,0,...) at vdev_mirror_io_start+0x14a >> zio_vdev_io_start(cb8218a0,c89efbc0,0,cb8218a0,c86c0600,...) at zio_vdev_io_start+0x228 >> zio_execute(cb8218a0,cb618000,cba1b640,cb900000,400,...) at zio_execute+0x106 >> spa_load_verify_cb(cb618000,0,cba1b640,cb884b40,c86c0600,...) at spa_load_verify_cb+0x89 >> traverse_visitbp(cb884b40,cba1b640,c86c0600,c86c0ba0,0,...) at traverse_visitbp+0x29f >> traverse_dnode(cb884b40,0,0,8b,0,...) at traverse_dnode+0x92 >> traverse_visitbp(cb884bb8,cba07200,c86c0890,cb884bf4,c16ce7e0,...) at traverse_visitbp+0xe47 >> traverse_visitbp(cb884bf4,cb9bf840,c86c0968,c86c0ba0,0,...) at traverse_visitbp+0xf32 >> traverse_dnode(cb884bf4,0,0,0,0,...) at traverse_dnode+0x92 >> traverse_visitbp(0,cb618398,c86c0b50,2,cb9f1c78,...) at traverse_visitbp+0x96d >> traverse_impl(0,0,cb618398,3e1,0,...) at traverse_impl+0x268 >> traverse_pool(cb618000,3e1,0,d,c1727830,...) at traverse_pool+0x79 >> spa_load(0,1,c86c0ec4,1e,0,...) at spa_load+0x1dde >> spa_load(0,0,c13d8c94,1,3,...) at spa_load+0x11a5 >> spa_load_best(0,ffffffff,ffffffff,1,c0adc395,...) at spa_load_best+0x71 >> spa_open_common(c17e0e1e,0,0,c86c1190,c16f5a1c,...) at spa_open_common+0x11a >> spa_open(c86c1078,c86c1074,c17e0e1e,c135d918,c1fd7798,...) at spa_open+0x27 >> dsl_dir_open_spa(0,cb770030,c17e11b1,c86c11f8,c86c11f4,...) at dsl_dir_open_spa+0x6c >> dsl_dataset_hold(cb770030,cb613800,c86c1240,cb613800,cb613800,...) at dsl_dataset_hold+0x3a >> dsl_dataset_own(cb770030,0,cb613800,c86c1240,c1684e30,...) at dsl_dataset_own+0x21 >> dmu_objset_own(cb770030,2,1,cb613800,c86c1290,...) at dmu_objset_own+0x2a >> zfsvfs_create(cb770030,c86c13ac,c17ee05d,681,0,...) at zfsvfs_create+0x4c >> zfs_mount(cb78ed20,c17f411c,c9ff4600,c89cae80,0,...) at zfs_mount+0x42c >> vfs_donmount(c89efbc0,4000,0,c86c1790,cb6c0800,...) at vfs_donmount+0xc6d >> kernel_mount(cb7700b0,4000,0,0,1,...) at kernel_mount+0x6b >> parse_mount(cb7700e0,c1194498,0,1,0,...) at parse_mount+0x606 >> vfs_mountroot(c13d95b0,4,c105c042,2bb,0,...) at vfs_mountroot+0x6cf >> start_init(0,c86c1d08,c105e94c,3db,0,...) at start_init+0x6a >> fork_exit(c0a42090,0,c86c1d08) at fork_exit+0x7f >> fork_trampoline() at fork_trampoline+0x8 >> --- trap 0, eip = 0, esp = 0xc86c1d40, ebp = 0 --- >> db> 43 deep (before the double fault) is disgusting, but even if clang has broken stack alignment due to a wrong default and no -mpreferred-stack-boundary to fix it, that's still only about 8*43 extra bytes (8 for the average extra stack to align to 16 bytes). Probably zfs is also putting large data structures on the stack. It would be useful if the stack trace printed the the stack pointer on every function call, so that you could see how much stack each function used. All those ', ...' printed after 5 args show further apparent clang lossage. vfs_mountroot takes no args at all, but the above shows ot taking 5 known args and further unknown args. ddb's stack tracer does limited parsing of the bytes in function's caller to guess the number of args. It usually guesses right for gcc. It apparently usually guesses wrong for clang. I can't test this for i386 since ref10-i386 is down, but for amd64 the parsing seems to be already too dificult for gcc and impossible for clang: for the function call in `void bar(void); void foo(void) { bar(); }', clang generates a tail call: % test: # @test % .cfi_startproc % # BB#0: # %entry % xorb %al, %al % jmp bar # TAILCALL % .Ltmp0: The bytes after the tail call are unparseable. The bytes before the tail call are difficult to parse, and AFAIR ddb doesn't look at them (the xorb says that there are no xmm args, but that is of no interest in the kernel). Gcc generates: % test: % .LFB2: % subq $8, %rsp % .LCFI0: % movl $0, %eax % call bar % addq $8, %rsp % ret The addq $8,%rsp after the call looks like it is popping 1 arg and is probably misguessed as such -- it is actually to clean up the stack alignment before return. Actually, this guess doesn't apply to amd64 -- the first few args are in registers where there number is almost impossible to guess even for gcc if there are none on stack; it is only when there are some on the stack that the number there can possibly be guessed and the total number = #number in registers + #on stack. With 1 arg: `void bar(int x); void foo(int) { bar(x); }', the generated code is identical for both compilers. So have any chance of guessing the number of args for bar(), bytes in bar()'s caller's callers would have to be checked, recursively. This is too hard for ddb. With tail calls, the checking is unbounded. Tail calls also break the stack trace, but save space so running out of kernel stack is less likely :-). It is arguable a bug to compile kernels with options that make stack tracing impossible. Even passing args in registers makes correct arg printing impossible -- ddb would have to assume that there are always at least 3 (?) args on amd64. The above shows 3 being printed for fork_exit(), which is correct, and 3 for kmem_malloc(), which is correct, and none for fork_trampoline(), which depends on ddb having special knowledge of this and a few other low-level/asm functions. All the others are broken. An very old bug in this area are that functions written in asm don't have normal frames (at least on amd64 and i386) unless profiling is configured. They are mostly leaf functions that shouldn't trap, so this is not very important for interactive use of ddb. But it makes the args of a function like memcpy() harder to find in interactive use, and breaks stack traces if an interrupt or trap occurs in the functions. Compilers may also omit frame pointers for leaf functions, but this is easy to prevent. The efficiency gains from not using frame pointers are small or negative, at least on modern x86 since the frame pointer management can run in parallel and is especially fast in leaf functions where it is not actually used (except for debugging), so frame pointers should never be omitted if there is any chance that the code needs to be debugged. The chance is almost 100% for kernels since they can be debugged if GDB or DDB or stack tracing is configured or if panic dumps are enabled and occur. A not so old bug in this area is that compiling with -O2 or just with with -march tends to break things. Even on old i386 kernels compiled with -O -march=i386, I mostly see 5 args (but no ', ...'). For a just a few functions, the calling sequence is "call foo; addl $4*N,%esp" and ddb correctly guesses that this means N args. Another sequence is "call foo; movl %eax,%esi; addl $8,%esp". ddb doesn't understand this at all, and misguesses that there are 5 args. Bruce From owner-freebsd-fs@FreeBSD.ORG Mon Nov 26 22:10:21 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3A1FE8B8 for ; Mon, 26 Nov 2012 22:10:21 +0000 (UTC) (envelope-from raymondj@caltech.edu) Received: from outgoing-mail.its.caltech.edu (outgoing-mail.its.caltech.edu [131.215.239.19]) by mx1.freebsd.org (Postfix) with ESMTP id 17E818FC17 for ; Mon, 26 Nov 2012 22:10:20 +0000 (UTC) Received: from earth-doxen.imss.caltech.edu (localhost [127.0.0.1]) by earth-doxen-postvirus (Postfix) with ESMTP id 71E0066E01C9 for ; Mon, 26 Nov 2012 14:00:53 -0800 (PST) X-Spam-Scanned: at Caltech-IMSS on earth-doxen by amavisd-new Received: from [127.0.0.1] (mitsuki.caltech.edu [131.215.167.33]) (Authenticated sender: raymondj) by earth-doxen-submit (Postfix) with ESMTP id 513AF66E01E5 for ; Mon, 26 Nov 2012 14:00:51 -0800 (PST) Message-ID: <50B3E680.8060606@caltech.edu> Date: Mon, 26 Nov 2012 14:00:32 -0800 From: Raymond Jimenez User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: ZFS kernel panics due to corrupt DVAs (despite RAIDZ) Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2012 22:10:21 -0000 Hello, We recently sent our drives out for data recovery (blown drive electronics), and when we got the new drives/data back, ZFS started to kernel panic whenever listing certain items in a directory, or whenever a scrub is close to finishing (~99.97%) The zpool worked fine before data recovery, and most of the files are accessible (only a couple hundred unavailable out of several million). Here's the kernel panic output if I scrub the disk: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x38 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff810792d1 stack pointer = 0x28:0xffffff8235122720 frame pointer = 0x28:0xffffff8235122750 code segment = base 0x0, limit 0xffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 52 (txg_thread_enter) [thread pid 52 tid 101230 ] Stopped at vdev_is_dead+0x1: cmpq $0x5, 0x38(%rdi) $rdi is zero, so this seems to be just a null pointer exception. The vdev setup looks like: pool: mfs-zpool004 state: ONLINE scan: scrub canceled on Mon Nov 26 05:40:49 2012 config: NAME STATE READ WRITE CKSUM mfs-zpool004 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gpt/lenin3-drive8 ONLINE 0 0 0 gpt/lenin3-drive9.eli ONLINE 0 0 0 gpt/lenin3-drive10 ONLINE 0 0 0 gpt/lenin3-drive11.eli ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 gpt/lenin3-drive12 ONLINE 0 0 0 gpt/lenin3-drive13.eli ONLINE 0 0 0 gpt/lenin3-drive14 ONLINE 0 0 0 gpt/lenin3-drive15.eli ONLINE 0 0 0 errors: No known data errors The initial scrub fixed some data (~24k) in the early stages, but also crashed at 99.97%. Right now, I'm using an interim work-around patch[1] so that our users can get files without worrying about crashing the server. It's a small check in dbuf_findbp() that checks if the DVA that will be returned has a small (=<16) vdev number, and if not, returns EIO. This just results in ZFS returning I/O errors for any of the corrupt files I try to access, which at least lets us get at our data for now. My suspicion is that somehow, bad data is getting interpreted as a block pointer/shift constant, and this sends ZFS into the woods. I haven't been able to track down how this data could get past checksum verification, especially with RAIDZ. Backtraces: (both crashes due to vdev_is_dead() dereferencing a null pointer) Scrub crash: http://wsyntax.com/~raymond/zfs/zfs-scrub-bt.txt Prefetch off, ls -al of "/06/chunk_0000000001417E06_00000001.mfs": http://wsyntax.com/~raymond/zfs/zfs-ls-bt.txt Regards, Raymond Jimenez [1] http://wsyntax.com/~raymond/zfs/zfs-dva-corrupt-workaround.patch From owner-freebsd-fs@FreeBSD.ORG Mon Nov 26 22:21:53 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C9A40C6 for ; Mon, 26 Nov 2012 22:21:53 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-la0-f54.google.com (mail-la0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 47DA98FC12 for ; Mon, 26 Nov 2012 22:21:52 +0000 (UTC) Received: by mail-la0-f54.google.com with SMTP id j13so11205181lah.13 for ; Mon, 26 Nov 2012 14:21:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=TP/qGWgceSySL1zKNubUXepMd5j3tEtxbab43hdLqrc=; b=KoDseEuvo9FwNBPQ8Mfye9zTIqLslW9eFzDJVrJ+7Wu/p71dm+O2bwMzlOmi0ryJRk qR/wVvXl8ZL+mmqUVZ9FnCx3J5zvkH7MwtEweVu6kxycFVVQMSui+yduMONC8XSYoIOF x9mXqlJUQ9bLV6p3G8KIeKk7VCE1Rau3wt0+sWIra/O44RxSUH75mq3w4FZQT4V3txYP sq4FlxLcsJinB3I2RWdKA838u0oR2K7ldfp3vHC0nQRbclNSqPdFCFi3RNWc354MZBf5 7Ep7z/zH5b/a5eHXjT5vXSr1U3yKlPpTrBeIuvRc1rU9bquuIxDrNJ7PAEJTbVx+Ailv 9EZw== MIME-Version: 1.0 Received: by 10.152.106.212 with SMTP id gw20mr12616349lab.8.1353968511609; Mon, 26 Nov 2012 14:21:51 -0800 (PST) Received: by 10.112.144.101 with HTTP; Mon, 26 Nov 2012 14:21:51 -0800 (PST) In-Reply-To: <50B3E680.8060606@caltech.edu> References: <50B3E680.8060606@caltech.edu> Date: Mon, 26 Nov 2012 14:21:51 -0800 Message-ID: Subject: Re: ZFS kernel panics due to corrupt DVAs (despite RAIDZ) From: Garrett Cooper To: Raymond Jimenez Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2012 22:21:53 -0000 On Mon, Nov 26, 2012 at 2:00 PM, Raymond Jimenez wrote: > Hello, > > We recently sent our drives out for data recovery (blown drive > electronics), and when we got the new drives/data back, ZFS > started to kernel panic whenever listing certain items in a > directory, or whenever a scrub is close to finishing (~99.97%) > > The zpool worked fine before data recovery, and most of the > files are accessible (only a couple hundred unavailable out of > several million). > > Here's the kernel panic output if I scrub the disk: > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x38 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff810792d1 > stack pointer = 0x28:0xffffff8235122720 > frame pointer = 0x28:0xffffff8235122750 > code segment = base 0x0, limit 0xffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 52 (txg_thread_enter) > [thread pid 52 tid 101230 ] > Stopped at vdev_is_dead+0x1: cmpq $0x5, 0x38(%rdi) > > $rdi is zero, so this seems to be just a null pointer exception. > > The vdev setup looks like: > > pool: mfs-zpool004 > state: ONLINE > scan: scrub canceled on Mon Nov 26 05:40:49 2012 > config: > > NAME STATE READ WRITE CKSUM > mfs-zpool004 ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > gpt/lenin3-drive8 ONLINE 0 0 0 > gpt/lenin3-drive9.eli ONLINE 0 0 0 > gpt/lenin3-drive10 ONLINE 0 0 0 > gpt/lenin3-drive11.eli ONLINE 0 0 0 > raidz1-1 ONLINE 0 0 0 > gpt/lenin3-drive12 ONLINE 0 0 0 > gpt/lenin3-drive13.eli ONLINE 0 0 0 > gpt/lenin3-drive14 ONLINE 0 0 0 > gpt/lenin3-drive15.eli ONLINE 0 0 0 > > errors: No known data errors > > The initial scrub fixed some data (~24k) in the early stages, but > also crashed at 99.97%. > > Right now, I'm using an interim work-around patch[1] so that our > users can get files without worrying about crashing the server. > It's a small check in dbuf_findbp() that checks if the DVA that will > be returned has a small (=<16) vdev number, and if not, returns EIO. > This just results in ZFS returning I/O errors for any of the corrupt > files I try to access, which at least lets us get at our data for now. > > My suspicion is that somehow, bad data is getting interpreted as > a block pointer/shift constant, and this sends ZFS into the woods. > I haven't been able to track down how this data could get past > checksum verification, especially with RAIDZ. > > Backtraces: > > (both crashes due to vdev_is_dead() dereferencing a null pointer) > > Scrub crash: > http://wsyntax.com/~raymond/zfs/zfs-scrub-bt.txt > > Prefetch off, ls -al of "/06/chunk_0000000001417E06_00000001.mfs": > http://wsyntax.com/~raymond/zfs/zfs-ls-bt.txt This is missing key details like uname, zpool version, etc. Thanks, -Garrett From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 00:54:21 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4A101250 for ; Tue, 27 Nov 2012 00:54:21 +0000 (UTC) (envelope-from raymondj@caltech.edu) Received: from outgoing-mail.its.caltech.edu (outgoing-mail.its.caltech.edu [131.215.239.19]) by mx1.freebsd.org (Postfix) with ESMTP id 266308FC0C for ; Tue, 27 Nov 2012 00:54:20 +0000 (UTC) Received: from fire-doxen.imss.caltech.edu (localhost [127.0.0.1]) by fire-doxen-postvirus (Postfix) with ESMTP id 9AEC12E50E6C; Mon, 26 Nov 2012 16:54:20 -0800 (PST) X-Spam-Scanned: at Caltech-IMSS on fire-doxen by amavisd-new Received: from [127.0.0.1] (mitsuki.caltech.edu [131.215.167.33]) (Authenticated sender: raymondj) by fire-doxen-submit (Postfix) with ESMTP id 577A42E50E69; Mon, 26 Nov 2012 16:54:17 -0800 (PST) Message-ID: <50B40F26.7070608@caltech.edu> Date: Mon, 26 Nov 2012 16:53:58 -0800 From: Raymond Jimenez User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Garrett Cooper Subject: Re: ZFS kernel panics due to corrupt DVAs (despite RAIDZ) References: <50B3E680.8060606@caltech.edu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 00:54:21 -0000 On 11/26/2012 2:21 PM, Garrett Cooper wrote: > On Mon, Nov 26, 2012 at 2:00 PM, Raymond Jimenez wrote: >> Hello, >> >> We recently sent our drives out for data recovery (blown drive >> electronics), and when we got the new drives/data back, ZFS >> started to kernel panic whenever listing certain items in a >> directory, or whenever a scrub is close to finishing (~99.97%) >> >> The zpool worked fine before data recovery, and most of the >> files are accessible (only a couple hundred unavailable out of >> several million). >> >> Here's the kernel panic output if I scrub the disk: >> >> Fatal trap 12: page fault while in kernel mode >> cpuid = 0; apic id = 00 >> fault virtual address = 0x38 >> fault code = supervisor read data, page not present >> instruction pointer = 0x20:0xffffffff810792d1 >> stack pointer = 0x28:0xffffff8235122720 >> frame pointer = 0x28:0xffffff8235122750 >> code segment = base 0x0, limit 0xffff, type 0x1b >> = DPL 0, pres 1, long 1, def32 0, gran 1 >> processor eflags = interrupt enabled, resume, IOPL = 0 >> current process = 52 (txg_thread_enter) >> [thread pid 52 tid 101230 ] >> Stopped at vdev_is_dead+0x1: cmpq $0x5, 0x38(%rdi) >> >> $rdi is zero, so this seems to be just a null pointer exception. >> >> The vdev setup looks like: >> >> pool: mfs-zpool004 >> state: ONLINE >> scan: scrub canceled on Mon Nov 26 05:40:49 2012 >> config: >> >> NAME STATE READ WRITE CKSUM >> mfs-zpool004 ONLINE 0 0 0 >> raidz1-0 ONLINE 0 0 0 >> gpt/lenin3-drive8 ONLINE 0 0 0 >> gpt/lenin3-drive9.eli ONLINE 0 0 0 >> gpt/lenin3-drive10 ONLINE 0 0 0 >> gpt/lenin3-drive11.eli ONLINE 0 0 0 >> raidz1-1 ONLINE 0 0 0 >> gpt/lenin3-drive12 ONLINE 0 0 0 >> gpt/lenin3-drive13.eli ONLINE 0 0 0 >> gpt/lenin3-drive14 ONLINE 0 0 0 >> gpt/lenin3-drive15.eli ONLINE 0 0 0 >> >> errors: No known data errors >> >> The initial scrub fixed some data (~24k) in the early stages, but >> also crashed at 99.97%. >> >> Right now, I'm using an interim work-around patch[1] so that our >> users can get files without worrying about crashing the server. >> It's a small check in dbuf_findbp() that checks if the DVA that will >> be returned has a small (=<16) vdev number, and if not, returns EIO. >> This just results in ZFS returning I/O errors for any of the corrupt >> files I try to access, which at least lets us get at our data for now. >> >> My suspicion is that somehow, bad data is getting interpreted as >> a block pointer/shift constant, and this sends ZFS into the woods. >> I haven't been able to track down how this data could get past >> checksum verification, especially with RAIDZ. >> >> Backtraces: >> >> (both crashes due to vdev_is_dead() dereferencing a null pointer) >> >> Scrub crash: >> http://wsyntax.com/~raymond/zfs/zfs-scrub-bt.txt >> >> Prefetch off, ls -al of "/06/chunk_0000000001417E06_00000001.mfs": >> http://wsyntax.com/~raymond/zfs/zfs-ls-bt.txt > > This is missing key details like uname, zpool version, etc. Sorry, total oversight on my part. uname -a: FreeBSD 03.chunk.dabney 9.0-STABLE FreeBSD 9.0-STABLE #25: Sat Nov 24 05:02:35 PST 2012 root@mfsmaster.dabney:/usr/obj/usr/src/sys/LENIN amd64 (updated as of a couple months ago) ZFS pool version 28, ZFS filesystem version 5. All disks are 3TB Seagate Barracuda 7200.14 ST3000DM001's, on a LSI 9211-8i, shows up as: mps0: port 0xb000-0xb0ff mem 0xfb33c000-0xfb33ffff,0xfb340000-0xfb37ffff irq 16 at device 0.0 on pci1 mps0: Firmware: 07.00.00.00 mps0: IOCCapabilities: 185c /boot/loader.conf: vfs.zfs.prefetch_disable="0" kern.geom.label.gptid.enable="0" vfs.zfs.arc_max="5G" kern.ipc.nmbclusters="131072" kern.ipc.maxsockbuf=16777216 kern.ipc.nmbjumbo9="38300" boot_multicons="YES" boot_serial="YES" console="comconsole,vidconsole" No ZFS tunables in /etc/sysctl.conf. The system is limited to 5GB ARC since the system has 8GB memory and is a diskless client; we were running into some lockups if we didn't restrict the ARC. Thanks, Raymond Jimenez From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 11:07:57 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 00080996; Tue, 27 Nov 2012 11:07:56 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: from mail-qc0-f182.google.com (mail-qc0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 915BD8FC08; Tue, 27 Nov 2012 11:07:56 +0000 (UTC) Received: by mail-qc0-f182.google.com with SMTP id k19so11076179qcs.13 for ; Tue, 27 Nov 2012 03:07:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=ayJ7WV9dv3yx8T8euUnnu5gF13uF6oosYusF7R/vj9k=; b=H19CAvEqxnPr3oXOfCFaYeAxbYTr3VcKh7zr4fuWjJqNwKF5Oih8CzZDLRpHpLckwg RWtc3cXv42UvfBb61ryQtEsWZb2vtAooiVbbz8enCSfCJkPtXgcs1IEleO+2OJG/xwui JVJMvG9Qd2uuDi2g5OS9yxY0bvE9iaEmL01iRvZPskGEk4G2tArV0UejMSsLgryDcgGN GtFxoC+MYXztJI1Z9MNheSVoTLCjuKvMfKMMaoj4Iz4dIT66/FYyes9eOUqCwx7kDQ0D mITfC+oJSwvZIZXjjioaYSLHO+LnxDRPdCmuHEDieWV0qwZK0q8DMD54sY1Fqa0z4rYr K8dg== MIME-Version: 1.0 Received: by 10.49.82.98 with SMTP id h2mr17418638qey.14.1354014470177; Tue, 27 Nov 2012 03:07:50 -0800 (PST) Received: by 10.49.121.163 with HTTP; Tue, 27 Nov 2012 03:07:50 -0800 (PST) Date: Tue, 27 Nov 2012 12:07:50 +0100 Message-ID: Subject: Re-sizable UFS project From: =?ISO-8859-1?Q?Ermal_Lu=E7i?= To: freebsd-current@freebsd.org, freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 11:07:57 -0000 Hello, some time ago the FreeBSD Foundation published/approved a project for live resizing of UFS filesystems. Does any know if the project was successful and any outcome from it? -- Ermal From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 11:09:39 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 537B0BFC for ; Tue, 27 Nov 2012 11:09:39 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 82A798FC16 for ; Tue, 27 Nov 2012 11:09:38 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA11672; Tue, 27 Nov 2012 13:09:33 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1TdJ2i-000NUh-Og; Tue, 27 Nov 2012 13:09:32 +0200 Message-ID: <50B49F6A.2020509@FreeBSD.org> Date: Tue, 27 Nov 2012 13:09:30 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Raymond Jimenez Subject: Re: ZFS kernel panics due to corrupt DVAs (despite RAIDZ) References: <50B3E680.8060606@caltech.edu> In-Reply-To: <50B3E680.8060606@caltech.edu> X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 11:09:39 -0000 on 27/11/2012 00:00 Raymond Jimenez said the following: > Hello, > > We recently sent our drives out for data recovery (blown drive > electronics), and when we got the new drives/data back, ZFS > started to kernel panic whenever listing certain items in a > directory, or whenever a scrub is close to finishing (~99.97%) Perhaps this thread could be of some interest to you: http://thread.gmane.org/gmane.os.freebsd.devel.file-systems/15611/focus=15616 > The zpool worked fine before data recovery, and most of the > files are accessible (only a couple hundred unavailable out of > several million). > > Here's the kernel panic output if I scrub the disk: > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x38 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff810792d1 > stack pointer = 0x28:0xffffff8235122720 > frame pointer = 0x28:0xffffff8235122750 > code segment = base 0x0, limit 0xffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 52 (txg_thread_enter) > [thread pid 52 tid 101230 ] > Stopped at vdev_is_dead+0x1: cmpq $0x5, 0x38(%rdi) > > $rdi is zero, so this seems to be just a null pointer exception. > > The vdev setup looks like: > > pool: mfs-zpool004 > state: ONLINE > scan: scrub canceled on Mon Nov 26 05:40:49 2012 > config: > > NAME STATE READ WRITE CKSUM > mfs-zpool004 ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > gpt/lenin3-drive8 ONLINE 0 0 0 > gpt/lenin3-drive9.eli ONLINE 0 0 0 > gpt/lenin3-drive10 ONLINE 0 0 0 > gpt/lenin3-drive11.eli ONLINE 0 0 0 > raidz1-1 ONLINE 0 0 0 > gpt/lenin3-drive12 ONLINE 0 0 0 > gpt/lenin3-drive13.eli ONLINE 0 0 0 > gpt/lenin3-drive14 ONLINE 0 0 0 > gpt/lenin3-drive15.eli ONLINE 0 0 0 > > errors: No known data errors > > The initial scrub fixed some data (~24k) in the early stages, but > also crashed at 99.97%. > > Right now, I'm using an interim work-around patch[1] so that our > users can get files without worrying about crashing the server. > It's a small check in dbuf_findbp() that checks if the DVA that will > be returned has a small (=<16) vdev number, and if not, returns EIO. > This just results in ZFS returning I/O errors for any of the corrupt > files I try to access, which at least lets us get at our data for now. > > My suspicion is that somehow, bad data is getting interpreted as > a block pointer/shift constant, and this sends ZFS into the woods. > I haven't been able to track down how this data could get past > checksum verification, especially with RAIDZ. > > Backtraces: > > (both crashes due to vdev_is_dead() dereferencing a null pointer) > > Scrub crash: > http://wsyntax.com/~raymond/zfs/zfs-scrub-bt.txt > > Prefetch off, ls -al of "/06/chunk_0000000001417E06_00000001.mfs": > http://wsyntax.com/~raymond/zfs/zfs-ls-bt.txt > > Regards, > Raymond Jimenez > > [1] http://wsyntax.com/~raymond/zfs/zfs-dva-corrupt-workaround.patch For one reason or the other wrong data (but correct looking - proper checksums, etc) got written to the disk. I'd say use the patch, lift the data and re-create the pool. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 11:13:33 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4480DE71 for ; Tue, 27 Nov 2012 11:13:33 +0000 (UTC) (envelope-from ale@FreeBSD.org) Received: from andxor.it (relay.andxor.it [195.223.2.3]) by mx1.freebsd.org (Postfix) with SMTP id 5D0FF8FC08 for ; Tue, 27 Nov 2012 11:13:28 +0000 (UTC) Received: (qmail 22793 invoked from network); 27 Nov 2012 11:13:04 -0000 Received: from unknown (HELO alex.andxor.it) (192.168.2.30) by andxor.it with SMTP; 27 Nov 2012 11:13:04 -0000 Message-ID: <50B4A040.6060001@FreeBSD.org> Date: Tue, 27 Nov 2012 12:13:04 +0100 From: Alex Dupre User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Firefox/17.0 SeaMonkey/2.14 MIME-Version: 1.0 To: =?ISO-8859-1?Q?Ermal_Lu=E7i?= Subject: Re: Re-sizable UFS project References: In-Reply-To: X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 11:13:33 -0000 Ermal Luçi ha scritto: > some time ago the FreeBSD Foundation published/approved a project for live > resizing of UFS filesystems. > Does any know if the project was successful and any outcome from it? http://lists.freebsd.org/pipermail/svn-src-head/2012-November/042539.html -- Alex Dupre From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 13:50:41 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CA1E8A89; Tue, 27 Nov 2012 13:50:41 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: from mail-qc0-f182.google.com (mail-qc0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 21AD68FC13; Tue, 27 Nov 2012 13:50:40 +0000 (UTC) Received: by mail-qc0-f182.google.com with SMTP id k19so11219457qcs.13 for ; Tue, 27 Nov 2012 05:50:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=VNyaE/Q92yZk54cu1mnOrGVWANmOqFPdMZKUGXK5L0k=; b=TPRure96C4F41MSv985wQ13DcK+bfoUDDacLtYmZLduGRSzO8XlQQ802HWFhnGJuiW vRNjnBN74zsmAWW9XZUGkd1KU7Zx43KNS4PIRgdLfIiHmAlVzzJqSjfz/OhlnThl3cEG HclEpwkOvjJCZqg0u3yo8honvZEnRyPy16H0mZKzKFLDEeiQxoU4BcZD8qCR+6nnCfbm CzhYBALlTOIicw6dEZlreBsffSsoISF1TtKqeKKKuirQ36i6RmleUi6FNMNk0qmBUZsx 4vXsszqtnQJavbgNCYd6en4ZtQEPkNLc7S5CbUrkqKsL1iRRyMaCgk8Y0BVW3UnDX8lf Pt9Q== MIME-Version: 1.0 Received: by 10.49.48.111 with SMTP id k15mr17956216qen.28.1354024240315; Tue, 27 Nov 2012 05:50:40 -0800 (PST) Received: by 10.49.121.163 with HTTP; Tue, 27 Nov 2012 05:50:40 -0800 (PST) In-Reply-To: <50B4A040.6060001@FreeBSD.org> References: <50B4A040.6060001@FreeBSD.org> Date: Tue, 27 Nov 2012 14:50:40 +0100 Message-ID: Subject: Re: Re-sizable UFS project From: =?ISO-8859-1?Q?Ermal_Lu=E7i?= To: Alex Dupre Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 13:50:41 -0000 On Tue, Nov 27, 2012 at 12:13 PM, Alex Dupre wrote: > Ermal Lu=E7i ha scritto: > > some time ago the FreeBSD Foundation published/approved a project for > live > > resizing of UFS filesystems. > > Does any know if the project was successful and any outcome from it? > > http://lists.freebsd.org/pipermail/svn-src-head/2012-November/042539.html > > -- > Alex Dupre > Thank you, somehow my search terms must have been wrong. --=20 Ermal From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 14:02:12 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B22BD1F6; Tue, 27 Nov 2012 14:02:12 +0000 (UTC) (envelope-from tomek.cedro@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id CB4E08FC0C; Tue, 27 Nov 2012 14:02:11 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id go10so9297609lbb.13 for ; Tue, 27 Nov 2012 06:02:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=MYK4Ewm35cLl769wwbOJ3rCl9StZfMEAerkfsvSBOzY=; b=xWCYb3xh2PA/36uDwKx1FhaUDb6fK0hPwWYg52gZf6HrM+vJEqfI7YbRgBlTBsB4lf SM1XjRRL4NQuQiIW/9V8a2L28NivgIKcpJlNMwYAClr9FJnFKWmhAaRu2L66tWILmm4q KRCvj1Z+6dUPD/a658fgIq3a2Yibo4HDKH0VlUo8FnhKYvcPLjwG40D99WUJS78nU1Zh jxk7u6EBEAeSpNvjbCTSomB9UX2Db2eurHNuMngzApTzBuuZsZi+OvFbyLyMsvWzfYIJ Lp0EaKPdNnjZGMMnxE+wJdNXzhw4ZVNJrqNIX3iMjrpQYZbAL1pF5dz3C1AjdRRmx5CE BiyA== MIME-Version: 1.0 Received: by 10.112.87.40 with SMTP id u8mr6632581lbz.50.1354024930146; Tue, 27 Nov 2012 06:02:10 -0800 (PST) Sender: tomek.cedro@gmail.com Received: by 10.114.12.226 with HTTP; Tue, 27 Nov 2012 06:02:09 -0800 (PST) In-Reply-To: References: <50B4A040.6060001@FreeBSD.org> Date: Tue, 27 Nov 2012 15:02:09 +0100 X-Google-Sender-Auth: phXZSAJ5H1ymzDeW1O3aNy-o9Vs Message-ID: Subject: Re: Re-sizable UFS project From: CeDeROM To: =?UTF-8?Q?Ermal_Lu=C3=A7i?= Content-Type: text/plain; charset=UTF-8 Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org, Alex Dupre X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 14:02:12 -0000 Btw. are there any projects to make UFS natively available (something like fs-driver for ExtFS) on platforms such as Windows, Linux, MacOS? It would be nice to have native UFS instead Ext2 as universal filesystem among these operating systems... :-) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 16:41:32 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F1F54E12 for ; Tue, 27 Nov 2012 16:41:32 +0000 (UTC) (envelope-from kiwi@oav.net) Received: from mail02.oav.net (unknown [IPv6:2001:67c:ec:100::25:2]) by mx1.freebsd.org (Postfix) with ESMTP id 76E258FC13 for ; Tue, 27 Nov 2012 16:41:32 +0000 (UTC) Received: from amavis1.local.oav.net (amavis1.local.oav.net [IPv6:2001:67c:ec:100::25:41]) by mail02.oav.net (Postfix) with ESMTP id 5810261C77 for ; Tue, 27 Nov 2012 17:41:24 +0100 (CET) (envelope-from kiwi@oav.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=oav.net; h= content-transfer-encoding:content-type:content-type:in-reply-to :references:subject:subject:mime-version:user-agent:from:from :date:date:message-id:received:received; s=a; t=1354034478; x= 1355848879; bh=cWYPTi7AKZUWEMpoDnxZOCvMJtYglUlCpu3hBYiXKO4=; b=T +vWktsPXIH560KQlQlJmzxkMtCkwFIBNEw+N46Lo41znK2mZAv3aTz3hJLI5DxbL G7x+szNK9HEZSH8bHNLJ4yFQRheOIaFUDqyB9VFZMPylBGDZAppXclZ2R8ns202/ 9pY0e4mdWwE2bbT+L6HdYWqYW7HekkwomSo/cMEi4g= X-Virus-Scanned: Amavisd-new at amavis1.local.oav.net.local.oav.net Received: from mail02.oav.net ([172.31.1.2]) by amavis1.local.oav.net (amavis1.local.oav.net [172.31.1.41]) (amavisd-new, port 10026) with LMTP id li0CC9gTqlsB for ; Tue, 27 Nov 2012 17:41:18 +0100 (CET) Received: from calchas.nic.fr (calchas.tech.ipv6.nic.fr [IPv6:2001:67c:2219:7::86:40]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: kiwi@oav.net) by mail02.oav.net (Postfix) with ESMTPSA id 3A44061C76 for ; Tue, 27 Nov 2012 17:41:18 +0100 (CET) (envelope-from kiwi@oav.net) Message-ID: <50B4EC4A.1030107@oav.net> Date: Tue, 27 Nov 2012 17:37:30 +0100 From: Xavier Beaudouin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: Re-sizable UFS project References: <50B4A040.6060001@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 16:41:33 -0000 Hello, On 11/27/12 15:02, CeDeROM wrote: > Btw. are there any projects to make UFS natively available (something > like fs-driver for ExtFS) on platforms such as Windows, Linux, MacOS? > It would be nice to have native UFS instead Ext2 as universal > filesystem among these operating systems... :-) It is a long time I didn't recompile a Linux kernel, but there is a UFS driver (kernel mode) on linux kernel (I was on 2.6.32, as far as I remember). There is also UFS on OSX... But it is toooo slow for OS X and case sensitive that make it ... unusable on OS X. I think we lost UFS on OSX recently (I think when OS X Lion came out).... but I don't know if OS X UFS is compatible with FreeBSD's. Xavier From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 18:25:22 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 09BBFB0D for ; Tue, 27 Nov 2012 18:25:22 +0000 (UTC) (envelope-from josh@signalboxes.net) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id AEFCA8FC16 for ; Tue, 27 Nov 2012 18:25:21 +0000 (UTC) Received: by mail-ob0-f182.google.com with SMTP id 16so14466068obc.13 for ; Tue, 27 Nov 2012 10:25:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type :x-gm-message-state; bh=Qaa5KW9Nwv2Ega0CwGnpSxU8Aa/euVyzqBJA2OAvP9g=; b=m1YgeeHl1LYNmXwmpQdFcNGCk6VovzoAT5Y5zjKA3gCy61AzmEEJ0HsRqABx0Ijt3G WnK9OqJSdcI/lS1fAqB+H8oYlaNPNxLuhhrL0gERil2Ee9+6ruiokAmSGgno4OoZrXW/ 9hdzqmS5iZBfXCdN6L9kcCwFFrzBnbcHn5PJUpt26WUHsfKdeU1yoAEu5WgbvfO3976Z n2BOoZxbvT1voXqtJjpZPyj8ESYh7LvNF/n9OBZ0Sf+/j56+TTtUli6E3/uwSQq3JsIn m5vJlVTrVAccld+rL6G2SVmt9BKH/vwlZNXTZi0aDFogVzN2PBOTIlDh7gm1u2JIjSfK aQug== Received: by 10.60.171.164 with SMTP id av4mr13642238oec.59.1354040720479; Tue, 27 Nov 2012 10:25:20 -0800 (PST) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx.google.com with ESMTPS id o5sm13172616oec.0.2012.11.27.10.25.19 (version=SSLv3 cipher=OTHER); Tue, 27 Nov 2012 10:25:19 -0800 (PST) Received: by mail-ob0-f182.google.com with SMTP id 16so14466028obc.13 for ; Tue, 27 Nov 2012 10:25:18 -0800 (PST) MIME-Version: 1.0 Received: by 10.60.31.234 with SMTP id d10mr14070686oei.123.1354040718481; Tue, 27 Nov 2012 10:25:18 -0800 (PST) Received: by 10.60.14.194 with HTTP; Tue, 27 Nov 2012 10:25:18 -0800 (PST) Date: Tue, 27 Nov 2012 11:25:18 -0700 Message-ID: Subject: ZFS: Panic when attempting to delete certain data From: Josh Beard To: freebsd-fs@freebsd.org X-Gm-Message-State: ALoCoQkTRN/7rIHYpHy8KLUXVu92B20xVQl89lIwaBqQqpTSeLPsv5eM/Xz2NRyMJGExsIc7xAYh Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 18:25:22 -0000 Hello, I have a system that I can consistently reproduce a panic on when trying to delete certain data. The data is data that was rsynced from another system - nothing terribly unique. This has been ongoing from several months, starting with 9.0-RELEASE and now running 9.1-RC3. I can't find anything in common with the files that I can trigger the panics with. One is a simple gzipped archive where some are plain text. Strangely, I can only reproduce it with data that was rsynced from that particular system (which is a Mac). I seriously doubt it's hardware at this point, as virtually every piece of hardware in that system has been replaced (including motherboard and drives). That said, the zpools were rebuilt from scratch when the drives were replaced and the issue persists. I can't seem to trigger it with other actions, such as chmod, chown, or even mv. Simply attempting to unlink the files seems to do it. # uname -a (I can reproduce on a GENERIC kernel, too). FreeBSD bksys1 9.1-RC3 FreeBSD 9.1-RC3 #0 r242591: Sun Nov 4 19:17:25 MST 2012 root@bksys1:/usr/obj/usr/src/sys/BKSYS191 amd64 zpool version is 28; zfs version is 5. /boot/loader.conf doesn't have anything related in it, and an empty one produces the same results. zpool scrubs are done weekly and have returned no errors (most recent was 3 days ago). Any insight is very appreciated! Josh The message: Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 05 fault virtual address = 0x160 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80ebd45a stack pointer = 0x28:0xffffff8466534850 frame pointer = 0x28:0xffffff8466534910 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 3245 (rm) trap number = 12 panic: page fault cpuid = 3 KDB: stack backtrace: #0 0xffffffff80585c28 at kdb_backtrace+0x68 #1 0xffffffff805502cb at panic+0x21b #2 0xffffffff807a9fad at trap_fatal+0x39d #3 0xffffffff807aa0f0 at trap_pfault+0x120 #4 0xffffffff807aa7e9 at trap+0x3d9 #5 0xffffffff80794f4f at calltrap+0x8 #6 0xffffffff8081cf13 at VOP_REMOVE_APV+0x53 #7 0xffffffff805ed355 at kern_unlinkat+0x265 #8 0xffffffff805ed419 at kern_unlink+0x19 #9 0xffffffff805ed431 at sys_unlink+0x11 #10 0xffffffff807a95bd at amd64_syscall+0x2fd #11 0xffffffff80795237 at Xfast_syscall+0xf7 Uptime: 14m42s Dumping 2432 out of 16361 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from /boot/kernel/coretemp.ko.symbols...done. done. Loaded symbols for /boot/kernel/coretemp.ko Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/if_lagg.ko...Reading symbols from /boot/kernel/if_lagg.ko.symbols...done. done. Loaded symbols for /boot/kernel/if_lagg.ko Reading symbols from /boot/kernel/ng_ubt.ko...Reading symbols from /boot/kernel/ng_ubt.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_ubt.ko Reading symbols from /boot/kernel/ng_hci.ko...Reading symbols from /boot/kernel/ng_hci.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_hci.ko Reading symbols from /boot/kernel/ng_bluetooth.ko...Reading symbols from /boot/kernel/ng_bluetooth.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_bluetooth.ko Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /boot/kernel/netgraph.ko.symbols...done. done. Loaded symbols for /boot/kernel/netgraph.ko Reading symbols from /boot/kernel/ng_l2cap.ko...Reading symbols from /boot/kernel/ng_l2cap.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_l2cap.ko Reading symbols from /boot/kernel/ng_btsocket.ko...Reading symbols from /boot/kernel/ng_btsocket.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_btsocket.ko Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /boot/kernel/ng_socket.ko.symbols...done. done. Loaded symbols for /boot/kernel/ng_socket.ko Reading symbols from /boot/kernel/blank_saver.ko...Reading symbols from /boot/kernel/blank_saver.ko.symbols...done. done. Loaded symbols for /boot/kernel/blank_saver.ko #0 doadump (textdump=Variable "textdump" is not available. ) at pcpu.h:224 224 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=Variable "textdump" is not available. ) at pcpu.h:224 #1 0xffffffff8054ff87 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:448 #2 0xffffffff8055030f in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:636 #3 0xffffffff807a9fad in trap_fatal (frame=0xffffff84665347a0, eva=352) at /usr/src/sys/amd64/amd64/trap.c:857 #4 0xffffffff807aa0f0 in trap_pfault (frame=0xffffff84665347a0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:714 #5 0xffffffff807aa7e9 in trap (frame=0xffffff84665347a0) at /usr/src/sys/amd64/amd64/trap.c:456 #6 0xffffffff80794f4f in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228 #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 #8 0xffffffff8081cf13 in VOP_REMOVE_APV (vop=Variable "vop" is not available. ) at vnode_if.c:1333 #9 0xffffffff805ed355 in kern_unlinkat (td=0xfffffe000c4b1000, fd=-100, path=0x7fffffffdb2e
, pathseg=UIO_USERSPACE, oldinum=0) at vnode_if.h:575 #10 0xffffffff805ed419 in kern_unlink (td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_syscalls.c:1897 #11 0xffffffff805ed431 in sys_unlink (td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_syscalls.c:1867 #12 0xffffffff807a95bd in amd64_syscall (td=0xfffffe000c4b1000, traced=0) at subr_syscall.c:135 #13 0xffffffff80795237 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:387 #14 0x00000008009100bc in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) --- atapci0: port 0x2018-0x201f,0x2024-0x2027,0x2010-0x2017,0x2020-0x2023,0x2000-0x200f mem 0xf6100000-0xf61003ff irq 19 at device 0.0 on pci3 ahci0: at channel -1 on atapci0 ahci0: AHCI v1.00 with 4 3Gbps ports, Port Multiplier supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ata2: at channel 0 on atapci0 ahci1: port 0x4068-0x406f,0x4074-0x4077,0x4060-0x4067,0x4070-0x4073,0x4020-0x403f mem 0xf6325000-0xf63257ff irq 19 at device 31.2 on pci0 ahci1: AHCI v1.30 with 6 3Gbps ports, Port Multiplier not supported ada0 at ahcich1 bus 0 scbus1 target 0 lun 0 ada0: ATA-8 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad6 ada1 at ahcich4 bus 0 scbus5 target 0 lun 0 ada1: ATA-8 SATA 3.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada1: Previously was known as ad14 ada2 at ahcich5 bus 0 scbus6 target 0 lun 0 ada2: ATA-8 SATA 3.x device ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada2: Previously was known as ad16 ada3 at ahcich6 bus 0 scbus7 target 0 lun 0 ada3: ATA-8 SATA 3.x device ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada3: Previously was known as ad18 ada4 at ahcich7 bus 0 scbus8 target 0 lun 0 ada4: ATA-8 SATA 3.x device ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada4: Command Queueing enabled ada4: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada4: Previously was known as ad20 ada5 at ahcich8 bus 0 scbus9 target 0 lun 0 ada5: ATA-8 SATA 3.x device ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada5: Command Queueing enabled ada5: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada5: Previously was known as ad22 ada6 at ahcich9 bus 0 scbus10 target 0 lun 0 ada6: ATA-8 SATA 3.x device ada6: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada6: Command Queueing enabled ada6: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada6: Previously was known as ad24 From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 18:34:45 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 179B2D15 for ; Tue, 27 Nov 2012 18:34:45 +0000 (UTC) (envelope-from stevenschlansker@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id D50A28FC13 for ; Tue, 27 Nov 2012 18:34:44 +0000 (UTC) Received: by mail-pb0-f54.google.com with SMTP id wz12so9318606pbc.13 for ; Tue, 27 Nov 2012 10:34:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=9lfeYte7KN6FtZRjBOSdYMaHJL8rnYnYDc6mwZJN98M=; b=tZzvbwRRq9MAIG1qgyP4PSEhT5y4p7t2cAlb4QoDNfT16VmX3Bmlphva5eUFNrc2Ua QtDsozdcpyMKCRjGKh4QpqTBZYPK8nKtWuEJ5xz3LhwSsrqxlfiZfh3GGCHtRwZ2f4tN xlgsPNvZRnR7xWmdt4ieNpMmeT/4qTLJ6dWwJsRTmgSZGhBcu0uJykc86c+lAlotfX7o 25JujC3Wj3NVJts0zPhTIF/IzSIVv0n+yTOzVGSQr+taYw7TUNC+gJm6iMbmJu3sHMHt /5cpSqDB3TML6m8nm/ynSJhQWAmGbT5Bb5KGrusvu6moroamCfrDiS19gKSdF9i/fgc0 zRCg== Received: by 10.68.231.97 with SMTP id tf1mr49984897pbc.149.1354041284285; Tue, 27 Nov 2012 10:34:44 -0800 (PST) Received: from nessie.corp.trumpet.io ([207.86.77.58]) by mx.google.com with ESMTPS id ak10sm11009463pbd.24.2012.11.27.10.34.38 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 27 Nov 2012 10:34:39 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: ZFS: Panic when attempting to delete certain data From: Steven Schlansker In-Reply-To: Date: Tue, 27 Nov 2012 10:34:37 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <3F21E539-BBA9-4F9A-A7A0-7808042C600D@gmail.com> References: To: Josh Beard X-Mailer: Apple Mail (2.1499) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 18:34:45 -0000 On Nov 27, 2012, at 10:25 AM, Josh Beard wrote: > Hello, >=20 > I have a system that I can consistently reproduce a panic on when = trying to > delete certain data. The data is data that was rsynced from another = system > - nothing terribly unique. This has been ongoing from several months, > starting with 9.0-RELEASE and now running 9.1-RC3. > Do you have dedupe turned on? I (and others, from what I've gathered) = had a lot of trouble with panics and system hangs when deleting deduped data with insufficient = RAM. If that is the case, my only recourse turned out to be getting enough = RAM in the box to offload the data, destroy the filesystem, and start over=85 From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 18:35:31 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CBA35D8D for ; Tue, 27 Nov 2012 18:35:31 +0000 (UTC) (envelope-from jusher71@yahoo.com) Received: from nm5.bullet.mail.ne1.yahoo.com (nm5.bullet.mail.ne1.yahoo.com [98.138.90.68]) by mx1.freebsd.org (Postfix) with ESMTP id 7C8138FC0C for ; Tue, 27 Nov 2012 18:35:31 +0000 (UTC) Received: from [98.138.226.180] by nm5.bullet.mail.ne1.yahoo.com with NNFMP; 27 Nov 2012 18:32:10 -0000 Received: from [98.138.89.249] by tm15.bullet.mail.ne1.yahoo.com with NNFMP; 27 Nov 2012 18:32:10 -0000 Received: from [127.0.0.1] by omp1041.mail.ne1.yahoo.com with NNFMP; 27 Nov 2012 18:32:10 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 91992.53931.bm@omp1041.mail.ne1.yahoo.com Received: (qmail 25298 invoked by uid 60001); 27 Nov 2012 18:32:10 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1354041130; bh=VTSCwuONGgrudTv6Nbu5QaIJH9PdqgXFzC/K7sXbzZE=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:MIME-Version:Content-Type; b=JBYXk+yGrN5H8iFQMwccMJEoj5rgwTcXe9Yl+0L9qykuQpw9NYn4SCcVmQNV4SsXtfXeJ/tKmB5o0JA++T2B3omEw3kkL4vOL6GUmbjOBdMxx+nlg9EIZ07sihlzPNi2nX7+RIj4ZdohiJHSChRzTSAZcN/GodGmrq03UPoQ7no= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:MIME-Version:Content-Type; b=rOy/xhGZFhwgMaW3t7QJyZUoIDdSl22g64C4Q/uNW/H+lM4Qd/TXV7mjdKcfFpBXQDmLrw39/4NBil6fYHcaehuUv9fOSInUzMvTIV695TfCuV40CqcNinTJaBo8pMrAUPKfZTCHvdBh7G1RmQBpj3BJU+tuhQVmCFp6H3AywLw=; X-YMail-OSG: 83LStd0VM1nHX3ui_I5HXHZnmPlfA5PIpUqpMbIcm3hpIJF 7EThmAb8MYlZP4XrQdD3Q2Uga23SOcK3w6GYoU5q8Ywd6z7.25qlWn7nSKn5 _COBkMYsDZDygdzWiNszyaKmqdZALsJ7kW6DNVlTdtARCXasrzQYc2jHe8eZ CBKanzYk38Ehpuk3NI8w31kqNrUyimvE3A5hCqTgxJaJDoHd8zdrywD7v5Vq y4L8AObJScpcyt.ma0CA5JAJMuNbVBsmB8hfeCPdRqFT66zforzB9rCzZwyG sKfmIAdbWKZ28qW5p1pVUYoJIcGw9hUGatX3R3PJ5LR05kpyw2b3UMoPfHOK 07I48SBzSvJt_94WpDCfqjcikM2eGjMDqdNbkU3XC90njk24sKRkv.CNqXnQ 8SrPPBS9suuq7Nmjcq3.Ofrhq56M5JqvJ.2J4IYDYWzIJHGIxxvnacvOU Received: from [173.164.238.34] by web120705.mail.ne1.yahoo.com via HTTP; Tue, 27 Nov 2012 10:32:09 PST X-Rocket-MIMEInfo: 001.001, V2UgY3JlYXRlZCBhIGZldyA4LjMtUkVMRUFTRSBzeXN0ZW1zIHdpdGggYm9vdCBtaXJyb3JzIHVzaW5nIG9uYm9hcmQgSW50ZWwgSUNIMTBSLiAgVGhlIHN5c3RlbSBwaWNrcyB1cCB0aGUgbWlycm9yIHdpdGggImF0YXJhaWQiIGFuZCBJIHNlZSBpdCBpbiBkbWVzZyBsaWtlIHRoaXM6CgphcjA6IDExNDMxMk1CIDxEREYgUkFJRDE.IHN0YXR1czogUkVBRFkKCkJ1dCBpdCBzZWVtcyB0aGF0IGdyYWlkIGlzIGEgbXVjaCBiZXR0ZXIgb3B0aW9uLCBhbmQgYXRhcmFpZCBoYXMgcHJvYmxlbXMgKEkgYW0gc2VlaW4BMAEBAQE- X-Mailer: YahooMailClassic/15.0.8 YahooMailWebService/0.8.127.475 Message-ID: <1354041129.24091.YahooMailClassic@web120705.mail.ne1.yahoo.com> Date: Tue, 27 Nov 2012 10:32:09 -0800 (PST) From: Jason Usher Subject: Can we migrate a live ataraid system to graid ? To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 18:35:32 -0000 We created a few 8.3-RELEASE systems with boot mirrors using onboard Intel ICH10R. The system picks up the mirror with "ataraid" and I see it in dmesg like this: ar0: 114312MB status: READY But it seems that graid is a much better option, and ataraid has problems (I am seeing chatter on freebsd-fs and other places about problems). Is it possible for us to migrate this boot mirror to graid ? Can we just leave the mirror as-is (since it was bios formatted) and just start using graid ? Has anyone done this switch on a live system, or is it impossible (and we need to reformat and start over) ? Thanks. From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 18:48:40 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BEC165CF for ; Tue, 27 Nov 2012 18:48:40 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id CCAD88FC08 for ; Tue, 27 Nov 2012 18:48:39 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id UAA16140; Tue, 27 Nov 2012 20:48:36 +0200 (EET) (envelope-from avg@FreeBSD.org) Message-ID: <50B50B04.8020109@FreeBSD.org> Date: Tue, 27 Nov 2012 20:48:36 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Josh Beard Subject: Re: ZFS: Panic when attempting to delete certain data References: In-Reply-To: X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 18:48:41 -0000 on 27/11/2012 20:25 Josh Beard said the following: > Hello, > > I have a system that I can consistently reproduce a panic on when trying to > delete certain data. The data is data that was rsynced from another system > - nothing terribly unique. This has been ongoing from several months, > starting with 9.0-RELEASE and now running 9.1-RC3. > > I can't find anything in common with the files that I can trigger the > panics with. One is a simple gzipped archive where some are plain text. > Strangely, I can only reproduce it with data that was rsynced from that > particular system (which is a Mac). Josh, I am collecting these cases, thank you for another one. I had an interesting investigation of http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/173747 Unfortunately, for some reason the whole conversation stayed private. I see that also opened a PR earlier: http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/170238 Could you please provide the following info? >From kgdb: - list in frame 7 (zfs_freebsd_remove), so that I can see the code line - local variables from frame 7 (info local) Also, for one of the files that trigger the problem: - ls -i to obtain its inode number - zdb -ddddd Thank you. > I seriously doubt it's hardware at this point, as virtually every piece of > hardware in that system has been replaced (including motherboard and > drives). That said, the zpools were rebuilt from scratch when the drives > were replaced and the issue persists. > > I can't seem to trigger it with other actions, such as chmod, chown, or > even mv. Simply attempting to unlink the files seems to do it. > > # uname -a (I can reproduce on a GENERIC kernel, too). > FreeBSD bksys1 9.1-RC3 FreeBSD 9.1-RC3 #0 r242591: Sun Nov 4 19:17:25 MST > 2012 root@bksys1:/usr/obj/usr/src/sys/BKSYS191 amd64 > > zpool version is 28; zfs version is 5. > > /boot/loader.conf doesn't have anything related in it, and an empty one > produces the same results. > > zpool scrubs are done weekly and have returned no errors (most recent was 3 > days ago). > > Any insight is very appreciated! > > Josh > > > The message: > Fatal trap 12: page fault while in kernel mode > cpuid = 3; apic id = 05 > fault virtual address = 0x160 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff80ebd45a > stack pointer = 0x28:0xffffff8466534850 > frame pointer = 0x28:0xffffff8466534910 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 3245 (rm) > trap number = 12 > panic: page fault > cpuid = 3 > KDB: stack backtrace: > #0 0xffffffff80585c28 at kdb_backtrace+0x68 > #1 0xffffffff805502cb at panic+0x21b > #2 0xffffffff807a9fad at trap_fatal+0x39d > #3 0xffffffff807aa0f0 at trap_pfault+0x120 > #4 0xffffffff807aa7e9 at trap+0x3d9 > #5 0xffffffff80794f4f at calltrap+0x8 > #6 0xffffffff8081cf13 at VOP_REMOVE_APV+0x53 > #7 0xffffffff805ed355 at kern_unlinkat+0x265 > #8 0xffffffff805ed419 at kern_unlink+0x19 > #9 0xffffffff805ed431 at sys_unlink+0x11 > #10 0xffffffff807a95bd at amd64_syscall+0x2fd > #11 0xffffffff80795237 at Xfast_syscall+0xf7 > Uptime: 14m42s > Dumping 2432 out of 16361 > MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > > Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from > /boot/kernel/coretemp.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/coretemp.ko > Reading symbols from /boot/kernel/zfs.ko...Reading symbols from > /boot/kernel/zfs.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/zfs.ko > Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from > /boot/kernel/opensolaris.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/opensolaris.ko > Reading symbols from /boot/kernel/if_lagg.ko...Reading symbols from > /boot/kernel/if_lagg.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/if_lagg.ko > Reading symbols from /boot/kernel/ng_ubt.ko...Reading symbols from > /boot/kernel/ng_ubt.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/ng_ubt.ko > Reading symbols from /boot/kernel/ng_hci.ko...Reading symbols from > /boot/kernel/ng_hci.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/ng_hci.ko > Reading symbols from /boot/kernel/ng_bluetooth.ko...Reading symbols from > /boot/kernel/ng_bluetooth.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/ng_bluetooth.ko > Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from > /boot/kernel/netgraph.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/netgraph.ko > Reading symbols from /boot/kernel/ng_l2cap.ko...Reading symbols from > /boot/kernel/ng_l2cap.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/ng_l2cap.ko > Reading symbols from /boot/kernel/ng_btsocket.ko...Reading symbols from > /boot/kernel/ng_btsocket.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/ng_btsocket.ko > Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from > /boot/kernel/ng_socket.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/ng_socket.ko > Reading symbols from /boot/kernel/blank_saver.ko...Reading symbols from > /boot/kernel/blank_saver.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/blank_saver.ko > #0 doadump (textdump=Variable "textdump" is not available. > ) at pcpu.h:224 > 224 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) #0 doadump (textdump=Variable "textdump" is not available. > ) at pcpu.h:224 > #1 0xffffffff8054ff87 in kern_reboot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:448 > #2 0xffffffff8055030f in panic (fmt=Variable "fmt" is not available. > ) > at /usr/src/sys/kern/kern_shutdown.c:636 > #3 0xffffffff807a9fad in trap_fatal (frame=0xffffff84665347a0, eva=352) > at /usr/src/sys/amd64/amd64/trap.c:857 > #4 0xffffffff807aa0f0 in trap_pfault (frame=0xffffff84665347a0, usermode=0) > at /usr/src/sys/amd64/amd64/trap.c:714 > #5 0xffffffff807aa7e9 in trap (frame=0xffffff84665347a0) > at /usr/src/sys/amd64/amd64/trap.c:456 > #6 0xffffffff80794f4f in calltrap () > at /usr/src/sys/amd64/amd64/exception.S:228 > #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not > available. > ) > at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 > #8 0xffffffff8081cf13 in VOP_REMOVE_APV (vop=Variable "vop" is not > available. > ) at vnode_if.c:1333 > #9 0xffffffff805ed355 in kern_unlinkat (td=0xfffffe000c4b1000, fd=-100, > path=0x7fffffffdb2e
, > pathseg=UIO_USERSPACE, oldinum=0) at vnode_if.h:575 > #10 0xffffffff805ed419 in kern_unlink (td=Variable "td" is not available. > ) > at /usr/src/sys/kern/vfs_syscalls.c:1897 > #11 0xffffffff805ed431 in sys_unlink (td=Variable "td" is not available. > ) > at /usr/src/sys/kern/vfs_syscalls.c:1867 > #12 0xffffffff807a95bd in amd64_syscall (td=0xfffffe000c4b1000, traced=0) > at subr_syscall.c:135 > #13 0xffffffff80795237 in Xfast_syscall () > at /usr/src/sys/amd64/amd64/exception.S:387 > #14 0x00000008009100bc in ?? () > Previous frame inner to this frame (corrupt stack?) > (kgdb) [snip] -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 20:32:30 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 430DC714 for ; Tue, 27 Nov 2012 20:32:30 +0000 (UTC) (envelope-from josh@signalboxes.net) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id E152F8FC15 for ; Tue, 27 Nov 2012 20:32:29 +0000 (UTC) Received: by mail-ob0-f182.google.com with SMTP id 16so14634382obc.13 for ; Tue, 27 Nov 2012 12:32:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=E9Ooz3AMoQydQUUD+PQF6sAY4qPAIEbSum2rYxWUeB4=; b=CKjbTiW/8yZBUIB3ONE6VOShCjREz3WvK1tvrPnSkkmGjB7ooFDsnDvT3YHmb52eNi E7dsJ7mynhpUPEMaKClPO7P52WDdPux+KYoOREAy9525tRduxW3hXHrW9a1H+3MEZzy6 LScv0ea+EDr0AwKuutVgd6Fbh9UZlTslwgRGlk40A0Cu9i0uLbscugQ8UaDumLzDBkwl jeXimsefxzbZWCNDquXyKB1VlY6dSiHD3h8DyuMyrcf7VJoAh4DMX+lHEHU3eg03/hMQ jMZTgEQYlNjEmqVZEdsSAMvIGjkbiDWbpJnBuh1giTGDIqivdWTcUNMj7yhGgmJgQb/v 2iGw== Received: by 10.182.78.228 with SMTP id e4mr893672obx.77.1354048348254; Tue, 27 Nov 2012 12:32:28 -0800 (PST) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx.google.com with ESMTPS id m3sm15818148obm.21.2012.11.27.12.32.26 (version=SSLv3 cipher=OTHER); Tue, 27 Nov 2012 12:32:27 -0800 (PST) Received: by mail-oa0-f54.google.com with SMTP id n9so16520628oag.13 for ; Tue, 27 Nov 2012 12:32:26 -0800 (PST) MIME-Version: 1.0 Received: by 10.60.169.171 with SMTP id af11mr9739803oec.92.1354048346084; Tue, 27 Nov 2012 12:32:26 -0800 (PST) Received: by 10.60.14.194 with HTTP; Tue, 27 Nov 2012 12:32:25 -0800 (PST) In-Reply-To: <50B50B04.8020109@FreeBSD.org> References: <50B50B04.8020109@FreeBSD.org> Date: Tue, 27 Nov 2012 13:32:25 -0700 Message-ID: Subject: Re: ZFS: Panic when attempting to delete certain data From: Josh Beard To: Andriy Gapon X-Gm-Message-State: ALoCoQkhcZM8cfi/cbefpXwmyfcvsjgm3Cqyw4D6IUzdKirZvBKJilZ4/SqdYV0az4w1VCmW2L2i Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 20:32:30 -0000 On Tue, Nov 27, 2012 at 11:48 AM, Andriy Gapon wrote: > on 27/11/2012 20:25 Josh Beard said the following: > > Hello, > > > > I have a system that I can consistently reproduce a panic on when trying > to > > delete certain data. The data is data that was rsynced from another > system > > - nothing terribly unique. This has been ongoing from several months, > > starting with 9.0-RELEASE and now running 9.1-RC3. > > > > I can't find anything in common with the files that I can trigger the > > panics with. One is a simple gzipped archive where some are plain text. > > Strangely, I can only reproduce it with data that was rsynced from that > > particular system (which is a Mac). > > Josh, > > I am collecting these cases, thank you for another one. > I had an interesting investigation of > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/173747 > Unfortunately, for some reason the whole conversation stayed private. > I see that also opened a PR earlier: > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/170238 > > Could you please provide the following info? > From kgdb: > - list in frame 7 (zfs_freebsd_remove), so that I can see the code line > - local variables from frame 7 (info local) > > > Andriy, Thanks for your quick response. I've never used kgdb, so forgive my ignorance here. Is this what you're looking for? If not, if you could you elaborate on those? #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); (kgdb) list zfs_freebsd_remove 5796 struct vop_remove_args /* { 5797 struct vnode *a_dvp; 5798 struct vnode *a_vp; 5799 struct componentname *a_cnp; 5800 } */ *ap; 5801 { 5802 5803 ASSERT(ap->a_cnp->cn_flags & SAVENAME); 5804 5805 return (zfs_remove(ap->a_dvp, ap->a_cnp->cn_nameptr, (kgdb) info frame 7 Stack frame at 0xffffff8466a6a920: rip = 0xffffffff80ebd45a in zfs_freebsd_remove (/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855); saved rip 0xffffffff8081cf13 called by frame at 0xffffff8466a6a940, caller of frame at 0xffffff8466a6a7a0 source language c. Arglist at 0xffffff8466a6a910, args: ap=Variable "ap" is not available. Also, for one of the files that trigger the problem: > - ls -i to obtain its inode number > - zdb -ddddd > # ls -i kyofilter\ v2.2.pax.gz (this is a symlink. the file that it's linked to does *not* panic the system if I try to delete it). 247126 kyofilter v2.2.pax.gz # zdb -ddddd store/tdxs1 247126 Dataset store/tdxs1 [ZPL], ID 109, cr_txg 35014, 1.33T, 1106389 objects, rootbp DVA[0]=<0:80001a2400:400> DVA[1]=<0:30800610000:400> [L0 DMU objset] fletcher4 lzjb LE contiguous unique double size=800L/200P birth=1166838L/1166838P fill=1106389 cksum=19391f0f67:78eb24a9cca:1439005549d01:275015332d1bdf Object lvl iblk dblk dsize lsize %full type 247126 1 16K 512 0 512 0.00 ZFS plain file 201 bonus System attributes dnode flags: USERUSED_ACCOUNTED dnode maxblkid: 0 path /tech/2012-09-14-01-00/Drivers/Kyocera/.old/C2126.old/Kyocera OS X 10.5+ Web build 2011.01.27.mpkg/Contents/Packages/Kyocera OS X subinstaller.mpkg/Contents/Packages/kyofilter v2.2.pkg/Contents/Resources/kyofilter v2.2.pax.gz uid 1001 gid 80 atime Tue Nov 27 13:27:57 2012 mtime Tue Jul 12 14:17:16 2011 ctime Fri Sep 14 01:05:23 2012 crtime Fri Sep 14 01:04:11 2012 gen 81338 mode 120755 size 17 parent 247122 links 1 pflags 40800000104 xattr 155 Indirect blocks: Thank you. > > > I seriously doubt it's hardware at this point, as virtually every piece > of > > hardware in that system has been replaced (including motherboard and > > drives). That said, the zpools were rebuilt from scratch when the drives > > were replaced and the issue persists. > > > > I can't seem to trigger it with other actions, such as chmod, chown, or > > even mv. Simply attempting to unlink the files seems to do it. > > > > # uname -a (I can reproduce on a GENERIC kernel, too). > > FreeBSD bksys1 9.1-RC3 FreeBSD 9.1-RC3 #0 r242591: Sun Nov 4 19:17:25 > MST > > 2012 root@bksys1:/usr/obj/usr/src/sys/BKSYS191 amd64 > > > > zpool version is 28; zfs version is 5. > > > > /boot/loader.conf doesn't have anything related in it, and an empty one > > produces the same results. > > > > zpool scrubs are done weekly and have returned no errors (most recent > was 3 > > days ago). > > > > Any insight is very appreciated! > > > > Josh > > > > > > The message: > > Fatal trap 12: page fault while in kernel mode > > cpuid = 3; apic id = 05 > > fault virtual address = 0x160 > > fault code = supervisor read data, page not present > > instruction pointer = 0x20:0xffffffff80ebd45a > > stack pointer = 0x28:0xffffff8466534850 > > frame pointer = 0x28:0xffffff8466534910 > > code segment = base 0x0, limit 0xfffff, type 0x1b > > = DPL 0, pres 1, long 1, def32 0, gran 1 > > processor eflags = interrupt enabled, resume, IOPL = 0 > > current process = 3245 (rm) > > trap number = 12 > > panic: page fault > > cpuid = 3 > > KDB: stack backtrace: > > #0 0xffffffff80585c28 at kdb_backtrace+0x68 > > #1 0xffffffff805502cb at panic+0x21b > > #2 0xffffffff807a9fad at trap_fatal+0x39d > > #3 0xffffffff807aa0f0 at trap_pfault+0x120 > > #4 0xffffffff807aa7e9 at trap+0x3d9 > > #5 0xffffffff80794f4f at calltrap+0x8 > > #6 0xffffffff8081cf13 at VOP_REMOVE_APV+0x53 > > #7 0xffffffff805ed355 at kern_unlinkat+0x265 > > #8 0xffffffff805ed419 at kern_unlink+0x19 > > #9 0xffffffff805ed431 at sys_unlink+0x11 > > #10 0xffffffff807a95bd at amd64_syscall+0x2fd > > #11 0xffffffff80795237 at Xfast_syscall+0xf7 > > Uptime: 14m42s > > Dumping 2432 out of 16361 > > MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > > > > Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from > > /boot/kernel/coretemp.ko.symbols...done. > > done. > > Loaded symbols for /boot/kernel/coretemp.ko > > Reading symbols from /boot/kernel/zfs.ko...Reading symbols from > > /boot/kernel/zfs.ko.symbols...done. > > done. > > Loaded symbols for /boot/kernel/zfs.ko > > Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from > > /boot/kernel/opensolaris.ko.symbols...done. > > done. > > Loaded symbols for /boot/kernel/opensolaris.ko > > Reading symbols from /boot/kernel/if_lagg.ko...Reading symbols from > > /boot/kernel/if_lagg.ko.symbols...done. > > done. > > Loaded symbols for /boot/kernel/if_lagg.ko > > Reading symbols from /boot/kernel/ng_ubt.ko...Reading symbols from > > /boot/kernel/ng_ubt.ko.symbols...done. > > done. > > Loaded symbols for /boot/kernel/ng_ubt.ko > > Reading symbols from /boot/kernel/ng_hci.ko...Reading symbols from > > /boot/kernel/ng_hci.ko.symbols...done. > > done. > > Loaded symbols for /boot/kernel/ng_hci.ko > > Reading symbols from /boot/kernel/ng_bluetooth.ko...Reading symbols from > > /boot/kernel/ng_bluetooth.ko.symbols...done. > > done. > > Loaded symbols for /boot/kernel/ng_bluetooth.ko > > Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from > > /boot/kernel/netgraph.ko.symbols...done. > > done. > > Loaded symbols for /boot/kernel/netgraph.ko > > Reading symbols from /boot/kernel/ng_l2cap.ko...Reading symbols from > > /boot/kernel/ng_l2cap.ko.symbols...done. > > done. > > Loaded symbols for /boot/kernel/ng_l2cap.ko > > Reading symbols from /boot/kernel/ng_btsocket.ko...Reading symbols from > > /boot/kernel/ng_btsocket.ko.symbols...done. > > done. > > Loaded symbols for /boot/kernel/ng_btsocket.ko > > Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from > > /boot/kernel/ng_socket.ko.symbols...done. > > done. > > Loaded symbols for /boot/kernel/ng_socket.ko > > Reading symbols from /boot/kernel/blank_saver.ko...Reading symbols from > > /boot/kernel/blank_saver.ko.symbols...done. > > done. > > Loaded symbols for /boot/kernel/blank_saver.ko > > #0 doadump (textdump=Variable "textdump" is not available. > > ) at pcpu.h:224 > > 224 pcpu.h: No such file or directory. > > in pcpu.h > > (kgdb) #0 doadump (textdump=Variable "textdump" is not available. > > ) at pcpu.h:224 > > #1 0xffffffff8054ff87 in kern_reboot (howto=260) > > at /usr/src/sys/kern/kern_shutdown.c:448 > > #2 0xffffffff8055030f in panic (fmt=Variable "fmt" is not available. > > ) > > at /usr/src/sys/kern/kern_shutdown.c:636 > > #3 0xffffffff807a9fad in trap_fatal (frame=0xffffff84665347a0, eva=352) > > at /usr/src/sys/amd64/amd64/trap.c:857 > > #4 0xffffffff807aa0f0 in trap_pfault (frame=0xffffff84665347a0, > usermode=0) > > at /usr/src/sys/amd64/amd64/trap.c:714 > > #5 0xffffffff807aa7e9 in trap (frame=0xffffff84665347a0) > > at /usr/src/sys/amd64/amd64/trap.c:456 > > #6 0xffffffff80794f4f in calltrap () > > at /usr/src/sys/amd64/amd64/exception.S:228 > > #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not > > available. > > ) > > at > > > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 > > #8 0xffffffff8081cf13 in VOP_REMOVE_APV (vop=Variable "vop" is not > > available. > > ) at vnode_if.c:1333 > > #9 0xffffffff805ed355 in kern_unlinkat (td=0xfffffe000c4b1000, fd=-100, > > path=0x7fffffffdb2e
, > > pathseg=UIO_USERSPACE, oldinum=0) at vnode_if.h:575 > > #10 0xffffffff805ed419 in kern_unlink (td=Variable "td" is not available. > > ) > > at /usr/src/sys/kern/vfs_syscalls.c:1897 > > #11 0xffffffff805ed431 in sys_unlink (td=Variable "td" is not available. > > ) > > at /usr/src/sys/kern/vfs_syscalls.c:1867 > > #12 0xffffffff807a95bd in amd64_syscall (td=0xfffffe000c4b1000, traced=0) > > at subr_syscall.c:135 > > #13 0xffffffff80795237 in Xfast_syscall () > > at /usr/src/sys/amd64/amd64/exception.S:387 > > #14 0x00000008009100bc in ?? () > > Previous frame inner to this frame (corrupt stack?) > > (kgdb) > [snip] > > -- > Andriy Gapon > From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 21:13:27 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E0EDC29C for ; Tue, 27 Nov 2012 21:13:27 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 4754B8FC15 for ; Tue, 27 Nov 2012 21:13:26 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id XAA17498; Tue, 27 Nov 2012 23:13:18 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1TdST0-000OBN-7P; Tue, 27 Nov 2012 23:13:18 +0200 Message-ID: <50B52CEC.9080208@FreeBSD.org> Date: Tue, 27 Nov 2012 23:13:16 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Josh Beard Subject: Re: ZFS: Panic when attempting to delete certain data References: <50B50B04.8020109@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 21:13:27 -0000 on 27/11/2012 22:32 Josh Beard said the following: > > > On Tue, Nov 27, 2012 at 11:48 AM, Andriy Gapon > wrote: > > on 27/11/2012 20:25 Josh Beard said the following: > > Hello, > > > > I have a system that I can consistently reproduce a panic on when trying to > > delete certain data. The data is data that was rsynced from another system > > - nothing terribly unique. This has been ongoing from several months, > > starting with 9.0-RELEASE and now running 9.1-RC3. > > > > I can't find anything in common with the files that I can trigger the > > panics with. One is a simple gzipped archive where some are plain text. > > Strangely, I can only reproduce it with data that was rsynced from that > > particular system (which is a Mac). > > Josh, > > I am collecting these cases, thank you for another one. > I had an interesting investigation of > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/173747 > Unfortunately, for some reason the whole conversation stayed private. > I see that also opened a PR earlier: > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/170238 > > Could you please provide the following info? > >From kgdb: > - list in frame 7 (zfs_freebsd_remove), so that I can see the code line > - local variables from frame 7 (info local) > > > > Andriy, > > Thanks for your quick response. I've never used kgdb, so forgive my ignorance > here. Is this what you're looking for? If not, if you could you elaborate on > those? > > #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not available. > ) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > > > (kgdb) list zfs_freebsd_remove > 5796 struct vop_remove_args /* { > 5797 struct vnode *a_dvp; > 5798 struct vnode *a_vp; > 5799 struct componentname *a_cnp; > 5800 } */ *ap; > 5801 { > 5802 > 5803 ASSERT(ap->a_cnp->cn_flags & SAVENAME); > 5804 > 5805 return (zfs_remove(ap->a_dvp, ap->a_cnp->cn_nameptr, Not quite :-) frame 7 list > (kgdb) info frame 7 > Stack frame at 0xffffff8466a6a920: > rip = 0xffffffff80ebd45a in zfs_freebsd_remove > (/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855); > saved rip 0xffffffff8081cf13 > called by frame at 0xffffff8466a6a940, caller of frame at 0xffffff8466a6a7a0 > source language c. > Arglist at 0xffffff8466a6a910, args: ap=Variable "ap" is not available. frame 7 info local > Also, for one of the files that trigger the problem: > - ls -i to obtain its inode number > - zdb -ddddd > > > # ls -i kyofilter\ v2.2.pax.gz (this is a symlink. the file that it's linked > to does *not* panic the system if I try to delete it). Hmm, then could you please rather do 'ls -Pi kyofilter\ v2.2.pax.gz' and then use that value for the zdb command? > 247126 kyofilter v2.2.pax.gz > > # zdb -ddddd store/tdxs1 247126 > Dataset store/tdxs1 [ZPL], ID 109, cr_txg 35014, 1.33T, 1106389 objects, rootbp > DVA[0]=<0:80001a2400:400> DVA[1]=<0:30800610000:400> [L0 DMU objset] fletcher4 > lzjb LE contiguous unique double size=800L/200P birth=1166838L/1166838P > fill=1106389 cksum=19391f0f67:78eb24a9cca:1439005549d01:275015332d1bdf > > Object lvl iblk dblk dsize lsize %full type > 247126 1 16K 512 0 512 0.00 ZFS plain file > 201 bonus System attributes > dnode flags: USERUSED_ACCOUNTED > dnode maxblkid: 0 > path /tech/2012-09-14-01-00/Drivers/Kyocera/.old/C2126.old/Kyocera OS > X 10.5+ Web build 2011.01.27.mpkg/Contents/Packages/Kyocera OS X > subinstaller.mpkg/Contents/Packages/kyofilter > v2.2.pkg/Contents/Resources/kyofilter v2.2.pax.gz > uid 1001 > gid 80 > atime Tue Nov 27 13:27:57 2012 > mtime Tue Jul 12 14:17:16 2011 > ctime Fri Sep 14 01:05:23 2012 > crtime Fri Sep 14 01:04:11 2012 > gen 81338 > mode 120755 > size 17 > parent 247122 > links 1 > pflags 40800000104 > xattr 155 -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 21:47:31 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E0B5578 for ; Tue, 27 Nov 2012 21:47:31 +0000 (UTC) (envelope-from josh@signalboxes.net) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8B9E48FC15 for ; Tue, 27 Nov 2012 21:47:30 +0000 (UTC) Received: by mail-ob0-f182.google.com with SMTP id 16so14724903obc.13 for ; Tue, 27 Nov 2012 13:47:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=1NCWJfB7phAIetHJ7OfEkIy1yh8xr8FGAP58XSsTkfk=; b=dbCgqCz3KZV/XkwppAKsn2v+OuKpJwFU5X4H+HZkNpIgVlbliT8VtIlkLM/JOAugvA fbwmuBl2boFXAHJgqbZaqT8TQpvI/NrZrCCMurIfnjOd1X2dyHNqE3500nEzK7cc83Lo GVZQv5K8q/bfKjLkhVnrsc+JKtgreKPTzUdWpODbe2M737VXvrIMk6HCyAK89C4q6Y0f FzXdL45hxC9zZqOPEUDRuA/hMuJbbbmIsHt3+qnWjE+8FNYPb3tdXpl+gOw3tC9znei5 6XyqxHHFmg6q85QKfHZJLOPQvLAnz63O+dyZGjKEM6TOSh9oH4xWAHRWSPozTbiwTN20 XZZg== Received: by 10.182.131.100 with SMTP id ol4mr1068990obb.38.1354052850435; Tue, 27 Nov 2012 13:47:30 -0800 (PST) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx.google.com with ESMTPS id m8sm17726171oeb.3.2012.11.27.13.47.28 (version=SSLv3 cipher=OTHER); Tue, 27 Nov 2012 13:47:29 -0800 (PST) Received: by mail-oa0-f54.google.com with SMTP id n9so16609447oag.13 for ; Tue, 27 Nov 2012 13:47:28 -0800 (PST) MIME-Version: 1.0 Received: by 10.60.4.168 with SMTP id l8mr7315744oel.48.1354052848009; Tue, 27 Nov 2012 13:47:28 -0800 (PST) Received: by 10.60.14.194 with HTTP; Tue, 27 Nov 2012 13:47:27 -0800 (PST) In-Reply-To: <50B52CEC.9080208@FreeBSD.org> References: <50B50B04.8020109@FreeBSD.org> <50B52CEC.9080208@FreeBSD.org> Date: Tue, 27 Nov 2012 14:47:27 -0700 Message-ID: Subject: Re: ZFS: Panic when attempting to delete certain data From: Josh Beard To: Andriy Gapon X-Gm-Message-State: ALoCoQnTNv0KWkqOWw0Uf98pAOIUDCDEe0e5d8T1v3eG2dN4daHUu2nvwF467NpD1CyREE0BJqCa Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 21:47:31 -0000 On Tue, Nov 27, 2012 at 2:13 PM, Andriy Gapon wrote: > on 27/11/2012 22:32 Josh Beard said the following: > > > > > > On Tue, Nov 27, 2012 at 11:48 AM, Andriy Gapon > > wrote: > > > > on 27/11/2012 20:25 Josh Beard said the following: > > > Hello, > > > > > > I have a system that I can consistently reproduce a panic on when > trying to > > > delete certain data. The data is data that was rsynced from > another system > > > - nothing terribly unique. This has been ongoing from several > months, > > > starting with 9.0-RELEASE and now running 9.1-RC3. > > > > > > I can't find anything in common with the files that I can trigger > the > > > panics with. One is a simple gzipped archive where some are plain > text. > > > Strangely, I can only reproduce it with data that was rsynced > from that > > > particular system (which is a Mac). > > > > Josh, > > > > I am collecting these cases, thank you for another one. > > I had an interesting investigation of > > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/173747 > > Unfortunately, for some reason the whole conversation stayed private. > > I see that also opened a PR earlier: > > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/170238 > > > > Could you please provide the following info? > > >From kgdb: > > - list in frame 7 (zfs_freebsd_remove), so that I can see the code > line > > - local variables from frame 7 (info local) > > > > > > > > Andriy, > > > > Thanks for your quick response. I've never used kgdb, so forgive my > ignorance > > here. Is this what you're looking for? If not, if you could you > elaborate on > > those? > > > > #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not > available. > > ) at > > > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 > > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > > > > > > (kgdb) list zfs_freebsd_remove > > 5796 struct vop_remove_args /* { > > 5797 struct vnode *a_dvp; > > 5798 struct vnode *a_vp; > > 5799 struct componentname *a_cnp; > > 5800 } */ *ap; > > 5801 { > > 5802 > > 5803 ASSERT(ap->a_cnp->cn_flags & SAVENAME); > > 5804 > > 5805 return (zfs_remove(ap->a_dvp, ap->a_cnp->cn_nameptr, > > Not quite :-) > frame 7 > list > > > (kgdb) info frame 7 > > Stack frame at 0xffffff8466a6a920: > > rip = 0xffffffff80ebd45a in zfs_freebsd_remove > > > (/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855); > > saved rip 0xffffffff8081cf13 > > called by frame at 0xffffff8466a6a940, caller of frame at > 0xffffff8466a6a7a0 > > source language c. > > Arglist at 0xffffff8466a6a910, args: ap=Variable "ap" is not available. > > frame 7 > info local > Thanks! Here we go: (kgdb) frame 7 #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); (kgdb) list 1850 &xattr_obj, sizeof (xattr_obj)); 1851 if (error == 0 && xattr_obj) { 1852 error = zfs_zget(zfsvfs, xattr_obj, &xzp); 1853 ASSERT3U(error, ==, 0); 1854 dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE); 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); 1856 } 1857 1858 mutex_enter(&zp->z_lock); 1859 if ((acl_obj = zfs_external_acl(zp)) != 0 && may_delete_now) (kgdb) frame 7 #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); (kgdb) info local No locals. (kgdb) > > Also, for one of the files that trigger the problem: > > - ls -i to obtain its inode number > > - zdb -ddddd > > > > > > # ls -i kyofilter\ v2.2.pax.gz (this is a symlink. the file that it's > linked > > to does *not* panic the system if I try to delete it). > > Hmm, then could you please rather do 'ls -Pi kyofilter\ v2.2.pax.gz' and > then > use that value for the zdb command? > # ls -Pi /DSDK12_NHR.pax.gz (symlink to ../Archive.pax.gz) 249868 ./Imaging/Packages/DSDK12_NHR_2012-02-23.pkg/Contents/Resources/DSDK12_NHR.pax.gz # zdb -ddddd store/tdxs1 249868 Dataset store/tdxs1 [ZPL], ID 109, cr_txg 35014, 1.33T, 1106389 objects, rootbp DVA[0]=<0:8000204400:400> DVA[1]=<0:30800644400:400> [L0 DMU objset] fletcher4 lzjb LE contiguous unique double size=800L/200P birth=1167710L/1167710P fill=1106389 cksum=1966704b59:757ae6cb615:134bfd597bca9:254b2ee348393d Object lvl iblk dblk dsize lsize %full type 249868 1 16K 512 0 512 0.00 ZFS plain file 201 bonus System attributes dnode flags: USERUSED_ACCOUNTED dnode maxblkid: 0 path /tech/2012-09-14-01-00/Imaging/Packages/DSDK12_NHR_2012-02-23.pkg/Contents/Resources/DSDK12_NHR.pax.gz uid 300002 gid 80 atime Tue Nov 27 14:43:00 2012 mtime Thu Feb 23 08:59:21 2012 ctime Fri Sep 14 01:12:37 2012 crtime Fri Sep 14 01:11:50 2012 gen 81430 mode 120755 size 17 parent 249866 links 1 pflags 40800000104 xattr 230 Indirect blocks: > > > 247126 kyofilter v2.2.pax.gz > > > > # zdb -ddddd store/tdxs1 247126 > > Dataset store/tdxs1 [ZPL], ID 109, cr_txg 35014, 1.33T, 1106389 objects, > rootbp > > DVA[0]=<0:80001a2400:400> DVA[1]=<0:30800610000:400> [L0 DMU objset] > fletcher4 > > lzjb LE contiguous unique double size=800L/200P birth=1166838L/1166838P > > fill=1106389 cksum=19391f0f67:78eb24a9cca:1439005549d01:275015332d1bdf > > > > Object lvl iblk dblk dsize lsize %full type > > 247126 1 16K 512 0 512 0.00 ZFS plain file > > 201 bonus System attributes > > dnode flags: USERUSED_ACCOUNTED > > dnode maxblkid: 0 > > path > /tech/2012-09-14-01-00/Drivers/Kyocera/.old/C2126.old/Kyocera OS > > X 10.5+ Web build 2011.01.27.mpkg/Contents/Packages/Kyocera OS X > > subinstaller.mpkg/Contents/Packages/kyofilter > > v2.2.pkg/Contents/Resources/kyofilter v2.2.pax.gz > > uid 1001 > > gid 80 > > atime Tue Nov 27 13:27:57 2012 > > mtime Tue Jul 12 14:17:16 2011 > > ctime Fri Sep 14 01:05:23 2012 > > crtime Fri Sep 14 01:04:11 2012 > > gen 81338 > > mode 120755 > > size 17 > > parent 247122 > > links 1 > > pflags 40800000104 > > xattr 155 > > -- > Andriy Gapon > From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 22:06:13 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 74799446 for ; Tue, 27 Nov 2012 22:06:13 +0000 (UTC) (envelope-from josh@signalboxes.net) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx1.freebsd.org (Postfix) with ESMTP id 22DBD8FC14 for ; Tue, 27 Nov 2012 22:06:12 +0000 (UTC) Received: by mail-oa0-f54.google.com with SMTP id n9so16630055oag.13 for ; Tue, 27 Nov 2012 14:06:12 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=qeE29E7Zvqw422ToYXfegds+NC8+vvsK6GGaR3eoh4I=; b=GR6b8t3HU+As42kWwoOjj2bgio9PvG9ag0FOSNU/ZhZj34rCdeXSAQUuCv+fHuvJRQ cvD1YfD3uSk4wl36jH+N3/tgmWL4wn59GUTkbKOS0IBjBEIg+YpDXGPRhqchJuOyNG4l dmCKaWfOadmdV7+WJQLEBvPEBLT8ZaguVVuQAPr3yHD6Fxw+J0kiuVzwcxDvRHvZg64r LvWK2DQ9BLl3C18gcJY5DgQQ1moZcEJiCXtd0XGglMH1MzF6oYxoaMYUYvCp/kqE2ghD y7WpRcNepJ3Ro5U9sy499yH+Ud7ktkVGQtr7KLBDJIv5/90atIFT9WG4R/v87VH5o8f7 L+vw== Received: by 10.60.25.106 with SMTP id b10mr13776337oeg.20.1354053972497; Tue, 27 Nov 2012 14:06:12 -0800 (PST) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx.google.com with ESMTPS id h2sm16043662obn.11.2012.11.27.14.06.10 (version=SSLv3 cipher=OTHER); Tue, 27 Nov 2012 14:06:11 -0800 (PST) Received: by mail-ob0-f182.google.com with SMTP id 16so14747138obc.13 for ; Tue, 27 Nov 2012 14:06:09 -0800 (PST) MIME-Version: 1.0 Received: by 10.60.30.167 with SMTP id t7mr14541297oeh.44.1354053969919; Tue, 27 Nov 2012 14:06:09 -0800 (PST) Received: by 10.60.14.194 with HTTP; Tue, 27 Nov 2012 14:06:09 -0800 (PST) In-Reply-To: <3F21E539-BBA9-4F9A-A7A0-7808042C600D@gmail.com> References: <3F21E539-BBA9-4F9A-A7A0-7808042C600D@gmail.com> Date: Tue, 27 Nov 2012 15:06:09 -0700 Message-ID: Subject: Re: ZFS: Panic when attempting to delete certain data From: Josh Beard To: Steven Schlansker X-Gm-Message-State: ALoCoQmmJh1pMFzfPiGH8YQi41jVMY+xq+c2KjWv4Fy4ke+Ek54yyDK/ujcgsDOD7qvw6Z+yOmsT Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 22:06:13 -0000 On Tue, Nov 27, 2012 at 11:34 AM, Steven Schlansker < stevenschlansker@gmail.com> wrote: > On Nov 27, 2012, at 10:25 AM, Josh Beard wrote: > > > Hello, > > > > I have a system that I can consistently reproduce a panic on when tryin= g > to > > delete certain data. The data is data that was rsynced from another > system > > - nothing terribly unique. This has been ongoing from several months, > > starting with 9.0-RELEASE and now running 9.1-RC3. > > > > Do you have dedupe turned on? I (and others, from what I've gathered) ha= d > a lot of trouble with > panics and system hangs when deleting deduped data with insufficient RAM. > > If that is the case, my only recourse turned out to be getting enough RAM > in the box to offload > the data, destroy the filesystem, and start over=85 > > Steven, Thanks, but dedup isn't on :\ This particular zpool is defaults. The system has 16 GB of RAM. Josh From owner-freebsd-fs@FreeBSD.ORG Tue Nov 27 23:58:28 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C0D185DF for ; Tue, 27 Nov 2012 23:58:28 +0000 (UTC) (envelope-from break19@gmail.com) Received: from mail-yh0-f54.google.com (mail-yh0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id 6FFF48FC13 for ; Tue, 27 Nov 2012 23:58:28 +0000 (UTC) Received: by mail-yh0-f54.google.com with SMTP id s35so1249918yhf.13 for ; Tue, 27 Nov 2012 15:58:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=q8OWEohidUYicLLbHVt7DVgjxI9rW5Yod2SlESLsKyM=; b=sJk8cSIm1gZjnVw0DwM3cQHAYmYosLs/enPLV5VS30YfalFExpndqN90McdviGbMu8 TLsw7fMJDrdReyncWhLGJlzJqXF0fDNnqVGG/HuVfPc0Wr4abjTCEFAkLNb21llTFlHt hOAfOjTX7OASC3+GXdEoqvlV/jSLJ5sAxTUJM5vsTrigoQULnsUewIxJrisZdqhlmyCd EQvwmfr2spASrl7mf9QFXmCYrFSN/S0HOpD1ZdQWcE/Fn2/Jvwqi0IIv5gZCadazigRm ArLxCBfdKCMjXe4LvWLSOE2COVRFAiwx4nOot+3HUkCvIuwdoM8RWhj97CcaCzXxj+VL mGRw== Received: by 10.101.142.34 with SMTP id u34mr4814838ann.7.1354060707522; Tue, 27 Nov 2012 15:58:27 -0800 (PST) Received: from [192.168.2.20] (173-17-218-61.client.mchsi.com. [173.17.218.61]) by mx.google.com with ESMTPS id t14sm18016368anl.17.2012.11.27.15.58.26 (version=SSLv3 cipher=OTHER); Tue, 27 Nov 2012 15:58:26 -0800 (PST) Message-ID: <50B5539B.7000505@gmail.com> Date: Tue, 27 Nov 2012 17:58:19 -0600 From: Chuck Burns User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: Re-sizable UFS project References: <50B4A040.6060001@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 23:58:28 -0000 On 11/27/2012 8:02 AM, CeDeROM wrote: > Btw. are there any projects to make UFS natively available (something > like fs-driver for ExtFS) on platforms such as Windows, Linux, MacOS? > It would be nice to have native UFS instead Ext2 as universal > filesystem among these operating systems... :-) > Linux has a read-only driver for UFS, and has since the beginning of Linux. -- Chuck Burns From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 07:51:55 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BF76B290; Wed, 28 Nov 2012 07:51:55 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: from mail-la0-f54.google.com (mail-la0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 13EB98FC12; Wed, 28 Nov 2012 07:51:54 +0000 (UTC) Received: by mail-la0-f54.google.com with SMTP id j13so12713611lah.13 for ; Tue, 27 Nov 2012 23:51:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=DmLPgQb0+n+BA5vw0GeGhXWlKfqRKSuUzjzlhyaHx9s=; b=hMSKlQwkpQey4DhmKQUv6b0Fu1UBG/bBRExGYrALdGISnIcze8X7zpfby5QL/mI6DZ ofelympp7Vckadppfoe96UwCuBIVbrCKw4SRybFQXYZr4wNJ08EVD5PQm7trwWr+GB5w 2nD0USmUQ8a6JANlQFQbqAvS5ciF1bTK5lflls1AGBrA6f02RQN/MZqC+QyGYDYpeBRU I1XQiikJe97Xg2J0rny0xu0kbAjJLPjggwprbw/WpK35vqhGRs636KhO1y1pb2jK1tWQ 3lBdp1gYwekxLmmzCzqgFNu+NJyWO/w7oGFDVGKcEiOUBzDl8R6rSOCapWEWH2gtZDWm 9IDw== MIME-Version: 1.0 Received: by 10.112.27.99 with SMTP id s3mr7649135lbg.81.1354089113926; Tue, 27 Nov 2012 23:51:53 -0800 (PST) Received: by 10.112.4.2 with HTTP; Tue, 27 Nov 2012 23:51:53 -0800 (PST) Date: Wed, 28 Nov 2012 09:51:53 +0200 Message-ID: Subject: NANDFS eats itself up From: Boris Astardzhiev To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: gjb@semihalf.com, Grzegorz Bernacki , stanislav_galabov@smartcom.bg X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 07:51:55 -0000 Hi, I've been playing with the NANDFS implementation in 10.0-CURRENT on a SHEEVAPLUG device. So far i've noticed a strange problem having a simple test on it. Firstly I have a partition that is 32MBs and it is mounted in /mnt (ROOTFS is over NFS). I've made an repetitive scp transfer of a 2MB file to the /mnt on the sheevaplug. After a delay of 100s the file is rm'ed from /mnt and this is repeating for quite awhile now. It seems the FS is eating itself up. # uname -a: FreeBSD smartcpe 10.0-CURRENT FreeBSD 10.0-CURRENT #32 r243606:243615M: Tue Nov 27 15:34:50 EET 2012 root@freebsd9:/usr/obj/arm.arm/usr/src2/sys/SHEEVAPLUG arm # sysctl vfs.nandfs vfs.nandfs.cleaner_segments: 5 vfs.nandfs.cleaner_interval: 5 vfs.nandfs.cleaner_enable: 1 vfs.nandfs.cps_between_sblocks: 5 vfs.nandfs.max_dirty_segs: 5 vfs.nandfs.sync_interval: 5 vfs.nandfs.verbose: 0 root@smartcpe:/mnt % df -h Filesystem Size Used Avail Capacity Mounted on 10.100.100.34:/usr/sheeva-fs 7.5G 5.9G 1.0G 85% / devfs 1.0k 1.0k 0B 100% /dev /dev/gnand0s.root 31M *896k* 30M 3% /mnt In the beginning the "Used" value was 128k. 4 hours later we have a "Used" value of "2.4MB": root@smartcpe:/mnt % df -h Filesystem Size Used Avail Capacity Mounted on 10.100.100.34:/usr/sheeva-fs 7.5G 5.9G 1.0G 85% / devfs 1.0k 1.0k 0B 100% /dev /dev/gnand0s.root 31M *2.4M* 28M 8% /mnt After 14h30min it has drained even more space - 7.8MB: root@smartcpe:/mnt % df -h Filesystem Size Used Avail Capacity Mounted on 10.100.100.34:/usr/sheeva-fs 7.5G 5.9G 1.0G 85% / devfs 1.0k 1.0k 0B 100% /dev /dev/gnand0s.root 31M *7.8M* 23M 25% /mnt How do I fix this? Greetings, Boris From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 08:36:11 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E1009171 for ; Wed, 28 Nov 2012 08:36:11 +0000 (UTC) (envelope-from metindoslu@gmail.com) Received: from mail-qc0-f182.google.com (mail-qc0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 98F1D8FC08 for ; Wed, 28 Nov 2012 08:36:11 +0000 (UTC) Received: by mail-qc0-f182.google.com with SMTP id k19so12056631qcs.13 for ; Wed, 28 Nov 2012 00:36:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type :content-transfer-encoding; bh=vVPdE3+rUPTF5V3gNXOkhxHgSraWGqZezEHe7XWwUj0=; b=zSCfk8eNeXKy1fXgjExGJ192IKmUhXqjDQHe74KPDlFs+9SR2th8zr2oy0eIwW1M5L lSMGixoeKI3ETLpvKiv+DZQopuwVajxYFmgbMrvmg+OdpxScA0IaBNnXyIVkcljHwflC HcUpMAiZHv2r4UpDA9wOi1KC7eMWcniPem0Lh++alskveT9W8sAFFC8yGtILALiSl4Ut uQ1jsm3wbl9OLTdF8LusTooxpoz6nU5jkzTURYRXgkmc9M6TYSNje3OmZrvQhmaZ1vOm MNDQPz/FpuklbOIRlsbl9yCNzr8tqBTxNUfVN02hV7x8H0t/MCecxUcCgI6CauO42ce9 Vmyg== Received: by 10.49.75.226 with SMTP id f2mr14016114qew.43.1354091770060; Wed, 28 Nov 2012 00:36:10 -0800 (PST) MIME-Version: 1.0 Received: by 10.49.35.242 with HTTP; Wed, 28 Nov 2012 00:35:49 -0800 (PST) From: =?UTF-8?B?TWV0aW4gRMO2xZ9sw7w=?= Date: Wed, 28 Nov 2012 10:35:49 +0200 Message-ID: Subject: Performance Difference on UFS and ZFS To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 08:36:11 -0000 Hi - I installed PostgreSQL 9.1.6 on FreeBSD 9.0-RELEASE, and compared ZFS to UFS. I also made sure all data easily fit into memory, ran some sequential scan queries on the database. Query run times on UFS were 100-200% slower than those of ZFS. This was intriguing as all data came from memory. What could be causing such a large performance difference? Thanks, Metin D=C3=B6=C5=9Fl=C3=BC From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 08:47:50 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 65B0E2EA for ; Wed, 28 Nov 2012 08:47:50 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 8E3848FC13 for ; Wed, 28 Nov 2012 08:47:49 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id KAA23331; Wed, 28 Nov 2012 10:47:46 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1TddJ4-000Oyy-Hh; Wed, 28 Nov 2012 10:47:46 +0200 Message-ID: <50B5CFAF.8070306@FreeBSD.org> Date: Wed, 28 Nov 2012 10:47:43 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Josh Beard Subject: Re: ZFS: Panic when attempting to delete certain data References: <50B50B04.8020109@FreeBSD.org> <50B52CEC.9080208@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 08:47:50 -0000 on 27/11/2012 23:47 Josh Beard said the following: > Thanks! Here we go: > > (kgdb) frame 7 > #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not available. > ) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > (kgdb) list > 1850 &xattr_obj, sizeof (xattr_obj)); > 1851 if (error == 0 && xattr_obj) { > 1852 error = zfs_zget(zfsvfs, xattr_obj, &xzp); > 1853 ASSERT3U(error, ==, 0); > 1854 dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE); > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > 1856 } > 1857 > 1858 mutex_enter(&zp->z_lock); > 1859 if ((acl_obj = zfs_external_acl(zp)) != 0 && may_delete_now) That's what I suspected. > (kgdb) frame 7 > #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not available. > ) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > (kgdb) info local > No locals. > (kgdb) A little bit unfortunate. > # ls -Pi /DSDK12_NHR.pax.gz (symlink to ../Archive.pax.gz) > 249868 > ./Imaging/Packages/DSDK12_NHR_2012-02-23.pkg/Contents/Resources/DSDK12_NHR.pax.gz > > # zdb -ddddd store/tdxs1 249868 > Dataset store/tdxs1 [ZPL], ID 109, cr_txg 35014, 1.33T, 1106389 objects, rootbp > DVA[0]=<0:8000204400:400> DVA[1]=<0:30800644400:400> [L0 DMU objset] fletcher4 > lzjb LE contiguous unique double size=800L/200P birth=1167710L/1167710P > fill=1106389 cksum=1966704b59:757ae6cb615:134bfd597bca9:254b2ee348393d > > Object lvl iblk dblk dsize lsize %full type > 249868 1 16K 512 0 512 0.00 ZFS plain file > 201 bonus System attributes > dnode flags: USERUSED_ACCOUNTED > dnode maxblkid: 0 > path > /tech/2012-09-14-01-00/Imaging/Packages/DSDK12_NHR_2012-02-23.pkg/Contents/Resources/DSDK12_NHR.pax.gz > uid 300002 > gid 80 > atime Tue Nov 27 14:43:00 2012 > mtime Thu Feb 23 08:59:21 2012 > ctime Fri Sep 14 01:12:37 2012 > crtime Fri Sep 14 01:11:50 2012 > gen 81430 > mode 120755 > size 17 > parent 249866 > links 1 > pflags 40800000104 > xattr 230 Could you please now run 'zdb -ddddd store/tdxs1 230' ? -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 10:42:27 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D67D66BD; Wed, 28 Nov 2012 10:42:27 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-wg0-f50.google.com (mail-wg0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 37CED8FC0C; Wed, 28 Nov 2012 10:42:26 +0000 (UTC) Received: by mail-wg0-f50.google.com with SMTP id 12so1419539wgh.31 for ; Wed, 28 Nov 2012 02:42:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=opHNldbHX/1HXuHeqEC7JGDL4PBvqcpBZeYGMXaEBTA=; b=d7Gv1IvrDOpvo4hEJXV4SpeMQRd2udMQqdXLCKIaKPr/5etqJ0jOG2R4onCV1fAy/u H8+1XZRMqX5xseG5CGf/uHC/kFLVzni/SkMFxo0qMVl51fGTYiFYiF4QC5QmIt0wN/iw IVCp69NKvCWGIIk7jQfNtn9ek4icIn7B2LlVuOK6pR+/h+m8rSOSNX9LzyP+Hwkg07AX DiNcU3UsOeW/pSBZTiFjIFcruhl7Yv02XAIFNAW7tYy0LuU/+I7b+e7LpFoIj7lJ7oHE +7+8ZGrdiM12cK63QWgrEE4ykMcEccAjJsVwIdfzOxbhFkI4wAd4PsuKBX5pMCd+lVh9 eMiQ== Received: by 10.180.86.70 with SMTP id n6mr28540205wiz.4.1354099345931; Wed, 28 Nov 2012 02:42:25 -0800 (PST) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPS id ey2sm7292412wib.9.2012.11.28.02.42.23 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 28 Nov 2012 02:42:24 -0800 (PST) Date: Wed, 28 Nov 2012 11:42:18 +0100 From: Mateusz Guzik To: Boris Astardzhiev Subject: Re: NANDFS eats itself up Message-ID: <20121128104218.GA17871@dft-labs.eu> References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, gjb@semihalf.com, Grzegorz Bernacki , stanislav_galabov@smartcom.bg X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 10:42:27 -0000 On Wed, Nov 28, 2012 at 09:51:53AM +0200, Boris Astardzhiev wrote: > Hi, > > I've been playing with the NANDFS implementation in 10.0-CURRENT on a > SHEEVAPLUG device. So far i've noticed a strange problem having a simple > test on it. Firstly I have a partition that is 32MBs and it is mounted in > /mnt (ROOTFS is over NFS). > > I've made an repetitive scp transfer of a 2MB file to the /mnt on the > sheevaplug. After a delay of 100s the file is rm'ed from /mnt and this is > repeating for quite awhile now. It seems the FS is eating itself up. > It is unclear from your descripion: do you still scp and rm the file? If so, it may be just that the cleaner is not able to delete data fast enough. You can increase vfs.nandfs.cleaner_segments and decrease vfs.nandfs.cleaner_interval to get more cleaning. If you are not doing anything on the fs and used space is growing, can you reproduce that? Preferably on a md based device (just create a file with dd of the same size and mdconfig -f it). -- Mateusz Guzik From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 13:22:31 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8DBDBE14 for ; Wed, 28 Nov 2012 13:22:31 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) by mx1.freebsd.org (Postfix) with ESMTP id 391428FC12 for ; Wed, 28 Nov 2012 13:22:30 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1Tdhb4-0008Af-B1 for freebsd-fs@freebsd.org; Wed, 28 Nov 2012 14:22:38 +0100 Received: from ib-jtotz.ib.ic.ac.uk ([155.198.110.220]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 28 Nov 2012 14:22:38 +0100 Received: from johannes by ib-jtotz.ib.ic.ac.uk with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 28 Nov 2012 14:22:38 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Johannes Totz Subject: Re: Re-sizable UFS project Date: Wed, 28 Nov 2012 13:22:11 +0000 Lines: 10 Message-ID: References: <50B4A040.6060001@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: ib-jtotz.ib.ic.ac.uk User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 In-Reply-To: X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 13:22:31 -0000 On 27/11/2012 14:02, CeDeROM wrote: > Btw. are there any projects to make UFS natively available (something > like fs-driver for ExtFS) on platforms such as Windows, Linux, MacOS? > It would be nice to have native UFS instead Ext2 as universal > filesystem among these operating systems... :-) > There used to be a UFS driver for Windows too. I used it for a while, but was a bit unstable. Otherwise, the smallest common denominator is FAT? From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 13:41:04 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A036B42E for ; Wed, 28 Nov 2012 13:41:04 +0000 (UTC) (envelope-from josh@signalboxes.net) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 42FBE8FC1A for ; Wed, 28 Nov 2012 13:41:04 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id l1so6745454vba.13 for ; Wed, 28 Nov 2012 05:41:03 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=NJngtqxBYDPyrm5BlEI230Ptkvfc3CxuJs77LbzKusM=; b=kQIGaKXRPr9M9bCk4rMbR+Mu8dSWwBNCEMt83ZNBbHGbH+Y6z30MWlyZa721sJUhpr nPRpUoI45wEy3/3my2fsaHvCsweIj1s4zUh7HJtL0npGIR+HN7xR1hIBk+HuB2QFTUyZ 6z4miyB7qMXDJxVOwdiJYTa0SPgx1cmNmc02lm/ZRwjFK8Jn2hFnj6/lcVbiY0y/0xbn WyHfRdZTCiAbNkWwjit/P5oP5no7JK6THhOpGwZOEgcbyO9woGUT9bJz1Nm7KP+K+mpx asBXS9POjzSjHDunizGyNpy20GTwWwsPB7IBBkYJEXT2YvMn1Bx7GmcpNMhu+ZKudR+F TBJA== Received: by 10.220.226.200 with SMTP id ix8mr28036385vcb.67.1354110063062; Wed, 28 Nov 2012 05:41:03 -0800 (PST) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx.google.com with ESMTPS id gl6sm10743814vec.4.2012.11.28.05.41.01 (version=SSLv3 cipher=OTHER); Wed, 28 Nov 2012 05:41:02 -0800 (PST) Received: by mail-vc0-f182.google.com with SMTP id fo13so18412879vcb.13 for ; Wed, 28 Nov 2012 05:41:00 -0800 (PST) MIME-Version: 1.0 Received: by 10.52.175.167 with SMTP id cb7mr25004329vdc.58.1354110060750; Wed, 28 Nov 2012 05:41:00 -0800 (PST) Received: by 10.220.137.8 with HTTP; Wed, 28 Nov 2012 05:41:00 -0800 (PST) In-Reply-To: <50B5CFAF.8070306@FreeBSD.org> References: <50B50B04.8020109@FreeBSD.org> <50B52CEC.9080208@FreeBSD.org> <50B5CFAF.8070306@FreeBSD.org> Date: Wed, 28 Nov 2012 06:41:00 -0700 Message-ID: Subject: Re: ZFS: Panic when attempting to delete certain data From: Josh Beard To: Andriy Gapon X-Gm-Message-State: ALoCoQnvqwbJ6Xgut9MlJk7OKE4CmZ4jwhNv+HMEjPSByN/vcHvOt93GGtY43DWAHTmAlMeO8Rhg Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 13:41:04 -0000 On Wed, Nov 28, 2012 at 1:47 AM, Andriy Gapon wrote: > on 27/11/2012 23:47 Josh Beard said the following: > > Thanks! Here we go: > > > > (kgdb) frame 7 > > #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not > available. > > ) at > > > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 > > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > > (kgdb) list > > 1850 &xattr_obj, sizeof (xattr_obj)); > > 1851 if (error == 0 && xattr_obj) { > > 1852 error = zfs_zget(zfsvfs, xattr_obj, &xzp); > > 1853 ASSERT3U(error, ==, 0); > > 1854 dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE); > > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > > 1856 } > > 1857 > > 1858 mutex_enter(&zp->z_lock); > > 1859 if ((acl_obj = zfs_external_acl(zp)) != 0 && > may_delete_now) > > That's what I suspected. > > > (kgdb) frame 7 > > #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not > available. > > ) at > > > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 > > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > > (kgdb) info local > > No locals. > > (kgdb) > > A little bit unfortunate. > > > # ls -Pi /DSDK12_NHR.pax.gz (symlink to ../Archive.pax.gz) > > 249868 > > > ./Imaging/Packages/DSDK12_NHR_2012-02-23.pkg/Contents/Resources/DSDK12_NHR.pax.gz > > > > # zdb -ddddd store/tdxs1 249868 > > Dataset store/tdxs1 [ZPL], ID 109, cr_txg 35014, 1.33T, 1106389 objects, > rootbp > > DVA[0]=<0:8000204400:400> DVA[1]=<0:30800644400:400> [L0 DMU objset] > fletcher4 > > lzjb LE contiguous unique double size=800L/200P birth=1167710L/1167710P > > fill=1106389 cksum=1966704b59:757ae6cb615:134bfd597bca9:254b2ee348393d > > > > Object lvl iblk dblk dsize lsize %full type > > 249868 1 16K 512 0 512 0.00 ZFS plain file > > 201 bonus System attributes > > dnode flags: USERUSED_ACCOUNTED > > dnode maxblkid: 0 > > path > > > /tech/2012-09-14-01-00/Imaging/Packages/DSDK12_NHR_2012-02-23.pkg/Contents/Resources/DSDK12_NHR.pax.gz > > uid 300002 > > gid 80 > > atime Tue Nov 27 14:43:00 2012 > > mtime Thu Feb 23 08:59:21 2012 > > ctime Fri Sep 14 01:12:37 2012 > > crtime Fri Sep 14 01:11:50 2012 > > gen 81430 > > mode 120755 > > size 17 > > parent 249866 > > links 1 > > pflags 40800000104 > > xattr 230 > > > Could you please now run 'zdb -ddddd store/tdxs1 230' ? > > -- > Andriy Gapon > # zdb -ddddd store/tdxs1 230 Dataset store/tdxs1 [ZPL], ID 109, cr_txg 35014, 1.33T, 1106389 objects, rootbp DVA[0]=<0:8000284800:400> DVA[1]=<0:308006b3800:400> [L0 DMU objset] fletcher4 lzjb LE contiguous unique double size=800L/200P birth=1167748L/1167748P fill=1106389 cksum=16e1e08cb8:70a50f1ec5a:13419c71d1cda:260fbed28af8f5 Object lvl iblk dblk dsize lsize %full type zdb: dmu_bonus_hold(230) failed, errno 2 From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 13:50:13 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CCD77756 for ; Wed, 28 Nov 2012 13:50:13 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 331D18FC13 for ; Wed, 28 Nov 2012 13:50:12 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id PAA26928; Wed, 28 Nov 2012 15:49:39 +0200 (EET) (envelope-from avg@FreeBSD.org) Message-ID: <50B61672.2070406@FreeBSD.org> Date: Wed, 28 Nov 2012 15:49:38 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Josh Beard Subject: Re: ZFS: Panic when attempting to delete certain data References: <50B50B04.8020109@FreeBSD.org> <50B52CEC.9080208@FreeBSD.org> <50B5CFAF.8070306@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 13:50:13 -0000 on 28/11/2012 15:41 Josh Beard said the following: > > > On Wed, Nov 28, 2012 at 1:47 AM, Andriy Gapon > wrote: > > on 27/11/2012 23:47 Josh Beard said the following: > > Thanks! Here we go: > > > > (kgdb) frame 7 > > #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not available. > > ) at > > > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 > > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > > (kgdb) list > > 1850 &xattr_obj, sizeof (xattr_obj)); > > 1851 if (error == 0 && xattr_obj) { > > 1852 error = zfs_zget(zfsvfs, xattr_obj, &xzp); > > 1853 ASSERT3U(error, ==, 0); > > 1854 dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE); > > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > > 1856 } > > 1857 > > 1858 mutex_enter(&zp->z_lock); > > 1859 if ((acl_obj = zfs_external_acl(zp)) != 0 && may_delete_now) > > That's what I suspected. > > > (kgdb) frame 7 > > #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is not available. > > ) at > > > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 > > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > > (kgdb) info local > > No locals. > > (kgdb) > > A little bit unfortunate. > > > # ls -Pi /DSDK12_NHR.pax.gz (symlink to ../Archive.pax.gz) > > 249868 > > > ./Imaging/Packages/DSDK12_NHR_2012-02-23.pkg/Contents/Resources/DSDK12_NHR.pax.gz > > > > # zdb -ddddd store/tdxs1 249868 > > Dataset store/tdxs1 [ZPL], ID 109, cr_txg 35014, 1.33T, 1106389 objects, rootbp > > DVA[0]=<0:8000204400:400> DVA[1]=<0:30800644400:400> [L0 DMU objset] fletcher4 > > lzjb LE contiguous unique double size=800L/200P birth=1167710L/1167710P > > fill=1106389 cksum=1966704b59:757ae6cb615:134bfd597bca9:254b2ee348393d > > > > Object lvl iblk dblk dsize lsize %full type > > 249868 1 16K 512 0 512 0.00 ZFS plain file > > 201 bonus System attributes > > dnode flags: USERUSED_ACCOUNTED > > dnode maxblkid: 0 > > path > > > /tech/2012-09-14-01-00/Imaging/Packages/DSDK12_NHR_2012-02-23.pkg/Contents/Resources/DSDK12_NHR.pax.gz > > uid 300002 > > gid 80 > > atime Tue Nov 27 14:43:00 2012 > > mtime Thu Feb 23 08:59:21 2012 > > ctime Fri Sep 14 01:12:37 2012 > > crtime Fri Sep 14 01:11:50 2012 > > gen 81430 > > mode 120755 > > size 17 > > parent 249866 > > links 1 > > pflags 40800000104 > > xattr 230 > > > # zdb -ddddd store/tdxs1 230 > Dataset store/tdxs1 [ZPL], ID 109, cr_txg 35014, 1.33T, 1106389 objects, rootbp > DVA[0]=<0:8000284800:400> DVA[1]=<0:308006b3800:400> [L0 DMU objset] fletcher4 > lzjb LE contiguous unique double size=800L/200P birth=1167748L/1167748P > fill=1106389 cksum=16e1e08cb8:70a50f1ec5a:13419c71d1cda:260fbed28af8f5 > > Object lvl iblk dblk dsize lsize %full type > zdb: dmu_bonus_hold(230) failed, errno 2 > errno 2 is ENOENT, so all pieces match. The file appears to have extended attributes, but they do not actually exist. If your ZFS module was built with debug, then ASSERT3U(error, ==, 0) would be triggered. This is something that I said earlier in the private conversation that I mentioned: Andriy Gapon said: > The relevant commits are r240632 and r240345. I can't recall when I MFC-ed them > to stable/9. Most likely they are not in 9.1 releng branch. [snip] > I also don't have a good advice on how to fix the existing corruption. > I'd probably go with using something like tar/cpio/pax to move data to fresh > filesystem. Please be sure to use latest stable/9 or 8 to not run into the > issue again. > > zfs send/recv won't help, it would mindlessly replicate the corrupted attributes. To this I might add that the bugs were not FreeBSD-specifc and they may be still present in ZFS upstream and other ZFS ports. Thank you for the debugging information. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 14:00:48 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 047A0B13 for ; Wed, 28 Nov 2012 14:00:48 +0000 (UTC) (envelope-from lars@e-new.0x20.net) Received: from mail.0x20.net (mail.0x20.net [IPv6:2001:aa8:fffb:1::3]) by mx1.freebsd.org (Postfix) with ESMTP id 840E58FC14 for ; Wed, 28 Nov 2012 14:00:47 +0000 (UTC) Received: from e-new.0x20.net (mail.0x20.net [IPv6:2001:aa8:fffb:1::3]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.0x20.net (Postfix) with ESMTPS id B65256A6006; Wed, 28 Nov 2012 15:00:46 +0100 (CET) Received: from e-new.0x20.net (localhost [127.0.0.1]) by e-new.0x20.net (8.14.5/8.14.5) with ESMTP id qASE0kXQ051778; Wed, 28 Nov 2012 15:00:46 +0100 (CET) (envelope-from lars@e-new.0x20.net) Received: (from lars@localhost) by e-new.0x20.net (8.14.5/8.14.5/Submit) id qASE0keT050573; Wed, 28 Nov 2012 15:00:46 +0100 (CET) (envelope-from lars) Date: Wed, 28 Nov 2012 15:00:46 +0100 From: Lars Engels To: Johannes Totz Subject: Re: Re-sizable UFS project Message-ID: <20121128140046.GG849@e-new.0x20.net> References: <50B4A040.6060001@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="IuhbYIxU28t+Kd57" Content-Disposition: inline In-Reply-To: X-Editor: VIM - Vi IMproved 7.3 X-Operation-System: FreeBSD 8.3-RELEASE-p4 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 14:00:48 -0000 --IuhbYIxU28t+Kd57 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 28, 2012 at 01:22:11PM +0000, Johannes Totz wrote: > On 27/11/2012 14:02, CeDeROM wrote: > > Btw. are there any projects to make UFS natively available (something > > like fs-driver for ExtFS) on platforms such as Windows, Linux, MacOS? > > It would be nice to have native UFS instead Ext2 as universal > > filesystem among these operating systems... :-) > >=20 >=20 > There used to be a UFS driver for Windows too. I used it for a while, > but was a bit unstable. > Otherwise, the smallest common denominator is FAT? For Windows, there's UFSExplorer: http://www.ufsexplorer.de/ --IuhbYIxU28t+Kd57 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlC2GQ4ACgkQKc512sD3afihSQCfWa6UuLSiGftHd9NZMMOBj8Lx Y2sAn26Hg4OExH1eiYB9xcgp5JyI1zkP =Aosa -----END PGP SIGNATURE----- --IuhbYIxU28t+Kd57-- From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 14:07:45 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 99174E57 for ; Wed, 28 Nov 2012 14:07:45 +0000 (UTC) (envelope-from josh@signalboxes.net) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 332C78FC14 for ; Wed, 28 Nov 2012 14:07:44 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id l1so6785749vba.13 for ; Wed, 28 Nov 2012 06:07:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=8ohSdWJuOlW9Sh/TM6q41SpFguJmKAoO7COLeb2hfm4=; b=J1GTWxBxhhqPRo59SD3wMEJXdtPomtTy5M0axbGGww8RIyDQGv1oJs2kGGtoEVay8Y jLS8f4KuE0vzfe4zoWkTkIkAUukXHl+MaFdZ9V7+ll8lhzd7iCEZU8B/6vULY5c4f/0j gkdOi+pLeYM2jX4d+YQY4Xqo19D/8AdA0IZlw9L4VGGBS2tvjXxedwzxz8I0an9Zihza 7Usd1Eh1Djpk9NtFv3KTl65/1P22PqOqHFfxXBW8Kbjqz5kiFIc+B9wzqJlFT5wuGglQ ElnRfP9eQLUg9seUyHH3DDmjQxVm7KHcaPGq68YK3qopDvT8ho4OIp7CVxNTBXhbngRX /kYw== Received: by 10.220.221.5 with SMTP id ia5mr28223054vcb.65.1354111664221; Wed, 28 Nov 2012 06:07:44 -0800 (PST) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx.google.com with ESMTPS id q7sm11800287vdw.22.2012.11.28.06.07.42 (version=SSLv3 cipher=OTHER); Wed, 28 Nov 2012 06:07:43 -0800 (PST) Received: by mail-vc0-f182.google.com with SMTP id fo13so18450181vcb.13 for ; Wed, 28 Nov 2012 06:07:42 -0800 (PST) MIME-Version: 1.0 Received: by 10.52.91.73 with SMTP id cc9mr16204302vdb.48.1354111662158; Wed, 28 Nov 2012 06:07:42 -0800 (PST) Received: by 10.220.137.8 with HTTP; Wed, 28 Nov 2012 06:07:41 -0800 (PST) In-Reply-To: <50B61672.2070406@FreeBSD.org> References: <50B50B04.8020109@FreeBSD.org> <50B52CEC.9080208@FreeBSD.org> <50B5CFAF.8070306@FreeBSD.org> <50B61672.2070406@FreeBSD.org> Date: Wed, 28 Nov 2012 07:07:41 -0700 Message-ID: Subject: Re: ZFS: Panic when attempting to delete certain data From: Josh Beard To: Andriy Gapon X-Gm-Message-State: ALoCoQks9ltvXZUEiWfaHsQcb69L2LZGyFcvRZCK45bUJ/VEQriztCO3D8wJFPAtgM8uWQ7nUjCw Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 14:07:45 -0000 On Wed, Nov 28, 2012 at 6:49 AM, Andriy Gapon wrote: > on 28/11/2012 15:41 Josh Beard said the following: > > > > > > On Wed, Nov 28, 2012 at 1:47 AM, Andriy Gapon > > wrote: > > > > on 27/11/2012 23:47 Josh Beard said the following: > > > Thanks! Here we go: > > > > > > (kgdb) frame 7 > > > #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is > not available. > > > ) at > > > > > > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 > > > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > > > (kgdb) list > > > 1850 &xattr_obj, sizeof (xattr_obj)); > > > 1851 if (error == 0 && xattr_obj) { > > > 1852 error = zfs_zget(zfsvfs, xattr_obj, &xzp); > > > 1853 ASSERT3U(error, ==, 0); > > > 1854 dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE); > > > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > > > 1856 } > > > 1857 > > > 1858 mutex_enter(&zp->z_lock); > > > 1859 if ((acl_obj = zfs_external_acl(zp)) != 0 && > may_delete_now) > > > > That's what I suspected. > > > > > (kgdb) frame 7 > > > #7 0xffffffff80ebd45a in zfs_freebsd_remove (ap=Variable "ap" is > not available. > > > ) at > > > > > > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1855 > > > 1855 dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE); > > > (kgdb) info local > > > No locals. > > > (kgdb) > > > > A little bit unfortunate. > > > > > # ls -Pi /DSDK12_NHR.pax.gz (symlink to ../Archive.pax.gz) > > > 249868 > > > > > > ./Imaging/Packages/DSDK12_NHR_2012-02-23.pkg/Contents/Resources/DSDK12_NHR.pax.gz > > > > > > # zdb -ddddd store/tdxs1 249868 > > > Dataset store/tdxs1 [ZPL], ID 109, cr_txg 35014, 1.33T, 1106389 > objects, rootbp > > > DVA[0]=<0:8000204400:400> DVA[1]=<0:30800644400:400> [L0 DMU > objset] fletcher4 > > > lzjb LE contiguous unique double size=800L/200P > birth=1167710L/1167710P > > > fill=1106389 > cksum=1966704b59:757ae6cb615:134bfd597bca9:254b2ee348393d > > > > > > Object lvl iblk dblk dsize lsize %full type > > > 249868 1 16K 512 0 512 0.00 ZFS plain file > > > 201 bonus System > attributes > > > dnode flags: USERUSED_ACCOUNTED > > > dnode maxblkid: 0 > > > path > > > > > > /tech/2012-09-14-01-00/Imaging/Packages/DSDK12_NHR_2012-02-23.pkg/Contents/Resources/DSDK12_NHR.pax.gz > > > uid 300002 > > > gid 80 > > > atime Tue Nov 27 14:43:00 2012 > > > mtime Thu Feb 23 08:59:21 2012 > > > ctime Fri Sep 14 01:12:37 2012 > > > crtime Fri Sep 14 01:11:50 2012 > > > gen 81430 > > > mode 120755 > > > size 17 > > > parent 249866 > > > links 1 > > > pflags 40800000104 > > > xattr 230 > > > > > > # zdb -ddddd store/tdxs1 230 > > Dataset store/tdxs1 [ZPL], ID 109, cr_txg 35014, 1.33T, 1106389 objects, > rootbp > > DVA[0]=<0:8000284800:400> DVA[1]=<0:308006b3800:400> [L0 DMU objset] > fletcher4 > > lzjb LE contiguous unique double size=800L/200P birth=1167748L/1167748P > > fill=1106389 cksum=16e1e08cb8:70a50f1ec5a:13419c71d1cda:260fbed28af8f5 > > > > Object lvl iblk dblk dsize lsize %full type > > zdb: dmu_bonus_hold(230) failed, errno 2 > > > > errno 2 is ENOENT, so all pieces match. > The file appears to have extended attributes, but they do not actually > exist. > If your ZFS module was built with debug, then ASSERT3U(error, ==, 0) would > be > triggered. > > This is something that I said earlier in the private conversation that I > mentioned: > > Andriy Gapon said: > > The relevant commits are r240632 and r240345. I can't recall when I > MFC-ed them > > to stable/9. Most likely they are not in 9.1 releng branch. > [snip] > > I also don't have a good advice on how to fix the existing corruption. > > I'd probably go with using something like tar/cpio/pax to move data to > fresh > > filesystem. Please be sure to use latest stable/9 or 8 to not run into > the > > issue again. > > > > zfs send/recv won't help, it would mindlessly replicate the corrupted > attributes. > > To this I might add that the bugs were not FreeBSD-specifc and they may be > still > present in ZFS upstream and other ZFS ports. > > Thank you for the debugging information. > -- > Andriy Gapon > Andriy, Does this mean re-creating the dataset (or the pool?) would resolve this under a recent stable/9 build would resolve this? I have no problem doing so - as the data is redundant. Thanks for your help on this. Josh From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 14:13:21 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9B4C9E1 for ; Wed, 28 Nov 2012 14:13:21 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id DDC428FC15 for ; Wed, 28 Nov 2012 14:13:20 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id QAA27127; Wed, 28 Nov 2012 16:13:17 +0200 (EET) (envelope-from avg@FreeBSD.org) Message-ID: <50B61BFD.20509@FreeBSD.org> Date: Wed, 28 Nov 2012 16:13:17 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Josh Beard Subject: Re: ZFS: Panic when attempting to delete certain data References: <50B50B04.8020109@FreeBSD.org> <50B52CEC.9080208@FreeBSD.org> <50B5CFAF.8070306@FreeBSD.org> <50B61672.2070406@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 14:13:21 -0000 on 28/11/2012 16:07 Josh Beard said the following: > > > On Wed, Nov 28, 2012 at 6:49 AM, Andriy Gapon > wrote: > This is something that I said earlier in the private conversation that I > mentioned: > > Andriy Gapon said: > > The relevant commits are r240632 and r240345. I can't recall when I MFC-ed them > > to stable/9. Most likely they are not in 9.1 releng branch. > [snip] > > I also don't have a good advice on how to fix the existing corruption. > > I'd probably go with using something like tar/cpio/pax to move data to fresh > > filesystem. Please be sure to use latest stable/9 or 8 to not run into the > > issue again. > > > > zfs send/recv won't help, it would mindlessly replicate the corrupted > attributes. > > To this I might add that the bugs were not FreeBSD-specifc and they may be still > present in ZFS upstream and other ZFS ports. > > Thank you for the debugging information. > -- > Andriy Gapon > > > Andriy, > > Does this mean re-creating the dataset (or the pool?) would resolve this under a > recent stable/9 build would resolve this? > > I have no problem doing so - as the data is redundant. > > Thanks for your help on this. As I've said above - it depends. If you just do normal file operations then the problem should not re-appear. If it does, then please report. If you zfs send/recv data from older systems or other operating systems, then you may run into the problem again. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 14:49:13 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CBBB8A92 for ; Wed, 28 Nov 2012 14:49:13 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-qa0-f54.google.com (mail-qa0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 7D6608FC12 for ; Wed, 28 Nov 2012 14:49:13 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id g24so5613054qab.13 for ; Wed, 28 Nov 2012 06:49:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=8gwS7qI31Ig9E8iHrVz9jOFq8uM7Q9CXv2VvNRKj8h8=; b=cKw3Z6ZDymoLo7PwZW8i9085F1Aiz1zgNl94eQA2lPAR7Efol2Z49TkX6qRbKLmeiN cZvenXv2pMMqR+gSzDpf16hi/OLlqp15uljpwg86MId6zJHkmPo6xrGSMzribXbltWQb vLQwodTcD++vgvl7oB3WjsLF49PxUkxLM9Snj/ssITXrU2H8m8/UCOdHdTVmI7dEgedP 2uKWq1vjPrD4eNsunyRKc4/zWwRH82Tcb/c9sLqGZPivdSpBV3inLvYbNev3ufunC4lv kNTDxpizOzK8sL/Lvvt1ijddr6LjNWzRlS4ZnwWFlyd0OP+qcr1FcVx6DGNuDZF5HQNC jT+Q== MIME-Version: 1.0 Received: by 10.49.1.47 with SMTP id 15mr21045642qej.46.1354114152603; Wed, 28 Nov 2012 06:49:12 -0800 (PST) Received: by 10.229.78.96 with HTTP; Wed, 28 Nov 2012 06:49:12 -0800 (PST) In-Reply-To: References: Date: Wed, 28 Nov 2012 17:49:12 +0300 Message-ID: Subject: Re: Performance Difference on UFS and ZFS From: Sergey Kandaurov To: =?UTF-8?B?TWV0aW4gRMO2xZ9sw7w=?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 14:49:13 -0000 On 28 November 2012 12:35, Metin D=C3=B6=C5=9Fl=C3=BC wrote: > Hi - > > I installed PostgreSQL 9.1.6 on FreeBSD 9.0-RELEASE, and compared ZFS > to UFS. I also made sure all data easily fit into memory, ran some > sequential scan queries on the database. > > Query run times on UFS were 100-200% slower than those of ZFS. This > was intriguing as all data came from memory. What could be causing > such a large performance difference? Hi Metin D=C3=B6=C5=9Fl=C3=BC! That's interesting. Can you please share your settings? Hardware configuration, system tuning (sysctl, zfs), PostgreSQL settings, data access pattern (read/write ratio), and how you measure performance would be a good start. --=20 wbr, pluknet From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 15:06:14 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7A12DED9; Wed, 28 Nov 2012 15:06:14 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: from mail-la0-f54.google.com (mail-la0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 992CC8FC15; Wed, 28 Nov 2012 15:06:13 +0000 (UTC) Received: by mail-la0-f54.google.com with SMTP id j13so13125957lah.13 for ; Wed, 28 Nov 2012 07:06:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=z4k0S718ab35UPUhVUBK50rSRwNMqo7xiGft0+AjtJA=; b=Ll/8PSo+1P5AlLLzE5hwpbNKcDrr+wvCvhZztzP9B+3bv4CskL+f8A1Vb93eaNDtp1 A4NAPZHVKYr4BpEh7+LYkJ69ZmhtbFQms8gLzzhnNChdOgdHMGf1hTVVMANV0C+u6Dj1 qgIbfhmGHRoE7AdYkbt3kcp2UnDiqyFOFHvsm1y4zUyMgiOEh7mP8ubAl83dzjIx69w5 AiIJOiL+LzMUCMo3e+XPmh+PKFUSayz2j0kafzfNjDlvs2BIrqIu4JtPjVoCe4gwUBN3 X8TZTws3gsZg6gRvj2RJAcvWTfsp0ybupmdF2sfxF2FQqx0riGxf+vEN+gEn9vyznCT/ sMOg== MIME-Version: 1.0 Received: by 10.112.27.99 with SMTP id s3mr8164429lbg.81.1354115172427; Wed, 28 Nov 2012 07:06:12 -0800 (PST) Received: by 10.112.4.2 with HTTP; Wed, 28 Nov 2012 07:06:12 -0800 (PST) In-Reply-To: <20121128104218.GA17871@dft-labs.eu> References: <20121128104218.GA17871@dft-labs.eu> Date: Wed, 28 Nov 2012 17:06:12 +0200 Message-ID: Subject: Re: NANDFS eats itself up From: Boris Astardzhiev To: Mateusz Guzik Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org, gjb@semihalf.com, Grzegorz Bernacki , stanislav_galabov@smartcom.bg X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 15:06:14 -0000 Yes, I do. I've made this to repeat via a script. The interval between each SCP transfer is 100s. I thought it would give just enough time to the fs for reclaiming its space back. Now I stopped the transfers and doubled the vfs.nandfs.cleaner_segments to 10. The fs has no files in it but it reports "~9.6MB" of used space. 4+ hours later it has NOT changed at all it still reports "~9.6MB" Used. On Wed, Nov 28, 2012 at 12:42 PM, Mateusz Guzik wrote: > On Wed, Nov 28, 2012 at 09:51:53AM +0200, Boris Astardzhiev wrote: > > Hi, > > > > I've been playing with the NANDFS implementation in 10.0-CURRENT on a > > SHEEVAPLUG device. So far i've noticed a strange problem having a simple > > test on it. Firstly I have a partition that is 32MBs and it is mounted in > > /mnt (ROOTFS is over NFS). > > > > I've made an repetitive scp transfer of a 2MB file to the /mnt on the > > sheevaplug. After a delay of 100s the file is rm'ed from /mnt and this is > > repeating for quite awhile now. It seems the FS is eating itself up. > > > > It is unclear from your descripion: do you still scp and rm the file? > If so, it may be just that the cleaner is not able to delete data fast > enough. You can increase vfs.nandfs.cleaner_segments and decrease > vfs.nandfs.cleaner_interval to get more cleaning. > > If you are not doing anything on the fs and used space is growing, can > you reproduce that? Preferably on a md based device (just create a file > with dd of the same size and mdconfig -f it). > > -- > Mateusz Guzik > From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 15:13:51 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 43D783A0; Wed, 28 Nov 2012 15:13:51 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-wi0-f170.google.com (mail-wi0-f170.google.com [209.85.212.170]) by mx1.freebsd.org (Postfix) with ESMTP id 961148FC12; Wed, 28 Nov 2012 15:13:50 +0000 (UTC) Received: by mail-wi0-f170.google.com with SMTP id hq7so4843813wib.1 for ; Wed, 28 Nov 2012 07:13:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=A5jhlstTmUpHmRZpe1SJhVlhsyqUR1H72Zgn68IcUks=; b=wRkZio3yL2kYFjomzgZ/VmoNtLCKDoBOr7+u8/p+2nkuG0PtEvC+nAxdyAP/QA6Eh2 iNh7LvgibI/FLFQ3UWJa7bGzLjR+SR6Mq920o+7sJjj0VCQBxp2ii6kRdniYmHQfu7YE 0VeskJZvudd08L2ojtVgYKxITRsQvLnbw2Cib9SPwQZp8Z5JIX+b6FXaufm2bqoe1wEx qd4Ge56dNJQ3ouX4PUwrSiH7AQ04qCwbPxeJDIWwxXGJiNL60Nj5aW8EJmZu2JqQ4uLv dQ6JxnHDcszgf64QgUhCJuNa3TRPTvsQqeH+rM2ts0JifGAe6Wopa7kIG/OqWYZUKNsz 54Fw== Received: by 10.180.100.132 with SMTP id ey4mr33441340wib.9.1354115624390; Wed, 28 Nov 2012 07:13:44 -0800 (PST) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPS id w5sm8334414wiz.10.2012.11.28.07.13.42 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 28 Nov 2012 07:13:43 -0800 (PST) Date: Wed, 28 Nov 2012 16:13:36 +0100 From: Mateusz Guzik To: Boris Astardzhiev Subject: Re: NANDFS eats itself up Message-ID: <20121128151336.GC17871@dft-labs.eu> References: <20121128104218.GA17871@dft-labs.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, gjb@semihalf.com, Grzegorz Bernacki , stanislav_galabov@smartcom.bg X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 15:13:51 -0000 On Wed, Nov 28, 2012 at 05:06:12PM +0200, Boris Astardzhiev wrote: > Yes, I do. I've made this to repeat via a script. The interval between each > SCP transfer is 100s. I thought it would give just enough time to the fs > for reclaiming its space back. > > Now I stopped the transfers and doubled the vfs.nandfs.cleaner_segments to > 10. The fs has no files in it but it reports "~9.6MB" of used space. 4+ > hours later it has NOT changed at all it still reports "~9.6MB" Used. > Now this indeed sounds like a bug. Can you enable debug like this: # sysctl vfs.nandfs.verbose=0xffffffff The kernel will start printing a lot of debugging information. With default configuration of syslog this will end up in /var/log/messages. Capture something like 30 seconds of output and post it somewhere. -- Mateusz Guzik From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 15:36:13 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CD90C9DB; Wed, 28 Nov 2012 15:36:13 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id D44BD8FC0C; Wed, 28 Nov 2012 15:36:12 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id go10so10584989lbb.13 for ; Wed, 28 Nov 2012 07:36:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=1eY/bqDEDbsXmL5Eix8LDAgRB7mq4u8m9yUdEwBr/To=; b=i2lIgzc5IVHZqLFo4GdHJXkyBBZQWVjZhnS2XuC0upXAOtwViNPRZdU0Pdm6UlQFx9 Sx+jhtWwhoEl2141D+csebr7ThmkiIPjxAAO7i5/Dr5dnKlaQTED7rT0uKTUJNSZ28fC MKdow7/VsMSc0ERUqjTXblObygbvfgRcUfYg6W54itDJM5xPWl+EbzBvpyqQ9ka3FpHf +6RUu4zC7Z48Kmc7hKB/JoHRE4O6K01pmD3bHp8tgXd67Zqt7yGTCcHywUN/ke22xyWV mXq4lxzVktHz8EIBY/t4wyfuF9cG37OY4mC7dAowBmQzRoh3KWy4emlYXNnU9Jkqcr8n 9VRw== MIME-Version: 1.0 Received: by 10.152.108.42 with SMTP id hh10mr18390086lab.4.1354116971570; Wed, 28 Nov 2012 07:36:11 -0800 (PST) Received: by 10.112.4.2 with HTTP; Wed, 28 Nov 2012 07:36:11 -0800 (PST) In-Reply-To: <20121128151336.GC17871@dft-labs.eu> References: <20121128104218.GA17871@dft-labs.eu> <20121128151336.GC17871@dft-labs.eu> Date: Wed, 28 Nov 2012 17:36:11 +0200 Message-ID: Subject: Re: NANDFS eats itself up From: Boris Astardzhiev To: Mateusz Guzik Content-Type: multipart/mixed; boundary=bcaec54fb9c61327f904cf8fed4b X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org, gjb@semihalf.com, Grzegorz Bernacki , stanislav_galabov@smartcom.bg X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 15:36:13 -0000 --bcaec54fb9c61327f904cf8fed4b Content-Type: text/plain; charset=ISO-8859-1 If I do that I might be unable to set back the verbose level to 0 since the output is very very NOISY. This means that I will have to start again from the beginning if I need to reproduce it. Nevertheless here you go. Check the attachment (OUTPUTNAND.txt.bz2). On Wed, Nov 28, 2012 at 5:13 PM, Mateusz Guzik wrote: > On Wed, Nov 28, 2012 at 05:06:12PM +0200, Boris Astardzhiev wrote: > > Yes, I do. I've made this to repeat via a script. The interval between > each > > SCP transfer is 100s. I thought it would give just enough time to the fs > > for reclaiming its space back. > > > > Now I stopped the transfers and doubled the vfs.nandfs.cleaner_segments > to > > 10. The fs has no files in it but it reports "~9.6MB" of used space. 4+ > > hours later it has NOT changed at all it still reports "~9.6MB" Used. > > > > Now this indeed sounds like a bug. Can you enable debug like this: > # sysctl vfs.nandfs.verbose=0xffffffff > > The kernel will start printing a lot of debugging information. With > default configuration of syslog this will end up in /var/log/messages. > Capture something like 30 seconds of output and post it somewhere. > > -- > Mateusz Guzik > --bcaec54fb9c61327f904cf8fed4b-- From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 15:55:04 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DBCF1289 for ; Wed, 28 Nov 2012 15:55:04 +0000 (UTC) (envelope-from metindoslu@gmail.com) Received: from mail-qa0-f54.google.com (mail-qa0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 906D28FC0C for ; Wed, 28 Nov 2012 15:55:04 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id g24so5684770qab.13 for ; Wed, 28 Nov 2012 07:55:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; bh=VbHXriEGsr/88masLf5fbDS5zMLJOGhmoyF2MWbtTD8=; b=g2dukh1frmXsPP6OAlgcoSR0Z79NiXwJ8eZlQJBI8cTuFgWEfEOj9qteNbOEJenRoV wfR/Qha9DvIH78Annv8MsU6qXkl9a7g5g+QnoqdhigJEqG/bdUhaz/gR5LRRW10YpYMC H5QOfmZ1A6I4YhK8aX/9CI0MDM9E81pjoL4aA5Rilq8v1v1kYysgxgeEXla2GofCThhF c1LWAbqE36hg1y+O9RHWDDvFOgHaEijXL7YVGvy96UC4fr0Hj+WU2gOcCz367z0ArwXW iLyYWBpz2D77TOIGwSTYOJp0DLJZ01r+IEl7ROWKr83ClL30Z1mQKhCgWbhD/42hozrG Yi9A== Received: by 10.49.75.226 with SMTP id f2mr16265101qew.43.1354118102908; Wed, 28 Nov 2012 07:55:02 -0800 (PST) MIME-Version: 1.0 Received: by 10.49.35.242 with HTTP; Wed, 28 Nov 2012 07:54:42 -0800 (PST) In-Reply-To: References: From: =?UTF-8?B?TWV0aW4gRMO2xZ9sw7w=?= Date: Wed, 28 Nov 2012 17:54:42 +0200 Message-ID: Subject: Re: Performance Difference on UFS and ZFS To: Sergey Kandaurov Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 15:55:04 -0000 Hi Sergey, I tested it on a cc1.4xlarge EC2 instance, here is the specs: 23 GiB of memory 33.5 EC2 Compute Units (2 x Intel Xeon X5570, quad-core =E2=80=9CNehalem=E2= =80=9D architecture) 1690 GB of instance storage 64-bit platform I installed PostgreSQL from its port on FreeBSD. I didn't do any tuning for PostgreSQL or FreeBSD. Data access pattern consists of completely from sequential reads such "select count(*) from table_name". I measured performance with PostgreSQL's timing option. As as side note; all queries are served from memory, so there were no disk usage for these tests. Thanks, Metin On Wed, Nov 28, 2012 at 4:49 PM, Sergey Kandaurov wrote= : > On 28 November 2012 12:35, Metin D=C3=B6=C5=9Fl=C3=BC wrote: >> Hi - >> >> I installed PostgreSQL 9.1.6 on FreeBSD 9.0-RELEASE, and compared ZFS >> to UFS. I also made sure all data easily fit into memory, ran some >> sequential scan queries on the database. >> >> Query run times on UFS were 100-200% slower than those of ZFS. This >> was intriguing as all data came from memory. What could be causing >> such a large performance difference? > > Hi Metin D=C3=B6=C5=9Fl=C3=BC! > > That's interesting. Can you please share your settings? > Hardware configuration, system tuning (sysctl, zfs), PostgreSQL settings, > data access pattern (read/write ratio), and how you measure performance > would be a good start. > > -- > wbr, > pluknet --=20 Metin D=C3=B6=C5=9Fl=C3=BC From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 18:47:49 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 205B8230 for ; Wed, 28 Nov 2012 18:47:49 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8DB918FC15 for ; Wed, 28 Nov 2012 18:47:46 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id go10so10803078lbb.13 for ; Wed, 28 Nov 2012 10:47:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=UTL9/Zy6B5gfKMHri9oaTqCAPmJjU3fHhvpqi9so8CU=; b=G2jQ5TjQHwhkAQ3eZX3rxgFiCB6iM4U4I8txyGEOQqtZKWzW+T7VwcVClAMj5FQ9p2 KG5/gsMUARoa1gbAH7QZljuEOiQ5ttqTW/rF6KiK/iAP0LxwQxU35KNWeKcx9kUIHfi+ pdnStT/Hce5fG1aBappIlAiZrRBoim5eECPjbs3q7cwvwWCIgIOu2V1lOlvPxkIXnoDh 2iN9T5gEnZm6x1UOu6PNdwM2AHGDjEJowousS6236BAUOQbVNecJXeayaLkWILOMHgW8 dD7xcOfEQeewGjSQAqaOW6O8tEJPoKZF/+ZDtWxBk80Tk/FP2iC6SFrGFJZGDao935Yf F54g== MIME-Version: 1.0 Received: by 10.152.114.100 with SMTP id jf4mr19226084lab.47.1354128465550; Wed, 28 Nov 2012 10:47:45 -0800 (PST) Received: by 10.112.61.33 with HTTP; Wed, 28 Nov 2012 10:47:45 -0800 (PST) In-Reply-To: References: Date: Wed, 28 Nov 2012 13:47:45 -0500 Message-ID: Subject: Re: Performance Difference on UFS and ZFS From: Zaphod Beeblebrox To: =?UTF-8?B?TWV0aW4gRMO2xZ9sw7w=?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 18:47:49 -0000 On Wed, Nov 28, 2012 at 10:54 AM, Metin D=C3=B6=C5=9Fl=C3=BC wrote: > I tested it on a cc1.4xlarge EC2 instance, here is the specs: > 23 GiB of memory > 33.5 EC2 Compute Units (2 x Intel Xeon X5570, quad-core =E2=80=9CNehalem= =E2=80=9D architecture) > 1690 GB of instance storage > 64-bit platform I suppose the first _enormous_ question is whether this is true for real hardware rather than just virtual. From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 21:07:16 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C07294DD; Wed, 28 Nov 2012 21:07:16 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from cpsmtpb-ews10.kpnxchange.com (cpsmtpb-ews10.kpnxchange.com [213.75.39.15]) by mx1.freebsd.org (Postfix) with ESMTP id 14EFC8FC0C; Wed, 28 Nov 2012 21:07:13 +0000 (UTC) Received: from cpsps-ews07.kpnxchange.com ([10.94.84.174]) by cpsmtpb-ews10.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Wed, 28 Nov 2012 22:05:23 +0100 Received: from CPSMTPM-TLF103.kpnxchange.com ([195.121.3.6]) by cpsps-ews07.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Wed, 28 Nov 2012 22:05:23 +0100 Received: from sjakie.klop.ws ([212.182.167.131]) by CPSMTPM-TLF103.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Wed, 28 Nov 2012 22:06:05 +0100 Received: from 212-182-167-131.ip.telfort.nl (localhost [127.0.0.1]) by sjakie.klop.ws (Postfix) with ESMTP id 24B9C4680; Wed, 28 Nov 2012 22:06:05 +0100 (CET) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: "Mateusz Guzik" , "Boris Astardzhiev" Subject: Re: NANDFS eats itself up References: <20121128104218.GA17871@dft-labs.eu> <20121128151336.GC17871@dft-labs.eu> Date: Wed, 28 Nov 2012 22:06:05 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: User-Agent: Opera Mail/12.11 (FreeBSD) X-OriginalArrivalTime: 28 Nov 2012 21:06:05.0702 (UTC) FILETIME=[2E36BA60:01CDCDAC] X-RcptDomain: freebsd.org Cc: freebsd-fs@freebsd.org, gjb@semihalf.com, Grzegorz Bernacki , stanislav_galabov@smartcom.bg X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 21:07:17 -0000 On Wed, 28 Nov 2012 16:36:11 +0100, Boris Astardzhiev wrote: > If I do that I might be unable to set back the verbose level to 0 since > the > output is very very NOISY. This means that I will have to start again > from > the beginning if I need to reproduce it. Nevertheless here you go. Check > the attachment (OUTPUTNAND.txt.bz2). Your attachment is stripped by the mailinglist. Ronald. > On Wed, Nov 28, 2012 at 5:13 PM, Mateusz Guzik wrote: > >> On Wed, Nov 28, 2012 at 05:06:12PM +0200, Boris Astardzhiev wrote: >> > Yes, I do. I've made this to repeat via a script. The interval between >> each >> > SCP transfer is 100s. I thought it would give just enough time to the >> fs >> > for reclaiming its space back. >> > >> > Now I stopped the transfers and doubled the >> vfs.nandfs.cleaner_segments >> to >> > 10. The fs has no files in it but it reports "~9.6MB" of used space. >> 4+ >> > hours later it has NOT changed at all it still reports "~9.6MB" Used. >> > >> >> Now this indeed sounds like a bug. Can you enable debug like this: >> # sysctl vfs.nandfs.verbose=0xffffffff >> >> The kernel will start printing a lot of debugging information. With >> default configuration of syslog this will end up in /var/log/messages. >> Capture something like 30 seconds of output and post it somewhere. >> >> -- >> Mateusz Guzik From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 21:20:54 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5857A66E for ; Wed, 28 Nov 2012 21:20:54 +0000 (UTC) (envelope-from jas@cse.yorku.ca) Received: from bronze.cs.yorku.ca (bronze.cs.yorku.ca [130.63.95.34]) by mx1.freebsd.org (Postfix) with ESMTP id 113188FC14 for ; Wed, 28 Nov 2012 21:20:53 +0000 (UTC) Received: from [130.63.97.125] (ident=jas) by bronze.cs.yorku.ca with esmtp (Exim 4.76) (envelope-from ) id 1Tdp3m-0008K9-G4; Wed, 28 Nov 2012 16:20:46 -0500 Message-ID: <50B6802E.50009@cse.yorku.ca> Date: Wed, 28 Nov 2012 16:20:46 -0500 From: Jason Keltz User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:10.0.9) Gecko/20121011 Thunderbird/10.0.9 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: ZFS/NFS performance file server References: <50A130B7.4080604@cse.yorku.ca> <20121113043409.GA70601@neutralgood.org> <50A2B95D.4000400@cse.yorku.ca> <50A2F804.3010009@freebsd.org> <20121115001840.GA27399@FreeBSD.org> <20121115102704.6657ee52@suse3> <20121116091747.2c1bfc55@suse3> <16B803FB-0964-4237-8F25-291470E7EFB5@gmail.com> In-Reply-To: <16B803FB-0964-4237-8F25-291470E7EFB5@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -1.0 X-Spam-Level: - X-Spam-Report: Content preview: I've been experimenting with filebench for ZFS performance testing on my new file server running 9.1RC3. Using filebench's "fileserver" workload, and testing a variety of different pool configurations, I am clearly able to see the differences in performance between the various ZFS layouts. I'm able to repeat the tests and get similar results each time. I really like this tool! I feel that the fileserver workload is probably more representative of my actual file server workload than say iozone/bonnie (in fact, probably heavier than my server workload). Presently, I'm just using the fileserver workload with default configuration. [...] Content analysis details: (-1.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SHORTCIRCUIT Not all rules were run, due to a shortcircuited rule -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 21:20:54 -0000 I've been experimenting with filebench for ZFS performance testing on my new file server running 9.1RC3. Using filebench's "fileserver" workload, and testing a variety of different pool configurations, I am clearly able to see the differences in performance between the various ZFS layouts. I'm able to repeat the tests and get similar results each time. I really like this tool! I feel that the fileserver workload is probably more representative of my actual file server workload than say iozone/bonnie (in fact, probably heavier than my server workload). Presently, I'm just using the fileserver workload with default configuration. After configuring the zpool on my system (presently 11 mirrored vdevs, but soon to be triple mirror), filebench gave me these numbers: statfile1 297131ops 4945ops/s 0.0mb/s 1.1ms/op 0us/op-cpu [0ms - 254ms] deletefile1 297141ops 4945ops/s 0.0mb/s 2.6ms/op 0us/op-cpu [0ms - 269ms] closefile3 297143ops 4945ops/s 0.0mb/s 0.1ms/op 0us/op-cpu [0ms - 14ms] readfile1 297144ops 4945ops/s 657.2mb/s 0.2ms/op 0us/op-cpu [0ms - 29ms] openfile2 297150ops 4946ops/s 0.0mb/s 1.1ms/op 0us/op-cpu [0ms - 254ms] closefile2 297150ops 4946ops/s 0.0mb/s 0.0ms/op 0us/op-cpu [0ms - 13ms] appendfilerand1 297150ops 4946ops/s 38.6mb/s 0.7ms/op 0us/op-cpu [0ms - 247ms] openfile1 297154ops 4946ops/s 0.0mb/s 1.1ms/op 0us/op-cpu [0ms - 268ms] closefile1 297155ops 4946ops/s 0.0mb/s 0.0ms/op 0us/op-cpu [0ms - 13ms] wrtfile1 297168ops 4946ops/s 621.1mb/s 0.8ms/op 0us/op-cpu [0ms - 67ms] createfile1 297172ops 4946ops/s 0.0mb/s 2.2ms/op 0us/op-cpu [0ms - 55ms] 57845: 64.858: IO Summary: 3268658 ops, 54401.548 ops/s, (4945/9891 r/w), 1316.9mb/s, 0us cpu/op, 3.3ms latency Next, I wanted to try NFS testing from a Linux client. All of my NFS clients are running RHEL63. I believe that the filebench fileserver workload is intended to be run locally on the file server, but I don't see any reason why I cannot run it on the client as well. Later, I would probably change the default 50 threads of activity to less (say, 2), and run the tool on say 100 or more hosts simultaneously to get a better idea of whether all clients get the performance that I would expect without the server being too heavily loaded. However, that's for another day! From people who use filebench - does this approach make some sense? I ran into some interesting results on my NFS testing, after which, I'm not quite sure I will use NFSv4 for this file server project, but I'm interested in feedback.... Note that the file server has a 1 gbit/s network connection, but most clients are on 100 mbit/s. Here's filebench running from a 100mbps client connected to the new file server with NFSv4 with the above pool all to itself: 21512: 258.862: IO Summary: 38537 ops, 642.082 ops/s, (58/117 r/w), 15.0mb/s, 3430us cpu/op, 280.9ms latency With the client on a 100mbit/s link, maximum throughput I would expect would be 12.5 MB/s. I'm not using compression at the moment (I know that this could inflate the number). In addition, I exported the pool, and re-imported it before running the test, and unmounted and mounted it on the client. Where does the extra 2.5mb/s come from? I'm assuming it has to do with caching that occurs during the test. I need to understand more clearly how filebench is generating the files, but I suspect it would be hard to get away from caching altogether. I don't really expect to actually GET 100mbit/s out of a 100mbit/s link, so I'm curious how much of the 15mb/s is "real". When I run the identical filebench test from the same 100mbps client but to my OLD file server (CentOS 4, NFS v3, running ext3 on an old 3ware card with a 6 disk RAID10 also gigabit), the numbers that I get are: 22991: 257.779: IO Summary: 46672 ops, 777.635 ops/s, (71/142 r/w), 18.1mb/s, 1543us cpu/op, 227.9ms latency NFSv4 to new server: 15 mb/s NFSv3 to old server: 18 mb/s Hmmm... 100 mbps client to the new file server with NFSv3 instead of NFSv4: 22460: 369.760: IO Summary: 64934 ops, 1081.895 ops/s, (98/197 r/w), 25.4mb/s, 2203us cpu/op, 166.7ms latency If I repeat that test, as long as I zfs export/import and unmount/mount the filesystem, that number is very consistent.... So... NFSv4: 15 MB/s and NFSv3: 25.4 MB/s 10 MB/s difference? There has been some discussion on the list before about the difference in performance between NFSv3 and NFSv4. I suspect this is just a sample of that... (I don't think this is just a FreeBSD thing - I think it's an NFSv4 thing.) So what happens if the client is on the same gigabit network as the new file server... NFSv4: 27669: 180.309: IO Summary: 55844 ops, 930.619 ops/s, (84/170 r/w), 21.6mb/s, 763us cpu/op, 193.3ms latency NFSv3: 28891: 176.382: IO Summary: 72068 ops, 1201.054 ops/s, (109/219 r/w), 28.1mb/s, 555us cpu/op, 150.0ms latency Hey wait ... sure, they are "closer", but 28.1 mb/s? Before, I was trying to understand how my performance number was OVER 100 mbps -- now it's significantly under gigabit??? This got to me to thinking ... hey, I don't have a separate ZIL .... so just for overall performance, I thought I'd disable the ZIL (zfs set sync=disabled) and see what happens to performance NFSv3 and NFSv4 on the same gigabit client test: NFSv4: 30077: 128.518: IO Summary: 49346 ops, 822.368 ops/s, (75/150 r/w), 19.2mb/s, 736us cpu/op, 218.9ms latency NFSv3: 29484: 110.466: IO Summary: 293843 ops, 4897.058 ops/s, (445/891 r/w), 115.7mb/s, 532us cpu/op, 36.4ms latency The results were consistent. I repeated them many times. Rick -- any ideas on this one!? So NFSv3 version gets closer to gigabit speed, but NFSv4 stays really slow - slower than having no ZIL!? (Clearly, I must be doing something wrong.) I'm not going to disable ZIL on a production system, but this testing would indicate to me that: 1) I'm probably better off sticking with NFSv3 for this server. NFSv4 was just "simpler" and I believe that the new ACLs could be helpful in my environment. 2) Without a fast ZIL, 1 100 mbps client didn't seem to have performance issues... but when I have 100+ clients all 100mbps talking to the server, I suspect a ZIL will be more important. With just one server talking to the file server at gigabit speed, it looks like the ZIL is necessary.... Unfortunately, in this respect, I'm told that SSD ZILs under FreeBSD 9.X can be tricky business -- with no TRIM support, and no secure erase support (except in HEAD), when garbage collection happens after the SSD has been used for a long while, there may be performance degradation.... sigh. Can you win!? Feedback? Jason. From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 21:47:18 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 52B44222; Wed, 28 Nov 2012 21:47:18 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 7B3928FC12; Wed, 28 Nov 2012 21:47:17 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id go10so10980429lbb.13 for ; Wed, 28 Nov 2012 13:47:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=m6rcuq+Xk71YeakYF7dmdb/mAcsfmTmRoqyye6hq9YM=; b=VuG0c/fESrMrvc8Y/M9fnhJbhp0NvpJvrT9AWYCmIITQMlvjU//2SacPxiXppe2RM0 PKj9rgkQlqB7owFanV5WK0/3Uhcd8XgSwtUilEdVWPsKS0QELL5rMFhXhtNQHhqM066g TiqvwRyLYO1gcLrpx4Kq1E3Qo/qm+5eVsZlmtqLxSepl1Grd9pSTmkNUBiF0NnMT/mS4 kGkh3rFeWyFLzSQcM/ZjFCVriGB1xPhchP6Es4w2ch+fEs9MFqVrCodH08jnF16t0vpu 2q4WQ9u5W+Ekn8NNG8OjSZN+AXG6yt63N3qMq5Gfidb4wW9+hgC/CcK/DtXvNYt4Mj8N h+xw== MIME-Version: 1.0 Received: by 10.112.103.135 with SMTP id fw7mr3929840lbb.17.1354139235928; Wed, 28 Nov 2012 13:47:15 -0800 (PST) Received: by 10.112.4.2 with HTTP; Wed, 28 Nov 2012 13:47:15 -0800 (PST) In-Reply-To: References: <20121128104218.GA17871@dft-labs.eu> <20121128151336.GC17871@dft-labs.eu> Date: Wed, 28 Nov 2012 23:47:15 +0200 Message-ID: Subject: Re: NANDFS eats itself up From: Boris Astardzhiev To: Ronald Klop Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org, gjb@semihalf.com, stanislav_galabov@smartcom.bg, Grzegorz Bernacki X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 21:47:18 -0000 Sorry for that, I haven't noticed it. I hope that this will do. The attachment - http://justpaste.it/1kk5 On Wed, Nov 28, 2012 at 11:06 PM, Ronald Klop wrote: > On Wed, 28 Nov 2012 16:36:11 +0100, Boris Astardzhiev < > boris.astardzhiev@gmail.com> wrote: > > If I do that I might be unable to set back the verbose level to 0 since >> the >> output is very very NOISY. This means that I will have to start again from >> the beginning if I need to reproduce it. Nevertheless here you go. Check >> the attachment (OUTPUTNAND.txt.bz2). >> > > Your attachment is stripped by the mailinglist. > > Ronald. > > > > On Wed, Nov 28, 2012 at 5:13 PM, Mateusz Guzik wrote: >> >> On Wed, Nov 28, 2012 at 05:06:12PM +0200, Boris Astardzhiev wrote: >>> > Yes, I do. I've made this to repeat via a script. The interval between >>> each >>> > SCP transfer is 100s. I thought it would give just enough time to the >>> fs >>> > for reclaiming its space back. >>> > >>> > Now I stopped the transfers and doubled the vfs.nandfs.cleaner_segments >>> to >>> > 10. The fs has no files in it but it reports "~9.6MB" of used space. 4+ >>> > hours later it has NOT changed at all it still reports "~9.6MB" Used. >>> > >>> >>> Now this indeed sounds like a bug. Can you enable debug like this: >>> # sysctl vfs.nandfs.verbose=0xffffffff >>> >>> The kernel will start printing a lot of debugging information. With >>> default configuration of syslog this will end up in /var/log/messages. >>> Capture something like 30 seconds of output and post it somewhere. >>> >>> -- >>> Mateusz Guzik >>> >> From owner-freebsd-fs@FreeBSD.ORG Wed Nov 28 23:31:39 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A327CCF4 for ; Wed, 28 Nov 2012 23:31:39 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 3B2778FC08 for ; Wed, 28 Nov 2012 23:31:38 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEACuetlCDaFvO/2dsb2JhbABFhiq6DnOCHgEBAQMBAQEBICsgCwUWDgoCAg0ZAikBCSYGCAcEARwBA4dpBgysGpJ7gSKLOIMTgRMDiF6KdoItgRyPKIMQgUUFBBce X-IronPort-AV: E=Sophos;i="4.84,180,1355115600"; d="scan'208";a="2467989" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu.net.uoguelph.ca with ESMTP; 28 Nov 2012 18:31:31 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id B7FC4B3F13; Wed, 28 Nov 2012 18:31:31 -0500 (EST) Date: Wed, 28 Nov 2012 18:31:31 -0500 (EST) From: Rick Macklem To: Jason Keltz Message-ID: <1544373868.953885.1354145491686.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <50B6802E.50009@cse.yorku.ca> Subject: Re: ZFS/NFS performance file server MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE7 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 23:31:39 -0000 Jason Keltz wrote: > I've been experimenting with filebench for ZFS performance testing on > my > new file server running 9.1RC3. Using filebench's "fileserver" > workload, > and testing a variety of different pool configurations, I am clearly > able to see the differences in performance between the various ZFS > layouts. I'm able to repeat the tests and get similar results each > time. I really like this tool! I feel that the fileserver workload is > probably more representative of my actual file server workload than > say > iozone/bonnie (in fact, probably heavier than my server workload). > Presently, I'm just using the fileserver workload with default > configuration. > > After configuring the zpool on my system (presently 11 mirrored vdevs, > but soon to be triple mirror), filebench gave me these numbers: > > statfile1 297131ops 4945ops/s 0.0mb/s > 1.1ms/op 0us/op-cpu [0ms - 254ms] > deletefile1 297141ops 4945ops/s 0.0mb/s > 2.6ms/op 0us/op-cpu [0ms - 269ms] > closefile3 297143ops 4945ops/s 0.0mb/s > 0.1ms/op 0us/op-cpu [0ms - 14ms] > readfile1 297144ops 4945ops/s 657.2mb/s > 0.2ms/op 0us/op-cpu [0ms - 29ms] > openfile2 297150ops 4946ops/s 0.0mb/s > 1.1ms/op 0us/op-cpu [0ms - 254ms] > closefile2 297150ops 4946ops/s 0.0mb/s > 0.0ms/op 0us/op-cpu [0ms - 13ms] > appendfilerand1 297150ops 4946ops/s 38.6mb/s > 0.7ms/op 0us/op-cpu [0ms - 247ms] > openfile1 297154ops 4946ops/s 0.0mb/s > 1.1ms/op 0us/op-cpu [0ms - 268ms] > closefile1 297155ops 4946ops/s 0.0mb/s > 0.0ms/op 0us/op-cpu [0ms - 13ms] > wrtfile1 297168ops 4946ops/s 621.1mb/s > 0.8ms/op 0us/op-cpu [0ms - 67ms] > createfile1 297172ops 4946ops/s 0.0mb/s > 2.2ms/op 0us/op-cpu [0ms - 55ms] > 57845: 64.858: IO Summary: 3268658 ops, 54401.548 ops/s, (4945/9891 > r/w), 1316.9mb/s, 0us cpu/op, 3.3ms latency > > Next, I wanted to try NFS testing from a Linux client. All of my NFS > clients are running RHEL63. I believe that the filebench fileserver > workload is intended to be run locally on the file server, but I don't > see any reason why I cannot run it on the client as well. Later, I > would probably change the default 50 threads of activity to less (say, > 2), and run the tool on say 100 or more hosts simultaneously to get a > better idea of whether all clients get the performance that I would > expect without the server being too heavily loaded. However, that's > for > another day! From people who use filebench - does this approach make > some sense? > > I ran into some interesting results on my NFS testing, after which, > I'm > not quite sure I will use NFSv4 for this file server project, but I'm > interested in feedback.... > > Note that the file server has a 1 gbit/s network connection, but most > clients are on 100 mbit/s. > > Here's filebench running from a 100mbps client connected to the new > file > server with NFSv4 with the above pool all to itself: > > 21512: 258.862: IO Summary: 38537 ops, 642.082 ops/s, (58/117 r/w), > 15.0mb/s, 3430us cpu/op, 280.9ms latency > > With the client on a 100mbit/s link, maximum throughput I would expect > would be 12.5 MB/s. I'm not using compression at the moment (I know > that this could inflate the number). In addition, I exported the pool, > and re-imported it before running the test, and unmounted and mounted > it > on the client. Where does the extra 2.5mb/s come from? I'm assuming it > has to do with caching that occurs during the test. I need to > understand more clearly how filebench is generating the files, but I > suspect it would be hard to get away from caching altogether. I don't > really expect to actually GET 100mbit/s out of a 100mbit/s link, so > I'm > curious how much of the 15mb/s is "real". > > When I run the identical filebench test from the same 100mbps client > but > to my OLD file server (CentOS 4, NFS v3, running ext3 on an old 3ware > card with a 6 disk RAID10 also gigabit), the numbers that I get are: > > 22991: 257.779: IO Summary: 46672 ops, 777.635 ops/s, (71/142 r/w), > 18.1mb/s, 1543us cpu/op, 227.9ms latency > > NFSv4 to new server: 15 mb/s > NFSv3 to old server: 18 mb/s > > Hmmm... > > 100 mbps client to the new file server with NFSv3 instead of NFSv4: > > 22460: 369.760: IO Summary: 64934 ops, 1081.895 ops/s, (98/197 r/w), > 25.4mb/s, 2203us cpu/op, 166.7ms latency > > If I repeat that test, as long as I zfs export/import and > unmount/mount > the filesystem, that number is very consistent.... > > So... > NFSv4: 15 MB/s and > NFSv3: 25.4 MB/s > > 10 MB/s difference? There has been some discussion on the list before > about the difference in performance between NFSv3 and NFSv4. I suspect > this is just a sample of that... > > (I don't think this is just a FreeBSD thing - I think it's an NFSv4 > thing.) > > So what happens if the client is on the same gigabit network as the > new > file server... > > NFSv4: > 27669: 180.309: IO Summary: 55844 ops, 930.619 ops/s, (84/170 r/w), > 21.6mb/s, 763us cpu/op, 193.3ms latency > > NFSv3: > 28891: 176.382: IO Summary: 72068 ops, 1201.054 ops/s, (109/219 r/w), > 28.1mb/s, 555us cpu/op, 150.0ms latency > > Hey wait ... sure, they are "closer", but 28.1 mb/s? Before, I was > trying to understand how my performance number was OVER 100 mbps -- > now > it's significantly under gigabit??? > > This got to me to thinking ... hey, I don't have a separate ZIL .... > so > just for overall performance, I thought I'd disable the ZIL (zfs set > sync=disabled) and see what happens to performance NFSv3 and NFSv4 on > the same gigabit client test: > > NFSv4: > 30077: 128.518: IO Summary: 49346 ops, 822.368 ops/s, (75/150 r/w), > 19.2mb/s, 736us cpu/op, 218.9ms latency > > NFSv3: > > 29484: 110.466: IO Summary: 293843 ops, 4897.058 ops/s, (445/891 r/w), > 115.7mb/s, 532us cpu/op, 36.4ms latency > > The results were consistent. I repeated them many times. Rick -- any > ideas on this one!? > Since you are using a Linux client, I can't really guess. For a FreeBSD client, the I/O is done the same way, but there is extra overhead for doing the opens (a file needs to open'd first for NFSv4, unless the client holds a delegation). You could try enabling delegations, but I don't know what effect that has on the Linux client. For the FreeBSD|server: sysctl vfs.nfsd.issue_delegations=1 Also, you might want to experiment with different rsize/wsize. If you get a case where the sender (client or server) generates a burst of traffic that the receiver can't reliably receive, you'll see better performance with a smaller rsize,wsize. The main advantage of NFSv4 is better byte range locking. If your client apps. don't care (or lockd works well enough in your case), then I'm not sure NFSv4 is needed. rick > So NFSv3 version gets closer to gigabit speed, but NFSv4 stays really > slow - slower than having no ZIL!? > (Clearly, I must be doing something wrong.) > > I'm not going to disable ZIL on a production system, but this testing > would indicate to me that: > > 1) I'm probably better off sticking with NFSv3 for this server. NFSv4 > was just "simpler" and I believe that the new ACLs could be helpful in > my environment. > 2) Without a fast ZIL, 1 100 mbps client didn't seem to have > performance > issues... but when I have 100+ clients all 100mbps talking to the > server, I suspect a ZIL will be more important. With just one server > talking to the file server at gigabit speed, it looks like the ZIL is > necessary.... Unfortunately, in this respect, I'm told that SSD ZILs > under FreeBSD 9.X can be tricky business -- with no TRIM support, and > no > secure erase support (except in HEAD), when garbage collection happens > after the SSD has been used for a long while, there may be performance > degradation.... sigh. Can you win!? > > Feedback? > > Jason. > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Thu Nov 29 09:30:06 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C163FF00; Thu, 29 Nov 2012 09:30:06 +0000 (UTC) (envelope-from raymondj@caltech.edu) Received: from outgoing-mail.its.caltech.edu (outgoing-mail.its.caltech.edu [131.215.239.19]) by mx1.freebsd.org (Postfix) with ESMTP id A29EB8FC12; Thu, 29 Nov 2012 09:30:06 +0000 (UTC) Received: from fire-doxen.imss.caltech.edu (localhost [127.0.0.1]) by fire-doxen-postvirus (Postfix) with ESMTP id B9E8E328021; Thu, 29 Nov 2012 01:29:59 -0800 (PST) X-Spam-Scanned: at Caltech-IMSS on fire-doxen by amavisd-new Received: from [127.0.0.1] (mitsuki.caltech.edu [131.215.167.33]) (Authenticated sender: raymondj) by fire-doxen-submit (Postfix) with ESMTP id C58572E50D2F; Thu, 29 Nov 2012 01:29:56 -0800 (PST) Message-ID: <50B72AFD.3040902@caltech.edu> Date: Thu, 29 Nov 2012 01:29:33 -0800 From: Raymond Jimenez User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Andriy Gapon Subject: Re: ZFS kernel panics due to corrupt DVAs (despite RAIDZ) References: <50B3E680.8060606@caltech.edu> <50B49F6A.2020509@FreeBSD.org> In-Reply-To: <50B49F6A.2020509@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2012 09:30:06 -0000 Hi Andriy, On 11/27/2012 3:09 AM, Andriy Gapon wrote: > > Perhaps this thread could be of some interest to you: > http://thread.gmane.org/gmane.os.freebsd.devel.file-systems/15611/focus=15616 > Thank you for the pointer. Unfortunately, a scrub segfaults in the same place with the same output. > For one reason or the other wrong data (but correct looking - proper checksums, > etc) got written to the disk. I'd say use the patch, lift the data and > re-create the pool. Since it's corrupt data coming from higher levels, is there any possibility of getting this data back? Is it worth debugging more to add checks to catch this, or are these scenarios be vanishingly small? Thank you, Raymond Jimenez From owner-freebsd-fs@FreeBSD.ORG Thu Nov 29 09:47:39 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A032B426 for ; Thu, 29 Nov 2012 09:47:39 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id E4FC88FC12 for ; Thu, 29 Nov 2012 09:47:38 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id LAA08617; Thu, 29 Nov 2012 11:47:26 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Te0iM-0000qh-GY; Thu, 29 Nov 2012 11:47:26 +0200 Message-ID: <50B72F2E.6080808@FreeBSD.org> Date: Thu, 29 Nov 2012 11:47:26 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Raymond Jimenez Subject: Re: ZFS kernel panics due to corrupt DVAs (despite RAIDZ) References: <50B3E680.8060606@caltech.edu> <50B49F6A.2020509@FreeBSD.org> <50B72AFD.3040902@caltech.edu> In-Reply-To: <50B72AFD.3040902@caltech.edu> X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2012 09:47:40 -0000 on 29/11/2012 11:29 Raymond Jimenez said the following: > Hi Andriy, > > On 11/27/2012 3:09 AM, Andriy Gapon wrote: >> >> Perhaps this thread could be of some interest to you: >> http://thread.gmane.org/gmane.os.freebsd.devel.file-systems/15611/focus=15616 >> > > Thank you for the pointer. Unfortunately, a scrub segfaults in the same > place with the same output. My intention was to show the debugging techniques for examining that troublesome data. >> For one reason or the other wrong data (but correct looking - proper checksums, >> etc) got written to the disk. I'd say use the patch, lift the data and >> re-create the pool. > > Since it's corrupt data coming from higher levels, is there any > possibility of getting this data back? Is it worth debugging more to > add checks to catch this, or are these scenarios be vanishingly > small? I do not have very many scenarios in mind which could lead to problems like that. And that those that I have look very improbable and "uncatchable". E.g. buggy kernel code over-writing some portion of a data buffer. Or some "rogue" DMA transaction doing the same. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Thu Nov 29 10:13:57 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CA35AC76; Thu, 29 Nov 2012 10:13:57 +0000 (UTC) (envelope-from raymondj@caltech.edu) Received: from outgoing-mail.its.caltech.edu (outgoing-mail.its.caltech.edu [131.215.239.19]) by mx1.freebsd.org (Postfix) with ESMTP id A6D168FC12; Thu, 29 Nov 2012 10:13:57 +0000 (UTC) Received: from fire-doxen.imss.caltech.edu (localhost [127.0.0.1]) by fire-doxen-postvirus (Postfix) with ESMTP id 9267532802C; Thu, 29 Nov 2012 02:13:57 -0800 (PST) X-Spam-Scanned: at Caltech-IMSS on fire-doxen by amavisd-new Received: from [127.0.0.1] (mitsuki.caltech.edu [131.215.167.33]) (Authenticated sender: raymondj) by fire-doxen-submit (Postfix) with ESMTP id 9B981328021; Thu, 29 Nov 2012 02:13:51 -0800 (PST) Message-ID: <50B73548.4020802@caltech.edu> Date: Thu, 29 Nov 2012 02:13:28 -0800 From: Raymond Jimenez User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Andriy Gapon Subject: Re: ZFS kernel panics due to corrupt DVAs (despite RAIDZ) References: <50B3E680.8060606@caltech.edu> <50B49F6A.2020509@FreeBSD.org> <50B72AFD.3040902@caltech.edu> <50B72F2E.6080808@FreeBSD.org> In-Reply-To: <50B72F2E.6080808@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2012 10:13:57 -0000 On 11/29/2012 1:47 AM, Andriy Gapon wrote: > on 29/11/2012 11:29 Raymond Jimenez said the following: >> Hi Andriy, >> >> On 11/27/2012 3:09 AM, Andriy Gapon wrote: >>> >>> Perhaps this thread could be of some interest to you: >>> http://thread.gmane.org/gmane.os.freebsd.devel.file-systems/15611/focus=15616 >>> >> >> Thank you for the pointer. Unfortunately, a scrub segfaults in the same >> place with the same output. > > My intention was to show the debugging techniques for examining that troublesome > data. > I see, thank you very much. It did help a considerable bit, especially showing how to use zdb with the debugging information. I'll continue working on this and maybe if I'm lucky, I can come up with some patch for my system. Raymond Jimenez From owner-freebsd-fs@FreeBSD.ORG Thu Nov 29 10:30:23 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CAAB8EF1 for ; Thu, 29 Nov 2012 10:30:23 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 200578FC08 for ; Thu, 29 Nov 2012 10:30:22 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id MAA09165; Thu, 29 Nov 2012 12:30:15 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Te1Nm-0000u9-S2; Thu, 29 Nov 2012 12:30:14 +0200 Message-ID: <50B73935.2090805@FreeBSD.org> Date: Thu, 29 Nov 2012 12:30:13 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Raymond Jimenez Subject: Re: ZFS kernel panics due to corrupt DVAs (despite RAIDZ) References: <50B3E680.8060606@caltech.edu> <50B49F6A.2020509@FreeBSD.org> <50B72AFD.3040902@caltech.edu> <50B72F2E.6080808@FreeBSD.org> <50B73548.4020802@caltech.edu> In-Reply-To: <50B73548.4020802@caltech.edu> X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2012 10:30:23 -0000 on 29/11/2012 12:13 Raymond Jimenez said the following: > On 11/29/2012 1:47 AM, Andriy Gapon wrote: >> on 29/11/2012 11:29 Raymond Jimenez said the following: >>> Hi Andriy, >>> >>> On 11/27/2012 3:09 AM, Andriy Gapon wrote: >>>> >>>> Perhaps this thread could be of some interest to you: >>>> http://thread.gmane.org/gmane.os.freebsd.devel.file-systems/15611/focus=15616 >>>> >>> >>> Thank you for the pointer. Unfortunately, a scrub segfaults in the same >>> place with the same output. >> >> My intention was to show the debugging techniques for examining that troublesome >> data. >> > > I see, thank you very much. It did help a considerable bit, especially > showing how to use zdb with the debugging information. I'll continue > working on this and maybe if I'm lucky, I can come up with some patch > for my system. Just in case, there was a parallel thread about the same problem: http://thread.gmane.org/gmane.os.illumos.zfs/35 -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Thu Nov 29 11:07:29 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0D54A9B8; Thu, 29 Nov 2012 11:07:29 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) by mx1.freebsd.org (Postfix) with ESMTP id 5985E8FC14; Thu, 29 Nov 2012 11:07:28 +0000 (UTC) Received: by mail-wi0-f174.google.com with SMTP id hm9so5417504wib.13 for ; Thu, 29 Nov 2012 03:07:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=ub1vlJUCs1SG05tKXS+C6BsBugB7KjKBO86CPz894u8=; b=w+Jv5L9yjRaewXnkcI78QenEnH/01WV/Vjcra4djbk5V6BU5l+YZ5DKQKIw0mYz0nG 9U3AUfxA5t/Ea+qtTx4t4dAv3Sh97BeAlhD1JfMPoquqd660yU+7F+dD/KGta5BvC2bO N4sit2OFzNcPhO1SKQQVahQBKlNtAAQeoS7If9a470HK7+jhI721qvIGlqm/vq3tO0if jGbDY444oFUysCT6Lo4X1zyT0QaNKDXrNOMDb+EljErtVSE19PC4mdDkn7XBzd08gWnT 7ovpNplUDb7bGwGmeX7WqXLs8/sINXWS+lvFBeZlbhCCr5Ecdg2uU9LrQ+Uuh+QOTqXI z6Vw== Received: by 10.180.102.2 with SMTP id fk2mr34371368wib.10.1354187247251; Thu, 29 Nov 2012 03:07:27 -0800 (PST) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPS id b1sm1881223wix.11.2012.11.29.03.07.25 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 29 Nov 2012 03:07:25 -0800 (PST) Date: Thu, 29 Nov 2012 12:07:20 +0100 From: Mateusz Guzik To: Boris Astardzhiev Subject: Re: NANDFS eats itself up Message-ID: <20121129110719.GA26212@dft-labs.eu> References: <20121128104218.GA17871@dft-labs.eu> <20121128151336.GC17871@dft-labs.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, gjb@semihalf.com, Grzegorz Bernacki , stanislav_galabov@smartcom.bg X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2012 11:07:29 -0000 On Wed, Nov 28, 2012 at 05:36:11PM +0200, Boris Astardzhiev wrote: > If I do that I might be unable to set back the verbose level to 0 since the > output is very very NOISY. This means that I will have to start again from > the beginning if I need to reproduce it. Nevertheless here you go. Check > the attachment (OUTPUTNAND.txt.bz2). > I think I reproduced your problem. Possibly I'll have time to work on this next week, but no promises. -- Mateusz Guzik From owner-freebsd-fs@FreeBSD.ORG Thu Nov 29 11:13:58 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4B4C3B31; Thu, 29 Nov 2012 11:13:58 +0000 (UTC) (envelope-from boris.astardzhiev@gmail.com) Received: from mail-la0-f54.google.com (mail-la0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 829328FC12; Thu, 29 Nov 2012 11:13:56 +0000 (UTC) Received: by mail-la0-f54.google.com with SMTP id j13so14092778lah.13 for ; Thu, 29 Nov 2012 03:13:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=jjOYWiosRj2XyAr1Huesk0wU4s4TQ1RgL8nsRAoi7ec=; b=zHpX63je1QJM6nh8fo4yVgF+7Nbrt07+W53sL1GhkqTlx3t0EQtWN9tdPEMlIrNrh0 TspmJiiJwc7afc8iVVwtZnApWEpuY51At86GaRa0t9qS8WQIYFsXQPGdeYR3bC8FhD5J FcxQx7+8NZfL6PLAYcAKI6o6GTsLBM8/Jo1L4FTHw3DeD2pO8ImhFr7DowAH09LnW0MR IhANp6ZZJ8tjMMQHHKuw0GjV6/9DPiVPkwSWaJ5L+sYESFdMdsc7ALbZgekSrMoioWpQ 2W6yAPaIREd8pF0X8lwkDdletLu3b2XcboFX3NpQwHduODz9JGZSHsmmQe1QJD7/Wzr4 DLag== MIME-Version: 1.0 Received: by 10.112.16.106 with SMTP id f10mr9525768lbd.1.1354187636134; Thu, 29 Nov 2012 03:13:56 -0800 (PST) Received: by 10.112.4.2 with HTTP; Thu, 29 Nov 2012 03:13:55 -0800 (PST) In-Reply-To: <20121129110719.GA26212@dft-labs.eu> References: <20121128104218.GA17871@dft-labs.eu> <20121128151336.GC17871@dft-labs.eu> <20121129110719.GA26212@dft-labs.eu> Date: Thu, 29 Nov 2012 13:13:55 +0200 Message-ID: Subject: Re: NANDFS eats itself up From: Boris Astardzhiev To: Mateusz Guzik Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org, gjb@semihalf.com, Grzegorz Bernacki , stanislav_galabov@smartcom.bg X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2012 11:13:58 -0000 Good, If you need any assistance drop a mail. On Thu, Nov 29, 2012 at 1:07 PM, Mateusz Guzik wrote: > On Wed, Nov 28, 2012 at 05:36:11PM +0200, Boris Astardzhiev wrote: > > If I do that I might be unable to set back the verbose level to 0 since > the > > output is very very NOISY. This means that I will have to start again > from > > the beginning if I need to reproduce it. Nevertheless here you go. Check > > the attachment (OUTPUTNAND.txt.bz2). > > > > I think I reproduced your problem. Possibly I'll have time to work on > this next week, but no promises. > > -- > Mateusz Guzik > From owner-freebsd-fs@FreeBSD.ORG Thu Nov 29 13:30:55 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6B89D7C2 for ; Thu, 29 Nov 2012 13:30:55 +0000 (UTC) (envelope-from vhaisman@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id DB3958FC15 for ; Thu, 29 Nov 2012 13:30:54 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id go10so11655114lbb.13 for ; Thu, 29 Nov 2012 05:30:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=VMvxUAkxnj1r2MKaH+ke5kRuhLlk8jEkvLuNASuPETg=; b=O82jrKldmUsjsJ46a+xgMlT9RBPpbtCL5ekPdHR7hRfzzmRDsauZgdTsfUo8Ihi0ai 1gAh7b1Ji5NhARClLNH5ek2MP4P3tz/Z9Efn3ZUQKbexFIdR25xgshhIBg2ifQOJ+MUe rc/4rdCTMdq88drF9l/k8u44DQEkScQSVccoBTZlm0RedIERuRrAWx2pmDujEAeI69kO jkSlFZVoC9Ta0WDnIcKYh+SOqTkybMWJp6dcxrTmeXATwj+r02NXODEcHWWGJBIYqVKl VyNZ35y9p+PmPMCL91/gnMVkLlQ/DcGErVLm1K03qUzY9uv+SK3OhwpAVkgQpFKiU0mk BLgQ== MIME-Version: 1.0 Received: by 10.112.10.98 with SMTP id h2mr6116070lbb.127.1354195853576; Thu, 29 Nov 2012 05:30:53 -0800 (PST) Received: by 10.152.120.103 with HTTP; Thu, 29 Nov 2012 05:30:53 -0800 (PST) In-Reply-To: References: <50B4A040.6060001@FreeBSD.org> Date: Thu, 29 Nov 2012 14:30:53 +0100 Message-ID: Subject: Re: Re-sizable UFS project From: =?UTF-8?Q?V=C3=A1clav_Zeman?= To: Johannes Totz Content-Type: text/plain; charset=UTF-8 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2012 13:30:55 -0000 On 28 November 2012 14:22, Johannes Totz wrote: > On 27/11/2012 14:02, CeDeROM wrote: >> Btw. are there any projects to make UFS natively available (something >> like fs-driver for ExtFS) on platforms such as Windows, Linux, MacOS? >> It would be nice to have native UFS instead Ext2 as universal >> filesystem among these operating systems... :-) >> > > There used to be a UFS driver for Windows too. I used it for a while, > but was a bit unstable. > Otherwise, the smallest common denominator is FAT? If FUSE NTFS works as well as on Linux, then NTFS could be it. -- VZ From owner-freebsd-fs@FreeBSD.ORG Thu Nov 29 23:29:50 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 890AF71F; Thu, 29 Nov 2012 23:29:50 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id EA0CF8FC08; Thu, 29 Nov 2012 23:29:49 +0000 (UTC) Received: from tom.home (localhost [127.0.0.1]) by kib.kiev.ua (8.14.5/8.14.5) with ESMTP id qATNTiX6024876; Fri, 30 Nov 2012 01:29:44 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.1 kib.kiev.ua qATNTiX6024876 Received: (from kostik@localhost) by tom.home (8.14.5/8.14.5/Submit) id qATNTior024875; Fri, 30 Nov 2012 01:29:44 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 30 Nov 2012 01:29:44 +0200 From: Konstantin Belousov To: sig6247 Subject: Re: clang compiled kernel panic when mounting zfs root on i386 Message-ID: <20121129232944.GQ3013@kib.kiev.ua> References: <50b37d46.8584440a.735c.ffffb4e6@mx.google.com> <20121126171658.GD3013@kib.kiev.ua> <20121127071243.D1255@besplex.bde.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="KI6XeYrntNhU1GwB" Content-Disposition: inline In-Reply-To: <20121127071243.D1255@besplex.bde.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-current@freebsd.org, fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2012 23:29:50 -0000 --KI6XeYrntNhU1GwB Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Nov 27, 2012 at 08:21:05AM +1100, Bruce Evans wrote: > On Mon, 26 Nov 2012, Konstantin Belousov wrote: >=20 > > On Mon, Nov 26, 2012 at 06:31:34AM -0800, sig6247 wrote: > >> > >> Just checked out r243529, this only happens when the kernel is compiled > >> by clang, and only on i386, either recompiling the kernel with gcc or > >> booting from a UFS root works fine. Is it a known problem? > > It looks like that clang uses more stack than gcc, and zfs makes quite > > deep call chains. =2E.. > It would be useful if the stack trace printed the the stack pointer > on every function call, so that you could see how much stack each > function used. Please apply the patch below and obtain the backtrace of the double fault panic again. I will commit the patch later. diff --git a/sys/amd64/amd64/db_trace.c b/sys/amd64/amd64/db_trace.c index cba90f2..2c81f87 100644 --- a/sys/amd64/amd64/db_trace.c +++ b/sys/amd64/amd64/db_trace.c @@ -186,7 +186,8 @@ db_ss(struct db_variable *vp, db_expr_t *valuep, int op) =20 static void db_nextframe(struct amd64_frame **, db_addr_t *, struct thread= *); static int db_numargs(struct amd64_frame *); -static void db_print_stack_entry(const char *, int, char **, long *, db_ad= dr_t); +static void db_print_stack_entry(const char *, int, char **, long *, db_ad= dr_t, + void *); static void decode_syscall(int, struct thread *); =20 static const char * watchtype_str(int type); @@ -230,12 +231,13 @@ db_numargs(fp) } =20 static void -db_print_stack_entry(name, narg, argnp, argp, callpc) +db_print_stack_entry(name, narg, argnp, argp, callpc, frame) const char *name; int narg; char **argnp; long *argp; db_addr_t callpc; + void *frame; { db_printf("%s(", name); #if 0 @@ -250,6 +252,8 @@ db_print_stack_entry(name, narg, argnp, argp, callpc) #endif db_printf(") at "); db_printsym(callpc, DB_STGY_PROC); + if (frame !=3D NULL) + db_printf("/frame 0x%lx", (register_t)frame); db_printf("\n"); } =20 @@ -341,7 +345,7 @@ db_nextframe(struct amd64_frame **fp, db_addr_t *ip, st= ruct thread *td) return; } =20 - db_print_stack_entry(name, 0, 0, 0, rip); + db_print_stack_entry(name, 0, 0, 0, rip, &(*fp)->f_frame); =20 /* * Point to base of trapframe which is just above the @@ -437,7 +441,8 @@ db_backtrace(struct thread *td, struct trapframe *tf, * Don't try to walk back on a stack for a * process that hasn't actually been run yet. */ - db_print_stack_entry(name, 0, 0, 0, pc); + db_print_stack_entry(name, 0, 0, 0, pc, + actframe); break; } first =3D FALSE; @@ -451,7 +456,7 @@ db_backtrace(struct thread *td, struct trapframe *tf, narg =3D db_numargs(frame); } =20 - db_print_stack_entry(name, narg, argnp, argp, pc); + db_print_stack_entry(name, narg, argnp, argp, pc, actframe); =20 if (actframe !=3D frame) { /* `frame' belongs to caller. */ @@ -465,7 +470,7 @@ db_backtrace(struct thread *td, struct trapframe *tf, if (INKERNEL((long)pc) && !INKERNEL((long)frame)) { sym =3D db_search_symbol(pc, DB_STGY_ANY, &offset); db_symbol_values(sym, &name, NULL); - db_print_stack_entry(name, 0, 0, 0, pc); + db_print_stack_entry(name, 0, 0, 0, pc, frame); break; } if (!INKERNEL((long) frame)) { diff --git a/sys/i386/i386/db_trace.c b/sys/i386/i386/db_trace.c index 445d9c5..822cc56 100644 --- a/sys/i386/i386/db_trace.c +++ b/sys/i386/i386/db_trace.c @@ -176,7 +176,8 @@ db_ss(struct db_variable *vp, db_expr_t *valuep, int op) =20 static void db_nextframe(struct i386_frame **, db_addr_t *, struct thread = *); static int db_numargs(struct i386_frame *); -static void db_print_stack_entry(const char *, int, char **, int *, db_add= r_t); +static void db_print_stack_entry(const char *, int, char **, int *, db_add= r_t, + void *); static void decode_syscall(int, struct thread *); =20 static const char * watchtype_str(int type); @@ -220,12 +221,13 @@ retry: } =20 static void -db_print_stack_entry(name, narg, argnp, argp, callpc) +db_print_stack_entry(name, narg, argnp, argp, callpc, frame) const char *name; int narg; char **argnp; int *argp; db_addr_t callpc; + void *frame; { int n =3D narg >=3D 0 ? narg : 5; =20 @@ -242,6 +244,8 @@ db_print_stack_entry(name, narg, argnp, argp, callpc) db_printf(",..."); db_printf(") at "); db_printsym(callpc, DB_STGY_PROC); + if (frame !=3D NULL) + db_printf("/frame 0x%r", (register_t)frame); db_printf("\n"); } =20 @@ -326,7 +330,7 @@ db_nextframe(struct i386_frame **fp, db_addr_t *ip, str= uct thread *td) return; } =20 - db_print_stack_entry(name, 0, 0, 0, eip); + db_print_stack_entry(name, 0, 0, 0, eip, &(*fp)->f_frame); =20 /* * For a double fault, we have to snag the values from the @@ -467,7 +471,8 @@ db_backtrace(struct thread *td, struct trapframe *tf, s= truct i386_frame *frame, * Don't try to walk back on a stack for a * process that hasn't actually been run yet. */ - db_print_stack_entry(name, 0, 0, 0, pc); + db_print_stack_entry(name, 0, 0, 0, pc, + actframe); break; } first =3D FALSE; @@ -481,7 +486,7 @@ db_backtrace(struct thread *td, struct trapframe *tf, s= truct i386_frame *frame, narg =3D db_numargs(frame); } =20 - db_print_stack_entry(name, narg, argnp, argp, pc); + db_print_stack_entry(name, narg, argnp, argp, pc, actframe); =20 if (actframe !=3D frame) { /* `frame' belongs to caller. */ @@ -495,7 +500,7 @@ db_backtrace(struct thread *td, struct trapframe *tf, s= truct i386_frame *frame, if (INKERNEL((int)pc) && !INKERNEL((int) frame)) { sym =3D db_search_symbol(pc, DB_STGY_ANY, &offset); db_symbol_values(sym, &name, NULL); - db_print_stack_entry(name, 0, 0, 0, pc); + db_print_stack_entry(name, 0, 0, 0, pc, frame); break; } if (!INKERNEL((int) frame)) { --KI6XeYrntNhU1GwB Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQt+/nAAoJEJDCuSvBvK1BXUIP/jOiC4stZJ9EEWRosY4ELMDI WJnPA/mukK/tGNY/WXYm3Ro+6tkfe8rbrUoolkIuo3D/dmtunA8JYXA0bkGW3Rbq GsJNfi6Rie4e7I+VkUI5cRzEZ0Atcz5Mw+Vy0ix6UQzGC9TvWvCQI0khfWBVbeyX v1hLx/McVn9iRBCftFtQj0JfGvQmVxLVCMdMQJ59Ds7IwHyPtWbPLtq5f0AEjSfz fCxksMI8sKfvzBHjZA5Cxux5k2Hf97gppXUTAsnHOau2M8oQFNyNNWf7SPnF6iKS Qw5JAhpNePHy6x1VjdiecUjziC4jBeMONWPGRVVwSdYyDNTuzd8kg8bkzPhYaywJ T7+1hjdzvV5fCTVIj53GFnMP60g0cnfyQoEt9nDXW7d6AZofeZWPVtusJUD4bmTz 42btYYzCzMWG/UViGzc3eoYX9kqmi8mut+/0UbaZErLtSlzTb86SXZfmnC/n2OE7 GrnyFXpia1To1YluGWsDV6BrNzhgWZhGmy9C+GyV60alO0MVF18mMRn5OSP1aBIe y2WJhMDTIdL+gnbWnKK0J+AV17BptApVzTfU/3KM2avH7Z83LVngLQDyfSyMtTN0 m3qS7bGMKQXf5oUh+jcGPjGimW4SJYjL75OuyvkWxmXn9IC+RWFhzDR3PyzMf+zc kRpA9qMuQM81N4BSbYjI =Jtls -----END PGP SIGNATURE----- --KI6XeYrntNhU1GwB-- From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 12:19:37 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3E6D3C14 for ; Fri, 30 Nov 2012 12:19:37 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) by mx1.freebsd.org (Postfix) with ESMTP id D6CF88FC14 for ; Fri, 30 Nov 2012 12:19:36 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1TePZJ-0003sK-AS for freebsd-fs@freebsd.org; Fri, 30 Nov 2012 13:19:45 +0100 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 30 Nov 2012 13:19:45 +0100 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 30 Nov 2012 13:19:45 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Subject: Re: Performance Difference on UFS and ZFS Date: Fri, 30 Nov 2012 13:19:24 +0100 Lines: 52 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig876DB8D731C7B33EFE4A5AAF" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:14.0) Gecko/20120812 Thunderbird/14.0 In-Reply-To: X-Enigmail-Version: 1.4.3 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2012 12:19:37 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig876DB8D731C7B33EFE4A5AAF Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 28/11/2012 16:54, Metin D=C3=B6=C5=9Fl=C3=BC wrote: > Hi Sergey, >=20 > I tested it on a cc1.4xlarge EC2 instance, here is the specs: >=20 > 23 GiB of memory > 33.5 EC2 Compute Units (2 x Intel Xeon X5570, quad-core =E2=80=9CNehale= m=E2=80=9D architecture) > 1690 GB of instance storage > 64-bit platform >=20 > I installed PostgreSQL from its port on FreeBSD. I didn't do any > tuning for PostgreSQL or FreeBSD. Data access pattern consists of > completely from sequential reads such "select count(*) from > table_name". I measured performance with PostgreSQL's timing option. > As as side note; all queries are served from memory, so there were no > disk usage for these tests. As others said - this is interesting and unexpected. Are you sure everything is the same across benchmarks? Since you are running on a virtualized platform, it may be that other users of the same storage pool "steal" your IO performance. I did a benchmark with PostgreSQL and ZFS vs UFS a couple of years ago, and the conclusion was that, once tuned, the performance is very similar, with ZFS being slightly better. Since you are testing read-only sequential IO, can you run an alternative test with some other benchmark such as bonnie++? --------------enig876DB8D731C7B33EFE4A5AAF Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlC4pE0ACgkQ/QjVBj3/HSx+qgCgpZww46FBjLWYL73V9e4sWqLT 72kAoJbjsHMDfoIoPVGD33huToD6HEjn =sgwp -----END PGP SIGNATURE----- --------------enig876DB8D731C7B33EFE4A5AAF-- From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 12:48:32 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B967D3DB; Fri, 30 Nov 2012 12:48:32 +0000 (UTC) (envelope-from sig6247@gmail.com) Received: from mail-ye0-f182.google.com (mail-ye0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 550898FC14; Fri, 30 Nov 2012 12:48:32 +0000 (UTC) Received: by mail-ye0-f182.google.com with SMTP id q5so43083yen.13 for ; Fri, 30 Nov 2012 04:48:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:to:cc:subject:in-reply-to:references :mime-version:content-type; bh=qXp7rUZlhnsNhLRthaKsOZwbbwSEzH4phdoKUQXBtL0=; b=aAmIQ79HBueetspzT78j1DGGBwoalgk+ttC26wz6BmPawRfsirXLOT3TgxtvI9ZcPM 5ffGlwvuMuz48Jjp/6uJQMDdiUal6xDVPSrKQ7u6eRN2T7One3sZjcnyAb4eaRYv/C5A ewyo5W20L7VFYWAjiQZaRHNNisoy6d5k6XIgbfGTYqJNqN0gv8Y8syjFtJPP3/xJqkk3 99salmwbf7jPq/TLnGiweTwMPEgNC8OJS9TjcHp92UDKOn9OIyfz3uUt3HTZdX3C7DbP oHer/On4ipIG2CJbZAkRkq76uIDQAEs8CuCtOmKqJixO+5brY7pO97cvSEoNueXMcT/R NxjQ== Received: by 10.236.124.131 with SMTP id x3mr840252yhh.14.1354279366065; Fri, 30 Nov 2012 04:42:46 -0800 (PST) Received: from localhost ([165.254.32.192]) by mx.google.com with ESMTPS id d66sm4703080yhe.1.2012.11.30.04.42.42 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 30 Nov 2012 04:42:45 -0800 (PST) Message-ID: <50b8a9c5.e64dec0a.1d88.133a@mx.google.com> Date: Fri, 30 Nov 2012 04:42:45 -0800 (PST) From: sig6247 To: Konstantin Belousov Subject: Re: clang compiled kernel panic when mounting zfs root on i386 In-Reply-To: <20121129232944.GQ3013@kib.kiev.ua> (Konstantin Belousov's message of "Fri, 30 Nov 2012 01:29:44 +0200") References: <50b37d46.8584440a.735c.ffffb4e6@mx.google.com> <20121126171658.GD3013@kib.kiev.ua> <20121127071243.D1255@besplex.bde.org> <20121129232944.GQ3013@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-current@freebsd.org, fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2012 12:48:32 -0000 On Fri, 30 Nov 2012 01:29:44 +0200, Konstantin Belousov wrote: > Please apply the patch below and obtain the backtrace of the double fault > panic again. I will commit the patch later. Thanks for the patch. WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from zfs:zroot []... Fatal double fault: eip = 0xc0e93e07 esp = 0xc86bffbc ebp = 0xc86c0032 cpuid = 1; apic id = 01 panic: double fault cpuid = 1 KDB: enter: panic [ thread pid 1 tid 100002 ] Stopped at kdb_enter+0x3d: movl $0,kdb_why db> bt Tracing pid 1 tid 100002 td 0xc89efbc0 kdb_enter(c1065960,c1065960,c10b903b,c139f438,aef01785,...) at kdb_enter+0x3d/frame 0xc139f3f0 panic(c10b903b,1,1,1,c86c0032,...) at panic+0x14b/frame 0xc139f42c dblfault_handler() at dblfault_handler+0xab/frame 0xc139f42c --- trap 0x17, eip = 0xc0e93e07, esp = 0xc86bffbc, ebp = 0xc86c0032 --- __qdivrem(0,aba80000,c10d,98600000,68c119,...) at __qdivrem+0x197/frame 0xc86c0032 db> From owner-freebsd-fs@FreeBSD.ORG Fri Nov 30 16:47:24 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1274A4E0; Fri, 30 Nov 2012 16:47:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 9DEBB8FC08; Fri, 30 Nov 2012 16:47:23 +0000 (UTC) Received: from tom.home (localhost [127.0.0.1]) by kib.kiev.ua (8.14.5/8.14.5) with ESMTP id qAUGlGo8028201; Fri, 30 Nov 2012 18:47:16 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.1 kib.kiev.ua qAUGlGo8028201 Received: (from kostik@localhost) by tom.home (8.14.5/8.14.5/Submit) id qAUGlFdd028200; Fri, 30 Nov 2012 18:47:15 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 30 Nov 2012 18:47:15 +0200 From: Konstantin Belousov To: sig6247 Subject: Re: clang compiled kernel panic when mounting zfs root on i386 Message-ID: <20121130164715.GW3013@kib.kiev.ua> References: <50b37d46.8584440a.735c.ffffb4e6@mx.google.com> <20121126171658.GD3013@kib.kiev.ua> <20121127071243.D1255@besplex.bde.org> <20121129232944.GQ3013@kib.kiev.ua> <50b8a9c5.e64dec0a.1d88.133a@mx.google.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="iwX7oKFvAj2SwWc7" Content-Disposition: inline In-Reply-To: <50b8a9c5.e64dec0a.1d88.133a@mx.google.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-current@freebsd.org, fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2012 16:47:24 -0000 --iwX7oKFvAj2SwWc7 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Nov 30, 2012 at 04:42:45AM -0800, sig6247 wrote: > On Fri, 30 Nov 2012 01:29:44 +0200, Konstantin Belousov wrote: >=20 > > Please apply the patch below and obtain the backtrace of the double fau= lt > > panic again. I will commit the patch later. >=20 > Thanks for the patch.=20 >=20 > WARNING: WITNESS option enabled, expect reduced performance. > Trying to mount root from zfs:zroot []... >=20 > Fatal double fault: > eip =3D 0xc0e93e07 > esp =3D 0xc86bffbc > ebp =3D 0xc86c0032 > cpuid =3D 1; apic id =3D 01 > panic: double fault > cpuid =3D 1 > KDB: enter: panic > [ thread pid 1 tid 100002 ] > Stopped at kdb_enter+0x3d: movl $0,kdb_why > db> bt > Tracing pid 1 tid 100002 td 0xc89efbc0 > kdb_enter(c1065960,c1065960,c10b903b,c139f438,aef01785,...) at kdb_enter+= 0x3d/frame 0xc139f3f0 > panic(c10b903b,1,1,1,c86c0032,...) at panic+0x14b/frame 0xc139f42c > dblfault_handler() at dblfault_handler+0xab/frame 0xc139f42c > --- trap 0x17, eip =3D 0xc0e93e07, esp =3D 0xc86bffbc, ebp =3D 0xc86c0032= --- > __qdivrem(0,aba80000,c10d,98600000,68c119,...) at __qdivrem+0x197/frame 0= xc86c0032 > db>=20 Hm, this is not very useful. Although the panic is again caused by the stack overflow, most likely (please also include the output of the "show thread" =66rom ddb), it is at different place, and probably at the leaf function. Can you try some more times, so that we could see 'big' backtrace ? --iwX7oKFvAj2SwWc7 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQuOMTAAoJEJDCuSvBvK1Bfb4QAKfQlzWqbvLNxc0iMg7NQlJ8 ZlJanY91GWIuZbgtSYSwkLZ0mJ0xhHgCRGJLFnVncZGY56bPmIDnkV0h776RY7RU 2fH1BI2QLCEVqRH/y4n3uoJZRfdZ2xrqRuvw3qkWQ5xUbMgTFZamckv1Z0rTsp9w NKwzFcWBGvZeioJJALMDbKr4FtmPeFK1K61aQlAL2WzwbTn7xd3Iqs9CrMm8oUEi FMl3mHajW61g2dYY1LwxEpR4/GnMmJARFqMRugN68W0PyfXvYfX7QFJcL/TuZZHS 3pl3fvVIR00sCmEwT1xYyI4Q9oF/5Hs5ybFFFBkLX0CABXyVoAZbUo/Z+JldM6SQ VgjMF+qucbcacBCYerSXjTQv0Hv+3P4ALMepJBuwiE/DCwnWe4UO3LhpUx1rVR5X 8siL9sE4u0qkyxirti7TTlOQS30MEneyYtybNQaiY3cn5YCrCy5otovc047hd65w JIX6I5R0ZS84KoXQe/gfyOcWz87zfFInVfRWevz62x7rfQqF3srlIenpfE1d0ZP1 qRYKGZcRiR02RACoLerlTKukj6ZzXK0XXTt84GZaqzUZvb3tzYS6e22gAEKRd7T6 xcvNILDcPaUHRRLr8+OPGPJNj0ZpKkeGT9EtVWSXBz+Sy+v7jB+nF8Yw4Y85DvnO 4UPihvsOQq9VptkFTkn0 =rLF0 -----END PGP SIGNATURE----- --iwX7oKFvAj2SwWc7-- From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 09:34:12 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0351C43C; Sat, 1 Dec 2012 09:34:12 +0000 (UTC) (envelope-from sig6247@gmail.com) Received: from mail-ye0-f182.google.com (mail-ye0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9552E8FC08; Sat, 1 Dec 2012 09:34:11 +0000 (UTC) Received: by mail-ye0-f182.google.com with SMTP id q5so212125yen.13 for ; Sat, 01 Dec 2012 01:34:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:to:cc:subject:in-reply-to:references :mime-version:content-type; bh=B3xNG/8mTroDisWL43ugcBgjH5T1nGBrCW2MtPRo/nk=; b=hhGPMrlQ+NUsEN4o9YbpInEhaGhokk/GJ2x97NDDdY/PCZgolXXW6adNGSwzRZVitJ Aw8yOYUBzhiCgWtsPVPQnKAwME3aig6UFvUO0LvnGNOnjT/2PGxNymsParRdkBy6W+hu s3JDwqnzWknr84XtQaik0kPBO3JOEQaUhvBP/bzGrGkC4bpxOZRnoUTQ7uZCx0AkFkoJ GiPFu8Rbrfe8BLrJgCGSpvdHKjd+tsLQ7Ofix86AgCoH75FW/sYhkRFWeRdNIWMoXHbf ZlL3TB22nGqDYuXDdulORuPCtk4eHbKF80dzkr2wExg+AJZDEk+vyzMbaZggwu4N15vu FvBg== Received: by 10.101.128.5 with SMTP id f5mr1229916ann.6.1354354445120; Sat, 01 Dec 2012 01:34:05 -0800 (PST) Received: from localhost (bolobolo1.torservers.net. [96.47.226.20]) by mx.google.com with ESMTPS id u15sm6694664anq.14.2012.12.01.01.33.56 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 01 Dec 2012 01:34:04 -0800 (PST) Message-ID: <50b9cf0c.0fd9650a.5bbf.ffffb9b3@mx.google.com> Date: Sat, 01 Dec 2012 01:34:04 -0800 (PST) From: sig6247 To: Konstantin Belousov Subject: Re: clang compiled kernel panic when mounting zfs root on i386 In-Reply-To: <20121130164715.GW3013@kib.kiev.ua> (Konstantin Belousov's message of "Fri, 30 Nov 2012 18:47:15 +0200") References: <50b37d46.8584440a.735c.ffffb4e6@mx.google.com> <20121126171658.GD3013@kib.kiev.ua> <20121127071243.D1255@besplex.bde.org> <20121129232944.GQ3013@kib.kiev.ua> <50b8a9c5.e64dec0a.1d88.133a@mx.google.com> <20121130164715.GW3013@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-current@freebsd.org, fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2012 09:34:12 -0000 On Fri, 30 Nov 2012 18:47:15 +0200, Konstantin Belousov wrote: > Hm, this is not very useful. Although the panic is again caused by the stack > overflow, most likely (please also include the output of the "show thread" > from ddb), it is at different place, and probably at the leaf function. > > Can you try some more times, so that we could see 'big' backtrace ? Sure. Thanks. WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from zfs:zroot []... Fatal double fault: eip = 0xc0add15d esp = 0xc86bffc8 ebp = 0xc86c003c cpuid = 1; apic id = 01 panic: double fault cpuid = 1 KDB: enter: panic [ thread pid 1 tid 100002 ] Stopped at kdb_enter+0x3d: movl $0,kdb_why db> bt Tracing pid 1 tid 100002 td 0xc89efbc0 kdb_enter(c1065960,c1065960,c10b903b,c139f438,2243cdbd,...) at kdb_enter+0x3d/frame 0xc139f3f0 panic(c10b903b,1,1,1,c86c003c,...) at panic+0x14b/frame 0xc139f42c dblfault_handler() at dblfault_handler+0xab/frame 0xc139f42c --- trap 0x17, eip = 0xc0add15d, esp = 0xc86bffc8, ebp = 0xc86c003c --- witness_checkorder(c1fd7508,9,c109ee8c,7fa,0,...) at witness_checkorder+0x37d/frame 0xc86c003c __mtx_lock_flags(c1fd7518,0,c109ee8c,7fa,c135e998,...) at __mtx_lock_flags+0x87/frame 0xc86c007 0 uma_zalloc_arg(c1fd66c0,0,1,4d3,c86c0110,...) at uma_zalloc_arg+0x605/frame 0xc86c00c8 vm_map_insert(c1fd508c,c13e0ca0,bd3a000,0,cbc39000,...) at vm_map_insert+0x499/frame 0xc86c0130 kmem_back(c1fd508c,cbc39000,1000,3,c86c01d4,...) at kmem_back+0x76/frame 0xc86c018c kmem_malloc(c1fd508c,1000,3) at kmem_malloc+0x250/frame 0xc86c01c0 page_alloc(c1fd1d80,1000,c86c020b,3,c1fd1d80,...) at page_alloc+0x27/frame 0xc86c01d4 keg_alloc_slab(103,4,c109ee8c,870,cbb95f6c,...) at keg_alloc_slab+0xc3/frame 0xc86c0218 keg_fetch_slab(103,c1fd1d80,cbb95f6c,c1fc8230,c86c02c0,...) at keg_fetch_slab+0xe2/frame 0xc86c 0250 zone_fetch_slab(c1fd1d80,c1fd0480,103,826,0,...) at zone_fetch_slab+0x43/frame 0xc86c0268 uma_zalloc_arg(c1fd1d80,0,102,3,2,...) at uma_zalloc_arg+0x3f2/frame 0xc86c02c0 malloc(4c,c1826100,102,c86c0388,c173909a,...) at malloc+0xe9/frame 0xc86c02e8 zfs_kmem_alloc(4c,102,cb7d8820,c89efbc0,cb7d8820,...) at zfs_kmem_alloc+0x20/frame 0xc86c02fc vdev_mirror_io_start(cba232e0,10,cba232e0,1,0,...) at vdev_mirror_io_start+0x14a/frame 0xc86c03 88 zio_vdev_io_start(cba232e0,c89efbc0,0,cba232e0,c86c0600,...) at zio_vdev_io_start+0x228/frame 0 xc86c03e4 zio_execute(cba232e0,cb7d8000,cbbec640,cbbe2000,600,...) at zio_execute+0x106/frame 0xc86c0418 spa_load_verify_cb(cb7d8000,0,cbbec640,cba6bd20,c86c0600,...) at spa_load_verify_cb+0x89/frame 0xc86c0458 traverse_visitbp(cba6bd20,cbbec640,c86c0600,c86c0ba0,0,...) at traverse_visitbp+0x29f/frame 0xc 86c05e0 traverse_dnode(cba6bd20,0,0,23,0,...) at traverse_dnode+0x92/frame 0xc86c0638 traverse_visitbp(cba6bd98,cbbf0080,c86c0890,cba6bdd4,c16ca7e0,...) at traverse_visitbp+0xe47/fr ame 0xc86c07c0 traverse_visitbp(cba6bdd4,cbbe2840,c86c0968,c86c0ba0,0,...) at traverse_visitbp+0xf32/frame 0xc 86c0948 traverse_dnode(cba6bdd4,0,0,0,0,...) at traverse_dnode+0x92/frame 0xc86c09a0 traverse_visitbp(0,cb7d8398,c86c0b50,2,cbbdc214,...) at traverse_visitbp+0x96d/frame 0xc86c0b28 traverse_impl(0,0,cb7d8398,74,0,...) at traverse_impl+0x268/frame 0xc86c0be0 traverse_pool(cb7d8000,74,0,d,c1723830,...) at traverse_pool+0x79/frame 0xc86c0c88 spa_load(0,1,c86c0ec4,1e,0,...) at spa_load+0x1dde/frame 0xc86c0df0 spa_load(0,0,c13d9d14,1,3,...) at spa_load+0x11a5/frame 0xc86c0f58 spa_load_best(0,ffffffff,ffffffff,1,c0add175,...) at spa_load_best+0x71/frame 0xc86c0fb0 spa_open_common(c17dce4e,0,0,c86c1190,c16f1a1c,...) at spa_open_common+0x11a/frame 0xc86c100c spa_open(c86c1078,c86c1074,c17dce4e,c135e998,c1fd7798,...) at spa_open+0x27/frame 0xc86c1020 dsl_dir_open_spa(0,c89770b0,c17dd1e1,c86c11f8,c86c11f4,...) at dsl_dir_open_spa+0x6c/frame 0xc8 6c1190 dsl_dataset_hold(c89770b0,cb7d3800,c86c1240,cb7d3800,cb7d3800,...) at dsl_dataset_hold+0x3a/fra me 0xc86c120c dsl_dataset_own(c89770b0,0,cb7d3800,c86c1240,c1824e30,...) at dsl_dataset_own+0x21/frame 0xc86c 1228 dmu_objset_own(c89770b0,2,1,cb7d3800,c86c1290,...) at dmu_objset_own+0x2a/frame 0xc86c1250 zfsvfs_create(c89770b0,c86c13ac,c17ea09b,681,0,...) at zfsvfs_create+0x4c/frame 0xc86c12a8 zfs_mount(cb99b540,c17f0160,cb98b100,c89cae80,0,...) at zfs_mount+0x42c/frame 0xc86c14e0 vfs_donmount(c89efbc0,4000,0,c86c1790,cb98b180,...) at vfs_donmount+0xc6d/frame 0xc86c1778 kernel_mount(c8977490,4000,0,0,1,...) at kernel_mount+0x6b/frame 0xc86c17b8 parse_mount(cb96e0e0,c1195498,0,1,0,...) at parse_mount+0x606/frame 0xc86c19d8 vfs_mountroot(c13da634,4,c105ceba,2bb,0,...) at vfs_mountroot+0x6cf/frame 0xc86c1c60 start_init(0,c86c1d08,c105f7c4,3db,0,...) at start_init+0x6a/frame 0xc86c1ccc fork_exit(c0a429e0,0,c86c1d08) at fork_exit+0x7f/frame 0xc86c1cf4 fork_trampoline() at fork_trampoline+0x8/frame 0xc86c1cf4 --- trap 0, eip = 0, esp = 0xc86c1d40, ebp = 0 --- db> show thread Thread 100002 at 0xc89efbc0: proc (pid 1): 0xc89edb40 name: kernel stack: 0xc86c0000-0xc86c1fff flags: 0x4 pflags: 0x10000 state: RUNNING (CPU 1) priority: 84 container lock: sched lock 1 (0xc1220000) db> From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 18:34:40 2012 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A76436E9; Sat, 1 Dec 2012 18:34:40 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 77E948FC08; Sat, 1 Dec 2012 18:34:40 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id qB1IYeu3086434; Sat, 1 Dec 2012 18:34:40 GMT (envelope-from avg@freefall.freebsd.org) Received: (from avg@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id qB1IYeGa086430; Sat, 1 Dec 2012 18:34:40 GMT (envelope-from avg) Date: Sat, 1 Dec 2012 18:34:40 GMT Message-Id: <201212011834.qB1IYeGa086430@freefall.freebsd.org> To: josh@hewbert.com, avg@FreeBSD.org, freebsd-fs@FreeBSD.org From: avg@FreeBSD.org Subject: Re: kern/170238: [zfs] [panic] Panic when deleting data X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2012 18:34:40 -0000 Synopsis: [zfs] [panic] Panic when deleting data State-Changed-From-To: open->closed State-Changed-By: avg State-Changed-When: Sat Dec 1 18:34:32 UTC 2012 State-Changed-Why: http://www.freebsd.org/cgi/query-pr.cgi?pr=170238 From owner-freebsd-fs@FreeBSD.ORG Sat Dec 1 18:35:45 2012 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E88C76F6; Sat, 1 Dec 2012 18:35:45 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id AAFAB8FC14; Sat, 1 Dec 2012 18:35:45 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id qB1IZjr2086517; Sat, 1 Dec 2012 18:35:45 GMT (envelope-from avg@freefall.freebsd.org) Received: (from avg@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id qB1IZjPo086513; Sat, 1 Dec 2012 18:35:45 GMT (envelope-from avg) Date: Sat, 1 Dec 2012 18:35:45 GMT Message-Id: <201212011835.qB1IZjPo086513@freefall.freebsd.org> To: avg@FreeBSD.org, freebsd-fs@FreeBSD.org, avg@FreeBSD.org From: avg@FreeBSD.org Subject: Re: kern/170238: [zfs] [panic] Panic when deleting data X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Dec 2012 18:35:46 -0000 Synopsis: [zfs] [panic] Panic when deleting data Responsible-Changed-From-To: freebsd-fs->avg Responsible-Changed-By: avg Responsible-Changed-When: Sat Dec 1 18:35:33 UTC 2012 Responsible-Changed-Why: I've looked at this one. http://www.freebsd.org/cgi/query-pr.cgi?pr=170238