From owner-freebsd-fs@FreeBSD.ORG Sun Dec 23 00:10:39 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 356F16BF for ; Sun, 23 Dec 2012 00:10:39 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id E4CAD8FC0A for ; Sun, 23 Dec 2012 00:10:38 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap8EAFhL1lCDaFvO/2dsb2JhbABDFoYkt2hzgkhWNQINGQJfiCYMoyiRdYEijmWBEwOIYo0qgRyPLIMSggQ X-IronPort-AV: E=Sophos;i="4.84,338,1355115600"; d="scan'208";a="6097253" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 22 Dec 2012 19:10:32 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 15DF1B409B; Sat, 22 Dec 2012 19:10:32 -0500 (EST) Date: Sat, 22 Dec 2012 19:10:32 -0500 (EST) From: Rick Macklem To: FreeBSD Filesystems Message-ID: <1071529580.1558626.1356221432033.JavaMail.root@erie.cs.uoguelph.ca> Subject: NFS krb5 host based initiator credential patch MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Linux)/6.0.10_GA_2692) Cc: admin@ist.tugraz.at X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Dec 2012 00:10:39 -0000 Hi, For a long time, I've had a patch that adds support for host based credentials in a keytab file to the kerberized NFS client. Unfortunately, it only worked if the kind of encryption used to create the keytab entry was explicitly set via a sysctl. Because of this dfr@ understandably didn't want it commited. Also, the patch had a bug which caused crashes when the initial use of the credential failed for any reason. I now finally have a patch that doesn't require explicit setting of the encryption type to make it work. (It does essentially a "kinit -k" to acquire a TGT and put it in a credential cache, which is then used by gss_init_sec_context().) I'd appreciate testing and review of this patch. It can be found at: http://people.freebsd.org/~rmacklem/rpcsec_gss-hostbased-initiator.patch this patch should apply to the files in -current. If the patch doesn't apply cleanly, you can find patched copies of the files here. (These should be buildable in any 9.0 or later system, I think?) http://people.freebsd.org/~rmacklem/rpcsec_gss-hostbased-initiator-patched-files The patch has worked ok for me for some testing, but I have only used a des-cbc-crc encrypted keytab entry. (I believe other encryption types should work, so long as they result in an 8 byte session key, but I haven't tested this and suggest testers start with des-cbc-crc.) rick ps: RPCSEC_GSS version 1 uses des-cbc encryption for krb5p, so stronger encryption for the keytab entry probably doesn't make any difference. From owner-freebsd-fs@FreeBSD.ORG Sun Dec 23 10:32:47 2012 Return-Path: Delivered-To: fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 047C78C2 for ; Sun, 23 Dec 2012 10:32:47 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 433C58FC19 for ; Sun, 23 Dec 2012 10:32:46 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id MAA23613; Sun, 23 Dec 2012 12:32:33 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1TmirB-000FQW-6C; Sun, 23 Dec 2012 12:32:33 +0200 Message-ID: <50D6DDC0.9030608@FreeBSD.org> Date: Sun, 23 Dec 2012 12:32:32 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Henri Hennebert Subject: Re: clang compiled kernel panic when mounting zfs root on i386 References: <50b37d46.8584440a.735c.ffffb4e6@mx.google.com> <20121126171658.GD3013@kib.kiev.ua> <20121127071243.D1255@besplex.bde.org> <20121129232944.GQ3013@kib.kiev.ua> <50b8a9c5.e64dec0a.1d88.133a@mx.google.com> <20121130164715.GW3013@kib.kiev.ua> <50b9cf0c.0fd9650a.5bbf.ffffb9b3@mx.google.com> <20121203224132.GJ3013@kib.kiev.ua> <50C880D7.1040907@gmail.com> <50C8DC8E.1080204@FreeBSD.org> <50C9AD2C.7000301__8690.12248372219$1355394383$gmane$org@FreeBSD.org> <50CF16B0.9090104@gmail.com> <50CF9AA9.1030808@FreeBSD.org> <50D2E7C6.1030201@restart.be> In-Reply-To: <50D2E7C6.1030201@restart.be> X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Dec 2012 10:32:47 -0000 on 20/12/2012 12:26 Henri Hennebert said the following: > 'options KSTACK_PAGES=4' allow my 9.1-STABLE to mount the root fs from zfs. > > I think it whould be useful to add this trick to /usr/src/UPDATING for > other i386 users. This sounds like a good idea, thank you. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Sun Dec 23 22:49:09 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 61980372; Sun, 23 Dec 2012 22:49:09 +0000 (UTC) (envelope-from takeda@takeda.tk) Received: from chinatsu.takeda.tk (mail.takeda.tk [74.0.89.210]) by mx1.freebsd.org (Postfix) with ESMTP id 1ED238FC13; Sun, 23 Dec 2012 22:49:08 +0000 (UTC) Received: from localhost.takeda.tk (takeda-ws2.lan [10.0.0.123]) (authenticated bits=0) by chinatsu.takeda.tk (8.14.5/8.14.5) with ESMTP id qBNMXa63010389 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO); Sun, 23 Dec 2012 14:33:37 -0800 (PST) (envelope-from takeda@takeda.tk) Date: Sun, 23 Dec 2012 14:23:08 -0800 From: Derek Kulinski X-Priority: 3 (Normal) Message-ID: <1824023197.20121223142308@takeda.tk> To: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: FreeBSD 9.1-RELEASE crashes almost daily; backtraces always list zfs routines MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.97.6 at chinatsu.takeda.tk X-Virus-Status: Clean X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Dec 2012 22:49:09 -0000 Please help, I reported this issue on http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/174372 but the crashes are unbearable since they happen regularly at night, most of the time when periodic.daily is called (3am) but there are exceptions. It seems like it can be triggered by any heavy disk activity. In many of the crash dumps the current process is find command, but of course that's does not always is the cause. When calling scrub, it appears to pass successfully. Smartmon tools is not reporting any disk errors. I tested memory using memtest86 about a month ago and it passed tests successfully. I never had this type of issue on 9.0, and not much changed in my kernel config besides installing WiFi card. System: FreeBSD chinatsu.takeda.tk 9.1-RELEASE FreeBSD 9.1-RELEASE #2 r244482: Wed Dec 19 23:28:15 PST 2012 root@chinatsu.takeda.tk:/usr/obj/usr/src/sys/CHINATSU amd64 It's compiled from releng/9.1 branch. Thank you for any help, Derek Example crash messages: Today: panic: general protection fault cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 panic() at panic+0x1cd trap_fatal() at trap_fatal+0x285 trap() at trap+0x23e calltrap() at calltrap+0x8 --- trap 0x9, rip = 0xffffffff8101f5ac, rsp = 0xffffff8230ac18c0, rbp = 0xffffff8230ac18d0 --- list_prev() at list_prev+0xc arc_evict() at arc_evict+0x194 arc_adjust() at arc_adjust+0x1a1 arc_reclaim_thread() at arc_reclaim_thread+0x1a6 fork_exit() at fork_exit+0x11c fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8230ac1bb0, rbp = 0 --- Uptime: 20h44m51s Dumping 1157 out of 8072 MB:..2%..12%..21%..31%..41%..52%..61%..71%..81%..92% Yesterday: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0xffffffffffffffe8 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff8102d1e7 stack pointer = 0x28:0xffffff82315ec190 frame pointer = 0x28:0xffffff82315ec280 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 34744 (find) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 panic() at panic+0x1cd trap_fatal() at trap_fatal+0x285 trap_pfault() at trap_pfault+0x216 trap() at trap+0x363 calltrap() at calltrap+0x8 --- trap 0xc, rip = 0xffffffff8102d1e7, rsp = 0xffffff82315ec190, rbp = 0xffffff82315ec280 --- arc_evict() at arc_evict+0x1e7 arc_get_data_buf() at arc_get_data_buf+0x1d5 arc_read_nolock() at arc_read_nolock+0x1ec arc_read() at arc_read+0x93 dbuf_read() at dbuf_read+0x452 dmu_buf_hold() at dmu_buf_hold+0xe0 zap_lockdir() at zap_lockdir+0x58 zap_cursor_retrieve() at zap_cursor_retrieve+0x19b zfs_freebsd_readdir() at zfs_freebsd_readdir+0x2ee VOP_READDIR_APV() at VOP_READDIR_APV+0x4a kern_getdirentries() at kern_getdirentries+0x225 sys_getdirentries() at sys_getdirentries+0x23 amd64_syscall() at amd64_syscall+0x5d8 Xfast_syscall() at Xfast_syscall+0xf7 --- syscall (196, FreeBSD ELF64, sys_getdirentries), rip = 0x80089b29c, rsp = 0x7fffffffd8a8, rbp = 0x1 --- Uptime: 2d3h49m3s Dumping 3701 out of 8072 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% -- Best regards, Derek mailto:takeda@takeda.tk From owner-freebsd-fs@FreeBSD.ORG Mon Dec 24 11:06:42 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7BC55653 for ; Mon, 24 Dec 2012 11:06:42 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 5F45C8FC24 for ; Mon, 24 Dec 2012 11:06:42 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id qBOB6gis066019 for ; Mon, 24 Dec 2012 11:06:42 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id qBOB6gdh066017 for freebsd-fs@FreeBSD.org; Mon, 24 Dec 2012 11:06:42 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 24 Dec 2012 11:06:42 GMT Message-Id: <201212241106.qBOB6gdh066017@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2012 11:06:42 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/174372 fs [zfs] Pagefault appears to be related to ZFS o kern/174315 fs [zfs] chflags uchg not supported o kern/174310 fs [zfs] root point mounting broken on CURRENT with multi o kern/174279 fs [ufs] UFS2-SU+J journal and filesystem corruption o kern/174060 fs [ext2fs] Ext2FS system crashes (buffer overflow?) o kern/173830 fs [zfs] Brain-dead simple change to ZFS error descriptio o kern/173718 fs [zfs] phantom directory in zraid2 pool o kern/173657 fs [nfs] strange UID map with nfsuserd o kern/173363 fs [zfs] [panic] Panic on 'zpool replace' on readonly poo o kern/173136 fs [unionfs] mounting above the NFS read-only share panic o kern/172348 fs [unionfs] umount -f of filesystem in use with readonly o kern/172334 fs [unionfs] unionfs permits recursive union mounts; caus o kern/171626 fs [tmpfs] tmpfs should be noisier when the requested siz o kern/171415 fs [zfs] zfs recv fails with "cannot receive incremental o kern/170945 fs [gpt] disk layout not portable between direct connect o bin/170778 fs [zfs] [panic] FreeBSD panics randomly o kern/170680 fs [nfs] Multiple NFS Client bug in the FreeBSD 7.4-RELEA o kern/170497 fs [xfs][panic] kernel will panic whenever I ls a mounted o kern/169945 fs [zfs] [panic] Kernel panic while importing zpool (afte o kern/169480 fs [zfs] ZFS stalls on heavy I/O o kern/169398 fs [zfs] Can't remove file with permanent error o kern/169339 fs panic while " : > /etc/123" o kern/169319 fs [zfs] zfs resilver can't complete o kern/168947 fs [nfs] [zfs] .zfs/snapshot directory is messed up when o kern/168942 fs [nfs] [hang] nfsd hangs after being restarted (not -HU o kern/168158 fs [zfs] incorrect parsing of sharenfs options in zfs (fs o kern/167979 fs [ufs] DIOCGDINFO ioctl does not work on 8.2 file syste o kern/167977 fs [smbfs] mount_smbfs results are differ when utf-8 or U o kern/167688 fs [fusefs] Incorrect signal handling with direct_io o kern/167685 fs [zfs] ZFS on USB drive prevents shutdown / reboot o kern/167612 fs [portalfs] The portal file system gets stuck inside po o kern/167272 fs [zfs] ZFS Disks reordering causes ZFS to pick the wron o kern/167260 fs [msdosfs] msdosfs disk was mounted the second time whe o kern/167109 fs [zfs] [panic] zfs diff kernel panic Fatal trap 9: gene o kern/167105 fs [nfs] mount_nfs can not handle source exports wiht mor o kern/167067 fs [zfs] [panic] ZFS panics the server o kern/167065 fs [zfs] boot fails when a spare is the boot disk o kern/167048 fs [nfs] [patch] RELEASE-9 crash when using ZFS+NULLFS+NF o kern/166912 fs [ufs] [panic] Panic after converting Softupdates to jo o kern/166851 fs [zfs] [hang] Copying directory from the mounted UFS di o kern/166477 fs [nfs] NFS data corruption. o kern/165950 fs [ffs] SU+J and fsck problem o kern/165923 fs [nfs] Writing to NFS-backed mmapped files fails if flu o kern/165521 fs [zfs] [hang] livelock on 1 Gig of RAM with zfs when 31 o kern/165392 fs Multiple mkdir/rmdir fails with errno 31 o kern/165087 fs [unionfs] lock violation in unionfs o kern/164472 fs [ufs] fsck -B panics on particular data inconsistency o kern/164370 fs [zfs] zfs destroy for snapshot fails on i386 and sparc o kern/164261 fs [nullfs] [patch] fix panic with NFS served from NULLFS o kern/164256 fs [zfs] device entry for volume is not created after zfs o kern/164184 fs [ufs] [panic] Kernel panic with ufs_makeinode o kern/163801 fs [md] [request] allow mfsBSD legacy installed in 'swap' o kern/163770 fs [zfs] [hang] LOR between zfs&syncer + vnlru leading to o kern/163501 fs [nfs] NFS exporting a dir and a subdir in that dir to o kern/162944 fs [coda] Coda file system module looks broken in 9.0 o kern/162860 fs [zfs] Cannot share ZFS filesystem to hosts with a hyph o kern/162751 fs [zfs] [panic] kernel panics during file operations o kern/162591 fs [nullfs] cross-filesystem nullfs does not work as expe o kern/162519 fs [zfs] "zpool import" relies on buggy realpath() behavi o kern/162362 fs [snapshots] [panic] ufs with snapshot(s) panics when g o kern/161968 fs [zfs] [hang] renaming snapshot with -r including a zvo o kern/161864 fs [ufs] removing journaling from UFS partition fails on o bin/161807 fs [patch] add option for explicitly specifying metadata o kern/161579 fs [smbfs] FreeBSD sometimes panics when an smb share is o kern/161533 fs [zfs] [panic] zfs receive panic: system ioctl returnin o kern/161438 fs [zfs] [panic] recursed on non-recursive spa_namespace_ o kern/161424 fs [nullfs] __getcwd() calls fail when used on nullfs mou o kern/161280 fs [zfs] Stack overflow in gptzfsboot o kern/161205 fs [nfs] [pfsync] [regression] [build] Bug report freebsd o kern/161169 fs [zfs] [panic] ZFS causes kernel panic in dbuf_dirty o kern/161112 fs [ufs] [lor] filesystem LOR in FreeBSD 9.0-BETA3 o kern/160893 fs [zfs] [panic] 9.0-BETA2 kernel panic o kern/160860 fs [ufs] Random UFS root filesystem corruption with SU+J o kern/160801 fs [zfs] zfsboot on 8.2-RELEASE fails to boot from root-o o kern/160790 fs [fusefs] [panic] VPUTX: negative ref count with FUSE o kern/160777 fs [zfs] [hang] RAID-Z3 causes fatal hang upon scrub/impo o kern/160706 fs [zfs] zfs bootloader fails when a non-root vdev exists o kern/160591 fs [zfs] Fail to boot on zfs root with degraded raidz2 [r o kern/160410 fs [smbfs] [hang] smbfs hangs when transferring large fil o kern/160283 fs [zfs] [patch] 'zfs list' does abort in make_dataset_ha o kern/159930 fs [ufs] [panic] kernel core o kern/159402 fs [zfs][loader] symlinks cause I/O errors o kern/159357 fs [zfs] ZFS MAXNAMELEN macro has confusing name (off-by- o kern/159356 fs [zfs] [patch] ZFS NAME_ERR_DISKLIKE check is Solaris-s o kern/159351 fs [nfs] [patch] - divide by zero in mountnfs() o kern/159251 fs [zfs] [request]: add FLETCHER4 as DEDUP hash option o kern/159077 fs [zfs] Can't cd .. with latest zfs version o kern/159048 fs [smbfs] smb mount corrupts large files o kern/159045 fs [zfs] [hang] ZFS scrub freezes system o kern/158839 fs [zfs] ZFS Bootloader Fails if there is a Dead Disk o kern/158802 fs amd(8) ICMP storm and unkillable process. o kern/158231 fs [nullfs] panic on unmounting nullfs mounted over ufs o f kern/157929 fs [nfs] NFS slow read o kern/157399 fs [zfs] trouble with: mdconfig force delete && zfs strip o kern/157179 fs [zfs] zfs/dbuf.c: panic: solaris assert: arc_buf_remov o kern/156797 fs [zfs] [panic] Double panic with FreeBSD 9-CURRENT and o kern/156781 fs [zfs] zfs is losing the snapshot directory, p kern/156545 fs [ufs] mv could break UFS on SMP systems o kern/156193 fs [ufs] [hang] UFS snapshot hangs && deadlocks processes o kern/156039 fs [nullfs] [unionfs] nullfs + unionfs do not compose, re o kern/155615 fs [zfs] zfs v28 broken on sparc64 -current o kern/155587 fs [zfs] [panic] kernel panic with zfs p kern/155411 fs [regression] [8.2-release] [tmpfs]: mount: tmpfs : No o kern/155199 fs [ext2fs] ext3fs mounted as ext2fs gives I/O errors o bin/155104 fs [zfs][patch] use /dev prefix by default when importing o kern/154930 fs [zfs] cannot delete/unlink file from full volume -> EN o kern/154828 fs [msdosfs] Unable to create directories on external USB o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 p kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o bin/153142 fs [zfs] ls -l outputs `ls: ./.zfs: Operation not support o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an f bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis p kern/133174 fs [msdosfs] [patch] msdosfs must support multibyte inter o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o kern/118318 fs [nfs] NFS server hangs under special circumstances o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118126 fs [nfs] [patch] Poor NFS server write performance o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116583 fs [ffs] [hang] System freezes for short time when using o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 295 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Dec 24 11:48:09 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8070C38C for ; Mon, 24 Dec 2012 11:48:09 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: from mail-bk0-f47.google.com (mail-bk0-f47.google.com [209.85.214.47]) by mx1.freebsd.org (Postfix) with ESMTP id 019068FC0A for ; Mon, 24 Dec 2012 11:48:08 +0000 (UTC) Received: by mail-bk0-f47.google.com with SMTP id j4so3275605bkw.20 for ; Mon, 24 Dec 2012 03:48:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:date:from:user-agent:mime-version:to:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=T7wuagg/WZeAD/GTPmOTPVinISbDdFYXRfCFu9KxhX4=; b=PtHgg9RS0XspRvcX7kO25e71N4v230CJ7OkhCfK6YVoPcSooo0fuyuRSlK8IuiyipY oIaQFWevpExOQQYWDgeSUr+r014ocOjT/bH7D2Xc8bMWC6T3GNvfmcxR9DVAEyRbLWSD jUK6Il2eYFe8Vv084TdNnZqtpWprAoWBhrgTqyHLniOIjO7ee0nW7GSKvSHfIuGGMfeS 2o+hs8FRqgsWXQTWkK3hgJyx9tNiyBYt2wSUu3uUFaKsUDSpEahLueCwuYz/7SKxZyEj gkBviVMUe53+dsf6EMCzn51FCWhDY/+tlV6SEISGMKZ7EWutstpKgYPEdW/IAx9v6YMu qzPw== X-Received: by 10.204.13.28 with SMTP id z28mr10289462bkz.113.1356349681706; Mon, 24 Dec 2012 03:48:01 -0800 (PST) Received: from [192.168.50.105] (double-l.xs4all.nl. [80.126.205.144]) by mx.google.com with ESMTPS id u3sm14923613bkw.9.2012.12.24.03.47.59 (version=SSLv3 cipher=OTHER); Mon, 24 Dec 2012 03:48:00 -0800 (PST) Message-ID: <50D840ED.20507@gmail.com> Date: Mon, 24 Dec 2012 12:47:57 +0100 From: Johan Hendriks User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Derek Kulinski , freebsd-fs@freebsd.org Subject: Re: FreeBSD 9.1-RELEASE crashes almost daily; backtraces always list zfs routines References: <1824023197.20121223142308@takeda.tk> In-Reply-To: <1824023197.20121223142308@takeda.tk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2012 11:48:09 -0000 Derek Kulinski schreef: > Please help, I reported this issue on > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/174372 but the crashes > are unbearable since they happen regularly at night, most of the time > when periodic.daily is called (3am) but there are exceptions. It seems > like it can be triggered by any heavy disk activity. In many of the > crash dumps the current process is find command, but of course that's > does not always is the cause. > > When calling scrub, it appears to pass successfully. Smartmon tools is > not reporting any disk errors. I tested memory using memtest86 about a > month ago and it passed tests successfully. > > I never had this type of issue on 9.0, and not much changed in my > kernel config besides installing WiFi card. > > System: > FreeBSD chinatsu.takeda.tk 9.1-RELEASE FreeBSD 9.1-RELEASE #2 r244482: Wed Dec 19 23:28:15 PST 2012 root@chinatsu.takeda.tk:/usr/obj/usr/src/sys/CHINATSU amd64 > It's compiled from releng/9.1 branch. > > Thank you for any help, > Derek > > > Example crash messages: > > Today: > panic: general protection fault > cpuid = 1 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > kdb_backtrace() at kdb_backtrace+0x37 > panic() at panic+0x1cd > trap_fatal() at trap_fatal+0x285 > trap() at trap+0x23e > calltrap() at calltrap+0x8 > --- trap 0x9, rip = 0xffffffff8101f5ac, rsp = 0xffffff8230ac18c0, rbp = 0xffffff8230ac18d0 --- > list_prev() at list_prev+0xc > arc_evict() at arc_evict+0x194 > arc_adjust() at arc_adjust+0x1a1 > arc_reclaim_thread() at arc_reclaim_thread+0x1a6 > fork_exit() at fork_exit+0x11c > fork_trampoline() at fork_trampoline+0xe > --- trap 0, rip = 0, rsp = 0xffffff8230ac1bb0, rbp = 0 --- > Uptime: 20h44m51s > Dumping 1157 out of 8072 MB:..2%..12%..21%..31%..41%..52%..61%..71%..81%..92% > > > Yesterday: > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0xffffffffffffffe8 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff8102d1e7 > stack pointer = 0x28:0xffffff82315ec190 > frame pointer = 0x28:0xffffff82315ec280 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 34744 (find) > trap number = 12 > panic: page fault > cpuid = 0 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > kdb_backtrace() at kdb_backtrace+0x37 > panic() at panic+0x1cd > trap_fatal() at trap_fatal+0x285 > trap_pfault() at trap_pfault+0x216 > trap() at trap+0x363 > calltrap() at calltrap+0x8 > --- trap 0xc, rip = 0xffffffff8102d1e7, rsp = 0xffffff82315ec190, rbp = 0xffffff82315ec280 --- > arc_evict() at arc_evict+0x1e7 > arc_get_data_buf() at arc_get_data_buf+0x1d5 > arc_read_nolock() at arc_read_nolock+0x1ec > arc_read() at arc_read+0x93 > dbuf_read() at dbuf_read+0x452 > dmu_buf_hold() at dmu_buf_hold+0xe0 > zap_lockdir() at zap_lockdir+0x58 > zap_cursor_retrieve() at zap_cursor_retrieve+0x19b > zfs_freebsd_readdir() at zfs_freebsd_readdir+0x2ee > VOP_READDIR_APV() at VOP_READDIR_APV+0x4a > kern_getdirentries() at kern_getdirentries+0x225 > sys_getdirentries() at sys_getdirentries+0x23 > amd64_syscall() at amd64_syscall+0x5d8 > Xfast_syscall() at Xfast_syscall+0xf7 > --- syscall (196, FreeBSD ELF64, sys_getdirentries), rip = 0x80089b29c, rsp = 0x7fffffffd8a8, rbp = 0x1 --- > Uptime: 2d3h49m3s > Dumping 3701 out of 8072 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > > > I can not help you with the crash itself. The problem with ZFS and the daily run is the check for certain files changes and permissions. If you have a lot of snapshots, find will be looking for those file through all of your snapshots. To disable this you can look at the following thread. http://forums.freebsd.org/archive/index.php/t-29994.html or the following. http://forums.freebsd.org/archive/index.php/t-31846.html regards Johan Hendriks From owner-freebsd-fs@FreeBSD.ORG Mon Dec 24 12:00:01 2012 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6BB6A472 for ; Mon, 24 Dec 2012 12:00:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 50F698FC0C for ; Mon, 24 Dec 2012 12:00:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id qBOC010w070867 for ; Mon, 24 Dec 2012 12:00:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id qBOC01wp070864; Mon, 24 Dec 2012 12:00:01 GMT (envelope-from gnats) Date: Mon, 24 Dec 2012 12:00:01 GMT Message-Id: <201212241200.qBOC01wp070864@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: Nikolay Denev Subject: Re: kern/88266: [smbfs] smbfs does not implement UIO_NOCOPY and sendfile(2) on smbfs mounted files fails X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Nikolay Denev List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2012 12:00:01 -0000 The following reply was made to PR kern/88266; it has been noted by GNATS. From: Nikolay Denev To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/88266: [smbfs] smbfs does not implement UIO_NOCOPY and sendfile(2) on smbfs mounted files fails Date: Mon, 24 Dec 2012 13:41:32 +0200 This PR can be closed : src/lib/libc/sys/sendfile.2 Revision 1.29: download - view: text, markup, annotated - select for diffs Mon Oct 31 04:08:28 2005 UTC (7 years, 1 month ago) by jkoshy Branches: MAIN Diff to: previous 1.28: preferred, colored Changes since revision 1.28: +5 -0 lines Document the fact that sendfile(2) can EOPNOTSUPP if the underlying filesystem for the file being transferred doesn't support UIO_NOCOPY. Reported by: Niki Denev From owner-freebsd-fs@FreeBSD.ORG Mon Dec 24 16:01:33 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4F2FD6F1; Mon, 24 Dec 2012 16:01:33 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 512118FC14; Mon, 24 Dec 2012 16:01:31 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id SAA05000; Mon, 24 Dec 2012 18:01:26 +0200 (EET) (envelope-from avg@FreeBSD.org) Message-ID: <50D87C56.70709@FreeBSD.org> Date: Mon, 24 Dec 2012 18:01:26 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Derek Kulinski Subject: Re: FreeBSD 9.1-RELEASE crashes almost daily; backtraces always list zfs routines References: <1824023197.20121223142308@takeda.tk> In-Reply-To: <1824023197.20121223142308@takeda.tk> X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2012 16:01:33 -0000 on 24/12/2012 00:23 Derek Kulinski said the following: > Dumping 3701 out of 8072 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% So do you have the crash dump(s)? -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Mon Dec 24 18:19:05 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2392A6D8; Mon, 24 Dec 2012 18:19:05 +0000 (UTC) (envelope-from takeda@takeda.tk) Received: from chinatsu.takeda.tk (mail.takeda.tk [74.0.89.210]) by mx1.freebsd.org (Postfix) with ESMTP id 977B18FC16; Mon, 24 Dec 2012 18:19:04 +0000 (UTC) Received: from localhost.takeda.tk (takeda-ws2.lan [10.0.0.123]) (authenticated bits=0) by chinatsu.takeda.tk (8.14.5/8.14.5) with ESMTP id qBOIJ2Gs012272 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO); Mon, 24 Dec 2012 10:19:03 -0800 (PST) (envelope-from takeda@takeda.tk) Date: Mon, 24 Dec 2012 10:17:19 -0800 From: Derek Kulinski X-Priority: 3 (Normal) Message-ID: <331959998.20121224101719@takeda.tk> To: Andriy Gapon Subject: Re: FreeBSD 9.1-RELEASE crashes almost daily; backtraces always list zfs routines In-Reply-To: <50D87C56.70709@FreeBSD.org> References: <1824023197.20121223142308@takeda.tk> <50D87C56.70709@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.97.6 at chinatsu.takeda.tk X-Virus-Status: Clean Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2012 18:19:05 -0000 Hello Andriy, Monday, December 24, 2012, 8:01:26 AM, you wrote: > on 24/12/2012 00:23 Derek Kulinski said the following: >> Dumping 3701 out of 8072 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > So do you have the crash dump(s)? Yes, but they are 3.5GB each. I attached text dump to GNATS but I can resend it to you (I don't know if it's ok to send attachments to the mailing list). If you would prefer I could give you access to the box. -- Best regards, Derek mailto:takeda@takeda.tk -- Programmer - A red-eyed, mumbling mammal capable of conversing with inanimate objects. From owner-freebsd-fs@FreeBSD.ORG Mon Dec 24 20:46:55 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1A8DE1D8; Mon, 24 Dec 2012 20:46:55 +0000 (UTC) (envelope-from linimon@lonesome.com) Received: from mail.soaustin.net (pancho.soaustin.net [76.74.250.40]) by mx1.freebsd.org (Postfix) with ESMTP id E51BF8FC0A; Mon, 24 Dec 2012 20:46:54 +0000 (UTC) Received: by mail.soaustin.net (Postfix, from userid 502) id 8CE1256074; Mon, 24 Dec 2012 14:46:53 -0600 (CST) Date: Mon, 24 Dec 2012 14:46:53 -0600 From: Mark Linimon To: Derek Kulinski Subject: Re: FreeBSD 9.1-RELEASE crashes almost daily; backtraces always list zfs routines Message-ID: <20121224204653.GA23711@lonesome.com> References: <1824023197.20121223142308@takeda.tk> <50D87C56.70709@FreeBSD.org> <331959998.20121224101719@takeda.tk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <331959998.20121224101719@takeda.tk> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org, Andriy Gapon X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2012 20:46:55 -0000 On Mon, Dec 24, 2012 at 10:17:19AM -0800, Derek Kulinski wrote: > Yes, but they are 3.5GB each. I attached text dump to GNATS but I can > resend it to you We have a limit of 500K on GNATS PRs. For something that huge, a PR database is really not the right place for it -- please post the dumps somewhere and include a URL to them in a followup to the PR. Thanks. Mark Linimon, on behalf of bugmeister From owner-freebsd-fs@FreeBSD.ORG Mon Dec 24 20:49:11 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7B9EE356 for ; Mon, 24 Dec 2012 20:49:11 +0000 (UTC) (envelope-from takeda@takeda.tk) Received: from chinatsu.takeda.tk (mail.takeda.tk [74.0.89.210]) by mx1.freebsd.org (Postfix) with ESMTP id 4E50A8FC0C for ; Mon, 24 Dec 2012 20:49:11 +0000 (UTC) Received: from localhost.takeda.tk (takeda-ws2.lan [10.0.0.123]) (authenticated bits=0) by chinatsu.takeda.tk (8.14.5/8.14.5) with ESMTP id qBOKnAG2015141 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO); Mon, 24 Dec 2012 12:49:10 -0800 (PST) (envelope-from takeda@takeda.tk) Date: Mon, 24 Dec 2012 12:43:08 -0800 From: Derek Kulinski X-Priority: 3 (Normal) Message-ID: <1975381197.20121224124308@takeda.tk> To: Johan Hendriks Subject: Re: FreeBSD 9.1-RELEASE crashes almost daily; backtraces always list zfs routines In-Reply-To: <50D840ED.20507@gmail.com> References: <1824023197.20121223142308@takeda.tk> <50D840ED.20507@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.97.6 at chinatsu.takeda.tk X-Virus-Status: Clean Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2012 20:49:11 -0000 Hello Johan, Monday, December 24, 2012, 3:47:57 AM, you wrote: > I can not help you with the crash itself. > The problem with ZFS and the daily run is the check for certain files > changes and permissions. > If you have a lot of snapshots, find will be looking for those file > through all of your snapshots. > To disable this you can look at the following thread. > http://forums.freebsd.org/archive/index.php/t-29994.html > or the following. > http://forums.freebsd.org/archive/index.php/t-31846.html Looks like same issue or at very least a very similar one. I do have many snapshots but the snapdir is defined as hidden for filesystems. Anyway, the advices there look to me like are more about hiding the symptom rather than fixing the issue. I still had it happen at least once during regular operation. Well at worst at least I have way to reduce number of crashes. Thank you. -- Best regards, Derek mailto:takeda@takeda.tk After Perl everything else is just assembly language. From owner-freebsd-fs@FreeBSD.ORG Mon Dec 24 20:55:29 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 07C664C6; Mon, 24 Dec 2012 20:55:29 +0000 (UTC) (envelope-from takeda@takeda.tk) Received: from chinatsu.takeda.tk (mail.takeda.tk [74.0.89.210]) by mx1.freebsd.org (Postfix) with ESMTP id CC5DF8FC12; Mon, 24 Dec 2012 20:55:28 +0000 (UTC) Received: from localhost.takeda.tk (takeda-ws2.lan [10.0.0.123]) (authenticated bits=0) by chinatsu.takeda.tk (8.14.5/8.14.5) with ESMTP id qBOKtRZO015237 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO); Mon, 24 Dec 2012 12:55:28 -0800 (PST) (envelope-from takeda@takeda.tk) Date: Mon, 24 Dec 2012 12:55:38 -0800 From: Derek Kulinski X-Priority: 3 (Normal) Message-ID: <47474870.20121224125538@takeda.tk> To: Mark Linimon Subject: Re: FreeBSD 9.1-RELEASE crashes almost daily; backtraces always list zfs routines In-Reply-To: <20121224204653.GA23711@lonesome.com> References: <1824023197.20121223142308@takeda.tk> <50D87C56.70709@FreeBSD.org> <331959998.20121224101719@takeda.tk> <20121224204653.GA23711@lonesome.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.97.6 at chinatsu.takeda.tk X-Virus-Status: Clean Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org, Andriy Gapon X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2012 20:55:29 -0000 Hello Mark, Monday, December 24, 2012, 12:46:53 PM, you wrote: > On Mon, Dec 24, 2012 at 10:17:19AM -0800, Derek Kulinski wrote: >> Yes, but they are 3.5GB each. I attached text dump to GNATS but I can >> resend it to you > We have a limit of 500K on GNATS PRs. For something that huge, a PR > database is really not the right place for it -- please post the dumps > somewhere and include a URL to them in a followup to the PR. > Thanks. > Mark Linimon, on behalf of bugmeister I included the text dump, but I do not see it when I visit the web interface so I don't know if it was attached there or not. -- Best regards, Derek mailto:takeda@takeda.tk My new car runs at 56Kbps From owner-freebsd-fs@FreeBSD.ORG Mon Dec 24 22:32:07 2012 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 483AB79F; Mon, 24 Dec 2012 22:32:07 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 12BA78FC0A; Mon, 24 Dec 2012 22:32:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id qBOMW6V9096955; Mon, 24 Dec 2012 22:32:06 GMT (envelope-from eadler@freefall.freebsd.org) Received: (from eadler@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id qBOMW63x096951; Mon, 24 Dec 2012 22:32:06 GMT (envelope-from eadler) Date: Mon, 24 Dec 2012 22:32:06 GMT Message-Id: <201212242232.qBOMW63x096951@freefall.freebsd.org> To: nike_d@cytexbg.com, eadler@FreeBSD.org, freebsd-fs@FreeBSD.org From: eadler@FreeBSD.org Subject: Re: kern/88266: [smbfs] smbfs does not implement UIO_NOCOPY and sendfile(2) on smbfs mounted files fails X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2012 22:32:07 -0000 Synopsis: [smbfs] smbfs does not implement UIO_NOCOPY and sendfile(2) on smbfs mounted files fails State-Changed-From-To: open->closed State-Changed-By: eadler State-Changed-When: Mon Dec 24 22:32:06 UTC 2012 State-Changed-Why: Per submitter request http://www.freebsd.org/cgi/query-pr.cgi?pr=88266 From owner-freebsd-fs@FreeBSD.ORG Mon Dec 24 23:28:03 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C68BC347; Mon, 24 Dec 2012 23:28:03 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id DCC1D8FC0A; Mon, 24 Dec 2012 23:28:02 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id BAA07223; Tue, 25 Dec 2012 01:28:01 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1TnHRB-000Hxu-40; Tue, 25 Dec 2012 01:28:01 +0200 Message-ID: <50D8E500.1070408@FreeBSD.org> Date: Tue, 25 Dec 2012 01:28:00 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Derek Kulinski Subject: Re: FreeBSD 9.1-RELEASE crashes almost daily; backtraces always list zfs routines References: <1824023197.20121223142308@takeda.tk> <50D87C56.70709@FreeBSD.org> <331959998.20121224101719@takeda.tk> In-Reply-To: <331959998.20121224101719@takeda.tk> X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2012 23:28:03 -0000 on 24/12/2012 20:17 Derek Kulinski said the following: > Hello Andriy, > > Monday, December 24, 2012, 8:01:26 AM, you wrote: > >> on 24/12/2012 00:23 Derek Kulinski said the following: >>> Dumping 3701 out of 8072 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > >> So do you have the crash dump(s)? > > Yes, but they are 3.5GB each. I attached text dump to GNATS but I can > resend it to you (I don't know if it's ok to send attachments to the > mailing list). If you would prefer I could give you access to the > box. Derek, I've looked through the cores and it does look like in all cases some sort of memory corruption is a precursor to a subsequent crash. I can't decidedly say if the corruptions are caused by the hardware, by some code overwriting random memory locations ("rogue" driver) or by a "simpler" bug like use after free. I am always inclined to suspect the hardware first. You can try to reproduce the problem with some additional checks enabled in the kernel. Those should catch the problem earlier and thus make its source clearer. I recommend the following: options INVARIANTS options INVARIANT_SUPPORT options WITNESS options DEBUG_MEMGUARD makeoptions DEBUG+="-DDEBUG" The last is really needed only for the ZFS and OpenSolaris compat code. It make result in some extra noise from unrelated subsystems. Perhaps you could just add "#define DEBUG" to sys/cddl/contrib/opensolaris/uts/common/sys/debug.h. I haven't tested this approach though. Also, please put vm.memguard.desc="arc_buf_hdr_t" into loader.conf. Please note that these options will make your system significantly slower. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Tue Dec 25 00:11:53 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2B845A54; Tue, 25 Dec 2012 00:11:53 +0000 (UTC) (envelope-from takeda@takeda.tk) Received: from chinatsu.takeda.tk (mail.takeda.tk [74.0.89.210]) by mx1.freebsd.org (Postfix) with ESMTP id EF45A8FC0A; Tue, 25 Dec 2012 00:11:52 +0000 (UTC) Received: from localhost.takeda.tk (takeda-ws2.lan [10.0.0.123]) (authenticated bits=0) by chinatsu.takeda.tk (8.14.5/8.14.5) with ESMTP id qBP0BjNe002051 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO); Mon, 24 Dec 2012 16:11:46 -0800 (PST) (envelope-from takeda@takeda.tk) Date: Mon, 24 Dec 2012 16:11:56 -0800 From: Derek Kulinski X-Priority: 3 (Normal) Message-ID: <574019558.20121224161156@takeda.tk> To: Andriy Gapon Subject: Re: FreeBSD 9.1-RELEASE crashes almost daily; backtraces always list zfs routines In-Reply-To: <50D8E500.1070408@FreeBSD.org> References: <1824023197.20121223142308@takeda.tk> <50D87C56.70709@FreeBSD.org> <331959998.20121224101719@takeda.tk> <50D8E500.1070408@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.97.6 at chinatsu.takeda.tk X-Virus-Status: Clean Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Dec 2012 00:11:53 -0000 Hello Andriy, Monday, December 24, 2012, 3:28:00 PM, you wrote: > I've looked through the cores and it does look like in all cases some sort of > memory corruption is a precursor to a subsequent crash. > I can't decidedly say if the corruptions are caused by the hardware, by some > code overwriting random memory locations ("rogue" driver) or by a "simpler" bug > like use after free. > I am always inclined to suspect the hardware first. > You can try to reproduce the problem with some additional checks enabled in the > kernel. Those should catch the problem earlier and thus make its source clearer. > I recommend the following: > options INVARIANTS > options INVARIANT_SUPPORT > options WITNESS > options DEBUG_MEMGUARD > makeoptions DEBUG+="-DDEBUG" > The last is really needed only for the ZFS and OpenSolaris compat code. It make > result in some extra noise from unrelated subsystems. > Perhaps you could just add "#define DEBUG" to > sys/cddl/contrib/opensolaris/uts/common/sys/debug.h. I haven't tested this > approach though. > Also, please put vm.memguard.desc="arc_buf_hdr_t" into loader.conf. > Please note that these options will make your system significantly slower. I recompiled the kernel and is running with options you specified (I enabled DEBUG in the file). Anyway even at boot time I started getting following warnings, is this anything: Dec 24 16:06:03 chinatsu kernel: Creating and/or trimming log files Dec 24 16:06:03 chinatsu kernel: lock order reversal: Dec 24 16:06:03 chinatsu kernel: 1st 0xffffffff80bf5780 pf task mtx (pf task mtx) @ /usr/src/sys/contrib/pf/net/pf.c:3330 Dec 24 16:06:03 chinatsu kernel: . Dec 24 16:06:03 chinatsu kernel: 2nd 0xfffffe0009211af8 radix node head (radix node head) @ /usr/src/sys/net/route.c:384 Dec 24 16:06:03 chinatsu kernel: KDB: stack backtrace: Dec 24 16:06:03 chinatsu kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Dec 24 16:06:03 chinatsu kernel: kdb_backtrace() at kdb_backtrace+0x37 Dec 24 16:06:03 chinatsu kernel: _witness_debugger() at _witness_debugger+0x2c Dec 24 16:06:03 chinatsu kernel: witness_checkorder() at witness_checkorder+0x844 Dec 24 16:06:03 chinatsu kernel: _rw_rlock() at Dec 24 16:06:03 chinatsu kernel: Starting syslogd. Dec 24 16:06:03 chinatsu kernel: _rw_rlock+0x81 Dec 24 16:06:03 chinatsu kernel: rtalloc1_fib() at rtalloc1_fib+0x11c Dec 24 16:06:03 chinatsu kernel: rtalloc_ign_fib() at rtalloc_ign_fib+0xc5 Dec 24 16:06:03 chinatsu kernel: pf_routable() at pf_routable+0x1fd Dec 24 16:06:03 chinatsu kernel: pf_test_rule() at pf_test_rule+0x6cf Dec 24 16:06:03 chinatsu kernel: pf_test() at pf_test+0xf58 Dec 24 16:06:03 chinatsu kernel: pf_check_in() at pf_check_in+0x2b Dec 24 16:06:03 chinatsu kernel: pfil_run_hooks() at pfil_run_hooks+0xd2 Dec 24 16:06:03 chinatsu kernel: ip_input() at ip_input+0x2dc Dec 24 16:06:03 chinatsu kernel: netisr_dispatch_src() at netisr_dispatch_src+0x170 Dec 24 16:06:03 chinatsu kernel: ether_demux() at ether_demux+0x17d Dec 24 16:06:03 chinatsu kernel: ether_nh_input() at ether_nh_input+0x209 Dec 24 16:06:03 chinatsu kernel: netisr_dispatch_src() at netisr_dispatch_src+0x170 Dec 24 16:06:03 chinatsu kernel: alc_int_task() at alc_int_task+0x2ff Dec 24 16:06:03 chinatsu kernel: taskqueue_run_locked() at taskqueue_run_locked+0x93 Dec 24 16:06:03 chinatsu kernel: taskqueue_thread_loop() at taskqueue_thread_loop+0x3e Dec 24 16:06:03 chinatsu kernel: fork_exit() at fork_exit+0x133 Dec 24 16:06:03 chinatsu kernel: fork_trampoline() at fork_trampoline+0xe Dec 24 16:06:03 chinatsu kernel: --- trap 0, rip = 0, rsp = 0xffffff85fb2ebbb0, rbp = 0 --- Dec 24 16:06:03 chinatsu kernel: No core dumps found. Dec 24 16:06:04 chinatsu kernel: lock order reversal: Dec 24 16:06:04 chinatsu kernel: 1st 0xffffff85b9cb8dd8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2677 Dec 24 16:06:04 chinatsu kernel: 2nd 0xfffffe00092c5c00 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284 Dec 24 16:06:04 chinatsu kernel: KDB: stack backtrace: Dec 24 16:06:04 chinatsu kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Dec 24 16:06:04 chinatsu kernel: kdb_backtrace() at kdb_backtrace+0x37 Dec 24 16:06:04 chinatsu kernel: _witness_debugger() at _witness_debugger+0x2c Dec 24 16:06:04 chinatsu kernel: witness_checkorder() at witness_checkorder+0x844 Dec 24 16:06:04 chinatsu kernel: _sx_xlock() at _sx_xlock+0x61 Dec 24 16:06:04 chinatsu kernel: ufsdirhash_acquire() at ufsdirhash_acquire+0x33 Dec 24 16:06:04 chinatsu kernel: ufsdirhash_remove() at Dec 24 16:06:04 chinatsu kernel: ufsdirhash_remove+0x16 Dec 24 16:06:04 chinatsu kernel: ufs_dirremove() at ufs_dirremove+0x1bb Dec 24 16:06:04 chinatsu kernel: ufs_remove() at ufs_remove+0x92 Dec 24 16:06:04 chinatsu kernel: VOP_REMOVE_APV() at VOP_REMOVE_APV+0xb7 Dec 24 16:06:04 chinatsu kernel: kern_unlinkat() at kern_unlinkat+0x2eb Dec 24 16:06:04 chinatsu kernel: amd64_syscall() at amd64_syscall+0x30e Dec 24 16:06:04 chinatsu kernel: Xfast_syscall() at Xfast_syscall+0xf7 Dec 24 16:06:04 chinatsu kernel: --- syscall (10, FreeBSD ELF64, sys_unlink), rip = 0x80090a22c, rsp = 0x7fffffff Dec 24 16:06:04 chinatsu kernel: ca88, rbp = 0x7fffffffdf20 --- Dec 24 16:06:04 chinatsu kernel: lock order reversal: Dec 24 16:06:04 chinatsu kernel: 1st 0xfffffe00266ddbd8 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:849 Dec 24 16:06:04 chinatsu kernel: 2nd 0xfffffe002679a818 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2158 Dec 24 16:06:04 chinatsu kernel: KDB: stack backtrace: Dec 24 16:06:04 chinatsu kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Dec 24 16:06:04 chinatsu kernel: kdb_backtrace() at kdb_backtrace+0x37 Dec 24 16:06:04 chinatsu kernel: _witness_debugger() at _witness_debugger+0x2c Dec 24 16:06:04 chinatsu kernel: witness_checkorder() at witness_checkorder+0x844 Dec 24 16:06:04 chinatsu kernel: __lockmgr_args() at __lockmgr_args+0x10d9 Dec 24 16:06:04 chinatsu kernel: vop_stdlock() at vop_stdlock+0x39 Dec 24 16:06:04 chinatsu kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0xbf Dec 24 16:06:04 chinatsu kernel: _vn_lock() at _vn_lock+0x47 Dec 24 16:06:04 chinatsu kernel: vget() at vget+0x7b Dec 24 16:06:04 chinatsu kernel: devfs_allocv() at devfs_allocv+0x13f Dec 24 16:06:04 chinatsu kernel: devfs_root() at devfs_root+0x4d Dec 24 16:06:04 chinatsu kernel: vfs_donmount() at vfs_donmount+0xafa Dec 24 16:06:04 chinatsu kernel: sys_nmount() at sys_nmount+0x66 Dec 24 16:06:04 chinatsu kernel: amd64_syscall() at amd64_syscall+0x30e Dec 24 16:06:04 chinatsu kernel: Xfast_syscall() at Xfast_syscall+0xf7 Dec 24 16:06:04 chinatsu kernel: --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x800a8d71c, rsp = 0x7fffffffccc8, rbp = 0x801009048 --- Dec 24 16:06:05 chinatsu named[1387]: starting BIND 9.8.3-P4 -t /var/named -u bind Dec 24 16:06:05 chinatsu kernel: Starting named. -- Best regards, Derek mailto:takeda@takeda.tk People say Microsoft paid 14M$ for using the Rolling Stones song 'Start me up' in their commercials. This is wrong. Microsoft payed 14M$ only for a part of the song. For instance, they didn't use the line 'You'll make a grown man cry'. From owner-freebsd-fs@FreeBSD.ORG Tue Dec 25 05:42:45 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 98C7DF98; Tue, 25 Dec 2012 05:42:45 +0000 (UTC) (envelope-from olivier777a7@gmail.com) Received: from mail-la0-f53.google.com (mail-la0-f53.google.com [209.85.215.53]) by mx1.freebsd.org (Postfix) with ESMTP id 159E48FC14; Tue, 25 Dec 2012 05:42:43 +0000 (UTC) Received: by mail-la0-f53.google.com with SMTP id w12so9112316lag.26 for ; Mon, 24 Dec 2012 21:42:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=e53pbmEe4yus1pqAV51dzsbm7DtF2kbV9vSrhgKG4v0=; b=E81UhbflvqrcqOOwlnqI96AyYpzRACn5ykMI98GOTpJTT34LYSIMpGna1/eBT+cvG5 GZVS+8Rp6AGhMWteQg7wGZfjFTgng1HPAhqMi50GrbB849BYyVRqqu7snXsio4UyzUrf uocS58dp7G0PANMZAoKTs+JyO+PmWFl6wU3/UKYDsAxSzLI8SF9QsVXEkKQejNBWFtQy 5PKoo9vPj7UuJo8Y2ZIb/eJvp+UsSJcuqbc8P2ec0i7xVohndZBDpXOxvqVYYgwi+dnT boyIHdHtF4ooPEG7YFm80pGBE7NcuZevZ+yQ0ELHpDKYf5jrs1AWq/0b/QCTvvUbG4ej sFqw== MIME-Version: 1.0 Received: by 10.112.23.233 with SMTP id p9mr9517554lbf.6.1356414162381; Mon, 24 Dec 2012 21:42:42 -0800 (PST) Received: by 10.114.78.41 with HTTP; Mon, 24 Dec 2012 21:42:42 -0800 (PST) Date: Mon, 24 Dec 2012 21:42:42 -0800 Message-ID: Subject: CAM hangs in 9-STABLE? [Was: NFS/ZFS hangs after upgrading from 9.0-RELEASE to -STABLE] From: olivier To: Andriy Gapon Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org, ken@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Dec 2012 05:42:45 -0000 Dear All It turns out that reverting to an older version of the mps driver did not fix the ZFS hangs I've been struggling with in 9.1 and 9-STABLE after all (they just took a bit longer to occur again, possibly just by chance). I followed steps along lines suggested by Andriy to collect more information when the problem occurs. Hopefully this will help figure out what's going on. As far as I can tell, what happens is that at some point IO operations to a bunch of drives that belong to different pools get stuck. For these drives, gstat shows no activity but 1 pending operation, as such: L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da1 I've been running gstat in a loop (every 100s) to monitor the machine. Just before the hang occurs, everything seems fine (see full gstat output below). Right after the hang occurs a number of drives seem stuck (see full gstat output below). Notably, some stuck drives are seen through the mps driver and others through the mpt driver. So the problem doesn't seem to be driver-specific. I have had the problem occur (at a lower frequency) on similar machines that don't use the mpt driver (and only have 1 disk provided through mps), so the problem doesn't seem to be caused by the mpt driver (and is likely not caused by defective hardware). Since based on the information I provided earlier Andriy thinks the problem might not originate in ZFS, perhaps that means that the problem is in the CAM layer? camcontrol tags -v (as suggested by Andriy) in the hung state shows for example (pass56:mpt1:0:8:20): dev_openings 254 (pass56:mpt1:0:8:20): dev_active 1 (pass56:mpt1:0:8:20): devq_openings 254 (pass56:mpt1:0:8:20): devq_queued 0 (pass56:mpt1:0:8:20): held 0 (pass56:mpt1:0:8:20): mintags 2 (pass56:mpt1:0:8:20): maxtags 255 (I'm not providing full camcontrol tags output below because I couldn't get it to run during the specific hang I documented most thoroughly; the example above is from a different occurrence of the hang). The buses don't seem completely frozen: if I manually remove drives while the machine is hanging, that's picked up by the mpt driver, which prints out corresponding messages to the console. But camcontrol reset all or rescan all don't seem to do anything. I've tried reducing vfs.zfs.vdev.min_pending and vfs.zfs.vdev.max_pending to 1, to no avail. Any suggestions to resolve this problem, work around it, or further investigate it would be greatly appreciated! Thanks a lot Olivier Detailed information: Output of procstat -a -kk when the machine is hanging is available at http://pastebin.com/7D2KtT35 (not putting it here because it's pretty long) dmesg is available at http://pastebin.com/9zJQwWJG . Note that I'm using LUN masking, so the "illegal requests" reported aren't really errors. Maybe one day if I get my problems sorted out I'll use geom multipathing instead. My kernel config is include GENERIC ident MYKERNEL options IPSEC device crypto options OFED # Infiniband protocol device mlx4ib # ConnectX Infiniband support device mlxen # ConnectX Ethernet support device mthca # Infinihost cards device ipoib # IP over IB devices options ATA_CAM # Handle legacy controllers with CAM options ATA_STATIC_ID # Static device numbering options KDB options DDB Full output of gstat just before the hang (at most 100s before the hang): L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da2/da2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da0/da0 1 85 48 79 4.7 35 84 0.5 0 0 0.0 24.3 da1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da1/da1 1 83 47 77 4.3 34 79 0.5 0 0 0.0 22.1 da4 1 1324 1303 21433 0.6 19 42 0.7 0 0 0.0 79.8 da3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da5 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da6 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da7 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da8 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da9 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da10 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da11 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da12 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da13 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da14 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da15 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da16 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 0 97 57 93 3.5 38 84 0.3 0 0 0.0 21.3 da20 0 85 47 69 3.3 36 86 0.4 0 0 0.0 16.8 da21 0 1666 1641 18992 0.3 23 43 0.4 0 0 0.0 57.9 da22 0 93 55 98 3.5 36 87 0.4 0 0 0.0 20.6 da23 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da24 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da25 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da26 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da27 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da28 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da29 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da30 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da31 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da32 0 1200 0 0 0.0 1198 11751 0.6 0 0 0.0 67.3 da33 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da34 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da35 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da36 0 81 44 67 2.0 35 84 0.3 0 0 0.0 10.1 da37 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da38 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da39 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da40 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da41 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da42 1 1020 999 22028 0.8 19 42 0.7 0 0 0.0 84.8 da43 0 1050 1029 23479 0.8 19 47 0.7 0 0 0.0 83.3 da44 1 1006 984 22758 0.8 21 46 0.6 0 0 0.0 84.8 da45 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da46 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da47 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da48 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da49 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da50 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 cd0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da4/da4 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da3/da3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da5/da5 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da6/da6 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da7/da7 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da8/da8 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da9/da9 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da10/da10 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da11/da11 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da12/da12 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da13/da13 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da14/da14 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da15/da15 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da16/da16 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da17/da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da18/da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da19/da19 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da20/da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da21/da21 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da22/da22 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da23/da23 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da24/da24 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da25/da25 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da26/da26 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 PART/da26/da26 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da26p1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da26p2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da26p3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da27/da27 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da28/da28 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da29/da29 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da30/da30 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da31/da31 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da32/da32 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da33/da33 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da34/da34 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da35/da35 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da36/da36 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da37/da37 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da38/da38 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da39/da39 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da40/da40 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da41/da41 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da42/da42 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da43/da43 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da44/da44 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da45/da45 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da46/da46 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da47/da47 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da48/da48 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da49/da49 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da50/da50 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/cd0/cd0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da26p1/da26p1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da26p2/da26p2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 LABEL/da26p1/da26p1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 gptid/84d4487b-34e3-11e2-b773-00259058949a 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da26p3/da26p3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 LABEL/da26p2/da26p2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 gptid/b4255780-34e3-11e2-b773-00259058949a 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/gptid/84d4487b-34e3-11e2-b773-00259058949a/gptid/84d4487b-34e3-11e2-b773-00259058949a 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da25 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/gptid/b4255780-34e3-11e2-b773-00259058949a/gptid/b4255780-34e3-11e2-b773-00259058949a 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da40 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da41 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da26p3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da29 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da30 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da24 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da6 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da7 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da16 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da21 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da37 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da23 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da4 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da43 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da44 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da22 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da33 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da45 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da3 Full output of gstat just after the hang (at most 100s after the hang): L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da2/da2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da0/da0 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da1/da1 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da4 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da5 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da6 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da7 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da8 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da9 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da10 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da11 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da12 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da13 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da14 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da15 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da16 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da19 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da20 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da21 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da22 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da23 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da24 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da25 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da26 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da27 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da28 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da29 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da30 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da31 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da32 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da33 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da34 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da35 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da36 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da37 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da38 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da39 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da40 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da41 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da42 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da43 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da44 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da45 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da46 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da47 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da48 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da49 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da50 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 cd0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da4/da4 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da3/da3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da5/da5 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da6/da6 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da7/da7 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da8/da8 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da9/da9 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da10/da10 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da11/da11 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da12/da12 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da13/da13 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da14/da14 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da15/da15 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da16/da16 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da17/da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da18/da18 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da19/da19 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da20/da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da21/da21 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da22/da22 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da23/da23 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da24/da24 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da25/da25 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da26/da26 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 PART/da26/da26 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da26p1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da26p2 1 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 da26p3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da27/da27 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da28/da28 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da29/da29 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da30/da30 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da31/da31 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da32/da32 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da33/da33 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da34/da34 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da35/da35 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da36/da36 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da37/da37 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da38/da38 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da39/da39 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da40/da40 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da41/da41 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da42/da42 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da43/da43 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da44/da44 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da45/da45 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da46/da46 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da47/da47 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da48/da48 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da49/da49 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da50/da50 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/cd0/cd0 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da26p1/da26p1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da26p2/da26p2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 LABEL/da26p1/da26p1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 gptid/84d4487b-34e3-11e2-b773-00259058949a 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/da26p3/da26p3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 LABEL/da26p2/da26p2 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 gptid/b4255780-34e3-11e2-b773-00259058949a 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/gptid/84d4487b-34e3-11e2-b773-00259058949a/gptid/84d4487b-34e3-11e2-b773-00259058949a 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da25 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 DEV/gptid/b4255780-34e3-11e2-b773-00259058949a/gptid/b4255780-34e3-11e2-b773-00259058949a 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da40 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da41 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da26p3 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da29 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da30 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da24 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da6 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da7 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da16 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da17 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da20 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da21 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da37 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da23 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da1 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da4 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da43 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da44 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da22 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da33 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da45 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 ZFS::VDEV/zfs::vdev/da3 On Thu, Dec 13, 2012 at 10:14 PM, olivier wrote: > For what it's worth, I think I might have solved my problem by reverting > to an older version of the mps driver. I checked out a recent version of > 9-STABLE and reversed the changes in > http://svnweb.freebsd.org/base?view=revision&revision=230592 (perhaps > there was a simpler way of reverting to the older mps driver). So far so > good, no hang even when hammering the file system. > > This does not conclusively prove that the new LSI mps driver is at fault, > but that seems to be a likely explanation. > > Thanks to everybody who pointed me in the right direction. Hope this helps > others who run into similar problems with 9.1 > Olivier > > > On Thu, Dec 13, 2012 at 10:14 AM, olivier wrote: > >> >> >> On Thu, Dec 13, 2012 at 9:54 AM, Andriy Gapon wrote: >> >>> Google for "zfs deadman". This is already committed upstream and I >>> think that it >>> is imported into FreeBSD, but I am not sure... Maybe it's imported just >>> into the >>> vendor area and is not merged yet. >>> >> >> Yes, that's exactly what I had in mind. The logic for panicking makes >> sense. >> As far as I can tell you're correct that deadman is in the vendor area >> but not merged. Any idea when it might make it into 9-STABLE? >> Thanks >> Olivier >> >> >> >> >>> So, when enabled this logic would panic a system as a way of letting >>> know that >>> something is wrong. You can read in the links why panic was selected >>> for this job. >>> >>> And speaking FreeBSD-centric - I think that our CAM layer would be a >>> perfect place >>> to detect such issues in non-ZFS-specific way. >>> >>> -- >>> Andriy Gapon >>> >> >> > From owner-freebsd-fs@FreeBSD.ORG Tue Dec 25 08:18:42 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2856A5F7; Tue, 25 Dec 2012 08:18:42 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 324858FC0C; Tue, 25 Dec 2012 08:18:40 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id KAA10993; Tue, 25 Dec 2012 10:18:27 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1TnPiU-000Iee-Ul; Tue, 25 Dec 2012 10:18:27 +0200 Message-ID: <50D9614F.6040306@FreeBSD.org> Date: Tue, 25 Dec 2012 10:18:23 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Derek Kulinski Subject: Re: FreeBSD 9.1-RELEASE crashes almost daily; backtraces always list zfs routines References: <1824023197.20121223142308@takeda.tk> <50D87C56.70709@FreeBSD.org> <331959998.20121224101719@takeda.tk> <50D8E500.1070408@FreeBSD.org> <574019558.20121224161156@takeda.tk> In-Reply-To: <574019558.20121224161156@takeda.tk> X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Dec 2012 08:18:42 -0000 on 25/12/2012 02:11 Derek Kulinski said the following: > Hello Andriy, > > Monday, December 24, 2012, 3:28:00 PM, you wrote: > >> I've looked through the cores and it does look like in all cases some sort of >> memory corruption is a precursor to a subsequent crash. > >> I can't decidedly say if the corruptions are caused by the hardware, by some >> code overwriting random memory locations ("rogue" driver) or by a "simpler" bug >> like use after free. > >> I am always inclined to suspect the hardware first. > >> You can try to reproduce the problem with some additional checks enabled in the >> kernel. Those should catch the problem earlier and thus make its source clearer. > >> I recommend the following: >> options INVARIANTS >> options INVARIANT_SUPPORT >> options WITNESS >> options DEBUG_MEMGUARD >> makeoptions DEBUG+="-DDEBUG" > >> The last is really needed only for the ZFS and OpenSolaris compat code. It make >> result in some extra noise from unrelated subsystems. >> Perhaps you could just add "#define DEBUG" to >> sys/cddl/contrib/opensolaris/uts/common/sys/debug.h. I haven't tested this >> approach though. > >> Also, please put vm.memguard.desc="arc_buf_hdr_t" into loader.conf. > >> Please note that these options will make your system significantly slower. > > I recompiled the kernel and is running with options you specified (I > enabled DEBUG in the file). > > Anyway even at boot time I started getting following warnings, is this > anything: These witness warning are OK-ish. Watch for panics. BTW, I should have said this earlier. Whatever the kind of the corruptions it would be much worse if a corruption would get propagated to the stable storage. Especially if it would be in any kind of pool metadata. So, your data is at great risk now. Please also take measures to back it up. Preferably by using a different system. > Dec 24 16:06:03 chinatsu kernel: Creating and/or trimming log files > Dec 24 16:06:03 chinatsu kernel: lock order reversal: > Dec 24 16:06:03 chinatsu kernel: 1st 0xffffffff80bf5780 pf task mtx (pf task mtx) @ /usr/src/sys/contrib/pf/net/pf.c:3330 > Dec 24 16:06:03 chinatsu kernel: . > Dec 24 16:06:03 chinatsu kernel: 2nd 0xfffffe0009211af8 radix node head (radix node head) @ /usr/src/sys/net/route.c:384 > Dec 24 16:06:03 chinatsu kernel: KDB: stack backtrace: > Dec 24 16:06:03 chinatsu kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > Dec 24 16:06:03 chinatsu kernel: kdb_backtrace() at kdb_backtrace+0x37 > Dec 24 16:06:03 chinatsu kernel: _witness_debugger() at _witness_debugger+0x2c > Dec 24 16:06:03 chinatsu kernel: witness_checkorder() at witness_checkorder+0x844 > Dec 24 16:06:03 chinatsu kernel: _rw_rlock() at > Dec 24 16:06:03 chinatsu kernel: Starting syslogd. > Dec 24 16:06:03 chinatsu kernel: _rw_rlock+0x81 > Dec 24 16:06:03 chinatsu kernel: rtalloc1_fib() at rtalloc1_fib+0x11c > Dec 24 16:06:03 chinatsu kernel: rtalloc_ign_fib() at rtalloc_ign_fib+0xc5 > Dec 24 16:06:03 chinatsu kernel: pf_routable() at pf_routable+0x1fd > Dec 24 16:06:03 chinatsu kernel: pf_test_rule() at pf_test_rule+0x6cf > Dec 24 16:06:03 chinatsu kernel: pf_test() at pf_test+0xf58 > Dec 24 16:06:03 chinatsu kernel: pf_check_in() at pf_check_in+0x2b > Dec 24 16:06:03 chinatsu kernel: pfil_run_hooks() at pfil_run_hooks+0xd2 > Dec 24 16:06:03 chinatsu kernel: ip_input() at ip_input+0x2dc > Dec 24 16:06:03 chinatsu kernel: netisr_dispatch_src() at netisr_dispatch_src+0x170 > Dec 24 16:06:03 chinatsu kernel: ether_demux() at ether_demux+0x17d > Dec 24 16:06:03 chinatsu kernel: ether_nh_input() at ether_nh_input+0x209 > Dec 24 16:06:03 chinatsu kernel: netisr_dispatch_src() at netisr_dispatch_src+0x170 > Dec 24 16:06:03 chinatsu kernel: alc_int_task() at alc_int_task+0x2ff > Dec 24 16:06:03 chinatsu kernel: taskqueue_run_locked() at taskqueue_run_locked+0x93 > Dec 24 16:06:03 chinatsu kernel: taskqueue_thread_loop() at taskqueue_thread_loop+0x3e > Dec 24 16:06:03 chinatsu kernel: fork_exit() at fork_exit+0x133 > Dec 24 16:06:03 chinatsu kernel: fork_trampoline() at fork_trampoline+0xe > Dec 24 16:06:03 chinatsu kernel: --- trap 0, rip = 0, rsp = 0xffffff85fb2ebbb0, rbp = 0 --- > Dec 24 16:06:03 chinatsu kernel: No core dumps found. > Dec 24 16:06:04 chinatsu kernel: lock order reversal: > Dec 24 16:06:04 chinatsu kernel: 1st 0xffffff85b9cb8dd8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2677 > Dec 24 16:06:04 chinatsu kernel: 2nd 0xfffffe00092c5c00 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284 > Dec 24 16:06:04 chinatsu kernel: KDB: stack backtrace: > Dec 24 16:06:04 chinatsu kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > Dec 24 16:06:04 chinatsu kernel: kdb_backtrace() at kdb_backtrace+0x37 > Dec 24 16:06:04 chinatsu kernel: _witness_debugger() at _witness_debugger+0x2c > Dec 24 16:06:04 chinatsu kernel: witness_checkorder() at witness_checkorder+0x844 > Dec 24 16:06:04 chinatsu kernel: _sx_xlock() at _sx_xlock+0x61 > Dec 24 16:06:04 chinatsu kernel: ufsdirhash_acquire() at ufsdirhash_acquire+0x33 > Dec 24 16:06:04 chinatsu kernel: ufsdirhash_remove() at > Dec 24 16:06:04 chinatsu kernel: ufsdirhash_remove+0x16 > Dec 24 16:06:04 chinatsu kernel: ufs_dirremove() at ufs_dirremove+0x1bb > Dec 24 16:06:04 chinatsu kernel: ufs_remove() at ufs_remove+0x92 > Dec 24 16:06:04 chinatsu kernel: VOP_REMOVE_APV() at VOP_REMOVE_APV+0xb7 > Dec 24 16:06:04 chinatsu kernel: kern_unlinkat() at kern_unlinkat+0x2eb > Dec 24 16:06:04 chinatsu kernel: amd64_syscall() at amd64_syscall+0x30e > Dec 24 16:06:04 chinatsu kernel: Xfast_syscall() at Xfast_syscall+0xf7 > Dec 24 16:06:04 chinatsu kernel: --- syscall (10, FreeBSD ELF64, sys_unlink), rip = 0x80090a22c, rsp = 0x7fffffff > Dec 24 16:06:04 chinatsu kernel: ca88, rbp = 0x7fffffffdf20 --- > Dec 24 16:06:04 chinatsu kernel: lock order reversal: > Dec 24 16:06:04 chinatsu kernel: 1st 0xfffffe00266ddbd8 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:849 > Dec 24 16:06:04 chinatsu kernel: 2nd 0xfffffe002679a818 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2158 > Dec 24 16:06:04 chinatsu kernel: KDB: stack backtrace: > Dec 24 16:06:04 chinatsu kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > Dec 24 16:06:04 chinatsu kernel: kdb_backtrace() at kdb_backtrace+0x37 > Dec 24 16:06:04 chinatsu kernel: _witness_debugger() at _witness_debugger+0x2c > Dec 24 16:06:04 chinatsu kernel: witness_checkorder() at witness_checkorder+0x844 > Dec 24 16:06:04 chinatsu kernel: __lockmgr_args() at __lockmgr_args+0x10d9 > Dec 24 16:06:04 chinatsu kernel: vop_stdlock() at vop_stdlock+0x39 > Dec 24 16:06:04 chinatsu kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0xbf > Dec 24 16:06:04 chinatsu kernel: _vn_lock() at _vn_lock+0x47 > Dec 24 16:06:04 chinatsu kernel: vget() at vget+0x7b > Dec 24 16:06:04 chinatsu kernel: devfs_allocv() at devfs_allocv+0x13f > Dec 24 16:06:04 chinatsu kernel: devfs_root() at devfs_root+0x4d > Dec 24 16:06:04 chinatsu kernel: vfs_donmount() at vfs_donmount+0xafa > Dec 24 16:06:04 chinatsu kernel: sys_nmount() at sys_nmount+0x66 > Dec 24 16:06:04 chinatsu kernel: amd64_syscall() at amd64_syscall+0x30e > Dec 24 16:06:04 chinatsu kernel: Xfast_syscall() at Xfast_syscall+0xf7 > Dec 24 16:06:04 chinatsu kernel: --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x800a8d71c, rsp = 0x7fffffffccc8, rbp = 0x801009048 --- > Dec 24 16:06:05 chinatsu named[1387]: starting BIND 9.8.3-P4 -t /var/named -u bind > Dec 24 16:06:05 chinatsu kernel: Starting named. > > > -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Tue Dec 25 15:07:17 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8E59CDB5; Tue, 25 Dec 2012 15:07:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id E79418FC15; Tue, 25 Dec 2012 15:07:16 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.5/8.14.5) with ESMTP id qBPF7CO2082709; Tue, 25 Dec 2012 17:07:12 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.3 kib.kiev.ua qBPF7CO2082709 Received: (from kostik@localhost) by tom.home (8.14.5/8.14.5/Submit) id qBPF7CTr082708; Tue, 25 Dec 2012 17:07:12 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 25 Dec 2012 17:07:12 +0200 From: Konstantin Belousov To: Patrick Dung Subject: Re: About QUOTA support in stock kernel (resent) Message-ID: <20121225150712.GA82219@kib.kiev.ua> References: <1356111338.1283.YahooMailClassic@web190806.mail.sg3.yahoo.com> <1356442470.26290.YahooMailClassic@web190804.mail.sg3.yahoo.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="CE+1k2dSO48ffgeK" Content-Disposition: inline In-Reply-To: <1356442470.26290.YahooMailClassic@web190804.mail.sg3.yahoo.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd hackers , fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Dec 2012 15:07:17 -0000 --CE+1k2dSO48ffgeK Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Dec 25, 2012 at 09:34:30PM +0800, Patrick Dung wrote: > Hi, >=20 > I would like to know why quota is not enabled in the stock kernel.. >=20 > I remembered that it is not enabled since freebsd 3.5 or freebsd 4 genera= tion. > Now in freebsd 9.0, it still neeed a kernel rebuild. >=20 > I have heard it has performance issue (GIANT lock) about quota. Enabling quota by default would cause small overhead, like one mutex acquir= e, for each inode and block alloc/dealloc, even for mount without quotas enabl= ed. Might be, it is reasonable to just enable it now. Unless somebody provide valid objections and I do not forget, I will do it in a week for HEAD. --CE+1k2dSO48ffgeK Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ2cEfAAoJEJDCuSvBvK1BczoQAJn1t9gpjqh9fckBwd+wMHr9 TeaTYybuGhhY19nTkmodheyAJsRphKhXVw4zDKYTCkO21HE9HbAx4X+NyLaBB+IN mQNhaP7r//TmuCUR0JSi7Wzincrngm6bT4DjGNTxFi5QbsebB4M848LhBe1HJJBH Ojdt4yO8kMYhRLHqeAydlsnwarvSWfD+fVcaOKW/BmekXvz9osLHnkhd2AwxQsUM YLX+4dB7Gao3WcuUWIr5alGaE3Lo5Ehua8ndI+Fl5GWABpq2Q6V2GnFssvCZONo+ ya0l/Sc8iQTprFE3XK1+4+Iu7Pr9PqWhRxwMGOC4ewHjrZH7CVpriamawqfMbB3v VvaPPf0ICrwmA32hR2Wmvv6l+I+T+iXtMocdhkxYBqU202JIgSlbykNRyYg2ztJ8 DSjEijvzSrX4litK8IMHIzpl6PnQ6jmKtlRTEUTtxdaNxpWmdELN6/TqV0o9ekvU u8WuOfVclmIj9RNjwJYGaNcYh9nmEFvD4FOSU4Ac67knO0mi7lMtsL22vPN8F4vQ WGYw1d2u7AYmInPR8AJkvbc/Phv0ux8oECHWKCcSWsfAmBXyUjpIe/zXEQznKSl7 04xMt2/btNzyktqoe70j1HSZyc5xcQ/KjaEvjE2FFJzoe8OYiBFirZ2JqUh80U90 +5TDEVwdRF47jrAq27bY =AZmI -----END PGP SIGNATURE----- --CE+1k2dSO48ffgeK-- From owner-freebsd-fs@FreeBSD.ORG Tue Dec 25 15:23:58 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4B33CFC5 for ; Tue, 25 Dec 2012 15:23:58 +0000 (UTC) (envelope-from lists@eitanadler.com) Received: from mail-la0-f52.google.com (mail-la0-f52.google.com [209.85.215.52]) by mx1.freebsd.org (Postfix) with ESMTP id B26348FC14 for ; Tue, 25 Dec 2012 15:23:57 +0000 (UTC) Received: by mail-la0-f52.google.com with SMTP id l5so9538060lah.25 for ; Tue, 25 Dec 2012 07:23:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eitanadler.com; s=0xdeadbeef; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=ZDbvvS6JVUEXBhm/0h8gXb3SwOq82qu7T8chPAQfm4o=; b=IqMbHPxSEKpmyeb0tGYTmCKeM8ZP9J8dy5gz/5kHe/sfNhQ1GBuQ6FoinlGxA9HXvU wFfwct8NhZ545j+NYrs3kVzcNDsy1A62kry9YzZh6hwaV8TodzoNB4MLgiPRjQbHnuoc UapSSyrFx5nwY6pABibGEmwRAg1vZwpxiDFgs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:x-gm-message-state; bh=ZDbvvS6JVUEXBhm/0h8gXb3SwOq82qu7T8chPAQfm4o=; b=ETlH7KuEZsEXbnajyGR4osVUda4k/RmcyGdHbPpB3LQbfEpCQrIKnHGCNhPI8dH6kh +ftAaWXikQBaadY7ex+QUCNCQ6niOdGe1M4i3zMGO/cgRF3rikO22++owqUkXwuV6M9O sDzPFZdmSsBTPDtSlDFpzeMy7jmQWB8Q0UvbqpbGCUPDyUZxJuAHnJ6qpSvLGjfUEU+e f/qPNZesLLGzPQKcLahEoZUwf4fC6F3IK+cc7b6bjVTSdW506jj+cQX4eQ0lEEjv/FeL dzPcVUDELJrVylWIPg/Zu45t0R0Nh6z+Tpy6ZE0D6aIK9qow/vPd8LCo7u3Sg6naWVgG 1fuw== Received: by 10.112.29.104 with SMTP id j8mr10045772lbh.0.1356449036254; Tue, 25 Dec 2012 07:23:56 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.162.100 with HTTP; Tue, 25 Dec 2012 07:23:26 -0800 (PST) In-Reply-To: <20121225150712.GA82219@kib.kiev.ua> References: <1356111338.1283.YahooMailClassic@web190806.mail.sg3.yahoo.com> <1356442470.26290.YahooMailClassic@web190804.mail.sg3.yahoo.com> <20121225150712.GA82219@kib.kiev.ua> From: Eitan Adler Date: Tue, 25 Dec 2012 10:23:26 -0500 Message-ID: Subject: Re: About QUOTA support in stock kernel (resent) To: Konstantin Belousov Content-Type: text/plain; charset=UTF-8 X-Gm-Message-State: ALoCoQmkJP0nPPbCgR8LGU5UEdRnH/pkCObQnDYBh3xbUqm0Vza7ejs1qe6jN1r9poVh7aZJYIiV Cc: Patrick Dung , freebsd hackers , fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Dec 2012 15:23:58 -0000 On 25 December 2012 10:07, Konstantin Belousov wrote: > Enabling quota by default would cause small overhead, like one mutex acquire, > for each inode and block alloc/dealloc, even for mount without quotas enabled. Why is this, and can it be avoided (for mounts without quotas)? > Might be, it is reasonable to just enable it now. Unless somebody provide > valid objections and I do not forget, I will do it in a week for HEAD. -- Eitan Adler From owner-freebsd-fs@FreeBSD.ORG Tue Dec 25 15:29:14 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 262861D9; Tue, 25 Dec 2012 15:29:14 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 8039F8FC12; Tue, 25 Dec 2012 15:29:13 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.5/8.14.5) with ESMTP id qBPFT6oI084650; Tue, 25 Dec 2012 17:29:06 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.3 kib.kiev.ua qBPFT6oI084650 Received: (from kostik@localhost) by tom.home (8.14.5/8.14.5/Submit) id qBPFT67S084649; Tue, 25 Dec 2012 17:29:06 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 25 Dec 2012 17:29:06 +0200 From: Konstantin Belousov To: Eitan Adler Subject: Re: About QUOTA support in stock kernel (resent) Message-ID: <20121225152905.GB82219@kib.kiev.ua> References: <1356111338.1283.YahooMailClassic@web190806.mail.sg3.yahoo.com> <1356442470.26290.YahooMailClassic@web190804.mail.sg3.yahoo.com> <20121225150712.GA82219@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="XF85m9dhOBO43t/C" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: Patrick Dung , freebsd hackers , fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Dec 2012 15:29:14 -0000 --XF85m9dhOBO43t/C Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Dec 25, 2012 at 10:23:26AM -0500, Eitan Adler wrote: > On 25 December 2012 10:07, Konstantin Belousov wrot= e: > > Enabling quota by default would cause small overhead, like one mutex ac= quire, > > for each inode and block alloc/dealloc, even for mount without quotas e= nabled. >=20 > Why is this, and can it be avoided (for mounts without quotas)? Because system should check whether quota is enabled to do the accounting. >=20 > > Might be, it is reasonable to just enable it now. Unless somebody provi= de > > valid objections and I do not forget, I will do it in a week for HEAD. >=20 >=20 >=20 > --=20 > Eitan Adler --XF85m9dhOBO43t/C Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ2cY/AAoJEJDCuSvBvK1BfzEP/3ze3Wcu1n28ZIeCaXCEeDXk qnJmZXLuw4y57CZJ2+mASDGTCUkp91iJcCtPAP+UyN91LIqAGgQVkI71L70dCZ3c OO+eAxgCYOCB4OqvjwDhI4cXuTyarf2blIjov0qxjwh9vqp9Zy7hL2FtPttpxmPr FOukKMZ9wMzM3MmPoASULdAgu4jZQQHV7Hhw9H6pujRxgXGPOHHbDoIkT0WHbq8K q6nR9OUjPKHx2hm4/mLQTGTlUqyF8V883IWI7zOoiO2G9S7rpOT7FdTBb7CljLub EUUMKGf9RcheduIew60ZafSXUwbRvD1AHIktlm1xTuXG4ZSyyRluyYN+QmUoQ16Q yh/BA0LzxwrKo3wx9cFUVw5FCHjPRTIKUv91InnxX7TQeRGAxkjNi35LtjzE6nS8 p90NS6v9OCEWNHc++Ece/FhCI5mK80bwzXtsULioLAeBDEzc0d/5vMtntx4wE4pg IYsGHjq0ZJjWjlr4eCf3CUUsAIqkE0dV43SEevV2UTHmXe+VD+v4serfn89AddYP oSb3JyJAJ2eoi81TEH1y6WWEnIBKquT+LvaKOVbBPP/JOSIPRzYOvqwTl9IHC5Vb hTkcXR1JxtZRoPpgvgSlo0F3slLbVnobNex9QWLceI8JSMXexrPJxJgJqT76lxSh VklzlLPHVKpCVEgbP29Y =+ine -----END PGP SIGNATURE----- --XF85m9dhOBO43t/C-- From owner-freebsd-fs@FreeBSD.ORG Tue Dec 25 16:49:18 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D0AD3D55 for ; Tue, 25 Dec 2012 16:49:18 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [IPv6:2001:5a8:4:7e72:4a5b:39ff:fe12:452]) by mx1.freebsd.org (Postfix) with ESMTP id AA05E8FC0A for ; Tue, 25 Dec 2012 16:49:18 +0000 (UTC) Received: from chez.mckusick.com (localhost [127.0.0.1]) by chez.mckusick.com (8.14.3/8.14.3) with ESMTP id qBPGnEwl064636; Tue, 25 Dec 2012 08:49:14 -0800 (PST) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201212251649.qBPGnEwl064636@chez.mckusick.com> To: Konstantin Belousov Subject: Re: About QUOTA support in stock kernel (resent) In-reply-to: <20121225152905.GB82219@kib.kiev.ua> Date: Tue, 25 Dec 2012 08:49:14 -0800 From: Kirk McKusick X-Spam-Status: No, score=0.0 required=5.0 tests=MISSING_MID, UNPARSEABLE_RELAY autolearn=failed version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on chez.mckusick.com Cc: Patrick Dung , fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Dec 2012 16:49:18 -0000 I agree with your decision to include quotas by default. When I put them into the system back in the mid-1980's we had two 1 MIP machines with 4Mb of memory that supported the entire Computer Science department at Berkeley. Machine cycles and memory were at a premium and *any* overhead had to be justified. So I made sure that quotas were as lean and light weight as I could. In this day-and-age of huge memories and blindingly fast CPUs including quotas seems like a good idea. Not a lot of folks use them, but those that do would like to have them available. And for those that want a tiny footprint (embedded), they can easily strip them out. Kirk McKusick From owner-freebsd-fs@FreeBSD.ORG Tue Dec 25 21:16:06 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EEF94383; Tue, 25 Dec 2012 21:16:06 +0000 (UTC) (envelope-from takeda@takeda.tk) Received: from chinatsu.takeda.tk (mail.takeda.tk [74.0.89.210]) by mx1.freebsd.org (Postfix) with ESMTP id BD39D8FC12; Tue, 25 Dec 2012 21:16:06 +0000 (UTC) Received: from localhost.takeda.tk (takeda-ws2.lan [10.0.0.123]) (authenticated bits=0) by chinatsu.takeda.tk (8.14.5/8.14.5) with ESMTP id qBPLG58K022933 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO); Tue, 25 Dec 2012 13:16:05 -0800 (PST) (envelope-from takeda@takeda.tk) Date: Tue, 25 Dec 2012 13:06:35 -0800 From: Derek Kulinski X-Priority: 3 (Normal) Message-ID: <1501851114.20121225130635@takeda.tk> To: Andriy Gapon Subject: Re: FreeBSD 9.1-RELEASE crashes almost daily; backtraces always list zfs routines In-Reply-To: <50D9614F.6040306@FreeBSD.org> References: <1824023197.20121223142308@takeda.tk> <50D87C56.70709@FreeBSD.org> <331959998.20121224101719@takeda.tk> <50D8E500.1070408@FreeBSD.org> <574019558.20121224161156@takeda.tk> <50D9614F.6040306@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.97.6 at chinatsu.takeda.tk X-Virus-Status: Clean Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Dec 2012 21:16:07 -0000 Hello Andriy, Tuesday, December 25, 2012, 12:18:23 AM, you wrote: >> I recompiled the kernel and is running with options you specified (I >> enabled DEBUG in the file). >> >> Anyway even at boot time I started getting following warnings, is this >> anything: > These witness warning are OK-ish. > Watch for panics. Sadly (I guess :) I did not have crash today (I don't see even a warning at that time), I'll update when it happens. Will also try to run that task today and will see if it will do anything. > BTW, I should have said this earlier. Whatever the kind of the corruptions it > would be much worse if a corruption would get propagated to the stable storage. > Especially if it would be in any kind of pool metadata. > So, your data is at great risk now. > Please also take measures to back it up. Preferably by using a different system. I'm backing it up using zfs send since dump doesn't appear to work on ZFS. Anyway zpool scrub does not show any problems... I performed my last scan when I started this thread. -- Best regards, Derek mailto:takeda@takeda.tk Hand over the calculator, friends don't let friends derive drunk From owner-freebsd-fs@FreeBSD.ORG Wed Dec 26 07:34:26 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 53C465F1 for ; Wed, 26 Dec 2012 07:34:26 +0000 (UTC) (envelope-from patrick_dkt@yahoo.com.hk) Received: from nm9-vm7.bullet.mail.sg3.yahoo.com (nm9-vm7.bullet.mail.sg3.yahoo.com [106.10.148.214]) by mx1.freebsd.org (Postfix) with ESMTP id 0E8E38FC0A for ; Wed, 26 Dec 2012 07:34:24 +0000 (UTC) Received: from [106.10.166.117] by nm9.bullet.mail.sg3.yahoo.com with NNFMP; 26 Dec 2012 07:32:04 -0000 Received: from [106.10.151.186] by tm6.bullet.mail.sg3.yahoo.com with NNFMP; 26 Dec 2012 07:32:04 -0000 Received: from [127.0.0.1] by omp1012.mail.sg3.yahoo.com with NNFMP; 26 Dec 2012 07:32:04 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 727286.1437.bm@omp1012.mail.sg3.yahoo.com Received: (qmail 60558 invoked by uid 60001); 26 Dec 2012 07:32:04 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.hk; s=s1024; t=1356507124; bh=aCdyeRg5Gg0hiPcWxxqC2tirkY1XePozzG+xaS/gi/8=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=NNw6mUwYdLtS9pXhsmo2JIgQNwR/ecaQW5N5GHyx0fMmuOWMKMo2HN8arVJw2ZYudQ86FXZ9SCulFVQjLzswVWB63PqsjiJKxkqDqsh56KBvdTk8FtyEW0pFkvzfvFYZg9uVvCKT1mlDB38v+fOP/fboBsyI3Xo/rg3CGEnV8ss= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com.hk; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=VzUaNakN0l9ByUe7q1dVDq1GGCkOGnZ8qvi97btm8fUeoUdWLyVIO9LI0unrLPdxouMD0pJY2ZJVG3S1lytl3vook5NmDnrOkZp1rXh08tjk7VLXFSO1AaQ8Is0rVSuURTqswS20gBbJogReTvkNR7kICYxyyMQIsVpvZmN5G5E=; X-YMail-OSG: Ytgf2XAVM1mJdz.zyZNvcxKs1.7yRF8jAoS.jp4c7uoG0uK mot4EtdXA7vZw0ZXgOBzwVnjQnQhYpmTyygG2DJ4UVpfiie.mxqKy8otqWQA xijduTqulnlPvc.bepVCIeNwsD0D0OUMP4pfLru3_IyeTvTbejr1ZEH30eLy UU2p7eTYQhMe.nFX8K_WpeGbkpUQlw5Hagwg9e4C6MwPlaLeUt_whS1AXLlK aBwcEx57sM89pX5RbPUsH7qfeMiYADgiUMiSimbkY4tZc2.yPQLlgMV3WZtV rod05eRF1oOarRUb92jIRymNyqapkNuIxm_A1oPgYQFSdWF1y1UkH3uRQMeX uFtRZXJ9xfgOKhkhrdmF8NvrC2XJbKvGC2TvtuoOJu1lPNGngnJHtWPoDfcl cekaeBUptTy1T2xlirTdrTCCd.Erh5JpGralfthPwrlYkCbNkOndZQnsRdfZ l9Vom8ISD5ENV7GdC9a2M3z2wKwvso2bsZZePeMW5p6O6ehnKYQUj Received: from [61.15.240.116] by web190803.mail.sg3.yahoo.com via HTTP; Wed, 26 Dec 2012 15:32:04 SGT X-Rocket-MIMEInfo: 001.001, SSBhbSBjdXJpb3VzIGlmIG90aGVyIG9wZXJhdGluZyBzeXN0ZW1zIGhhdmUgdGhpcyBwZXJmb3JtYW5jZSBpbXBhY3QuDQoNCkNvdWxkIHdlIGhhdmUgc29tZSB3b3JrYXJvdW5kIG9yIG5lZWQgc29tZSBjb2RlIGltcHJvdmVtZW50Pw0KRm9yIGV4YW1wbGU6DQpEbyB0aGUgY2hlY2tpbmcvYWNjb3VudGluZyBvbmx5IGlmIHRoZSBzcGVjaWZpYyBtb3VudCBwb2ludCBoYXMgZW5hYmxlZCBxdW90YS4NCmV0Yy4uDQoNClJlZ2FyZHMsDQpQYXRyaWNrDQoNCi0tLSBPbiBUdWUsIDEyLzI1LzEyLCBLb25zdGFudGkBMAEBAQE- X-Mailer: YahooMailClassic/15.1.2 YahooMailWebService/0.8.129.483 Message-ID: <1356507124.60133.YahooMailClassic@web190803.mail.sg3.yahoo.com> Date: Wed, 26 Dec 2012 15:32:04 +0800 (SGT) From: Patrick Dung Subject: Re: About QUOTA support in stock kernel (resent) To: Eitan Adler , Konstantin Belousov In-Reply-To: <20121225152905.GB82219@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd hackers , fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Dec 2012 07:34:26 -0000 I am curious if other operating systems have this performance impact. Could we have some workaround or need some code improvement? For example: Do the checking/accounting only if the specific mount point has enabled quota. etc.. Regards, Patrick --- On Tue, 12/25/12, Konstantin Belousov wrote: From: Konstantin Belousov Subject: Re: About QUOTA support in stock kernel (resent) To: "Eitan Adler" Cc: "Patrick Dung" , "freebsd hackers" , fs@freebsd.org Date: Tuesday, December 25, 2012, 11:29 PM On Tue, Dec 25, 2012 at 10:23:26AM -0500, Eitan Adler wrote: > On 25 December 2012 10:07, Konstantin Belousov wrote: > > Enabling quota by default would cause small overhead, like one mutex acquire, > > for each inode and block alloc/dealloc, even for mount without quotas enabled. > > Why is this, and can it be avoided (for mounts without quotas)? Because system should check whether quota is enabled to do the accounting. > > > Might be, it is reasonable to just enable it now. Unless somebody provide > > valid objections and I do not forget, I will do it in a week for HEAD. > > > > -- > Eitan Adler From owner-freebsd-fs@FreeBSD.ORG Thu Dec 27 19:41:50 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D826E9A9; Thu, 27 Dec 2012 19:41:50 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 47AF48FC08; Thu, 27 Dec 2012 19:41:50 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.5/8.14.5) with ESMTP id qBRJfjEG051356; Thu, 27 Dec 2012 21:41:45 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.3 kib.kiev.ua qBRJfjEG051356 Received: (from kostik@localhost) by tom.home (8.14.5/8.14.5/Submit) id qBRJfjwi051355; Thu, 27 Dec 2012 21:41:45 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 27 Dec 2012 21:41:45 +0200 From: Konstantin Belousov To: Andreas Longwitz Subject: Re: FS hang with suspfs when creating snapshot on a UFS + GJOURNAL setup Message-ID: <20121227194145.GM82219@kib.kiev.ua> References: <50DC30F6.1050904@incore.de> <20121227133355.GI82219@kib.kiev.ua> <50DC8999.8000708@incore.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="zHDeOHGDnzKksZSU" Content-Disposition: inline In-Reply-To: <50DC8999.8000708@incore.de> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-stable@freebsd.org, fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2012 19:41:50 -0000 --zHDeOHGDnzKksZSU Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Dec 27, 2012 at 06:47:05PM +0100, Andreas Longwitz wrote: > Konstantin Belousov wrote: > > On Thu, Dec 27, 2012 at 12:28:54PM +0100, Andreas Longwitz wrote: > >> On a FreeBSD 8-Stable machine with UFS + GJOURNAL (no SU) I observed t= he > >> same behaviour as described for UFS+SU+J in > >> lists.freebsd.org/pipermail/freebsd-current/2012-January/030937.html. > >> > >> The snapshot was initiated by amanda with > >> dump 3ubLshf 64 1048576 0 - /dev/mirror/gm0p10.journal (pid 50347) > >> and the process creating the snapshot is > >> /sbin/mksnap_ffs /home /home/.snap/dump_snapshot (pid 50350). > >> > >> The process 50350 hangs and also all following processes that tried to > >> access the home partition, some of them work, but don't come to an end, > >> e.g. a shell after (Ctrl T): > >> load: 0.61 cmd: sh 52670 [suspfs] 43.46r 0.00u 0.00s 0% 2272k. > >> > >> All write I/O's to all gjournaled partitions are stopped. Under normal > >> circumstances the snapshot is taken in five seconds, so I have definit= iv > >> not the problem I have described in > >> lists.freebsd.org/pipermail/freebsd-geom/2012-May/005246.html. > >> > >> ..... > >> > >> It seems there is a deadlock on the suspfs lock, but I could not figure > >> out who holds this lock. > >> Any hints how to get better diagnostic information for next time the > >> error occurs are welcome. > >=20 > > The > > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/ke= rneldebug-deadlocks.html > > provides the instructions. > >=20 > > The suspfs is owned by the snapshot creator. The question is, where is = it > > blocked. >=20 > Thanks for answer. >=20 > In the meantime I can reproduce the problem and got some more > information. It looks that there is a deadlock between the two processes > with pid 18 (g_journal switcher) and pid 7126 (/sbin/mksnap_ffs /home > /home/.snap/my_snapshot): >=20 > 0 18 0 0 45 0 0 16 snaplk DL ?? 4:40,34 > [g_journal switcher] > 0 7126 1933 0 50 0 5836 1176 suspfs D ?? 0:00,44 > /sbin/mksnap_ffs /home /home/.snap/my_snapshot >=20 > procstat -t 18 --> > PID TID COMM TDNAME CPU PRI STATE WCHAN > 18 100076 g_journal switcher g_journal switch 0 129 sleep snaplk > procstat -t 7126 --> > PID TID COMM TDNAME CPU PRI STATE WCHAN > 7126 100157 mksnap_ffs - 1 134 sleep suspfs > procstat -kk 18 --> > PID TID COMM TDNAME KSTACK > 18 100076 g_journal switcher g_journal switch mi_switch+0x186 > sleepq_wait+0x42 __lockmgr_args+0x49b ffs_copyonwrite > +0x19a ffs_geom_strategy+0x1b5 bufwrite+0xe9 ffs_sbupdate+0x12a > g_journal_ufs_clean+0x3e g_journal_switcher+0xe5e fork > _exit+0x11f fork_trampoline+0xe > procstat -kk 7126 --> > PID TID COMM TDNAME KSTACK > 7126 100157 mksnap_ffs - mi_switch+0x186 > sleepq_wait+0x42 _sleep+0x373 vn_start_write+0xdf ffs_s > napshot+0xe2b ffs_mount+0x65a vfs_donmount+0xdc5 nmount+0x63 > amd64_syscall+0x1f4 Xfast_syscall+0xfc >=20 > >From DDB: > db> show lockedvnods > Locked vnodes >=20 > 0xffffff012281a938: tag ufs, type VREG > usecount 1, writecount 0, refcount 3339 mountedhere 0 > flags (VV_SYSTEM) > lock type snaplk: EXCL by thread 0xffffff000807a470 (pid 7126) > with exclusive waiters pending > ino 23552, on dev mirror/gm0p10.journal =2E.. > db> alltrace (pid 18 and 7126) >=20 > Tracing command g_journal switcher pid 18 tid 100076 td 0xffffff0002bd5000 > sched_switch() at sched_switch+0xde > mi_switch() at mi_switch+0x186 > sleepq_wait() at sleepq_wait+0x42 > __lockmgr_args() at __lockmgr_args+0x49b > ffs_copyonwrite() at ffs_copyonwrite+0x19a > ffs_geom_strategy() at ffs_geom_strategy+0x1b5 > bufwrite() at bufwrite+0xe9 > ffs_sbupdate() at ffs_sbupdate+0x12a > g_journal_ufs_clean() at g_journal_ufs_clean+0x3e > g_journal_switcher() at g_journal_switcher+0xe5e > fork_exit() at fork_exit+0x11f > fork_trampoline() at fork_trampoline+0xe > --- trap 0, rip =3D 0, rsp =3D 0xffffff8242ca8cf0, rbp =3D 0 --- >=20 > Tracing command mksnap_ffs pid 7126 tid 100157 td 0xffffff000807a470 > sched_switch() at sched_switch+0xde > mi_switch() at mi_switch+0x186 > sleepq_wait() at sleepq_wait+0x42 > _sleep() at _sleep+0x373 > vn_start_write() at vn_start_write+0xdf > ffs_snapshot() at ffs_snapshot+0xe2b Can you look up the line number for the ffs_snapshot+0xe2b ? I think the bug is that vn_start_write() is called while the snaplock is owned, after the out1 label in ffs_snapshot() (I am looking at the HEAD code). > ffs_mount() at ffs_mount+0x65a > vfs_donmount() at vfs_donmount+0xdc5 > nmount() at nmount+0x63 > amd64_syscall() at amd64_syscall+0x1f4 > Xfast_syscall() at Xfast_syscall+0xfc > --- syscall (378, FreeBSD ELF64, nmount), rip =3D 0x18069e35c, rsp =3D > 0x7fffffffe358, rbp =3D 0x7fffffffedc7 --- >=20 > There are a lot of other - but later started than pid 7126 - processes > sitting on the suspfs lock, most of them hang on a close with a stack > like this: >=20 > Tracing command cat pid 17726 tid 100387 td 0xffffff012e4bd470 > sched_switch() at sched_switch+0xde > mi_switch() at mi_switch+0x186 > sleepq_wait() at sleepq_wait+0x42 > _sleep() at _sleep+0x373 > vn_start_write() at vn_start_write+0xdf > vn_close() at vn_close+0x5b > vn_closefile() at vn_closefile+0x5a > _fdrop() at _fdrop+0x23 > closef() at closef+0x45 > kern_close() at kern_close+0x163 > amd64_syscall() at amd64_syscall+0x1f4 > Xfast_syscall() at Xfast_syscall+0xfc > --- syscall (6, FreeBSD ELF64, close), rip =3D 0x18073f07c, rsp =3D > 0x7fffffffeb68, rbp =3D 0 --- >=20 >=20 > If more information is necessary to catch this problem, I can try to > reproduce the problem with a suitable debug kernel. >=20 > --=20 > Andreas Longwitz --zHDeOHGDnzKksZSU Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ3KR4AAoJEJDCuSvBvK1BOTEP/RIsmiLgE2r+ansViY7TGdv9 a/P0lQK/+a/+TZ2K24h+J5/sqBHFNnBq9mxD+8OwLBdGjYtoKb4uDio2DyS/xjBd r1nP+XZxL5DS9cHTE7gzFJvX4i6T+g77L9PQkWaux1LhY06NW0gW/XytzWUcMl27 gQ66zXgpzvxC6IrQHOm8CN1F94N2wmu6o+3/mpP9XzbvShljkx+t2u7CqZpwQ7US ItqSLMo6oUzfYYcWXskTud2BFbfcDaZXZ4ZwNt5MwmpnGBti+Cxq9D04NDNy5TYa 0zEsELjWv7DHxSOviD2YNdQR1o3dYPtHrWrnnFTMOL2ZFOao/XoDQmT8oR1qbUoB I2BfpPBJ7rb52nS5oota19H3bjYqpTwQU/JFTQfoq9vkrt0WgktDPf+bgJdltw+y Iaeq5osr2pdLjbdTiPve0UGgbFD4ayQVuRUOYJ/2dwC/eSQ+suMVhFgwx37o+TfQ qKms7hpu3IuYKHBA+Von4RBB+g58PKbvEfAI9VY3D6xGj9hZ3R5wrh5GXvW3dI5U 4mBjPFaJXolubmwAgmJWNPcb5mNuJhEj0fhs/9XtN5V+bux6szt3cZdEN68zjA9F k8hkP+FfvOa3VqGCdF6EfjcItcyUXw/l2vYJcmWzcSmP+EO8geyiE/DYtcKANvQd hBoR2RLQOYj541So28Xd =ywrw -----END PGP SIGNATURE----- --zHDeOHGDnzKksZSU-- From owner-freebsd-fs@FreeBSD.ORG Thu Dec 27 20:05:48 2012 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9ED3EFE8; Thu, 27 Dec 2012 20:05:48 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 833DC8FC08; Thu, 27 Dec 2012 20:05:48 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id qBRK5m3H046958; Thu, 27 Dec 2012 20:05:48 GMT (envelope-from rmacklem@freefall.freebsd.org) Received: (from rmacklem@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id qBRK5lvv046954; Thu, 27 Dec 2012 20:05:47 GMT (envelope-from rmacklem) Date: Thu, 27 Dec 2012 20:05:47 GMT Message-Id: <201212272005.qBRK5lvv046954@freefall.freebsd.org> To: tomefrom@list.ru, rmacklem@FreeBSD.org, freebsd-fs@FreeBSD.org From: rmacklem@FreeBSD.org Subject: Re: kern/173657: [nfs] strange UID map with nfsuserd X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2012 20:05:48 -0000 Synopsis: [nfs] strange UID map with nfsuserd State-Changed-From-To: open->feedback State-Changed-By: rmacklem State-Changed-When: Thu Dec 27 20:03:29 UTC 2012 State-Changed-Why: I have emailed the reporter, asking them if they can test the patch: http://people.freebsd.org/~rmacklem/setattr-nfsuserd.patch which I think fixes the reported problem. It basically marks the "default" mappings to nobody for cases where there isn't a valid name<->uid mapping and doesn't allow Setattr to be done using those "default" mappings. http://www.freebsd.org/cgi/query-pr.cgi?pr=173657 From owner-freebsd-fs@FreeBSD.ORG Thu Dec 27 23:02:29 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 081A2C17; Thu, 27 Dec 2012 23:02:29 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 808478FC0A; Thu, 27 Dec 2012 23:02:28 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.5/8.14.5) with ESMTP id qBRN2Oi4070984; Fri, 28 Dec 2012 01:02:24 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.3 kib.kiev.ua qBRN2Oi4070984 Received: (from kostik@localhost) by tom.home (8.14.5/8.14.5/Submit) id qBRN2NG6070983; Fri, 28 Dec 2012 01:02:23 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 28 Dec 2012 01:02:23 +0200 From: Konstantin Belousov To: fs@freebsd.org Subject: Nandfs use of MNT_VNODE_FOREACH Message-ID: <20121227230223.GN82219@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Ll0BBk1HBk/f94B0" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: gber@freebsd.org, cognet@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2012 23:02:29 -0000 --Ll0BBk1HBk/f94B0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I took a look at removing MNT_VNODE_FOREACH interface in the HEAD, and apparently we still have one user of the said interface, probably, due to cross-commits. Namely, nandfs utilizes the interface. First, it is obsoleted and is going to be removed. Second, I believe that MNT_VNODE_FOREACH_ACTIVE would be better choice there, because intent of the loop is to do something with each dirty vnode, and dirty vnode must be active, because there are dirty buffers attached to it. That said, use of vget(LK_RETRY) and immediate check for VI_DOOMED is not useful, I removed the LK_RETRY from the lockflags. Another issue is that intent was to avoid sleep for locked vnode, but the check is racy. I used LK_NOWAIT instead. Check for mnt_syncer is also usually done as vp->v_type =3D=3D VNON, but lets postpone this. Could someone who uses the filesystem test the patch below, please ? diff --git a/sys/fs/nandfs/nandfs_segment.c b/sys/fs/nandfs/nandfs_segment.c index 836bead..a73b7f2 100644 --- a/sys/fs/nandfs/nandfs_segment.c +++ b/sys/fs/nandfs/nandfs_segment.c @@ -481,36 +481,20 @@ nandfs_iterate_dirty_vnodes(struct mount *mp, struct = nandfs_seginfo *seginfo) int error, lockreq, update; =20 td =3D curthread; - lockreq =3D LK_EXCLUSIVE | LK_INTERLOCK | LK_RETRY; + lockreq =3D LK_EXCLUSIVE | LK_INTERLOCK; =20 - MNT_ILOCK(mp); - - MNT_VNODE_FOREACH(vp, mp, mvp) { + MNT_VNODE_FOREACH_ACTIVE(vp, mp, mvp) { update =3D 0; =20 if (mp->mnt_syncer =3D=3D vp) continue; - if (VOP_ISLOCKED(vp)) - continue; - - VI_LOCK(vp); - MNT_IUNLOCK(mp); - if (vp->v_iflag & VI_DOOMED) { + if (VOP_ISLOCKED(vp)) { VI_UNLOCK(vp); - MNT_ILOCK(mp); - continue; - } - - if ((error =3D vget(vp, lockreq, td)) !=3D 0) { - MNT_ILOCK(mp); continue; } =20 - if (vp->v_iflag & VI_DOOMED) { - vput(vp); - MNT_ILOCK(mp); + if (vget(vp, lockreq, td) !=3D 0) continue; - } =20 nandfs_node =3D VTON(vp); if (nandfs_node->nn_flags & IN_MODIFIED) { @@ -532,12 +516,8 @@ nandfs_iterate_dirty_vnodes(struct mount *mp, struct n= andfs_seginfo *seginfo) =20 if (update) nandfs_node_update(nandfs_node); - - MNT_ILOCK(mp); } =20 - MNT_IUNLOCK(mp); - return (0); } =20 --Ll0BBk1HBk/f94B0 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ3NN/AAoJEJDCuSvBvK1BDVAP/32k/OjDCGKRKnNgRSMRl2Y7 Kya0raDE68w545/AmsYFMSbiOGNqhsDL6wyG/scbkBzCcsgf5gyNhehqGds2u/it gV09Eucf0+sUv6Z+yy3414FDpAdDswFVJ8KpK39BQ9Q0zXV+QQzYD+M7ghrX9fQZ l5jLSFCzPngmWkg70ohdHD4xBRnO3vH+q+jW3dAiQ1uzY6mW9WP28qokrxgBDPMm 4ZaTLxcTAzZ0wdxvcE4OUAbu7Mwg2MiSNGp01vC/sCdShCe4sJ3OJ/Vwcgd5bEiv sjbIIc6dE+Tehg4LDNZf3W4VtNqLAYY0s2HDNDIxJf0OrFUN2ZDfLgOy1zupuPZi NpwE4TfRkOXR7DRVPC50eiT+kcqvtYmrVmKvUkn1/xpAkMt+ERZIwVmw5X3r7ZnN 34uZfonn5o8FCVNSJE2rBYRreQUPrb7EDgc64hW5B9aXM0J58xUNvy3PbCaK0Efe uH3UsVMKn05PGIOP54lMhidin4XEm0Bg4WqIyPvBnGzu06hCMnkSszw/3JV07n4Z mC9JClN8nP49OV/4rOhdxD6fByDO32vSfdJ7EJZYjIS73BtC39g+4uAoA6Kf6IgI A10ijTMHA63vP3hHPF9gAb273sYBVI0kQ+S0TtFSbTo75dH58o00R5BTtIq8GqiP qvjmn55mnJVoV22GKdjs =zm+D -----END PGP SIGNATURE----- --Ll0BBk1HBk/f94B0-- From owner-freebsd-fs@FreeBSD.ORG Fri Dec 28 09:19:34 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2DC51803; Fri, 28 Dec 2012 09:19:34 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) by mx1.freebsd.org (Postfix) with ESMTP id D3CAA8FC12; Fri, 28 Dec 2012 09:19:33 +0000 (UTC) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id D880A5C36A; Fri, 28 Dec 2012 10:19:32 +0100 (CET) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id rwPEqDXUl2Da; Fri, 28 Dec 2012 10:19:32 +0100 (CET) Received: from mail.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id 001015C36E; Fri, 28 Dec 2012 10:19:31 +0100 (CET) Received: from bsdlo.incore (bsdlo.incore [192.168.0.84]) by mail.incore (Postfix) with ESMTP id EA0615083F; Fri, 28 Dec 2012 10:19:31 +0100 (CET) Message-ID: <50DD6423.5090305@incore.de> Date: Fri, 28 Dec 2012 10:19:31 +0100 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: FS hang with suspfs when creating snapshot on a UFS + GJOURNAL setup References: <50DC30F6.1050904@incore.de> <20121227133355.GI82219@kib.kiev.ua> <50DC8999.8000708@incore.de> <20121227194145.GM82219@kib.kiev.ua> In-Reply-To: <20121227194145.GM82219@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org, fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2012 09:19:34 -0000 Konstantin Belousov wrote: >>> On Thu, Dec 27, 2012 at 12:28:54PM +0100, Andreas Longwitz wrote: >> db> alltrace (pid 18 and 7126) >> >> Tracing command g_journal switcher pid 18 tid 100076 td 0xffffff0002bd5000 >> sched_switch() at sched_switch+0xde >> mi_switch() at mi_switch+0x186 >> sleepq_wait() at sleepq_wait+0x42 >> __lockmgr_args() at __lockmgr_args+0x49b >> ffs_copyonwrite() at ffs_copyonwrite+0x19a >> ffs_geom_strategy() at ffs_geom_strategy+0x1b5 >> bufwrite() at bufwrite+0xe9 >> ffs_sbupdate() at ffs_sbupdate+0x12a >> g_journal_ufs_clean() at g_journal_ufs_clean+0x3e >> g_journal_switcher() at g_journal_switcher+0xe5e >> fork_exit() at fork_exit+0x11f >> fork_trampoline() at fork_trampoline+0xe >> --- trap 0, rip = 0, rsp = 0xffffff8242ca8cf0, rbp = 0 --- >> >> Tracing command mksnap_ffs pid 7126 tid 100157 td 0xffffff000807a470 >> sched_switch() at sched_switch+0xde >> mi_switch() at mi_switch+0x186 >> sleepq_wait() at sleepq_wait+0x42 >> _sleep() at _sleep+0x373 >> vn_start_write() at vn_start_write+0xdf >> ffs_snapshot() at ffs_snapshot+0xe2b > Can you look up the line number for the ffs_snapshot+0xe2b ? (kgdb) list *ffs_snapshot+0xe2b 0xffffffff8056287b is in ffs_snapshot (/usr/src/sys/ufs/ffs/ffs_snapshot.c:676). 671 /* 672 * Resume operation on filesystem. 673 */ 674 vfs_write_resume(vp->v_mount); 675 vn_start_write(NULL, &wrtmp, V_WAIT); 676 if (collectsnapstats && starttime.tv_sec > 0) { 677 nanotime(&endtime); 678 timespecsub(&endtime, &starttime); 679 printf("%s: suspended %ld.%03ld sec, redo %ld of %d\n", 680 vp->v_mount->mnt_stat.f_mntonname, (long)endtime.tv_sec, > I think the bug is that vn_start_write() is called while the snaplock > is owned, after the out1 label in ffs_snapshot() (I am looking at the > HEAD code). You are right, the vn_start_write() is just after the out1 label. >> ffs_mount() at ffs_mount+0x65a >> vfs_donmount() at vfs_donmount+0xdc5 >> nmount() at nmount+0x63 >> amd64_syscall() at amd64_syscall+0x1f4 >> Xfast_syscall() at Xfast_syscall+0xfc >> --- syscall (378, FreeBSD ELF64, nmount), rip = 0x18069e35c, rsp = >> 0x7fffffffe358, rbp = 0x7fffffffedc7 --- -- Andreas Longwitz From owner-freebsd-fs@FreeBSD.ORG Fri Dec 28 11:27:31 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 07BC8F61; Fri, 28 Dec 2012 11:27:31 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 514388FC0A; Fri, 28 Dec 2012 11:27:30 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.5/8.14.5) with ESMTP id qBSBROYj042785; Fri, 28 Dec 2012 13:27:24 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.7.3 kib.kiev.ua qBSBROYj042785 Received: (from kostik@localhost) by tom.home (8.14.5/8.14.5/Submit) id qBSBRO4f042784; Fri, 28 Dec 2012 13:27:24 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 28 Dec 2012 13:27:24 +0200 From: Konstantin Belousov To: Andreas Longwitz Subject: Re: FS hang with suspfs when creating snapshot on a UFS + GJOURNAL setup Message-ID: <20121228112724.GR82219@kib.kiev.ua> References: <50DC30F6.1050904@incore.de> <20121227133355.GI82219@kib.kiev.ua> <50DC8999.8000708@incore.de> <20121227194145.GM82219@kib.kiev.ua> <50DD6423.5090305@incore.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="r7tUYVWcdYzDJoZW" Content-Disposition: inline In-Reply-To: <50DD6423.5090305@incore.de> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: pho@freebsd.org, freebsd-stable@freebsd.org, fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2012 11:27:31 -0000 --r7tUYVWcdYzDJoZW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Dec 28, 2012 at 10:19:31AM +0100, Andreas Longwitz wrote: > Konstantin Belousov wrote: > >>> On Thu, Dec 27, 2012 at 12:28:54PM +0100, Andreas Longwitz wrote: > >> db> alltrace (pid 18 and 7126) > >> > >> Tracing command g_journal switcher pid 18 tid 100076 td 0xffffff0002bd= 5000 > >> sched_switch() at sched_switch+0xde > >> mi_switch() at mi_switch+0x186 > >> sleepq_wait() at sleepq_wait+0x42 > >> __lockmgr_args() at __lockmgr_args+0x49b > >> ffs_copyonwrite() at ffs_copyonwrite+0x19a > >> ffs_geom_strategy() at ffs_geom_strategy+0x1b5 > >> bufwrite() at bufwrite+0xe9 > >> ffs_sbupdate() at ffs_sbupdate+0x12a > >> g_journal_ufs_clean() at g_journal_ufs_clean+0x3e > >> g_journal_switcher() at g_journal_switcher+0xe5e > >> fork_exit() at fork_exit+0x11f > >> fork_trampoline() at fork_trampoline+0xe > >> --- trap 0, rip =3D 0, rsp =3D 0xffffff8242ca8cf0, rbp =3D 0 --- > >> > >> Tracing command mksnap_ffs pid 7126 tid 100157 td 0xffffff000807a470 > >> sched_switch() at sched_switch+0xde > >> mi_switch() at mi_switch+0x186 > >> sleepq_wait() at sleepq_wait+0x42 > >> _sleep() at _sleep+0x373 > >> vn_start_write() at vn_start_write+0xdf > >> ffs_snapshot() at ffs_snapshot+0xe2b > > Can you look up the line number for the ffs_snapshot+0xe2b ? >=20 > (kgdb) list *ffs_snapshot+0xe2b > 0xffffffff8056287b is in ffs_snapshot > (/usr/src/sys/ufs/ffs/ffs_snapshot.c:676). > 671 /* > 672 * Resume operation on filesystem. > 673 */ > 674 vfs_write_resume(vp->v_mount); > 675 vn_start_write(NULL, &wrtmp, V_WAIT); > 676 if (collectsnapstats && starttime.tv_sec > 0) { > 677 nanotime(&endtime); > 678 timespecsub(&endtime, &starttime); > 679 printf("%s: suspended %ld.%03ld sec, redo %ld of %d\n", > 680 vp->v_mount->mnt_stat.f_mntonname, (long)endtime.tv_sec, >=20 > > I think the bug is that vn_start_write() is called while the snaplock > > is owned, after the out1 label in ffs_snapshot() (I am looking at the > > HEAD code). >=20 > You are right, the vn_start_write() is just after the out1 label. Please try the following patch. It is against HEAD, might need some adjustments for 8. I do the resume and write accounting atomically, not allowing other suspension to intervent between. diff --git a/sys/kern/vfs_vnops.c b/sys/kern/vfs_vnops.c index 3f65b05..cf49ecb 100644 --- a/sys/kern/vfs_vnops.c +++ b/sys/kern/vfs_vnops.c @@ -1434,6 +1434,40 @@ vn_closefile(fp, td) * proceed. If a suspend request is in progress, we wait until the * suspension is over, and then proceed. */ +static int +vn_start_write_locked(struct mount *mp, int flags) +{ + int error; + + mtx_assert(MNT_MTX(mp), MA_OWNED); + error =3D 0; + + /* + * Check on status of suspension. + */ + if ((curthread->td_pflags & TDP_IGNSUSP) =3D=3D 0 || + mp->mnt_susp_owner !=3D curthread) { + while ((mp->mnt_kern_flag & MNTK_SUSPEND) !=3D 0) { + if (flags & V_NOWAIT) { + error =3D EWOULDBLOCK; + goto unlock; + } + error =3D msleep(&mp->mnt_flag, MNT_MTX(mp), + (PUSER - 1) | (flags & PCATCH), "suspfs", 0); + if (error) + goto unlock; + } + } + if (flags & V_XSLEEP) + goto unlock; + mp->mnt_writeopcount++; +unlock: + if (error !=3D 0 || (flags & V_XSLEEP) !=3D 0) + MNT_REL(mp); + MNT_IUNLOCK(mp); + return (error); +} + int vn_start_write(vp, mpp, flags) struct vnode *vp; @@ -1470,30 +1504,7 @@ vn_start_write(vp, mpp, flags) if (vp =3D=3D NULL) MNT_REF(mp); =20 - /* - * Check on status of suspension. - */ - if ((curthread->td_pflags & TDP_IGNSUSP) =3D=3D 0 || - mp->mnt_susp_owner !=3D curthread) { - while ((mp->mnt_kern_flag & MNTK_SUSPEND) !=3D 0) { - if (flags & V_NOWAIT) { - error =3D EWOULDBLOCK; - goto unlock; - } - error =3D msleep(&mp->mnt_flag, MNT_MTX(mp), - (PUSER - 1) | (flags & PCATCH), "suspfs", 0); - if (error) - goto unlock; - } - } - if (flags & V_XSLEEP) - goto unlock; - mp->mnt_writeopcount++; -unlock: - if (error !=3D 0 || (flags & V_XSLEEP) !=3D 0) - MNT_REL(mp); - MNT_IUNLOCK(mp); - return (error); + return (vn_start_write_locked(mp, flags)); } =20 /* @@ -1639,8 +1650,7 @@ vfs_write_suspend(mp) * Request a filesystem to resume write operations. */ void -vfs_write_resume(mp) - struct mount *mp; +vfs_write_resume_flags(struct mount *mp, int flags) { =20 MNT_ILOCK(mp); @@ -1652,10 +1662,25 @@ vfs_write_resume(mp) wakeup(&mp->mnt_writeopcount); wakeup(&mp->mnt_flag); curthread->td_pflags &=3D ~TDP_IGNSUSP; + if ((flags & VR_START_WRITE) !=3D 0) { + MNT_REF(mp); + mp->mnt_writeopcount++; + } MNT_IUNLOCK(mp); VFS_SUSP_CLEAN(mp); - } else + } else if ((flags & VR_START_WRITE) !=3D 0) { + MNT_REF(mp); + vn_start_write_locked(mp, 0); + } else { MNT_IUNLOCK(mp); + } +} + +void +vfs_write_resume(struct mount *mp) +{ + + vfs_write_resume_flags(mp, 0); } =20 /* diff --git a/sys/sys/vnode.h b/sys/sys/vnode.h index 42f9e5f..4371b40 100644 --- a/sys/sys/vnode.h +++ b/sys/sys/vnode.h @@ -392,6 +392,8 @@ extern int vttoif_tab[]; #define V_NOWAIT 0x0002 /* vn_start_write: don't sleep for suspend */ #define V_XSLEEP 0x0004 /* vn_start_write: just return after sleep */ =20 +#define VR_START_WRITE 0x0001 /* vfs_write_resume: start write atomically = */ + #define VREF(vp) vref(vp) =20 #ifdef DIAGNOSTIC @@ -701,6 +703,7 @@ int vn_io_fault_uiomove(char *data, int xfersize, struc= t uio *uio); int vfs_cache_lookup(struct vop_lookup_args *ap); void vfs_timestamp(struct timespec *); void vfs_write_resume(struct mount *mp); +void vfs_write_resume_flags(struct mount *mp, int flags); int vfs_write_suspend(struct mount *mp); int vop_stdbmap(struct vop_bmap_args *); int vop_stdfsync(struct vop_fsync_args *); diff --git a/sys/ufs/ffs/ffs_snapshot.c b/sys/ufs/ffs/ffs_snapshot.c index e528509..25ad79c 100644 --- a/sys/ufs/ffs/ffs_snapshot.c +++ b/sys/ufs/ffs/ffs_snapshot.c @@ -687,8 +687,7 @@ out1: /* * Resume operation on filesystem. */ - vfs_write_resume(vp->v_mount); - vn_start_write(NULL, &wrtmp, V_WAIT); + vfs_write_resume_flags(vp->v_mount, VR_START_WRITE); if (collectsnapstats && starttime.tv_sec > 0) { nanotime(&endtime); timespecsub(&endtime, &starttime); --r7tUYVWcdYzDJoZW Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJQ3YIbAAoJEJDCuSvBvK1BencP/0168lx07OvV4T7toyComatr aMz0nFicRqIkTzK52yDACCt8XMarnzV87EzkHfirRbtu4xwjxF27BHIgowDUCeDW U4Cne6f9wt8zVX/9NkGYmbJAZ5Osuz22nAUz5AAd9H6S05IC4HJ/+ovsWw57riEU ljWNkCS+tRs8uFjAK6aKUiE7c5eoaL+IXVCrdAVW2Puc48eJ9B6dl9NesYdbCqEs RkTD6RyG0qdHPEsBpYmyxq+uklJVlrm3uxEzKhbmbCEcfrwynXvb/wR0d4Kfmyb2 tje334D/C7+nv6oITdMOqfkdbBDJuKGMR9aWh+8lkuCgZnK7SLC3tYsEOGhk3EBy /mo1W7DpcWu2qciEDsJDBCQQ1r8ZJXFT8Rtx0c2KzioNbsGriTiVkcghJBMzvUMi nNbUkoGu/37w9J4t+kxbDgLtRiPI9P5YfOEUHJgGnVP3o/4JQkmcDnDiKZnw6NhF s1qXtk7pVFHgrJ2sN40NkX2rIZvE4Ih/Ueti83UWmWwr9p9sF/O/0TIDXMmImNAa MYnGvTx8pPgmXFjfGKkAc2EYG24PAuXhSD1oXZjY+z2m3UKgXpQ01oCLv2i6kkRL XCSuN6b/vQ6VyDvUCnZt1u+gbg7hgiM3htHnXHjDlRu/QaY/EiJHcUFWljpClb63 Po93TZJCvHxewZsuu1Kf =EJ0n -----END PGP SIGNATURE----- --r7tUYVWcdYzDJoZW-- From owner-freebsd-fs@FreeBSD.ORG Fri Dec 28 19:42:00 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6F8FF6AD; Fri, 28 Dec 2012 19:42:00 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [IPv6:2001:5a8:4:7e72:4a5b:39ff:fe12:452]) by mx1.freebsd.org (Postfix) with ESMTP id 291DB8FC0C; Fri, 28 Dec 2012 19:42:00 +0000 (UTC) Received: from chez.mckusick.com (localhost [127.0.0.1]) by chez.mckusick.com (8.14.3/8.14.3) with ESMTP id qBSJfwIw029811; Fri, 28 Dec 2012 11:41:58 -0800 (PST) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201212281941.qBSJfwIw029811@chez.mckusick.com> To: Konstantin Belousov Subject: Re: FS hang with suspfs when creating snapshot on a UFS + GJOURNAL setup In-reply-to: <20121228112724.GR82219@kib.kiev.ua> Date: Fri, 28 Dec 2012 11:41:57 -0800 From: Kirk McKusick X-Spam-Status: No, score=0.0 required=5.0 tests=MISSING_MID, UNPARSEABLE_RELAY autolearn=failed version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on chez.mckusick.com Cc: pho@freebsd.org, Andreas Longwitz , fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2012 19:42:00 -0000 Your proposed fix looks correct to me. Kirk McKusick From owner-freebsd-fs@FreeBSD.ORG Fri Dec 28 21:37:08 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8163DEFD; Fri, 28 Dec 2012 21:37:08 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) by mx1.freebsd.org (Postfix) with ESMTP id 21C528FC0A; Fri, 28 Dec 2012 21:37:07 +0000 (UTC) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id 04DDD5C3A0; Fri, 28 Dec 2012 22:37:01 +0100 (CET) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id s91mKSocLEi5; Fri, 28 Dec 2012 22:36:59 +0100 (CET) Received: from mail.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id C6BCD5C39D; Fri, 28 Dec 2012 22:36:59 +0100 (CET) Received: from bsdmhs.longwitz (unknown [192.168.99.6]) by mail.incore (Postfix) with ESMTP id 09FEA5083F; Fri, 28 Dec 2012 22:36:58 +0100 (CET) Message-ID: <50DE10FA.30102@incore.de> Date: Fri, 28 Dec 2012 22:36:58 +0100 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: FS hang with suspfs when creating snapshot on a UFS + GJOURNAL setup References: <50DC30F6.1050904@incore.de> <20121227133355.GI82219@kib.kiev.ua> <50DC8999.8000708@incore.de> <20121227194145.GM82219@kib.kiev.ua> <50DD6423.5090305@incore.de> <20121228112724.GR82219@kib.kiev.ua> In-Reply-To: <20121228112724.GR82219@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit Cc: pho@freebsd.org, freebsd-stable@freebsd.org, fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2012 21:37:08 -0000 Konstantin Belousov wrote: > > Please try the following patch. It is against HEAD, might need some > adjustments for 8. I do the resume and write accounting atomically, > not allowing other suspension to intervent between. > > diff --git a/sys/kern/vfs_vnops.c b/sys/kern/vfs_vnops.c > index 3f65b05..cf49ecb 100644 > --- a/sys/kern/vfs_vnops.c > +++ b/sys/kern/vfs_vnops.c > @@ -1434,6 +1434,40 @@ vn_closefile(fp, td) > * proceed. If a suspend request is in progress, we wait until the > * suspension is over, and then proceed. > */ > +static int > +vn_start_write_locked(struct mount *mp, int flags) > +{ > + int error; > + > + mtx_assert(MNT_MTX(mp), MA_OWNED); > + error = 0; > + > + /* > + * Check on status of suspension. > + */ > + if ((curthread->td_pflags & TDP_IGNSUSP) == 0 || > + mp->mnt_susp_owner != curthread) { > + while ((mp->mnt_kern_flag & MNTK_SUSPEND) != 0) { > + if (flags & V_NOWAIT) { > + error = EWOULDBLOCK; > + goto unlock; > + } > + error = msleep(&mp->mnt_flag, MNT_MTX(mp), > + (PUSER - 1) | (flags & PCATCH), "suspfs", 0); > + if (error) > + goto unlock; > + } > + } > + if (flags & V_XSLEEP) > + goto unlock; > + mp->mnt_writeopcount++; > +unlock: > + if (error != 0 || (flags & V_XSLEEP) != 0) > + MNT_REL(mp); > + MNT_IUNLOCK(mp); > + return (error); > +} > + > int > vn_start_write(vp, mpp, flags) > struct vnode *vp; > @@ -1470,30 +1504,7 @@ vn_start_write(vp, mpp, flags) > if (vp == NULL) > MNT_REF(mp); > > - /* > - * Check on status of suspension. > - */ > - if ((curthread->td_pflags & TDP_IGNSUSP) == 0 || > - mp->mnt_susp_owner != curthread) { > - while ((mp->mnt_kern_flag & MNTK_SUSPEND) != 0) { > - if (flags & V_NOWAIT) { > - error = EWOULDBLOCK; > - goto unlock; > - } > - error = msleep(&mp->mnt_flag, MNT_MTX(mp), > - (PUSER - 1) | (flags & PCATCH), "suspfs", 0); > - if (error) > - goto unlock; > - } > - } > - if (flags & V_XSLEEP) > - goto unlock; > - mp->mnt_writeopcount++; > -unlock: > - if (error != 0 || (flags & V_XSLEEP) != 0) > - MNT_REL(mp); > - MNT_IUNLOCK(mp); > - return (error); > + return (vn_start_write_locked(mp, flags)); > } > > /* > @@ -1639,8 +1650,7 @@ vfs_write_suspend(mp) > * Request a filesystem to resume write operations. > */ > void > -vfs_write_resume(mp) > - struct mount *mp; > +vfs_write_resume_flags(struct mount *mp, int flags) > { > > MNT_ILOCK(mp); > @@ -1652,10 +1662,25 @@ vfs_write_resume(mp) > wakeup(&mp->mnt_writeopcount); > wakeup(&mp->mnt_flag); > curthread->td_pflags &= ~TDP_IGNSUSP; > + if ((flags & VR_START_WRITE) != 0) { > + MNT_REF(mp); > + mp->mnt_writeopcount++; > + } > MNT_IUNLOCK(mp); > VFS_SUSP_CLEAN(mp); > - } else > + } else if ((flags & VR_START_WRITE) != 0) { > + MNT_REF(mp); > + vn_start_write_locked(mp, 0); > + } else { > MNT_IUNLOCK(mp); > + } > +} > + > +void > +vfs_write_resume(struct mount *mp) > +{ > + > + vfs_write_resume_flags(mp, 0); > } > > /* > diff --git a/sys/sys/vnode.h b/sys/sys/vnode.h > index 42f9e5f..4371b40 100644 > --- a/sys/sys/vnode.h > +++ b/sys/sys/vnode.h > @@ -392,6 +392,8 @@ extern int vttoif_tab[]; > #define V_NOWAIT 0x0002 /* vn_start_write: don't sleep for suspend */ > #define V_XSLEEP 0x0004 /* vn_start_write: just return after sleep */ > > +#define VR_START_WRITE 0x0001 /* vfs_write_resume: start write atomically */ > + > #define VREF(vp) vref(vp) > > #ifdef DIAGNOSTIC > @@ -701,6 +703,7 @@ int vn_io_fault_uiomove(char *data, int xfersize, struct uio *uio); > int vfs_cache_lookup(struct vop_lookup_args *ap); > void vfs_timestamp(struct timespec *); > void vfs_write_resume(struct mount *mp); > +void vfs_write_resume_flags(struct mount *mp, int flags); > int vfs_write_suspend(struct mount *mp); > int vop_stdbmap(struct vop_bmap_args *); > int vop_stdfsync(struct vop_fsync_args *); > diff --git a/sys/ufs/ffs/ffs_snapshot.c b/sys/ufs/ffs/ffs_snapshot.c > index e528509..25ad79c 100644 > --- a/sys/ufs/ffs/ffs_snapshot.c > +++ b/sys/ufs/ffs/ffs_snapshot.c > @@ -687,8 +687,7 @@ out1: > /* > * Resume operation on filesystem. > */ > - vfs_write_resume(vp->v_mount); > - vn_start_write(NULL, &wrtmp, V_WAIT); > + vfs_write_resume_flags(vp->v_mount, VR_START_WRITE); > if (collectsnapstats && starttime.tv_sec > 0) { > nanotime(&endtime); > timespecsub(&endtime, &starttime); Your patch adjusted to FreeBSD Stable 8 works fine. Creating a snapshot every minute runs without errors for more than 6 hours now, without the patch my test machine hangs not later than one hour. -- Andreas Longwitz From owner-freebsd-fs@FreeBSD.ORG Sat Dec 29 13:02:29 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 29B09E11 for ; Sat, 29 Dec 2012 13:02:29 +0000 (UTC) (envelope-from kpaasial@gmail.com) Received: from mail-wi0-f172.google.com (mail-wi0-f172.google.com [209.85.212.172]) by mx1.freebsd.org (Postfix) with ESMTP id B48868FC0A for ; Sat, 29 Dec 2012 13:02:28 +0000 (UTC) Received: by mail-wi0-f172.google.com with SMTP id o1so8712893wic.11 for ; Sat, 29 Dec 2012 05:02:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=UJjsCItHGPnnKIZvcUboSDNQ8mFO6AiqFrk8zWjopNA=; b=DH+NvvazOi0JBgzyR1oAW/Nt/D9Ru06klvG5Jqa6CwVErxZ5r2G4eJW45G2VjLLekC MXfFYBZ9sDTl0VGM47K4yoiYa3Lm7tFwmAXZAb1L3cpr9iS0AOgvAu3HQUnDcLSz6wGP d0kct6yY4eAHwlvAThxFyZUqefWw7mANkzsUfKlxDIzLyYB9AHViIycJ4N9MbuJAzgmw 2kaXGtpVErK0nvjSM8hff3OWeQ2Wk8+tuEFtzzeJvYsfTbNf4ugAy1yNlrr3ggWmd7Oq cPB3U1xcc783ApnKSUzBpsWIBoPlWEg+kvg3PYqNYpl81lZthr3Mv05zjVVyHHES9sxr oCtw== MIME-Version: 1.0 Received: by 10.180.109.201 with SMTP id hu9mr6067237wib.32.1356786141717; Sat, 29 Dec 2012 05:02:21 -0800 (PST) Received: by 10.216.172.197 with HTTP; Sat, 29 Dec 2012 05:02:21 -0800 (PST) Date: Sat, 29 Dec 2012 15:02:21 +0200 Message-ID: Subject: Zpool made of ZVOLs? From: Kimmo Paasiala To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Dec 2012 13:02:29 -0000 Is it possible and even recommended to create a zpool out of block devices that are actually ZVOLs? I'd like to have a separate zfs pool for poudriere because building ports with it disrupts the snapshotting on my main pool if the poudirere jails are created on it. At the same time I don't want to resort to using external disks to create a pool that is only used for a jail or two. Regards, -Kimmo