From owner-freebsd-fs@FreeBSD.ORG Sun Oct 13 16:52:59 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 12C7515C for ; Sun, 13 Oct 2013 16:52:59 +0000 (UTC) (envelope-from zaphod@berentweb.com) Received: from sam.nabble.com (sam.nabble.com [216.139.236.26]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E6C722C01 for ; Sun, 13 Oct 2013 16:52:58 +0000 (UTC) Received: from [192.168.236.26] (helo=sam.nabble.com) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1VVOuS-0006UO-H1 for freebsd-fs@freebsd.org; Sun, 13 Oct 2013 09:52:52 -0700 Date: Sun, 13 Oct 2013 09:52:52 -0700 (PDT) From: Beeblebrox To: freebsd-fs@freebsd.org Message-ID: <1381683172518-5851539.post@n5.nabble.com> In-Reply-To: References: <1380880223590-5848720.post@n5.nabble.com> <524EEE40.5060208@gmail.com> <1381060830753-5849397.post@n5.nabble.com> <20131006123350.GJ3287@sludge.elizium.za.net> Subject: Questions re swap-on-zfs MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Oct 2013 16:52:59 -0000 Thanks everyone for the assistance and the explanation for why swap "pri" will not work. @Ronald Klop: Wouldn't going through ccd(4) slow the throughput to the HDD by reason of extra processing layer? I want to make sure there is close to none of performance penalty before implementing. Thanks. ----- FreeBSD-9.2-stable_amd64_root-on-zfs_clang-only-world -- View this message in context: http://freebsd.1045724.n5.nabble.com/Questions-re-swap-on-zfs-tp5848720p5851539.html Sent from the freebsd-fs mailing list archive at Nabble.com. From owner-freebsd-fs@FreeBSD.ORG Mon Oct 14 11:06:47 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 8042E4AD for ; Mon, 14 Oct 2013 11:06:47 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6CFE12C73 for ; Mon, 14 Oct 2013 11:06:47 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r9EB6lhs035178 for ; Mon, 14 Oct 2013 11:06:47 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r9EB6lsw035176 for freebsd-fs@FreeBSD.org; Mon, 14 Oct 2013 11:06:47 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 14 Oct 2013 11:06:47 GMT Message-Id: <201310141106.r9EB6lsw035176@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Oct 2013 11:06:47 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/182570 fs [zfs] [patch] ZFS panic in receive o kern/182536 fs [zfs] zfs deadlock o kern/181966 fs [zfs] Kernel panic in ZFS I/O: solaris assert: BP_EQUA o kern/181834 fs [nfs] amd mounting NFS directories can drive a dead-lo o kern/181565 fs [swap] Problem with vnode-backed swap space. o kern/181377 fs [zfs] zfs recv causes an inconsistant pool o kern/181281 fs [msdosfs] stack trace after successfull 'umount /mnt' o kern/181082 fs [fuse] [ntfs] Write to mounted NTFS filesystem using F o kern/180979 fs [netsmb][patch]: Fix large files handling o kern/180876 fs [zfs] [hast] ZFS with trim,bio_flush or bio_delete loc o kern/180678 fs [NFS] succesfully exported filesystems being reported o kern/180438 fs [smbfs] [patch] mount_smbfs fails on arm because of wr p kern/180236 fs [zfs] [nullfs] Leakage free space using ZFS with nullf o kern/178854 fs [ufs] FreeBSD kernel crash in UFS s kern/178467 fs [zfs] [request] Optimized Checksum Code for ZFS o kern/178412 fs [smbfs] Coredump when smbfs mounted o kern/178388 fs [zfs] [patch] allow up to 8MB recordsize o kern/178387 fs [zfs] [patch] sparse files performance improvements o kern/178349 fs [zfs] zfs scrub on deduped data could be much less see o kern/178329 fs [zfs] extended attributes leak o kern/178238 fs [nullfs] nullfs don't release i-nodes on unlink. f kern/178231 fs [nfs] 8.3 nfsv4 client reports "nfsv4 client/server pr o kern/177985 fs [zfs] disk usage problem when copying from one zfs dat o kern/177971 fs [nfs] FreeBSD 9.1 nfs client dirlist problem w/ nfsv3, o kern/177966 fs [zfs] resilver completes but subsequent scrub reports o kern/177658 fs [ufs] FreeBSD panics after get full filesystem with uf o kern/177536 fs [zfs] zfs livelock (deadlock) with high write-to-disk o kern/177445 fs [hast] HAST panic o kern/177240 fs [zfs] zpool import failed with state UNAVAIL but all d o kern/176978 fs [zfs] [panic] zfs send -D causes "panic: System call i o kern/176857 fs [softupdates] [panic] 9.1-RELEASE/amd64/GENERIC panic o bin/176253 fs zpool(8): zfs pool indentation is misleading/wrong o kern/176141 fs [zfs] sharesmb=on makes errors for sharenfs, and still o kern/175950 fs [zfs] Possible deadlock in zfs after long uptime o kern/175897 fs [zfs] operations on readonly zpool hang o kern/175449 fs [unionfs] unionfs and devfs misbehaviour o kern/175179 fs [zfs] ZFS may attach wrong device on move o kern/175071 fs [ufs] [panic] softdep_deallocate_dependencies: unrecov o kern/174372 fs [zfs] Pagefault appears to be related to ZFS o kern/174315 fs [zfs] chflags uchg not supported o kern/174310 fs [zfs] root point mounting broken on CURRENT with multi o kern/174279 fs [ufs] UFS2-SU+J journal and filesystem corruption o kern/173830 fs [zfs] Brain-dead simple change to ZFS error descriptio o kern/173718 fs [zfs] phantom directory in zraid2 pool f kern/173657 fs [nfs] strange UID map with nfsuserd o kern/173363 fs [zfs] [panic] Panic on 'zpool replace' on readonly poo o kern/173136 fs [unionfs] mounting above the NFS read-only share panic o kern/172942 fs [smbfs] Unmounting a smb mount when the server became o kern/172348 fs [unionfs] umount -f of filesystem in use with readonly o kern/172334 fs [unionfs] unionfs permits recursive union mounts; caus o kern/171626 fs [tmpfs] tmpfs should be noisier when the requested siz o kern/171415 fs [zfs] zfs recv fails with "cannot receive incremental o kern/170945 fs [gpt] disk layout not portable between direct connect o bin/170778 fs [zfs] [panic] FreeBSD panics randomly o kern/170680 fs [nfs] Multiple NFS Client bug in the FreeBSD 7.4-RELEA o kern/170497 fs [xfs][panic] kernel will panic whenever I ls a mounted o kern/169945 fs [zfs] [panic] Kernel panic while importing zpool (afte o kern/169480 fs [zfs] ZFS stalls on heavy I/O o kern/169398 fs [zfs] Can't remove file with permanent error o kern/169339 fs panic while " : > /etc/123" o kern/169319 fs [zfs] zfs resilver can't complete o kern/168947 fs [nfs] [zfs] .zfs/snapshot directory is messed up when o kern/168942 fs [nfs] [hang] nfsd hangs after being restarted (not -HU o kern/168158 fs [zfs] incorrect parsing of sharenfs options in zfs (fs o kern/167979 fs [ufs] DIOCGDINFO ioctl does not work on 8.2 file syste o kern/167977 fs [smbfs] mount_smbfs results are differ when utf-8 or U o kern/167688 fs [fusefs] Incorrect signal handling with direct_io o kern/167685 fs [zfs] ZFS on USB drive prevents shutdown / reboot o kern/167612 fs [portalfs] The portal file system gets stuck inside po o kern/167272 fs [zfs] ZFS Disks reordering causes ZFS to pick the wron o kern/167260 fs [msdosfs] msdosfs disk was mounted the second time whe o kern/167109 fs [zfs] [panic] zfs diff kernel panic Fatal trap 9: gene o kern/167105 fs [nfs] mount_nfs can not handle source exports wiht mor o kern/167067 fs [zfs] [panic] ZFS panics the server o kern/167065 fs [zfs] boot fails when a spare is the boot disk o kern/167048 fs [nfs] [patch] RELEASE-9 crash when using ZFS+NULLFS+NF o kern/166912 fs [ufs] [panic] Panic after converting Softupdates to jo o kern/166851 fs [zfs] [hang] Copying directory from the mounted UFS di o kern/166477 fs [nfs] NFS data corruption. o kern/165950 fs [ffs] SU+J and fsck problem o kern/165521 fs [zfs] [hang] livelock on 1 Gig of RAM with zfs when 31 o kern/165392 fs Multiple mkdir/rmdir fails with errno 31 o kern/165087 fs [unionfs] lock violation in unionfs o kern/164472 fs [ufs] fsck -B panics on particular data inconsistency o kern/164370 fs [zfs] zfs destroy for snapshot fails on i386 and sparc o kern/164261 fs [nullfs] [patch] fix panic with NFS served from NULLFS o kern/164256 fs [zfs] device entry for volume is not created after zfs o kern/164184 fs [ufs] [panic] Kernel panic with ufs_makeinode o kern/163801 fs [md] [request] allow mfsBSD legacy installed in 'swap' o kern/163770 fs [zfs] [hang] LOR between zfs&syncer + vnlru leading to o kern/163501 fs [nfs] NFS exporting a dir and a subdir in that dir to o kern/162944 fs [coda] Coda file system module looks broken in 9.0 o kern/162860 fs [zfs] Cannot share ZFS filesystem to hosts with a hyph o kern/162751 fs [zfs] [panic] kernel panics during file operations o kern/162591 fs [nullfs] cross-filesystem nullfs does not work as expe o kern/162519 fs [zfs] "zpool import" relies on buggy realpath() behavi o kern/161968 fs [zfs] [hang] renaming snapshot with -r including a zvo o kern/161864 fs [ufs] removing journaling from UFS partition fails on o kern/161579 fs [smbfs] FreeBSD sometimes panics when an smb share is o kern/161533 fs [zfs] [panic] zfs receive panic: system ioctl returnin o kern/161438 fs [zfs] [panic] recursed on non-recursive spa_namespace_ o kern/161424 fs [nullfs] __getcwd() calls fail when used on nullfs mou o kern/161280 fs [zfs] Stack overflow in gptzfsboot o kern/161205 fs [nfs] [pfsync] [regression] [build] Bug report freebsd o kern/161169 fs [zfs] [panic] ZFS causes kernel panic in dbuf_dirty o kern/161112 fs [ufs] [lor] filesystem LOR in FreeBSD 9.0-BETA3 o kern/160893 fs [zfs] [panic] 9.0-BETA2 kernel panic f kern/160860 fs [ufs] Random UFS root filesystem corruption with SU+J o kern/160801 fs [zfs] zfsboot on 8.2-RELEASE fails to boot from root-o o kern/160790 fs [fusefs] [panic] VPUTX: negative ref count with FUSE o kern/160777 fs [zfs] [hang] RAID-Z3 causes fatal hang upon scrub/impo o kern/160706 fs [zfs] zfs bootloader fails when a non-root vdev exists o kern/160591 fs [zfs] Fail to boot on zfs root with degraded raidz2 [r o kern/160410 fs [smbfs] [hang] smbfs hangs when transferring large fil o kern/160283 fs [zfs] [patch] 'zfs list' does abort in make_dataset_ha o kern/159930 fs [ufs] [panic] kernel core o kern/159402 fs [zfs][loader] symlinks cause I/O errors o kern/159357 fs [zfs] ZFS MAXNAMELEN macro has confusing name (off-by- o kern/159356 fs [zfs] [patch] ZFS NAME_ERR_DISKLIKE check is Solaris-s o kern/159351 fs [nfs] [patch] - divide by zero in mountnfs() o kern/159251 fs [zfs] [request]: add FLETCHER4 as DEDUP hash option o kern/159077 fs [zfs] Can't cd .. with latest zfs version o kern/159048 fs [smbfs] smb mount corrupts large files o kern/159045 fs [zfs] [hang] ZFS scrub freezes system o kern/158839 fs [zfs] ZFS Bootloader Fails if there is a Dead Disk o kern/158802 fs amd(8) ICMP storm and unkillable process. o kern/158231 fs [nullfs] panic on unmounting nullfs mounted over ufs o f kern/157929 fs [nfs] NFS slow read o kern/157399 fs [zfs] trouble with: mdconfig force delete && zfs strip o kern/157179 fs [zfs] zfs/dbuf.c: panic: solaris assert: arc_buf_remov o kern/156797 fs [zfs] [panic] Double panic with FreeBSD 9-CURRENT and o kern/156781 fs [zfs] zfs is losing the snapshot directory, p kern/156545 fs [ufs] mv could break UFS on SMP systems o kern/156193 fs [ufs] [hang] UFS snapshot hangs && deadlocks processes o kern/156039 fs [nullfs] [unionfs] nullfs + unionfs do not compose, re o kern/155615 fs [zfs] zfs v28 broken on sparc64 -current o kern/155587 fs [zfs] [panic] kernel panic with zfs p kern/155411 fs [regression] [8.2-release] [tmpfs]: mount: tmpfs : No o kern/155199 fs [ext2fs] ext3fs mounted as ext2fs gives I/O errors o bin/155104 fs [zfs][patch] use /dev prefix by default when importing o kern/154930 fs [zfs] cannot delete/unlink file from full volume -> EN o kern/154828 fs [msdosfs] Unable to create directories on external USB o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 p kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o bin/153142 fs [zfs] ls -l outputs `ls: ./.zfs: Operation not support o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server o kern/145750 fs [unionfs] [hang] unionfs locks the machine s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an f bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141950 fs [unionfs] [lor] ufs/unionfs/ufs Lock order reversal o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/137588 fs [unionfs] [lor] LOR nfs/ufs/nfs o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis p kern/133174 fs [msdosfs] [patch] msdosfs must support multibyte inter o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126973 fs [unionfs] [hang] System hang with unionfs and init chr o kern/126553 fs [unionfs] unionfs move directory problem 2 (files appe o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files o bin/123574 fs [unionfs] df(1) -t option destroys info for unionfs (a o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o kern/121385 fs [unionfs] unionfs cross mount -> kernel panic o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o kern/118318 fs [nfs] NFS server hangs under special circumstances o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118126 fs [nfs] [patch] Poor NFS server write performance o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116583 fs [ffs] [hang] System freezes for short time when using o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/67326 fs [msdosfs] crash after attempt to mount write protected o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t o kern/9619 fs [nfs] Restarting mountd kills existing mounts 336 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Oct 14 20:08:07 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id AB5757C8 for ; Mon, 14 Oct 2013 20:08:07 +0000 (UTC) (envelope-from rick@havokmon.com) Received: from smtp101-5.vfemail.net (nine.vfemail.net [108.76.175.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 612A8238D for ; Mon, 14 Oct 2013 20:08:06 +0000 (UTC) Received: (qmail 20994 invoked by uid 89); 14 Oct 2013 20:08:05 -0000 Received: by simscan 1.4.0 ppid: 20987, pid: 20990, t: 0.0637s scanners:none Received: from unknown (HELO www110) (cmlja0BoYXZva21vbi5jb20=@172.16.100.92) by 172.16.100.61 with ESMTPA; 14 Oct 2013 20:08:05 -0000 Received: from rrcs-98-103-53-237.central.biz.rr.com (rrcs-98-103-53-237.central.biz.rr.com [98.103.53.237]) by beta.vfemail.net (Horde Framework) with HTTP; Mon, 14 Oct 2013 15:08:05 -0500 Date: Mon, 14 Oct 2013 15:08:04 -0500 Message-ID: <20131014150804.Horde.9q7sJBy75i220K9FmcJ1nw1@beta.vfemail.net> From: Rick Romero To: freebsd-fs@freebsd.org Subject: NFS locks, rpcbind port = 0 failed? - try #2 User-Agent: Internet Messaging Program (IMP) H5 (6.1.4) X-VFEmail-Originating-IP: OTguMTAzLjUzLjIzNw== X-VFEmail-Remote-Browser: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0 @ X-VFEmail-AntiSpam: Notify admin@vfemail.net of any spam, and include VFEmail headers X-Remote-Browser: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes Content-Transfer-Encoding: 8bit Content-Disposition: inline Content-Description: Plaintext Message X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Oct 2013 20:08:07 -0000 This is a continuation of "9.1 VM nfs3 & locks over VPN" from freebsd-questions - trying a different angle maybe it'll jostle someones memory.  Don't mean to cross-post, but as I pay more attention to the lists I'm reading, this seems to be the better list for NFS issues. I have a FreeBSD 9.2 VM at an offsite hosting company.  hostname nl101vpn OpenVPN is installed on it, routed not bridged mode. I have multiple OSs installed on local network. I'm already exportings NFS off 9.1 with working file locks. What I see - export nfsv3 or nfsv4 from nl101vpn, mount on local FreeBSD or Linux - locks do not work. export nfsv3 from any local system, mount on nl101vpn - locks work. export nfsv3 from locally installed VM, mount on any local host or nl101vpn - locks work.  No OpenVPN installed on it though. This was to test if virtio drivers might be causing the problem. I even ran a tcpdump to see if something was getting lost - both sides match, nothing is getting dropped nl101vpn - /var/log/messages: Oct 14 12:21:01 nl101 kernel: NLM: failed to contact remote rpcbind, stat = 0, port = 0  (why port 0?) Oct 14 12:23:02 nl101 last message repeated 109 times Oct 14 12:25:48 nl101 last message repeated 177 times I tried binding rpcbind to the VPN interface, but that doesn't seem to work.  tcpdump shows no packets trying to leave the 'Internet' interface. So I haven't exhausted every combination, or completely 100% replicated whats happening offsite, but it's getting pretty ridiculous now... I'm lost, and I need NFS locking to work. Help :) Thanks, Rick From owner-freebsd-fs@FreeBSD.ORG Mon Oct 14 22:16:25 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4351AE60 for ; Mon, 14 Oct 2013 22:16:25 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 094972BE4 for ; Mon, 14 Oct 2013 22:16:24 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Am0PAHtsXFKDaFve/2dsb2JhbABZgz9Sgym+HEuBQHQEgiEBAQEDAQEBASArIAsFFhgCAg0ZAikBCSYGCAcEARwEh18GDKskkkiBKYxvgQUBMweCaoE5A5U6g3qQU4NAIDGBAzk X-IronPort-AV: E=Sophos;i="4.93,494,1378872000"; d="scan'208";a="60075954" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 14 Oct 2013 18:16:17 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id A11D0B40A3; Mon, 14 Oct 2013 18:16:17 -0400 (EDT) Date: Mon, 14 Oct 2013 18:16:17 -0400 (EDT) From: Rick Macklem To: Rick Romero Message-ID: <485946006.40991171.1381788977647.JavaMail.root@uoguelph.ca> In-Reply-To: <20131014150804.Horde.9q7sJBy75i220K9FmcJ1nw1@beta.vfemail.net> Subject: Re: NFS locks, rpcbind port = 0 failed? - try #2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Oct 2013 22:16:25 -0000 Rick Romero wrote: > This is a continuation of "9.1 VM nfs3 & locks over VPN" from > freebsd-questions - trying a > different angle maybe it'll jostle someones memory.=C2=A0 Don't mean to > cross-post, but as I pay more attention to the lists I'm reading, > this > seems to be the better list for NFS issues. >=20 > I have a FreeBSD 9.2 VM at an offsite hosting company.=C2=A0 hostname > nl101vpn > OpenVPN is installed on it, routed not bridged mode. > I have multiple OSs installed on local network. I'm already > exportings NFS > off 9.1 with working file locks. >=20 > What I see - > export nfsv3 or nfsv4 from nl101vpn, mount on local FreeBSD or Linux > - > locks do not work. > export nfsv3 from any local system, mount on nl101vpn - locks work. > export nfsv3 from locally installed VM, mount on any local host or > nl101vpn > - locks work.=C2=A0 No OpenVPN installed on it though. This was to test i= f > virtio drivers might be causing the problem. >=20 > I even ran a tcpdump to see if something was getting lost - both > sides > match, nothing is getting dropped >=20 > nl101vpn - /var/log/messages: > Oct 14 12:21:01 nl101 kernel: NLM: failed to contact remote rpcbind, > stat =3D > 0, port =3D 0=C2=A0 (why port 0?) > Oct 14 12:23:02 nl101 last message repeated 109 times > Oct 14 12:25:48 nl101 last message repeated 177 times >=20 > I tried binding rpcbind to the VPN interface, but that doesn't seem > to > work.=C2=A0 tcpdump shows no packets trying to leave the 'Internet' > interface. >=20 > So I haven't exhausted every combination, or completely 100% > replicated > whats happening offsite, but it's getting pretty ridiculous now... > I'm > lost, and I need NFS locking to work. > Help :) >=20 For rpcbind to work, IP broadcast needs to work between the hosts and I suspect that the VPN doesn't support that. Without rpcbind, I don't think you can get rpc.lockd/rpc.statd to work, but I am not sure. (There are command line options for these daemons that allow you to set specific port #s, but I don't think that will fix the problem, since they still need rpcbind to tell them the port# for the remote machines.) These protocols were designed in the 1980s for use on a LAN. Now, nfsv4 shouldn't care less about rpcbind, rpc.lockd. NFSv4 locking is handled as a part of the NFSv4 protocol and always uses port #2049. I'd suggest you try NFSv4 again and make sure it is using NFSv4 and the mount has not fallen back to NFSv3. (For FreeBSD, specify "nfsv4" as a mount option. For Linux, specify "vers=3D4" as a mount option.) You can check what the mount is actually using via "nfsstat -m". If you assumed the locking for NFSv4 wasn't working because of these messages, that isn't the case. If you are using NFSv4 for all mounts, you don't need to run rpc.lockd at all (at least for FreeBSD, I'm not sure what the daemons do w.r.t. Linux). Good luck with it, rick > Thanks, > Rick > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 01:05:06 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id B0C864D2 for ; Tue, 15 Oct 2013 01:05:06 +0000 (UTC) (envelope-from rick@havokmon.com) Received: from smtp101-5.vfemail.net (nine.vfemail.net [108.76.175.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4C6D82493 for ; Tue, 15 Oct 2013 01:05:05 +0000 (UTC) Received: (qmail 1270 invoked by uid 89); 15 Oct 2013 01:04:58 -0000 Received: by simscan 1.4.0 ppid: 1257, pid: 1262, t: 0.0870s scanners:none Received: from unknown (HELO www110) (cmlja0BoYXZva21vbi5jb20=@172.16.100.92) by 172.16.100.61 with ESMTPA; 15 Oct 2013 01:04:58 -0000 Received: from fw.vfemail.net (fw.vfemail.net [108.76.175.13]) by beta.vfemail.net (Horde Framework) with HTTP; Mon, 14 Oct 2013 20:04:58 -0500 Date: Mon, 14 Oct 2013 20:04:58 -0500 Message-ID: <20131014200458.Horde.QjqQXEfm9A5k9357kfSZGQ1@beta.vfemail.net> From: Rick Romero To: freebsd-fs@freebsd.org Subject: Re: NFS locks, rpcbind port = 0 failed? - try #2 References: <485946006.40991171.1381788977647.JavaMail.root@uoguelph.ca> In-Reply-To: <485946006.40991171.1381788977647.JavaMail.root@uoguelph.ca> User-Agent: Internet Messaging Program (IMP) H5 (6.1.4) X-VFEmail-Originating-IP: MTA4Ljc2LjE3NS4xMw== X-VFEmail-Remote-Browser: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0 @ X-VFEmail-AntiSpam: Notify admin@vfemail.net of any spam, and include VFEmail headers X-Remote-Browser: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes Content-Transfer-Encoding: 8bit Content-Disposition: inline Content-Description: Plaintext Message X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 01:05:06 -0000 Quoting Rick Macklem : > Rick Romero wrote: >> This is a continuation of "9.1 VM nfs3 & locks over VPN" from >> freebsd-questions - trying a >> different angle maybe it'll jostle someones memory.  Don't mean to >> cross-post, but as I pay more attention to the lists I'm reading, >> this >> seems to be the better list for NFS issues. >> >> I have a FreeBSD 9.2 VM at an offsite hosting company.  hostname >> nl101vpn >> OpenVPN is installed on it, routed not bridged mode. >> I have multiple OSs installed on local network. I'm already >> exportings NFS >> off 9.1 with working file locks. >> >> What I see - >> export nfsv3 or nfsv4 from nl101vpn, mount on local FreeBSD or Linux >> - >> locks do not work. >> export nfsv3 from any local system, mount on nl101vpn - locks work. >> export nfsv3 from locally installed VM, mount on any local host or >> nl101vpn >> - locks work.  No OpenVPN installed on it though. This was to test if >> virtio drivers might be causing the problem. >> >> I even ran a tcpdump to see if something was getting lost - both >> sides >> match, nothing is getting dropped >> >> nl101vpn - /var/log/messages: >> Oct 14 12:21:01 nl101 kernel: NLM: failed to contact remote rpcbind, >> stat = >> 0, port = 0  (why port 0?) >> Oct 14 12:23:02 nl101 last message repeated 109 times >> Oct 14 12:25:48 nl101 last message repeated 177 times >> >> I tried binding rpcbind to the VPN interface, but that doesn't seem >> to >> work.  tcpdump shows no packets trying to leave the 'Internet' >> interface. >> >> So I haven't exhausted every combination, or completely 100% >> replicated >> whats happening offsite, but it's getting pretty ridiculous now... >> I'm >> lost, and I need NFS locking to work. >> Help :) > > For rpcbind to work, IP broadcast needs to work between the hosts > and I suspect that the VPN doesn't support that. > > Without rpcbind, I don't think you can get rpc.lockd/rpc.statd > to work, but I am not sure. (There are command line options for > these daemons that allow you to set specific port #s, but I don't > think that will fix the problem, since they still need rpcbind to > tell them the port# for the remote machines.) These protocols were > designed in the 1980s for use on a LAN. > > Now, nfsv4 shouldn't care less about rpcbind, rpc.lockd. NFSv4 locking > is handled as a part of the NFSv4 protocol and always uses port #2049. > I'd suggest you try NFSv4 again and make sure it is using NFSv4 and > the mount has not fallen back to NFSv3. (For FreeBSD, specify "nfsv4" > as a mount option. For Linux, specify "vers=4" as a mount option.) > You can check what the mount is actually using via "nfsstat -m". > If you assumed the locking for NFSv4 wasn't working because of these > messages, that isn't the case. If you are using NFSv4 for all mounts, > you don't need to run rpc.lockd at all (at least for FreeBSD, I'm > not sure what the daemons do w.r.t. Linux). Hi Rick, Yeah - I thought the VPN might pose a problem, but I can get locks from the VM side (nl101vpn) via NFS3 back to the main site.  So it doesn't seem to be an issue with the VPN. After that I created a local VM to ensure it wasn't a virtio thing, and then upgraded the remote VM to 9.2 (to rule out any funky custom options the host may have thrown into their 9.1 installer).  nada. So I'm re-trying with NFS4 - though my mount does show it was mounted nfs4 (from Linux) last time I tried: nl101vpn:/first on /mnt type nfs4 (rw,relatime,vers=4,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.16.1.92,minorversion=0,local_lock=none,addr=10.9.8.6) After trying again, still doesn't work.  Though now I noticed the error is different.  I have a little perl script that I test with, and the line is: flock(LOCKFILE, LOCK_SH) or die "Can't get shared lock on $lock_file: $!\n"; Before I would get 'pemission denied' - which (IIRC) would also happen when I forgot to run lockd or statd. Now with NFS4 it says, 'Bad file descriptor' After much more testing I've gotten a single Linux VM (but no other VMs on the same host, or my other host, yet they're all from the same template) to get a lock (n NFSv3) Failed locks show in the logs: NLM: failed to contact remote rpcbind, stat = 0, port = 0    (FreeBSD) NLM: failed to contact remote rpcbind, stat = 7, port = 28416  (Linux) I have 2 FreeBSD boxes, 9.1 and 7.2.  Both don't seem to be relaying their ports?  The Linux ones that fail have a port, but apparently can't be contacted. Or is 'port' in the log not really referring to a port number? The OpenVPN 'server' is another Linux VM - but on a different host than the working Linux VM :P  None of the Linux VMs on that host can get a lock.  How's THAT for weird? :) I need some aspirin.  Rick   From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 04:01:12 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id A43A113E; Tue, 15 Oct 2013 04:01:12 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 48A732D61; Tue, 15 Oct 2013 04:01:12 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.5/8.14.5) with ESMTP id r9F41AO4023339; Tue, 15 Oct 2013 00:01:10 -0400 (EDT) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.5/8.14.4/Submit) id r9F41Av9023336; Tue, 15 Oct 2013 00:01:10 -0400 (EDT) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <21084.48646.196295.776944@hergotha.csail.mit.edu> Date: Tue, 15 Oct 2013 00:01:10 -0400 From: Garrett Wollman To: freebsd-stable@freebsd.org Subject: How to unstick ZFS resilver? X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (hergotha.csail.mit.edu [127.0.0.1]); Tue, 15 Oct 2013 00:01:10 -0400 (EDT) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hergotha.csail.mit.edu Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 04:01:12 -0000 I have a large (88-drive) zpool in which a drive was recently replaced. (The pool has a bunch of duff Toshiba MK2001TRKB drives -- never ever pay money for these! -- and I'm trying to replace them one by one before they fail completely.) The resilver on the first drive replacement has been taking much much too long, and currently it's stuck in this state: pool: export state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Wed Oct 9 14:54:47 2013 86.5T scanned out of 86.8T at 1/s, (scan is slow, no estimated time) 982G resilvered, 99.62% done The overall progress hasn't changed in twelve hours, even across a reboot, and the server is fairly lightly loaded. Searching the Web is no help; can anyone suggest a remedial action? (This is on 9.1-RELEASE, with our local patches, and all the drives are SAS.) In exchange, I offer the following DTrace script which I used to identify the slow SAS drives: #!/usr/sbin/dtrace -s #pragma D option quiet #pragma D option dynvarsize=2m inline int TOO_SLOW = 100000000; /* 100 ms */ dtrace:::BEGIN { printf("Tracing... Hit Ctrl-C to end.\n"); } fbt::dastrategy:entry { start_time[(struct buf *)arg0] = timestamp; } fbt::dadone:entry /(this->bp = (struct buf *)args[1]->ccb_h.periph_priv.entries[1].ptr) && start_time[this->bp] && (timestamp - start_time[this->bp]) > TOO_SLOW/ { @[strjoin("da", lltostr(args[0]->unit_number))] = count(); start_time[this->bp] = 0; } -GAWollman From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 05:45:05 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id BAD6EE7B for ; Tue, 15 Oct 2013 05:45:05 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: from mail-la0-x22e.google.com (mail-la0-x22e.google.com [IPv6:2a00:1450:4010:c03::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3EEDC234B for ; Tue, 15 Oct 2013 05:45:05 +0000 (UTC) Received: by mail-la0-f46.google.com with SMTP id eh20so6353038lab.5 for ; Mon, 14 Oct 2013 22:45:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphix.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ORhTp3Tg5cbggnO/TMORgiJU/gsWh6z+0Ewe/P1L5NE=; b=Ph9BcCR9vOjtKHJUvRIul7hHsBaiW/r2FXu0GFaPgq2tyAxX29fx5SYk6j/yQXvma8 ZpgqylmRiJn7BbuE5gqOrpV7nSvheVW1fRl/J6SyXs3By7YpmGMjZ0p8M6iAiHZHVqvb L1kHyi1l9svpYjnxIQAJD1YfzNtRtpWE44n2o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=ORhTp3Tg5cbggnO/TMORgiJU/gsWh6z+0Ewe/P1L5NE=; b=kovIACxjyt1lYzTt7rKv/j8TOmOQ7vxDgjbWO/iLiKyIRkwpl1SgF+lKRnax9c8Asu nbM292iY5Yh6ay4pkw+fRvduMenm6Wu2JSDINgj0e0dNSQPB9cJuTKhGLppfARi+1/7x /xeFP1QYTmJvz0TLPIvLm/Dk4tsJi7QmZSsLEdACPX86zVrc9cWLKUIoS7AJPQA0rXts rh0jsjX6m0+0YR2WKj3/28nJuKzwgfQ0qkeSnXP8SZS4vDqWNw5/zzHas9LVbgtNFUHA +Kw7DS5oxM2+7Gm7sY/TxtmK5ivPO23iw4khquNUXxPKeXOdYMauDHvmpCSOkAqFEVfb kf2g== X-Gm-Message-State: ALoCoQlbme1gELfdY9tJ7raMGaJGFqk+OZNgkr9yERDq+fCXyqmXWGQ5a8fL5qigmH/39aQxv+Ck MIME-Version: 1.0 X-Received: by 10.152.36.98 with SMTP id p2mr34129152laj.14.1381815903238; Mon, 14 Oct 2013 22:45:03 -0700 (PDT) Received: by 10.114.92.200 with HTTP; Mon, 14 Oct 2013 22:45:03 -0700 (PDT) In-Reply-To: References: <52457A32.2090105@fsn.hu> <77F6465C-4E76-4EE9-88B5-238FFB4E0161@sarenet.es> <20130930234401.GA68360@neutralgood.org> Date: Mon, 14 Oct 2013 22:45:03 -0700 Message-ID: Subject: Re: zfs: the exponential file system from hell From: Matthew Ahrens To: Jordan Hubbard Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 05:45:05 -0000 On Tue, Oct 1, 2013 at 9:33 AM, Jordan Hubbard wrot= e: > > On Sep 30, 2013, at 4:44 PM, kpneal@pobox.com wrote: > > > Bottom line: > > The replacement for the 'df' command when using ZFS is 'zfs list'. > > Given that we have the sources to df, I guess we should consider the > question begged: Do we want to change it to DTRT for zfs filesystems? > There's no Unix Law=99 that says "df(1) must use the output of statfs(2) > directly and can use no longer sources of information!" > > At the end of the day, df(1) is just a convenient status reporting tool > aimed at human consumption. It could easily reach out to "zfs list" for > the data it prints for zfs volumes if what's reported by statfs(2) just > isn't suitable. > Indeed. This is what illumos df(1) does, so that the "Size" reported is the size of the pool, rather than the "Used + Available" of the individual filesystem. It would be great to get a similar change in gnu df (which is also used on many illumos distros). https://github.com/illumos/illumos-gate/blob/master/usr/src/cmd/fs.d/df.c#L= 1236-1306 --matt From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 08:25:34 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 5DD7197D for ; Tue, 15 Oct 2013 08:25:34 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-qa0-x234.google.com (mail-qa0-x234.google.com [IPv6:2607:f8b0:400d:c00::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1DC4A2C31 for ; Tue, 15 Oct 2013 08:25:34 +0000 (UTC) Received: by mail-qa0-f52.google.com with SMTP id w8so2869981qac.11 for ; Tue, 15 Oct 2013 01:25:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=FtIL6bhCT63uwQEYBu+eD8WIiTPMmXnkl8igIvA3E6g=; b=cOa75PCKnTSFh0WLSOraO/dmtVwygWovN8HYpEndL1wfBZmRuIBN3Da+2UX8b/b9Of NCiR50hfJNQXwRyA9OnORcDzM9orKuy+nIW1FAeE6Px57vwHVY0cNQKlWb2C4Kazf7kH eTETtnouNO/MxfjLLKEjCxVtdPBCw4mSRF0+3aGfMC6HhCD7UQpj0EjuPFGiaQXQuIgp FanYCvN5QGevTSqQ1gJTMWj5wyMCH+clrzAT4nvzqM8t5T3rCDilfOLvC7pmO5u9a1M+ zvulYOCc07PJwB/cN+Cx48JePZAt7ap9+0tNHjtq/JV6V2ffHi2NrggK5TjyGHMmKdEB ZWSw== MIME-Version: 1.0 X-Received: by 10.49.127.195 with SMTP id ni3mr43965127qeb.21.1381825533069; Tue, 15 Oct 2013 01:25:33 -0700 (PDT) Received: by 10.224.55.77 with HTTP; Tue, 15 Oct 2013 01:25:32 -0700 (PDT) In-Reply-To: References: <20131010170724.GA19751@potato.growveg.org> Date: Tue, 15 Oct 2013 09:25:32 +0100 Message-ID: Subject: Re: RAM and zfs and multiple disks From: krad To: Zaphod Beeblebrox Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Tom Evans , john@potato.growveg.org, FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 08:25:34 -0000 on thing worth noting about dedup is that its not an all or nothing option as you can tune it per file system. Also you only need the humongous amount of ram when you do writes, which of course is quite a common thing to do. However there are scenarios where a dataset will dedup quite well, but isnt written to often (os repo archive etc). You can potentially enable dedup on these and take the write performance hit. You can sometimes get away with is on databases as well, where the dataset is large and highly dedupable, but doesnt change much. What I have done in the past is enable dedup on the initial data load, and then disable it before deploying to production. In this case you get the benefits of the space saving, but not the hits when you write, as newly written data isnt deduped. This is a special case though so wont suit most as over time depending on write patterns the dataset could grow much bigger. As previously mentioned one of the cheapest ways to make it viable is to add a big chunk of SSD for L2ARC. Before you think doing this though check whether its even worth considering on your data by doing a zdb -S . It will give you the expected dedup ratio eg os pool from my main server root@#:/home/krad# zdb -S rpool Simulated DDT histogram: bucket allocated referenced ______ ______________________________ ______________________________ refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE ------ ------ ----- ----- ----- ------ ----- ----- ----- 1 460K 31.4G 26.1G 26.1G 460K 31.4G 26.1G 26.1G 2 33.8K 822M 580M 580M 69.8K 1.67G 1.17G 1.17G 4 2.34K 50.1M 31.3M 31.3M 10.8K 254M 156M 156M 8 417 12.8M 5.26M 5.26M 4.29K 134M 55.5M 55.5M 16 131 1.52M 555K 555K 2.70K 29.4M 9.54M 9.54M 32 45 62.5K 51K 51K 1.73K 2.48M 2.02M 2.02M 64 7 3.50K 3.50K 3.50K 602 301K 301K 301K 128 5 130K 130K 130K 991 17.3M 17.3M 17.3M Total 497K 32.3G 26.7G 26.7G 551K 33.5G 27.5G 27.5G dedup = 1.03, compress = 1.22, copies = 1.00, dedup * compress / copies = 1.25 and a vm pool root@radical:/home/krad# zdb -S vm1 Simulated DDT histogram: bucket allocated referenced ______ ______________________________ ______________________________ refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE ------ ------ ----- ----- ----- ------ ----- ----- ----- 1 1.28M 164G 102G 102G 1.28M 164G 102G 102G 2 17.8K 2.23G 1.21G 1.21G 40.9K 5.11G 2.75G 2.75G 4 1.25K 160M 85.7M 85.7M 5.70K 730M 374M 374M 8 176 22M 6.33M 6.33M 1.72K 220M 63.6M 63.6M 16 24 3M 1019K 1019K 540 67.5M 22.8M 22.8M 32 4 512K 71.5K 71.5K 163 20.4M 2.49M 2.49M 64 5 640K 22.5K 22.5K 383 47.9M 1.68M 1.68M 128 2 256K 12K 12K 342 42.8M 1.92M 1.92M 256 2 256K 9K 9K 715 89.4M 3.14M 3.14M 512 1 128K 4.50K 4.50K 570 71.2M 2.50M 2.50M Total 1.30M 167G 103G 103G 1.33M 171G 105G 105G dedup = 1.02, compress = 1.62, copies = 1.00, dedup * compress / copies = 1.65 Having said all that as everyone else has said I wouldnt thought it would be much use on a desktop, and 8 GB of ram it will run fine. Limiting arc as people have suggested, shouldnt really be necessary as its mainly there to stop memory fragmentation, and a few special application cases, generally though it just works and you can leave it alone. On 11 October 2013 16:58, Zaphod Beeblebrox wrote: > At home I have a large ZFS server serving up 18x 2T disks (raidZ1 x2). The > server machine is core2-duo with 8G RAM and an 'em' GigE added (because the > motherboard GigE was crap). I use SATA port expanders to multiply the 6 > motherboard ports. The machine has a UFS boot and there is a bit of a > "fight" when something uses memory and/or UFS to do something. > > I would have had ZFS on my desktop, but instead I use a diskless NFS setup > from the ZFS server. Originally, when I setup these systems, the NFS > diskless desktop was avoiding the conflict between the nvidia driver being > 32 bit only and ZFS liking more memory. This is long moot, but the setup > remains. The only remaining task here is to use NFSv4 diskless. I'm not > sure how easy/hard that is. The root mount seems to be NFSv3. > > As for ZFS on desktops, all the PC-BSD setups I've done for the family are > ZFS. They typically have either 2 or 4 gig of RAM. I haven't really tried > a low memory ZFS (say 512 meg). They seem to run acceptably well, but they > aren't really challenging anything --- running a FreeBSD/gnome or > FreeBSD/KDE desktop on a machine with 2 or 4 gig of RAM is really not a > challenge. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 11:47:20 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 1040B268 for ; Tue, 15 Oct 2013 11:47:20 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-wi0-x235.google.com (mail-wi0-x235.google.com [IPv6:2a00:1450:400c:c05::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8F1A92919 for ; Tue, 15 Oct 2013 11:47:19 +0000 (UTC) Received: by mail-wi0-f181.google.com with SMTP id l12so4856076wiv.2 for ; Tue, 15 Oct 2013 04:47:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=0f+3xdJ5Z/4nIq+6+Lf3h39Iv7FeziM6Np2SnbX+X7I=; b=Y1c57FJESa2iQRcQPmMtwhonhXxqv98eHP/p7S/6SQOWEOYxxwlApda9NjYzkLPTEj NtlVUQJd+uSPO0Qp7vAPYvTB51yqhAzw+1JqrRrFa+MhlGBl+X84m4Qfshbf8pFOFmv4 HF/L2INyvX/1yidqUuomiF0rQe58D8N8amJzMmTfJGJHYaV3JUM8+BHBn52sQtSYMvlj LfjALisCKEnFNz3S7tgri0jlR41BDG8YfSFqcadl+HZZOI9G3QCYbNoD4AKT3/44hJF8 V0GfBWbDPUa02WHsCfuVIxzmANHSUq+1DLyn2NELLpdTcuti7emYs0c1u3dAisDqyD1y jVag== MIME-Version: 1.0 X-Received: by 10.194.109.35 with SMTP id hp3mr1570644wjb.55.1381837637517; Tue, 15 Oct 2013 04:47:17 -0700 (PDT) Received: by 10.216.67.16 with HTTP; Tue, 15 Oct 2013 04:47:16 -0700 (PDT) In-Reply-To: References: <1345367028.18318718.1378328159276.JavaMail.root@uoguelph.ca> Date: Tue, 15 Oct 2013 19:47:16 +0800 Message-ID: Subject: Re: fixing "umount -f" for the NFS client From: Marcelo Araujo To: Benjamin Kaduk Content-Type: multipart/mixed; boundary=089e010d857485b28b04e8c62591 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: araujo@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 11:47:20 -0000 --089e010d857485b28b04e8c62591 Content-Type: text/plain; charset=ISO-8859-1 2013/9/5 Benjamin Kaduk > On Wed, 4 Sep 2013, Rick Macklem wrote: > > Benjamin Kaduk wrote: >> >>> >>> I think there are spare vfsops fields, so the MFC can be done in an >>> ABI-compatible way. The new routine is for optional functionality, >>> so it >>> seems fine. >>> >>> There are spares vfs ops in 10/current, but not in stable/9. An MFC will >> result in a VFS ABI change. (Since 10.0 hasn't been released yet, I didn't >> use one of the recently added spares.) >> > > Oh, right, I was looking at 10/current. > > Unless there are pressing calls for the feature in the stable branches, > it's probably best to hold off on the MFC, then. OpenAFS has encountered a > few KBI incompatibilities over the years (mostly in the networking bits, if > I remember correctly), and we can deal in the future, but not having to is > nice. > > Hello Guys, Is it possible to have it on 9-STABLE? I tried to port the changes of revision 255136 made by rmacklem@ to a 9.1-RELEASE but the bug is still there. * * Any change to make it works on 9.1, 9.2 or 9-STABLE? The patch attached is based on 9.1-RELEASE. Best Regards, -- Marcelo Araujo araujo@FreeBSD.org --089e010d857485b28b04e8c62591 Content-Type: application/octet-stream; name="nfs_purge.patch" Content-Disposition: attachment; filename="nfs_purge.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hmt2flmd0 ZGlmZiAtciBmYTM4NDIwNWJiYjcgc3lzL2ZzL25mc2NsaWVudC9uZnNfY2x2ZnNvcHMuYwotLS0g YS9zeXMvZnMvbmZzY2xpZW50L25mc19jbHZmc29wcy5jCVdlZCBPY3QgMDkgMTc6NDA6MjEgMjAx MyArMDgwMAorKysgYi9zeXMvZnMvbmZzY2xpZW50L25mc19jbHZmc29wcy5jCVR1ZSBPY3QgMTUg MTk6NDE6NDcgMjAxMyArMDgwMApAQCAtMTE1LDYgKzExNSw3IEBACiBzdGF0aWMgdmZzX3N0YXRm c190IG5mc19zdGF0ZnM7CiBzdGF0aWMgdmZzX3N5bmNfdCBuZnNfc3luYzsKIHN0YXRpYyB2ZnNf c3lzY3RsX3QgbmZzX3N5c2N0bDsKK3N0YXRpYyB2ZnNfcHVyZ2VfdCBuZnNfcHVyZ2U7CiAKIC8q CiAgKiBuZnMgdmZzIG9wZXJhdGlvbnMuCkBAIC0xMjksNiArMTMwLDcgQEAKIAkudmZzX3VuaW5p dCA9CQluY2xfdW5pbml0LAogCS52ZnNfdW5tb3VudCA9CQluZnNfdW5tb3VudCwKIAkudmZzX3N5 c2N0bCA9CQluZnNfc3lzY3RsLAorICAgICAgIC52ZnNfcHVyZ2UgPSAgICAgICAgIG5mc19wdXJn ZSwKIH07CiBWRlNfU0VUKG5mc192ZnNvcHMsIG5mcywgVkZDRl9ORVRXT1JLKTsKIApAQCAtMTYw Miw4ICsxNjA0LDIyIEBACiB9CiAKIC8qCisgKiBQdXJnZSBhbnkgUlBDcyBpbiBwcm9ncmVzcywg c28gdGhhdCB0aGV5IHdpbGwgYWxsIHJldHVybiBlcnJvcnMuCisgKiBUaGlzIGFsbG93cyBkb3Vu bW91bnQoKSB0byBjb250aW51ZSBhcyBmYXIgYXMgVkZTX1VOTU9VTlQoKSBmb3IgYQorICogZm9y Y2VkIGRpc21vdW50LgorICovCitzdGF0aWMgdm9pZAorbmZzX3B1cmdlKHN0cnVjdCBtb3VudCAq bXApCit7CisJc3RydWN0IG5mc21vdW50ICpubXAgPSBWRlNUT05GUyhtcCk7CisKKwluZXduZnNf bm1jYW5jZWxyZXFzKG5tcCk7Cit9CisKKy8qCiAgKiBFeHRyYWN0IHRoZSBpbmZvcm1hdGlvbiBu ZWVkZWQgYnkgdGhlIG5sbSBmcm9tIHRoZSBuZnMgdm5vZGUuCiAgKi8KKwogc3RhdGljIHZvaWQK IG5mc19nZXRubG1pbmZvKHN0cnVjdCB2bm9kZSAqdnAsIHVpbnQ4X3QgKmZocCwgc2l6ZV90ICpm aGxlbnAsCiAgICAgc3RydWN0IHNvY2thZGRyX3N0b3JhZ2UgKnNwLCBpbnQgKmlzX3YzcCwgb2Zm X3QgKnNpemVwLApkaWZmIC1yIGZhMzg0MjA1YmJiNyBzeXMva2Vybi92ZnNfbW91bnQuYwotLS0g YS9zeXMva2Vybi92ZnNfbW91bnQuYwlXZWQgT2N0IDA5IDE3OjQwOjIxIDIwMTMgKzA4MDAKKysr IGIvc3lzL2tlcm4vdmZzX21vdW50LmMJVHVlIE9jdCAxNSAxOTo0MTo0NyAyMDEzICswODAwCkBA IC0xMjY4LDggKzEyNjgsMTYgQEAKIAl9CiAJbXAtPm1udF9rZXJuX2ZsYWcgfD0gTU5US19VTk1P VU5UIHwgTU5US19OT0lOU01OVFE7CiAJLyogQWxsb3cgZmlsZXN5c3RlbXMgdG8gZGV0ZWN0IHRo YXQgYSBmb3JjZWQgdW5tb3VudCBpcyBpbiBwcm9ncmVzcy4gKi8KLQlpZiAoZmxhZ3MgJiBNTlRf Rk9SQ0UpCisJaWYgKGZsYWdzICYgTU5UX0ZPUkNFKSB7CiAJCW1wLT5tbnRfa2Vybl9mbGFnIHw9 IE1OVEtfVU5NT1VOVEY7CisJCU1OVF9JVU5MT0NLKG1wKTsKKwkJLyoKKwkJICogTXVzdCBiZSBk b25lIGFmdGVyIHNldHRpbmcgTU5US19VTk1PVU5URiBhbmQgYmVmb3JlCisJCSAqIHdhaXRpbmcg Zm9yIG1udF9sb2NrcmVmIHRvIGJlY29tZSAwLgorCQkgKi8KKwkJVkZTX1BVUkdFKG1wKTsKKwkJ TU5UX0lMT0NLKG1wKTsKKwl9CiAJZXJyb3IgPSAwOwogCWlmIChtcC0+bW50X2xvY2tyZWYpIHsK IAkJbXAtPm1udF9rZXJuX2ZsYWcgfD0gTU5US19EUkFJTklORzsKZGlmZiAtciBmYTM4NDIwNWJi Yjcgc3lzL3N5cy9tb3VudC5oCi0tLSBhL3N5cy9zeXMvbW91bnQuaAlXZWQgT2N0IDA5IDE3OjQw OjIxIDIwMTMgKzA4MDAKKysrIGIvc3lzL3N5cy9tb3VudC5oCVR1ZSBPY3QgMTUgMTk6NDE6NDcg MjAxMyArMDgwMApAQCAtNjM0LDYgKzYzNCw3IEBACiB0eXBlZGVmIGludCB2ZnNfc3lzY3RsX3Qo c3RydWN0IG1vdW50ICptcCwgZnNjdGxvcF90IG9wLAogCQkgICAgc3RydWN0IHN5c2N0bF9yZXEg KnJlcSk7CiB0eXBlZGVmIHZvaWQgdmZzX3N1c3BfY2xlYW5fdChzdHJ1Y3QgbW91bnQgKm1wKTsK K3R5cGVkZWYgdm9pZCB2ZnNfcHVyZ2VfdChzdHJ1Y3QgbW91bnQgKm1wKTsKIAogc3RydWN0IHZm c29wcyB7CiAJdmZzX21vdW50X3QJCSp2ZnNfbW91bnQ7CkBAIC02NTEsNiArNjUyLDcgQEAKIAl2 ZnNfZXh0YXR0cmN0bF90CSp2ZnNfZXh0YXR0cmN0bDsKIAl2ZnNfc3lzY3RsX3QJCSp2ZnNfc3lz Y3RsOwogCXZmc19zdXNwX2NsZWFuX3QJKnZmc19zdXNwX2NsZWFuOworCXZmc19wdXJnZV90CQkq dmZzX3B1cmdlOwogfTsKIAogdmZzX3N0YXRmc190CV9fdmZzX3N0YXRmczsKQEAgLTcxMiw2ICs3 MTQsMTMgQEAKIAkJbXR4X2Fzc2VydCgmR2lhbnQsIE1BX09XTkVEKTsJCQkJXAogfSB3aGlsZSAo MCkKIAorI2RlZmluZSBWRlNfUFVSR0UoTVApIGRvIHsgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgXAorCWlmICgqKE1QKS0+bW50X29wLT52ZnNfcHVyZ2UgIT0g TlVMTCkgeyAgICAgICAgICAgICAgICAgICAgICAgICBcCisJCSgqKE1QKS0+bW50X29wLT52ZnNf cHVyZ2UpKE1QKTsgICAgICAgICAgICAgICAgICAgICAgICAgXAorCX0gICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBcCit9IHdoaWxl ICgwKQorCisKICNkZWZpbmUgVkZTX0tOT1RFX0xPQ0tFRCh2cCwgaGludCkgZG8JCQkJCVwKIHsJ CQkJCQkJCQlcCiAJaWYgKCgodnApLT52X3ZmbGFnICYgVlZfTk9LTk9URSkgPT0gMCkJCQkJXAo= --089e010d857485b28b04e8c62591-- From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 12:54:13 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 38F3FD14 for ; Tue, 15 Oct 2013 12:54:13 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from cpsmtpb-ews01.kpnxchange.com (cpsmtpb-ews01.kpnxchange.com [213.75.39.4]) by mx1.freebsd.org (Postfix) with ESMTP id C58142023 for ; Tue, 15 Oct 2013 12:54:12 +0000 (UTC) Received: from cpsps-ews09.kpnxchange.com ([10.94.84.176]) by cpsmtpb-ews01.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Tue, 15 Oct 2013 14:53:04 +0200 Received: from CPSMTPM-TLF102.kpnxchange.com ([195.121.3.5]) by cpsps-ews09.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Tue, 15 Oct 2013 14:53:04 +0200 Received: from sjakie.klop.ws ([212.182.167.131]) by CPSMTPM-TLF102.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Tue, 15 Oct 2013 14:53:04 +0200 Received: from 212-182-167-131.ip.telfort.nl (localhost [127.0.0.1]) by sjakie.klop.ws (Postfix) with ESMTP id 3AD6534CF for ; Tue, 15 Oct 2013 14:53:04 +0200 (CEST) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: freebsd-fs@freebsd.org Subject: Re: Questions re swap-on-zfs References: <1380880223590-5848720.post@n5.nabble.com> <524EEE40.5060208@gmail.com> <1381060830753-5849397.post@n5.nabble.com> <20131006123350.GJ3287@sludge.elizium.za.net> <1381683172518-5851539.post@n5.nabble.com> Date: Tue, 15 Oct 2013 14:53:04 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <1381683172518-5851539.post@n5.nabble.com> User-Agent: Opera Mail/12.16 (FreeBSD) X-OriginalArrivalTime: 15 Oct 2013 12:53:04.0634 (UTC) FILETIME=[7D1FB5A0:01CEC9A5] X-RcptDomain: freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 12:54:13 -0000 On Sun, 13 Oct 2013 18:52:52 +0200, Beeblebrox wrote: > Thanks everyone for the assistance and the explanation for why swap "pri" > will not work. > > @Ronald Klop: Wouldn't going through ccd(4) slow the throughput to the > HDD > by reason of extra processing layer? I want to make sure there is close > to > none of performance penalty before implementing. Theoretically yes, but in practice I think you will not notice a difference. You can also use gconcat for about the same effect. Remind that you will loose all swap if one of the concatenated disks fails. Ronald. > > Thanks. > > > > > > ----- > FreeBSD-9.2-stable_amd64_root-on-zfs_clang-only-world > -- > View this message in context: > http://freebsd.1045724.n5.nabble.com/Questions-re-swap-on-zfs-tp5848720p5851539.html > Sent from the freebsd-fs mailing list archive at Nabble.com. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 13:06:02 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 29D7B61D; Tue, 15 Oct 2013 13:06:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id D073B2123; Tue, 15 Oct 2013 13:06:01 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ArgEAFo8XVKDaFve/2dsb2JhbABahBGDKb1kgRSBOHSCJQEBBAEjVhsYAgINGQJZGYgABqsKkkGBKY1tNAcWglSBOwORXpgog0AggW0 X-IronPort-AV: E=Sophos;i="4.93,498,1378872000"; d="scan'208";a="59437952" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 15 Oct 2013 09:05:55 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 4E8A5B4064; Tue, 15 Oct 2013 09:05:55 -0400 (EDT) Date: Tue, 15 Oct 2013 09:05:55 -0400 (EDT) From: Rick Macklem To: araujo@FreeBSD.org Message-ID: <748522744.41194273.1381842355314.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: fixing "umount -f" for the NFS client MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 13:06:02 -0000 araujo wrote: > > > > > > > 2013/9/5 Benjamin Kaduk < kaduk@mit.edu > > > > > On Wed, 4 Sep 2013, Rick Macklem wrote: > > > > > Benjamin Kaduk wrote: > > > > I think there are spare vfsops fields, so the MFC can be done in an > ABI-compatible way. The new routine is for optional functionality, > so it > seems fine. > > There are spares vfs ops in 10/current, but not in stable/9. An MFC > will > result in a VFS ABI change. (Since 10.0 hasn't been released yet, I > didn't > use one of the recently added spares.) > > Oh, right, I was looking at 10/current. > > Unless there are pressing calls for the feature in the stable > branches, it's probably best to hold off on the MFC, then. OpenAFS > has encountered a few KBI incompatibilities over the years (mostly > in the networking bits, if I remember correctly), and we can deal in > the future, but not having to is nice. > > > > > Hello Guys, > > > Is it possible to have it on 9-STABLE? > I tried to port the changes of revision 255136 made by rmacklem@ to a > 9.1-RELEASE but the bug is still there. > > > Any change to make it works on 9.1, 9.2 or 9-STABLE? > > The patch attached is based on 9.1-RELEASE. > The patch looks ok at a glance. Note that it can take up to 2-3minutes for a forced dismount to complete, depending on where the threads are waiting. If the mount is still there 5minutes after doing "umount -f", do a "ps axhl" and post the output of that to me. It may be getting stuck somewhere else than where I've seen during testing. rick > > Best Regards, -- > Marcelo Araujo > araujo@FreeBSD.org From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 13:34:32 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6EE9A284 for ; Tue, 15 Oct 2013 13:34:32 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 04E092407 for ; Tue, 15 Oct 2013 13:34:31 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AokNAHdDXVKDaFve/2dsb2JhbABagz9Sgym+L0uBOXQEgiEBAQEDAQEBASArIAsFFhgCAg0ZAikBCSYGCAcEARwEh18GDKp7kj6BKYxoDXgBMweCaoE7A5U5g3qQU4NAIDF8Bxci X-IronPort-AV: E=Sophos;i="4.93,499,1378872000"; d="scan'208";a="60207430" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 15 Oct 2013 09:34:30 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 72F13B4037; Tue, 15 Oct 2013 09:34:30 -0400 (EDT) Date: Tue, 15 Oct 2013 09:34:30 -0400 (EDT) From: Rick Macklem To: Rick Romero Message-ID: <124950221.41251023.1381844070456.JavaMail.root@uoguelph.ca> In-Reply-To: <20131014200458.Horde.QjqQXEfm9A5k9357kfSZGQ1@beta.vfemail.net> Subject: Re: NFS locks, rpcbind port = 0 failed? - try #2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 13:34:32 -0000 Rick Romero wrote: > Quoting Rick Macklem : >=20 > > Rick Romero wrote: > >> This is a continuation of "9.1 VM nfs3 & locks over VPN" from > >> freebsd-questions - trying a > >> different angle maybe it'll jostle someones memory.=C2=A0 Don't mean t= o > >> cross-post, but as I pay more attention to the lists I'm reading, > >> this > >> seems to be the better list for NFS issues. > >> > >> I have a FreeBSD 9.2 VM at an offsite hosting company.=C2=A0 hostname > >> nl101vpn > >> OpenVPN is installed on it, routed not bridged mode. > >> I have multiple OSs installed on local network. I'm already > >> exportings NFS > >> off 9.1 with working file locks. > >> > >> What I see - > >> export nfsv3 or nfsv4 from nl101vpn, mount on local FreeBSD or > >> Linux > >> - > >> locks do not work. > >> export nfsv3 from any local system, mount on nl101vpn - locks > >> work. > >> export nfsv3 from locally installed VM, mount on any local host or > >> nl101vpn > >> - locks work.=C2=A0 No OpenVPN installed on it though. This was to tes= t > >> if > >> virtio drivers might be causing the problem. > >> > >> I even ran a tcpdump to see if something was getting lost - both > >> sides > >> match, nothing is getting dropped > >> > >> nl101vpn - /var/log/messages: > >> Oct 14 12:21:01 nl101 kernel: NLM: failed to contact remote > >> rpcbind, > >> stat =3D > >> 0, port =3D 0=C2=A0 (why port 0?) > >> Oct 14 12:23:02 nl101 last message repeated 109 times > >> Oct 14 12:25:48 nl101 last message repeated 177 times > >> > >> I tried binding rpcbind to the VPN interface, but that doesn't > >> seem > >> to > >> work.=C2=A0 tcpdump shows no packets trying to leave the 'Internet' > >> interface. > >> > >> So I haven't exhausted every combination, or completely 100% > >> replicated > >> whats happening offsite, but it's getting pretty ridiculous now... > >> I'm > >> lost, and I need NFS locking to work. > >> Help :) > > > > For rpcbind to work, IP broadcast needs to work between the hosts > > and I suspect that the VPN doesn't support that. > > > > Without rpcbind, I don't think you can get rpc.lockd/rpc.statd > > to work, but I am not sure. (There are command line options for > > these daemons that allow you to set specific port #s, but I don't > > think that will fix the problem, since they still need rpcbind to > > tell them the port# for the remote machines.) These protocols were > > designed in the 1980s for use on a LAN. > > > > Now, nfsv4 shouldn't care less about rpcbind, rpc.lockd. NFSv4 > > locking > > is handled as a part of the NFSv4 protocol and always uses port > > #2049. > > I'd suggest you try NFSv4 again and make sure it is using NFSv4 and > > the mount has not fallen back to NFSv3. (For FreeBSD, specify > > "nfsv4" > > as a mount option. For Linux, specify "vers=3D4" as a mount option.) > > You can check what the mount is actually using via "nfsstat -m". > > If you assumed the locking for NFSv4 wasn't working because of > > these > > messages, that isn't the case. If you are using NFSv4 for all > > mounts, > > you don't need to run rpc.lockd at all (at least for FreeBSD, I'm > > not sure what the daemons do w.r.t. Linux). >=20 > Hi Rick, >=20 > Yeah - I thought the VPN might pose a problem, but I can get locks > from the > VM side (nl101vpn) via NFS3 back to the main site.=C2=A0 So it doesn't > seem to > be an issue with the VPN. After that I created a local VM to ensure > it > wasn't a virtio thing, and then upgraded the remote VM to 9.2 (to > rule out > any funky custom options the host may have thrown into their 9.1 > installer).=C2=A0 nada. >=20 > So I'm re-trying with NFS4 - though my mount does show it was mounted > nfs4 > (from Linux) last time I tried: > nl101vpn:/first on /mnt type nfs4 > (rw,relatime,vers=3D4,rsize=3D65536,wsize=3D65536,namlen=3D255,hard,proto= =3Dtcp,timeo=3D600,retrans=3D2,sec=3Dsys,clientaddr=3D172.16.1.92,minorvers= ion=3D0,local_lock=3Dnone,addr=3D10.9.8.6) >=20 > After trying again, still doesn't work.=C2=A0 Though now I noticed the > error is > different.=C2=A0 I have a little perl script that I test with, and the > line is: > flock(LOCKFILE, LOCK_SH) or die "Can't get shared lock on $lock_file: > $!\n"; > Before I would get 'pemission denied' - which (IIRC) would also > happen when > I forgot to run lockd or statd. > Now with NFS4 it says, 'Bad file descriptor' >=20 Well NFSv4 supports POSIX byte range locking (the fcntl() stuff), so if you can switch your testing/apps to that, you might find it works. (I have no idea what a Linux nfs4 mount will do with a flock() call.) > After much more testing I've gotten a single Linux VM (but no other > VMs on > the same host, or my other host, yet they're all from the same > template) to > get a lock (n NFSv3) > Failed locks show in the logs: > NLM: failed to contact remote rpcbind, stat =3D 0, port =3D 0 > =C2=A0=C2=A0=C2=A0 (FreeBSD) > NLM: failed to contact remote rpcbind, stat =3D 7, port =3D 28416 > =C2=A0 (Linux) >=20 > I have 2 FreeBSD boxes, 9.1 and 7.2.=C2=A0 Both don't seem to be relaying > their > ports? > The Linux ones that fail have a port, but apparently can't be > contacted. Or > is 'port' in the log not really referring to a port number? >=20 > The OpenVPN 'server' is another Linux VM - but on a different host > than the > working Linux VM :P=C2=A0 None of the Linux VMs on that host can get a > lock. > How's THAT for weird? :) >=20 > I need some aspirin. >=20 Maybe someone else can suggest something that will help. If your machines/VMs have multiple IP host addresses, the "-h" option can be useful on all these daemons. I'll also mention that, although I don't know the NSM (rpc.statd) well, my basic understanding of it is that it "pings" other hosts to see if they are "up" (sometimes using IP broadcast, I think?). If it doesn't get a response, it marks the host "down" and then rpc.lockd doesn't use that host. I've never trusted rpc.lockd and friends, but others have found them useful. I believe a reliable LAN/WAN with support for IP broadcast and no port# blocking is needed for it to have any chance of working reliably. rick > Rick > =C2=A0 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 14:06:43 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 5E5B41FA for ; Tue, 15 Oct 2013 14:06:43 +0000 (UTC) (envelope-from krichy@cflinux.hu) Received: from pi.nmdps.net (pi.nmdps.net [IPv6:2a01:be00:10:201:0:80:0:1]) by mx1.freebsd.org (Postfix) with ESMTP id 2358626EF for ; Tue, 15 Oct 2013 14:06:42 +0000 (UTC) Received: from pi.nmdps.net (pi.nmdps.net [109.61.102.5]) (Authenticated sender: krichy@cflinux.hu) by pi.nmdps.net (Postfix) with ESMTPSA id 0C6D21639 for ; Tue, 15 Oct 2013 16:06:34 +0200 (CEST) Date: Tue, 15 Oct 2013 16:06:31 +0200 (CEST) From: Richard Kojedzinszky X-X-Sender: krichy@pi.nmdps.net To: freebsd-fs@freebsd.org Subject: zfs zvol fsync() Message-ID: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 14:06:43 -0000 Dear FS devs, I run in to the problem that it seems that when a zvol is opened, and an fsync() is issued to is, nothing happens. Mainly I use a zvol to export it as an iscsi volume, and net/istgt issues fsync() to the device upon a scsi synchronize cache request. Any comments on this? Thanks in advance, Kojedzinszky Richard From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 14:31:39 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 155D4ABD; Tue, 15 Oct 2013 14:31:39 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-pd0-x22f.google.com (mail-pd0-x22f.google.com [IPv6:2607:f8b0:400e:c02::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id DD04B295A; Tue, 15 Oct 2013 14:31:38 +0000 (UTC) Received: by mail-pd0-f175.google.com with SMTP id q10so8845697pdj.6 for ; Tue, 15 Oct 2013 07:31:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:from:subject:date:to; bh=2tmgoXkfGHd0NTYFjEhti4rNlisHU8+UN99eHcHGiog=; b=m3L8FPjMFsfPWYQRZVFDfCoSyzwl6LO86tqDqrfMKy9L/E6oY4DjZeK0hQSZxQ4t8y QpFjeGOefaLkCG2co/2O0seCkOvX72pYshLtrReQhwOVlOElIcivApf6MBXwFu6fLJRW bF2mfo9Hie0Plw6azsG1LpRVgtgfp1P8ulNXY2ETGVYfKJTwNAFBoJxUzq/Z2H6SuaOv 0oI3JQ7X0tAT+gtaMoBtbmYnGsZVakaQFn9EczxWl2B5L0hXxPpRtI3xUu6Kol5uTOW4 yzKWreyzwtU0tZz/phTW31FqUz82zXz2Nv13bLaJp7EUmUseUgWVTB7bZTFJDmATRCKS 9sjA== X-Received: by 10.68.170.225 with SMTP id ap1mr8915855pbc.133.1381847498510; Tue, 15 Oct 2013 07:31:38 -0700 (PDT) Received: from [10.0.1.5] (host-219-68-127-128.dynamic.kbtelecom.net. [219.68.127.128]) by mx.google.com with ESMTPSA id wp8sm84968726pbc.26.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 15 Oct 2013 07:31:36 -0700 (PDT) References: <748522744.41194273.1381842355314.JavaMail.root@uoguelph.ca> Mime-Version: 1.0 (1.0) In-Reply-To: <748522744.41194273.1381842355314.JavaMail.root@uoguelph.ca> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: <8374DB4D-C659-4400-AFC9-8E56B692C71E@gmail.com> X-Mailer: iPhone Mail (10B146) From: araujobsdport@gmail.com Subject: Re: fixing "umount -f" for the NFS client Date: Tue, 15 Oct 2013 22:31:32 +0800 To: Rick Macklem Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 14:31:39 -0000 On 2013/10/15, at 21:05, Rick Macklem wrote: > araujo wrote: >>=20 >>=20 >>=20 >>=20 >>=20 >>=20 >> 2013/9/5 Benjamin Kaduk < kaduk@mit.edu > >>=20 >>=20 >>=20 >> On Wed, 4 Sep 2013, Rick Macklem wrote: >>=20 >>=20 >>=20 >>=20 >> Benjamin Kaduk wrote: >>=20 >>=20 >>=20 >> I think there are spare vfsops fields, so the MFC can be done in an >> ABI-compatible way. The new routine is for optional functionality, >> so it >> seems fine. >>=20 >> There are spares vfs ops in 10/current, but not in stable/9. An MFC >> will >> result in a VFS ABI change. (Since 10.0 hasn't been released yet, I >> didn't >> use one of the recently added spares.) >>=20 >> Oh, right, I was looking at 10/current. >>=20 >> Unless there are pressing calls for the feature in the stable >> branches, it's probably best to hold off on the MFC, then. OpenAFS >> has encountered a few KBI incompatibilities over the years (mostly >> in the networking bits, if I remember correctly), and we can deal in >> the future, but not having to is nice. >>=20 >>=20 >>=20 >>=20 >> Hello Guys, >>=20 >>=20 >> Is it possible to have it on 9-STABLE? >> I tried to port the changes of revision 255136 made by rmacklem@ to a >> 9.1-RELEASE but the bug is still there. >>=20 >>=20 >> Any change to make it works on 9.1, 9.2 or 9-STABLE? >>=20 >> The patch attached is based on 9.1-RELEASE. > The patch looks ok at a glance. Note that it can take > up to 2-3minutes for a forced dismount to complete, > depending on where the threads are waiting. >=20 > If the mount is still there 5minutes after doing > "umount -f", do a "ps axhl" and post the output > of that to me. It may be getting stuck somewhere > else than where I've seen during testing. Hello Rick, Thanks by the prompt reply, I'm gonna make more tests tomorrow, and give you= the output if necessary!=20 However, is there any way to improve this time to force the umount? >=20 > rick >=20 >>=20 >> Best Regards, -- >> Marcelo Araujo >> araujo@FreeBSD.org From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 15:17:54 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D5A243D4 for ; Tue, 15 Oct 2013 15:17:54 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ie0-x233.google.com (mail-ie0-x233.google.com [IPv6:2607:f8b0:4001:c03::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AAC352E1E for ; Tue, 15 Oct 2013 15:17:54 +0000 (UTC) Received: by mail-ie0-f179.google.com with SMTP id aq17so13904959iec.10 for ; Tue, 15 Oct 2013 08:17:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=g/d0jBD9Iy/9zK8A3u/mr4VN9CdFKXXYvZnC+JhMXVQ=; b=lUutqgtplQENOo0U/FChwUfRWB+F0SM1jx8pgSuQ90wdnU6w+Vhn+PQ7yIcXGaQ15R vZqsq9jk2vKaf/MZX3GEeG3/N0jnlqjiv1OBJoXc+m0Lv1QvwoovzEt1qAgap97ba111 2jBL4j5V4El+OAHyripSmb2u8U7fmi83hNhAKn9MdIQRl9LDqL5oNkK9/VnHir0jMuIV uryEiTvL37zemCBESehMTFVNTZS14dmfYFPbMZEmGWGtEN1uJwxiz8KMWwk6YQ690M3S Ghd7M4RHA1tFiWpjgHXC/cqyCjoRoBId6iGAKV+hrZn/iIZLZ/io5PeXzsHFam4BQGGS H2MA== MIME-Version: 1.0 X-Received: by 10.50.117.40 with SMTP id kb8mr17508871igb.60.1381850274078; Tue, 15 Oct 2013 08:17:54 -0700 (PDT) Sender: jdavidlists@gmail.com Received: by 10.43.180.131 with HTTP; Tue, 15 Oct 2013 08:17:53 -0700 (PDT) Date: Tue, 15 Oct 2013 11:17:53 -0400 X-Google-Sender-Auth: VF1eXMvrHxHFju05Y25BDcSgUs8 Message-ID: Subject: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page From: J David To: "freebsd-fs@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 15:17:55 -0000 What are the necessary loader.conf / kernel config invocations to make ZFS stable on 9.2-RELEASE i386 node with 3GB RAM? This machine was rock solid under 8.4, but since upgrading to 9.2 it has been a disaster. It crashes every few hours with "panic: pmap_enter: attempted pmap_enter on 4MB page." Here are a couple of stack traces: panic: pmap_enter: attempted pmap_enter on 4MB page cpuid = 0 KDB: stack backtrace: #0 0xc0afc092 at kdb_backtrace+0x52 #1 0xc0ac249c at panic+0x1bc #2 0xc0f1825d at pmap_enter+0x63d #3 0xc0d34ee5 at vm_fault_hold+0x1c45 #4 0xc0d33262 at vm_fault+0x82 #5 0xc0f1f5d6 at trap_pfault+0x186 #6 0xc0f1ed4b at trap+0x51b #7 0xc0f0887c at calltrap+0x6 #8 0xc16770b6 at zio_execute+0x116 #9 0xc15d9460 at taskq_run_safe+0x10 #10 0xc0b08f26 at taskqueue_run_locked+0xe6 #11 0xc0b097f7 at taskqueue_thread_loop+0xb7 #12 0xc0a90a73 at fork_exit+0xa3 #13 0xc0f08924 at fork_trampoline+0x8 panic: pmap_enter: attempted pmap_enter on 4MB page cpuid = 1 KDB: stack backtrace: #0 0xc0afc092 at kdb_backtrace+0x52 #1 0xc0ac249c at panic+0x1bc #2 0xc0f1825d at pmap_enter+0x63d #3 0xc0d34ee5 at vm_fault_hold+0x1c45 #4 0xc0d33262 at vm_fault+0x82 #5 0xc0f1f5d6 at trap_pfault+0x186 #6 0xc0f1ed4b at trap+0x51b #7 0xc0f0887c at calltrap+0x6 #8 0xc1679bdf at zio_dva_allocate+0x9f #9 0xc16770b6 at zio_execute+0x116 #10 0xc15d9460 at taskq_run_safe+0x10 #11 0xc0b08f26 at taskqueue_run_locked+0xe6 #12 0xc0b097f7 at taskqueue_thread_loop+0xb7 #13 0xc0a90a73 at fork_exit+0xa3 #14 0xc0f08924 at fork_trampoline+0x8 For the last crash, top was running and this is what was on the screen when it died: CPU: 0.6% user, 0.0% nice, 2.2% system, 0.2% interrupt, 97.1% idle Mem: 442M Active, 91M Inact, 79M Wired, 3764K Cache, 4960K Buf, 2380M Free ARC: 40M Total, 8520K MFU, 30M MRU, 304K Anon, 494K Header, 1062K Other Swap: 4096M Total, 4096M Free Write failed: Broken pipe It doesn't seem to matter what KVA_PAGES, vm.kmem_size, vfs.zfs.arc_max or vfs.zfs.vdev.cache.size is set to, and the ZFS tuning guides in the wiki (albeit appearing to be dating to the 7.x era) provides guidelines for tuning down to 768M. So 3GB should be enough for a machine that is 99% idle. (Particularly given that it dies with >2GB free.) This seems to be specific to i386, this problem hasn't cropped up on any amd64 nodes. No compression, deduplication, snapshots or anything like that is in use. Thanks for any suggestions! From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 15:29:25 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D6C9A875 for ; Tue, 15 Oct 2013 15:29:25 +0000 (UTC) (envelope-from rick@havokmon.com) Received: from smtp101-5.vfemail.net (nine.vfemail.net [108.76.175.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7CD232F26 for ; Tue, 15 Oct 2013 15:29:25 +0000 (UTC) Received: (qmail 27680 invoked by uid 89); 15 Oct 2013 15:29:22 -0000 Received: by simscan 1.4.0 ppid: 27674, pid: 27677, t: 0.0932s scanners:none Received: from unknown (HELO www110) (cmlja0BoYXZva21vbi5jb20=@172.16.100.92) by 172.16.100.61 with ESMTPA; 15 Oct 2013 15:29:22 -0000 Received: from rrcs-98-103-53-237.central.biz.rr.com (rrcs-98-103-53-237.central.biz.rr.com [98.103.53.237]) by beta.vfemail.net (Horde Framework) with HTTP; Tue, 15 Oct 2013 10:29:22 -0500 Date: Tue, 15 Oct 2013 10:29:22 -0500 Message-ID: <20131015102922.Horde.Wr6o_T_ZY1aX0Z7nkPNeAw2@beta.vfemail.net> From: Rick Romero To: freebsd-fs@freebsd.org Subject: Re: NFS locks, rpcbind port = 0 failed? - try #2 References: <124950221.41251023.1381844070456.JavaMail.root@uoguelph.ca> In-Reply-To: <124950221.41251023.1381844070456.JavaMail.root@uoguelph.ca> User-Agent: Internet Messaging Program (IMP) H5 (6.1.4) X-VFEmail-Originating-IP: OTguMTAzLjUzLjIzNw== X-VFEmail-Remote-Browser: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0 @ X-VFEmail-AntiSpam: Notify admin@vfemail.net of any spam, and include VFEmail headers X-Remote-Browser: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes Content-Transfer-Encoding: 8bit Content-Disposition: inline Content-Description: Plaintext Message X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 15:29:26 -0000 Quoting Rick Macklem : > Rick Romero wrote: >> Quoting Rick Macklem : >> >> Rick Romero wrote: >> This is a continuation of "9.1 VM nfs3 & locks over VPN" from >> freebsd-questions - trying a >> different angle maybe it'll jostle someones memory.  Don't mean to >> cross-post, but as I pay more attention to the lists I'm reading, >> this >> seems to be the better list for NFS issues. >> >> I have a FreeBSD 9.2 VM at an offsite hosting company.  hostname >> nl101vpn >> OpenVPN is installed on it, routed not bridged mode. >> I have multiple OSs installed on local network. I'm already >> exportings NFS >> off 9.1 with working file locks. >> >> What I see - >> export nfsv3 or nfsv4 from nl101vpn, mount on local FreeBSD or >> Linux >> - >> locks do not work. >> export nfsv3 from any local system, mount on nl101vpn - locks >> work. >> export nfsv3 from locally installed VM, mount on any local host or >> nl101vpn >> - locks work.  No OpenVPN installed on it though. This was to test >> if >> virtio drivers might be causing the problem. >> >> I even ran a tcpdump to see if something was getting lost - both >> sides >> match, nothing is getting dropped >> >> nl101vpn - /var/log/messages: >> Oct 14 12:21:01 nl101 kernel: NLM: failed to contact remote >> rpcbind, >> stat = >> 0, port = 0  (why port 0?) >> Oct 14 12:23:02 nl101 last message repeated 109 times >> Oct 14 12:25:48 nl101 last message repeated 177 times >> >> I tried binding rpcbind to the VPN interface, but that doesn't >> seem >> to >> work.  tcpdump shows no packets trying to leave the 'Internet' >> interface. >> >> So I haven't exhausted every combination, or completely 100% >> replicated >> whats happening offsite, but it's getting pretty ridiculous now... >> I'm >> lost, and I need NFS locking to work. >> Help :) >> >> For rpcbind to work, IP broadcast needs to work between the hosts >> and I suspect that the VPN doesn't support that. >> >> Without rpcbind, I don't think you can get rpc.lockd/rpc.statd >> to work, but I am not sure. (There are command line options for >> these daemons that allow you to set specific port #s, but I don't >> think that will fix the problem, since they still need rpcbind to >> tell them the port# for the remote machines.) These protocols were >> designed in the 1980s for use on a LAN. >> >> Now, nfsv4 shouldn't care less about rpcbind, rpc.lockd. NFSv4 >> locking >> is handled as a part of the NFSv4 protocol and always uses port >> #2049. >> I'd suggest you try NFSv4 again and make sure it is using NFSv4 and >> the mount has not fallen back to NFSv3. (For FreeBSD, specify >> "nfsv4" >> as a mount option. For Linux, specify "vers=4" as a mount option.) >> You can check what the mount is actually using via "nfsstat -m". >> If you assumed the locking for NFSv4 wasn't working because of >> these >> messages, that isn't the case. If you are using NFSv4 for all >> mounts, >> you don't need to run rpc.lockd at all (at least for FreeBSD, I'm >> not sure what the daemons do w.r.t. Linux). >> >>   Hi Rick, >> >> Yeah - I thought the VPN might pose a problem, but I can get locks >> from the >> VM side (nl101vpn) via NFS3 back to the main site.  So it doesn't >> seem to >> be an issue with the VPN. After that I created a local VM to ensure >> it >> wasn't a virtio thing, and then upgraded the remote VM to 9.2 (to >> rule out >> any funky custom options the host may have thrown into their 9.1 >> installer).  nada. >> >> So I'm re-trying with NFS4 - though my mount does show it was mounted >> nfs4 >> (from Linux) last time I tried: >> nl101vpn:/first on /mnt type nfs4 >> (rw,relatime,vers=4,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.16.1.92,minorversion=0,local_lock=none,addr=10.9.8.6) >> >> After trying again, still doesn't work.  Though now I noticed the >> error is >> different.  I have a little perl script that I test with, and the >> line is: >> flock(LOCKFILE, LOCK_SH) or die "Can't get shared lock on $lock_file: >> $!\n"; >> Before I would get 'pemission denied' - which (IIRC) would also >> happen when >> I forgot to run lockd or statd. >> Now with NFS4 it says, 'Bad file descriptor' > > Well NFSv4 supports POSIX byte range locking (the fcntl() stuff), > so if you can switch your testing/apps to that, you might find > it works. (I have no idea what a Linux nfs4 mount will do with a > flock() call.) > >> After much more testing I've gotten a single Linux VM (but no other >> VMs on >> the same host, or my other host, yet they're all from the same >> template) to >> get a lock (n NFSv3) >> Failed locks show in the logs: >> NLM: failed to contact remote rpcbind, stat = 0, port = 0 >>     (FreeBSD) >> NLM: failed to contact remote rpcbind, stat = 7, port = 28416 >>   (Linux) >> >> I have 2 FreeBSD boxes, 9.1 and 7.2.  Both don't seem to be relaying >> their >> ports? >> The Linux ones that fail have a port, but apparently can't be >> contacted. Or >> is 'port' in the log not really referring to a port number? >> >> The OpenVPN 'server' is another Linux VM - but on a different host >> than the >> working Linux VM :P  None of the Linux VMs on that host can get a >> lock. >> How's THAT for weird? :) >> >> I need some aspirin. > > Maybe someone else can suggest something that will help. If your > machines/VMs have multiple IP host addresses, the "-h" option can > be useful on all these daemons. > > I'll also mention that, although I don't know the NSM (rpc.statd) > well, my basic understanding of it is that it "pings" other hosts > to see if they are "up" (sometimes using IP broadcast, I think?). > If it doesn't get a response, it marks the host "down" and then > rpc.lockd doesn't use that host. > > I've never trusted rpc.lockd and friends, but others have found > them useful. I believe a reliable LAN/WAN with support for IP > broadcast and no port# blocking is needed for it to have any chance > of working reliably. Long story short - while my perl script doesn't properly test for locks (thanks for that, I wouldn't have moved on without it) - the applications I need to run seem to be working correctly over NFSv4. Thanks for your time! Rick From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 16:45:49 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 28842877; Tue, 15 Oct 2013 16:45:49 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A74B125C8; Tue, 15 Oct 2013 16:45:48 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r9FGjcLr036440; Tue, 15 Oct 2013 19:45:38 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r9FGjcLr036440 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r9FGjblB036431; Tue, 15 Oct 2013 19:45:37 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 15 Oct 2013 19:45:37 +0300 From: Konstantin Belousov To: J David Subject: Re: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page Message-ID: <20131015164537.GH3865@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="8/pVXlBMPtxfSuJG" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: "freebsd-fs@freebsd.org" , alc@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 16:45:49 -0000 --8/pVXlBMPtxfSuJG Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Oct 15, 2013 at 11:17:53AM -0400, J David wrote: > What are the necessary loader.conf / kernel config invocations to make > ZFS stable on 9.2-RELEASE i386 node with 3GB RAM? >=20 > This machine was rock solid under 8.4, but since upgrading to 9.2 it > has been a disaster. It crashes every few hours with "panic: > pmap_enter: attempted pmap_enter on 4MB page." >=20 > Here are a couple of stack traces: >=20 > panic: pmap_enter: attempted pmap_enter on 4MB page > cpuid =3D 0 > KDB: stack backtrace: > #0 0xc0afc092 at kdb_backtrace+0x52 > #1 0xc0ac249c at panic+0x1bc > #2 0xc0f1825d at pmap_enter+0x63d > #3 0xc0d34ee5 at vm_fault_hold+0x1c45 > #4 0xc0d33262 at vm_fault+0x82 > #5 0xc0f1f5d6 at trap_pfault+0x186 > #6 0xc0f1ed4b at trap+0x51b > #7 0xc0f0887c at calltrap+0x6 > #8 0xc16770b6 at zio_execute+0x116 > #9 0xc15d9460 at taskq_run_safe+0x10 > #10 0xc0b08f26 at taskqueue_run_locked+0xe6 > #11 0xc0b097f7 at taskqueue_thread_loop+0xb7 > #12 0xc0a90a73 at fork_exit+0xa3 > #13 0xc0f08924 at fork_trampoline+0x8 >=20 > panic: pmap_enter: attempted pmap_enter on 4MB page > cpuid =3D 1 > KDB: stack backtrace: > #0 0xc0afc092 at kdb_backtrace+0x52 > #1 0xc0ac249c at panic+0x1bc > #2 0xc0f1825d at pmap_enter+0x63d > #3 0xc0d34ee5 at vm_fault_hold+0x1c45 > #4 0xc0d33262 at vm_fault+0x82 > #5 0xc0f1f5d6 at trap_pfault+0x186 > #6 0xc0f1ed4b at trap+0x51b > #7 0xc0f0887c at calltrap+0x6 > #8 0xc1679bdf at zio_dva_allocate+0x9f > #9 0xc16770b6 at zio_execute+0x116 > #10 0xc15d9460 at taskq_run_safe+0x10 > #11 0xc0b08f26 at taskqueue_run_locked+0xe6 > #12 0xc0b097f7 at taskqueue_thread_loop+0xb7 > #13 0xc0a90a73 at fork_exit+0xa3 > #14 0xc0f08924 at fork_trampoline+0x8 >=20 > For the last crash, top was running and this is what was on the screen > when it died: >=20 > CPU: 0.6% user, 0.0% nice, 2.2% system, 0.2% interrupt, 97.1% idle > Mem: 442M Active, 91M Inact, 79M Wired, 3764K Cache, 4960K Buf, 2380M Free > ARC: 40M Total, 8520K MFU, 30M MRU, 304K Anon, 494K Header, 1062K Other > Swap: 4096M Total, 4096M Free > Write failed: Broken pipe >=20 > It doesn't seem to matter what KVA_PAGES, vm.kmem_size, > vfs.zfs.arc_max or vfs.zfs.vdev.cache.size is set to, and the ZFS > tuning guides in the wiki (albeit appearing to be dating to the 7.x > era) provides guidelines for tuning down to 768M. So 3GB should be > enough for a machine that is 99% idle. (Particularly given that it > dies with >2GB free.) >=20 > This seems to be specific to i386, this problem hasn't cropped up on > any amd64 nodes. No compression, deduplication, snapshots or anything > like that is in use. It seems that i386 pmap_enter() cannot deal with the superpages, while amd64 can. On the other hand, I am not quite sure that this is your problem, because you get traps on accessing KVA, and I suspect that ZFS uses wired memory. Obtain the kernel dump and do full backtrace for the panic. Anyway, I believe that i386 pmap_enter() should do a demotion when needed. Below is the prototyped change for this. diff --git a/sys/i386/i386/pmap.c b/sys/i386/i386/pmap.c index 64bf1a3..91453da 100644 --- a/sys/i386/i386/pmap.c +++ b/sys/i386/i386/pmap.c @@ -3473,17 +3473,21 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_prot_t a= ccess, vm_page_t m, PMAP_LOCK(pmap); sched_pin(); =20 - /* - * In the case that a page table page is not - * resident, we are creating it here. - */ - if (va < VM_MAXUSER_ADDRESS) { + pde =3D pmap_pde(pmap, va); + if ((*pde & PG_PS) !=3D 0) { + /* PG_V is asserted by pmap_demote_pde */ + pmap_demote_pde(pmap, pde, va); + if (va < VM_MAXUSER_ADDRESS) { + mpte =3D PHYS_TO_VM_PAGE(*pde & PG_FRAME); + mpte->wire_count++; + } + } else if (va < VM_MAXUSER_ADDRESS) { + /* + * In the case that a page table page is not resident, + * we are creating it here. + */ mpte =3D pmap_allocpte(pmap, va, M_WAITOK); } - - pde =3D pmap_pde(pmap, va); - if ((*pde & PG_PS) !=3D 0) - panic("pmap_enter: attempted pmap_enter on 4MB page"); pte =3D pmap_pte_quick(pmap, va); =20 /* --8/pVXlBMPtxfSuJG Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJSXXEwAAoJEJDCuSvBvK1BcTQP/j2cVxIx9ic7Cqqq5F4jPWid /3wT3wxmZsfjrMlu3FGHEUtyx4G9EmF7UVIhiQcA9Chf2GQNHoYowUaFT/DK6zY/ zlNe/p7Evgx4xNg94Rl5/bMDSvjNkwIiqHvp43tJbxGaKANhZt0kzK0lqEux6h0Q VGjpXMw9ERJrcjj3eQQgVvDOOvyaEyGgzrYIIebWN5lL34Bz/c9W3k5XezlTYQ96 dJevealL0u93D2ptWKzUs41Xnrgt+etRSiDX7tRG0CVoZ6eYuH6FigzTEjIkuF7n DJie+eza1b8KZ6Aeeo7wlI4FxiI/x37CVTC3LrMu4l6F83f/nvQbYwI6QnhZiydz gSG7zHh/l0ifNavjNjAMcKOIhNfEU1+OEqSzJ3Wxw7q3a3ZnAMhix7frNT1h8eHi PDQCHa5ZlGw/h3vJ1OZbOnq0AgEjBcn01vfO85fz+FX19gqI15aVIqN26ISqMCDQ 2wocg2pwODdfcD+YphoDM7tfy6agGin8KGIKkij6/BQ0tlJDZOl1ahcZjv699iQJ z/1vJ41S6LmmOmlnHy7b9rBMqz/4qPTR0k6j6iTPjxvHjUgdibWgtuZrx4aqhsh1 z0U+nS02+sCKiW+LK08po+7U8mZ0fFxOvRwmU9+Bx3dApR7uYuqUaAQ306f8i4Ro B4NPnP6dpCETioiVvHzn =wRK3 -----END PGP SIGNATURE----- --8/pVXlBMPtxfSuJG-- From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 17:13:49 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7D631961 for ; Tue, 15 Oct 2013 17:13:49 +0000 (UTC) (envelope-from jlh@FreeBSD.org) Received: from caravan.chchile.org (caravan.chchile.org [178.32.125.136]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E294B27FE for ; Tue, 15 Oct 2013 17:13:48 +0000 (UTC) Received: by caravan.chchile.org (Postfix, from userid 1000) id 27ACEC22A2; Tue, 15 Oct 2013 17:13:41 +0000 (UTC) Date: Tue, 15 Oct 2013 19:13:41 +0200 From: Jeremie Le Hen To: freebsd-fs@FreeBSD.org Subject: Re: ARC_SPACE_OTHER exceeds arc_max Message-ID: <20131015171340.GA30906@caravan.chchile.org> Mail-Followup-To: freebsd-fs@FreeBSD.org References: <20131011184206.GA60057@caravan.chchile.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20131011184206.GA60057@caravan.chchile.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 17:13:49 -0000 On Fri, Oct 11, 2013 at 08:42:06PM +0200, Jeremie Le Hen wrote: > Hi, > > (Please Cc: me on reply, as I'm not subscribed.) > > On my FreeBSD 9.1 machine, roughly 2/3 of the times the daily scripts > are run, my ARC size is outgrowing like crazy the vfs.zfs.arc_max being > set to 536870912 (512 MB). > > The consquence of this is that userland processes are killed (!). OK > this box has no swap space and should have but still, this sounds really > crazy that a filesystem cache is able to "reclaim" memory for running > processes :). This has hit me again three times since this email. Any idea? > > I've used top -b every 30 seconds to get an idea of the system's memory, > logs below. Here is also a snippet of my sysctls related to zfs: > > vfs.zfs.l2arc_norw: 1 > vfs.zfs.l2arc_feed_again: 1 > vfs.zfs.l2arc_noprefetch: 1 > vfs.zfs.l2arc_feed_min_ms: 200 > vfs.zfs.l2arc_feed_secs: 1 > vfs.zfs.l2arc_headroom: 2 > vfs.zfs.l2arc_write_boost: 8388608 > vfs.zfs.l2arc_write_max: 8388608 > vfs.zfs.arc_meta_limit: 134217728 > vfs.zfs.arc_meta_used: 236528568 > vfs.zfs.arc_min: 67108864 > vfs.zfs.arc_max: 536870912 > debug.adaptive_machine_arch: 1 > hw.machine_arch: amd64 > kstat.zfs.misc.arcstats.hits: 77149964 > kstat.zfs.misc.arcstats.misses: 13690743 > kstat.zfs.misc.arcstats.demand_data_hits: 8073317 > kstat.zfs.misc.arcstats.demand_data_misses: 274103 > kstat.zfs.misc.arcstats.demand_metadata_hits: 69076639 > kstat.zfs.misc.arcstats.demand_metadata_misses: 13416617 > kstat.zfs.misc.arcstats.prefetch_data_hits: 0 > kstat.zfs.misc.arcstats.prefetch_data_misses: 0 > kstat.zfs.misc.arcstats.prefetch_metadata_hits: 8 > kstat.zfs.misc.arcstats.prefetch_metadata_misses: 23 > kstat.zfs.misc.arcstats.mru_hits: 33260238 > kstat.zfs.misc.arcstats.mru_ghost_hits: 2830869 > kstat.zfs.misc.arcstats.mfu_hits: 43889719 > kstat.zfs.misc.arcstats.mfu_ghost_hits: 3452884 > kstat.zfs.misc.arcstats.allocated: 14361097 > kstat.zfs.misc.arcstats.deleted: 7860017 > kstat.zfs.misc.arcstats.stolen: 6994054 > kstat.zfs.misc.arcstats.recycle_miss: 7051205 > kstat.zfs.misc.arcstats.mutex_miss: 4479 > kstat.zfs.misc.arcstats.evict_skip: 4052583 > kstat.zfs.misc.arcstats.evict_l2_cached: 0 > kstat.zfs.misc.arcstats.evict_l2_eligible: 129211340800 > kstat.zfs.misc.arcstats.evict_l2_ineligible: 14336 > kstat.zfs.misc.arcstats.hash_elements: 88387 > kstat.zfs.misc.arcstats.hash_elements_max: 133658 > kstat.zfs.misc.arcstats.hash_collisions: 14197499 > kstat.zfs.misc.arcstats.hash_chains: 16044 > kstat.zfs.misc.arcstats.hash_chain_max: 27 > kstat.zfs.misc.arcstats.p: 188161024 > kstat.zfs.misc.arcstats.c: 536870912 > kstat.zfs.misc.arcstats.c_min: 67108864 > kstat.zfs.misc.arcstats.c_max: 536870912 > kstat.zfs.misc.arcstats.size: 534264760 > kstat.zfs.misc.arcstats.hdr_size: 20228688 > kstat.zfs.misc.arcstats.data_size: 406874112 > kstat.zfs.misc.arcstats.other_size: 107161960 > kstat.zfs.misc.arcstats.l2_hits: 0 > kstat.zfs.misc.arcstats.l2_misses: 0 > kstat.zfs.misc.arcstats.l2_feeds: 0 > kstat.zfs.misc.arcstats.l2_rw_clash: 0 > kstat.zfs.misc.arcstats.l2_read_bytes: 0 > kstat.zfs.misc.arcstats.l2_write_bytes: 0 > kstat.zfs.misc.arcstats.l2_writes_sent: 0 > kstat.zfs.misc.arcstats.l2_writes_done: 0 > kstat.zfs.misc.arcstats.l2_writes_error: 0 > kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0 > kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0 > kstat.zfs.misc.arcstats.l2_evict_reading: 0 > kstat.zfs.misc.arcstats.l2_free_on_write: 0 > kstat.zfs.misc.arcstats.l2_abort_lowmem: 0 > kstat.zfs.misc.arcstats.l2_cksum_bad: 0 > kstat.zfs.misc.arcstats.l2_io_error: 0 > kstat.zfs.misc.arcstats.l2_size: 0 > kstat.zfs.misc.arcstats.l2_hdr_size: 0 > kstat.zfs.misc.arcstats.l2_write_trylock_fail: 0 > kstat.zfs.misc.arcstats.l2_write_passed_headroom: 0 > kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 0 > kstat.zfs.misc.arcstats.l2_write_in_l2: 0 > kstat.zfs.misc.arcstats.l2_write_io_in_progress: 0 > kstat.zfs.misc.arcstats.l2_write_not_cacheable: 5 > kstat.zfs.misc.arcstats.l2_write_full: 0 > kstat.zfs.misc.arcstats.l2_write_buffer_iter: 0 > kstat.zfs.misc.arcstats.l2_write_pios: 0 > kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 0 > kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 0 > kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 0 > kstat.zfs.misc.arcstats.memory_throttle_count: 0 > kstat.zfs.misc.arcstats.duplicate_buffers: 0 > kstat.zfs.misc.arcstats.duplicate_buffers_size: 0 > kstat.zfs.misc.arcstats.duplicate_reads: 0 > > > > > > > > > ct 8 03:00:11 CEST 2013 > Mem: 97M Active, 112M Inact, 1258M Wired, 552K Cache, 482M Free > ARC: 310M Total, 42M MFU, 147M MRU, 912K Anon, 20M Header, 101M Other > > Tue Oct 8 03:00:41 CEST 2013 > Mem: 97M Active, 112M Inact, 1258M Wired, 552K Cache, 482M Free > ARC: 310M Total, 42M MFU, 147M MRU, 912K Anon, 20M Header, 101M Other > > Tue Oct 8 03:01:11 CEST 2013 > Mem: 105M Active, 114M Inact, 1274M Wired, 552K Cache, 457M Free > ARC: 330M Total, 42M MFU, 159M MRU, 2241K Anon, 20M Header, 107M Other > > Tue Oct 8 03:01:41 CEST 2013 > Mem: 105M Active, 115M Inact, 1241M Wired, 552K Cache, 490M Free > ARC: 332M Total, 32M MFU, 137M MRU, 912K Anon, 21M Header, 141M Other > > Tue Oct 8 03:02:11 CEST 2013 > Mem: 105M Active, 121M Inact, 1251M Wired, 552K Cache, 473M Free > ARC: 326M Total, 34M MFU, 145M MRU, 912K Anon, 22M Header, 124M Other > > Tue Oct 8 03:02:42 CEST 2013 > Mem: 105M Active, 121M Inact, 1247M Wired, 552K Cache, 478M Free > ARC: 350M Total, 32M MFU, 143M MRU, 1265K Anon, 23M Header, 150M Other > > Tue Oct 8 03:03:12 CEST 2013 > Mem: 104M Active, 121M Inact, 1232M Wired, 552K Cache, 493M Free > ARC: 293M Total, 35M MFU, 126M MRU, 928K Anon, 24M Header, 107M Other > > Tue Oct 8 03:03:42 CEST 2013 > Mem: 104M Active, 119M Inact, 1253M Wired, 552K Cache, 475M Free > ARC: 367M Total, 37M MFU, 145M MRU, 928K Anon, 24M Header, 160M Other > > Tue Oct 8 03:04:12 CEST 2013 > Mem: 104M Active, 119M Inact, 1284M Wired, 552K Cache, 443M Free > ARC: 316M Total, 55M MFU, 157M MRU, 912K Anon, 24M Header, 78M Other > > Tue Oct 8 03:04:42 CEST 2013 > Mem: 104M Active, 119M Inact, 1379M Wired, 552K Cache, 348M Free > ARC: 413M Total, 84M MFU, 223M MRU, 1743K Anon, 24M Header, 81M Other > > Tue Oct 8 03:05:12 CEST 2013 > Mem: 104M Active, 119M Inact, 1451M Wired, 552K Cache, 276M Free > ARC: 484M Total, 106M MFU, 274M MRU, 928K Anon, 23M Header, 80M Other > > Tue Oct 8 03:05:42 CEST 2013 > Mem: 104M Active, 119M Inact, 1451M Wired, 552K Cache, 276M Free > ARC: 494M Total, 119M MFU, 261M MRU, 930K Anon, 23M Header, 90M Other > > Tue Oct 8 03:06:12 CEST 2013 > Mem: 104M Active, 119M Inact, 1451M Wired, 552K Cache, 276M Free > ARC: 501M Total, 151M MFU, 228M MRU, 1226K Anon, 24M Header, 96M Other > > Tue Oct 8 03:06:42 CEST 2013 > Mem: 104M Active, 119M Inact, 1450M Wired, 552K Cache, 277M Free > ARC: 502M Total, 191M MFU, 188M MRU, 944K Anon, 24M Header, 99M Other > > Tue Oct 8 03:07:12 CEST 2013 > Mem: 104M Active, 119M Inact, 1451M Wired, 552K Cache, 276M Free > ARC: 507M Total, 226M MFU, 153M MRU, 1094K Anon, 24M Header, 103M Other > > Tue Oct 8 03:07:42 CEST 2013 > Mem: 104M Active, 118M Inact, 1407M Wired, 552K Cache, 321M Free > ARC: 479M Total, 175M MFU, 161M MRU, 915K Anon, 25M Header, 117M Other > > Tue Oct 8 03:08:13 CEST 2013 > Mem: 94M Active, 116M Inact, 1387M Wired, 552K Cache, 353M Free > ARC: 506M Total, 90M MFU, 231M MRU, 912K Anon, 26M Header, 158M Other > > Tue Oct 8 03:08:43 CEST 2013 > Mem: 96M Active, 117M Inact, 1351M Wired, 552K Cache, 386M Free > ARC: 436M Total, 52M MFU, 231M MRU, 1322K Anon, 23M Header, 129M Other > > Tue Oct 8 03:09:13 CEST 2013 > Mem: 95M Active, 117M Inact, 1314M Wired, 552K Cache, 423M Free > ARC: 440M Total, 73M MFU, 173M MRU, 941K Anon, 23M Header, 170M Other > > Tue Oct 8 03:09:43 CEST 2013 > Mem: 94M Active, 117M Inact, 1327M Wired, 552K Cache, 412M Free > ARC: 510M Total, 80M MFU, 182M MRU, 912K Anon, 23M Header, 223M Other > > Tue Oct 8 03:10:13 CEST 2013 > Mem: 94M Active, 117M Inact, 1276M Wired, 552K Cache, 463M Free > ARC: 578M Total, 65M MFU, 150M MRU, 912K Anon, 27M Header, 336M Other > > Tue Oct 8 03:10:43 CEST 2013 > Mem: 95M Active, 117M Inact, 1262M Wired, 552K Cache, 476M Free > ARC: 585M Total, 56M MFU, 144M MRU, 913K Anon, 26M Header, 358M Other > > Tue Oct 8 03:11:13 CEST 2013 > Mem: 103M Active, 114M Inact, 1269M Wired, 552K Cache, 464M Free > ARC: 590M Total, 66M MFU, 140M MRU, 912K Anon, 26M Header, 357M Other > > Tue Oct 8 03:11:44 CEST 2013 > Mem: 103M Active, 114M Inact, 1294M Wired, 552K Cache, 439M Free > ARC: 667M Total, 70M MFU, 155M MRU, 1056K Anon, 27M Header, 415M Other > > Tue Oct 8 03:12:14 CEST 2013 > Mem: 103M Active, 114M Inact, 1389M Wired, 552K Cache, 343M Free > ARC: 792M Total, 73M MFU, 185M MRU, 1040K Anon, 27M Header, 507M Other > > Tue Oct 8 03:12:44 CEST 2013 > Mem: 94M Active, 119M Inact, 1472M Wired, 552K Cache, 265M Free > ARC: 891M Total, 82M MFU, 212M MRU, 1056K Anon, 27M Header, 568M Other > > Tue Oct 8 03:13:15 CEST 2013 > Mem: 94M Active, 119M Inact, 1538M Wired, 544K Cache, 199M Free > ARC: 951M Total, 85M MFU, 220M MRU, 928K Anon, 28M Header, 618M Other > > Tue Oct 8 03:13:45 CEST 2013 > Mem: 136M Active, 22M Inact, 1682M Wired, 46M Cache, 65M Free > ARC: 1113M Total, 90M MFU, 259M MRU, 912K Anon, 26M Header, 738M Other > > Tue Oct 8 03:14:15 CEST 2013 > Mem: 153M Active, 3936K Inact, 1736M Wired, 36M Cache, 21M Free > ARC: 1159M Total, 95M MFU, 271M MRU, 1040K Anon, 25M Header, 767M Other > > Tue Oct 8 03:14:46 CEST 2013 > Mem: 62M Active, 15M Inact, 1808M Wired, 35M Cache, 30M Free > ARC: 1213M Total, 80M MFU, 294M MRU, 819K Anon, 19M Header, 819M Other > > Tue Oct 8 03:15:17 CEST 2013 > Mem: 61M Active, 6488K Inact, 1816M Wired, 33M Cache, 34M Free > ARC: 1194M Total, 73M MFU, 293M MRU, 1040K Anon, 19M Header, 808M Other > > Tue Oct 8 03:15:47 CEST 2013 > Mem: 75M Active, 2548K Inact, 1808M Wired, 25M Cache, 41M Free > ARC: 1189M Total, 72M MFU, 292M MRU, 1475K Anon, 19M Header, 804M Other > > Tue Oct 8 03:16:17 CEST 2013 > Mem: 75M Active, 2928K Inact, 1806M Wired, 24M Cache, 43M Free > ARC: 1183M Total, 72M MFU, 291M MRU, 912K Anon, 18M Header, 801M Other > > Tue Oct 8 03:16:47 CEST 2013 > Mem: 64M Active, 14M Inact, 1805M Wired, 24M Cache, 44M Free > ARC: 1179M Total, 71M MFU, 290M MRU, 912K Anon, 18M Header, 798M Other > > Tue Oct 8 03:17:17 CEST 2013 > Mem: 21M Active, 57M Inact, 1802M Wired, 24M Cache, 47M Free > ARC: 1174M Total, 71M MFU, 289M MRU, 912K Anon, 18M Header, 795M Other > > Tue Oct 8 03:17:48 CEST 2013 > Mem: 16M Active, 61M Inact, 1801M Wired, 24M Cache, 49M Free > ARC: 1170M Total, 70M MFU, 288M MRU, 912K Anon, 18M Header, 792M Other > > Tue Oct 8 03:18:18 CEST 2013 > Mem: 16M Active, 61M Inact, 1800M Wired, 24M Cache, 50M Free > ARC: 1165M Total, 70M MFU, 287M MRU, 912K Anon, 18M Header, 789M Other > > Tue Oct 8 03:18:48 CEST 2013 > Mem: 17M Active, 62M Inact, 1799M Wired, 23M Cache, 50M Free > ARC: 1162M Total, 70M MFU, 286M MRU, 912K Anon, 18M Header, 787M Other > > Tue Oct 8 03:19:18 CEST 2013 > Mem: 14M Active, 57M Inact, 1797M Wired, 23M Cache, 60M Free > ARC: 1157M Total, 69M MFU, 285M MRU, 912K Anon, 18M Header, 784M Other > > Tue Oct 8 03:19:48 CEST 2013 > Mem: 14M Active, 57M Inact, 1796M Wired, 23M Cache, 61M Free > ARC: 1153M Total, 69M MFU, 284M MRU, 912K Anon, 18M Header, 781M Other > > Tue Oct 8 03:20:18 CEST 2013 > Mem: 15M Active, 56M Inact, 1794M Wired, 22M Cache, 63M Free > ARC: 1148M Total, 69M MFU, 283M MRU, 912K Anon, 18M Header, 777M Other > > Tue Oct 8 03:20:48 CEST 2013 > Mem: 14M Active, 56M Inact, 1793M Wired, 22M Cache, 65M Free > ARC: 1145M Total, 69M MFU, 282M MRU, 912K Anon, 18M Header, 775M Other > > > -- > Jeremie Le Hen > > Scientists say the world is made up of Protons, Neutrons and Electrons. > They forgot to mention Morons. > -- Jeremie Le Hen Scientists say the world is made up of Protons, Neutrons and Electrons. They forgot to mention Morons. From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 17:41:36 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 8C72C4C5; Tue, 15 Oct 2013 17:41:36 +0000 (UTC) (envelope-from alc@rice.edu) Received: from pp1.rice.edu (proofpoint1.mail.rice.edu [128.42.201.100]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4DF2D29E8; Tue, 15 Oct 2013 17:41:35 +0000 (UTC) Received: from pps.filterd (pp1.rice.edu [127.0.0.1]) by pp1.rice.edu (8.14.5/8.14.5) with SMTP id r9FH7qR8001595; Tue, 15 Oct 2013 12:12:37 -0500 Received: from mh11.mail.rice.edu (mh11.mail.rice.edu [128.42.199.30]) by pp1.rice.edu with ESMTP id 1fe4ma9v4c-1; Tue, 15 Oct 2013 12:12:37 -0500 X-Virus-Scanned: by amavis-2.7.0 at mh11.mail.rice.edu, auth channel Received: from adsl-216-63-78-18.dsl.hstntx.swbell.net (108-254-203-201.lightspeed.hstntx.sbcglobal.net [108.254.203.201]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh11.mail.rice.edu (Postfix) with ESMTPSA id B7F064C01C0; Tue, 15 Oct 2013 12:12:36 -0500 (CDT) Message-ID: <525D7784.5000808@rice.edu> Date: Tue, 15 Oct 2013 12:12:36 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:24.0) Gecko/20100101 Thunderbird/24.0 MIME-Version: 1.0 To: Konstantin Belousov , J David Subject: Re: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page References: <20131015164537.GH3865@kib.kiev.ua> In-Reply-To: <20131015164537.GH3865@kib.kiev.ua> X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 kscore.is_bulkscore=0 kscore.compositescore=0 circleOfTrustscore=0 compositescore=0.600226699279213 urlsuspect_oldscore=0.000226699279213129 suspectscore=38 recipient_domain_to_sender_totalscore=0 phishscore=0 bulkscore=0 kscore.is_spamscore=0 recipient_to_sender_totalscore=0 recipient_domain_to_sender_domain_totalscore=448 rbsscore=0.600226699279213 spamscore=0 recipient_to_sender_domain_totalscore=0 urlsuspectscore=0.9 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1305240000 definitions=main-1310150077 Cc: "freebsd-fs@freebsd.org" , alc@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 17:41:36 -0000 On 10/15/2013 11:45, Konstantin Belousov wrote: > On Tue, Oct 15, 2013 at 11:17:53AM -0400, J David wrote: >> What are the necessary loader.conf / kernel config invocations to make >> ZFS stable on 9.2-RELEASE i386 node with 3GB RAM? >> >> This machine was rock solid under 8.4, but since upgrading to 9.2 it >> has been a disaster. It crashes every few hours with "panic: >> pmap_enter: attempted pmap_enter on 4MB page." >> >> Here are a couple of stack traces: >> >> panic: pmap_enter: attempted pmap_enter on 4MB page >> cpuid = 0 >> KDB: stack backtrace: >> #0 0xc0afc092 at kdb_backtrace+0x52 >> #1 0xc0ac249c at panic+0x1bc >> #2 0xc0f1825d at pmap_enter+0x63d >> #3 0xc0d34ee5 at vm_fault_hold+0x1c45 >> #4 0xc0d33262 at vm_fault+0x82 >> #5 0xc0f1f5d6 at trap_pfault+0x186 >> #6 0xc0f1ed4b at trap+0x51b >> #7 0xc0f0887c at calltrap+0x6 >> #8 0xc16770b6 at zio_execute+0x116 >> #9 0xc15d9460 at taskq_run_safe+0x10 >> #10 0xc0b08f26 at taskqueue_run_locked+0xe6 >> #11 0xc0b097f7 at taskqueue_thread_loop+0xb7 >> #12 0xc0a90a73 at fork_exit+0xa3 >> #13 0xc0f08924 at fork_trampoline+0x8 >> >> panic: pmap_enter: attempted pmap_enter on 4MB page >> cpuid = 1 >> KDB: stack backtrace: >> #0 0xc0afc092 at kdb_backtrace+0x52 >> #1 0xc0ac249c at panic+0x1bc >> #2 0xc0f1825d at pmap_enter+0x63d >> #3 0xc0d34ee5 at vm_fault_hold+0x1c45 >> #4 0xc0d33262 at vm_fault+0x82 >> #5 0xc0f1f5d6 at trap_pfault+0x186 >> #6 0xc0f1ed4b at trap+0x51b >> #7 0xc0f0887c at calltrap+0x6 >> #8 0xc1679bdf at zio_dva_allocate+0x9f >> #9 0xc16770b6 at zio_execute+0x116 >> #10 0xc15d9460 at taskq_run_safe+0x10 >> #11 0xc0b08f26 at taskqueue_run_locked+0xe6 >> #12 0xc0b097f7 at taskqueue_thread_loop+0xb7 >> #13 0xc0a90a73 at fork_exit+0xa3 >> #14 0xc0f08924 at fork_trampoline+0x8 >> >> For the last crash, top was running and this is what was on the screen >> when it died: >> >> CPU: 0.6% user, 0.0% nice, 2.2% system, 0.2% interrupt, 97.1% idle >> Mem: 442M Active, 91M Inact, 79M Wired, 3764K Cache, 4960K Buf, 2380M Free >> ARC: 40M Total, 8520K MFU, 30M MRU, 304K Anon, 494K Header, 1062K Other >> Swap: 4096M Total, 4096M Free >> Write failed: Broken pipe >> >> It doesn't seem to matter what KVA_PAGES, vm.kmem_size, >> vfs.zfs.arc_max or vfs.zfs.vdev.cache.size is set to, and the ZFS >> tuning guides in the wiki (albeit appearing to be dating to the 7.x >> era) provides guidelines for tuning down to 768M. So 3GB should be >> enough for a machine that is 99% idle. (Particularly given that it >> dies with >2GB free.) >> >> This seems to be specific to i386, this problem hasn't cropped up on >> any amd64 nodes. No compression, deduplication, snapshots or anything >> like that is in use. > It seems that i386 pmap_enter() cannot deal with the superpages, while > amd64 can. On the other hand, I am not quite sure that this is your > problem, because you get traps on accessing KVA, and I suspect that > ZFS uses wired memory. Obtain the kernel dump and do full backtrace > for the panic. Try MFCing r253949. I'm not confident that r253949 is the fix for this particular problem, but it should be MFCed. > Anyway, I believe that i386 pmap_enter() should do a demotion when needed. A kernel space pmap_enter should never be overwriting an existing superpage mapping. > Below is the prototyped change for this. > > diff --git a/sys/i386/i386/pmap.c b/sys/i386/i386/pmap.c > index 64bf1a3..91453da 100644 > --- a/sys/i386/i386/pmap.c > +++ b/sys/i386/i386/pmap.c > @@ -3473,17 +3473,21 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_prot_t access, vm_page_t m, > PMAP_LOCK(pmap); > sched_pin(); > > - /* > - * In the case that a page table page is not > - * resident, we are creating it here. > - */ > - if (va < VM_MAXUSER_ADDRESS) { > + pde = pmap_pde(pmap, va); > + if ((*pde & PG_PS) != 0) { > + /* PG_V is asserted by pmap_demote_pde */ > + pmap_demote_pde(pmap, pde, va); > + if (va < VM_MAXUSER_ADDRESS) { > + mpte = PHYS_TO_VM_PAGE(*pde & PG_FRAME); > + mpte->wire_count++; > + } > + } else if (va < VM_MAXUSER_ADDRESS) { > + /* > + * In the case that a page table page is not resident, > + * we are creating it here. > + */ > mpte = pmap_allocpte(pmap, va, M_WAITOK); > } > - > - pde = pmap_pde(pmap, va); > - if ((*pde & PG_PS) != 0) > - panic("pmap_enter: attempted pmap_enter on 4MB page"); > pte = pmap_pte_quick(pmap, va); > > /* From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 19:53:22 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id A6491B4D for ; Tue, 15 Oct 2013 19:53:22 +0000 (UTC) (envelope-from ericbrowning@skaggscatholiccenter.org) Received: from mail-pd0-f175.google.com (mail-pd0-f175.google.com [209.85.192.175]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 81F3923E1 for ; Tue, 15 Oct 2013 19:53:22 +0000 (UTC) Received: by mail-pd0-f175.google.com with SMTP id q10so9249914pdj.6 for ; Tue, 15 Oct 2013 12:53:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=nu+JXxwMOqyl1InB3ZUDjd7TwXEM4V7zQyZ+drSMxC0=; b=PMieDtrD8xc+VZi3pDFgkSyLx6PEwC07zaS7y+CxObI0oHhdCxBYLfElpTDFGvft/r sh6nkwqsQrGFEZftD18g6ROmnaMF661KA7cZNvdejMKfwJ6WWystcYhibrzotv7LDTjc r69FNJgE9FC1jCziI/rtx02ehQIvONmWmzeTbSxnzslKuHWcErrO09sVF9idO9gFZ2P+ eeQPXkTrOI9pJ3n2fF6zOCMzQsq9jyIsEDByHShKQU/c+fkGiB6slJ39RvkL5YqaFH13 DUuvg1B7BsoKmXnLgBzUJn3vAtEVyWJXh/av4Up6eKEkOMxS4ec9nn5RkIu1+21YS0py ENHA== X-Gm-Message-State: ALoCoQnXueYFgOzfN+i3Jyz28y6gb5PidWTMm2sXyugW6nufVc33CrwJVJQoAAyh6Wv7mvxNHLxg MIME-Version: 1.0 X-Received: by 10.66.100.227 with SMTP id fb3mr45570725pab.26.1381866796499; Tue, 15 Oct 2013 12:53:16 -0700 (PDT) Received: by 10.70.102.133 with HTTP; Tue, 15 Oct 2013 12:53:16 -0700 (PDT) Date: Tue, 15 Oct 2013 13:53:16 -0600 Message-ID: Subject: What is the NFS max thread limit From: Eric Browning To: FreeBSD FS Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 19:53:22 -0000 I'm currently still debugging slow NFS performance on mac clients. Running 9 stable with a mix of 200 Mac OS X 10.8, 10.7 and 10.6 clients. Periodically I get this error on the mac side: automountd[1766]: set_and_fake_mapent_mntlevel: subdir=/Contents error: Contents not found in map=-fstab I'm wondering with 200 clients with five nfsiod threads (~1000 nfsiod threads total) am I oversubscribing the nfs server? I can't seem to configure any more than 256 threads (nfs_server_flags="-t -n 256"). When I configure above 256 I get an error about it resetting to 4: Starting nfsd. nfsd: nfsd count 512; reset to 4 Tomorrow when all the mac clients have restarted I have changed their nfsiod threads down to 1 each to see if this is indeed the issue. Thanks in advance, -- Eric Browning Systems Administrator 801-984-7623 Skaggs Catholic Center Juan Diego Catholic High School Saint John the Baptist Middle Saint John the Baptist Elementary From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 22:08:43 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0DD3EDFC; Tue, 15 Oct 2013 22:08:43 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id B63A52BDF; Tue, 15 Oct 2013 22:08:42 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqgEACK8XVKDaFve/2dsb2JhbABahBGDKb1sgRWBO3SCJQEBBSNWGxgCAg0ZAlkZiAarQpJMgSmNbTQHFoJUgTsDkV6YKINAIIFt X-IronPort-AV: E=Sophos;i="4.93,502,1378872000"; d="scan'208";a="60393121" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 15 Oct 2013 18:08:41 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 2995BB3F44; Tue, 15 Oct 2013 18:08:41 -0400 (EDT) Date: Tue, 15 Oct 2013 18:08:41 -0400 (EDT) From: Rick Macklem To: araujobsdport@gmail.com Message-ID: <883922185.41691052.1381874921157.JavaMail.root@uoguelph.ca> In-Reply-To: <8374DB4D-C659-4400-AFC9-8E56B692C71E@gmail.com> Subject: Re: fixing "umount -f" for the NFS client MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 22:08:43 -0000 Marcelo Araujo wrote: > > > On 2013/10/15, at 21:05, Rick Macklem wrote: > > > araujo wrote: > >> > >> > >> > >> > >> > >> > >> 2013/9/5 Benjamin Kaduk < kaduk@mit.edu > > >> > >> > >> > >> On Wed, 4 Sep 2013, Rick Macklem wrote: > >> > >> > >> > >> > >> Benjamin Kaduk wrote: > >> > >> > >> > >> I think there are spare vfsops fields, so the MFC can be done in > >> an > >> ABI-compatible way. The new routine is for optional functionality, > >> so it > >> seems fine. > >> > >> There are spares vfs ops in 10/current, but not in stable/9. An > >> MFC > >> will > >> result in a VFS ABI change. (Since 10.0 hasn't been released yet, > >> I > >> didn't > >> use one of the recently added spares.) > >> > >> Oh, right, I was looking at 10/current. > >> > >> Unless there are pressing calls for the feature in the stable > >> branches, it's probably best to hold off on the MFC, then. OpenAFS > >> has encountered a few KBI incompatibilities over the years (mostly > >> in the networking bits, if I remember correctly), and we can deal > >> in > >> the future, but not having to is nice. > >> > >> > >> > >> > >> Hello Guys, > >> > >> > >> Is it possible to have it on 9-STABLE? > >> I tried to port the changes of revision 255136 made by rmacklem@ > >> to a > >> 9.1-RELEASE but the bug is still there. > >> > >> > >> Any change to make it works on 9.1, 9.2 or 9-STABLE? > >> > >> The patch attached is based on 9.1-RELEASE. > > The patch looks ok at a glance. Note that it can take > > up to 2-3minutes for a forced dismount to complete, > > depending on where the threads are waiting. > > > > If the mount is still there 5minutes after doing > > "umount -f", do a "ps axhl" and post the output > > of that to me. It may be getting stuck somewhere > > else than where I've seen during testing. > > Hello Rick, > > Thanks by the prompt reply, I'm gonna make more tests tomorrow, and > give you the output if necessary! > > However, is there any way to improve this time to force the umount? > To be honest, I have enough trouble making it work at all. I think the long delays happen when the kernel RPC is stuck somewhere in the TCP code waiting for timers to go off. (The "umount -f" forces a soclose() and then the RPCs stuck in sosend() and friends will eventually fail.) rick > > > > > > rick > > > >> > >> Best Regards, -- > >> Marcelo Araujo > >> araujo@FreeBSD.org > From owner-freebsd-fs@FreeBSD.ORG Tue Oct 15 22:41:55 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 9F10E613 for ; Tue, 15 Oct 2013 22:41:55 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 651832D8D for ; Tue, 15 Oct 2013 22:41:55 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ai4FAGDDXVKDaFve/2dsb2JhbABXA4M/UoMpvWxKS4E4dIIlAQEBAwEBAQEgKyALBRYYAgINGQIpAQkmBggHBAEcBIdfBgyrNJJLgSmMYgEFgQUkEAcRglmBOwOVOYN6kFODQCAxegEIFyI X-IronPort-AV: E=Sophos;i="4.93,503,1378872000"; d="scan'208";a="60403522" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 15 Oct 2013 18:41:47 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id DCD15B3F48; Tue, 15 Oct 2013 18:41:47 -0400 (EDT) Date: Tue, 15 Oct 2013 18:41:47 -0400 (EDT) From: Rick Macklem To: Eric Browning Message-ID: <744364739.41705471.1381876907892.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: What is the NFS max thread limit MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 22:41:55 -0000 Eric Browning wrote: > I'm currently still debugging slow NFS performance on mac clients. > Running > 9 stable with a mix of 200 Mac OS X 10.8, 10.7 and 10.6 clients. > > Periodically I get this error on the mac side: > automountd[1766]: set_and_fake_mapent_mntlevel: subdir=/Contents > error: > Contents not found in map=-fstab > > I'm wondering with 200 clients with five nfsiod threads (~1000 nfsiod > threads total) am I oversubscribing the nfs server? I can't seem to > configure any more than 256 threads (nfs_server_flags="-t -n 256"). > When I > configure above 256 I get an error about it resetting to 4: > > Starting nfsd. > nfsd: nfsd count 512; reset to 4 > > Tomorrow when all the mac clients have restarted I have changed their > nfsiod threads down to 1 each to see if this is indeed the issue. > Well, normally not all clients would be actively reading/writing files concurrently, so I wouldn't expect your load to need more than 256 nfsd threads. I also recall that you had a very large Access RPC load and Access RPCs are not done by nfsiod threads. (nfsiod threads only do readaheads and write behinds for buffer cache blocks of files.) However, if you do want to try more than 256 nfsd threads, you will need to edit nfsd.c and increase MAXNFSDCNT and then rebuild the nfsd daemon from these modified sources. I have no idea what causes the Mac OS X error, but you could try posting on one of the Apple mailing lists like darwin-kernel@lists.apple.com. rick ps: The nfsd.c in head/current/10.0 also has a "--maxthreads" option that I don't think is limited to MAXNFSDCNT from a glance at the code. > Thanks in advance, > -- > Eric Browning > Systems Administrator > 801-984-7623 > > Skaggs Catholic Center > Juan Diego Catholic High School > Saint John the Baptist Middle > Saint John the Baptist Elementary > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Wed Oct 16 01:23:59 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 000FE894; Wed, 16 Oct 2013 01:23:58 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ie0-x236.google.com (mail-ie0-x236.google.com [IPv6:2607:f8b0:4001:c03::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BBFF12536; Wed, 16 Oct 2013 01:23:58 +0000 (UTC) Received: by mail-ie0-f182.google.com with SMTP id as1so119775iec.13 for ; Tue, 15 Oct 2013 18:23:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=JltTIRrtAkC2EOkPkscEpzyyu7VRF6+WDRNDT9YbiUw=; b=Bsxihad/F6CHujYUZ+q3YOrUnnLsqmYjYf/OHK5ZcHheRtoMXHzR79LZbKtYF9zufk hR8/2lNlK+SH40PEqwDhV4BZcbnNoUYtThfHrlN6mWKCN8xBFgPhV92jZXjrPiZxax/q jEYfCzZtMD/s0t8cStOjakgJD78ivO2Oeiylumg5b3hMQaqUHKCkiNwzG74DFT0Rml4a rEwpxzMUyPX5b/a42c5xBvcsOn55LPolbvjFSI6v0AOhkYP5EcUCmobXs2bZW4fv4otD tqle+5dUq5u0yursg4THxOlR+M/GdM0/Be+YLbZHK2dfvZU2iV2HQaxqdYEj49a4MKXr FWNw== MIME-Version: 1.0 X-Received: by 10.50.73.135 with SMTP id l7mr98292igv.60.1381886638234; Tue, 15 Oct 2013 18:23:58 -0700 (PDT) Sender: jdavidlists@gmail.com Received: by 10.43.180.131 with HTTP; Tue, 15 Oct 2013 18:23:58 -0700 (PDT) In-Reply-To: <525D7784.5000808@rice.edu> References: <20131015164537.GH3865@kib.kiev.ua> <525D7784.5000808@rice.edu> Date: Tue, 15 Oct 2013 21:23:58 -0400 X-Google-Sender-Auth: kWZiMw_7qlLaMsc2KNnuU65hi9I Message-ID: Subject: Re: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page From: J David To: Alan Cox Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-fs@freebsd.org" , alc@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 01:23:59 -0000 With a kernel updated with debugging code and r253949, it panics during boot: fifo_open: 0x894212d0 is not exclusive locked but should be KDB: enter: lock violation -[ thread pid 963 tid 100353 ] Stopped at kdb_enter+0x3d: movl $0,kdb_why db> where Tracing pid 963 tid 100353 td 0x883aa5e0 kdb_enter(8112e312,8112e4ee,894212d0,8112e24d,812746d4,...) at kdb_enter+0x3d/frame 0xbf656980 assert_vop_elocked(894212d0,811069b9,df,1,0,...) at assert_vop_elocked+0xbe/frame 0xbf6569b0 fifo_open(bf656ae8,8117a85a,8114f47a,81423bac,bf656a74,...) at fifo_open+0x36/frame 0xbf656a10 VOP_OPEN_APV(818a3f30,bf656ae8,100,10000,10000,...) at VOP_OPEN_APV+0xca/frame 0xbf656a40 vn_open_cred(bf656b60,bf656bec,c08,0,879ae180,8832d428) at vn_open_cred+0x5a5/frame 0xbf656b18 vn_open(bf656b60,bf656bec,c08,8832d428,0,...) at vn_open+0x3b/frame 0xbf656b38 kern_openat(883aa5e0,ffffff9c,804a7b2,0,4,7fbfde08) at kern_openat+0x1cb/frame 0xbf656c20 sys_open(883aa5e0,bf656cc8,811748f4,81126582,1,...) at sys_open+0x38/frame 0xbf656c40 syscall(bf656d08) at syscall+0x2da/frame 0xbf656cfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xbf656cfc --- syscall (5, FreeBSD ELF32, sys_open), eip = 0x2815e29b, esp = 0x7fbfdd6c, ebp = 0x7fbfdd78 --- If possible, I'll see about booting it in single user mode and disabling as much junk as possible that might be touching FIFO's, as it's unlike that problem is related to the ZFS issue. From owner-freebsd-fs@FreeBSD.ORG Wed Oct 16 02:09:28 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 08A8EFB2 for ; Wed, 16 Oct 2013 02:09:28 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-pd0-x234.google.com (mail-pd0-x234.google.com [IPv6:2607:f8b0:400e:c02::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D2EF2270A for ; Wed, 16 Oct 2013 02:09:27 +0000 (UTC) Received: by mail-pd0-f180.google.com with SMTP id y10so135751pdj.25 for ; Tue, 15 Oct 2013 19:09:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=cCwo21OX1J/+8zHzIGopBS40uUbsOZrFcwKvYl2q0x0=; b=uTKBZAe/naIK+tuxSO5hIMdXsyv6xrN4rTJ0RQZlmc6HjO4d87yUswD95ZtjJTpL0k qF6HylaGFFDIM4QLyh4ySvqTTaBWbGhJrQrwbeJGEjiVbBkJ1nQ47K7klcw0UGnL3iUJ WKu5qQ521Zck8v1EIGGR0yNdpqL5cHKoxLXjhC25M/7Kahb/WS/ze1dsPhGJ/1XKRQrr FHvYMYQN2bdP8wulDV1lF80prjDKATniMyVzuZC87w5k+01oQ++5qn3RxvIkgraFTsER Y39vGolDtthrsDAu7WcZdtNoOwJHmn8DZhGFNp9y0W6D8FKHyqqnoiFt7/xeAyr6EPiw MrHQ== MIME-Version: 1.0 X-Received: by 10.68.40.169 with SMTP id y9mr18529pbk.193.1381889367460; Tue, 15 Oct 2013 19:09:27 -0700 (PDT) Received: by 10.68.127.202 with HTTP; Tue, 15 Oct 2013 19:09:27 -0700 (PDT) In-Reply-To: <748522744.41194273.1381842355314.JavaMail.root@uoguelph.ca> References: <748522744.41194273.1381842355314.JavaMail.root@uoguelph.ca> Date: Wed, 16 Oct 2013 10:09:27 +0800 Message-ID: Subject: Re: fixing "umount -f" for the NFS client From: Marcelo Araujo To: Rick Macklem Content-Type: multipart/mixed; boundary=bcaec51a7546ddd38e04e8d230fb X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: araujo@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 02:09:28 -0000 --bcaec51a7546ddd38e04e8d230fb Content-Type: text/plain; charset=ISO-8859-1 2013/10/15 Rick Macklem > araujo wrote: > > > > > > > > > > > > > > 2013/9/5 Benjamin Kaduk < kaduk@mit.edu > > > > > > > > > On Wed, 4 Sep 2013, Rick Macklem wrote: > > > > > > > > > > Benjamin Kaduk wrote: > > > > > > > > I think there are spare vfsops fields, so the MFC can be done in an > > ABI-compatible way. The new routine is for optional functionality, > > so it > > seems fine. > > > > There are spares vfs ops in 10/current, but not in stable/9. An MFC > > will > > result in a VFS ABI change. (Since 10.0 hasn't been released yet, I > > didn't > > use one of the recently added spares.) > > > > Oh, right, I was looking at 10/current. > > > > Unless there are pressing calls for the feature in the stable > > branches, it's probably best to hold off on the MFC, then. OpenAFS > > has encountered a few KBI incompatibilities over the years (mostly > > in the networking bits, if I remember correctly), and we can deal in > > the future, but not having to is nice. > > > > > > > > > > Hello Guys, > > > > > > Is it possible to have it on 9-STABLE? > > I tried to port the changes of revision 255136 made by rmacklem@ to a > > 9.1-RELEASE but the bug is still there. > > > > > > Any change to make it works on 9.1, 9.2 or 9-STABLE? > > > > The patch attached is based on 9.1-RELEASE. > > > The patch looks ok at a glance. Note that it can take > up to 2-3minutes for a forced dismount to complete, > depending on where the threads are waiting. > > If the mount is still there 5minutes after doing > "umount -f", do a "ps axhl" and post the output > of that to me. It may be getting stuck somewhere > else than where I've seen during testing. > > Hello Rick, I made a test right now and the "umount -f" hang for almost 5 minutes and nothing happens. Here attached is the output of "ps axhl" as you requested. Any idea what could be? Thank you so much by all support. -- Marcelo Araujo araujo@FreeBSD.org --bcaec51a7546ddd38e04e8d230fb Content-Type: text/plain; charset=US-ASCII; name="ps.txt" Content-Disposition: attachment; filename="ps.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hmtx8wci0 VUlEICAgUElEICBQUElEIENQVSAgUFJJICBOSSAgICBWU1ogICBSU1MgTVdDSEFOICAgU1RBVCBU VCAgICAgICAgIFRJTUUgQ09NTUFORAogIDAgICAgIDAgICAgIDAgICAwIC0xMDAgICAwICAgICAg MCAgIDgwMCAtICAgICAgICBETHMgID8/ICAgICAgMDowMC4zOSBba2VybmVsXQogIDAgICAgIDEg ICAgIDAgICAwICAgNTIgICAwICAgNjI3NiAgIDU1MiB3YWl0ICAgICBJTHMgID8/ICAgICAgMDow MC4wMSAvc2Jpbi9pbml0IC0tCiAgMCAgICAgMiAgICAgMCAgIDAgIC0xNiAgIDAgICAgICAwICAg IDE2IGNyeXB0b193IERMICAgPz8gICAgICAwOjAwLjAwIFtjcnlwdG9dCiAgMCAgICAgMyAgICAg MCAgIDAgIC0xNiAgIDAgICAgICAwICAgIDE2IGNyeXB0b19yIERMICAgPz8gICAgICAwOjAwLjAw IFtjcnlwdG8gcmV0dXJuc10KICAwICAgICA0ICAgICAwICAgMCAgLTE2ICAxMCAgICAgIDAgICAg MTYgc3dhaXQgICAgRE5MICA/PyAgICAgIDA6MDAuMDAgW3Njc3RkMF0KICAwICAgICA1ICAgICAw ICAgMCAgLTE2ICAxMCAgICAgIDAgICAgMTYgc3dhaXQgICAgRE5MICA/PyAgICAgIDA6MDAuMDAg W3Njc3RkMV0KICAwICAgICA2ICAgICAwICAgMCAgLTE2ICAxMCAgICAgIDAgICAgMTYgc3dhaXQg ICAgRE5MICA/PyAgICAgIDA6MDAuMDAgW3Njc3RkMl0KICAwICAgICA3ICAgICAwICAgMCAgLTE2 ICAxMCAgICAgIDAgICAgMTYgc3dhaXQgICAgRE5MICA/PyAgICAgIDA6MDAuMDAgW3Njc3RkM10K ICAwICAgICA4ICAgICAwICAgMCAgLTE2ICAxMCAgICAgIDAgICAgMTYgc3dhaXQgICAgRE5MICA/ PyAgICAgIDA6MDAuMDAgW3Njc3RkNF0KICAwICAgICA5ICAgICAwICAgMCAgLTE2ICAxMCAgICAg IDAgICAgMTYgc3dhaXQgICAgRE5MICA/PyAgICAgIDA6MDAuMDAgW3Njc3RkNV0KICAwICAgIDEw ICAgICAwICAgMCAgLTE2ICAgMCAgICAgIDAgICAgMTYgYXVkaXRfd28gREwgICA/PyAgICAgIDA6 MDAuMDAgW2F1ZGl0XQogIDAgICAgMTEgICAgIDAgICAwICAxNTUgICAwICAgICAgMCAgIDI1NiAt ICAgICAgICBSTCAgID8/ICAxMzE5MDo0OS43NSBbaWRsZV0KICAwICAgIDEyICAgICAwICAgMCAg LTc2ICAgMCAgICAgIDAgICA3NjggLSAgICAgICAgV0wgICA/PyAgICAgIDE6MjkuMDMgW2ludHJd CiAgMCAgICAxMyAgICAgMCAgIDAgICAtOCAgIDAgICAgICAwICAgIDQ4IC0gICAgICAgIERMICAg Pz8gICAgICAwOjA2LjAxIFtnZW9tXQogIDAgICAgMTQgICAgIDAgICAwICAtMTYgICAwICAgICAg MCAgICAxNiAtICAgICAgICBETCAgID8/ICAgICAgMDowMS40MiBbeWFycm93XQogIDAgICAgMTUg ICAgIDAgICAwICAtMTYgIDEwICAgICAgMCAgICAxNiBzd2FpdCAgICBETkwgID8/ICAgICAgMDow MC4wMCBbc2NzdGQ2XQogIDAgICAgMTYgICAgIDAgICAwICAtMTYgIDEwICAgICAgMCAgICAxNiBz d2FpdCAgICBETkwgID8/ICAgICAgMDowMC4wMCBbc2NzdGQ3XQogIDAgICAgMTcgICAgIDAgICAw ICAtMTYgIDEwICAgICAgMCAgICAxNiBzd2FpdCAgICBETkwgID8/ICAgICAgMDowMC4wMCBbc2Nz dGQ4XQogIDAgICAgMTggICAgIDAgICAwICAtMTYgIDEwICAgICAgMCAgICAxNiBzd2FpdCAgICBE TkwgID8/ICAgICAgMDowMC4wMCBbc2NzdGQ5XQogIDAgICAgMTkgICAgIDAgICAwICAtMTYgIDEw ICAgICAgMCAgICAxNiBzd2FpdCAgICBETkwgID8/ICAgICAgMDowMC4wMCBbc2NzdGQxMF0KICAw ICAgIDIwICAgICAwICAgMCAgLTE2ICAxMCAgICAgIDAgICAgMTYgc3dhaXQgICAgRE5MICA/PyAg ICAgIDA6MDAuMDAgW3Njc3RkMTFdCiAgMCAgICAyMSAgICAgMCAgIDAgIC0xNiAgMTAgICAgICAw ICAgIDE2IHN3YWl0ICAgIEROTCAgPz8gICAgICAwOjAwLjAwIFtzY3N0ZDEyXQogIDAgICAgMjIg ICAgIDAgICAwICAtMTYgIDEwICAgICAgMCAgICAxNiBzd2FpdCAgICBETkwgID8/ICAgICAgMDow MC4wMCBbc2NzdGQxM10KICAwICAgIDIzICAgICAwICAgMCAgLTE2ICAxMCAgICAgIDAgICAgMTYg c3dhaXQgICAgRE5MICA/PyAgICAgIDA6MDAuMDAgW3Njc3RkMTRdCiAgMCAgICAyNCAgICAgMCAg IDAgIC0xNiAgMTAgICAgICAwICAgIDE2IHN3YWl0ICAgIEROTCAgPz8gICAgICAwOjAwLjAwIFtz Y3N0ZDE1XQogIDAgICAgMjUgICAgIDAgICAwICAtMTYgLTEwICAgICAgMCAgICAxNiBzd2FpdCAg ICBEPEwgID8/ICAgICAgMDowMC4wMCBbc2NzdF9pbml0ZF0KICAwICAgIDI2ICAgICAwICAgMCAg LTE2IC0xMCAgICAgIDAgICAgMTYgc3dhaXQgICAgRDxMICA/PyAgICAgIDA6MDAuMDAgW3Njc3Rf dG1dCiAgMCAgICAyNyAgICAgMCAgIDAgIC0xNiAtMTAgICAgICAwICAgIDE2IHN3YWl0ICAgIEQ8 TCAgPz8gICAgICAwOjAwLjAwIFtzY3N0X21nbXRkXQogIDAgICAgMjggICAgIDAgICAwICAtMTYg ICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBETCAgID8/ICAgICAgMDowMC4wMCBbdmFhaXRocjBd CiAgMCAgICAyOSAgICAgMCAgIDAgIC0xNiAgIDAgICAgICAwICAgIDE2IHN3YWl0ICAgIERMICAg Pz8gICAgICAwOjAwLjAwIFt2YWFpdGhyMV0KICAwICAgIDMwICAgICAwICAgMCAgLTE2ICAgMCAg ICAgIDAgICAgMTYgc3dhaXQgICAgREwgICA/PyAgICAgIDA6MDAuMDAgW3ZhYWl0aHIyXQogIDAg ICAgMzEgICAgIDAgICAwICAtMTYgICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBETCAgID8/ICAg ICAgMDowMC4wMCBbdmFhaXRocjNdCiAgMCAgICAzMiAgICAgMCAgIDAgIC0xNiAgIDAgICAgICAw ICAgIDE2IHN3YWl0ICAgIERMICAgPz8gICAgICAwOjAwLjAwIFt2YWFpdGhyNF0KICAwICAgIDMz ICAgICAwICAgMCAgLTE2ICAgMCAgICAgIDAgICAgMTYgc3dhaXQgICAgREwgICA/PyAgICAgIDA6 MDAuMDAgW3ZhYWl0aHI1XQogIDAgICAgMzQgICAgIDAgICAwICAtMTYgICAwICAgICAgMCAgICAx NiBzd2FpdCAgICBETCAgID8/ICAgICAgMDowMC4wMCBbdmFhaXRocjZdCiAgMCAgICAzNSAgICAg MCAgIDAgIC0xNiAgIDAgICAgICAwICAgIDE2IHN3YWl0ICAgIERMICAgPz8gICAgICAwOjAwLjAw IFt2YWFpdGhyN10KICAwICAgIDM2ICAgICAwICAgMCAgLTE2ICAgMCAgICAgIDAgICAgMTYgc3dh aXQgICAgREwgICA/PyAgICAgIDA6MDAuMDAgW2lzY3NpcmQwXQogIDAgICAgMzcgICAgIDAgICAw ICAtMTYgICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBETCAgID8/ICAgICAgMDowMC4wMCBbaXNj c2lyZDFdCiAgMCAgICAzOCAgICAgMCAgIDAgIC0xNiAgIDAgICAgICAwICAgIDE2IHN3YWl0ICAg IERMICAgPz8gICAgICAwOjAwLjAwIFtpc2NzaXJkMl0KICAwICAgIDM5ICAgICAwICAgMCAgLTE2 ICAgMCAgICAgIDAgICAgMTYgc3dhaXQgICAgREwgICA/PyAgICAgIDA6MDAuMDAgW2lzY3NpcmQz XQogIDAgICAgNDAgICAgIDAgICAwICAtMTYgICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBETCAg ID8/ICAgICAgMDowMC4wMCBbaXNjc2lyZDRdCiAgMCAgICA0MSAgICAgMCAgIDAgIC0xNiAgIDAg ICAgICAwICAgIDE2IHN3YWl0ICAgIERMICAgPz8gICAgICAwOjAwLjAwIFtpc2NzaXJkNV0KICAw ICAgIDQyICAgICAwICAgMCAgLTE2ICAgMCAgICAgIDAgICAgMTYgc3dhaXQgICAgREwgICA/PyAg ICAgIDA6MDAuMDAgW2lzY3NpcmQ2XQogIDAgICAgNDMgICAgIDAgICAwICAtMTYgICAwICAgICAg MCAgICAxNiBzd2FpdCAgICBETCAgID8/ICAgICAgMDowMC4wMCBbaXNjc2lyZDddCiAgMCAgICA0 NCAgICAgMCAgIDAgIC0xNiAgIDAgICAgICAwICAgIDE2IHN3YWl0ICAgIERMICAgPz8gICAgICAw OjAwLjAwIFtpc2NzaXJkOF0KICAwICAgIDQ1ICAgICAwICAgMCAgLTE2ICAgMCAgICAgIDAgICAg MTYgc3dhaXQgICAgREwgICA/PyAgICAgIDA6MDAuMDAgW2lzY3NpcmQ5XQogIDAgICAgNDYgICAg IDAgICAwICAtMTYgICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBETCAgID8/ICAgICAgMDowMC4w MCBbaXNjc2lyZDEwXQogIDAgICAgNDcgICAgIDAgICAwICAtMTYgICAwICAgICAgMCAgICAxNiBz d2FpdCAgICBETCAgID8/ICAgICAgMDowMC4wMCBbaXNjc2lyZDExXQogIDAgICAgNDggICAgIDAg ICAwICAtMTYgICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBETCAgID8/ICAgICAgMDowMC4wMCBb aXNjc2lyZDEyXQogIDAgICAgNDkgICAgIDAgICAwICAtMTYgICAwICAgICAgMCAgICAxNiBzd2Fp dCAgICBETCAgID8/ICAgICAgMDowMC4wMCBbaXNjc2lyZDEzXQoKVUlEICAgUElEICBQUElEIENQ VSAgUFJJICBOSSAgICBWU1ogICBSU1MgTVdDSEFOICAgU1RBVCBUVCAgICAgICAgIFRJTUUgQ09N TUFORAogIDAgICAgNTAgICAgIDAgICAwICAtMTYgICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBE TCAgID8/ICAgICAgMDowMC4wMCBbaXNjc2lyZDE0XQogIDAgICAgNTEgICAgIDAgICAwICAtMTYg ICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBETCAgID8/ICAgICAgMDowMC4wMCBbaXNjc2lyZDE1 XQogIDAgICAgNTIgICAgIDAgICAwICAtMTYgICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBETCAg ID8/ICAgICAgMDowMC4wMCBbaXNjc2l3cjBdCiAgMCAgICA1MyAgICAgMCAgIDAgIC0xNiAgIDAg ICAgICAwICAgIDE2IHN3YWl0ICAgIERMICAgPz8gICAgICAwOjAwLjAwIFtpc2NzaXdyMV0KICAw ICAgIDU0ICAgICAwICAgMCAgLTE2ICAgMCAgICAgIDAgICAgMTYgc3dhaXQgICAgREwgICA/PyAg ICAgIDA6MDAuMDAgW2lzY3Npd3IyXQogIDAgICAgNTUgICAgIDAgICAwICAtMTYgICAwICAgICAg MCAgICAxNiBzd2FpdCAgICBETCAgID8/ICAgICAgMDowMC4wMCBbaXNjc2l3cjNdCiAgMCAgICA1 NiAgICAgMCAgIDAgIC0xNiAgIDAgICAgICAwICAgIDE2IHN3YWl0ICAgIERMICAgPz8gICAgICAw OjAwLjAwIFtpc2NzaXdyNF0KICAwICAgIDU3ICAgICAwICAgMCAgLTE2ICAgMCAgICAgIDAgICAg MTYgc3dhaXQgICAgREwgICA/PyAgICAgIDA6MDAuMDAgW2lzY3Npd3I1XQogIDAgICAgNTggICAg IDAgICAwICAtMTYgICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBETCAgID8/ICAgICAgMDowMC4w MCBbaXNjc2l3cjZdCiAgMCAgICA1OSAgICAgMCAgIDAgIC0xNiAgIDAgICAgICAwICAgIDE2IHN3 YWl0ICAgIERMICAgPz8gICAgICAwOjAwLjAwIFtpc2NzaXdyN10KICAwICAgIDYwICAgICAwICAg MCAgLTE2ICAgMCAgICAgIDAgICAgMTYgc3dhaXQgICAgREwgICA/PyAgICAgIDA6MDAuMDAgW2lz Y3Npd3I4XQogIDAgICAgNjEgICAgIDAgICAwICAtMTYgICAwICAgICAgMCAgICAxNiBzd2FpdCAg ICBETCAgID8/ICAgICAgMDowMC4wMCBbaXNjc2l3cjldCiAgMCAgICA2MiAgICAgMCAgIDAgIC0x NiAgIDAgICAgICAwICAgIDE2IHN3YWl0ICAgIERMICAgPz8gICAgICAwOjAwLjAwIFtpc2NzaXdy MTBdCiAgMCAgICA2MyAgICAgMCAgIDAgIC0xNiAgIDAgICAgICAwICAgIDE2IHN3YWl0ICAgIERM ICAgPz8gICAgICAwOjAwLjAwIFtpc2NzaXdyMTFdCiAgMCAgICA2NCAgICAgMCAgIDAgIC0xNiAg IDAgICAgICAwICAgIDE2IHN3YWl0ICAgIERMICAgPz8gICAgICAwOjAwLjAwIFtpc2NzaXdyMTJd CiAgMCAgICA2NSAgICAgMCAgIDAgIC0xNiAgIDAgICAgICAwICAgIDE2IHN3YWl0ICAgIERMICAg Pz8gICAgICAwOjAwLjAwIFtpc2NzaXdyMTNdCiAgMCAgICA2NiAgICAgMCAgIDAgIC0xNiAgIDAg ICAgICAwICAgIDE2IHN3YWl0ICAgIERMICAgPz8gICAgICAwOjAwLjAwIFtpc2NzaXdyMTRdCiAg MCAgICA2NyAgICAgMCAgIDAgIC0xNiAgIDAgICAgICAwICAgIDE2IHN3YWl0ICAgIERMICAgPz8g ICAgICAwOjAwLjAwIFtpc2NzaXdyMTVdCiAgMCAgICA2OCAgICAgMCAgIDAgIC0xNiAgIDAgICAg ICAwICAgIDE2IG1wc19zY2FuIERMICAgPz8gICAgICAwOjAwLjE0IFttcHNfc2NhbjBdCiAgMCAg ICA2OSAgICAgMCAgIDAgIC02OCAgIDAgICAgICAwICAgMjA4IC0gICAgICAgIERMICAgPz8gICAg ICAwOjA1LjM2IFt1c2JdCiAgMCAgICA3MCAgICAgMCAgIDAgICAtOCAgIDAgICAgICAwICAgIDMy IGwyYXJjX2ZlIERMICAgPz8gICAgICAwOjAwLjQxIFt6ZnNrZXJuXQogIDAgICAgNzEgICAgIDAg ICAwICAtMTYgICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBETCAgID8/ICAgICAgMDowMC4wMCBb dmFhaV90aHIwXQogIDAgICAgNzIgICAgIDAgICAwICAtMTYgICAwICAgICAgMCAgICAxNiBzd2Fp dCAgICBETCAgID8/ICAgICAgMDowMC4wMCBbdmFhaV90aHIxXQogIDAgICAgNzMgICAgIDAgICAw ICAtMTYgICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBETCAgID8/ICAgICAgMDowMC4wMCBbdmFh aV90aHIyXQogIDAgICAgNzQgICAgIDAgICAwICAtMTYgICAwICAgICAgMCAgICAxNiBzd2FpdCAg ICBETCAgID8/ICAgICAgMDowMC4wMCBbdmFhaV90aHIzXQogIDAgICAgNzUgICAgIDAgICAwICAt MTYgICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBETCAgID8/ICAgICAgMDowMC4wMCBbdmFhaV90 aHI0XQogIDAgICAgNzYgICAgIDAgICAwICAtMTYgICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBE TCAgID8/ICAgICAgMDowMC4wMCBbdmFhaV90aHI1XQogIDAgICAgNzcgICAgIDAgICAwICAtMTYg ICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBETCAgID8/ICAgICAgMDowMC4wMCBbdmFhaV90aHI2 XQogIDAgICAgNzggICAgIDAgICAwICAtMTYgICAwICAgICAgMCAgICAxNiBzd2FpdCAgICBETCAg ID8/ICAgICAgMDowMC4wMCBbdmFhaV90aHI3XQogIDAgICAgNzkgICAgIDAgICAwICAtMTYgICAw ICAgICAgMCAgICAxNiBwZnRtICAgICBETCAgID8/ICAgICAgMDowMC4xNyBbcGZwdXJnZV0KICAw ICAgIDgwICAgICAwICAgMCAgLTE2ICAgMCAgICAgIDAgICAgMTYgd2FpdGluZ18gREwgICA/PyAg ICAgIDA6MDAuMDAgW3NjdHBfaXRlcmF0b3JdCiAgMCAgICA4MSAgICAgMCAgIDAgIC0xNiAgIDAg ICAgICAwICAgIDE2IGNjYl9zY2FuIERMICAgPz8gICAgICAwOjAwLjAwIFt4cHRfdGhyZF0KICAw ICAgIDgyICAgICAwICAgMCAgLTE2ICAgMCAgICAgIDAgICAgMTYgZ2t0OndhaXQgREwgICA/PyAg ICAgIDA6MDAuMDAgW2dfbXBfa3RdCiAgMCAgICA4MyAgICAgMCAgIDAgIC0xNiAgIDAgICAgICAw ICAgIDE2IHBzbGVlcCAgIERMICAgPz8gICAgICAwOjAwLjAzIFtwYWdlZGFlbW9uXQogIDAgICAg ODQgICAgIDAgICAwICAtMTYgICAwICAgICAgMCAgICAxNiBwc2xlZXAgICBETCAgID8/ICAgICAg MDowMC4wMCBbdm1kYWVtb25dCiAgMCAgICA4NSAgICAgMCAgIDAgIDE1NSAgIDAgICAgICAwICAg IDE2IHBnemVybyAgIERMICAgPz8gICAgICAwOjAwLjAwIFtwYWdlemVyb10KICAwICAgIDg2ICAg ICAwICAgMCAgLTE2ICAgMCAgICAgIDAgICAgMTYgcHNsZWVwICAgREwgICA/PyAgICAgIDA6MDEu MzcgW2J1ZmRhZW1vbl0KICAwICAgIDg3ICAgICAwICAgMCAgIDE2ICAgMCAgICAgIDAgICAgMTYg c3luY2VyICAgREwgICA/PyAgICAgIDA6MDEuMjMgW3N5bmNlcl0KICAwICAgIDg4ICAgICAwICAg MCAgLTE2ICAgMCAgICAgIDAgICAgMTYgdmxydXd0ICAgREwgICA/PyAgICAgIDA6MDAuMTggW3Zu bHJ1XQogIDAgICAgODkgICAgIDAgICAwICAtMTYgICAwICAgICAgMCAgICAxNiBzZGZsdXNoICBE TCAgID8/ICAgICAgMDoxMC4zMiBbc29mdGRlcGZsdXNoXQogIDAgICAxMDcgICAgIDAgICAwICAg LTggICAwICAgICAgMCAgICAxNiBtZHdhaXQgICBETCAgID8/ICAgICAgMDowMS4xNyBbbWQwXQog IDAgICAxMTcgICAgIDAgICAwICAgLTggICAwICAgICAgMCAgICAxNiBtZHdhaXQgICBETCAgID8/ ICAgICAgMDowMC4wMCBbbWQxXQogIDAgICAxMjcgICAgIDAgICAwICAgLTggICAwICAgICAgMCAg ICAxNiBtZHdhaXQgICBETCAgID8/ICAgICAgMDowMC4wMCBbbWQyXQogIDAgICAxMzcgICAgIDAg ICAwICAgLTggICAwICAgICAgMCAgICAxNiBtZHdhaXQgICBETCAgID8/ICAgICAgMDowMC4wMCBb bWQzXQogIDAgICAxNDcgICAgIDAgICAwICAgLTggICAwICAgICAgMCAgICAxNiBtZHdhaXQgICBE TCAgID8/ICAgICAgMDoxMS43MyBbbWQ0XQogIDAgICAxNTcgICAgIDAgICAwICAgLTggICAwICAg ICAgMCAgICAxNiBtZHdhaXQgICBETCAgID8/ICAgICAgMDowMC4wMCBbbWQ1XQogIDAgIDEzNTMg ICAgIDEgICAwICAgMjAgICAwICAxMDM3MiAgNDYwOCBzZWxlY3QgICBJcyAgID8/ICAgICAgMDow MC4wMCAvc2Jpbi9kZXZkCiAgMCAgMTczNCAgICAgMSAgIDAgICAyMCAgIDAgIDkzMTI4IDE3MDEy IHNlbGVjdCAgIFNzICAgPz8gICAgICAxOjE4Ljc3IC91c3IvbG9jYWwvYmluL3B5dGhvbjIuNyAv dXNyL2xvY2FsL2Jpbi9zdXBlcnZpc29yZAogIDAgIDE3NjMgICAgIDEgICAwICAgIDEgICAwICAx Mzk5MiAgMTc4MCBzZWxlY3QgICBTcyAgID8/ICAgICAgMDoxMC43MiAvdXNyL3NiaW4vc3lzbG9n ZCAtcwoKVUlEICAgUElEICBQUElEIENQVSAgUFJJICBOSSAgICBWU1ogICBSU1MgTVdDSEFOICAg U1RBVCBUVCAgICAgICAgIFRJTUUgQ09NTUFORAogIDAgIDE4MzkgICAgIDEgICAwICAgMjAgICAw ICAxNjA3NiAgMTg2MCBzZWxlY3QgICBTcyAgID8/ICAgICAgMDowMC4wNyAvdXNyL3NiaW4vcnBj YmluZAogIDAgIDIwMDkgICAgIDEgICAwICAgMjAgICAwICA0NjQ2MCAgNDcwNCBzZWxlY3QgICBJ cyAgID8/ICAgICAgMDowMC4wMCAvdXNyL3NiaW4vc3NoZAogIDAgIDIwMTIgICAgIDEgICAwICAg NTIgICAwICA0NjQ2MCAgNDk1NiBzZWxlY3QgICBJcyAgID8/ICAgICAgMDowMC4wMCAvdXNyL3Ni aW4vc3NoZCAtZiAvZXRjL3NzaF9mYWlsb3Zlci9zc2hkX2NvbmZpZyAtbyBQaWRmaWxlPS92YXIv cnVuL3NzaGRfZmFpbG92ZXIucGlkCiAgMCAgMjAyMCAgICAgMSAgIDAgICAyMCAgIDAgIDE2MDcy ICAxNzI4IG5hbnNscCAgIElzICAgPz8gICAgICAwOjAwLjAzIC91c3Ivc2Jpbi9jcm9uIC1zCiAg MCAgMjE4MiAgICAgMSAgIDAgICAyOSAgIDAgIDE4MTQ4ICAxNzE2IHNlbGVjdCAgIElzICAgPz8g ICAgICAwOjAwLjAwIC91c3Ivc2Jpbi9pbmV0ZCAtd1cgLUMgNjAKICAwIDQyNjc1ICAyMDA5ICAg MCAgIDIwICAgMCAgNjU1MjQgIDUzOTYgc2VsZWN0ICAgU3MgICA/PyAgICAgIDA6MDAuMDUgc3No ZDogcm9vdEBwdHMvMCAoc3NoZCkKICAwICAyMjUzICAgICAxICAgMCAgIDUyICAgMCAgMTE5Mjgg IDE1ODAgdHR5aW4gICAgU3MrICB1MCAgICAgIDA6MDAuNzUgL3Vzci9saWJleGVjL2dldHR5IHN0 ZC4xMTUyMDAgdHR5dTAKICAwICAxOTYyICAgICAxICAgMCAgIDUyICAgMCAgMzIyNDQgIDQyMTIg dXdhaXQgICAgSSAgICB2MC0gICAgIDA6MDAuMDEgL3Vzci9sb2NhbC9zYmluL3FldnRkCiAgMCAg MjI0NSAgICAgMSAgIDAgICAyMyAgIDAgIDM2ODA4ICAyMDY4IHdhaXQgICAgIElzICAgdjAgICAg ICAwOjAwLjAxIGxvZ2luIFtwYW1dIChsb2dpbikKICAwICAyMjU4ICAyMjQ1ICAgMCAgIDI1ICAg MCAgMTQzNzYgIDIxNzYgd2FpdCAgICAgSSAgICB2MCAgICAgIDA6MDAuMDAgLXNoIChzaCkKICAw ICAyMjY1ICAyMjU4ICAgMCAgIDIwICAgMCAgMTQyMjQgIDI3MDggdHR5aW4gICAgSSsgICB2MCAg ICAgIDA6MDAuMDEgYmFzaAogIDAgIDIyNDYgICAgIDEgICAwICAgMjAgICAwICAzNjgwOCAgMjA2 OCB3YWl0ICAgICBJcyAgIHYxICAgICAgMDowMC4wMSBsb2dpbiBbcGFtXSAobG9naW4pCiAgMCAz NzI1OSAgMjI0NiAgIDAgICAyMCAgIDAgIDE0Mzc2ICAyMTg4IHdhaXQgICAgIEkgICAgdjEgICAg ICAwOjAwLjAwIC1zaCAoc2gpCiAgMCAzNzI3NSAzNzI1OSAgIDAgICAyMCAgIDAgIDE0MjI0ICAy NzMyIHdhaXQgICAgIEkgICAgdjEgICAgICAwOjAwLjAwIGJhc2gKICAwIDM3ODEyIDM3Mjc1ICAg MCAgIDIwICAgMCAgMTE4NTYgIDE0ODAgcnBjY29uICAgRCsgICB2MSAgICAgIDA6MDAuMDQgdW1v dW50IC1mIC9tbnQvCiAgMCAgMjI0NyAgICAgMSAgIDAgICA1MiAgIDAgIDExOTI4ICAxNTgwIHR0 eWluICAgIElzKyAgdjIgICAgICAwOjAwLjAwIC91c3IvbGliZXhlYy9nZXR0eSBQYyB0dHl2Mgog IDAgIDIyNDggICAgIDEgICAwICAgNTIgICAwICAxMTkyOCAgMTU4MCB0dHlpbiAgICBJcysgIHYz ICAgICAgMDowMC4wMCAvdXNyL2xpYmV4ZWMvZ2V0dHkgUGMgdHR5djMKICAwICAyMjQ5ICAgICAx ICAgMCAgIDUyICAgMCAgMTE5MjggIDE1ODAgdHR5aW4gICAgSXMrICB2NCAgICAgIDA6MDAuMDAg L3Vzci9saWJleGVjL2dldHR5IFBjIHR0eXY0CiAgMCAgMjI1MCAgICAgMSAgIDAgICA1MiAgIDAg IDExOTI4ICAxNTgwIHR0eWluICAgIElzKyAgdjUgICAgICAwOjAwLjAwIC91c3IvbGliZXhlYy9n ZXR0eSBQYyB0dHl2NQogIDAgIDIyNTEgICAgIDEgICAwICAgNTIgICAwICAxMTkyOCAgMTU4MCB0 dHlpbiAgICBJcysgIHY2ICAgICAgMDowMC4wMCAvdXNyL2xpYmV4ZWMvZ2V0dHkgUGMgdHR5djYK ICAwICAyMjUyICAgICAxICAgMCAgIDUyICAgMCAgMTE5MjggIDE1ODAgdHR5aW4gICAgSXMrICB2 NyAgICAgIDA6MDAuMDAgL3Vzci9saWJleGVjL2dldHR5IFBjIHR0eXY3CiAgMCA0MjcxNiA0MjY3 NSAgIDAgICAyMSAgIDAgIDE0Mzc2ICAyMTQ0IHdhaXQgICAgIElzICAgIDAgICAgICAwOjAwLjAw IC1zaCAoc2gpCiAgMCA0Mjc0NiA0MjcxNiAgIDAgICAyMCAgIDAgIDE0MjI0ICAyNjc2IHdhaXQg ICAgIFMgICAgIDAgICAgICAwOjAwLjAxIGJhc2gKICAwIDQzNDE0IDQyNzQ2ICAgMCAgIDIwICAg MCAgMTYxMTYgIDE4NDQgLSAgICAgICAgUisgICAgMCAgICAgIDA6MDAuMDAgcHMgYXhobAo= --bcaec51a7546ddd38e04e8d230fb-- From owner-freebsd-fs@FreeBSD.ORG Wed Oct 16 03:00:24 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 5EC45EF8 for ; Wed, 16 Oct 2013 03:00:24 +0000 (UTC) (envelope-from ericbrowning@skaggscatholiccenter.org) Received: from mail-pd0-f171.google.com (mail-pd0-f171.google.com [209.85.192.171]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3725F29D3 for ; Wed, 16 Oct 2013 03:00:23 +0000 (UTC) Received: by mail-pd0-f171.google.com with SMTP id z10so196068pdj.2 for ; Tue, 15 Oct 2013 20:00:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=OjLPCvtOhx55EJd8QWbta4aCwaw4mhhnAVhS8TovXs0=; b=Ah/iVHoGtVsIPGy11hr8M1IunUxFjUWG++lq6scdmPrg1P4/M8cA5dPDybWmIaWKaA 5oHQLqxPEw18pzr/sky6rwVFyx8OvhYEe1cYsHZkEoGUN7w4QMAX89jKi16TcH+jxjC9 orfN0+k6XL9DTvi42bskp3v95A1N2c7/sNiHO0T7+kDR54b91R2PcRAfvGa1sdc+g3PE WMIqgz/338cAdma8LEwcIdw6soziXsdeJyMoIsZlwhee3TE7vi2jkZpPtfJ82JCb7bvJ aYpoO6tJt5XkLfN1Cc58zDFRpkh9bmxKZsP2BdRekp+L5i4ZtduB8zi7P/wWkmYTEY0n gjpQ== X-Gm-Message-State: ALoCoQnM9JR+dpvlq8pAjVyPdlk0Qc2va0y7MMHhp3Nidlxh1PvhZTH6zpIl5Z5vUgBtvrnjd8fc MIME-Version: 1.0 X-Received: by 10.68.4.197 with SMTP id m5mr542485pbm.46.1381892423308; Tue, 15 Oct 2013 20:00:23 -0700 (PDT) Received: by 10.70.102.133 with HTTP; Tue, 15 Oct 2013 20:00:23 -0700 (PDT) In-Reply-To: <744364739.41705471.1381876907892.JavaMail.root@uoguelph.ca> References: <744364739.41705471.1381876907892.JavaMail.root@uoguelph.ca> Date: Tue, 15 Oct 2013 21:00:23 -0600 Message-ID: Subject: Re: What is the NFS max thread limit From: Eric Browning To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 03:00:24 -0000 Here is the nfsstat -c output for one mac client for one day from one of my busy workers. Are these numbers unreasonable? There are only 14 retries which doesn't seem high so I think the network is ok. Any thoughts on client side optimizations to reduce RPC load? Client Info: RPC Counts: Getattr Setattr Lookup Readlink Read Write 72725986 7586 56972 63 17082 22317 Create Remove Rename Link Symlink Mkdir 4583 2041 3108 0 5 45 Rmdir Readdir RdirPlus Access Mknod Fsstat 140 0 1106 49382068 0 26197 Fsinfo PathConf Commit 1 1 3650 RPC Info: TimedOut Invalid X Replies Retries Requests 0 0 0 14 122252951 Cache Info: Attr Hits Misses Lkup Hits Misses BioR Hits Misses 761115632 3278207 110222766 53251 132864002 16282 BioW Hits Misses BioRLHits Misses BioD Hits Misses 185211 22316 16004 63 12257339 5883 DirE Hits Misses 2990343 23 On Tue, Oct 15, 2013 at 4:41 PM, Rick Macklem wrote: > Eric Browning wrote: > > I'm currently still debugging slow NFS performance on mac clients. > > Running > > 9 stable with a mix of 200 Mac OS X 10.8, 10.7 and 10.6 clients. > > > > Periodically I get this error on the mac side: > > automountd[1766]: set_and_fake_mapent_mntlevel: subdir=/Contents > > error: > > Contents not found in map=-fstab > > > > I'm wondering with 200 clients with five nfsiod threads (~1000 nfsiod > > threads total) am I oversubscribing the nfs server? I can't seem to > > configure any more than 256 threads (nfs_server_flags="-t -n 256"). > > When I > > configure above 256 I get an error about it resetting to 4: > > > > Starting nfsd. > > nfsd: nfsd count 512; reset to 4 > > > > Tomorrow when all the mac clients have restarted I have changed their > > nfsiod threads down to 1 each to see if this is indeed the issue. > > > Well, normally not all clients would be actively reading/writing files > concurrently, so I wouldn't expect your load to need more than 256 nfsd > threads. I also recall that you had a very large Access RPC load and > Access RPCs are not done by nfsiod threads. (nfsiod threads only do > readaheads and write behinds for buffer cache blocks of files.) > > However, if you do want to try more than 256 nfsd threads, you will > need to edit nfsd.c and increase MAXNFSDCNT and then rebuild the nfsd > daemon from these modified sources. > > I have no idea what causes the Mac OS X error, but you could try > posting on one of the Apple mailing lists like > darwin-kernel@lists.apple.com. > > rick > ps: The nfsd.c in head/current/10.0 also has a "--maxthreads" option > that I don't think is limited to MAXNFSDCNT from a glance at the code. > > > Thanks in advance, > > -- > > Eric Browning > > Systems Administrator > > 801-984-7623 > > > > Skaggs Catholic Center > > Juan Diego Catholic High School > > Saint John the Baptist Middle > > Saint John the Baptist Elementary > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > -- Eric Browning Systems Administrator 801-984-7623 Skaggs Catholic Center Juan Diego Catholic High School Saint John the Baptist Middle Saint John the Baptist Elementary From owner-freebsd-fs@FreeBSD.ORG Wed Oct 16 05:49:49 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id ECBA8D52 for ; Wed, 16 Oct 2013 05:49:48 +0000 (UTC) (envelope-from jedrzejczak.michal@gmail.com) Received: from mail-ea0-x22e.google.com (mail-ea0-x22e.google.com [IPv6:2a00:1450:4013:c01::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 774082230 for ; Wed, 16 Oct 2013 05:49:48 +0000 (UTC) Received: by mail-ea0-f174.google.com with SMTP id z15so103556ead.33 for ; Tue, 15 Oct 2013 22:49:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=XJLtg3ErgZ/co3Q8YjgYxww+jjFBPlsVnhbQkKUJ/8g=; b=iUdeXXRjxOBtc+UeWWgNsALLfTmOBWK0kLuaI34eNyoUhKA68aamBb5o8LERAe7tHh VWxI72RwjSZBAPhxNILN287bNorgGg1wWyHAaxgnRJQGpblOV1GJgpXWZAwxVnjqre3h 9q4rnUB2VOyZQeGoE1rcLZ8difzBwhiNq3Bu//++6chkWsq6SAJQ3+KDcBrumKZBHVYS LUMGi3jliWmbTvfjNL0oO2UHcyPs5fghbaj3mav7U+e3U0htnqYcNVPquloBhVV/XhUn 8nvsPJs0npg7TUYnjk2IJjF/t9zrANF1KRIVluOqeSLz5GFBdCc9G7Hjcc5vWB5WX5PU UveA== MIME-Version: 1.0 X-Received: by 10.14.251.76 with SMTP id a52mr173825ees.76.1381902586801; Tue, 15 Oct 2013 22:49:46 -0700 (PDT) Received: by 10.223.2.5 with HTTP; Tue, 15 Oct 2013 22:49:46 -0700 (PDT) Date: Wed, 16 Oct 2013 07:49:46 +0200 Message-ID: Subject: ARC Summary: (THROTTLED) From: =?ISO-8859-2?Q?Micha=B3_J=EAdrzejczak?= To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 05:49:49 -0000 Hello list. I've problem with ARC. ZFSonRoot FreeBSD 9.2 (amd64). zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT zroot 7.25T 6.24T 1.01T 86% 1.00x ONLINE - System Memory: 20.38% 3.10 GiB Active, 13.24% 2.01 GiB Inact 62.48% 9.50 GiB Wired, 3.26% 507.51 MiB Cache 0.63% 97.76 MiB Free, 0.01% 1.92 MiB Gap Real Installed: 16.00 GiB Real Available: 98.17% 15.71 GiB Real Managed: 96.82% 15.21 GiB Logical Total: 16.00 GiB Logical Used: 83.73% 13.40 GiB Logical Free: 16.27% 2.60 GiB Kernel Memory: 5.78 GiB Data: 99.59% 5.76 GiB Text: 0.41% 24.11 MiB Kernel Memory Map: 8.56 GiB Size: 65.15% 5.57 GiB Free: 34.85% 2.98 GiB Page: 1 ------------------------------------------------------------------------ *ARC Summary: (THROTTLED)* Storage pool Version: 5000 Filesystem Version: 5 Memory Throttle Count: 4.86k ARC Misc: Deleted: 719.32m Recycle Misses: 18.46m Mutex Misses: 4.56m Evict Skips: 4.56m ARC Size: 41.41% 5.88 GiB Target Size: (Adaptive) 41.41% 5.88 GiB Min Size (Hard Limit): 12.50% 1.78 GiB Max Size (High Water): 8:1 14.21 GiB ARC Size Breakdown: Recently Used Cache Size: 92.95% 5.47 GiB Frequently Used Cache Size: 7.05% 424.58 MiB ARC Hash Breakdown: Elements Max: 1.16m Elements Current: 32.80% 381.24k Collisions: 643.24m Chain Max: 30 Chains: 103.78k Page: 2 ------------------------------------------------------------------------ ARC Efficiency: 839.63m Cache Hit Ratio: 93.39% 784.14m Cache Miss Ratio: 6.61% 55.50m Actual Hit Ratio: 85.30% 716.21m Data Demand Efficiency: 99.13% 487.70m Data Prefetch Efficiency: 27.32% 57.62m CACHE HITS BY CACHE LIST: Anonymously Used: 8.12% 63.70m Most Recently Used: 21.00% 164.64m Most Frequently Used: 70.34% 551.58m Most Recently Used Ghost: 0.15% 1.17m Most Frequently Used Ghost: 0.39% 3.05m CACHE HITS BY DATA TYPE: Demand Data: 61.65% 483.44m Prefetch Data: 2.01% 15.74m Demand Metadata: 29.69% 232.78m Prefetch Metadata: 6.65% 52.18m CACHE MISSES BY DATA TYPE: Demand Data: 7.68% 4.26m Prefetch Data: 75.47% 41.88m Demand Metadata: 12.49% 6.93m Prefetch Metadata: 4.36% 2.42m Do you have any suggestion? What is wrong? Regards From owner-freebsd-fs@FreeBSD.ORG Wed Oct 16 05:58:21 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 1176BE51; Wed, 16 Oct 2013 05:58:21 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A3D8322C0; Wed, 16 Oct 2013 05:58:20 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r9G5wCTg004405; Wed, 16 Oct 2013 08:58:12 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r9G5wCTg004405 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r9G5wBR7004404; Wed, 16 Oct 2013 08:58:11 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 16 Oct 2013 08:58:11 +0300 From: Konstantin Belousov To: J David Subject: Re: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page Message-ID: <20131016055811.GN3865@kib.kiev.ua> References: <20131015164537.GH3865@kib.kiev.ua> <525D7784.5000808@rice.edu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="kUBUi7JBpjcBtem/" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: "freebsd-fs@freebsd.org" , alc@freebsd.org, Alan Cox X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 05:58:21 -0000 --kUBUi7JBpjcBtem/ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Oct 15, 2013 at 09:23:58PM -0400, J David wrote: > With a kernel updated with debugging code and r253949, it panics during b= oot: >=20 > fifo_open: 0x894212d0 is not exclusive locked but should be > KDB: enter: lock violation > -[ thread pid 963 tid 100353 ] > Stopped at kdb_enter+0x3d: movl $0,kdb_why > db> where > Tracing pid 963 tid 100353 td 0x883aa5e0 > kdb_enter(8112e312,8112e4ee,894212d0,8112e24d,812746d4,...) at > kdb_enter+0x3d/frame 0xbf656980 > assert_vop_elocked(894212d0,811069b9,df,1,0,...) at > assert_vop_elocked+0xbe/frame 0xbf6569b0 > fifo_open(bf656ae8,8117a85a,8114f47a,81423bac,bf656a74,...) at > fifo_open+0x36/frame 0xbf656a10 > VOP_OPEN_APV(818a3f30,bf656ae8,100,10000,10000,...) at > VOP_OPEN_APV+0xca/frame 0xbf656a40 > vn_open_cred(bf656b60,bf656bec,c08,0,879ae180,8832d428) at > vn_open_cred+0x5a5/frame 0xbf656b18 > vn_open(bf656b60,bf656bec,c08,8832d428,0,...) at vn_open+0x3b/frame 0xbf6= 56b38 > kern_openat(883aa5e0,ffffff9c,804a7b2,0,4,7fbfde08) at > kern_openat+0x1cb/frame 0xbf656c20 > sys_open(883aa5e0,bf656cc8,811748f4,81126582,1,...) at > sys_open+0x38/frame 0xbf656c40 > syscall(bf656d08) at syscall+0x2da/frame 0xbf656cfc > Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xbf656cfc > --- syscall (5, FreeBSD ELF32, sys_open), eip =3D 0x2815e29b, esp =3D > 0x7fbfdd6c, ebp =3D 0x7fbfdd78 --- >=20 > If possible, I'll see about booting it in single user mode and > disabling as much junk as possible that might be touching FIFO's, as > it's unlike that problem is related to the ZFS issue. You want r255728. --kUBUi7JBpjcBtem/ Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJSXiryAAoJEJDCuSvBvK1BW4YP/jiHwNUk+PyXDB2oURLWYfSH McC0sUGyBHQOAtuL7cZuXB6IRViSUfVkhfsCR3oiEujBMTTNlasTHMtluGSe9w7E r3Koy3UyWrBcsBYNYUTWQre0y8SzPNSAYJDH3jhzDW3tb5vB0k6D3XEz32zTOt7G 60Iw1SpEvm8aoxeL/Yvv6Uy/HtL+lxbQ96bcvFvyZo3F1NhDGa7jq3fBSu+j9DXF d2MuI+IB3DyxvexnGZboALEadxOxP9bqCZxfZO0nuiFZgy6/iRd7bqZvLs6v++AC np+n/P6horPTopTi7VtzUjHBwfRRc0D5pIbLUsp+PWLbM4HRsUPJa6Lq846aeqvc jQxKDw5R6ZPVjK/2sWv8n677WlpDnoSeb+vpa1GkJYVrZIjUmn0ajzshzDcFYh2l e2+aj0Wp8CTElg7wq8iJMJPaR+nHWtneGRT8FGyEEa9bHLK9eqTWtawy1jXfSqtG 1FeB9XsvHKPyCrU1GrMaekfa+EuO7+eMQTuFiEVxSRaa9PqpLNYz2QK7vSHFKqZR TI1ENtlHlXo4MHm1yMKUu69XvL5UDFVsH9txESPgFPn2hjwcHBGwTok9ZxpdhyPn OKLfIV6wUucZftzWfxYHaTWtEAmGDkaZ4VGcoHAg0q9OF6sThyii0hJjTyvopdAu WmKzL8oWg1BZO913c0A0 =O5M3 -----END PGP SIGNATURE----- --kUBUi7JBpjcBtem/-- From owner-freebsd-fs@FreeBSD.ORG Wed Oct 16 06:00:16 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id DC98BEEE; Wed, 16 Oct 2013 06:00:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5E65522E3; Wed, 16 Oct 2013 06:00:16 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r9G60AXs004609; Wed, 16 Oct 2013 09:00:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r9G60AXs004609 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r9G60AVt004608; Wed, 16 Oct 2013 09:00:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 16 Oct 2013 09:00:10 +0300 From: Konstantin Belousov To: Alan Cox Subject: Re: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page Message-ID: <20131016060010.GO3865@kib.kiev.ua> References: <20131015164537.GH3865@kib.kiev.ua> <525D7784.5000808@rice.edu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MmQIYbZiCoQ2kDro" Content-Disposition: inline In-Reply-To: <525D7784.5000808@rice.edu> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: "freebsd-fs@freebsd.org" , alc@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 06:00:16 -0000 --MmQIYbZiCoQ2kDro Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Oct 15, 2013 at 12:12:36PM -0500, Alan Cox wrote: > On 10/15/2013 11:45, Konstantin Belousov wrote: > > On Tue, Oct 15, 2013 at 11:17:53AM -0400, J David wrote: > >> What are the necessary loader.conf / kernel config invocations to make > >> ZFS stable on 9.2-RELEASE i386 node with 3GB RAM? > >> > >> This machine was rock solid under 8.4, but since upgrading to 9.2 it > >> has been a disaster. It crashes every few hours with "panic: > >> pmap_enter: attempted pmap_enter on 4MB page." > >> > >> Here are a couple of stack traces: > >> > >> panic: pmap_enter: attempted pmap_enter on 4MB page > >> cpuid =3D 0 > >> KDB: stack backtrace: > >> #0 0xc0afc092 at kdb_backtrace+0x52 > >> #1 0xc0ac249c at panic+0x1bc > >> #2 0xc0f1825d at pmap_enter+0x63d > >> #3 0xc0d34ee5 at vm_fault_hold+0x1c45 > >> #4 0xc0d33262 at vm_fault+0x82 > >> #5 0xc0f1f5d6 at trap_pfault+0x186 > >> #6 0xc0f1ed4b at trap+0x51b > >> #7 0xc0f0887c at calltrap+0x6 > >> #8 0xc16770b6 at zio_execute+0x116 > >> #9 0xc15d9460 at taskq_run_safe+0x10 > >> #10 0xc0b08f26 at taskqueue_run_locked+0xe6 > >> #11 0xc0b097f7 at taskqueue_thread_loop+0xb7 > >> #12 0xc0a90a73 at fork_exit+0xa3 > >> #13 0xc0f08924 at fork_trampoline+0x8 > >> > >> panic: pmap_enter: attempted pmap_enter on 4MB page > >> cpuid =3D 1 > >> KDB: stack backtrace: > >> #0 0xc0afc092 at kdb_backtrace+0x52 > >> #1 0xc0ac249c at panic+0x1bc > >> #2 0xc0f1825d at pmap_enter+0x63d > >> #3 0xc0d34ee5 at vm_fault_hold+0x1c45 > >> #4 0xc0d33262 at vm_fault+0x82 > >> #5 0xc0f1f5d6 at trap_pfault+0x186 > >> #6 0xc0f1ed4b at trap+0x51b > >> #7 0xc0f0887c at calltrap+0x6 > >> #8 0xc1679bdf at zio_dva_allocate+0x9f > >> #9 0xc16770b6 at zio_execute+0x116 > >> #10 0xc15d9460 at taskq_run_safe+0x10 > >> #11 0xc0b08f26 at taskqueue_run_locked+0xe6 > >> #12 0xc0b097f7 at taskqueue_thread_loop+0xb7 > >> #13 0xc0a90a73 at fork_exit+0xa3 > >> #14 0xc0f08924 at fork_trampoline+0x8 > >> > >> For the last crash, top was running and this is what was on the screen > >> when it died: > >> > >> CPU: 0.6% user, 0.0% nice, 2.2% system, 0.2% interrupt, 97.1% idle > >> Mem: 442M Active, 91M Inact, 79M Wired, 3764K Cache, 4960K Buf, 2380M = Free > >> ARC: 40M Total, 8520K MFU, 30M MRU, 304K Anon, 494K Header, 1062K Other > >> Swap: 4096M Total, 4096M Free > >> Write failed: Broken pipe > >> > >> It doesn't seem to matter what KVA_PAGES, vm.kmem_size, > >> vfs.zfs.arc_max or vfs.zfs.vdev.cache.size is set to, and the ZFS > >> tuning guides in the wiki (albeit appearing to be dating to the 7.x > >> era) provides guidelines for tuning down to 768M. So 3GB should be > >> enough for a machine that is 99% idle. (Particularly given that it > >> dies with >2GB free.) > >> > >> This seems to be specific to i386, this problem hasn't cropped up on > >> any amd64 nodes. No compression, deduplication, snapshots or anything > >> like that is in use. > > It seems that i386 pmap_enter() cannot deal with the superpages, while > > amd64 can. On the other hand, I am not quite sure that this is your > > problem, because you get traps on accessing KVA, and I suspect that > > ZFS uses wired memory. Obtain the kernel dump and do full backtrace > > for the panic. >=20 > Try MFCing r253949. I'm not confident that r253949 is the fix for this > particular problem, but it should be MFCed. Probably not. In addition, please note that the reporter has i386. >=20 > > Anyway, I believe that i386 pmap_enter() should do a demotion when need= ed. >=20 > A kernel space pmap_enter should never be overwriting an existing > superpage mapping. Yes, this is why I requested the backtrace. Still, I also think that demotion is missed in pmap_enter() on i386. >=20 > > Below is the prototyped change for this. > > > > diff --git a/sys/i386/i386/pmap.c b/sys/i386/i386/pmap.c > > index 64bf1a3..91453da 100644 > > --- a/sys/i386/i386/pmap.c > > +++ b/sys/i386/i386/pmap.c > > @@ -3473,17 +3473,21 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_prot= _t access, vm_page_t m, > > PMAP_LOCK(pmap); > > sched_pin(); > > =20 > > - /* > > - * In the case that a page table page is not > > - * resident, we are creating it here. > > - */ > > - if (va < VM_MAXUSER_ADDRESS) { > > + pde =3D pmap_pde(pmap, va); > > + if ((*pde & PG_PS) !=3D 0) { > > + /* PG_V is asserted by pmap_demote_pde */ > > + pmap_demote_pde(pmap, pde, va); > > + if (va < VM_MAXUSER_ADDRESS) { > > + mpte =3D PHYS_TO_VM_PAGE(*pde & PG_FRAME); > > + mpte->wire_count++; > > + } > > + } else if (va < VM_MAXUSER_ADDRESS) { > > + /* > > + * In the case that a page table page is not resident, > > + * we are creating it here. > > + */ > > mpte =3D pmap_allocpte(pmap, va, M_WAITOK); > > } > > - > > - pde =3D pmap_pde(pmap, va); > > - if ((*pde & PG_PS) !=3D 0) > > - panic("pmap_enter: attempted pmap_enter on 4MB page"); > > pte =3D pmap_pte_quick(pmap, va); > > =20 > > /* --MmQIYbZiCoQ2kDro Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJSXitpAAoJEJDCuSvBvK1B0HEQAIxJilHSStiud8mlKHWYjFdt idgzYT3pOSdmaYTUNSyi5j9SG8PZ2vU+y5ROrxwD5kZfDyA2IXbF/8jqOU/hTzJd gjE3gZnHTlc0eEaFmyy5MmXgrKnAgEcPwk+IN+7kBjzldnHpC1m0AKHDUcIpZiUd Pih3gIFXHF/234tw6iXw3DWhPvIwjiKoZrTVBg6/CqFXytnN3MZNj3WNn0mcL4V6 OigUH3+vDjbEUYD/Q3PQI9Fixu5WyEsk8BWvP6GvvTsh8FBE7Ztlzm4R505L41TW LoVHryke3web8rlFtSl7Jd6NDn+5XCBIEWFw79loZ73uiqAb6JXK/31LH5wGvP9v 57CMBIbDgn89G4s6vt3lP2gOl3UdneIa2xT0lx8cuEWaO+YmQFLw+C1M76IXP1U2 D9Pbch5u1lvETV+ODne6DMX/jg5poRWoebODhyW1R8F9xx7Bll9G2a7Y6niZbog9 il6B8+XVWouqEMW3R5lhOu5cg/4kdsrp3Ddw/F2fv9ezRgNTcHIH3GhADfio58u+ x3zEtaUYujnF8+T416U2xtWlLuXbViTw+KRoMqfX7cz3CHY8wL6NWJYgNeE9YNB4 OxYDQJsKGnGA8JBpRdhsShAcBmaeu41ny3KHv8VggiYDJ0m2IF6HhmTh7a5yKPTy bOUJU1poHgv7taDv3bcJ =7P+/ -----END PGP SIGNATURE----- --MmQIYbZiCoQ2kDro-- From owner-freebsd-fs@FreeBSD.ORG Wed Oct 16 12:23:39 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6505EAFB for ; Wed, 16 Oct 2013 12:23:39 +0000 (UTC) (envelope-from maurizio.vairani@cloverinformatica.it) Received: from smtpdb6.aruba.it (smtpdb6.aruba.it [62.149.158.248]) by mx1.freebsd.org (Postfix) with ESMTP id C3FE72949 for ; Wed, 16 Oct 2013 12:23:38 +0000 (UTC) Received: from cloverinformatica.it ([188.10.129.202]) by smtpcmd02.ad.aruba.it with bizsmtp id doNR1m00N4N8xN401oNRTW; Wed, 16 Oct 2013 14:22:26 +0200 Received: from [192.168.0.99] (MAURIZIO-PC [192.168.0.99]) by cloverinformatica.it (Postfix) with ESMTP id BD9A067E2; Wed, 16 Oct 2013 14:22:25 +0200 (CEST) Message-ID: <525E8501.7080302@cloverinformatica.it> Date: Wed, 16 Oct 2013 14:22:25 +0200 From: Maurizio Vairani User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1 MIME-Version: 1.0 To: Garrett Wollman Subject: Re: How to unstick ZFS resilver? References: <21084.48646.196295.776944@hergotha.csail.mit.edu> In-Reply-To: <21084.48646.196295.776944@hergotha.csail.mit.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 12:23:39 -0000 On 15/10/2013 06:01, Garrett Wollman wrote: > I have a large (88-drive) zpool in which a drive was recently > replaced. (The pool has a bunch of duff Toshiba MK2001TRKB drives -- > never ever pay money for these! -- and I'm trying to replace them one > by one before they fail completely.) The resilver on the first drive > replacement has been taking much much too long, and currently it's > stuck in this state: > > pool: export > state: DEGRADED > status: One or more devices is currently being resilvered. The pool will > continue to function, possibly in a degraded state. > action: Wait for the resilver to complete. > scan: resilver in progress since Wed Oct 9 14:54:47 2013 > 86.5T scanned out of 86.8T at 1/s, (scan is slow, no estimated time) > 982G resilvered, 99.62% done > > The overall progress hasn't changed in twelve hours, even across a > reboot, and the server is fairly lightly loaded. Searching the Web is > no help; can anyone suggest a remedial action? (This is on > 9.1-RELEASE, with our local patches, and all the drives are SAS.) > > In exchange, I offer the following DTrace script which I used to > identify the slow SAS drives: > > #!/usr/sbin/dtrace -s > > #pragma D option quiet > #pragma D option dynvarsize=2m > > inline int TOO_SLOW = 100000000; /* 100 ms */ > > dtrace:::BEGIN > { > printf("Tracing... Hit Ctrl-C to end.\n"); > } > > fbt::dastrategy:entry > { > start_time[(struct buf *)arg0] = timestamp; > } > > fbt::dadone:entry > /(this->bp = (struct buf *)args[1]->ccb_h.periph_priv.entries[1].ptr)&& start_time[this->bp]&& (timestamp - start_time[this->bp])> TOO_SLOW/ > { > @[strjoin("da", lltostr(args[0]->unit_number))] = count(); > start_time[this->bp] = 0; > } > > -GAWollman > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" Before replace the drives. Have you upgrade the MK2001TRKB firmware ? From newegg.com: "Make sure the firmware is at 0106. The issue with the old firmware (0105) has to do with the drive going into and then coming out of Idle mode B. An over voltage condition can possibly occur which will damage the head on the drive. This damage can lead to an increased level of read errors. 0106 firmware fixes the issue in drives that have not yet failed but it does nothing for drives that have already failed and unfortunately there is no way to recover those drives." Maurizio From owner-freebsd-fs@FreeBSD.ORG Wed Oct 16 18:45:14 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C15D0A95; Wed, 16 Oct 2013 18:45:14 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ie0-x22a.google.com (mail-ie0-x22a.google.com [IPv6:2607:f8b0:4001:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8A3972332; Wed, 16 Oct 2013 18:45:14 +0000 (UTC) Received: by mail-ie0-f170.google.com with SMTP id at1so2139158iec.15 for ; Wed, 16 Oct 2013 11:45:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=0fqe3I1rVBHBygeF/MgvNym047FmmWbgNE2nfP21K6U=; b=BWn/Rk3OE79d8CVCFGAkJRSwzbty3ATxKXLwtjYqhhEzHN383Ai/shrU3AR152oKFi Tz9/xcNW0OjYMUi1dUlrKho0UBsjnZ2AsB/3OShvrolZdngBU9qhk5KfrxiLINzsLIxx NJ8RDntcXisjLjxo9QMU4U2D+P9isNpyw2Tl/BbtPWLLGi555tPqhIegle6HMfFx9LEK BfRU2tI0R5J8TQoVl4M76OtE12n0cdti8+YMnT+rv23QcqZOIb2fhQSk2PYGwmH83xWe FL61GvqnKzKn0VLEjT/GbD5I6q9fXHIcOobyTOgZZLN1siqrWs21Bg72+22gWxrAQBl7 GzOg== MIME-Version: 1.0 X-Received: by 10.50.117.40 with SMTP id kb8mr22788197igb.60.1381949113825; Wed, 16 Oct 2013 11:45:13 -0700 (PDT) Sender: jdavidlists@gmail.com Received: by 10.43.180.131 with HTTP; Wed, 16 Oct 2013 11:45:13 -0700 (PDT) In-Reply-To: <20131016060010.GO3865@kib.kiev.ua> References: <20131015164537.GH3865@kib.kiev.ua> <525D7784.5000808@rice.edu> <20131016060010.GO3865@kib.kiev.ua> Date: Wed, 16 Oct 2013 14:45:13 -0400 X-Google-Sender-Auth: DNnYna5aKetNDTlkZQZdxCNeRTA Message-ID: Subject: Re: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page From: J David To: Konstantin Belousov Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-fs@freebsd.org" , alc@freebsd.org, Alan Cox X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 18:45:14 -0000 On Tue, Oct 15, 2013 at 12:12:36PM -0500, Alan Cox wrote: > Try MFCing r253949. I'm not confident that r253949 is the fix for this > particular problem, but it should be MFCed. On Wed, Oct 16, 2013 at 1:58 AM, Konstantin Belousov wrote: > You want r255728. OK, that node is now running with r253949 (i386 portion only), r255728, and a full complement of debug options. If it crashes again, I will report back with the stack trace. If it does not crash in a reasonable time (24-48 hours), I will report that as well. So far it does not have the "prototyped change" since it sounds like the stack trace of the situation as it stands should be done first to confirm suspicions about the nature of the problem. Also, given that r253949 is already present, it seems better to limit the number of changes being made at once. Thanks! From owner-freebsd-fs@FreeBSD.ORG Wed Oct 16 20:58:48 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0597559A; Wed, 16 Oct 2013 20:58:48 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-ie0-x22b.google.com (mail-ie0-x22b.google.com [IPv6:2607:f8b0:4001:c03::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BE5D62A67; Wed, 16 Oct 2013 20:58:47 +0000 (UTC) Received: by mail-ie0-f171.google.com with SMTP id tp5so2442147ieb.30 for ; Wed, 16 Oct 2013 13:58:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=ZN/tQu1bp1WckDKV72eif8WoZLKsTcELiGQuTAKpyV4=; b=OYUSF22ohDNFIGLAs24L4tt+uSr6kfjkgizlOSqiZQjIVv/VD5xekmp9u5G49yaNsp aSeJnTNZLaY6fG5ESJ0BhXs34DizmY78buy+QTy6XBG2k7CXyoW2xyLlgThIBSx44hx8 5o9XcU6gG/9DqTsr66CFzCcF+sbs7osWSanQWkmDKye2Gl2K3onQGsIs9OTSba0G39Q8 u2MoMIw1As1DCPiczAzrJx48HEO+oZVB/Qv/dCtRf3d81uqwmwuVxKnFSNNflOtjubWU WkCiup4xFeLXGV+LC8lVHWlWZdqhMWvBhibGA8x8e94ciXmcjL2vq4dfyoZePt9jpvaY EP5Q== MIME-Version: 1.0 X-Received: by 10.50.107.70 with SMTP id ha6mr2352347igb.60.1381957127235; Wed, 16 Oct 2013 13:58:47 -0700 (PDT) Sender: jdavidlists@gmail.com Received: by 10.43.180.131 with HTTP; Wed, 16 Oct 2013 13:58:47 -0700 (PDT) In-Reply-To: References: <20131015164537.GH3865@kib.kiev.ua> <525D7784.5000808@rice.edu> <20131016060010.GO3865@kib.kiev.ua> Date: Wed, 16 Oct 2013 16:58:47 -0400 X-Google-Sender-Auth: ld55cn20Li6A0iJOOnP32paQMfo Message-ID: Subject: Re: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page From: J David To: Konstantin Belousov Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-fs@freebsd.org" , alc@freebsd.org, Alan Cox X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 20:58:48 -0000 OK, that didn't take long: Tracing pid 3153 tid 100432 td 0x885d28d0 kdb_enter(8111d67f,8111d67f,811736e8,bf7511c0,80a9c47f,...) at kdb_enter+0x3d/frame 0xbf751170 panic(811736e8,0,811733cf,dbf,81479a24,...) at panic+0x1dc/frame 0xbf7511b4 pmap_enter(81487c78,88d92000,7,82fab0c8,7,...) at pmap_enter+0x6cc/frame 0xbf751200 vm_fault_hold(81feb000,88d92000,1,0,0,...) at vm_fault_hold+0x1b41/frame 0xbf751368 vm_fault(81feb000,88d92000,1,0,bf7513f4,...) at vm_fault+0x82/frame 0xbf751390 trap_pfault(88d92334,81fed780,246,81feca80,88271b80,...) at trap_pfault+0x21e/frame 0xbf751408 trap(bf751558) at trap+0x58c/frame 0xbf75154c calltrap() at calltrap+0x6/frame 0xbf75154c --- trap 0xc, eip = 0x80afbc7d, esp = 0xbf751598, ebp = 0xbf7515dc --- witness_checkorder(81fed988,9,81156ee6,9f9,0,...) at witness_checkorder+0x34d/frame 0xbf7515dc _mtx_lock_flags(81fed988,0,81156ee6,9f9,80a98adb,...) at _mtx_lock_flags+0x75/frame 0xbf75160c uma_zfree_arg(81fecd80,8945b800,88c5a9fc,ffffffff,0,...) at uma_zfree_arg+0x54/frame 0xbf751648 free(8945b800,818dc100,bf7516e8,817f24a9,8945b800,...) at free+0xb6/frame 0xbf751678 zfs_kmem_free(8945b800,124,892f71e0,0,0,...) at zfs_kmem_free+0x19/frame 0xbf751688 zap_lockdir(88452400,115,0,0,1,...) at zap_lockdir+0x359/frame 0xbf7516e8 zap_lookup_norm(88452400,115,0,bf7518b8,8,...) at zap_lookup_norm+0x49/frame 0xbf75172c zap_lookup(88452400,115,0,bf7518b8,8,...) at zap_lookup+0x69/frame 0xbf751768 zfs_dirent_lock(bf751830,88a5b0fc,bf7518b8,bf75182c,6,...) at zfs_dirent_lock+0x4e9/frame 0xbf7517e8 zfs_dirlook(88a5b0fc,bf7518b8,bf751bbc,0,0,...) at zfs_dirlook+0x1bb/frame 0xbf751848 zfs_lookup(bf751bbc,bf751bd0,0,89eee080,0,...) at zfs_lookup+0x288/frame 0xbf75189c zfs_freebsd_lookup(bf751a18,8117a717,bf7519e4,80b3ead8,88d92328,...) at zfs_freebsd_lookup+0x6c/frame 0xbf7519c8 VOP_CACHEDLOOKUP_APV(818a3e1c,bf751a18,bf751bd0,0,0,...) at VOP_CACHEDLOOKUP_APV+0xca/frame 0xbf7519f8 vfs_cache_lookup(bf751ab8,8117a697,8114f476,8112db4f,812746d4,...) at vfs_cache_lookup+0xec/frame 0xbf751a40 VOP_LOOKUP_APV(818a3e1c,bf751ab8,bf751bd0,33a,885d28d0,...) at VOP_LOOKUP_APV+0xca/frame 0xbf751a70 lookup(bf751b90,8112cf73,106,cc,8,...) at lookup+0x646/frame 0xbf751ae0 namei(bf751b90,89ed5904,885d28d0,bf751c18,0,...) at namei+0x591/frame 0xbf751b68 kern_accessat(885d28d0,ffffff9c,280ff800,0,0,0) at kern_accessat+0xdd/frame 0xbf751c20 sys_access(885d28d0,bf751cc8,81174954,8112194d,8111ddc0,...) at sys_access+0x39/frame 0xbf751c40 syscall(bf751d08) at syscall+0x2da/frame 0xbf751cfc Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xbf751cfc --- syscall (33, FreeBSD ELF32, sys_access), eip = 0x280f0ec7, esp = 0x7fbfc4ec, ebp = 0x7fbfc510 --- From owner-freebsd-fs@FreeBSD.ORG Thu Oct 17 03:28:32 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 27281B9 for ; Thu, 17 Oct 2013 03:28:32 +0000 (UTC) (envelope-from shawn.wallbridge@imaginaryforces.com) Received: from barracuda.imaginaryforces.com (199.193.209.132.static.oneiricsys.com [199.193.209.132]) by mx1.freebsd.org (Postfix) with ESMTP id F1ABC2D97 for ; Thu, 17 Oct 2013 03:28:31 +0000 (UTC) X-ASG-Debug-ID: 1381980510-0413ad4d7f229330001-3nHGF7 Received: from newman.IMAGINARYFORCES.LOCAL (newman.imaginaryforces.local [192.168.23.34]) by barracuda.imaginaryforces.com with ESMTP id aHyCz0hvL0f1FPOI for ; Wed, 16 Oct 2013 20:28:30 -0700 (PDT) X-Barracuda-Envelope-From: shawn.wallbridge@imaginaryforces.com Received: from NEWMAN.IMAGINARYFORCES.LOCAL ([192.168.23.34]) by newman.IMAGINARYFORCES.LOCAL ([192.168.23.34]) with mapi id 14.01.0438.000; Wed, 16 Oct 2013 20:32:27 -0700 From: Shawn Wallbridge To: "freebsd-fs@freebsd.org" Subject: Urgent help needed, fsck segfault's on a 9.2-RELEASE machine Thread-Topic: Urgent help needed, fsck segfault's on a 9.2-RELEASE machine X-ASG-Orig-Subj: Urgent help needed, fsck segfault's on a 9.2-RELEASE machine Thread-Index: AQHOyumA7j6gT1qkakqopQBPm1f4bQ== Date: Thu, 17 Oct 2013 03:32:26 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.23.68] Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Barracuda-Connect: newman.imaginaryforces.local[192.168.23.34] X-Barracuda-Start-Time: 1381980510 X-Barracuda-URL: http://barracuda.imaginaryforces.com:8000/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at imaginaryforces.com X-Barracuda-Bayes: INNOCENT GLOBAL 0.0000 1.0000 -2.0210 X-Barracuda-Spam-Score: -2.02 X-Barracuda-Spam-Status: No, SCORE=-2.02 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=8.0 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.2.141528 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Oct 2013 03:28:32 -0000 Our fileserver panic'ed tonight, after it came back up, it couldn't background fsck one of the file systems. I brought it back up by commenting that filesystem out. Now I am trying to do an fsck on the filesystem, but it segfaults. root@mercury:/var/log # fsck /dev/da0p1 fsck: Could not determine filesystem type root@mercury:/var/log # fsck -t ufs /dev/da0p1 ** /dev/da0p1 fsck: /dev/da0p1: Segmentation fault: 11 It's running 9.2-RELEASE uname -a FreeBSD mercury 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 The filesystem is on a LSI RAID Card, 9750-24i4e. The controller says everything is fine. I tried looking at the core file, but it doesn't mean anything to me. # gdb core fsck_ufs.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...core: No such file or directory. Core was generated by `fsck_ufs'. Program terminated with signal 11, Segmentation fault. #0 0x0000000000405cce in ?? () (gdb) bt #0 0x0000000000405cce in ?? () #1 0xffffffff90000000 in ?? () #2 0x000000000040ef6b in ?? () #3 0x0000000000624e10 in ?? () #4 0x000000080061f8cb in ?? () #5 0x0000007b7100ff00 in ?? () #6 0x00000000000121a0 in ?? () #7 0x0000007b00000005 in ?? () #8 0x00000000525f4c3b in ?? () #9 0x00000000021036a8 in ?? () #10 0x00000000525f4c3b in ?? () #11 0x00000000021036a8 in ?? () #12 0x00000000525f4c3b in ?? () #13 0x00000000021036a8 in ?? () #14 0x0000000000000000 in ?? () (gdb) quit ________________________________ This e-mail is intended only for the named person or entity to which it is = addressed and contains valuable business information that is proprietary, p= rivileged, confidential and/or otherwise protected from disclosure. If you = received this e-mail in error, any review, use, dissemination, distribution= or copying of this e-mail is strictly prohibited. Please notify us immedia= tely of the error via e-mail to postmaster@imaginaryforces.c= om and please delete the e-mail from your system, retaining no copies in an= y media. We appreciate your cooperation. ...imaginaryforces.com...=0D From owner-freebsd-fs@FreeBSD.ORG Thu Oct 17 03:33:29 2013 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id B8F0E266 for ; Thu, 17 Oct 2013 03:33:29 +0000 (UTC) (envelope-from shawn.wallbridge@imaginaryforces.com) Received: from barracuda.imaginaryforces.com (199.193.209.132.static.oneiricsys.com [199.193.209.132]) by mx1.freebsd.org (Postfix) with ESMTP id 90BB02DF7 for ; Thu, 17 Oct 2013 03:33:29 +0000 (UTC) X-ASG-Debug-ID: 1381979945-0413ad4d7f2291d0001-ETJDRd Received: from newman.IMAGINARYFORCES.LOCAL (newman.imaginaryforces.local [192.168.23.34]) by barracuda.imaginaryforces.com with ESMTP id 6sWwGaXM0vywgNfB for ; Wed, 16 Oct 2013 20:19:05 -0700 (PDT) X-Barracuda-Envelope-From: shawn.wallbridge@imaginaryforces.com Received: from NEWMAN.IMAGINARYFORCES.LOCAL ([192.168.23.34]) by newman.IMAGINARYFORCES.LOCAL ([192.168.23.34]) with mapi id 14.01.0438.000; Wed, 16 Oct 2013 20:23:02 -0700 From: Shawn Wallbridge To: "fs@freebsd.org" Subject: Urgent help needed, fsck segfault's on a 9.2-RELEASE machine Thread-Topic: Urgent help needed, fsck segfault's on a 9.2-RELEASE machine X-ASG-Orig-Subj: Urgent help needed, fsck segfault's on a 9.2-RELEASE machine Thread-Index: AQHOyugvdmlxAGVTWkSV4jSHudZUxg== Date: Thu, 17 Oct 2013 03:23:02 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.23.68] Content-Type: text/plain; charset="us-ascii" Content-ID: <3C992BFC4161F04580A8C69ECD5301B5@IMAGINARYFORCES.COM> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Barracuda-Connect: newman.imaginaryforces.local[192.168.23.34] X-Barracuda-Start-Time: 1381979945 X-Barracuda-URL: http://barracuda.imaginaryforces.com:8000/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at imaginaryforces.com X-Barracuda-Bayes: INNOCENT GLOBAL 0.0000 1.0000 -2.0210 X-Barracuda-Spam-Score: -2.02 X-Barracuda-Spam-Status: No, SCORE=-2.02 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=8.0 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.2.141528 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Oct 2013 03:33:29 -0000 Our fileserver panic'ed tonight, after it came back up, it couldn't background fsck one of the file systems. I brought it back up by commenting that filesystem out. Now I am trying to do an fsck on the filesystem, but it segfaults. root@mercury:/var/log # fsck /dev/da0p1 fsck: Could not determine filesystem type root@mercury:/var/log # fsck -t ufs /dev/da0p1 ** /dev/da0p1 fsck: /dev/da0p1: Segmentation fault: 11 It's running 9.2-RELEASE uname -a FreeBSD mercury 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 The filesystem is on a LSI RAID Card, 9750-24i4e. The controller says everything is fine. I tried looking at the core file, but it doesn't mean anything to me. # gdb core fsck_ufs.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...core: No such file or directory. Core was generated by `fsck_ufs'. Program terminated with signal 11, Segmentation fault. #0 0x0000000000405cce in ?? () (gdb) bt #0 0x0000000000405cce in ?? () #1 0xffffffff90000000 in ?? () #2 0x000000000040ef6b in ?? () #3 0x0000000000624e10 in ?? () #4 0x000000080061f8cb in ?? () #5 0x0000007b7100ff00 in ?? () #6 0x00000000000121a0 in ?? () #7 0x0000007b00000005 in ?? () #8 0x00000000525f4c3b in ?? () #9 0x00000000021036a8 in ?? () #10 0x00000000525f4c3b in ?? () #11 0x00000000021036a8 in ?? () #12 0x00000000525f4c3b in ?? () #13 0x00000000021036a8 in ?? () #14 0x0000000000000000 in ?? () (gdb) quit ________________________________ This e-mail is intended only for the named person or entity to which it is = addressed and contains valuable business information that is proprietary, p= rivileged, confidential and/or otherwise protected from disclosure. If you = received this e-mail in error, any review, use, dissemination, distribution= or copying of this e-mail is strictly prohibited. Please notify us immedia= tely of the error via e-mail to postmaster@imaginaryforces.c= om and please delete the e-mail from your system, retaining no copies in an= y media. We appreciate your cooperation. ...imaginaryforces.com...=0D From owner-freebsd-fs@FreeBSD.ORG Thu Oct 17 07:10:01 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0E767A3F; Thu, 17 Oct 2013 07:10:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8899C2805; Thu, 17 Oct 2013 07:10:00 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r9H79o3x028442; Thu, 17 Oct 2013 10:09:50 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r9H79o3x028442 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r9H79nx1028441; Thu, 17 Oct 2013 10:09:49 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 Oct 2013 10:09:49 +0300 From: Konstantin Belousov To: J David Subject: Re: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page Message-ID: <20131017070949.GA3865@kib.kiev.ua> References: <20131015164537.GH3865@kib.kiev.ua> <525D7784.5000808@rice.edu> <20131016060010.GO3865@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="azskX66S5GHWoEK7" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: "freebsd-fs@freebsd.org" , alc@freebsd.org, Alan Cox X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Oct 2013 07:10:01 -0000 --azskX66S5GHWoEK7 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Oct 16, 2013 at 04:58:47PM -0400, J David wrote: > OK, that didn't take long: >=20 > Tracing pid 3153 tid 100432 td 0x885d28d0 > kdb_enter(8111d67f,8111d67f,811736e8,bf7511c0,80a9c47f,...) at > kdb_enter+0x3d/frame 0xbf751170 > panic(811736e8,0,811733cf,dbf,81479a24,...) at panic+0x1dc/frame 0xbf7511= b4 > pmap_enter(81487c78,88d92000,7,82fab0c8,7,...) at > pmap_enter+0x6cc/frame 0xbf751200 > vm_fault_hold(81feb000,88d92000,1,0,0,...) at > vm_fault_hold+0x1b41/frame 0xbf751368 > vm_fault(81feb000,88d92000,1,0,bf7513f4,...) at vm_fault+0x82/frame 0xbf7= 51390 > trap_pfault(88d92334,81fed780,246,81feca80,88271b80,...) at > trap_pfault+0x21e/frame 0xbf751408 > trap(bf751558) at trap+0x58c/frame 0xbf75154c > calltrap() at calltrap+0x6/frame 0xbf75154c > --- trap 0xc, eip =3D 0x80afbc7d, esp =3D 0xbf751598, ebp =3D 0xbf7515dc = --- > witness_checkorder(81fed988,9,81156ee6,9f9,0,...) at > witness_checkorder+0x34d/frame 0xbf7515dc > _mtx_lock_flags(81fed988,0,81156ee6,9f9,80a98adb,...) at > _mtx_lock_flags+0x75/frame 0xbf75160c > uma_zfree_arg(81fecd80,8945b800,88c5a9fc,ffffffff,0,...) at > uma_zfree_arg+0x54/frame 0xbf751648 > free(8945b800,818dc100,bf7516e8,817f24a9,8945b800,...) at > free+0xb6/frame 0xbf751678 > zfs_kmem_free(8945b800,124,892f71e0,0,0,...) at > zfs_kmem_free+0x19/frame 0xbf751688 > zap_lockdir(88452400,115,0,0,1,...) at zap_lockdir+0x359/frame 0xbf7516e8 > zap_lookup_norm(88452400,115,0,bf7518b8,8,...) at > zap_lookup_norm+0x49/frame 0xbf75172c > zap_lookup(88452400,115,0,bf7518b8,8,...) at zap_lookup+0x69/frame 0xbf75= 1768 > zfs_dirent_lock(bf751830,88a5b0fc,bf7518b8,bf75182c,6,...) at > zfs_dirent_lock+0x4e9/frame 0xbf7517e8 > zfs_dirlook(88a5b0fc,bf7518b8,bf751bbc,0,0,...) at > zfs_dirlook+0x1bb/frame 0xbf751848 > zfs_lookup(bf751bbc,bf751bd0,0,89eee080,0,...) at > zfs_lookup+0x288/frame 0xbf75189c > zfs_freebsd_lookup(bf751a18,8117a717,bf7519e4,80b3ead8,88d92328,...) > at zfs_freebsd_lookup+0x6c/frame 0xbf7519c8 > VOP_CACHEDLOOKUP_APV(818a3e1c,bf751a18,bf751bd0,0,0,...) at > VOP_CACHEDLOOKUP_APV+0xca/frame 0xbf7519f8 > vfs_cache_lookup(bf751ab8,8117a697,8114f476,8112db4f,812746d4,...) at > vfs_cache_lookup+0xec/frame 0xbf751a40 > VOP_LOOKUP_APV(818a3e1c,bf751ab8,bf751bd0,33a,885d28d0,...) at > VOP_LOOKUP_APV+0xca/frame 0xbf751a70 > lookup(bf751b90,8112cf73,106,cc,8,...) at lookup+0x646/frame 0xbf751ae0 > namei(bf751b90,89ed5904,885d28d0,bf751c18,0,...) at namei+0x591/frame 0xb= f751b68 > kern_accessat(885d28d0,ffffff9c,280ff800,0,0,0) at > kern_accessat+0xdd/frame 0xbf751c20 > sys_access(885d28d0,bf751cc8,81174954,8112194d,8111ddc0,...) at > sys_access+0x39/frame 0xbf751c40 > syscall(bf751d08) at syscall+0x2da/frame 0xbf751cfc > Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xbf751cfc > --- syscall (33, FreeBSD ELF32, sys_access), eip =3D 0x280f0ec7, esp =3D > 0x7fbfc4ec, ebp =3D 0x7fbfc510 --- As I said you before, obtain the core and get backtrace from kgdb. I want to see both ddb and kgdb backtraces, the trap from the guts of witness means that somebody passed incorrect pointer in the kernel. --azskX66S5GHWoEK7 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJSX409AAoJEJDCuSvBvK1Bp38QAKaWosbc0l+crS198eXQiXjk /M4lzq94xTs8BSi0Oc8iKsDp9mvjUR33BStH9JUICT9EMagrIXc2FTv2y5ZrhaWK 1lBJNPc88pm4+htBEdVdy4S1Q3+t6iXjzQwH2TpO+iIXFxU0B9IuRVIn9xfkejoN C9sFIPW+TfULJZB4AL6x+VI+7C/Vd3YwOSxKV8iOUHA78kJ4ITTB2jwfLfLtdRsU awU3sBhOE0j+WlmLnbShV5/Mw/mlpzYvgV/qD3MomXedh2c+/2GNli6re+j1VG2k Zu1yRxLkHr8yPtsLtF/zMwPdwwi8eN2YWm0L3uItMA0AYGuXnw+1z+NNWOHz6ZdU PvQcj1u2YOGOR0U4j9wxxb8Gr0p8cnrqYM19c6gTBZ0E2IOqnShQK55tIYmxC0+T DYp59uT6QpaJymLJN6zw5ecxtpQWufFbumEeiuzVUwhOCVLwCR1MtFmTIY3rRkTt kxYKirThwEXE3G3KuKeIUQW+kDCCHr97BpeugZLa0GAemhFwkzpYZA7Z/spYE0Hy rY9G8E1TOtCpZ+MUygS/4j/5N8m2rkM0Ip2OYn0UWqWKV2pH9Z8LUpiBUWYUHycY aj6L2BDpRdH3mzvDQRpGaJm/8YsL7pGMvroA9nJHq+pQNEzsbK1Mr72Cgefy4Hvf ipPZXom2ZfqOMt9r9cF4 =+5/P -----END PGP SIGNATURE----- --azskX66S5GHWoEK7-- From owner-freebsd-fs@FreeBSD.ORG Fri Oct 18 13:44:13 2013 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 8DD47EA1 for ; Fri, 18 Oct 2013 13:44:13 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from land.berklix.org (land.berklix.org [144.76.10.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1C2E7220D for ; Fri, 18 Oct 2013 13:44:12 +0000 (UTC) Received: from mart.js.berklix.net (p5DCBCA9A.dip0.t-ipconnect.de [93.203.202.154]) (authenticated bits=128) by land.berklix.org (8.14.5/8.14.5) with ESMTP id r9IDi9qQ046322; Fri, 18 Oct 2013 13:44:09 GMT (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by mart.js.berklix.net (8.14.3/8.14.3) with ESMTP id r9IDi5u3056458; Fri, 18 Oct 2013 15:44:05 +0200 (CEST) (envelope-from jhs@berklix.com) Received: from fire.js.berklix.net (localhost.js.berklix.net [127.0.0.1]) by fire.js.berklix.net (8.14.4/8.14.4) with ESMTP id r9IDhrmu068968; Fri, 18 Oct 2013 15:43:59 +0200 (CEST) (envelope-from jhs@fire.js.berklix.net) Message-Id: <201310181343.r9IDhrmu068968@fire.js.berklix.net> To: Shawn Wallbridge Subject: Re: Urgent help needed, fsck segfault's on a 9.2-RELEASE machine From: "Julian H. Stacey" Organization: http://berklix.com BSD Unix Linux Consultancy, Munich Germany User-agent: EXMH on FreeBSD http://berklix.com/free/ X-URL: http://www.berklix.com In-reply-to: Your message "Thu, 17 Oct 2013 03:23:02 -0000." Date: Fri, 18 Oct 2013 15:43:53 +0200 Sender: jhs@berklix.com Cc: "fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Oct 2013 13:44:13 -0000 > root@mercury:/var/log # fsck -t ufs /dev/da0p1 > ** /dev/da0p1 > fsck: /dev/da0p1: Segmentation fault: 11 > > It's running 9.2-RELEASE Well, taking your word Urgent, & as /dev/da0p1 seems a low disk number, maybe the partition housing fsck_ufs is also corrupt ? Have you tried running fsck from removable media ie cd/dvd/usb stick ? Before you do that you might or not, want to copy fsck binary & corrupt partitions somewheer to do an autopsy later when you'r back OK. > (gdb) bt Hex Ugh, only as last resort, better rebuild fsck_ufs with eg CFLAGS += -g in /etc/make.conf or on command line. Good luck Cheers, Julian -- Julian Stacey, BSD Unix Linux C Sys Eng Consultant, Munich http://berklix.com Interleave replies below like a play script. Indent old text with "> ". Send plain text, not quoted-printable, HTML, base64, or multipart/alternative. From owner-freebsd-fs@FreeBSD.ORG Fri Oct 18 15:25:51 2013 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0106FDF for ; Fri, 18 Oct 2013 15:25:50 +0000 (UTC) (envelope-from shawn.wallbridge@imaginaryforces.com) Received: from barracuda.imaginaryforces.com (199.193.209.132.static.oneiricsys.com [199.193.209.132]) by mx1.freebsd.org (Postfix) with ESMTP id CCA732AF4 for ; Fri, 18 Oct 2013 15:25:50 +0000 (UTC) X-ASG-Debug-ID: 1382109943-0413ad4d7f2427e0001-ETJDRd Received: from newman.IMAGINARYFORCES.LOCAL (newman.imaginaryforces.local [192.168.23.34]) by barracuda.imaginaryforces.com with ESMTP id Fu8oTiv92oPCsEtD; Fri, 18 Oct 2013 08:25:43 -0700 (PDT) X-Barracuda-Envelope-From: shawn.wallbridge@imaginaryforces.com Received: from NEWMAN.IMAGINARYFORCES.LOCAL ([192.168.23.34]) by newman.IMAGINARYFORCES.LOCAL ([192.168.23.34]) with mapi id 14.01.0438.000; Fri, 18 Oct 2013 08:30:39 -0700 From: Shawn Wallbridge To: "Julian H. Stacey" Subject: Re: Urgent help needed, fsck segfault's on a 9.2-RELEASE machine Thread-Topic: Urgent help needed, fsck segfault's on a 9.2-RELEASE machine X-ASG-Orig-Subj: Re: Urgent help needed, fsck segfault's on a 9.2-RELEASE machine Thread-Index: AQHOzAjU4uc0cbAc5UycvhhiLLIp0Jn6leuA Date: Fri, 18 Oct 2013 15:30:39 +0000 Message-ID: In-Reply-To: <201310181343.r9IDhrmu068968@fire.js.berklix.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.23.68] Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Barracuda-Connect: newman.imaginaryforces.local[192.168.23.34] X-Barracuda-Start-Time: 1382109943 X-Barracuda-URL: http://barracuda.imaginaryforces.com:8000/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at imaginaryforces.com X-Barracuda-Bayes: INNOCENT GLOBAL 0.0000 1.0000 -2.0210 X-Barracuda-Spam-Score: -2.02 X-Barracuda-Spam-Status: No, SCORE=-2.02 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=8.0 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.2.141570 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Cc: "fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Oct 2013 15:25:51 -0000 On 10/18/13 6:43 AM, "Julian H. Stacey" wrote: >> root@mercury:/var/log # fsck -t ufs /dev/da0p1 >> ** /dev/da0p1 >> fsck: /dev/da0p1: Segmentation fault: 11 >> >> It's running 9.2-RELEASE > >Well, taking your word Urgent, & as /dev/da0p1 seems a low disk number, >maybe the partition housing fsck_ufs is also corrupt ? >Have you tried running fsck from removable media ie cd/dvd/usb stick ? > >Before you do that you might or not, want to copy fsck binary & >corrupt partitions somewheer to do an autopsy later when you'r back OK. > > >> (gdb) bt > >Hex Ugh, only as last resort, better >rebuild fsck_ufs with eg CFLAGS +=3D -g in /etc/make.conf or on command >line. > >Good luck > >Cheers, >Julian >-- >Julian Stacey, BSD Unix Linux C Sys Eng Consultant, Munich >http://berklix.com > Interleave replies below like a play script. Indent old text with "> ". > Send plain text, not quoted-printable, HTML, base64, or >multipart/alternative. > Thank you for responding Julian. I ended up creating a PR and a thread on forums.freebsd.org.. http://forums.freebsd.org/showthread.php?p=3D236842 http://www.freebsd.org/cgi/query-pr.cgi?pr=3D183042 I booted off of the 9.2 CD into LiveCD and had the same issue, fsck_ufs segfaulting. After many many hours of troubleshooting (documented in the forum thread), I ended up using the binary from a 9.0-RELEASE box, which worked fine. Thank you shawn ________________________________ This e-mail is intended only for the named person or entity to which it is = addressed and contains valuable business information that is proprietary, p= rivileged, confidential and/or otherwise protected from disclosure. If you = received this e-mail in error, any review, use, dissemination, distribution= or copying of this e-mail is strictly prohibited. Please notify us immedia= tely of the error via e-mail to postmaster@imaginaryforces.c= om and please delete the e-mail from your system, retaining no copies in an= y media. We appreciate your cooperation. ...imaginaryforces.com...=0D