From owner-freebsd-fs@FreeBSD.ORG Sun Mar 25 02:32:16 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 00E6A106566C for ; Sun, 25 Mar 2012 02:32:16 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from nm27-vm0.bullet.mail.bf1.yahoo.com (nm27-vm0.bullet.mail.bf1.yahoo.com [98.139.213.139]) by mx1.freebsd.org (Postfix) with SMTP id A248D8FC19 for ; Sun, 25 Mar 2012 02:32:15 +0000 (UTC) Received: from [98.139.212.147] by nm27.bullet.mail.bf1.yahoo.com with NNFMP; 25 Mar 2012 02:32:15 -0000 Received: from [98.139.211.201] by tm4.bullet.mail.bf1.yahoo.com with NNFMP; 25 Mar 2012 02:32:15 -0000 Received: from [127.0.0.1] by smtp210.mail.bf1.yahoo.com with NNFMP; 25 Mar 2012 02:32:15 -0000 X-Yahoo-Newman-Id: 135946.76426.bm@smtp210.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: kA.NdO0VM1nrhe4a9i6le_W83YtQHNS05l5cTasGzjB_9Wj SJgLdUr1aYq91CZFaywLyJ6kudcml6B9FKrXFB9UMId1S2L.xQ4Td.7Axnjh sX4R2t_QIDCyLMPKvgk5W0RHLan_QtPBm8Jv2h59mSx3JpxCA9emkGseVPGs Ccr5lZZAMlAGB4GIDxPXTEjozOz1h_92_oM92_e_.WfKz9v0gQyznMbRL6zU mWzjHekcYPGpdG4P5Q5AADA.CVJDswpEZCqI1zKSir7T7iYQ_9MypID.oU9w L2FfLY6j5UsR1Va5Z4u6Og9UXzfcug0ADXEEozoCO4L_l2oSs0v0nJuSKAm8 URRXpUbwc4aXMdxrVSp2jLVrhJBQCfeasMpuooKH3L4P7GlpVrszUK3p12nO _LVvjEdvXqtIdYhU0JfdbddrDJVeQRxZrpVFV0amv2M6l119aM7xonu0W0IL ZCsa0cCY6y2H81l7VdA9EVcLXkczwhVUULTgTeq4GtwZU8uexUEPIFsx1Aqg 9mWX8UomlRHyK3lS1wZFUdIAjYsDh4r1gqpqktXUfUQ-- X-Yahoo-SMTP: xcjD0guswBAZaPPIbxpWwLcp9Unf Received: from [192.168.10.102] (pfg@200.118.157.7 with plain) by smtp210.mail.bf1.yahoo.com with SMTP; 24 Mar 2012 19:32:14 -0700 PDT Message-ID: <4F6E83AD.5040403@FreeBSD.org> Date: Sat, 24 Mar 2012 21:32:13 -0500 From: Pedro Giffuni User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0.2) Gecko/20120226 Thunderbird/10.0.2 MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Re: [patch] ext4fs read only mode X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Mar 2012 02:32:16 -0000 Hello; On Aug. 22 2010 Zheng Liu posted a patch to add read-only support for ext4 extents in our ext2fs driver that was part of his GSoC. The ext2fs module has received many enhancements, mainly to make it look and behave more like the base UFS1 and the old patch doesn't apply anymore. I updated the patch just cleaning up some things that are not strictly necessary and adapting it to the underlying changes. http://people.freebsd.org/~pfg/patches/patch-ext2-extents The patch should apply cleanly in both 10-current and 9-stable. No review of this patch has been done, beyond compiling on amd64, and I have no idea if it works at all. AFAICT, there was no feedback on the original patch so I am actually wondering if this feature is desired at all. I will be glad to keep it around for a while and eventually update it but committing it is not yet in the horizon until we see more interest and at least some testing. best regards, Pedro. From owner-freebsd-fs@FreeBSD.ORG Mon Mar 26 10:09:40 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A0A35106566C for ; Mon, 26 Mar 2012 10:09:40 +0000 (UTC) (envelope-from sven@crashme.org) Received: from celaeno.tauri.mw.lg.virgo.supercluster.net (celaeno.tauri.mw.lg.virgo.supercluster.net [213.252.140.33]) by mx1.freebsd.org (Postfix) with ESMTP id 5A4348FC1E for ; Mon, 26 Mar 2012 10:09:39 +0000 (UTC) Received: from miram.persei.mw.lg.virgo.supercluster.net ([213.252.140.37] helo=[192.168.20.6]) by celaeno.tauri.mw.lg.virgo.supercluster.net with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.76) (envelope-from ) id 1SC6JV-0004OP-VK for freebsd-fs@freebsd.org; Mon, 26 Mar 2012 09:34:11 +0000 Message-ID: <4F703815.8070809@crashme.org> Date: Mon, 26 Mar 2012 11:34:13 +0200 From: Sven Brandenburg User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:11.0) Gecko/20120310 Thunderbird/11.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 2.5 (++) Subject: NFSv3, ZFS, 10GE performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2012 10:09:40 -0000 Hi, I'm toying around with a potential ZFS based Fileserver and I'm curious as to what level of performance can be expected from FreeBSD 9 and ZFS exports via NFS(v3) over 10GE. My current testing shows that while I can easily saturate the 10GE links with ttcp, I 'only' manage about 300MB/s through NFS. In my tests, the exported files should fit into L1ARC entirely (and can be copied to /dev/null on the local machine with several GB/s.. which seems about right reading from L1). I would expect something in the region of ~900MB/s via NFS in my test case. What can be expected for 10GE in terms of NFS throughput for a 9-STABLE client+server? Are there sysctl's or similar knobs to fiddle with? If helpful, I can provide more details. Thanks :) Sven From owner-freebsd-fs@FreeBSD.ORG Mon Mar 26 10:37:42 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6112B1065674 for ; Mon, 26 Mar 2012 10:37:42 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) by mx1.freebsd.org (Postfix) with ESMTP id 176D48FC15 for ; Mon, 26 Mar 2012 10:37:41 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1SC7Is-0002gU-4v for freebsd-fs@freebsd.org; Mon, 26 Mar 2012 12:37:34 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 26 Mar 2012 12:37:34 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 26 Mar 2012 12:37:34 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Mon, 26 Mar 2012 12:37:24 +0200 Lines: 34 Message-ID: References: <4F703815.8070809@crashme.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigD38BDCC770CEA48ACCD613FD" X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0) Gecko/20120213 Thunderbird/10.0 In-Reply-To: <4F703815.8070809@crashme.org> X-Enigmail-Version: 1.3.5 Subject: Re: NFSv3, ZFS, 10GE performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2012 10:37:42 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigD38BDCC770CEA48ACCD613FD Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 26/03/2012 11:34, Sven Brandenburg wrote: > I would expect something in the region of ~900MB/s via NFS in my test c= ase. >=20 > Are there sysctl's or similar knobs to fiddle with? You could try modifying the rsize and wsize NFS options (read mount_nfs(8)), they help with UFS. If you get some good results with it, post them back here! --------------enigD38BDCC770CEA48ACCD613FD Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9wRuQACgkQldnAQVacBchtmQCg5A1Ie4XnN/ZOeyO9bX6Su+ge exsAniUMVwihUfrtgVGMtBDcml1652SF =g8J7 -----END PGP SIGNATURE----- --------------enigD38BDCC770CEA48ACCD613FD-- From owner-freebsd-fs@FreeBSD.ORG Mon Mar 26 11:07:03 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9D6AB1065676 for ; Mon, 26 Mar 2012 11:07:03 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 874568FC1A for ; Mon, 26 Mar 2012 11:07:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q2QB73xJ018257 for ; Mon, 26 Mar 2012 11:07:03 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q2QB723l018255 for freebsd-fs@FreeBSD.org; Mon, 26 Mar 2012 11:07:02 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 26 Mar 2012 11:07:02 GMT Message-Id: <201203261107.q2QB723l018255@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2012 11:07:03 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/165950 fs [ffs] SU+J and fsck problem o kern/165923 fs [nfs] Writing to NFS-backed mmapped files fails if flu o kern/165392 fs Multiple mkdir/rmdir fails with errno 31 o kern/165087 fs [unionfs] lock violation in unionfs o kern/164472 fs [ufs] fsck -B panics on particular data inconsistency o kern/164370 fs [zfs] zfs destroy for snapshot fails on i386 and sparc o kern/164261 fs [nullfs] [patch] fix panic with NFS served from NULLFS o kern/164256 fs [zfs] device entry for volume is not created after zfs o kern/164184 fs [ufs] [panic] Kernel panic with ufs_makeinode o kern/163801 fs [md] [request] allow mfsBSD legacy installed in 'swap' o kern/163770 fs [zfs] [hang] LOR between zfs&syncer + vnlru leading to o kern/163501 fs [nfs] NFS exporting a dir and a subdir in that dir to o kern/162944 fs [coda] Coda file system module looks broken in 9.0 o kern/162860 fs [zfs] Cannot share ZFS filesystem to hosts with a hyph o kern/162751 fs [zfs] [panic] kernel panics during file operations o kern/162591 fs [nullfs] cross-filesystem nullfs does not work as expe o kern/162519 fs [zfs] "zpool import" relies on buggy realpath() behavi o kern/162362 fs [snapshots] [panic] ufs with snapshot(s) panics when g o kern/162083 fs [zfs] [panic] zfs unmount -f pool o kern/161968 fs [zfs] [hang] renaming snapshot with -r including a zvo o kern/161897 fs [zfs] [patch] zfs partition probing causing long delay o kern/161864 fs [ufs] removing journaling from UFS partition fails on o bin/161807 fs [patch] add option for explicitly specifying metadata o kern/161579 fs [smbfs] FreeBSD sometimes panics when an smb share is o kern/161533 fs [zfs] [panic] zfs receive panic: system ioctl returnin o kern/161511 fs [unionfs] Filesystem deadlocks when using multiple uni o kern/161438 fs [zfs] [panic] recursed on non-recursive spa_namespace_ o kern/161424 fs [nullfs] __getcwd() calls fail when used on nullfs mou o kern/161280 fs [zfs] Stack overflow in gptzfsboot o kern/161205 fs [nfs] [pfsync] [regression] [build] Bug report freebsd o kern/161169 fs [zfs] [panic] ZFS causes kernel panic in dbuf_dirty o kern/161112 fs [ufs] [lor] filesystem LOR in FreeBSD 9.0-BETA3 o kern/160893 fs [zfs] [panic] 9.0-BETA2 kernel panic o kern/160860 fs [ufs] Random UFS root filesystem corruption with SU+J o kern/160801 fs [zfs] zfsboot on 8.2-RELEASE fails to boot from root-o o kern/160790 fs [fusefs] [panic] VPUTX: negative ref count with FUSE o kern/160777 fs [zfs] [hang] RAID-Z3 causes fatal hang upon scrub/impo o kern/160706 fs [zfs] zfs bootloader fails when a non-root vdev exists o kern/160591 fs [zfs] Fail to boot on zfs root with degraded raidz2 [r o kern/160410 fs [smbfs] [hang] smbfs hangs when transferring large fil o kern/160283 fs [zfs] [patch] 'zfs list' does abort in make_dataset_ha o kern/159930 fs [ufs] [panic] kernel core o kern/159663 fs [socket] [nullfs] sockets don't work though nullfs mou o kern/159402 fs [zfs][loader] symlinks cause I/O errors o kern/159357 fs [zfs] ZFS MAXNAMELEN macro has confusing name (off-by- o kern/159356 fs [zfs] [patch] ZFS NAME_ERR_DISKLIKE check is Solaris-s o kern/159351 fs [nfs] [patch] - divide by zero in mountnfs() o kern/159251 fs [zfs] [request]: add FLETCHER4 as DEDUP hash option o kern/159077 fs [zfs] Can't cd .. with latest zfs version o kern/159048 fs [smbfs] smb mount corrupts large files o kern/159045 fs [zfs] [hang] ZFS scrub freezes system o kern/158839 fs [zfs] ZFS Bootloader Fails if there is a Dead Disk o kern/158802 fs amd(8) ICMP storm and unkillable process. o kern/158231 fs [nullfs] panic on unmounting nullfs mounted over ufs o f kern/157929 fs [nfs] NFS slow read o kern/157399 fs [zfs] trouble with: mdconfig force delete && zfs strip o kern/157179 fs [zfs] zfs/dbuf.c: panic: solaris assert: arc_buf_remov o kern/156797 fs [zfs] [panic] Double panic with FreeBSD 9-CURRENT and o kern/156781 fs [zfs] zfs is losing the snapshot directory, p kern/156545 fs [ufs] mv could break UFS on SMP systems o kern/156193 fs [ufs] [hang] UFS snapshot hangs && deadlocks processes o kern/156039 fs [nullfs] [unionfs] nullfs + unionfs do not compose, re o kern/155615 fs [zfs] zfs v28 broken on sparc64 -current o kern/155587 fs [zfs] [panic] kernel panic with zfs f kern/155411 fs [regression] [8.2-release] [tmpfs]: mount: tmpfs : No o kern/155199 fs [ext2fs] ext3fs mounted as ext2fs gives I/O errors o bin/155104 fs [zfs][patch] use /dev prefix by default when importing o kern/154930 fs [zfs] cannot delete/unlink file from full volume -> EN o kern/154828 fs [msdosfs] Unable to create directories on external USB o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 p kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153520 fs [zfs] Boot from GPT ZFS root on HP BL460c G1 unstable o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/151111 fs [zfs] vnodes leakage during zfs unmount o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147560 fs [zfs] [boot] Booting 8.1-PRERELEASE raidz system take o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an f bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139597 fs [patch] [tmpfs] tmpfs initializes va_gen but doesn't u o kern/139564 fs [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files f sparc/123566 fs [zfs] zpool import issue: EOVERFLOW o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118126 fs [nfs] [patch] Poor NFS server write performance o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116583 fs [ffs] [hang] System freezes for short time when using o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] [iconv] mount_msdosfs: msdosfs_iconv: Operat o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 263 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Mar 26 14:43:05 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 18040106564A for ; Mon, 26 Mar 2012 14:43:05 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id CEC578FC15 for ; Mon, 26 Mar 2012 14:43:04 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id q2QEUXkS027498; Mon, 26 Mar 2012 09:30:35 -0500 (CDT) Date: Mon, 26 Mar 2012 09:30:33 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Sven Brandenburg In-Reply-To: <4F703815.8070809@crashme.org> Message-ID: References: <4F703815.8070809@crashme.org> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Mon, 26 Mar 2012 09:30:36 -0500 (CDT) Cc: freebsd-fs@freebsd.org Subject: Re: NFSv3, ZFS, 10GE performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2012 14:43:05 -0000 On Mon, 26 Mar 2012, Sven Brandenburg wrote: > Hi, > > I'm toying around with a potential ZFS based Fileserver and I'm curious as to > what level of performance can be expected from FreeBSD 9 and ZFS exports via > NFS(v3) over 10GE. > My current testing shows that while I can easily saturate the 10GE links with > ttcp, I 'only' manage about 300MB/s through NFS. In my tests, the exported How are you performing your testing? Do you have a zfs server directory and are copying files from the server to a client? What size are the test files on your server? How much RAM does your server have and how much is zfs allowed to use? What program are you using to perform the copy? Is the copy sequential or can it be done in parallel via multiple processes/threads on the client? Are you only interested in single threaded read performance to a single client? Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@FreeBSD.ORG Mon Mar 26 16:32:24 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0CDA81065672 for ; Mon, 26 Mar 2012 16:32:24 +0000 (UTC) (envelope-from sven@crashme.org) Received: from celaeno.tauri.mw.lg.virgo.supercluster.net (celaeno.tauri.mw.lg.virgo.supercluster.net [213.252.140.33]) by mx1.freebsd.org (Postfix) with ESMTP id B99E68FC1B for ; Mon, 26 Mar 2012 16:32:23 +0000 (UTC) Received: from miram.persei.mw.lg.virgo.supercluster.net ([213.252.140.37] helo=[192.168.20.6]) by celaeno.tauri.mw.lg.virgo.supercluster.net with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.76) (envelope-from ) id 1SCCqC-0004TL-TH for freebsd-fs@freebsd.org; Mon, 26 Mar 2012 16:32:22 +0000 Message-ID: <4F709A18.50907@crashme.org> Date: Mon, 26 Mar 2012 18:32:24 +0200 From: Sven Brandenburg User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:11.0) Gecko/20120310 Thunderbird/11.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4F703815.8070809@crashme.org> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) Subject: Re: NFSv3, ZFS, 10GE performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2012 16:32:24 -0000 On 03/26/2012 12:37 PM, Ivan Voras wrote: > You could try modifying the rsize and wsize NFS options (read > mount_nfs(8)), they help with UFS. I tried this a few days ago and fiddling rsize alters performance from "ok" to "terrible". However, you made me revisit this and mount_nfs(8) seems to have a gem in its options: readahead. This did the trick for me and my (long and sequential) reads. While the manpage says its limited to 0-4, the best results were achieved with readahead=8 : 1.1GB/s - which is what I had hoped for. On a tangent: gnu-dd 1GB/s is 10^9 Bytes/s, not 2^30. Yes, I fell for it at first :) The good news is that there was no fiddling on the NFS server side. (Apart from MTU increases, PCI settings and more buffers to get TCP performance to full tilt in the first place) Hopefully, readahead doesn't kill performance for smaller files.. :-) regards, Sven From owner-freebsd-fs@FreeBSD.ORG Mon Mar 26 16:32:43 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ACAA01065674 for ; Mon, 26 Mar 2012 16:32:43 +0000 (UTC) (envelope-from sven@crashme.org) Received: from celaeno.tauri.mw.lg.virgo.supercluster.net (celaeno.tauri.mw.lg.virgo.supercluster.net [213.252.140.33]) by mx1.freebsd.org (Postfix) with ESMTP id 6545C8FC1F for ; Mon, 26 Mar 2012 16:32:43 +0000 (UTC) Received: from miram.persei.mw.lg.virgo.supercluster.net ([213.252.140.37] helo=[192.168.20.6]) by celaeno.tauri.mw.lg.virgo.supercluster.net with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.76) (envelope-from ) id 1SCCqW-0004TO-Am; Mon, 26 Mar 2012 16:32:42 +0000 Message-ID: <4F709A2C.2080806@crashme.org> Date: Mon, 26 Mar 2012 18:32:44 +0200 From: Sven Brandenburg User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:11.0) Gecko/20120310 Thunderbird/11.0 MIME-Version: 1.0 To: Bob Friesenhahn References: <4F703815.8070809@crashme.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) Cc: freebsd-fs@freebsd.org Subject: Re: NFSv3, ZFS, 10GE performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2012 16:32:43 -0000 On 03/26/2012 04:30 PM, Bob Friesenhahn wrote: > How are you performing your testing? ZFS directoy is mounted via nfs on client. I created several files (4,8,16GB sizes) with random data in the exported fs. Since I'm currently only interested in network saturation/nfs optimization part of the equation, I only dd'ed those files to /dev/null (testing several block sizes) on the client in one go, no random reads. On a tangent: files made entirely out of zeroes are not a good benchmark for measuring nfs performance. It knows(TM) :) The zpool consists of only one slow disk to work out if data comes out of L1ARC or from disk. After the first dd run all subsequent runs are served from ARC, just as expected (the server has 96GB of RAM, so the files should fit). On another tangent: at first I tried to use md(4) as source for the nfs exports before complicating things with zfs - as it turns out md is rather slow, only about 0.7-1.0GB/s with ufs on it. Local reads of my files once the ARC is 'seeded' are several times faster. > Are you only interested in single threaded read performance to a single > client? My first item on the list is serving one client as fast as possible, next step is multiple client machines (maybe these are conflicting goals?). However, I figure that I should at least be able to press the right buttons to dial in "fast" for one client :-) regards, Sven From owner-freebsd-fs@FreeBSD.ORG Mon Mar 26 16:57:07 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A34CE106568A for ; Mon, 26 Mar 2012 16:57:07 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id 63D488FC1F for ; Mon, 26 Mar 2012 16:57:07 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id q2QGv1Q7028082; Mon, 26 Mar 2012 11:57:01 -0500 (CDT) Date: Mon, 26 Mar 2012 11:57:01 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Sven Brandenburg In-Reply-To: <4F709A18.50907@crashme.org> Message-ID: References: <4F703815.8070809@crashme.org> <4F709A18.50907@crashme.org> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Mon, 26 Mar 2012 11:57:01 -0500 (CDT) Cc: freebsd-fs@freebsd.org Subject: Re: NFSv3, ZFS, 10GE performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2012 16:57:07 -0000 On Mon, 26 Mar 2012, Sven Brandenburg wrote: > > Hopefully, readahead doesn't kill performance for smaller files.. :-) You are right to be concerned. There are plenty of cases where read-ahead damages application performance. Reading data which is never actually used is expensive. It would be useful if FreeBSD would support posix_fadvise() so that applications can specify the type of access they will use, and if this advice can be used by NFS and the filesystem layer to decide if read-ahead should be used, and how much. Here are some notes as to how Linux treats the avise options: "Under Linux, POSIX_FADV_NORMAL sets the readahead window to the default size for the backing device; POSIX_FADV_SEQUENTIAL doubles this size, and POSIX_FADV_RANDOM disables file readahead entirely. These changes affect the entire file, not just the specified region (but other open file handles to the same file are unaffected). POSIX_FADV_WILLNEED initiates a non-blocking read of the specified region into the page cache. The amount of data read may be decreased by the kernel depending on virtual memory load. (A few megabytes will usually be fully satisfied, and more is rarely useful.)" The system can learn how an application performs its accesses, but it takes time and many accesses before correct tunings can be learned, and by then it may be too late to be of benefit since the I/Os have already completed. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@FreeBSD.ORG Mon Mar 26 17:09:54 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26CBC106564A for ; Mon, 26 Mar 2012 17:09:54 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay06.ispgateway.de (smtprelay06.ispgateway.de [80.67.31.103]) by mx1.freebsd.org (Postfix) with ESMTP id D48A28FC15 for ; Mon, 26 Mar 2012 17:09:53 +0000 (UTC) Received: from [109.46.100.95] (helo=fabiankeil.de) by smtprelay06.ispgateway.de with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.68) (envelope-from ) id 1SCDKb-0005Hh-Ir for freebsd-fs@freebsd.org; Mon, 26 Mar 2012 19:03:45 +0200 Date: Mon, 26 Mar 2012 19:03:42 +0200 From: Fabian Keil To: freebsd-fs@freebsd.org Message-ID: <20120326190342.0b78cbc8@fabiankeil.de> In-Reply-To: References: <4F703815.8070809@crashme.org> <4F709A18.50907@crashme.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=PGP-SHA1; boundary="Sig_/GAqmTSOICOxqnc/ZiWsvYw."; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 Subject: Re: NFSv3, ZFS, 10GE performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2012 17:09:54 -0000 --Sig_/GAqmTSOICOxqnc/ZiWsvYw. Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Bob Friesenhahn wrote: > On Mon, 26 Mar 2012, Sven Brandenburg wrote: > > > > Hopefully, readahead doesn't kill performance for smaller files.. :-) >=20 > You are right to be concerned. There are plenty of cases where=20 > read-ahead damages application performance. Reading data which is=20 > never actually used is expensive. >=20 > It would be useful if FreeBSD would support posix_fadvise() so that=20 > applications can specify the type of access they will use, and if this=20 > advice can be used by NFS and the filesystem layer to decide if=20 > read-ahead should be used, and how much. posix_fadvise() is already available in FreeBSD 10.0-CURRENT. Fabian --Sig_/GAqmTSOICOxqnc/ZiWsvYw. Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk9woXEACgkQBYqIVf93VJ21HQCfbY2lvS2Nk31YdWC2S00WZH9x 9kwAoLd0l6a70JXXD774XJqnXSha++1o =dtoo -----END PGP SIGNATURE----- --Sig_/GAqmTSOICOxqnc/ZiWsvYw.-- From owner-freebsd-fs@FreeBSD.ORG Mon Mar 26 17:33:22 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0A935106564A for ; Mon, 26 Mar 2012 17:33:22 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id AECA18FC08 for ; Mon, 26 Mar 2012 17:33:21 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id q2QHQotM028173; Mon, 26 Mar 2012 12:26:50 -0500 (CDT) Date: Mon, 26 Mar 2012 12:26:50 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Sven Brandenburg In-Reply-To: <4F709A2C.2080806@crashme.org> Message-ID: References: <4F703815.8070809@crashme.org> <4F709A2C.2080806@crashme.org> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Mon, 26 Mar 2012 12:26:50 -0500 (CDT) Cc: freebsd-fs@freebsd.org Subject: Re: NFSv3, ZFS, 10GE performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2012 17:33:22 -0000 On Mon, 26 Mar 2012, Sven Brandenburg wrote: > >> Are you only interested in single threaded read performance to a single >> client? > My first item on the list is serving one client as fast as possible, next > step is multiple client machines (maybe these are conflicting goals?). > However, I figure that I should at least be able to press the right buttons > to dial in "fast" for one client :-) It really depends on if the client will actually consume all of the data as requested or if the client application will be competing with NFS because it transfers data it will never need. An issue which is quite familiar to me is the TIFF image file format. This format stores the image data in sequential strips, yet it uses more complex data structures (which require many seek + small read) as well. In my testing, I found that excessive read-ahead hindered performance with this file format. More than 16k of read-ahead often resulted in a performance penalty over NFS. Memory mapped files can also cause a performance problem since the MMU page size is typically 4K and the program is blocked from running until the current page fault completes. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@FreeBSD.ORG Mon Mar 26 21:48:04 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DCF11106566C for ; Mon, 26 Mar 2012 21:48:04 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 94DCA8FC08 for ; Mon, 26 Mar 2012 21:48:04 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap8EAAvjcE+DaFvO/2dsb2JhbABEhT+0BoIJAQEBAwEBAQEgBCcgCwUWGAICDRkCKQEJJgYIBwQBGQMEh2QFC6c6kW2BL4kzBYUQgRgEkzWCK4ERjxyDA4E4CA X-IronPort-AV: E=Sophos;i="4.75,322,1330923600"; d="scan'208";a="165358377" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 26 Mar 2012 17:47:57 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 86B43B3FA0; Mon, 26 Mar 2012 17:47:57 -0400 (EDT) Date: Mon, 26 Mar 2012 17:47:57 -0400 (EDT) From: Rick Macklem To: Sven Brandenburg Message-ID: <40825357.1703430.1332798477468.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <4F709A18.50907@crashme.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE8 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org Subject: Re: NFSv3, ZFS, 10GE performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Mar 2012 21:48:04 -0000 Sven Brandenburg wrote: > On 03/26/2012 12:37 PM, Ivan Voras wrote: > > You could try modifying the rsize and wsize NFS options (read > > mount_nfs(8)), they help with UFS. > > I tried this a few days ago and fiddling rsize alters performance from > "ok" to "terrible". > However, you made me revisit this and mount_nfs(8) seems to have a gem > in its options: readahead. > This did the trick for me and my (long and sequential) reads. > While the manpage says its limited to 0-4, the best results were > achieved with readahead=8 : 1.1GB/s - which is what I had hoped for. > The new NFS client (which is the default in 9) will use the largest size supported by the server, limited to MAX_BSIZE as default rsize, wsize. (And the server will allow a rsize/wsize of MAX_BSIZE.) MAX_BSIZE is 64kb. I'd like to try making that bigger, but haven't gotten around to it yet. (If you wanted to try bumping MAX_BSIZE to 128Kb on both client and server and seeing what happens, that might be interesting, since my understanding is that ZFS uses a 128Kb block size.) I'd guess that you needed a readahead of 8 to fill the TCP pipe, but I have no idea what packet transit time you have between client<->server. (In other words 64Kb * 8 fills the data pipe. Anything less doesn't do so. My experience for LANs is that a larger block size with smaller readahead works about as well. For example 128Kb * 4 or 512Kb * 1, if MAX_BSIZE could be bumped to 512Kb. Solaris10 servers allow 1Mbyte rsize/wsize, if I recall correctly?) So, beyond what you've done, all I can suggest is trying bumping MAX_BSIZE up (but I have no idea if such a system even boots;-). Have fun with it, rick > On a tangent: gnu-dd 1GB/s is 10^9 Bytes/s, not 2^30. Yes, I fell for > it > at first :) > The good news is that there was no fiddling on the NFS server side. > (Apart from MTU increases, PCI settings and more buffers to get TCP > performance to full tilt in the first place) > > Hopefully, readahead doesn't kill performance for smaller files.. :-) > > regards, > Sven > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Tue Mar 27 14:30:14 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EB837106566C for ; Tue, 27 Mar 2012 14:30:14 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id C11448FC1B for ; Tue, 27 Mar 2012 14:30:14 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 440A3B966; Tue, 27 Mar 2012 10:30:13 -0400 (EDT) From: John Baldwin To: freebsd-fs@freebsd.org Date: Tue, 27 Mar 2012 07:48:55 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p10; KDE/4.5.5; amd64; ; ) References: <4F703815.8070809@crashme.org> <20120326190342.0b78cbc8@fabiankeil.de> In-Reply-To: <20120326190342.0b78cbc8@fabiankeil.de> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201203270748.55392.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 27 Mar 2012 10:30:14 -0400 (EDT) Cc: Subject: Re: NFSv3, ZFS, 10GE performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Mar 2012 14:30:15 -0000 On Monday, March 26, 2012 1:03:42 pm Fabian Keil wrote: > Bob Friesenhahn wrote: > > > On Mon, 26 Mar 2012, Sven Brandenburg wrote: > > > > > > Hopefully, readahead doesn't kill performance for smaller files.. :-) > > > > You are right to be concerned. There are plenty of cases where > > read-ahead damages application performance. Reading data which is > > never actually used is expensive. > > > > It would be useful if FreeBSD would support posix_fadvise() so that > > applications can specify the type of access they will use, and if this > > advice can be used by NFS and the filesystem layer to decide if > > read-ahead should be used, and how much. > > posix_fadvise() is already available in FreeBSD 10.0-CURRENT. It doesn't quite do the trick of bumping up the read ahead amount for sequential yet (though that is an easy change). Implementing WILLNEED is a bit trickier as it requires per-FS support. I have patches to implement it for UFS. -- John Baldwin From owner-freebsd-fs@FreeBSD.ORG Tue Mar 27 16:10:14 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 06FC41065673 for ; Tue, 27 Mar 2012 16:10:14 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id E30568FC1B for ; Tue, 27 Mar 2012 16:10:13 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q2RGADQA071095 for ; Tue, 27 Mar 2012 16:10:13 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q2RGADj3071094; Tue, 27 Mar 2012 16:10:13 GMT (envelope-from gnats) Date: Tue, 27 Mar 2012 16:10:13 GMT Message-Id: <201203271610.q2RGADj3071094@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Quentin Rameau Cc: Subject: Re: kern/147881: [zfs] [patch] ZFS " sharenfs" doesn' t allow different "exports" options for different hosts X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Quentin Rameau List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Mar 2012 16:10:14 -0000 The following reply was made to PR kern/147881; it has been noted by GNATS. From: Quentin Rameau To: bug-followup@freebsd.org, Richard.Conto@gmail.com Cc: Subject: Re: kern/147881: [zfs] [patch] ZFS "sharenfs" doesn't allow different "exports" options for different hosts Date: Tue, 27 Mar 2012 18:07:22 +0200 --f46d042f92881cc9da04bc3bb206 Content-Type: text/plain; charset=UTF-8 I made another similar patch which does not include commented lines. It's for RELENG_9_0 -- Quentin Rameau --f46d042f92881cc9da04bc3bb206 Content-Type: text/x-patch; charset=US-ASCII; name="fsshare.c-multipleexports.patch" Content-Disposition: attachment; filename="fsshare.c-multipleexports.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_h0b50y1k0 LS0tIHNyYy9jZGRsL2NvbXBhdC9vcGVuc29sYXJpcy9taXNjL2Zzc2hhcmUuYy5vcmlnCTIwMTIt MDMtMjcgMTQ6MjM6MDkuNTI4NTU5NDUzICswMjAwCisrKyBzcmMvY2RkbC9jb21wYXQvb3BlbnNv bGFyaXMvbWlzYy9mc3NoYXJlLmMJMjAxMi0wMy0yNyAxNzoyODoyNi42OTA1NjUyMzggKzAyMDAK QEAgLTE1MSw3ICsxNTEsOCBAQAogICAgIGludCBzaGFyZSkKIHsKIAljaGFyIHRtcGZpbGVbUEFU SF9NQVhdOwotCWNoYXIgKmxpbmU7CisJY2hhciBvbGRvcHRzW09QVFNTSVpFXTsKKwljaGFyICps aW5lLCAqcywgKmV4cG9ydDsKIAlGSUxFICpuZXdmZCwgKm9sZGZkOwogCWludCBmZCwgZXJyb3I7 CiAKQEAgLTIxMSw4ICsyMTIsMTQgQEAKIAkJZ290byBvdXQ7CiAJfQogCWlmIChzaGFyZSkgewot CQlmcHJpbnRmKG5ld2ZkLCAiJXNcdCVzXG4iLCBtb3VudHBvaW50LAotCQkgICAgdHJhbnNsYXRl X29wdHMoc2hhcmVvcHRzKSk7CisJCXN0cmxjcHkob2xkb3B0cywgc2hhcmVvcHRzLCBzaXplb2Yo b2xkb3B0cykpOworCQlzID0gb2xkb3B0czsKKwkJd2hpbGUgKChleHBvcnQgPSBzdHJzZXAoJnMs ICI7IikpICE9IE5VTEwpIHsKKwkJCWlmIChleHBvcnRbMF0gPT0gJ1wwJykKKwkJCQljb250aW51 ZTsKKwkJCWZwcmludGYobmV3ZmQsICIlc1x0JXNcbiIsIG1vdW50cG9pbnQsCisJCQkgICAgdHJh bnNsYXRlX29wdHMoZXhwb3J0KSk7CisJCX0KIAl9CiAKIG91dDoK --f46d042f92881cc9da04bc3bb206-- From owner-freebsd-fs@FreeBSD.ORG Tue Mar 27 16:36:26 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 02FC51065670; Tue, 27 Mar 2012 16:36:26 +0000 (UTC) (envelope-from db@bsdsystems.de) Received: from fop.bsdsystems.de (mx.bsdsystems.de [88.198.57.43]) by mx1.freebsd.org (Postfix) with ESMTP id 0BF1A8FC16; Tue, 27 Mar 2012 16:36:25 +0000 (UTC) Received: from [172.16.10.254] (e176108054.adsl.alicedsl.de [85.176.108.54]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by fop.bsdsystems.de (Postfix) with ESMTP id F04ED7BA51; Tue, 27 Mar 2012 18:18:13 +0200 (CEST) Mime-Version: 1.0 (Apple Message framework v1081) Content-Type: text/plain; charset=us-ascii From: Dennis Berger In-Reply-To: Date: Tue, 27 Mar 2012 18:18:09 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <9A143905-EAC1-4280-89F1-B7A742159610@bsdsystems.de> References: To: andy thomas X-Pgp-Agent: GPGMail 1.3.3 X-Mailer: Apple Mail (2.1081) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS read/write performance slows with time X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Mar 2012 16:36:26 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 We have seen the same issue on freenas 8.2 which is a patched freebsd = 8.2. Performance dropped from somewhat 700MByte/sec to 2MByte/sec on the main = NAS system. I couldn't find the problem in time so we rebooted. Uptime was around = 150 days. After reboot everything was back to normal. We will reinstall the system with a stock freebsd 9 as no GUI is needed = anymore. best, - -dennis Am 22.03.2012 um 01:18 schrieb andy thomas: > On Wed, 21 Mar 2012, Artem Belevich wrote: >=20 >> On Wed, Mar 21, 2012 at 4:56 PM, andy thomas = wrote: >>> A server running 64-bit FreeBSD 8.0 boots from a SATA disk and then = mounts a >> ... >>> But over a period of time, this performance begins to deteriorate = and after >>> 180 days of uptime >>=20 >> If it's indeed 8.0 that you are running, I would strongly recommend >> upgrading to 8-STABLE or 8.3 when it's released. >> A *lot* of things in ZFS got fixed/improved since ZFSv13 in 8.0. >=20 > It's running 8.0-RELEASE. I've not seen these gradual deterioration = problems with 8.2-RELEASE (or indeed on OpenSolaris SPARC and = OpenIndiana servers). >=20 > Thanks for the quick reply. >=20 > Andy > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJPcehFAAoJEBgcZwoN9xrD0qQIAKtxp8EOaQQS6vaJfqWYvg8Y XQCgRrcOElzCeBU3YiBpCVXzn8EwJu+ngKCm1e71Tfa0gUTEMPxp87C+9qV0GRbo fqZTk64kEozAYQxxZTUawpePhif/7aEw/48o3GbnUwyYMpzuoysp/XtumBbhTWQm pTp2OYGimbRvnYIyxxhjC3Krxgp+wh6WTcJKIqhXWvkbYuU+30noCSdJRS7IqlNX gziQ2CcZeAAx4Ebm+MMKaIBwUZ4cvosqU4d+xaMpprXKJNoOylsZWKWpnqU/A6Ah AqO/XgW8oYf22+7Dk9SNxBb003nYCvEuHGJttziTKjrQ1TfJjWqptclScphhhD4=3D =3DOYDT -----END PGP SIGNATURE----- From owner-freebsd-fs@FreeBSD.ORG Tue Mar 27 17:46:46 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CB041106567A for ; Tue, 27 Mar 2012 17:46:46 +0000 (UTC) (envelope-from prvs=1433e93ef2=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 5777C8FC15 for ; Tue, 27 Mar 2012 17:46:45 +0000 (UTC) X-Spam-Processed: mail1.multiplay.co.uk, Tue, 27 Mar 2012 18:45:56 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50019008620.msg for ; Tue, 27 Mar 2012 18:45:56 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1433e93ef2=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: From: "Steven Hartland" To: "Dennis Berger" , "andy thomas" References: <9A143905-EAC1-4280-89F1-B7A742159610@bsdsystems.de> Date: Tue, 27 Mar 2012 18:45:57 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS read/write performance slows with time X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Mar 2012 17:46:46 -0000 8.2 is effected by overflow bugs which can cause behaviour like this. Regards Steve ----- Original Message ----- From: "Dennis Berger" To: "andy thomas" Cc: Sent: Tuesday, March 27, 2012 5:18 PM Subject: Re: ZFS read/write performance slows with time -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 We have seen the same issue on freenas 8.2 which is a patched freebsd 8.2. Performance dropped from somewhat 700MByte/sec to 2MByte/sec on the main NAS system. I couldn't find the problem in time so we rebooted. Uptime was around 150 days. After reboot everything was back to normal. We will reinstall the system with a stock freebsd 9 as no GUI is needed anymore. ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Tue Mar 27 18:14:58 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 847E9106564A for ; Tue, 27 Mar 2012 18:14:58 +0000 (UTC) (envelope-from tdb@carrick.bishnet.net) Received: from carrick.bishnet.net (carrick.bishnet.net [IPv6:2a01:348:132:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 457148FC0A for ; Tue, 27 Mar 2012 18:14:58 +0000 (UTC) Received: from carrick-users.bishnet.net ([2a01:348:132:51::10]) by carrick.bishnet.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.76 (FreeBSD)) (envelope-from ) id 1SCav3-0000cp-RE for freebsd-fs@freebsd.org; Tue, 27 Mar 2012 19:14:57 +0100 Received: (from tdb@localhost) by carrick-users.bishnet.net (8.14.4/8.14.4/Submit) id q2RIEv8g002406 for freebsd-fs@freebsd.org; Tue, 27 Mar 2012 19:14:57 +0100 (BST) (envelope-from tdb) Date: Tue, 27 Mar 2012 19:14:57 +0100 From: Tim Bishop To: freebsd-fs@freebsd.org Message-ID: <20120327181457.GC24787@carrick-users.bishnet.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-PGP-Key: 0x5AE7D984, http://www.bishnet.net/tim/tim-bishnet-net.asc X-PGP-Fingerprint: 1453 086E 9376 1A50 ECF6 AE05 7DCE D659 5AE7 D984 User-Agent: Mutt/1.5.21 (2010-09-15) Subject: ZFS: processes hanging when trying to access filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Mar 2012 18:14:58 -0000 I have a machine running 8-STABLE amd64 from the end of last week. I have a problem where the machine starts to freeze up. Any process accessing the ZFS filesystems hangs, which eventually causes more and more processes to be spawned (cronjobs, etc, never complete). Although the root filesystem is on UFS (the machine hosts jails on ZFS), eventually I can't log in anymore. The problem occurs when the frequently used part of the ARC gets too large. See this graph: http://dl.dropbox.com/u/318044/zfs_arc_utilization-day.png At the right of the graph things started to hang. At the same time I see a high amount of context switching. I picked a hanging process and procstat showed the following: PID TID COMM TDNAME KSTACK 24787 100303 mutt - mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 txg_wait_open+0x85 dmu_tx_assign+0x170 zfs_inactive+0xf1 zfs_freebsd_inactive+0x1a vinactive+0x71 vputx+0x2d8 null_reclaim+0xb3 vgonel+0x119 vrecycle+0x7b null_inactive+0x1f vinactive+0x71 vputx+0x2d8 vn_close+0xa1 vn_closefile+0x5a _fdrop+0x23 I'm running a reduced amount of jails on the machine at the moment which is limiting the speed at which the machine freezes up completely. I'd like to debug this problem further, so any advice on useful information to collect would be appreciated. I've had this problem on the machine before[1] but adding more RAM allievated the issue. Tim. [1] http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058541.html -- Tim Bishop http://www.bishnet.net/tim/ PGP Key: 0x5AE7D984 From owner-freebsd-fs@FreeBSD.ORG Tue Mar 27 18:34:58 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7EE731065673 for ; Tue, 27 Mar 2012 18:34:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id E3FAF8FC1C for ; Tue, 27 Mar 2012 18:34:57 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q2RIYgNt079693 for ; Tue, 27 Mar 2012 21:34:42 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q2RIYeCw080261 for ; Tue, 27 Mar 2012 21:34:40 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q2RIYeOG080260 for fs@freebsd.org; Tue, 27 Mar 2012 21:34:40 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 27 Mar 2012 21:34:40 +0300 From: Konstantin Belousov To: fs@freebsd.org Message-ID: <20120327183440.GS2358@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="DBwjNljafULFACtk" Content-Disposition: inline User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Subject: RFC: SEEK_HOLE/SEEK_DATA common implementation X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Mar 2012 18:34:58 -0000 --DBwjNljafULFACtk Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hello, Please find at http://people.freebsd.org/~kib/misc/seek_hole.1.patch a prototype generic implementation of the SEEK_HOLE/SEEK_DATA lseek(2) whence command for any filesystem that properly supports bmap. I was unable to find any test suite for the feature, and the only more or less reasonable case I found was from lklm, extracted at http://people.freebsd.org/~kib/misc/seek_hole.c The block for file with hole at the end is commented out because UFS never puts holes at EOF. The block_size is tuned for default block size on UFS on recent FreeBSD. Filesystem-specific implementations could greatly increase the performance of the call, because the presented implementation does linear search through the blocks until hole/data boundary is found. E.g., for UFS, the fast tree-like structure of indirect blocks can be traversed to point at the next boundary. But this implementation is generic. Please comment. --DBwjNljafULFACtk Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk9yCEAACgkQC3+MBN1Mb4i9IwCgjNfxloBEPUGl6vVAPPEldo9M YJYAn1jeVfualFuYW5AqnRO5QFoBwJT9 =SJml -----END PGP SIGNATURE----- --DBwjNljafULFACtk-- From owner-freebsd-fs@FreeBSD.ORG Wed Mar 28 00:00:22 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49B6D106567B for ; Wed, 28 Mar 2012 00:00:21 +0000 (UTC) (envelope-from dan@3geeks.org) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 231A18FC15 for ; Wed, 28 Mar 2012 00:00:20 +0000 (UTC) Received: by obbuo13 with SMTP id uo13so655301obb.13 for ; Tue, 27 Mar 2012 17:00:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=from:content-type:content-transfer-encoding:subject:date:message-id :to:mime-version:x-mailer:x-gm-message-state; bh=aDvy85UuTySyudOjD7XEHBPS8nie53t69Oe8se5rJRA=; b=Xt2odITvfNm0NLJz15U3HQHs4e7CUTPWvQQkx9acaD5i4OStGENcdDRiGEUFdnm1Xy S3lE/jOD1Zdwi6lMaWJ8lpg9HHAG7VrhDK8YwdJl28cYhdyDYEgTdLDlpdicyd8CrYvn isUojEUEr8XmWdBSdVR9ep/3RKsZL0cirVfu2bOTu8sqFwB9M+SG5P9MgeVCHvqdU3cO Q434FdyfOBJmF1HX6wAKf3KS8Cufw43/CT9VvNjVN+GVFNLt7uAuLcnTKmbZ7mShWlOZ SQxAe6SzWVXfuDQpFUdarbNU92ST5oNVEueMdM8ckqT/E0twABFp/Ly+0LppDMwCWMPx ZgCA== Received: by 10.182.174.101 with SMTP id br5mr35942304obc.0.1332892820183; Tue, 27 Mar 2012 17:00:20 -0700 (PDT) Received: from [10.0.1.2] (99-126-192-237.lightspeed.austtx.sbcglobal.net. [99.126.192.237]) by mx.google.com with ESMTPS id q6sm1398398obz.17.2012.03.27.17.00.17 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 27 Mar 2012 17:00:19 -0700 (PDT) From: Daniel Mayfield Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Tue, 27 Mar 2012 19:00:06 -0500 Message-Id: <6275E6A5-E7A5-49A2-B15F-1BE2EF131A34@3geeks.org> To: freebsd-fs@freebsd.org Mime-Version: 1.0 (Apple Message framework v1084) X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQnd5Pzg35VTAmuMOtmeYzB/8vkIcBsHRe33Oj70IspBh1UljkiwmB5h3PABB0UH0rZYe+pz Subject: mismatch between zfs and zpool X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2012 00:00:22 -0000 Any idea why these mismatch? These are WD20EARS disks in a USB JBOD = enclosure. [dan@rooter ~] $ zfs list NAME USED AVAIL REFER MOUNTPOINT mediavg 1.86T 3.33T 314K /media [dan@rooter ~] $ zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT mediavg 7.25T 2.56T 4.69T 35% 1.00x ONLINE - [dan@rooter ~/Documents/Books]$ zdb mediavg: version: 28 name: 'mediavg' state: 0 txg: 440650 pool_guid: 11730812331217717046 hostid: 530641825 hostname: 'rooter.home.3geeks.org' vdev_children: 1 vdev_tree: type: 'root' id: 0 guid: 11730812331217717046 children[0]: type: 'raidz' id: 0 guid: 2141238275387851622 nparity: 1 metaslab_array: 30 metaslab_shift: 36 ashift: 12 asize: 7984384049152 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 8222953817252947832 path: '/dev/da0p3' phys_path: '/dev/da0p3' whole_disk: 1 DTL: 123 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 10598772377339347959 path: '/dev/da2p3' phys_path: '/dev/da2p3' whole_disk: 1 DTL: 122 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 17878786601530240982 path: '/dev/da1p3' phys_path: '/dev/da1p3' whole_disk: 1 DTL: 121 create_txg: 4 children[3]: type: 'disk' id: 3 guid: 2602895078436511407 path: '/dev/da3p3' phys_path: '/dev/da3p3' whole_disk: 1 DTL: 120 create_txg: 4 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 28 00:03:03 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B8EBE106566C for ; Wed, 28 Mar 2012 00:03:03 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: from mail-qc0-f182.google.com (mail-qc0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6F4968FC18 for ; Wed, 28 Mar 2012 00:03:03 +0000 (UTC) Received: by qcsg15 with SMTP id g15so394413qcs.13 for ; Tue, 27 Mar 2012 17:02:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=CKwju//dJ3wGdBXnkP7LdCDDdhinGaBe2+nJma53iws=; b=y//ucK+nFrk2DcKLrTAVM3IpDGvq2FsgoHcBDAnfPOI4IRY4SFI5acFtr1dgVFYEvY ALmvqzUrsssF6Z73nnsn2mGMdlQJN9xsgdfUbH8D4KWWCYkcjXDjw9WM/lWnrsBoPy4t M2xNZhiVV2xn74WIeiff0goA76NCWH4Lp4PvO5elwoaVrDwR3DMnRq87ERZ/fTyhzfUX VB8JXXRRtFB+FW2SRJQE+HozfS5JNKqZblSHCnEA3c3Y2Exupm26Vrt/6yUtGPBr17Cq WKK91SLXjZrI2+HN/qGQ8kdE08N1//djd/lhh54w7j615WYp1W2sDeABzrEL4sWVjkfV nTqg== MIME-Version: 1.0 Received: by 10.229.75.145 with SMTP id y17mr10369940qcj.116.1332892977561; Tue, 27 Mar 2012 17:02:57 -0700 (PDT) Sender: rincebrain@gmail.com Received: by 10.229.157.16 with HTTP; Tue, 27 Mar 2012 17:02:57 -0700 (PDT) In-Reply-To: <6275E6A5-E7A5-49A2-B15F-1BE2EF131A34@3geeks.org> References: <6275E6A5-E7A5-49A2-B15F-1BE2EF131A34@3geeks.org> Date: Tue, 27 Mar 2012 20:02:57 -0400 X-Google-Sender-Auth: 6IRFgGr4BX8l9bQvTDS1Isx-4yE Message-ID: From: Rich To: Daniel Mayfield Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: mismatch between zfs and zpool X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2012 00:03:03 -0000 zfs list -t all -r, perhaps? - Rich On Tue, Mar 27, 2012 at 8:00 PM, Daniel Mayfield wrote: > Any idea why these mismatch? =A0These are WD20EARS disks in a USB JBOD en= closure. > > [dan@rooter ~] $ zfs list > NAME =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 USED =A0AVAIL = =A0REFER =A0MOUNTPOINT > mediavg =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 1.86T =A03.33T =A0 31= 4K =A0/media > [dan@rooter ~] $ zpool list > NAME =A0 =A0 =A0 =A0 =A0 =A0SIZE =A0ALLOC =A0 FREE =A0 =A0CAP =A0DEDUP = =A0HEALTH =A0ALTROOT > mediavg =A0 =A0 =A0 =A07.25T =A02.56T =A04.69T =A0 =A035% =A01.00x =A0ONL= INE =A0- > [dan@rooter ~/Documents/Books]$ zdb > mediavg: > =A0 =A0version: 28 > =A0 =A0name: 'mediavg' > =A0 =A0state: 0 > =A0 =A0txg: 440650 > =A0 =A0pool_guid: 11730812331217717046 > =A0 =A0hostid: 530641825 > =A0 =A0hostname: 'rooter.home.3geeks.org' > =A0 =A0vdev_children: 1 > =A0 =A0vdev_tree: > =A0 =A0 =A0 =A0type: 'root' > =A0 =A0 =A0 =A0id: 0 > =A0 =A0 =A0 =A0guid: 11730812331217717046 > =A0 =A0 =A0 =A0children[0]: > =A0 =A0 =A0 =A0 =A0 =A0type: 'raidz' > =A0 =A0 =A0 =A0 =A0 =A0id: 0 > =A0 =A0 =A0 =A0 =A0 =A0guid: 2141238275387851622 > =A0 =A0 =A0 =A0 =A0 =A0nparity: 1 > =A0 =A0 =A0 =A0 =A0 =A0metaslab_array: 30 > =A0 =A0 =A0 =A0 =A0 =A0metaslab_shift: 36 > =A0 =A0 =A0 =A0 =A0 =A0ashift: 12 > =A0 =A0 =A0 =A0 =A0 =A0asize: 7984384049152 > =A0 =A0 =A0 =A0 =A0 =A0is_log: 0 > =A0 =A0 =A0 =A0 =A0 =A0create_txg: 4 > =A0 =A0 =A0 =A0 =A0 =A0children[0]: > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0type: 'disk' > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0id: 0 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0guid: 8222953817252947832 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0path: '/dev/da0p3' > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0phys_path: '/dev/da0p3' > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0whole_disk: 1 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0DTL: 123 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0create_txg: 4 > =A0 =A0 =A0 =A0 =A0 =A0children[1]: > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0type: 'disk' > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0id: 1 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0guid: 10598772377339347959 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0path: '/dev/da2p3' > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0phys_path: '/dev/da2p3' > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0whole_disk: 1 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0DTL: 122 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0create_txg: 4 > =A0 =A0 =A0 =A0 =A0 =A0children[2]: > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0type: 'disk' > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0id: 2 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0guid: 17878786601530240982 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0path: '/dev/da1p3' > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0phys_path: '/dev/da1p3' > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0whole_disk: 1 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0DTL: 121 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0create_txg: 4 > =A0 =A0 =A0 =A0 =A0 =A0children[3]: > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0type: 'disk' > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0id: 3 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0guid: 2602895078436511407 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0path: '/dev/da3p3' > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0phys_path: '/dev/da3p3' > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0whole_disk: 1 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0DTL: 120 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0create_txg: 4 > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Wed Mar 28 00:09:33 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E71F1065675 for ; Wed, 28 Mar 2012 00:09:33 +0000 (UTC) (envelope-from dan@3geeks.org) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0C72A8FC1F for ; Wed, 28 Mar 2012 00:09:32 +0000 (UTC) Received: by obbuo13 with SMTP id uo13so668120obb.13 for ; Tue, 27 Mar 2012 17:09:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=kfbrTs0LmlsqMJoDNdY/IyNcjtdN1Y2Hll1edTqoQmE=; b=oJEOEA1eI12dovM9IJ21IhaQ++kN7ptkUH7tJrHDzX7FFbScDWs06BaMIcxM2NWUri flj816AdVPFvcxQS9LeQrVWgm7RQmijNQH1U2gdU7Dz8W7C1ukrvnA/XgLGPaGYsME+v Tzao1paRkldB0Wrd7EJ8UEy/ViuHXywYAjKnXM0onbpEHOYncmWJYJzj4kBUjG2ne6nV oMxORTBMvNm+zL9q1KM4ntS/mtN6g5f6Y2sx17JiCCoHRbzZKQAFYwOg5RpjvRt1t374 e4DzcRGWgfg9tMrpjmq9DiMNVzWBhyAraeUMTx0y8lvBLJjDAuDwY/EMfD4Z2toN43EZ nu/A== Received: by 10.182.0.48 with SMTP id 16mr35825258obb.23.1332893372523; Tue, 27 Mar 2012 17:09:32 -0700 (PDT) Received: from [10.0.1.2] (99-126-192-237.lightspeed.austtx.sbcglobal.net. [99.126.192.237]) by mx.google.com with ESMTPS id k7sm1183895oei.0.2012.03.27.17.09.30 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 27 Mar 2012 17:09:31 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Daniel Mayfield In-Reply-To: Date: Tue, 27 Mar 2012 19:09:26 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <4E85511B-3181-4C46-B71D-237C397DCCAB@3geeks.org> References: <6275E6A5-E7A5-49A2-B15F-1BE2EF131A34@3geeks.org> To: Rich X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQmZW8xHkG/r2URZITu8xa8U4T3XsNv0Sde0vqv39o2D1UY90sKrQp1C4Od9YbMFFH7UPM5N Cc: freebsd-fs@freebsd.org Subject: Re: mismatch between zfs and zpool X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2012 00:09:33 -0000 [dan@rooter ~/Documents/Books]$ zfs list -t all -r NAME USED AVAIL REFER MOUNTPOINT mediavg 1.86T 3.33T 314K /media mediavg/foo 6.43G 3.33T 6.43G /media/foo mediavg/bar 18.9G 3.33T 18.9G /media/bar mediavg/baz 535G 3.33T 535G /media/baz mediavg/qux 16.8G 3.33T 16.8G /media/qux mediavg/quux 132G 3.33T 132G /media/quux mediavg/corge 18.0G 3.33T 18.0G /media/corge mediavggrault 33.8G 3.33T 33.8G /media/grault mediavg/garply 1.12T 3.33T 1.12T /media/garply Daniel On Mar 27, 2012, at 7:02 PM, Rich wrote: > zfs list -t all -r, perhaps? >=20 > - Rich >=20 > On Tue, Mar 27, 2012 at 8:00 PM, Daniel Mayfield = wrote: >> Any idea why these mismatch? These are WD20EARS disks in a USB JBOD = enclosure. >>=20 >> [dan@rooter ~] $ zfs list >> NAME USED AVAIL REFER MOUNTPOINT >> mediavg 1.86T 3.33T 314K /media >> [dan@rooter ~] $ zpool list >> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >> mediavg 7.25T 2.56T 4.69T 35% 1.00x ONLINE - >> [dan@rooter ~/Documents/Books]$ zdb >> mediavg: >> version: 28 >> name: 'mediavg' >> state: 0 >> txg: 440650 >> pool_guid: 11730812331217717046 >> hostid: 530641825 >> hostname: 'rooter.home.3geeks.org' >> vdev_children: 1 >> vdev_tree: >> type: 'root' >> id: 0 >> guid: 11730812331217717046 >> children[0]: >> type: 'raidz' >> id: 0 >> guid: 2141238275387851622 >> nparity: 1 >> metaslab_array: 30 >> metaslab_shift: 36 >> ashift: 12 >> asize: 7984384049152 >> is_log: 0 >> create_txg: 4 >> children[0]: >> type: 'disk' >> id: 0 >> guid: 8222953817252947832 >> path: '/dev/da0p3' >> phys_path: '/dev/da0p3' >> whole_disk: 1 >> DTL: 123 >> create_txg: 4 >> children[1]: >> type: 'disk' >> id: 1 >> guid: 10598772377339347959 >> path: '/dev/da2p3' >> phys_path: '/dev/da2p3' >> whole_disk: 1 >> DTL: 122 >> create_txg: 4 >> children[2]: >> type: 'disk' >> id: 2 >> guid: 17878786601530240982 >> path: '/dev/da1p3' >> phys_path: '/dev/da1p3' >> whole_disk: 1 >> DTL: 121 >> create_txg: 4 >> children[3]: >> type: 'disk' >> id: 3 >> guid: 2602895078436511407 >> path: '/dev/da3p3' >> phys_path: '/dev/da3p3' >> whole_disk: 1 >> DTL: 120 >> create_txg: 4 >>=20 >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Wed Mar 28 00:24:05 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 16BB01065670 for ; Wed, 28 Mar 2012 00:24:05 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: from mail-qc0-f182.google.com (mail-qc0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id C0DDB8FC16 for ; Wed, 28 Mar 2012 00:24:04 +0000 (UTC) Received: by qcsg15 with SMTP id g15so403717qcs.13 for ; Tue, 27 Mar 2012 17:24:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=YJsIsJs5URdnygCZ0OqmQp1/1U/EXuqTc2wYL2Rd344=; b=DaNn+GyXl+NICZD1SGbTeI39tTSDbRiMDeUTzeUWEX29+4FGL/Ojfib0NepfF6durb I+b7kLx9PbX8rf7rmMBbq7u05YliFere2oB8b5BcJ3ZhMdJUk4uiukC3uxmlKe5E65H5 apYTpjdRPcdhdUWVr1mGqjjqSVWAfPacOgaWRreR/pDXjxBHzWUgPw/CLOai1u6BT7bs iZA7th2fxTOYX77k1KXKLIJyH/v2IyVT1JQ9j42WOw5M/ocqKajEFBg1tlclQ0ZjkWij UGk0EjQlSj1iw/79sLeiH1nox5FnG1f8vntCB9PRil40qtJXoANFmikuYkUgguA4DAOl hSzg== MIME-Version: 1.0 Received: by 10.224.73.12 with SMTP id o12mr35060549qaj.98.1332894244061; Tue, 27 Mar 2012 17:24:04 -0700 (PDT) Sender: rincebrain@gmail.com Received: by 10.229.157.16 with HTTP; Tue, 27 Mar 2012 17:24:04 -0700 (PDT) In-Reply-To: <4E85511B-3181-4C46-B71D-237C397DCCAB@3geeks.org> References: <6275E6A5-E7A5-49A2-B15F-1BE2EF131A34@3geeks.org> <4E85511B-3181-4C46-B71D-237C397DCCAB@3geeks.org> Date: Tue, 27 Mar 2012 20:24:04 -0400 X-Google-Sender-Auth: S-H3ALstgLq2eJ2o2j4oO25G5vY Message-ID: From: Rich To: Daniel Mayfield Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: mismatch between zfs and zpool X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2012 00:24:05 -0000 I'd guess RAID-Z1 + metadata of asize=3D12 overhead, but I can't actually swear to it. - Rich On Tue, Mar 27, 2012 at 8:09 PM, Daniel Mayfield wrote: > [dan@rooter ~/Documents/Books]$ zfs list -t all -r > NAME =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 USED =A0AVAIL = =A0REFER =A0MOUNTPOINT > mediavg =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 1.86T =A03.33T =A0 31= 4K =A0/media > mediavg/foo =A0 =A0 =A0 =A0 =A0 =A06.43G =A03.33T =A06.43G =A0/media/foo > mediavg/bar =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 18.9G =A03.33T =A018.9G =A0/m= edia/bar > mediavg/baz =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 535G =A03.33T =A0 535G =A0/me= dia/baz > mediavg/qux =A0 =A0 =A0 =A0 =A0 =A016.8G =A03.33T =A016.8G =A0/media/qux > mediavg/quux =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 132G =A03.33T =A0 132G =A0/m= edia/quux > mediavg/corge =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 18.0G =A03.33T =A018.0G =A0= /media/corge > mediavggrault =A0 =A0 =A0 =A0 =A0 =A0 =A033.8G =A03.33T =A033.8G =A0/medi= a/grault > mediavg/garply =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 1.12T =A03.33T =A01.12= T =A0/media/garply > > Daniel > > On Mar 27, 2012, at 7:02 PM, Rich wrote: > >> zfs list -t all -r, perhaps? >> >> - Rich >> >> On Tue, Mar 27, 2012 at 8:00 PM, Daniel Mayfield wrote: >>> Any idea why these mismatch? =A0These are WD20EARS disks in a USB JBOD = enclosure. >>> >>> [dan@rooter ~] $ zfs list >>> NAME =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 USED =A0AVAIL = =A0REFER =A0MOUNTPOINT >>> mediavg =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 1.86T =A03.33T =A0 = 314K =A0/media >>> [dan@rooter ~] $ zpool list >>> NAME =A0 =A0 =A0 =A0 =A0 =A0SIZE =A0ALLOC =A0 FREE =A0 =A0CAP =A0DEDUP = =A0HEALTH =A0ALTROOT >>> mediavg =A0 =A0 =A0 =A07.25T =A02.56T =A04.69T =A0 =A035% =A01.00x =A0O= NLINE =A0- >>> [dan@rooter ~/Documents/Books]$ zdb >>> mediavg: >>> =A0 =A0version: 28 >>> =A0 =A0name: 'mediavg' >>> =A0 =A0state: 0 >>> =A0 =A0txg: 440650 >>> =A0 =A0pool_guid: 11730812331217717046 >>> =A0 =A0hostid: 530641825 >>> =A0 =A0hostname: 'rooter.home.3geeks.org' >>> =A0 =A0vdev_children: 1 >>> =A0 =A0vdev_tree: >>> =A0 =A0 =A0 =A0type: 'root' >>> =A0 =A0 =A0 =A0id: 0 >>> =A0 =A0 =A0 =A0guid: 11730812331217717046 >>> =A0 =A0 =A0 =A0children[0]: >>> =A0 =A0 =A0 =A0 =A0 =A0type: 'raidz' >>> =A0 =A0 =A0 =A0 =A0 =A0id: 0 >>> =A0 =A0 =A0 =A0 =A0 =A0guid: 2141238275387851622 >>> =A0 =A0 =A0 =A0 =A0 =A0nparity: 1 >>> =A0 =A0 =A0 =A0 =A0 =A0metaslab_array: 30 >>> =A0 =A0 =A0 =A0 =A0 =A0metaslab_shift: 36 >>> =A0 =A0 =A0 =A0 =A0 =A0ashift: 12 >>> =A0 =A0 =A0 =A0 =A0 =A0asize: 7984384049152 >>> =A0 =A0 =A0 =A0 =A0 =A0is_log: 0 >>> =A0 =A0 =A0 =A0 =A0 =A0create_txg: 4 >>> =A0 =A0 =A0 =A0 =A0 =A0children[0]: >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0type: 'disk' >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0id: 0 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0guid: 8222953817252947832 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0path: '/dev/da0p3' >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0phys_path: '/dev/da0p3' >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0whole_disk: 1 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0DTL: 123 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0create_txg: 4 >>> =A0 =A0 =A0 =A0 =A0 =A0children[1]: >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0type: 'disk' >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0id: 1 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0guid: 10598772377339347959 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0path: '/dev/da2p3' >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0phys_path: '/dev/da2p3' >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0whole_disk: 1 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0DTL: 122 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0create_txg: 4 >>> =A0 =A0 =A0 =A0 =A0 =A0children[2]: >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0type: 'disk' >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0id: 2 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0guid: 17878786601530240982 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0path: '/dev/da1p3' >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0phys_path: '/dev/da1p3' >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0whole_disk: 1 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0DTL: 121 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0create_txg: 4 >>> =A0 =A0 =A0 =A0 =A0 =A0children[3]: >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0type: 'disk' >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0id: 3 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0guid: 2602895078436511407 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0path: '/dev/da3p3' >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0phys_path: '/dev/da3p3' >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0whole_disk: 1 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0DTL: 120 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0create_txg: 4 >>> >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Wed Mar 28 00:55:30 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 72C0F1065670 for ; Wed, 28 Mar 2012 00:55:30 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id CB9668FC0A for ; Wed, 28 Mar 2012 00:55:29 +0000 (UTC) Received: by ghrr20 with SMTP id r20so528749ghr.13 for ; Tue, 27 Mar 2012 17:55:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=xNOPly5UWfhpXIbFVVAhjEMJWxbWTZe7i8l0VgXL6lE=; b=PlGOsr8m7wCqIXXlvZqArkvSXdRh4LLtYmZywIcf98mNvDI7LaOzmxT3e+HBBAMOsN P+F78ttXioTJCHk6Q0ra0IM+b9zaBCgIwJ0C2CC+PX7nDETpEJiWCrJtmaORTzxkprOG Cx4CYV73AEhxIgqiVXSQioVzKz5sM9dA9ZvTfjhvPdCs9Rvk9oLz1yCVj3X5lmzvUPu1 ba6M7ANt9+UGBWG5qPrXICeTvx+S7sifo1rG30O/wLH8gUXZRKrLJnMRZgb/6WcLuhjQ PBelKoGvVQgftr8v5WmeYEZ+uQy3eZ/oMP3kfqXFa6fcKiL2wn+rfKzcWFhf6IOeQ383 SQZw== MIME-Version: 1.0 Received: by 10.68.235.106 with SMTP id ul10mr21719779pbc.91.1332896128616; Tue, 27 Mar 2012 17:55:28 -0700 (PDT) Received: by 10.68.135.234 with HTTP; Tue, 27 Mar 2012 17:55:28 -0700 (PDT) Received: by 10.68.135.234 with HTTP; Tue, 27 Mar 2012 17:55:28 -0700 (PDT) In-Reply-To: <6275E6A5-E7A5-49A2-B15F-1BE2EF131A34@3geeks.org> References: <6275E6A5-E7A5-49A2-B15F-1BE2EF131A34@3geeks.org> Date: Tue, 27 Mar 2012 17:55:28 -0700 Message-ID: From: Freddie Cash To: Daniel Mayfield Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: mismatch between zfs and zpool X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2012 00:55:30 -0000 On Mar 27, 2012 5:00 PM, "Daniel Mayfield" wrote: > > Any idea why these mismatch? These are WD20EARS disks in a USB JBOD enclosure. > > [dan@rooter ~] $ zfs list > NAME USED AVAIL REFER MOUNTPOINT > mediavg 1.86T 3.33T 314K /media > [dan@rooter ~] $ zpool list > NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT > mediavg 7.25T 2.56T 4.69T 35% 1.00x ONLINE - If you the differences in used/alloc and avail/free between zfs and zpool, it's because they show different things. Search the archives and the zfs-discuss archives for all the gory details. The cliff's notes version is that zpool shows raw storage available in the pool, and zfs shows usable storage space after redundancy is taken into account. From owner-freebsd-fs@FreeBSD.ORG Wed Mar 28 09:26:45 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 04548106566C for ; Wed, 28 Mar 2012 09:26:45 +0000 (UTC) (envelope-from rsb@berentweb.com) Received: from mail-lpp01m010-f54.google.com (mail-lpp01m010-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 6BA3C8FC0A for ; Wed, 28 Mar 2012 09:26:44 +0000 (UTC) Received: by lagv3 with SMTP id v3so1380033lag.13 for ; Wed, 28 Mar 2012 02:26:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=berentweb.com; s=google; h=mime-version:sender:x-originating-ip:date:x-google-sender-auth :message-id:subject:from:to:content-type; bh=ptULYrTOOjDr2AD+mFcRy/E49rT0SlrgiuVhAy2vsUQ=; b=YpR/Y22zNV64hXShgGOMsmp14aNG4cy66thvkg2Cr1zlNZuKxxBZpzpEwh76dpvTdZ IGafbrxGGfsEg1al3ZeaVGbgt6NSayz2o+yh+2V5TOuT+775yM9EHq5eLLZUVT3RPNb1 B67GbeUXSBMrmDD0t+tBIfaMTh2i/OAS4rQzw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:x-originating-ip:date:x-google-sender-auth :message-id:subject:from:to:content-type:x-gm-message-state; bh=ptULYrTOOjDr2AD+mFcRy/E49rT0SlrgiuVhAy2vsUQ=; b=TZhK1f8pND4uHq0HVP9B3+dM5t/Zf/dzrHneo2JvrtES1aRX6w8BiEgOT4JGipLVou qHYPdtIu7+JDul3R2WD/6sZhWVcOYaOUHQ6C3KoOzY40Mvo9o77lMVor0gWYR/SsGtPs qZlAcyfafDudGwhqzQ/WN3QzxxoTo05PSX586x0E+ihGjuA7sUZUyth+Wl0NniU0OwCo broQzzCTjkJlYD5xI2EbEp0zCVpCvNOtuZmExW7sSSY7iU8RbSRcH5kFGKf2erQaU244 SJiyyuP1rkbu3dHuC4A9awOLxC23TnPYASSZglW1lF8rUbJNSnEWasUrHCnKtn2Az9Bw r8uQ== MIME-Version: 1.0 Received: by 10.152.110.170 with SMTP id ib10mr24604968lab.7.1332926802956; Wed, 28 Mar 2012 02:26:42 -0700 (PDT) Sender: rsb@berentweb.com Received: by 10.112.77.15 with HTTP; Wed, 28 Mar 2012 02:26:42 -0700 (PDT) X-Originating-IP: [85.110.15.53] Date: Wed, 28 Mar 2012 12:26:42 +0300 X-Google-Sender-Auth: _Njl4CENt9EaS4FRJC_J7SlqbLk Message-ID: From: Beeblebrox To: freebsd-fs@freebsd.org X-Gm-Message-State: ALoCoQnVAWs6OSuGCc1JbAqhwkmR0Z29MTAm1ct0bFUHGvCJquCA6e27bdrNJa7pUsynWPUcdg0B Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: unionfs: strange error when mounted in jail. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2012 09:26:45 -0000 I am using ezjail to setup some jailed services. Ezjail sets up a light-jail and a base(fat)-jail which is mounted "ro" under the light-jail. I modified this to experiment with an idea. In addition to the standard mount_nullfs ezjail does, I placed in /etc/fstab.pxe: > > /data/jails/base/usr/local /data/jails/pxe/usr/local unionfs rw,below,noatime,copymode=masquerade 0 0 > > /data/jails/base/var/db/pkg /data/jails/pxe/var/db/pkg unionfs rw,below,noatime,copymode=masquerade 0 0 What this does, is it allows to create a "base ports" fat-jail where all "base ports" will be included in subsequently created thin-jails, not as copy but as a unionfs layer. When updating ports in the jails, the "base ports" package will have to be updated only once, instead of repeated update for each thin-jail. I also have zfs dedup=on for the /data/jails folder. I install jail-specific ports into the thin-jail by normal process -> start the jail normally & pkg_add. I then shut-down jail, modify the jail's fstab (fstab.pxe) and re-start jail. The problem I run into: Start the jail, then #jexec into jailed env. Execute an app or service like portmaster (installed into /data/jails/base) or dhcpd installed into /data/jails/pxe) portmaster will start-up, but hangs after a short run. dhcpd hangs and freezes the jail immediately - I cannot kill any of the processes and have to do a poweroff. Both executables run normally without the unionfs structure. Regards. From owner-freebsd-fs@FreeBSD.ORG Wed Mar 28 09:52:18 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01A2B1065680 for ; Wed, 28 Mar 2012 09:52:18 +0000 (UTC) (envelope-from peter.maloney@brockmann-consult.de) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.17.9]) by mx1.freebsd.org (Postfix) with ESMTP id 7D6668FC08 for ; Wed, 28 Mar 2012 09:52:17 +0000 (UTC) Received: from [10.3.0.26] ([141.4.215.32]) by mrelayeu.kundenserver.de (node=mreu0) with ESMTP (Nemesis) id 0Lr6Rz-1ShCDy39pj-00efGj; Wed, 28 Mar 2012 11:52:09 +0200 Message-ID: <4F72DF48.6040108@brockmann-consult.de> Date: Wed, 28 Mar 2012 11:52:08 +0200 From: Peter Maloney User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:6mMhQujBIhb1pKponF+3Om1Wgn1EsVCdYSui6ekb2rX rDFbE2gZ176b9E33VZHBkT+kaRuEgy97bY3QROn0I/LO2JfcrF sYwjGzr0HaVQWPitvModSZIzUTh3Uz8mL7MGgcS1plcFaQN9ww gvviWoYyBzVHC2A0CMaY+DQHjar+CKfNRqetZTf+2RD2QocQnZ Jk6joMEp0Nix3ynV26UIw/1yMjeinodqc2P5c9BZMSY+tdZwyV PKOexegIbKKjQjEo5aXAp1HGhsUrvBqmIwwAe3Jya1erdGoHZr mYz22tQoR0Mx3bRn7kw+aOFCSVqBhjPiXgFmmatQhR+cwGoVqw FE4h2TuRsD5ztL4l3nwOksnjoGK0OXbIZfeXq4Pry Subject: Strange ZFS deadlock today X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2012 09:52:18 -0000 I believe it is caused by ZFS, since the state looks like the code in this page about ARC: [url]http://dtrace.org/blogs/brendan/2012/01/09/activity-of-the-zfs-arc/[/url] To cause/reveal this problem, first, I tried to run something with OpenJDK 6. In one window: [CODE]# ./CopyFromHd.sh load: 0.00 cmd: java 51239 [ucond] 44.56r 0.18u 0.06s 0% 19588k load: 0.00 cmd: java 51239 [ucond] 46.22r 0.18u 0.06s 0% 19588k load: 0.00 cmd: java 51239 [ucond] 51.14r 0.18u 0.06s 0% 19588k load: 0.00 cmd: java 51239 [ucond] 52.90r 0.18u 0.06s 0% 19588k ^C load: 0.00 cmd: java 51239 [buf_hash_table.ht_locks[i].ht_lock] 58.35r 0.18u 0.06s 0% 19872k load: 0.00 cmd: java 51239 [buf_hash_table.ht_locks[i].ht_lock] 61.73r 0.18u 0.06s 0% 19872k ^C load: 0.00 cmd: java 51239 [buf_hash_table.ht_locks[i].ht_lock] 89.67r 0.18u 0.06s 0% 19872k load: 0.00 cmd: java 51239 [buf_hash_table.ht_locks[i].ht_lock] 141.07r 0.18u 0.06s 0% 19872k ^C[/CODE] Then since it was hung, I thought maybe it was zfs' fault rather than OpenJDK, so tried with du. Second window: [CODE] # du -shx /data/archive2/2011/09/11/x 3.1G /data/archive2/2011/09/11/x # du -shx /data/archive2/2011/09/11 24G /data/archive2/2011/09/11 # du -shx /data/archive2/2011/ load: 0.00 cmd: du 72503 [buf_hash_table.ht_locks[i].ht_lock] 13.75r 0.00u 0.00s 0% 1012k ^C^C^C^Z^C load: 0.00 cmd: du 72503 [buf_hash_table.ht_locks[i].ht_lock] 221.97r 0.00u 0.00s 0% 1012k [/CODE] Then I thought I could just run the simplest version of the program (which does pretty much no IO) Third window: [CODE]# ./CopyFromHd.sh --help ^C^C^C^C^C^C load: 0.00 cmd: java 52339 [suspended] 26.33r 0.15u 0.04s 0% 25644k load: 0.00 cmd: java 52339 [suspended] 27.38r 0.15u 0.04s 0% 25644k ^C load: 0.00 cmd: java 52339 [suspended] 285.23r 0.15u 0.04s 0% 25644k ^Z [1]+ Stopped ./CopyFromHd.sh # jobs -p 51988 # kill -9 51988 # jobs -p 51988 [1]+ Killed: 9 ./CopyFromHd.sh # jobs -p [/CODE] Back to the first window: [CODE]load: 0.00 cmd: java 51239 [buf_hash_table.ht_locks[i].ht_lock] 459.38r 0.18u 0.06s 0% 19872k ^C^Z [1]+ Stopped ./CopyFromHd.sh ... # jobs -p 51128 # kill -9 51128 # jobs -p 51128 [1]+ Interrupt: 2 ./CopyFromHd.sh ... # jobs -p # ps axl | grep java 0 51239 1 0 44 0 1264940 19904 - T 0 0:00.25 [java] 0 76933 77797 0 44 0 9124 1180 piperd S+ 0 0:00.00 grep java 0 52339 1 0 44 0 1266988 25676 - T 1 0:00.20 /usr/local/openjdk6 //bin/java -Xmx1024M -classpath ... [/CODE] The java above is the "--help" one from the 2nd window. So I guess the first one ended. But "dd" commands I ran are all stuck, and so is the 2nd java, and also "zdb" "zdb zroot" "zdb tank" and "zdb data" are all stuck. Also running "find" on either tank or data hangs. [CMD="#"]zpool iostat[/CMD] [CMD="#"]zpool status [-v][/CMD] and [CMD="#"]zfs list[/CMD] all run without hanging. Then I thought maybe I would disable primarycache to see what happens. [CODE]# zfs set primarycache=none data # zfs set primarycache=none tank load: 0.00 cmd: zfs 80750 [tx->tx_sync_done_cv)] 5.73r 0.00u 0.00s 0% 1636k (hang) ^Z^C load: 0.00 cmd: zfs 80750 [tx->tx_sync_done_cv)] 87.28r 0.00u 0.00s 0% 1636k [/CODE] Another window: [CODE]# zfs get primarycache data NAME PROPERTY VALUE SOURCE data primarycache none local # zfs get primarycache tank NAME PROPERTY VALUE SOURCE tank primarycache none local[/CODE] On a Linux server that nfs mounts /data on this FreeBSD server, [CMD="#"]df[/CMD] hangs at the point where the nfs mount should be listed. (So I have to reboot now rather than later) I have not run memtest on this machine. I have no l2arc. Uptime is 22 days. It is normally perfectly stable. I ran CopyFromHd to process over 20 TB of data so far (2 weeks ago), so it is not normal for this to happen. Other similar servers don't hang this way. (most hangs I've caused in the past are not in this "buf_hash_table.ht_locks[i].ht_lock" state. The last thing I did before this was running "zdb -S tank" yesterday. (tank is not the same pool as data) FreeBSD version is 8.2-STABLE csupped on 2012-02-04. And then on restart, the console showed: "some processes would not die; ps axl advised" and "shutdown terminated abnormally" From owner-freebsd-fs@FreeBSD.ORG Wed Mar 28 10:50:41 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B8A4F106564A for ; Wed, 28 Mar 2012 10:50:41 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: from mail-bk0-f54.google.com (mail-bk0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id EDE558FC15 for ; Wed, 28 Mar 2012 10:50:40 +0000 (UTC) Received: by bkcjc3 with SMTP id jc3so1002851bkc.13 for ; Wed, 28 Mar 2012 03:50:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type; bh=wgEUxJeD2qJ7kywF2pJbsmbIwZ8kFRDaPVn6tlX5yRY=; b=CacpEHCpRQmLwfM5O0V2SEyRVbZDbCKOmZgZZWf/qls2utgr3w8M4Ra76DJYg2GLFt Rxv5Cddk4+Gl/QC0Kl9eIeq4USip3Wo82NwkRk9lseLty0m/AH1Bu63pJeUjTiTYT/6t 0VsaDPB0lDe4+AMOmElOqNdRGtnG/018qIu/dCF8FW8xPI4N8AZUIE6bcNqe6v/9Vwxf tcrEzL+IDuw6FYAoRAEAEZcxdC0j60NCNJa6Uyg+kT0YSAK6YW08CfBH5nHLHoc09vMu 0ZgMukIvf/zM0KCwCWzJJ3mLONU3bd4Pb6/GYaCYay4ZxuFLblCrareUoxqZsubZV1ky Ikdg== Received: by 10.205.133.10 with SMTP id hw10mr11344361bkc.61.1332931839691; Wed, 28 Mar 2012 03:50:39 -0700 (PDT) Received: from [192.168.50.103] (double-l.xs4all.nl. [80.126.205.144]) by mx.google.com with ESMTPS id zx16sm5561133bkb.13.2012.03.28.03.50.37 (version=SSLv3 cipher=OTHER); Wed, 28 Mar 2012 03:50:38 -0700 (PDT) Message-ID: <4F72ECFC.4040902@gmail.com> Date: Wed, 28 Mar 2012 12:50:36 +0200 From: Johan Hendriks User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:11.0) Gecko/20120312 Thunderbird/11.0 MIME-Version: 1.0 To: Tim Bishop , freebsd-fs@freebsd.org References: <20120327181457.GC24787@carrick-users.bishnet.net> In-Reply-To: <20120327181457.GC24787@carrick-users.bishnet.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: ZFS: processes hanging when trying to access filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2012 10:50:41 -0000 Tim Bishop schreef: > I have a machine running 8-STABLE amd64 from the end of last week. I > have a problem where the machine starts to freeze up. Any process > accessing the ZFS filesystems hangs, which eventually causes more and > more processes to be spawned (cronjobs, etc, never complete). Although > the root filesystem is on UFS (the machine hosts jails on ZFS), > eventually I can't log in anymore. > > The problem occurs when the frequently used part of the ARC gets too > large. See this graph: > > http://dl.dropbox.com/u/318044/zfs_arc_utilization-day.png > > At the right of the graph things started to hang. > > At the same time I see a high amount of context switching. > > I picked a hanging process and procstat showed the following: > > PID TID COMM TDNAME KSTACK > 24787 100303 mutt - mi_switch+0x176 sleepq_wait+0x42 _cv_wait+0x129 txg_wait_open+0x85 dmu_tx_assign+0x170 zfs_inactive+0xf1 zfs_freebsd_inactive+0x1a vinactive+0x71 vputx+0x2d8 null_reclaim+0xb3 vgonel+0x119 vrecycle+0x7b null_inactive+0x1f vinactive+0x71 vputx+0x2d8 vn_close+0xa1 vn_closefile+0x5a _fdrop+0x23 > > I'm running a reduced amount of jails on the machine at the moment which > is limiting the speed at which the machine freezes up completely. I'd > like to debug this problem further, so any advice on useful information > to collect would be appreciated. > > I've had this problem on the machine before[1] but adding more RAM > allievated the issue. > > Tim. > > [1] http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058541.html > Just a me too, only i am on FreeBSD 9.0-RELEASE AMD64 Once a week the system starts to hang. System boots from a normal disk in a mirror on UFS, the zpool is on the SAS drives in the bays. NFS is in tx >-tx state , but can not be restarted, same goes for mountd and samba leaves a lot of smbd processen in a zfs state. The only way to come out of it is a reset and restart the machine. We use the machine as NFS server for two ESXi5.0 machines. The strange thing is we have an almost identical machine that does not show this behaviour, same board, memory and raid controller. The only difference is that that machine is a 4U case. Settings in loader.conf are. # ZFS zfs_load="YES" # Tuning vfs.zfs.arc_max="12G" It gets really frustrating, there is an exchange server running on it, and i am becoming an eseutil expert which is not something i want :D. Second problem is that the whole company can not work, and i need to get the things up as fast as i can. It happened on moday only till now, but the time is random. The sytem is a 16 bay supermicro with a LSI 9211-8i card and 16 GB mem on the X9SCM-F board. Nothing fancy My arc stats at this moment are. The only thing is that i do not fully know which stats are important. zfs-stat -AE ZFS Subsystem Report Wed Mar 28 12:34:54 2012 ------------------------------------------------------------------------ ARC Summary: (HEALTHY) Memory Throttle Count: 0 ARC Misc: Deleted: 12.40m Recycle Misses: 22.95k Mutex Misses: 12.53k Evict Skips: 64.35k ARC Size: 95.44% 11.45 GiB Target Size: (Adaptive) 95.44% 11.45 GiB Min Size (Hard Limit): 12.50% 1.50 GiB Max Size (High Water): 8:1 12.00 GiB ARC Size Breakdown: Recently Used Cache Size: 93.75% 10.74 GiB Frequently Used Cache Size: 6.25% 733.11 MiB ARC Hash Breakdown: Elements Max: 308.73k Elements Current: 99.68% 307.75k Collisions: 13.69m Chain Max: 16 Chains: 77.43k ------------------------------------------------------------------------ ARC Efficiency: 69.94m Cache Hit Ratio: 83.10% 58.12m Cache Miss Ratio: 16.90% 11.82m Actual Hit Ratio: 67.26% 47.04m Data Demand Efficiency: 94.55% 35.86m Data Prefetch Efficiency: 43.37% 17.21m CACHE HITS BY CACHE LIST: Anonymously Used: 17.49% 10.16m Most Recently Used: 24.32% 14.13m Most Frequently Used: 56.62% 32.91m Most Recently Used Ghost: 0.24% 140.34k Most Frequently Used Ghost: 1.34% 776.73k CACHE HITS BY DATA TYPE: Demand Data: 58.33% 33.90m Prefetch Data: 12.84% 7.46m Demand Metadata: 22.47% 13.06m Prefetch Metadata: 6.35% 3.69m CACHE MISSES BY DATA TYPE: Demand Data: 16.53% 1.95m Prefetch Data: 82.46% 9.75m Demand Metadata: 0.42% 50.02k Prefetch Metadata: 0.60% 70.50k ------------------------------------------------------------------------ The latest top when it hangs was. i already did shutdown samba and restarted nfs, mountd but these also hangs. 1719 root4200 10000K1300K tx->tx038:030.00% nfsd 1884 root1200 24380K3112K select30:070.00% ntpd 1933 root1260 18500K1860K nanslp00:000.00% cron 1606 root1200 16424K1776K select30:000.00% syslogd 1695 root1200 14292K1828K select10:000.00% nfsuserd 1692 root1200 14292K1828K select30:000.00% nfsuserd 1693 root1200 14292K1828K select00:000.00% nfsuserd 1694 root1200 14292K1828K select20:000.00% nfsuserd 19312 adminusr1200 70184K5524K select10:000.00% sshd 19412 root1200 20940K2536K CPU000:000.00% top 19164 root1200 70184K5440K sbwait00:000.00% sshd 19309 root1210 70184K5440K sbwait00:000.00% sshd 19175 root1200 70184K5440K sbwait00:000.00% sshd 19228 root1200 70184K5440K sbwait00:000.00% sshd 19240 root1200 80784K 12068K zfs30:000.00% smbd 19286 root1200 80784K 12064K zfs00:000.00% smbd 19131 root1200 80784K 12060K zfs30:000.00% smbd 18887 root1200 80784K 12060K zfs10:000.00% smbd 19095 root1200 80784K 12064K zfs00:000.00% smbd 19089 root1200 80784K 12064K zfs10:000.00% smbd 18929 root1200 80784K 12064K zfs00:000.00% smbd 18977 root1210 80784K 12056K zfs10:000.00% smbd 19062 root1200 80784K 12056K zfs10:000.00% smbd 18944 root1200 80784K 12056K zfs 10:000.00% smbd 19063 root1200 80784K 12056K zfs10:000.00% smbd 19231 adminusr1200 70184K5524K select20:000.00% sshd 19317 root1200 21812K3152K wait00:000.00% bash 19178 adminusr1200 70184K5524K select10:000.00% sshd 19236 root1210 21812K3156K wait00:000.00% bash 18924 root1200 80784K 12060K zfs30:000.00% smbd At the topic starter, how do you get the graphs for the arc data? regards Johan Hendriks From owner-freebsd-fs@FreeBSD.ORG Wed Mar 28 11:13:33 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 783E91065674 for ; Wed, 28 Mar 2012 11:13:33 +0000 (UTC) (envelope-from peter.maloney@brockmann-consult.de) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.171]) by mx1.freebsd.org (Postfix) with ESMTP id 21CDB8FC1A for ; Wed, 28 Mar 2012 11:13:33 +0000 (UTC) Received: from [10.3.0.26] ([141.4.215.32]) by mrelayeu.kundenserver.de (node=mreu1) with ESMTP (Nemesis) id 0LlrKG-1SluVS05Ym-00Zcgd; Wed, 28 Mar 2012 13:08:22 +0200 Message-ID: <4F72F125.6070404@brockmann-consult.de> Date: Wed, 28 Mar 2012 13:08:21 +0200 From: Peter Maloney User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4F72DF48.6040108@brockmann-consult.de> In-Reply-To: <4F72DF48.6040108@brockmann-consult.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:OzBrQgME5BZk5IyGmwNhb+xEh3tFuYpIEdzZR/V+yNs A7wUdXZPG2l3yDgEE0GjhjdjkJWMWllz79nQEPKSHYdkl/18jh Sc/CdY+gAIOeJENCdHMxyTY9gl52xOPJhY0SHdyyRDpbAcxzOa MeWmlI5BtnYZ9WQwhx2ZjBTX4k4Rm1rEvAADPPCO2MQ8gE4ga1 Gzrr9iB1PK+++MJeAJ1kZ5thXolAcqzbtdsRgSy+6gtpPxd6ZO +s5x9eo2XsioXAUknq6n2UxTDJmLk9yvw7HQxNnKvziO2qo7Ka 2JyvV9huh0xP4/Tv6xdcBRN/dFtL1Ah3ryz0LrymfowLxLst26 Kp154Pq1J/IEiFcxyqrcZ5X/vJD4OxIbqGGoxlFTF Subject: Re: Strange ZFS deadlock today X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2012 11:13:33 -0000 And the top command reported (I think this was before running the first java program): 54 processes: 1 running, 52 sleeping, 1 stopped CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 46M Active, 5504K Inact, 34G Wired, 5808K Cache, 2592K Buf, 13G Free Swap: 80G Total, 25M Used, 80G Free And after running "zdb -S" I am fairly sure zfs-stats also showed arc around 10-13 GB below the arc max I set. But I'm not sure I ran the command when "zdb -S" was done, or just after a long time of it running. And compression and dedup are not enabled on any dataset. /boot/loader.conf snippet: vm.kmem_size="40g" vm.kmem_size_max="40g" vfs.zfs.arc_min="80m" vfs.zfs.arc_max="38g" vfs.zfs.arc_meta_limit="24g" vfs.zfs.vdev.cache.size="32m" vfs.zfs.vdev.cache.max="256m" On 03/28/2012 11:52 AM, Peter Maloney wrote: > I believe it is caused by ZFS, since the state looks like the code in > this page about ARC: > [url]http://dtrace.org/blogs/brendan/2012/01/09/activity-of-the-zfs-arc/[/url] > > To cause/reveal this problem, first, I tried to run something with > OpenJDK 6. > > In one window: > [CODE]# ./CopyFromHd.sh > load: 0.00 cmd: java 51239 [ucond] 44.56r 0.18u 0.06s 0% 19588k > load: 0.00 cmd: java 51239 [ucond] 46.22r 0.18u 0.06s 0% 19588k > load: 0.00 cmd: java 51239 [ucond] 51.14r 0.18u 0.06s 0% 19588k > load: 0.00 cmd: java 51239 [ucond] 52.90r 0.18u 0.06s 0% 19588k > ^C > load: 0.00 cmd: java 51239 [buf_hash_table.ht_locks[i].ht_lock] > 58.35r 0.18u 0.06s 0% 19872k > load: 0.00 cmd: java 51239 [buf_hash_table.ht_locks[i].ht_lock] > 61.73r 0.18u 0.06s 0% 19872k > ^C > load: 0.00 cmd: java 51239 [buf_hash_table.ht_locks[i].ht_lock] > 89.67r 0.18u 0.06s 0% 19872k > load: 0.00 cmd: java 51239 [buf_hash_table.ht_locks[i].ht_lock] > 141.07r 0.18u 0.06s 0% 19872k > ^C[/CODE] > > Then since it was hung, I thought maybe it was zfs' fault rather than > OpenJDK, so tried with du. > > Second window: > [CODE] > # du -shx /data/archive2/2011/09/11/x > 3.1G /data/archive2/2011/09/11/x > > # du -shx /data/archive2/2011/09/11 > 24G /data/archive2/2011/09/11 > > # du -shx /data/archive2/2011/ > load: 0.00 cmd: du 72503 [buf_hash_table.ht_locks[i].ht_lock] 13.75r > 0.00u 0.00s 0% 1012k > ^C^C^C^Z^C > load: 0.00 cmd: du 72503 [buf_hash_table.ht_locks[i].ht_lock] 221.97r > 0.00u 0.00s 0% 1012k > [/CODE] > > Then I thought I could just run the simplest version of the program > (which does pretty much no IO) > > Third window: > [CODE]# ./CopyFromHd.sh --help > ^C^C^C^C^C^C > load: 0.00 cmd: java 52339 [suspended] 26.33r 0.15u 0.04s 0% 25644k > load: 0.00 cmd: java 52339 [suspended] 27.38r 0.15u 0.04s 0% 25644k > ^C > load: 0.00 cmd: java 52339 [suspended] 285.23r 0.15u 0.04s 0% 25644k > ^Z > [1]+ Stopped ./CopyFromHd.sh > # jobs -p > 51988 > # kill -9 51988 > # jobs -p > 51988 > [1]+ Killed: 9 ./CopyFromHd.sh > # jobs -p > [/CODE] > > Back to the first window: > [CODE]load: 0.00 cmd: java 51239 [buf_hash_table.ht_locks[i].ht_lock] > 459.38r 0.18u 0.06s 0% 19872k > ^C^Z > [1]+ Stopped ./CopyFromHd.sh ... > # jobs -p > 51128 > # kill -9 51128 > # jobs -p > 51128 > [1]+ Interrupt: 2 ./CopyFromHd.sh ... > # jobs -p > # ps axl | grep java > 0 51239 1 0 44 0 1264940 19904 - T 0 0:00.25 > [java] > 0 76933 77797 0 44 0 9124 1180 piperd S+ 0 0:00.00 > grep java > 0 52339 1 0 44 0 1266988 25676 - T 1 0:00.20 > /usr/local/openjdk6 //bin/java -Xmx1024M -classpath ... > [/CODE] > > > The java above is the "--help" one from the 2nd window. So I guess the > first one ended. > > But "dd" commands I ran are all stuck, and so is the 2nd java, and > also "zdb" "zdb zroot" "zdb tank" and "zdb data" are all stuck. Also > running "find" on either tank or data hangs. > > [CMD="#"]zpool iostat[/CMD] [CMD="#"]zpool status [-v][/CMD] and > [CMD="#"]zfs list[/CMD] all run without hanging. > > Then I thought maybe I would disable primarycache to see what happens. > > [CODE]# zfs set primarycache=none data > # zfs set primarycache=none tank > load: 0.00 cmd: zfs 80750 [tx->tx_sync_done_cv)] 5.73r 0.00u 0.00s 0% > 1636k > (hang) > ^Z^C > load: 0.00 cmd: zfs 80750 [tx->tx_sync_done_cv)] 87.28r 0.00u 0.00s > 0% 1636k > [/CODE] > > Another window: > [CODE]# zfs get primarycache data > NAME PROPERTY VALUE SOURCE > data primarycache none local > # zfs get primarycache tank > NAME PROPERTY VALUE SOURCE > tank primarycache none local[/CODE] > > > On a Linux server that nfs mounts /data on this FreeBSD server, > [CMD="#"]df[/CMD] hangs at the point where the nfs mount should be > listed. (So I have to reboot now rather than later) > > I have not run memtest on this machine. > > I have no l2arc. > > Uptime is 22 days. > > It is normally perfectly stable. I ran CopyFromHd to process over 20 > TB of data so far (2 weeks ago), so it is not normal for this to > happen. Other similar servers don't hang this way. (most hangs I've > caused in the past are not in this > "buf_hash_table.ht_locks[i].ht_lock" state. > > The last thing I did before this was running "zdb -S tank" yesterday. > (tank is not the same pool as data) > > FreeBSD version is 8.2-STABLE csupped on 2012-02-04. > > > And then on restart, the console showed: "some processes would not > die; ps axl advised" and "shutdown terminated abnormally" > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" -- -------------------------------------------- Peter Maloney Brockmann Consult Max-Planck-Str. 2 21502 Geesthacht Germany Tel: +49 4152 889 300 Fax: +49 4152 889 333 E-mail: peter.maloney@brockmann-consult.de Internet: http://www.brockmann-consult.de -------------------------------------------- From owner-freebsd-fs@FreeBSD.ORG Wed Mar 28 20:20:38 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9D15B106566B for ; Wed, 28 Mar 2012 20:20:38 +0000 (UTC) (envelope-from rsb@berentweb.com) Received: from mail-lpp01m010-f54.google.com (mail-lpp01m010-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 15B8B8FC12 for ; Wed, 28 Mar 2012 20:20:37 +0000 (UTC) Received: by lagv3 with SMTP id v3so2344486lag.13 for ; Wed, 28 Mar 2012 13:20:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=berentweb.com; s=google; h=mime-version:sender:x-originating-ip:date:x-google-sender-auth :message-id:subject:from:to:content-type; bh=tc9QPNc7Hwp/6Fl0GaxpFQuws+gnpf2/4fex0djlHcc=; b=MjauLXxIBrC2nCM4+jT9xax2bNTE3zyzocZWGQmCGdRF4Lt6Y+pvm3UEjlG52FKOYu tuIgOMz2XP90lDofQlqtNeLWruTuxyVFQWG380ILXIdjVpYBcKwV4Hbz7KcN/e1QmYDX c+XZ9BENtVzkjVFJf9k3xychvoc41Q3c9043s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:x-originating-ip:date:x-google-sender-auth :message-id:subject:from:to:content-type:x-gm-message-state; bh=tc9QPNc7Hwp/6Fl0GaxpFQuws+gnpf2/4fex0djlHcc=; b=fEg8DHAFRv698iFq97jPpqYw7UBv2hNPlX5XmffS9zY/hOjiT1h2YQsXqHAvoBk66D 5OQ2qq/oGDyd2lbnrGSzoPTQz22aLiUG0x/+Zbql82LBrLTizAwo7+YS0VaoKgKmRc+F 0tAMdGktp2Ybi6lWuE/Pw2vWBkWRsGlyU0NdtWx2Riw4MLF4lkf6jbibDbH1g5lWX05w z6p4qjTKjkWlLN/NvrMAckmIKdahnFjLil3g3lZt0F407Mw3kRKZuJbmMr+lVvi3in4v BOkKOEYBf9ue28sQbU9tzeulXz9sorzZ98n/SgTmbBz4ZBuKANrSs0sxgmdmKkXQuZHU Jl+Q== MIME-Version: 1.0 Received: by 10.152.105.241 with SMTP id gp17mr26858987lab.21.1332966036843; Wed, 28 Mar 2012 13:20:36 -0700 (PDT) Sender: rsb@berentweb.com Received: by 10.112.77.15 with HTTP; Wed, 28 Mar 2012 13:20:36 -0700 (PDT) X-Originating-IP: [85.110.91.209] Date: Wed, 28 Mar 2012 23:20:36 +0300 X-Google-Sender-Auth: G_sY4JSUX-GSbeHATTJSb73rRDc Message-ID: From: Beeblebrox To: freebsd-fs@freebsd.org X-Gm-Message-State: ALoCoQkcrZkJlICeHQS4Mz6P53UWOtaDidFFwh31Jtkby2rDGlX62rYeSn7YvR3Fq1STceAWkzdU Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: jailed NFS server X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2012 20:20:38 -0000 Is it possible to get an NFS server working from inside a Jail, where host storage is on ZFS? I get errors from mountd and nfsd when started inside jail (exports file has no V4 line and simple one-line test export). mountd & nfsd errors are: mountd[2580]: Can't delete exports for V4: mountd[2580]: can't delete exports for /: Operation not permitted mountd[2580]: can't change attributes for /home mountd[2580]: bad exports list line /home -network 192.168.2.0/24 nfsd[2583]: Can't read stable storage file I have a modified host /etc/sysctl.conf as below, per post by PJD: http://www.mailinglistarchive.com/html/freebsd-current@freebsd.org/2007-07/msg01185.html Not that I really know whether these settings are valid, but at least I got rid of rpcbind errors. > > > security.jail.jailed: 1 > > > security.jail.mount_allowed: 1 > > > security.jail.chflags_allowed: 1 > > > security.jail.allow_raw_sockets: 0 > > > security.jail.enforce_statfs: 2 > > > security.jail.sysvipc_allowed: 1 > > > security.jail.socket_unixiproute_only: 1 > > > security.jail.set_hostname_allowed: 1 > > > ## security.jail.enforce_statfs=0 > > > vfs.nfsd.nfs_privport=1 > > > vfs.nfsd.server_max_nfsvers=4 Then I start NFS manually form inside jail to observe any faults (Jail IP is 192.168.2.1): #> service rpcbind onestart -h 192.168.2.1 #> service mountd onestart -r -n -p 59 -l -h 192.168.2.1 #> service nfsd onestart -u -t -n 4 -l -h 192.168.2.1 Thanks & Regards. From owner-freebsd-fs@FreeBSD.ORG Wed Mar 28 20:26:28 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A54E21065670 for ; Wed, 28 Mar 2012 20:26:28 +0000 (UTC) (envelope-from phillip.nordwall@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 5DA638FC17 for ; Wed, 28 Mar 2012 20:26:28 +0000 (UTC) Received: by obbuo13 with SMTP id uo13so2358565obb.13 for ; Wed, 28 Mar 2012 13:26:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=mMLg97R3tIZE1AuU7rCwOIT1kolEDmPB6rs4awsNpA8=; b=jOPRK8mkAqvKPjkSJs7cxsYYi5fXWKUrof47oldbNFIo3BXelgh9N5vhzsFyJdKnOe 0pkRfMvC0CYR0g9zWoMwNh6+7CfsIe3ZDC8+yI7lXqExXRottpJfVv7BPc9mF6yVDDGt dy+EddQdsu17wVhjSQjXRarJskpC5M6xUpW/0G0DVLMMMRQuncLo0BGfmPHYUE8DPSwz EiPAqzaj45dxyn+XGCPKXI8zCwJhF0nJ6hg2naTfRb3Dr6DngmfmKCD3BY+s+zh2vdFH aVWZ7dmfxF/BbcKEBYIXHJEXM2JCAsnJQvK8d/lkVpLetpFGbUcRDGJM92ZIA4J/VDtd 4wPw== MIME-Version: 1.0 Received: by 10.182.127.20 with SMTP id nc20mr39495574obb.66.1332966387482; Wed, 28 Mar 2012 13:26:27 -0700 (PDT) Sender: phillip.nordwall@gmail.com Received: by 10.182.43.225 with HTTP; Wed, 28 Mar 2012 13:26:27 -0700 (PDT) In-Reply-To: References: Date: Wed, 28 Mar 2012 13:26:27 -0700 X-Google-Sender-Auth: v-2hBzEEZNyMD8pWM3XeuWz77xY Message-ID: From: Phillip Nordwall To: Beeblebrox Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: jailed NFS server X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2012 20:26:28 -0000 Good luck, and if you have success let us know. We had 2 people with FreeBSD server experience spend 3 days each on getting this to work on FreeBSD 8.0 before giving up do to other pressing business. Phillip Nordwall On Wed, Mar 28, 2012 at 1:20 PM, Beeblebrox wrote: > Is it possible to get an NFS server working from inside a Jail, where host > storage is on ZFS? I get errors from mountd and nfsd when started inside > jail (exports file has no V4 line and simple one-line test export). mountd > & nfsd errors are: > mountd[2580]: Can't delete exports for V4: > mountd[2580]: can't delete exports for /: Operation not permitted > mountd[2580]: can't change attributes for /home > mountd[2580]: bad exports list line /home -network 192.168.2.0/24 > nfsd[2583 ]: Can't read stable storage > file > > I have a modified host /etc/sysctl.conf as below, per post by PJD: > > http://www.mailinglistarchive.com/html/freebsd-current@freebsd.org/2007-07/msg01185.html > Not that I really know whether these settings are valid, but at least I got > rid of rpcbind errors. > > > > security.jail.jailed: 1 > > > > security.jail.mount_allowed: 1 > > > > security.jail.chflags_allowed: 1 > > > > security.jail.allow_raw_sockets: 0 > > > > security.jail.enforce_statfs: 2 > > > > security.jail.sysvipc_allowed: 1 > > > > security.jail.socket_unixiproute_only: 1 > > > > security.jail.set_hostname_allowed: 1 > > > > ## security.jail.enforce_statfs=0 > > > > vfs.nfsd.nfs_privport=1 > > > > vfs.nfsd.server_max_nfsvers=4 > > Then I start NFS manually form inside jail to observe any faults (Jail IP > is 192.168.2.1): > #> service rpcbind onestart -h 192.168.2.1 > #> service mountd onestart -r -n -p 59 -l -h 192.168.2.1 > #> service nfsd onestart -u -t -n 4 -l -h 192.168.2.1 > > Thanks & Regards. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Wed Mar 28 20:30:03 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 56E161065670 for ; Wed, 28 Mar 2012 20:30:03 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: from mail.ultra-secure.de (mail.ultra-secure.de [78.47.114.122]) by mx1.freebsd.org (Postfix) with ESMTP id 8913E8FC12 for ; Wed, 28 Mar 2012 20:30:02 +0000 (UTC) Received: (qmail 69682 invoked by uid 89); 28 Mar 2012 20:29:11 -0000 Received: by simscan 1.4.0 ppid: 69677, pid: 69679, t: 0.0467s scanners: attach: 1.4.0 clamav: 0.97.3/m:54/d:14712 Received: from unknown (HELO ?192.168.1.201?) (rainer@ultra-secure.de@217.71.83.52) by mail.ultra-secure.de with ESMTPA; 28 Mar 2012 20:29:11 -0000 Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: text/plain; charset=iso-8859-1 From: Rainer Duffner In-Reply-To: Date: Wed, 28 Mar 2012 22:29:10 +0200 Content-Transfer-Encoding: 7bit Message-Id: <0685CC3A-753B-4C5B-9E15-C0565B48F885@ultra-secure.de> References: To: Beeblebrox X-Mailer: Apple Mail (2.1257) Cc: freebsd-fs@freebsd.org Subject: Re: jailed NFS server X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2012 20:30:03 -0000 Am 28.03.2012 um 22:20 schrieb Beeblebrox: > Is it possible to get an NFS server working from inside a Jail, where host > storage is on ZFS? Maybe try unfs3: http://www.freshports.org/net/unfs3/ Regards Rainer From owner-freebsd-fs@FreeBSD.ORG Wed Mar 28 21:36:55 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B8572106564A for ; Wed, 28 Mar 2012 21:36:55 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 6703E8FC16 for ; Wed, 28 Mar 2012 21:36:55 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap8EAPyCc0+DaFvO/2dsb2JhbABFhUC0NIIJAQEBAwEBAQEgKyALBRYYAgINGQIpAQkmBggHBAEZAwSHZAULqF2SG4EviUMFhQOBGASTNoIrgRGPHIMDgTgI X-IronPort-AV: E=Sophos;i="4.75,333,1330923600"; d="scan'208";a="162715640" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 28 Mar 2012 17:35:47 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 929B2B3F97; Wed, 28 Mar 2012 17:35:47 -0400 (EDT) Date: Wed, 28 Mar 2012 17:35:47 -0400 (EDT) From: Rick Macklem To: Sven Brandenburg Message-ID: <1023490904.1870197.1332970547582.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <4F709A18.50907@crashme.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE8 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org Subject: Re: NFSv3, ZFS, 10GE performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2012 21:36:55 -0000 Sven Brandenburg wrote: > On 03/26/2012 12:37 PM, Ivan Voras wrote: > > You could try modifying the rsize and wsize NFS options (read > > mount_nfs(8)), they help with UFS. > > I tried this a few days ago and fiddling rsize alters performance from > "ok" to "terrible". > However, you made me revisit this and mount_nfs(8) seems to have a gem > in its options: readahead. > This did the trick for me and my (long and sequential) reads. > While the manpage says its limited to 0-4, the best results were > achieved with readahead=8 : 1.1GB/s - which is what I had hoped for. > Yea, the new NFS client allows a readahead of up to 16. The man page for FreeBSD-9 should probably be changed, since the new NFS client is the default. Btw, without readahead, the client will do Read RPCs serially. (In other words, the next Read RPC won't start until the reply to the previous one has been received.) > On a tangent: gnu-dd 1GB/s is 10^9 Bytes/s, not 2^30. Yes, I fell for > it > at first :) > The good news is that there was no fiddling on the NFS server side. > (Apart from MTU increases, PCI settings and more buffers to get TCP > performance to full tilt in the first place) > > Hopefully, readahead doesn't kill performance for smaller files.. :-) > Well, readaheads only happen if the file is large enough for the readahead to be before EOF. As such, they just won't happen for small files. rick > regards, > Sven > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Wed Mar 28 21:58:08 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B197D106566B for ; Wed, 28 Mar 2012 21:58:08 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id 72E1C8FC1A for ; Wed, 28 Mar 2012 21:58:08 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id q2SLtM5N014328; Wed, 28 Mar 2012 16:55:22 -0500 (CDT) Date: Wed, 28 Mar 2012 16:55:22 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Rick Macklem In-Reply-To: <1023490904.1870197.1332970547582.JavaMail.root@erie.cs.uoguelph.ca> Message-ID: References: <1023490904.1870197.1332970547582.JavaMail.root@erie.cs.uoguelph.ca> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Wed, 28 Mar 2012 16:55:22 -0500 (CDT) Cc: freebsd-fs@freebsd.org Subject: Re: NFSv3, ZFS, 10GE performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2012 21:58:08 -0000 On Wed, 28 Mar 2012, Rick Macklem wrote: >> >> Hopefully, readahead doesn't kill performance for smaller files.. :-) >> > Well, readaheads only happen if the file is large enough for the readahead > to be before EOF. As such, they just won't happen for small files. The real problem is for applications which do a seek and a new read prior to consuming all the data which is being read ahead on its behalf. This results in read amplification and increased latency for the next seek+read. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 02:12:45 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 011A8106564A for ; Thu, 29 Mar 2012 02:12:45 +0000 (UTC) (envelope-from rsb@berentweb.com) Received: from mail-lpp01m010-f54.google.com (mail-lpp01m010-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 677388FC15 for ; Thu, 29 Mar 2012 02:12:44 +0000 (UTC) Received: by lagv3 with SMTP id v3so2798475lag.13 for ; Wed, 28 Mar 2012 19:12:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=berentweb.com; s=google; h=mime-version:sender:x-originating-ip:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=Byfg/B212/6XGhj56UJyidNYa3lix06M5sJHLhB3YoE=; b=TYGMVScTlehdoWEEXtKOiB89T6vHBg7hsIGQYmNABPnuoajsk9I/s1oftd+OsOfeg8 GXo/GgLtLBD9f2NKk2ZQ0zTDHIYp7j+Yyi5lMJvE51bWCyI60uVCofwdDVuXq3tTmmAI j4fhk9TpIR6gDsewOoObGcgwTBmQeRy8xB3is= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:x-originating-ip:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :x-gm-message-state; bh=Byfg/B212/6XGhj56UJyidNYa3lix06M5sJHLhB3YoE=; b=oMca8h9kLpM/zt2+uSEhJeLrgtNP8XgpFxmnzcL623AQ2uQ3xSeh40jS3IPHHARB/N WEwiGlfjDyok7Lw7UMirNfdkKXfDlb8hjWATZ4LND8HE3owgc/7sggJfX7UvJnTEYqsj MztKaGMNvAeVK2UMhI6Xiw326RmR8CO4MFka8SP3zJpyHqeUSDRPIAlgmp2lpitE4ivB zAA6yvRUm8cc4Mq9tsEwkRx8VOwsf92wMYZ8z7TqKKYfryHMPBPhTtHO6Wx3THjuDX22 TZXnobfjLO5meZiFjQsc0bWugG4kZCh40GS7WQMokZqOHrj0Wmjicsvh8/OZb/7e3yIN fGsw== MIME-Version: 1.0 Received: by 10.152.133.144 with SMTP id pc16mr25655501lab.0.1332987163118; Wed, 28 Mar 2012 19:12:43 -0700 (PDT) Sender: rsb@berentweb.com Received: by 10.112.77.15 with HTTP; Wed, 28 Mar 2012 19:12:43 -0700 (PDT) X-Originating-IP: [78.162.12.68] In-Reply-To: <0685CC3A-753B-4C5B-9E15-C0565B48F885@ultra-secure.de> References: <0685CC3A-753B-4C5B-9E15-C0565B48F885@ultra-secure.de> Date: Thu, 29 Mar 2012 05:12:43 +0300 X-Google-Sender-Auth: H66UaSJLYt9cy2qpQpP8JB7mMbw Message-ID: From: Beeblebrox To: freebsd-fs@freebsd.org X-Gm-Message-State: ALoCoQkJC6/W6ApChrSf6ya8SedLLE/pGd9rwQwHYYZwMMUilKt6SZd0mN28DMjR4LGQ2KIlOvII Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: jailed NFS server X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 02:12:45 -0000 Maybe I will give unfs3 a try. However, One of the reasons I'm trying to set it up is to be able to run Tinderbox on that jail for distributed compiling. When I did a little searching about unfs3 + Tinderbox + jail, it came up with posts about problems and that such setup "does not give good results". Any feedback about such setup? Also, is such "bad performance result" also valid for all types of HPC/parallel computing run from jails so that "Just Run From Host" becomes the defaul solution in these cases? Thanks and Regards. On Wed, Mar 28, 2012 at 11:29 PM, Rainer Duffner wrote: > > Am 28.03.2012 um 22:20 schrieb Beeblebrox: > > > Is it possible to get an NFS server working from inside a Jail, where > host > > storage is on ZFS? > > > Maybe try unfs3: > > http://www.freshports.org/net/unfs3/ > > > > Regards > Rainer > From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 08:13:51 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B45B4106564A; Thu, 29 Mar 2012 08:13:51 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 515598FC12; Thu, 29 Mar 2012 08:13:51 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (89.112.15.178.pppoe.eltel.net [89.112.15.178]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 76ADA4AC32; Thu, 29 Mar 2012 12:13:49 +0400 (MSK) Date: Thu, 29 Mar 2012 12:13:44 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <1884361371.20120329121344@serebryakov.spb.ru> To: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: Subject: What is actual status of SUJ in 9-STABLE? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 08:13:51 -0000 Hello, Freebsd-fs. My server crashed today morning due to PSU failure, and now it is checking (in foreground!) 8Tb UFS2+SU volume for 6200 seconds, and it is only "Phase 1b" (!!!). I don't want even think about background check of this FS. Is SUJ stable enough to migrate to it? It was marked as stable some time ago, and was included into 9-RELEASE, but later I seen some messages on fs@ list, that it still has some problems, and even some references to McKusick's message about this instability (but I've failed to find message itself). BTW, this check reveals many softupdate inconsistences (mostly DUPs), and= most of them are in files, which was not written for sure in time of crash (old archives, which could be only read!). --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 09:09:04 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 321CF1065687 for ; Thu, 29 Mar 2012 09:09:04 +0000 (UTC) (envelope-from mailer@mail.firmos.at) Received: from mail.firmos.at (mail.firmos.at [80.123.225.49]) by mx1.freebsd.org (Postfix) with ESMTP id 392C88FC20 for ; Thu, 29 Mar 2012 09:09:02 +0000 (UTC) Received: from [91.114.28.42] (helo=BigMac.local) by mail.firmos.at with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.76 (FreeBSD)) (envelope-from ) id 1SDB2W-000B2P-6P for freebsd-fs@freebsd.org; Thu, 29 Mar 2012 10:49:04 +0200 Message-ID: <4F74220E.9070707@firmos.at> Date: Thu, 29 Mar 2012 10:49:18 +0200 From: Franz Schober Organization: FirmOS Business Solutions Gmbh User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Sender: mailer@mail.firmos.at X-SA-Exim-Connect-IP: 91.114.28.42 X-SA-Exim-Mail-From: mailer@mail.firmos.at X-SA-Exim-Scanned: No (on mail.firmos.at); SAEximRunCond expanded to false Subject: ZFS Performance FreeBSD 9.0 vs. Openindiana X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 09:09:04 -0000 Hi, I am testing a new storage hardware (Xeon E5606, 24 GB Ram, LSI1068 controller in non-raid mode, 8 7200 SATA Disks, 2 OCZ SSD ZIL Mirror), for a NFS/iSCSI storage system. Currently I'm doing basic performance tests of the ZFS filesystem. We are using FreeBSD for a couple of years for various services in our software development company, so FreeBSD is our first choice. For performance tests, I use the iozone benchmark with a multi-streaming concurrency test, iozone -o -c -t 8 -r 128k -s 4G (Sync Mode, 8 concurrent workers, 128 k Recordsize, 4G working file for every worker to run not only in cache). ZPool ist a stripe of the 8 SATA Disks and a mirror of the two SSD Log disks (here the FreeBSD zpool status): pool: pools state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM pools ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 logs mirror-8 ONLINE 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 Comparing FreeBSD 9.0-RELEASE (amd64) to OpenIndiana 151a, I get a performance difference, where FreeBSD is slower in most tests, the pools is freshly created before each test: FreeBSD: Iozone: Performance Test of File I/O Version $Revision: 3.397 $ Compiled for 64 bit mode. Build: freebsd Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer. Ben England. Run began: Thu Mar 29 10:39:47 2012 SYNC Mode. Include close in write timing Record Size 128 KB File size set to 4194304 KB Command line used: iozone -o -c -t 8 -r 128k -s 4G Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. Throughput test with 8 processes Each process writes a 4194304 Kbyte file in 128 Kbyte records Children see throughput for 8 initial writers = 150780.28 KB/sec Parent sees throughput for 8 initial writers = 150767.08 KB/sec Min throughput per process = 18846.89 KB/sec Max throughput per process = 18849.44 KB/sec Avg throughput per process = 18847.53 KB/sec Min xfer = 4193792.00 KB Children see throughput for 8 rewriters = 150051.28 KB/sec Parent sees throughput for 8 rewriters = 150048.26 KB/sec Min throughput per process = 18615.49 KB/sec Max throughput per process = 18837.79 KB/sec Avg throughput per process = 18756.41 KB/sec Min xfer = 4144896.00 KB Children see throughput for 8 readers = 1001472.78 KB/sec Parent sees throughput for 8 readers = 1001417.37 KB/sec Min throughput per process = 118845.12 KB/sec Max throughput per process = 138027.58 KB/sec Avg throughput per process = 125184.10 KB/sec Min xfer = 3611520.00 KB Children see throughput for 8 re-readers = 876152.78 KB/sec Parent sees throughput for 8 re-readers = 870610.71 KB/sec Min throughput per process = 106323.10 KB/sec Max throughput per process = 114136.91 KB/sec Avg throughput per process = 109519.10 KB/sec Min xfer = 3909120.00 KB Children see throughput for 8 reverse readers = 1148130.64 KB/sec Parent sees throughput for 8 reverse readers = 1142916.22 KB/sec Min throughput per process = 126814.09 KB/sec Max throughput per process = 172099.73 KB/sec Avg throughput per process = 143516.33 KB/sec Min xfer = 3117440.00 KB Children see throughput for 8 stride readers = 299141.80 KB/sec Parent sees throughput for 8 stride readers = 298399.38 KB/sec Min throughput per process = 29238.70 KB/sec Max throughput per process = 45995.27 KB/sec Avg throughput per process = 37392.73 KB/sec Min xfer = 2667264.00 KB Children see throughput for 8 random readers = 115979.46 KB/sec Parent sees throughput for 8 random readers = 115974.77 KB/sec Min throughput per process = 14312.92 KB/sec Max throughput per process = 14753.71 KB/sec Avg throughput per process = 14497.43 KB/sec Min xfer = 4068992.00 KB Children see throughput for 8 mixed workload = 188689.19 KB/sec Parent sees throughput for 8 mixed workload = 188673.96 KB/sec Min throughput per process = 12175.37 KB/sec Max throughput per process = 34954.12 KB/sec Avg throughput per process = 23586.15 KB/sec Min xfer = 1460992.00 KB Children see throughput for 8 random writers = 147947.42 KB/sec Parent sees throughput for 8 random writers = 147939.94 KB/sec Min throughput per process = 18492.69 KB/sec Max throughput per process = 18494.70 KB/sec Avg throughput per process = 18493.43 KB/sec Min xfer = 4193920.00 KB Children see throughput for 8 pwrite writers = 149780.00 KB/sec Parent sees throughput for 8 pwrite writers = 149772.91 KB/sec Min throughput per process = 18721.60 KB/sec Max throughput per process = 18723.81 KB/sec Avg throughput per process = 18722.50 KB/sec Min xfer = 4193792.00 KB Children see throughput for 8 pread readers = 1014824.33 KB/sec Parent sees throughput for 8 pread readers = 1014743.46 KB/sec Min throughput per process = 114311.04 KB/sec Max throughput per process = 150336.52 KB/sec Avg throughput per process = 126853.04 KB/sec Min xfer = 3189376.00 KB iozone test complete. OpenIndiana: Iozone: Performance Test of File I/O Version $Revision: 3.323 $ Compiled for 32 bit mode. Build: Solaris10cc-64 Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root. Run began: Fri Mar 16 14:11:48 2012 SYNC Mode. Include close in write timing Record Size 128 KB File size set to 4194304 KB Command line used: /usr/benchmarks/iozone/iozone -o -c -t 8 -r 128k -s 4G Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. Throughput test with 8 processes Each process writes a 4194304 Kbyte file in 128 Kbyte records Children see throughput for 8 initial writers = 163315.97 KB/sec Parent sees throughput for 8 initial writers = 163238.81 KB/sec Min throughput per process = 20399.37 KB/sec Max throughput per process = 20423.74 KB/sec Avg throughput per process = 20414.50 KB/sec Min xfer = 4189440.00 KB Children see throughput for 8 rewriters = 163415.44 KB/sec Parent sees throughput for 8 rewriters = 163298.96 KB/sec Min throughput per process = 20266.25 KB/sec Max throughput per process = 20611.95 KB/sec Avg throughput per process = 20426.93 KB/sec Min xfer = 4123904.00 KB Children see throughput for 8 readers = 1180533.94 KB/sec Parent sees throughput for 8 readers = 1180090.09 KB/sec Min throughput per process = 121104.02 KB/sec Max throughput per process = 198012.62 KB/sec Avg throughput per process = 147566.74 KB/sec Min xfer = 2565376.00 KB Children see throughput for 8 re-readers = 951965.89 KB/sec Parent sees throughput for 8 re-readers = 948751.85 KB/sec Min throughput per process = 118049.84 KB/sec Max throughput per process = 120618.28 KB/sec Avg throughput per process = 118995.74 KB/sec Min xfer = 4113536.00 KB Children see throughput for 8 reverse readers = 3024070.31 KB/sec Parent sees throughput for 8 reverse readers = 2867010.77 KB/sec Min throughput per process = 318376.59 KB/sec Max throughput per process = 557625.75 KB/sec Avg throughput per process = 378008.79 KB/sec Min xfer = 2563968.00 KB Children see throughput for 8 stride readers = 335398.05 KB/sec Parent sees throughput for 8 stride readers = 331540.81 KB/sec Min throughput per process = 32520.95 KB/sec Max throughput per process = 62529.34 KB/sec Avg throughput per process = 41924.76 KB/sec Min xfer = 2210048.00 KB Children see throughput for 8 random readers = 120422.72 KB/sec Parent sees throughput for 8 random readers = 120418.39 KB/sec Min throughput per process = 14456.03 KB/sec Max throughput per process = 15568.85 KB/sec Avg throughput per process = 15052.84 KB/sec Min xfer = 3894528.00 KB Children see throughput for 8 mixed workload = 191391.66 KB/sec Parent sees throughput for 8 mixed workload = 191249.48 KB/sec Min throughput per process = 12364.50 KB/sec Max throughput per process = 35442.05 KB/sec Avg throughput per process = 23923.96 KB/sec Min xfer = 1463296.00 KB Children see throughput for 8 random writers = 163556.06 KB/sec Parent sees throughput for 8 random writers = 163218.36 KB/sec Min throughput per process = 20411.16 KB/sec Max throughput per process = 20484.72 KB/sec Avg throughput per process = 20444.51 KB/sec Min xfer = 4179328.00 KB iozone test complete. Now my question: Which parameters in the ZFS Subsystem of FreeBSD are tuneable to reach the same performance in FreeBSD, especially pushing the synchronous write performance ? Thx a lot, Franz Schober From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 09:14:20 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 31B5B106566C for ; Thu, 29 Mar 2012 09:14:20 +0000 (UTC) (envelope-from gleb.kurtsou@gmail.com) Received: from mail-bk0-f54.google.com (mail-bk0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id AF8DE8FC17 for ; Thu, 29 Mar 2012 09:14:19 +0000 (UTC) Received: by bkcjc3 with SMTP id jc3so2282116bkc.13 for ; Thu, 29 Mar 2012 02:14:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=0EwvKQens6xOxttbRIo2p4YzNPcRAOsUArdNNZjgaNY=; b=sphCwdCnj16WCN6A/An764x7tbSxZx+wnv3CBJ1TAjPwOZcM6y7jC5wVmnj1scI7HX MVJkqSTrw5Zfu/jodjQbN7MXn8K6j45SYeMf8BgTTiRJd9/EzBQQJYPKSN7ujbMyUufX L7HPL5kVxz2p4WdX6OHTMPrMLI8dxkfq0NNDS/PeaZYl3s1dhnq7FNQagDtnaO4JH5Jr OqmIHUg5tskCVXW0zzpCAQKcHdX8pSbSHX23azfemIQWuogyVkGtvwUW0SiXG7Kngd+d D5ONaSyPMNekBbq0G7+8ZYaH1WI1lYhmdpqfBlGYaZinhCPV9aYhFHl0cFu00bBeFjJt Tqow== Received: by 10.204.154.83 with SMTP id n19mr6251024bkw.69.1333012458458; Thu, 29 Mar 2012 02:14:18 -0700 (PDT) Received: from localhost ([78.157.92.5]) by mx.google.com with ESMTPS id u5sm12189593bka.5.2012.03.29.02.14.16 (version=SSLv3 cipher=OTHER); Thu, 29 Mar 2012 02:14:17 -0700 (PDT) Date: Thu, 29 Mar 2012 12:14:15 +0300 From: Gleb Kurtsou To: Konstantin Belousov Message-ID: <20120329091415.GA1316@reks> References: <20120327183440.GS2358@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20120327183440.GS2358@deviant.kiev.zoral.com.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: fs@freebsd.org Subject: Re: RFC: SEEK_HOLE/SEEK_DATA common implementation X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 09:14:20 -0000 On (27/03/2012 21:34), Konstantin Belousov wrote: > Hello, > Please find at > http://people.freebsd.org/~kib/misc/seek_hole.1.patch > a prototype generic implementation of the SEEK_HOLE/SEEK_DATA lseek(2) > whence command for any filesystem that properly supports bmap. > > I was unable to find any test suite for the feature, and the only more > or less reasonable case I found was from lklm, extracted at > http://people.freebsd.org/~kib/misc/seek_hole.c > The block for file with hole at the end is commented out because > UFS never puts holes at EOF. The block_size is tuned for default block > size on UFS on recent FreeBSD. > > Filesystem-specific implementations could greatly increase the performance > of the call, because the presented implementation does linear search > through the blocks until hole/data boundary is found. E.g., for UFS, > the fast tree-like structure of indirect blocks can be traversed to > point at the next boundary. But this implementation is generic. What do you think about replacing FIOSEEKHOLE/FIOSEEKDATA ioctls with VOP_SEEKHOLE (or similar). That would be more straightforward. Currently there is no way for a file system to use standard SEEKHOLE implementation without invoking vop_stdioctl. It handles only FIOSEEKHOLE now but that is likely to change in future. I wish we had no default VOP_IOCTL implementation and keep it file system specific. We should also return valid _PC_MIN_HOLE_SIZE (=mnt_stat.f_iosize) in VOP_PATHCONF for userland to see SEEK_HOLE/SEEK_DATA support. I'm not sure enabling it by default is desirable, performance penalty can be considerable. Consider creating tar archive of large files with O(file size) SEEK_HOLE enabled. On the other hand default implementation can be useful, e.g. if file system had internal flag for sparse files it could simply reuse default vop_stdseekhole() avoiding overhead for non-sparse files. Thanks, Gleb. > > Please comment. From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 09:34:37 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 03A0D106566B for ; Thu, 29 Mar 2012 09:34:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 6D23B8FC16 for ; Thu, 29 Mar 2012 09:34:35 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q2T9YRTC079746; Thu, 29 Mar 2012 12:34:27 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q2T9YRrn094569; Thu, 29 Mar 2012 12:34:27 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q2T9YRp6094568; Thu, 29 Mar 2012 12:34:27 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 29 Mar 2012 12:34:27 +0300 From: Konstantin Belousov To: Gleb Kurtsou Message-ID: <20120329093427.GA2358@deviant.kiev.zoral.com.ua> References: <20120327183440.GS2358@deviant.kiev.zoral.com.ua> <20120329091415.GA1316@reks> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="KKlV3Q509gERURet" Content-Disposition: inline In-Reply-To: <20120329091415.GA1316@reks> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: fs@freebsd.org Subject: Re: RFC: SEEK_HOLE/SEEK_DATA common implementation X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 09:34:37 -0000 --KKlV3Q509gERURet Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Mar 29, 2012 at 12:14:15PM +0300, Gleb Kurtsou wrote: > On (27/03/2012 21:34), Konstantin Belousov wrote: > > Hello, > > Please find at > > http://people.freebsd.org/~kib/misc/seek_hole.1.patch > > a prototype generic implementation of the SEEK_HOLE/SEEK_DATA lseek(2) > > whence command for any filesystem that properly supports bmap. > >=20 > > I was unable to find any test suite for the feature, and the only more > > or less reasonable case I found was from lklm, extracted at > > http://people.freebsd.org/~kib/misc/seek_hole.c > > The block for file with hole at the end is commented out because > > UFS never puts holes at EOF. The block_size is tuned for default block > > size on UFS on recent FreeBSD. > >=20 > > Filesystem-specific implementations could greatly increase the performa= nce > > of the call, because the presented implementation does linear search > > through the blocks until hole/data boundary is found. E.g., for UFS, > > the fast tree-like structure of indirect blocks can be traversed to > > point at the next boundary. But this implementation is generic. >=20 > What do you think about replacing FIOSEEKHOLE/FIOSEEKDATA ioctls with > VOP_SEEKHOLE (or similar). That would be more straightforward. Currently > there is no way for a file system to use standard SEEKHOLE > implementation without invoking vop_stdioctl. It handles only > FIOSEEKHOLE now but that is likely to change in future. I wish we had no > default VOP_IOCTL implementation and keep it file system specific. This should be driven by ZFS guys. If current structure makes the imports from Illumos easier, then I prefer to not introduce new VOP. Otherwise, feel free to change. Filesystem probably should always fall back to calling vop_stdioctl() manually if it ever implements VOP_IOCTL, regardless of seek_hole/data. >=20 > We should also return valid _PC_MIN_HOLE_SIZE (=3Dmnt_stat.f_iosize) in > VOP_PATHCONF for userland to see SEEK_HOLE/SEEK_DATA support. Yes, but there is more inconsitencies between manpage and ZFS implementatio= n, at least it seems so from my reading. Currently I am more interested in the comments about the approach itself. >=20 > I'm not sure enabling it by default is desirable, performance penalty > can be considerable. Consider creating tar archive of large files with > O(file size) SEEK_HOLE enabled. On the other hand default implementation > can be useful, e.g. if file system had internal flag for sparse files it > could simply reuse default vop_stdseekhole() avoiding overhead for > non-sparse files. I do not understand this. Can you provide some numbers illustrating your concerns ? And yes, I would be happy to get some testing results from applications that already use the interface, but this is a blue dream. There are too many corner cases in the interface that seems to be not documented and I was unable to find test giving good coverage. --KKlV3Q509gERURet Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk90LKMACgkQC3+MBN1Mb4jFPACfXjnd9s3pk6nr/fDeDszTt7B/ SakAoM+v6PqHuykiRqoRjoOsB1u+q30N =foX6 -----END PGP SIGNATURE----- --KKlV3Q509gERURet-- From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 09:59:34 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7A967106564A for ; Thu, 29 Mar 2012 09:59:34 +0000 (UTC) (envelope-from tdb@carrick.bishnet.net) Received: from carrick.bishnet.net (carrick.bishnet.net [IPv6:2a01:348:132:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 3481A8FC18 for ; Thu, 29 Mar 2012 09:59:34 +0000 (UTC) Received: from carrick-users.bishnet.net ([2a01:348:132:51::10]) by carrick.bishnet.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.76 (FreeBSD)) (envelope-from ) id 1SDC8k-000Pv6-Eb; Thu, 29 Mar 2012 10:59:34 +0100 Received: (from tdb@localhost) by carrick-users.bishnet.net (8.14.4/8.14.4/Submit) id q2T9xYCh099639; Thu, 29 Mar 2012 10:59:34 +0100 (BST) (envelope-from tdb) Date: Thu, 29 Mar 2012 10:59:34 +0100 From: Tim Bishop To: Johan Hendriks Message-ID: <20120329095934.GJ24787@carrick-users.bishnet.net> References: <20120327181457.GC24787@carrick-users.bishnet.net> <4F72ECFC.4040902@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4F72ECFC.4040902@gmail.com> X-PGP-Key: 0x5AE7D984, http://www.bishnet.net/tim/tim-bishnet-net.asc X-PGP-Fingerprint: 1453 086E 9376 1A50 ECF6 AE05 7DCE D659 5AE7 D984 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS: processes hanging when trying to access filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 09:59:34 -0000 On Wed, Mar 28, 2012 at 12:50:36PM +0200, Johan Hendriks wrote: > At the topic starter, how do you get the graphs for the arc data? I'm using munin. Master is on another machine so it doesn't get directly affected by the zfs problems. Munin has a plugin that collects zfs stats using the zfs-stats script. I find it useful to correlate the behaviour I'm seeing with the status of the ARC and other system bits. Tim -- Tim Bishop http://www.bishnet.net/tim/ PGP Key: 0x5AE7D984 From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 10:51:19 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 10D5B106566C for ; Thu, 29 Mar 2012 10:51:19 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9CFF48FC12 for ; Thu, 29 Mar 2012 10:51:18 +0000 (UTC) Received: by eekd17 with SMTP id d17so1156143eek.13 for ; Thu, 29 Mar 2012 03:51:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=u9pS8K7TsivqzGkWMQG6/T2JLbvUoM//K857LOsF6es=; b=WrdsLFuGpcPIAM8iBr+Q9bupr8DvK0ht/p25TEEonRiBrNHLV9dR4huweIwjebfeVX 8noBCIVftmq+BHMEpztLYmj9twUrgKPfawzhoVMh6qyLEF87FVqxenZZStLHo/Jg4uKR khwqjS+jh0vmFNCwgQaljb8r2ybxcRZqiS+yMMiJVmIbhS4sQWQsHCATbhrpbyZ8VPAe N/JHef1VA154DpcYH7V5tGgonJll+iq+yzbnrMrRsu47srVAy+x7PYZojAs+mHBpxuBD 8z7Hmx2TFAh8yKf2zpru5y6VzAUHr9FB1T5fk6GwDAt7mELUeafCSLkmJii7ReI1mk4s Gl5g== MIME-Version: 1.0 Received: by 10.180.24.7 with SMTP id q7mr4440675wif.11.1333018277456; Thu, 29 Mar 2012 03:51:17 -0700 (PDT) Received: by 10.180.102.67 with HTTP; Thu, 29 Mar 2012 03:51:17 -0700 (PDT) Date: Thu, 29 Mar 2012 06:51:17 -0400 Message-ID: From: grarpamp To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 Subject: Re; ZFS Performance FreeBSD 9.0 vs. Openindiana X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 10:51:19 -0000 Knowing of no better... I find Iozone quite nice. However... I would first resolve these differences in the test methodology before trying to compare anything 74 revisions apart... and perhaps the address width as well. Version $Revision: 3.323 $ Compiled for 32 bit mode. Build: Solaris10cc-64 File size set to 4194304 KB Version $Revision: 3.397 $ Compiled for 64 bit mode. Build: freebsd File size set to 4194304 KB Then try to see where the average 45% speedup is occuring. > to run not only in cache > Processor cache size set to 1024 Kbytes. 128k could very well fit in a cache somewhere. Such as say, the one above. As shown on the iozone site, I'd try graphing the full range of record sizes as a first baseline comparison. And poke around the sysctls to see how the zfs parameters match up. > -c Include close() in the timing calculations. This is useful only > if you suspect that close() is broken in the operating system > currently under test. Really, you do? From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 13:56:32 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ACC0A1065672 for ; Thu, 29 Mar 2012 13:56:32 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 65B1B8FC1D for ; Thu, 29 Mar 2012 13:56:32 +0000 (UTC) Received: by ggnk4 with SMTP id k4so1767915ggn.13 for ; Thu, 29 Mar 2012 06:56:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=HlkLGfabG/IXrk7NDbzx23/WZ8edGB5EE7BxpeTo7BE=; b=wi3aICTiwL5MgHueKZGOh5LUtIZt2F37++UKBaFJlt78M46je0Gexl76ulfS7C2LnL kpysk/XCU+XRqbbuw+79LSRzcm+gQmnKWjJ8VRW2uMYmUb9FDTc2EA2KRujcCHPjX+kM T2CgM3ugS6V5sb0cdbQA03KrTMEyDoKdBMjF3kR37oRN8lwjLa6vXrQVat02O1tD5xDK X/ORYnxRZSUfGK8A7iviYFroHTjzGZGkXTU1mkknb4ybUCQQApQqmGRNCS2lYhP4Ryes /Cq9udRMQliC7QpsB8qU5FBpsz2YJFLQsBCj9AhSSSFuw9Li3dgsftxzQajjJyJrycpU g89w== MIME-Version: 1.0 Received: by 10.68.189.170 with SMTP id gj10mr214586pbc.121.1333029391072; Thu, 29 Mar 2012 06:56:31 -0700 (PDT) Sender: mdf356@gmail.com Received: by 10.68.52.40 with HTTP; Thu, 29 Mar 2012 06:56:31 -0700 (PDT) In-Reply-To: <20120329093427.GA2358@deviant.kiev.zoral.com.ua> References: <20120327183440.GS2358@deviant.kiev.zoral.com.ua> <20120329091415.GA1316@reks> <20120329093427.GA2358@deviant.kiev.zoral.com.ua> Date: Thu, 29 Mar 2012 06:56:31 -0700 X-Google-Sender-Auth: -U-DiqKNhUsGkFoA0v5jizPRXOI Message-ID: From: mdf@FreeBSD.org To: Konstantin Belousov Content-Type: text/plain; charset=ISO-8859-1 Cc: fs@freebsd.org Subject: Re: RFC: SEEK_HOLE/SEEK_DATA common implementation X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 13:56:32 -0000 On Thu, Mar 29, 2012 at 2:34 AM, Konstantin Belousov wrote: > Filesystem probably should always fall back to calling vop_stdioctl() > manually if it ever implements VOP_IOCTL, regardless of seek_hole/data. We can't if there's a use of VOP_BMAP() in one of the ioctls. For OneFS, a call to VOP_BMAP is fatal. That operation doesn't work with our system. We could implement SEEK_HOLE/SEEK_DATA semi-efficiently, but implementing a custom filesystem becomes very complex once one has not only to manage replacing all VOPs where the default doesn't work but also has to know there's a bunch of ioctls in VOP_IOCTL that must be handled as well. Thanks, matthew From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 14:38:46 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2174D106564A for ; Thu, 29 Mar 2012 14:38:46 +0000 (UTC) (envelope-from sven@crashme.org) Received: from celaeno.tauri.mw.lg.virgo.supercluster.net (celaeno.tauri.mw.lg.virgo.supercluster.net [213.252.140.33]) by mx1.freebsd.org (Postfix) with ESMTP id CE17F8FC15 for ; Thu, 29 Mar 2012 14:38:45 +0000 (UTC) Received: from miram.persei.mw.lg.virgo.supercluster.net ([213.252.140.37] helo=[192.168.20.6]) by celaeno.tauri.mw.lg.virgo.supercluster.net with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.76) (envelope-from ) id 1SDG4N-0000kJ-GQ; Thu, 29 Mar 2012 14:11:19 +0000 Message-ID: <4F746D8C.8010903@crashme.org> Date: Thu, 29 Mar 2012 16:11:24 +0200 From: Sven Brandenburg User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:11.0) Gecko/20120310 Thunderbird/11.0 MIME-Version: 1.0 To: Rick Macklem References: <40825357.1703430.1332798477468.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <40825357.1703430.1332798477468.JavaMail.root@erie.cs.uoguelph.ca> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) Cc: freebsd-fs@freebsd.org Subject: Re: NFSv3, ZFS, 10GE performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 14:38:46 -0000 On 03/26/2012 11:47 PM, Rick Macklem wrote: > MAX_BSIZE is 64kb. I'd like to try making that bigger, but haven't gotten > around to it yet. (If you wanted to try bumping MAX_BSIZE to 128Kb on both > client and server and seeing what happens, that might be interesting, since > my understanding is that ZFS uses a 128Kb block size.) I finally came around and tested it (with 256k and 1M) - there is good and bad news. The good news is the system does indeed boot (off of zfs at least, no idea on ufs) and it does increase performance. I am now seeing roughly 800MB/s off the bat which is quite nice. The bad news is that I had to use a Linux client because the FreeBSD client declined to work: mount_nfs: /mnt, : No buffer space available (Although I will freely admit that my knowledge of where to ajdust this value is rather limited: What I did was changing MAXBSIZE MAXPHYS to 1M in /usr/src/sys/sys/param.h, remaking world+kernel then reboot. I forgot MAXPHYS in my first try and this crashed the client machine as soon as I tried to mount something via nfs. Notably, the server seems to be working ok even with a mismatched MAXPHYS/MAXBSIZE). So far, the results are very promising. regards, Sven From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 14:46:28 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EB61A106566C for ; Thu, 29 Mar 2012 14:46:28 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id AD4A48FC14 for ; Thu, 29 Mar 2012 14:46:28 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id q2TEkGj5019846; Thu, 29 Mar 2012 09:46:17 -0500 (CDT) Date: Thu, 29 Mar 2012 09:46:16 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Franz Schober In-Reply-To: <4F74220E.9070707@firmos.at> Message-ID: References: <4F74220E.9070707@firmos.at> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Thu, 29 Mar 2012 09:46:17 -0500 (CDT) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS Performance FreeBSD 9.0 vs. Openindiana X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 14:46:29 -0000 On Thu, 29 Mar 2012, Franz Schober wrote: > > For performance tests, I use the iozone benchmark with a multi-streaming > concurrency test, > iozone -o -c -t 8 -r 128k -s 4G (Sync Mode, 8 concurrent workers, 128 k > Recordsize, 4G working file for every worker to run not only in cache). The 128k record size is special. I think that this is the transition point where zfs writes sync data directly to the pool disks rather than to the log disks. I don't know which way it goes (pool/log) at exactly 128k. If your log disks are idle during the benchmark, then the answer to this question would be clear. > Now my question: Which parameters in the ZFS Subsystem of FreeBSD are > tuneable to reach the same performance in FreeBSD, > especially pushing the synchronous write performance ? I do not see much difference with the writes. I do see a large difference with the reads. This could easily be due to a difference in how Solaris and FreeBSD manage kernel memory. In Solaris, substantial caching may still be taking place even though you tried to avoid it. The ARC size might be too small (by default) under FreeBSD to offer similar caching. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 17:35:03 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 68D9B106566B; Thu, 29 Mar 2012 17:35:03 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (unknown [IPv6:2001:5a8:4:7e72:4a5b:39ff:fe12:452]) by mx1.freebsd.org (Postfix) with ESMTP id 1FFA38FC16; Thu, 29 Mar 2012 17:35:03 +0000 (UTC) Received: from chez.mckusick.com (localhost [127.0.0.1]) by chez.mckusick.com (8.14.3/8.14.3) with ESMTP id q2THYwYi026461; Thu, 29 Mar 2012 10:34:58 -0700 (PDT) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201203291734.q2THYwYi026461@chez.mckusick.com> To: lev@freebsd.org In-reply-to: <1884361371.20120329121344@serebryakov.spb.ru> Date: Thu, 29 Mar 2012 10:34:58 -0700 From: Kirk McKusick X-Spam-Status: No, score=0.0 required=5.0 tests=MISSING_MID, UNPARSEABLE_RELAY autolearn=failed version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on chez.mckusick.com Cc: freebsd-fs@freebsd.org Subject: Re: What is actual status of SUJ in 9-STABLE? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 17:35:03 -0000 > Date: Thu, 29 Mar 2012 12:13:44 +0400 > From: Lev Serebryakov > To: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org > Subject: What is actual status of SUJ in 9-STABLE? > > Hello, Freebsd-fs. > > My server crashed today morning due to PSU failure, and now it is > checking (in foreground!) 8Tb UFS2+SU volume for 6200 seconds, and it > is only "Phase 1b" (!!!). I don't want even think about background > check of this FS. > > Is SUJ stable enough to migrate to it? It was marked as stable some > time ago, and was included into 9-RELEASE, but later I seen some > messages on fs@ list, that it still has some problems, and even some > references to McKusick's message about this instability (but I've failed > to find message itself). Most of the issues with SUJ are related to their interaction with snapshots. At the moment 9-head (head of the 9 branch) has had the taking of snapshots disabled on filesystems running with SUJ (but not with SU). There have been some important bug fixes to SUJ since 9-release. So if you wish to use SUJ, I recommend using 9-head rather than 9-release. > BTW, this check reveals many softupdate inconsistences (mostly DUPs), > and most of them are in files, which was not written for sure in time > of crash (old archives, which could be only read!). > > -- > // Black Lion AKA Lev Serebryakov It sounds like you have a disk sector containing a block of inodes trashed or gone bad. Kirk McKusick From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 18:28:53 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C99B1065678; Thu, 29 Mar 2012 18:28:53 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id A78D08FC14; Thu, 29 Mar 2012 18:28:52 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q2TISl83025398; Thu, 29 Mar 2012 21:28:47 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q2TISlE4007768; Thu, 29 Mar 2012 21:28:47 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q2TISl5h007767; Thu, 29 Mar 2012 21:28:47 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 29 Mar 2012 21:28:47 +0300 From: Konstantin Belousov To: mdf@freebsd.org Message-ID: <20120329182847.GD2358@deviant.kiev.zoral.com.ua> References: <20120327183440.GS2358@deviant.kiev.zoral.com.ua> <20120329091415.GA1316@reks> <20120329093427.GA2358@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="J/iAXrWP4xH+aVs6" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: fs@freebsd.org Subject: Re: RFC: SEEK_HOLE/SEEK_DATA common implementation X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 18:28:53 -0000 --J/iAXrWP4xH+aVs6 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Mar 29, 2012 at 06:56:31AM -0700, mdf@freebsd.org wrote: > On Thu, Mar 29, 2012 at 2:34 AM, Konstantin Belousov > wrote: > > Filesystem probably should always fall back to calling vop_stdioctl() > > manually if it ever implements VOP_IOCTL, regardless of seek_hole/data. >=20 > We can't if there's a use of VOP_BMAP() in one of the ioctls. For > OneFS, a call to VOP_BMAP is fatal. That operation doesn't work with > our system. But you had to already take some measure to disable the call to VOP_BMAP() for your filesystem, otherwise mmap() is fatal too (whatever the word fatal means) ? >=20 > We could implement SEEK_HOLE/SEEK_DATA semi-efficiently, but > implementing a custom filesystem becomes very complex once one has not > only to manage replacing all VOPs where the default doesn't work but > also has to know there's a bunch of ioctls in VOP_IOCTL that must be > handled as well. Does anything change if VOP_SEEKDATA() is added which default implementation using VOP_BMAP() ? --J/iAXrWP4xH+aVs6 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk90qd8ACgkQC3+MBN1Mb4iW0gCfRL7N7wSLpIWEQYZqr1c5nhFH V2gAoMPffMz7oLWi6Ihnxj/f9C5IZf5u =Djew -----END PGP SIGNATURE----- --J/iAXrWP4xH+aVs6-- From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 18:34:35 2012 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 760AB106564A for ; Thu, 29 Mar 2012 18:34:35 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 451918FC15 for ; Thu, 29 Mar 2012 18:34:35 +0000 (UTC) Received: by pbcwz17 with SMTP id wz17so554440pbc.13 for ; Thu, 29 Mar 2012 11:34:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=zNR6fuRgu4D/BKOV+2NR7aUOWPQAjV8Xk7IKFEdR8J8=; b=FOZDIqO6f47F01pR9JFrzF+BA45mxyAOkyOfXOd690Cg8qUZhq0scgipVcVYPTHlN1 KNTD4Cd/Ce9w61mQSFQDrD7qcX3cCUUyl/5x9qaX+te0CIIUTgi+lpELKta4LqYTgbat TnnkCBCYZ2BWjpi+Yl2wi5hm9kxltoVd5Yvm6O7KIHHiVJqugCUNmyA5/Ru4PgNDbL4S SsR/aK8sErJk9YO3K/t6ZAjkmiWLeckoxnSWSNnSVKNVOOkFFu87ft6siF/H9OWMeyeW xlDzuWQVVAsIPQ/QgxTF9b6so7xDnrxHsBK+kpeKL4xn8nl7qnwTiRIaxHWCtBVxlYw6 c/ng== MIME-Version: 1.0 Received: by 10.68.223.161 with SMTP id qv1mr2063229pbc.100.1333046074660; Thu, 29 Mar 2012 11:34:34 -0700 (PDT) Sender: mdf356@gmail.com Received: by 10.68.52.40 with HTTP; Thu, 29 Mar 2012 11:34:34 -0700 (PDT) In-Reply-To: <20120329182847.GD2358@deviant.kiev.zoral.com.ua> References: <20120327183440.GS2358@deviant.kiev.zoral.com.ua> <20120329091415.GA1316@reks> <20120329093427.GA2358@deviant.kiev.zoral.com.ua> <20120329182847.GD2358@deviant.kiev.zoral.com.ua> Date: Thu, 29 Mar 2012 11:34:34 -0700 X-Google-Sender-Auth: Y9Bwm7peO2ax0PGduNc6nXm-lA0 Message-ID: From: mdf@FreeBSD.org To: Konstantin Belousov Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: fs@freebsd.org Subject: Re: RFC: SEEK_HOLE/SEEK_DATA common implementation X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 18:34:35 -0000 On Thu, Mar 29, 2012 at 11:28 AM, Konstantin Belousov wrote: > On Thu, Mar 29, 2012 at 06:56:31AM -0700, mdf@freebsd.org wrote: >> On Thu, Mar 29, 2012 at 2:34 AM, Konstantin Belousov >> wrote: >> > Filesystem probably should always fall back to calling vop_stdioctl() >> > manually if it ever implements VOP_IOCTL, regardless of seek_hole/data= . >> >> We can't if there's a use of VOP_BMAP() in one of the ioctls. =A0For >> OneFS, a call to VOP_BMAP is fatal. =A0That operation doesn't work with >> our system. > But you had to already take some measure to disable the call to VOP_BMAP(= ) > for your filesystem, otherwise mmap() is fatal too (whatever the word > fatal means) ? Yes, OneFS does not support mmap(). >> We could implement SEEK_HOLE/SEEK_DATA semi-efficiently, but >> implementing a custom filesystem becomes very complex once one has not >> only to manage replacing all VOPs where the default doesn't work but >> also has to know there's a bunch of ioctls in VOP_IOCTL that must be >> handled as well. > > Does anything change if VOP_SEEKDATA() is added which default implementat= ion > using VOP_BMAP() ? If it's a VOP it makes it easier to e.g. write a consistency checker when creating a vnode, that the FS defines certain VOPs. This could be in conjunction with a mnt flag indicating support for various features. At the least, it's more expected, I'd think, for a filesystem to know it has to handle all the VOPs listed in the table. There's no global list of all ioctls a filesystem is expected to handle. Thanks, matthew From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 20:13:02 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA7211065673 for ; Thu, 29 Mar 2012 20:13:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id A34068FC14 for ; Thu, 29 Mar 2012 20:13:02 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap4EAHPBdE+DaFvO/2dsb2JhbABEhUS0VoIJAQEEASNWBRYOCgICDRkCWQYTCYd8BQupG5IqgS+ORoEYBJVhkC6DAw X-IronPort-AV: E=Sophos;i="4.75,338,1330923600"; d="scan'208";a="165858640" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 29 Mar 2012 16:11:54 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 187E6B407B; Thu, 29 Mar 2012 16:11:54 -0400 (EDT) Date: Thu, 29 Mar 2012 16:11:54 -0400 (EDT) From: Rick Macklem To: Bob Friesenhahn Message-ID: <2042316157.1946865.1333051914085.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE8 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org Subject: Re: NFSv3, ZFS, 10GE performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 20:13:03 -0000 Bob Friesenhahn wrote: > On Wed, 28 Mar 2012, Rick Macklem wrote: > >> > >> Hopefully, readahead doesn't kill performance for smaller files.. > >> :-) > >> > > Well, readaheads only happen if the file is large enough for the > > readahead > > to be before EOF. As such, they just won't happen for small files. > > The real problem is for applications which do a seek and a new read > prior to consuming all the data which is being read ahead on its > behalf. This results in read amplification and increased latency for > the next seek+read. > Yes, random reads on a large file is a problem, as you noted before. (I was just clarifying the "small file" case and it's good that you are clarifying the "large file, non-sequential" case.) As jhb@ noted, disabling read-ahead may be a good idea for the POSIX hint. Thanks, rick > Bob > -- > Bob Friesenhahn > bfriesen@simple.dallas.tx.us, > http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 20:29:02 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 68CF4106566C for ; Thu, 29 Mar 2012 20:29:02 +0000 (UTC) (envelope-from peterjeremy@acm.org) Received: from mail12.syd.optusnet.com.au (mail12.syd.optusnet.com.au [211.29.132.193]) by mx1.freebsd.org (Postfix) with ESMTP id ECBEC8FC14 for ; Thu, 29 Mar 2012 20:29:01 +0000 (UTC) Received: from server.vk2pj.dyndns.org (c220-239-116-103.belrs4.nsw.optusnet.com.au [220.239.116.103]) by mail12.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id q2TKSmwj027516 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 30 Mar 2012 07:28:50 +1100 X-Bogosity: Ham, spamicity=0.000000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.5/8.14.4) with ESMTP id q2TKSmv2077281; Fri, 30 Mar 2012 07:28:48 +1100 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.5/8.14.5/Submit) id q2TKSldL077280; Fri, 30 Mar 2012 07:28:47 +1100 (EST) (envelope-from peter) Date: Fri, 30 Mar 2012 07:28:46 +1100 From: Peter Jeremy To: Beeblebrox Message-ID: <20120329202846.GB76833@server.vk2pj.dyndns.org> References: <0685CC3A-753B-4C5B-9E15-C0565B48F885@ultra-secure.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="b5gNqxB1S1yM7hjW" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: jailed NFS server X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 20:29:02 -0000 --b5gNqxB1S1yM7hjW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2012-Mar-29 05:12:43 +0300, Beeblebrox wrote: >Maybe I will give unfs3 a try. However, One of the reasons I'm trying to >set it up is to be able to run Tinderbox on that jail for distributed >compiling. When I did a little searching about unfs3 + Tinderbox + jail, it >came up with posts about problems and that such setup "does not give good >results". Whilst I've not used unfs3 on FreeBSD, I do use it on Solaris to allow me to NFS export a (ZFS) filesystem from within a zone. My experience is that it works reasonably well, given its limitations: - It's single-threaded. This isn't an issue for me because there are only a couple of light users. It would be useless as a server for more than that. - There's no support for locking (lockd/statd). - A user who has shell access to the server and can mount a filesystem via unfs3 can DoS the NFS server by killing the unfs3 daemon. I did find it necessary to fix a number of bugs along the way. --=20 Peter Jeremy --b5gNqxB1S1yM7hjW Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk90xf4ACgkQ/opHv/APuIesSQCguoXSaV6TSmkRYFULzOXK0IbR 6GQAn0S4CejOvOpK9oJVBF1ePLRx8fr8 =ONrg -----END PGP SIGNATURE----- --b5gNqxB1S1yM7hjW-- From owner-freebsd-fs@FreeBSD.ORG Fri Mar 30 07:08:17 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9F61E1065678 for ; Fri, 30 Mar 2012 07:08:16 +0000 (UTC) (envelope-from mailer@mail.firmos.at) Received: from mail.firmos.at (mail.firmos.at [80.123.225.49]) by mx1.freebsd.org (Postfix) with ESMTP id 8C0F68FC12 for ; Fri, 30 Mar 2012 07:08:15 +0000 (UTC) Received: from [91.114.28.42] (helo=BigMac.local) by mail.firmos.at with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.76 (FreeBSD)) (envelope-from ) id 1SDVw9-000H7U-1v for freebsd-fs@freebsd.org; Fri, 30 Mar 2012 09:07:53 +0200 Message-ID: <4F755BD7.7040308@firmos.at> Date: Fri, 30 Mar 2012 09:08:07 +0200 From: Franz Schober Organization: FirmOS Business Solutions Gmbh User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: mailer@mail.firmos.at X-SA-Exim-Connect-IP: 91.114.28.42 X-SA-Exim-Mail-From: mailer@mail.firmos.at X-SA-Exim-Scanned: No (on mail.firmos.at); SAEximRunCond expanded to false Subject: Re: Re; ZFS Performance FreeBSD 9.0 vs. Openindiana X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2012 07:08:17 -0000 Thx for your input ! Am 29.03.12 12:51, schrieb grarpamp: > Knowing of no better... I find Iozone quite nice. > > However... I would first resolve these differences in > the test methodology before trying to compare > anything 74 revisions apart... and perhaps the > address width as well. > > Version $Revision: 3.323 $ > Compiled for 32 bit mode. > Build: Solaris10cc-64 > File size set to 4194304 KB > > Version $Revision: 3.397 $ > Compiled for 64 bit mode. > Build: freebsd > File size set to 4194304 KB I recompiled both iozone binaries to 3.398 64 bit, results are similar. Messurement files are below. > Then try to see where the average 45% speedup is occuring. > >> to run not only in cache >> Processor cache size set to 1024 Kbytes. > 128k could very well fit in a cache somewhere. > Such as say, the one above. > > As shown on the iozone site, I'd try graphing the full range > of record sizes as a first baseline comparison. In progress. > And poke around the sysctls to see how the zfs parameters > match up. Any hints about the very interresting sysctls about zfs tuning for FreeBSD here ? >> -c Include close() in the timing calculations. This is useful only >> if you suspect that close() is broken in the operating system >> currently under test. > Really, you do? Used the parameters here from an NFS testing paper, for the new tests I omitted this one. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > Iozone: Performance Test of File I/O Version $Revision: 3.398 $ Compiled for 64 bit mode. Build: freebsd Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer. Ben England. Run began: Thu Mar 29 14:19:38 2012 SYNC Mode. Record Size 128 KB File size set to 4194304 KB Command line used: iozone -o -t 8 -r 128k -s 4G Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. Throughput test with 8 processes Each process writes a 4194304 Kbyte file in 128 Kbyte records Children see throughput for 8 initial writers = 150868.63 KB/sec Parent sees throughput for 8 initial writers = 150863.48 KB/sec Min throughput per process = 18857.70 KB/sec Max throughput per process = 18859.75 KB/sec Avg throughput per process = 18858.58 KB/sec Min xfer = 4193920.00 KB Children see throughput for 8 rewriters = 149426.29 KB/sec Parent sees throughput for 8 rewriters = 149424.16 KB/sec Min throughput per process = 18593.01 KB/sec Max throughput per process = 18746.05 KB/sec Avg throughput per process = 18678.29 KB/sec Min xfer = 4160128.00 KB Children see throughput for 8 readers = 1042745.14 KB/sec Parent sees throughput for 8 readers = 1041299.44 KB/sec Min throughput per process = 122961.57 KB/sec Max throughput per process = 133874.20 KB/sec Avg throughput per process = 130343.14 KB/sec Min xfer = 3852288.00 KB Children see throughput for 8 re-readers = 866942.31 KB/sec Parent sees throughput for 8 re-readers = 862920.82 KB/sec Min throughput per process = 104678.79 KB/sec Max throughput per process = 111929.87 KB/sec Avg throughput per process = 108367.79 KB/sec Min xfer = 3944832.00 KB Children see throughput for 8 reverse readers = 997806.75 KB/sec Parent sees throughput for 8 reverse readers = 988615.08 KB/sec Min throughput per process = 108911.76 KB/sec Max throughput per process = 144133.22 KB/sec Avg throughput per process = 124725.84 KB/sec Min xfer = 3181952.00 KB Children see throughput for 8 stride readers = 288693.36 KB/sec Parent sees throughput for 8 stride readers = 287774.92 KB/sec Min throughput per process = 33722.81 KB/sec Max throughput per process = 40996.01 KB/sec Avg throughput per process = 36086.67 KB/sec Min xfer = 3455104.00 KB Children see throughput for 8 random readers = 115986.33 KB/sec Parent sees throughput for 8 random readers = 115983.24 KB/sec Min throughput per process = 14144.19 KB/sec Max throughput per process = 14719.62 KB/sec Avg throughput per process = 14498.29 KB/sec Min xfer = 4030464.00 KB Children see throughput for 8 mixed workload = 189486.43 KB/sec Parent sees throughput for 8 mixed workload = 189470.53 KB/sec Min throughput per process = 12340.80 KB/sec Max throughput per process = 35010.00 KB/sec Avg throughput per process = 23685.80 KB/sec Min xfer = 1478528.00 KB Children see throughput for 8 random writers = 147573.26 KB/sec Parent sees throughput for 8 random writers = 147564.68 KB/sec Min throughput per process = 18446.08 KB/sec Max throughput per process = 18447.95 KB/sec Avg throughput per process = 18446.66 KB/sec Min xfer = 4193920.00 KB Children see throughput for 8 pwrite writers = 150153.40 KB/sec Parent sees throughput for 8 pwrite writers = 150151.15 KB/sec Min throughput per process = 18768.81 KB/sec Max throughput per process = 18769.74 KB/sec Avg throughput per process = 18769.18 KB/sec Min xfer = 4194176.00 KB Children see throughput for 8 pread readers = 1132129.56 KB/sec Parent sees throughput for 8 pread readers = 1132038.43 KB/sec Min throughput per process = 132049.78 KB/sec Max throughput per process = 149369.56 KB/sec Avg throughput per process = 141516.20 KB/sec Min xfer = 3708288.00 KB Children see throughput for 8 fwriters = 479942.93 KB/sec Parent sees throughput for 8 fwriters = 405355.22 KB/sec Min throughput per process = 51488.59 KB/sec Max throughput per process = 79697.76 KB/sec Avg throughput per process = 59992.87 KB/sec Min xfer = 4194304.00 KB Children see throughput for 8 freaders = 1305623.56 KB/sec Parent sees throughput for 8 freaders = 1274297.58 KB/sec Min throughput per process = 159303.33 KB/sec Max throughput per process = 170930.03 KB/sec Avg throughput per process = 163202.95 KB/sec Min xfer = 4194304.00 KB iozone test complete. Iozone: Performance Test of File I/O Version $Revision: 3.398 $ Compiled for 64 bit mode. Build: Solaris10gcc-64 Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer. Ben England. Run began: Thu Mar 29 14:26:32 2012 SYNC Mode. Record Size 128 KB File size set to 4194304 KB Command line used: /usr/benchmarks/iozone/iozone -o -t 8 -r 128k -s 4G Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. Throughput test with 8 processes Each process writes a 4194304 Kbyte file in 128 Kbyte records Children see throughput for 8 initial writers = 163928.73 KB/sec Parent sees throughput for 8 initial writers = 163742.79 KB/sec Min throughput per process = 20472.19 KB/sec Max throughput per process = 20514.05 KB/sec Avg throughput per process = 20491.09 KB/sec Min xfer = 4185728.00 KB Children see throughput for 8 rewriters = 166042.96 KB/sec Parent sees throughput for 8 rewriters = 166022.58 KB/sec Min throughput per process = 20738.16 KB/sec Max throughput per process = 20785.44 KB/sec Avg throughput per process = 20755.37 KB/sec Min xfer = 4184960.00 KB Children see throughput for 8 readers = 1204767.22 KB/sec Parent sees throughput for 8 readers = 1204261.66 KB/sec Min throughput per process = 129381.34 KB/sec Max throughput per process = 193455.00 KB/sec Avg throughput per process = 150595.90 KB/sec Min xfer = 2804864.00 KB Children see throughput for 8 re-readers = 996214.55 KB/sec Parent sees throughput for 8 re-readers = 991045.12 KB/sec Min throughput per process = 123043.02 KB/sec Max throughput per process = 126982.37 KB/sec Avg throughput per process = 124526.82 KB/sec Min xfer = 4094208.00 KB Children see throughput for 8 reverse readers = 2712166.84 KB/sec Parent sees throughput for 8 reverse readers = 2634185.61 KB/sec Min throughput per process = 271423.66 KB/sec Max throughput per process = 581808.50 KB/sec Avg throughput per process = 339020.86 KB/sec Min xfer = 1995136.00 KB Children see throughput for 8 stride readers = 318718.81 KB/sec Parent sees throughput for 8 stride readers = 318551.66 KB/sec Min throughput per process = 32463.41 KB/sec Max throughput per process = 50197.65 KB/sec Avg throughput per process = 39839.85 KB/sec Min xfer = 2714112.00 KB Children see throughput for 8 random readers = 121252.01 KB/sec Parent sees throughput for 8 random readers = 121242.78 KB/sec Min throughput per process = 14592.07 KB/sec Max throughput per process = 16029.80 KB/sec Avg throughput per process = 15156.50 KB/sec Min xfer = 3818112.00 KB Children see throughput for 8 mixed workload = 192510.71 KB/sec Parent sees throughput for 8 mixed workload = 192398.09 KB/sec Min throughput per process = 12424.72 KB/sec Max throughput per process = 35641.26 KB/sec Avg throughput per process = 24063.84 KB/sec Min xfer = 1462912.00 KB Children see throughput for 8 random writers = 163482.42 KB/sec Parent sees throughput for 8 random writers = 163297.29 KB/sec Min throughput per process = 20411.26 KB/sec Max throughput per process = 20456.37 KB/sec Avg throughput per process = 20435.30 KB/sec Min xfer = 4185216.00 KB Children see throughput for 8 fwriters = 132004.80 KB/sec Parent sees throughput for 8 fwriters = 130492.16 KB/sec Min throughput per process = 16337.71 KB/sec Max throughput per process = 16703.73 KB/sec Avg throughput per process = 16500.60 KB/sec Min xfer = 4194304.00 KB Children see throughput for 8 freaders = 1069833.95 KB/sec Parent sees throughput for 8 freaders = 1064831.11 KB/sec Min throughput per process = 133137.16 KB/sec Max throughput per process = 135073.42 KB/sec Avg throughput per process = 133729.24 KB/sec Min xfer = 4194304.00 KB iozone test complete. Thx, Franz Schober From owner-freebsd-fs@FreeBSD.ORG Fri Mar 30 07:56:29 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 19518106566C for ; Fri, 30 Mar 2012 07:56:29 +0000 (UTC) (envelope-from rsb@berentweb.com) Received: from mail-lpp01m010-f54.google.com (mail-lpp01m010-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 826558FC14 for ; Fri, 30 Mar 2012 07:56:28 +0000 (UTC) Received: by lagv3 with SMTP id v3so621162lag.13 for ; Fri, 30 Mar 2012 00:56:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=berentweb.com; s=google; h=mime-version:sender:x-originating-ip:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=NWZJTc6Tx5JqOw+zrUDK/TNntWYYFCcXdLFENZ+BvMw=; b=QRIAcXavqxSXoK7TC4sKmhhxceqKnJ3Xxp4hX+5/wQycyU6AXVYaLB0X7/2TG+m0uZ ftBOZUL7jYygziBHnoxVVZHbGVEw/PscfwpeEM5v/AyIYbFGk86Su4WAOc/F/4lacKSi /yBZIbNXxU41Y20Q0FTT+Mfmpic84gs3CJI6I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:x-originating-ip:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:x-gm-message-state :content-type; bh=NWZJTc6Tx5JqOw+zrUDK/TNntWYYFCcXdLFENZ+BvMw=; b=Cp3IZih0ylUZlkFn3wO4T9MH04lBVY7s799tRw7Qgsdy3tme5UXKIYnkN5ayq2/PWc l+pus7SLmOwAnCWCh8A410MrsCLfxLonYROumWNIrjbzTsXV6/y2qGJ5xrD0jNhstosm /o4XXHKWUqvz/3A1CNY02IVe4KhYHCPkoZu0EWq9rOMWfheLVOdQAFST7HGwRCPNqZfE zrneCrc3SFNFrqI/y5zc0fKGHxEO32y7b0TRvW94UVegBn7BqjDjk4U1hvqemnI11Hwh 4Yi6T1Drc1mg+nuwQLHzF3W2hffVK086NL9zlEZCMh0NJsR/ae61hmEkToBv2nVBPwE+ S94g== MIME-Version: 1.0 Received: by 10.152.105.211 with SMTP id go19mr1365426lab.51.1333094186985; Fri, 30 Mar 2012 00:56:26 -0700 (PDT) Sender: rsb@berentweb.com Received: by 10.112.77.15 with HTTP; Fri, 30 Mar 2012 00:56:26 -0700 (PDT) X-Originating-IP: [78.162.12.68] In-Reply-To: <20120329202846.GB76833@server.vk2pj.dyndns.org> References: <0685CC3A-753B-4C5B-9E15-C0565B48F885@ultra-secure.de> <20120329202846.GB76833@server.vk2pj.dyndns.org> Date: Fri, 30 Mar 2012 10:56:26 +0300 X-Google-Sender-Auth: D5e3tpl18Q2ka8642ltn7t9D2ZM Message-ID: From: Beeblebrox To: freebsd-fs@freebsd.org X-Gm-Message-State: ALoCoQkiyIJZsPQmCSdQB5EWmiHT3HyRyjnMiGBx/A+1EcMPRc0GnAEJpMb6ol9H/5C1tFl0GXHU Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: jailed NFS server X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2012 07:56:29 -0000 All NFS clients are "slave nodes" booting diskless and the network structure is a build/compute farm. I therefore have nothing to be concerned with regards to security on the internal network side, as long as I have my PF working correctly. In this structure I do need NFS responding as fast as possible and I really do not want to fiddle with strange errors tinderbox might start throwing at build time because of jail incompatibility. Thanks for the input everyone, but I've decided to serve NFS from host under the circumstances. So SOLVED, although not as initially intended... On Thu, Mar 29, 2012 at 11:28 PM, Peter Jeremy wrote: > On 2012-Mar-29 05:12:43 +0300, Beeblebrox wrote: > >Maybe I will give unfs3 a try. However, One of the reasons I'm trying to > >set it up is to be able to run Tinderbox on that jail for distributed > >compiling. When I did a little searching about unfs3 + Tinderbox + jail, > it > >came up with posts about problems and that such setup "does not give good > >results". > > Whilst I've not used unfs3 on FreeBSD, I do use it on Solaris to allow > me to NFS export a (ZFS) filesystem from within a zone. My experience > is that it works reasonably well, given its limitations: > - It's single-threaded. This isn't an issue for me because there are > only a couple of light users. It would be useless as a server for > more than that. > - There's no support for locking (lockd/statd). > - A user who has shell access to the server and can mount a filesystem > via unfs3 can DoS the NFS server by killing the unfs3 daemon. > > I did find it necessary to fix a number of bugs along the way. > > -- > Peter Jeremy > From owner-freebsd-fs@FreeBSD.ORG Fri Mar 30 08:06:17 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 46213106564A for ; Fri, 30 Mar 2012 08:06:17 +0000 (UTC) (envelope-from johnny@joonix.se) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id ED5DD8FC1A for ; Fri, 30 Mar 2012 08:06:16 +0000 (UTC) Received: by vcmm1 with SMTP id m1so329407vcm.13 for ; Fri, 30 Mar 2012 01:06:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:x-originating-ip:date:x-google-sender-auth :message-id:subject:from:to:x-gm-message-state:content-type; bh=kVtDTpjenEZ3ro1BEewaD1jAy/MUUbXgmBqeVZUJbEo=; b=LGIqcdS+yC/ARUKynv8iIgHuJhr0b66Bndm1hI8hFGcqnRCimHfxEeRLzGmml0zPKq cYl9dqlz0AHq461/s/cdKXf7dWJBtJG8WMjXV/V0JwdZE/epbFooIE3xzuXsT7yokQ14 8nWzyPuoiro3GdnEb1RdtSUw3U5lQFUmF+pq5kKNwFwp7sEfwOo1X2I7f3Kym/dumjT/ 5snRmN1je68zB7nsSWAzNgpxzyBt1Kiz1JvlDwiKKJr2WcwI/qNYTWwUMMj2ppl1Zk3Y vbNThjbfQGm5gqEOiGvv+w7sZl1YM5j4iBXcvbsm4J14Mr80qslAeG60wcqKwxxehJi3 7BzA== MIME-Version: 1.0 Received: by 10.220.151.198 with SMTP id d6mr160493vcw.70.1333094775894; Fri, 30 Mar 2012 01:06:15 -0700 (PDT) Sender: johnny@joonix.se Received: by 10.52.66.161 with HTTP; Fri, 30 Mar 2012 01:06:15 -0700 (PDT) X-Originating-IP: [195.178.174.2] Date: Fri, 30 Mar 2012 10:06:15 +0200 X-Google-Sender-Auth: 1QRpqV5Ri1ZZMpUqojR2X1IH3wQ Message-ID: From: =?ISO-8859-1?Q?Johnny_Bergstr=F6m?= To: freebsd-fs@freebsd.org X-Gm-Message-State: ALoCoQl0bGCNt6D0WFm+qSYbMwRFfqjEl5kEiifqCc+WAy7UTWhKRqN8Veh7IFljFRw90lwc0L6T Content-Type: text/plain; charset=ISO-8859-1 Subject: CARP + HAST + ZFS + NFS = problems? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2012 08:06:17 -0000 Hello! This is my first post in the mailing list so please bare with me. I have been looking for a good solution to provide a storage server with high availability that will serve photos via nfs. Stumbled upon this article: http://www.aisecure.net/2012/02/07/hast-freebsd-zfs-with-carp-failover/ which is mostly what my current lab testing looks like. This looked like a promising setup and I trust ZFS since I've used it in more simple setups before. But unfortunately there has been some problems in my tests. The whole idea is that the slave machine it should handle failures from the master such as dropping the network interface or even crashing / resetting. Many times my tests has worked flawlessly, clients get stalled on the NFS mount until the new server wakes up and then continue writing data as if nothing happened. But more than once, I've had ZFS metadata corruption or even unrecoverable errors where zfs import -F doesn't work and it tells me to restore the pool from backup. My tests consists of simply doing ifconfig down while a client is writing data and also doing hard reset on the master machine. This is what the lab setup looks like: 2 virtual machines running with 1G of ram as Xen guests. 2 ZFS mirrored virtual drives via hast devices. CARP setup which is monitored by devd and that executes my script similar to the one in the article, with additions to start/stop nfsd. The problems with corruptions has never happened when I try to use ZFS directly on the drives without having HAST in between. Also I've noticed the virtual harddrives have problems flushing: Mar 29 10:01:01 storage1 hastd[6690]: [disk2] (primary) Remote request failed (Operation not supported by device): FLUSH. Mar 29 10:01:02 storage1 hastd[6690]: [disk2] (primary) Unable to flush disk cache on activemap update: Operation not supported by device. I'm wondering if anyone has used this setup on real machines and on production, it seems a bit sketchy to me right now but in theory the solution would be perfect. My idea was to apply this on real machines with raidz of 3 drives, which later would be expanded by an additional 3 drives. Any suggestions on fixing this problem or an alternative configuration that might be more stable? From owner-freebsd-fs@FreeBSD.ORG Fri Mar 30 11:52:29 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B2A3106564A for ; Fri, 30 Mar 2012 11:52:29 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-bk0-f54.google.com (mail-bk0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id C017B8FC14 for ; Fri, 30 Mar 2012 11:52:28 +0000 (UTC) Received: by bkcjc3 with SMTP id jc3so534707bkc.13 for ; Fri, 30 Mar 2012 04:52:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=MriBTkdUOa4RbdqOCizNn1i8BrknkyDV84m1/HobqrI=; b=oMpC9FVW3Z9JucEz9RcIdZ8AgfzJ1MWU8C/qwNx9vDHyD5GJJ0txgIS7mEAtjs/lPU k1LDlOe6Xk4BFJdO0Vo5oEazzfq4J67rKyV+K3ecn2sQ8aL3bmBgvbqlDIc7ySPKKTVF hRKo0CguqbzLgMTsAJYAdXPsByuTSAYDRLMZw0Dmuf1oN9BZ61tDDa6H78jaJkDZrArT PoHY6NIEGrsQlulInKOcixqH30on6dT+R8Vb2kI1qBmqIQHtVzN5OURsIGartfWmecds DPm8opoXPSHnTx0UJ9nz8kZUY1vTqP/JGEb7nDuYIWICsTBoqlvlKpTLWqRn+fMo23yF VqDA== MIME-Version: 1.0 Received: by 10.205.117.15 with SMTP id fk15mr800231bkc.133.1333108347717; Fri, 30 Mar 2012 04:52:27 -0700 (PDT) Received: by 10.204.202.142 with HTTP; Fri, 30 Mar 2012 04:52:27 -0700 (PDT) Received: by 10.204.202.142 with HTTP; Fri, 30 Mar 2012 04:52:27 -0700 (PDT) In-Reply-To: References: <0685CC3A-753B-4C5B-9E15-C0565B48F885@ultra-secure.de> Date: Fri, 30 Mar 2012 11:52:27 +0000 Message-ID: From: Chris Rees To: Beeblebrox Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: jailed NFS server X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2012 11:52:29 -0000 On 29 Mar 2012 03:13, "Beeblebrox" wrote: > > Maybe I will give unfs3 a try. However, One of the reasons I'm trying to > set it up is to be able to run Tinderbox on that jail for distributed > compiling. When I did a little searching about unfs3 + Tinderbox + jail, it > came up with posts about problems and that such setup "does not give good > results". > Any feedback about such setup? Also, is such "bad performance result" also > valid for all types of HPC/parallel computing run from jails so that "Just > Run From Host" becomes the defaul solution in these cases? > Jails aren't really virtualised, so performance shouldn't be an issue. Look at the Tinderbox README for how to run it in a jail; until nullfs is jail-safe it's impossible to do 'properly'- you have to fake around it by chrooting tinderd. Chris From owner-freebsd-fs@FreeBSD.ORG Fri Mar 30 14:49:22 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D0E36106566C for ; Fri, 30 Mar 2012 14:49:22 +0000 (UTC) (envelope-from c.kworr@gmail.com) Received: from mail-bk0-f54.google.com (mail-bk0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 5C6288FC18 for ; Fri, 30 Mar 2012 14:49:22 +0000 (UTC) Received: by bkcjc3 with SMTP id jc3so770988bkc.13 for ; Fri, 30 Mar 2012 07:49:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=2nWUtw+3/y9+9bpBzBdmnFiburdbVZ62BBWCUKkRbLQ=; b=lZP4FZcNewMtWmjkcKlu+CiC0b4j0Cn8pBtdhxlOmNp7JRggn/0LMl3yDBZO6IEBHb KV4Lg6+7CauNTKC/YVYnsHhUXY2clMWTwQwLA/xYyvtQN6xZgWIQQWYNpEfBTHhIA4Ec A5Y/kiRv6oHzOvBGrw8i1XNXfNkXDFlUt/41FE7DsV7HPooaYQugftq7BGySNxiloj+1 E6f0SeJXYWuiRDN+ezauIFuBY/QRL4KAMLh94UVskvHsSLD8OEElKI+Nv5euoA3RbvnB 3I1sBqfuRKntm/GJIUNdTL5Bq/Pfbw7IQn6xPyE9AoRWG/ngMXzSnt314/q8+A+7WtKE 0rwg== Received: by 10.204.145.81 with SMTP id c17mr1147917bkv.39.1333118961413; Fri, 30 Mar 2012 07:49:21 -0700 (PDT) Received: from green.tandem.local (255-125-132-95.pool.ukrtel.net. [95.132.125.255]) by mx.google.com with ESMTPS id r14sm21164083bkv.11.2012.03.30.07.49.19 (version=SSLv3 cipher=OTHER); Fri, 30 Mar 2012 07:49:20 -0700 (PDT) Message-ID: <4F75C7EC.30606@gmail.com> Date: Fri, 30 Mar 2012 17:49:16 +0300 From: Volodymyr Kostyrko User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:11.0) Gecko/20120315 Firefox/11.0 SeaMonkey/2.8 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: ZFS v28 and free space leakage X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2012 14:49:22 -0000 Hi all. Maybe I'm missing something, but when I delete all data from the disk how much space should empty disk occupy? kohrah1 136G 22,6M 136G 0% 1.00x ONLINE - I.e. I have 22 hidden megs. Scrub doesn't touch them. There were no snapshots. Disk was deduped in the past. Server hit core dump several times when running bonnie++ over deduped ZFS. If I create a snapshot and dump it down it would be 50 Kb file. -- Sphinx of black quartz judge my vow. From owner-freebsd-fs@FreeBSD.ORG Fri Mar 30 16:33:42 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6BEAE106564A for ; Fri, 30 Mar 2012 16:33:42 +0000 (UTC) (envelope-from peter.maloney@brockmann-consult.de) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.171]) by mx1.freebsd.org (Postfix) with ESMTP id 00E448FC1E for ; Fri, 30 Mar 2012 16:33:41 +0000 (UTC) Received: from [192.168.179.43] (hmbg-4d06de95.pool.mediaWays.net [77.6.222.149]) by mrelayeu.kundenserver.de (node=mrbap2) with ESMTP (Nemesis) id 0MUlX2-1RjswM3Wjk-00Ydu6; Fri, 30 Mar 2012 18:33:35 +0200 Message-ID: <4F75E05D.2060206@brockmann-consult.de> Date: Fri, 30 Mar 2012 18:33:33 +0200 From: Peter Maloney User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20120327 Thunderbird/11.0.1 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4F75C7EC.30606@gmail.com> In-Reply-To: <4F75C7EC.30606@gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:6yeG1YvdCRmeoolA4V9Aw037quLNqlh1oky2uypZ2jx IDTA8XHyFE3IGCQhJaupQQxvXQ3JcVqejfdBwunG/uaKRH7+1q Wb/9hL56sGX/xA7g2QyzE7Wyg7d7Kjh8oHQRt9MoDQ4brf5kju oW8jWRSc230s+ORh2h714MJYA78Kq1zO7nu4GGjcb5zbGJJfum FsmoMuV7Y949zRqq5a7Ckvm/VSRK3+eWqimcx5T2ka2gaDf611 b/IC5cWlS1sZFfXi6NU7lAq4ez7YjbRmLNcgYnwLTiDgKxFOlZ Uh/GfdWXuFp6Rbir3YEgxoAV1POzqW0FzOQwMfJafO5jw7OdVo iiFJ4pD2pDeMX/4ACQj+sIlh4zlvUGlgcDgQrtYI1wSRVT0mY6 eot83ue17hqWw== Subject: Re: ZFS v28 and free space leakage X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2012 16:33:42 -0000 I think you ran zpool list... Does zfs list show the same? Do you have any snapshots or clones? What sort of vdevs do you have? Does creating an empty pool show 0 used? What about after adding more datasets? Do you have datasets? They might use some for metadata. Here begins the guessing and/or babbling... And I haven't tried this with zfs, but I know with ext on Linux, if you fill up a directory, and delete all the files in it, the directory takes more space than before it was filled (du will include this space when run). So be very thorough with how you calculate it. Maybe zfs did the same thing with metadata structures, and just left them allocated empty (just a guess). To prove there is a leak, you would need to fill up the disk, delete everything, and then fill it again to see if it fit less. If I did such a test and it was the same, I would just forget about the problem. Perhaps another interesting experiment would be to zfs send the pool to see if the destination pool ends up in the same state. On 30.03.2012 16:49, Volodymyr Kostyrko wrote: > Hi all. > > Maybe I'm missing something, but when I delete all data from the disk > how much space should empty disk occupy? > > kohrah1 136G 22,6M 136G 0% 1.00x ONLINE - > > I.e. I have 22 hidden megs. Scrub doesn't touch them. There were no > snapshots. Disk was deduped in the past. Server hit core dump several > times when running bonnie++ over deduped ZFS. > > If I create a snapshot and dump it down it would be 50 Kb file. > From owner-freebsd-fs@FreeBSD.ORG Fri Mar 30 23:51:48 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 222311065674 for ; Fri, 30 Mar 2012 23:51:48 +0000 (UTC) (envelope-from josh@signalboxes.net) Received: from mail.signaltotrust.net (hewbert.com [69.164.207.85]) by mx1.freebsd.org (Postfix) with ESMTP id BC0A48FC14 for ; Fri, 30 Mar 2012 23:51:47 +0000 (UTC) Received: from [192.168.1.51] (unknown [67.158.43.219]) (Authenticated sender: josh) by mail.signaltotrust.net (Postfix) with ESMTPSA id 14BB862E1 for ; Fri, 30 Mar 2012 17:51:46 -0600 (MDT) Message-ID: <4F764712.2010407@signalboxes.net> Date: Fri, 30 Mar 2012 17:51:46 -0600 From: Josh Beard User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:11.0) Gecko/20120313 Thunderbird/11.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: NFS: rpc.statd/lockd becomes unresponsive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2012 23:51:48 -0000 Originally sent to freebsd-net, but I realized this is probably a more appropriate list. Sorry! Hello, We've recently setup a FreeBSD 9.0-RELEASE (x64) system to test as an NFS server for "live" network homes for Mac clients (mostly 10.5 and 10.6 clients). We're a public school district and normally have around 150-200 users logged in at a time with network homes. Currently, we're using netatalk (AFP) on a Linux box, after migrating from an aging Mac OS X server. Unfortunately, netatalk has some serious performance issues under the load we're putting it under and we'd like to migrate to NFS. We've tried several Linux distributions and various kernels and we're now testing FreeBSD (and tested FreeNAS) with similar setups. Unfortunately, they all suffer the same issue. As a test, I have a series of scripts to simulate user activity on the clients (e.g. opening Word, opening a browser, doing some read/writes with dd, etc). After a while, NFS on the server runs into an issue where (what I think happens) rpc.statd can't talk to rpc.lockd. Being Mac clients, they all get a rather ugly dialog box stating that their connection to the server has been lost. It's worth mentioning that this server is a KVM 'guest' on a Linux server. I'm aware of some I/O issues there, but I don't have a decent piece of hardware to really test this on. I allocated 4 CPUs to it and 10GB of RAM. I've tested with the virtio net drivers and without. Considering I've seen the same symptoms on around 6 Linux distributions, with various kernels, FreeNAS, and FreeBSD, I wouldn't be surprised to get the same results if I weren't virtualized. I haven't really done any tuning on the FreeBSD server, it's fairly vanilla. We have around ~2600 machines throughout our campus, with limited remote management capabilities (that's on the big agenda to tackle), so changing NFS mount options there would be rather difficult. These are LDAP accounts with the NFS mounts in LDAP as well, for what it's worth. The clients mount it pretty vanilla (output of 'mount' on client): freenas.dsdk12.schoollocal:/mnt/homes on /net/freenas.dsdk12.schoollocal/mnt/homes (nfs, nodev, nosuid, automounted, nobrowse) On the server, my /etc/exports looks like this: /srv/homes -alldirs -network 172.30.0.0/16 This export doesn't have a lot of data - it's 150 small home directories of test accounts. No other activity is being done on this server. The filesystem if UFS. /etc/rc.conf on the server: rpcbind_enable="YES" nfs_server_enable="YES" mountd_flags="-r -l" nfsd_enable="YES" mountd_enable="YES" rpc_lockd_enable="YES" rpc_statd_enable="YES" nfs_server_flags="-t -n 128" When this occurs, /var/log/messages starts to fill up with this: Mar 30 16:35:18 freefs kernel: Failed to contact local NSM - rpc error 5 Mar 30 16:35:20 freefs rpc.statd: unmon request from localhost, no matching monitor Mar 30 16:35:44 freefs rpc.statd: unmon request from localhost, no matching monitor -- repeated a few times every few seconds -- Mar 30 16:54:50 freefs rpc.statd: Unsolicited notification from host hs00508s4434.dsdk12.schoollocal Mar 30 16:55:01 freefs rpc.statd: Unsolicited notification from host hs00520s4539.dsdk12.schoollocal Mar 30 16:55:10 freefs rpc.statd: Failed to call rpc.statd client at host localhost nfsstat shortly after a failure: Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 0 0 1208 Cache Info: Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses 177 951 226 28 3 6 0 2 BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses 49 3 13 5 9 0 148 9 Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 262698 101012 1575347 29 1924761 2172712 0 43792 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 27447 0 21 5596 1691 118073 0 2596146 Mknod Fsstat Fsinfo PathConf Commit 0 83638 108 108 183632 Server Ret-Failed 0 Server Faults 0 Server Cache Stats: Inprog Idem Non-idem Misses 0 0 0 9172982 Server Write Gathering: WriteOps WriteRPC Opsaved 2172712 2172712 0 rpcinfo shortly after a failure: program version netid address service owner 100000 4 tcp 0.0.0.0.0.111 rpcbind superuser 100000 3 tcp 0.0.0.0.0.111 rpcbind superuser 100000 2 tcp 0.0.0.0.0.111 rpcbind superuser 100000 4 udp 0.0.0.0.0.111 rpcbind superuser 100000 3 udp 0.0.0.0.0.111 rpcbind superuser 100000 2 udp 0.0.0.0.0.111 rpcbind superuser 100000 4 tcp6 ::.0.111 rpcbind superuser 100000 3 tcp6 ::.0.111 rpcbind superuser 100000 4 udp6 ::.0.111 rpcbind superuser 100000 3 udp6 ::.0.111 rpcbind superuser 100000 4 local /var/run/rpcbind.sock rpcbind superuser 100000 3 local /var/run/rpcbind.sock rpcbind superuser 100000 2 local /var/run/rpcbind.sock rpcbind superuser 100005 1 udp6 ::.2.119 mountd superuser 100005 3 udp6 ::.2.119 mountd superuser 100005 1 tcp6 ::.2.119 mountd superuser 100005 3 tcp6 ::.2.119 mountd superuser 100005 1 udp 0.0.0.0.2.119 mountd superuser 100005 3 udp 0.0.0.0.2.119 mountd superuser 100005 1 tcp 0.0.0.0.2.119 mountd superuser 100005 3 tcp 0.0.0.0.2.119 mountd superuser 100024 1 udp6 ::.3.191 status superuser 100024 1 tcp6 ::.3.191 status superuser 100024 1 udp 0.0.0.0.3.191 status superuser 100024 1 tcp 0.0.0.0.3.191 status superuser 100003 2 tcp 0.0.0.0.8.1 nfs superuser 100003 3 tcp 0.0.0.0.8.1 nfs superuser 100003 2 tcp6 ::.8.1 nfs superuser 100003 3 tcp6 ::.8.1 nfs superuser 100021 0 udp6 ::.3.248 nlockmgr superuser 100021 0 tcp6 ::.2.220 nlockmgr superuser 100021 0 udp 0.0.0.0.3.202 nlockmgr superuser 100021 0 tcp 0.0.0.0.2.255 nlockmgr superuser 100021 1 udp6 ::.3.248 nlockmgr superuser 100021 1 tcp6 ::.2.220 nlockmgr superuser 100021 1 udp 0.0.0.0.3.202 nlockmgr superuser 100021 1 tcp 0.0.0.0.2.255 nlockmgr superuser 100021 3 udp6 ::.3.248 nlockmgr superuser 100021 3 tcp6 ::.2.220 nlockmgr superuser 100021 3 udp 0.0.0.0.3.202 nlockmgr superuser 100021 3 tcp 0.0.0.0.2.255 nlockmgr superuser 100021 4 udp6 ::.3.248 nlockmgr superuser 100021 4 tcp6 ::.2.220 nlockmgr superuser 100021 4 udp 0.0.0.0.3.202 nlockmgr superuser 100021 4 tcp 0.0.0.0.2.255 nlockmgr superuser 300019 1 tcp 0.0.0.0.2.185 amd superuser 300019 1 udp 0.0.0.0.2.162 amd superuser The load can get fairly high during my 'stress' tests, but not *that* high. I'm surprised to see these particular symptoms that affect every connected user at the same time and would expect slowdowns rather than the issue I'm seeing. Any ideas or nudges in the right direction are most welcome. This is severely plaguing us and our students :\ Thanks, Josh From owner-freebsd-fs@FreeBSD.ORG Sat Mar 31 19:06:09 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1A3AD1065673; Sat, 31 Mar 2012 19:06:09 +0000 (UTC) (envelope-from jh@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id E19DF8FC1A; Sat, 31 Mar 2012 19:06:08 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q2VJ68ir046929; Sat, 31 Mar 2012 19:06:08 GMT (envelope-from jh@freefall.freebsd.org) Received: (from jh@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q2VJ68B1046925; Sat, 31 Mar 2012 19:06:08 GMT (envelope-from jh) Date: Sat, 31 Mar 2012 19:06:08 GMT Message-Id: <201203311906.q2VJ68B1046925@freefall.freebsd.org> To: pgollucci@FreeBSD.org, jh@FreeBSD.org, freebsd-fs@FreeBSD.org From: jh@FreeBSD.org Cc: Subject: Re: kern/155411: [regression] [8.2-release] [tmpfs]: mount: tmpfs : No space left on device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2012 19:06:09 -0000 Synopsis: [regression] [8.2-release] [tmpfs]: mount: tmpfs : No space left on device State-Changed-From-To: feedback->closed State-Changed-By: jh State-Changed-When: Sat Mar 31 19:06:08 UTC 2012 State-Changed-Why: Feedback timeout. http://www.freebsd.org/cgi/query-pr.cgi?pr=155411 From owner-freebsd-fs@FreeBSD.ORG Sat Mar 31 19:12:01 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A98A7106564A; Sat, 31 Mar 2012 19:12:01 +0000 (UTC) (envelope-from jh@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7E5598FC08; Sat, 31 Mar 2012 19:12:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q2VJC1ba055286; Sat, 31 Mar 2012 19:12:01 GMT (envelope-from jh@freefall.freebsd.org) Received: (from jh@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q2VJC1Fe055273; Sat, 31 Mar 2012 19:12:01 GMT (envelope-from jh) Date: Sat, 31 Mar 2012 19:12:01 GMT Message-Id: <201203311912.q2VJC1Fe055273@freefall.freebsd.org> To: nemo@ikkoku.de, jh@FreeBSD.org, freebsd-fs@FreeBSD.org From: jh@FreeBSD.org Cc: Subject: Re: sparc64/123566: [zfs] zpool import issue: EOVERFLOW X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2012 19:12:01 -0000 Synopsis: [zfs] zpool import issue: EOVERFLOW State-Changed-From-To: feedback->closed State-Changed-By: jh State-Changed-When: Sat Mar 31 19:12:00 UTC 2012 State-Changed-Why: Feedback timeout. http://www.freebsd.org/cgi/query-pr.cgi?pr=123566 From owner-freebsd-fs@FreeBSD.ORG Sat Mar 31 19:29:02 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0C3511065670 for ; Sat, 31 Mar 2012 19:29:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id A0D188FC21 for ; Sat, 31 Mar 2012 19:29:01 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap8EAFVad0+DaFvO/2dsb2JhbABChVS0PYIJAQEBAwEBAQEgBCcgCwUWGAICDRkCKQEJJgYIBwQBHASHYwULrRGJXoEviVsFhHWBGASTNoIrgRGPHoMDgTgI X-IronPort-AV: E=Sophos;i="4.75,349,1330923600"; d="scan'208";a="163153918" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 31 Mar 2012 15:28:54 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id A01F5B4005; Sat, 31 Mar 2012 15:28:54 -0400 (EDT) Date: Sat, 31 Mar 2012 15:28:54 -0400 (EDT) From: Rick Macklem To: Josh Beard Message-ID: <2115805571.2042848.1333222134633.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <4F764712.2010407@signalboxes.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE8 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org Subject: Re: NFS: rpc.statd/lockd becomes unresponsive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2012 19:29:02 -0000 Josh Beard wrote: > Originally sent to freebsd-net, but I realized this is probably a more > appropriate list. Sorry! > > > Hello, > > We've recently setup a FreeBSD 9.0-RELEASE (x64) system to test as an > NFS server for "live" network homes for Mac clients (mostly 10.5 and > 10.6 clients). > > We're a public school district and normally have around 150-200 users > logged in at a time with network homes. Currently, we're using > netatalk > (AFP) on a Linux box, after migrating from an aging Mac OS X server. > Unfortunately, netatalk has some serious performance issues under the > load we're putting it under and we'd like to migrate to NFS. > > We've tried several Linux distributions and various kernels and we're > now testing FreeBSD (and tested FreeNAS) with similar setups. > Unfortunately, they all suffer the same issue. > > As a test, I have a series of scripts to simulate user activity on the > clients (e.g. opening Word, opening a browser, doing some read/writes > with dd, etc). After a while, NFS on the server runs into an issue > where (what I think happens) rpc.statd can't talk to rpc.lockd. Being > Mac clients, they all get a rather ugly dialog box stating that their > connection to the server has been lost. > > It's worth mentioning that this server is a KVM 'guest' on a Linux > server. I'm aware of some I/O issues there, but I don't have a decent > piece of hardware to really test this on. I allocated 4 CPUs to it and > 10GB of RAM. I've tested with the virtio net drivers and without. > Considering I've seen the same symptoms on around 6 Linux > distributions, > with various kernels, FreeNAS, and FreeBSD, I wouldn't be surprised to > get the same results if I weren't virtualized. > > I haven't really done any tuning on the FreeBSD server, it's fairly > vanilla. > > We have around ~2600 machines throughout our campus, with limited > remote > management capabilities (that's on the big agenda to tackle), so > changing NFS mount options there would be rather difficult. These are > LDAP accounts with the NFS mounts in LDAP as well, for what it's > worth. > The clients mount it pretty vanilla (output of 'mount' on client): > freenas.dsdk12.schoollocal:/mnt/homes on > /net/freenas.dsdk12.schoollocal/mnt/homes (nfs, nodev, nosuid, > automounted, nobrowse) > Well, if you look at the mailing list archives, you'll figure out what I think about the NLM and NSM (ie. avoid using them if at all possible). If multiple clients do not do locking on the same files concurrently in such a way as they need to see each others file locks (home directories are typically accessed by one client at any time), then have the client mounts use "nolockd" (I think it's "nolock" on Linux and not sure what it's called on Mac OS X), which makes the file locking happen locally within the client. --> Then you can get rid of rpc.lockd and rpc.statd. The only other way to avoid rpc.lockd, rpc.statd is to do all the mounts as NFSv4. The Linux NFSv4 client is now stable from what I've seen and Lion shipped with a client, although I haven't had a chance to test it. (If your Macs are Snow Leopard, this isn't an option.) I can't explain why statd wouldn't be able to talk to lockd, but will note that, for these to work, they must have a reliable network connection without any firewall stuff, because rpc.statd uses IP broadcast and they typically choose port#s dynamically and use rpcbind to find out what they are. Good luck with it, rick > On the server, my /etc/exports looks like this: > /srv/homes -alldirs -network 172.30.0.0/16 > > This export doesn't have a lot of data - it's 150 small home > directories > of test accounts. No other activity is being done on this server. The > filesystem if UFS. > > /etc/rc.conf on the server: > rpcbind_enable="YES" > nfs_server_enable="YES" > mountd_flags="-r -l" > nfsd_enable="YES" > mountd_enable="YES" > rpc_lockd_enable="YES" > rpc_statd_enable="YES" > nfs_server_flags="-t -n 128" > > When this occurs, /var/log/messages starts to fill up with this: > > Mar 30 16:35:18 freefs kernel: Failed to contact local NSM - rpc error > 5 > Mar 30 16:35:20 freefs rpc.statd: unmon request from localhost, no > matching monitor > Mar 30 16:35:44 freefs rpc.statd: unmon request from localhost, no > matching monitor > -- repeated a few times every few seconds -- > Mar 30 16:54:50 freefs rpc.statd: Unsolicited notification from host > hs00508s4434.dsdk12.schoollocal > Mar 30 16:55:01 freefs rpc.statd: Unsolicited notification from host > hs00520s4539.dsdk12.schoollocal > Mar 30 16:55:10 freefs rpc.statd: Failed to call rpc.statd client at > host localhost > > nfsstat shortly after a failure: > Rpc Info: > TimedOut Invalid X Replies Retries Requests > 0 0 0 0 1208 > Cache Info: > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits > Misses > 177 951 226 28 3 6 0 > 2 > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits > Misses > 49 3 13 5 9 0 148 > 9 > > Server Info: > Getattr Setattr Lookup Readlink Read Write Create > Remove > 262698 101012 1575347 29 1924761 2172712 0 > 43792 > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus > Access > 27447 0 21 5596 1691 118073 0 > 2596146 > Mknod Fsstat Fsinfo PathConf Commit > 0 83638 108 108 183632 > Server Ret-Failed > 0 > Server Faults > 0 > Server Cache Stats: > Inprog Idem Non-idem Misses > 0 0 0 9172982 > Server Write Gathering: > WriteOps WriteRPC Opsaved > 2172712 2172712 0 > > rpcinfo shortly after a failure: > program version netid address service owner > 100000 4 tcp 0.0.0.0.0.111 rpcbind superuser > 100000 3 tcp 0.0.0.0.0.111 rpcbind superuser > 100000 2 tcp 0.0.0.0.0.111 rpcbind superuser > 100000 4 udp 0.0.0.0.0.111 rpcbind superuser > 100000 3 udp 0.0.0.0.0.111 rpcbind superuser > 100000 2 udp 0.0.0.0.0.111 rpcbind superuser > 100000 4 tcp6 ::.0.111 rpcbind superuser > 100000 3 tcp6 ::.0.111 rpcbind superuser > 100000 4 udp6 ::.0.111 rpcbind superuser > 100000 3 udp6 ::.0.111 rpcbind superuser > 100000 4 local /var/run/rpcbind.sock rpcbind superuser > 100000 3 local /var/run/rpcbind.sock rpcbind superuser > 100000 2 local /var/run/rpcbind.sock rpcbind superuser > 100005 1 udp6 ::.2.119 mountd superuser > 100005 3 udp6 ::.2.119 mountd superuser > 100005 1 tcp6 ::.2.119 mountd superuser > 100005 3 tcp6 ::.2.119 mountd superuser > 100005 1 udp 0.0.0.0.2.119 mountd superuser > 100005 3 udp 0.0.0.0.2.119 mountd superuser > 100005 1 tcp 0.0.0.0.2.119 mountd superuser > 100005 3 tcp 0.0.0.0.2.119 mountd superuser > 100024 1 udp6 ::.3.191 status superuser > 100024 1 tcp6 ::.3.191 status superuser > 100024 1 udp 0.0.0.0.3.191 status superuser > 100024 1 tcp 0.0.0.0.3.191 status superuser > 100003 2 tcp 0.0.0.0.8.1 nfs superuser > 100003 3 tcp 0.0.0.0.8.1 nfs superuser > 100003 2 tcp6 ::.8.1 nfs superuser > 100003 3 tcp6 ::.8.1 nfs superuser > 100021 0 udp6 ::.3.248 nlockmgr superuser > 100021 0 tcp6 ::.2.220 nlockmgr superuser > 100021 0 udp 0.0.0.0.3.202 nlockmgr superuser > 100021 0 tcp 0.0.0.0.2.255 nlockmgr superuser > 100021 1 udp6 ::.3.248 nlockmgr superuser > 100021 1 tcp6 ::.2.220 nlockmgr superuser > 100021 1 udp 0.0.0.0.3.202 nlockmgr superuser > 100021 1 tcp 0.0.0.0.2.255 nlockmgr superuser > 100021 3 udp6 ::.3.248 nlockmgr superuser > 100021 3 tcp6 ::.2.220 nlockmgr superuser > 100021 3 udp 0.0.0.0.3.202 nlockmgr superuser > 100021 3 tcp 0.0.0.0.2.255 nlockmgr superuser > 100021 4 udp6 ::.3.248 nlockmgr superuser > 100021 4 tcp6 ::.2.220 nlockmgr superuser > 100021 4 udp 0.0.0.0.3.202 nlockmgr superuser > 100021 4 tcp 0.0.0.0.2.255 nlockmgr superuser > 300019 1 tcp 0.0.0.0.2.185 amd superuser > 300019 1 udp 0.0.0.0.2.162 amd superuser > > The load can get fairly high during my 'stress' tests, but not *that* > high. I'm surprised to see these particular symptoms that affect every > connected user at the same time and would expect slowdowns rather than > the issue I'm seeing. > > Any ideas or nudges in the right direction are most welcome. This is > severely plaguing us and our students :\ > > Thanks, > Josh > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"