From owner-freebsd-fs@FreeBSD.ORG Sun Feb 6 23:15:12 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CAAA01065670 for ; Sun, 6 Feb 2011 23:15:12 +0000 (UTC) (envelope-from roberto@keltia.freenix.fr) Received: from keltia.net (centre.keltia.net [IPv6:2a01:240:fe5c::41]) by mx1.freebsd.org (Postfix) with ESMTP id 7E8348FC12 for ; Sun, 6 Feb 2011 23:15:12 +0000 (UTC) Received: from iqpano.freenix.org (unknown [IPv6:2a01:240:fe5c:0:d69a:20ff:fed0:3a83]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: roberto) by keltia.net (Postfix/TLS) with ESMTPSA id AAB99DE1D; Mon, 7 Feb 2011 00:15:09 +0100 (CET) Date: Mon, 7 Feb 2011 00:15:05 +0100 From: Ollivier Robert To: freebsd-fs@freebsd.org, Graham Todd Message-ID: <20110206231505.GA55764@iqpano.freenix.org> References: <20110131095752.00001957@unknown> <4D46C2E9.8000603@bellanet.org> <20110131181355.00006075@unknown> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110131181355.00006075@unknown> X-Operating-System: MacOS X / MBP 4,1 - FreeBSD 8.0 / T3500-E5520 Nehalem User-Agent: Mutt/1.5.20 (2009-06-14) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.3 (keltia.net); Mon, 07 Feb 2011 00:15:09 +0100 (CET) Cc: Subject: Re: Configuring which pool to load zfsloader from X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2011 23:15:12 -0000 According to Bruce Cran: > This is before the loader: it's the first stage (boot2?) that _loads_ > the loader. I created the pool after installation, but for some > reason the bootloader wants to load from the new pool. The "bootfs" property is supposed to handle this situation. You put it on the pool you want to boot from. See [1] for details. [1] http://www.keltia.net/howtos/zfsboot -- Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- roberto@keltia.freenix.fr In memoriam to Ondine : http://ondine.keltia.net/ From owner-freebsd-fs@FreeBSD.ORG Mon Feb 7 09:17:03 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C6A6106566B for ; Mon, 7 Feb 2011 09:17:03 +0000 (UTC) (envelope-from l.pizzamiglio@bally-wulff.de) Received: from mail2.bally-wulff-berlin.de (mail2.bally-wulff-berlin.de [212.144.118.9]) by mx1.freebsd.org (Postfix) with ESMTP id 17B718FC1B for ; Mon, 7 Feb 2011 09:17:02 +0000 (UTC) Received: from bwex.bally-wulff.de (unknown [192.9.204.106]) by mail2.bally-wulff-berlin.de (Postfix) with ESMTP id C856999078; Mon, 7 Feb 2011 10:17:00 +0100 (CET) Received: from [192.9.205.177] ([192.9.205.177]) by bwex.bally-wulff.de with Microsoft SMTPSVC(6.0.3790.4675); Mon, 7 Feb 2011 10:17:01 +0100 Message-ID: <4D4FB88A.90906@bally-wulff.de> Date: Mon, 07 Feb 2011 10:16:58 +0100 From: Luca Pizzamiglio User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.13) Gecko/20101213 Thunderbird/3.1.7 MIME-Version: 1.0 To: Gleb Kurtsou References: <4D4BE50A.7000705@bally-wulff.de> <20110204141254.1dd98de8@ernst.jennejohn.org> <20110204224856.GA78229@tops.skynet.lt> <20110205100728.46f5ca92@ernst.jennejohn.org> <20110205211405.GA7212@tops.skynet.lt> In-Reply-To: <20110205211405.GA7212@tops.skynet.lt> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 07 Feb 2011 09:17:01.0260 (UTC) FILETIME=[C71D10C0:01CBC6A7] Cc: freebsd-fs@freebsd.org Subject: Re: building kernel including GEOM_VINUM X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 09:17:03 -0000 On 02/05/2011 22:14, Gleb Kurtsou wrote: > On (05/02/2011 10:07), Gary Jennejohn wrote: >> On Sat, 5 Feb 2011 00:48:56 +0200 >> Gleb Kurtsou wrote: >>> On (04/02/2011 14:12), Gary Jennejohn wrote: >>>> It would probably require some major hacking. Apparently gvinum was >>>> designed with using it only as a KLD in mind. >>> The change it trivial, just add files listed in >>> sys/modules/geom/geom_vinum/Makefile to sys/files. >>> It was also necessary it tweak /sbin/gvinum to check if module loaded >>> during startup to eliminate useless warning (afair). >> >> For someone not used to working with the kernel I'd say that what you >> describe is major hacking. Especially the modification to gvinum. > The patch attached. Would you test if it's ok in both cases: built-in > and module. vinum(4) man page states that loading it as module is > preferred, I see no reason for it to still remain so, the text was > written in ~1999 and vinum was rewritten on top of geom afterwards. Hi Gleb! thanks a lot for the patch!! I'll start to test it in both cases! The handbook doesn't contain an error, but it's inaccurate. The vinum(4) man page, instead, contains an error: the SYNOPSIS "device vinum" is wrong. I could open a PR to fix at least the man page and then Gleb could "officially" submit its patch (after tests). I'm impressed by the quality of FreeBSD and I'd love to contribute with my 5 cents... Thanks again! > > Thanks, > Gleb. Best regards, Luca Pizzamiglio From owner-freebsd-fs@FreeBSD.ORG Mon Feb 7 11:07:00 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2DCE910656C6 for ; Mon, 7 Feb 2011 11:07:00 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 018FD8FC22 for ; Mon, 7 Feb 2011 11:07:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p17B6x0h027731 for ; Mon, 7 Feb 2011 11:06:59 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p17B6xvw027729 for freebsd-fs@FreeBSD.org; Mon, 7 Feb 2011 11:06:59 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 7 Feb 2011 11:06:59 GMT Message-Id: <201102071106.p17B6xvw027729@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 11:07:00 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 f kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153847 fs [nfs] [panic] Kernel panic from incorrect m_free in nf o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153552 fs [zfs] zfsboot from 8.2-RC1 freeze at boot time o kern/153520 fs [zfs] Boot from GPT ZFS root on HP BL460c G1 unstable o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small o kern/152488 fs [tmpfs] [patch] mtime of file updated when only inode o kern/152079 fs [msdosfs] [patch] Small cleanups from the other NetBSD o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o kern/151845 fs [smbfs] [patch] smbfs should be upgraded to support Un o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/151111 fs [zfs] vnodes leakage during zfs unmount o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/150207 fs zpool(1): zpool import -d /dev tries to open weird dev o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149022 fs [hang] File system operations hangs with suspfs state o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o bin/148296 fs [zfs] [loader] [patch] Very slow probe in /usr/src/sys o kern/148204 fs [nfs] UDP NFS causes overload o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147790 fs [zfs] zfs set acl(mode|inherit) fails on existing zfs o kern/147560 fs [zfs] [boot] Booting 8.1-PRERELEASE raidz system take o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server o kern/146375 fs [nfs] [patch] Typos in macro variables names in sys/fs s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an o bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c o kern/144458 fs [nfs] [patch] nfsd fails as a kld p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142914 fs [zfs] ZFS performance degradation over time o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142401 fs [ntfs] [patch] Minor updates to NTFS from NetBSD o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140134 fs [msdosfs] write and fsck destroy filesystem integrity o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139597 fs [patch] [tmpfs] tmpfs initializes va_gen but doesn't u o kern/139564 fs [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138790 fs [zfs] ZFS ceases caching when mem demand is high o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133614 fs [panic] panic: ffs_truncate: read-only filesystem o kern/133174 fs [msdosfs] [patch] msdosfs must support utf-encoded int o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121779 fs [ufs] snapinfo(8) (and related tools?) only work for t o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha f kern/120991 fs [panic] [ffs] [snapshot] System crashes when manipulat o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with p kern/116608 fs [msdosfs] [patch] msdosfs fails to check mount options o kern/116583 fs [ffs] [hang] System freezes for short time when using o kern/116170 fs [panic] Kernel panic when mounting /tmp o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] [iconv] mount_msdosfs: msdosfs_iconv: Operat o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/106030 fs [ufs] [panic] panic in ufs from geom when a dead disk o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o bin/94635 fs snapinfo(8)/libufs only works for disk-backed filesyst o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/33464 fs [ufs] soft update inconsistencies after system crash o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 219 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Feb 7 13:38:09 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 103111065670; Mon, 7 Feb 2011 13:38:09 +0000 (UTC) (envelope-from gleb.kurtsou@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 41C608FC2B; Mon, 7 Feb 2011 13:38:07 +0000 (UTC) Received: by eyf6 with SMTP id 6so2187805eyf.13 for ; Mon, 07 Feb 2011 05:38:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:date:from:to:cc:subject:message-id:references :mime-version:content-type:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=Gr/xYSgbW5jls1s50a5kLAtrAEN2TY0rnCdFU32OorI=; b=fm6oQ7z4s69GZLqToD9G0BqhqLI87UAMPliG6RIwb/Ho/vOxtZwfqhKx5TInSyvUDd 5pIscrNs79NayEDx+SA7Mr2J3sWC7uLkeis2f1odWASeRgCgSF91pSEiYRuQZY8/FpSK PRIPjVxEcx0zrr6owzrDBparnm8UP7qx0Cyok= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:content-transfer-encoding :in-reply-to:user-agent; b=AVmSTS8w8DAj3wbrvtsQ0UR+nvxvFIUkshpJR7EKgNuQc+agzYT5PYEz3U/2EwFoXt GqwQNrShad4mBzzKXBkOl/4QVZCHCrCKE811uoYdqP7GGvydDnG4VmNz9JZZ/h4pmZ4e Iwbxes8zfO0ZbZvm17VM+bqiezFCHT5YNbgL4= Received: by 10.103.233.16 with SMTP id k16mr10857347mur.132.1297085887121; Mon, 07 Feb 2011 05:38:07 -0800 (PST) Received: from localhost (lan-78-157-92-5.vln.skynet.lt [78.157.92.5]) by mx.google.com with ESMTPS id a6sm819372fak.3.2011.02.07.05.38.05 (version=SSLv3 cipher=OTHER); Mon, 07 Feb 2011 05:38:06 -0800 (PST) Date: Mon, 7 Feb 2011 15:37:48 +0200 From: Gleb Kurtsou To: Ivan Voras Message-ID: <20110207133748.GA16327@tops.skynet.lt> References: <4D36A2CF.1080508@fsn.hu> <20110119084648.GA28278@icarus.home.lan> <4D36B85B.8070201@fsn.hu> <20110119150200.GY2518@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: tmpfs is zero bytes (no free space), maybe a zfs bug? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 13:38:09 -0000 On (19/01/2011 17:27), Ivan Voras wrote: > On 19 January 2011 16:02, Kostik Belousov wrote: > > >> http://people.freebsd.org/~ivoras/diffs/tmpfs.h.patch > >> > >> I don't think this is a complete solution but it's a start. If you can, > >> try it and see if it helps. > > This is not a start, and actually a step in the wrong direction. > > Tmpfs is wrong now, but the patch would make the wrongness even bigger. > > > > Issue is that the current tmpfs calculation should not depend on the > > length of the inactive queue or the amount of free pages. This data only > > measures  the pressure on the pagedaemon, and has absolutely no relation > > to the amount of data that can be put into anonymous objects before the > > system comes out of swap. > > > > vm_lowmem handler is invoked in two situations: > > - when KVA cannot satisfy the request for the space allocation; > > - when pagedaemon have to start the scan. > > None of the situations has any direct correlation with the fact that > > tmpfs needs to check, that is "Is there enough swap to keep all my > > future anonymous memory requests ?". > > > > Might be, swap reservation numbers can be useful to the tmpfs reporting. > > Also might be, tmpfs should reserve the swap explicitely on start, instead > > of making attempts to guess how much can be allocated at random moment. > > Thank you for your explanation! I'm still not very familiar with VM > and VFS. Could you also read my report at > http://www.mail-archive.com/freebsd-current@freebsd.org/msg126491.html > ? I'm curious about the fact that there is lots of 'free' memory here > in the same situation. > > Do you think that there is something which can be done as a band-aid > without a major modification to tmpfs? It's up to user to mount tmpfs filesystems of reasonable size to prevent resource exhaustion. Anyway, enormously large tmpfs killing all your process is not the way to go. Unless there are objections, I'm planning to do the following: 1. By default set tmpfs size to max(all swap/2, all memory/2) and print warning that filesystem size should be specified manually. Max(swap/2,mem/2) is used as a band-aid for the case when no swap is setup. 3. Remove "live" filesystem size checks, i.e. do not depend on free/inact memory. 2. Add support for resizing tmpfs on the fly: mount -u -o size= /tmpfs Reserving swap for tmpfs might not be what user expects: generally I use tmpfs for work dir for building ports, it's unused most of the time. btw, what linux and opensolaris do when available mem/swap gets low due to tmpfs and how filesystem size determined at real-time? Thanks, Gleb. From owner-freebsd-fs@FreeBSD.ORG Mon Feb 7 14:36:18 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B2A18106566C; Mon, 7 Feb 2011 14:36:18 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 533A38FC16; Mon, 7 Feb 2011 14:36:17 +0000 (UTC) Received: by gxk8 with SMTP id 8so1800504gxk.13 for ; Mon, 07 Feb 2011 06:36:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:from :date:x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=rmxLzAOtmwo26WcUYyMy8B5BD+SN/mbYkxoyjrU940I=; b=JGu4HVCUzuwIVcY0CCF8W/AsvxgT4qts2jmEb3c7Ymm96GclxvgUqqrBf0E3WU1SRB B03GTJOwI993I51OYfAQIXuHgpGvRm8ULZRwL+qUHfhrUl0FcBRrZbWAXbluy0h6xuMl /WoydmmVfvS/YDRljSXmQbead1tUVi9Aswg2U= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding; b=lTMispOnfVAD71Xe4+QozDpg8z0SFrAIyOa4XuJ62GEXZEifRd3scSqNz1/VJo8DzK Dx7HWReGwHTAERgYVD/ujzj8rdf8E2w1uYoqxRMNNFBo8ROgDHTfZsdjpEminNUWn7CP qh54zrHdHqDRzjNsoGxfXzlWTzYqlEvy8XJdw= Received: by 10.90.89.11 with SMTP id m11mr19712995agb.18.1297089377455; Mon, 07 Feb 2011 06:36:17 -0800 (PST) MIME-Version: 1.0 Sender: ivoras@gmail.com Received: by 10.151.11.12 with HTTP; Mon, 7 Feb 2011 06:35:37 -0800 (PST) In-Reply-To: <20110207133748.GA16327@tops.skynet.lt> References: <4D36A2CF.1080508@fsn.hu> <20110119084648.GA28278@icarus.home.lan> <4D36B85B.8070201@fsn.hu> <20110119150200.GY2518@deviant.kiev.zoral.com.ua> <20110207133748.GA16327@tops.skynet.lt> From: Ivan Voras Date: Mon, 7 Feb 2011 15:35:37 +0100 X-Google-Sender-Auth: YneYLcrX7-KEG3tpy4EENC0bbgQ Message-ID: To: Gleb Kurtsou Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: tmpfs is zero bytes (no free space), maybe a zfs bug? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 14:36:18 -0000 On 7 February 2011 14:37, Gleb Kurtsou wrote: > It's up to user to mount tmpfs filesystems of reasonable size to prevent > resource exhaustion. Anyway, enormously large tmpfs killing all your > process is not the way to go. Of course not, but as I see it (from admin perspective), tmpfs should behave as close to regular processes in consuming memory as possible (where possible; obviously it cannot be subject to the OOM killer :) ). The problem described in this thread is that there is enough memory in various lists and tmpfs still reports "0 bytes free". See my message: the machine had more than 8 GB of "free" memory (reported by "top") and still "0 bytes free" in tmpfs - and that's not counting inactive and other forms of used memory which could be freed or swapped out (and also not counting swap). By "as close to regular processes in consuming memory" I mean that I would expect tmpfs to allocate from the same total pool of memory as processes and be subject to the same mechanisms of VM, including swap. If that is not possible, I would (again, as an admin) like to extend the tmpfs(5) man page and other documentation with information about what types of memory will and will not count towards available to tmpfs. > Unless there are objections, I'm planning to do the following: > > 1. By default set tmpfs size to max(all swap/2, all memory/2) and print > warning that filesystem size should be specified manually. > Max(swap/2,mem/2) is used as a band-aid for the case when no swap is setu= p. You mean as a reservation, maximum limit or something else? If a tmpfs with "size" of e.g. 16 GB is configured, will the memory be preallocated? wired? I don't think there should be default hard size limits to tmpfs - it should be able to hold sudden bursts of large temp files (using swap if needed), but that could be achieved by configuring a tmpfs whose size is RAM+swap if the memory is not preallocated so not a big problem. > 3. Remove "live" filesystem size checks, i.e. do not depend on > free/inact memory. I'm for it, if it's possible in the light of #1 > 2. Add support for resizing tmpfs on the fly: > =C2=A0 =C2=A0 =C2=A0 =C2=A0mount -u -o size=3D /tmpfs ditto. > Reserving swap for tmpfs might not be what user expects: generally I use > tmpfs for work dir for building ports, it's unused most of the time. It looks like we think the opposite of it :) I would like it to be swapped out if needed, making room for running processes etc. as regular VM paging algorithms decide. Of course, if that could be controlled with a flag we'd both be happy :) > btw, what linux and opensolaris do when available mem/swap gets low due > to tmpfs and how filesystem size determined at real-time? There's some information here: http://en.wikipedia.org/wiki/Tmpfs From owner-freebsd-fs@FreeBSD.ORG Mon Feb 7 15:29:25 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CE565106566C; Mon, 7 Feb 2011 15:29:25 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id A2A548FC16; Mon, 7 Feb 2011 15:29:25 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 61E3446B09; Mon, 7 Feb 2011 10:29:25 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.10]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 85F3B8A01D; Mon, 7 Feb 2011 10:29:24 -0500 (EST) From: John Baldwin To: Doug Barton Date: Mon, 7 Feb 2011 10:17:42 -0500 User-Agent: KMail/1.13.5 (FreeBSD/7.4-CBSD-20110107; KDE/4.4.5; amd64; ; ) References: <4D47B954.3010600@FreeBSD.org> <201102040759.51736.jhb@freebsd.org> <4D4C6DF4.2060004@FreeBSD.org> In-Reply-To: <4D4C6DF4.2060004@FreeBSD.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201102071017.42616.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Mon, 07 Feb 2011 10:29:24 -0500 (EST) X-Virus-Scanned: clamav-milter 0.96.3 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=0.5 required=4.2 tests=BAYES_00,MAY_BE_FORGED, RDNS_DYNAMIC autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on bigwig.baldwin.cx Cc: freebsd-fs@freebsd.org Subject: Re: ext2fs crash in -current (r218056) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 15:29:25 -0000 On Friday, February 04, 2011 4:21:56 pm Doug Barton wrote: > On 02/04/2011 04:59, John Baldwin wrote: > > On Thursday, February 03, 2011 6:06:32 pm Doug Barton wrote: > >> On 02/02/2011 14:04, John Baldwin wrote: > >>> Err, this is a different panic than what you reported earlier. Your disk died > >>> and spewed a bunch of EIO errors. I can look at the locking assertion failure > >>> tomorrow, but this is a differnt issue. Even UFS needed a good bit of work to > >>> handle disks dying gracefully. > >> > >> Can you defined "died" a bit? :-/ I just plugged it back in and it > >> seems to be working Ok, but it's my backup disk so if I'm looking at a > >> potential failure I'm a bit worried ... > > > > It's hard to say as the other errors have already scrolled off the screen. > > There weren't any other errors. I just trimmed the jpeg down to a more > manageable level. No other messages at all before the 'da0: autosense failed' line? -- John Baldwin From owner-freebsd-fs@FreeBSD.ORG Mon Feb 7 15:33:39 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 14A9B106566B for ; Mon, 7 Feb 2011 15:33:39 +0000 (UTC) (envelope-from drb@karlov.mff.cuni.cz) Received: from mail.karlov.mff.cuni.cz (mail.karlov.mff.cuni.cz [195.113.27.114]) by mx1.freebsd.org (Postfix) with ESMTP id CABDE8FC1B for ; Mon, 7 Feb 2011 15:33:38 +0000 (UTC) Received: from nat-r.karlov.mff.cuni.cz ([195.113.27.73] helo=[10.32.82.27]) by mail.karlov.mff.cuni.cz with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.68 (FreeBSD)) (envelope-from ) id 1PmScF-000MYM-Ma for freebsd-fs@freebsd.org; Mon, 07 Feb 2011 16:03:00 +0100 Message-ID: <4D50099C.1030401@karlov.mff.cuni.cz> Date: Mon, 07 Feb 2011 16:02:52 +0100 From: =?UTF-8?B?VG9tw6HFoSBEcmJvaGxhdg==?= User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: NFSv4 ACLs on NFS4 mount ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 15:33:39 -0000 Hello, I have 8.1-RELEASE running ZFS and NFS4 experimantal server and I wonder, what is the status of NFS4 ACLs? I see them work on ZFS fileystems directly mounted but when mounted via NFS4 (even on the same machine), getfacl returns 'old' POSIX form instead of NFSv4 ACLs. By digging mailng list archive I got to know that this 'should' work but for me it doesn't. Is it me setting something wrong or is that an unimplemented stuff? Thanks a lot (do not hesitate to point to some man pages digging or what ever... I am just stuck with this), Tomas Drbohlav From owner-freebsd-fs@FreeBSD.ORG Mon Feb 7 15:34:30 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4CEE3106566B; Mon, 7 Feb 2011 15:34:30 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (unknown [IPv6:2a01:348:0:15:5d59:5c40:0:1]) by mx1.freebsd.org (Postfix) with ESMTP id 6DF748FC1D; Mon, 7 Feb 2011 15:34:29 +0000 (UTC) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id 52A28E76AD; Mon, 7 Feb 2011 15:34:22 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cran.org.uk; h=date:from :to:cc:subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=mail; bh=0/ezqAX4I5Bk 80Ig9wd2WycWVio=; b=lFzpogA/U13tM60r6bll9H8yCcZewYtR45/+YYKoUwga crW5UUSVbfW/Ux14dgGRd6eAip0ayO/YLzd27EsxaAP8uGwBZqkPRIyBndHfFUZ+ GFFCvhGiq0AV+gKh8X07Qr/RSMSmGJoshIeQ9OVZeMyHgp7Y0eVluJjqjrNwgn8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cran.org.uk; h=date:from:to :cc:subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=mail; b=f/kSNi laIpGzXFQhMZrOVO1DW3w4dT/dbTqInn5djEZZ0wxg08W9GMl68Q4Y6omh1V+jvN CLdf6EQB9c670OO5uWZU/+md8sgHiF9kAK+CcD4KqCaaS2QXF3RptL8ZbIshh47J bmQKeokyCp3sDM2t8W+0HRNNfuM0TtjjzlFEM= Received: from unknown (client-86-23-50-224.brhm.adsl.virginmedia.com [86.23.50.224]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id 093FAE7695; Mon, 7 Feb 2011 15:34:22 +0000 (GMT) Date: Mon, 7 Feb 2011 15:34:19 +0000 From: Bruce Cran To: John Baldwin Message-ID: <20110207153419.00003583@unknown> In-Reply-To: <201102071017.42616.jhb@freebsd.org> References: <4D47B954.3010600@FreeBSD.org> <201102040759.51736.jhb@freebsd.org> <4D4C6DF4.2060004@FreeBSD.org> <201102071017.42616.jhb@freebsd.org> X-Mailer: Claws Mail 3.7.8cvs9 (GTK+ 2.16.6; i586-pc-mingw32msvc) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ext2fs crash in -current (r218056) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 15:34:30 -0000 On Mon, 7 Feb 2011 10:17:42 -0500 John Baldwin wrote: > No other messages at all before the 'da0: autosense failed' line? I've seen this with msdosfs: a memory stick I'm using just stops working after a while with the "autosense failed" error - and nothing before it. I'm wondering what debug sysctls to use to see what's going wrong. -- Bruce Cran From owner-freebsd-fs@FreeBSD.ORG Mon Feb 7 15:44:27 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A3D541065670; Mon, 7 Feb 2011 15:44:27 +0000 (UTC) (envelope-from gleb.kurtsou@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id C94C68FC0A; Mon, 7 Feb 2011 15:44:26 +0000 (UTC) Received: by bwz12 with SMTP id 12so4961089bwz.13 for ; Mon, 07 Feb 2011 07:44:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:date:from:to:cc:subject:message-id:references :mime-version:content-type:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=IfXnB0x7Z/1fRIbu2fAzfG6yeefBDobpgX5DbG4BEmQ=; b=NFukoqj1s0L83nEhFc+FmSoJHA6SRx7Wfotbd60avEybV1rVQ5MAyhtcPwUNvkWBRC rWFVXl6+I+SxziAyFmJvu1jHN495KVSYg77x03dldVHcSSkWY8auZVRaLkW3M96ram5V aQE4yhjxe56xXAhvv/7tIRBqTq4GXmtUmNQaU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:content-transfer-encoding :in-reply-to:user-agent; b=qtRojSjuZgMXMebDFLd2zT/q+rDyDMw3+o3K6lIFY0rCkPPabTOAvqgemC90DOkCAq loAdbWyRMqxf569KgoVttAaB8FsMi5s2/XYQWTXkIUmuXrV4NnmidF2+1IwK6XqQmWQV WcQ4UmVjePH3oIs25AUnAs9CThKyLUbqr6H2Y= Received: by 10.204.16.138 with SMTP id o10mr24739bka.157.1297093465620; Mon, 07 Feb 2011 07:44:25 -0800 (PST) Received: from localhost (lan-78-157-92-5.vln.skynet.lt [78.157.92.5]) by mx.google.com with ESMTPS id u23sm2114791bkw.21.2011.02.07.07.44.24 (version=SSLv3 cipher=RC4-MD5); Mon, 07 Feb 2011 07:44:24 -0800 (PST) Date: Mon, 7 Feb 2011 17:44:06 +0200 From: Gleb Kurtsou To: Ivan Voras Message-ID: <20110207154406.GA28877@tops.skynet.lt> References: <4D36A2CF.1080508@fsn.hu> <20110119084648.GA28278@icarus.home.lan> <4D36B85B.8070201@fsn.hu> <20110119150200.GY2518@deviant.kiev.zoral.com.ua> <20110207133748.GA16327@tops.skynet.lt> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: tmpfs is zero bytes (no free space), maybe a zfs bug? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 15:44:27 -0000 On (07/02/2011 15:35), Ivan Voras wrote: > On 7 February 2011 14:37, Gleb Kurtsou wrote: > > > It's up to user to mount tmpfs filesystems of reasonable size to prevent > > resource exhaustion. Anyway, enormously large tmpfs killing all your > > process is not the way to go. > > Of course not, but as I see it (from admin perspective), tmpfs should > behave as close to regular processes in consuming memory as possible > (where possible; obviously it cannot be subject to the OOM killer :) > ). Here is key difference it's not subject to be killed by OOM killer. Thus exhausting all resource for real. I propose to enforce specifying upper limit of filesystem size by user. > The problem described in this thread is that there is enough memory in > various lists and tmpfs still reports "0 bytes free". See my message: > the machine had more than 8 GB of "free" memory (reported by "top") > and still "0 bytes free" in tmpfs - and that's not counting inactive > and other forms of used memory which could be freed or swapped out > (and also not counting swap). That's because tmpfs incorrectly checks how much memory is available including both swap and ram. In VM world that's not so easy. > By "as close to regular processes in consuming memory" I mean that I > would expect tmpfs to allocate from the same total pool of memory as > processes and be subject to the same mechanisms of VM, including swap. > If that is not possible, I would (again, as an admin) like to extend > the tmpfs(5) man page and other documentation with information about > what types of memory will and will not count towards available to > tmpfs. > > > Unless there are objections, I'm planning to do the following: > > > > 1. By default set tmpfs size to max(all swap/2, all memory/2) and print > > warning that filesystem size should be specified manually. > > Max(swap/2,mem/2) is used as a band-aid for the case when no swap is setup. > > You mean as a reservation, maximum limit or something else? If a tmpfs > with "size" of e.g. 16 GB is configured, will the memory be > preallocated? wired? Memory in tmpfs is allocated/freed as needed, there is no preallocated or reserved memory/swap. It already behaves the way you've described. I'm against preallocating or reserving memory. There is ramfs in linux that does preallocation, but it looks deprecated. > I don't think there should be default hard size limits to tmpfs - it > should be able to hold sudden bursts of large temp files (using swap > if needed), but that could be achieved by configuring a tmpfs whose > size is RAM+swap if the memory is not preallocated so not a big > problem. But there is one in Linux (Documentation/filesystems/tmpfs.txt): 59 size: The limit of allocated bytes for this tmpfs instance. The 60 default is half of your physical RAM without swap. If you 61 oversize your tmpfs instances the machine will deadlock 62 since the OOM handler will not be able to free that memory. That's actually what I've proposed: size=mem/2 vs size=max(mem/2,swap/2) Limit should be there not to panic the system. > > 3. Remove "live" filesystem size checks, i.e. do not depend on > > free/inact memory. > > I'm for it, if it's possible in the light of #1 > > > 2. Add support for resizing tmpfs on the fly: > >        mount -u -o size= /tmpfs > > ditto. It's trivial. If it can be resized, change maxsize in struct, fail otherwise. > > Reserving swap for tmpfs might not be what user expects: generally I use > > tmpfs for work dir for building ports, it's unused most of the time. > > It looks like we think the opposite of it :) I would like it to be > swapped out if needed, making room for running processes etc. as > regular VM paging algorithms decide. Of course, if that could be > controlled with a flag we'd both be happy :) Perhaps there is a bit of misunderstanding, it will be swapped out, will behave exactly as it does now, but will have sane default filesystem size limit by default, will change semantics of calculating available memory: I want it to try hard to allocate memory unless filesystem limit is hit, failing only if there is clearly memory shortage (as I said, it is for user to properly configure it). > > btw, what linux and opensolaris do when available mem/swap gets low due > > to tmpfs and how filesystem size determined at real-time? > > There's some information here: http://en.wikipedia.org/wiki/Tmpfs From owner-freebsd-fs@FreeBSD.ORG Mon Feb 7 19:15:41 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 142D1106566B for ; Mon, 7 Feb 2011 19:15:41 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from mail2.fluidhosting.com (mx22.fluidhosting.com [204.14.89.5]) by mx1.freebsd.org (Postfix) with ESMTP id 9A7418FC13 for ; Mon, 7 Feb 2011 19:15:40 +0000 (UTC) Received: (qmail 1197 invoked by uid 399); 7 Feb 2011 19:15:39 -0000 Received: from localhost (HELO doug-optiplex.ka9q.net) (dougb@dougbarton.us@127.0.0.1) by localhost with ESMTPAM; 7 Feb 2011 19:15:39 -0000 X-Originating-IP: 127.0.0.1 X-Sender: dougb@dougbarton.us Message-ID: <4D5044DA.7000406@FreeBSD.org> Date: Mon, 07 Feb 2011 11:15:38 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.13) Gecko/20110129 Thunderbird/3.1.7 MIME-Version: 1.0 To: Bruce Cran References: <4D47B954.3010600@FreeBSD.org> <201102040759.51736.jhb@freebsd.org> <4D4C6DF4.2060004@FreeBSD.org> <201102071017.42616.jhb@freebsd.org> <20110207153419.00003583@unknown> In-Reply-To: <20110207153419.00003583@unknown> X-Enigmail-Version: 1.1.2 OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ext2fs crash in -current (r218056) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 19:15:41 -0000 On 02/07/2011 07:34, Bruce Cran wrote: > On Mon, 7 Feb 2011 10:17:42 -0500 > John Baldwin wrote: > >> No other messages at all before the 'da0: autosense failed' line? Right. > I've seen this with msdosfs: a memory stick I'm using just stops > working after a while with the "autosense failed" error - and nothing > before it. I'm wondering what debug sysctls to use to see what's going > wrong. I was copying from a fat32 partition to an ext2fs partition on the same USB drive, so this fits. Meanwhile, I finally had a chance to stress test John's locking patch. It successfully passed a make -j2 buildworld/buildkernel with src and obj on the same partition, and a ports build of libreoffice (and its several dozen deps) with ports and WRKDIRPREFIX on the same partition, part of which overlapped with the world build. Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-fs@FreeBSD.ORG Mon Feb 7 20:15:10 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 199CC1065674 for ; Mon, 7 Feb 2011 20:15:10 +0000 (UTC) (envelope-from mlmichael70@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id A3BA48FC1E for ; Mon, 7 Feb 2011 20:15:09 +0000 (UTC) Received: by wwf26 with SMTP id 26so5035515wwf.31 for ; Mon, 07 Feb 2011 12:15:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:message-id:date:from:user-agent:mime-version:to :subject:content-type:content-transfer-encoding; bh=3+lMQtVofegPbqku2OSE4MlwM2DCC7UBQkdVxxFEeyE=; b=Ov8aUerN8zOgMSJAoJcq5BU12s7W27B2l5q8/mMsMXHRW96ATgIscu37Ht00K0vUx9 9bk0ySDThb0HhujAZBo9sprT0LURtoeuQv5jWo0JEjxxEO4MsIBXiHWS2gLe0WEkGCFx 7A3lwZmWNwft/QAmYzudRTsMMmXHF7riWmLVY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; b=HrQHgUTPO+4Gb+dgGu5lMxqv0Eb8aCbwTYZYG6PIdRNvSHs+ZsDL+/LpgmkB5YXwZp 5T1JElEsh8acptts6XztfF86fn/+ApSuwknOebEp9UCaynr4BzpcfyV8dIjMvk4wuWGc iGAIIGY2jp/5du0Ox2wmLXXj/uYJwCLNE5W08= Received: by 10.227.145.208 with SMTP id e16mr14241225wbv.5.1297108358398; Mon, 07 Feb 2011 11:52:38 -0800 (PST) Received: from prime.nonspace ([82.132.211.58]) by mx.google.com with ESMTPS id x1sm3661542wbh.2.2011.02.07.11.52.37 (version=SSLv3 cipher=RC4-MD5); Mon, 07 Feb 2011 11:52:38 -0800 (PST) Message-ID: <4D504D96.2040305@gmail.com> Date: Mon, 07 Feb 2011 19:52:54 +0000 From: Michael User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.13) Gecko/20101215 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: chflags (uappnd) on ZFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 20:15:10 -0000 Hello, Is uappnd flag supported on ZFS? I'm using 8.1-R and when I try to: chflags uappnd file.txt then I get: chflags: file.txt: Operation not supported Thank you in advance, Michael From owner-freebsd-fs@FreeBSD.ORG Mon Feb 7 21:41:57 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 12C14106564A; Mon, 7 Feb 2011 21:41:57 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id A46938FC0C; Mon, 7 Feb 2011 21:41:56 +0000 (UTC) Received: by qwj9 with SMTP id 9so3704484qwj.13 for ; Mon, 07 Feb 2011 13:41:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=a2fQ5Z0WFJOEDLFGi7lo3BW1yHav4jLl4AyWOZ32Nd8=; b=lq9pUdKAh6RX/sY1li9qjq2FOQvwRec6xOtIsqWMTf27x+hQX0t5q+8YCF01eaS2Hv hjfrymiyl0DluhuxM/SY6KiPnd/MNjx0CaZDsh1ElF1j/r86bEwLqQU/94hJCYqXXkr1 /iAzWp0Q43yIxUMIuBD24YCIKDmcWgwmr3nVg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=rHfqtFi9eYPD34q90Zhzvo46AyJZEuitM5N0ADvALF/E3bpYWPQvFHOiKb9yIAXauw PWMusqeEWm7cuC65Pg7H20/aFE1ZB1FOy5FwxSOGNR/jBCivm5HVZzX0k9c1tCftK+fk e3yG3Fpde99y2BC+9HMQpVZ+Tf3GJSeh5MgoM= MIME-Version: 1.0 Received: by 10.224.73.132 with SMTP id q4mr15039443qaj.62.1297114915831; Mon, 07 Feb 2011 13:41:55 -0800 (PST) Received: by 10.220.85.66 with HTTP; Mon, 7 Feb 2011 13:41:55 -0800 (PST) In-Reply-To: <4D46D0CF.8090103@chreo.net> References: <4D0A09AF.3040005@FreeBSD.org> <4D46D0CF.8090103@chreo.net> Date: Mon, 7 Feb 2011 16:41:55 -0500 Message-ID: From: Rich To: Chreo Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 21:41:57 -0000 Hi world, I've tested with ZFSv28 on RELENG_8 using the patch http://people.freebsd.org/~mm/patches/zfs/v28/releng-8.2-zfsv28-20110204.patch.xz . I can reproducibly panic it using the following sequence of three steps: # dd if=/dev/zero of=sparse_file bs=1M count=2000000 # zpool create testv28 raidz1 ad20 ad26 `pwd`/sparse_file # zpool offline testv28 `pwd`/sparse_file At step 3, I panic every time. I am attempting to get a crash dump, but for whatever reason, the dump appears to be hanging instead of making any progress. - Rich From owner-freebsd-fs@FreeBSD.ORG Mon Feb 7 21:48:07 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5F7841065672; Mon, 7 Feb 2011 21:48:07 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: from mail-qy0-f175.google.com (mail-qy0-f175.google.com [209.85.216.175]) by mx1.freebsd.org (Postfix) with ESMTP id EB2E48FC08; Mon, 7 Feb 2011 21:48:06 +0000 (UTC) Received: by qyk8 with SMTP id 8so1564604qyk.13 for ; Mon, 07 Feb 2011 13:48:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=eSKbqa+tD/sKThsxhftSt6T3jo99AoK4PYJ/uu7z7VI=; b=eG6BzQow6SpIe4KbybrwCFSzmMcTndk0I4hMcDNq3UFJw+Q9j+ADk7i6DtNeFWQrP0 /gsP16etVEGXBO433maf/DlrD4Upsy/+N4hjzBzXvPDhY/6yXZYCH4w1LSolk4vlQHtQ OfBGmrGCtqB03F3CxDh6lLKrvDgttDdfdQPPI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=XD5Xtue7xkzeFdjHwnc72eXJ3tQuh6coa247aXJl/15jkjATj5twVGmjVFr70ou5V9 Hd9ZVmUFCbVYoD3PVT7GeU+G41DlI33sS9mipxIZwICdWq21v0tTV4nUPHA/JffJ1BVe NPc1EZjPbbgbjsqdtRtkkIswVFMF/SN6VPeyE= MIME-Version: 1.0 Received: by 10.224.89.85 with SMTP id d21mr14919335qam.162.1297115285656; Mon, 07 Feb 2011 13:48:05 -0800 (PST) Received: by 10.220.85.66 with HTTP; Mon, 7 Feb 2011 13:48:05 -0800 (PST) In-Reply-To: References: <4D0A09AF.3040005@FreeBSD.org> <4D46D0CF.8090103@chreo.net> Date: Mon, 7 Feb 2011 16:48:05 -0500 Message-ID: From: Rich To: Chreo Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 21:48:07 -0000 My apologies, the actual command is count=1 seek=2000000, but that doesn't end up changing the observed behavior even if I had a 2T disk that I was using for the flat file. - Rich On Mon, Feb 7, 2011 at 4:41 PM, Rich wrote: > Hi world, > I've tested with ZFSv28 on RELENG_8 using the patch > http://people.freebsd.org/~mm/patches/zfs/v28/releng-8.2-zfsv28-20110204.patch.xz > . > > I can reproducibly panic it using the following sequence of three steps: > # dd if=/dev/zero of=sparse_file bs=1M count=2000000 > # zpool create testv28 raidz1 ad20 ad26 `pwd`/sparse_file > # zpool offline testv28 `pwd`/sparse_file > > At step 3, I panic every time. I am attempting to get a crash dump, but for > whatever reason, the dump appears to be hanging instead of making any > progress. > > - Rich > From owner-freebsd-fs@FreeBSD.ORG Mon Feb 7 21:56:38 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E5AA11065670 for ; Mon, 7 Feb 2011 21:56:38 +0000 (UTC) (envelope-from gpm@hotplug.ru) Received: from shadow.rusoft.ru (shadow.rusoft.ru [217.16.18.75]) by mx1.freebsd.org (Postfix) with ESMTP id 8D5688FC0A for ; Mon, 7 Feb 2011 21:56:37 +0000 (UTC) Received: from shadow.rusoft.ru (localhost [127.0.0.1]) by shadow.rusoft.ru (Postfix) with ESMTP id 1A0B42A56 for ; Tue, 8 Feb 2011 00:37:11 +0300 (MSK) Received: from jazz.rusoft.ru (unknown [83.222.3.162]) by shadow.rusoft.ru (Postfix) with ESMTP id 005F92A55 for ; Tue, 8 Feb 2011 00:37:10 +0300 (MSK) Received: from ghost-pc.home.lan (ghostmaster.static.corbina.ru [78.107.250.255]) by jazz.rusoft.ru (Postfix) with ESMTPA id E5BB41CC00F for ; Tue, 8 Feb 2011 00:37:09 +0300 (MSK) Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: freebsd-fs@freebsd.org References: Date: Tue, 08 Feb 2011 00:37:04 +0300 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Emil Muratov" Message-ID: In-Reply-To: User-Agent: Opera Mail/11.01 (Win32) X-Virus-Scanned: ClamAV using ClamSMTP Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 21:56:39 -0000 > For the past few weeks, I noticed that the amount of memory reported in > top > (sum of active, inact, wired, cache buf and free) keeps decreasing as the > uptime increases. I can't pinpoint to when I first noticed this, as I > have > updated the system a few times just in case this has been fixed. Yes, I have the same issue on my home file storage. My system is 8.1 amd64, 2G ram, zfs on root raidz with 4x1,5T drives. After updating to stable a couple of days ago I noticed that the system leaks memory very fast. Checking here and there I found that the issue concerns sendfile (yep, again!). How to reproduce: Configure samba with aio and sendfile (mine is version 3.5.6) smb.conf [global] use sendfile=true aio read size = 16384 Download a couple of large samba shared files (8-10 gigs). While downloading files I can see that memory decreazes to nowhere very-very fast, several MBs per second! First it drains free mem, than active and inactive, than comes wired until the whole system commits suicide suffocating itself to the death. The only way to free memory is to reboot the system. I can't unload zfs module like PJD suggested to do, 'cause my root is on zfs :( I'll try to make a bootable flash and move root to the flash to try to unload module and what will happen. Everything was OK in stable before the new year, sendfile used to pump free and wired memory to inactive than slowly reclaiming it back. But it seems something was changed after NY holydays? From owner-freebsd-fs@FreeBSD.ORG Mon Feb 7 22:59:12 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6FEA2106566B for ; Mon, 7 Feb 2011 22:59:12 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 26E418FC0C for ; Mon, 7 Feb 2011 22:59:11 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApwEAAMIUE2DaFvO/2dsb2JhbACEG6F9qR6QJYEngz12BIR6hm0 X-IronPort-AV: E=Sophos;i="4.60,439,1291611600"; d="scan'208";a="110035298" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 07 Feb 2011 17:59:10 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id DC432B3F36; Mon, 7 Feb 2011 17:59:10 -0500 (EST) Date: Mon, 7 Feb 2011 17:59:10 -0500 (EST) From: Rick Macklem To: Andrey Simonenko Message-ID: <511655722.1552410.1297119550835.JavaMail.root@erie.cs.uoguelph.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE8 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org, Mickael Canevet Subject: Re: "rpc mount export: RPC: Can't decode result" when export list is to long X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 22:59:12 -0000 > This this the update (this is the minimal version, without optimization): > > --- svc_vc.c.orig 2009-08-03 11:13:06.000000000 +0300 > +++ svc_vc.c 2011-01-31 11:31:28.000000000 +0200 > @@ -546,7 +546,7 @@ write_vc(xprtp, buf, len) > cd->strm_stat = XPRT_DIED; > return (-1); > } > - if (cd->nonblock && i != cnt) { > + if (cd->nonblock) { > /* > * For non-blocking connections, do not > * take more than 2 seconds writing the > @@ -560,6 +560,7 @@ write_vc(xprtp, buf, len) > return (-1); > } > } > + i = 0; > } > } First off, good catch Andrey. Second, sorry for taking soo long to respond. I'm in a location where my internet access is limited and I missed this message on the first pass through my email. I'm somewhat familiar with the kernel RPC, where the RPC messages are all in one record, which means they're limited to the size set by svc_vc_create(). I looked and see that the userland RPC library does split large messages into multiple records over TCP, so the limitation doesn't exist for TCP for userland. (ie. What I said before only applies to the UDP case and not the TCP one.) I won't be able to commit the above fix until April (due to the limited internet access before then), but will commit it then. Thanks for the fix. W.r.t. The "brief" case, I don't know, but my guess was it was done to help make things work for UDP clients, which are limited to 8800 byte RPC messages. (That limit could be increased tojust under 64K, but since the message ends up being IP fragmented, it's not a great plan, imho.) I don't know if there are any "showmount" commands out there still using UDP only. rick ps: Mickael, did you have a chance to try Andrey's patch to see if it fixed your Linux automount problem? From owner-freebsd-fs@FreeBSD.ORG Mon Feb 7 23:08:28 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 51CFE106566C for ; Mon, 7 Feb 2011 23:08:28 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 0F1688FC20 for ; Mon, 7 Feb 2011 23:08:27 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApwEAGgKUE2DaFvO/2dsb2JhbACEG6F+qR2QI4Engz12BIR6hm0 X-IronPort-AV: E=Sophos;i="4.60,439,1291611600"; d="scan'208";a="110036169" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 07 Feb 2011 18:08:27 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 79F67B3F23; Mon, 7 Feb 2011 18:08:27 -0500 (EST) Date: Mon, 7 Feb 2011 18:08:27 -0500 (EST) From: Rick Macklem To: =?utf-8?Q?Tom=C3=A1=C5=A1_Drbohlav?= Message-ID: <1174528658.1552842.1297120107416.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <4D50099C.1030401@karlov.mff.cuni.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE8 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org Subject: Re: NFSv4 ACLs on NFS4 mount ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 23:08:28 -0000 > Hello, > > I have 8.1-RELEASE running ZFS and NFS4 experimantal server and I > wonder, what is the status of NFS4 ACLs? I see them work on ZFS > fileystems directly mounted but when mounted via NFS4 (even on the > same > machine), getfacl returns 'old' POSIX form instead of NFSv4 ACLs. By > digging mailng list archive I got to know that this 'should' work but > for me it doesn't. > Thanks to George spotting it recently, they should work if you apply the ACL related patches found at: http://people.freebsd.org/~rmacklem For 8.1/ZFS, you'll see that there is one for the client side (adds VOP_PATHCONF() support to the client), plus one for the server that is in head, stable/8 but not 8.1. (It replaces a VOP_ACCESS() with a VOP_ACCESSX().) Also, although I think that 8.1 has everything else you need except the two patches, I'll admit that I haven't tested anything that is pre-8.2 with these patches. (ie. You might also need an 8.2 upgrade, but hopefully not.) Good luck with them, rick From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 09:42:26 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26130106566B for ; Tue, 8 Feb 2011 09:42:26 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id B19538FC16 for ; Tue, 8 Feb 2011 09:42:25 +0000 (UTC) Received: by bwz12 with SMTP id 12so5747802bwz.13 for ; Tue, 08 Feb 2011 01:42:24 -0800 (PST) Received: by 10.204.128.200 with SMTP id l8mr6366378bks.60.1297156738460; Tue, 08 Feb 2011 01:18:58 -0800 (PST) Received: from dfleuriot.local (tui75-3-88-168-239-38.fbx.proxad.net [88.168.239.38]) by mx.google.com with ESMTPS id v1sm2586695bkt.5.2011.02.08.01.18.56 (version=SSLv3 cipher=RC4-MD5); Tue, 08 Feb 2011 01:18:57 -0800 (PST) Message-ID: <4D510A7F.3070708@my.gd> Date: Tue, 08 Feb 2011 10:18:55 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4D504D96.2040305@gmail.com> In-Reply-To: <4D504D96.2040305@gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Subject: Re: chflags (uappnd) on ZFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 09:42:26 -0000 Hi Michael, list, Getting the very same error on 8.2-RC3 amd64 FreeBSD mybsd 8.2-RC3 FreeBSD 8.2-RC3 #1: Thu Feb 3 11:03:48 CET 2011 root@mybsd:/usr/obj/usr/src/sys/DAM amd64 mybsd# zpool get version data NAME PROPERTY VALUE SOURCE data version 15 default mybsd# zfs get version NAME PROPERTY VALUE SOURCE data version 4 On 2/7/11 8:52 PM, Michael wrote: > Hello, > > Is uappnd flag supported on ZFS? I'm using 8.1-R and when I try to: > chflags uappnd file.txt > then I get: > chflags: file.txt: Operation not supported > > Thank you in advance, > Michael > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 09:58:25 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 67790106564A for ; Tue, 8 Feb 2011 09:58:25 +0000 (UTC) (envelope-from admin@kkip.pl) Received: from mainframe.kkip.pl (kkip.pl [87.105.164.78]) by mx1.freebsd.org (Postfix) with ESMTP id 17DAE8FC1D for ; Tue, 8 Feb 2011 09:58:24 +0000 (UTC) Received: from static-78-8-144-74.ssp.dialog.net.pl ([78.8.144.74] helo=[192.168.0.2]) by mainframe.kkip.pl with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.73 (FreeBSD)) (envelope-from ) id 1Pmjny-000Nrw-P0; Tue, 08 Feb 2011 10:24:22 +0100 Message-ID: <4D510BBB.1060708@kkip.pl> Date: Tue, 08 Feb 2011 10:24:11 +0100 From: Bartosz Stec User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: Emil Muratov References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Authenticated-User: admin@kkip.pl X-Authenticator: plain X-Sender-Verify: SUCCEEDED (sender exists & accepts mail) X-Spam-Score: -8.5 X-Spam-Score-Int: -84 X-Exim-Version: 4.73 (build at 10-Jan-2011 16:29:01) X-Date: 2011-02-08 10:24:22 X-Connected-IP: 78.8.144.74:54351 X-Message-Linecount: 66 X-Body-Linecount: 53 X-Message-Size: 2711 X-Body-Size: 2099 X-Received-Count: 1 X-Recipient-Count: 3 X-Local-Recipient-Count: 3 X-Local-Recipient-Defer-Count: 0 X-Local-Recipient-Fail-Count: 0 Cc: freebsd-fs@freebsd.org, pjd@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 09:58:25 -0000 W dniu 2011-02-07 22:37, Emil Muratov pisze: > >> For the past few weeks, I noticed that the amount of memory reported >> in top >> (sum of active, inact, wired, cache buf and free) keeps decreasing as >> the >> uptime increases. I can't pinpoint to when I first noticed this, as I >> have >> updated the system a few times just in case this has been fixed. > > Yes, I have the same issue on my home file storage. My system is 8.1 > amd64, 2G ram, zfs on root raidz with 4x1,5T drives. > After updating to stable a couple of days ago I noticed that the > system leaks memory very fast. Checking here and there I found that > the issue concerns sendfile (yep, again!). > > How to reproduce: > Configure samba with aio and sendfile (mine is version 3.5.6) > > smb.conf > [global] > use sendfile=true > aio read size = 16384 > > Download a couple of large samba shared files (8-10 gigs). > > > While downloading files I can see that memory decreazes to nowhere > very-very fast, several MBs per second! First it drains free mem, than > active and inactive, than comes wired until the whole system commits > suicide suffocating itself to the death. > The only way to free memory is to reboot the system. I can't unload > zfs module like PJD suggested to do, 'cause my root is on zfs :( > I'll try to make a bootable flash and move root to the flash to try to > unload module and what will happen. > > Everything was OK in stable before the new year, sendfile used to pump > free and wired memory to inactive than slowly reclaiming it back. But > it seems something was changed after NY holydays? I'm glad someone else finally picked that problem, so there's appareantly no memory-eating ghost in my machine ;) Here's my thread on stable list about this issue: http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061247.html And in fact, PC reported in thread above is also SAMBA server with aio/sendfile enabled and ZFS. I would be happy testing some patches if necessary, because until now I need to monitor memory and reboot this server before it dies. -- Bartosz Stec From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 10:12:51 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EDAFB106564A for ; Tue, 8 Feb 2011 10:12:51 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta03.westchester.pa.mail.comcast.net (qmta03.westchester.pa.mail.comcast.net [76.96.62.32]) by mx1.freebsd.org (Postfix) with ESMTP id 9202A8FC0A for ; Tue, 8 Feb 2011 10:12:51 +0000 (UTC) Received: from omta03.westchester.pa.mail.comcast.net ([76.96.62.27]) by qmta03.westchester.pa.mail.comcast.net with comcast id 5NAq1g0010bG4ec53NCrLy; Tue, 08 Feb 2011 10:12:51 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta03.westchester.pa.mail.comcast.net with comcast id 5NCq1g00f2tehsa3PNCrzm; Tue, 08 Feb 2011 10:12:51 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 198459B422; Tue, 8 Feb 2011 02:12:49 -0800 (PST) Date: Tue, 8 Feb 2011 02:12:49 -0800 From: Jeremy Chadwick To: Damien Fleuriot Message-ID: <20110208101249.GA8057@icarus.home.lan> References: <4D504D96.2040305@gmail.com> <4D510A7F.3070708@my.gd> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D510A7F.3070708@my.gd> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: chflags (uappnd) on ZFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 10:12:52 -0000 On Tue, Feb 08, 2011 at 10:18:55AM +0100, Damien Fleuriot wrote: > Getting the very same error on 8.2-RC3 amd64 > > FreeBSD mybsd 8.2-RC3 FreeBSD 8.2-RC3 #1: Thu Feb 3 11:03:48 CET 2011 > root@mybsd:/usr/obj/usr/src/sys/DAM amd64 > > mybsd# zpool get version data > NAME PROPERTY VALUE SOURCE > data version 15 default > > mybsd# zfs get version > NAME PROPERTY VALUE SOURCE > data version 4 > > > On 2/7/11 8:52 PM, Michael wrote: > > Hello, > > > > Is uappnd flag supported on ZFS? I'm using 8.1-R and when I try to: > > chflags uappnd file.txt > > then I get: > > chflags: file.txt: Operation not supported It looks like the implemented/translated chflags(2) bits are: SF_IMMUTABLE ("chflags schg") <--> ZFS_IMMUTABLE SF_APPEND ("chflags sappnd") <--> ZFS_APPENDONLY SF_NOUNLINK ("chflags sunlnk") <--> ZFS_NOUNLINK UF_NODUMP ("chflags nodump") <--> ZFS_NODUMP This is based on "grep -r FLAG_CHANGE /usr/src/sys/cddl", and reading src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zfs_znode.h. I assume that one can use the inverse operations (clearing bits) as well. Please see the chflags(2) man page (don't skim please) for details of what the difference is between SF_xxx and UF_xxx flags. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 10:15:51 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7326D106566B for ; Tue, 8 Feb 2011 10:15:51 +0000 (UTC) (envelope-from simon@comsys.ntu-kpi.kiev.ua) Received: from comsys.kpi.ua (comsys.kpi.ua [77.47.192.42]) by mx1.freebsd.org (Postfix) with ESMTP id 27F428FC08 for ; Tue, 8 Feb 2011 10:15:51 +0000 (UTC) Received: from pm513-1.comsys.kpi.ua ([10.18.52.101] helo=pm513-1.comsys.ntu-kpi.kiev.ua) by comsys.kpi.ua with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1PmkbF-00074L-Ad; Tue, 08 Feb 2011 12:15:09 +0200 Received: by pm513-1.comsys.ntu-kpi.kiev.ua (Postfix, from userid 1001) id F3F9B1CC1E; Tue, 8 Feb 2011 12:15:48 +0200 (EET) Date: Tue, 8 Feb 2011 12:15:48 +0200 From: Andrey Simonenko To: Rick Macklem Message-ID: <20110208101548.GA7747@pm513-1.comsys.ntu-kpi.kiev.ua> References: <511655722.1552410.1297119550835.JavaMail.root@erie.cs.uoguelph.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <511655722.1552410.1297119550835.JavaMail.root@erie.cs.uoguelph.ca> User-Agent: Mutt/1.5.21 (2010-09-15) X-Authenticated-User: simon@comsys.ntu-kpi.kiev.ua X-Authenticator: plain X-Sender-Verify: SUCCEEDED (sender exists & accepts mail) X-Exim-Version: 4.63 (build at 10-Dec-2010 16:36:10) X-Date: 2011-02-08 12:15:09 X-Connected-IP: 10.18.52.101:24413 X-Message-Linecount: 67 X-Body-Linecount: 51 X-Message-Size: 2854 X-Body-Size: 2093 Cc: freebsd-fs@freebsd.org, Mickael Canevet Subject: Re: "rpc mount export: RPC: Can't decode result" when export list is to long X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 10:15:51 -0000 On Mon, Feb 07, 2011 at 05:59:10PM -0500, Rick Macklem wrote: > > This this the update (this is the minimal version, without optimization): > > > > --- svc_vc.c.orig 2009-08-03 11:13:06.000000000 +0300 > > +++ svc_vc.c 2011-01-31 11:31:28.000000000 +0200 > > @@ -546,7 +546,7 @@ write_vc(xprtp, buf, len) > > cd->strm_stat = XPRT_DIED; > > return (-1); > > } > > - if (cd->nonblock && i != cnt) { > > + if (cd->nonblock) { > > /* > > * For non-blocking connections, do not > > * take more than 2 seconds writing the > > @@ -560,6 +560,7 @@ write_vc(xprtp, buf, len) > > return (-1); > > } > > } > > + i = 0; > > } > > } > > First off, good catch Andrey. > > Second, sorry for taking soo long to respond. I'm in a location where my internet > access is limited and I missed this message on the first pass through my email. > > I'm somewhat familiar with the kernel RPC, where the RPC messages are all in one > record, which means they're limited to the size set by svc_vc_create(). I looked > and see that the userland RPC library does split large messages into multiple > records over TCP, so the limitation doesn't exist for TCP for userland. (ie. What > I said before only applies to the UDP case and not the TCP one.) > > I won't be able to commit the above fix until April (due to the limited internet > access before then), but will commit it then. Thanks for the fix. I created PR about this mistake: http://www.freebsd.org/cgi/query-pr.cgi?pr=154505 > > W.r.t. The "brief" case, I don't know, but my guess was it was done to help > make things work for UDP clients, which are limited to 8800 byte RPC messages. > (That limit could be increased tojust under 64K, but since the message ends up > being IP fragmented, it's not a great plan, imho.) I don't know if there are > any "showmount" commands out there still using UDP only. This change was made to support some automounter that uses UDP, but such behaviour for MOUNT EXPORTS request is not mentioned in RFC1813: http://www.freebsd.org/cgi/query-pr.cgi?pr=26320 From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 10:27:31 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 521F6106566C for ; Tue, 8 Feb 2011 10:27:31 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA11.westchester.pa.mail.comcast.net (qmta11.westchester.pa.mail.comcast.net [76.96.59.211]) by mx1.freebsd.org (Postfix) with ESMTP id F1E058FC16 for ; Tue, 8 Feb 2011 10:27:30 +0000 (UTC) Received: from omta08.westchester.pa.mail.comcast.net ([76.96.62.12]) by QMTA11.westchester.pa.mail.comcast.net with comcast id 5NSW1g0020Fqzac5BNTWqX; Tue, 08 Feb 2011 10:27:30 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta08.westchester.pa.mail.comcast.net with comcast id 5NTV1g00B2tehsa3UNTVCX; Tue, 08 Feb 2011 10:27:30 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id D225D9B422; Tue, 8 Feb 2011 02:27:27 -0800 (PST) Date: Tue, 8 Feb 2011 02:27:27 -0800 From: Jeremy Chadwick To: Bartosz Stec Message-ID: <20110208102727.GA8555@icarus.home.lan> References: <4D510BBB.1060708@kkip.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D510BBB.1060708@kkip.pl> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org, pjd@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 10:27:31 -0000 On Tue, Feb 08, 2011 at 10:24:11AM +0100, Bartosz Stec wrote: > W dniu 2011-02-07 22:37, Emil Muratov pisze: > > > >>For the past few weeks, I noticed that the amount of memory > >>reported in top > >>(sum of active, inact, wired, cache buf and free) keeps > >>decreasing as the > >>uptime increases. I can't pinpoint to when I first noticed this, > >>as I have > >>updated the system a few times just in case this has been fixed. > > > >Yes, I have the same issue on my home file storage. My system is > >8.1 amd64, 2G ram, zfs on root raidz with 4x1,5T drives. > >After updating to stable a couple of days ago I noticed that the > >system leaks memory very fast. Checking here and there I found > >that the issue concerns sendfile (yep, again!). > > > >How to reproduce: > >Configure samba with aio and sendfile (mine is version 3.5.6) > > > >smb.conf > >[global] > >use sendfile=true > >aio read size = 16384 > > > >Download a couple of large samba shared files (8-10 gigs). > > > > > >While downloading files I can see that memory decreazes to nowhere > >very-very fast, several MBs per second! First it drains free mem, > >than active and inactive, than comes wired until the whole system > >commits suicide suffocating itself to the death. > >The only way to free memory is to reboot the system. I can't > >unload zfs module like PJD suggested to do, 'cause my root is on > >zfs :( > >I'll try to make a bootable flash and move root to the flash to > >try to unload module and what will happen. > > > >Everything was OK in stable before the new year, sendfile used to > >pump free and wired memory to inactive than slowly reclaiming it > >back. But it seems something was changed after NY holydays? > > I'm glad someone else finally picked that problem, so there's > appareantly no memory-eating ghost in my machine ;) > Here's my thread on stable list about this issue: > http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061247.html > > And in fact, PC reported in thread above is also SAMBA server with > aio/sendfile enabled and ZFS. > > I would be happy testing some patches if necessary, because until > now I need to monitor memory and reboot this server before it dies. The source and build date of your kernel will matter greatly here. I can't speak about the memory utilisation aspect, but I tend to disable sendfile everywhere possible when ZFS is in use on a system. The reason is based on something I and another user experienced back in October 2010 pertaining to sendfile() on ZFS locking up processes (making them unkillable). See here[1] for details; this problem has since been fixed[2] (look for commits around October). You'll also find some commits that went through in November pertaining to ZFS and sendfile. This is why I said the date of your kernel/sources matters. :-) The issue I referenced in [1] is not related to memory utilisation, but does indicate use of sendfile with ZFS may be a bad idea (by this I mean, there may be aspects of its implementation when mixed with ZFS that have been overlooked). Simple test: if you disable use of sendfile (but not AIO) in Samba, does the problem go away? Comparatively, Apache out-of-the-box defaults to mmap() and sendfile() being disabled. BSD ftpd, however, is known to use sendfile exclusively. [1]: http://lists.freebsd.org/pipermail/freebsd-fs/2010-October/thread.html#9729 [2]: http://www.freshbsd.org/?branch=RELENG_8&project=freebsd&committer=&module=src&q=sendfile -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 10:48:19 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CE8801065724; Tue, 8 Feb 2011 10:48:19 +0000 (UTC) (envelope-from admin@kkip.pl) Received: from mainframe.kkip.pl (kkip.pl [87.105.164.78]) by mx1.freebsd.org (Postfix) with ESMTP id 672188FC0C; Tue, 8 Feb 2011 10:48:18 +0000 (UTC) Received: from static-78-8-144-74.ssp.dialog.net.pl ([78.8.144.74] helo=[192.168.0.2]) by mainframe.kkip.pl with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.73 (FreeBSD)) (envelope-from ) id 1Pml7A-000OLY-VB; Tue, 08 Feb 2011 11:48:16 +0100 Message-ID: <4D511F65.2050503@kkip.pl> Date: Tue, 08 Feb 2011 11:48:05 +0100 From: Bartosz Stec User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jeremy Chadwick References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> In-Reply-To: <20110208102727.GA8555@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Authenticated-User: admin@kkip.pl X-Authenticator: plain X-Sender-Verify: SUCCEEDED (sender exists & accepts mail) X-Spam-Score: -8.5 X-Spam-Score-Int: -84 X-Exim-Version: 4.73 (build at 10-Jan-2011 16:29:01) X-Date: 2011-02-08 11:48:16 X-Connected-IP: 78.8.144.74:52059 X-Message-Linecount: 95 X-Body-Linecount: 82 X-Message-Size: 4465 X-Body-Size: 3766 X-Received-Count: 1 X-Recipient-Count: 3 X-Local-Recipient-Count: 3 X-Local-Recipient-Defer-Count: 0 X-Local-Recipient-Fail-Count: 0 Cc: freebsd-fs@freebsd.org, pjd@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 10:48:20 -0000 W dniu 2011-02-08 11:27, Jeremy Chadwick pisze: > On Tue, Feb 08, 2011 at 10:24:11AM +0100, Bartosz Stec wrote: >> W dniu 2011-02-07 22:37, Emil Muratov pisze: >>>> For the past few weeks, I noticed that the amount of memory >>>> reported in top >>>> (sum of active, inact, wired, cache buf and free) keeps >>>> decreasing as the >>>> uptime increases. I can't pinpoint to when I first noticed this, >>>> as I have >>>> updated the system a few times just in case this has been fixed. >>> Yes, I have the same issue on my home file storage. My system is >>> 8.1 amd64, 2G ram, zfs on root raidz with 4x1,5T drives. >>> After updating to stable a couple of days ago I noticed that the >>> system leaks memory very fast. Checking here and there I found >>> that the issue concerns sendfile (yep, again!). >>> >>> How to reproduce: >>> Configure samba with aio and sendfile (mine is version 3.5.6) >>> >>> smb.conf >>> [global] >>> use sendfile=true >>> aio read size = 16384 >>> >>> Download a couple of large samba shared files (8-10 gigs). >>> >>> >>> While downloading files I can see that memory decreazes to nowhere >>> very-very fast, several MBs per second! First it drains free mem, >>> than active and inactive, than comes wired until the whole system >>> commits suicide suffocating itself to the death. >>> The only way to free memory is to reboot the system. I can't >>> unload zfs module like PJD suggested to do, 'cause my root is on >>> zfs :( >>> I'll try to make a bootable flash and move root to the flash to >>> try to unload module and what will happen. >>> >>> Everything was OK in stable before the new year, sendfile used to >>> pump free and wired memory to inactive than slowly reclaiming it >>> back. But it seems something was changed after NY holydays? >> I'm glad someone else finally picked that problem, so there's >> appareantly no memory-eating ghost in my machine ;) >> Here's my thread on stable list about this issue: >> http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061247.html >> >> And in fact, PC reported in thread above is also SAMBA server with >> aio/sendfile enabled and ZFS. >> >> I would be happy testing some patches if necessary, because until >> now I need to monitor memory and reboot this server before it dies. > The source and build date of your kernel will matter greatly here. > > I can't speak about the memory utilisation aspect, but I tend to disable > sendfile everywhere possible when ZFS is in use on a system. The reason > is based on something I and another user experienced back in October > 2010 pertaining to sendfile() on ZFS locking up processes (making them > unkillable). See here[1] for details; this problem has since been > fixed[2] (look for commits around October). You'll also find some > commits that went through in November pertaining to ZFS and sendfile. > This is why I said the date of your kernel/sources matters. :-) I tried rebuild since original thread, hoping that problem is fixed already, so now it's very fresh: 8.2-PRERELEASE #18: Sun Feb 6 03:04:47 CET 2011. Problem is still here: Mem: 37M Active, 78M Inact, 1154M Wired, 64M Cache, 199M Buf, 40M Free About 1373MB instead of 2GB, and it's not even 2 days of uptime. > The issue I referenced in [1] is not related to memory utilisation, but > does indicate use of sendfile with ZFS may be a bad idea (by this I > mean, there may be aspects of its implementation when mixed with ZFS > that have been overlooked). > > Simple test: if you disable use of sendfile (but not AIO) in Samba, does > the problem go away? I've just disabled sendfile in smb.conf and I'll report in about 2 days, after reboot which I will perform tonight. I hope it won't hit samba performance too much ;) -- Bartosz Stec From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 11:00:28 2011 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 314FA106566C for ; Tue, 8 Feb 2011 11:00:28 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 06A828FC17 for ; Tue, 8 Feb 2011 11:00:28 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p18B0RWT096454 for ; Tue, 8 Feb 2011 11:00:27 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p18B0RFq096430; Tue, 8 Feb 2011 11:00:27 GMT (envelope-from gnats) Date: Tue, 8 Feb 2011 11:00:27 GMT Message-Id: <201102081100.p18B0RFq096430@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Carl Cc: Subject: Re: kern/154228: [md] md getting stuck in wdrain state X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Carl List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 11:00:28 -0000 The following reply was made to PR kern/154228; it has been noted by GNATS. From: Carl To: bug-followup@FreeBSD.org, k0802647@telus.net Cc: Subject: Re: kern/154228: [md] md getting stuck in wdrain state Date: Tue, 08 Feb 2011 02:59:41 -0800 For whatever reason I was not copied on the patch message, despite being the bug reporter. The explanation for that patch is more than a little obscure. In simpler terms, what have you uncovered? Does that patch implement a complete fix, partial fix, a workaround, or what? Is it recommended I try it? Did someone manage to reproduce my problem scenario? Yesterday I ran into the same bug. Similar but different exercise. Again on a remote production system. I had no choice but to try again, so I repeated the procedure, only using a non-sparse file instead. It hung yet again, so that should rule out sparse files as part of the problem. I noticed in the mdconfig(8) man page this description for the "-o [no]async" option: 'For vnode backed devices: avoid IO_SYNC for increased performance but at the risk of deadlocking the entire kernel.' It seems to me the default would be "-o noasync" and that this is supposed to avoid that particular risk for deadlock, but what command can I use to verify whether a particular enabled memory disk is actually using IO_SYNC or not? Carl / K0802647 From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 11:12:36 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 236311065674 for ; Tue, 8 Feb 2011 11:12:36 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta10.emeryville.ca.mail.comcast.net (qmta10.emeryville.ca.mail.comcast.net [76.96.30.17]) by mx1.freebsd.org (Postfix) with ESMTP id 04ABC8FC14 for ; Tue, 8 Feb 2011 11:12:35 +0000 (UTC) Received: from omta22.emeryville.ca.mail.comcast.net ([76.96.30.89]) by qmta10.emeryville.ca.mail.comcast.net with comcast id 5PCa1g0011vN32cAAPCbeL; Tue, 08 Feb 2011 11:12:35 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta22.emeryville.ca.mail.comcast.net with comcast id 5PCZ1g00P2tehsa8iPCaVK; Tue, 08 Feb 2011 11:12:35 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id B644A9B422; Tue, 8 Feb 2011 03:12:33 -0800 (PST) Date: Tue, 8 Feb 2011 03:12:33 -0800 From: Jeremy Chadwick To: Bartosz Stec Message-ID: <20110208111233.GA10895@icarus.home.lan> References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110208102727.GA8555@icarus.home.lan> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org, pjd@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 11:12:36 -0000 On Tue, Feb 08, 2011 at 02:27:27AM -0800, Jeremy Chadwick wrote: > Comparatively, Apache out-of-the-box defaults to mmap() and sendfile() > being disabled. BSD ftpd, however, is known to use sendfile > exclusively. Tom Evans pointed out to me that this is not the case -- Apache defaults to using "EnableMMAP On" and "EnableSendfile On", despite the commented-out defaults in the stock httpd.conf file saying "Off". So to clarify: I was wrong; disabling sendfile is something you have to do administratively. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 11:23:30 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5F0A610656C2 for ; Tue, 8 Feb 2011 11:23:30 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from mail-qy0-f175.google.com (mail-qy0-f175.google.com [209.85.216.175]) by mx1.freebsd.org (Postfix) with ESMTP id 142B88FC1F for ; Tue, 8 Feb 2011 11:23:29 +0000 (UTC) Received: by qyk8 with SMTP id 8so242926qyk.13 for ; Tue, 08 Feb 2011 03:23:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type:content-transfer-encoding; bh=cBbADovv+9irDSs3MLg0qIZ2QXJo69DzfGeBEDLNpYs=; b=BkqAgGg5TcLeX43V9OhCl4GWR5IQDnCKKCzE17spTB1t7+gOWg6dapenoNvZ09sTZE cY7xbYsE0lVn7OIPGxHxAAcvVYo01QQHZtN3g6/ixt0sAw93tAhcQnvYO8AZDOGEUAmW 346mkUkeQqBWJz0CMyc1ckCRQVVCMkG+kzVNc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=teNXoCurcZ7wE4GEGQmioO7pp9JWHwNKm1THqSotnjiM815VxM6LF5XwLww8nKcl1a 1TVnhGITyXNUeeGzkE/UfXiGEfvc3EgVT06zkj9CoJz0wxWRiFpT+0gxWcjAWqYJu1D0 0i2xVc+7ilx8YmymOUsQwV41rLdkFZNMbTJUA= MIME-Version: 1.0 Received: by 10.229.249.78 with SMTP id mj14mr10240432qcb.25.1297162612921; Tue, 08 Feb 2011 02:56:52 -0800 (PST) Received: by 10.229.246.8 with HTTP; Tue, 8 Feb 2011 02:56:52 -0800 (PST) In-Reply-To: <20110208102727.GA8555@icarus.home.lan> References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> Date: Tue, 8 Feb 2011 10:56:52 +0000 Message-ID: From: Tom Evans To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 11:23:30 -0000 On Tue, Feb 8, 2011 at 10:27 AM, Jeremy Chadwick wrote: > Comparatively, Apache out-of-the-box defaults to mmap() and sendfile() > being disabled. =C2=A0BSD ftpd, however, is known to use sendfile > exclusively. > No, it doesn't. http://httpd.apache.org/docs/2.2/mod/core.html#enablesendfile Cheers Tom From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 12:36:43 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 163C91065672; Tue, 8 Feb 2011 12:36:43 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (mail.vx.sk [IPv6:2a01:4f8:100:1043::3]) by mx1.freebsd.org (Postfix) with ESMTP id 398C88FC13; Tue, 8 Feb 2011 12:36:42 +0000 (UTC) Received: from core.vx.sk (localhost [127.0.0.1]) by mail.vx.sk (Postfix) with ESMTP id 5FC70137605; Tue, 8 Feb 2011 13:36:41 +0100 (CET) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk ([127.0.0.1]) by core.vx.sk (mail.vx.sk [127.0.0.1]) (amavisd-new, port 10024) with LMTP id ZOBDmnhKrXuR; Tue, 8 Feb 2011 13:36:39 +0100 (CET) Received: from [10.0.3.144] (188-167-50-235.dynamic.chello.sk [188.167.50.235]) by mail.vx.sk (Postfix) with ESMTPSA id 1B99F1375FB; Tue, 8 Feb 2011 13:36:37 +0100 (CET) Message-ID: <4D5138D6.6020007@FreeBSD.org> Date: Tue, 08 Feb 2011 13:36:38 +0100 From: Martin Matuska User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jeremy Chadwick References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> In-Reply-To: <20110208102727.GA8555@icarus.home.lan> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org, pjd@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 12:36:43 -0000 I am using ZFS and sendfile(2) with apache, lighttpd, nginx and on FreeBSD 8.2-RC (v15 or with ZFS v28 patch). The systems concerned are quite busy. E.g. lighttpd servers delivering 30-50 Megabytes per second (half a gigabit) or samba servers delivering to hundreds of users. Since FreeBSD 8.1, really a lot of bugfixes have been ported to ZFS including very important sendfile(2) fixes and I highly recommend upgrading to 8.2 or 8-STABLE. As of v28, I am using the patch for 8-STABLE on several production servers. When 8.2-RELEASE is out, I plan to put a v28 patch + v28 binary upgrade kit on my webpage. As of the memory usage shown by top(1), I have noticed that memory used by some programs using IPC shared memory (e.g. postgresql) do not show up in the counts. After stopping the program the count is "fixed" again. Dňa 08.02.2011 11:27, Jeremy Chadwick wrote / napísal(a): > On Tue, Feb 08, 2011 at 10:24:11AM +0100, Bartosz Stec wrote: >> W dniu 2011-02-07 22:37, Emil Muratov pisze: >>>> For the past few weeks, I noticed that the amount of memory >>>> reported in top >>>> (sum of active, inact, wired, cache buf and free) keeps >>>> decreasing as the >>>> uptime increases. I can't pinpoint to when I first noticed this, >>>> as I have >>>> updated the system a few times just in case this has been fixed. >>> Yes, I have the same issue on my home file storage. My system is >>> 8.1 amd64, 2G ram, zfs on root raidz with 4x1,5T drives. >>> After updating to stable a couple of days ago I noticed that the >>> system leaks memory very fast. Checking here and there I found >>> that the issue concerns sendfile (yep, again!). >>> >>> How to reproduce: >>> Configure samba with aio and sendfile (mine is version 3.5.6) >>> >>> smb.conf >>> [global] >>> use sendfile=true >>> aio read size = 16384 >>> >>> Download a couple of large samba shared files (8-10 gigs). >>> >>> >>> While downloading files I can see that memory decreazes to nowhere >>> very-very fast, several MBs per second! First it drains free mem, >>> than active and inactive, than comes wired until the whole system >>> commits suicide suffocating itself to the death. >>> The only way to free memory is to reboot the system. I can't >>> unload zfs module like PJD suggested to do, 'cause my root is on >>> zfs :( >>> I'll try to make a bootable flash and move root to the flash to >>> try to unload module and what will happen. >>> >>> Everything was OK in stable before the new year, sendfile used to >>> pump free and wired memory to inactive than slowly reclaiming it >>> back. But it seems something was changed after NY holydays? >> I'm glad someone else finally picked that problem, so there's >> appareantly no memory-eating ghost in my machine ;) >> Here's my thread on stable list about this issue: >> http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061247.html >> >> And in fact, PC reported in thread above is also SAMBA server with >> aio/sendfile enabled and ZFS. >> >> I would be happy testing some patches if necessary, because until >> now I need to monitor memory and reboot this server before it dies. > The source and build date of your kernel will matter greatly here. > > I can't speak about the memory utilisation aspect, but I tend to disable > sendfile everywhere possible when ZFS is in use on a system. The reason > is based on something I and another user experienced back in October > 2010 pertaining to sendfile() on ZFS locking up processes (making them > unkillable). See here[1] for details; this problem has since been > fixed[2] (look for commits around October). You'll also find some > commits that went through in November pertaining to ZFS and sendfile. > This is why I said the date of your kernel/sources matters. :-) > > The issue I referenced in [1] is not related to memory utilisation, but > does indicate use of sendfile with ZFS may be a bad idea (by this I > mean, there may be aspects of its implementation when mixed with ZFS > that have been overlooked). > > Simple test: if you disable use of sendfile (but not AIO) in Samba, does > the problem go away? > > Comparatively, Apache out-of-the-box defaults to mmap() and sendfile() > being disabled. BSD ftpd, however, is known to use sendfile > exclusively. > > [1]: http://lists.freebsd.org/pipermail/freebsd-fs/2010-October/thread.html#9729 > [2]: http://www.freshbsd.org/?branch=RELENG_8&project=freebsd&committer=&module=src&q=sendfile > From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 18:44:18 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3ED25106564A for ; Tue, 8 Feb 2011 18:44:18 +0000 (UTC) (envelope-from admin@kkip.pl) Received: from mainframe.kkip.pl (kkip.pl [87.105.164.78]) by mx1.freebsd.org (Postfix) with ESMTP id E32428FC13 for ; Tue, 8 Feb 2011 18:44:17 +0000 (UTC) Received: from static-78-8-144-74.ssp.dialog.net.pl ([78.8.144.74] helo=[192.168.0.2]) by mainframe.kkip.pl with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.73 (FreeBSD)) (envelope-from ) id 1PmsXp-0000t7-7y; Tue, 08 Feb 2011 19:44:16 +0100 Message-ID: <4D518EF6.7000408@kkip.pl> Date: Tue, 08 Feb 2011 19:44:06 +0100 From: Bartosz Stec User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: Martin Matuska References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> <4D5138D6.6020007@FreeBSD.org> In-Reply-To: <4D5138D6.6020007@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Authenticated-User: admin@kkip.pl X-Authenticator: plain X-Sender-Verify: SUCCEEDED (sender exists & accepts mail) X-Spam-Score: -8.5 X-Spam-Score-Int: -84 X-Exim-Version: 4.73 (build at 10-Jan-2011 16:29:01) X-Date: 2011-02-08 19:44:16 X-Connected-IP: 78.8.144.74:59634 X-Message-Linecount: 42 X-Body-Linecount: 29 X-Message-Size: 1949 X-Body-Size: 1261 X-Received-Count: 1 X-Recipient-Count: 2 X-Local-Recipient-Count: 2 X-Local-Recipient-Defer-Count: 0 X-Local-Recipient-Fail-Count: 0 Cc: freebsd-fs@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 18:44:18 -0000 W dniu 2011-02-08 13:36, Martin Matuska pisze: > I am using ZFS and sendfile(2) with apache, lighttpd, nginx and on > FreeBSD 8.2-RC (v15 or with ZFS v28 patch). The systems concerned are > quite busy. > E.g. lighttpd servers delivering 30-50 Megabytes per second (half a > gigabit) or samba servers delivering to hundreds of users. > > Since FreeBSD 8.1, really a lot of bugfixes have been ported to ZFS > including very important sendfile(2) fixes and I highly recommend > upgrading to 8.2 or 8-STABLE. > As of v28, I am using the patch for 8-STABLE on several production servers. > > When 8.2-RELEASE is out, I plan to put a v28 patch + v28 binary upgrade > kit on my webpage. > > As of the memory usage shown by top(1), I have noticed that memory used > by some programs using IPC shared memory (e.g. postgresql) do not show > up in the counts. > After stopping the program the count is "fixed" again. > This machine was rebuilded from fresh sources two days ago, so whatever this bug is, it must be quite new (probably appeared in january 2011). By the way, I wrote (very) simple script for myself, quite helpful with leakage monitoring. Maybe you'll find it useful too if you're experiencing same issue: http://pastebin.com/sQUyQbmm -- Bartosz Stec From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 19:55:20 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE525106566B; Tue, 8 Feb 2011 19:55:19 +0000 (UTC) (envelope-from admin@kkip.pl) Received: from mainframe.kkip.pl (kkip.pl [87.105.164.78]) by mx1.freebsd.org (Postfix) with ESMTP id 3BE4C8FC15; Tue, 8 Feb 2011 19:55:18 +0000 (UTC) Received: from static-78-8-144-74.ssp.dialog.net.pl ([78.8.144.74] helo=[192.168.0.2]) by mainframe.kkip.pl with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.73 (FreeBSD)) (envelope-from ) id 1PmteU-0001Cs-ON; Tue, 08 Feb 2011 20:55:17 +0100 Message-ID: <4D519F97.2000805@kkip.pl> Date: Tue, 08 Feb 2011 20:55:03 +0100 From: Bartosz Stec User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> <4D511F65.2050503@kkip.pl> In-Reply-To: <4D511F65.2050503@kkip.pl> X-Authenticated-User: admin@kkip.pl X-Authenticator: plain X-Sender-Verify: SUCCEEDED (sender exists & accepts mail) X-Spam-Score: -8.0 X-Spam-Score-Int: -79 X-Exim-Version: 4.73 (build at 10-Jan-2011 16:29:01) X-Date: 2011-02-08 20:55:17 X-Connected-IP: 78.8.144.74:62227 X-Message-Linecount: 495 X-Body-Linecount: 482 X-Message-Size: 16494 X-Body-Size: 15788 X-Received-Count: 1 X-Recipient-Count: 3 X-Local-Recipient-Count: 3 X-Local-Recipient-Defer-Count: 0 X-Local-Recipient-Fail-Count: 0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org, pjd@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 19:55:20 -0000 W dniu 2011-02-08 11:48, Bartosz Stec pisze: > W dniu 2011-02-08 11:27, Jeremy Chadwick pisze: >> On Tue, Feb 08, 2011 at 10:24:11AM +0100, Bartosz Stec wrote: >>> W dniu 2011-02-07 22:37, Emil Muratov pisze: >>>>> For the past few weeks, I noticed that the amount of memory >>>>> reported in top >>>>> (sum of active, inact, wired, cache buf and free) keeps >>>>> decreasing as the >>>>> uptime increases. I can't pinpoint to when I first noticed this, >>>>> as I have >>>>> updated the system a few times just in case this has been fixed. >>>> Yes, I have the same issue on my home file storage. My system is >>>> 8.1 amd64, 2G ram, zfs on root raidz with 4x1,5T drives. >>>> After updating to stable a couple of days ago I noticed that the >>>> system leaks memory very fast. Checking here and there I found >>>> that the issue concerns sendfile (yep, again!). >>>> >>>> How to reproduce: >>>> Configure samba with aio and sendfile (mine is version 3.5.6) >>>> >>>> smb.conf >>>> [global] >>>> use sendfile=true >>>> aio read size = 16384 >>>> >>>> Download a couple of large samba shared files (8-10 gigs). >>>> >>>> >>>> While downloading files I can see that memory decreazes to nowhere >>>> very-very fast, several MBs per second! First it drains free mem, >>>> than active and inactive, than comes wired until the whole system >>>> commits suicide suffocating itself to the death. >>>> The only way to free memory is to reboot the system. I can't >>>> unload zfs module like PJD suggested to do, 'cause my root is on >>>> zfs :( >>>> I'll try to make a bootable flash and move root to the flash to >>>> try to unload module and what will happen. >>>> >>>> Everything was OK in stable before the new year, sendfile used to >>>> pump free and wired memory to inactive than slowly reclaiming it >>>> back. But it seems something was changed after NY holydays? >>> I'm glad someone else finally picked that problem, so there's >>> appareantly no memory-eating ghost in my machine ;) >>> Here's my thread on stable list about this issue: >>> http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061247.html >>> >>> >>> And in fact, PC reported in thread above is also SAMBA server with >>> aio/sendfile enabled and ZFS. >>> >>> I would be happy testing some patches if necessary, because until >>> now I need to monitor memory and reboot this server before it dies. >> The source and build date of your kernel will matter greatly here. >> >> I can't speak about the memory utilisation aspect, but I tend to disable >> sendfile everywhere possible when ZFS is in use on a system. The reason >> is based on something I and another user experienced back in October >> 2010 pertaining to sendfile() on ZFS locking up processes (making them >> unkillable). See here[1] for details; this problem has since been >> fixed[2] (look for commits around October). You'll also find some >> commits that went through in November pertaining to ZFS and sendfile. >> This is why I said the date of your kernel/sources matters. :-) > > I tried rebuild since original thread, hoping that problem is fixed > already, so now it's very fresh: 8.2-PRERELEASE #18: Sun Feb 6 > 03:04:47 CET 2011. > Problem is still here: > > Mem: 37M Active, 78M Inact, 1154M Wired, 64M Cache, 199M Buf, 40M Free > About 1373MB instead of 2GB, and it's not even 2 days of uptime. > >> The issue I referenced in [1] is not related to memory utilisation, but >> does indicate use of sendfile with ZFS may be a bad idea (by this I >> mean, there may be aspects of its implementation when mixed with ZFS >> that have been overlooked). >> >> Simple test: if you disable use of sendfile (but not AIO) in Samba, does >> the problem go away? > I've just disabled sendfile in smb.conf and I'll report in about 2 > days, after reboot which I will perform tonight. > I hope it won't hit samba performance too much ;) > We didn't need to wait 2 days :) Now I can confirm that sendfile under SAMBA + ZFS are responsible for issue. Here's sample output from my monitoring script[1] (update every 2 seconds): PHYSMEM: 2027.41 MB ACTIVE: 61.14 MB INACTIVE: 40.01 MB WIRED: 1303.86 MB CACHED: .50 MB FREE: 552.30 MB SUM: 1957.82 MB ------------------------ MISSING: 69.58 MB PHYSMEM: 2027.41 MB ACTIVE: 61.14 MB INACTIVE: 40.07 MB WIRED: 1303.86 MB CACHED: .50 MB FREE: 551.80 MB SUM: 1957.38 MB ------------------------ MISSING: 70.02 MB PHYSMEM: 2027.41 MB ACTIVE: 61.14 MB INACTIVE: 40.13 MB WIRED: 1303.86 MB CACHED: .50 MB FREE: 551.30 MB SUM: 1956.94 MB ------------------------ MISSING: 70.46 MB PHYSMEM: 2027.41 MB ACTIVE: 61.14 MB INACTIVE: 40.19 MB WIRED: 1303.86 MB CACHED: .50 MB FREE: 550.80 MB SUM: 1956.51 MB ------------------------ MISSING: 70.89 MB PHYSMEM: 2027.41 MB ACTIVE: 61.14 MB INACTIVE: 40.24 MB WIRED: 1303.86 MB CACHED: .50 MB FREE: 550.42 MB SUM: 1956.18 MB ------------------------ MISSING: 71.22 MB PHYSMEM: 2027.41 MB ACTIVE: 61.14 MB INACTIVE: 40.30 MB WIRED: 1303.86 MB CACHED: .50 MB FREE: 549.92 MB SUM: 1955.74 MB ------------------------ MISSING: 71.66 MB PHYSMEM: 2027.41 MB ACTIVE: 61.14 MB INACTIVE: 40.38 MB WIRED: 1303.86 MB CACHED: .50 MB FREE: 549.30 MB SUM: 1955.19 MB ------------------------ MISSING: 72.21 MB PHYSMEM: 2027.41 MB ACTIVE: 61.14 MB INACTIVE: 40.44 MB WIRED: 1303.86 MB CACHED: .50 MB FREE: 548.80 MB SUM: 1954.76 MB ------------------------ MISSING: 72.64 MB This behaviour has been seen while copying 600MB file from SAMBA share with sendfile enabled. It doesn't happen when writing to samba share, and it doesn't happen with sendfile disabled, both ways. For me it looks like memory which leaks should be added to wired pool and belongs to ARC, but appareantly this doesn't work well and WIRED: 1303.86 MB all the time. [1] http://pastebin.com/sQUyQbmm -- Bartosz Stec From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 20:36:59 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 79686106564A; Tue, 8 Feb 2011 20:36:59 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id C2AC18FC1A; Tue, 8 Feb 2011 20:36:58 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id p18Kasjw078827 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 8 Feb 2011 22:36:54 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id p18Kasa6085086; Tue, 8 Feb 2011 22:36:54 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id p18KarHm085085; Tue, 8 Feb 2011 22:36:53 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 8 Feb 2011 22:36:53 +0200 From: Kostik Belousov To: Bartosz Stec Message-ID: <20110208203653.GC78089@deviant.kiev.zoral.com.ua> References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> <4D511F65.2050503@kkip.pl> <4D519F97.2000805@kkip.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="UIrAl4r1g2eOkvhC" Content-Disposition: inline In-Reply-To: <4D519F97.2000805@kkip.pl> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org, pjd@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 20:36:59 -0000 --UIrAl4r1g2eOkvhC Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Feb 08, 2011 at 08:55:03PM +0100, Bartosz Stec wrote: > W dniu 2011-02-08 11:48, Bartosz Stec pisze: > >W dniu 2011-02-08 11:27, Jeremy Chadwick pisze: > >>On Tue, Feb 08, 2011 at 10:24:11AM +0100, Bartosz Stec wrote: > >>>W dniu 2011-02-07 22:37, Emil Muratov pisze: > >>>>>For the past few weeks, I noticed that the amount of memory > >>>>>reported in top > >>>>>(sum of active, inact, wired, cache buf and free) keeps > >>>>>decreasing as the > >>>>>uptime increases. I can't pinpoint to when I first noticed this, > >>>>>as I have > >>>>>updated the system a few times just in case this has been fixed. > >>>>Yes, I have the same issue on my home file storage. My system is > >>>>8.1 amd64, 2G ram, zfs on root raidz with 4x1,5T drives. > >>>>After updating to stable a couple of days ago I noticed that the > >>>>system leaks memory very fast. Checking here and there I found > >>>>that the issue concerns sendfile (yep, again!). > >>>> > >>>>How to reproduce: > >>>>Configure samba with aio and sendfile (mine is version 3.5.6) > >>>> > >>>>smb.conf > >>>>[global] > >>>>use sendfile=3Dtrue > >>>>aio read size =3D 16384 > >>>> > >>>>Download a couple of large samba shared files (8-10 gigs). > >>>> > >>>> > >>>>While downloading files I can see that memory decreazes to nowhere > >>>>very-very fast, several MBs per second! First it drains free mem, > >>>>than active and inactive, than comes wired until the whole system > >>>>commits suicide suffocating itself to the death. > >>>>The only way to free memory is to reboot the system. I can't > >>>>unload zfs module like PJD suggested to do, 'cause my root is on > >>>>zfs :( > >>>>I'll try to make a bootable flash and move root to the flash to > >>>>try to unload module and what will happen. > >>>> > >>>>Everything was OK in stable before the new year, sendfile used to > >>>>pump free and wired memory to inactive than slowly reclaiming it > >>>>back. But it seems something was changed after NY holydays? > >>>I'm glad someone else finally picked that problem, so there's > >>>appareantly no memory-eating ghost in my machine ;) > >>>Here's my thread on stable list about this issue: > >>>http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061247.= html=20 > >>> > >>> > >>>And in fact, PC reported in thread above is also SAMBA server with > >>>aio/sendfile enabled and ZFS. > >>> > >>>I would be happy testing some patches if necessary, because until > >>>now I need to monitor memory and reboot this server before it dies. > >>The source and build date of your kernel will matter greatly here. > >> > >>I can't speak about the memory utilisation aspect, but I tend to disable > >>sendfile everywhere possible when ZFS is in use on a system. The reason > >>is based on something I and another user experienced back in October > >>2010 pertaining to sendfile() on ZFS locking up processes (making them > >>unkillable). See here[1] for details; this problem has since been > >>fixed[2] (look for commits around October). You'll also find some > >>commits that went through in November pertaining to ZFS and sendfile. > >>This is why I said the date of your kernel/sources matters. :-) > > > >I tried rebuild since original thread, hoping that problem is fixed=20 > >already, so now it's very fresh: 8.2-PRERELEASE #18: Sun Feb 6=20 > >03:04:47 CET 2011. > >Problem is still here: > > > >Mem: 37M Active, 78M Inact, 1154M Wired, 64M Cache, 199M Buf, 40M Free > >About 1373MB instead of 2GB, and it's not even 2 days of uptime. > > > >>The issue I referenced in [1] is not related to memory utilisation, but > >>does indicate use of sendfile with ZFS may be a bad idea (by this I > >>mean, there may be aspects of its implementation when mixed with ZFS > >>that have been overlooked). > >> > >>Simple test: if you disable use of sendfile (but not AIO) in Samba, does > >>the problem go away? > >I've just disabled sendfile in smb.conf and I'll report in about 2=20 > >days, after reboot which I will perform tonight. > >I hope it won't hit samba performance too much ;) > > > We didn't need to wait 2 days :) > Now I can confirm that sendfile under SAMBA + ZFS are responsible for=20 > issue. Here's sample output from my monitoring script[1] (update every 2= =20 > seconds): >=20 > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.01 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 552.30 MB > SUM: 1957.82 MB > ------------------------ > MISSING: 69.58 MB >=20 > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.07 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 551.80 MB > SUM: 1957.38 MB > ------------------------ > MISSING: 70.02 MB >=20 > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.13 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 551.30 MB > SUM: 1956.94 MB > ------------------------ > MISSING: 70.46 MB >=20 > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.19 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 550.80 MB > SUM: 1956.51 MB > ------------------------ > MISSING: 70.89 MB >=20 > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.24 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 550.42 MB > SUM: 1956.18 MB > ------------------------ > MISSING: 71.22 MB >=20 > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.30 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 549.92 MB > SUM: 1955.74 MB > ------------------------ > MISSING: 71.66 MB >=20 > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.38 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 549.30 MB > SUM: 1955.19 MB > ------------------------ > MISSING: 72.21 MB >=20 > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.44 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 548.80 MB > SUM: 1954.76 MB > ------------------------ > MISSING: 72.64 MB >=20 > This behaviour has been seen while copying 600MB file from SAMBA share=20 > with sendfile enabled. > It doesn't happen when writing to samba share, and it doesn't happen=20 > with sendfile disabled, both ways. > For me it looks like memory which leaks should be added to wired pool=20 > and belongs to ARC, but appareantly this doesn't work well and WIRED:=20 > 1303.86 MB all the time. >=20 > [1] http://pastebin.com/sQUyQbmm Try this. I the similar fix is needed for tmpfs, but there are some more issues and pending rewrite, so I decided not to touch it. commit 8e5885bce1afecd419e40240a2d7ab90deb0392a Author: Konstantin Belousov Date: Tue Feb 8 22:35:29 2011 +0200 Do not forget to activate the page diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c b/s= ys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c index e8191b3..7343c72 100644 --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c @@ -353,6 +353,9 @@ page_unlock(vm_page_t pp) { =20 vm_page_wakeup(pp); + vm_page_lock(pp); + vm_page_activate(pp); + vm_page_unlock(pp); } =20 static caddr_t @@ -480,7 +483,7 @@ again: if (error =3D=3D 0) uiomove_fromphys(&m, off, bytes, uio); VM_OBJECT_LOCK(obj); - vm_page_wakeup(m); + page_unlock(m); } else if (uio->uio_segflg =3D=3D UIO_NOCOPY) { /* * The code below is here to make sendfile(2) work @@ -527,9 +530,15 @@ again: zfs_unmap_page(sf); } VM_OBJECT_LOCK(obj); - if (error =3D=3D 0) - m->valid =3D VM_PAGE_BITS_ALL; vm_page_io_finish(m); + vm_page_lock(m); + if (error =3D=3D 0) { + m->valid =3D VM_PAGE_BITS_ALL; + vm_page_activate(m); + } else + vm_page_free(m); + vm_page_unlock(m); + if (error =3D=3D 0) { uio->uio_resid -=3D bytes; uio->uio_offset +=3D bytes; --UIrAl4r1g2eOkvhC Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk1RqWUACgkQC3+MBN1Mb4izHwCfUbU4AceFZ3BnReYfVOety+gw 1gUAoLYpyMdSOEsmxeI3g1E05wy9ntdQ =wjUH -----END PGP SIGNATURE----- --UIrAl4r1g2eOkvhC-- From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 20:51:45 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EB849106564A; Tue, 8 Feb 2011 20:51:45 +0000 (UTC) (envelope-from gpm@hotplug.ru) Received: from shadow.rusoft.ru (shadow.rusoft.ru [217.16.18.75]) by mx1.freebsd.org (Postfix) with ESMTP id 9F2908FC14; Tue, 8 Feb 2011 20:51:45 +0000 (UTC) Received: from shadow.rusoft.ru (localhost [127.0.0.1]) by shadow.rusoft.ru (Postfix) with ESMTP id 388B9498D; Tue, 8 Feb 2011 23:51:44 +0300 (MSK) Received: from jazz.rusoft.ru (unknown [83.222.3.162]) by shadow.rusoft.ru (Postfix) with ESMTP id A7B07498C; Tue, 8 Feb 2011 23:51:43 +0300 (MSK) Received: from ghost-pc.home.lan (ghostmaster.static.corbina.ru [78.107.250.255]) by jazz.rusoft.ru (Postfix) with ESMTPA id F21451CC011; Tue, 8 Feb 2011 23:51:42 +0300 (MSK) Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: "Jeremy Chadwick" , "Bartosz Stec" References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> <4D511F65.2050503@kkip.pl> Date: Tue, 08 Feb 2011 23:51:43 +0300 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Emil Muratov" Message-ID: In-Reply-To: <4D511F65.2050503@kkip.pl> User-Agent: Opera Mail/11.01 (Win32) X-Virus-Scanned: ClamAV using ClamSMTP Cc: freebsd-fs@freebsd.org, pjd@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 20:51:46 -0000 >> I can't speak about the memory utilisation aspect, but I tend to disable >> sendfile everywhere possible when ZFS is in use on a system. The reason >> is based on something I and another user experienced back in October >> 2010 pertaining to sendfile() on ZFS locking up processes (making them >> unkillable). See here[1] for details; this problem has since been >> fixed[2] (look for commits around October). You'll also find some >> commits that went through in November pertaining to ZFS and sendfile. >> This is why I said the date of your kernel/sources matters. :-) > > I tried rebuild since original thread, hoping that problem is fixed > already, so now it's very fresh: 8.2-PRERELEASE #18: Sun Feb 6 03:04:47 > CET 2011. > Problem is still here: > > Mem: 37M Active, 78M Inact, 1154M Wired, 64M Cache, 199M Buf, 40M Free > About 1373MB instead of 2GB, and it's not even 2 days of uptime. > >> The issue I referenced in [1] is not related to memory utilisation, but >> does indicate use of sendfile with ZFS may be a bad idea (by this I >> mean, there may be aspects of its implementation when mixed with ZFS >> that have been overlooked). Yes, I remember those commits. It helped a lot for the sendfile problem that days. So my system was built based on stable as of december fist decade. And everything worked fine more or less for my home server. But after updating a couple of days ago this memory hole issue arised, thats why I mentioned that maybe some of the latest january commits broke something? >> Simple test: if you disable use of sendfile (but not AIO) in Samba, does >> the problem go away? > I've just disabled sendfile in smb.conf and I'll report in about 2 days, > after reboot which I will perform tonight. > I hope it won't hit samba performance too much ;) Yes, for me setting "use sendfile" to "false" in smb.conf fixes everything. I can download tons of data via smb without any memory leaks. From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 21:03:03 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 912A1106564A for ; Tue, 8 Feb 2011 21:03:03 +0000 (UTC) (envelope-from gldisater@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 462668FC14 for ; Tue, 8 Feb 2011 21:03:02 +0000 (UTC) Received: by yxh35 with SMTP id 35so2565683yxh.13 for ; Tue, 08 Feb 2011 13:03:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=ivdqO4A8PtoqGzP734T0qWvdwFwUGf62+eGo+YXl6Fk=; b=dhAIRYFUedXQ/Kz64xDWKRG7ISJY10ScHTJMRig7Zm6y3gAKYpmqHruH9gkirCgMYy sc10jEVOQIaPyfkgXBcqSvU1dcn+BTOW4lgHWPeKHI1AYmzYtZMGJepUFF5YCTW9G+La eqJl+oi9SE/niyZf/7z78xHL7sOyI3nmyv3Tw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; b=gxjeCwJb7FhsWsFKxT6bLsa9mwuXegPhLR76+ClRuyIkONZzBerBG+CjsTrioe/HUl 8jgLEfTEfF4By446hKUDeyeE25z/MA6B7x/r4tdfUzfqpnSvsTm7LTsJBZllbn/Y9j0r QBfVQRCfREI3MKk6CDhfU5m42nqWBOJtNHAV4= Received: by 10.90.52.20 with SMTP id z20mr417711agz.107.1297197558507; Tue, 08 Feb 2011 12:39:18 -0800 (PST) Received: from [192.168.1.2] (CPE00236909a9fe-CM00195eca698c.cpe.net.cable.rogers.com [99.236.13.52]) by mx.google.com with ESMTPS id b11sm7501926ana.38.2011.02.08.12.39.14 (version=SSLv3 cipher=RC4-MD5); Tue, 08 Feb 2011 12:39:14 -0800 (PST) Message-ID: <4D51A969.5090203@gmail.com> Date: Tue, 08 Feb 2011 15:36:57 -0500 From: Jeremy Faulkner User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.13) Gecko/20110130 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> <4D511F65.2050503@kkip.pl> <4D519F97.2000805@kkip.pl> In-Reply-To: <4D519F97.2000805@kkip.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 21:03:03 -0000 On 02/08/11 14:55, Bartosz Stec wrote: > W dniu 2011-02-08 11:48, Bartosz Stec pisze: > We didn't need to wait 2 days :) > Now I can confirm that sendfile under SAMBA + ZFS are responsible for > issue. Here's sample output from my monitoring script[1] (update every 2 > seconds): > > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.01 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 552.30 MB > SUM: 1957.82 MB > ------------------------ > MISSING: 69.58 MB > > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.07 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 551.80 MB > SUM: 1957.38 MB > ------------------------ > MISSING: 70.02 MB > > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.13 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 551.30 MB > SUM: 1956.94 MB > ------------------------ > MISSING: 70.46 MB > > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.19 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 550.80 MB > SUM: 1956.51 MB > ------------------------ > MISSING: 70.89 MB > > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.24 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 550.42 MB > SUM: 1956.18 MB > ------------------------ > MISSING: 71.22 MB > > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.30 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 549.92 MB > SUM: 1955.74 MB > ------------------------ > MISSING: 71.66 MB > > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.38 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 549.30 MB > SUM: 1955.19 MB > ------------------------ > MISSING: 72.21 MB > > PHYSMEM: 2027.41 MB > ACTIVE: 61.14 MB > INACTIVE: 40.44 MB > WIRED: 1303.86 MB > CACHED: .50 MB > FREE: 548.80 MB > SUM: 1954.76 MB > ------------------------ > MISSING: 72.64 MB > > This behaviour has been seen while copying 600MB file from SAMBA share > with sendfile enabled. > It doesn't happen when writing to samba share, and it doesn't happen > with sendfile disabled, both ways. > For me it looks like memory which leaks should be added to wired pool > and belongs to ARC, but appareantly this doesn't work well and WIRED: > 1303.86 MB all the time. > > [1] http://pastebin.com/sQUyQbmm > Also occurring with the zfsv28 patch for 8-STABLE gldisater@constans:~% zpool upgrade This system is currently running ZFS pool version 28. All pools are formatted using this version. gldisater@constans:~% sh memleak-detect.sh PHYSMEM: 12268.94 MB ACTIVE: 427.57 MB INACTIVE: 724.98 MB WIRED: 10155.28 MB CACHED: 28.32 MB FREE: 556.16 MB SUM: 11892.32 MB ------------------------ MISSING: 376.61 MB gldisater@constans:~% uname -a FreeBSD constans 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #14: Tue Jan 18 15:26:02 EST 2011 root@constans:/usr/obj/usr/src/sys/CONSTANS amd64 gldisater@constans:~% uptime 3:22PM up 8 days, 5:07, 2 users, load averages: 0.21, 0.22, 0.24 -- Jeremy Faulkner From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 21:22:04 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 91AE8106566C for ; Tue, 8 Feb 2011 21:22:04 +0000 (UTC) (envelope-from gpm@hotplug.ru) Received: from shadow.rusoft.ru (shadow.rusoft.ru [217.16.18.75]) by mx1.freebsd.org (Postfix) with ESMTP id 440098FC0A for ; Tue, 8 Feb 2011 21:22:04 +0000 (UTC) Received: from shadow.rusoft.ru (localhost [127.0.0.1]) by shadow.rusoft.ru (Postfix) with ESMTP id B478B4FD9; Wed, 9 Feb 2011 00:22:02 +0300 (MSK) Received: from jazz.rusoft.ru (unknown [83.222.3.162]) by shadow.rusoft.ru (Postfix) with ESMTP id 995174FD8; Wed, 9 Feb 2011 00:22:02 +0300 (MSK) Received: from ghost-pc.home.lan (ghostmaster.static.corbina.ru [78.107.250.255]) by jazz.rusoft.ru (Postfix) with ESMTPA id 3E8631CC00F; Wed, 9 Feb 2011 00:22:02 +0300 (MSK) Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: "Bartosz Stec" , "Kostik Belousov" References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> <4D511F65.2050503@kkip.pl> <4D519F97.2000805@kkip.pl> <20110208203653.GC78089@deviant.kiev.zoral.com.ua> Date: Wed, 09 Feb 2011 00:22:01 +0300 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Emil Muratov" Message-ID: In-Reply-To: <20110208203653.GC78089@deviant.kiev.zoral.com.ua> User-Agent: Opera Mail/11.01 (Win32) X-Virus-Scanned: ClamAV using ClamSMTP Cc: freebsd-fs@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 21:22:04 -0000 >> >>Simple test: if you disable use of sendfile (but not AIO) in Samba, >> does >> >>the problem go away? >> >I've just disabled sendfile in smb.conf and I'll report in about 2 >> >days, after reboot which I will perform tonight. >> >I hope it won't hit samba performance too much ;) >> > >> We didn't need to wait 2 days :) >> Now I can confirm that sendfile under SAMBA + ZFS are responsible for >> issue. Here's sample output from my monitoring script[1] (update every 2 >> seconds): >> >> >> This behaviour has been seen while copying 600MB file from SAMBA share >> with sendfile enabled. >> It doesn't happen when writing to samba share, and it doesn't happen >> with sendfile disabled, both ways. >> For me it looks like memory which leaks should be added to wired pool >> and belongs to ARC, but appareantly this doesn't work well and WIRED: >> 1303.86 MB all the time. >> >> [1] http://pastebin.com/sQUyQbmm > > Try this. I the similar fix is needed for tmpfs, but there are some > more issues and pending rewrite, so I decided not to touch it. > > commit 8e5885bce1afecd419e40240a2d7ab90deb0392a > Author: Konstantin Belousov > Date: Tue Feb 8 22:35:29 2011 +0200 > > Do not forget to activate the page > > diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c > b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c > index e8191b3..7343c72 100644 > --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c > +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c What is this patch up against? I can't apply it to the STABLE. :( From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 21:37:05 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2FB46106566B for ; Tue, 8 Feb 2011 21:37:05 +0000 (UTC) (envelope-from admin@kkip.pl) Received: from mainframe.kkip.pl (kkip.pl [87.105.164.78]) by mx1.freebsd.org (Postfix) with ESMTP id D2D4D8FC08 for ; Tue, 8 Feb 2011 21:37:04 +0000 (UTC) Received: from static-78-8-144-74.ssp.dialog.net.pl ([78.8.144.74] helo=[192.168.0.2]) by mainframe.kkip.pl with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.73 (FreeBSD)) (envelope-from ) id 1PmvF2-0001fk-2V; Tue, 08 Feb 2011 22:37:03 +0100 Message-ID: <4D51B775.2010100@kkip.pl> Date: Tue, 08 Feb 2011 22:36:53 +0100 From: Bartosz Stec User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jeremy Faulkner References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> <4D511F65.2050503@kkip.pl> <4D519F97.2000805@kkip.pl> <4D51A969.5090203@gmail.com> In-Reply-To: <4D51A969.5090203@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Authenticated-User: admin@kkip.pl X-Authenticator: plain X-Sender-Verify: SUCCEEDED (sender exists & accepts mail) X-Spam-Score: -8.5 X-Spam-Score-Int: -84 X-Exim-Version: 4.73 (build at 10-Jan-2011 16:29:01) X-Date: 2011-02-08 22:37:03 X-Connected-IP: 78.8.144.74:62062 X-Message-Linecount: 53 X-Body-Linecount: 40 X-Message-Size: 2101 X-Body-Size: 1352 X-Received-Count: 1 X-Recipient-Count: 2 X-Local-Recipient-Count: 2 X-Local-Recipient-Defer-Count: 0 X-Local-Recipient-Fail-Count: 0 Cc: freebsd-fs@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 21:37:05 -0000 > gldisater@constans:~% zpool upgrade > This system is currently running ZFS pool version 28. > > All pools are formatted using this version. > gldisater@constans:~% sh memleak-detect.sh > PHYSMEM: 12268.94 MB > ACTIVE: 427.57 MB > INACTIVE: 724.98 MB > WIRED: 10155.28 MB > CACHED: 28.32 MB > FREE: 556.16 MB > SUM: 11892.32 MB > ------------------------ > MISSING: 376.61 MB > > gldisater@constans:~% uname -a > FreeBSD constans 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #14: Tue Jan 18 > 15:26:02 EST 2011 root@constans:/usr/obj/usr/src/sys/CONSTANS amd64 > gldisater@constans:~% uptime > 3:22PM up 8 days, 5:07, 2 users, load averages: 0.21, 0.22, 0.24 > > Have you noticed MISSING count increasing over time or you just looking at this exact calculation? This script is far from being perfect and I don't know as much about FreeBSD internals to calculate it to show exactly zero at this point (if it is possible at all). It only calculates physical memory minus memory pools visible in top(1), and on my system it gives MISSING count about 36MB just after reboot. What's important - check if this variable is stable over time. You can confirm memory leakage if MISSING is increasing over time or while using sendfile. You can try for instance 'sh memleak-detect.sh -m -r 60' to check it every one minute. Cheers. -- Bartosz Stec From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 21:37:26 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 18700106566B for ; Tue, 8 Feb 2011 21:37:26 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id A6EAE8FC18 for ; Tue, 8 Feb 2011 21:37:25 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id p18LbLLc083154 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 8 Feb 2011 23:37:21 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id p18LbL8X086125; Tue, 8 Feb 2011 23:37:21 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id p18LbL2M086124; Tue, 8 Feb 2011 23:37:21 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 8 Feb 2011 23:37:21 +0200 From: Kostik Belousov To: Emil Muratov Message-ID: <20110208213721.GE78089@deviant.kiev.zoral.com.ua> References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> <4D511F65.2050503@kkip.pl> <4D519F97.2000805@kkip.pl> <20110208203653.GC78089@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="g7y1ijQH9DuGXLTD" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 21:37:26 -0000 --g7y1ijQH9DuGXLTD Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Feb 09, 2011 at 12:22:01AM +0300, Emil Muratov wrote: > >Try this. I the similar fix is needed for tmpfs, but there are some > >more issues and pending rewrite, so I decided not to touch it. > > > >commit 8e5885bce1afecd419e40240a2d7ab90deb0392a > >Author: Konstantin Belousov > >Date: Tue Feb 8 22:35:29 2011 +0200 > > > > Do not forget to activate the page > > > >diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c = =20 > >b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c > >index e8191b3..7343c72 100644 > >--- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c > >+++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c >=20 >=20 > What is this patch up against? I can't apply it to the STABLE. :( HEAD. I adopted the patch to stable/8, but did not even compiled it. Index: cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c (revision 218456) +++ cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c (working copy) @@ -347,6 +347,9 @@ page_unlock(vm_page_t pp) { =20 vm_page_wakeup(pp); + vm_page_lock_queues(); + vm_page_activate(pp); + vm_page_unlock_queues(); } =20 static caddr_t @@ -465,7 +468,7 @@ again: if (error =3D=3D 0) uiomove_fromphys(&m, off, bytes, uio); VM_OBJECT_LOCK(obj); - vm_page_wakeup(m); + page_unlock(m); } else if (uio->uio_segflg =3D=3D UIO_NOCOPY) { /* * The code below is here to make sendfile(2) work @@ -504,10 +507,16 @@ again: zfs_unmap_page(sf); } VM_OBJECT_LOCK(obj); - if (error =3D=3D 0) - m->valid =3D VM_PAGE_BITS_ALL; vm_page_io_finish(m); + vm_page_lock_queues(); if (error =3D=3D 0) { + m->valid =3D VM_PAGE_BITS_ALL; + vm_page_activate(m); + } else + vm_page_free(m); + vm_page_unlock_queues(); + + if (error =3D=3D 0) { uio->uio_resid -=3D bytes; uio->uio_offset +=3D bytes; } --g7y1ijQH9DuGXLTD Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk1Rt5EACgkQC3+MBN1Mb4gBrACfYLFi1Ynk7Xi5WgF7A0WfNguT hFIAnRTy4cJuvwUBUEnKbjJFyEGbqibH =ecGZ -----END PGP SIGNATURE----- --g7y1ijQH9DuGXLTD-- From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 21:52:09 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1888D106566B for ; Tue, 8 Feb 2011 21:52:09 +0000 (UTC) (envelope-from admin@kkip.pl) Received: from mainframe.kkip.pl (kkip.pl [87.105.164.78]) by mx1.freebsd.org (Postfix) with ESMTP id A3CAA8FC0A for ; Tue, 8 Feb 2011 21:52:08 +0000 (UTC) Received: from static-78-8-144-74.ssp.dialog.net.pl ([78.8.144.74] helo=[192.168.0.2]) by mainframe.kkip.pl with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.73 (FreeBSD)) (envelope-from ) id 1PmvTa-0001jK-TH; Tue, 08 Feb 2011 22:52:06 +0100 Message-ID: <4D51BAFC.50806@kkip.pl> Date: Tue, 08 Feb 2011 22:51:56 +0100 From: Bartosz Stec User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: Kostik Belousov References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> <4D511F65.2050503@kkip.pl> <4D519F97.2000805@kkip.pl> <20110208203653.GC78089@deviant.kiev.zoral.com.ua> <20110208213721.GE78089@deviant.kiev.zoral.com.ua> In-Reply-To: <20110208213721.GE78089@deviant.kiev.zoral.com.ua> X-Authenticated-User: admin@kkip.pl X-Authenticator: plain X-Sender-Verify: SUCCEEDED (sender exists & accepts mail) X-Spam-Score: -8.5 X-Spam-Score-Int: -84 X-Exim-Version: 4.73 (build at 10-Jan-2011 16:29:01) X-Date: 2011-02-08 22:52:06 X-Connected-IP: 78.8.144.74:56574 X-Message-Linecount: 224 X-Body-Linecount: 211 X-Message-Size: 7879 X-Body-Size: 7005 X-Received-Count: 1 X-Recipient-Count: 2 X-Local-Recipient-Count: 2 X-Local-Recipient-Defer-Count: 0 X-Local-Recipient-Fail-Count: 0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 21:52:09 -0000 W dniu 2011-02-08 22:37, Kostik Belousov pisze: > On Wed, Feb 09, 2011 at 12:22:01AM +0300, Emil Muratov wrote: >>> Try this. I the similar fix is needed for tmpfs, but there are some >>> more issues and pending rewrite, so I decided not to touch it. >>> >>> commit 8e5885bce1afecd419e40240a2d7ab90deb0392a >>> Author: Konstantin Belousov >>> Date: Tue Feb 8 22:35:29 2011 +0200 >>> >>> Do not forget to activate the page >>> >>> diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c >>> b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c >>> index e8191b3..7343c72 100644 >>> --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c >>> +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c >> >> What is this patch up against? I can't apply it to the STABLE. :( > HEAD. > > I adopted the patch to stable/8, but did not even compiled it. > > Index: cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c > =================================================================== > --- cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c (revision 218456) > +++ cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c (working copy) > @@ -347,6 +347,9 @@ page_unlock(vm_page_t pp) > { > > vm_page_wakeup(pp); > + vm_page_lock_queues(); > + vm_page_activate(pp); > + vm_page_unlock_queues(); > } > > static caddr_t > @@ -465,7 +468,7 @@ again: > if (error == 0) > uiomove_fromphys(&m, off, bytes, uio); > VM_OBJECT_LOCK(obj); > - vm_page_wakeup(m); > + page_unlock(m); > } else if (uio->uio_segflg == UIO_NOCOPY) { > /* > * The code below is here to make sendfile(2) work > @@ -504,10 +507,16 @@ again: > zfs_unmap_page(sf); > } > VM_OBJECT_LOCK(obj); > - if (error == 0) > - m->valid = VM_PAGE_BITS_ALL; > vm_page_io_finish(m); > + vm_page_lock_queues(); > if (error == 0) { > + m->valid = VM_PAGE_BITS_ALL; > + vm_page_activate(m); > + } else > + vm_page_free(m); > + vm_page_unlock_queues(); > + > + if (error == 0) { > uio->uio_resid -= bytes; > uio->uio_offset += bytes; > } Patching was succesfull # patch < patch Hmm... Looks like a unified diff to me... The text leading up to this was: -------------------------- |Index: cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c |=================================================================== |--- cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c (revision 218456) |+++ cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c (working copy) -------------------------- Patching file zfs_vnops.c using Plan A... Hunk #1 succeeded at 347. Hunk #2 succeeded at 468. Hunk #3 succeeded at 507. Hmm... Ignoring the trailing garbage. done Now what's the recommended way to apply above changes to system (without rebuilding whole universe...)? -- Bartosz Stec From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 22:58:48 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C910E106566B for ; Tue, 8 Feb 2011 22:58:48 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 5597F8FC08 for ; Tue, 8 Feb 2011 22:58:48 +0000 (UTC) Received: by bwz12 with SMTP id 12so323865bwz.13 for ; Tue, 08 Feb 2011 14:58:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:to:cc:subject:date:message-id:user-agent :mime-version:content-type; bh=cbOMrn0fmql/pi1qD06fvV+TEAcpAuGyBDIuBs4ohUk=; b=aAE1au4VCjxJk1nCzBRQKl0MzQ9gkuMb75JwPPAiSqPmBfZZ7sDQh61FquM6slqdZj 4LqDdC2Gl3WjresOL41plAOw4tBMXseYpB3X2fPOXqn8pQ5sz5vgsuULafkh1Ca1DgHb e3JauyhPAiPjAzVkSzzI+ehjMtAkReempj6E0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:date:message-id:user-agent:mime-version :content-type; b=EvgV4HemW1HeCxcBf6qBUXKaF1axj3YYo/NL0xQDK7DKHTVn2haT5Ou7BNf8yfZBnY 8TJR+gUWjymDR3NxR4cyORPXFnfH2b7H1NtMQzvviQTI+ariZo7KhOj1jwId+yz6CDQ+ nUguJpx3YYT3Q8l/78ZrTgdoTCbC64HmSsPjY= Received: by 10.204.98.130 with SMTP id q2mr18212860bkn.31.1297204295620; Tue, 08 Feb 2011 14:31:35 -0800 (PST) Received: from localhost ([95.69.174.185]) by mx.google.com with ESMTPS id x38sm3049507bkj.13.2011.02.08.14.31.33 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 08 Feb 2011 14:31:34 -0800 (PST) From: Mikolaj Golub To: freebsd-fs@FreeBSD.org Date: Wed, 09 Feb 2011 00:31:31 +0200 Message-ID: <86lj1qgnuk.fsf@kopusha.home.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Cc: Pawel Jakub Dawidek Subject: HAST: unix socket issue X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 22:58:48 -0000 --=-=-= Hi, Currently hastd on termination does not remove control socket file. This is because of the bug in uds_accept(): in accept() it passes the old sockaddr instead of the new one. As a result sun_path is reseted in the initial socket while sockaddr for the new socket is left uninitialized. See the attached patch. Later when unlink is called on a socket close it fails because the path is empty or some garbage. But after fixing this issue another problem shows up, which is masked by the first bug: in the child process we close all unneeded descriptors. For a unix socket uds_close() will unlink the socket file, which we still need for parent. E.g. control socket file will be removed on a worker start. It looks like we need some way to tell uds_close() not to remove the file in such cases, e.g. setting a flag in socket structure before proto_close() or adding proto_close_without_unlink() (can't think out a good name :-) -- Mikolaj Golub --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=proto_uds.c.uds_accept.patch Index: sbin/hastd/proto_uds.c =================================================================== --- sbin/hastd/proto_uds.c (revision 218456) +++ sbin/hastd/proto_uds.c (working copy) @@ -201,7 +201,7 @@ uds_accept(void *ctx, void **newctxp) return (errno); fromlen = sizeof(uctx->uc_sun); - newuctx->uc_fd = accept(uctx->uc_fd, (struct sockaddr *)&uctx->uc_sun, + newuctx->uc_fd = accept(uctx->uc_fd, (struct sockaddr *)&newuctx->uc_sun, &fromlen); if (newuctx->uc_fd < 0) { ret = errno; --=-=-=-- From owner-freebsd-fs@FreeBSD.ORG Tue Feb 8 23:16:40 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 47CC8106564A for ; Tue, 8 Feb 2011 23:16:40 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.garage.freebsd.pl (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id E626A8FC12 for ; Tue, 8 Feb 2011 23:16:39 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 16BEC45E87; Wed, 9 Feb 2011 00:16:38 +0100 (CET) Received: from localhost (89-73-195-149.dynamic.chello.pl [89.73.195.149]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 0443745CA6; Wed, 9 Feb 2011 00:16:32 +0100 (CET) Date: Wed, 9 Feb 2011 00:16:14 +0100 From: Pawel Jakub Dawidek To: Mikolaj Golub Message-ID: <20110208231614.GB1822@garage.freebsd.pl> References: <86lj1qgnuk.fsf@kopusha.home.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="yNb1oOkm5a9FJOVX" Content-Disposition: inline In-Reply-To: <86lj1qgnuk.fsf@kopusha.home.net> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@FreeBSD.org Subject: Re: HAST: unix socket issue X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 23:16:40 -0000 --yNb1oOkm5a9FJOVX Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Feb 09, 2011 at 12:31:31AM +0200, Mikolaj Golub wrote: > Hi, >=20 > Currently hastd on termination does not remove control socket file. This = is > because of the bug in uds_accept(): in accept() it passes the old sockaddr > instead of the new one. As a result sun_path is reseted in the initial so= cket > while sockaddr for the new socket is left uninitialized. See the attached > patch. >=20 > Later when unlink is called on a socket close it fails because the path is > empty or some garbage. >=20 > But after fixing this issue another problem shows up, which is masked by = the > first bug: in the child process we close all unneeded descriptors. For a = unix > socket uds_close() will unlink the socket file, which we still need for > parent. E.g. control socket file will be removed on a worker start. >=20 > It looks like we need some way to tell uds_close() not to remove the file= in > such cases, e.g. setting a flag in socket structure before proto_close() = or > adding proto_close_without_unlink() (can't think out a good name :-) Good catch. I fixed it a bit differently. Actually in my test with your patch we still don't have valid socket path in sun_path. It was of course bogus to unlink(2) socket file on each close. We should do it only when this is descriptor we called bind(2) and listen(2) on, not when this is descriptor created on connect(2) or accept(2). Another problem was that after fork(2) it is wrong to unlink(2) the socket file even for descriptor we are listening on. I fixed this by storing process PID on bind(2), so on close we can tell if we did create socket file and unlink it only then. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --yNb1oOkm5a9FJOVX Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk1Rzr4ACgkQForvXbEpPzSBMwCggR8K9449898R9GtngNyyhSCr oMcAoOJP73wvL7LVLFpKmkhmEW8egxEg =41/r -----END PGP SIGNATURE----- --yNb1oOkm5a9FJOVX-- From owner-freebsd-fs@FreeBSD.ORG Wed Feb 9 00:49:43 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 416FB10656AC for ; Wed, 9 Feb 2011 00:49:43 +0000 (UTC) (envelope-from gldisater@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 03A318FC29 for ; Wed, 9 Feb 2011 00:49:42 +0000 (UTC) Received: by iwn39 with SMTP id 39so6399837iwn.13 for ; Tue, 08 Feb 2011 16:49:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=//Tf9VHsEkuPWi9MOcfm54h/aa+naI0c7BksmPV0C/s=; b=UPovi5YC913nBMiCk2NT6+in162czjFpCbNNVKHcDI5HI7DqjhoRFLDTGFdno7SPUF h+q+MLfGRfJCsygSsH6bhLsL5nOJUpZowXn5SeS2NGRCO4jrLtYftJAUYIVjSDtjTgpI tgaxUww1qBVsRSMgopDKsMR8j4R6s1uXSgiu4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; b=tD6YrKns/oOJ0OBlkYVkb3uvunyArSntcjWHfCtiJ8nnFparcxXA9pAti92fW+o0DB vvckDsLS4eiE1HDze/IZ0rWPTLmSE3H7nG0RYycQX8jVlqWfoeisxhLd4KmQiGf1Z16n Pai2fDXfNBdzda5CtfgRnlFyPQC/OTA7LXK8c= Received: by 10.42.217.197 with SMTP id hn5mr1542025icb.529.1297212582177; Tue, 08 Feb 2011 16:49:42 -0800 (PST) Received: from [192.168.1.2] (CPE00236909a9fe-CM00195eca698c.cpe.net.cable.rogers.com [99.236.13.52]) by mx.google.com with ESMTPS id 8sm114329iba.16.2011.02.08.16.49.40 (version=SSLv3 cipher=OTHER); Tue, 08 Feb 2011 16:49:40 -0800 (PST) Message-ID: <4D51E42A.2090208@gmail.com> Date: Tue, 08 Feb 2011 19:47:38 -0500 From: Jeremy Faulkner User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.13) Gecko/20110130 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> <4D511F65.2050503@kkip.pl> <4D519F97.2000805@kkip.pl> <4D51A969.5090203@gmail.com> <4D51B775.2010100@kkip.pl> In-Reply-To: <4D51B775.2010100@kkip.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 00:49:43 -0000 On 02/08/11 16:36, Bartosz Stec wrote: > >> gldisater@constans:~% zpool upgrade >> This system is currently running ZFS pool version 28. >> >> All pools are formatted using this version. >> gldisater@constans:~% sh memleak-detect.sh >> PHYSMEM: 12268.94 MB >> ACTIVE: 427.57 MB >> INACTIVE: 724.98 MB >> WIRED: 10155.28 MB >> CACHED: 28.32 MB >> FREE: 556.16 MB >> SUM: 11892.32 MB >> ------------------------ >> MISSING: 376.61 MB >> >> gldisater@constans:~% uname -a >> FreeBSD constans 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #14: Tue Jan 18 >> 15:26:02 EST 2011 root@constans:/usr/obj/usr/src/sys/CONSTANS amd64 >> gldisater@constans:~% uptime >> 3:22PM up 8 days, 5:07, 2 users, load averages: 0.21, 0.22, 0.24 >> >> > Have you noticed MISSING count increasing over time or you just looking > at this exact calculation? > This script is far from being perfect and I don't know as much about > FreeBSD internals to calculate it to show exactly zero at this point (if > it is possible at all). > It only calculates physical memory minus memory pools visible in top(1), > and on my system it gives MISSING count about 36MB just after reboot. > > What's important - check if this variable is stable over time. You can > confirm memory leakage if MISSING is increasing over time or while using > sendfile. You can try for instance 'sh memleak-detect.sh -m -r 60' to > check it every one minute. > > Cheers. > My MISSING has increased to 376.62 MB. My usage of this server is primary reading not a lot of writing so a minutely check won't show anything. I'll try to write a lot of data to it later. -- Jeremy Faulkner From owner-freebsd-fs@FreeBSD.ORG Wed Feb 9 20:10:04 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 847481065672; Wed, 9 Feb 2011 20:10:04 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (mail.vx.sk [IPv6:2a01:4f8:100:1043::3]) by mx1.freebsd.org (Postfix) with ESMTP id C4E488FC13; Wed, 9 Feb 2011 20:10:03 +0000 (UTC) Received: from core.vx.sk (localhost [127.0.0.1]) by mail.vx.sk (Postfix) with ESMTP id E9D67131E82; Wed, 9 Feb 2011 21:10:02 +0100 (CET) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk ([127.0.0.1]) by core.vx.sk (mail.vx.sk [127.0.0.1]) (amavisd-new, port 10024) with LMTP id mAnN9ooDWH2X; Wed, 9 Feb 2011 21:10:00 +0100 (CET) Received: from [10.9.8.1] (chello085216231078.chello.sk [85.216.231.78]) by mail.vx.sk (Postfix) with ESMTPSA id 3017F131E6E; Wed, 9 Feb 2011 21:10:00 +0100 (CET) Message-ID: <4D52F498.2090000@FreeBSD.org> Date: Wed, 09 Feb 2011 21:10:00 +0100 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; sk; rv:1.8.1.23) Gecko/20090812 Lightning/0.9 Thunderbird/2.0.0.23 Mnenhy/0.7.5.0 MIME-Version: 1.0 To: Kostik Belousov References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> <4D511F65.2050503@kkip.pl> <4D519F97.2000805@kkip.pl> <20110208203653.GC78089@deviant.kiev.zoral.com.ua> In-Reply-To: <20110208203653.GC78089@deviant.kiev.zoral.com.ua> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=windows-1250 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org, pjd@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 20:10:04 -0000 I think this should go to HEAD and I have already prepared ported patches for v28 (head and stable). Dòa 08.02.2011 21:36, Kostik Belousov wrote / napísal(a): > On Tue, Feb 08, 2011 at 08:55:03PM +0100, Bartosz Stec wrote: >> W dniu 2011-02-08 11:48, Bartosz Stec pisze: >>> W dniu 2011-02-08 11:27, Jeremy Chadwick pisze: >>>> On Tue, Feb 08, 2011 at 10:24:11AM +0100, Bartosz Stec wrote: >>>>> W dniu 2011-02-07 22:37, Emil Muratov pisze: >>>>>>> For the past few weeks, I noticed that the amount of memory >>>>>>> reported in top >>>>>>> (sum of active, inact, wired, cache buf and free) keeps >>>>>>> decreasing as the >>>>>>> uptime increases. I can't pinpoint to when I first noticed this, >>>>>>> as I have >>>>>>> updated the system a few times just in case this has been fixed. >>>>>> Yes, I have the same issue on my home file storage. My system is >>>>>> 8.1 amd64, 2G ram, zfs on root raidz with 4x1,5T drives. >>>>>> After updating to stable a couple of days ago I noticed that the >>>>>> system leaks memory very fast. Checking here and there I found >>>>>> that the issue concerns sendfile (yep, again!). >>>>>> >>>>>> How to reproduce: >>>>>> Configure samba with aio and sendfile (mine is version 3.5.6) >>>>>> >>>>>> smb.conf >>>>>> [global] >>>>>> use sendfile=true >>>>>> aio read size = 16384 >>>>>> >>>>>> Download a couple of large samba shared files (8-10 gigs). >>>>>> >>>>>> >>>>>> While downloading files I can see that memory decreazes to nowhere >>>>>> very-very fast, several MBs per second! First it drains free mem, >>>>>> than active and inactive, than comes wired until the whole system >>>>>> commits suicide suffocating itself to the death. >>>>>> The only way to free memory is to reboot the system. I can't >>>>>> unload zfs module like PJD suggested to do, 'cause my root is on >>>>>> zfs :( >>>>>> I'll try to make a bootable flash and move root to the flash to >>>>>> try to unload module and what will happen. >>>>>> >>>>>> Everything was OK in stable before the new year, sendfile used to >>>>>> pump free and wired memory to inactive than slowly reclaiming it >>>>>> back. But it seems something was changed after NY holydays? >>>>> I'm glad someone else finally picked that problem, so there's >>>>> appareantly no memory-eating ghost in my machine ;) >>>>> Here's my thread on stable list about this issue: >>>>> http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061247.html >>>>> >>>>> >>>>> And in fact, PC reported in thread above is also SAMBA server with >>>>> aio/sendfile enabled and ZFS. >>>>> >>>>> I would be happy testing some patches if necessary, because until >>>>> now I need to monitor memory and reboot this server before it dies. >>>> The source and build date of your kernel will matter greatly here. >>>> >>>> I can't speak about the memory utilisation aspect, but I tend to disable >>>> sendfile everywhere possible when ZFS is in use on a system. The reason >>>> is based on something I and another user experienced back in October >>>> 2010 pertaining to sendfile() on ZFS locking up processes (making them >>>> unkillable). See here[1] for details; this problem has since been >>>> fixed[2] (look for commits around October). You'll also find some >>>> commits that went through in November pertaining to ZFS and sendfile. >>>> This is why I said the date of your kernel/sources matters. :-) >>> >>> I tried rebuild since original thread, hoping that problem is fixed >>> already, so now it's very fresh: 8.2-PRERELEASE #18: Sun Feb 6 >>> 03:04:47 CET 2011. >>> Problem is still here: >>> >>> Mem: 37M Active, 78M Inact, 1154M Wired, 64M Cache, 199M Buf, 40M Free >>> About 1373MB instead of 2GB, and it's not even 2 days of uptime. >>> >>>> The issue I referenced in [1] is not related to memory utilisation, but >>>> does indicate use of sendfile with ZFS may be a bad idea (by this I >>>> mean, there may be aspects of its implementation when mixed with ZFS >>>> that have been overlooked). >>>> >>>> Simple test: if you disable use of sendfile (but not AIO) in Samba, does >>>> the problem go away? >>> I've just disabled sendfile in smb.conf and I'll report in about 2 >>> days, after reboot which I will perform tonight. >>> I hope it won't hit samba performance too much ;) >>> >> We didn't need to wait 2 days :) >> Now I can confirm that sendfile under SAMBA + ZFS are responsible for >> issue. Here's sample output from my monitoring script[1] (update every 2 >> seconds): >> >> PHYSMEM: 2027.41 MB >> ACTIVE: 61.14 MB >> INACTIVE: 40.01 MB >> WIRED: 1303.86 MB >> CACHED: .50 MB >> FREE: 552.30 MB >> SUM: 1957.82 MB >> ------------------------ >> MISSING: 69.58 MB >> >> PHYSMEM: 2027.41 MB >> ACTIVE: 61.14 MB >> INACTIVE: 40.07 MB >> WIRED: 1303.86 MB >> CACHED: .50 MB >> FREE: 551.80 MB >> SUM: 1957.38 MB >> ------------------------ >> MISSING: 70.02 MB >> >> PHYSMEM: 2027.41 MB >> ACTIVE: 61.14 MB >> INACTIVE: 40.13 MB >> WIRED: 1303.86 MB >> CACHED: .50 MB >> FREE: 551.30 MB >> SUM: 1956.94 MB >> ------------------------ >> MISSING: 70.46 MB >> >> PHYSMEM: 2027.41 MB >> ACTIVE: 61.14 MB >> INACTIVE: 40.19 MB >> WIRED: 1303.86 MB >> CACHED: .50 MB >> FREE: 550.80 MB >> SUM: 1956.51 MB >> ------------------------ >> MISSING: 70.89 MB >> >> PHYSMEM: 2027.41 MB >> ACTIVE: 61.14 MB >> INACTIVE: 40.24 MB >> WIRED: 1303.86 MB >> CACHED: .50 MB >> FREE: 550.42 MB >> SUM: 1956.18 MB >> ------------------------ >> MISSING: 71.22 MB >> >> PHYSMEM: 2027.41 MB >> ACTIVE: 61.14 MB >> INACTIVE: 40.30 MB >> WIRED: 1303.86 MB >> CACHED: .50 MB >> FREE: 549.92 MB >> SUM: 1955.74 MB >> ------------------------ >> MISSING: 71.66 MB >> >> PHYSMEM: 2027.41 MB >> ACTIVE: 61.14 MB >> INACTIVE: 40.38 MB >> WIRED: 1303.86 MB >> CACHED: .50 MB >> FREE: 549.30 MB >> SUM: 1955.19 MB >> ------------------------ >> MISSING: 72.21 MB >> >> PHYSMEM: 2027.41 MB >> ACTIVE: 61.14 MB >> INACTIVE: 40.44 MB >> WIRED: 1303.86 MB >> CACHED: .50 MB >> FREE: 548.80 MB >> SUM: 1954.76 MB >> ------------------------ >> MISSING: 72.64 MB >> >> This behaviour has been seen while copying 600MB file from SAMBA share >> with sendfile enabled. >> It doesn't happen when writing to samba share, and it doesn't happen >> with sendfile disabled, both ways. >> For me it looks like memory which leaks should be added to wired pool >> and belongs to ARC, but appareantly this doesn't work well and WIRED: >> 1303.86 MB all the time. >> >> [1] http://pastebin.com/sQUyQbmm > > Try this. I the similar fix is needed for tmpfs, but there are some > more issues and pending rewrite, so I decided not to touch it. > > commit 8e5885bce1afecd419e40240a2d7ab90deb0392a > Author: Konstantin Belousov > Date: Tue Feb 8 22:35:29 2011 +0200 > > Do not forget to activate the page > > diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c > index e8191b3..7343c72 100644 > --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c > +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c > @@ -353,6 +353,9 @@ page_unlock(vm_page_t pp) > { > > vm_page_wakeup(pp); > + vm_page_lock(pp); > + vm_page_activate(pp); > + vm_page_unlock(pp); > } > > static caddr_t > @@ -480,7 +483,7 @@ again: > if (error == 0) > uiomove_fromphys(&m, off, bytes, uio); > VM_OBJECT_LOCK(obj); > - vm_page_wakeup(m); > + page_unlock(m); > } else if (uio->uio_segflg == UIO_NOCOPY) { > /* > * The code below is here to make sendfile(2) work > @@ -527,9 +530,15 @@ again: > zfs_unmap_page(sf); > } > VM_OBJECT_LOCK(obj); > - if (error == 0) > - m->valid = VM_PAGE_BITS_ALL; > vm_page_io_finish(m); > + vm_page_lock(m); > + if (error == 0) { > + m->valid = VM_PAGE_BITS_ALL; > + vm_page_activate(m); > + } else > + vm_page_free(m); > + vm_page_unlock(m); > + > if (error == 0) { > uio->uio_resid -= bytes; > uio->uio_offset += bytes; From owner-freebsd-fs@FreeBSD.ORG Wed Feb 9 20:18:33 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3450C1065674; Wed, 9 Feb 2011 20:18:33 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.garage.freebsd.pl (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 2F2758FC0A; Wed, 9 Feb 2011 20:18:31 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 77C4645C9F; Wed, 9 Feb 2011 21:18:30 +0100 (CET) Received: from localhost (89-73-195-149.dynamic.chello.pl [89.73.195.149]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 8FF2C4569A; Wed, 9 Feb 2011 21:18:25 +0100 (CET) Date: Wed, 9 Feb 2011 21:18:09 +0100 From: Pawel Jakub Dawidek To: Martin Matuska Message-ID: <20110209201809.GA3433@garage.freebsd.pl> References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> <4D511F65.2050503@kkip.pl> <4D519F97.2000805@kkip.pl> <20110208203653.GC78089@deviant.kiev.zoral.com.ua> <4D52F498.2090000@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="wRRV7LY7NUeQGEoC" Content-Disposition: inline In-Reply-To: <4D52F498.2090000@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 20:18:33 -0000 --wRRV7LY7NUeQGEoC Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Feb 09, 2011 at 09:10:00PM +0100, Martin Matuska wrote: > I think this should go to HEAD and I have already prepared ported > patches for v28 (head and stable). Did someone who sees the issue confirm it fixes the problem for him? If yes, I'm fine with the patch going in. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --wRRV7LY7NUeQGEoC Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk1S9oEACgkQForvXbEpPzQxQgCgtfHxYvIK4P/fy+B4LOXBpmWn SxUAoIQi37KBgexTTCg4b1ZDiR12Chv6 =tlKX -----END PGP SIGNATURE----- --wRRV7LY7NUeQGEoC-- From owner-freebsd-fs@FreeBSD.ORG Wed Feb 9 21:11:10 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8952A1065697; Wed, 9 Feb 2011 21:11:10 +0000 (UTC) (envelope-from gpm@hotplug.ru) Received: from shadow.rusoft.ru (shadow.rusoft.ru [217.16.18.75]) by mx1.freebsd.org (Postfix) with ESMTP id 3AC778FC0C; Wed, 9 Feb 2011 21:11:10 +0000 (UTC) Received: from shadow.rusoft.ru (localhost [127.0.0.1]) by shadow.rusoft.ru (Postfix) with ESMTP id 971613982; Thu, 10 Feb 2011 00:11:08 +0300 (MSK) Received: from jazz.rusoft.ru (unknown [83.222.3.162]) by shadow.rusoft.ru (Postfix) with ESMTP id 78CB93981; Thu, 10 Feb 2011 00:11:08 +0300 (MSK) Received: from ghost-pc.home.lan (ghostmaster.static.corbina.ru [78.107.250.255]) by jazz.rusoft.ru (Postfix) with ESMTPA id 65DE51CC00F; Thu, 10 Feb 2011 00:11:07 +0300 (MSK) Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: "Martin Matuska" , "Pawel Jakub Dawidek" References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> <4D511F65.2050503@kkip.pl> <4D519F97.2000805@kkip.pl> <20110208203653.GC78089@deviant.kiev.zoral.com.ua> <4D52F498.2090000@FreeBSD.org> <20110209201809.GA3433@garage.freebsd.pl> Date: Thu, 10 Feb 2011 00:11:09 +0300 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Emil Muratov" Message-ID: In-Reply-To: <20110209201809.GA3433@garage.freebsd.pl> User-Agent: Opera Mail/11.01 (Win32) X-Virus-Scanned: ClamAV using ClamSMTP Cc: freebsd-fs@freebsd.org Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 21:11:10 -0000 > On Wed, Feb 09, 2011 at 09:10:00PM +0100, Martin Matuska wrote: >> I think this should go to HEAD and I have already prepared ported >> patches for v28 (head and stable). > > Did someone who sees the issue confirm it fixes the problem for him? > If yes, I'm fine with the patch going in. > Yes, I confirm. I had applied STABLE version and rebuild kernel. Some quick tests confirmed that memory leaking is gone, samba with aio sendfile works fine. Will do more other tests later, but I'd say thanks to all involved for such a quick fix! :) From owner-freebsd-fs@FreeBSD.ORG Wed Feb 9 22:17:21 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D2AB106564A; Wed, 9 Feb 2011 22:17:21 +0000 (UTC) (envelope-from admin@kkip.pl) Received: from mainframe.kkip.pl (kkip.pl [87.105.164.78]) by mx1.freebsd.org (Postfix) with ESMTP id AC9DD8FC15; Wed, 9 Feb 2011 22:17:20 +0000 (UTC) Received: from static-78-8-144-74.ssp.dialog.net.pl ([78.8.144.74] helo=[192.168.0.2]) by mainframe.kkip.pl with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.73 (FreeBSD)) (envelope-from ) id 1PnILW-00074J-LC; Wed, 09 Feb 2011 23:17:18 +0100 Message-ID: <4D531262.6060900@kkip.pl> Date: Wed, 09 Feb 2011 23:17:06 +0100 From: Bartosz Stec User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: Emil Muratov References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> <4D511F65.2050503@kkip.pl> <4D519F97.2000805@kkip.pl> <20110208203653.GC78089@deviant.kiev.zoral.com.ua> <4D52F498.2090000@FreeBSD.org> <20110209201809.GA3433@garage.freebsd.pl> In-Reply-To: X-Authenticated-User: admin@kkip.pl X-Authenticator: plain X-Sender-Verify: SUCCEEDED (sender exists & accepts mail) X-Spam-Score: -8.5 X-Spam-Score-Int: -84 X-Exim-Version: 4.73 (build at 10-Jan-2011 16:29:01) X-Date: 2011-02-09 23:17:18 X-Connected-IP: 78.8.144.74:55479 X-Message-Linecount: 121 X-Body-Linecount: 106 X-Message-Size: 4105 X-Body-Size: 3153 X-Received-Count: 1 X-Recipient-Count: 4 X-Local-Recipient-Count: 4 X-Local-Recipient-Defer-Count: 0 X-Local-Recipient-Fail-Count: 0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 22:17:21 -0000 W dniu 2011-02-09 22:11, Emil Muratov pisze: > > >> On Wed, Feb 09, 2011 at 09:10:00PM +0100, Martin Matuska wrote: >>> I think this should go to HEAD and I have already prepared ported >>> patches for v28 (head and stable). >> >> Did someone who sees the issue confirm it fixes the problem for him? >> If yes, I'm fine with the patch going in. >> > > Yes, I confirm. I had applied STABLE version and rebuild kernel. Some > quick tests confirmed that memory leaking is gone, samba with aio > sendfile works fine. Will do more other tests later, but I'd say > thanks to all involved for such a quick fix! :) Yes, I confirm too, now MISSING count seems to be stable with heavy reading from samba: # ./mem_monitor -m -r 2 MISSING: 36.70 MB MISSING: 36.65 MB MISSING: 36.64 MB MISSING: 36.59 MB MISSING: 36.64 MB MISSING: 36.59 MB MISSING: 36.51 MB MISSING: 36.52 MB MISSING: 36.60 MB MISSING: 36.60 MB MISSING: 36.59 MB MISSING: 36.59 MB MISSING: 36.59 MB -- Bartosz Stec From owner-freebsd-fs@FreeBSD.ORG Thu Feb 10 12:25:09 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0E71E1065672 for ; Thu, 10 Feb 2011 12:25:09 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id B5F1D8FC16 for ; Thu, 10 Feb 2011 12:25:08 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1PnVa7-0004TT-5i for freebsd-fs@freebsd.org; Thu, 10 Feb 2011 13:25:07 +0100 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 10 Feb 2011 13:25:07 +0100 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 10 Feb 2011 13:25:07 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Thu, 10 Feb 2011 13:24:54 +0100 Lines: 9 Message-ID: References: <4D510BBB.1060708@kkip.pl> <20110208102727.GA8555@icarus.home.lan> <4D511F65.2050503@kkip.pl> <4D519F97.2000805@kkip.pl> <20110208203653.GC78089@deviant.kiev.zoral.com.ua> <20110208213721.GE78089@deviant.kiev.zoral.com.ua> <4D51BAFC.50806@kkip.pl> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.12) Gecko/20101102 Thunderbird/3.1.6 In-Reply-To: <4D51BAFC.50806@kkip.pl> X-Enigmail-Version: 1.1.2 Subject: Re: Memory leak in ZFS? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 12:25:09 -0000 On 08/02/2011 22:51, Bartosz Stec wrote: >> Index: cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c > > Now what's the recommended way to apply above changes to system (without > rebuilding whole universe...)? This is a kernel change - you only need to rebuild the kernel. From owner-freebsd-fs@FreeBSD.ORG Thu Feb 10 16:53:07 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D776106564A; Thu, 10 Feb 2011 16:53:07 +0000 (UTC) (envelope-from gleb.kurtsou@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4A3E98FC0A; Thu, 10 Feb 2011 16:53:05 +0000 (UTC) Received: by fxm16 with SMTP id 16so1675372fxm.13 for ; Thu, 10 Feb 2011 08:53:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:date:from:to:cc:subject:message-id:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=/UYza3rj0sq7Qn10VHYSj/uDWGzyA/BiNAJ3IslxEGs=; b=TRF+E7SJFoZdg8ljnAmsqPQY1CRFYxsCFAW/PqNwEvVlk+my5LeUSjxNIA5jTtb5J6 HomJFqopOAOxy7eEEr2wqH0mns3QBRXCG1UWW5vMIo1GJLQAZC2gllxArxqX6meR0cnS 19AkGQ6MriX3c1i8ovxVQjvpm7MtghDVscPUE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=u8EwzFIASZ5P65iizY5QSPqaiyNw8pcbp7p+VmYRgp3m0WlhzbRoj5jqlGcZ+NK5YD zefnMXexLCprHZQHzYrgvlfSQM6rcXuqqs/fSTyz8bYgdNrRPs3C3M8AfMLpDAtHUwFe EsqJVyD/nlEvgXrb2xGawD3xA2Ea+0ORcOBrY= Received: by 10.223.101.135 with SMTP id c7mr14121354fao.76.1297356785003; Thu, 10 Feb 2011 08:53:05 -0800 (PST) Received: from localhost ([91.187.7.158]) by mx.google.com with ESMTPS id z1sm128338fau.21.2011.02.10.08.53.01 (version=SSLv3 cipher=OTHER); Thu, 10 Feb 2011 08:53:03 -0800 (PST) Date: Thu, 10 Feb 2011 18:52:38 +0200 From: Gleb Kurtsou To: Ivan Voras Message-ID: <20110210165237.GA15601@tops> References: <4D36A2CF.1080508@fsn.hu> <20110119084648.GA28278@icarus.home.lan> <4D36B85B.8070201@fsn.hu> <20110119150200.GY2518@deviant.kiev.zoral.com.ua> <20110207133748.GA16327@tops.skynet.lt> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="IS0zKkzwUGydFO0o" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: tmpfs is zero bytes (no free space), maybe a zfs bug? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 16:53:07 -0000 --IS0zKkzwUGydFO0o Content-Type: text/plain; charset=utf-8 Content-Disposition: inline On (07/02/2011 15:35), Ivan Voras wrote: > On 7 February 2011 14:37, Gleb Kurtsou wrote: > > > It's up to user to mount tmpfs filesystems of reasonable size to prevent > > resource exhaustion. Anyway, enormously large tmpfs killing all your > > process is not the way to go. > > Of course not, but as I see it (from admin perspective), tmpfs should > behave as close to regular processes in consuming memory as possible > (where possible; obviously it cannot be subject to the OOM killer :) > ). Could you test the patch. It sets file system size to half of RAM by default and makes tmpfs behave much like regular process for vm subsystem. It no longer depends on inactive/wired memory stats, but checks if swap is nearly full. I've added vfs.tmpfs.swap_reserved sysctl to limit tmpfs growth. In my tests system didn't panic nor invoked OOM killer while consuming all available ram and swap. Unfortunately I wasn't able to test it with ZFS, I'd appreciate if you could run several test to see how ZFS and tmpfs will behave in low memory situation. If it works as expected I'm going to implement resize feature, update man page and change mount option parsing to allow specifying size in human readable form, e.g. size=1g. Thanks, Gleb. --IS0zKkzwUGydFO0o Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="tmpfs-memfix.patch.txt" commit 185bc042f0647a38b86aa78c5dda25a4bf0ea3dd Author: Gleb Kurtsou Date: Thu Feb 10 18:38:44 2011 +0200 tmpfs: Change the way available memory is calculated Try to allocate pages until filesystem size limit hit, fail in low memory situation. By default set filesystem size to half of available memory Add vfs.tmpfs.swap_reserved sysctl; set default to 2048 pages (8m or 16m) Check if free pages available before allocating new node Reorganize limits and mount option parsing diff --git a/sys/fs/tmpfs/tmpfs.h b/sys/fs/tmpfs/tmpfs.h index b1c4249..07f521c 100644 --- a/sys/fs/tmpfs/tmpfs.h +++ b/sys/fs/tmpfs/tmpfs.h @@ -487,61 +487,30 @@ int tmpfs_truncate(struct vnode *, off_t); * Memory management stuff. */ -/* Amount of memory pages to reserve for the system (e.g., to not use by - * tmpfs). - * XXX: Should this be tunable through sysctl, for instance? */ -#define TMPFS_PAGES_RESERVED (4 * 1024 * 1024 / PAGE_SIZE) - /* - * Returns information about the number of available memory pages, - * including physical and virtual ones. - * - * Remember to remove TMPFS_PAGES_RESERVED from the returned value to avoid - * excessive memory usage. - * + * Number of reserved swap pages should not be lower than + * swap_pager_almost_full high water mark. */ +#define TMPFS_SWAP_MINRESERVED 1024 + static __inline size_t -tmpfs_mem_info(void) +tmpfs_pages_max(struct tmpfs_mount *tmp) { - size_t size; - - size = swap_pager_avail + cnt.v_free_count + cnt.v_inactive_count; - size -= size > cnt.v_wire_count ? cnt.v_wire_count : size; - return size; + return (tmp->tm_pages_max); } -/* Returns the maximum size allowed for a tmpfs file system. This macro - * must be used instead of directly retrieving the value from tm_pages_max. - * The reason is that the size of a tmpfs file system is dynamic: it lets - * the user store files as long as there is enough free memory (including - * physical memory and swap space). Therefore, the amount of memory to be - * used is either the limit imposed by the user during mount time or the - * amount of available memory, whichever is lower. To avoid consuming all - * the memory for a given mount point, the system will always reserve a - * minimum of TMPFS_PAGES_RESERVED pages, which is also taken into account - * by this macro (see above). */ static __inline size_t -TMPFS_PAGES_MAX(struct tmpfs_mount *tmp) +tmpfs_pages_used(struct tmpfs_mount *tmp) { - size_t freepages; - - freepages = tmpfs_mem_info(); - freepages -= freepages < TMPFS_PAGES_RESERVED ? - freepages : TMPFS_PAGES_RESERVED; + const size_t node_size = sizeof(struct tmpfs_node) + + sizeof(struct tmpfs_dirent); + size_t meta_pages; - return MIN(tmp->tm_pages_max, freepages + tmp->tm_pages_used); + meta_pages = howmany((uintmax_t)tmp->tm_nodes_inuse * node_size, + PAGE_SIZE); + return (meta_pages + tmp->tm_pages_used); } -/* Returns the available space for the given file system. */ -#define TMPFS_META_PAGES(tmp) (howmany((tmp)->tm_nodes_inuse * (sizeof(struct tmpfs_node) \ - + sizeof(struct tmpfs_dirent)), PAGE_SIZE)) -#define TMPFS_FILE_PAGES(tmp) ((tmp)->tm_pages_used) - -#define TMPFS_PAGES_AVAIL(tmp) (TMPFS_PAGES_MAX(tmp) > \ - TMPFS_META_PAGES(tmp)+TMPFS_FILE_PAGES(tmp)? \ - TMPFS_PAGES_MAX(tmp) - TMPFS_META_PAGES(tmp) \ - - TMPFS_FILE_PAGES(tmp):0) - #endif /* --------------------------------------------------------------------- */ diff --git a/sys/fs/tmpfs/tmpfs_subr.c b/sys/fs/tmpfs/tmpfs_subr.c index 84a2038..259e205 100644 --- a/sys/fs/tmpfs/tmpfs_subr.c +++ b/sys/fs/tmpfs/tmpfs_subr.c @@ -41,6 +41,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -55,6 +56,60 @@ __FBSDID("$FreeBSD$"); #include #include +static long tmpfs_swap_reserved = TMPFS_SWAP_MINRESERVED * 2; + +SYSCTL_NODE(_vfs, OID_AUTO, tmpfs, CTLFLAG_RW, 0, "tmpfs memory file system"); + +static int +sysctl_swap_reserved(SYSCTL_HANDLER_ARGS) +{ + int error; + long pages, bytes; + + pages = *(long *)arg1; + bytes = pages * PAGE_SIZE; + + error = sysctl_handle_long(oidp, &bytes, 0, req); + if (error || !req->newptr) + return (error); + + pages = bytes / PAGE_SIZE; + if (pages < TMPFS_SWAP_MINRESERVED) + return (EINVAL); + + *(long *)arg1 = pages; + return (0); +} + +SYSCTL_PROC(_vfs_tmpfs, OID_AUTO, swap_reserved, CTLTYPE_LONG|CTLFLAG_RW, + &tmpfs_swap_reserved, 0, sysctl_swap_reserved, "L", "reserved swap space"); + +static __inline size_t +tmpfs_pages_avail(struct tmpfs_mount *tmp, size_t req_pages) +{ + vm_ooffset_t avail; + + if (tmpfs_pages_max(tmp) < tmpfs_pages_used(tmp) + req_pages) + return (0); + + if (!vm_page_count_target()) + return (1); + + /* + * Fail if pagedaemon wasn't able to free desired number of pages and + * we are running out of swap. + */ + avail = swap_pager_avail - vm_paging_target() - req_pages; + if (avail < tmpfs_swap_reserved) { /* avail is signed */ + printf("tmpfs: low memory: available %jd, " + "paging target %d, requested %zd\n", + (intmax_t)swap_pager_avail, vm_paging_target(), req_pages); + return (0); + } + + return (1); +} + /* --------------------------------------------------------------------- */ /* @@ -95,6 +150,8 @@ tmpfs_alloc_node(struct tmpfs_mount *tmp, enum vtype type, if (tmp->tm_nodes_inuse >= tmp->tm_nodes_max) return (ENOSPC); + if (tmpfs_pages_avail(tmp, 1) == 0) + return (ENOSPC); nnode = (struct tmpfs_node *)uma_zalloc_arg( tmp->tm_node_pool, tmp, M_WAITOK); @@ -905,7 +962,7 @@ tmpfs_reg_resize(struct vnode *vp, off_t newsize) newpages = round_page(newsize) / PAGE_SIZE; if (newpages > oldpages && - newpages - oldpages > TMPFS_PAGES_AVAIL(tmp)) { + tmpfs_pages_avail(tmp, newpages - oldpages) == 0) { error = ENOSPC; goto out; } diff --git a/sys/fs/tmpfs/tmpfs_vfsops.c b/sys/fs/tmpfs/tmpfs_vfsops.c index 356be5e..128200f 100644 --- a/sys/fs/tmpfs/tmpfs_vfsops.c +++ b/sys/fs/tmpfs/tmpfs_vfsops.c @@ -129,14 +129,14 @@ tmpfs_node_fini(void *mem, int size) static int tmpfs_mount(struct mount *mp) { + const size_t nodes_per_page = howmany(PAGE_SIZE, + sizeof(struct tmpfs_dirent) + sizeof(struct tmpfs_node)); struct tmpfs_mount *tmp; struct tmpfs_node *root; - size_t pages; - uint32_t nodes; int error; /* Size counters. */ - u_int nodes_max; - u_quad_t size_max, maxfilesize; + u_quad_t pages; + u_quad_t nodes_max, size_max, maxfilesize; /* Root node attributes. */ uid_t root_uid; @@ -173,7 +173,7 @@ tmpfs_mount(struct mount *mp) if (mp->mnt_cred->cr_ruid != 0 || vfs_scanopt(mp->mnt_optnew, "mode", "%ho", &root_mode) != 1) root_mode = va.va_mode; - if (vfs_scanopt(mp->mnt_optnew, "inodes", "%u", &nodes_max) != 1) + if (vfs_scanopt(mp->mnt_optnew, "inodes", "%qu", &nodes_max) != 1) nodes_max = 0; if (vfs_scanopt(mp->mnt_optnew, "size", "%qu", &size_max) != 1) size_max = 0; @@ -181,38 +181,49 @@ tmpfs_mount(struct mount *mp) &maxfilesize) != 1) maxfilesize = 0; - /* Do not allow mounts if we do not have enough memory to preserve - * the minimum reserved pages. */ - if (tmpfs_mem_info() < TMPFS_PAGES_RESERVED) + /* + * XXX Deny mounts if pagedaemon wasn't able to recovery desired + * number of pages. + */ + if (vm_page_count_target()) return ENOSPC; /* Get the maximum number of memory pages this file system is * allowed to use, based on the maximum size the user passed in - * the mount structure. A value of zero is treated as if the - * maximum available space was requested. */ - if (size_max < PAGE_SIZE || size_max > SIZE_MAX - PAGE_SIZE) - pages = SIZE_MAX; + * the mount structure. Use half of RAM by default. */ + if (size_max < PAGE_SIZE*4 || size_max > SIZE_MAX - PAGE_SIZE) + pages = cnt.v_page_count / 2; else pages = howmany(size_max, PAGE_SIZE); MPASS(pages > 0); + MPASS(pages < SIZE_MAX); + + if (pages < SIZE_MAX / PAGE_SIZE) + size_max = pages * PAGE_SIZE; + else + size_max = SIZE_MAX; if (nodes_max <= 3) { - if (pages > UINT32_MAX - 3) - nodes = UINT32_MAX; + if (pages < UINT32_MAX / nodes_per_page) + nodes_max = pages * nodes_per_page; else - nodes = pages + 3; - } else - nodes = nodes_max; - MPASS(nodes >= 3); + nodes_max = UINT32_MAX; + } + if (nodes_max > UINT32_MAX) + nodes_max = UINT32_MAX; + MPASS(nodes_max >= 3); + + if (maxfilesize < PAGE_SIZE || maxfilesize > size_max) + maxfilesize = size_max; /* Allocate the tmpfs mount structure and fill it. */ tmp = (struct tmpfs_mount *)malloc(sizeof(struct tmpfs_mount), M_TMPFSMNT, M_WAITOK | M_ZERO); mtx_init(&tmp->allnode_lock, "tmpfs allnode lock", NULL, MTX_DEF); - tmp->tm_nodes_max = nodes; + tmp->tm_nodes_max = nodes_max; tmp->tm_nodes_inuse = 0; - tmp->tm_maxfilesize = maxfilesize > 0 ? maxfilesize : UINT64_MAX; + tmp->tm_maxfilesize = maxfilesize; LIST_INIT(&tmp->tm_nodes_used); tmp->tm_pages_max = pages; @@ -381,22 +392,23 @@ tmpfs_fhtovp(struct mount *mp, struct fid *fhp, struct vnode **vpp) static int tmpfs_statfs(struct mount *mp, struct statfs *sbp) { - fsfilcnt_t freenodes; struct tmpfs_mount *tmp; + size_t used; tmp = VFS_TO_TMPFS(mp); sbp->f_iosize = PAGE_SIZE; sbp->f_bsize = PAGE_SIZE; - sbp->f_blocks = TMPFS_PAGES_MAX(tmp); - sbp->f_bavail = sbp->f_bfree = TMPFS_PAGES_AVAIL(tmp); - - freenodes = MIN(tmp->tm_nodes_max - tmp->tm_nodes_inuse, - TMPFS_PAGES_AVAIL(tmp) * PAGE_SIZE / sizeof(struct tmpfs_node)); - - sbp->f_files = freenodes + tmp->tm_nodes_inuse; - sbp->f_ffree = freenodes; + sbp->f_blocks = tmpfs_pages_max(tmp); + used = tmpfs_pages_used(tmp); + if (tmpfs_pages_max(tmp) <= used) + sbp->f_bavail = 0; + else + sbp->f_bavail = tmpfs_pages_max(tmp) - used; + sbp->f_bfree = sbp->f_bavail; + sbp->f_files = tmp->tm_nodes_max; + sbp->f_ffree = tmp->tm_nodes_max - tmp->tm_nodes_inuse; /* sbp->f_owner = tmp->tn_uid; */ return 0; --IS0zKkzwUGydFO0o-- From owner-freebsd-fs@FreeBSD.ORG Thu Feb 10 16:56:51 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2FBFC106566B; Thu, 10 Feb 2011 16:56:51 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (unknown [IPv6:2a01:348:0:15:5d59:5c40:0:1]) by mx1.freebsd.org (Postfix) with ESMTP id 582D98FC14; Thu, 10 Feb 2011 16:56:50 +0000 (UTC) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id 4A826E76A8; Thu, 10 Feb 2011 16:56:44 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cran.org.uk; h=date:from :to:cc:subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=mail; bh=pAvIZN0EdlMD S6SycAio4YvY9DY=; b=Nmgxyjb93vt+qWJ+vmWfjwu8uywxWY6Vs/VjLtREeE3U rLp2EWfCihLxUSJNvPGRWD0BRHD35Xy+QQQwSSj8MaVaEXPPUG1zNk0fSZOHsaKZ Qi4atO2j/s+YwoI6CF+bMHWXysJw9cIObv/hjfJwA3gRZgkPv1KiuKZGi1s2A24= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cran.org.uk; h=date:from:to :cc:subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=mail; b=PGDVpN xkvXMIUP3LAXqq6TFZmwnjxyY5FVC/vGIiR4COGUCNNK7v4e+Fz1TJ9j3lBn4L3T esgIzLIoLgVJdi97PKn35CP6a6ifHMFNPRwjfH5wjSxuBoZvjxnsrRejC6HgMBgT 8WBXCbE1RedW7UkLSEFne7koxE7h2BBzlDDFY= Received: from unknown (client-86-23-69-78.brhm.adsl.virginmedia.com [86.23.69.78]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id 12CCEE7621; Thu, 10 Feb 2011 16:56:44 +0000 (GMT) Date: Thu, 10 Feb 2011 16:56:30 +0000 From: Bruce Cran To: Attila Nagy Message-ID: <20110210165630.00000ee8@unknown> In-Reply-To: <4D36B85B.8070201@fsn.hu> References: <4D36A2CF.1080508@fsn.hu> <20110119084648.GA28278@icarus.home.lan> <4D36B85B.8070201@fsn.hu> X-Mailer: Claws Mail 3.7.8cvs9 (GTK+ 2.16.6; i586-pc-mingw32msvc) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable Subject: Re: tmpfs is zero bytes (no free space), maybe a zfs bug? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 16:56:51 -0000 On Wed, 19 Jan 2011 11:09:31 +0100 Attila Nagy wrote: > On 01/19/11 09:46, Jeremy Chadwick wrote: > > On Wed, Jan 19, 2011 at 09:37:35AM +0100, Attila Nagy wrote: > >> I first noticed this problem on machines with more memory (32GB > >> eg.), but now it happens on 4G machines too: > >> tmpfs 0B 0B 0B > >> 100% /tmp > >> FreeBSD builder 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #0: Sat Jan > >> 8 22:11:54 CET 2011 > >> > >> Maybe it's related, that I use zfs on these machines... > >> > >> Sometimes it grows and shrinks, but generally there is no space > >> even for a small file, or a socket to create. > > http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/060867.html > > > Oh crap. :( > > I hope somebody can find the time to look into this, it's pretty > annoying... It's also listed as a bug on OpenSolaris: http://bugs.opensolaris.org/bugdatabase/view_bug.do;?bug_id=6804661 -- Bruce Cran From owner-freebsd-fs@FreeBSD.ORG Thu Feb 10 17:27:01 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 60270106564A; Thu, 10 Feb 2011 17:27:01 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id B42D08FC14; Thu, 10 Feb 2011 17:26:59 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id C5DF472A8DB; Thu, 10 Feb 2011 18:26:57 +0100 (CET) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000063, version=1.2.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 13.1730] X-CRM114-CacheID: sfid-20110210_18265_4C20929E X-CRM114-Status: Good ( pR: 13.1730 ) X-Spambayes-Classification: ham; 0.00 Message-ID: <4D541FE0.6090205@fsn.hu> Date: Thu, 10 Feb 2011 18:26:56 +0100 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Bruce Cran References: <4D36A2CF.1080508@fsn.hu> <20110119084648.GA28278@icarus.home.lan> <4D36B85B.8070201@fsn.hu> <20110210165630.00000ee8@unknown> In-Reply-To: <20110210165630.00000ee8@unknown> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable Subject: Re: tmpfs is zero bytes (no free space), maybe a zfs bug? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Feb 2011 17:27:01 -0000 On 02/10/2011 05:56 PM, Bruce Cran wrote: > On Wed, 19 Jan 2011 11:09:31 +0100 > Attila Nagy wrote: > >> On 01/19/11 09:46, Jeremy Chadwick wrote: >>> On Wed, Jan 19, 2011 at 09:37:35AM +0100, Attila Nagy wrote: >>>> I first noticed this problem on machines with more memory (32GB >>>> eg.), but now it happens on 4G machines too: >>>> tmpfs 0B 0B 0B >>>> 100% /tmp >>>> FreeBSD builder 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #0: Sat Jan >>>> 8 22:11:54 CET 2011 >>>> >>>> Maybe it's related, that I use zfs on these machines... >>>> >>>> Sometimes it grows and shrinks, but generally there is no space >>>> even for a small file, or a socket to create. >>> http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/060867.html >>> >> Oh crap. :( >> >> I hope somebody can find the time to look into this, it's pretty >> annoying... > It's also listed as a bug on OpenSolaris: > http://bugs.opensolaris.org/bugdatabase/view_bug.do;?bug_id=6804661 > ZFS is a great innovation, which forces sysadmins to learn kernel and VM internals. :-O From owner-freebsd-fs@FreeBSD.ORG Fri Feb 11 10:50:12 2011 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 45809106564A for ; Fri, 11 Feb 2011 10:50:12 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1AD628FC20 for ; Fri, 11 Feb 2011 10:50:12 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p1BAoB4L093496 for ; Fri, 11 Feb 2011 10:50:11 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p1BAoBSu093495; Fri, 11 Feb 2011 10:50:11 GMT (envelope-from gnats) Date: Fri, 11 Feb 2011 10:50:11 GMT Message-Id: <201102111050.p1BAoBSu093495@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Robert Schulze Cc: Subject: Re: kern/136944: [ffs] [lor] bufwait/snaplk (fsync) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Robert Schulze List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Feb 2011 10:50:12 -0000 The following reply was made to PR kern/136944; it has been noted by GNATS. From: Robert Schulze To: rene@freebsd.org Cc: bug-followup@freebsd.org Subject: Re: kern/136944: [ffs] [lor] bufwait/snaplk (fsync) Date: Fri, 11 Feb 2011 11:18:46 +0100 Hi, I can confirm that, too: lock order reversal: 1st 0xffffff0019291cc8 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:423 2nd 0xffffff81ef169978 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2636 3rd 0xffffff00074647e8 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:544 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x49 witness_checkorder() at witness_checkorder+0x7d3 __lockmgr_args() at __lockmgr_args+0xd0b ffs_lock() at ffs_lock+0xac VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x5d ffs_snapshot() at ffs_snapshot+0x1b79 ffs_mount() at ffs_mount+0x5c5 vfs_donmount() at vfs_donmount+0xcd4 nmount() at nmount+0x74 syscallenter() at syscallenter+0xe5 syscall() at syscall+0x55 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (378, FreeBSD ELF64, nmount), rip = 0x8007acfdc, rsp = 0x7fffffffe9a8, rbp = 0x800a04530 --- lock order reversal: 1st 0xffffff81ef06e6f8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2636 2nd 0xffffff0015bcc630 snaplk (snaplk) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:2223 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x49 witness_checkorder() at witness_checkorder+0x7d3 __lockmgr_args() at __lockmgr_args+0xd0b ffs_copyonwrite() at ffs_copyonwrite+0x161 ffs_geom_strategy() at ffs_geom_strategy+0x159 bufwrite() at bufwrite+0xff ffs_sbupdate() at ffs_sbupdate+0x94 ffs_sync() at ffs_sync+0x489 sync_fsync() at sync_fsync+0x136 VOP_FSYNC_APV() at VOP_FSYNC_APV+0xb5 sync_vnode() at sync_vnode+0x143 sched_sync() at sched_sync+0x1c6 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8245289cf0, rbp = 0 --- lock order reversal: 1st 0xffffff0015bcc630 snaplk (snaplk) @ /usr/src/sys/kern/vfs_vnops.c:296 2nd 0xffffff0019291cc8 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:1587 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 _witness_debugger() at _witness_debugger+0x49 witness_checkorder() at witness_checkorder+0x7d3 __lockmgr_args() at __lockmgr_args+0xd0b ffs_snapremove() at ffs_snapremove+0xe2 softdep_releasefile() at softdep_releasefile+0x133 ufs_inactive() at ufs_inactive+0x153 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb9 vinactive() at vinactive+0x89 vputx() at vputx+0x3c6 vn_close() at vn_close+0x10b vn_closefile() at vn_closefile+0x51 _fdrop() at _fdrop+0x20 closef() at closef+0x58 kern_close() at kern_close+0xff syscallenter() at syscallenter+0xe5 syscall() at syscall+0x55 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (6, FreeBSD ELF64, close), rip = 0x800844b1c, rsp = 0x7fffffffe9a8, rbp = 0 --- this was after hot-resetting, maybe due to a background fsck. with kind regards, Robert Schulze From owner-freebsd-fs@FreeBSD.ORG Fri Feb 11 11:22:56 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 930DE106566C for ; Fri, 11 Feb 2011 11:22:56 +0000 (UTC) (envelope-from drb@karlov.mff.cuni.cz) Received: from mail.karlov.mff.cuni.cz (mail.karlov.mff.cuni.cz [195.113.27.114]) by mx1.freebsd.org (Postfix) with ESMTP id 4DA5C8FC08 for ; Fri, 11 Feb 2011 11:22:56 +0000 (UTC) Received: from nat-r.karlov.mff.cuni.cz ([195.113.27.73] helo=[10.32.82.27]) by mail.karlov.mff.cuni.cz with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.68 (FreeBSD)) (envelope-from ) id 1Pnr5S-000F0a-PT; Fri, 11 Feb 2011 12:22:54 +0100 Message-ID: <4D551C0C.1050600@karlov.mff.cuni.cz> Date: Fri, 11 Feb 2011 12:22:52 +0100 From: =?UTF-8?B?VG9tw6HFoSBEcmJvaGxhdg==?= User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: Rick Macklem References: <1174528658.1552842.1297120107416.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <1174528658.1552842.1297120107416.JavaMail.root@erie.cs.uoguelph.ca> Content-Type: multipart/mixed; boundary="------------000904090805010105020403" Cc: freebsd-fs@freebsd.org Subject: Re: NFSv4 ACLs on NFS4 mount ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Feb 2011 11:22:56 -0000 This is a multi-part message in MIME format. --------------000904090805010105020403 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 8.2.2011 0:08, Rick Macklem wrote: >> Hello, >> >> I have 8.1-RELEASE running ZFS and NFS4 experimantal server and I >> wonder, what is the status of NFS4 ACLs? I see them work on ZFS >> fileystems directly mounted but when mounted via NFS4 (even on the >> same >> machine), getfacl returns 'old' POSIX form instead of NFSv4 ACLs. By >> digging mailng list archive I got to know that this 'should' work but >> for me it doesn't. >> > Thanks to George spotting it recently, they should work if you apply > the ACL related patches found at: > http://people.freebsd.org/~rmacklem > > For 8.1/ZFS, you'll see that there is one for the client side (adds > VOP_PATHCONF() support to the client), plus one for the server that > is in head, stable/8 but not 8.1. (It replaces a VOP_ACCESS() with a > VOP_ACCESSX().) > > Also, although I think that 8.1 has everything else you need except the > two patches, I'll admit that I haven't tested anything that is pre-8.2 > with these patches. (ie. You might also need an 8.2 upgrade, but hopefully > not.) Lovely, works as expected. Thanks! Only minor note, server-nfsv4acl.patch expect slightly different code before (and after): ... NFSVOPLOCK(vp, LK_EXCLUSIVE | LK_RETRY, p); ... in 8.1-p1 vs. ... if (vn_lock(vp, LK_SHARED) == 0) { ... I attach patch against 8.1-p1 I finally used. T:D > Good luck with them, rick --------------000904090805010105020403 Content-Type: text/x-patch; name="server-nfsv4acl-8.1rel-p1.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="server-nfsv4acl-8.1rel-p1.patch" --- nfs_commonsubs.c 2011-02-08 10:03:10.000000000 +0100 +++ nfs_commonsubs.c.orig 2010-06-14 04:09:06.000000000 +0200 @@ -1987,7 +1987,7 @@ #ifdef NFS4_ACL_EXTATTR_NAME } else if (naclp != NULL) { NFSVOPLOCK(vp, LK_EXCLUSIVE | LK_RETRY, p); - error = VOP_ACCESSX(vp, VREAD_ACL, cred, p); + error = VOP_ACCESS(vp, VREAD_ACL, cred, p); if (error == 0) error = VOP_GETACL(vp, ACL_TYPE_NFS4, naclp, cred, p); --------------000904090805010105020403-- From owner-freebsd-fs@FreeBSD.ORG Sat Feb 12 01:20:07 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CDFA0106564A; Sat, 12 Feb 2011 01:20:07 +0000 (UTC) (envelope-from gleb.kurtsou@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 324A28FC0C; Sat, 12 Feb 2011 01:20:06 +0000 (UTC) Received: by fxm16 with SMTP id 16so3600751fxm.13 for ; Fri, 11 Feb 2011 17:20:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:date:from:to:cc:subject:message-id:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=Fb8bH16kHkNThhn/Pr61q+0DAtrvIRJcx4eVF4dKyV8=; b=miVZIngBdZHSCNzlYoOmTIu+a6vU3K4Gh/2dL0lPK6lFvGbLyyecFdt20HQLXEo8km DgpOpWv8BJlI4qplEbojzKUk5yglNd6qIWktpwVFH/vhJi77fqi89LWejM+XWQggPVn9 rPH8WGDJRG8ZizgR/2ZZXBv6IBu9DGkYFVyH8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=tMskGD0IkDWShS5DD6axoN4DFPD9EKoqUkCSaiPjlT0hVQF3l4dwMoADAqG6TFx0yP 1GydLfGknmN3IF1oK/zHmKPr4i/a9mgDjM0fqST+N5hkdlOfDSKw9qVt5BZhoQ1gAB8h iR9yf6Sg6NV5up27zTTcR2c4dPnTzKq7hn+s8= Received: by 10.223.98.200 with SMTP id r8mr514485fan.30.1297432640381; Fri, 11 Feb 2011 05:57:20 -0800 (PST) Received: from localhost ([91.187.13.237]) by mx.google.com with ESMTPS id n15sm370428fam.12.2011.02.11.05.57.18 (version=SSLv3 cipher=OTHER); Fri, 11 Feb 2011 05:57:19 -0800 (PST) Date: Fri, 11 Feb 2011 15:56:53 +0200 From: Gleb Kurtsou To: Bruce Cran Message-ID: <20110211135652.GA22733@tops> References: <4D36A2CF.1080508@fsn.hu> <20110119084648.GA28278@icarus.home.lan> <4D36B85B.8070201@fsn.hu> <20110210165630.00000ee8@unknown> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20110210165630.00000ee8@unknown> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org, freebsd-stable Subject: Re: tmpfs is zero bytes (no free space), maybe a zfs bug? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Feb 2011 01:20:07 -0000 On (10/02/2011 16:56), Bruce Cran wrote: > On Wed, 19 Jan 2011 11:09:31 +0100 > Attila Nagy wrote: > > I hope somebody can find the time to look into this, it's pretty > > annoying... > > It's also listed as a bug on OpenSolaris: > http://bugs.opensolaris.org/bugdatabase/view_bug.do;?bug_id=6804661 Could you try my patch I've mentioned above in the thread: http://marc.info/?l=freebsd-fs&m=129735686129438&w=2 I've reproduce test scenario in OpenSolaris bug report and it worked as expected for me. System: amd64, 4GB RAM, ~5GB swap /boot/loader.conf: vm.kmem_size="6G" vfs.zfs.prefetch_disable="1" vfs.zfs.txg.timeout="5" # mount -t tmpfs -o size=$((5*1024*1024*1024)) none /mnt # dd if=/dev/zero of=test bs=1m count=$((3*1024)) # dd if=test of=/dev/zero bs=1m # dd if=test of=/dev/zero bs=1m # dd if=test of=/dev/zero bs=1m top statistics: Mem: 429M Active, 272M Inact, 2889M Wired, 96K Cache, 1328K Buf, 196M Free Swap: 5120M Total, 5120M Free ZFS seems to consume most of RAM # cp test /mnt top statistics: Mem: 2808M Active, 247M Inact, 623M Wired, 104M Cache, 1328K Buf, 5052K Free Swap: 5120M Total, 619M Used, 4501M Free, 12% Inuse ZFS cache has shrinked, swap increased, most of tmpfs remains in memory # df -h /mnt Filesystem Size Used Avail Capacity Mounted on tmpfs 5.0G 3.0G 2.0G 60% /mnt Thanks, Gleb. > > -- > Bruce Cran > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Sat Feb 12 07:30:10 2011 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 834D6106566C for ; Sat, 12 Feb 2011 07:30:10 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 574538FC13 for ; Sat, 12 Feb 2011 07:30:10 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p1C7UA58045871 for ; Sat, 12 Feb 2011 07:30:10 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p1C7UA7k045869; Sat, 12 Feb 2011 07:30:10 GMT (envelope-from gnats) Date: Sat, 12 Feb 2011 07:30:10 GMT Message-Id: <201102120730.p1C7UA7k045869@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Carl Cc: Subject: Re: kern/154228: [md] md getting stuck in wdrain state X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Carl List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Feb 2011 07:30:10 -0000 The following reply was made to PR kern/154228; it has been noted by GNATS. From: Carl To: bug-followup@FreeBSD.org, k0802647@telus.net Cc: Subject: Re: kern/154228: [md] md getting stuck in wdrain state Date: Fri, 11 Feb 2011 23:26:05 -0800 For the sake of end users suffering from this problem, please elaborate on the patch. Carl / K0802647 From owner-freebsd-fs@FreeBSD.ORG Sat Feb 12 09:54:45 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2E369106566C for ; Sat, 12 Feb 2011 09:54:45 +0000 (UTC) (envelope-from freebsd-list@nikazam.com) Received: from mail-gw0-f54.google.com (mail-gw0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id E6B978FC08 for ; Sat, 12 Feb 2011 09:54:44 +0000 (UTC) Received: by gwj21 with SMTP id 21so1454869gwj.13 for ; Sat, 12 Feb 2011 01:54:44 -0800 (PST) MIME-Version: 1.0 Received: by 10.236.103.175 with SMTP id f35mr502798yhg.27.1297504483879; Sat, 12 Feb 2011 01:54:43 -0800 (PST) Received: by 10.147.32.10 with HTTP; Sat, 12 Feb 2011 01:54:43 -0800 (PST) In-Reply-To: <4D46D0CF.8090103@chreo.net> References: <4D0A09AF.3040005@FreeBSD.org> <4D46D0CF.8090103@chreo.net> Date: Sat, 12 Feb 2011 09:54:43 +0000 Message-ID: From: Nik A Azam To: freebsd-fs@freebsd.org, mm@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Feb 2011 09:54:45 -0000 Hi Martin, all I'm testing the ZFS v28 on FreeBSD stable (r218583M , ZFS patch from http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20110208-nopython.patch.xz) and been getting this panic everytime I issue any zfs/zpool command. This is 100% reproducible. panic: _sx_xlock_hard: recursed on non-recursive sx GEOM topology @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c:380 cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a kdb_backtrace() at kdb_backtrace+0x37 panic() at panic+0x182 _sx_xunlock_hard() at _sx_xunlock_hard _sx_xlock() at _sx_xlock+0xa9 vdev_geom_open_by_path() at vdev_geom_open_by_path+0x45 vdev_geom_open() at vdev_geom_open+0x100 vdeev_open() at vdeev_open+0xc9 vdev_open_children() at vdev_open_children+0x39 vdev_raidz_open() at vdev_raidz_open+0x4f vdev_open() at vdev_open+0xc9 vdev_open_children() at vdev_open_children+0x39 vdev_root_open() at vdev_root_open+0x40 vdev_open() at den)open+0xc9 spa_load() at spa_load+0x23f spa_load_best() at spa_load_best+0x4a pool_status_check() at pool_status_check+0x19 zfsdev_ioctl() at zfsdev_ioctl+0x208 devfs_ioctl_f() at devfs_ioctl_f+0x73 kern_ioctl() at kern_ioctl+0x8b ioctl() at 90ctl+0xec syscall() at syscall+0x41 Xfast_syscall() at Xfast_syscall+0x2e I'm more than happy to investigate this further given instructions on how to do so. Really appreciate the work that you guys have put in FreeBSD/ZFS! Thanks, Nik On Mon, Jan 31, 2011 at 3:10 PM, Chreo wrote: > Hello Martin, > > On 2010-12-16 13:44, Martin Matuska wrote: > >> Following the announcement of Pawel Jakub Dawidek (pjd@FreeBSD.org) I am >> providing a ZFSv28 testing patch for 8-STABLE. >> >> Link to the patch: >> >> >> http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101215.patch.xz >> >> > I've tested > http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20110116-nopython.patch.xz > with 8-STABLE from 2011-01-18 > > Seems to work nicely except for a panic when importing a degraded pool on > GELI vdevs: > (captured from the screen and OCR'd) > vdev.geom_detach:156[1]: Closing access to label/Disk.4.eli. > vdev.geom_detach:160[1]: Destroyed consumer to label/Disk.4.eli. > vdev.geomdetach:156[1]: Closing access to label/Disk.5.eli. > vdev.geom_detach:160[1]: Destroyed consumer to label/Disk.5.eli. > Solaris: WARNING: can't open objset for Ocean/Images > panic: solaris assert: bpobj_iterate(defer_bpo, spa_free_sync_cb, zio, tx) > == 0 (0x6 == 0x0), file: > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c, > line: 5576 > cpuid = 1 > KDB: stack backtrace: > #0 0xffffffff802f14ce at kdb_backtrace+0x5e > #1 0xffffffff802bf877 at panic+Ox187 > #2 0xffffffff808e0c48 at spa_sync+0x978 > #3 0xffffffff808f1011 at txg_sync_thread+0x271 > #4 0xffffffff802960b7 at fork_exit+0x117 > #5 0xffffffff804b7a7e at fork_trampoline+0xe > GEOM_ELI: Device label/Disk.5.eli destroyed. > GEOM_ELI: Device label/Disk.4.eli destroyed. > > The command run was: > # zpool import -F Ocean > and that worked with ZFS v15 > > The panic is 100% reproducible. The reason for this import was that I > wanted to try and clear the log (something which is possible on v28 but not > v15 it seems) with: zpool clear Ocean, and that caused a panic. An export > was done and the import was tried. Using the same command on v15 works and > imports the pool but it is faulted (due to the log). > > Anything I can test or do about this? I've also tried importing with -o > failmode=continue and that does absolutely nothing to prevent the panic. > > The other pool on the same system works perfectly so far with v28. Many > thanks to you and PJD for your work on ZFS. > > Regards, > Christian Elmerot > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Sat Feb 12 19:01:14 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 19E751065675; Sat, 12 Feb 2011 19:01:14 +0000 (UTC) (envelope-from freebsd-list@nikazam.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id BF4898FC18; Sat, 12 Feb 2011 19:01:13 +0000 (UTC) Received: by gyc15 with SMTP id 15so1566807gyc.13 for ; Sat, 12 Feb 2011 11:01:12 -0800 (PST) MIME-Version: 1.0 Received: by 10.150.178.20 with SMTP id a20mr2317063ybf.23.1297537272487; Sat, 12 Feb 2011 11:01:12 -0800 (PST) Received: by 10.147.32.10 with HTTP; Sat, 12 Feb 2011 11:01:12 -0800 (PST) In-Reply-To: References: <4D0A09AF.3040005@FreeBSD.org> <4D46D0CF.8090103@chreo.net> Date: Sat, 12 Feb 2011 19:01:12 +0000 Message-ID: From: Nik A Azam To: freebsd-fs@freebsd.org, mm@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Feb 2011 19:01:14 -0000 Ok, looking at SVN log there was a change to vdev_geom.c recently after mm's patch. I synced to revision r218540 and all is good. Sorry for the noise! nik On Sat, Feb 12, 2011 at 9:54 AM, Nik A Azam wrote: > Hi Martin, all > > I'm testing the ZFS v28 on FreeBSD stable (r218583M , ZFS patch from > http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20110208-nopython.patch.xz) > and been getting this panic everytime I issue any zfs/zpool command. This is > 100% reproducible. > > panic: _sx_xlock_hard: recursed on non-recursive sx GEOM topology @ > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c:380 > > > cpuid = 1 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > kdb_backtrace() at kdb_backtrace+0x37 > panic() at panic+0x182 > _sx_xunlock_hard() at _sx_xunlock_hard > _sx_xlock() at _sx_xlock+0xa9 > vdev_geom_open_by_path() at vdev_geom_open_by_path+0x45 > vdev_geom_open() at vdev_geom_open+0x100 > vdeev_open() at vdeev_open+0xc9 > vdev_open_children() at vdev_open_children+0x39 > vdev_raidz_open() at vdev_raidz_open+0x4f > vdev_open() at vdev_open+0xc9 > vdev_open_children() at vdev_open_children+0x39 > vdev_root_open() at vdev_root_open+0x40 > vdev_open() at den)open+0xc9 > spa_load() at spa_load+0x23f > spa_load_best() at spa_load_best+0x4a > pool_status_check() at pool_status_check+0x19 > zfsdev_ioctl() at zfsdev_ioctl+0x208 > devfs_ioctl_f() at devfs_ioctl_f+0x73 > kern_ioctl() at kern_ioctl+0x8b > ioctl() at 90ctl+0xec > syscall() at syscall+0x41 > Xfast_syscall() at Xfast_syscall+0x2e > > I'm more than happy to investigate this further given instructions on how > to do so. Really appreciate the work that you guys have put in FreeBSD/ZFS! > > Thanks, > Nik > > > On Mon, Jan 31, 2011 at 3:10 PM, Chreo wrote: > >> Hello Martin, >> >> On 2010-12-16 13:44, Martin Matuska wrote: >> >>> Following the announcement of Pawel Jakub Dawidek (pjd@FreeBSD.org) I am >>> providing a ZFSv28 testing patch for 8-STABLE. >>> >>> Link to the patch: >>> >>> >>> http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101215.patch.xz >>> >>> >> I've tested >> http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20110116-nopython.patch.xz >> with 8-STABLE from 2011-01-18 >> >> Seems to work nicely except for a panic when importing a degraded pool on >> GELI vdevs: >> (captured from the screen and OCR'd) >> vdev.geom_detach:156[1]: Closing access to label/Disk.4.eli. >> vdev.geom_detach:160[1]: Destroyed consumer to label/Disk.4.eli. >> vdev.geomdetach:156[1]: Closing access to label/Disk.5.eli. >> vdev.geom_detach:160[1]: Destroyed consumer to label/Disk.5.eli. >> Solaris: WARNING: can't open objset for Ocean/Images >> panic: solaris assert: bpobj_iterate(defer_bpo, spa_free_sync_cb, zio, tx) >> == 0 (0x6 == 0x0), file: >> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c, >> line: 5576 >> cpuid = 1 >> KDB: stack backtrace: >> #0 0xffffffff802f14ce at kdb_backtrace+0x5e >> #1 0xffffffff802bf877 at panic+Ox187 >> #2 0xffffffff808e0c48 at spa_sync+0x978 >> #3 0xffffffff808f1011 at txg_sync_thread+0x271 >> #4 0xffffffff802960b7 at fork_exit+0x117 >> #5 0xffffffff804b7a7e at fork_trampoline+0xe >> GEOM_ELI: Device label/Disk.5.eli destroyed. >> GEOM_ELI: Device label/Disk.4.eli destroyed. >> >> The command run was: >> # zpool import -F Ocean >> and that worked with ZFS v15 >> >> The panic is 100% reproducible. The reason for this import was that I >> wanted to try and clear the log (something which is possible on v28 but not >> v15 it seems) with: zpool clear Ocean, and that caused a panic. An export >> was done and the import was tried. Using the same command on v15 works and >> imports the pool but it is faulted (due to the log). >> >> Anything I can test or do about this? I've also tried importing with -o >> failmode=continue and that does absolutely nothing to prevent the panic. >> >> The other pool on the same system works perfectly so far with v28. Many >> thanks to you and PJD for your work on ZFS. >> >> Regards, >> Christian Elmerot >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> > > From owner-freebsd-fs@FreeBSD.ORG Sat Feb 12 21:51:27 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CB9A5106566B for ; Sat, 12 Feb 2011 21:51:27 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 8BA918FC16 for ; Sat, 12 Feb 2011 21:51:27 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApwEAFaPVk2DaFvO/2dsb2JhbACEHqJYqU6PXoEng0F2BIUEhns X-IronPort-AV: E=Sophos;i="4.60,463,1291611600"; d="scan'208";a="110683433" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 12 Feb 2011 16:51:26 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id C4F47B3F2A; Sat, 12 Feb 2011 16:51:26 -0500 (EST) Date: Sat, 12 Feb 2011 16:51:26 -0500 (EST) From: Rick Macklem To: =?utf-8?Q?Tom=C3=A1=C5=A1_Drbohlav?= Message-ID: <801619581.1863246.1297547486791.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <4D551C0C.1050600@karlov.mff.cuni.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE8 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org Subject: Re: NFSv4 ACLs on NFS4 mount ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Feb 2011 21:51:27 -0000 > > Lovely, works as expected. Thanks! > Good to hear. I'll admit neither George or I had tested ZFS. > Only minor note, server-nfsv4acl.patch expect slightly different code > before (and after): > > ... > NFSVOPLOCK(vp, LK_EXCLUSIVE | LK_RETRY, p); > ... > > in 8.1-p1 vs. > > ... > if (vn_lock(vp, LK_SHARED) == 0) { > ... > > I attach patch against 8.1-p1 I finally used. > Yep, your patch is fine. (All that needs to be done is replace the VOP_ACCESS() call with VOP_ACCESSX() as you've done.) rick