From owner-freebsd-fs@FreeBSD.ORG Sun Jun 27 04:18:52 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DEE21106566C; Sun, 27 Jun 2010 04:18:52 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B728D8FC08; Sun, 27 Jun 2010 04:18:52 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o5R4IqZf027962; Sun, 27 Jun 2010 04:18:52 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o5R4IqY4027958; Sun, 27 Jun 2010 04:18:52 GMT (envelope-from linimon) Date: Sun, 27 Jun 2010 04:18:52 GMT Message-Id: <201006270418.o5R4IqY4027958@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-amd64@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/148138: [zfs] zfs raidz pool commands freeze X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Jun 2010 04:18:53 -0000 Old Synopsis: zfs raidz pool commands freeze New Synopsis: [zfs] zfs raidz pool commands freeze Responsible-Changed-From-To: freebsd-amd64->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sun Jun 27 04:18:36 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=148138 From owner-freebsd-fs@FreeBSD.ORG Mon Jun 28 05:31:23 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5D3B41065672 for ; Mon, 28 Jun 2010 05:31:23 +0000 (UTC) (envelope-from peterjeremy@acm.org) Received: from mail16.syd.optusnet.com.au (mail16.syd.optusnet.com.au [211.29.132.197]) by mx1.freebsd.org (Postfix) with ESMTP id C800B8FC13 for ; Mon, 28 Jun 2010 05:31:22 +0000 (UTC) Received: from server.vk2pj.dyndns.org (c211-30-160-13.belrs4.nsw.optusnet.com.au [211.30.160.13]) by mail16.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id o5S5VJBu018483 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 28 Jun 2010 15:31:20 +1000 X-Bogosity: Ham, spamicity=0.000000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.4/8.14.4) with ESMTP id o5S5VHIg034516; Mon, 28 Jun 2010 15:31:17 +1000 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.4/8.14.4/Submit) id o5S5VHvE034515; Mon, 28 Jun 2010 15:31:17 +1000 (EST) (envelope-from peter) Date: Mon, 28 Jun 2010 15:31:17 +1000 From: Peter Jeremy To: =?iso-8859-1?Q?Micka=EBl?= Maillot Message-ID: <20100628053117.GA32123@server.vk2pj.dyndns.org> References: <20100625231708.GB29793@server.vk2pj.dyndns.org> <20100626141038.0d9f488a@r500.local> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="SUOF0GtieIMvvwua" Content-Disposition: inline In-Reply-To: 0;258;0cFrom: Peter Jeremy X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org Subject: Re: mdconfig on ZFS leaks disk space X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jun 2010 05:31:23 -0000 --SUOF0GtieIMvvwua Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable >> Peter Jeremy wrote: >> >>> I recently did a quick experiment to create an 8TB UFS filesystem >>> via mdconfig and after destroying the md and deleting the file, >>> the disk space used by the md was not returned - even after a >>> reboot. =A0Has anyone else seen this? Soem further experimenting showed that send|recv does not free up the space. On 2010-Jun-26 18:29:41 +0200, Micka=EBl Maillot wrote: >what is your svn rev ? >because r208869: Fix freeing space after deleting large files with holes >dated: Sun Jun 6 13:08:36 2010 Thanks. Upgrading to a recent -stable has fixed the problem. Even better, just booting an updated kernel and mounting the problematic ZFS is sufficient to release the extraneous free space. --=20 Peter Jeremy --SUOF0GtieIMvvwua Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkwoM6UACgkQ/opHv/APuIfWWQCfbs03u5R225FouI0FgRxpYvWl f40AoJEl4yV09drkY24H9omUlnUZfpOM =Ecwx -----END PGP SIGNATURE----- --SUOF0GtieIMvvwua-- From owner-freebsd-fs@FreeBSD.ORG Mon Jun 28 11:06:52 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2ACB1106566B for ; Mon, 28 Jun 2010 11:06:52 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 188D78FC1D for ; Mon, 28 Jun 2010 11:06:52 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o5SB6pAv086504 for ; Mon, 28 Jun 2010 11:06:51 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o5SB6pOg086502 for freebsd-fs@FreeBSD.org; Mon, 28 Jun 2010 11:06:51 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 28 Jun 2010 11:06:51 GMT Message-Id: <201006281106.o5SB6pOg086502@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jun 2010 11:06:52 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147790 fs [zfs] zfs set acl(mode|inherit) fails on existing zfs o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/147292 fs [nfs] [patch] readahead missing in nfs client options o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server o kern/146375 fs [nfs] [patch] Typos in macro variables names in sys/fs o kern/145778 fs [zfs] [panic] panic in zfs_fuid_map_id (known issue fi s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat s kern/145424 fs [zfs] [patch] move source closer to v15 o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an o kern/145309 fs [disklabel]: Editing disk label invalidates the whole o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c o kern/144458 fs [nfs] [patch] nfsd fails as a kld p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o kern/143345 fs [ext2fs] [patch] extfs minor header cleanups to better o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142924 fs [ext2fs] [patch] Small cleanup for the inode struct in o kern/142914 fs [zfs] ZFS performance degradation over time o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142401 fs [ntfs] [patch] Minor updates to NTFS from NetBSD o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140134 fs [msdosfs] write and fsck destroy filesystem integrity o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs o bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139597 fs [patch] [tmpfs] tmpfs initializes va_gen but doesn't u o kern/139564 fs [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/139363 fs [nfs] diskless root nfs mount from non FreeBSD server o kern/138790 fs [zfs] ZFS ceases caching when mem demand is high o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb f kern/137037 fs [zfs] [hang] zfs rollback on root causes FreeBSD to fr o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic o kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133614 fs [panic] panic: ffs_truncate: read-only filesystem o kern/133174 fs [msdosfs] [patch] msdosfs must support utf-encoded int f kern/133150 fs [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130229 fs [iconv] usermount fails on fs that need iconv o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/129059 fs [zfs] [patch] ZFS bootloader whitelistable via WITHOUT f kern/128829 fs smbd(8) causes periodic panic on 7-RELEASE o kern/127420 fs [gjournal] [panic] Journal overflow on gmirrored gjour o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS p kern/124621 fs [ext3] [patch] Cannot mount ext2fs partition f bin/124424 fs [zfs] zfs(8): zfs list -r shows strange snapshots' siz o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121779 fs [ufs] snapinfo(8) (and related tools?) only work for t o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha f kern/120991 fs [panic] [fs] [snapshot] System crashes when manipulati o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F f kern/119735 fs [zfs] geli + ZFS + samba starting on boot panics 7.0-B o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o bin/118249 fs mv(1): moving a directory changes its mtime o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116913 fs [ffs] [panic] ffs_blkfree: freeing free block p kern/116608 fs [msdosfs] [patch] msdosfs fails to check mount options o kern/116583 fs [ffs] [hang] System freezes for short time when using o kern/116170 fs [panic] Kernel panic when mounting /tmp o kern/115645 fs [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/106030 fs [ufs] [panic] panic in ufs from geom when a dead disk o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [iso9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna f kern/91568 fs [ufs] [panic] writing to UFS/softupdates DVD media in o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o kern/85326 fs [smbfs] [panic] saving a file via samba to an overquot o kern/84589 fs [2TB] 5.4-STABLE unresponsive during background fsck 2 o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/53137 fs [ffs] [panic] background fscking causing ffs_valloc pa o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/33464 fs [ufs] soft update inconsistencies after system crash o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 177 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Jun 28 11:38:42 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C44AF106566B for ; Mon, 28 Jun 2010 11:38:42 +0000 (UTC) (envelope-from bsd@nezmer.info) Received: from mail.nezmer.info (nezmer.info [97.107.142.36]) by mx1.freebsd.org (Postfix) with ESMTP id A6BAC8FC1B for ; Mon, 28 Jun 2010 11:38:42 +0000 (UTC) Date: Mon, 28 Jun 2010 14:38:37 +0300 From: Nezmer To: freebsd-fs@freebsd.org Message-ID: <20100628113837.GA98334@mail> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) Subject: nullmounting zfs fs with children X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jun 2010 11:38:42 -0000 Hi, Is this normal behaviour? # zfs create -p -o mountpoint=/zfstest/subdir0/subdir1/subdir2 POOL/zfstest/subdir0/subdir1/subdir2 # echo 2 > /zfstest/subdir0/subdir1/subdir2/file2 # echo 1 > /zfstest/subdir0/subdir1/file1 # find /zfstest /zfstest /zfstest/subdir0 /zfstest/subdir0/subdir1 /zfstest/subdir0/subdir1/file1 /zfstest/subdir0/subdir1/subdir2 /zfstest/subdir0/subdir1/subdir2/file2 # mkdir /nulltest # mount_nullfs /zfstest /nulltest # find /nulltest /nulltest /nulltest/subdir0 /nulltest/subdir0/subdir1 /nulltest/subdir0/subdir1/file1 /nulltest/subdir0/subdir1/subdir2 # echo 0 > /zfstest/subdir0/file0 # find /zfstest /zfstest /zfstest/subdir0 /zfstest/subdir0/subdir1 /zfstest/subdir0/subdir1/file1 /zfstest/subdir0/subdir1/subdir2 /zfstest/subdir0/subdir1/subdir2/file2 /zfstest/subdir0/file0 # find /nulltest /nulltest /nulltest/subdir0 /nulltest/subdir0/subdir1 /nulltest/subdir0/subdir1/file1 /nulltest/subdir0/subdir1/subdir2 /nulltest/subdir0/file0 # umount /nulltest # zfs destroy -r POOL/zfstest # find /zfstest /zfstest /zfstest/subdir0 /zfstest/subdir0/subdir1 /zfstest/subdir0/subdir1/file1 /zfstest/subdir0/subdir1/subdir2 /zfstest/subdir0/file0 # find /nulltest /nulltest /nulltest/subdir0 /nulltest/subdir0/subdir1 /nulltest/subdir0/subdir1/file1 /nulltest/subdir0/subdir1/subdir2 /nulltest/subdir0/file0 I noticed this behaviour when I wanted to nullmount my "/usr/home" inside a chroot: # mount|grep /usr/home POOL/usr/home on /usr/home (zfs, local, noatime) POOL/usr/home/nezmer on /usr/home/nezmer (zfs, local, noatime) POOL/usr/home/nezmer/Mail on /usr/home/nezmer/Mail (zfs, local, noatime) POOL/usr/home/nezmer/pkgs on /usr/home/nezmer/pkgs (zfs, local, noatime) POOL/usr/home/nezmer/srcs on /usr/home/nezmer/srcs (zfs, local, noatime) /usr/home on /chroot/usr/home (nullfs, local) # find /usr/home -maxdepth 4|wc 7267 7289 390127 # find /chroot/usr/home /chroot/usr/home /chroot/usr/home/nezmer From owner-freebsd-fs@FreeBSD.ORG Mon Jun 28 11:45:47 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7C0BD106567A for ; Mon, 28 Jun 2010 11:45:47 +0000 (UTC) (envelope-from bsd@nezmer.info) Received: from mail.nezmer.info (nezmer.info [97.107.142.36]) by mx1.freebsd.org (Postfix) with ESMTP id 5C7788FC1A for ; Mon, 28 Jun 2010 11:45:47 +0000 (UTC) Date: Mon, 28 Jun 2010 14:45:42 +0300 From: Nezmer To: freebsd-fs@freebsd.org Message-ID: <20100628114542.GA11720@mail> References: <20100628113837.GA98334@mail> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100628113837.GA98334@mail> User-Agent: Mutt/1.5.20 (2009-06-14) Subject: Re: nullmounting zfs fs with children X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jun 2010 11:45:47 -0000 On Mon, Jun 28, 2010 at 02:38:37PM +0300, Nezmer wrote: > Hi, > > Is this normal behaviour? > > # zfs create -p -o mountpoint=/zfstest/subdir0/subdir1/subdir2 POOL/zfstest/subdir0/subdir1/subdir2 > # echo 2 > /zfstest/subdir0/subdir1/subdir2/file2 > # echo 1 > /zfstest/subdir0/subdir1/file1 > > # find /zfstest > /zfstest > /zfstest/subdir0 > /zfstest/subdir0/subdir1 > /zfstest/subdir0/subdir1/file1 > /zfstest/subdir0/subdir1/subdir2 > /zfstest/subdir0/subdir1/subdir2/file2 > > # mkdir /nulltest > # mount_nullfs /zfstest /nulltest > > # find /nulltest > /nulltest > /nulltest/subdir0 > /nulltest/subdir0/subdir1 > /nulltest/subdir0/subdir1/file1 > /nulltest/subdir0/subdir1/subdir2 > > # echo 0 > /zfstest/subdir0/file0 > > # find /zfstest > /zfstest > /zfstest/subdir0 > /zfstest/subdir0/subdir1 > /zfstest/subdir0/subdir1/file1 > /zfstest/subdir0/subdir1/subdir2 > /zfstest/subdir0/subdir1/subdir2/file2 > /zfstest/subdir0/file0 > > # find /nulltest > /nulltest > /nulltest/subdir0 > /nulltest/subdir0/subdir1 > /nulltest/subdir0/subdir1/file1 > /nulltest/subdir0/subdir1/subdir2 > /nulltest/subdir0/file0 > > # umount /nulltest > # zfs destroy -r POOL/zfstest > > # find /zfstest > /zfstest > /zfstest/subdir0 > /zfstest/subdir0/subdir1 > /zfstest/subdir0/subdir1/file1 > /zfstest/subdir0/subdir1/subdir2 > /zfstest/subdir0/file0 # mount_nullfs /zfstest /nulltest > > # find /nulltest > /nulltest > /nulltest/subdir0 > /nulltest/subdir0/subdir1 > /nulltest/subdir0/subdir1/file1 > /nulltest/subdir0/subdir1/subdir2 > /nulltest/subdir0/file0 > > I noticed this behaviour when I wanted to nullmount my "/usr/home" inside a chroot: > > # mount|grep /usr/home > POOL/usr/home on /usr/home (zfs, local, noatime) > POOL/usr/home/nezmer on /usr/home/nezmer (zfs, local, noatime) > POOL/usr/home/nezmer/Mail on /usr/home/nezmer/Mail (zfs, local, noatime) > POOL/usr/home/nezmer/pkgs on /usr/home/nezmer/pkgs (zfs, local, noatime) > POOL/usr/home/nezmer/srcs on /usr/home/nezmer/srcs (zfs, local, noatime) > /usr/home on /chroot/usr/home (nullfs, local) > > # find /usr/home -maxdepth 4|wc > 7267 7289 390127 > > # find /chroot/usr/home > /chroot/usr/home > /chroot/usr/home/nezmer > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Mon Jun 28 12:32:53 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BD9E1106566B for ; Mon, 28 Jun 2010 12:32:53 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 48C2F8FC15 for ; Mon, 28 Jun 2010 12:32:52 +0000 (UTC) Received: by bwz12 with SMTP id 12so231770bwz.13 for ; Mon, 28 Jun 2010 05:32:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:subject:references :x-comment-to:date:in-reply-to:message-id:user-agent:mime-version :content-type; bh=bDZFOMrbJhMLlCWwrqu9RtP9m/KSpaIXU/U96o4VsDM=; b=uhV7vHZ8ofxxT6ETxMVmHrTt2e2iFvIHx8U54L9pu6uKkA+ejD6frh91WicCch6UuH tcqbpAjnHRZti7gAGZ78vfp6J/Xu/RJfo5VQI6SqW5ubwe1OZFQ5lt5ZBRDEAOTVQLvG crcaQvl5ku1RjNPz2r3MTAS8HkFJCjM1yOp08= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:references:x-comment-to:date:in-reply-to :message-id:user-agent:mime-version:content-type; b=X9UPxwW1l24VJA+31ybmdoPFV3BUydHBtXTnhpUz15xHIw/oKU2WJL3nKSPXzOqrzL xp1zJB7b40eX2LBR5K72dSjTIHr4o3eijjcYUw2975cu4/EsN6wVq8yfPFFby29x/BcR EtgMV6t0t7iEDeZmyzdr07tEyLttMoHna8p9k= Received: by 10.204.162.138 with SMTP id v10mr3541360bkx.10.1277728361055; Mon, 28 Jun 2010 05:32:41 -0700 (PDT) Received: from localhost ([95.69.160.105]) by mx.google.com with ESMTPS id u9sm16760132bkf.5.2010.06.28.05.32.39 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 28 Jun 2010 05:32:40 -0700 (PDT) From: Mikolaj Golub To: Nezmer References: <20100628113837.GA98334@mail> <20100628114542.GA11720@mail> X-Comment-To: Nezmer Date: Mon, 28 Jun 2010 15:32:37 +0300 In-Reply-To: <20100628114542.GA11720@mail> (Nezmer's message of "Mon, 28 Jun 2010 14:45:42 +0300") Message-ID: <8663132v8a.fsf@kopusha.home.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-fs@freebsd.org Subject: Re: nullmounting zfs fs with children X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jun 2010 12:32:53 -0000 On Mon, 28 Jun 2010 14:45:42 +0300 Nezmer wrote: N> On Mon, Jun 28, 2010 at 02:38:37PM +0300, Nezmer wrote: >> Hi, >> >> Is this normal behaviour? >> >> # zfs create -p -o mountpoint=/zfstest/subdir0/subdir1/subdir2 POOL/zfstest/subdir0/subdir1/subdir2 >> # echo 2 > /zfstest/subdir0/subdir1/subdir2/file2 >> # echo 1 > /zfstest/subdir0/subdir1/file1 >> >> # find /zfstest >> /zfstest >> /zfstest/subdir0 >> /zfstest/subdir0/subdir1 >> /zfstest/subdir0/subdir1/file1 >> /zfstest/subdir0/subdir1/subdir2 >> /zfstest/subdir0/subdir1/subdir2/file2 >> >> # mkdir /nulltest >> # mount_nullfs /zfstest /nulltest >> >> # find /nulltest >> /nulltest >> /nulltest/subdir0 >> /nulltest/subdir0/subdir1 >> /nulltest/subdir0/subdir1/file1 >> /nulltest/subdir0/subdir1/subdir2 This is normal. POOL/zfstest and POOL/zfstest/subdir0/subdir1/subdir2 are two file systems. You have null mounted only the first one. You need to null mount subdir2 too: # mount_nullfs /zfstest/subdir0/subdir1/subdir2 /nulltest/subdir0/subdir1/subdir2 -- Mikolaj Golub From owner-freebsd-fs@FreeBSD.ORG Mon Jun 28 22:29:30 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 15F801065673 for ; Mon, 28 Jun 2010 22:29:30 +0000 (UTC) (envelope-from peterjeremy@acm.org) Received: from mail35.syd.optusnet.com.au (mail35.syd.optusnet.com.au [211.29.133.51]) by mx1.freebsd.org (Postfix) with ESMTP id 819F58FC13 for ; Mon, 28 Jun 2010 22:29:29 +0000 (UTC) Received: from server.vk2pj.dyndns.org (c211-30-160-13.belrs4.nsw.optusnet.com.au [211.30.160.13]) by mail35.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id o5SMTQ3N018182 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 29 Jun 2010 08:29:27 +1000 X-Bogosity: Ham, spamicity=0.000000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.4/8.14.4) with ESMTP id o5SMTPG9079895 for ; Tue, 29 Jun 2010 08:29:25 +1000 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.4/8.14.4/Submit) id o5SMTPX6079894 for freebsd-fs@freebsd.org; Tue, 29 Jun 2010 08:29:25 +1000 (EST) (envelope-from peter) Date: Tue, 29 Jun 2010 08:29:25 +1000 From: Peter Jeremy To: freebsd-fs@freebsd.org Message-ID: <20100628222924.GA79347@server.vk2pj.dyndns.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ew6BAiZeqk4r7MaW" Content-Disposition: inline X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.20 (2009-06-14) Subject: Possible ZFS data loss following unclean shutdown X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jun 2010 22:29:30 -0000 --ew6BAiZeqk4r7MaW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable http://blog.lastinfirstout.net/2010/06/sunoracle-finally-announces-zfs-data= =2Ehtml and its references talk about a ZFS bug where large synchronous writes (>32kB) not being properly committed before fsync() returns. I've looked through the SVN commit logs but don't see any of the associated bugids (2178259, 6440499, 6754011, 6791106, 6860045, 6860045, 6877373, 6880764, 6892298, 6900938) referenced. I've had a report of a problem that could potentially be caused by this but am still investigating it. Does anyone know if these bugs affect ZFS on FreeBSD? --=20 Peter Jeremy --ew6BAiZeqk4r7MaW Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkwpIkQACgkQ/opHv/APuIeuFACgmohkJtTtGX1FiYz8OIopdv/g F2gAn2ZH8ejQFo204vbSy3DRuJe4qRiu =iROR -----END PGP SIGNATURE----- --ew6BAiZeqk4r7MaW-- From owner-freebsd-fs@FreeBSD.ORG Wed Jun 30 04:52:16 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 90D401065672 for ; Wed, 30 Jun 2010 04:52:16 +0000 (UTC) (envelope-from spork@bway.net) Received: from xena.bway.net (xena.bway.net [216.220.96.26]) by mx1.freebsd.org (Postfix) with ESMTP id 30ED38FC19 for ; Wed, 30 Jun 2010 04:52:15 +0000 (UTC) Received: (qmail 13673 invoked by uid 0); 30 Jun 2010 04:52:15 -0000 Received: from unknown (HELO ?10.3.2.41?) (spork@96.57.144.66) by smtp.bway.net with (AES128-SHA encrypted) SMTP; 30 Jun 2010 04:52:15 -0000 Message-Id: From: Charles Sprickman To: Pawel Jakub Dawidek In-Reply-To: <20100616132504.GK1739@garage.freebsd.pl> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Wed, 30 Jun 2010 00:52:14 -0400 References: <20100616132504.GK1739@garage.freebsd.pl> X-Mailer: Apple Mail (2.936) Cc: freebsd-fs@freebsd.org Subject: Re: zfs panic "evicting znode" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jun 2010 04:52:16 -0000 On Jun 16, 2010, at 9:25 AM, Pawel Jakub Dawidek wrote: > On Tue, Jun 15, 2010 at 10:34:20PM -0400, Charles Sprickman wrote: >> Howdy, >> >> I have a box running 8.0-RELEASE that recently started panicing >> every few >> hours with the following message: >> >> panic: evicting znode 0xa1eafe80 > > Do you have at least backtrace from DDB? > > What kind of workload do you have? I was no able to find a way to > trigger znode_evict_error() call (this is what you're seeing). > You could try changing '#if 1' to '#if 0' in > zfs_znode.c:znode_evict_error(), but I don't think this code path was > ever tested, because as I said, I was unable to trigger it. > > -- > Pawel Jakub Dawidek http://www.wheelsystems.com > pjd@FreeBSD.org http://www.FreeBSD.org > FreeBSD committer Am I Evil? Yes, I Am! I'm not too clever with ddb, but here's what I found after the last panic: http://img191.imageshack.us/slideshow/webplayer.php?id=img0505g.jpg Serial port is dead on this box, so I had to snap pics. I did a "where" and then looked at each thread. I now have the box at home, so I might have another chance with it before we nuke it - what other info could you use? Thanks, Charles From owner-freebsd-fs@FreeBSD.ORG Thu Jul 1 07:29:27 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF7F7106564A; Thu, 1 Jul 2010 07:29:27 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id A824D8FC18; Thu, 1 Jul 2010 07:29:27 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id 36DA86E549; Thu, 1 Jul 2010 07:21:19 +0000 (UTC) Message-ID: <4C2C43D5.1080907@soupacific.com> Date: Thu, 01 Jul 2010 16:29:25 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1 MIME-Version: 1.0 To: Mikolaj Golub References: <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> <868w6hwt2w.fsf@kopusha.home.net> <20100614153746.GN1721@garage.freebsd.pl> <86zkyxvc4v.fsf@kopusha.home.net> In-Reply-To: <86zkyxvc4v.fsf@kopusha.home.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: HAST and CARP X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Jul 2010 07:29:28 -0000 HI folks ! I finished to run HAST with CARP and ifstated. Question I haove now. SERVERA is master for CARP and HAST then boot SERVERB ServerB is using CARP and ifstated. First CARP state is INIT then BACKUP Similar to ucarp's method, vip-down.sh is called. vip-down.sh calls carp_down.sh Inside is almost same to ucarp_down.sh except delete ucarp staff. if I do not put hastctl create xxxx, then hastd refuse connection and message says Split-brain detected secondary ! I put debug call but I can not figure out what kind value is correct. } else if (res->hr_secondary_localcnt > res->hr_primary_remotecnt && res->hr_primary_localcnt > res->hr_secondary_remotecnt) { /* * Not good, we have split-brain condition. */ //hiroshi debug pjdlog_error("secondary_remotecnt %lu, primary_remotecnt %lu", res->hr_secondary_remotecnt, res->hr_primary_remotecnt); pjdlog_error("secondary_localcnt %lu, primary_localcnt %lu", res->hr_secondary_localcnt, res->hr_primary_localcnt); pjdlog_error("Split-brain detected, exiting."); nv_add_string(nvout, "Split-brain condition!", "errmsg"); free(map); map = NULL; mapsize = 0; } else /* if (res->hr_secondary_localcnt < res->hr_primary_remotecnt || hastctl status return perfect as secondary. Only looks like connection error. When using ucarp with debug Booting as secondary vip-up.sh then vip-down.sh But CARP, only calls vip-dwon.sh directly. Once I put hastctl create xxx before hastctl role secondary. Things works fine. Do I need hastctl create xxx or split-brain secondary is wrong? My complete files are here. http://www.soupacific.com/VivaFreeBSD_HAST/hastquick.html Anybody to try HAST with CARP, please try! but any mistake I made, All risk is your own please. My HAST, CARP and ifstated works fine! Thanks Hiroshi Katayama Hiorhis From owner-freebsd-fs@FreeBSD.ORG Thu Jul 1 07:59:45 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 92F47106566B for ; Thu, 1 Jul 2010 07:59:45 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (core.vx.sk [188.40.32.143]) by mx1.freebsd.org (Postfix) with ESMTP id 1DF178FC0A for ; Thu, 1 Jul 2010 07:59:44 +0000 (UTC) Received: from core.vx.sk (localhost [127.0.0.1]) by mail.vx.sk (Postfix) with ESMTP id 14D01DA7F4; Thu, 1 Jul 2010 09:59:44 +0200 (CEST) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk ([127.0.0.1]) by core.vx.sk (mail.vx.sk [127.0.0.1]) (amavisd-new, port 10024) with LMTP id If2BrcZjWf4L; Thu, 1 Jul 2010 09:59:41 +0200 (CEST) Received: from [10.9.8.1] (188-167-78-139.dynamic.chello.sk [188.167.78.139]) by mail.vx.sk (Postfix) with ESMTPSA id 94F3BDA7EC; Thu, 1 Jul 2010 09:59:41 +0200 (CEST) Message-ID: <4C2C4AEC.1060704@FreeBSD.org> Date: Thu, 01 Jul 2010 09:59:40 +0200 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; sk; rv:1.8.1.23) Gecko/20090812 Lightning/0.9 Thunderbird/2.0.0.23 Mnenhy/0.7.5.0 MIME-Version: 1.0 To: Peter Jeremy References: <20100628222924.GA79347@server.vk2pj.dyndns.org> In-Reply-To: <20100628222924.GA79347@server.vk2pj.dyndns.org> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=windows-1250 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org Subject: Re: Possible ZFS data loss following unclean shutdown X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Jul 2010 07:59:45 -0000 This issue was fixed in OpenSolaris revision 10800:469478b180d9 Bug information: http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6880764 This bugfix has not been imported yet, as it requires imported revision 8227:f7d7be9b1f56 to apply cleanly as a whole. My latest zfs-v16 patches (v2,v3) contain this and other bugfixes from OpenSolaris. http://people.freebsd.org/~mm/patches/zfs/v16/ FYI: these patches apply against 8-stable as well Dňa 29. 6. 2010 0:29, Peter Jeremy wrote / napísal(a): > http://blog.lastinfirstout.net/2010/06/sunoracle-finally-announces-zfs-data.html > and its references talk about a ZFS bug where large synchronous writes > (>32kB) not being properly committed before fsync() returns. I've > looked through the SVN commit logs but don't see any of the associated > bugids (2178259, 6440499, 6754011, 6791106, 6860045, 6860045, 6877373, > 6880764, 6892298, 6900938) referenced. I've had a report of a problem > that could potentially be caused by this but am still investigating it. > > Does anyone know if these bugs affect ZFS on FreeBSD? > > From owner-freebsd-fs@FreeBSD.ORG Thu Jul 1 20:33:30 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0FA09106564A; Thu, 1 Jul 2010 20:33:30 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id CE8988FC17; Thu, 1 Jul 2010 20:33:28 +0000 (UTC) Received: by bwz12 with SMTP id 12so1411927bwz.13 for ; Thu, 01 Jul 2010 13:33:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:subject:references :x-comment-to:date:in-reply-to:message-id:user-agent:mime-version :content-type; bh=YPCOPZLL3duf53FRgSAMyl3qQLQydgfjTaYFbSp84KQ=; b=Q/cHdeO7lswtsF6IS8ie2a2bmM/ZMDVuRlToVuU9Ohr0A8l8t3ZqiGeRGWyYfSkXpP 2YcwndW8ImqpOpirMRL+tKkPjg3F1OQz07Xq8z6ygdaX0upsXuZtPJv/dN9l1d+i9qN8 z6fSJsA6K1PCuYdqDgz9ltoBCTxGOpXJyhkI8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:references:x-comment-to:date:in-reply-to :message-id:user-agent:mime-version:content-type; b=sK09Ux7uP+5sokie9t6qdpSMgARDEw7J8NTWWKjX9I2ni+zf0i5Tsd1TlTvQPWxPgR mOd8apgf25rONijf7mVxy9akocqps5upt+Bz96yaAPCMoaUMiHGi5nB+5uvU8eAqEde1 w970CuwqFICThvFL9indtAX83H/yUuy+w/3Hs= Received: by 10.204.82.130 with SMTP id b2mr47220bkl.12.1278016401315; Thu, 01 Jul 2010 13:33:21 -0700 (PDT) Received: from localhost ([95.69.160.105]) by mx.google.com with ESMTPS id bq20sm13969754bkb.3.2010.07.01.13.33.17 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 01 Jul 2010 13:33:18 -0700 (PDT) From: Mikolaj Golub To: "hiroshi\@soupacific.com" References: <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> <868w6hwt2w.fsf@kopusha.home.net> <20100614153746.GN1721@garage.freebsd.pl> <86zkyxvc4v.fsf@kopusha.home.net> <4C2C43D5.1080907@soupacific.com> X-Comment-To: hiroshi@soupacific.com Date: Thu, 01 Jul 2010 23:33:14 +0300 In-Reply-To: <4C2C43D5.1080907@soupacific.com> (hiroshi@soupacific.com's message of "Thu, 01 Jul 2010 16:29:25 +0900") Message-ID: <86mxubndrp.fsf@kopusha.home.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: HAST and CARP X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Jul 2010 20:33:30 -0000 On Thu, 01 Jul 2010 16:29:25 +0900 hiroshi@soupacific.com wrote: h> SERVERA is master for CARP and HAST h> then boot SERVERB h> ServerB is using CARP and ifstated. h> First CARP state is INIT then BACKUP h> Similar to ucarp's method, vip-down.sh is called. h> vip-down.sh calls carp_down.sh h> Inside is almost same to ucarp_down.sh except delete ucarp staff. h> if I do not put hastctl create xxxx, then hastd refuse connection and h> message says h> Split-brain detected secondary ! Split-brain means that you or your scripts did something wrong: both nodes acted as primary (either simultaneously or one then another but there was no communication between them so both made changes to the data that was not synced to another node). The easy way to get split-brain is to change the role on secondary to primary without changing the role on the primary host, make some changes on the secondary (acting as a primary) and change back its role to secondary. You should check your setup if something like this may happen. Doing 'hastctl create' on every switching is wrong. Note, after 'hastctl create' hast metadata on the disk are lost and synchronization of all blocks from the primary is initiated, which is rather long. You can observe the status of this synchronization looking at "dirty" field of hastctl status output on the primary. You can't switch to failover until synchronization is complete. So you should avoid 'hastctl create' -- this is only for initial setup or repairing your hast. -- Mikolaj Golub From owner-freebsd-fs@FreeBSD.ORG Fri Jul 2 05:16:11 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 85AEC1065670; Fri, 2 Jul 2010 05:16:11 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id 2C4748FC13; Fri, 2 Jul 2010 05:16:10 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id 8DD1B6E9B1; Fri, 2 Jul 2010 05:08:03 +0000 (UTC) Message-ID: <4C2D7615.5070606@soupacific.com> Date: Fri, 02 Jul 2010 14:16:05 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1 MIME-Version: 1.0 To: Mikolaj Golub References: <4C139F9C.2090305@soupacific.com><86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com><20100613003635.GA60012@icarus.home.lan><20100613074921.GB1320@garage.freebsd.pl><4C149A5C.3070401@soupacific.com><20100613102401.GE1320@garage.freebsd.pl><86eigavzsg.fsf@kopusha.home.net><20100614095044.GH1721@garage.freebsd.pl><868w6hwt2w.fsf@kopusha.home.net><20100614153746.GN1721@garage.freebsd.pl><86zkyxvc4v.fsf@kopusha.home.net> <4C2C43D5.1080907@soupacific.com> <86mxubndrp.fsf@kopusha.home.net> In-Reply-To: <86mxubndrp.fsf@kopusha.home.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: HAST and CARP X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jul 2010 05:16:11 -0000 Thanks for your info. > Doing 'hastctl create' on every switching is wrong. Note, after 'hastctl > create' hast metadata on the disk are lost and synchronization of all blocks That's I was afraid of it ! > Split-brain means that you or your scripts did something wrong: both nodes > acted as primary (either simultaneously or one then another but there was no > communication between them so both made changes to the data that was not > synced to another node). > > The easy way to get split-brain is to change the role on secondary to primary > without changing the role on the primary host, make some changes on the > secondary (acting as a primary) and change back its role to secondary. I checked that both node communication is established. And here is log without hastctl create. Seems ServerB once became MASTER, then back to BACKUP. This situation cause unhappy split-brain happened. hastctl dump shows prevrole: primary error debug los is this Jul 2 12:31:37 fw01B kernel: Clearing /tmp (X related). Jul 2 12:31:37 fw01B kernel: Updating motd: Jul 2 12:31:37 fw01B kernel: . Jul 2 12:31:37 fw01B kernel: Configuring syscons: Jul 2 12:31:37 fw01B kernel: blanktime Jul 2 12:31:37 fw01B kernel: . Jul 2 12:31:38 fw01B sm-mta[879]: gethostbyaddr(211.19.53.206) failed: 2 Jul 2 12:31:38 fw01B sm-mta[879]: gethostbyaddr(211.19.53.202) failed: 2 Jul 2 12:31:38 fw01B sm-mta[880]: starting daemon (8.14.4): SMTP+queueing@00:30:00 Jul 2 12:31:38 fw01B sm-msp-queue[884]: starting daemon (8.14.4): queueing@00:30:00 Jul 2 12:31:38 fw01B kernel: Starting cron. Jul 2 12:31:38 fw01B kernel: Starting background file system checks in 60 seconds. Jul 2 12:31:38 fw01B kernel: Jul 2 12:31:38 fw01B kernel: Fri Jul 2 12:31:38 UTC 2010 Jul 2 12:31:40 fw01B kernel: carp0: INIT -> BACKUP Jul 2 12:31:40 fw01B kernel: alc0: link state changed to UP Jul 2 12:31:40 fw01B kernel: carp0: 2 link states coalesced Jul 2 12:31:40 fw01B kernel: carp0: link state changed to DOWN Jul 2 12:31:43 fw01B login: login on ttyv0 as root Jul 2 12:31:43 fw01B login: ROOT LOGIN (root) ON ttyv0 Jul 2 12:31:43 fw01B kernel: Jul 2 12:31:43 fw01B login: ROOT LOGIN (root) ON ttyv0 Jul 2 12:31:48 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jul 2 12:31:48 fw01B hastd: Connection from tcp4://211.19.53.201:20070 to tcp4://211.19.53.206:8457. Jul 2 12:31:48 fw01B hastd: tcp4://211.19.53.201:20070: resource=zfshast Jul 2 12:31:48 fw01B hastd: [zfshast] (init) We act as init for the resource and not as secondary as requested by tcp4://211.19.53.201:20070. Jul 2 12:31:48 fw01B kernel: Jul 2 12:31:48 fw01B hastd: [zfshast] (init) We act as init for the resource and not as secondary as requested by tcp4://211.19.53.201:20070. Jul 2 12:31:53 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jul 2 12:31:53 fw01B hastd: Connection from tcp4://211.19.53.201:11542 to tcp4://211.19.53.206:8457. Jul 2 12:31:53 fw01B hastd: tcp4://211.19.53.201:11542: resource=zfshast Jul 2 12:31:53 fw01B hastd: [zfshast] (init) We act as init for the resource and not as secondary as requested by tcp4://211.19.53.201:11542. Jul 2 12:31:53 fw01B kernel: Jul 2 12:31:53 fw01B hastd: [zfshast] (init) We act as init for the resource and not as secondary as requested by tcp4://211.19.53.201:11542. Jul 2 12:31:58 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jul 2 12:31:58 fw01B hastd: Connection from tcp4://211.19.53.201:49777 to tcp4://211.19.53.206:8457. Jul 2 12:31:58 fw01B hastd: tcp4://211.19.53.201:49777: resource=zfshast Jul 2 12:31:58 fw01B hastd: [zfshast] (init) We act as init for the resource and not as secondary as requested by tcp4://211.19.53.201:49777. Jul 2 12:31:58 fw01B kernel: Jul 2 12:31:58 fw01B hastd: [zfshast] (init) We act as init for the resource and not as secondary as requested by tcp4://211.19.53.201:49777. Jul 2 12:31:58 fw01B hastd: [zfshast] (init) Role changed to secondary. Jul 2 12:32:03 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jul 2 12:32:03 fw01B hastd: Connection from tcp4://211.19.53.201:17014 to tcp4://211.19.53.206:8457. Jul 2 12:32:03 fw01B hastd: tcp4://211.19.53.201:17014: resource=zfshast Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) Initial connection from tcp4://211.19.53.201:17014. Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) Incoming connection from tcp4://211.19.53.201:17014 configured. Jul 2 12:32:03 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jul 2 12:32:03 fw01B hastd: Connection from tcp4://211.19.53.201:42420 to tcp4://211.19.53.206:8457. Jul 2 12:32:03 fw01B hastd: tcp4://211.19.53.201:42420: resource=zfshast Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) Outgoing connection to tcp4://211.19.53.201:42420 configured. Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) hastd_secondary Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) calling init_local() Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) init_local Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) Obtained info about /dev/ad4p4. Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) Locked /dev/ad4p4. Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) inside metadata.c res->hr_role !=HAST_ROLE_PRIMAR Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) inside mettadata secondary_localcnt: 1 secondary_remotecnt: 0 Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) calling init_remote() Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) init_remote() Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) humhum secondary local 1: secondary remote 0 Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) init hr_secondary_remotecnt: 0 hr_primary_remotecnt: 0 Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) secondary_remotecnt 0, primary_remotecnt 0 Jul 2 12:32:03 fw01B kernel: Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) secondary_remotecnt 0, primary_remotecnt 0 Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) secondary_localcnt 1, primary_localcnt 1 Jul 2 12:32:03 fw01B kernel: Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) secondary_localcnt 1, primary_localcnt 1 Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) Split-brain detected, exiting. Jul 2 12:32:03 fw01B kernel: Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) Split-brain detected, exiting. Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) Worker process exited ungracefully (pid=979, exitcode=78). Jul 2 12:32:03 fw01B kernel: Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) Worker process exited ungracefully (pid=979, exitcode=78). Jul 2 12:32:08 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jul 2 12:32:08 fw01B hastd: Connection from tcp4://211.19.53.201:53033 to tcp4://211.19.53.206:8457. Jul 2 12:32:08 fw01B hastd: tcp4://211.19.53.201:53033: resource=zfshast Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) Initial connection from tcp4://211.19.53.201:53033. Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) Incoming connection from tcp4://211.19.53.201:53033 configured. Jul 2 12:32:08 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jul 2 12:32:08 fw01B hastd: Connection from tcp4://211.19.53.201:50656 to tcp4://211.19.53.206:8457. Jul 2 12:32:08 fw01B hastd: tcp4://211.19.53.201:50656: resource=zfshast Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) Outgoing connection to tcp4://211.19.53.201:50656 configured. Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) hastd_secondary Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) calling init_local() Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) init_local Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) Obtained info about /dev/ad4p4. Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) Locked /dev/ad4p4. Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) inside metadata.c res->hr_role !=HAST_ROLE_PRIMAR Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) inside mettadata secondary_localcnt: 1 secondary_remotecnt: 0 Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) calling init_remote() Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) init_remote() Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) humhum secondary local 1: secondary remote 0 Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) init hr_secondary_remotecnt: 0 hr_primary_remotecnt: 0 Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) secondary_remotecnt 0, primary_remotecnt 0 Jul 2 12:32:08 fw01B kernel: Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) secondary_remotecnt 0, primary_remotecnt 0 Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) secondary_localcnt 1, primary_localcnt 1 Jul 2 12:32:08 fw01B kernel: Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) secondary_localcnt 1, primary_localcnt 1 Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) Split-brain detected, exiting. Jul 2 12:32:08 fw01B kernel: Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) Split-brain detected, exiting. Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) Worker process exited ungracefully (pid=980, exitcode=78). Jul 2 12:32:08 fw01B kernel: Jul 2 12:32:08 fw01B hastd: [zfshast] (secondary) Worker process exited ungracefully (pid=980, exitcode=78). Jul 2 12:32:13 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. When debuglog working properly. Jul 2 10:24:10 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jul 2 10:24:10 fw01B hastd: tcp4://211.19.53.201:26965: resource=zfshast Jul 2 10:24:15 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jul 2 10:24:15 fw01B hastd: tcp4://211.19.53.201:50280: resource=zfshast Jul 2 10:24:20 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jul 2 10:24:20 fw01B hastd: tcp4://211.19.53.201:27929: resource=zfshast Jul 2 10:24:25 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jul 2 10:24:25 fw01B hastd: tcp4://211.19.53.201:24357: resource=zfshast Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) Initial connection from tcp4://211.19.53.201:24357. Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) Incoming connection from tcp4://211.19.53.201:24357 configured. Jul 2 10:24:25 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. Jul 2 10:24:25 fw01B hastd: tcp4://211.19.53.201:18217: resource=zfshast Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) Outgoing connection to tcp4://211.19.53.201:18217 configured. Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) hastd_secondary Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) calling init_local() Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) init_local Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) Obtained info about /dev/ad4p4. Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) Locked /dev/ad4p4. Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) inside metadata.c res->hr_role !=HAST_ROLE_PRIMAR Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) inside mettadata secondary_localcnt: 0 secondary_remotecnt: 0 Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) calling init_remote() Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) init_remote() Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) humhum secondary local 0: secondary remote 0 Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) init hr_secondary_remotecnt: 0 hr_primary_remotecnt: 0 Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) recv: Taking free request. Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) disk: Taking request. Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) disk: No requests, waiting. Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) recv: (0x8013ea2e0) Got request. Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) send: Taking request. Jul 2 10:24:25 fw01B hastd: [zfshast] (secondary) send: No requests, waiting. Jul 2 10:24:26 fw01B hastd: [zfshast] (secondary) disk: (0x8013ea2e0) Moving request to the send queue. Jul 2 10:24:26 fw01B hastd: [zfshast] (secondary) disk: Taking request. Jul 2 10:24:26 fw01B hastd: [zfshast] (secondary) disk: No requests, waiting. hastctl role seconary xxx should reset some value of master to backup? Hope this logs can help you ! If you need to make me debug bit more, give me some idea to check! Thanks Hiroshi Katayama From owner-freebsd-fs@FreeBSD.ORG Fri Jul 2 07:11:30 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0051A1065672; Fri, 2 Jul 2010 07:11:29 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-ww0-f42.google.com (mail-ww0-f42.google.com [74.125.82.42]) by mx1.freebsd.org (Postfix) with ESMTP id 2203E8FC15; Fri, 2 Jul 2010 07:11:28 +0000 (UTC) Received: by wwd20 with SMTP id 20so453001wwd.1 for ; Fri, 02 Jul 2010 00:11:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:subject :organization:references:date:in-reply-to:message-id:user-agent :mime-version:content-type; bh=tO3PXRbtLGKfFhTmayeMDwn8xyFswFan22vcV/HsdFY=; b=jk075YSde8NFcbHkENKcGP+XTApS0RZKKp7u11HDEdF27sFhuHLIviiOFhYhM31KRt Jp75ktAFb2W4b9LJfP1bCC5GUkyRRpNzdiy0DlB/ABvMQBv3B2X7Zc4ZyUvm+I82ZK7+ VOLNl5V44JmRdGF/jG77CcY9E8eIfyWkA76x0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:organization:references:date:in-reply-to :message-id:user-agent:mime-version:content-type; b=o/emMoxf3ifiXnmnvufQd6HQlfABdI52dRidu29wV2SyVu4R1sevlbE8WVp2lcZtj7 TvIv43F9s2RfUPr7gKmdJRG5vE+tktxb3VY1RmuzspLhpiyW2yqD0xQKK2MGRmXv7qAa xa3Rp9Ah4qjTjhuWwVAKZnJvpIzsURUu3qs5M= Received: by 10.227.129.85 with SMTP id n21mr169785wbs.81.1278054677820; Fri, 02 Jul 2010 00:11:17 -0700 (PDT) Received: from localhost (ua1.etadirect.net [91.198.140.16]) by mx.google.com with ESMTPS id a27sm2390525wbe.18.2010.07.02.00.11.15 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 02 Jul 2010 00:11:16 -0700 (PDT) From: Mikolaj Golub To: "hiroshi\@soupacific.com" Organization: TOA Ukraine References: <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> <868w6hwt2w.fsf@kopusha.home.net> <20100614153746.GN1721@garage.freebsd.pl> <86zkyxvc4v.fsf@kopusha.home.net> <4C2C43D5.1080907@soupacific.com> <86mxubndrp.fsf@kopusha.home.net> <4C2D7615.5070606@soupacific.com> Date: Fri, 02 Jul 2010 10:11:12 +0300 In-Reply-To: <4C2D7615.5070606@soupacific.com> (hiroshi@soupacific.com's message of "Fri, 02 Jul 2010 14:16:05 +0900") Message-ID: <861vbm1hpr.fsf@zhuzha.ua1> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: HAST and CARP X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jul 2010 07:11:30 -0000 On Fri, 02 Jul 2010 14:16:05 +0900 hiroshi@soupacific.com wrote: h> I checked that both node communication is established. And here is log h> without hastctl create. h> Seems ServerB once became MASTER, then back to BACKUP. h> This situation cause unhappy split-brain happened. h> hastctl dump shows prevrole: primary h> error debug los is this h> Jul 2 12:31:37 fw01B kernel: Clearing /tmp (X related). h> Jul 2 12:31:37 fw01B kernel: Updating motd: h> Jul 2 12:31:37 fw01B kernel: . h> Jul 2 12:31:37 fw01B kernel: Configuring syscons: h> Jul 2 12:31:37 fw01B kernel: blanktime h> Jul 2 12:31:37 fw01B kernel: . h> Jul 2 12:31:38 fw01B sm-mta[879]: gethostbyaddr(211.19.53.206) failed: 2 h> Jul 2 12:31:38 fw01B sm-mta[879]: gethostbyaddr(211.19.53.202) failed: 2 h> Jul 2 12:31:38 fw01B sm-mta[880]: starting daemon (8.14.4): h> SMTP+queueing@00:30:00 h> Jul 2 12:31:38 fw01B sm-msp-queue[884]: starting daemon (8.14.4): h> queueing@00:30:00 h> Jul 2 12:31:38 fw01B kernel: Starting cron. h> Jul 2 12:31:38 fw01B kernel: Starting background file system checks h> in 60 seconds. h> Jul 2 12:31:38 fw01B kernel: h> Jul 2 12:31:38 fw01B kernel: Fri Jul 2 12:31:38 UTC 2010 h> Jul 2 12:31:40 fw01B kernel: carp0: INIT -> BACKUP h> Jul 2 12:31:40 fw01B kernel: alc0: link state changed to UP h> Jul 2 12:31:40 fw01B kernel: carp0: 2 link states coalesced h> Jul 2 12:31:40 fw01B kernel: carp0: link state changed to DOWN h> Jul 2 12:31:43 fw01B login: login on ttyv0 as root h> Jul 2 12:31:43 fw01B login: ROOT LOGIN (root) ON ttyv0 h> Jul 2 12:31:43 fw01B kernel: Jul 2 12:31:43 fw01B login: ROOT LOGIN h> (root) ON ttyv0 h> Jul 2 12:31:48 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. h> Jul 2 12:31:48 fw01B hastd: Connection from h> tcp4://211.19.53.201:20070 to tcp4://211.19.53.206:8457. h> Jul 2 12:31:48 fw01B hastd: tcp4://211.19.53.201:20070: resource=zfshast h> Jul 2 12:31:48 fw01B hastd: [zfshast] (init) We act as init for the h> resource and not as secondary as requested by h> tcp4://211.19.53.201:20070. h> Jul 2 12:31:48 fw01B kernel: Jul 2 12:31:48 fw01B hastd: [zfshast] h> (init) We act as init for the resource and not as secondary as h> requested by tcp4://211.19.53.201:20070. h> Jul 2 12:31:53 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. h> Jul 2 12:31:53 fw01B hastd: Connection from h> tcp4://211.19.53.201:11542 to tcp4://211.19.53.206:8457. h> Jul 2 12:31:53 fw01B hastd: tcp4://211.19.53.201:11542: resource=zfshast h> Jul 2 12:31:53 fw01B hastd: [zfshast] (init) We act as init for the h> resource and not as secondary as requested by h> tcp4://211.19.53.201:11542. h> Jul 2 12:31:53 fw01B kernel: Jul 2 12:31:53 fw01B hastd: [zfshast] h> (init) We act as init for the resource and not as secondary as h> requested by tcp4://211.19.53.201:11542. h> Jul 2 12:31:58 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. h> Jul 2 12:31:58 fw01B hastd: Connection from h> tcp4://211.19.53.201:49777 to tcp4://211.19.53.206:8457. h> Jul 2 12:31:58 fw01B hastd: tcp4://211.19.53.201:49777: resource=zfshast h> Jul 2 12:31:58 fw01B hastd: [zfshast] (init) We act as init for the h> resource and not as secondary as requested by h> tcp4://211.19.53.201:49777. h> Jul 2 12:31:58 fw01B kernel: Jul 2 12:31:58 fw01B hastd: [zfshast] h> (init) We act as init for the resource and not as secondary as h> requested by tcp4://211.19.53.201:49777. h> Jul 2 12:31:58 fw01B hastd: [zfshast] (init) Role changed to secondary. h> Jul 2 12:32:03 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. h> Jul 2 12:32:03 fw01B hastd: Connection from h> tcp4://211.19.53.201:17014 to tcp4://211.19.53.206:8457. h> Jul 2 12:32:03 fw01B hastd: tcp4://211.19.53.201:17014: resource=zfshast h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) Initial connection h> from tcp4://211.19.53.201:17014. h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) Incoming connection h> from tcp4://211.19.53.201:17014 configured. h> Jul 2 12:32:03 fw01B hastd: Accepting connection to tcp4://0.0.0.0:8457. h> Jul 2 12:32:03 fw01B hastd: Connection from h> tcp4://211.19.53.201:42420 to tcp4://211.19.53.206:8457. h> Jul 2 12:32:03 fw01B hastd: tcp4://211.19.53.201:42420: resource=zfshast h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) Outgoing connection h> to tcp4://211.19.53.201:42420 configured. h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) hastd_secondary h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) calling init_local() h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) init_local h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) Obtained info about h> /dev/ad4p4. h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) Locked /dev/ad4p4. h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) inside metadata.c h> res->hr_role !=HAST_ROLE_PRIMAR h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) inside mettadata h> secondary_localcnt: 1 secondary_remotecnt: 0 h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) calling init_remote() h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) init_remote() h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) humhum secondary h> local 1: secondary remote 0 h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) init h> hr_secondary_remotecnt: 0 hr_primary_remotecnt: 0 h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) secondary_remotecnt h> 0, primary_remotecnt 0 h> Jul 2 12:32:03 fw01B kernel: Jul 2 12:32:03 fw01B hastd: [zfshast] h> (secondary) secondary_remotecnt 0, primary_remotecnt 0 h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) secondary_localcnt h> 1, primary_localcnt 1 h> Jul 2 12:32:03 fw01B kernel: Jul 2 12:32:03 fw01B hastd: [zfshast] h> (secondary) secondary_localcnt 1, primary_localcnt 1 h> Jul 2 12:32:03 fw01B hastd: [zfshast] (secondary) Split-brain h> detected, exiting. h> Jul 2 12:32:03 fw01B kernel: Jul 2 12:32:03 fw01B hastd: [zfshast] h> (secondary) Split-brain detected, exiting. So you have: secondary localcnt: 1 secondary remotecnt: 0 primary localcnt: 1 primary remotecnt: 0 This is a split-brain condition as described on wiki: primary's localcnt is greater than secondary's remotecnt (primary [fw01A] was modified while fw01B wasn't watching) and secondary's localcnt is greater than primary's remotecnt (fw01B was modified while fw01A wasn't watching). h> Hope this logs can help you ! If you need to make me debug bit more, h> give me some idea to check! Actually the logs you have provided are not very interesting as they shows the state after bad things happened. It is more interesting to look at the logs (both hosts) before split brain. I would recommend: 1) Configure hast manually and ensure that both primary and secondary function properly and data are synchronized between the nodes. Also make sure the clock on both hosts is in sync (needed when comparing logs). 2) Reboot both servers so your carp/hast setup auto starts and see what happens. 3) If it sets primary and secondary automatically and status is ok on both nodes initiate switching to failover. 4) If after switching (or earlier) split brain is detected, provide logs from both nodes since hosts reboot. -- Mikolaj Golub From owner-freebsd-fs@FreeBSD.ORG Fri Jul 2 07:50:03 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E2F11065670 for ; Fri, 2 Jul 2010 07:50:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id E806D8FC0A for ; Fri, 2 Jul 2010 07:50:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o627o2Qt034625 for ; Fri, 2 Jul 2010 07:50:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o627o2Tc034624; Fri, 2 Jul 2010 07:50:02 GMT (envelope-from gnats) Date: Fri, 2 Jul 2010 07:50:02 GMT Message-Id: <201007020750.o627o2Tc034624@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Antonios Anastasiadis Cc: Subject: Re: kern/145272: [zfs] [panic] Panic during boot when accessing zfs on a gmirror X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Antonios Anastasiadis List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jul 2010 07:50:03 -0000 The following reply was made to PR kern/145272; it has been noted by GNATS. From: Antonios Anastasiadis To: bug-followup@freebsd.org, chris@chacal.cjkey.org.uk Cc: Subject: Re: kern/145272: [zfs] [panic] Panic during boot when accessing zfs on a gmirror Date: Fri, 2 Jul 2010 10:15:29 +0300 --001485f7784041a2bf048a625bf8 Content-Type: text/plain; charset=UTF-8 This crash also occurs in 8.1-RC2 to me. --001485f7784041a2bf048a625bf8 Content-Type: text/html; charset=UTF-8 This crash also occurs in 8.1-RC2 to me.
--001485f7784041a2bf048a625bf8-- From owner-freebsd-fs@FreeBSD.ORG Fri Jul 2 07:59:35 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C65C106566B; Fri, 2 Jul 2010 07:59:35 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id 0842C8FC1A; Fri, 2 Jul 2010 07:59:34 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id CA6336EA78; Fri, 2 Jul 2010 07:51:28 +0000 (UTC) Message-ID: <4C2D9C62.4050105@soupacific.com> Date: Fri, 02 Jul 2010 16:59:30 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1 MIME-Version: 1.0 To: Mikolaj Golub References: <4C139F9C.2090305@soupacific.com><86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com><20100613003635.GA60012@icarus.home.lan><20100613074921.GB1320@garage.freebsd.pl><4C149A5C.3070401@soupacific.com><20100613102401.GE1320@garage.freebsd.pl><86eigavzsg.fsf@kopusha.home.net><20100614095044.GH1721@garage.freebsd.pl><868w6hwt2w.fsf@kopusha.home.net><20100614153746.GN1721@garage.freebsd.pl><86zkyxvc4v.fsf@kopusha.home.net> <4C2C43D5.1080907@soupacific.com><86mxubndrp.fsf@kopusha.home.net> <4C2D7615.5070606@soupacific.com> <861vbm1hpr.fsf@zhuzha.ua1> In-Reply-To: <861vbm1hpr.fsf@zhuzha.ua1> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: HAST and CARP X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jul 2010 07:59:35 -0000 On 7/2/2010 4:11 PM, Mikolaj Golub wrote: > > So you have: > > secondary localcnt: 1 > secondary remotecnt: 0 > primary localcnt: 1 > primary remotecnt: 0 > > This is a split-brain condition as described on wiki: primary's localcnt is > greater than secondary's remotecnt (primary [fw01A] was modified while fw01B > wasn't watching) and secondary's localcnt is greater than primary's remotecnt > (fw01B was modified while fw01A wasn't watching). So hasctl role secondary xxx does not change cnt values ? Scenario is this ServerA failed, then ServerB became MASTER. Only ServerA is started(say after fixed something) , both servers are connected,then ServerB starts, BUT during failure of ServerA, ServerB was MASTER. ServerA was started before ServerB is started, thus ServerA should be MASTER! On this situation, CARP will set ServerA is MASTER and late comer ServerB is set as BACKUP by CARP. hastctl role secondary xxx set > secondary localcnt: 1 > secondary remotecnt: 0 > primary localcnt: 1 > primary remotecnt: 0 above values to NOT split-brain. It sounds more favoritabel way ???? hastctl role is managed by ifstated watching CARP status. Is this strange idea ? Thanks Hiroshi > > h> Hope this logs can help you ! If you need to make me debug bit more, > h> give me some idea to check! > > Actually the logs you have provided are not very interesting as they shows the > state after bad things happened. It is more interesting to look at the logs > (both hosts) before split brain. > > I would recommend: > > 1) Configure hast manually and ensure that both primary and secondary function > properly and data are synchronized between the nodes. Also make sure the clock > on both hosts is in sync (needed when comparing logs). > > 2) Reboot both servers so your carp/hast setup auto starts and see what > happens. > > 3) If it sets primary and secondary automatically and status is ok on both > nodes initiate switching to failover. > > 4) If after switching (or earlier) split brain is detected, provide logs from > both nodes since hosts reboot. > From owner-freebsd-fs@FreeBSD.ORG Fri Jul 2 09:25:25 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ACB77106564A; Fri, 2 Jul 2010 09:25:25 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id E6F598FC14; Fri, 2 Jul 2010 09:25:24 +0000 (UTC) Received: by wyb34 with SMTP id 34so2054764wyb.13 for ; Fri, 02 Jul 2010 02:25:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:subject :organization:references:date:in-reply-to:message-id:user-agent :mime-version:content-type; bh=gHAWtdlf4lv+jSmTEIDT4ON5ci/OTzKPqth0e7/EAPo=; b=UUZbvEQn/w/2zJbV45mQpRfV+E+Ox7qLeBKHETD2nL4bgoaGkWiqdvTy1tXUIF4voH GGnEBDfQgE1XzRpfdiobXV8IGl319Z7d9u2r3+yNWa19AhUgRqG/hndpDmni9dbLzOMC J0zeKDP2zo+0NmZK1O32CdoZguWE4Um6hqCTU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:organization:references:date:in-reply-to :message-id:user-agent:mime-version:content-type; b=EUN03vvyC3/f+24EBRcNMKbPisSmR0UlDfnO+sWBn3YWrACRaqpG6wSk85E4GJRz4e HptpC3QtEkP5jncberU9CEf1HSgsZ5qRLZEPvBJrz/VkyO4Afs92YDQJpT10zEm41+4d RkMOh8l3ZkdeQv7loZmADE0cG6j6/cXTGDWik= Received: by 10.227.137.81 with SMTP id v17mr259571wbt.128.1278062720635; Fri, 02 Jul 2010 02:25:20 -0700 (PDT) Received: from localhost (ua1.etadirect.net [91.198.140.16]) by mx.google.com with ESMTPS id i25sm3245529wbi.4.2010.07.02.02.25.18 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 02 Jul 2010 02:25:19 -0700 (PDT) From: Mikolaj Golub To: "hiroshi\@soupacific.com" Organization: TOA Ukraine References: <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> <868w6hwt2w.fsf@kopusha.home.net> <20100614153746.GN1721@garage.freebsd.pl> <86zkyxvc4v.fsf@kopusha.home.net> <4C2C43D5.1080907@soupacific.com> <86mxubndrp.fsf@kopusha.home.net> <4C2D7615.5070606@soupacific.com> <861vbm1hpr.fsf@zhuzha.ua1> <4C2D9C62.4050105@soupacific.com> Date: Fri, 02 Jul 2010 12:25:16 +0300 In-Reply-To: <4C2D9C62.4050105@soupacific.com> (hiroshi@soupacific.com's message of "Fri, 02 Jul 2010 16:59:30 +0900") Message-ID: <86wrtez14z.fsf@zhuzha.ua1> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: HAST and CARP X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jul 2010 09:25:25 -0000 On Fri, 02 Jul 2010 16:59:30 +0900 hiroshi@soupacific.com wrote: h> On 7/2/2010 4:11 PM, Mikolaj Golub wrote: >> >> So you have: >> >> secondary localcnt: 1 >> secondary remotecnt: 0 >> primary localcnt: 1 >> primary remotecnt: 0 >> >> This is a split-brain condition as described on wiki: primary's localcnt is >> greater than secondary's remotecnt (primary [fw01A] was modified while fw01B >> wasn't watching) and secondary's localcnt is greater than primary's remotecnt >> (fw01B was modified while fw01A wasn't watching). h> So hasctl role secondary xxx does not change cnt values ? Every node actually has only two values: localcnt and remotecnt. These values are kept in disk metadata. So if a node behaves as a secondary, it reads these values from the disk and stores in secondary_localcnt and secondary_remotecnt. Other two values (primary_localcnt, primary_remotecnt) are received from primary host. If I haven't overlooked something in the code secondary does not modify localcnt and remotecnt in metadata. These counters can be modified only when the node behaves as primary: on initialization primary sets them to localcnt=1 and remotecnt=0, then if data are synchronized between the nodes it sets the counters to the same values as on secondary. If primary can't send data to secondary it increases localcnt. So only primary can modify counters and if split-brain is detected that means that secondary in past was primary for some time and another node was not aware about this (or data was not synchronized). h> Scenario is this h> ServerA failed, then ServerB became MASTER. h> Only ServerA is started(say after fixed something) , both servers are h> connected,then ServerB starts, BUT during failure of ServerA, ServerB h> was MASTER. h> ServerA was started before ServerB is started, thus ServerA should be h> MASTER! h> On this situation, CARP will set ServerA is MASTER and late comer h> ServerB is set as BACKUP by CARP. h> hastctl role secondary xxx set >> secondary localcnt: 1 >> secondary remotecnt: 0 >> primary localcnt: 1 >> primary remotecnt: 0 h> above values to NOT split-brain. It sounds more favoritabel way ???? I am not sure I understand what you mean here :-) First of all you can't modify counters manually. This is maintained by hast internally. When ServerB has become master and modified some data, ServerA, before setting to primary again, should be set to secondary to synchronize all changes and only after this be switched to primary, otherwise you will have split-brain and should synchronize full storage recreating provider on secondary. -- Mikolaj Golub From owner-freebsd-fs@FreeBSD.ORG Fri Jul 2 11:05:43 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA334106564A; Fri, 2 Jul 2010 11:05:43 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id 7238B8FC1A; Fri, 2 Jul 2010 11:05:43 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id EE5366EB57; Fri, 2 Jul 2010 10:57:36 +0000 (UTC) Message-ID: <4C2DC801.5080108@soupacific.com> Date: Fri, 02 Jul 2010 20:05:37 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1 MIME-Version: 1.0 To: Mikolaj Golub References: <4C139F9C.2090305@soupacific.com><86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com><20100613003635.GA60012@icarus.home.lan><20100613074921.GB1320@garage.freebsd.pl><4C149A5C.3070401@soupacific.com><20100613102401.GE1320@garage.freebsd.pl><86eigavzsg.fsf@kopusha.home.net><20100614095044.GH1721@garage.freebsd.pl><868w6hwt2w.fsf@kopusha.home.net><20100614153746.GN1721@garage.freebsd.pl><86zkyxvc4v.fsf@kopusha.home.net> <4C2C43D5.1080907@soupacific.com><86mxubndrp.fsf@kopusha.home.net> <4C2D7615.5070606@soupacific.com><861vbm1hpr.fsf@zhuzha.ua1> <4C2D9C62.4050105@soupacific.com> <86wrtez14z.fsf@zhuzha.ua1> In-Reply-To: <86wrtez14z.fsf@zhuzha.ua1> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: HAST and CARP X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jul 2010 11:05:43 -0000 Thanks ! > > When ServerB has become master and modified some data, ServerA, before setting > to primary again, should be set to secondary to synchronize all changes and > only after this be switched to primary, otherwise you will have split-brain > and should synchronize full storage recreating provider on secondary. > I understand this idea! and agree with you this situation. BUT once we go into split-brain mode, there is no way to solve this except hastctl create xxxx. hastctl create xxx initialize hast device and not recommend idea!! I agree this. Even split-brain condition, hastctl status returns as BACKUP. I could not find the way to resolve split-brain by script. But managing ServerB became MASTER, that time we can change advskew to prevent booting ServerA keeping as BACKUP. Of course before booting SerberA, change advskew manually! But this manner can not solve split-brain. Is there any idea to solve split-brain condition? Thanks Hiroshi From owner-freebsd-fs@FreeBSD.ORG Fri Jul 2 12:25:03 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F8CC106564A; Fri, 2 Jul 2010 12:25:03 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id DE71D8FC19; Fri, 2 Jul 2010 12:25:02 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id 2DCC56EBC2; Fri, 2 Jul 2010 12:16:56 +0000 (UTC) Message-ID: <4C2DDA98.9090702@soupacific.com> Date: Fri, 02 Jul 2010 21:24:56 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1 MIME-Version: 1.0 To: Mikolaj Golub References: <4C139F9C.2090305@soupacific.com><86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com><20100613003635.GA60012@icarus.home.lan><20100613074921.GB1320@garage.freebsd.pl><4C149A5C.3070401@soupacific.com><20100613102401.GE1320@garage.freebsd.pl><86eigavzsg.fsf@kopusha.home.net><20100614095044.GH1721@garage.freebsd.pl><868w6hwt2w.fsf@kopusha.home.net><20100614153746.GN1721@garage.freebsd.pl><86zkyxvc4v.fsf@kopusha.home.net> <4C2C43D5.1080907@soupacific.com><86mxubndrp.fsf@kopusha.home.net> <4C2D7615.5070606@soupacific.com><861vbm1hpr.fsf@zhuzha.ua1> <4C2D9C62.4050105@soupacific.com> <86wrtez14z.fsf@zhuzha.ua1> In-Reply-To: <86wrtez14z.fsf@zhuzha.ua1> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: HAST and CARP X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jul 2010 12:25:03 -0000 Still I am thinking what is the better way. As Mikolaj Golub mentioned to avoide damage ServerB' hast data by ServerA boot as MASTER. Split-brain warns us to think about the situation. and save data on hast device. Such situation we can think and hastctl create xxx to late comer server. u--mm Is there any idea by other people? Hiorhsi From owner-freebsd-fs@FreeBSD.ORG Fri Jul 2 19:31:29 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 895C01065675; Fri, 2 Jul 2010 19:31:29 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 3B94D8FC0A; Fri, 2 Jul 2010 19:31:29 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o62JVTAw076500; Fri, 2 Jul 2010 19:31:29 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o62JVTNY076496; Fri, 2 Jul 2010 19:31:29 GMT (envelope-from linimon) Date: Fri, 2 Jul 2010 19:31:29 GMT Message-Id: <201007021931.o62JVTNY076496@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: bin/86765: [patch] bsdlabel(8) assigning wrong fs type. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jul 2010 19:31:29 -0000 Old Synopsis: bsdlabel(8) assigning wrong fs type. New Synopsis: [patch] bsdlabel(8) assigning wrong fs type. Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Fri Jul 2 19:30:20 UTC 2010 Responsible-Changed-Why: New patch submitted. Although it applies to bin/, perhaps the folks on -fs could review it. http://www.freebsd.org/cgi/query-pr.cgi?pr=86765 From owner-freebsd-fs@FreeBSD.ORG Sat Jul 3 08:21:21 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8A596106566C; Sat, 3 Jul 2010 08:21:21 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id A7A988FC16; Sat, 3 Jul 2010 08:21:20 +0000 (UTC) Received: by bwz12 with SMTP id 12so2411589bwz.13 for ; Sat, 03 Jul 2010 01:21:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:subject:references :x-comment-to:date:in-reply-to:message-id:user-agent:mime-version :content-type; bh=ZFg8AhBARthGdCy4EuEm3YqxQWG9W2QHkDTdajOc2hE=; b=MmTOwi4GURUQiaACo+kn4bZoGF89Fn04DRexrD4HDm0IR8QKfvaBoHrqP0TxPa4weY /w1OPf007NR+2vr0h7HP7hiLuk15K9h67IfR9jt5JOBNeAL13NqcIHQXGcxfMvWJzAp6 O5H0OrCWZKnD8TTTdxKv0tA/VSMHA+i9OXqJ8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:references:x-comment-to:date:in-reply-to :message-id:user-agent:mime-version:content-type; b=B1hxrLKFUY7Sh9oe7wHFmZFzGha+PF6tx9tvV6dGJ7Z9N3A1W6VGvWvMKTo7vzJ8/9 TLsX4R7s6+b9S7L6AI4pqmzYShTcG4K3mhUqMXhTI8k5bLY9aDPXeuAMhKCmiZ9uemM3 5po2K4EOcy36PZRmBBoMuYl1/QHD5iusaefSs= Received: by 10.204.83.204 with SMTP id g12mr93024bkl.25.1278145269327; Sat, 03 Jul 2010 01:21:09 -0700 (PDT) Received: from localhost ([95.69.160.105]) by mx.google.com with ESMTPS id 24sm6839845bkr.19.2010.07.03.01.21.07 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sat, 03 Jul 2010 01:21:07 -0700 (PDT) From: Mikolaj Golub To: "hiroshi\@soupacific.com" References: <4C139F9C.2090305@soupacific.com> <86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com> <20100613003635.GA60012@icarus.home.lan> <20100613074921.GB1320@garage.freebsd.pl> <4C149A5C.3070401@soupacific.com> <20100613102401.GE1320@garage.freebsd.pl> <86eigavzsg.fsf@kopusha.home.net> <20100614095044.GH1721@garage.freebsd.pl> <868w6hwt2w.fsf@kopusha.home.net> <20100614153746.GN1721@garage.freebsd.pl> <86zkyxvc4v.fsf@kopusha.home.net> <4C2C43D5.1080907@soupacific.com> <86mxubndrp.fsf@kopusha.home.net> <4C2D7615.5070606@soupacific.com> <861vbm1hpr.fsf@zhuzha.ua1> <4C2D9C62.4050105@soupacific.com> <86wrtez14z.fsf@zhuzha.ua1> <4C2DC801.5080108@soupacific.com> X-Comment-To: hiroshi@soupacific.com Date: Sat, 03 Jul 2010 11:21:05 +0300 In-Reply-To: <4C2DC801.5080108@soupacific.com> (hiroshi@soupacific.com's message of "Fri, 02 Jul 2010 20:05:37 +0900") Message-ID: <86iq4xx9fy.fsf@kopusha.home.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: HAST and CARP X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Jul 2010 08:21:21 -0000 On Fri, 02 Jul 2010 20:05:37 +0900 hiroshi@soupacific.com wrote: h> Thanks ! >> >> When ServerB has become master and modified some data, ServerA, before setting >> to primary again, should be set to secondary to synchronize all changes and >> only after this be switched to primary, otherwise you will have split-brain >> and should synchronize full storage recreating provider on secondary. >> h> I understand this idea! and agree with you this situation. h> BUT once we go into split-brain mode, there is no way to solve this h> except hastctl create xxxx. h> hastctl create xxx initialize hast device and not recommend idea!! I h> agree this. h> Even split-brain condition, hastctl status returns as BACKUP. h> I could not find the way to resolve split-brain by script. I don't think resolving split-brain by script is a good idea. You can detect split-brain on primary monitoring provider status and then resolve it manually. It is much better to have scripts/setup that prevent split-brain situations than scripts that resolve it :-) h> But managing ServerB became MASTER, that time we can change advskew to h> prevent booting ServerA keeping as BACKUP. h> Of course before booting SerberA, change advskew manually! But this h> manner can not solve split-brain. h> Is there any idea to solve split-brain condition? You should have a setup so when the master is rebooted after the reboot it checks the status of other node and sets its own role accordingly (so there would not be two masters simultaneously). Software I use in my setup (our home made application) does this well. sysutils/heartbeat should work fine too. As for me carp might not do well for this but I am not very experienced with carp so I can be wrong. -- Mikolaj Golub From owner-freebsd-fs@FreeBSD.ORG Sat Jul 3 08:55:17 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CB2AB1065687; Sat, 3 Jul 2010 08:55:17 +0000 (UTC) (envelope-from nwf@cs.jhu.edu) Received: from blaze.cs.jhu.edu (blaze.cs.jhu.edu [128.220.13.50]) by mx1.freebsd.org (Postfix) with ESMTP id 98FDF8FC0C; Sat, 3 Jul 2010 08:55:17 +0000 (UTC) Received: from gradx.cs.jhu.edu (gradx.cs.jhu.edu [128.220.13.52]) by blaze.cs.jhu.edu (8.14.3/8.14.3) with ESMTP id o638tG9O028698 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sat, 3 Jul 2010 04:55:16 -0400 (EDT) Received: from gradx.cs.jhu.edu (localhost [127.0.0.1]) by gradx.cs.jhu.edu (8.14.3/8.13.1) with ESMTP id o638tGZh004073; Sat, 3 Jul 2010 04:55:16 -0400 Received: (from nwf@localhost) by gradx.cs.jhu.edu (8.14.3/8.13.8/Submit) id o638tGli004072; Sat, 3 Jul 2010 04:55:16 -0400 Date: Sat, 3 Jul 2010 04:55:16 -0400 From: Nathaniel W Filardo To: alc@freebsd.org Message-ID: <20100703085516.GH21929@gradx.cs.jhu.edu> References: <20100609212747.GF21929@gradx.cs.jhu.edu> <201006100816.24077.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Bk6I6VqSt1TlXFl0" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-08-17) Cc: freebsd-fs@freebsd.org Subject: [sparc64] [ZFS] panic: mutex vnode interlock not owned X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Jul 2010 08:55:17 -0000 --Bk6I6VqSt1TlXFl0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable (hello freebsd-fs@; I'm cc:ing you since the latest part of my story involves a ZFS-related panic and I hear you're the right place to go with those. It began attempting to debug a VM locking panic and has moved a little...) On Thu, Jun 10, 2010 at 12:23:24PM -0500, Alan Cox wrote: > On Thu, Jun 10, 2010 at 7:16 AM, John Baldwin wrote: >=20 > > On Wednesday 09 June 2010 5:27:47 pm Nathaniel W Filardo wrote: > > > Attempting to boot on (2-way SMP; SUN Fire V240) sparc64 a 9.0-CURRENT > > > kernel built on Jun 9 at 14:41, and fully csup'd before building (I d= on't > > > have the SVN revision number, sorry) yields, surprisingly late in the > > boot > > > process, this panic: > > > > > > panic: mutex vm object not owned at /systank/src/sys/vm/vm_object.c:1= 692 > > > cpuid =3D 0 > > > KDB: stack backtrace: > > > panic() at panic+0x1c8 > > > _mtx_assert() at _mtx_assert+0xb0 > > > vm_object_collapse() at vm_object_collapse+0x28 > > > vm_object_deallocate() at vm_object_deallocate+0x538 > > > _vm_map_unlock() at _vm_map_unlock+0x64 > > > vm_map_remove() at vm_map_remove+0x64 > > > vmspace_exit() at vmspace_exit+0x100 > > > exit1() at exit1+0x788 > > > sys_exit() at sys_exit+0x10 > > > syscallenter() at syscallenter+0x268 > > > syscall() at syscall+0x74 > > > -- syscall (1, FreeBSD ELF64, sys_exit) %o7=3D0x11980c -- > > > userland() at 0x406fe8c8 > > > user trace: trap %o7=3D0x11980c > > > pc 0x406fe8c8, sp 0x7fdffff7611 > > > done > > > Uptime: 4m7s > > > > > > The system was, at the time, attempting to bring up its jails. > > > > > > Anything else that would be helpful to know? > > > > Can you get a crashdump? If so, it would be good to pull up gdb and ch= eck > > the > > value sof 'object' and 'robject' in the vm_object_deallocate() frame. > > > > > That would be useful. None of the locking changes of the last few weeks > have altered the vm object locking, so this assertion failure and stack > trace come as something of a surprise. >=20 > Alan Well, I thought that no longer delegating ZFS (with "zfs jail") to the jail whose startup was causing the above panic might solve the problem and indeed the system made it slightly further. A few minutes after reaching the login: prompt, though, it produced panic: mutex vnode interlock not owned at /systank/src/sys/kern/kern_mutex.= c:223 cpuid =3D 0 KDB: stack backtrace: panic() at panic+0x1c8 _mtx_assert() at _mtx_assert+0xb0 _mtx_unlock_flags() at _mtx_unlock_flags+0x144 vnlru_free() at vnlru_free+0x500 getnewvnode() at getnewvnode+0x7c zfs_znode_cache_constructor() at zfs_znode_cache_constructor+0x4c zfs_znode_alloc() at zfs_znode_alloc+0x34 zfs_zget() at zfs_zget+0x2b8 zfs_dirent_lock() at zfs_dirent_lock+0x508 zfs_dirlook() at zfs_dirlook+0x50 zfs_lookup() at zfs_lookup+0x1bc zfs_freebsd_lookup() at zfs_freebsd_lookup+0x6c VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x108 vfs_cache_lookup() at vfs_cache_lookup+0xfc VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x110 lookup() at lookup+0x7d0 namei() at namei+0x69c kern_statat_vnhook() at kern_statat_vnhook+0x48 kern_statat() at kern_statat+0x1c kern_lstat() at kern_lstat+0x18 lstat() at lstat+0x14 syscallenter() at syscallenter+0x27c syscall() at syscall+0x74 -- syscall (190, FreeBSD ELF64, lstat) %o7=3D0x12b830 -- =2E.. which at least is consistent with my hunch that the original panic had something to do with ZFS. The system is as of svn 209653 (git c65b199...) with http://people.freebsd.org/~marius/sparc64_pin_ipis.diff applied. The old kernel has uname FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #20: Sun Apr 4 20:31:58 EDT 2010 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN sparc= 64 which is probably too old to be of use to anybody, but just in case, there it is. I don't suspect the machine of having bad hardware since this old kernel runs apparently fine on it and zpool scrubs haven't found anything yet. I can't easily get a crash dump on the system (if somebody could tell me how to get one from a ddb(4) prompt, I could try that, but otherwise the system just ceases to do anything after panic; I have swap and dump set, so I'm not sure what's not happening there...). Anything more I should do? --nwf; --Bk6I6VqSt1TlXFl0 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iEYEARECAAYFAkwu+vQACgkQTeQabvr9Tc8keACgg+a5xGtLbR8EfCubtv1IJvup rGsAnioap+9N+Yl7t2ivE61COSbycHEH =RnDE -----END PGP SIGNATURE----- --Bk6I6VqSt1TlXFl0-- From owner-freebsd-fs@FreeBSD.ORG Sat Jul 3 13:41:49 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 83F831065673; Sat, 3 Jul 2010 13:41:49 +0000 (UTC) (envelope-from hiroshi@soupacific.com) Received: from mail.soupacific.com (mail.soupacific.com [211.19.53.201]) by mx1.freebsd.org (Postfix) with ESMTP id 4DE268FC17; Sat, 3 Jul 2010 13:41:49 +0000 (UTC) Received: from [127.0.0.1] (unknown [192.168.1.239]) by mail.soupacific.com (Postfix) with ESMTP id 255F56C0A1; Sat, 3 Jul 2010 13:33:44 +0000 (UTC) Message-ID: <4C2F3E14.1080601@soupacific.com> Date: Sat, 03 Jul 2010 22:41:40 +0900 From: "hiroshi@soupacific.com" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1 MIME-Version: 1.0 To: Mikolaj Golub References: <4C139F9C.2090305@soupacific.com><86iq5oc82y.fsf@kopusha.home.net> <4C14215D.9090304@soupacific.com><20100613003635.GA60012@icarus.home.lan><20100613074921.GB1320@garage.freebsd.pl><4C149A5C.3070401@soupacific.com><20100613102401.GE1320@garage.freebsd.pl><86eigavzsg.fsf@kopusha.home.net><20100614095044.GH1721@garage.freebsd.pl><868w6hwt2w.fsf@kopusha.home.net><20100614153746.GN1721@garage.freebsd.pl><86zkyxvc4v.fsf@kopusha.home.net> <4C2C43D5.1080907@soupacific.com><86mxubndrp.fsf@kopusha.home.net> <4C2D7615.5070606@soupacific.com><861vbm1hpr.fsf@zhuzha.ua1> <4C2D9C62.4050105@soupacific.com><86wrtez14z.fsf@zhuzha.ua1> <4C2DC801.5080108@soupacific.com> <86iq4xx9fy.fsf@kopusha.home.net> In-Reply-To: <86iq4xx9fy.fsf@kopusha.home.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: HAST and CARP X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Jul 2010 13:41:49 -0000 Hi Mikolaj! > > I don't think resolving split-brain by script is a good idea. You can detect > split-brain on primary monitoring provider status and then resolve it > manually. It is much better to have scripts/setup that prevent split-brain > situations than scripts that resolve it :-) I agree with you solving split-brain by script. Today I was thinking about this situation resolving, and conclusion is some thing like hastctl role secondary -f xxx But this command can be used only for emergency situation and not inside script. ifstated is enabled in rc.conf, once reboot ServerB without ServerA connection. Such case ServerB became CARP's MASTER then ifstated also set hast as primary. This may happen often when setting hast testing time. So I think some rescue command may help to get out split-brain situation caused by any accident or mistake. > > You should have a setup so when the master is rebooted after the reboot it > checks the status of other node and sets its own role accordingly (so there > would not be two masters simultaneously). Software I use in my setup (our home > made application) does this well. sysutils/heartbeat should work fine too. As > for me carp might not do well for this but I am not very experienced with carp > so I can be wrong. > By CARP, ifconfig carp0 advskew {bigger value than secondary} on console sets CARP as secondary. How do you think this idea ? Hiroshi From owner-freebsd-fs@FreeBSD.ORG Sat Jul 3 18:28:04 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 46C3C106566C for ; Sat, 3 Jul 2010 18:28:04 +0000 (UTC) (envelope-from bounces@nabble.com) Received: from kuber.nabble.com (kuber.nabble.com [216.139.236.158]) by mx1.freebsd.org (Postfix) with ESMTP id 22B8A8FC12 for ; Sat, 3 Jul 2010 18:28:03 +0000 (UTC) Received: from isper.nabble.com ([192.168.236.156]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from ) id 1OV7Rb-0002Wv-BH for freebsd-fs@freebsd.org; Sat, 03 Jul 2010 11:28:03 -0700 Message-ID: <29065362.post@talk.nabble.com> Date: Sat, 3 Jul 2010 11:28:03 -0700 (PDT) From: Bucarr To: freebsd-fs@freebsd.org In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: bucarr@gmail.com References: Subject: Re: zfs on 4k sector disks X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Jul 2010 18:28:04 -0000 I was wondering that as well. So I set up a new FreeBSD8.0 box with a UFS2 disk as my boot disk and two WD 2TB EARS drives ("Advanced Format") for a zfs data array. I made no attempt to alter the WD drives in any way and ignored the 512b vs 4096b issues. 'zpool create tank raidz da0 da1' set up the array in a few seconds and I've copied to/from with no troubles and no apparent performance issues that I care about. If I had more of the WD drives lying around, I'd add those to the array and see what gives then. HTH, Bucky mailinglists-19 wrote: > > Hello, > > I've got 3 1.5TB drives that are using the "Advanced Format" from > Western Digital, see the link for details: > http://www.wdc.com/wdproducts/library/WhitePapers/ENG/2579-771430.pdf > This "Advanced Format" basically means the sectors size on disk is > changed to 4k in stead of 512k. > > I'm wondering if freeBSD 8.0 using ZFS raidz and drives like these will > work properly, I've seen a lot of articles and postings about partitions > not being properly aligned with the physical disk layout and suffering > big performance hits. My question is; should I worry about these issues > or does ZFS already align the data on a 4k border? > > My setup would be 3 * 1.5TB WD drives (WD15EARS) using the whole drive > for a raidz pool. > > Thanks, > Rob Evers > ***deze e-mail is gescand door Onlinespamfilter.nl*** > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > -- View this message in context: http://old.nabble.com/zfs-on-4k-sector-disks-tp27664932p29065362.html Sent from the freebsd-fs mailing list archive at Nabble.com. From owner-freebsd-fs@FreeBSD.ORG Sat Jul 3 21:12:18 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49B111065673 for ; Sat, 3 Jul 2010 21:12:18 +0000 (UTC) (envelope-from sfourman@gmail.com) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 051EB8FC0C for ; Sat, 3 Jul 2010 21:12:17 +0000 (UTC) Received: by qyk7 with SMTP id 7so1139056qyk.13 for ; Sat, 03 Jul 2010 14:12:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=kVMB8nFVGUcAoZmuJHwCiLlP6j1a3RVhgnpwmYhVpo4=; b=f0wEBkjS+Q6ZVRjWHJ51vt+1I9jc2iaj7FqN2ylEPhfDO80e/ZpgSICemlolOJmWBL Pjl6NVAgDRTarSCSTSeX1/fB9r8/0R7kuLYyxBFOE98x3kpqlkRFasV8u7YPYG+ORLDN 6gUzs03U+UI+7KvOSi0FtTTR+6BFIYSlR9iEY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=tmTBi4iZz2LjwlxwSuBWgkxkEeBkZymjvNb9k+uksy7V6H6AvFRgN6MgJ1YFkpVH4g 4AsAOFNIPz5Pap22FfyIPKKjPW5hMYzToXPuWTSvcc9oYEZbhcfwcFE4CeW79UVQ/mHy 1iSD6M4QXUEu97PCKz/RpooYW01WQ/jgh1GjI= MIME-Version: 1.0 Received: by 10.229.225.16 with SMTP id iq16mr412332qcb.67.1278191527460; Sat, 03 Jul 2010 14:12:07 -0700 (PDT) Received: by 10.229.41.203 with HTTP; Sat, 3 Jul 2010 14:12:07 -0700 (PDT) Date: Sat, 3 Jul 2010 16:12:07 -0500 Message-ID: From: "Sam Fourman Jr." To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Help with Faulted zpool X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Jul 2010 21:12:18 -0000 Hello list, I have a File server that runs FreeBSD 8.1 (zfs v14) after a poweroutage, I am unable to import my zpool named Network my pool is made up of 6 1TB disks configured in raidz. there is ~1.9TB of actual data on this pool. FreeBSD does not have many option for restoring a pool from corrupt meta data I have loaded OpenSolaris svn_134 on a septate boot disk, in hopes of recovering my zpool. on Opensolaris 134, I am not able to import my zpool almost everything I try gives me cannot import 'Network': I/O error I have done quite a bit of searching, and I found that import -fFX Network should work however after ~ 20 hours this hard locks Opensolaris (however it does return a ping) here is a list of commands that I have run on Open Solaris http://www.puffybsd.com/zfsv14.txt if anyone could help me use zdb or mdb to recover my pool I would very much appreciate it. I believe the metadata is corrupt on my zpool -- Sam Fourman Jr. Fourman Networks http://www.fourmannetworks.com From owner-freebsd-fs@FreeBSD.ORG Sat Jul 3 23:53:43 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89292106566B for ; Sat, 3 Jul 2010 23:53:43 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (muon.cran.org.uk [204.109.60.94]) by mx1.freebsd.org (Postfix) with ESMTP id 4F4938FC08 for ; Sat, 3 Jul 2010 23:53:43 +0000 (UTC) Received: from unknown (87-194-158-129.bethere.co.uk [87.194.158.129]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id 9A93760D2; Sat, 3 Jul 2010 23:53:55 +0000 (UTC) Date: Sun, 4 Jul 2010 00:53:36 +0100 From: Bruce Cran To: Bucarr Message-ID: <20100704005336.00003234@unknown> In-Reply-To: <29065362.post@talk.nabble.com> References: <29065362.post@talk.nabble.com> X-Mailer: Claws Mail 3.7.4cvs1 (GTK+ 2.16.0; i586-pc-mingw32msvc) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: zfs on 4k sector disks X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Jul 2010 23:53:43 -0000 On Sat, 3 Jul 2010 11:28:03 -0700 (PDT) Bucarr wrote: > I was wondering that as well. So I set up a new FreeBSD8.0 box with > a UFS2 disk as my boot disk and two WD 2TB EARS drives ("Advanced > Format") for a zfs data array. I made no attempt to alter the WD > drives in any way and ignored the 512b vs 4096b issues. 'zpool > create tank raidz da0 da1' set up the array in a few seconds and I've > copied to/from with no troubles and no apparent performance issues > that I care about. If I had more of the WD drives lying around, I'd > add those to the array and see what gives then. I suspect the performance may be acceptable, but probably not great. I thought I didn't have any problems with RAIDZ's variable stripe size on my EARS drives either until I found some writes would suddenly take ages to finish, causing applications to hang. -- Bruce Cran