From owner-freebsd-fs@FreeBSD.ORG Sun Jun 19 14:20:12 2011 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E26171065670 for ; Sun, 19 Jun 2011 14:20:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id C7FF88FC18 for ; Sun, 19 Jun 2011 14:20:11 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p5JEKBY6091398 for ; Sun, 19 Jun 2011 14:20:11 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p5JEKBIQ091397; Sun, 19 Jun 2011 14:20:11 GMT (envelope-from gnats) Date: Sun, 19 Jun 2011 14:20:11 GMT Message-Id: <201106191420.p5JEKBIQ091397@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Cc: Subject: Re: kern/154930: [zfs] cannot delete/unlink file from full volume -> ENOSPC X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: soralx@cydem.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Jun 2011 14:20:12 -0000 The following reply was made to PR kern/154930; it has been noted by GNATS. From: To: , Cc: Subject: Re: kern/154930: [zfs] cannot delete/unlink file from full volume -> ENOSPC Date: Sun, 19 Jun 2011 06:54:26 -0700 All, I encountered a similar snag, only worse, with no solution. The server has a 7 TB ZFS volume, which consists of 5 WD2001FASS connected to PERC 5/i in RAID5 (originally there were 4 disks, but the RAID was expanded by adding another disk and rebuilding). On June 12th (the date of OS rebuild, both world and kernel; the last update was on June 6), the pool was at v15, and had 12 GB free. On June 14, the pool already had 0 bytes free (found out from periodic daily run output email). It is unknown whether the FS was accessed at all during these two days; however, `find ./ -newerBt 2011-06-10 -print` returns just one small file dated 2011-06-11, and `find ./ -newermt 2011-06-10` returns ~20 files with total size <1 GB (BTW, these are backups, and nobody could have touched them, so it's a mystery why they have a recent modified time). So, the question is: where could have 12 GB suddenly disappear to? Further, on June 18th, the pool was mounted when a `zpool upgrade tst` command was issued. The upgrade to v28 succeeded, but then I found out that deleting files, large or small, is impossible: # rm ./qemu0.raw rm: ./qemu0.raw: No space left on device # truncate -s0 ./qemu0.raw truncate: ./qemu0.raw: No space left on device # cat /dev/null > ./qemu0.raw ./qemu0.raw: No space left on device. Snapshots, compression, or deduplication have never been used on the volume. Also, these messages appear in dmesg: Solaris: WARNING: metaslab_free_dva(): bad DVA 0:5978620460544 Solaris: WARNING: metaslab_free_dva(): bad DVA 0:5978620461568 Solaris: WARNING: metaslab_free_dva(): bad DVA 0:5993168926208 Contrary to what the pool's name might suggest, this is not test storage, but has valuable data on it. Help! >Environment: System: FreeBSD cydem.org 8.2-STABLE FreeBSD 8.2-STABLE #0: Sun Jun 12 07:55:32 PDT 2011 soralx@cydem.org:/usr/obj/usr/src/sys/CYDEM amd64 `df`: Filesystem 1K-blocks Used Avail Capacity Mounted on tst 7650115271 7650115271 0 100% /stor1-tst `zpool list`: NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT tst 5.44T 5.36T 80.6G 98% 1.00x ONLINE - `zfs list`: NAME USED AVAIL REFER MOUNTPOINT tst 7.12T 0 7.12T /stor1-tst `zpool status`: pool: tst state: ONLINE scan: scrub in progress since Sat Jun 18 06:32:37 2011 1.82T scanned out of 5.36T at 350M/s, 2h56m to go 0 repaired, 34.02% done config: NAME STATE READ WRITE CKSUM tst ONLINE 0 0 0 mfid1 ONLINE 0 0 0 errors: No known data errors [this completed with 'scrub repaired 0 in 5h47m with 0 errors' at 133% done, i.e. it scrubbed all 7.12 TB]. -- [SorAlx] ridin' VN2000 Classic LT