From owner-freebsd-fs@FreeBSD.ORG Wed Oct 31 17:25:17 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EDBBC805 for ; Wed, 31 Oct 2012 17:25:17 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 503C08FC1A for ; Wed, 31 Oct 2012 17:25:16 +0000 (UTC) Received: by mail-we0-f182.google.com with SMTP id x43so936011wey.13 for ; Wed, 31 Oct 2012 10:25:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:from:to:subject:date:mime-version:content-type :content-transfer-encoding:x-priority:x-msmail-priority:x-mailer :x-mimeole:x-gm-message-state; bh=/UdVNw6/TtBky4k6o1nrrKoz8JR6OpY1IC9KmusreE0=; b=NfEd3zz/4UImDK47Ceai5+LcB4jybDJHOwsQ8pgXlWJJTrSXYO1JlMnyEYL5H4FON7 A0CVKMWUMqiZiMxEWTrEdKCFg7EOlXZ30pfLQVfHg3l4rSqCl8JI/T0+/A/t3jLMNl8U i5oEe8cN2AZL+DqhxBNYeC5Q2dS7Gc2D/Uf9DoJ6DAI7YraAsFzfhjfu0OSULVNQvRZn /c9ELHZAtfpCJbQeFVwXg3glBWIrGcJXSh2c4x/wUVMz7XYPZeIf3fc7FPmqkvLNIB0R gyDsgwSdR1EtK6JXKG3MxO+sfLzoVtGsBhVHuQ/W97sbTMUYFFwIDXXnndQ/pGAqnuzw /YVQ== Received: by 10.216.200.150 with SMTP id z22mr18350489wen.97.1351704309968; Wed, 31 Oct 2012 10:25:09 -0700 (PDT) Received: from r2d2 (188-220-16-49.zone11.bethere.co.uk. [188.220.16.49]) by mx.google.com with ESMTPS id eq2sm6687299wib.1.2012.10.31.10.25.08 (version=SSLv3 cipher=OTHER); Wed, 31 Oct 2012 10:25:08 -0700 (PDT) Message-ID: <27087376D1C14132A3CC1B4016912F6D@multiplay.co.uk> From: "Steven Hartland" To: , Subject: ZFS corruption due to lack of space? Date: Wed, 31 Oct 2012 17:25:09 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-Gm-Message-State: ALoCoQkEI8wXqpj6AnyQhgucAaDUsafj5x2dlzlb4LntFGSDRm7T4rpXK2J/mvJYa2HRJdf+CSDO X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Oct 2012 17:25:18 -0000 Been running some tests on new hardware here to verify all is good. One of the tests was to fill the zfs array which seems like its totally corrupted the tank. The HW is 7 x 3TB disks in RAIDZ2 with dual 13GB ZIL partitions and dual 100GB L2ARC on Enterprise SSD's. All disks are connected to an LSI 2208 RAID controller run by mfi driver. HD's via a SAS2X28 backplane and SSD's via a passive blackplane backplane. The file system has 31 test files most random data from /dev/random and one blank from /dev/zero. The test running was multiple ~20 dd's under screen with all but one from /dev/random and to final one from /dev/zero e.g. dd if=/dev/random bs=1m of=/tank2/random10 No hardware errors have raised, so no disk timeouts etc. On completion each dd reported no space as you would expect e.g. dd if=/dev/random bs=1m of=/tank2/random13 dd: /tank2/random13: No space left on device 503478+0 records in 503477+0 records out 527933898752 bytes transferred in 126718.731762 secs (4166187 bytes/sec) You have new mail. At that point with the test seemingly successful I went to delete test files which resulted in:- rm random* rm: random1: Unknown error: 122 rm: random10: Unknown error: 122 rm: random11: Unknown error: 122 rm: random12: Unknown error: 122 rm: random13: Unknown error: 122 rm: random14: Unknown error: 122 rm: random18: Unknown error: 122 rm: random2: Unknown error: 122 rm: random3: Unknown error: 122 rm: random4: Unknown error: 122 rm: random5: Unknown error: 122 rm: random6: Unknown error: 122 rm: random7: Unknown error: 122 rm: random9: Unknown error: 122 Error 122 I assume is ECKSUM At this point the pool was showing checksum errors zpool status pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gptid/41fb7e5c-21cf-11e2-92a3-002590881138 ONLINE 0 0 0 gptid/42a1b53c-21cf-11e2-92a3-002590881138 ONLINE 0 0 0 errors: No known data errors pool: tank2 state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: none requested config: NAME STATE READ WRITE CKSUM tank2 ONLINE 0 0 4.22K raidz2-0 ONLINE 0 0 16.9K mfisyspd0 ONLINE 0 0 0 mfisyspd1 ONLINE 0 0 0 mfisyspd2 ONLINE 0 0 0 mfisyspd3 ONLINE 0 0 0 mfisyspd4 ONLINE 0 0 0 mfisyspd5 ONLINE 0 0 0 mfisyspd6 ONLINE 0 0 0 logs mfisyspd7p3 ONLINE 0 0 0 mfisyspd8p3 ONLINE 0 0 0 cache mfisyspd9 ONLINE 0 0 0 mfisyspd10 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: tank2:<0x3> tank2:<0x8> tank2:<0x9> tank2:<0xa> tank2:<0xb> tank2:<0xf> tank2:<0x10> tank2:<0x11> tank2:<0x12> tank2:<0x13> tank2:<0x14> tank2:<0x15> So I tried a scrub, which looks like its going to take 5 days to complete and is reporting many many more errors:- pool: tank2 state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub in progress since Wed Oct 31 16:13:53 2012 118G scanned out of 18.7T at 42.2M/s, 128h19m to go 49.0M repaired, 0.62% done config: NAME STATE READ WRITE CKSUM tank2 ONLINE 0 0 596K raidz2-0 ONLINE 0 0 1.20M mfisyspd0 ONLINE 0 0 0 (repairing) mfisyspd1 ONLINE 0 0 0 (repairing) mfisyspd2 ONLINE 0 0 0 (repairing) mfisyspd3 ONLINE 0 0 2 (repairing) mfisyspd4 ONLINE 0 0 1 (repairing) mfisyspd5 ONLINE 0 0 0 (repairing) mfisyspd6 ONLINE 0 0 1 (repairing) logs mfisyspd7p3 ONLINE 0 0 0 mfisyspd8p3 ONLINE 0 0 0 cache mfisyspd9 ONLINE 0 0 0 mfisyspd10 ONLINE 0 0 0 errors: 596965 data errors, use '-v' for a list At this point I decided to cancel the scrub but no joy on that zpool scrub -s tank2 cannot cancel scrubbing tank2: out of space So questions:- 1. Given the information it seems like the multiple writes filling the disk may have caused metadata corruption? 2. Is there anyway to stop the scrub? 3. Surely low space should never prevent stopping a scrub? Regards Steve