From owner-freebsd-fs@FreeBSD.ORG Thu Dec 25 19:31:33 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8E87C376 for ; Thu, 25 Dec 2014 19:31:33 +0000 (UTC) Received: from mail-wi0-f181.google.com (mail-wi0-f181.google.com [209.85.212.181]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2339664798 for ; Thu, 25 Dec 2014 19:31:32 +0000 (UTC) Received: by mail-wi0-f181.google.com with SMTP id r20so15920843wiv.8 for ; Thu, 25 Dec 2014 11:31:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=8zCkVobR2PIPpQs5SHUsqYmgoiHhabpDKEUcoNiQnTU=; b=Eky25uJmWhAy7r7qPQp8g//ahLavb/eLa2PI4t3iD2zJoyvs203GQO8arjAKtuPUj4 z7epLpHDv+OBAl63sTtX0a2X2J7OwG32+0xvvbDgXzSmFQpvBW1CfQVLUV/hgEq8SeBZ +wFiAtUbIFfloW/g0sYaOjyCpodh583YFFq0r4KgKLt2ViTWkT/xO0YF3uhJBjVDgJCc vU8eRlxjW3DTKn3zq/QUiucK5FUrXCrmnPh1Eof7F7ExVYqJYdOfaymnqy0hq2AiAlE7 WlIxDgkoxfqSstinueOcqWpd2UoLaHXg3ZHgmjIxVwYo+DEsi02XxRREHyiJ6ULlQ3Mz WG+w== X-Gm-Message-State: ALoCoQnt0YbcWwEcAEIljTC2JAbmOJZHUb+esNlQ/l67SWikgutZ0hJVcZkmWIo7oBonNH24pdRE X-Received: by 10.194.79.199 with SMTP id l7mr76187099wjx.136.1419535885084; Thu, 25 Dec 2014 11:31:25 -0800 (PST) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id gf6sm36132249wjc.11.2014.12.25.11.31.24 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 25 Dec 2014 11:31:24 -0800 (PST) Message-ID: <549C65FF.4010702@multiplay.co.uk> Date: Thu, 25 Dec 2014 19:31:11 +0000 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: LSI SAS 9300-8i weird ZFS checksum errors References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Dec 2014 19:31:33 -0000 On 25/12/2014 14:39, George Kontostanos wrote: > Hello, list and Merry Christmas to all > > I am facing some weird checksum errors during scrub. The configuration is > the following: > > Board: Supermicro Motherboard X10DRi-T4+ ( > http://www.supermicro.com/products/motherboard/xeon/c600/x10dri-t4_.cfm) > Controller: LSI SAS 9300-8i ( > http://www.lsi.com/products/host-bus-adapters/pages/lsi-sas-9300-8i.aspx) > HDD: 21X6TB Western Digital WD60EFRX > HDD: 2XIntel SATA 600GB Solid-State Drive SSDSC2BB600G401 DC S3500 > (SWAP, ZIL, CACHE) > Chassis: Supermicro 847BE1C-R1K28LPB 4U Storage Chassis > RAM: 64 GB > > I installed initially FreeBSD 10.1-RELEASE created one pool consistent by 3 > X7disk VDEVs in RAIDZ3. I used NFS to start copying some data. After > copying around 3TB I initiated a scrub. > The result was the following: http://pastebin.com/rswgCY2A and > http://pastebin.com/DQ2urGXk > > I tried to flash the controller but the LSI utility did not recognize the > controller. I installed FreeBSD 9.3-RELEASE and used LSI's mpslsi3 driver. > I was able to flash the latest bios and firmware that way. > > LSI Corporation SAS3 Flash Utility > Version 07.00.00.00 (2014.08.14) > Copyright (c) 2008-2014 LSI Corporation. All rights reserved > > Adapter Selected is a LSI SAS: SAS3008(C0) > > Controller Number : 0 > Controller : SAS3008(C0) > PCI Address : 00:82:00:00 > SAS Address : 500605b-0-06ce-27e0 > NVDATA Version (Default) : 06.03.00.05 > NVDATA Version (Persistent) : 06.03.00.05 > Firmware Product ID : 0x2221 (IT) > Firmware Version : 06.00.00.00 > NVDATA Vendor : LSI > NVDATA Product ID : SAS9300-8i > BIOS Version : 08.13.00.00 > UEFI BSD Version : 02.00.00.00 > FCODE Version : N/A > Board Name : SAS9300-8i > Board Assembly : H3-25573-00E > Board Tracer Number : SV32928040 > > I recreated the pool again and started writing data via NFS again. After 3 > TB of data I started a scrub and I am still getting checksum errors though > there are no messages regarding the drives anymore in /var/log/messages > > pool: Pool > state: ONLINE > status: One or more devices has experienced an unrecoverable error. An > attempt was made to correct the error. Applications are unaffected. > action: Determine if the device needs to be replaced, and clear the errors > using 'zpool clear' or replace the device with 'zpool replace'. > see: http://illumos.org/msg/ZFS-8000-9P > > scan: scrub in progress since Thu Dec 25 08:46:21 2014 > 2.28T scanned out of 5.54T at 816M/s, 1h9m to go > 11.9M repaired, 41.26% done > config: > > NAME STATE READ WRITE CKSUM > Pool ONLINE 0 0 0 > raidz3-0 ONLINE 0 0 0 > gpt/WD-WX41D94RN5A3 ONLINE 0 0 15 (repairing) > gpt/WD-WX41D948YE1U ONLINE 0 0 14 (repairing) > gpt/WD-WX41D94RN879 ONLINE 0 0 16 (repairing) > gpt/WD-WX21D947NC83 ONLINE 0 0 24 (repairing) > gpt/WD-WX21D947NT77 ONLINE 0 0 15 (repairing) > gpt/WD-WX41D948YAKV ONLINE 0 0 19 (repairing) > gpt/WD-WX21D9421SCV ONLINE 0 0 20 (repairing) > raidz3-1 ONLINE 0 0 0 > gpt/WD-WX21D9421F6F ONLINE 0 0 16 (repairing) > gpt/WD-WX41D948YPN4 ONLINE 0 0 14 (repairing) > gpt/WD-WX21D947NE2K ONLINE 0 0 22 (repairing) > gpt/WD-WX41D948Y2PX ONLINE 0 0 19 (repairing) > gpt/WD-WX41D94RNAX7 ONLINE 0 0 17 (repairing) > gpt/WD-WX21D947N1RP ONLINE 0 0 12 (repairing) > gpt/WD-WX21D94216X7 ONLINE 0 0 20 (repairing) > raidz3-2 ONLINE 0 0 0 > gpt/WD-WX41D948YAHP ONLINE 0 0 25 (repairing) > gpt/WD-WX21D947N06F ONLINE 0 0 18 (repairing) > gpt/WD-WX21D947N3T1 ONLINE 0 0 21 (repairing) > gpt/WD-WX41D94RNT7D ONLINE 0 0 5 (repairing) > gpt/WD-WX41D948Y9VV ONLINE 0 0 18 (repairing) > gpt/WD-WX41D94RNS62 ONLINE 0 0 24 (repairing) > gpt/WD-WX21D9421ZP9 ONLINE 0 0 28 (repairing) > logs > mirror-3 ONLINE 0 0 0 > gpt/zil0 ONLINE 0 0 0 > gpt/zil1 ONLINE 0 0 0 > cache > gpt/cache0 ONLINE 0 0 0 > gpt/cache1 ONLINE 0 0 0 > > errors: No known data errors > > This is really driving me crazy since smartmon tools do not display any > errors on the drives. > > Any suggestions are most welcomed!!! > Check for bad hardware, first guess would be memory, next would be hotswap backplane. Regards Steve