From owner-freebsd-stable@FreeBSD.ORG Sun Aug 19 18:07:15 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23E9C106566C; Sun, 19 Aug 2012 18:07:15 +0000 (UTC) (envelope-from gkontos.mail@gmail.com) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id B86B28FC1E; Sun, 19 Aug 2012 18:07:14 +0000 (UTC) Received: by vcbgb22 with SMTP id gb22so5643227vcb.13 for ; Sun, 19 Aug 2012 11:07:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=SQPFuNf/OUqihfQOyITzeqiXVUVED0ZU7/mpqCVSklY=; b=cHtAiYp9A9QkPV8obZbLy1HDQ6rtgfqFch2CCd8xJurOqjHY/73R2uViTkdqIQ3hqb 4ykDEf32cbdNli0KQqX0/WPO45BIuwM8ecQn5Sl7t34pCc/LQ+FF+3R5GstnYLNSqedg gWZ52OusGFZhTjPWxK0oizS4zKy0tgVFp7UoG4GQXTl/reByaEBJQGP1YEEkilmtTWtV fjjvgkvlgajYBS5cDEiBiTfCxAEhlJ9Pf9BPfrK4PJ81DBTJLgrUx1KaMosNFUjBMVIj HfojTUsqAV7n0g+svFrC0O72EzpEvk5YDIWoKN1n6lbwebMstLlBHQ8sEfVfVm2efBuP 121w== MIME-Version: 1.0 Received: by 10.58.211.71 with SMTP id na7mr8840121vec.39.1345399633803; Sun, 19 Aug 2012 11:07:13 -0700 (PDT) Received: by 10.220.157.8 with HTTP; Sun, 19 Aug 2012 11:07:13 -0700 (PDT) In-Reply-To: <201208191758.q7JHw5xG010417@gw.catspoiler.org> References: <201208191758.q7JHw5xG010417@gw.catspoiler.org> Date: Sun, 19 Aug 2012 21:07:13 +0300 Message-ID: From: George Kontostanos To: Don Lewis Content-Type: text/plain; charset=ISO-8859-1 Cc: FreeBSD Stable Subject: Re: LSI 9240-4i 4K alignment X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Aug 2012 18:07:15 -0000 On Sun, Aug 19, 2012 at 8:58 PM, Don Lewis wrote: > On 8 Aug, George Kontostanos wrote: >> Hi all, >> >> We have a server with a LSI 9240-4i controller configured in JBOD with >> 4 SATA disks. Running FreeBSD 9.1-Beta1: >> >> Relevant dmesg: >> >> FreeBSD 9.1-BETA1 #0: Thu Jul 12 09:38:51 UTC 2012 >> root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 >> CPU: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz (3200.09-MHz K8-class CPU) >> Origin = "GenuineIntel" Id = 0x206a7 Family = 6 Model = 2a Stepping = 7 >> Features=0xbfebfbff >> Features2=0x1fbae3ff >> AMD Features=0x28100800 >> AMD Features2=0x1 >> TSC: P-state invariant, performance statistics >> real memory = 17179869184 (16384 MB) >> avail memory = 16471670784 (15708 MB) >> ... >> mfi0: port 0xe000-0xe0ff mem >> 0xf7a60000-0xf7a63fff,0xf7a00000-0xf7a3ffff irq 16 at device 0.0 on >> pci1 >> mfi0: Using MSI >> mfi0: Megaraid SAS driver Ver 4.23 >> ... >> mfi0: 321 (397672301s/0x0020/info) - Shutdown command received from host >> mfi0: 322 (boot + 3s/0x0020/info) - Firmware initialization started >> (PCI ID 0073/1000/9241/1000) >> mfi0: 323 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664 >> mfi0: 324 (boot + 3s/0x0020/info) - Firmware initialization started >> (PCI ID 0073/1000/9241/1000) >> mfi0: 325 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664 >> mfi0: 326 (boot + 5s/0x0020/info) - Package version 20.10.1-0107 >> mfi0: 327 (boot + 5s/0x0020/info) - Board Revision 03A >> mfi0: 328 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) >> ... >> mfisyspd0 on mfi0 >> mfisyspd0: 1907729MB (3907029168 sectors) SYSPD volume >> mfisyspd0: SYSPD volume attached >> mfisyspd1 on mfi0 >> mfisyspd1: 1907729MB (3907029168 sectors) SYSPD volume >> mfisyspd1: SYSPD volume attached >> mfisyspd2 on mfi0 >> mfisyspd2: 1907729MB (3907029168 sectors) SYSPD volume >> mfisyspd2: SYSPD volume attached >> mfisyspd3 on mfi0 >> mfisyspd3: 1907729MB (3907029168 sectors) SYSPD volume >> mfisyspd3: SYSPD volume attached >> ... >> mfi0: 329 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) Info: >> enclPd=ffff, scsiType=0, portMap=00, >> sasAddr=4433221100000000,0000000000000000 >> mfi0: 330 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) >> mfi0: 331 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) Info: >> enclPd=ffff, scsiType=0, portMap=02, >> sasAddr=4433221102000000,0000000000000000 >> mfi0: 332 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) >> mfi0: 333 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) Info: >> enclPd=ffff, scsiType=0, portMap=03, >> sasAddr=4433221101000000,0000000000000000 >> mfi0: 334 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) >> mfi0: 335 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) Info: >> enclPd=ffff, scsiType=0, portMap=01, >> sasAddr=4433221103000000,0000000000000000 >> mfi0: 336 (397672376s/0x0020/info) - Time established as 08/07/12 >> 16:32:56; (28 seconds since power on) >> >> The problem: >> >> When trying to create a RaidZ pool using gpart and perform a 4K >> alignment using gnop, we get the follwoing error immediately after >> exporting the pool and destroying the .nop devices: >> >> id: 8043746387654554958 >> state: FAULTED >> status: One or more devices contains corrupted data. >> action: The pool cannot be imported due to damaged devices or data. >> The pool may be active on another system, but can be imported using >> the '-f' flag. >> see: http://illumos.org/msg/ZFS-8000-5E >> config: >> >> Pool FAULTED corrupted data >> raidz1-0 ONLINE >> 13283347160590042564 UNAVAIL corrupted data >> 16981727992215676534 UNAVAIL corrupted data >> 6607570030658834339 UNAVAIL corrupted data >> 3435463242860701988 UNAVAIL corrupted data > > I'm planning on doing something similar, but I'm curious about how gnop > and GPT labels interact. I want to partition the drives for my pool > slightly on the small side so that I'm less likely to run into problems > if I have to replace a drive in the future. If I used gpart to create > and label a GPT partition on the drive, the partition will show up as > /dev/gpt/label. The gnop man page says that running gnop on dev creates > /dev/dev.nop. What happens if you gnop /dev/gpt/label? Is this what > you are doing? > >> When we use glabel for the same purpose with the combination of gnop, >> the pool imports fine. >> >> Any suggestions? > > It should be sufficient to only gnop one of the devices. You should be > able to create the pool with only one gnop device to get the ashift > value that you desire, export the pool, destroy the .nop device, and > import the pool. If it things the device is corrupted (which seems like > a bug of some sort), then ZFS should be able to resilver it. > After a few tests with many scenarios, the pool shows as ASHIFT=12 with the devices gnoped directly. However, the performance is decreasing dramatically after adding a few TB of data. We haven't used more than 30% of its capacity yet on a RaidZ1 pool. Scrub speed has been reduced to 20M max. We will park the controller for now and use the standard HBA supported by the mps driver :) -- George Kontostanos Aicom telecoms ltd http://www.aisecure.net