Date: Sun, 19 Aug 2012 21:07:13 +0300 From: George Kontostanos <gkontos.mail@gmail.com> To: Don Lewis <truckman@freebsd.org> Cc: FreeBSD Stable <freebsd-stable@freebsd.org> Subject: Re: LSI 9240-4i 4K alignment Message-ID: <CA%2BdUSyrdQm7ON1awHZT8Dp0hMZ2TOBoUf9O9Kmmo1TLsgGJreQ@mail.gmail.com> In-Reply-To: <201208191758.q7JHw5xG010417@gw.catspoiler.org> References: <CA%2BdUSyrdPsDJyZ0Ow3%2B6sNzc10we2BGV10tUYtzkuE5u9Ook2A@mail.gmail.com> <201208191758.q7JHw5xG010417@gw.catspoiler.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Aug 19, 2012 at 8:58 PM, Don Lewis <truckman@freebsd.org> wrote: > On 8 Aug, George Kontostanos wrote: >> Hi all, >> >> We have a server with a LSI 9240-4i controller configured in JBOD with >> 4 SATA disks. Running FreeBSD 9.1-Beta1: >> >> Relevant dmesg: >> >> FreeBSD 9.1-BETA1 #0: Thu Jul 12 09:38:51 UTC 2012 >> root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 >> CPU: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz (3200.09-MHz K8-class CPU) >> Origin = "GenuineIntel" Id = 0x206a7 Family = 6 Model = 2a Stepping = 7 >> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> >> Features2=0x1fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX> >> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> >> AMD Features2=0x1<LAHF> >> TSC: P-state invariant, performance statistics >> real memory = 17179869184 (16384 MB) >> avail memory = 16471670784 (15708 MB) >> ... >> mfi0: <Drake Skinny> port 0xe000-0xe0ff mem >> 0xf7a60000-0xf7a63fff,0xf7a00000-0xf7a3ffff irq 16 at device 0.0 on >> pci1 >> mfi0: Using MSI >> mfi0: Megaraid SAS driver Ver 4.23 >> ... >> mfi0: 321 (397672301s/0x0020/info) - Shutdown command received from host >> mfi0: 322 (boot + 3s/0x0020/info) - Firmware initialization started >> (PCI ID 0073/1000/9241/1000) >> mfi0: 323 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664 >> mfi0: 324 (boot + 3s/0x0020/info) - Firmware initialization started >> (PCI ID 0073/1000/9241/1000) >> mfi0: 325 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664 >> mfi0: 326 (boot + 5s/0x0020/info) - Package version 20.10.1-0107 >> mfi0: 327 (boot + 5s/0x0020/info) - Board Revision 03A >> mfi0: 328 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) >> ... >> mfisyspd0 on mfi0 >> mfisyspd0: 1907729MB (3907029168 sectors) SYSPD volume >> mfisyspd0: SYSPD volume attached >> mfisyspd1 on mfi0 >> mfisyspd1: 1907729MB (3907029168 sectors) SYSPD volume >> mfisyspd1: SYSPD volume attached >> mfisyspd2 on mfi0 >> mfisyspd2: 1907729MB (3907029168 sectors) SYSPD volume >> mfisyspd2: SYSPD volume attached >> mfisyspd3 on mfi0 >> mfisyspd3: 1907729MB (3907029168 sectors) SYSPD volume >> mfisyspd3: SYSPD volume attached >> ... >> mfi0: 329 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) Info: >> enclPd=ffff, scsiType=0, portMap=00, >> sasAddr=4433221100000000,0000000000000000 >> mfi0: 330 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) >> mfi0: 331 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) Info: >> enclPd=ffff, scsiType=0, portMap=02, >> sasAddr=4433221102000000,0000000000000000 >> mfi0: 332 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) >> mfi0: 333 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) Info: >> enclPd=ffff, scsiType=0, portMap=03, >> sasAddr=4433221101000000,0000000000000000 >> mfi0: 334 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) >> mfi0: 335 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) Info: >> enclPd=ffff, scsiType=0, portMap=01, >> sasAddr=4433221103000000,0000000000000000 >> mfi0: 336 (397672376s/0x0020/info) - Time established as 08/07/12 >> 16:32:56; (28 seconds since power on) >> >> The problem: >> >> When trying to create a RaidZ pool using gpart and perform a 4K >> alignment using gnop, we get the follwoing error immediately after >> exporting the pool and destroying the .nop devices: >> >> id: 8043746387654554958 >> state: FAULTED >> status: One or more devices contains corrupted data. >> action: The pool cannot be imported due to damaged devices or data. >> The pool may be active on another system, but can be imported using >> the '-f' flag. >> see: http://illumos.org/msg/ZFS-8000-5E >> config: >> >> Pool FAULTED corrupted data >> raidz1-0 ONLINE >> 13283347160590042564 UNAVAIL corrupted data >> 16981727992215676534 UNAVAIL corrupted data >> 6607570030658834339 UNAVAIL corrupted data >> 3435463242860701988 UNAVAIL corrupted data > > I'm planning on doing something similar, but I'm curious about how gnop > and GPT labels interact. I want to partition the drives for my pool > slightly on the small side so that I'm less likely to run into problems > if I have to replace a drive in the future. If I used gpart to create > and label a GPT partition on the drive, the partition will show up as > /dev/gpt/label. The gnop man page says that running gnop on dev creates > /dev/dev.nop. What happens if you gnop /dev/gpt/label? Is this what > you are doing? > >> When we use glabel for the same purpose with the combination of gnop, >> the pool imports fine. >> >> Any suggestions? > > It should be sufficient to only gnop one of the devices. You should be > able to create the pool with only one gnop device to get the ashift > value that you desire, export the pool, destroy the .nop device, and > import the pool. If it things the device is corrupted (which seems like > a bug of some sort), then ZFS should be able to resilver it. > After a few tests with many scenarios, the pool shows as ASHIFT=12 with the devices gnoped directly. However, the performance is decreasing dramatically after adding a few TB of data. We haven't used more than 30% of its capacity yet on a RaidZ1 pool. Scrub speed has been reduced to 20M max. We will park the controller for now and use the standard HBA supported by the mps driver :) -- George Kontostanos Aicom telecoms ltd http://www.aisecure.net
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BdUSyrdQm7ON1awHZT8Dp0hMZ2TOBoUf9O9Kmmo1TLsgGJreQ>