Date: Fri, 2 Nov 2012 10:32:56 -0000 From: "Steven Hartland" <killing@multiplay.co.uk> To: <freebsd-fs@freebsd.org>, <freebsd-stable@freebsd.org>, <freebsd-scsi@freebsd.org> Subject: Re: mfi corrupts JBOD disks >2TB due to LBA overflow (was: ZFS corruption due to lack of space?) Message-ID: <1A4A7E2C4EDD4BAC8F21275BA647782A@multiplay.co.uk> References: <27087376D1C14132A3CC1B4016912F6D@multiplay.co.uk> <20121031212346.GL3309@server.rulingia.com> <F569C535F56843F3B16691C206AD6492@multiplay.co.uk> <A5515D2F4E974E619E24892D660E4DFE@multiplay.co.uk> <9DB937FEA7634C4BAC49EF5823F93CA3@multiplay.co.uk> <B5D0827141BD4B228F1F68318B67DC97@multiplay.co.uk> <5846347C20554E549FA512C1D59F6427@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Copying in freebsd-scsi@ for visability. ----- Original Message ----- From: "Steven Hartland" > Ok after revisiting all the facts and spotting that > the corruption only seemed to happen after my zpool > was nearly full I came up with a wild idea, could > the corruption be being caused by writes after 2TB? > > A few command lines latter and this was confirmed > writes to the 3TB disks under mfi are wrapping at > 2TB!!! > > Steps to prove:- > 1. zero out block 1 on the disk > dd if=/dev/zero bs=512 count=1 of=/dev/mfisyspd0 > 1+0 records in > 1+0 records out > 512 bytes transferred in 0.000728 secs (703171 bytes/sec) > > 2. confirm the first block is zeros > dd if=/dev/mfisyspd0 bs=512 count=1 | hexdump -C > 1+0 records in > 1+0 records out > 512 bytes transferred in 0.000250 secs (2047172 bytes/sec) > 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| > * > 00000200 > > 3. write 1 block random after the 2TB boundary > dd if=/dev/random bs=512 count=1 of=/dev/mfisyspd0 oseek=4294967296 > 1+0 records in > 1+0 records out > 512 bytes transferred in 0.000717 secs (714162 bytes/sec) > > 4. first block of the disk now contains random data > dd if=/dev/mfisyspd0 bs=512 count=8 | hexdump -C > 00000000 9c d1 d2 1d 9f 2c fc 30 ab 09 7a f7 64 16 2a 58 |.....,.0..z.d.*X| > 00000010 18 27 9d 1f ae 4d 27 53 1a 50 e7 c1 b1 3a 9b e4 |.'...M'S.P...:..| > 00000020 c3 7c d0 25 83 e2 bd 85 33 f2 33 8e 71 55 70 7c |.|.%....3.3.qUp|| > 00000030 8c 15 af 55 f6 88 8d 6e 40 1c f3 1a 5c e7 80 4b |...U...n@...\..K| > ... > > Looking at the driver code the problem is that IO on syspd > disks aka JBOD is always done using 10 byte CDB commands > in mfi_build_syspdio. This is clearly a serious problem as > it results in total corruption on disks > 2^32 sectors > when sectors above 2^32 are accessed. > > The fix doesn't seem too hard and I think I've already > got a basic version working, just needs more testing need. > > The bug also effects kernel mfi_dump_blocks but thats > less likely to trigger due to how its used. > > Will create PR when I've finished testing and am happy > with the patch, but wanted to let others know in the > mean time given how serious the bug is. PR which includes a patch which fixes this issue is:- http://www.freebsd.org/cgi/query-pr.cgi?pr=173291 Given its critical nature I would strongly advise this gets MFC'ed to all branches ASAP. While someone is looking at this would be good to get the following mfi related PR's I've submitted could also be committed as well ;-) Add deviceid to mfi disk startup output http://www.freebsd.org/cgi/query-pr.cgi?pr=173290 Improvements to mfi support including foreign disks / configs in mfiutil http://www.freebsd.org/cgi/query-pr.cgi?pr=172091 Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1A4A7E2C4EDD4BAC8F21275BA647782A>