Date: Thu, 1 Nov 2012 23:36:13 -0000 From: "Steven Hartland" <killing@multiplay.co.uk> To: <freebsd-fs@freebsd.org>, <freebsd-stable@freebsd.org>, "Peter Jeremy" <peter@rulingia.com> Subject: Re: mfi corrupts JBOD disks >2TB due to LBA overflow (was: ZFS corruption due to lack of space?) Message-ID: <5846347C20554E549FA512C1D59F6427@multiplay.co.uk> References: <27087376D1C14132A3CC1B4016912F6D@multiplay.co.uk> <20121031212346.GL3309@server.rulingia.com> <F569C535F56843F3B16691C206AD6492@multiplay.co.uk> <A5515D2F4E974E619E24892D660E4DFE@multiplay.co.uk> <9DB937FEA7634C4BAC49EF5823F93CA3@multiplay.co.uk> <B5D0827141BD4B228F1F68318B67DC97@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Ok after revisiting all the facts and spotting that
the corruption only seemed to happen after my zpool
was nearly full I came up with a wild idea, could
the corruption be being caused by writes after 2TB?
A few command lines latter and this was confirmed
writes to the 3TB disks under mfi are wrapping at
2TB!!!
Steps to prove:-
1. zero out block 1 on the disk
dd if=/dev/zero bs=512 count=1 of=/dev/mfisyspd0
1+0 records in
1+0 records out
512 bytes transferred in 0.000728 secs (703171 bytes/sec)
2. confirm the first block is zeros
dd if=/dev/mfisyspd0 bs=512 count=1 | hexdump -C
1+0 records in
1+0 records out
512 bytes transferred in 0.000250 secs (2047172 bytes/sec)
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000200
3. write 1 block random after the 2TB boundary
dd if=/dev/random bs=512 count=1 of=/dev/mfisyspd0 oseek=4294967296
1+0 records in
1+0 records out
512 bytes transferred in 0.000717 secs (714162 bytes/sec)
4. first block of the disk now contains random data
dd if=/dev/mfisyspd0 bs=512 count=8 | hexdump -C
00000000 9c d1 d2 1d 9f 2c fc 30 ab 09 7a f7 64 16 2a 58 |.....,.0..z.d.*X|
00000010 18 27 9d 1f ae 4d 27 53 1a 50 e7 c1 b1 3a 9b e4 |.'...M'S.P...:..|
00000020 c3 7c d0 25 83 e2 bd 85 33 f2 33 8e 71 55 70 7c |.|.%....3.3.qUp||
00000030 8c 15 af 55 f6 88 8d 6e 40 1c f3 1a 5c e7 80 4b |...U...n@...\..K|
...
Looking at the driver code the problem is that IO on syspd
disks aka JBOD is always done using 10 byte CDB commands
in mfi_build_syspdio. This is clearly a serious problem as
it results in total corruption on disks > 2^32 sectors
when sectors above 2^32 are accessed.
The fix doesn't seem too hard and I think I've already
got a basic version working, just needs more testing need.
The bug also effects kernel mfi_dump_blocks but thats
less likely to trigger due to how its used.
Will create PR when I've finished testing and am happy
with the patch, but wanted to let others know in the
mean time given how serious the bug is.
Regards
Steve
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5846347C20554E549FA512C1D59F6427>