Date: Fri, 3 Mar 2017 23:22:09 +0100 From: Dirk-Willem van Gulik <dirkx@webweaving.org> To: "Kenneth D. Merry" <ken@freebsd.org>, freebsd-hackers@freebsd.org Subject: Re: sa and mpt in FreeBSD 11.0-RELEASE-p2 Message-ID: <5DC8B4FC-71BD-4218-9209-1D52A47AA1CB@webweaving.org> In-Reply-To: <89CFE344-6D31-4E72-A79A-3DB638D5D8AC@webweaving.org> References: <C2E846C2-D96B-4429-8DC9-5612EF529F3E@webweaving.org> <20170220205621.GA14597@mithlond.kdm.org> <9A1C3971-0614-4E7C-9A99-C40346D794CB@webweaving.org> <20170221204551.GA32056@mithlond.kdm.org> <89CFE344-6D31-4E72-A79A-3DB638D5D8AC@webweaving.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 21 Feb 2017, at 21:45, Kenneth D. Merry <ken@freebsd.org> wrote: >=20 > On Tue, Feb 21, 2017 at 18:15:06 +0100, Dirk-Willem van Gulik wrote: >>=20 >>> On 20 Feb 2017, at 21:56, Kenneth D. Merry <ken@FreeBSD.ORG> wrote: >>>=20 >>> On Mon, Feb 20, 2017 at 21:04:11 +0100, Dirk-Willem van Gulik wrote: >>>> A machine, recently upgraded from 8.4 to FreeBSD 11.0-RELEASE-p2 = seems to have lost pretty much any and all performance on mpt(4) with = its attached tape drives >>>>=20 >>>> Read performance is around 50Mbyte/second - and write a paltry = 100-200kbyte/second (and occasionally hitting 800kbyte/second) (from/to = memory disk&/dev/null; no risk of shoe shining, compression off). >>>>=20 >>>> SCSI bus looks happy; with no kernel messages. Output of the DLT = script below. >>>>=20 >>>> Normal LTO drives in a tape robot, G9 server; pretty much all SAS = *except* for a U320 to tape drive with the performance issue. Active = terminator. >>>>=20 >>>> Does this ring a bell with anyone? Was anything changed in either = the sa or mpt driver since 8.x ?=20 >>>>=20 >>>> One odd thing is that a 'dd(1)' -without- a block size (compression = is off, data is a prepared urandom file on md disk) writes much faster = -- while on 8.x one -had- to use a sane blockwise to get the 'normal' = speed of around 120Mbyte/second. Could it be that one needs to fiddle = with MAXPHYS (which AFAIK is a readonly sysctl). >>>>=20 >>>=20 >>> What blocksize are you using? What blocksize were you using before? = What >>> application do you normally use to talk to the tape drive? >>=20 >> Amanda (or plain DD); we increased Amanda???s default of 32 to 64k = (which stopped any and all shoe shining at that time. It was tested in = 2013 (so likely in FreeBSD 8) up to 16 Mbyte in factors of 2 according = to our notes - but with no further improvements in performance - and = left at 64k). >>=20 > This tells me that Amanda is keeping the tape driver well supplied = with > I/O. Since 64K was probably the blocksize going down to the hardware > previously, it makes sense that increasing it in Amanda didn't make a > difference. Ok. clear. Thanks. ! >=20 >> The actual drive (mt status -v) reports (prior to increasing MAXPHYS) = on 11.0-p2 with default kernel config: >>=20 >>> Drive: sa0: <QUANTUM ULTRIUM 4 W52T> Serial Number: HU1027B53Y >>> --------------------------------- >>> Mode Density Blocksize bpi Compression >>> Current: 0x46:LTO-4 variable 323215 enabled (0x1) >>> --------------------------------- >>> Current Driver State: at rest. >>> --------------------------------- >>> Partition: 0 Calc File Number: 0 Calc Record Number: 0 >>> Residual: 0 Reported File Number: 0 Reported Record Number: 0 >>> Flags: BOP >>> --------------------------------- >>> Tape I/O parameters: >>> Maximum I/O size allowed by driver and controller (maxio): 131072 = bytes >>> Maximum I/O size reported by controller (cpi_maxio): 131072 bytes >>> Maximum block size supported by tape drive and media (max_blk): = 16777215 bytes >>> Minimum block size supported by tape drive and media (min_blk): 1 = bytes >>> Block granularity supported by tape drive and media (blk_gran): 0 = bytes >>> Maximum possible I/O size (max_effective_iosize): 131072 bytes >>=20 >> ???. >>> If you want to re-enable I/O splitting (which I should have disabled = by >>> now), you can set the following loader tunable in /boot/loader.conf: >>>=20 >>> kern.cam.sa.allow_io_split=3D1 >>=20 >> Ok - this changes behaviour - non-blocksized limited writes are now = up; 2Mbyte/second; any block-sized write is 200kb/second (regardless of = bs). >>=20 >>> If so, you probably want to look at the blocksize you're using, and = bump up >>> MAXPHYS in your kernel configuration to a larger value. >>=20 >> Post MAXPHYS & DFLTPHYS increase to 4MB (4x1024x1024) (as 16MB caused = the kernel to hang during boot just post usb/knd - likely as it is there = that it starts on SAS disk & tape discovery) I get >> as ???mt status -v??? the output: >>=20 >>> Tape I/O parameters: >>> Maximum I/O size allowed by driver and controller (maxio): 1372160 = bytes >>> Maximum I/O size reported by controller (cpi_maxio): 1372160 bytes >>> Maximum block size supported by tape drive and media (max_blk): = 16777215 bytes >>> Minimum block size supported by tape drive and media (min_blk): 1 = bytes >>> Block granularity supported by tape drive and media (blk_gran): 0 = bytes >>> Maximum possible I/O size (max_effective_iosize): 1372160 bytes >>=20 >>=20 >> And see the same speeds; non blocksize limited (with dd) are = 2.7Mbyte/second (so slightly higher); the blocksized writes are better = (but still) low; 200kbyte/second for 16k, 600k for 32k, 800k for 64k, = 500k for 128k, 1M at 256k, 1M at 1024k. The writes at 2048k and 4096k = fail with a ???hardware failure asc:44,0 ??? tape is now frozen??? = error). >>=20 >> This is with `kern.cam.sa.allow_io_split=3D1??? in = /boot/loader.conf???. Doing so without that (after a reboot) give = substantially similar results: (1.8Mbyte/second (0), 450k (16k), 560k = (32k), 0.9M (64), 560k (128k), 730k (256k) and 600k (1M) ??? it fairly = errors out with a si_iosize_max=3D1372160 for sizes above that. >>=20 >=20 > The fact that you're getting substantially higher performance with I/O > splitting enabled points to probable delays in getting the data into = the > kernel. >=20 > Instead of doing a single threaded read/write dd test, I would suggest > doing something like this: >=20 > dd if=3Dinputfile bs=3D10m | dd of=3D/dev/nsa0 bs=3D1m This does not change the numbers at all (with a PHYSMEM=3D4M, = kern.cam.sa.0.*maxio reporting 1372160 and with or without splitting). We've also tried a different SCSI card (albeit also an LSI one - so also = mpt(4)); which gives near idential numbers - around 600k.=20 > Also, the fact that you're getting a hardware failure with larger = writes > may mean that the tape drive might not support what it claims to. The > hardware failure is probably coming from the tape drive. Do we've downgraded the machine back to FreeBSD 8.4-- and tested the = same setup against the same drive. That gave us normal LTO4 speeds. We = tried two mpt cards and one adaptec card. I guess that suggest it is not a drive, card or cable issue.=20 However we are seeing a difference in the output of the pciconf command = with respect to the busses and order of the cards detected; and see an = adaptec card work fine in both 8.4 and 11.0-p7 (and the adaptec cards = are reported identical ordered). So this may well be sa(4) unrelated - but PCI or LSI card specific. But thanks to your help - we do have a working solution (and able to rid = the last machines off LTO3 and LTO4). Thanks! Dw
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5DC8B4FC-71BD-4218-9209-1D52A47AA1CB>