Date: Tue, 21 Feb 2017 15:45:52 -0500 From: "Kenneth D. Merry" <ken@FreeBSD.ORG> To: Dirk-Willem van Gulik <dirkx@webweaving.org> Cc: freebsd-hackers@freebsd.org Subject: Re: sa and mpt in FreeBSD 11.0-RELEASE-p2 Message-ID: <20170221204551.GA32056@mithlond.kdm.org> In-Reply-To: <9A1C3971-0614-4E7C-9A99-C40346D794CB@webweaving.org> References: <C2E846C2-D96B-4429-8DC9-5612EF529F3E@webweaving.org> <20170220205621.GA14597@mithlond.kdm.org> <9A1C3971-0614-4E7C-9A99-C40346D794CB@webweaving.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Feb 21, 2017 at 18:15:06 +0100, Dirk-Willem van Gulik wrote:
>
> > On 20 Feb 2017, at 21:56, Kenneth D. Merry <ken@FreeBSD.ORG> wrote:
> >
> > On Mon, Feb 20, 2017 at 21:04:11 +0100, Dirk-Willem van Gulik wrote:
> >> A machine, recently upgraded from 8.4 to FreeBSD 11.0-RELEASE-p2 seems to have lost pretty much any and all performance on mpt(4) with its attached tape drives
> >>
> >> Read performance is around 50Mbyte/second - and write a paltry 100-200kbyte/second (and occasionally hitting 800kbyte/second) (from/to memory disk&/dev/null; no risk of shoe shining, compression off).
> >>
> >> SCSI bus looks happy; with no kernel messages. Output of the DLT script below.
> >>
> >> Normal LTO drives in a tape robot, G9 server; pretty much all SAS *except* for a U320 to tape drive with the performance issue. Active terminator.
> >>
> >> Does this ring a bell with anyone? Was anything changed in either the sa or mpt driver since 8.x ?
> >>
> >> One odd thing is that a 'dd(1)' -without- a block size (compression is off, data is a prepared urandom file on md disk) writes much faster -- while on 8.x one -had- to use a sane blockwise to get the 'normal' speed of around 120Mbyte/second. Could it be that one needs to fiddle with MAXPHYS (which AFAIK is a readonly sysctl).
> >>
> >
> > What blocksize are you using? What blocksize were you using before? What
> > application do you normally use to talk to the tape drive?
>
> Amanda (or plain DD); we increased Amanda???s default of 32 to 64k (which stopped any and all shoe shining at that time. It was tested in 2013 (so likely in FreeBSD 8) up to 16 Mbyte in factors of 2 according to our notes - but with no further improvements in performance - and left at 64k).
>
This tells me that Amanda is keeping the tape driver well supplied with
I/O. Since 64K was probably the blocksize going down to the hardware
previously, it makes sense that increasing it in Amanda didn't make a
difference.
> The actual drive (mt status -v) reports (prior to increasing MAXPHYS) on 11.0-p2 with default kernel config:
>
> > Drive: sa0: <QUANTUM ULTRIUM 4 W52T> Serial Number: HU1027B53Y
> > ---------------------------------
> > Mode Density Blocksize bpi Compression
> > Current: 0x46:LTO-4 variable 323215 enabled (0x1)
> > ---------------------------------
> > Current Driver State: at rest.
> > ---------------------------------
> > Partition: 0 Calc File Number: 0 Calc Record Number: 0
> > Residual: 0 Reported File Number: 0 Reported Record Number: 0
> > Flags: BOP
> > ---------------------------------
> > Tape I/O parameters:
> > Maximum I/O size allowed by driver and controller (maxio): 131072 bytes
> > Maximum I/O size reported by controller (cpi_maxio): 131072 bytes
> > Maximum block size supported by tape drive and media (max_blk): 16777215 bytes
> > Minimum block size supported by tape drive and media (min_blk): 1 bytes
> > Block granularity supported by tape drive and media (blk_gran): 0 bytes
> > Maximum possible I/O size (max_effective_iosize): 131072 bytes
>
> ???.
> > If you want to re-enable I/O splitting (which I should have disabled by
> > now), you can set the following loader tunable in /boot/loader.conf:
> >
> > kern.cam.sa.allow_io_split=1
>
> Ok - this changes behaviour - non-blocksized limited writes are now up; 2Mbyte/second; any block-sized write is 200kb/second (regardless of bs).
>
> > If so, you probably want to look at the blocksize you're using, and bump up
> > MAXPHYS in your kernel configuration to a larger value.
>
> Post MAXPHYS & DFLTPHYS increase to 4MB (4x1024x1024) (as 16MB caused the kernel to hang during boot just post usb/knd - likely as it is there that it starts on SAS disk & tape discovery) I get
> as ???mt status -v??? the output:
>
> > Tape I/O parameters:
> > Maximum I/O size allowed by driver and controller (maxio): 1372160 bytes
> > Maximum I/O size reported by controller (cpi_maxio): 1372160 bytes
> > Maximum block size supported by tape drive and media (max_blk): 16777215 bytes
> > Minimum block size supported by tape drive and media (min_blk): 1 bytes
> > Block granularity supported by tape drive and media (blk_gran): 0 bytes
> > Maximum possible I/O size (max_effective_iosize): 1372160 bytes
>
>
> And see the same speeds; non blocksize limited (with dd) are 2.7Mbyte/second (so slightly higher); the blocksized writes are better (but still) low; 200kbyte/second for 16k, 600k for 32k, 800k for 64k, 500k for 128k, 1M at 256k, 1M at 1024k. The writes at 2048k and 4096k fail with a ???hardware failure asc:44,0 ??? tape is now frozen??? error).
>
> This is with `kern.cam.sa.allow_io_split=1??? in /boot/loader.conf???. Doing so without that (after a reboot) give substantially similar results: (1.8Mbyte/second (0), 450k (16k), 560k (32k), 0.9M (64), 560k (128k), 730k (256k) and 600k (1M) ??? it fairly errors out with a si_iosize_max=1372160 for sizes above that.
>
What do you mean by 0 for a blocksize? Do you mean that you didn't specify
a blocksize? If not, it appears that dd defaults to 512 bytes, although it
could be different in your case.
If the kernel is well supplied with data, there should be very little
difference between the I/O splitting case and the non-I/O splitting case.
The fact that you're getting substantially higher performance with I/O
splitting enabled points to probable delays in getting the data into the
kernel.
Instead of doing a single threaded read/write dd test, I would suggest
doing something like this:
dd if=inputfile bs=10m | dd of=/dev/nsa0 bs=1m
That will insure that the dd process writing to the tape has plenty of data
to write. Even though the test script below uses a ramdisk for the source
file, it looks like there is probably a delay between writes, which might
account for the lower performance.
Also, the fact that you're getting a hardware failure with larger writes
may mean that the tape drive might not support what it claims to. The
hardware failure is probably coming from the tape drive.
> Dw
>
> #!/bin/sh
> set -x
>
> # create a ram disk of about 1 Gbyte with a noncompressible file on it.
> #
> mdconfig -a -s 1200M
> newfs md0
> mount /dev/md0 /mnt
> dd if=/dev/urandom bs=1m count=1024 of=/mnt/test-1024m
>
> for i in "" bs=16k bs=32k bs=64k bs=128k bs=256k bs=1024k bs=2048k bs=4196k
> do
> mt rewind
> echo Run started with ${i:-no blocksize set}
> dd if=/mnt/test-1024m $i of=/dev/nsa0
> done
Ken
--
Kenneth Merry
ken@FreeBSD.ORG
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20170221204551.GA32056>