From owner-freebsd-hackers@freebsd.org Tue Feb 21 17:15:56 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C49B9CE8B05 for ; Tue, 21 Feb 2017 17:15:56 +0000 (UTC) (envelope-from dirkx@webweaving.org) Received: from weser.webweaving.org (weser.webweaving.org [148.251.234.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.webweaving.org", Issuer "RapidSSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5C879E2D for ; Tue, 21 Feb 2017 17:15:56 +0000 (UTC) (envelope-from dirkx@webweaving.org) Received: from beeb.leiden.webweaving.org (5ED06D14.cm-7-1b.dynamic.ziggo.nl [94.208.109.20]) (authenticated bits=0) by weser.webweaving.org (8.15.2/8.15.2) with ESMTPSA id v1LHF7Ca006030 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 21 Feb 2017 18:15:07 +0100 (CET) (envelope-from dirkx@webweaving.org) X-Authentication-Warning: weser.webweaving.org: Host 5ED06D14.cm-7-1b.dynamic.ziggo.nl [94.208.109.20] claimed to be beeb.leiden.webweaving.org Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3265\)) Subject: Re: sa and mpt in FreeBSD 11.0-RELEASE-p2 From: Dirk-Willem van Gulik In-Reply-To: <20170220205621.GA14597@mithlond.kdm.org> Date: Tue, 21 Feb 2017 18:15:06 +0100 Cc: freebsd-hackers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <9A1C3971-0614-4E7C-9A99-C40346D794CB@webweaving.org> References: <20170220205621.GA14597@mithlond.kdm.org> To: "Kenneth D. Merry" X-Mailer: Apple Mail (2.3265) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (weser.webweaving.org [148.251.234.232]); Tue, 21 Feb 2017 18:15:07 +0100 (CET) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Feb 2017 17:15:56 -0000 > On 20 Feb 2017, at 21:56, Kenneth D. Merry wrote: >=20 > On Mon, Feb 20, 2017 at 21:04:11 +0100, Dirk-Willem van Gulik wrote: >> A machine, recently upgraded from 8.4 to FreeBSD 11.0-RELEASE-p2 = seems to have lost pretty much any and all performance on mpt(4) with = its attached tape drives >>=20 >> Read performance is around 50Mbyte/second - and write a paltry = 100-200kbyte/second (and occasionally hitting 800kbyte/second) (from/to = memory disk&/dev/null; no risk of shoe shining, compression off). >>=20 >> SCSI bus looks happy; with no kernel messages. Output of the DLT = script below. >>=20 >> Normal LTO drives in a tape robot, G9 server; pretty much all SAS = *except* for a U320 to tape drive with the performance issue. Active = terminator. >>=20 >> Does this ring a bell with anyone? Was anything changed in either the = sa or mpt driver since 8.x ?=20 >>=20 >> One odd thing is that a 'dd(1)' -without- a block size (compression = is off, data is a prepared urandom file on md disk) writes much faster = -- while on 8.x one -had- to use a sane blockwise to get the 'normal' = speed of around 120Mbyte/second. Could it be that one needs to fiddle = with MAXPHYS (which AFAIK is a readonly sysctl). >>=20 >=20 > What blocksize are you using? What blocksize were you using before? = What > application do you normally use to talk to the tape drive? Amanda (or plain DD); we increased Amanda=E2=80=99s default of 32 to 64k = (which stopped any and all shoe shining at that time. It was tested in = 2013 (so likely in FreeBSD 8) up to 16 Mbyte in factors of 2 according = to our notes - but with no further improvements in performance - and = left at 64k). The actual drive (mt status -v) reports (prior to increasing MAXPHYS) on = 11.0-p2 with default kernel config: > Drive: sa0: Serial Number: HU1027B53Y > --------------------------------- > Mode Density Blocksize bpi Compression > Current: 0x46:LTO-4 variable 323215 enabled (0x1) > --------------------------------- > Current Driver State: at rest. > --------------------------------- > Partition: 0 Calc File Number: 0 Calc Record Number: 0 > Residual: 0 Reported File Number: 0 Reported Record Number: 0 > Flags: BOP > --------------------------------- > Tape I/O parameters: > Maximum I/O size allowed by driver and controller (maxio): 131072 = bytes > Maximum I/O size reported by controller (cpi_maxio): 131072 bytes > Maximum block size supported by tape drive and media (max_blk): = 16777215 bytes > Minimum block size supported by tape drive and media (min_blk): 1 = bytes > Block granularity supported by tape drive and media (blk_gran): 0 = bytes > Maximum possible I/O size (max_effective_iosize): 131072 bytes =E2=80=A6. > If you want to re-enable I/O splitting (which I should have disabled = by > now), you can set the following loader tunable in /boot/loader.conf: >=20 > kern.cam.sa.allow_io_split=3D1 Ok - this changes behaviour - non-blocksized limited writes are now up; = 2Mbyte/second; any block-sized write is 200kb/second (regardless of bs). > If so, you probably want to look at the blocksize you're using, and = bump up > MAXPHYS in your kernel configuration to a larger value. Post MAXPHYS & DFLTPHYS increase to 4MB (4x1024x1024) (as 16MB caused = the kernel to hang during boot just post usb/knd - likely as it is there = that it starts on SAS disk & tape discovery) I get as =E2=80=98mt status -v=E2=80=99 the output: > Tape I/O parameters: > Maximum I/O size allowed by driver and controller (maxio): 1372160 = bytes > Maximum I/O size reported by controller (cpi_maxio): 1372160 bytes > Maximum block size supported by tape drive and media (max_blk): = 16777215 bytes > Minimum block size supported by tape drive and media (min_blk): 1 = bytes > Block granularity supported by tape drive and media (blk_gran): 0 = bytes > Maximum possible I/O size (max_effective_iosize): 1372160 bytes And see the same speeds; non blocksize limited (with dd) are = 2.7Mbyte/second (so slightly higher); the blocksized writes are better = (but still) low; 200kbyte/second for 16k, 600k for 32k, 800k for 64k, = 500k for 128k, 1M at 256k, 1M at 1024k. The writes at 2048k and 4096k = fail with a =E2=80=98hardware failure asc:44,0 =E2=80=94 tape is now = frozen=E2=80=9D error). This is with `kern.cam.sa.allow_io_split=3D1=E2=80=99 in = /boot/loader.conf=E2=80=99. Doing so without that (after a reboot) give = substantially similar results: (1.8Mbyte/second (0), 450k (16k), 560k = (32k), 0.9M (64), 560k (128k), 730k (256k) and 600k (1M) =E2=80=94 it = fairly errors out with a si_iosize_max=3D1372160 for sizes above that. Dw #!/bin/sh set -x # create a ram disk of about 1 Gbyte with a noncompressible file on it. # mdconfig -a -s 1200M newfs md0 mount /dev/md0 /mnt dd if=3D/dev/urandom bs=3D1m count=3D1024 of=3D/mnt/test-1024m for i in "" bs=3D16k bs=3D32k bs=3D64k bs=3D128k bs=3D256k bs=3D1024k = bs=3D2048k bs=3D4196k do mt rewind echo Run started with ${i:-no blocksize set} dd if=3D/mnt/test-1024m $i of=3D/dev/nsa0 done