From owner-freebsd-hackers@freebsd.org Mon Feb 20 20:56:26 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7526DCE6FF7 for ; Mon, 20 Feb 2017 20:56:26 +0000 (UTC) (envelope-from ken@kdm.org) Received: from mithlond.kdm.org (mithlond.kdm.org [96.89.93.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "A1-33714", Issuer "A1-33714" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4945EBB8 for ; Mon, 20 Feb 2017 20:56:25 +0000 (UTC) (envelope-from ken@kdm.org) Received: from mithlond.kdm.org (localhost [127.0.0.1]) by mithlond.kdm.org (8.15.2/8.14.9) with ESMTPS id v1KKuMFe014865 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 20 Feb 2017 15:56:22 -0500 (EST) (envelope-from ken@mithlond.kdm.org) Received: (from ken@localhost) by mithlond.kdm.org (8.15.2/8.14.9/Submit) id v1KKuMtX014864; Mon, 20 Feb 2017 15:56:22 -0500 (EST) (envelope-from ken) Date: Mon, 20 Feb 2017 15:56:22 -0500 From: "Kenneth D. Merry" To: Dirk-Willem van Gulik Cc: freebsd-hackers@freebsd.org Subject: Re: sa and mpt in FreeBSD 11.0-RELEASE-p2 Message-ID: <20170220205621.GA14597@mithlond.kdm.org> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mithlond.kdm.org [127.0.0.1]); Mon, 20 Feb 2017 15:56:22 -0500 (EST) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mithlond.kdm.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Feb 2017 20:56:26 -0000 On Mon, Feb 20, 2017 at 21:04:11 +0100, Dirk-Willem van Gulik wrote: > A machine, recently upgraded from 8.4 to FreeBSD 11.0-RELEASE-p2 seems to have lost pretty much any and all performance on mpt(4) with its attached tape drives > > Read performance is around 50Mbyte/second - and write a paltry 100-200kbyte/second (and occasionally hitting 800kbyte/second) (from/to memory disk&/dev/null; no risk of shoe shining, compression off). > > SCSI bus looks happy; with no kernel messages. Output of the DLT script below. > > Normal LTO drives in a tape robot, G9 server; pretty much all SAS *except* for a U320 to tape drive with the performance issue. Active terminator. > > Does this ring a bell with anyone? Was anything changed in either the sa or mpt driver since 8.x ? > > One odd thing is that a 'dd(1)' -without- a block size (compression is off, data is a prepared urandom file on md disk) writes much faster -- while on 8.x one -had- to use a sane blockwise to get the 'normal' speed of around 120Mbyte/second. Could it be that one needs to fiddle with MAXPHYS (which AFAIK is a readonly sysctl). > What blocksize are you using? What blocksize were you using before? What application do you normally use to talk to the tape drive? One thing that did change is that I disabled I/O splitting by default in the sa(4) driver. That means that previously, you could write a 10MB block to the sa(4) character device, and it would get split up by physio(9) into a bunch of MAXPHYS-size chunks. The problem with that behavior is that you can get an error back, and not have a good indication of which block caused the problem, and how many blocks made it out to tape if a write fails. You can also have applications that think they're writing large blocks, but in reality aren't actually doing that. The application doesn't have control of what size block is going to tape. Now, you can't write more than the lowest common denominator supported by MAXPHYS, the controller and the tape drive. This lets the application see on a block by block basis what succeeded and what failed. It also gives the application control of the blocksize, up to the limit supported by the configuration. You can see what is possible with your current configuration by doing a 'mt status -v'. For example: # mt status -v Drive: sa0: Serial Number: 101300520A --------------------------------- Mode Density Blocksize bpi Compression Current: 0x58:LTO-5 variable 384607 enabled (0x1) --------------------------------- Current Driver State: at rest. --------------------------------- Partition: 0 Calc File Number: 0 Calc Record Number: 0 Residual: 0 Reported File Number: 0 Reported Record Number: 0 Flags: BOP --------------------------------- Tape I/O parameters: Maximum I/O size allowed by driver and controller (maxio): 1081344 bytes Maximum I/O size reported by controller (cpi_maxio): 5197824 bytes Maximum block size supported by tape drive and media (max_blk): 8388608 bytes Minimum block size supported by tape drive and media (min_blk): 1 bytes Block granularity supported by tape drive and media (blk_gran): 0 bytes Maximum possible I/O size (max_effective_iosize): 1081344 bytes This particular machine has an LTO-5 attached via a Qlogic FC controller, and MAXPHYS is set to 1024*1056 (just over 1MB) in the kernel config file. If you want to re-enable I/O splitting (which I should have disabled by now), you can set the following loader tunable in /boot/loader.conf: kern.cam.sa.allow_io_split=1 Then reboot and re-run your test and see if the performance is any different. If so, you probably want to look at the blocksize you're using, and bump up MAXPHYS in your kernel configuration to a larger value. > Dw. > > DLT / http://www.freebsddiary.org/tape-testing.php > READING > Corrected errors with substantial delay: 0 > Corrected errors with possible delay : 0 > Total errors : 0 > Total errors corrected : 0 > Total times correction algorithm used : 0 > Total bytes processed : 8590352388 > Total corrected errors / GB : 0 > Total uncorrected errors : 0 > Read compression ratio : 101% > On tape Mbytes read : 2 > On tape kbytes read residual : 329320 > WRITING > Corrected errors with substantial delay: 154 > Corrected errors with possible delay : 0 > Total errors : 0 > Total errors corrected : 0 > Total times correction algorithm used : 168 > Total bytes processed : 4909148037124 > Total corrected errors / GB : 0 > Total uncorrected errors : 0 > Write compression ratio : 99% > Host requested Mbytes written : 1442 > Host requested kbytes written residual : 115996 > On tape Mbytes written : 1443 > On tape kbytes written residual : 0 > > camcontrol devlist > > at scbus3 target 5 lun 0 (sa0,pass0) > at scbus3 target 5 lun 1 (ch0,pass1) > at scbus4 target 0 lun 0 (pass2,da0) > at scbus6 target 0 lun 0 (pass3,da1) > at scbus6 target 1 lun 0 (pass4,da2) > at scbus7 target 5 lun 0 (sa1,pass5) > at scbus7 target 5 lun 1 (ch1,pass6) > > mt rblim > /dev/nsa0: > min blocksize 1 byte > max blocksize 16777215 bytes > granularity 1 byte > > mt status, ostatus and errstatus > > Mode Density Blocksize bpi Compression > Current: 0x46:LTO-4 variable 323215 disabled > ---------available modes--------- > 0: 0x46:LTO-4 variable 323215 0x1 > 1: 0x46:LTO-4 variable 323215 0x1 > 2: 0x46:LTO-4 variable 323215 0x1 > 3: 0x46:LTO-4 variable 323215 0x1 > --------------------------------- > Current Driver State: at rest. > --------------------------------- > File Number: 0 Record Number: 0 Residual Count 0 > > > Drive: sa0: Serial Number: HU1027B53Y > --------------------------------- > Mode Density Blocksize bpi Compression > Current: 0x46:LTO-4 variable 323215 disabled > --------------------------------- > Current Driver State: at rest. > --------------------------------- > Partition: 0 Calc File Number: 0 Calc Record Number: 0 > Residual: 0 Reported File Number: 0 Reported Record Number: 0 > Flags: BOP What does 'mt status -v' show? That will show you the maximum blocksize that you can effectively send to the device. Ken -- Kenneth Merry ken@FreeBSD.ORG