Date: Fri, 11 Jun 2004 11:47:57 +0200 From: "Frantisek Rysanek" <Frantisek.Rysanek@post.cz> To: freebsd-scsi@freebsd.org Subject: how to limit SCSI TCQ depth in FreeBSD? Message-ID: <40C99BED.12482.159E333F@localhost>
next in thread | raw e-mail | index | archive | help
Dear SCSI experts and FreeBSD veterans, I'm looking for advice with an apparently simple SCSI issue: how do I set a maximum TCQ depth in FreeBSD? Scenario: - an U320 SCSI disk drive - Adaptec 39320 (aic7902) == U320 & PCI-X - FreeBSD 5.2.1 (and Linux 2.4.26, for comparison) - Bonnie (-s 1500, 4x in parallel, to trash the RAM buffers) - the system is a dual Xeon @ 2.6 GHz, 1 GB DDR266, ServerWorks GC-LE (PCI-X) I have encountered a SCSI disk device that tends to hang under load in FreeBSD - namely I have discovered that using Bonnie. The disk hangs when Bonnie says "rewriting" and in iostat you can see the TPS skyrocket. In Linux, with the default compile-time parameters to aic79xx.o, the disk works without a problem, no matter what Bonnie does to it. I figured out that it could be a problem with TCQ depth. The default in Linux/aic79xx is 32, so I cranked it up to 250, recompiled and ran Bonnie. Guess what - the disk hung at just the same point as in FreeBSD, only the TPS in linux managed to crawl yet somewhat higher. The hang invokes a flurry of error messages on the host system, both in Linux and FreeBSD - the precise wording is different, but essentially the drivers say that the SCSI bus is stuck and they keep trying to reset it, to no avail. The culprit device remains stuck until it's power-cycled. The datasheets for the disk device say that it can cope with 256 commands in the TCQ. Yet my experiments show that the highest safe value is about 224. 240 already makes the device crash. I tried to find the breakdown point by interval halving. An increase from 32 to 128 on the host side results in a significant performance improvement (about +50 %). A further increase to 192 or 224 is not much of an advantage. At 240, the disk chokes. To sum up, I'm fairly convinced that the disk can run just fine at a lower TCQ depth. The trouble is, that I can't find a way to set the TCQ depth in FreeBSD. Although the directory structure seems similar, the FreeBSD drivers for aic7xxx/aic79xx are significantly different from the Linux variety. The preprocessor macros and structs that handle TCQ settings in Linux are clearly missing in FreeBSD. I understand that FreeBSD is using a common SCSI and CAM layer, so perhaps I'd better be looking somewhere in there - only I can't find a hint of what to do. I tried looking into /usr/src/sys/conf/options /usr/src/sys/conf/NOTES /usr/src/sys/dev/aic7xxx/* etc. to no avail. I'm not too much of a coder, I get lost quickly in the CAM and SCSI layers. Any ideas are welcome. Thanks for your time reading this message :-) Frank Rysanek
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?40C99BED.12482.159E333F>