Date: Wed, 19 Jun 2013 21:44:40 +0200 From: =?iso-8859-1?Q?Dennis_K=F6gel?= <dk@neveragain.de> To: Jeremy Chadwick <jdc@koitsu.org> Cc: freebsd-stable@freebsd.org, Steven Hartland <killing@multiplay.co.uk>, Ronald Klop <ronald-freebsd8@klop.yi.org> Subject: Re: Weird I/O hangs (9.1R, arcsas, interrupt spikes on uhci0) Message-ID: <464554DB-2903-44DE-9EAD-52CD0E3C2823@neveragain.de> In-Reply-To: <20130619151652.GB72566@icarus.home.lan> References: <C2AA9591-CBF4-4956-BABE-08BD8994FF8C@neveragain.de> <op.wyxg11zc8527sy@ronaldradial.versatec.local> <FD9290D8-1A12-4F28-816B-94EFB4516DA4@neveragain.de> <EA2D201C731C46CB8F7BE4972847A53B@multiplay.co.uk> <B199EA9B-6E1C-4B1A-A8F3-4574FF61AEC0@neveragain.de> <15E6A1D4AB1D43D49C1DA02EAF463126@multiplay.co.uk> <27EED7A0-AB0B-43B5-8B7F-B424852DBD65@neveragain.de> <20130619151652.GB72566@icarus.home.lan>
index | next in thread | previous in thread | raw e-mail
Am 19.06.2013 um 17:16 schrieb Jeremy Chadwick <jdc@koitsu.org>: > Which model of the ARC1320 are you using (there are 2). It has four internal connectors, so it should be the ARC-1320ix-16. No port multipliers. >>> Also when you see hangs can you access the disk directly or not >>> e.g. dd if=/dev/da0 of=/dev/null bs=1m count=10 ? >> >> Interesting idea. The dd then hangs right until everything else resumes as well. >> >> ^T during hang says: load: 12.39 cmd: dd 7847 [physrd] 6.36r 0.00u 0.00s 0% 1632k > > Is this ***while** you have immense amounts of ZFS write I/O going to > those drives (your zpool iostat was showing ~250-300MB/sec to the pool)? > [...] It's important to note that the interrupt spikes (and the I/O hangs) happen just as frequently on an idle system. Having a bunch of dd processes writiing + iostat just visualizes it better. So, with or without actual write load: dd with if=/dev/daX (arcsas device) hangs when the interrupt counters for uhci0 soar for these ~10 seconds phases, as shown above. Noteworthy: dd'ing from if=/dev/ada1 (onboard controller) during such a hang phase returns immediately, i.e. works fine. (ada1 is part of ZFS -- the other 'zroot' pool -- but is not an arcsas device, so a driver issue sounds more likely). > Can you please try putting this in /boot/loader.conf + reboot and > see if the behaviour for you changes? > > vfs.zfs.no_write_throttle="1" This produces quite interesting burst numbers, but does not affect the problem behaviour at all. Am 19.06.2013 um 17:10 schrieb Steven Hartland <killing@multiplay.co.uk>: > You might want to try adding a seperate disk (different type) > to the controller which isn't used and perform the same test to > try and eliminate disk's as the source of the issue. That's currently not an option, as the zpool already contains data; but I tried against a disk on another controller, see above. > Also see what "gstat -d" shows during this? Do you see a big spike > of activity either side? The picture is pretty much the same as with zpool iostat: Healthy values, all disks from 70-100% busy; during a hang phase, every column just drops to zero -- except for L(q), which remains frozen at some low value for the duration of the hang (e.g. 4 or 10). Sample outputs here: http://pub.neveragain.de/arcsas/gstat.txt Thanks, D.home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?464554DB-2903-44DE-9EAD-52CD0E3C2823>