Date: Tue, 29 May 2012 12:34:19 -0700 From: Freddie Cash <fjwcash@gmail.com> To: Kees Jan Koster <kjkoster@gmail.com> Cc: freebsd-stable@freebsd.org Subject: Re: FreeBSD 9.0 hangs on heavy I/O Message-ID: <CAOjFWZ40LX%2B8Lw15mHDG8F3nN0aex5EpqVdjPxRPS89t1Fqkiw@mail.gmail.com> In-Reply-To: <BD5D6BB6-8CFF-456A-B03E-05454EB03AB6@gmail.com> References: <BD5D6BB6-8CFF-456A-B03E-05454EB03AB6@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, May 29, 2012 at 12:26 PM, Kees Jan Koster <kjkoster@gmail.com> wrot= e: > I seem to have a problem where really heavy disk I/O is drowning my machi= ne. I see hangs in the shell where I am logged on using ssh. Network connec= tions get dropped for no apparent reason and some HTTP requests are served = really slowly. Profiling the app code shows that the hangs are in completel= y random places. Operations that are no more than a few lines of code apart= suddenly take seconds to complete. > > In my search I seem to find that my machine is quite slow on the disk. I = find that rather odd, given that the device in question is an SSD drive and= it is a good bit faster than the WD drive that used to carry the data set = that is accessed heavily. This drive is doing 1.5 times the throughput, but= the hangs have not gone away. > > To clarify, the data set used to live on ada2 (see the devlist below) whi= ch is a spinning disk. When I experienced intermittent hangs I plugged in a= n SSD drive (ada3 on the devlist) and moved the data there. This improved t= he MB's per second that are being written (it is mostly-write data) but has= not changed the hangs. If anything, they got worse since. > > Using gstat I notice that I/O service time is quite high. From the gstat = below you can see that it takes just over 2s to servr the requests. The L(q= ) seems to never drop far below 100 and %busy hovers around 100% all day lo= ng. Can someone please help me troubleshoot that further? What can I do to = make the underlying problem visible? > > I should mention all data is referenced through cross-mountpoint symlinks= , would that make a difference? Should I use canonical paths in the code in= stead? > > All file systems are mounted "noatime, soft-updates". You may want to play around with gshed, the GEOM Scheduler. Matt Dillon did a bunch of tests comparing FreeBSD+UFS to DragonflyBSD+HAMMER and found that FreeBSD starves read threads in order to satisfy write threads (or the other way around?). But, adding gsched into the mix helped things immensely, allowing mixed reads/writes to better shares disk I/O resources. I'll see if I can dig up a link to his testing e-mail messages. --=20 Freddie Cash fjwcash@gmail.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOjFWZ40LX%2B8Lw15mHDG8F3nN0aex5EpqVdjPxRPS89t1Fqkiw>