Date: Tue, 29 May 2012 12:40:47 -0700 From: Freddie Cash <fjwcash@gmail.com> To: Kees Jan Koster <kjkoster@gmail.com> Cc: freebsd-stable@freebsd.org Subject: Re: FreeBSD 9.0 hangs on heavy I/O Message-ID: <CAOjFWZ7964oeTNZqADj4cRt3kkdOf5Mwyx8GQDnJnZ8vyONckg@mail.gmail.com> In-Reply-To: <CAOjFWZ40LX%2B8Lw15mHDG8F3nN0aex5EpqVdjPxRPS89t1Fqkiw@mail.gmail.com> References: <BD5D6BB6-8CFF-456A-B03E-05454EB03AB6@gmail.com> <CAOjFWZ40LX%2B8Lw15mHDG8F3nN0aex5EpqVdjPxRPS89t1Fqkiw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, May 29, 2012 at 12:34 PM, Freddie Cash <fjwcash@gmail.com> wrote: > On Tue, May 29, 2012 at 12:26 PM, Kees Jan Koster <kjkoster@gmail.com> wr= ote: >> I seem to have a problem where really heavy disk I/O is drowning my mach= ine. I see hangs in the shell where I am logged on using ssh. Network conne= ctions get dropped for no apparent reason and some HTTP requests are served= really slowly. Profiling the app code shows that the hangs are in complete= ly random places. Operations that are no more than a few lines of code apar= t suddenly take seconds to complete. >> >> In my search I seem to find that my machine is quite slow on the disk. I= find that rather odd, given that the device in question is an SSD drive an= d it is a good bit faster than the WD drive that used to carry the data set= that is accessed heavily. This drive is doing 1.5 times the throughput, bu= t the hangs have not gone away. >> >> To clarify, the data set used to live on ada2 (see the devlist below) wh= ich is a spinning disk. When I experienced intermittent hangs I plugged in = an SSD drive (ada3 on the devlist) and moved the data there. This improved = the MB's per second that are being written (it is mostly-write data) but ha= s not changed the hangs. If anything, they got worse since. >> >> Using gstat I notice that I/O service time is quite high. From the gstat= below you can see that it takes just over 2s to servr the requests. The L(= q) seems to never drop far below 100 and %busy hovers around 100% all day l= ong. Can someone please help me troubleshoot that further? What can I do to= make the underlying problem visible? >> >> I should mention all data is referenced through cross-mountpoint symlink= s, would that make a difference? Should I use canonical paths in the code i= nstead? >> >> All file systems are mounted "noatime, soft-updates". > > You may want to play around with gshed, the GEOM Scheduler. > > Matt Dillon did a bunch of tests comparing FreeBSD+UFS to > DragonflyBSD+HAMMER and found that FreeBSD starves read threads in > order to satisfy write threads (or the other way around?). =C2=A0But, > adding gsched into the mix helped things immensely, allowing mixed > reads/writes to better shares disk I/O resources. > > I'll see if I can dig up a link to his testing e-mail messages. Here's the post, part of a thread on benchmarking RAID controllers: http://leaf.dragonflybsd.org/mailarchive/kernel/2011-07/msg00034.html --=20 Freddie Cash fjwcash@gmail.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOjFWZ7964oeTNZqADj4cRt3kkdOf5Mwyx8GQDnJnZ8vyONckg>