Date: Mon, 15 Oct 2018 17:05:54 -0400 From: James Shank <james.shank@gmail.com> To: freebsd-hackers@freebsd.org Subject: High I/O RAID on UFS2 Message-ID: <1BA62E58-93D7-4763-B27A-E21B7DA362B2@gmail.com>
next in thread | raw e-mail | index | archive | help
Hi FreeBSD Hackers, I am running FreeBSD 11.1-RELEASE-p15 with a UFS2 filesystem block size 64K,= with a RAID 6 of 16 SSD drives on a MRSAS compatible RAID card stripe size o= f 256K (and yes, I know about the misalignment) and 384GB memory on 48 real c= ores with hyper threading enabled. I invite critiques of this configuration= , but am more keen on getting help making it run efficiently. I am running 11.1 because 11.2-RELEASE repeatably panics after failing somew= here after allocbuf() tries to allocate a page (sorry, I don=E2=80=99t still= have the output from this state). I am doing very read heavy operations, via multithreaded readers. My writes= tend to be small, but come in every 15 seconds or so. I=E2=80=99d estimate= read to write ratio of about 100:1 or more. What I see happening is pagedaemon coming alive and bufferspacedaemon doing t= he same to clear the corresponding resources. bufferspacedaemon runs nearly= all the time at 100%. I haven=E2=80=99t yet had much success with tuning t= hings to keep enough resources free to allow my read rates to be sustained. My read rates can hit and exceed 6GBps when everything is humming along, but= only for short periods of time, then they drop off to much lower rates afte= r the kernel starts shuffling resources. For long periods of time, my best s= ustainable rate seems to be 3.5GBps, and I can=E2=80=99t seem to get the ful= l read rate for sustained periods. I have tried several things, but I believe now that I may be missing somethi= ng that the more experienced gurus will find obvious. Any pointers on where I should look next? Thanks! James
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1BA62E58-93D7-4763-B27A-E21B7DA362B2>