Date: Mon, 13 Feb 2012 09:48:08 -0500 From: Stephen Sanders <ssanders@softhammer.net> To: freebsd-hackers@freebsd.org Subject: Odd RAID Performance Issue Message-ID: <4F3922A8.2090808@softhammer.net>
next in thread | raw e-mail | index | archive | help
We've an application that logs data on one very large raid6 array and updates/accesses a database on another smaller raid5 array. Both arrays are connected to the same PCIe 3ware RAID controller. The system has 2 six core 3Ghz processors and 24 GB of RAM. The system is running FreeBSD 8.1. The averaged read/write rate to the database is 2MB/s while the averaged write raid to the data logging array is 300MB/s. Writes to the logging array are somewhat bursty. The problem we're encountering is that the disk subsystem appears to 'pause' periodically. It looks as if this is a result of disk read/write operations from the database array taking a very long time to complete (up to 8 sec). When the disk read operation takes such a long time, it appears that the system starts to run out of memory due to bio block buffering. Most processes end up in either getblk() or waithighrunning(). We've instrumented g_vfs_strategie() and bufdone_finish() using dtrace. The indication from this effort is that a number of reads and writes are taking 4-8 seconds. So far, it looks as if the disk driver and hardware are OK as read/write operations appear to be in the milli-second region. We believe that our instrumentation is pointing to something between the VFS layer and the CAM as the culprit. We've gotten the same result from FreeBSD 8.2 but have not tried FreeBSD 9 as yet. This scenario is not limited to a single system and is occurring on a couple of systems. Does this sound familiar to anyone out there? Thanks
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F3922A8.2090808>