Date: Mon, 1 Jun 2015 07:49:56 +0000 From: =?iso-8859-1?Q?Karli_Sj=F6berg?= <karli.sjoberg@slu.se> To: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Strange networking behaviour in storage server Message-ID: <1433145014066.28625@slu.se>
next in thread | raw e-mail | index | archive | help
Hey! So we have this ZFS storage server upgraded from 9.3-RELEASE to 10.1-STABLE to overcome not being able to 1) use SSD drives as L2ARC[1] and 2) not being able to hotswap SATA drives[2]. After the upgrade we=B4ve noticed a very odd networking behaviour, it sends/receives full speed for a while, then there is a couple of minutes of complete silence where even terminal commands like an "ls" just waits until they are executed and then it starts sending full speed again. I =B4ve linked to a screenshot showing this send and pause behaviour. The blue line is the total, green is SMB and turquoise is NFS over jumbo frames. It behaves this way regardless of the protocol. http://oi62.tinypic.com/33xvjb6.jpg The problem is that these pauses can sometimes be so long that connections drop. Like someone is copying files over SMB or iSCSI and suddenly they get an error message saying that the transfer failed and they have to start over with the file(s). That=B4s horrible! So far NFS has proven to be the most resillient, it=B4s stupid simple nature just waits and resumes transfer when pause is over. Kudus for that. The server is driven by a Supermicro X9SRL-F, a Xeon 1620v2 and 64GB ECC RAM. The hardware has been ruled out, we happened to have a identical MB and CPU lying around and that didn=B4t improve things. We have also installed a Intel PRO 100/1000 Quad-port ethernet adapter to test if that would change things, but it hasn=B4t, it still behaves this way. The two built-in NIC's are Intel 82574L and the Quad-port NIC's are Intel 82571EB, so both em(4) driven. I happen to know that the em driver has updated between 9.3 and 10.1. Perhaps that is to blame, but I have no idea. Is there anyone that can make sense of this? [1]: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D197164 [2]: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D191348 /K
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1433145014066.28625>