Date: Sun, 9 Feb 2014 19:54:15 -0500 (EST) From: Rick Macklem <rmacklem@uoguelph.ca> To: Christian Weisgerber <naddy@mips.inka.de> Cc: freebsd-net@freebsd.org Subject: Re: Terrible NFS performance under 9.2-RELEASE? Message-ID: <2059385169.3089582.1391993655512.JavaMail.root@uoguelph.ca> In-Reply-To: <ld8ft9$25o0$1@lorvorc.mips.inka.de>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
Christian Weisgerber wrote:
> Rick Macklem <rmacklem@uoguelph.ca> wrote:
>
> > I have a "hunch" that might explain why 64K NFS reads/writes
> > perform
> > poorly for some network environments.
> > A 64K NFS read reply/write request consists of a list of 34 mbufs
> > when
> > passed to TCP via sosend() and a total data length of around
> > 65680bytes.
> > Looking at a couple of drivers (virtio and ixgbe), they seem to
> > expect
> > no more than 32-33 mbufs in a list for a 65535 byte TSO xmit. I
> > think
> > (I don't have anything that does TSO to confirm this) that NFS will
> > pass
> > a list that is longer (34 plus a TCP/IP header).
>
> This may or may not be the same problem:
>
> When I switched my desktop box from FreeBSD 7 to 9, NFS read
> performance from my media server (running OpenBSD) became extremely
> poor. I couldn't even stream a movie any longer. Disabling TSO
> on the nfe(4) interface had no effect. My workaround was to switch
> from a TCP mount to a UDP one. The problem has persisted to FreeBSD
> 10.
>
> I can now report that switching to [rw]size=32768 with a TCP mount
> also works fine.
>
If it is convenient, trying a 64K TCP mount with a kernel that has the
attached patch (which makes it use page size clusters and reduces the
# of segments to 18) to see if it works well, would be interesting.
rick
ps: This is not a patch destined for head and shouldn't be applied to
a production box, but I think it is ok for testing.
> --
> Christian "naddy" Weisgerber
> naddy@mips.inka.de
>
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to
> "freebsd-net-unsubscribe@freebsd.org"
>
[-- Attachment #2 --]
--- fs/nfs/nfsport.h.sav2 2014-01-26 18:43:47.000000000 -0500
+++ fs/nfs/nfsport.h 2014-01-26 19:04:27.000000000 -0500
@@ -153,14 +153,27 @@
MGETHDR((m), M_WAITOK, MT_DATA); \
} \
} while (0)
-#define NFSMCLGET(m, w) do { \
- MGET((m), M_WAITOK, MT_DATA); \
- while ((m) == NULL ) { \
- (void) nfs_catnap(PZERO, 0, "nfsmget"); \
- MGET((m), M_WAITOK, MT_DATA); \
- } \
- MCLGET((m), (w)); \
+#if MJUMPAGESIZE > MCLBYTES
+#define NFSMCLGET(m, w) do { \
+ (m) = m_getjcl(M_WAITOK, MT_DATA, 0, MJUMPAGESIZE); \
+ while ((m) == NULL) { \
+ (void)nfs_catnap(PZERO, 0, "nfsmget"); \
+ MGET((m), M_WAITOK, MT_DATA); \
+ if ((m) != NULL) \
+ MCLGET((m), (w)); \
+ } \
} while (0)
+#else
+#define NFSMCLGET(m, w) do { \
+ (m) = m_getjcl(M_WAITOK, MT_DATA, 0, MCLBYTES); \
+ while ((m) == NULL) { \
+ (void)nfs_catnap(PZERO, 0, "nfsmget"); \
+ MGET((m), M_WAITOK, MT_DATA); \
+ if ((m) != NULL) \
+ MCLGET((m), (w)); \
+ } \
+ } while (0)
+#endif
#define NFSMCLGETHDR(m, w) do { \
MGETHDR((m), M_WAITOK, MT_DATA); \
while ((m) == NULL ) { \
--- fs/nfsserver/nfs_nfsdport.c.sav2 2014-01-26 18:54:29.000000000 -0500
+++ fs/nfsserver/nfs_nfsdport.c 2014-01-26 18:56:08.000000000 -0500
@@ -566,8 +566,7 @@ nfsvno_readlink(struct vnode *vp, struct
len = 0;
i = 0;
while (len < NFS_MAXPATHLEN) {
- NFSMGET(mp);
- MCLGET(mp, M_WAITOK);
+ NFSMCLGET(mp, M_WAITOK);
mp->m_len = NFSMSIZ(mp);
if (len == 0) {
mp3 = mp2 = mp;
@@ -636,8 +635,7 @@ nfsvno_read(struct vnode *vp, off_t off,
*/
i = 0;
while (left > 0) {
- NFSMGET(m);
- MCLGET(m, M_WAITOK);
+ NFSMCLGET(m, M_WAITOK);
m->m_len = 0;
siz = min(M_TRAILINGSPACE(m), left);
left -= siz;
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2059385169.3089582.1391993655512.JavaMail.root>