From owner-freebsd-net@FreeBSD.ORG Wed Jul 6 19:02:35 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id CE4901065670 for ; Wed, 6 Jul 2011 19:02:35 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from xps.daemonology.net (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx2.freebsd.org (Postfix) with SMTP id 4CCF414DF12 for ; Wed, 6 Jul 2011 19:02:34 +0000 (UTC) Received: (qmail 16082 invoked from network); 6 Jul 2011 19:02:33 -0000 Received: from unknown (HELO xps.daemonology.net) (127.0.0.1) by localhost with SMTP; 6 Jul 2011 19:02:33 -0000 Message-ID: <4E14B149.6030307@freebsd.org> Date: Wed, 06 Jul 2011 12:02:33 -0700 From: Colin Percival User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.16) Gecko/20101220 Thunderbird/3.0.11 MIME-Version: 1.0 To: freebsd-net@freebsd.org, "freebsd-xen@freebsd.org" X-Enigmail-Version: 1.0.1 Content-Type: multipart/mixed; boundary="------------060605090108080700050202" Cc: Subject: how to get "max # of mbufs in a packet" from xn to the tcp stack? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2011 19:02:35 -0000 This is a multi-part message in MIME format. --------------060605090108080700050202 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi all, I've attached a patch which fixes the "nfrags > 18, netback won't be able to handle it" problem with xen netfront when TSO is enabled. It's not finished, though: + int max_mbuf_chain_len = 16; /* XXX Set this based on interface? */ I'm not sure what the right way is to feed a value from the interface up into tcp_output; can someone advise? -- Colin Percival Security Officer, FreeBSD | freebsd.org | The power to serve Founder / author, Tarsnap | tarsnap.com | Online backups for the truly paranoid --------------060605090108080700050202 Content-Type: text/x-diff; name="tcp_mbuf_chain_limit.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="tcp_mbuf_chain_limit.patch" Index: kern/uipc_mbuf.c =================================================================== --- kern/uipc_mbuf.c (revision 223824) +++ kern/uipc_mbuf.c (working copy) @@ -525,12 +525,14 @@ * only their reference counts are incremented. */ struct mbuf * -m_copym(struct mbuf *m, int off0, int len, int wait) +m_copy_nbufs(struct mbuf *m, int off0, int len, int wait, long * outlen, + int nbufmax) { struct mbuf *n, **np; int off = off0; struct mbuf *top; int copyhdr = 0; + int len_orig = len; KASSERT(off >= 0, ("m_copym, negative off %d", off)); KASSERT(len >= 0, ("m_copym, negative len %d", len)); @@ -546,7 +548,7 @@ } np = ⊤ top = 0; - while (len > 0) { + while (len > 0 && nbufmax-- > 0) { if (m == NULL) { KASSERT(len == M_COPYALL, ("m_copym, length > size of mbuf chain")); @@ -584,6 +586,9 @@ if (top == NULL) mbstat.m_mcfail++; /* XXX: No consistency. */ + if (outlen) + *outlen = len_orig - len; + return (top); nospace: m_freem(top); @@ -591,6 +596,13 @@ return (NULL); } +struct mbuf * +m_copym(struct mbuf *m, int off0, int len, int wait) +{ + + return (m_copy_nbufs(m, off0, len, wait, NULL, INT_MAX)); +} + /* * Returns mbuf chain with new head for the prepending case. * Copies from mbuf (chain) n from off for len to mbuf (chain) m Index: netinet/tcp_output.c =================================================================== --- netinet/tcp_output.c (revision 223824) +++ netinet/tcp_output.c (working copy) @@ -183,6 +183,7 @@ int sack_rxmit, sack_bytes_rxmt; struct sackhole *p; int tso; + int max_mbuf_chain_len = 16; /* XXX Set this based on interface? */ struct tcpopt to; #if 0 int maxburst = TCP_MAXBURST; @@ -806,16 +807,6 @@ struct mbuf *mb; u_int moff; - if ((tp->t_flags & TF_FORCEDATA) && len == 1) - TCPSTAT_INC(tcps_sndprobe); - else if (SEQ_LT(tp->snd_nxt, tp->snd_max) || sack_rxmit) { - tp->t_sndrexmitpack++; - TCPSTAT_INC(tcps_sndrexmitpack); - TCPSTAT_ADD(tcps_sndrexmitbyte, len); - } else { - TCPSTAT_INC(tcps_sndpack); - TCPSTAT_ADD(tcps_sndbyte, len); - } MGETHDR(m, M_DONTWAIT, MT_DATA); if (m == NULL) { SOCKBUF_UNLOCK(&so->so_snd); @@ -847,7 +838,8 @@ mtod(m, caddr_t) + hdrlen); m->m_len += len; } else { - m->m_next = m_copy(mb, moff, (int)len); + m->m_next = m_copy_nbufs(mb, moff, len, M_DONTWAIT, + &len, max_mbuf_chain_len); if (m->m_next == NULL) { SOCKBUF_UNLOCK(&so->so_snd); (void) m_free(m); @@ -856,6 +848,18 @@ } } + /* Update stats here as m_copy_nbufs may have adjusted len. */ + if ((tp->t_flags & TF_FORCEDATA) && len == 1) + TCPSTAT_INC(tcps_sndprobe); + else if (SEQ_LT(tp->snd_nxt, tp->snd_max) || sack_rxmit) { + tp->t_sndrexmitpack++; + TCPSTAT_INC(tcps_sndrexmitpack); + TCPSTAT_ADD(tcps_sndrexmitbyte, len); + } else { + TCPSTAT_INC(tcps_sndpack); + TCPSTAT_ADD(tcps_sndbyte, len); + } + /* * If we're sending everything we've got, set PUSH. * (This will keep happy those implementations which only Index: sys/mbuf.h =================================================================== --- sys/mbuf.h (revision 223824) +++ sys/mbuf.h (working copy) @@ -849,6 +849,7 @@ int, int, int, int); struct mbuf *m_copypacket(struct mbuf *, int); void m_copy_pkthdr(struct mbuf *, struct mbuf *); +struct mbuf *m_copy_nbufs(struct mbuf *, int, int, int, long *, int); struct mbuf *m_copyup(struct mbuf *n, int len, int dstoff); struct mbuf *m_defrag(struct mbuf *, int); void m_demote(struct mbuf *, int); --------------060605090108080700050202--