From owner-freebsd-xen@FreeBSD.ORG Thu Mar 7 12:42:56 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 52C2B125 for ; Thu, 7 Mar 2013 12:42:56 +0000 (UTC) (envelope-from bounces+73574-0602-freebsd-xen=freebsd.org@sendgrid.me) Received: from o3.shared.sendgrid.net (o3.shared.sendgrid.net [208.117.48.85]) by mx1.freebsd.org (Postfix) with SMTP id EC6177CA for ; Thu, 7 Mar 2013 12:42:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sendgrid.info; h=from :mime-version:to:cc:subject:references:in-reply-to:content-type; s=smtpapi; bh=yyW+lyLw3rs13AMwk5hoXERmzRM=; b=f8SZma2PEcCHYD0e2 LsnTfQQhLuu4N37WcQxY6Wt62T40CdsG39WjcB8Eyny3yVbY9q2J9Q20BgjP19j4 ptMUY80WUllsd4QyxkSRe05qJCQz6QcBeD9AXmDBqwsQ+dsuASDef5Gis4kt8Sc8 A00U6i7Nom8JSzKtuh2Q/w9boo= Received: by 10.4.35.240 with SMTP id mf66.16374.51388B491 Thu, 07 Mar 2013 06:42:49 -0600 (CST) Received: from mail.tarsnap.com (unknown [10.60.208.13]) by mi15 (SG) with ESMTP id 51388b49.37d6.188dad1 for ; Thu, 07 Mar 2013 06:42:49 -0600 (CST) Received: (qmail 46775 invoked from network); 7 Mar 2013 12:42:48 -0000 Received: from unknown (HELO clamshell.daemonology.net) (127.0.0.1) by ec2-107-20-205-189.compute-1.amazonaws.com with ESMTP; 7 Mar 2013 12:42:48 -0000 Received: (qmail 34920 invoked from network); 7 Mar 2013 12:41:57 -0000 Received: from unknown (HELO clamshell.daemonology.net) (127.0.0.1) by clamshell.daemonology.net with SMTP; 7 Mar 2013 12:41:57 -0000 Message-ID: <51388B15.1060105@freebsd.org> Date: Thu, 07 Mar 2013 04:41:57 -0800 From: Colin Percival User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130131 Thunderbird/17.0.2 MIME-Version: 1.0 To: Mark Felder Subject: Re: misc. questions References: <000901ce1aa7$90704320$b150c960$@ezwind.net> <5137C5E6.2070802@freebsd.org> In-Reply-To: X-Enigmail-Version: 1.4.6 Content-Type: multipart/mixed; boundary="------------020105010609080700090803" X-Sendgrid-EID: RUbAm5H8PjswBj/QH+sYVehaJogg3iBnZcyVi1bw/IzwwHsH9R1DI2ZrN2/OCZrbpeFjGLcQeytHwJz4DoEsfc6qmJdUUxKB52PEn86S3XgK0agFtMa2R21d8azUAPWR Cc: freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2013 12:42:56 -0000 This is a multi-part message in MIME format. --------------020105010609080700090803 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit On 03/07/13 04:35, Mark Felder wrote: > On Wed, 06 Mar 2013 16:40:38 -0600, Colin Percival wrote: >> You'll want to turn off tso, since it produces long mbuf chains which most >> xn netbacks choke on. (I have a very ugly workaround patch for this which I >> use on EC2, but simply turning off tso is enough unless you need Gbps+ speeds). > > Can you link me to this patch? I have an environment that might warrant using it > for now. Attached. And remember that I said it was a *very ugly* workaround... :-) -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid --------------020105010609080700090803 Content-Type: text/plain; charset=us-ascii; name="tcp_mbuf_chain_limit.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="tcp_mbuf_chain_limit.patch" --- sys/kern/uipc_mbuf.c (revision 223824) +++ sys/kern/uipc_mbuf.c (working copy) @@ -525,12 +525,14 @@ * only their reference counts are incremented. */ struct mbuf * -m_copym(struct mbuf *m, int off0, int len, int wait) +m_copy_nbufs(struct mbuf *m, int off0, int len, int wait, long * outlen, + int nbufmax) { struct mbuf *n, **np; int off = off0; struct mbuf *top; int copyhdr = 0; + int len_orig = len; KASSERT(off >= 0, ("m_copym, negative off %d", off)); KASSERT(len >= 0, ("m_copym, negative len %d", len)); @@ -546,7 +548,7 @@ } np = ⊤ top = 0; - while (len > 0) { + while (len > 0 && nbufmax-- > 0) { if (m == NULL) { KASSERT(len == M_COPYALL, ("m_copym, length > size of mbuf chain")); @@ -584,6 +586,9 @@ if (top == NULL) mbstat.m_mcfail++; /* XXX: No consistency. */ + if (outlen) + *outlen = len_orig - len; + return (top); nospace: m_freem(top); @@ -591,6 +596,13 @@ return (NULL); } +struct mbuf * +m_copym(struct mbuf *m, int off0, int len, int wait) +{ + + return (m_copy_nbufs(m, off0, len, wait, NULL, INT_MAX)); +} + /* * Returns mbuf chain with new head for the prepending case. * Copies from mbuf (chain) n from off for len to mbuf (chain) m --- sys/netinet/tcp_output.c (revision 228872) +++ sys/netinet/tcp_output.c (working copy) @@ -183,6 +183,7 @@ int sack_rxmit, sack_bytes_rxmt; struct sackhole *p; int tso; + int max_mbuf_chain_len = 16; /* XXX Set this based on interface? */ struct tcpopt to; #if 0 int maxburst = TCP_MAXBURST; @@ -806,16 +807,6 @@ struct mbuf *mb; u_int moff; - if ((tp->t_flags & TF_FORCEDATA) && len == 1) - TCPSTAT_INC(tcps_sndprobe); - else if (SEQ_LT(tp->snd_nxt, tp->snd_max) || sack_rxmit) { - tp->t_sndrexmitpack++; - TCPSTAT_INC(tcps_sndrexmitpack); - TCPSTAT_ADD(tcps_sndrexmitbyte, len); - } else { - TCPSTAT_INC(tcps_sndpack); - TCPSTAT_ADD(tcps_sndbyte, len); - } MGETHDR(m, M_DONTWAIT, MT_DATA); if (m == NULL) { SOCKBUF_UNLOCK(&so->so_snd); @@ -847,7 +838,8 @@ mtod(m, caddr_t) + hdrlen); m->m_len += len; } else { - m->m_next = m_copy(mb, moff, (int)len); + m->m_next = m_copy_nbufs(mb, moff, len, M_DONTWAIT, + &len, max_mbuf_chain_len); if (m->m_next == NULL) { SOCKBUF_UNLOCK(&so->so_snd); (void) m_free(m); @@ -856,6 +848,18 @@ } } + /* Update stats here as m_copy_nbufs may have adjusted len. */ + if ((tp->t_flags & TF_FORCEDATA) && len == 1) + TCPSTAT_INC(tcps_sndprobe); + else if (SEQ_LT(tp->snd_nxt, tp->snd_max) || sack_rxmit) { + tp->t_sndrexmitpack++; + TCPSTAT_INC(tcps_sndrexmitpack); + TCPSTAT_ADD(tcps_sndrexmitbyte, len); + } else { + TCPSTAT_INC(tcps_sndpack); + TCPSTAT_ADD(tcps_sndbyte, len); + } + /* * If we're sending everything we've got, set PUSH. * (This will keep happy those implementations which only --- sys/sys/mbuf.h (revision 223824) +++ sys/sys/mbuf.h (working copy) @@ -849,6 +849,7 @@ int, int, int, int); struct mbuf *m_copypacket(struct mbuf *, int); void m_copy_pkthdr(struct mbuf *, struct mbuf *); +struct mbuf *m_copy_nbufs(struct mbuf *, int, int, int, long *, int); struct mbuf *m_copyup(struct mbuf *n, int len, int dstoff); struct mbuf *m_defrag(struct mbuf *, int); void m_demote(struct mbuf *, int); --------------020105010609080700090803--