Date: Thu, 10 Apr 2003 23:51:13 +0300 From: "Petri Helenius" <pete@he.iki.fi> To: "Jin Guojun [DSD]" <j_guojun@lbl.gov>, <freebsd-hackers@freebsd.org>, <freebsd-performance@freebsd.org> Subject: Re: tcp_output starving -- is due to mbuf get delay? Message-ID: <05b601c2ffa2$ed87a5b0$932a40c1@PHE> References: <3E94A22D.174321F0@lbl.gov> <3E94A8C4.3A196E42@lbl.gov>
next in thread | previous in thread | raw e-mail | index | archive | help
There was a discussion on mballoc performance on freebsd-net about a month ago
but it has since died without conclusion.
Pete
----- Original Message -----
From: "Jin Guojun [DSD]" <j_guojun@lbl.gov>
To: <freebsd-hackers@freebsd.org>; <freebsd-performance@freebsd.org>
Sent: Thursday, April 10, 2003 2:12 AM
Subject: Re: tcp_output starving -- is due to mbuf get delay?
> Some details was left behind --
>
> The machine is 2 GHz Intel P4 with 1 GB memory, so the delay is not from
> either CPU or lack of memory.
>
> -Jin
>
> "Jin Guojun [DSD]" wrote:
>
> > When testing GigE path that has 67 ms RTT, the maximum TCP throughput is
> > limited at 250 Mb/s. By tracing the problem, I found that tcp_output() is
> > starving
> > where snd_wnd and snd_cwnd are fully open. The snd_cc is never filled beyond
> > the 4.05MB even though the snd_hiwat is 10MB and snd_sbmax is 8MB. That is,
> > sosend never stopped at sbwait. So only place can slow down is the mbuf
> > allocation
> > in sosend(). The attached trace file shows that each MGET and MCLGET takes
> > significant time -- around 8 us at slow start time, and gradually increasing
> > after that
> > in an range 18 to 648 us.
> > Each packet Tx on GigE takes 12 us. It average mbuf allocation takes 18 us, then
> >
> > the performance will be reduced to 40%, in fact it is down to 25%, which means
> > higher average delay.
> >
> > I have change NMBCLUSTER from 2446 to 6566 to 10240, and nothing is improved.
> >
> > Any one can tell what factors would cause MGET / MCLGET to wait?
> > Is there any way to make MGET/MCLGET not to wait?
> >
> > -Jin
> >
> > ----------- system info -------------
> >
> > kern.ipc.maxsockbuf: 10485760
> > net.inet.tcp.sendspace: 8388608
> > kern.ipc.nmbclusters: 10240
> > kern.ipc.mbuf_wait: 32
> > kern.ipc.mbtypes: 2606 322 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> > kern.ipc.nmbufs: 40960
> >
> > -------------- code trace and explanation ----------
> >
> > sosend()
> > {
> > ...
> > if (space < resid + clen &&
> > (atomic || space < so->so_snd.sb_lowat || space < clen)) {
> > if (so->so_state & SS_NBIO)
> > snderr(EWOULDBLOCK);
> > sbunlock(&so->so_snd);
> > error = sbwait(&so->so_snd); /***** never come
> > down to here ****/
> > splx(s);
> > if (error)
> > goto out;
> > goto restart;
> > }
> > splx(s);
> > mp = ⊤
> > space -= clen;
> > do {
> > if (uio == NULL) {
> > /*
> > * Data is prepackaged in "top".
> > */
> > resid = 0;
> > if (flags & MSG_EOR)
> > top->m_flags |= M_EOR;
> > } else do {
> > if (top == 0) {
> > microtime(&t1);
> > MGETHDR(m, M_WAIT, MT_DATA);
> > if (m == NULL) {
> > error = ENOBUFS;
> > goto release;
> > }
> > mlen = MHLEN;
> > m->m_pkthdr.len = 0;
> > m->m_pkthdr.rcvif = (struct ifnet *)0;
> > } else {
> > MGET(m, M_WAIT, MT_DATA);
> > if (m == NULL) {
> > error = ENOBUFS;
> > goto release;
> > }
> > mlen = MLEN;
> > }
> > if (resid >= MINCLSIZE) {
> > MCLGET(m, M_WAIT);
> > if ((m->m_flags & M_EXT) == 0)
> > goto nopages;
> > mlen = MCLBYTES;
> > len = min(min(mlen, resid), space);
> > } else {
> > nopages:
> > len = min(min(mlen, resid), space);
> > /*
> > * For datagram protocols, leave room
> > * for protocol headers in first mbuf.
> > */
> > if (atomic && top == 0 && len < mlen)
> > MH_ALIGN(m, len);
> > }
> > microtime(&t2);
> > td = time_diff(&t2, &t1);
> > if ((td > 5 && (++tcnt & 31) == 0) || td > 50)
> > log( ... "td %d %d\n", td, tcnt);
> >
> > ...
> >
> > } /* end of sosend */
>
> _______________________________________________
> freebsd-performance@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?05b601c2ffa2$ed87a5b0$932a40c1>