Date: Thu, 11 Feb 1999 09:15:29 -0800 From: "Justin C. Walker" <justin@apple.com> To: Chris Csanady <ccsanady@friley-185-205.res.iastate.edu> Cc: freebsd-current@FreeBSD.ORG, freebsd-net@FreeBSD.ORG Subject: Re: Serious mbuf cluster leak.. Message-ID: <199902111715.JAA00622@walker3.apple.com> In-Reply-To: "Your message of Wed, 10 Feb 1999 17:49:20 CST."<19990210234920.2A11B6@friley-185-205.res.iastate.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
<=
bold><flushleft><color><param>7540,3813,2F35</param><fontfamily><param>Hel=
vetica</param>Ouch. I can say that our implementation doesn't seem to =
suffer from this problem. Could be there's an issue in the use of =
PRUS_* v. the socket state we use. The code in my kernel looks like:
in sosend():
if (dontroute)
so->so_options |=3D SO_DONTROUTE;
if (resid > 0)
so->so_state |=3D SS_MORETOCOME;
s =3D splnet(); /* XXX =
*/
error =3D (*so->so_proto->pr_usrreq)(so,
(flags & MSG_OOB) ? PRU_SENDOOB : PRU_SEND,
top, addr, control);
splx(s);
if (dontroute)
so->so_options &=3D ~SO_DONTROUTE;
so->so_state &=3D ~SS_MORETOCOME;
and in tcp_output():
if (len) {
if (len =3D=3D tp->t_maxseg)
goto send;
if (!(so->so_state & SS_MORETOCOME)) {
if ((idle || tp->t_flags & TF_NODELAY) &&
len + off >=3D so->so_snd.sb_cc)
goto send;
}
if (tp->t_force)
goto send;
We've subjected this to countless (well, some, I'm sure, can count them =
:-}) hours of thrashing for web server, file server, and other-server =
types of uses, and haven't seen any (reports of) leakage like this.
I'll look more closely at the results we see, to verify that we don't =
have a problem.
Regards,
Justin
From: =
</fontfamily></color></flushleft></bold><color><param>7540,3813,2F35</para=
m><fontfamily><param>Helvetica</param>Chris Csanady =
<<ccsanady@friley-185-205.res.iastate.edu><color><param>0000,0000,0000</pa=
ram>
<bold><color><param>7540,3813,2F35</param>Date: =
</color></bold><color><param>7540,3813,2F35</param>1999-02-11 01:51:17 =
-0800<color><param>0000,0000,0000</param>
<bold><color><param>7540,3813,2F35</param>To: =
</color></bold><color><param>7540,3813,2F35</param>freebsd-current@FreeBSD=
.ORG<color><param>0000,0000,0000</param>
<bold><color><param>7540,3813,2F35</param>Subject: =
</color></bold><color><param>7540,3813,2F35</param>Re: Serious mbuf =
cluster leak..<color><param>0000,0000,0000</param>
<bold><color><param>7540,3813,2F35</param>Cc: =
</color></bold><color><param>7540,3813,2F35</param>freebsd-net@FreeBSD.ORG=
<color><param>0000,0000,0000</param>
<bold><color><param>7540,3813,2F35</param>In-reply-to: =
</color></bold><color><param>7540,3813,2F35</param>"Your message of Wed, =
10 Feb 1999 17:49:20 =
CST."<<19990210234920.2A11B6@friley-185-205.res.iastate.edu><color><param>=
0000,0000,0000</param>
<bold><color><param>7540,3813,2F35</param>X-Mailer: =
</color></bold><color><param>7540,3813,2F35</param>exmh version 2.0.2 =
2/24/98<color><param>0000,0000,0000</param>
<bold><color><param>7540,3813,2F35</param>X-Loop: =
</color></bold><color><param>7540,3813,2F35</param>FreeBSD.org<color><para=
m>0000,0000,0000</param>
=
</color></color></color></color></color></color></color></color></color></=
color></color></color></color></color></color></fontfamily><fixed><fontfam=
ily><param>Ohlfs</param>After a while, I have determined the cause of =
the leak to be the<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>following commit. Although, I can't =
seem to find any reason why<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>it would cause this =
behavior--reverting these files fixes =
it.<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>Any =
thoughts?<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>fenner 1999/01/20 09:32:01 =
PST<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> Modified =
files:<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> sys/kern =
uipc_socket.c <color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> sys/netinet =
tcp_output.c tcp_usrreq.c tcp_var.h <color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> sys/sys protosw.h =
<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> =
Log:<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> Add a flag, passed to pru_send =
routines, PRUS_MORETOCOME. This<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> flag means that there is more data =
to be put into the socket buffer.<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> Use it in TCP to reduce the =
interaction between mbuf sizes and =
the<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> Nagle =
algorithm.<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> =
<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> Based on: "Justin C. Walker" =
<<justin@apple.com>'s description of =
Apple's<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> fix for this =
problem.<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> =
<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> Revision Changes =
Path<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> 1.50 +4 -2 =
src/sys/kern/uipc_socket.c<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> 1.32 +3 -2 =
src/sys/netinet/tcp_output.c<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> 1.40 +7 -2 =
src/sys/netinet/tcp_usrreq.c<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> 1.49 +18 -17 =
src/sys/netinet/tcp_var.h<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param> 1.26 +2 -1 =
src/sys/sys/protosw.h<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>I have been seeing a nasty cluster =
leak in both 3.0 stable and 4.0<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>current as of today. Until now, I =
thougt maybe it was something in<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>my driver, athough after much =
careful looking over my code, it<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>simply does not look possible. =
Also, I downgraded to current of<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>Dec 12, and the problem =
dissappears. <color><param>0000,0000,0000</param>
=
<color><param>7540,3813,2F35</param>><color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>The odd thing is that the clusters =
that leak don't seem to be<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>attached to mbufs. Or at least =
there is not a 1-1 ratio. Following<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>is netstat output after a while of =
running netpipe in streaming<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>mode. (NPtcp -s; see =
ftp://ftp.scl.ameslab.gov/pub/netpipe) =
Also,<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>the leak only becomes apparent when =
the send write size is very<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>large--several hundred K to several =
megabytes.<color><param>0000,0000,0000</param>
=
<color><param>7540,3813,2F35</param>><color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>Does anyone have any idea what this =
may be? I really am not sure<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>where to look aside from trying =
prorgressively newer kernels. Also,<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>I only have alphas to test on right =
now..<color><param>0000,0000,0000</param>
=
<color><param>7540,3813,2F35</param>><color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>puck:~> netstat =
-m<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>211/416 mbufs in =
use:<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>> 116 mbufs allocated to =
data<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>> 95 mbufs allocated to =
packet headers<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>1674/1688/2048 mbuf clusters in use =
(current/peak/max)<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>3480 Kbytes allocated to network =
(97% in use)<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>0 requests for memory =
denied<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>0 requests for memory =
delayed<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>0 calls to protocol drain =
routines<color><param>0000,0000,0000</param>
=
<color><param>7540,3813,2F35</param>><color><param>0000,0000,0000</param>
=
<color><param>7540,3813,2F35</param>><color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>Chris =
Csanady<color><param>0000,0000,0000</param>
=
<color><param>7540,3813,2F35</param>><color><param>0000,0000,0000</param>
=
<color><param>7540,3813,2F35</param>><color><param>0000,0000,0000</param>
=
<color><param>7540,3813,2F35</param>><color><param>0000,0000,0000</param>
=
<color><param>7540,3813,2F35</param>><color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>To Unsubscribe: send mail to =
majordomo@FreeBSD.org<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>>with "unsubscribe freebsd-net" in =
the body of the message<color><param>0000,0000,0000</param>
=
<color><param>7540,3813,2F35</param>><color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>To Unsubscribe: send mail to =
majordomo@FreeBSD.org<color><param>0000,0000,0000</param>
<color><param>7540,3813,2F35</param>with "unsubscribe freebsd-net" in =
the body of the message<color><param>0000,0000,0000</param>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-current" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199902111715.JAA00622>