From owner-freebsd-current Fri Feb 12 11:59:38 1999 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id LAA27340 for freebsd-current-outgoing; Fri, 12 Feb 1999 11:59:38 -0800 (PST) (envelope-from owner-freebsd-current@FreeBSD.ORG) Received: from mail-out1.apple.com (mail-out1.apple.com [17.254.0.52]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id LAA27317; Fri, 12 Feb 1999 11:59:34 -0800 (PST) (envelope-from justin@rhapture.apple.com) Received: from mailgate1.apple.com (A17-128-100-225.apple.com [17.128.100.225]) by mail-out1.apple.com (8.8.5/8.8.5) with ESMTP id LAA50880; Fri, 12 Feb 1999 11:48:50 -0800 Received: from scv2.apple.com (scv2.apple.com) by mailgate1.apple.com (mailgate1.apple.com- SMTPRS 2.0.15) with ESMTP id ; Fri, 12 Feb 1999 11:48:43 -0800 Received: from rhapture.apple.com (rhapture.apple.com [17.202.40.59]) by scv2.apple.com (8.8.5/8.8.5) with ESMTP id LAA12162; Fri, 12 Feb 1999 11:48:41 -0800 Received: (from justin@localhost) by walker3.apple.com (8.9.1/8.9.1) id JAA00622; Thu, 11 Feb 1999 09:15:30 -0800 (PST) Message-Id: <199902111715.JAA00622@walker3.apple.com> To: Chris Csanady Subject: Re: Serious mbuf cluster leak.. Cc: freebsd-current@FreeBSD.ORG, freebsd-net@FreeBSD.ORG In-Reply-To: "Your message of Wed, 10 Feb 1999 17:49:20 CST."<19990210234920.2A11B6@friley-185-205.res.iastate.edu> Date: Thu, 11 Feb 1999 09:15:29 -0800 From: "Justin C. Walker" Reply-To: justin@apple.com X-Mailer: by Apple MailViewer (2.104) MIME-Version: 1.0 (Apple Message framework v104) Content-Type: text/enriched; charset=us-ascii Content-Transfer-Encoding: quoted-printable Sender: owner-freebsd-current@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG <= bold>7540,3813,2F35Hel= veticaOuch. I can say that our implementation doesn't seem to = suffer from this problem. Could be there's an issue in the use of = PRUS_* v. the socket state we use. The code in my kernel looks like: in sosend(): if (dontroute) so->so_options |=3D SO_DONTROUTE; if (resid > 0) so->so_state |=3D SS_MORETOCOME; s =3D splnet(); /* XXX = */ error =3D (*so->so_proto->pr_usrreq)(so, (flags & MSG_OOB) ? PRU_SENDOOB : PRU_SEND, top, addr, control); splx(s); if (dontroute) so->so_options &=3D ~SO_DONTROUTE; so->so_state &=3D ~SS_MORETOCOME; and in tcp_output(): if (len) { if (len =3D=3D tp->t_maxseg) goto send; if (!(so->so_state & SS_MORETOCOME)) { if ((idle || tp->t_flags & TF_NODELAY) && len + off >=3D so->so_snd.sb_cc) goto send; } if (tp->t_force) goto send; We've subjected this to countless (well, some, I'm sure, can count them = :-}) hours of thrashing for web server, file server, and other-server = types of uses, and haven't seen any (reports of) leakage like this. I'll look more closely at the results we see, to verify that we don't = have a problem. Regards, Justin From: = 7540,3813,2F35HelveticaChris Csanady = <0000,0000,0000 7540,3813,2F35Date: = 7540,3813,2F351999-02-11 01:51:17 = -08000000,0000,0000 7540,3813,2F35To: = 7540,3813,2F35freebsd-current@FreeBSD= .ORG0000,0000,0000 7540,3813,2F35Subject: = 7540,3813,2F35Re: Serious mbuf = cluster leak..0000,0000,0000 7540,3813,2F35Cc: = 7540,3813,2F35freebsd-net@FreeBSD.ORG= 0000,0000,0000 7540,3813,2F35In-reply-to: = 7540,3813,2F35"Your message of Wed, = 10 Feb 1999 17:49:20 = CST."<<19990210234920.2A11B6@friley-185-205.res.iastate.edu>= 0000,0000,0000 7540,3813,2F35X-Mailer: = 7540,3813,2F35exmh version 2.0.2 = 2/24/980000,0000,0000 7540,3813,2F35X-Loop: = 7540,3813,2F35FreeBSD.org0000,0000,0000 = OhlfsAfter a while, I have determined the cause of = the leak to be the0000,0000,0000 7540,3813,2F35following commit. Although, I can't = seem to find any reason why0000,0000,0000 7540,3813,2F35it would cause this = behavior--reverting these files fixes = it.0000,0000,0000 7540,3813,2F35Any = thoughts?0000,0000,0000 7540,3813,2F35fenner 1999/01/20 09:32:01 = PST0000,0000,0000 7540,3813,2F35 Modified = files:0000,0000,0000 7540,3813,2F35 sys/kern = uipc_socket.c 0000,0000,0000 7540,3813,2F35 sys/netinet = tcp_output.c tcp_usrreq.c tcp_var.h 0000,0000,0000 7540,3813,2F35 sys/sys protosw.h = 0000,0000,0000 7540,3813,2F35 = Log:0000,0000,0000 7540,3813,2F35 Add a flag, passed to pru_send = routines, PRUS_MORETOCOME. This0000,0000,0000 7540,3813,2F35 flag means that there is more data = to be put into the socket buffer.0000,0000,0000 7540,3813,2F35 Use it in TCP to reduce the = interaction between mbuf sizes and = the0000,0000,0000 7540,3813,2F35 Nagle = algorithm.0000,0000,0000 7540,3813,2F35 = 0000,0000,0000 7540,3813,2F35 Based on: "Justin C. Walker" = <'s description of = Apple's0000,0000,0000 7540,3813,2F35 fix for this = problem.0000,0000,0000 7540,3813,2F35 = 0000,0000,0000 7540,3813,2F35 Revision Changes = Path0000,0000,0000 7540,3813,2F35 1.50 +4 -2 = src/sys/kern/uipc_socket.c0000,0000,0000 7540,3813,2F35 1.32 +3 -2 = src/sys/netinet/tcp_output.c0000,0000,0000 7540,3813,2F35 1.40 +7 -2 = src/sys/netinet/tcp_usrreq.c0000,0000,0000 7540,3813,2F35 1.49 +18 -17 = src/sys/netinet/tcp_var.h0000,0000,0000 7540,3813,2F35 1.26 +2 -1 = src/sys/sys/protosw.h0000,0000,0000 7540,3813,2F35>I have been seeing a nasty cluster = leak in both 3.0 stable and 4.00000,0000,0000 7540,3813,2F35>current as of today. Until now, I = thougt maybe it was something in0000,0000,0000 7540,3813,2F35>my driver, athough after much = careful looking over my code, it0000,0000,0000 7540,3813,2F35>simply does not look possible. = Also, I downgraded to current of0000,0000,0000 7540,3813,2F35>Dec 12, and the problem = dissappears. 0000,0000,0000 = 7540,3813,2F35>0000,0000,0000 7540,3813,2F35>The odd thing is that the clusters = that leak don't seem to be0000,0000,0000 7540,3813,2F35>attached to mbufs. Or at least = there is not a 1-1 ratio. Following0000,0000,0000 7540,3813,2F35>is netstat output after a while of = running netpipe in streaming0000,0000,0000 7540,3813,2F35>mode. (NPtcp -s; see = ftp://ftp.scl.ameslab.gov/pub/netpipe) = Also,0000,0000,0000 7540,3813,2F35>the leak only becomes apparent when = the send write size is very0000,0000,0000 7540,3813,2F35>large--several hundred K to several = megabytes.0000,0000,0000 = 7540,3813,2F35>0000,0000,0000 7540,3813,2F35>Does anyone have any idea what this = may be? I really am not sure0000,0000,0000 7540,3813,2F35>where to look aside from trying = prorgressively newer kernels. Also,0000,0000,0000 7540,3813,2F35>I only have alphas to test on right = now..0000,0000,0000 = 7540,3813,2F35>0000,0000,0000 7540,3813,2F35>puck:~> netstat = -m0000,0000,0000 7540,3813,2F35>211/416 mbufs in = use:0000,0000,0000 7540,3813,2F35> 116 mbufs allocated to = data0000,0000,0000 7540,3813,2F35> 95 mbufs allocated to = packet headers0000,0000,0000 7540,3813,2F35>1674/1688/2048 mbuf clusters in use = (current/peak/max)0000,0000,0000 7540,3813,2F35>3480 Kbytes allocated to network = (97% in use)0000,0000,0000 7540,3813,2F35>0 requests for memory = denied0000,0000,0000 7540,3813,2F35>0 requests for memory = delayed0000,0000,0000 7540,3813,2F35>0 calls to protocol drain = routines0000,0000,0000 = 7540,3813,2F35>0000,0000,0000 = 7540,3813,2F35>0000,0000,0000 7540,3813,2F35>Chris = Csanady0000,0000,0000 = 7540,3813,2F35>0000,0000,0000 = 7540,3813,2F35>0000,0000,0000 = 7540,3813,2F35>0000,0000,0000 = 7540,3813,2F35>0000,0000,0000 7540,3813,2F35>To Unsubscribe: send mail to = majordomo@FreeBSD.org0000,0000,0000 7540,3813,2F35>with "unsubscribe freebsd-net" in = the body of the message0000,0000,0000 = 7540,3813,2F35>0000,0000,0000 7540,3813,2F35To Unsubscribe: send mail to = majordomo@FreeBSD.org0000,0000,0000 7540,3813,2F35with "unsubscribe freebsd-net" in = the body of the message0000,0000,0000 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-current" in the body of the message