From owner-freebsd-net@FreeBSD.ORG Tue Jun 24 08:56:50 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D550D55D for ; Tue, 24 Jun 2014 08:56:50 +0000 (UTC) Received: from nm41.bullet.mail.ne1.yahoo.com (nm41.bullet.mail.ne1.yahoo.com [98.138.120.48]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8D5A02F83 for ; Tue, 24 Jun 2014 08:56:49 +0000 (UTC) Received: from [127.0.0.1] by nm41.bullet.mail.ne1.yahoo.com with NNFMP; 24 Jun 2014 08:56:42 -0000 Received: from [98.138.226.177] by nm41.bullet.mail.ne1.yahoo.com with NNFMP; 24 Jun 2014 08:53:42 -0000 Received: from [66.196.81.174] by tm12.bullet.mail.ne1.yahoo.com with NNFMP; 24 Jun 2014 08:53:21 -0000 Received: from [98.139.212.224] by tm20.bullet.mail.bf1.yahoo.com with NNFMP; 24 Jun 2014 08:53:21 -0000 Received: from [127.0.0.1] by omp1033.mail.bf1.yahoo.com with NNFMP; 24 Jun 2014 08:53:21 -0000 X-Yahoo-Newman-Property: ymail-4 X-Yahoo-Newman-Id: 297825.95222.bm@omp1033.mail.bf1.yahoo.com Received: (qmail 87466 invoked by uid 60001); 24 Jun 2014 08:53:21 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1403600001; bh=9wIe3YsxzjYdMc2jGWtGsXE8MzX6B+HMdqbWI08O4YY=; h=References:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=GJli6KdzjIUZgBVJiNwkIf2IJ8ZSbiSdwLtZwnX8AP2rSL+d3QlMW6D7PG1Gt0UzMnobABzhhaMuUnwmgsWdzt9xwcqNamUjEtMDjtbjwdyQx2c1xn8E68geUAhGwbFtrDoJeQumNrwXXFyAR0sUTJm2vlpkT5qsojoNO/cJUjc= X-YMail-OSG: OssDxbQVM1ltf1N.SgrBzzYhRFfDBigVdHSiOaVXzDht_Ux EbQ4Lg1_GNzgkkzhs4stkzWywO.hkHh9bPa0RrE_.rlH.CGE9msdBGWjqsls KcBt736WEeHbvJFcmTUAMcFgUeqROEQVWT45aoEDDqWWlF8Snv8TvUWsySE9 VeRH9AABaXmeakukzJVFNh5GnipHSwwhS5q4SkjDRzIum.Pi3t1nz4HEx4qd F_vynWBPukkSQUcKTgffFz3fMwt9o872ptbI1P2etL.U2zBOhtXUSKtzOM0B Dz7G.NUwjmaYpCt8NsQP.V1oLrTp0TnTX1lTKb2_Ix0ma4x18Zzg1pGV2r4p 9_gnNWukRuVCX.WtWS22ngnz2_UK1pj2R.iVeJotaVF9vLnNCsaXyj8ydMmi WoOuGBqhMk4WhtPkmK5nG0ep3_DewOdm_en66VT21LiSiDfBKWUfPhTCTOrh 2vw4_7JoBSIKZU3AgBLl_1q2ypew4Y9UbkU1O0YEyXBOoBzujiOAXah9Uyup D2o2hsGdDNb4t2F9SWgdJG49sAxa4han0ZZXpq8yqV1vJ Received: from [89.122.203.240] by web162506.mail.bf1.yahoo.com via HTTP; Tue, 24 Jun 2014 01:53:21 PDT X-Rocket-MIMEInfo: 002.001, SGksCgpJIGhhdmUgYSB2bXdhcmUgdmlydHVhbCBzZXJ2ZXIgcnVubmluZyBGcmVlQlNEIDEwLjAgU1RBQkxFClRoZSB2aXJ0dWFsIHNlcnZlciBoYXMgMTAwbWJwcyBwb3J0LgoKSXQgaXMgcnVubmluZyBhIFRvciByb3V0ZXIsIGNvbnN1bWluZyBhbiBhdmVyYWdlIG9mIDYtNyBUQiBvZiBtb250aGx5IHRyYWZmaWMuIEl0cyB0aGUgb25seSBwdXJwb3NlIG9mIHRoZSBzZXJ2ZXIuCgpMYXN0IG5pZ2h0IGl0IHdlbnQgZG93biwgYW5kIHZtd2FyZSBjb25zb2xlIGxvZyB3YXMgc2F5aW5nOgpbem9uZTogTWJ1Zl8BMAEBAQE- X-RocketYMMF: stere01 X-Mailer: YahooMailWebService/0.8.191.1 References: Message-ID: <1403600001.74170.YahooMailNeo@web162506.mail.bf1.yahoo.com> Date: Tue, 24 Jun 2014 01:53:21 -0700 From: Stefan Stere Reply-To: : ; Subject: [zone: Mbuf_cluster] kern.ipc.nmbclusters limit reached in Virtual machine causes downtime To: "freebsd-net@freebsd.org" MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jun 2014 08:56:50 -0000 Hi, I have a vmware virtual server running FreeBSD 10.0 STABLE The virtual server has 100mbps port. It is running a Tor router, consuming an average of 6-7 TB of monthly traffic. Its the only purpose of the server. Last night it went down, and vmware console log was saying: [zone: Mbuf_cluster] kern.ipc.nmbclusters limit reached I don't know what this means - the traffic of the server is unlimited and nothing is capped in any way. What can I do to fix this? I have read on freebsd wiki that I might need to add some lines to sysctl ? can you please confirm? Thank you in advance. From owner-freebsd-net@FreeBSD.ORG Tue Jun 24 09:09:02 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B77B9A36; Tue, 24 Jun 2014 09:09:02 +0000 (UTC) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.69.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebius.int.ru", Issuer "cell.glebius.int.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 502AB20A7; Tue, 24 Jun 2014 09:09:01 +0000 (UTC) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.14.8/8.14.8) with ESMTP id s5O98m6t046655 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 24 Jun 2014 13:08:48 +0400 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.14.8/8.14.8/Submit) id s5O98lQK046654; Tue, 24 Jun 2014 13:08:47 +0400 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Tue, 24 Jun 2014 13:08:47 +0400 From: Gleb Smirnoff To: Alan Somers Subject: Re: ifaddr refcount problem Message-ID: <20140624090847.GB28199@glebius.int.ru> References: <53A48849.8080504@chelsio.com> <20140623085229.GQ28199@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="jL2BoiuKMElzg3CS" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: "freebsd-net@freebsd.org" , Navdeep Parhar X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jun 2014 09:09:02 -0000 --jL2BoiuKMElzg3CS Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Mon, Jun 23, 2014 at 10:44:58AM -0600, Alan Somers wrote: A> > On Fri, Jun 20, 2014 at 12:15:21PM -0700, Navdeep Parhar wrote: A> > N> Revision 264905 and 266860 that followed it seem to leak ifaddr A> > N> references. ifa_ifwithdstaddr and ifa_ifwithnet both install a A> > N> reference on the ifaddr returned to the caller but ip_output does not A> > N> release it, eventually leading to a panic when the refcount wraps over A> > N> to 0 and the ifaddr is freed while it is still on various lists. A> > N> A> > N> I'm using this patch for now. Thoughts? A> > N> A> > N> Regards, A> > N> Navdeep A> > N> A> > N> A> > N> diff -r 6dfcecd314af sys/netinet/ip_output.c A> > N> --- a/sys/netinet/ip_output.c Fri Jun 20 10:33:22 2014 -0700 A> > N> +++ b/sys/netinet/ip_output.c Fri Jun 20 12:07:12 2014 -0700 A> > N> @@ -243,6 +243,7 @@ again: A> > N> ifp = ia->ia_ifp; A> > N> ip->ip_ttl = 1; A> > N> isbroadcast = 1; A> > N> + ifa_free((void *)ia); A> > N> } else if (flags & IP_ROUTETOIF) { A> > N> if ((ia = ifatoia(ifa_ifwithdstaddr(sintosa(dst)))) == NULL && A> > N> (ia = ifatoia(ifa_ifwithnet(sintosa(dst), 0))) == NULL) { A> > N> @@ -253,6 +254,7 @@ again: A> > N> ifp = ia->ia_ifp; A> > N> ip->ip_ttl = 1; A> > N> isbroadcast = in_broadcast(dst->sin_addr, ifp); A> > N> + ifa_free((void *)ia); A> > N> } else if (IN_MULTICAST(ntohl(ip->ip_dst.s_addr)) && A> > N> imo != NULL && imo->imo_multicast_ifp != NULL) { A> > N> /* A> > A> > The patch shouldn't use void * casts, but instead specify explicit member: A> > A> > ifa_free(&ia->ia_ifa); A> > A> > Apart from that it, the patch looks entirely correct and plugging a leak. A> > Thanks! A> A> I still don't see how this patch would work without breaking stuff A> like the statistics collection at line 673 of ip_output.c. If we call A> ifa_free immediately after choosing our ifp, then ia won't be A> available at lines 630 or 673, and ip_output will never record A> statistics, right? You are right, thanks. In case of IP_SENDONES/IP_ROUTETOIF we should hold the reference to ia throughout the function and free it at the end. Suggested patch, not tested. -- Totus tuus, Glebius. --jL2BoiuKMElzg3CS Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="ip_output.diff" Index: sys/netinet/ip_output.c =================================================================== --- sys/netinet/ip_output.c (revision 267536) +++ sys/netinet/ip_output.c (working copy) @@ -552,8 +552,11 @@ sendit: #endif error = netisr_queue(NETISR_IP, m); goto done; - } else + } else { + if (flags & (IP_SENDONES | IP_ROUTETOIF)) + ifa_free(&ia->ia_ifa); goto again; /* Redo the routing table lookup. */ + } } /* See if local, if yes, send it to netisr with IP_FASTFWD_OURS. */ @@ -582,6 +585,8 @@ sendit: m->m_flags |= M_SKIP_FIREWALL; m->m_flags &= ~M_IP_NEXTHOP; m_tag_delete(m, fwd_tag); + if (flags & (IP_SENDONES | IP_ROUTETOIF)) + ifa_free(&ia->ia_ifa); goto again; } @@ -694,6 +699,8 @@ passout: done: if (ro == &iproute) RO_RTFREE(ro); + if (flags & (IP_SENDONES | IP_ROUTETOIF)) + ifa_free(&ia->ia_ifa); return (error); bad: m_freem(m); --jL2BoiuKMElzg3CS--