Date: Tue, 24 Jun 2014 01:53:21 -0700 From: Stefan Stere <icetor.is@bitmessage.ro> To: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: [zone: Mbuf_cluster] kern.ipc.nmbclusters limit reached in Virtual machine causes downtime Message-ID: <1403600001.74170.YahooMailNeo@web162506.mail.bf1.yahoo.com>
next in thread | raw e-mail | index | archive | help
Hi, I have a vmware virtual server running FreeBSD 10.0 STABLE The virtual server has 100mbps port. It is running a Tor router, consuming an average of 6-7 TB of monthly traffic. Its the only purpose of the server. Last night it went down, and vmware console log was saying: [zone: Mbuf_cluster] kern.ipc.nmbclusters limit reached I don't know what this means - the traffic of the server is unlimited and nothing is capped in any way. What can I do to fix this? I have read on freebsd wiki that I might need to add some lines to sysctl ? can you please confirm? Thank you in advance. From owner-freebsd-net@FreeBSD.ORG Tue Jun 24 09:09:02 2014 Return-Path: <owner-freebsd-net@FreeBSD.ORG> Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B77B9A36; Tue, 24 Jun 2014 09:09:02 +0000 (UTC) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.69.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebius.int.ru", Issuer "cell.glebius.int.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 502AB20A7; Tue, 24 Jun 2014 09:09:01 +0000 (UTC) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.14.8/8.14.8) with ESMTP id s5O98m6t046655 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 24 Jun 2014 13:08:48 +0400 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.14.8/8.14.8/Submit) id s5O98lQK046654; Tue, 24 Jun 2014 13:08:47 +0400 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Tue, 24 Jun 2014 13:08:47 +0400 From: Gleb Smirnoff <glebius@FreeBSD.org> To: Alan Somers <asomers@freebsd.org> Subject: Re: ifaddr refcount problem Message-ID: <20140624090847.GB28199@glebius.int.ru> References: <53A48849.8080504@chelsio.com> <20140623085229.GQ28199@FreeBSD.org> <CAOtMX2hGpvhK5TkBUfXvXisu5E2zen=h6MEv_Bxecn=aKsnnnQ@mail.gmail.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="jL2BoiuKMElzg3CS" Content-Disposition: inline In-Reply-To: <CAOtMX2hGpvhK5TkBUfXvXisu5E2zen=h6MEv_Bxecn=aKsnnnQ@mail.gmail.com> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Navdeep Parhar <navdeep@chelsio.com> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>; List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Tue, 24 Jun 2014 09:09:02 -0000 --jL2BoiuKMElzg3CS Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Mon, Jun 23, 2014 at 10:44:58AM -0600, Alan Somers wrote: A> > On Fri, Jun 20, 2014 at 12:15:21PM -0700, Navdeep Parhar wrote: A> > N> Revision 264905 and 266860 that followed it seem to leak ifaddr A> > N> references. ifa_ifwithdstaddr and ifa_ifwithnet both install a A> > N> reference on the ifaddr returned to the caller but ip_output does not A> > N> release it, eventually leading to a panic when the refcount wraps over A> > N> to 0 and the ifaddr is freed while it is still on various lists. A> > N> A> > N> I'm using this patch for now. Thoughts? A> > N> A> > N> Regards, A> > N> Navdeep A> > N> A> > N> A> > N> diff -r 6dfcecd314af sys/netinet/ip_output.c A> > N> --- a/sys/netinet/ip_output.c Fri Jun 20 10:33:22 2014 -0700 A> > N> +++ b/sys/netinet/ip_output.c Fri Jun 20 12:07:12 2014 -0700 A> > N> @@ -243,6 +243,7 @@ again: A> > N> ifp = ia->ia_ifp; A> > N> ip->ip_ttl = 1; A> > N> isbroadcast = 1; A> > N> + ifa_free((void *)ia); A> > N> } else if (flags & IP_ROUTETOIF) { A> > N> if ((ia = ifatoia(ifa_ifwithdstaddr(sintosa(dst)))) == NULL && A> > N> (ia = ifatoia(ifa_ifwithnet(sintosa(dst), 0))) == NULL) { A> > N> @@ -253,6 +254,7 @@ again: A> > N> ifp = ia->ia_ifp; A> > N> ip->ip_ttl = 1; A> > N> isbroadcast = in_broadcast(dst->sin_addr, ifp); A> > N> + ifa_free((void *)ia); A> > N> } else if (IN_MULTICAST(ntohl(ip->ip_dst.s_addr)) && A> > N> imo != NULL && imo->imo_multicast_ifp != NULL) { A> > N> /* A> > A> > The patch shouldn't use void * casts, but instead specify explicit member: A> > A> > ifa_free(&ia->ia_ifa); A> > A> > Apart from that it, the patch looks entirely correct and plugging a leak. A> > Thanks! A> A> I still don't see how this patch would work without breaking stuff A> like the statistics collection at line 673 of ip_output.c. If we call A> ifa_free immediately after choosing our ifp, then ia won't be A> available at lines 630 or 673, and ip_output will never record A> statistics, right? You are right, thanks. In case of IP_SENDONES/IP_ROUTETOIF we should hold the reference to ia throughout the function and free it at the end. Suggested patch, not tested. -- Totus tuus, Glebius. --jL2BoiuKMElzg3CS Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="ip_output.diff" Index: sys/netinet/ip_output.c =================================================================== --- sys/netinet/ip_output.c (revision 267536) +++ sys/netinet/ip_output.c (working copy) @@ -552,8 +552,11 @@ sendit: #endif error = netisr_queue(NETISR_IP, m); goto done; - } else + } else { + if (flags & (IP_SENDONES | IP_ROUTETOIF)) + ifa_free(&ia->ia_ifa); goto again; /* Redo the routing table lookup. */ + } } /* See if local, if yes, send it to netisr with IP_FASTFWD_OURS. */ @@ -582,6 +585,8 @@ sendit: m->m_flags |= M_SKIP_FIREWALL; m->m_flags &= ~M_IP_NEXTHOP; m_tag_delete(m, fwd_tag); + if (flags & (IP_SENDONES | IP_ROUTETOIF)) + ifa_free(&ia->ia_ifa); goto again; } @@ -694,6 +699,8 @@ passout: done: if (ro == &iproute) RO_RTFREE(ro); + if (flags & (IP_SENDONES | IP_ROUTETOIF)) + ifa_free(&ia->ia_ifa); return (error); bad: m_freem(m); --jL2BoiuKMElzg3CS--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1403600001.74170.YahooMailNeo>