From owner-freebsd-stable@freebsd.org Tue May 3 10:30:15 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A9510B2B58C for ; Tue, 3 May 2016 10:30:15 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from smtp.fagskolen.gjovik.no (smtp.fagskolen.gjovik.no [IPv6:2001:700:1100:1:200:ff:fe00:b]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.fagskolen.gjovik.no", Issuer "Fagskolen i Gj??vik" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 38F9A1762 for ; Tue, 3 May 2016 10:30:15 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from mail.fig.ol.no (localhost [127.0.0.1]) by mail.fig.ol.no (8.15.2/8.15.2) with ESMTPS id u43AU73T099030 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 3 May 2016 12:30:07 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) Received: from localhost (trond@localhost) by mail.fig.ol.no (8.15.2/8.15.2/Submit) with ESMTP id u43AU6W9099027; Tue, 3 May 2016 12:30:07 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) X-Authentication-Warning: mail.fig.ol.no: trond owned process doing -bs Date: Tue, 3 May 2016 12:30:06 +0200 (CEST) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= Sender: Trond.Endrestol@fagskolen.gjovik.no To: Wolfgang Zenker cc: freebsd-stable@FreeBSD.org Subject: Re: Recent stable: bsnmpd eats up memory and cpu In-Reply-To: Message-ID: References: <20160501220107.GA58930@lyxys.ka.sub.org> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) Organization: Fagskolen Innlandet OpenPGP: url=http://fig.ol.no/~trond/trond.key MIME-Version: 1.0 X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, AWL autolearn=ham autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail.fig.ol.no Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2016 10:30:15 -0000 On Mon, 2 May 2016 09:10+0200, Trond Endrestøl wrote: > On Mon, 2 May 2016 00:01+0200, Wolfgang Zenker wrote: > > > Hi, > > > > after updating some 10-STABLE systems a few days ago, I noticed that on > > two of those systems bsnmpd started to use up a lot of cpu time, and the > > available memory shrinked until rendering the system unusable. Killing > > bsnmpd stops the cpu usage but does not free up memory. > > Both affected systems are amd64, one having moved from r297555 to > > r298723, the other from r297555 to r298722. Another amd64 system > > that went from r297555 to r298722 appears to be not affected. > > The two affected systems are on an internal LAN segment and there > > is currently no application connecting to snmp on those machines. > > > > What would be useful debugging data to collect in this case? > > I believe I've seen the very same on my systems. All of them got > updated last Friday due to the recent NTP fix. Prior to last Friday, > they all ran stable/10 from early March, r296648-ish. Neither of them > run bsnmpd, but they offer a lot of network services. > > Three of my i386 systems each with 1 GiB of memory ran out of swap > space, Sunday afternoon. > > This night a mail server running i386 with 4 GiB of memory died while > handling mail. From the messages I could glean on /dev/ttyvb (due to > custom logging) before rebooting, is that it's all networking related. > > SpamAssassin and syslogd on the mail server managed to transmit these > lines to the central log host before dying: > > May 2 00:05:17 [HOSTNAME] spamc[63613]: connect to spamd on ::1 failed, retrying (#1 of 3): Connection refused > May 2 00:05:17 [HOSTNAME] spamc[63613]: connect to spamd on 127.0.0.1 failed, retrying (#1 of 3): Connection refused > May 2 00:05:18 [HOSTNAME] spamc[63613]: connect to spamd on ::1 failed, retrying (#2 of 3): Connection refused > May 2 00:05:18 [HOSTNAME] spamc[63613]: connect to spamd on 127.0.0.1 failed, retrying (#2 of 3): Connection refused > May 2 00:05:19 [HOSTNAME] spamc[63613]: connect to spamd on ::1 failed, retrying (#3 of 3): Connection refused > May 2 00:05:19 [HOSTNAME] spamc[63613]: connect to spamd on 127.0.0.1 failed, retrying (#3 of 3): Connection refused > May 2 00:05:19 [HOSTNAME] spamc[63613]: connection attempt to spamd aborted after 3 retries > > May 2 00:52:17 [HOSTNAME] sm-mta[63740]: u41Mp86h063740: Milter (spamassassin): error creating socket: No buffer space available > May 2 00:52:17 [HOSTNAME] sm-mta[63739]: u41Mp8r9063739: Milter (spamassassin): error creating socket: No buffer space available > May 2 00:52:17 [HOSTNAME] sm-mta[63740]: u41Mp86h063740: Milter (spamassassin): to error state > May 2 00:52:17 [HOSTNAME] sm-mta[63739]: u41Mp8r9063739: Milter (spamassassin): to error state > > All of the amd64 systems with 4 GiB or 8 GiB of memory are apparently > unaffected. > > Maybe it's time to convert the remaining i386 systems to amd64 > systems, and add some memory while I'm at it. > > The bug is either in the kernel or in libc, or both. You might want to try out the patch created by Mark Johnston: http://lists.freebsd.org/pipermail/freebsd-current/2016-May/061015.html I'm in the process of testing this patch on one i386 stable/10 system and on one amd64 stable/10 system. -- +-------------------------------+------------------------------------+ | Vennlig hilsen, | Best regards, | | Trond Endrestøl, | Trond Endrestøl, | | IT-ansvarlig, | System administrator, | | Fagskolen Innlandet, | Gjøvik Technical College, Norway, | | tlf. mob. 952 62 567, | Cellular...: +47 952 62 567, | | sentralbord 61 14 54 00. | Switchboard: +47 61 14 54 00. | +-------------------------------+------------------------------------+ From owner-freebsd-stable@freebsd.org Tue May 3 11:41:14 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA8DEB2B718 for ; Tue, 3 May 2016 11:41:14 +0000 (UTC) (envelope-from jenkins-admin@FreeBSD.org) Received: from jenkins-9.freebsd.org (jenkins-9.freebsd.org [8.8.178.209]) by mx1.freebsd.org (Postfix) with ESMTP id 9C5A519C5; Tue, 3 May 2016 11:41:14 +0000 (UTC) (envelope-from jenkins-admin@FreeBSD.org) Received: from jenkins-9.freebsd.org (localhost [127.0.0.1]) by jenkins-9.freebsd.org (Postfix) with ESMTP id 9A6211143; Tue, 3 May 2016 11:41:14 +0000 (UTC) Date: Tue, 3 May 2016 11:41:14 +0000 (GMT) From: jenkins-admin@FreeBSD.org To: jenkins-admin@FreeBSD.org, freebsd-stable@FreeBSD.org Message-ID: <125543220.54.1462275674418.JavaMail.jenkins@jenkins-9.freebsd.org> Subject: Jenkins build became unstable: FreeBSD_stable_10 #241 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAkKKb2VAfYQKfu1t7qk4nR5qzUBEI+UqT4BPec4qHVhqUy0FFdq50sMH+3y9bCDNOufctov6VqTNffZ3YXArnZK95YF0OX97fh+E9txYOUX1adc+TikcKjuYpHmL5dE62eaZTI+4A5jnRonskQ1PaoIFz0Kbu4mWzkFsmdiXTraGzomXq4cHUCATA2+K4eDYgjXEQI30z3GOMmmZ4t/+6QGk1cMb/BqMWHbn80AsRCb4tU7Hpd72XLDpsuO7YRP1Q0CjmNAuBOTj+sFiiOe6U9HpqOlQN+iFUvBdZo/ybuy5Kh71cAaYQNL68cYdZJ6binH/DkG3KY/fS7DFYAeuwjwIDAQAB X-Jenkins-Job: FreeBSD_stable_10 X-Jenkins-Result: UNSTABLE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2016 11:41:14 -0000 See