From owner-freebsd-net@freebsd.org Wed Oct 12 14:54:07 2016 Return-Path: <owner-freebsd-net@freebsd.org> Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5956BC0F42A for <freebsd-net@mailman.ysv.freebsd.org>; Wed, 12 Oct 2016 14:54:07 +0000 (UTC) (envelope-from donaldbaud@yahoo.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 38F3D1D6 for <freebsd-net@freebsd.org>; Wed, 12 Oct 2016 14:54:07 +0000 (UTC) (envelope-from donaldbaud@yahoo.com) Received: by mailman.ysv.freebsd.org (Postfix) id 35700C0F429; Wed, 12 Oct 2016 14:54:07 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3361FC0F428 for <net@mailman.ysv.freebsd.org>; Wed, 12 Oct 2016 14:54:07 +0000 (UTC) (envelope-from donaldbaud@yahoo.com) Received: from nm22-vm8.bullet.mail.gq1.yahoo.com (nm22-vm8.bullet.mail.gq1.yahoo.com [98.136.217.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 039A91D5 for <net@freebsd.org>; Wed, 12 Oct 2016 14:54:06 +0000 (UTC) (envelope-from donaldbaud@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1476283887; bh=D+IihGdmm6CRuUF18KCZuJ5BrDcKxYVadQzmjZFGp+E=; h=Subject:To:References:From:Date:In-Reply-To:From:Subject; b=Gu4NGbrv4HWjlTQ6OGa6Vg/yz7yL3YOyefIr0z4wAzKY/XElbSGp3kbb6aMKoCpyBimkJtOawAACXmg0O+Jzx8ux6u8CTIlF+hh1+7rm5ITkhvS0nEeJ1/1jnC7tGe3bd9opWduaaThZVbmaHS6hce3XoWWJR+RgD63eN62iC+Wk7ftJRm1FKb9vhjix/TWOU3l6ttX3saJ8D4EJXsS0ng0e06waiRRQiUKGINBsjIqEjqv8L8Qqg5UrnC9exivDDN4RoOE4khmUGFhf9p9jBnKtMfUPTHoJ2Rkkj9zvbuoTinLhM+8QZEt2sqCSd5Qzzb6aoqqfcMDyKp8Rno3uRQ== Received: from [98.137.12.58] by nm22.bullet.mail.gq1.yahoo.com with NNFMP; 12 Oct 2016 14:51:27 -0000 Received: from [208.71.42.213] by tm3.bullet.mail.gq1.yahoo.com with NNFMP; 12 Oct 2016 14:51:26 -0000 Received: from [127.0.0.1] by smtp224.mail.gq1.yahoo.com with NNFMP; 12 Oct 2016 14:51:26 -0000 X-Yahoo-Newman-Id: 958998.41214.bm@smtp224.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: RmaPDhkVM1kmNmkJPi5lz0eUzf915AuT_ybQftD7DgQv8v0 AkU4SlKtsJyRtUAbfCA5spuryREm5xrJNuJi6rSFYgcV.OQIyRdbwCaWA7Lr zHPnLs4ZWotTUtNCR8N1d2oZJL3spb9cGdQhgnQkBGGLlRHkHyVAEKneGGj. Vy.x.9wSwTn.My2MolfOXSOqoKKoU1tvo7gYkgurPksvK5wY53kCQ4EOKjj6 OtPkpSmRSDgUyj93vyW.xBJvtTsVqqd6ZxzUAWLOa42U8mMXG0gfu5rhdKP1 tOIQtWwEqi_MMiF14c2bT.8sd0fpqYpcaW1Jd0ifbVrXlPKh_E8gYTJrRAmn NFUV2TmwYiGpVcENHyZUtOIWw0Zt.NYRbjcXyJAUXVW087cmL5Kc8dWB0UtW Eauo4IDD33IPc32BLZYDgaMHdrq4tpxxhkvbHiAvmk5mse0JFyZKLuicIA6y y6w8XaShftYQzeqED3EdT_F_tNF2rZLvADSeTxW32eaHcxZPENLQObgnUO2H JR6tZllghxrxFei9.7V8nYlMzB8lUVuvCj0g6zFsdYpo7IZ0W5HX4YanB X-Yahoo-SMTP: fRRiyyGswBB1GYfBhw8frTz_QYRdpxY- Subject: Re: FreeBSD10.3-RELEASE. Kernel panic. To: "net@freebsd.org" <net@freebsd.org> References: <CAAFYNruF4gFAiTCAhyRUQzcovW2osrKn4ehiuNR0btJCZbnOGg@mail.gmail.com> <57FC859F.5000200@grosbein.net> <CAJajdNUXOrzWDKVmSB1Xm_G6zqBhMsZ2vesDcAw2CPGFBU0xtg@mail.gmail.com> <2033449965.65391.1476244568309@mail.yahoo.com> <a450f0eb-378a-2bd5-2f24-a0eb6b941856@freebsd.org> From: Donald Baud <donaldbaud@yahoo.com> Message-ID: <86183ea5-5855-5fb3-22f6-d25454859186@yahoo.com> Date: Wed, 12 Oct 2016 10:51:25 -0400 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <a450f0eb-378a-2bd5-2f24-a0eb6b941856@freebsd.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/> List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Wed, 12 Oct 2016 14:54:07 -0000 On 10/12/16 1:13 AM, Julian Elischer wrote: > On 11/10/2016 8:56 PM, Donald Baud via freebsd-net wrote: >> I've been plagued with these =daily= panics until I tried the >> following recipes and the server has been up for 30 days so far: >> >> Normally I should expermient more to see which one of the receipes is >> really the fix, but I'm just glad that the server is stable for now. > > this is really great information. > It makes debugging a lot more possible. > I know it is a hard question, but do you have a way to simulate this > workload? > > I have no real way to simulate this kind of workload Sadly, I don't have a way to simulate the workload but I am very interested to help fix these crashes since as Cassiano said, this makes mpd5/freebsd useless for pppoe/l2tp termination. At this point, I would suggest that Cassiano and Ðндрей confirm that they don't get panics when they apply the recipes that I am using. I am still running many other cisco-vpdn gateways that I would convert into mpd5/freebsd but my plan was stalled with the daily crashes. I'll wait a couple of weeks to be sure that my recipes are a valid workaround before converting my remaining cisco gateways to mpd5. -Dbaud >> >> >> recipe-1: Don't let mpd5 start automatically when server boots: >> i.e. in: /etc/rc.conf >> mpd5_enable="NO" >> and wait about 5 minutes after server boots then issue: >> /usr/local/etc/rc.d/mpd5 onestart >> >> >> recipe-2: recompile the kernel with the NETGRAPH_DEBUG option: >> options NETGRAPH >> options NETGRAPH_DEBUG >> options NETGRAPH_KSOCKET >> options NETGRAPH_L2TP >> options NETGRAPH_SOCKET >> options NETGRAPH_TEE >> options NETGRAPH_VJC >> options NETGRAPH_PPP >> options NETGRAPH_IFACE >> options NETGRAPH_MPPC_COMPRESSION >> options NETGRAPH_MPPC_ENCRYPTION >> options NETGRAPH_TCPMSS >> options IPFIREWALL >> >> recipe-3: recompile the kernel and disable the IPv6 and SCTP options: >> nooptions INET6 >> nooptions SCTP >> >> recipe-4: Don't use any of the sysctl optimizations >> in other words I commented out all values in sysctl.conf: >> # net.graph.maxdgram=20480 (this is the default) >> # net.graph.recvspace=20480 (this is the default) >> >> recipe-5: Don't use any of the loader.conf optimizations >> in other words I commented out all values in loader.conf >> # net.graph.maxdata=4096 (this is the default) >> # net.graph.maxalloc=4096 (this is the default) >> >> ================================ >> In my case, I had the panics with 10.3 and 11-PRERELEASE >> 11.0-PRERELEASE FreeBSD 11.0-PRERELEASE #2 r305587 >> >> With those recipes, I have been running without any crash for a month >> and counting. Thats' 300 l2tp tunnels and 1400 l2tp sessions >> generating 700Mbit/s. >> >> >> -DBaud >> >> >> On Tuesday, October 11, 2016 7:30 AM, Cassiano Peixoto >> <peixotocassiano@gmail.com> wrote: >> Hi, >> >> There are many users complaining about this: >> >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=186114 >> >> I've been dealing with this issue for one year with no solution. mpd5 as >> pppoe server on FreeBSD is useless with this bug. >> >> I really would like to see it working again, i think it's quite >> important >> to both project and many users. >> >> Thanks. >> >> On Tue, Oct 11, 2016 at 3:24 AM, Eugene Grosbein <eugen@grosbein.net> >> wrote: >> >>> 11.10.2016 11:02, Ðндрей Леушкин пишет: >>> >>>> Hello. I have problem with "FreeBSD nas 10.3-RELEASE FreeBSD >>>> 10.3-RELEASE >>>> #0: Fri Oct 7 21:12:56 YEKT 2016 nas@nas:/usr/obj/usr/src/sys/nasv3 >>>> amd64" >>>> >>>> Kernel panic is repeated at intervals of 2-3 days. At first I >>>> thought that >>>> the problem is in the hardware, but the problem did not go away after >>>> replacing the server platform. >>>> >>>> Coredumps and more info on link >>>> https://drive.google.com/open?id=0BxciMy2q7ZjTTkIxem9wTE1tM2M >>>> >>>> Sorry for my english. >>>> I'll wait for an answer. >>>> >>> This is known and long-stanging problem in the FreeBSD network stack. >>> It shows up when you have lots of network interfaced created/removed >>> frequently >>> like in your case of Network Access Server (PPtP, PPPoE etc). >>> >>> Generally, people run into this problem using mpd5 network daemon. >>> mpd5 uses NETGRAPH kernel subsystem to process traffic and >>> if an interface disappears (f.e., ,user disconnected) >>> while kernel still processes traffic obtained from this interface, it >>> panices. >>> >>> There were lots of reports of this problem. Noone seems to be >>> working on >>> it at the moment. >>> You should fill a PR using Bugzilla and attach your logs to it. >>> >>> Eugene Grosbein >>>