From owner-freebsd-net@freebsd.org Wed Oct 12 19:24:19 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 607C1C0FE4C for ; Wed, 12 Oct 2016 19:24:19 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 3CC2A75F for ; Wed, 12 Oct 2016 19:24:19 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 3C0AAC0FE45; Wed, 12 Oct 2016 19:24:19 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B8E3C0FE44 for ; Wed, 12 Oct 2016 19:24:19 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-yw0-x234.google.com (mail-yw0-x234.google.com [IPv6:2607:f8b0:4002:c05::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DEDD2754 for ; Wed, 12 Oct 2016 19:24:18 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-yw0-x234.google.com with SMTP id t192so39508494ywf.0 for ; Wed, 12 Oct 2016 12:24:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=dzrto6iOdU2evHhBWVY2Qrg8tZes9dgEEMJNBox9ums=; b=V8F0M3ScQJ9SRZP3sW7dk1Q8wVLsjLHO6reTtYmMGwPNuhAkriWHJqNexb9EQ4ZLfw sAXTgcFGY44J9QdRFERtwzaIM6UzPrsUoK9zKBF6Oq0tozar4B8/COQXOvUe6p66NEVe lJyxrQLUvZPz9Cj+qbyKt4lvsEJ3W1ti8w46if3x+S4t47saDgnlD4IBXVO3079AP7Or Bvr9EyOfDFrbAPmCWPbfGZxwYKAdeKN32llLZR158G5XnkyKm5pqOCyjQMq/xQTQ5iUe CJHFjwpcjsjnfDGk9TtPCActi9lGZ4jNM71QdJsjr/cqHLssS2PteTpcFGmUKlPV7VqC q+dQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=dzrto6iOdU2evHhBWVY2Qrg8tZes9dgEEMJNBox9ums=; b=CmKHuLfPep2uSf75Zeh2/3Mb9jNx7OPNPXWu03NuarG67FxJ2ZRJJlmaQJX6n8wUJc 9jBzBAXtF0fo8fku/HNwVekFvYQAXIyKvuMG12jst4QCS07J0FWlA4g5cWGQB1DPhyrz tpqjx5NduU9QtWIx0ktFenrjj38umrdYl6XX7IAzifrzTdMr7po81N2OZJKNayTfGfkz zhvrK13XimLnxJKI3fIzADzlSrQedTRDrk0+ZXphgLvjJMYL0D7ZeaIIVCxC6rSfdHtL Jy4CphPPjwcwsa+k7Ce7PxYwMLpzZxaSlDjb5oHWLMMAhDGwoGQGEzS6evQedouIUg1s rVLQ== X-Gm-Message-State: AA6/9RnNqTUPSkQ4q5twx0A4LzgKft/3Q1qdUF2NEVgWYpIesZtIfMawpZ9Meo6ylWeDrh7CoyPNlfYdZUf7mw== X-Received: by 10.129.120.74 with SMTP id t71mr2470948ywc.51.1476300257944; Wed, 12 Oct 2016 12:24:17 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.161.37 with HTTP; Wed, 12 Oct 2016 12:24:17 -0700 (PDT) In-Reply-To: <86183ea5-5855-5fb3-22f6-d25454859186@yahoo.com> References: <57FC859F.5000200@grosbein.net> <2033449965.65391.1476244568309@mail.yahoo.com> <86183ea5-5855-5fb3-22f6-d25454859186@yahoo.com> From: Zaphod Beeblebrox Date: Wed, 12 Oct 2016 15:24:17 -0400 Message-ID: Subject: Re: FreeBSD10.3-RELEASE. Kernel panic. To: Donald Baud Cc: "net@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2016 19:24:19 -0000 While my mp5 servers are possibly less busy (I havn't had common crashes), I have noticed a "group" of problems. 1. The carrier dropping communication (ie: fiber cut or l2 switch breakage) of the L2TP streams can leave mpd5 in a state where it will not die and will not destroy interfaces (requires reboot to clear). 2. There are race conditions between quagga and mpd5 for adding/dropping routes. 3. if A is a pppoe client and B is the mpd5 server, A cannot access TCP services on B. It can access tcp services _beyond_ B, but not on B. (there is a ticket open for this). On Wed, Oct 12, 2016 at 10:51 AM, Donald Baud via freebsd-net < freebsd-net@freebsd.org> wrote: > > On 10/12/16 1:13 AM, Julian Elischer wrote: > >> On 11/10/2016 8:56 PM, Donald Baud via freebsd-net wrote: >> >>> I've been plagued with these =3Ddaily=3D panics until I tried the follo= wing >>> recipes and the server has been up for 30 days so far: >>> >>> Normally I should expermient more to see which one of the receipes is >>> really the fix, but I'm just glad that the server is stable for now. >>> >> >> this is really great information. >> It makes debugging a lot more possible. >> I know it is a hard question, but do you have a way to simulate this >> workload? >> >> I have no real way to simulate this kind of workload >> > > Sadly, I don't have a way to simulate the workload but I am very > interested to help fix these crashes since as Cassiano said, this makes > mpd5/freebsd useless for pppoe/l2tp termination. > > At this point, I would suggest that Cassiano and =D0=90=D0=BD=D0=B4=D1=80= =D0=B5=D0=B9 confirm that they > don't get panics when they apply the recipes that I am using. > > I am still running many other cisco-vpdn gateways that I would convert > into mpd5/freebsd but my plan was stalled with the daily crashes. > I'll wait a couple of weeks to be sure that my recipes are a valid > workaround before converting my remaining cisco gateways to mpd5. > > -Dbaud > > >>> >>> recipe-1: Don't let mpd5 start automatically when server boots: >>> i.e. in: /etc/rc.conf >>> mpd5_enable=3D"NO" >>> and wait about 5 minutes after server boots then issue: >>> /usr/local/etc/rc.d/mpd5 onestart >>> >>> >>> recipe-2: recompile the kernel with the NETGRAPH_DEBUG option: >>> options NETGRAPH >>> options NETGRAPH_DEBUG >>> options NETGRAPH_KSOCKET >>> options NETGRAPH_L2TP >>> options NETGRAPH_SOCKET >>> options NETGRAPH_TEE >>> options NETGRAPH_VJC >>> options NETGRAPH_PPP >>> options NETGRAPH_IFACE >>> options NETGRAPH_MPPC_COMPRESSION >>> options NETGRAPH_MPPC_ENCRYPTION >>> options NETGRAPH_TCPMSS >>> options IPFIREWALL >>> >>> recipe-3: recompile the kernel and disable the IPv6 and SCTP options: >>> nooptions INET6 >>> nooptions SCTP >>> >>> recipe-4: Don't use any of the sysctl optimizations >>> in other words I commented out all values in sysctl.conf: >>> # net.graph.maxdgram=3D20480 (this is the default) >>> # net.graph.recvspace=3D20480 (this is the default) >>> >>> recipe-5: Don't use any of the loader.conf optimizations >>> in other words I commented out all values in loader.conf >>> # net.graph.maxdata=3D4096 (this is the default) >>> # net.graph.maxalloc=3D4096 (this is the default) >>> >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D >>> In my case, I had the panics with 10.3 and 11-PRERELEASE >>> 11.0-PRERELEASE FreeBSD 11.0-PRERELEASE #2 r305587 >>> >>> With those recipes, I have been running without any crash for a month >>> and counting. Thats' 300 l2tp tunnels and 1400 l2tp sessions generatin= g >>> 700Mbit/s. >>> >>> >>> -DBaud >>> >>> >>> On Tuesday, October 11, 2016 7:30 AM, Cassiano Peixoto < >>> peixotocassiano@gmail.com> wrote: >>> Hi, >>> >>> There are many users complaining about this: >>> >>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D186114 >>> >>> I've been dealing with this issue for one year with no solution. mpd5 a= s >>> pppoe server on FreeBSD is useless with this bug. >>> >>> I really would like to see it working again, i think it's quite importa= nt >>> to both project and many users. >>> >>> Thanks. >>> >>> On Tue, Oct 11, 2016 at 3:24 AM, Eugene Grosbein >>> wrote: >>> >>> 11.10.2016 11:02, =D0=90=D0=BD=D0=B4=D1=80=D0=B5=D0=B9 =D0=9B=D0=B5=D1= =83=D1=88=D0=BA=D0=B8=D0=BD =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >>>> >>>> Hello. I have problem with "FreeBSD nas 10.3-RELEASE FreeBSD >>>>> 10.3-RELEASE >>>>> #0: Fri Oct 7 21:12:56 YEKT 2016 nas@nas:/usr/obj/usr/src/sys/nasv3 >>>>> amd64" >>>>> >>>>> Kernel panic is repeated at intervals of 2-3 days. At first I thought >>>>> that >>>>> the problem is in the hardware, but the problem did not go away after >>>>> replacing the server platform. >>>>> >>>>> Coredumps and more info on link >>>>> https://drive.google.com/open?id=3D0BxciMy2q7ZjTTkIxem9wTE1tM2M >>>>> >>>>> Sorry for my english. >>>>> I'll wait for an answer. >>>>> >>>>> This is known and long-stanging problem in the FreeBSD network stack. >>>> It shows up when you have lots of network interfaced created/removed >>>> frequently >>>> like in your case of Network Access Server (PPtP, PPPoE etc). >>>> >>>> Generally, people run into this problem using mpd5 network daemon. >>>> mpd5 uses NETGRAPH kernel subsystem to process traffic and >>>> if an interface disappears (f.e., ,user disconnected) >>>> while kernel still processes traffic obtained from this interface, it >>>> panices. >>>> >>>> There were lots of reports of this problem. Noone seems to be working = on >>>> it at the moment. >>>> You should fill a PR using Bugzilla and attach your logs to it. >>>> >>>> Eugene Grosbein >>>> >>>> > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >