From owner-freebsd-net@freebsd.org Thu Jan 12 14:14:16 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC3B0CACD5D for ; Thu, 12 Jan 2017 14:14:16 +0000 (UTC) (envelope-from jpa@semihalf.com) Received: from mail-ua0-x22e.google.com (mail-ua0-x22e.google.com [IPv6:2607:f8b0:400c:c08::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7FE041768 for ; Thu, 12 Jan 2017 14:14:16 +0000 (UTC) (envelope-from jpa@semihalf.com) Received: by mail-ua0-x22e.google.com with SMTP id 96so14789989uaq.3 for ; Thu, 12 Jan 2017 06:14:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=ig06pvn7qzcu0Uu2io3V3i/sVOZzo0s45u/KhRUgHxQ=; b=V//BXZUXaAtukprEP5XTmVslOKbJkLJiuXn/Vbhx3SDS2m9Yqtk27ctnOSCSOcyldr mEd7LlBRYIv06QYMQw8WyzsOUuUi4yhZWjAvLWkvwhFjZcWqQcVzHaRkEwGYdioafLDy TU77LorlOQdOgrySBuHQbwdDJeeeLEzYsRsozuTvY81rShUpWSP/KMdbFk062SjlBCTm bJwOd7JpugI1v5MTEkDjc+Bx9x40jXayO+hE23jCn1q2zn/iryKCKK7z+v4klwPr20A8 tcuqbrectMavRCrQplRSw0b3x8HboHHdaPvZ4rqL5WLiYKpVlymYgrRxdRr10xXrumXu oMEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=ig06pvn7qzcu0Uu2io3V3i/sVOZzo0s45u/KhRUgHxQ=; b=AO0GsaZk1zR5ffvLSJmverQT2igTnkxfgkBIrD5X/Y2zAFH1oGEwObpCRJvMtcqwKU 2U2EryhuSB1D1X9+WN87yaHqN2u7payb0AthlEDZaEVw139ZXMzI0MUx0/JE6QDe8tI5 i4gs1/uI1tnsXq5g4a3sPwqE2xIom06jB30cIsqPvQQOnC3qD9hJJl/qVxF1LFZgS33O nsZUGf/CJHlZ1Q7DcVlInS3muhstxndvLvwmDAwSGT795zaBzwdzPqYYI2vZl57AtbG/ 038sPhi09LKdLRVVfrmEJh9w+kNXDMGHDgJBjxXnwVhk7CqJWX4jqawL853henJ5Kt5L qh+w== X-Gm-Message-State: AIkVDXLO9wxVSO9T1g/nfCIelOZlfqV+XLEOGHYXl6ynDbJAqLfAfLb0AIOyILPnY/TYWzMU5gjy87ZQhNS6eA== X-Received: by 10.159.48.131 with SMTP id j3mr5982468uab.43.1484230455540; Thu, 12 Jan 2017 06:14:15 -0800 (PST) MIME-Version: 1.0 Received: by 10.31.46.18 with HTTP; Thu, 12 Jan 2017 06:13:45 -0800 (PST) In-Reply-To: References: <20161128221033.GM55731@strugglingcoder.info> <20161129190509.GO27748@FreeBSD.org> From: Jakub Palider Date: Thu, 12 Jan 2017 15:13:45 +0100 Message-ID: Subject: Re: FLOWTABLE aka TCP route caching panic To: Gleb Smirnoff Cc: hiren panchasara , freebsd-net@freebsd.org, karels@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 14:14:16 -0000 An update: eventually, FLOWTABLE option also resulted in crash with __rw_lock_hard on FreeBSD release/11.0.0 (also on VM). This time, however, the backtrace was different, instead of tcp_output->ip_output->ether_output() chain, it happened in flowtable_clean_vnet() On Wed, Jan 11, 2017 at 10:42 AM, Jakub Palider wrote: > On Tue, 29 Nov 2016 11:05:09 -0800 > Gleb Smirnoff wrote: > > > On Mon, Nov 28, 2016 at 02:10:33PM -0800, hiren panchasara wrote: > > h> > Hi, > > h> > I have found that last month (19 Oct) the problem appeared on this > list, > > h> > and to my experience it persists, both on VM and bare metal > installation > > h> > (HEAD from yesterday). I looks that enabling FLOWTABLE option is > the only > > h> > source of this fault happening. It appears on our setup in 80% > cases within > > h> > one hour from boot up. > > h> > From our debugging, it is caused by lock on DESTROYED lock. Did you > find a > > h> > solution to this problem? > > > > Not yet. > > > > I'm pretty sure that reverting my r307234 will fix your crashes. > However, I still > > believe that r307234 is a proper way of fixing things, not r300854 which > just > > plugged the problem in the nearest place to the crash. But as we all see > r307234 > > is definitely missing some code path, which still allows for stale route > to be > > referenced. > > > > -- > > Totus tuus, Glebius. > > Thank you for your pointer, in helped at some point. The problem > however returned, especially when under heavy, continuous load. We are > running 4 iperf3 processes, each having 4 threads, and the machine > dies after 30-60 minutes of TCP traffic. I am running rS30911112 from > Nov 24. > On FreeBSD 11.0-RELEASE this problem (1 full day of testing) was not > noticed. Could you point to related commits that might also influence > this behaviour? > Thanks, > Jakub >