From owner-freebsd-stable@FreeBSD.ORG Mon Sep 22 18:11:05 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7AB5758F; Mon, 22 Sep 2014 18:11:05 +0000 (UTC) Received: from mail-ig0-x230.google.com (mail-ig0-x230.google.com [IPv6:2607:f8b0:4001:c05::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 299C1EF5; Mon, 22 Sep 2014 18:11:05 +0000 (UTC) Received: by mail-ig0-f176.google.com with SMTP id hn15so3208738igb.15 for ; Mon, 22 Sep 2014 11:11:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=z+VHVqff8dslmYkWAEaIeBgmXxtg5QuGXZEKPvidAmM=; b=rXDN19LtMZuGm/FP/FIKlk96uzxUK5OKESQ6aODvWLWr8rThZq4tUBC4VXEfr4NijN 8PpLVLqyxjrYSx9O7bEJU9mVzHxpnXbDzKXRYVMornCvdb2DRQxs+Q2gVMgZDTIS3vMP zbxKnnNxwtoTZHV7VaPOxtc9agJIlzMzmtvPO+go+UKus1ilbTbjmT9+MvaO/m4qkOec PnkqesEEjxawfaPohO0C5n3IFcsQTlI7a7vJ50l93Snqhit+a/I99K0LGXPR6PqUgREt QXct5eMIaYr3uBuwlNkNE8Yr8kgMzUvooJsm6izR2LeGAihEiSz/FXT0DOYqlgZStbG0 NkDg== MIME-Version: 1.0 X-Received: by 10.43.151.5 with SMTP id kq5mr2435548icc.87.1411409464533; Mon, 22 Sep 2014 11:11:04 -0700 (PDT) Received: by 10.50.87.130 with HTTP; Mon, 22 Sep 2014 11:11:04 -0700 (PDT) In-Reply-To: References: <1411259605.674669006.339g4pd4@frv35.fwdcdn.com> Date: Mon, 22 Sep 2014 11:11:04 -0700 Message-ID: Subject: Re: FreeBSD 10 network performance problems From: Rumen Telbizov To: Adrian Chadd Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Tom Elite , "freebsd-stable@freebsd.org" , "K. Macy" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Sep 2014 18:11:05 -0000 Thank you all for the answers and directions. I tried all of the suggested changes to sysctl.conf and loader.conf settings above - they made no difference whatsoever. I believe they might help in certain situations but will improve things marginally. What I am dealing with is a pretty hard and sudden drop of performance due to lock contention. Adrian: What you're saying makes sense. If we can avoid the locking completely, this problem might go away. I'll see if I can get some help and try to tackle this challenge. Cheers, Rumen Telbizov On Sun, Sep 21, 2014 at 11:29 PM, Adrian Chadd wrote: > Hi! > > On 21 September 2014 15:08, K. Macy wrote: > > What you're dealing with is hardly an edge case. Most people don't need > to > > push more than a couple of Gbps in production. > > > > Flowtable is hardly "untested." However, it has been a source of friction > > at times because it can be somewhat brittle, having limits on the number > of > > cache entries that it can store that are frequently too low for people > with > > very large numbers of active flows. Without raising this limit > > substantially these systems will fail in a rather spectacular fashion. > > Additionally, flowtable was not written with the intent of being a > routing > > cache. It was developed to support stateful multipath routing for load > > balancing. In its current incarnation, stripped of much of the code for > its > > initial purpose, it's really just a band-aid around locking problems in > > routing. That said, the handful of commercial users of FreeBSD that do > have > > large amounts of traffic (10s of Gbps) per system that I personally know > of > > all have flowtable enabled. > > > > Unfortunately, at least in terms of what is in HEAD, little has been done > > to fix the contention that flowtable works around. For your purposes the > > response that Adrian gave you is the closest to "optimal." > > Gleb and I spent a bunch of time late last year and early this year > finding and fixing a lot of the corner cases with Flowtable handling. > > I'm pretty sure that it'll behave predictably and reliably for people > now. If it doesn't then please file PRs. It's no longer some corner of > the codebase that isn't well understood. At least two people besides > the author (Gleb and I) understand what it is, how it works and how it > ties into things. > > What's missing is someone sitting down and adding flowtable support to > the rest of the forwarding path(s). It shouldn't be too hard - as long > as you have an mbuf that has the IPv4/IPv6 header setup as that's what > flowtable currently expects when doing lookups - but it has to be > done. > > So I thoroughly recommend someone who has the test setup and the > desire to do so and post results. I have enough equipment now to test > this out and develop it but I'm really busy doing work, wireless RSS > stuff. So I just don't have the spare cycles to do it. > > I do think it'll be a pretty simple task. > > In theory - once you have the flowtable code working correctly in the > forwarding path you shouldn't see any rtentry lock contention except > during route changes (which will invalidate flowtable entries and > cause normal routing table lookups until the flowtable has all the > route entries in question.) > > > > -a > -- Rumen Telbizov Unix Systems Administrator