From owner-freebsd-net@FreeBSD.ORG Sat Feb 8 01:56:31 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0C84E9E5; Sat, 8 Feb 2014 01:56:31 +0000 (UTC) Received: from mail-pa0-x22a.google.com (mail-pa0-x22a.google.com [IPv6:2607:f8b0:400e:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D09101FE4; Sat, 8 Feb 2014 01:56:30 +0000 (UTC) Received: by mail-pa0-f42.google.com with SMTP id kl14so3956256pab.1 for ; Fri, 07 Feb 2014 17:56:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=AYYGh5i5fu5dRqBRMnnDew3pMyPWxg2dWO2b7l4RjkU=; b=W4gvYc/GkDY/RyvL8+eBNSSf7gE3bFUetfEMopoAZbDMxwC/l+fKuKebyY/kat4PBV apiQNhD5AmYndoN8tebL/y0vfwE+vmZnmKTK+VtpxLnGg04aLHBlHRLt3Ctpa1t6TdS/ rXHIjRn+GOWXVccR8Lgk20nM5JtuHuLkXxKf2/6M5GGVjk0pM1YbL6piBfBnoW/8L+y/ Aoa9DTVFg+mpfm5qjMhuS9tuPI3ExRnr4pjZGk5+iIltvWZieleryXicjEqhWO3+U4cC thkBka3uaqmv3OJYa44ACV0PIFy1onNTCIP6hze9oyTPmYWNMx30mqUUfl83vGhXVek9 EM7w== MIME-Version: 1.0 X-Received: by 10.66.144.102 with SMTP id sl6mr11367016pab.96.1391824590415; Fri, 07 Feb 2014 17:56:30 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.70.103.174 with HTTP; Fri, 7 Feb 2014 17:56:29 -0800 (PST) In-Reply-To: References: Date: Fri, 7 Feb 2014 17:56:29 -0800 X-Google-Sender-Auth: DlLOcpG-6OiTJIEw5vOsRxf6oOc Message-ID: Subject: Re: flowtable, collisions, locking and CPU affinity From: Adrian Chadd To: Ryan Stone Content-Type: text/plain; charset=ISO-8859-1 Cc: FreeBSD Net , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Feb 2014 01:56:31 -0000 On 7 February 2014 17:05, Ryan Stone wrote: > On Fri, Feb 7, 2014 at 7:12 PM, Adrian Chadd wrote: >> In any case - the reason it's happening above is because there's no >> actual lock held over the whole lookup/insert path. It's a per-CPU >> critical enter/exit path, so the only way to guarantee consistency is >> to use sched_pin() for the entirety of the function. > > sched_pin seems like a very heavy hammer for what has to be a very > rare and mostly harmless race. Why not redo the lookup after you have > reacquired the lock, and if you don't have to do the insert anymore > then don't and move on? You say "rare and harmless race"; I can trick the system to do this: flowtable for IPv4: 57448 lookups 39543 hits 17905 misses 17820 collisions 382 free checks 82 frees .. and it gets stuck in a loop of never quite correctly updating/using the flowtable entries. So, it is mostly harmless, except exactly when it bites you in the ass. Yes, we could just reacquire the lock and insert if required, but I still have to be absolutely sure that the thread isn't preempted and migrated to another core. Otherwie we'd have teh same issue again. I'll keep tickling it. Thanks, -a