From owner-freebsd-net@FreeBSD.ORG Fri Apr 26 16:05:33 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2A9F4C6E; Fri, 26 Apr 2013 16:05:33 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-ob0-x235.google.com (mail-ob0-x235.google.com [IPv6:2607:f8b0:4003:c01::235]) by mx1.freebsd.org (Postfix) with ESMTP id E05C4108E; Fri, 26 Apr 2013 16:05:32 +0000 (UTC) Received: by mail-ob0-f181.google.com with SMTP id ta17so3710411obb.12 for ; Fri, 26 Apr 2013 09:05:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=MPmd2DknYQmHvzdZsL1ogzIbaPeqmbkUfhiN1afHquM=; b=f+T6YGEpLiaA+mqOaCDyDzZp0e9EyZNptnJ22glGSsk5o0DJtwWk0FHBISwLL33sNN gQ9U6uMg4SEcgSA5kvZofFcgN1zErYN3rj7FbmNXbvjNzsnbNFQr9mVhaoeJlKO0Rba3 0mFkgvzM0cs0jZhmcZkkvojDgIxsl14Sfz87fY++pL4APxJDvF5DRr34Yboo9ZoA9Yhk +3PlADiQbkYnUq/4je5kSZuO5GRz+sbkjano35mfpgvQ42EIAx98+xj2VOMmqmhZSLjS 3JcBLyv3YoK6MbVwTbOU4kw/9ybt3m3viJceCKOt/pgOAyGuQi/kJ5UkPSeW4bUtLuLA Qq3A== MIME-Version: 1.0 X-Received: by 10.182.96.1 with SMTP id do1mr17765581obb.17.1366992332536; Fri, 26 Apr 2013 09:05:32 -0700 (PDT) Received: by 10.76.156.198 with HTTP; Fri, 26 Apr 2013 09:05:32 -0700 (PDT) In-Reply-To: <517A93FE.7020209@soe.ucsc.edu> References: <5176E5C1.9090601@soe.ucsc.edu> <20130426134224.GV76816@FreeBSD.org> <517A93FE.7020209@soe.ucsc.edu> Date: Fri, 26 Apr 2013 12:05:32 -0400 Message-ID: Subject: Re: pf performance? From: Ryan Stone To: Erich Weiler Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Apr 2013 16:05:33 -0000 On Fri, Apr 26, 2013 at 10:49 AM, Erich Weiler wrote: > The pf isn't a process, so you can't see it in top. pf has some helper >> threads however, but packet processing isn't performed by any of them. >> > > But the work pf does would show up in 'system' on top right? So if I see > all my CPUs tied up 100% in 'interrupts' and very little 'system', would it > be a reasonable assumption to think that if I got more CPU cores to handle > the interrupts that eventually I would see 'system' load increase as the > interrupt load became faster to be handled? And thus increase my bandwidth? > > In other words, until I see like 100% system usage in one core, I would > have room to grow? > Probably not. Mutexes in FreeBSD use "adaptive spinning". This means that when a thread is unable to acquire a mutex, if the owner of the mutex is still running on another CPU core then the thread will spin and wait for the mutex to become free rather than block. This improves the latency of acquiring the mutex and saves us many expensive calls through the scheduler, but the downside is that you have one heavily contended mutex you can have many cores wasted spinning on the lock. In your case it is likely that your interrupt threads are spinning on the pf lock. You can partially confirm this by profiling the system. The quick way is: kldload hwpmc pmcstat -S unhalted-cycles -T (if unhalted-cycles does not work, try instructions). If _mtx_lock_sleep is the top function then you are hitting mutex contention and more cores are unlikely to help. In this case you might actually be able to improve performance by *reducing* the number of threads that are contending in pf, for example by using fewer receive queues in your NIC. If this machine is dedicated for pf then setting sysctl net.isr.direct=0 might also improve performance, by forcing all packets to go through a single netisr thread (assuming that net.isr.maxthreads is 1). Note that this will apply to traffic that does not go through pf, so if this machine were doing other network things (e.g. serving NFS) then it would bottleneck your other traffic behind your pf traffic.