From owner-freebsd-smp@FreeBSD.ORG Thu Mar 1 15:11:52 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A233A16A403 for ; Thu, 1 Mar 2007 15:11:52 +0000 (UTC) (envelope-from keith.arner@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.226]) by mx1.freebsd.org (Postfix) with ESMTP id 5668713C474 for ; Thu, 1 Mar 2007 15:11:52 +0000 (UTC) (envelope-from keith.arner@gmail.com) Received: by nz-out-0506.google.com with SMTP id r28so524726nza for ; Thu, 01 Mar 2007 07:11:51 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:mime-version:content-type:x-google-sender-auth; b=i4v9a4hne+SJyHNM1boav+thsrZL8pshpHU54HMQHnxvpP/L2fOXIDVLXQaHt5baudCJeP3ZwvOOaIx1/UH/eXqxM+1evlrwGTOD8ZZYqw3kkN43taM7StUdA0cTWx90VWEeI6BjihZWdhiUz1O3Xg5Iv2PxCsgfR0ssCyjLcnU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:mime-version:content-type:x-google-sender-auth; b=Drx3eBcMwu6sYJ+TAsutIH2uGCIDk9fSCSZHcr4ACgOgJi2XLVVDfvU2ju9JmzvwkXeBY1Do7L9Br95NBFFcI1jBhvcEI4WFTojnXSfgd7o86DyOcsnTPM3005gw8Me44UzT5BiPRWfCgm3rVpJ+tQMrZjAng9ROzr8Hat9o5NE= Received: by 10.65.206.7 with SMTP id i7mr3179997qbq.1172760354972; Thu, 01 Mar 2007 06:45:54 -0800 (PST) Received: by 10.64.201.3 with HTTP; Thu, 1 Mar 2007 06:45:54 -0800 (PST) Message-ID: <8e552a500703010645x61d9b064w21c475ecc00a0e0e@mail.gmail.com> Date: Thu, 1 Mar 2007 09:45:54 -0500 From: "Keith Arner" Sender: keith.arner@gmail.com To: freebsd-smp@freebsd.org MIME-Version: 1.0 X-Google-Sender-Auth: a893e1d06f40a209 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: INP_INFO_WLOCK(&tcbinfo) bottleneck X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Mar 2007 15:11:52 -0000 I'm working on a project that needs high throughput, low latency TCP traffic. I'd like to be able to parallelize as much of the processing as possible, across several cores. I've done some early investigation in FreeBSD with direct dispatch enabled (`sysctl net.isr.direct=1`). What I've discovered is that the system tends to bottleneck with a lot of contention for INP_INFO_WLOCK(&tcbinfo). Looking at the source code, it's apparent why this is; both tcp_input() and tcp_usr_send() hold this mutex for the majority of their processing, which effectively means that TCP input and output becomes single threaded. (Though I see that this is relaxed in the output path in -CURRENT.) I see from Robert Watson's todo list (at http://wiki.freebsd.org/NetworkTodo ) that using a read/write lock over the pcb list is a work in progress. Are there any patches available that reflect whatever work has been done so far? Or even any descriptions of what has been tried, and what the hard parts are? Keith