From owner-freebsd-smp@FreeBSD.ORG Thu Mar 1 20:37:42 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 338F616A404 for ; Thu, 1 Mar 2007 20:37:41 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.244]) by mx1.freebsd.org (Postfix) with ESMTP id A37B013C4BC for ; Thu, 1 Mar 2007 20:37:41 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by an-out-0708.google.com with SMTP id c24so463960ana for ; Thu, 01 Mar 2007 12:37:41 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=irymc1AZNeOK6TiuVVjoCcaOS9UPiA5iRIgM+A8Gm8+HlOF/zeydYckpF5M0e88ezoeEKx/UOU4myffwWHcESHkpkuEj04El1CxziXDZmhMfMpR5/GJP7QCFMPEMZ9/4qBCqhgw2Xdl7EXaq7KAiKEOtfQxfFmejGeTnvpcngnk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=LKYkE6wBhdTQjvT5E3PqjCABMramA96JMprXeoy74g0XQJOTjQkg3UlHOMkto8+twTLYmdjWl6wGUvczfT8niOrNmxudkZv3pYtWyY7Gu8nvCR++Mwqx8m3++5b7702xq3Dem1e9zvl8Z6ARmw+u5nqWDTSrQq/dRRcCmLL3ERg= Received: by 10.100.57.14 with SMTP id f14mr1795254ana.1172779866030; Thu, 01 Mar 2007 12:11:06 -0800 (PST) Received: by 10.100.191.1 with HTTP; Thu, 1 Mar 2007 12:11:05 -0800 (PST) Message-ID: <3bbf2fe10703011211i6b0ad2b6gf75067ca17d0ac3b@mail.gmail.com> Date: Thu, 1 Mar 2007 21:11:05 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "Keith Arner" In-Reply-To: <8e552a500703010645x61d9b064w21c475ecc00a0e0e@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8e552a500703010645x61d9b064w21c475ecc00a0e0e@mail.gmail.com> X-Google-Sender-Auth: 89d8169ea68659df Cc: freebsd-smp@freebsd.org Subject: Re: INP_INFO_WLOCK(&tcbinfo) bottleneck X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Mar 2007 20:37:42 -0000 2007/3/1, Keith Arner : > I'm working on a project that needs high throughput, low latency TCP > traffic. > I'd like to be able to parallelize as much of the processing as possible, > across > several cores. > > I've done some early investigation in FreeBSD with direct dispatch enabled > (`sysctl net.isr.direct=1`). What I've discovered is that the system tends > to bottleneck with a lot of contention for INP_INFO_WLOCK(&tcbinfo). > Looking at the source code, it's apparent why this is; both tcp_input() > and tcp_usr_send() hold this mutex for the majority of their processing, > which effectively means that TCP input and output becomes single threaded. > (Though I see that this is relaxed in the output path in -CURRENT.) > > I see from Robert Watson's todo list (at http://wiki.freebsd.org/NetworkTodo > ) > that using a read/write lock over the pcb list is a work in progress. Are > there > any patches available that reflect whatever work has been done so far? Or > even any descriptions of what has been tried, and what the hard parts are? > > Keith Currently, the main problem for this is that somewhere (TM) there mutex gets recursed and recursion is not handled alredy for rwlocks, so kernel starts panicing. Patch for recursion in rwlock is not trivial, but there are ongoing discussions on it. Attilio -- Peace can only be achieved by understanding - A. Einstein