From owner-svn-src-all@FreeBSD.ORG Thu Jan 22 05:26:37 2015 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A5060956 for ; Thu, 22 Jan 2015 05:26:37 +0000 (UTC) Received: from mail-ob0-f180.google.com (mail-ob0-f180.google.com [209.85.214.180]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 671E19BA for ; Thu, 22 Jan 2015 05:26:37 +0000 (UTC) Received: by mail-ob0-f180.google.com with SMTP id uz6so28541196obc.11 for ; Wed, 21 Jan 2015 21:26:36 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=2RCmvLyW7SEixhwB/8BhFfcLXIySteIgbdvYcMZ2N98=; b=T3wgfjlYiKZWVxJrbAayLY1hd0PVlldFdWP0VzgqxXKSQBMqX5hvLNeSFvFlgdBCiA UcpRCgroK58l9TYj1A9bEMISKfD/CEY4QqhVSHw+r0GwBLhAFZ5k7lmd8szs5bA+mKkG +RKSz0KNo/s//wriWy3LnQzeOJFgNnIYRjKSzp8XNS+4wQ35aA7hUbOXfYteH6Zh+1Q0 M4sFC99q0BqUCGO/TLF2UwNldHt26ASE5SpnxfNRVdKAWcQKau5b1slFtralcQ+1MgkT iIO0j5ScR0zFAer+eM2diUTJT/HJpP1UJGtbinlbVn/0y7by0VMFd1xxn5KlYfBbGduY cpHA== X-Gm-Message-State: ALoCoQlK9yzU04C83uOXISBdd8T7XglnfwAPxRtH1JEA4GZPolkjeupvsKNY7JXSbtVvd32aA55r X-Received: by 10.60.133.141 with SMTP id pc13mr27616787oeb.68.1421904396587; Wed, 21 Jan 2015 21:26:36 -0800 (PST) Received: from [192.168.43.169] ([172.56.8.11]) by mx.google.com with ESMTPSA id d198sm4759185oih.12.2015.01.21.21.26.35 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 21 Jan 2015 21:26:36 -0800 (PST) Sender: Warner Losh Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) Subject: Re: svn commit: r277213 - in head: share/man/man9 sys/kern sys/ofed/include/linux sys/sys From: Warner Losh In-Reply-To: <54BE21F0.6010602@selasky.org> Date: Wed, 21 Jan 2015 22:26:33 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <7C692107-51CF-4DFA-BD6C-623D56893150@bsdimp.com> References: <201501151532.t0FFWV2Y037455@svn.freebsd.org> <54BDD9E1.6090505@selasky.org> <20150120075126.GA42409@kib.kiev.ua> <54BE0AAA.4050104@selasky.org> <20150120090057.GD42409@kib.kiev.ua> <54BE21F0.6010602@selasky.org> To: Hans Petter Selasky X-Mailer: Apple Mail (2.1993) Cc: Konstantin Belousov , "svn-src-head@freebsd.org" , Adrian Chadd , "src-committers@freebsd.org" , "svn-src-all@freebsd.org" X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jan 2015 05:26:37 -0000 > On Jan 20, 2015, at 2:37 AM, Hans Petter Selasky = wrote: >=20 > On 01/20/15 10:00, Konstantin Belousov wrote: >> On Tue, Jan 20, 2015 at 08:58:34AM +0100, Hans Petter Selasky wrote: >>> On 01/20/15 08:51, Konstantin Belousov wrote: >>>> On Tue, Jan 20, 2015 at 05:30:25AM +0100, Hans Petter Selasky = wrote: >>>>> On 01/19/15 22:59, Adrian Chadd wrote: >>>>>> Hi, >>>>>>=20 >>>>>> Would you please check what the results of this are with CPU = specific >>>>>> callwheels? >>>>>>=20 >>>>>> I'm doing some 10+ gig traffic testing on -HEAD with RSS enabled = (on >>>>>> ixgbe) and with this setup, the per-CPU TCP callwheel stuff is >>>>>> enabled. But all the callwheels are now back on clock(0) and so = is the >>>>>> lock contention. :( >>>>>>=20 >>>>>> Thanks, >>>>>>=20 >>>>>=20 >>>>> Hi, >>>>>=20 >>>>> Like stated in the manual page, callout_reset_curcpu/on() does not = work >>>>> with MPSAFE callouts any more! >>>> I.e. you 'fixed' some undeterminate bugs in callout migration by = not >>>> doing migration at all anymore. >>>>=20 >>>>>=20 >>>>> You need to use callout_init_{mtx,rm,rw} and remove the custom = locking >>>>> inside the callback in the TCP stack to get it working like = before! >>>>=20 >>>> No, you need to do this, if you think that whole callout KPI must = be >>>> rototiled. It is up to the person who modifies the KPI, to ensure = that >>>> existing code is not broken. >=20 > Hi, >=20 > It is not very hard to update existing callout clients and you can do = it too, if you need the extra bits of performance. >=20 > Are there more API's than the TCP stack which you think needs an = update and are performance critical? >=20 >>>>=20 >>>> As I understand, currently we are back to the one-cpu callouts. >>>> Do other people consider this situation acceptable ? >=20 > For the TCP stack - yes, but not for other clients like cv_timedwait() = and such. >=20 > If you think you have a better way to solve the callout problems, = please tell me! In order for a callout to change its CPU you need a lock = to protect which CPU the callout is on. Instead of introducing a third = lock in the callout path, which will be a congestion point, to protect = against changing the CPU number, I decided that we will use the client's = mutex and the MPSAFE implies the client doesn't have any mutex. So it = won't work with callout clients which use the CALLOUT_MPSAFE flag. = Honestly CALLOUT_MPSAFE should not be used, because it leads to extra = complexity in the clients catching the race when tearing down the = callouts and any pending callbacks. Then it is incumbent on you to fix them. You can=92t just fix one = instance and wash your hands of the problem. Maybe this is a real and legitimate bug. However, until you=92ve = followed your solution through by actually fixing the abusers of it, my = confidence that another issue won=92t present itself is quite low. The = code seems half baked to me. And from reading this thread, it seems like = perhaps I=92m not the only one. >>> Please read the callout 9 manual page first. >>=20 >> Assume I read it. How this changes any of my points above ? >> """ >> A change in the CPU selection cannot happen if this function is >> re-scheduled inside a callout function. Else the callback function = given >> by the func argument will be executed on the same CPU like previously >> done. >> """ >> You cannot do this without fixing consumers. >>=20 >=20 > The code simply needs an update. It is not broken in any ways - right? = If it is not broken, fixing it is not that urgent. Radically changing the performance characteristics is breaking the code. = Performance regression in the TCP stack is urgent to fix. Not being able = to enumerate what all the consumers are that use this and provide an = analysis about why they aren=92t important to fix is a bug in your = process, and in your interaction with the project. We simply do not = operate that way. Warner=