From owner-freebsd-net@freebsd.org Sun Aug 21 17:33:07 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41E33BC0C09 for ; Sun, 21 Aug 2016 17:33:07 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x231.google.com (mail-io0-x231.google.com [IPv6:2607:f8b0:4001:c06::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0406E121B; Sun, 21 Aug 2016 17:33:07 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-io0-x231.google.com with SMTP id q83so91181534iod.1; Sun, 21 Aug 2016 10:33:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-transfer-encoding; bh=He0VSw3dYWnZNdfbDRlNPcCQw+OeRs8/0W623NsoEj8=; b=0RDulcZA2/AEolwWbEEVzW5sDtuS6EJemTL3kXjp33DqjILKALovKWLFAcwi4r5fsD s5nOO00rdie4kC8J01ZSjslWLud/6Nhh8ej7GvRhGo6h2xKa3gTip/YFz31WvnZc7bk3 P9BlAc6o+Zh7pTVwQo2UcClPepJCINOnh/XC+q4AEd/Qj2I/3PzYyGpioENU6yhI68Ue HppxfS6bqZLU5gVLI4gT2vW0QRzuC+GB1pEFPhB18ME/Wt0ndLm2dOuwRYt39JFDXHGR dlXWum6ms6sJ0pD9lDn9acaHfn+KW11AgRmLy2LEjUEymNDp124NclH+lp2a7iuBQCwP 0JqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-transfer-encoding; bh=He0VSw3dYWnZNdfbDRlNPcCQw+OeRs8/0W623NsoEj8=; b=lkQBdQQ5ozFVOnqJRPle4FT9YcxnKvNyg4088TEREOWNF90rRHPetrMb0koFdbXbtq posvPtkBP8HBYvUgtMRdTM+mOlmUexyR2auxPE/wkyz+V7bYNKBFg+EQrlkZGSmQzgAb gWx+PqoxnN9yGnCSI+Ejfshd4/GkVJuBZUwKrZzp8FvL4Yr51IQNmcNPjqWetgE/LkMQ 5lfKJxTy0PHsfhU8PU4JMPrncya24soGiVN8ap+31ptmCKHc0BJ0HCPewT08VDKnTCnV 6w/oWnNfFYgdD+bNBcP2WByxwbB+uBiqe5foXGAo5WnwJk9JGx0wUQgH9qqZz+WsB2pf gWEQ== X-Gm-Message-State: AEkooutP1229Xi0JaMGGb6M6ue4OqheI5aA0OvN8vSludwBWuZUDw2Typ2l74enxUnvnGTSywFKU4YyeeeCh+Q== X-Received: by 10.107.53.163 with SMTP id k35mr19996168ioo.75.1471800786122; Sun, 21 Aug 2016 10:33:06 -0700 (PDT) MIME-Version: 1.0 Sender: adrian.chadd@gmail.com Received: by 10.36.141.129 with HTTP; Sun, 21 Aug 2016 10:33:05 -0700 (PDT) In-Reply-To: <28FA9F29-FA29-4547-875D-0734DA120636@FreeBSD.org> References: <201608172021.u7HKLXJ4001584@repo.freebsd.org> <28FA9F29-FA29-4547-875D-0734DA120636@FreeBSD.org> From: Adrian Chadd Date: Sun, 21 Aug 2016 10:33:05 -0700 X-Google-Sender-Auth: iv_4XFCc2pEyZguBrdvNXsul3KU Message-ID: Subject: Re: svn commit: r304313 - head/sys/net To: "Robert N. M. Watson" Cc: "Andrey V. Elsukov" , FreeBSD Net Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Aug 2016 17:33:07 -0000 On 21 August 2016 at 07:42, Robert N. M. Watson wrote= : > On 20 Aug 2016, at 21:27, Adrian Chadd wrote: > >> I wonder if the right-er thing to do here is to allow the cpuid to be >> whatever it needs to be, but limit the cpuid lookups when it resolves >> to a netisr array. >> >> that'd mean the hybrid model would still return the current CPU up to >> the max CPU id, but anything trying to queue into a netisr would not >> overflow the netisr queue count. > > > I think some refinement of model is required. The original intent was tha= t =E2=80=9Cworkstreams=E2=80=9D represented ordered sets of queues being de= livered for deferred dispatch across the set of CPUs (hence =E2=80=9Cnws=E2= =80=9D being allocated using DPCPU). When performing direct dispatch, as a = matter of convenience, we also bill work performed on a CPU to its workstre= am: > > 1109 /* > 1110 * If direct dispatch is forced, then unconditionally dispa= tch > 1111 * without a formal CPU selection. Borrow the current CPU'= s stats, > 1112 * even if there's no worker on it. In this case we don't = update > 1113 * nws_flags because all netisr processing will be source o= rdered due > 1114 * to always being forced to directly dispatch. > 1115 */ > > In hybrid mode, the world is a little different, because we are willing t= o direct dispatch hybrid work *only* if the worker thread isn=E2=80=99t alr= eady processing the workstream: > > 1147 /*- > 1148 * We are willing to direct dispatch only if three conditio= ns hold: > 1149 * > 1150 * (1) The netisr worker isn't already running, > 1151 * (2) Another thread isn't already directly dispatching, a= nd > 1152 * (3) The netisr hasn't already been woken up. > 1153 */ > > This is important because if the workstream already has a backlog of work= , but the thread isn=E2=80=99t yet running, we must enqueue the work to ens= ure it remains ordered with respect to other work in the workstream. > > Assuming we wish to retain the current workstream model, then we need to = adapt the code a bit to handle the case where we still have one workstream = per CPU, but where we are only willing to perform deferred dispatch (or hyb= rid dispatch) to a CPU that has a worker thread. We would then bill to any = CPU=E2=80=99s workstream for true direct dispatch, but for hybrid dispatch,= continue to always bill a CPU that has a worker thread to which we were wi= lling to assign the work. > > I.e., as I think you=E2=80=99re suggesting, we probably need to tweak the= functions slightly to differentiate between =E2=80=9Cyou can run it here a= nd must use curcpu=E2=80=99s workstream=E2=80=9D and =E2=80=9Cyou can run i= t here but must use a specific CPU=E2=80=99s workstream=E2=80=9D =E2=80=94 = the former being true direct dispatch, and the latter being hybrid dispatch= where the netisr in question isn=E2=80=99t already running, but we bill it= to that workstream/netisr even though we run it on the current CPU. > > Does this make sense? Right. Let me go and look into it a little more. I think we may want to revert the change (which just landed to -11, so maybe revert that too) so I can test both of them out for correctness. Andrey, I'm sorry for suggesting we back it out, but I'd like to make sure we don't break networking on 11. :) Is that okay? I will look at this tonight/tomorrow. -adrian