From owner-freebsd-net@freebsd.org Sun Aug 21 14:42:21 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 53004BC1FF1 for ; Sun, 21 Aug 2016 14:42:21 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [198.74.231.69]) by mx1.freebsd.org (Postfix) with ESMTP id 36A761F90; Sun, 21 Aug 2016 14:42:21 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from [10.0.1.11] (host109-151-51-164.range109-151.btcentralplus.com [109.151.51.164]) by cyrus.watson.org (Postfix) with ESMTPSA id D056646BC2; Sun, 21 Aug 2016 10:42:19 -0400 (EDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: svn commit: r304313 - head/sys/net From: "Robert N. M. Watson" In-Reply-To: Date: Sun, 21 Aug 2016 15:42:26 +0100 Cc: "Andrey V. Elsukov" , FreeBSD Net Content-Transfer-Encoding: quoted-printable Message-Id: <28FA9F29-FA29-4547-875D-0734DA120636@FreeBSD.org> References: <201608172021.u7HKLXJ4001584@repo.freebsd.org> To: Adrian Chadd X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Aug 2016 14:42:21 -0000 On 20 Aug 2016, at 21:27, Adrian Chadd wrote: > I wonder if the right-er thing to do here is to allow the cpuid to be > whatever it needs to be, but limit the cpuid lookups when it resolves > to a netisr array. >=20 > that'd mean the hybrid model would still return the current CPU up to > the max CPU id, but anything trying to queue into a netisr would not > overflow the netisr queue count. I think some refinement of model is required. The original intent was = that =E2=80=9Cworkstreams=E2=80=9D represented ordered sets of queues = being delivered for deferred dispatch across the set of CPUs (hence = =E2=80=9Cnws=E2=80=9D being allocated using DPCPU). When performing = direct dispatch, as a matter of convenience, we also bill work performed = on a CPU to its workstream: 1109 /* 1110 * If direct dispatch is forced, then unconditionally = dispatch 1111 * without a formal CPU selection. Borrow the current = CPU's stats, 1112 * even if there's no worker on it. In this case we don't = update 1113 * nws_flags because all netisr processing will be source = ordered due 1114 * to always being forced to directly dispatch. 1115 */ In hybrid mode, the world is a little different, because we are willing = to direct dispatch hybrid work *only* if the worker thread isn=E2=80=99t = already processing the workstream: 1147 /*- 1148 * We are willing to direct dispatch only if three = conditions hold: 1149 * 1150 * (1) The netisr worker isn't already running, 1151 * (2) Another thread isn't already directly dispatching, = and 1152 * (3) The netisr hasn't already been woken up. 1153 */ This is important because if the workstream already has a backlog of = work, but the thread isn=E2=80=99t yet running, we must enqueue the work = to ensure it remains ordered with respect to other work in the = workstream. Assuming we wish to retain the current workstream model, then we need to = adapt the code a bit to handle the case where we still have one = workstream per CPU, but where we are only willing to perform deferred = dispatch (or hybrid dispatch) to a CPU that has a worker thread. We = would then bill to any CPU=E2=80=99s workstream for true direct = dispatch, but for hybrid dispatch, continue to always bill a CPU that = has a worker thread to which we were willing to assign the work. I.e., as I think you=E2=80=99re suggesting, we probably need to tweak = the functions slightly to differentiate between =E2=80=9Cyou can run it = here and must use curcpu=E2=80=99s workstream=E2=80=9D and =E2=80=9Cyou = can run it here but must use a specific CPU=E2=80=99s workstream=E2=80=9D = =E2=80=94 the former being true direct dispatch, and the latter being = hybrid dispatch where the netisr in question isn=E2=80=99t already = running, but we bill it to that workstream/netisr even though we run it = on the current CPU. Does this make sense? Robert=