From owner-freebsd-net@FreeBSD.ORG Fri Oct 21 07:46:20 2005 Return-Path: X-Original-To: freebsd-net@FreeBSD.org Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CEB6016A421; Fri, 21 Oct 2005 07:46:20 +0000 (GMT) (envelope-from yar@comp.chem.msu.su) Received: from comp.chem.msu.su (comp.chem.msu.su [158.250.32.97]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7B0DC43D46; Fri, 21 Oct 2005 07:46:16 +0000 (GMT) (envelope-from yar@comp.chem.msu.su) Received: from comp.chem.msu.su (localhost [127.0.0.1]) by comp.chem.msu.su (8.13.3/8.13.3) with ESMTP id j9L7kApx001387; Fri, 21 Oct 2005 11:46:10 +0400 (MSD) (envelope-from yar@comp.chem.msu.su) Received: (from yar@localhost) by comp.chem.msu.su (8.13.3/8.13.3/Submit) id j9L7k7QD001386; Fri, 21 Oct 2005 11:46:07 +0400 (MSD) (envelope-from yar) Date: Fri, 21 Oct 2005 11:46:07 +0400 From: Yar Tikhiy To: Gleb Smirnoff , freebsd-net@FreeBSD.org, ru@FreeBSD.org, Andrew Thompson Message-ID: <20051021074606.GF69814@comp.chem.msu.su> References: <20051019102559.GA45909@heff.fud.org.nz> <20051020070054.GZ59364@cell.sick.ru> <20051020085721.GC27114@comp.chem.msu.su> <20051021053033.GW59364@cell.sick.ru> <20051021060654.GC69814@comp.chem.msu.su> <20051021064028.GZ59364@cell.sick.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20051021064028.GZ59364@cell.sick.ru> User-Agent: Mutt/1.5.9i Cc: Subject: Re: vlan patch X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Oct 2005 07:46:21 -0000 On Fri, Oct 21, 2005 at 10:40:28AM +0400, Gleb Smirnoff wrote: > On Fri, Oct 21, 2005 at 10:06:55AM +0400, Yar Tikhiy wrote: > Y> On Fri, Oct 21, 2005 at 09:30:33AM +0400, Gleb Smirnoff wrote: > Y> > On Thu, Oct 20, 2005 at 12:57:21PM +0400, Yar Tikhiy wrote: > Y> > Y> The hash code consists of literally a couple of #define's. And the > Y> > Y> difference between ng_vlan(4) and vlan(4) is that each ng_vlan node > Y> > Y> gets its own instance of the hash table. OTOH, in vlan(4) we need > Y> > Y> to decide if the hash table will be per parent interface or a single > Y> > Y> global instance. In the latter case we could hash by a combination > Y> > Y> of the VLAN tag and parent's ifindex. Perhaps this approach will > Y> > Y> yield more CPU cache hits during hash table lookups. In addition, > Y> > Y> it will be thriftier in using memory. Locking the global hash table > Y> > Y> should not be an issue as we can use an sx lock in this case for > Y> > Y> optimal read access. > Y> > > Y> > The sx lock is slow. We'd better use per interface hash, and thus > Y> > get locking instantly, with per-vlan lock. In other case, we will > Y> > acquire per-vlan lock + the sx lock on every packet. The sx lock > Y> > actually means mtx_lock+mtx_unlock, thus we will make 3 mutex > Y> > operations instead of one. > Y> > Y> OK, let's forget about sx locks. However, a per-interface hash is > Y> associated with a _physical_ interface, hence we must find the vlan > Y> to lock using the hash first. If there were a physical interface > Y> lock held by its driver in each case, it could protect the hash as > Y> well. Can we rely on this? > > Oops. When speaking about per-vlan lock, I really meant per-trunk lock. > > Or are you going to implement per-vlan lock? Is this going to be a benefit? > Since all packets on trunk are serialized by NIC driver, can there be any > benefit in creating a mutex per vlan interface, not per vlan trunk? I think you make a good point. Indeed, today 2 threads won't process at once 2 vlan packets coming from the same parent interface. The curious question is whether this state of matters will stay forever. The other approach would be to have separate threads for ip_input, ipfw, ip_output, and the rest of network stack functions, but it appears too inefficient in our case due to the overhead of context switches and locking. So I agree that locking vlans per trunk is better today. Do you think we should stick the vlan mutex into struct ifnet, too, along with the pointer to the vlan hash? Perhaps, it's time to separate vlan-related fields into a separate struct? -- Yar