From owner-freebsd-net@FreeBSD.ORG Tue Jul 26 13:13:39 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A943810657A6; Tue, 26 Jul 2011 13:13:39 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 5577C8FC15; Tue, 26 Jul 2011 13:13:38 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 4BDB47300A; Tue, 26 Jul 2011 15:12:15 +0200 (CEST) Date: Tue, 26 Jul 2011 15:12:15 +0200 From: Luigi Rizzo To: "Robert N. M. Watson" Message-ID: <20110726131215.GC6415@onelab2.iet.unipi.it> References: <20110714154457.GI70776@FreeBSD.org> <201107251300.20832.Daan@vitsch.nl> <1836450A-72C5-4796-811E-91C3974C1932@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1836450A-72C5-4796-811E-91C3974C1932@freebsd.org> User-Agent: Mutt/1.4.2.3i Cc: Daan Vreeken , gnn@freebsd.org, bz@freebsd.org, Ryan Stone , freebsd-net@freebsd.org Subject: Re: m_pkthdr.rcvif dangling pointer problem X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jul 2011 13:13:39 -0000 On Tue, Jul 26, 2011 at 10:09:09AM +0100, Robert N. M. Watson wrote: > > On 25 Jul 2011, at 12:00, Daan Vreeken wrote: > > > Couldn't the dangling pointer problem be solved by adding a 'generation' field > > to the mbuf structure? > > The 'generation' could be a system-wide number that gets incremented whenever > > an interface is removed. The mbuf* functions could keep a (per CPU?) > > reference count on the number of mbufs allocated/freed during > > that 'generation'. After interface removal, the ifnet structure could be > > freed when all the reference counters of generations before the current > > generation reach zero (whenever that happens). > > I think a hybrid approach makes sense, combining a number of the ideas we've been kicking about: > > (1) Add per-CPU ifnet refcounts that don't imply cache-line misses on each mbuf alloc/free > (2) Add optional subsystem drain functions so that subsystems that may have unbounded queueing times for mbufs deterministically ensure reference release, perhaps by substituting a common deadif for outstanding dying references. > > The former gives us actual correctness in terms of avoiding races, the latter gives us deterministic freeing by subsystems that potentially queue mbufs forever (i.e., TCP) but no longer require the ifnet reference. I'd like to suggest that before doing all this work we could try and see which subsystems have a real need to de-reference the reference, which fields they use, and how often. Because maybe just copying into the mbuf a blob of 8-16 bytes with useful info (a cookie, fib index, some flags, etc) could perhaps cover the majority of cases (in terms of usage frequency, not locations in the code) and let us deal with other cases by looking up the cookie in some data structure. As an example: - some functions just use rcvif to tell whether this is an incoming packet. No actual dereference; - others might only care that rcvif equals some other (already refcounted) value, so we don't have a race there. cheers luigi