From owner-freebsd-hackers@FreeBSD.ORG Fri Feb 5 14:31:49 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AD992106566C for ; Fri, 5 Feb 2010 14:31:49 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.freebsd.org (Postfix) with ESMTP id 7CE188FC36 for ; Fri, 5 Feb 2010 14:31:49 +0000 (UTC) Received: from [172.31.193.10] (rrcs-98-101-145-84.midsouth.biz.rr.com [98.101.145.84]) (authenticated bits=0) by duke.cs.duke.edu (8.14.2/8.14.2) with ESMTP id o15EVmWe007864 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 5 Feb 2010 09:31:48 -0500 (EST) X-DKIM: Sendmail DKIM Filter v2.8.3 duke.cs.duke.edu o15EVmWe007864 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=cs.duke.edu; s=mail; t=1265380309; bh=3WvF5yHGwNkBvaAsAHpgn23MW5YfCBVM0ogEfGdmPAQ=; h=Message-ID:Date:From:MIME-Version:To:CC:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=mB1V2cjQMZ+zrFxlivDfffRoO+v5f6p6WfjRTG1iVSvxZSfmJSVlJNRBfV4NuDv6U YkW7zUFHPoI7eqGk66V/71HVe4M7zr16v8W8WP2B6ZMCWMMxAyX4EY4cL3otDpWpod ktYzLQCGP/cOAqFukCYNUJdi3E82f1uHXkEE1cik= Message-ID: <4B6C2BCF.7090104@cs.duke.edu> Date: Fri, 05 Feb 2010 09:31:43 -0500 From: Andrew Gallatin User-Agent: Thunderbird 2.0.0.23 (X11/20090817) MIME-Version: 1.0 To: Kostik Belousov References: <4B6B30BC.7030107@cs.duke.edu> <20100205100643.GQ15587@deviant.kiev.zoral.com.ua> <4B6C225D.3020306@cs.duke.edu> <20100205140038.GR15587@deviant.kiev.zoral.com.ua> In-Reply-To: <20100205140038.GR15587@deviant.kiev.zoral.com.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: devfs panic w/INVARIANTS X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2010 14:31:49 -0000 Kostik Belousov wrote: > On Fri, Feb 05, 2010 at 08:51:25AM -0500, Andrew Gallatin wrote: >> Kostik Belousov wrote: >>> On Thu, Feb 04, 2010 at 03:40:28PM -0500, Andrew Gallatin wrote: >>>> I've got a commercial driver that uses device cloning. >>>> At unload time, the driver calls clone_cleanup(). When I unload >>>> the driver when the kernel is built with INVARIANTS, I'll see a >>>> panic in devfs_populate_loop(). This happens in 6-stable, >>>> as well as 8-stable. >>>> >>> >From what I can see the clone has been freed, but it >>>> remains on the devfs cdevp_list. Then the next time >>>> devfs_populate_loop() is called, it trips over the bad >>>> entry (cdp->cdp_dirents points to 0xdeadc0dedeadc0de) >>>> See appended kgdb session. >>>> >>>> If I trace the code path, it looks like clone_cleanup() >>>> calls destroy_devl(). And destroy_devl() will eventually >>>> call devfs_free() if the si_refcnt is zero. But I don't >>>> see anything which will get the cdev removed from >>>> the cdevp_list prior to it being freed. >>>> >>>> The only code I see which will get the cdev removed from >>>> the cdevp_list() seems to be the "GC any lingering devices" >>>> block in devfs_populate_loop >>>> >>>> What am I missing? >>> You did not mentioned it, but my guess is that you create clones from >>> the dev_clone event handler. Please note that devfs_lookup() that fires >> Yes, I do. >> >>> dev_clone event, consumes a device reference. Thus clone handlers shall >>> do dev_ref(). >>> >>> Due to races with cleanup, you should use MAKEDEV_REF flag for >>> make_dev_credv(9) KPI instead of doing make_dev()/dev_ref() pair. >> I need to support FreeBSD going all the way back to 6, so that's not an >> option in some versions. >> >> But, I'm talking about device removal time. If I call clone_cleanup() >> where the clones have dev->si_refcount==1, then I get the use-after-free >> panic. If I hack things to elevate the reference count (such that >> dev->si_refcount==2 when clone_cleanup() is called), then I don't >> get the panic. >> >> Are you saying I should have been taking the extra reference >> via my dev_clone eventhandler? Won't having the extra reference >> lead to a memory leak? Or am I just mis-reading the code, and >> this will lead to things being freed normally? > Yes, clone handler shall do dev_ref(). Either by doing race-free > make_dev_credf(MAKEDEV_REF) call, or by using dev_ref() after make_dev(). OK, cool. The man pages are handy. When I started this back in the FreeBSD 5 days, the man pages didn't exist :) >>> That said, do you really need clones at all ? >> I need to support FreeBSD back to 6.x, and I need to support the >> linux-like model of opening the "same" /dev/node multiple times >> and getting unique handles. So I think I need clones. > > Wouldn't it be cleaner to use cdevpriv for the 7/8/HEAD where it is > present ? And have special #ifdef-ed code for 6, that could be > eventually dropped. Yes, the cdevpriv() is a much cleaner interface. I'll probably add support for that soon. Thanks for the help, Drew