From owner-svn-src-head@freebsd.org Sun Nov 24 16:45:47 2019 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DB7711B4069; Sun, 24 Nov 2019 16:45:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47LbdQ281kz4CYf; Sun, 24 Nov 2019 16:45:46 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id xAOGjbH5086102 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 24 Nov 2019 18:45:40 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua xAOGjbH5086102 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id xAOGjaGZ086101; Sun, 24 Nov 2019 18:45:36 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 24 Nov 2019 18:45:36 +0200 From: Konstantin Belousov To: Warner Losh Cc: Warner Losh , src-committers , svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: Re: svn commit: r355037 - head/sys/dev/pci Message-ID: <20191124164536.GC2707@kib.kiev.ua> References: <201911232343.xANNhqkQ097797@repo.freebsd.org> <20191124131010.GB2707@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.2 (2019-09-21) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 47LbdQ281kz4CYf X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; RCPT_COUNT_FIVE(0.00)[5]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(0.00)[ip: (-2.75), ipnet: 2001:470::/32(-4.63), asn: 6939(-3.51), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2019 16:45:47 -0000 On Sun, Nov 24, 2019 at 07:37:01AM -0700, Warner Losh wrote: > On Sun, Nov 24, 2019, 6:10 AM Konstantin Belousov > wrote: > > > On Sat, Nov 23, 2019 at 11:43:52PM +0000, Warner Losh wrote: > > > Author: imp > > > Date: Sat Nov 23 23:43:52 2019 > > > New Revision: 355037 > > > URL: https://svnweb.freebsd.org/changeset/base/355037 > > > > > > Log: > > > Push Giant down one layer > > > > > > The /dev/pci device doesn't need GIANT, per se. However, one routine > > > that it calls, pci_find_dbsf implicitly does. It walks a list that can > > > change when PCI scans a new bus. With hotplug, this means we could > > > have a race with that scanning. To prevent that, take out Giant around > > > scanning the list. > > > > > > However, given that we have places in the tree that drop giant, if > > > held when we call into them, the whole use of Giant to protect newbus > > > may be less effective that we desire, so add a comment about why we're > > > talking it out, and we'll address the issue when we lock newbus with > > > something other than Giant. > > > > > > Modified: > > > head/sys/dev/pci/pci.c > > > head/sys/dev/pci/pci_user.c > > > > > > Modified: head/sys/dev/pci/pci.c > > > > > ============================================================================== > > > --- head/sys/dev/pci/pci.c Sat Nov 23 23:41:21 2019 (r355036) > > > +++ head/sys/dev/pci/pci.c Sat Nov 23 23:43:52 2019 (r355037) > > > @@ -445,18 +445,21 @@ pci_find_bsf(uint8_t bus, uint8_t slot, uint8_t > > func) > > > device_t > > > pci_find_dbsf(uint32_t domain, uint8_t bus, uint8_t slot, uint8_t func) > > > { > > > - struct pci_devinfo *dinfo; > > > + struct pci_devinfo *dinfo = NULL; > > > > > > + /* Giant because newbus is Giant locked revisit with newbus > > locking */ > > > + mtx_lock(&Giant); > > > STAILQ_FOREACH(dinfo, &pci_devq, pci_links) { > > > if ((dinfo->cfg.domain == domain) && > > > (dinfo->cfg.bus == bus) && > > > (dinfo->cfg.slot == slot) && > > > (dinfo->cfg.func == func)) { > > > - return (dinfo->cfg.dev); > > > + break; > > > } > > > } > > > + mtx_unlock(&Giant); > > > > > > - return (NULL); > > > + return (dinfo != NULL ? dinfo->cfg.dev : NULL); > > I do not think this change is correct. If the parallel hotplug, or > > rather, hot-unplug event occurs, then dinfo potentially becomes invalid > > right after the Giant unlock, which makes both this function and its > > callers to access freed memory. Having caller to lock a newbus lock > > around both the call and consumption of the returned data is required. > > > There are many data lifetime issues. If anything the PCI user device calls > drops Giant and then picks it back up again we are in the same boat... I > totally agree this is a bad situation, but can only really be fixed by > locking newbus with a different lock than Giant and likely using some kind > of reference count for device_t that are handed out... > > In the mean time, I'll move giant back up into the ioctl routine and hope > it isn't dropped by things it calls..m I think we can start at least by marking the Giant acqusitions that are related to newbus. I never saw anybody talking publically why the naive translation of newbus Giant into a sleepable lock, e.g. sx in exclusive mode, cannot work. From my memory, one of the big issues is that many sleeps done at probe/attach, need to drop the newbus lock.