From owner-freebsd-current@FreeBSD.ORG Wed May 6 16:49:40 2015 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5614D2FD; Wed, 6 May 2015 16:49:40 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 302D61E8F; Wed, 6 May 2015 16:49:40 +0000 (UTC) Received: from ralph.baldwin.cx (pool-173-54-116-245.nwrknj.fios.verizon.net [173.54.116.245]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 0D9F0B9B3; Wed, 6 May 2015 12:49:39 -0400 (EDT) From: John Baldwin To: freebsd-current@freebsd.org Cc: Ryan Stone , Eric Badger , "current@freebsd.org" Subject: Re: PCI PF memory decode disable when sizing VF BARs Date: Wed, 06 May 2015 11:45:52 -0400 Message-ID: <11092809.7nmbPfKl0V@ralph.baldwin.cx> User-Agent: KMail/4.14.2 (FreeBSD/10.1-STABLE; KDE/4.14.2; amd64; ; ) In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 06 May 2015 12:49:39 -0400 (EDT) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2015 16:49:40 -0000 On Tuesday, May 05, 2015 07:15:21 PM Ryan Stone wrote: > On Tue, May 5, 2015 at 9:17 AM, Eric Badger wrote: > > > Hi Ryan and -current, > > > > During IOV config, when setting up VF bars, several calls are made to > > 'pci_read_bar' (in sys/dev/pci/pci.c) in order to size VF BARs, which > > causes memory decoding to be turned off temporarily for the PF associated > > with those VFs. I'm finding that this can interfere with an already running > > PF. > > > > Ouch. That's a nasty bug. Thanks for tracking this down. > > > > 1. Check the value of the 'reg' arg to 'pci_read_bar' and, if it is > > outside a standard BAR range, don't disable memory decoding. This is > > simple, but feels a little hackish and may have consequences I'm missing. > > > 2. Pass some flag/context through such that pci_read_bar knows it is > > configuring VF BARs (we might instead disable VF MSE in this case, if it is > > enabled). It would be necessary to carry this flag/context through several > > function calls before reaching pci_read_bar, which might end up being ugly. > > > 3. Rearrange the calls so that VF BARs are sized when the PF is not yet > > running, and that info saved until VFs are created. Probably it would be > > done when the PF BARs are sized for any device supporting IOV, even if that > > device never creates VFs. > > > > I don't think that #3 is possible. Unfortunately the BAR is sized again > when the BAR is allocated so it's difficult so it wouldn't be enough to > just size the BAR during attach. I would have to reserve the memory space > during attach, but that might reserve physical address space that other > devices need to function. > > Actually, that problem will prevent #2 from being easy to implement too. > We'd have to add an extra flag to pci_alloc_multi_resource. I think that > #1 is the best option. There's already a precedent for something similar > has it has a special base for the device ROM BAR. > > I haven't had a chance to test this yet, but I believe that this patch will > solve the problem: > > https://people.freebsd.org/~rstone/patches/iov_bar.diff There are some devices with BARs in non-standard locations. :( If there is a flag to just disable the VF bar decoding, then ideally we should just be doing that and leaving the global decoding flag alone while sizing the VF BAR. -- John Baldwin