From owner-freebsd-current@FreeBSD.ORG Tue May 5 23:15:22 2015 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3716A938 for ; Tue, 5 May 2015 23:15:22 +0000 (UTC) Received: from mail-ig0-x233.google.com (mail-ig0-x233.google.com [IPv6:2607:f8b0:4001:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0121A19C8 for ; Tue, 5 May 2015 23:15:22 +0000 (UTC) Received: by igblo3 with SMTP id lo3so115225998igb.1 for ; Tue, 05 May 2015 16:15:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=1GVVVz6cNJ9Mj+Qbwdb70cle93cVEt4k40BM53xJXos=; b=Rc3i/sS78yiBtLR+4h9nMAdNCnUCmXaswygNcyFTFpT85N3dYLllr/lKZ9baESmD+M qLARA/1YGwIGCbQvz6W3xOeM7i5VanQ3Xa+cFcQyCrH7Oz3FBQPopkcYp5Mbl4qHBeCZ ANwZiA7J9DfZaWVt+YnPSN6hwMVDO9qH+EYCZWeoQ2hofi/SE6R40tS+SZ8TYZuD/NDL lcAMJ1dHq/pRKXhH8yz7Ys5a+6pAPyA8NbP5RZXgrHD0j080N12n0fcfRMEiSrkSZ8li 4vXcjAkKUl6L5mcArXlkMukrWu7d9rFAkoyloiz0bhtl9Q3jN9qKlFBstSVGaPcVszj1 V7SA== MIME-Version: 1.0 X-Received: by 10.107.33.21 with SMTP id h21mr37247195ioh.1.1430867721402; Tue, 05 May 2015 16:15:21 -0700 (PDT) Received: by 10.107.40.194 with HTTP; Tue, 5 May 2015 16:15:21 -0700 (PDT) In-Reply-To: References: Date: Tue, 5 May 2015 19:15:21 -0400 Message-ID: Subject: Re: PCI PF memory decode disable when sizing VF BARs From: Ryan Stone To: Eric Badger Cc: "current@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 May 2015 23:15:22 -0000 On Tue, May 5, 2015 at 9:17 AM, Eric Badger wrote: > Hi Ryan and -current, > > During IOV config, when setting up VF bars, several calls are made to > 'pci_read_bar' (in sys/dev/pci/pci.c) in order to size VF BARs, which > causes memory decoding to be turned off temporarily for the PF associated > with those VFs. I'm finding that this can interfere with an already running > PF. > Ouch. That's a nasty bug. Thanks for tracking this down. > 1. Check the value of the 'reg' arg to 'pci_read_bar' and, if it is > outside a standard BAR range, don't disable memory decoding. This is > simple, but feels a little hackish and may have consequences I'm missing. > 2. Pass some flag/context through such that pci_read_bar knows it is > configuring VF BARs (we might instead disable VF MSE in this case, if it is > enabled). It would be necessary to carry this flag/context through several > function calls before reaching pci_read_bar, which might end up being ugly. > 3. Rearrange the calls so that VF BARs are sized when the PF is not yet > running, and that info saved until VFs are created. Probably it would be > done when the PF BARs are sized for any device supporting IOV, even if that > device never creates VFs. > I don't think that #3 is possible. Unfortunately the BAR is sized again when the BAR is allocated so it's difficult so it wouldn't be enough to just size the BAR during attach. I would have to reserve the memory space during attach, but that might reserve physical address space that other devices need to function. Actually, that problem will prevent #2 from being easy to implement too. We'd have to add an extra flag to pci_alloc_multi_resource. I think that #1 is the best option. There's already a precedent for something similar has it has a special base for the device ROM BAR. I haven't had a chance to test this yet, but I believe that this patch will solve the problem: https://people.freebsd.org/~rstone/patches/iov_bar.diff