From owner-dev-commits-src-main@freebsd.org Thu Apr 8 19:45:08 2021 Return-Path: Delivered-To: dev-commits-src-main@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C7E0A5BD579; Thu, 8 Apr 2021 19:45:08 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-qt1-x830.google.com (mail-qt1-x830.google.com [IPv6:2607:f8b0:4864:20::830]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FGWv84fHKz4WJJ; Thu, 8 Apr 2021 19:45:08 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: by mail-qt1-x830.google.com with SMTP id h7so2403863qtx.3; Thu, 08 Apr 2021 12:45:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:to:references:from:autocrypt:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=qvjnpIVRvNYI1v/VkpsN9uFXafOG2umrY9fYzqNFE5U=; b=D6R/uYJxDmlFBHkyORoOGoKREueO9q7ojpEM088uoS7bF8vS4bBzKboCV6vUjJxkXS LEuaYWRxd3Wq/y7AMRyrsR2FFvi95LRaED58ebPwShs5qH4hbx1cMXxbB5w7c+XwHAKM MxdYILvIteuYG69kGqjAe7ERmLqdu7v6IzFUBMyif7Gfn/8zxHZBkxFg+xno1go/OvPh F5GclqZ5J05+6Qq3KzsbBBxPrQw9eQL4jl0EIzHjoAukcgZdt0t9Mz98lUyYaqoS724p hkMg4JBkIRUM+YuGMGfL11W7lZMNAXA4+z8wg1OBzyA7XCWRZKibhgetFPYtzALRtXKm WjKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:to:references:from:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=qvjnpIVRvNYI1v/VkpsN9uFXafOG2umrY9fYzqNFE5U=; b=kA8IC/qTqkU12RdaFmVunMX/0H/iQ+vXNzMi150OLhLfCgn4R9rhcsQiWfHHEseaQ0 7HYJ7LWtUQQniUW+UAwA/MNlSpOkEAgdhr3LL3qjOTpobjd/IeoFMOttCp2A1V2Ud3Rd kzovNJSKNSSbbcaFWWLpaKYw6bK3DBjIHPT0Ak7zOOtwVGS23rFNLYcCpiHvC8YvM6v2 QIqWdhtbeMcQeKKvVdg+qHv0smYdCv+nUCBWf47hIH+xYZYHdF1GD6LOCUL5cgVTovy7 zJwyTl9CoP+6jsuglEj5x/D9RtFbV3eu1iVUSgZGVGu3ZET1CqUbptywHWhL3VpkKyd6 Gtog== X-Gm-Message-State: AOAM533/QVa8MSzD0bsxyPluzjJNbaLEuKfiJzvhTrjcbwLm3Ivz7DYr wogWfen+DIyLeYlsp+6fXNSTtAD1/gjrZQ== X-Google-Smtp-Source: ABdhPJw1Hp7dpOi+/Js+yr7DF9gPE2vOvbSqb+oTIOmvk7Qo7nYWIG0JtOQP0WBxHU4BM2kj2mpn9A== X-Received: by 2002:ac8:7381:: with SMTP id t1mr9345340qtp.3.1617911106766; Thu, 08 Apr 2021 12:45:06 -0700 (PDT) Received: from mavoffice.ixsystems.com ([38.32.73.2]) by smtp.gmail.com with ESMTPSA id j30sm318535qtv.90.2021.04.08.12.45.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 08 Apr 2021 12:45:06 -0700 (PDT) Sender: Alexander Motin Subject: Re: git: 5a898b2b78ce - main - Set PCIe device's Max_Payload_Size to match PCIe root's. To: John Baldwin , src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org References: <202104051440.135EeeTZ095177@gitrepo.freebsd.org> <4091ea80-5269-9c78-9fe6-6bba2ed85fbd@FreeBSD.org> <6239ed61-4d35-2cfa-69a7-16da733d8091@FreeBSD.org> From: Alexander Motin Autocrypt: addr=mav@FreeBSD.org; prefer-encrypt=mutual; keydata= mQENBFOzxAwBCADkPrax0pI2W/ig0CK9nRJJwsHitAGEZ2HZiFEuti+6/4UVxj81yr4ak/4g 9bKUyC7rMEAp/ZHNhd+MFCPAAcHPvtovnfykqE/vuosCS3wlSLloix2iKVLks0CwbLHGAyne 46lTQW74Xl/33c3W1Z6d8jD9gVFT/xaVzZ0U9xdzOmsYAZaAj4ki0tuxO9F7L+ct9grRe7iP g8t9hai7BL4ee3VRwk2JXnKb7UvBiVITKYWKz1jRvZIrjPokgEcCLOSlv7x/1kjuFnj3xWZU 7HSFFT8J93epBbrSSCsYsppIk2fZH41kaaFXsMQfTPH8wkeM6qwrvOh4HiQM08R+9tThABEB AAG0IUFsZXhhbmRlciBNb3RpbiA8bWF2QEZyZWVCU0Qub3JnPokBVwQTAQoAQQIbAwULCQgH AwUVCgkICwUWAwIBAAIeAQIXgAIZARYhBOmM88TmnMPNDledVYMYw5VbqyJ/BQJZYMKuBQkN McyiAAoJEIMYw5VbqyJ/tuUIAOG3ONOSNYqjK4eTZ1TVh9jdUBAhWk5nhDFnODN49Wj0AbYm 7aIqy8O1hnCDSZG5LttjSAo3UfXJZDKQM0BLb0gpRMBnAYqO6tdolLNqAbPGJBnGoPjsh24y 6KcbDaNnis+lD4GwPXwQM+92wZGhCUFElPV9NciZGVS65TNIgk7X+yEjjhD1MSWKKijZ1r9Z zIt4OzUTxxNOvzdlABZS88nNRdJkatOQJPmFdd1mpP6UzTNCiLUo1pIqOEtJgvVVDYq5WHY6 tciWWYdmZG/tIBexJmv2mV2OLVjXR6ZeKmntVH14H72/wRHJuYHQC+r5SVRcWWayrThsY6jZ Yr4+raS5AQ0EU7PEDAEIAOZgWf2cJIu+58IzP2dkXE/urj3tr4OqrB/yHGWUf71Lz6D0Fi6Z AXgDtmcFLGPfMyWuLAvSM+xmoguk7zC4hRBYvQycmIhuqBq1jO1Wp/Z+lpoPM/1cDYLn8Flv mI/c40MhUZh345DA4jYWWaZNjQHUWVQ1fPf595vdVVMPT/abE8E5DaF6fSkRmqFTmfYRkfbt 3ytU8NdUapDcJVY7cEP2nJBVNZPnOIObR/ZIgSxjjrG5o34yXoqeup8JvwEv+/NylzzuyXEZ R1EdEIzQ/a1nh/0j4NXtzZEqKW4aTWlmSqb6wN8jh1OSOOqkYsfnE3nfxcZbxi4IRoNQYlm5 9R8AEQEAAYkBPAQYAQoAJgIbDBYhBOmM88TmnMPNDledVYMYw5VbqyJ/BQJZYMLYBQkNMczM AAoJEIMYw5VbqyJ/TqgH/RQHClkvecE0262lwKoP/m0Mh4I5TLRgoJJn8S7G1BnqohYJkiLq A6xe6urGD7OqdNAl12UbrjWbdJV+zvea3vJoM4MZuYiYrGaXWxzFXqWJcPwMU9sAh8MRghHu uC5vgPb45Tnftw9/+n0i8GfVhQhOqepUGdQg4NPcXviSkoAvig6pp9Lcxisn0groUQKt15Gc sS9YcQWg3j9Hnipc6Mu416HX98Fb113NHJqc2geTHLkRyuBFOoyIqB6N9GKjzOAIzxxsVdl9 TevwGsrp4M4/RFzWbSgsbOnbE7454lmuVZGfReEjnUm8RHp9Q2UWKXlp3exlZjvOp/uVEpCg lz65AQ0EU7PEDAEIAOZgWf2cJIu+58IzP2dkXE/urj3tr4OqrB/yHGWUf71Lz6D0Fi6ZAXgD tmcFLGPfMyWuLAvSM+xmoguk7zC4hRBYvQycmIhuqBq1jO1Wp/Z+lpoPM/1cDYLn8FlvmI/c 40MhUZh345DA4jYWWaZNjQHUWVQ1fPf595vdVVMPT/abE8E5DaF6fSkRmqFTmfYRkfbt3ytU 8NdUapDcJVY7cEP2nJBVNZPnOIObR/ZIgSxjjrG5o34yXoqeup8JvwEv+/NylzzuyXEZR1Ed EIzQ/a1nh/0j4NXtzZEqKW4aTWlmSqb6wN8jh1OSOOqkYsfnE3nfxcZbxi4IRoNQYlm59R8A EQEAAYkBPAQYAQoAJgIbDBYhBOmM88TmnMPNDledVYMYw5VbqyJ/BQJZYMLYBQkNMczMAAoJ EIMYw5VbqyJ/TqgH/RQHClkvecE0262lwKoP/m0Mh4I5TLRgoJJn8S7G1BnqohYJkiLqA6xe 6urGD7OqdNAl12UbrjWbdJV+zvea3vJoM4MZuYiYrGaXWxzFXqWJcPwMU9sAh8MRghHuuC5v gPb45Tnftw9/+n0i8GfVhQhOqepUGdQg4NPcXviSkoAvig6pp9Lcxisn0groUQKt15GcsS9Y cQWg3j9Hnipc6Mu416HX98Fb113NHJqc2geTHLkRyuBFOoyIqB6N9GKjzOAIzxxsVdl9Tevw Gsrp4M4/RFzWbSgsbOnbE7454lmuVZGfReEjnUm8RHp9Q2UWKXlp3exlZjvOp/uVEpCglz4= Message-ID: Date: Thu, 8 Apr 2021 15:45:05 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.8.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4FGWv84fHKz4WJJ X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: dev-commits-src-main@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for the main branch of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Apr 2021 19:45:08 -0000 On 08.04.2021 15:26, John Baldwin wrote: > On 4/8/21 11:02 AM, Alexander Motin wrote: >> On 08.04.2021 13:36, John Baldwin wrote: >>> On 4/5/21 7:40 AM, Alexander Motin wrote: >>>> +    rmps = pcie_read_config(root, PCIER_DEVICE_CTL, 2) & >>>> +        PCIEM_CTL_MAX_PAYLOAD; >>>> +    mps = pcie_read_config(dev, PCIER_DEVICE_CTL, 2) & >>>> +        PCIEM_CTL_MAX_PAYLOAD; >>>> +    if (mps == rmps) >>>> +        return; >>>> +    /* Check whether the device is capable of the root's MPS. */ >>>> +    mmps = (pcie_read_config(dev, PCIER_DEVICE_CAP, 2) & >>>> +        PCIEM_CAP_MAX_PAYLOAD) << 5; >>>> +    if (rmps > mmps) { >>>> +        /* >>>> +         * The device is unable to handle root's MPS.  Limit root. >>>> +         * XXX: We should traverse through all the tree, applying >>>> +         * it to all the devices. >>>> +         */ >>> >>> Hmmm, limiting the root seems very dubious here.  Do you really need >>> this? >> >> All devices under the same root (at least ones that talk to each other) >> must have the same MPS value, otherwise some may consider larger >> transfer as an error.  In case of direct PCIe connection the root is the >> only other device, so this code should be sufficient. > > I mean, had you seen any cases where you needed to adjust the root? No. I don't think I have NVMe device incapable of 256 byte transfers to which my Supermicro boards BIOS default. But I can easily imagine either such device or BIOS with bigger default. > I > do worry about this breaking other systems due to it not doing the walk > of the full sub-tree.  Maybe only do it if the root port is the grandparent > of the device in question since that is the safe case here and punt if > there are switches in the middle? I have no problem limiting it to grandparent, if you prefer, just not sure it is much better. If there are more devices under that root, then without full tree traversal either some old or the new device get MPS mismatch. The only question whether it will cause a receive of too large transfer by the root capable of it but limited by the MPS setting (formally a protocol violation, but at least Intel Optane 905p seem to ignore it), or the new device that is not even capable of it. >>> If not, I'd put it behind a tunable sysctl that defaults to off.  >>> Ideally >>> what you'd do here is use an and of the two masks and select one of >>> those >>> bits to choose the value rather than assuming the root can do the >>> device's >>> value. >> >> It is not a bitmask, it is a power-of-2 between 128 bytes and the >> maximum device capability.  So if the root is already configured for >> higher value, then it must support the lover one too. > > I thought the DEVICE_CAP is a bitmask, and that is what I was thinking > of using > the and with.  Nominally I think we should be doing the AND of the > DEVICE_CAP > field for all the devices under a port and then programming them to some > value > remaining in the mask.  Most devices just DMA to/from memory though > rather than > to each other, so generally you just care about the path from a device > to a root > port. "Max_Payload_Size Supported – This field indicates the maximum payload size that the Function can support for TLPs." -- this field in DEVICE_CAP is not a bitmask. So nominally we should take minimum for all the devices. -- Alexander Motin