From owner-freebsd-scsi@freebsd.org Mon Jun 11 18:28:50 2018 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 636F31004014 for ; Mon, 11 Jun 2018 18:28:50 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id DA247750E4 for ; Mon, 11 Jun 2018 18:28:49 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: by mailman.ysv.freebsd.org (Postfix) id 9B32B100400B; Mon, 11 Jun 2018 18:28:49 +0000 (UTC) Delivered-To: scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 790431004002 for ; Mon, 11 Jun 2018 18:28:49 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EED72750E2 for ; Mon, 11 Jun 2018 18:28:48 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [78.138.80.135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id w5BISjAU041137; Mon, 11 Jun 2018 20:28:45 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 56B07806; Mon, 11 Jun 2018 20:28:45 +0200 (CEST) Subject: =?UTF-8?Q?Re:_What_is_ENXIO_=e2=80=93_MSI_allocation_regression_in_?= =?UTF-8?Q?:[Was_Re:_svn_commit:_r321714_-_in_head/sys/dev:_mpr_mps]?= To: Scott Long Cc: scsi@freebsd.org References: <201707300653.v6U6rwLN099096@repo.freebsd.org> <597DA578.6030101@omnilan.de> <597F56A8.1060603@omnilan.de> <59804C8C.1020003@omnilan.de> <78611650-D7A4-4B1D-A254-DB058E1AC1C6@samsco.org> From: Harry Schmalzbauer Organization: OmniLAN Message-ID: <6e1e5f9f-4ece-dc9d-b059-08d52c9e6965@omnilan.de> Date: Mon, 11 Jun 2018 20:28:44 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Greylist: ACL 130 matched, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [78.138.80.130]); Mon, 11 Jun 2018 20:28:46 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: 78.138.80.135; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jun 2018 18:28:50 -0000 Am 05.06.2018 um 19:54 schrieb Scott Long: … >>>> Late in the 11.2 phase, I identified this commit as a regression for MSI (non-x) alloctaion. >>>> I have an idea what probably causes the problem here (INTx allocation, although MSI (and MSI-x) capability): >>>> disable_msix is not 0 (I need to disable MSI-x because of ESXi-passthru…). >>>> >>>> Corresponding lines: >>>> { >>>> device_t dev; >>>> int error, msgs; >>>> >>>> dev = sc->mps_dev; >>>> error = 0; >>>> msgs = 0; >>>> >>>> if ((sc->disable_msix == 0) && >>>> ((msgs = pci_msix_count(dev)) >= MPS_MSI_COUNT)) >>>> error = mps_alloc_msix(sc, MPS_MSI_COUNT); >>>> if ((error != 0) && (sc->disable_msi == 0) && >>>> ((msgs = pci_msi_count(dev)) >= MPS_MSI_COUNT)) >>>> error = mps_alloc_msi(sc, MPS_MSI_COUNT); >>>> if (error != 0) >>>> msgs = 0; >>>> >>>> sc->msi_msgs = msgs; >>>> return (error); >>>> } >>>> >>>> Before r321714, error was assigned ENXIO, which, if != 0, could help make me understand the problem. >>>> Unfortunately I have no idea what ENXIO means, where it's defined and most important, how to find the place where the declaration/definition happens. Only joe and vi available here, any hints highly appreciated. >>>> >>>> I can confirm that MSI allocation works with mps.ko_21.02.00.00-fbsd-r321415 with my ESXi-passthru-non_msi-x setup. >>>> Although the dirver emits no message that an MSI was allocated, like toher drivers do. That's a cosmetic one though. >>>> But the MSI->INTx regression is a severe one for me, which I'd like to fix myself but I'm missing so many fundamental skills :-( >>>> >>> Hi Harry, >>> You are correct about the bug. Please change the line at the top of the function that reads >>> error = 0; >>> to >>> error = ENXIO; >>> Let me know if that fixes the MSI problem for you. … >> BTW, does anybody have a link where I can get info about ENXIO? > > ENXIO means that the device is not available. I use it in the driver to signal when the hardware cannot be accessed. The manual page for error codes is “man errno" Oic, there's a man page :-) Haven't had time to look into it, but since you confirmed that ENXIO!=0, I simply changed that and now mps(4) allocates MSI again in my setup. For completeness my diff: Index: src/sys/dev/mps/mps_pci.c =================================================================== --- sys/dev/mps/mps_pci.c (Revision 334948) +++ sys/dev/mps/mps_pci.c (Arbeitskopie) @@ -244,7 +244,7 @@ int error, msgs; dev = sc->mps_dev; - error = 0; + error = ENXIO; msgs = 0; if ((sc->disable_msix == 0) && Unfortunately my other real problem persists – iSCSI sessions lock up the machine (11.2-RC2). No deadlock, since it will recover within some minutes, but otherwise a complete lock until iSCSI sessions time out, since no single ethernet/ip packet get's processed. Unfortunately I'm very short on testing resources here and don't know how to trace ctld/whatelse to find the lock-circle. So far I can only tell that it happens only with Server2016 iSCSI connections (using 4k block size). Will open a different thread/PR as soon as I found out anything… Thanks, -harry P.S.: I guess it's far too late to get that into 11.2?