Date: Tue, 5 Jun 2018 11:54:32 -0600 From: Scott Long <scottl@samsco.org> To: Harry Schmalzbauer <freebsd@omnilan.de> Cc: scsi@freebsd.org Subject: =?utf-8?Q?Re=3A_What_is_ENXIO_=E2=80=93_MSI_allocation_regression?= =?utf-8?Q?_in_=3A=5BWas_Re=3A_svn_commit=3A_r321714_-_in_head/sys/dev=3A_?= =?utf-8?Q?mpr_mps=5D?= Message-ID: <A2AECF9F-2BB1-4FC0-8330-658336A3A4F0@samsco.org> In-Reply-To: <d99e383d-b09a-f3bd-f1e2-a6a808016347@omnilan.de> References: <201707300653.v6U6rwLN099096@repo.freebsd.org> <597DA578.6030101@omnilan.de> <597F56A8.1060603@omnilan.de> <D18DFAD4-6E93-4AE2-BE15-EFF4D8ABCB2A@samsco.org> <59804C8C.1020003@omnilan.de> <e7d94e6a-89e8-ffa1-40da-7fb67e6bfc2b@omnilan.de> <78611650-D7A4-4B1D-A254-DB058E1AC1C6@samsco.org> <d99e383d-b09a-f3bd-f1e2-a6a808016347@omnilan.de>
next in thread | previous in thread | raw e-mail | index | archive | help
> On Jun 5, 2018, at 1:18 AM, Harry Schmalzbauer <freebsd@omnilan.de> wrote:
>
> Am 05.06.2018 um 00:22 schrieb Scott Long:
>>> On Jun 4, 2018, at 4:51 AM, Harry Schmalzbauer <freebsd@omnilan.de> wrote:
>>>
>>> Am 01.08.2017 um 11:40 schrieb Harry Schmalzbauer:
>>>> Bezüglich Scott Long's Nachricht vom 31.07.2017 18:56 (localtime):
>>>>
>>>> …
>>>>>> I'd like to report one I hadn't expected:
>>>>>>
>>>>>> mps0: <Avago Technologies (LSI) SAS2008> port 0x4000-0x40ff mem 0xc3bc0000-0xc3bc3fff,0xc3b80000-0xc3bbffff irq 19 at device 0.0 on pci7
>>>>>>
>>>>>> mps0: Firmware: 20.00.04.00, Driver: 21.02.00.00-fbsd
>>>>>> mps0: IOCCapabilities:
>>>>>> 185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR>
>>>>>> mps0: Cannot allocate INTx interrupt
>>>>>> mps0: mps_iocfacts_allocate failed to setup interrupts
>>>>>> mps0: mps_attach IOC Facts based allocation failed with error 6
>>>>>> panic: resource_list_release: resource entry is not busy
>>>>>> cpuid = 6
>>>>>> KDB: stack backtrace:
>>>>>> #0 0xffffffff805e32d7 at kdb_backtrace+0x67
>>>>>> #1 0xffffffff805a1d26 at vpanic+0x186
>>>>>> #2 0xffffffff805a1b93 at panic+0x43
>>>>>> #3 0xffffffff805d71c6 at resource_list_release+0x1c6
>>>>>> #4 0xffffffff8040fef1 at mps_pci_free+0xe1
>>>>>> #5 0xffffffff8040fa23 at mps_pci_attach+0x1b3
>>>>>> #6 0xffffffff805d6594 at device_attach+0x3a4
>>>>>> #7 0xffffffff805d774d at bus_generic_attach+0x3d
>>>>>> #8 0xffffffff8044ac05 at pci_attach+0xd5
>>>>>> #9 0xffffffff805d6594 at device_attach+0x3a4
>>>>>> #10 0xffffffff805d774d at bus_generic_attach+0x3d
>>>>>> #11 0xffffffff80364761 at acpi_pcib_pci_attach+0xa1
>>>>>> #12 0xffffffff805d6594 at device_attach+0x3a4
>>>>>> #13 0xffffffff805d774d at bus_generic_attach+0x3d
>>>>>> #14 0xffffffff8044ac05 at pci_attach+0xd5
>>>>>> #15 0xffffffff805d6594 at device_attach+0x3a4
>>>>>> #16 0xffffffff805d774d at bus_generic_attach+0x3d
>>>>>> #17 0xffffffff80363e4d at acpi_pcib_acpi_attach+0x42d
>>>>>> Uptime: 1s
>>>> …
>>>>
>>>>> Fixed in r321799, thanks for the report.
>>>> Fix confiremd; merged together with r321733 (and 321737) to 11.1 and
>>>> panic vanished.
>>>
>>> Late in the 11.2 phase, I identified this commit as a regression for MSI (non-x) alloctaion.
>>> I have an idea what probably causes the problem here (INTx allocation, although MSI (and MSI-x) capability):
>>> disable_msix is not 0 (I need to disable MSI-x because of ESXi-passthru…).
>>>
>>> Corresponding lines:
>>> {
>>> device_t dev;
>>> int error, msgs;
>>>
>>> dev = sc->mps_dev;
>>> error = 0;
>>> msgs = 0;
>>>
>>> if ((sc->disable_msix == 0) &&
>>> ((msgs = pci_msix_count(dev)) >= MPS_MSI_COUNT))
>>> error = mps_alloc_msix(sc, MPS_MSI_COUNT);
>>> if ((error != 0) && (sc->disable_msi == 0) &&
>>> ((msgs = pci_msi_count(dev)) >= MPS_MSI_COUNT))
>>> error = mps_alloc_msi(sc, MPS_MSI_COUNT);
>>> if (error != 0)
>>> msgs = 0;
>>>
>>> sc->msi_msgs = msgs;
>>> return (error);
>>> }
>>>
>>> Before r321714, error was assigned ENXIO, which, if != 0, could help make me understand the problem.
>>> Unfortunately I have no idea what ENXIO means, where it's defined and most important, how to find the place where the declaration/definition happens. Only joe and vi available here, any hints highly appreciated.
>>>
>>> I can confirm that MSI allocation works with mps.ko_21.02.00.00-fbsd-r321415 with my ESXi-passthru-non_msi-x setup.
>>> Although the dirver emits no message that an MSI was allocated, like toher drivers do. That's a cosmetic one though.
>>> But the MSI->INTx regression is a severe one for me, which I'd like to fix myself but I'm missing so many fundamental skills :-(
>>>
>> Hi Harry,
>> You are correct about the bug. Please change the line at the top of the function that reads
>> error = 0;
>> to
>> error = ENXIO;
>> Let me know if that fixes the MSI problem for you.
>
> Hello Scott,
>
> thanks for your hint.
> Unfortunately I have a lot more problems – the system (11.2-RC1) deadlocks for some soconds with iSCSI load...
> This is far easyer reproducable / heavier impact with mps(4) and INTx allocation than with MSI, but backup runs over night triggered that extreme slowdown although mps(4) was allocating MSI – up to 20 sec locks, where even no terminal update happes.
> All those update ar queued though, so after about 10-20 sedonds, the screen flickers, showing all queued output.
>
> One symptom is that systat(1) shows 25% intr usage which is one core.
> It's a ZFS machine, so high sys usage is normal, but intr usually is about 10% with GbE traffic.
> Only when the slowdown/lockup happens, intr usage constantly stays at 25%.
>
> Can't imagine ctld(8) or zfs is causing this, but who knows – I don't at the moment.
> Will have to revert to 11.1 and see if things change, the machine was 10.? before – without such problems.
>
> BTW, does anybody have a link where I can get info about ENXIO?
ENXIO means that the device is not available. I use it in the driver to signal when the hardware cannot be accessed. The manual page for error codes is “man errno"
Scott
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A2AECF9F-2BB1-4FC0-8330-658336A3A4F0>