Date: Thu, 10 Nov 2011 10:27:03 -0800 From: Adrian Chadd <adrian@freebsd.org> To: Hooman Fazaeli <hoomanfazaeli@gmail.com> Cc: pyunyh@gmail.com, freebsd-net@freebsd.org, Jason Wolfe <nitroboost@gmail.com>, Jack Vogel <jfvogel@gmail.com>, Emil Muratov <gpm@hotplug.ru> Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled Message-ID: <CAJ-VmokfZOHUptqVcfz49ogzk0qCAdSXTO2p8_M=-fAcF4h=1g@mail.gmail.com> In-Reply-To: <4EBB9CDF.9090300@gmail.com> References: <CAAAm0r0RXEJo4UiKS=Ui0e5OQTg6sg-xcYf3mYB5%2Bvk8i8557w@mail.gmail.com> <CAAAm0r1DKvoL9=Ket9up=4%2B5xiCzTTZJK99FhF9jcCA28B0M%2BA@mail.gmail.com> <CAAAm0r3XdsMHZh%2BP_NF-txZasdExzwZ8ymmGQgGhJQds0fOiBQ@mail.gmail.com> <CAAAm0r1iS3z-7CBJ=xYDf%2BJOA1Q2nU0O54Twbyb7FjvgWHjKVw@mail.gmail.com> <4EA7E203.3020306@sepehrs.com> <CAAAm0r3Nr2t8cCetPkFnLQ-3KwqHw_0SpqbtvYPRUkSP=9n8CA@mail.gmail.com> <4EA80818.3030504@sentex.net> <4EA80F88.4000400@hotplug.ru> <4EA82715.2000404@gmail.com> <4EA8FA40.7010504@hotplug.ru> <4EA91836.2040508@gmail.com> <4EA959EE.2070806@hotplug.ru> <4EAD116A.8090006@gmail.com> <CAAAm0r3qm=nQQuAmZDD4k4X8K-xW6_kM9TukRT=1GoG9dYR3zw@mail.gmail.com> <4EAE58A2.9040803@gmail.com> <CAAAm0r0uoPPEQbq5rHkFr6ZLp-WJ4YVjDVvxxV6y%2BUh4eEKDEA@mail.gmail.com> <4EB96511.50701@gmail.com> <CAJ-Vmomf-wxb8dY7YF7qT_FGK5d-YLPU3BkPOeHnOtKZ%2BUrYeQ@mail.gmail.com> <4EBA3F22.2060204@gmail.com> <CAJ-Vmok8kX9F5eXTghx_s7diNiLTWY1-eMDUdCOUHQCz6zW%2BPg@mail.gmail.com> <4EBB9CDF.9090300@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Wait, but low resources which are stopping _what_ ? The whole point behind OACTIVE is to say "oi, give me time to flush my TX queue." I'll tell you when I'm ready. So when you clear OACTIVE after a TX completion handler, you would then call _start (or _start_locked) to reschedule some more frames. There shouldn't be a resource shortage here - ie, once you run out of tx descriptor slots, you'd set OACTIVE, then you will call the TX completion handler (on interrupt) until you've handled the pending entries in the TX descriptor ring/list. Once you've handled at least -one- TXed frame, that slot is now available and you can free OACTIVE + call _start/_start_locked. The problem is whether this is working right for drivers that implement _transmit, do multiqueue, etc. Ie, there's now multiple places where OACTIVE is being set/cleared, and from my experience with a few other > 100mbit ethernet/wifi drivers, I've seen situations where the TX queue stalled because the right mix of "clear OACTIVE to tell the stack they can start firing off more frames" and "call _start / _start_locked once there are actually TX buffers/descriptors available" didn't happen correctly. At this point, if you never call _start() and OACTIVE is set, TX stalls. It doesn't matter if you _clear_ OACTIVE at this point. The queue is full and thus nothing will happen. Some drivers even work around this silliness by calling the tx start routine from the end of the RX completion routine. :-) I don't think that's strictly needed if your hardware is posting interrupt notifications sensibly and your interrupt handler isn't buggy, but hey, maybe it isn't (eg, you disable interrupts, you do your TX/RX work, in the meantime some more TX completion has occured, then you restore interrupts and clear the bits you were working on - there's stuff in the TX completion queue, but now that the queue is full you won't _get_ another interrupt for it.) So the magic is: * is OACTIVE being set/cleared sensibly; * are there calls to _start or _start_locked at the right spots (ie, after TX completion has occured); * what's going on with the _transmit stuff; * how's the descriptor aggregation stuff for _transmit working and are you hitting concurrency issues where that queue/queue gets stalled because you never call the TX completion function and then _start or _transmit to drain that queue? * Check the interrupt handling and see if when you disable/enable interrupts, you're clearing the status bits wrong (if the hardware even lets you do the sensible thing.) Eg, don't clear the TX completion bits _after_ you handle TX completion but before you re-enable interrupts, as the hardware may have completed some more frames and filled the TX descriptor queue/ring. At this point if you're lucky you'll get a "yo, TX queue FULL!" interrupt/error, but if you're even _more_ unlucky, you've just handled a TX completion + TX queue full event and whilst handling that full queue, you fill the queue up again. Then you won't get a subsequent interrupt for the now filled queue. Now, I've not got any em hardware (legacy or shiny) so I'm simply offering observations from having had to debug this stuff in the recent past. I just think you've hit the same issue. :) Adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-VmokfZOHUptqVcfz49ogzk0qCAdSXTO2p8_M=-fAcF4h=1g>