From owner-svn-src-all@freebsd.org Wed May 1 16:45:58 2019 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DF081159B4FD; Wed, 1 May 2019 16:45:57 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7045E84BAC; Wed, 1 May 2019 16:45:57 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from John-Baldwins-MacBook-Pro-3.local (ralph.baldwin.cx [66.234.199.215]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id 723EA354D; Wed, 1 May 2019 16:45:56 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Subject: Re: svn commit: r341586 - head/sys/dev/mlx5/mlx5_en To: slavash@FreeBSD.org, Hans Petter Selasky , "Andrey V. Elsukov" , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org References: <201812051425.wB5EP38T004562@repo.freebsd.org> <7cea5305-4136-a0b6-487b-51307b1c6db9@FreeBSD.org> <123654bf-59f7-1db8-55ce-36306bdac43d@FreeBSD.org> <67b638e0-8e6f-f960-b263-62eed8ce28b7@FreeBSD.org> <0e80cb1d-20b9-5b8c-330e-5f64a603ff10@FreeBSD.org> <737bfa62-eb3e-7c48-6356-0e0b76bb0593@FreeBSD.org> <76a05c08-b851-c50a-16fa-ac42c872413c@FreeBSD.org> From: John Baldwin Openpgp: preference=signencrypt Autocrypt: addr=jhb@FreeBSD.org; keydata= mQGiBETQ+XcRBADMFybiq69u+fJRy/0wzqTNS8jFfWaBTs5/OfcV7wWezVmf9sgwn8TW0Dk0 c9MBl0pz+H01dA2ZSGZ5fXlmFIsee1WEzqeJzpiwd/pejPgSzXB9ijbLHZ2/E0jhGBcVy5Yo /Tw5+U/+laeYKu2xb0XPvM0zMNls1ah5OnP9a6Ql6wCgupaoMySb7DXm2LHD1Z9jTsHcAQMD /1jzh2BoHriy/Q2s4KzzjVp/mQO5DSm2z14BvbQRcXU48oAosHA1u3Wrov6LfPY+0U1tG47X 1BGfnQH+rNAaH0livoSBQ0IPI/8WfIW7ub4qV6HYwWKVqkDkqwcpmGNDbz3gfaDht6nsie5Z pcuCcul4M9CW7Md6zzyvktjnbz61BADGDCopfZC4of0Z3Ka0u8Wik6UJOuqShBt1WcFS8ya1 oB4rc4tXfSHyMF63aPUBMxHR5DXeH+EO2edoSwViDMqWk1jTnYza51rbGY+pebLQOVOxAY7k do5Ordl3wklBPMVEPWoZ61SdbcjhHVwaC5zfiskcxj5wwXd2E9qYlBqRg7QeSm9obiBCYWxk d2luIDxqaGJARnJlZUJTRC5vcmc+iGAEExECACAFAkTQ+awCGwMGCwkIBwMCBBUCCAMEFgID AQIeAQIXgAAKCRBy3lIGd+N/BI6RAJ9S97fvbME+3hxzE3JUyUZ6vTewDACdE1stFuSfqMvM jomvZdYxIYyTUpC5Ag0ERND5ghAIAPwsO0B7BL+bz8sLlLoQktGxXwXQfS5cInvL17Dsgnr3 1AKa94j9EnXQyPEj7u0d+LmEe6CGEGDh1OcGFTMVrof2ZzkSy4+FkZwMKJpTiqeaShMh+Goj XlwIMDxyADYvBIg3eN5YdFKaPQpfgSqhT+7El7w+wSZZD8pPQuLAnie5iz9C8iKy4/cMSOrH YUK/tO+Nhw8Jjlw94Ik0T80iEhI2t+XBVjwdfjbq3HrJ0ehqdBwukyeJRYKmbn298KOFQVHO EVbHA4rF/37jzaMadK43FgJ0SAhPPF5l4l89z5oPu0b/+5e2inA3b8J3iGZxywjM+Csq1tqz hltEc7Q+E08AAwUIAL+15XH8bPbjNJdVyg2CMl10JNW2wWg2Q6qdljeaRqeR6zFus7EZTwtX sNzs5bP8y51PSUDJbeiy2RNCNKWFMndM22TZnk3GNG45nQd4OwYK0RZVrikalmJY5Q6m7Z16 4yrZgIXFdKj2t8F+x613/SJW1lIr9/bDp4U9tw0V1g3l2dFtD3p3ZrQ3hpoDtoK70ioIAjjH aIXIAcm3FGZFXy503DOA0KaTWwvOVdYCFLm3zWuSOmrX/GsEc7ovasOWwjPn878qVjbUKWwx Q4QkF4OhUV9zPtf9tDSAZ3x7QSwoKbCoRCZ/xbyTUPyQ1VvNy/mYrBcYlzHodsaqUDjHuW+I SQQYEQIACQUCRND5ggIbDAAKCRBy3lIGd+N/BCO8AJ9j1dWVQWxw/YdTbEyrRKOY8YZNwwCf afMAg8QvmOWnHx3wl8WslCaXaE8= Message-ID: <054b81ee-0bdc-184a-9928-8679b53d52cc@FreeBSD.org> Date: Wed, 1 May 2019 09:45:54 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <76a05c08-b851-c50a-16fa-ac42c872413c@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 7045E84BAC X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.98 / 15.00]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.98)[-0.979,0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 May 2019 16:45:58 -0000 On 5/1/19 1:05 AM, Slava Shwartsman wrote: > > > On 01-May-19 10:28, Slava Shwartsman wrote: >> >> >> On 01-May-19 10:09, Slava Shwartsman wrote: >>> >>> >>> On 30-Apr-19 00:14, John Baldwin wrote: >>>> On 4/25/19 12:10 AM, Slava Shwartsman wrote: >>>>> >>>>> >>>>> On 17-Apr-19 00:28, John Baldwin wrote: >>>>>> On 4/16/19 8:32 AM, Hans Petter Selasky wrote: >>>>>>> On 4/16/19 4:39 PM, Andrey V. Elsukov wrote: >>>>>>>> On 05.12.2018 17:25, Slava Shwartsman wrote: >>>>>>>>> Author: slavash >>>>>>>>> Date: Wed Dec  5 14:25:03 2018 >>>>>>>>> New Revision: 341586 >>>>>>>>> URL: https://svnweb.freebsd.org/changeset/base/341586 >>>>>>>>> >>>>>>>>> Log: >>>>>>>>>      mlx5en: Implement backpressure indication. >>>>>>>>>      The backpressure indication is implemented using an >>>>>>>>> unlimited rate type of >>>>>>>>>      mbuf send tag. When the upper layers typically the socket >>>>>>>>> layer has obtained such >>>>>>>>>      a tag, it can then query the destination driver queue for >>>>>>>>> the current >>>>>>>>>      amount of space available in the send queue. >>>>>>>>>      A single mbuf send tag may be referenced multiple times and >>>>>>>>> a refcount has been added >>>>>>>>>      to the mlx5e_priv structure to track its usage. Because the >>>>>>>>> send tag resides >>>>>>>>>      in the mlx5e_channel structure, there is no need to wait >>>>>>>>> for refcounts to reach >>>>>>>>>      zero until the mlx4en(4) driver is detached. The channels >>>>>>>>> structure is persistant >>>>>>>>>      during the lifetime of the mlx5en(4) driver it belongs to >>>>>>>>> and can so be accessed >>>>>>>>>      without any need of synchronization. >>>>>>>>>      The mlx5e_snd_tag structure was extended to contain a type >>>>>>>>> field, because there are now >>>>>>>>>      two different tag types which end up in the driver which >>>>>>>>> need to be distinguished. >>>>>>>>>      Submitted by:   hselasky@ >>>>>>>>>      Approved by:    hselasky (mentor) >>>>>>>>>      MFC after:      1 week >>>>>>>>>      Sponsored by:   Mellanox Technologies >>>>>>>>> @@ -587,27 +609,33 @@ mlx5e_xmit(struct ifnet *ifp, struct mbuf >>>>>>>>> *mb) >>>>>>>>>         struct mlx5e_sq *sq; >>>>>>>>>         int ret; >>>>>>>>> -    sq = mlx5e_select_queue(ifp, mb); >>>>>>>>> -    if (unlikely(sq == NULL)) { >>>>>>>>> -#ifdef RATELIMIT >>>>>>>>> -        /* Check for route change */ >>>>>>>>> -        if (mb->m_pkthdr.snd_tag != NULL && >>>>>>>>> -            mb->m_pkthdr.snd_tag->ifp != ifp) { >>>>>>>>> +    if (mb->m_pkthdr.snd_tag != NULL) { >>>>>>>>> +        sq = mlx5e_select_queue_by_send_tag(ifp, mb); >>>>>>>>> +        if (unlikely(sq == NULL)) { >>>>>>>>> +            /* Check for route change */ >>>>>>>>> +            if (mb->m_pkthdr.snd_tag->ifp != ifp) { >>>>>>>>> +                /* Free mbuf */ >>>>>>>>> +                m_freem(mb); >>>>>>>>> + >>>>>>>>> +                /* >>>>>>>>> +                 * Tell upper layers about route >>>>>>>>> +                 * change and to re-transmit this >>>>>>>>> +                 * packet: >>>>>>>>> +                 */ >>>>>>>>> +                return (EAGAIN); >>>>>>>>> +            } >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I just discovered something strange and found that this commit is >>>>>>>> the >>>>>>>> cause. >>>>>>>> The test system has mlx5en 100G interface. It has two vlans: >>>>>>>> vlan500 and >>>>>>>> vlan100. >>>>>>>> Via vlan500 it receives some packets flows. Then it routes these >>>>>>>> packets >>>>>>>> into vlan100. >>>>>>>> But packets are dropped in mlx5e_xmit() with EAGAIN error code. >>>>>>>> >>>>>>>> # dtrace -n 'fbt::ip6_output:return {printf("%d", arg1);}' >>>>>>>> dtrace: description 'fbt::ip6_output:return ' matched 1 probe >>>>>>>> CPU     ID                    FUNCTION:NAME >>>>>>>>     23  54338                ip6_output:return 35 >>>>>>>>     16  54338                ip6_output:return 35 >>>>>>>>     21  54338                ip6_output:return 35 >>>>>>>>     22  54338                ip6_output:return 35 >>>>>>>>     24  54338                ip6_output:return 35 >>>>>>>>     23  54338                ip6_output:return 35 >>>>>>>>     14  54338                ip6_output:return 35 >>>>>>>> ^C >>>>>>>> >>>>>>>> # dtrace -n 'fbt::mlx5e_xmit:return {printf("%d", arg1);}' >>>>>>>> dtrace: description 'fbt::mlx5e_xmit:return ' matched 1 probe >>>>>>>> CPU     ID                    FUNCTION:NAME >>>>>>>>     16  69030                mlx5e_xmit:return 35 >>>>>>>>     23  69030                mlx5e_xmit:return 35 >>>>>>>>     26  69030                mlx5e_xmit:return 35 >>>>>>>>     25  69030                mlx5e_xmit:return 35 >>>>>>>>     24  69030                mlx5e_xmit:return 35 >>>>>>>>     21  69030                mlx5e_xmit:return 35 >>>>>>>>     26  69030                mlx5e_xmit:return 35 >>>>>>>> ^C >>>>>>>> >>>>>>>> The kernel config is GENERIC. >>>>>>>> 13.0-CURRENT #9 r345758+82f3d57(svn_head)-dirty >>>>>>>> >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> This might be a case where rcvif in the mbuf's pktheader is not >>>>>>> cleared >>>>>>> before the packet is fed back on the wire. >>>>>>> >>>>>>> John Baldwin is working on the send tags implementation, to eliminate >>>>>>> the EAGAIN handling in the network drivers. >>>>>> >>>>>> I will try to push this branch sooner then since it affects more >>>>>> than just >>>>>> TLS.  Part of the change includes a new flag we can use to assert >>>>>> that we >>>>> Thanks John! >>>>>> aren't just getting a stale rcvif (though there are also now >>>>>> assertions in >>>>>> ip_output that should catch this case I think). >>>>>> >>>>> >>>>> >>>>> >>>>> Hi Andrey, >>>>> >>>>> Yes, we were able to reproduce this issue in house. If you don't >>>>> mind, I >>>>> prefer to wait for John's update - where he eliminates the EAGAIN >>>>> handling in the network drivers. >>>> >>>> I have rebased the branch for this, but for now it will just panic >>>> sooner >>>> I believe by tripping an assertion.  Can you grab the diff (or just >>>> the branch) >>>> from the 'send_tags' branch at github/bsdjhb/freebsd and reproduce >>>> under a >>>> kernel with INVARIANTS?  I think we will have to explicitly clear the >>>> 'rcvif' >>>> pointer somewhere, but I want to see what the stack trace looks like >>>> so I can >>>> think about the "right" place to clear it. >>>> >>> >>> Hi John, >>> >>> I grabbed your branch (which doesn't build BTW due to libbe(3): Fix >>> mis-application of patch (SHLIBDIR) so I just reverted it). >>> >>> The kernel doesn't panic in this scenario - it just that the packets >>> are being dropped. So I added a kdb_backtrace right before the return >>> (EAGAIN) in mlx5e_xmit: >>> >>> KDB: stack backtrace: >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >>> 0xfffffe0000547d90 >>> mlx5e_xmit() at mlx5e_xmit+0x3d/frame 0xfffffe0000548160 >>> vlan_transmit() at vlan_transmit+0xdc/frame 0xfffffe00005481d0 >>> ether_output_frame() at ether_output_frame+0xa2/frame 0xfffffe0000548200 >>> ether_output() at ether_output+0x689/frame 0xfffffe00005482a0 >>> ip_output() at ip_output+0x13a4/frame 0xfffffe00005483f0 >>> ip_forward() at ip_forward+0x344/frame 0xfffffe00005484b0 >>> ip_input() at ip_input+0x7f5/frame 0xfffffe0000548560 >>> netisr_dispatch_src() at netisr_dispatch_src+0xa2/frame >>> 0xfffffe00005485d0 >>> ether_demux() at ether_demux+0x147/frame 0xfffffe0000548600 >>> ether_nh_input() at ether_nh_input+0x403/frame 0xfffffe0000548660 >>> netisr_dispatch_src() at netisr_dispatch_src+0xa2/frame >>> 0xfffffe00005486d0 >>> ether_input() at ether_input+0x73/frame 0xfffffe0000548700 >>> vlan_input() at vlan_input+0x1e7/frame 0xfffffe0000548750 >>> ether_demux() at ether_demux+0x12d/frame 0xfffffe0000548780 >>> ether_nh_input() at ether_nh_input+0x403/frame 0xfffffe00005487e0 >>> netisr_dispatch_src() at netisr_dispatch_src+0xa2/frame >>> 0xfffffe0000548850 >>> ether_input() at ether_input+0x73/frame 0xfffffe0000548880 >>> mlx5e_rx_cq_comp() at mlx5e_rx_cq_comp+0x8b4/frame 0xfffffe00005489a0 >>> mlx5_cq_completion() at mlx5_cq_completion+0x5e/frame 0xfffffe00005489d0 >>> mlx5_msix_handler() at mlx5_msix_handler+0x1ba/frame 0xfffffe0000548a10 >>> ithread_loop() at ithread_loop+0x187/frame 0xfffffe0000548a70 >>> fork_exit() at fork_exit+0x84/frame 0xfffffe0000548ab0 >>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000548ab0 >>> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- >>> >>> >>> Please ping me if you want me to try anything else. >>> >>> >>> Slava >> >> My bad - tested with master. Re-testing now. >> >> >> Slava > > Got it now: > > panic: Assertion m->m_pkthdr.snd_tag == NULL failed at > /usr/src/sys/netinet/ip_output.c:213 > cpuid = 0 > time = 1556697834 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfffffe00005ac1f0 > vpanic() at vpanic+0x19d/frame 0xfffffe00005ac240 > panic() at panic+0x43/frame 0xfffffe00005ac2a0 > ip_output() at ip_output+0x159f/frame 0xfffffe00005ac3f0 > ip_forward() at ip_forward+0x38c/frame 0xfffffe00005ac4b0 > ip_input() at ip_input+0x7f5/frame 0xfffffe00005ac560 > netisr_dispatch_src() at netisr_dispatch_src+0xa2/frame 0xfffffe00005ac5d0 > ether_demux() at ether_demux+0x147/frame 0xfffffe00005ac600 > ether_nh_input() at ether_nh_input+0x403/frame 0xfffffe00005ac660 > netisr_dispatch_src() at netisr_dispatch_src+0xa2/frame 0xfffffe00005ac6d0 > ether_input() at ether_input+0x7d/frame 0xfffffe00005ac700 > vlan_input() at vlan_input+0x1e7/frame 0xfffffe00005ac750 > ether_demux() at ether_demux+0x12d/frame 0xfffffe00005ac780 > ether_nh_input() at ether_nh_input+0x403/frame 0xfffffe00005ac7e0 > netisr_dispatch_src() at netisr_dispatch_src+0xa2/frame 0xfffffe00005ac850 > ether_input() at ether_input+0x7d/frame 0xfffffe00005ac880 > mlx5e_rx_cq_comp() at mlx5e_rx_cq_comp+0x8b4/frame 0xfffffe00005ac9a0 > mlx5_cq_completion() at mlx5_cq_completion+0x5e/frame 0xfffffe00005ac9d0 > mlx5_msix_handler() at mlx5_msix_handler+0x1ba/frame 0xfffffe00005aca10 > ithread_loop() at ithread_loop+0x187/frame 0xfffffe00005aca70 > fork_exit() at fork_exit+0x84/frame 0xfffffe00005acab0 > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005acab0 > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > KDB: enter: panic > [ thread pid 12 tid 100113 ] > Stopped at kdb_enter+0x3b: movq $0,kdb_why > > > I can keep the machine in this state for a while now if you want to take > a look at anything specific. I pushed further changes to this branch (that are now in the review that Hans noted) that I think should fix the panic. You can either pull the branch (I'll try to rebase it today) or from the review. Please let me know if this resolves the panic. Thanks! -- John Baldwin