From owner-svn-src-head@freebsd.org Thu Apr 25 07:15:27 2019 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1CFFD158EA50; Thu, 25 Apr 2019 07:15:27 +0000 (UTC) (envelope-from slava.shwartsman@gmail.com) Received: from mail-wm1-f66.google.com (mail-wm1-f66.google.com [209.85.128.66]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 121FA6DBB4; Thu, 25 Apr 2019 07:15:25 +0000 (UTC) (envelope-from slava.shwartsman@gmail.com) Received: by mail-wm1-f66.google.com with SMTP id 10so7894419wmk.0; Thu, 25 Apr 2019 00:15:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:subject:to:references:from:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=nBSqTqqOWk5ANCtfxmll8aIJDxSrLHn4pJ8rNrq/PHs=; b=R1jqM+NbdCC/kXIOyqqz6wqlQkhDJ++vPkE1W1SUdmyvR6zXE7Qr7yG6atIB74vfwW UIDvjr2XSnU7vre42n0riUU18HXvZXWcVPqwqL8zzD3M2YbJ/ZoMss4g1bjm2rpuC/hC 7C+ve6LiJU/av+xwJ2kWHyHVjd37B9ZbUqOAMEhzTJSS3btYvJBWCO62eRA+5VWtNi+K OYmapP27FRu+pip423/7NlsHEwHpW7X8KMj095qeV0JQ4g1DOx5TJsRvJyoErFB1krEJ raKdiGt/jz+MDYssFRyoCororUy43r1vdgffc9YfEWLQmAeVcW7hpDNtkuYSoZyGCTht zk3w== X-Gm-Message-State: APjAAAXQM/We4/PaZfhThvMrUxTH0CcB7mQl0NnO+Xau/Fv6O2EvhmPw RWMA5D20YKtipoCVetKEaELrj4nt X-Google-Smtp-Source: APXvYqxCdTd/6bMdUoJxyyO89pHy+nuYs3vLftfv5NwF8/YhaRpVzsZ3l+x1SV5vA7YHS9Q3zVj3eQ== X-Received: by 2002:a1c:eb12:: with SMTP id j18mr2424879wmh.48.1556176212915; Thu, 25 Apr 2019 00:10:12 -0700 (PDT) Received: from [10.0.0.30] (bzq-79-182-94-172.red.bezeqint.net. [79.182.94.172]) by smtp.gmail.com with ESMTPSA id v1sm21579478wrd.47.2019.04.25.00.10.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 25 Apr 2019 00:10:12 -0700 (PDT) Reply-To: slavash@FreeBSD.org Subject: Re: svn commit: r341586 - head/sys/dev/mlx5/mlx5_en To: John Baldwin , Hans Petter Selasky , "Andrey V. Elsukov" , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org References: <201812051425.wB5EP38T004562@repo.freebsd.org> <7cea5305-4136-a0b6-487b-51307b1c6db9@FreeBSD.org> From: Slava Shwartsman Message-ID: <123654bf-59f7-1db8-55ce-36306bdac43d@FreeBSD.org> Date: Thu, 25 Apr 2019 10:10:09 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <7cea5305-4136-a0b6-487b-51307b1c6db9@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 121FA6DBB4 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of slavashwartsman@gmail.com designates 209.85.128.66 as permitted sender) smtp.mailfrom=slavashwartsman@gmail.com X-Spamd-Result: default: False [-4.13 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_REPLYTO(0.00)[slavash@FreeBSD.org]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.87)[-0.873,0]; FORGED_SENDER(0.30)[slavash@FreeBSD.org,slavashwartsman@gmail.com]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TAGGED_FROM(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_NEQ_ENVFROM(0.00)[slavash@FreeBSD.org,slavashwartsman@gmail.com]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; R_DKIM_NA(0.00)[]; DMARC_NA(0.00)[FreeBSD.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[66.128.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.24)[ipnet: 209.85.128.0/17(-3.88), asn: 15169(-2.26), country: US(-0.06)] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Apr 2019 07:15:27 -0000 On 17-Apr-19 00:28, John Baldwin wrote: > On 4/16/19 8:32 AM, Hans Petter Selasky wrote: >> On 4/16/19 4:39 PM, Andrey V. Elsukov wrote: >>> On 05.12.2018 17:25, Slava Shwartsman wrote: >>>> Author: slavash >>>> Date: Wed Dec 5 14:25:03 2018 >>>> New Revision: 341586 >>>> URL: https://svnweb.freebsd.org/changeset/base/341586 >>>> >>>> Log: >>>> mlx5en: Implement backpressure indication. >>>> >>>> The backpressure indication is implemented using an unlimited rate type of >>>> mbuf send tag. When the upper layers typically the socket layer has obtained such >>>> a tag, it can then query the destination driver queue for the current >>>> amount of space available in the send queue. >>>> >>>> A single mbuf send tag may be referenced multiple times and a refcount has been added >>>> to the mlx5e_priv structure to track its usage. Because the send tag resides >>>> in the mlx5e_channel structure, there is no need to wait for refcounts to reach >>>> zero until the mlx4en(4) driver is detached. The channels structure is persistant >>>> during the lifetime of the mlx5en(4) driver it belongs to and can so be accessed >>>> without any need of synchronization. >>>> >>>> The mlx5e_snd_tag structure was extended to contain a type field, because there are now >>>> two different tag types which end up in the driver which need to be distinguished. >>>> >>>> Submitted by: hselasky@ >>>> Approved by: hselasky (mentor) >>>> MFC after: 1 week >>>> Sponsored by: Mellanox Technologies >>>> @@ -587,27 +609,33 @@ mlx5e_xmit(struct ifnet *ifp, struct mbuf *mb) >>>> struct mlx5e_sq *sq; >>>> int ret; >>>> >>>> - sq = mlx5e_select_queue(ifp, mb); >>>> - if (unlikely(sq == NULL)) { >>>> -#ifdef RATELIMIT >>>> - /* Check for route change */ >>>> - if (mb->m_pkthdr.snd_tag != NULL && >>>> - mb->m_pkthdr.snd_tag->ifp != ifp) { >>>> + if (mb->m_pkthdr.snd_tag != NULL) { >>>> + sq = mlx5e_select_queue_by_send_tag(ifp, mb); >>>> + if (unlikely(sq == NULL)) { >>>> + /* Check for route change */ >>>> + if (mb->m_pkthdr.snd_tag->ifp != ifp) { >>>> + /* Free mbuf */ >>>> + m_freem(mb); >>>> + >>>> + /* >>>> + * Tell upper layers about route >>>> + * change and to re-transmit this >>>> + * packet: >>>> + */ >>>> + return (EAGAIN); >>>> + } >>> >>> Hi, >>> >>> I just discovered something strange and found that this commit is the >>> cause. >>> The test system has mlx5en 100G interface. It has two vlans: vlan500 and >>> vlan100. >>> Via vlan500 it receives some packets flows. Then it routes these packets >>> into vlan100. >>> But packets are dropped in mlx5e_xmit() with EAGAIN error code. >>> >>> # dtrace -n 'fbt::ip6_output:return {printf("%d", arg1);}' >>> dtrace: description 'fbt::ip6_output:return ' matched 1 probe >>> CPU ID FUNCTION:NAME >>> 23 54338 ip6_output:return 35 >>> 16 54338 ip6_output:return 35 >>> 21 54338 ip6_output:return 35 >>> 22 54338 ip6_output:return 35 >>> 24 54338 ip6_output:return 35 >>> 23 54338 ip6_output:return 35 >>> 14 54338 ip6_output:return 35 >>> ^C >>> >>> # dtrace -n 'fbt::mlx5e_xmit:return {printf("%d", arg1);}' >>> dtrace: description 'fbt::mlx5e_xmit:return ' matched 1 probe >>> CPU ID FUNCTION:NAME >>> 16 69030 mlx5e_xmit:return 35 >>> 23 69030 mlx5e_xmit:return 35 >>> 26 69030 mlx5e_xmit:return 35 >>> 25 69030 mlx5e_xmit:return 35 >>> 24 69030 mlx5e_xmit:return 35 >>> 21 69030 mlx5e_xmit:return 35 >>> 26 69030 mlx5e_xmit:return 35 >>> ^C >>> >>> The kernel config is GENERIC. >>> 13.0-CURRENT #9 r345758+82f3d57(svn_head)-dirty >>> >> >> Hi, >> >> This might be a case where rcvif in the mbuf's pktheader is not cleared >> before the packet is fed back on the wire. >> >> John Baldwin is working on the send tags implementation, to eliminate >> the EAGAIN handling in the network drivers. > > I will try to push this branch sooner then since it affects more than just > TLS. Part of the change includes a new flag we can use to assert that we Thanks John! > aren't just getting a stale rcvif (though there are also now assertions in > ip_output that should catch this case I think). > Hi Andrey, Yes, we were able to reproduce this issue in house. If you don't mind, I prefer to wait for John's update - where he eliminates the EAGAIN handling in the network drivers. Slava