From owner-freebsd-net@freebsd.org Wed Jun 17 00:13:21 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6DEE233A832 for ; Wed, 17 Jun 2020 00:13:21 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49mlsC5nnyz40Ss for ; Wed, 17 Jun 2020 00:13:19 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pl1-x635.google.com with SMTP id x11so95654plo.7 for ; Tue, 16 Jun 2020 17:13:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:mail-followup-to :references:mime-version:content-disposition:in-reply-to; bh=QOviLWZRVjmnZr9s/LkXo4YY6G1GUXimMSilJB02+Sc=; b=B9KvsynFcUd/p+bDYB9p6fwlTu5fOhemFfPlvBgixHVtAwomlA9o84k3D55OkZJibt CVTLRq5koHyKwS4BrP/PlUl9q2uNfbcXkIdEPLPbjEwWUhLMe/0/Kmm7/ltehd+bTVhL 4EqYylPMCUDYdPHEomSfua8Q2cw6OdjM9aAs/jAHwOualkyRWLdokqfirlNgOVCrUG22 Y8rJmJLvJeShR5bnI/B8ArWf2sZcU90iL6ukbtuZoAlkpYwKpEmnpvifV5aAbiealNKf qm0UJDm/mOFxCQpxayI6PmrLn8L0QoHKC7QYMKRpFolFDgbeW8eFWXy6XsqJE1LU4GwY I93A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :in-reply-to; bh=QOviLWZRVjmnZr9s/LkXo4YY6G1GUXimMSilJB02+Sc=; b=uIT2mBMxFUkMecEEmYYovfyI1GTI1HMJAW8t5g59gLCpxqP8FooMN3y06Xq4GWht1u 3YFyAIbFlmq4lVjcIwvERG5j73C5pH7HFqR+GYQGphAeHwV/Tjp6JuPV3OOce8dAtNcc /2h+w2cSTCswVdV2f+Oss4eaO6defL4VpbwnYJEqBSz5QHRyEplJvPfGOZ6UukBqEJD1 9OxuMy04JxJXhmsGMpG5pwrzyZWGlnt7jEih86I0e3Poy1BUG1cM/buDQUdmWpRyTDv5 9xfErRAFcDf8G0LJV6AEPzHo7D416jkWS6Ckamj9oACdLtLdHZdrTAujrBEqBOmgnLUJ vMuA== X-Gm-Message-State: AOAM532hGPQaBoohaQFq370Sbrkw91Rn++B8S0GtmFjjEiqETJhVkrv4 ipbVWHGRUyDweTtPkl6t6ro= X-Google-Smtp-Source: ABdhPJw5g/TWjTRTQSBHrTOIsqF4pUsgTymjWFVcq9CGRU9bleWNfEopg1TDUcz+L2fs5BYhQGeH1A== X-Received: by 2002:a17:902:6bc5:: with SMTP id m5mr4343127plt.101.1592352797816; Tue, 16 Jun 2020 17:13:17 -0700 (PDT) Received: from x270 ([2601:641:c000:b500::1353]) by smtp.gmail.com with ESMTPSA id m5sm15444288pga.3.2020.06.16.17.13.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jun 2020 17:13:17 -0700 (PDT) Sender: Navdeep Parhar Date: Tue, 16 Jun 2020 17:13:15 -0700 From: Navdeep Parhar To: "Saad, Mark" Cc: "Foster, Greg" , "freebsd-net@freebsd.org" Subject: Re: How to Increase TX Queue Priority for LACP Packets Message-ID: <20200617001315.GB80914@x270> Mail-Followup-To: "Saad, Mark" , "Foster, Greg" , "freebsd-net@freebsd.org" References: <20200613025031.GA34653@x270> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 49mlsC5nnyz40Ss X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=B9KvsynF; dmarc=none; spf=pass (mx1.freebsd.org: domain of nparhar@gmail.com designates 2607:f8b0:4864:20::635 as permitted sender) smtp.mailfrom=nparhar@gmail.com X-Spamd-Result: default: False [-2.15 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; NEURAL_HAM_LONG(-1.03)[-1.033]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; NEURAL_HAM_SHORT(-0.50)[-0.498]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::635:from]; NEURAL_HAM_MEDIUM(-0.92)[-0.918]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[np@FreeBSD.org,nparhar@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[np@FreeBSD.org,nparhar@gmail.com]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jun 2020 00:13:21 -0000 We could have a global knob that tells all NIC drivers to use a reserved queue for non-RSS traffic, but that would be advisory at best because the tx queue selection takes place inside the driver's (or iflib's) transmit routine. The meat of the change is going to be in iflib and all non-iflib drivers' if_transmit. Regards, Navdeep On Tue, Jun 16, 2020 at 09:48:19PM +0000, Saad, Mark wrote: > All > Is there any way to make this change on other nic's like Intel ix and > Solarflare sfxge ? I have seen similar issues on both with 12.1 > mainly with solarflare nics. > > --- > Mark Saad > mark.saad@lucera.com > > > ________________________________________ > From: owner-freebsd-net@freebsd.org on behalf of Foster, Greg > Sent: Tuesday, June 16, 2020 3:56 PM > To: Navdeep Parhar > Cc: freebsd-net@freebsd.org > Subject: RE: How to Increase TX Queue Priority for LACP Packets > > HI Navdeep, > > Thanks for the information! I've integrated the changes and will be > testing more today. > > We have seen the LACP port flapping under different scenarios, most we > believe are traffic/load based. > > I did see the flapping unexpectedly when I just enabled LACP debug > (e.g., sysctl net.link.lagg.lacp.debug=1). Is this a known > problem? > > Thanks > Greg > > -----Original Message----- > From: Navdeep Parhar On Behalf Of Navdeep Parhar > Sent: Friday, June 12, 2020 7:51 PM > To: Foster, Greg > Cc: freebsd-net@freebsd.org > Subject: Re: How to Increase TX Queue Priority for LACP Packets > > On Fri, Jun 12, 2020 at 11:47:41PM +0000, Foster, Greg wrote: > > FreeBSD Networkers, > > > > We are seeing LACP port flapping on our FreeBSD 10.4/12.1 systems > > under different conditions. > > > > Can someone explain or point me to the information on how to queue > > the LACP packets to a higher priority queue ? > > > > We are using the Chelsio T580-LP-CR adapter/cxgbe driver. The > > Cheslio NICs have 8 TX/RX queues each, but I don't know how to > > explicitly put the LACP packets in the higher priority TX queue. > > > > I've read about PF/ALTQ and think this may be overkill our needs, > > and was wondering if there was a simpler method. > > This is cxgbe specific but that's what you're using so it'll do. > > Add "hw.cxgbe.rsrv_noflowq=1" to your /boot/loader.conf. That > reserves one tx queue for non-RSS traffic (like ARP, LACP). You might > also want to increase the number of tx queues to compensate for the > one that's now reserved. Use "hw.cxgbe.ntxq=9" for that. The ntxq > knob might be different on 10.4 but the man page matching the driver > should have its exact name. > > Regards, > Navdeep > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >