From owner-freebsd-net@freebsd.org Wed Jun 17 00:43:56 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D556933BA21 for ; Wed, 17 Jun 2020 00:43:56 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49mmXW6pvWz42F8 for ; Wed, 17 Jun 2020 00:43:55 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pl1-x634.google.com with SMTP id x11so130042plo.7 for ; Tue, 16 Jun 2020 17:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:mail-followup-to :references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=aa+uZg9aej4ZtZus9Tf1mkuh+PrMLHMrI/Cvg3G8oYQ=; b=sOt4rPHovE2a0N16q4Kf+B8oc3mtncs0ApG8Dnj88anI8gyubml9JMNQiSyHtzUoqE xUnSTWpTZzh37dmoRgH8HB5qTo2FcWn9Pp5Tc1AhbaNzxP5CpKEWP/27vQRXryf2sZHn L+NcMIK2agBF3qncFhIH15VR7nKb0QgR6M9OJ6w9p429+6dwPIpTkpiFUC/azT4iODWu rbMzQvPRvmo4hA1KwyzxBBZY7O+y4KaVDTH1We2DaqWXzhLFEpfCGN224263evzb1RW7 QpNSmcE7tvDOX6zwEvrv/vpYlWznFfgBlsFvQuSkMWnqaMyhSQJLNs4OPVfALD1aaBSU s7fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=aa+uZg9aej4ZtZus9Tf1mkuh+PrMLHMrI/Cvg3G8oYQ=; b=Bxdy9L8JNpIwffxI5eFVowZJZakIngDBJPW2HSLKFaIwuyIKGkJNSdXgJyZL4N1T8i RGV961+YFYEt7bopOpKiTrnJw7AZihqfKY6s3XvXXW5L+W2VZflbvXvBC6zSSNU1yRD9 5CFZUimXNyedehpDkCok1w86ySp9dcpOCPIeqAMTSWvY1x98rs/2LeUd6+++n1xu3KEr pzW4UkIUc/BUmnIoZgNgl1TOffeqSzpuWtIm0R68oLRjLMf3Tg9HMYn7X5Eb9wnlPPsy Uh1q3yZKnwt3RVZEzFHeQWZazc5XjVmVaswpR1Ir9Vw5iUJ8bVTwtzRJ5KxyKaPJ/3jz yWEA== X-Gm-Message-State: AOAM533lGtTApHtR7SPnhzRhnFwIM8KOjvr0CPf2ovi3y2pwW0IUrWmc rDWU/kTBeZOFtd3OSt9Vdnc= X-Google-Smtp-Source: ABdhPJz8WghttjqdbZ7EijZpkLxaTz8Jj8b5iQzTcMehhn6TeibfNRfL5rG3yl7fNistV61fBoBxxg== X-Received: by 2002:a17:90b:3c6:: with SMTP id go6mr5476600pjb.224.1592354634303; Tue, 16 Jun 2020 17:43:54 -0700 (PDT) Received: from x270 ([2601:641:c000:b500::1353]) by smtp.gmail.com with ESMTPSA id q1sm19670087pfk.132.2020.06.16.17.43.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jun 2020 17:43:53 -0700 (PDT) Sender: Navdeep Parhar Date: Tue, 16 Jun 2020 17:43:51 -0700 From: Navdeep Parhar To: "Saad, Mark" Cc: "Foster, Greg" , "freebsd-net@freebsd.org" Subject: Re: How to Increase TX Queue Priority for LACP Packets Message-ID: <20200617004351.GC80914@x270> Mail-Followup-To: "Saad, Mark" , "Foster, Greg" , "freebsd-net@freebsd.org" References: <20200613025031.GA34653@x270> <20200617001315.GB80914@x270> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 49mmXW6pvWz42F8 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=sOt4rPHo; dmarc=none; spf=pass (mx1.freebsd.org: domain of nparhar@gmail.com designates 2607:f8b0:4864:20::634 as permitted sender) smtp.mailfrom=nparhar@gmail.com X-Spamd-Result: default: False [-2.15 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_LONG(-1.03)[-1.034]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; NEURAL_HAM_SHORT(-0.50)[-0.497]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::634:from]; NEURAL_HAM_MEDIUM(-0.92)[-0.918]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[np@freebsd.org,nparhar@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[np@freebsd.org,nparhar@gmail.com]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jun 2020 00:43:56 -0000 No, it just enables some printf's in the LACP code and has no effect on anything else. Regards, Navdeep On Wed, Jun 17, 2020 at 12:31:33AM +0000, Saad, Mark wrote: > Navdeep > Thanks for getting back ; I’ll do some digging. Back to the > question about running with LACP debug on . Does this put the nics > into promiscuous mode ? > --- > Mark Saad | mark.saad@lucera.com > > > On Jun 16, 2020, at 8:13 PM, Navdeep Parhar wrote: > > > > We could have a global knob that tells all NIC drivers to use a reserved > > queue for non-RSS traffic, but that would be advisory at best because > > the tx queue selection takes place inside the driver's (or iflib's) > > transmit routine. The meat of the change is going to be in iflib and > > all non-iflib drivers' if_transmit. > > > > Regards, > > Navdeep > > > >> On Tue, Jun 16, 2020 at 09:48:19PM +0000, Saad, Mark wrote: > >> All > >> Is there any way to make this change on other nic's like Intel ix and > >> Solarflare sfxge ? I have seen similar issues on both with 12.1 > >> mainly with solarflare nics. > >> > >> --- > >> Mark Saad > >> mark.saad@lucera.com > >> > >> > >> ________________________________________ > >> From: owner-freebsd-net@freebsd.org on behalf of Foster, Greg > >> Sent: Tuesday, June 16, 2020 3:56 PM > >> To: Navdeep Parhar > >> Cc: freebsd-net@freebsd.org > >> Subject: RE: How to Increase TX Queue Priority for LACP Packets > >> > >> HI Navdeep, > >> > >> Thanks for the information! I've integrated the changes and will be > >> testing more today. > >> > >> We have seen the LACP port flapping under different scenarios, most we > >> believe are traffic/load based. > >> > >> I did see the flapping unexpectedly when I just enabled LACP debug > >> (e.g., sysctl net.link.lagg.lacp.debug=1). Is this a known > >> problem? > >> > >> Thanks > >> Greg > >> > >> -----Original Message----- > >> From: Navdeep Parhar On Behalf Of Navdeep Parhar > >> Sent: Friday, June 12, 2020 7:51 PM > >> To: Foster, Greg > >> Cc: freebsd-net@freebsd.org > >> Subject: Re: How to Increase TX Queue Priority for LACP Packets > >> > >>> On Fri, Jun 12, 2020 at 11:47:41PM +0000, Foster, Greg wrote: > >>> FreeBSD Networkers, > >>> > >>> We are seeing LACP port flapping on our FreeBSD 10.4/12.1 systems > >>> under different conditions. > >>> > >>> Can someone explain or point me to the information on how to queue > >>> the LACP packets to a higher priority queue ? > >>> > >>> We are using the Chelsio T580-LP-CR adapter/cxgbe driver. The > >>> Cheslio NICs have 8 TX/RX queues each, but I don't know how to > >>> explicitly put the LACP packets in the higher priority TX queue. > >>> > >>> I've read about PF/ALTQ and think this may be overkill our needs, > >>> and was wondering if there was a simpler method. > >> > >> This is cxgbe specific but that's what you're using so it'll do. > >> > >> Add "hw.cxgbe.rsrv_noflowq=1" to your /boot/loader.conf. That > >> reserves one tx queue for non-RSS traffic (like ARP, LACP). You might > >> also want to increase the number of tx queues to compensate for the > >> one that's now reserved. Use "hw.cxgbe.ntxq=9" for that. The ntxq > >> knob might be different on 10.4 but the man page matching the driver > >> should have its exact name. > >> > >> Regards, > >> Navdeep > >> > >> _______________________________________________ > >> freebsd-net@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-net > >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > >> > >