From owner-svn-src-head@freebsd.org Mon Dec 17 14:25:56 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9277D134155D; Mon, 17 Dec 2018 14:25:56 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D1BCE6FF4F; Mon, 17 Dec 2018 14:25:55 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from [192.168.200.4] (c-71-56-186-158.hsd1.va.comcast.net [71.56.186.158]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: gallatin) by duke.cs.duke.edu (Postfix) with ESMTPSA id 2FA6727002EB; Mon, 17 Dec 2018 09:25:48 -0500 (EST) DMARC-Filter: OpenDMARC Filter v1.3.1 duke.cs.duke.edu 2FA6727002EB DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=cs.duke.edu; s=mail0816; t=1545056748; bh=R4VQuk+BCU7VuQFj76xOoaj9TIpWQ2atUDeslu6APo0=; h=Subject:To:From:Date:From; b=tU4Hz9lyW4Ue+zZ6lcuPCj8wVt8rAX/K4oVocb+z9ae0PKEg8fgJ0hQeBm/MU+mQS 5VZjID1KEOnRNhVgOvTZ6JXHC1apIYftPIn7waRjyyjdoSFrnDZkJUDkzM2ohFe1m/ 1tvQJscbsJyMqYxw8p2DYZBiPJodKQtjHLzebUxBITf+eeAkgughnlT7q+1azEB9ry BvmOLtEUTUjP73mDybeeybC9ORsU1B7mpGM0xDmRgYlcai7wIEdy9D6OiXFv827RTT MxM0aQNYpQyEUX23mVPCIzF9HCIFufF/AXu8Jr0nJ3mWUwBjhaHiV3VMOTiWeE4mdu IZdhIFoWSQhlA== Subject: Re: svn commit: r341578 - head/sys/dev/mlx5/mlx5_en To: Slava Shwartsman , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org References: <201812051420.wB5EKwxr099242@repo.freebsd.org> From: Andrew Gallatin Message-ID: <9e09a2f8-d9fd-7fde-8e5a-b7c566cdb6a9@cs.duke.edu> Date: Mon, 17 Dec 2018 09:25:47 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <201812051420.wB5EKwxr099242@repo.freebsd.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: D1BCE6FF4F X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.duke.edu header.s=mail0816 header.b=tU4Hz9ly; dmarc=pass (policy=none) header.from=cs.duke.edu; spf=pass (mx1.freebsd.org: domain of gallatin@cs.duke.edu designates 152.3.140.1 as permitted sender) smtp.mailfrom=gallatin@cs.duke.edu X-Spamd-Result: default: False [-4.45 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[cs.duke.edu:s=mail0816]; NEURAL_HAM_MEDIUM(-0.95)[-0.954,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:152.3.140.0/23]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.83)[-0.827,0]; DKIM_TRACE(0.00)[cs.duke.edu:+]; RCVD_IN_DNSWL_MED(-0.20)[1.140.3.152.list.dnswl.org : 127.0.11.2]; DMARC_POLICY_ALLOW(-0.50)[cs.duke.edu,none]; MX_GOOD(-0.01)[mx.oit.duke.edu]; NEURAL_HAM_SHORT(-0.94)[-0.944,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-0.02)[country: US(-0.08)]; ASN(0.00)[asn:13371, ipnet:152.3.128.0/17, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Dec 2018 14:25:56 -0000 On 12/5/18 9:20 AM, Slava Shwartsman wrote: > Author: slavash > Date: Wed Dec 5 14:20:57 2018 > New Revision: 341578 > URL: https://urldefense.proofpoint.com/v2/url?u=https-3A__svnweb.freebsd.org_changeset_base_341578&d=DwIDaQ&c=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc&r=Ed-falealxPeqc22ehgAUCLh8zlZbibZLSMWJeZro4A&m=BFp2c_-S0jnzRZJF2APwvTwmnmVFcyjcnBvHRZ3Locc&s=b7fvhOzf_b5bMVGquu4SaBhMNql5N8dVPAvpfKtz53Q&e= > > Log: > mlx5en: Remove the DRBR and associated logic in the transmit path. > > The hardware queues are deep enough currently and using the DRBR and associated > callbacks only leads to more task switching in the TX path. The is also a race > setting the queue_state which can lead to hung TX rings. > The point of DRBR in the tx path is not simply to provide a software ring for queuing excess packets. Rather it provides a mechanism to avoid lock contention by shoving a packet into the software ring, where it will later be found & processed, rather than blocking the caller on a mtx lock. I'm concerned you may have introduced a performance regression for use cases where you have N:1 or N:M lock contention where many threads on different cores are contending for the same tx queue. The state of the art for this is no longer DRBR, but mp_ring, as used by both cxgbe and iflib. For well behaved workloads (like Netflix's), I don't anticipate this being a performance issue. However, I worry that this will impact other workloads and that you should consider running some testing of N:1 contention. Eg, 128 netperfs running in parallel with only a few nic tx rings. Sorry for the late reply.. I'm behind on my -committers email. If you have not already MFC'ed this, you may want to reconsider. Drew