From owner-svn-src-head@freebsd.org Wed Dec 18 09:08:13 2019 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 320641D94AF; Wed, 18 Dec 2019 09:08:13 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [IPv6:2a01:4f8:c17:6c4b::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47d8LN2dSXz4MJm; Wed, 18 Dec 2019 09:08:12 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2016.home.selasky.org (unknown [62.141.129.235]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 06392260197; Wed, 18 Dec 2019 09:57:29 +0100 (CET) Subject: Re: svn commit: r341578 - head/sys/dev/mlx5/mlx5_en From: Hans Petter Selasky To: "Andrey V. Elsukov" , Slava Shwartsman , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org References: <201812051420.wB5EKwxr099242@repo.freebsd.org> <8715aa7b-ceea-7cfd-1980-a260c98070d5@yandex.ru> <6151923d-05bb-b945-359e-d632708ba021@selasky.org> <60698d75-ca26-da71-682c-ec7d24f0eab6@yandex.ru> <8a1d1216-f15d-7fe0-b55c-04b999f8a947@selasky.org> Message-ID: Date: Wed, 18 Dec 2019 09:56:01 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.1.2 MIME-Version: 1.0 In-Reply-To: <8a1d1216-f15d-7fe0-b55c-04b999f8a947@selasky.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 47d8LN2dSXz4MJm X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 2a01:4f8:c17:6c4b::2 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-4.93 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[selasky.org]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-2.63)[ip: (-9.18), ipnet: 2a01:4f8::/29(-2.42), asn: 24940(-1.55), country: DE(-0.02)]; FREEMAIL_TO(0.00)[yandex.ru]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Dec 2019 09:08:13 -0000 On 2019-12-18 07:26, Hans Petter Selasky wrote: > On 2019-12-17 18:14, Andrey V. Elsukov wrote: >> On 13.12.2019 17:27, Hans Petter Selasky wrote: >>> On 2019-12-13 14:40, Andrey V. Elsukov wrote: >>>> On 05.12.2018 17:20, Slava Shwartsman wrote: >>>>> Author: slavash >>>>> Date: Wed Dec  5 14:20:57 2018 >>>>> New Revision: 341578 >>>>> URL: https://svnweb.freebsd.org/changeset/base/341578 >>>>> >>>>> Log: >>>>>     mlx5en: Remove the DRBR and associated logic in the transmit path. >>>>>        The hardware queues are deep enough currently and using the >>>>> DRBR and associated >>>>>     callbacks only leads to more task switching in the TX path. The is >>>>> also a race >>>>>     setting the queue_state which can lead to hung TX rings. >>>> >>>> JFYI. We have compared the same router+firewall workloads on the host >>>> with this change and before, and I can say, that without DRBR on TX now >>>> we constantly have several percents of packets drops due to ENOBUFS >>>> error from mlx5e_xmit(). >>>> >>> >>> Have you tried to tune the TX/RX parameters? >>> >>> Especially the tx_queue_size . >> >> We use the following settings: >> % sysctl dev.mce.4.conf. | grep que >> dev.mce.4.conf.rx_queue_size: 16384 >> dev.mce.4.conf.tx_queue_size: 16384 >> dev.mce.4.conf.rx_queue_size_max: 16384 >> dev.mce.4.conf.tx_queue_size_max: 16384 >> >> Also, previously I have patched MLX5E_SQ_TX_QUEUE_SIZE value up to 16384. > > Hi, > > What about the other parameters. Did you tune any of those? > > At what rate does this happen? > > Can you send me the full dev.mce.4 sysctl tree off-list? > Are you using any performance options like RSS in the kernel? How many NUMA domains does this machine have? Have you tuned the driver threads, like binding interrupt threads to CPU's? --HPS