From owner-svn-src-all@freebsd.org Wed Dec 18 06:27:57 2019 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E63F11D5753; Wed, 18 Dec 2019 06:27:57 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [IPv6:2a01:4f8:c17:6c4b::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47d4nS6Rmsz4Dsw; Wed, 18 Dec 2019 06:27:56 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2016.home.selasky.org (unknown [62.141.129.235]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 39750260197; Wed, 18 Dec 2019 07:27:47 +0100 (CET) Subject: Re: svn commit: r341578 - head/sys/dev/mlx5/mlx5_en To: "Andrey V. Elsukov" , Slava Shwartsman , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org References: <201812051420.wB5EKwxr099242@repo.freebsd.org> <8715aa7b-ceea-7cfd-1980-a260c98070d5@yandex.ru> <6151923d-05bb-b945-359e-d632708ba021@selasky.org> <60698d75-ca26-da71-682c-ec7d24f0eab6@yandex.ru> From: Hans Petter Selasky Message-ID: <8a1d1216-f15d-7fe0-b55c-04b999f8a947@selasky.org> Date: Wed, 18 Dec 2019 07:26:46 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.1.2 MIME-Version: 1.0 In-Reply-To: <60698d75-ca26-da71-682c-ec7d24f0eab6@yandex.ru> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 47d4nS6Rmsz4Dsw X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 2a01:4f8:c17:6c4b::2 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-4.93 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[selasky.org]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-2.63)[ip: (-9.18), ipnet: 2a01:4f8::/29(-2.42), asn: 24940(-1.55), country: DE(-0.02)]; FREEMAIL_TO(0.00)[yandex.ru]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Dec 2019 06:27:58 -0000 On 2019-12-17 18:14, Andrey V. Elsukov wrote: > On 13.12.2019 17:27, Hans Petter Selasky wrote: >> On 2019-12-13 14:40, Andrey V. Elsukov wrote: >>> On 05.12.2018 17:20, Slava Shwartsman wrote: >>>> Author: slavash >>>> Date: Wed Dec  5 14:20:57 2018 >>>> New Revision: 341578 >>>> URL: https://svnweb.freebsd.org/changeset/base/341578 >>>> >>>> Log: >>>>    mlx5en: Remove the DRBR and associated logic in the transmit path. >>>>       The hardware queues are deep enough currently and using the >>>> DRBR and associated >>>>    callbacks only leads to more task switching in the TX path. The is >>>> also a race >>>>    setting the queue_state which can lead to hung TX rings. >>> >>> JFYI. We have compared the same router+firewall workloads on the host >>> with this change and before, and I can say, that without DRBR on TX now >>> we constantly have several percents of packets drops due to ENOBUFS >>> error from mlx5e_xmit(). >>> >> >> Have you tried to tune the TX/RX parameters? >> >> Especially the tx_queue_size . > > We use the following settings: > % sysctl dev.mce.4.conf. | grep que > dev.mce.4.conf.rx_queue_size: 16384 > dev.mce.4.conf.tx_queue_size: 16384 > dev.mce.4.conf.rx_queue_size_max: 16384 > dev.mce.4.conf.tx_queue_size_max: 16384 > > Also, previously I have patched MLX5E_SQ_TX_QUEUE_SIZE value up to 16384. Hi, What about the other parameters. Did you tune any of those? At what rate does this happen? Can you send me the full dev.mce.4 sysctl tree off-list? --HPS