From owner-freebsd-net@freebsd.org Tue Mar 24 19:14:12 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 058C6267EAB for ; Tue, 24 Mar 2020 19:14:12 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-qv1-xf32.google.com (mail-qv1-xf32.google.com [IPv6:2607:f8b0:4864:20::f32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48n1Bd0KKBz4dqr; Tue, 24 Mar 2020 19:14:00 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-qv1-xf32.google.com with SMTP id g4so3135228qvo.12; Tue, 24 Mar 2020 12:14:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=kQpei7Qexd8vUpQ/2UdVMwkEyplf683x+1z/ERfqFNY=; b=hCgIpi+rbuctyHIo90fKCpCmVPq79xbCGvQWj1WrvzKs46hBKAc6jVKEnamP4uni7J 3FVm1a7X0GQnJtFVMegXhILOF9Gu2VKA+pckAdIRBqIUxRGw+SePkJqB2V31DEl5ug8q NuidHEnltwAzQn+AN7flAvj7phAtKehFHm8iwsTeCjvLVtGR7ZfTq68UXBwqRd7+TsMe ViYSF8oe9fjzz9tqCpFPVjgPtIIndFKbwaShqomJByumqi3q3a/CpLRYfjZJ8RGjJw8D wGvRlMAzWUmQlf4tSBQhIGVCNWcddU1CNNMGEWmFskMkP1bjfXFkDTvW+kcAb9weIMvz Ct6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=kQpei7Qexd8vUpQ/2UdVMwkEyplf683x+1z/ERfqFNY=; b=qxwwb6V2IfSAbsQ13ZtoxnlI61VWebWN9T6jfvmlkoTCVdQZ471M1qj5OK85502uSL aAgyHcyidUgZldSfmr5Simzgx+hvZljrLDTDVJk9l87fmhTp9qTuxQr48B2kx+Cb9cA+ ww2BO9/v2Te+o+TmrqTC4RRsyPpDfTmFfglSD7dux28gApGfhb4wxuZSlvC+1tXs3JGe XIW4v93Qqebp1SV3DT6jHGoJpDv6Rdl8wM23R/9xViJ4D787MTpiTwPTW427Tc4U4+Yj a3N2kzeVEtbYlBSoY/U13+TtEIWi9qryGnzwdByciWzsYptth8NY2Ui/vIfMBA0bmtG2 R1UA== X-Gm-Message-State: ANhLgQ0hnt8WvfGjeC26D08CHL666Ds9Or1MUs8d+XqV/RD69eaH2Zex cV7L8nbH4+nAk9ogcAs88oYbaI3uOuobGKLHrb22dJOk X-Google-Smtp-Source: ADFU+vu53+lFaFOnfBGQYh3mUNocDOvNzoBqP6RiH/kpJsRrzkDI2anmRF1etImwqrmgnx+UhiOSvzxrRlzkUpXLEes= X-Received: by 2002:a0c:aee5:: with SMTP id n37mr25958857qvd.173.1585077231245; Tue, 24 Mar 2020 12:13:51 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ryan Stone Date: Tue, 24 Mar 2020 15:13:40 -0400 Message-ID: Subject: Re: Can net.iflib.min_tx_latency=0 introduce an unbounded amount of latency? To: Patrick Kelsey Cc: freebsd-net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 48n1Bd0KKBz4dqr X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=hCgIpi+r; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rysto32@gmail.com designates 2607:f8b0:4864:20::f32 as permitted sender) smtp.mailfrom=rysto32@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE_FREEMAIL(0.00)[]; IP_SCORE(0.00)[ip: (-7.67), ipnet: 2607:f8b0::/32(-0.38), asn: 15169(-0.49), country: US(-0.05)]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2.3.f.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 19:14:13 -0000 Thanks for the analysis. I haven't had much of an opportunity to get into the guts of iflib and I'm not very successful at following the round trips through mpring yet. On Mon, Mar 23, 2020 at 12:50 AM Patrick Kelsey wrote= : > > > > On Sun, Mar 22, 2020 at 2:31 PM Ryan Stone wrote: >> >> I've been tracking down a bug at work that appears to be due to >> excessive latency introduced on the TX side of a TCP connection. In >> looking through the iflib code, I noticed the tunable >> net.iflib.min_tx_latency. My reading of the iflib code is that if a >> packet is enqueued to a tx ring but not posted (when >> min_tx_latency=3D0), then iflib can introduce an unbounded amount of >> latency if the ring is idle. Is my reading of the code correct? >> > > Let's see...in the case described above, the next iflib_timer() invocatio= n (max 500ms later) should find that txq->ift_db_pending is non-zero, which= will cause it to enqueue the TX task, which will invoke _task_fn_tx(), whi= ch will then enqueue the special marker (void **)&txq, and one way or anoth= er, iflib_txq_drain() will be called. > > Looking at iflib_txq_drain() it appears that the first call to iflib_txd_= db_check() should post the pending packets either because a non-zero number= of slots were just reclaimed, or the check for txq->ift_db_pending >=3D TX= Q_MAX_DB_DEFERRED(...) is satisfied, unless there are too many posted packe= ts in the hardware queue that aren't being drained by the hardware (this wo= uld result in the in_use parameter remaining elevated such that a small num= ber of pending packets would not exceed the threshold, and also no slot rec= laim occurring). > > -Patrick