From owner-svn-src-head@freebsd.org Sun Jan 26 18:44:59 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0D88C1F7724 for ; Sun, 26 Jan 2020 18:44:59 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from mail-pl1-x641.google.com (mail-pl1-x641.google.com [IPv6:2607:f8b0:4864:20::641]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 485MHt0qxpz4Myg for ; Sun, 26 Jan 2020 18:44:57 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: by mail-pl1-x641.google.com with SMTP id t14so2928574plr.8 for ; Sun, 26 Jan 2020 10:44:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jroberson-net.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version; bh=XNu/ssFXMbOE/CMfDR9xJEQcXIATni4dw2ediEV7/M4=; b=ctrm0F64PIhLKbDsPtN7sZJ+i5xCRK9fxoY3h3xa3MAVg7Y5iMnLH+HK6CwePYZTSi KTJoJE6ZI10z5L5jdrJ16MIKd8iGK7vTYugjp5HlpfJXv16c3SrVjm7nnhLKpOVdAf0U A1KbjKkE1yQaKxxj+wVDGHYgP9Oj7PrybZLQt6eA7rmjNh4dR9R9bHZX17HD+c+TXceR IqKGjgtzqsVdEYvA6UJMjE6U2xWzZfvPHEsQI1E5Gw9F5pXWofdF+aeDThf7ddkCht9v 9RkoiWhq5/WapdC4YhyUH1ApBiatbRNWbUzwYS5BrLHxfCVruVEp5+moZzQkiUZcsEl8 oG7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version; bh=XNu/ssFXMbOE/CMfDR9xJEQcXIATni4dw2ediEV7/M4=; b=nyClJSL3QnNS+StXZM/mTtIbRfx6wSicGoDlujNqXi7kyh3Y6OGRpixLPfamQ34KTw QU8gd1hGyXYd9Fqg8/TratY+DqqEEz+S/ZZbeBi9pCzJBJP/xd3UF5Rk+KVMS/jnddac lTHX1W/tupskbyqixK0i6P9kuk21EtBZonJsxsVNQGIfev3fGd33ThQxOMfGNVuofAjm v1ASk2nBjLVtmvMN5cJDFIF29lZKut6gygLfC+G1hDJZLh1yKy0WrkjToed4WUPnJwwd Ux2nYNW0SsxP4wUpEKMO5JwO1Dk0XRPbdXZy0BT3UVtKCN2huvIhN1yBpVs+Hc+yuMlY 5yJA== X-Gm-Message-State: APjAAAVm+hVSgsTxL6clnINNOIlOwB1ntLHA1RmTWI22zVg9cISxVQqo JK3s3yzq3eB5aFYLtRPtmgCQLw== X-Google-Smtp-Source: APXvYqyYRefMaAXjXom2RXow2LY8pcfBjEaf37UdU+JkPRUt+vw/l7E6eDSnyV9uQY8Vip5rQKlD6A== X-Received: by 2002:a17:90a:7187:: with SMTP id i7mr6110005pjk.6.1580064296530; Sun, 26 Jan 2020 10:44:56 -0800 (PST) Received: from rrcs-76-81-105-82.west.biz.rr.com (rrcs-76-81-105-82.west.biz.rr.com. [76.81.105.82]) by smtp.gmail.com with ESMTPSA id b185sm13360033pfa.102.2020.01.26.10.44.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 26 Jan 2020 10:44:56 -0800 (PST) Date: Sun, 26 Jan 2020 08:44:53 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: John Baldwin cc: Gleb Smirnoff , Ryan Stone , src-committers , svn-src-all , svn-src-head Subject: Re: svn commit: r357051 - head/sys/dev/bge In-Reply-To: <6a5485eb-8623-3316-0385-b0cda1624b05@FreeBSD.org> Message-ID: References: <202001231636.00NGawrr080128@repo.freebsd.org> <20200123230546.GG1268@FreeBSD.org> <20200124012458.GI1268@FreeBSD.org> <20200124024356.GK1268@FreeBSD.org> <20200124033243.GL1268@FreeBSD.org> <6a5485eb-8623-3316-0385-b0cda1624b05@FreeBSD.org> User-Agent: Alpine 2.21.9999 (BSF 287 2018-06-16) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 485MHt0qxpz4Myg X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=jroberson-net.20150623.gappssmtp.com header.s=20150623 header.b=ctrm0F64; dmarc=none; spf=none (mx1.freebsd.org: domain of jroberson@jroberson.net has no SPF policy when checking 2607:f8b0:4864:20::641) smtp.mailfrom=jroberson@jroberson.net X-Spamd-Result: default: False [-2.71 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[jroberson-net.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[svn-src-head@freebsd.org]; DMARC_NA(0.00)[jroberson.net]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[jroberson-net.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[1.4.6.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(-0.91)[ip: (-0.66), ipnet: 2607:f8b0::/32(-2.05), asn: 15169(-1.79), country: US(-0.05)] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Jan 2020 18:44:59 -0000 On Sun, 26 Jan 2020, John Baldwin wrote: > On 1/23/20 7:32 PM, Gleb Smirnoff wrote: >> On Thu, Jan 23, 2020 at 05:09:14PM -1000, Jeff Roberson wrote: >> J> While we don't have a policy strictly requiring reviews it is the norm to >> J> have substantial changes socialized and reviewed. I appreciate the work >> J> that you are doing but it likely should've been discussed somewhere >> J> more publicly. I apologized if I missed it but I don't see reference to >> J> anything. >> >> That was https://reviews.freebsd.org/D23242 > > A review alone isn't sufficient for large, sweeping changes in my mind. > For major changes, a thread on arch@ or net@ or the like is probably more > appropriate. You can include a link to a review or git branch, etc. in > that e-mail, but phabricator aren't as well suited to higher-level > design-review type discussion, more for implementation-review. >> J> Architecturally I am more concerned with the coarseness of net_epoch and >> J> the duration of hold becoming a resource utilization problem in high >> J> turn-over workloads. Like short connection tcp. Has anyone done >> J> substantial testing here? epoch as it is today will hold every free >> J> callback for a minimum of several clock ticks and a maximum of 2x the >> J> duration of the longest epoch section time. With preemption, etc. this >> J> could be 100s of ms of PCBs held. >> >> We also are concerned about that theoretically. Haven't yet seen effect >> in practice, but our sessions are mostly longer living. First we have the >> tunable to limit batching. Second, there are some ideas on how to improve >> the garbage collector performance if it becomes an issue. > > There are other workloads than Netflix. ;) Verisign has incredibly short-lived > connections with very high turnover. I think though that they have already > abandoned the in-tree network stack for a userland stack built on netmap. Still, > I think that there are probably other FreeBSD users that are probably somewhere > in the middle that shouldn't be ignored. > > Packet batching would not be impossible by simply using m_nextpkt chains in > mbufs passed up to ether_input and having ether_input pass them in a loop to > the next higher loop (as a first step). That would reduce unlock/lock operations > in drivers (for those still using locks on receive) as well as permitting > ether_input to process batches under a single epoch invocation. This is actually the approach that I took for nokia. You could prefetch m->m_nextpkt at the top of the loop iteration. It was very effective there. Seeing how many drivers and unexpected entry points had to have the NET_EPOCH added I would want to review again once it's stable and see if there is a way to simplify through API changes. It seems like more than expected slipped through the cracks and I worry about long-term maintenance. Thanks, Jeff > > -- > John Baldwin >