From owner-freebsd-arch@freebsd.org Fri Nov 13 19:09:33 2020 Return-Path: Delivered-To: freebsd-arch@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8A2C52EA756 for ; Fri, 13 Nov 2020 19:09:33 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CXp1R3vfpz4mKb for ; Fri, 13 Nov 2020 19:09:31 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: by mail-wm1-x332.google.com with SMTP id c9so9760000wml.5 for ; Fri, 13 Nov 2020 11:09:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:subject:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=bmH1E7Qgud8GfIefNnEsEP2XSb1QSnZNld06YaPmBFg=; b=VrcqP/+crwIXRp/JAhwPCCEKR+By1LNVoIojPmzdNk6CwslEnJEcvgVAhsGsikwzZl 7KXi5TIdxJ12FbaoeFGbl0LGSofkMbFz2eI1o8Cow1FlY13bmcFpIUOayUFr48dDSp86 e3wIaPXarNHUylLMCKwvl4k/tAkPFpyVVqE5iB5ZFTXIrw4dwOJkB+IjXNnrdrgriKi2 /cgif8ZA1C/tndd02pwGq5KpzpABfdg6GLsUbB4xm5kU3BzBHKgRWvJ0NKZaaSw6nqRl fT0mRjczZu/a3F6oUVKHrNmnEWBdFa60IzWv3wXkDUJUqiQFd8P4XuWVi2o/nDczrFkb prEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=bmH1E7Qgud8GfIefNnEsEP2XSb1QSnZNld06YaPmBFg=; b=gaZqKLNlPozq30qsbKeNSS00U8ZYUjIfMoM15ohN4+IVBx8rSiw4QuZZkMbghvJfc+ WSrX+XnlbN3OhwyWJNeGVuNxNkYgoOVsH13TrSD6wXp5cQ3AYk6LNBo/kaltO1tloQZo Ux9vmpSeZmwnwNd38RKdsXgee9j0YWJcswQZQ/0nj5DA/sDZoN+k9lr3BJrKDcO/+u89 hj+aKqR3hfN8oH/aWWUR1DbbLgvKa68BEhO5Ps3t1ubYLCoww7HG+QEMYU0ut0NhjsEk KukTRP3RT+dlwIAL7uE+0C2uBeX3Mbf07jnzcEtEPQTpAQ89uALPZLa4g4bzCh5b+2BG SgSg== X-Gm-Message-State: AOAM533Gm4dFG6/5VwQ4lZDnw6Yw5pOV/MQF92bW3Vvx+inD79COD7DX l2RQrXK2bmm0UBtEIgQINgcDbn8f9fI= X-Google-Smtp-Source: ABdhPJw30l3rLLin/fGKS5EN+68ayi6MXzqkg2md51cPvSM3vT8Jai6Gsy3LMFdPoN4viE/n2/7pcA== X-Received: by 2002:a1c:4b18:: with SMTP id y24mr4141654wma.154.1605294568717; Fri, 13 Nov 2020 11:09:28 -0800 (PST) Received: from ernst.home (p5b02350e.dip0.t-ipconnect.de. [91.2.53.14]) by smtp.gmail.com with ESMTPSA id h15sm12055119wrw.15.2020.11.13.11.09.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Nov 2020 11:09:27 -0800 (PST) Date: Fri, 13 Nov 2020 20:09:26 +0100 From: Gary Jennejohn To: "freebsd-arch@freebsd.org" Subject: Re: MAXPHYS bump for FreeBSD 13 Message-ID: <20201113190926.082fbdaf@ernst.home> In-Reply-To: References: Reply-To: gljennjohn@gmail.com X-Mailer: Claws Mail 3.17.5 (GTK+ 2.24.32; amd64-portbld-freebsd13.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4CXp1R3vfpz4mKb X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=VrcqP/+c; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of gljennjohn@gmail.com designates 2a00:1450:4864:20::332 as permitted sender) smtp.mailfrom=gljennjohn@gmail.com X-Spamd-Result: default: False [-3.96 / 15.00]; HAS_REPLYTO(0.00)[gljennjohn@gmail.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.96)[-0.961]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::332:from]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RECEIVED_SPAMHAUS_PBL(0.00)[91.2.53.14:received]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arch@freebsd.org]; FREEMAIL_REPLYTO(0.00)[gmail.com]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::332:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::332:from]; TO_DN_EQ_ADDR_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-arch] X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Nov 2020 19:09:33 -0000 On Fri, 13 Nov 2020 11:33:30 -0700 Warner Losh wrote: > Greetings, > > We currently have a MAXPHYS of 128k. This is the maximum size of I/Os that > we normally use (though there are exceptions). > > I'd like to propose that we bump MAXPHYS to 1MB, as well as bumping > DFLTPHYS to 1MB. > > 128k was good back in the 90s/2000s when memory was smaller, drives did > smaller I/Os, etc. Now, however, it doesn't make much sense. Modern I/O > devices can easily do 1MB or more and there's performance benefits from > scheduling larger I/Os. > > Bumping this will mean larger struct buf and struct bio. Without some > concerted effort, it's hard to make this be a sysctl tunable. While that's > desirable, perhaps, it shouldn't gate this bump. The increase in size for > 1MB is modest enough. > > The NVMe driver currently is limited to 1MB transfers due to limitations in > the NVMe scatter gather lists and a desire to preallocate as much as > possible up front. Most NVMe drivers have maximum transfer sizes between > 128k and 1MB, with larger being the trend. > > The mp[rs] drivers can use larger MAXPHYS, though resource limitations on > some cards hamper bumping it beyond about 2MB. > > The AHCI driver is happy with 1MB and larger sizes. > > Netflix has run MAXPHYS of 8MB for years, though that's likely 2x too large > even for our needs due to limiting factors in the upper layers making it > hard to schedule I/Os larger than 3-4MB reliably. > > So this should be a relatively low risk, and high benefit. > > I don't think other kernel tunables need to change, but I always run into > trouble with runningbufs :) > > Comments? Anything I forgot? > Seems like a good idea to me. I tried 1MB a few months ago and saw no problems, although that change had little effect on transfers to my SSDs. Still, it could be useful for spinning rust. -- Gary Jennejohn