From owner-freebsd-arch@freebsd.org Fri Nov 13 18:33:46 2020 Return-Path: Delivered-To: freebsd-arch@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7D38D2EA27E for ; Fri, 13 Nov 2020 18:33:46 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x734.google.com (mail-qk1-x734.google.com [IPv6:2607:f8b0:4864:20::734]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CXnD91gg2z4kD0 for ; Fri, 13 Nov 2020 18:33:44 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x734.google.com with SMTP id r7so9724971qkf.3 for ; Fri, 13 Nov 2020 10:33:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=r1PowvalaeaThrpHZinHU3BJs6Oitirr5nvgxz9+jgw=; b=nnZdg38BNOWjLhVuNjIgUSksEveh2M31XzFEqEk/xmEoShcqZXiLGng5ArNRb5W2nc EIB5/aBu7UplFewjoCcEAgQSn6ZLZaB3Rf9QXBwImCxITnhnVbVnWEHZMu5nEia1nkqS RcDWMve9D4dkSiXoSbFwVJ+xE+kWknwGDmcXv7MRP5EVDN1/15gi5T/yN/a3dO7kQsPi dFyGTAPLpgIygfQFowUnr+ZFApiPqoI9crXoAfuseN5Y0jB3Ok8eMpyPdC2mAlOcSByy 3qT8I8rvXMIgr4JA/7wQTd/J9vceowA0KIjTbuXVcQHwcPbFYpgwZ74m4wqc848P/JTw WJ/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=r1PowvalaeaThrpHZinHU3BJs6Oitirr5nvgxz9+jgw=; b=lkQA62cyr62RH2LNmMNRGEpxxSTmG1RePlaj4vW2yfxb5oi9iWfWKzZ8GOs9sAzoYJ xdLjFgsugee9dC/1KMjPm2Y4QV7PH8my/OK0hRnhUTH/StOEt5C5DAJ9BK9xCqjgm1B0 0ITay9oD42e17vXacyuglnseq+4O+LV+/eao3ntGqsUO/dQUL1RorpZV0xMWTWgXgrg1 Vj4l5qSAINX6CXvAek5R/0JDKa9vC++rL0YVskMNFxqOTycSakmbe6jPmYetI9TNcxqF 4Aqrb2oH5GXtpQ6WN8W9KRJuZjVGvUr02DeNBuBc4VZQPB1rB2tQx7VeW4W14bSrUB75 b6aw== X-Gm-Message-State: AOAM533jPxSAx8OXTAcVLo5Qf7o38Mrh2ceSJelHl7mQlxwvEWeGIrB+ 4WEOiuhA4Nncqh9HPdt0n/XH14GFMv9o4WfU7wCUKXP2BGPVNg== X-Google-Smtp-Source: ABdhPJxWYe4/yN3dgjHmpquJ1LDoOdfZ3+i12lBq9BqOG+6xp+en/ahwg+Le9lyzagFZYMqVyvObO9XZ5+b6WP160oc= X-Received: by 2002:a37:e207:: with SMTP id g7mr3361018qki.44.1605292421511; Fri, 13 Nov 2020 10:33:41 -0800 (PST) MIME-Version: 1.0 From: Warner Losh Date: Fri, 13 Nov 2020 11:33:30 -0700 Message-ID: Subject: MAXPHYS bump for FreeBSD 13 To: "freebsd-arch@freebsd.org" X-Rspamd-Queue-Id: 4CXnD91gg2z4kD0 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=nnZdg38B; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::734) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RCVD_COUNT_TWO(0.00)[2]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arch@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::734:from:127.0.2.255]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-0.998]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::734:from]; TO_DN_EQ_ADDR_ALL(0.00)[]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::734:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_ALL(0.00)[]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; MAILMAN_DEST(0.00)[freebsd-arch] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Nov 2020 18:33:46 -0000 Greetings, We currently have a MAXPHYS of 128k. This is the maximum size of I/Os that we normally use (though there are exceptions). I'd like to propose that we bump MAXPHYS to 1MB, as well as bumping DFLTPHYS to 1MB. 128k was good back in the 90s/2000s when memory was smaller, drives did smaller I/Os, etc. Now, however, it doesn't make much sense. Modern I/O devices can easily do 1MB or more and there's performance benefits from scheduling larger I/Os. Bumping this will mean larger struct buf and struct bio. Without some concerted effort, it's hard to make this be a sysctl tunable. While that's desirable, perhaps, it shouldn't gate this bump. The increase in size for 1MB is modest enough. The NVMe driver currently is limited to 1MB transfers due to limitations in the NVMe scatter gather lists and a desire to preallocate as much as possible up front. Most NVMe drivers have maximum transfer sizes between 128k and 1MB, with larger being the trend. The mp[rs] drivers can use larger MAXPHYS, though resource limitations on some cards hamper bumping it beyond about 2MB. The AHCI driver is happy with 1MB and larger sizes. Netflix has run MAXPHYS of 8MB for years, though that's likely 2x too large even for our needs due to limiting factors in the upper layers making it hard to schedule I/Os larger than 3-4MB reliably. So this should be a relatively low risk, and high benefit. I don't think other kernel tunables need to change, but I always run into trouble with runningbufs :) Comments? Anything I forgot? Warner