From owner-svn-src-all@freebsd.org Fri Feb 15 23:36:24 2019 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0344714EDADE; Fri, 15 Feb 2019 23:36:24 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 965F68BA7A; Fri, 15 Feb 2019 23:36:23 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 89B4625BF8; Fri, 15 Feb 2019 23:36:23 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x1FNaN57039323; Fri, 15 Feb 2019 23:36:23 GMT (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x1FNaNUo039321; Fri, 15 Feb 2019 23:36:23 GMT (envelope-from glebius@FreeBSD.org) Message-Id: <201902152336.x1FNaNUo039321@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: glebius set sender to glebius@FreeBSD.org using -f From: Gleb Smirnoff Date: Fri, 15 Feb 2019 23:36:23 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r344188 - in head: lib/libc/sys sys/vm X-SVN-Group: head X-SVN-Commit-Author: glebius X-SVN-Commit-Paths: in head: lib/libc/sys sys/vm X-SVN-Commit-Revision: 344188 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 965F68BA7A X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.97 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.997,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.97)[-0.973,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Feb 2019 23:36:24 -0000 Author: glebius Date: Fri Feb 15 23:36:22 2019 New Revision: 344188 URL: https://svnweb.freebsd.org/changeset/base/344188 Log: For 32-bit machines rollback the default number of vnode pager pbufs back to the lever before r343030. For 64-bit machines reduce it slightly, too. Together with r343030 I bumped the limit up to the value we use at Netflix to serve 100 Gbit/s of sendfile traffic, and it probably isn't a good default. Provide a loader tunable to change vnode pager pbufs count. Document it. Modified: head/lib/libc/sys/sendfile.2 head/sys/vm/vnode_pager.c Modified: head/lib/libc/sys/sendfile.2 ============================================================================== --- head/lib/libc/sys/sendfile.2 Fri Feb 15 22:55:13 2019 (r344187) +++ head/lib/libc/sys/sendfile.2 Fri Feb 15 23:36:22 2019 (r344188) @@ -25,7 +25,7 @@ .\" .\" $FreeBSD$ .\" -.Dd January 25, 2019 +.Dd February 15, 2019 .Dt SENDFILE 2 .Os .Sh NAME @@ -48,6 +48,7 @@ The system call sends a regular file or shared memory object specified by descriptor .Fa fd +h out a stream socket specified by descriptor .Fa s . .Pp @@ -224,6 +225,19 @@ implementation of .Fn sendfile is "zero-copy", meaning that it has been optimized so that copying of the file data is avoided. .Sh TUNING +.Ss physical paging buffers +.Fn sendfile +uses vnode pager to read file pages into memory. +The pager uses a pool of physical buffers to run its I/O operations. +When system runs out of pbufs, sendfile will block and report state +.Dq Li zonelimit . +Size of the pool can be tuned with +.Va vm.vnode_pbufs +.Xr loader.conf 5 +tunable and can be checked with +.Xr sysctl 8 +OID of the same name at runtime. +.Ss sendfile(2) buffers On some architectures, this system call internally uses a special .Fn sendfile buffer @@ -279,9 +293,11 @@ buffers usage respectively. These values may also be viewed through .Nm netstat Fl m . .Pp -If a value of zero is reported for -.Va kern.ipc.nsfbufs , -your architecture does not need to use +If +.Xr sysctl 8 +OID +.Va kern.ipc.nsfbufs +doesn't exist, your architecture does not need to use .Fn sendfile buffers because their task can be efficiently performed by the generic virtual memory structures. @@ -363,11 +379,13 @@ does not support The socket peer has closed the connection. .El .Sh SEE ALSO +.Xr loader.conf 5 , .Xr netstat 1 , .Xr open 2 , .Xr send 2 , .Xr socket 2 , .Xr writev 2 , +.Xr sysctl 8 , .Xr tuning 7 .Rs .%A K. Elmeleegy Modified: head/sys/vm/vnode_pager.c ============================================================================== --- head/sys/vm/vnode_pager.c Fri Feb 15 22:55:13 2019 (r344187) +++ head/sys/vm/vnode_pager.c Fri Feb 15 23:36:22 2019 (r344188) @@ -115,13 +115,23 @@ SYSCTL_PROC(_debug, OID_AUTO, vnode_domainset, CTLTYPE &vnode_domainset, 0, sysctl_handle_domainset, "A", "Default vnode NUMA policy"); +static int nvnpbufs; +SYSCTL_INT(_vm, OID_AUTO, vnode_pbufs, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, + &nvnpbufs, 0, "number of physical buffers allocated for vnode pager"); + static uma_zone_t vnode_pbuf_zone; static void vnode_pager_init(void *dummy) { - vnode_pbuf_zone = pbuf_zsecond_create("vnpbuf", nswbuf * 8); +#ifdef __LP64__ + nvnpbufs = nswbuf * 2; +#else + nvnpbufs = nswbuf / 2; +#endif + TUNABLE_INT_FETCH("vm.vnode_pbufs", &nvnpbufs); + vnode_pbuf_zone = pbuf_zsecond_create("vnpbuf", nvnpbufs); } SYSINIT(vnode_pager, SI_SUB_CPU, SI_ORDER_ANY, vnode_pager_init, NULL);