From nobody Wed Nov 10 05:05:33 2021 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6853618419EC for ; Wed, 10 Nov 2021 05:05:36 +0000 (UTC) (envelope-from jschauma@netmeister.org) Received: from panix.netmeister.org (panix.netmeister.org [166.84.7.99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hpt8b1vH4z3h6w for ; Wed, 10 Nov 2021 05:05:34 +0000 (UTC) (envelope-from jschauma@netmeister.org) Received: by panix.netmeister.org (Postfix, from userid 1000) id 02A968586F; Wed, 10 Nov 2021 00:05:33 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=netmeister.org; s=2021; t=1636520734; bh=QDAc5uHeEgE3NOasY+xBFJwMz2kH9OT08J5Z/vBC9zc=; h=Date:From:To:Subject:References:In-Reply-To; b=BTHxFB9lReHX08Bf6uOJqW1DkL+xHac7TBAlkvF0JxTI14CugXlVxEkGNAJ6qmUmf joEcH0tfn8ewZEAa87aIer3Qd+ByGrWC5KseCqJ+KgGe6rkKi8hbMUn1dW5MbH6fD2 2s+a/P2X3UT1TxHH4MvKAiNeoNdLABs7VXAOPhJY= Date: Wed, 10 Nov 2021 00:05:33 -0500 To: freebsd-net@freebsd.org Subject: Re: AF_UNIX socketpair dgram queue sizes Message-ID: <20211110050533.GA11277@netmeister.org> References: <20211110015719.GY3553@netmeister.org> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.1 (2018-12-01) X-Rspamd-Queue-Id: 4Hpt8b1vH4z3h6w X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=netmeister.org header.s=2021 header.b=BTHxFB9l; dmarc=pass (policy=quarantine) header.from=netmeister.org; spf=pass (mx1.freebsd.org: domain of jschauma@netmeister.org designates 166.84.7.99 as permitted sender) smtp.mailfrom=jschauma@netmeister.org X-Spamd-Result: default: False [-3.96 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.994]; R_DKIM_ALLOW(-0.20)[netmeister.org:s=2021]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-0.99)[-0.989]; RCPT_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[netmeister.org:+]; DMARC_POLICY_ALLOW(-0.50)[netmeister.org,quarantine]; NEURAL_HAM_SHORT(-0.98)[-0.977]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:2033, ipnet:166.84.0.0/16, country:US]; MID_RHS_MATCH_FROM(0.00)[] Reply-To: jschauma@netmeister.org From: Jan Schaumann via freebsd-net X-Original-From: Jan Schaumann X-ThisMailContainsUnwantedMimeParts: N Mark Johnston wrote: > There is an additional factor: wasted space. When writing data to a > socket, the kernel buffers that data in mbufs. All mbufs have some > amount of embedded storage, and the kernel accounts for that storage, > whether or not it's used. With small byte datagrams there can be a lot > of overhead; I'm observing two mbufs being allocated for each datagram for small datagrams, but only one mbuf for larger datagrams. That seems counter-intuitive to me? > The kern.ipc.sockbuf_waste_factor sysctl controls the upper limit on > total bytes (used or not) that may be enqueued in a socket buffer. The > default value of 8 means that we'll waste up to 7 bytes per byte of > data, I think. Setting it higher should let you enqueue more messages. Ah, this looks like something relevant. Setting kern.ipc.sockbuf_waste_factor=1, I can only write 8 1-byte datagrams. For any increase of the waste factor by one, I get another 8 1-byte datagrams, up until waste factor > 29, at which point we hit recvspace: 30 * 8 = 240, so 240 1-byte datagrams with 16 bytes dgram overhead means we get 240*17 = 4080 bytes, which just fits (well, with room for one empty 16-byte dgram) into the recvspace = 4096. But I still don't get the direct relationship between the waste factor and the recvspace / buffer queue: with a waste_factor of 1 and a datagram with 1972 bytes, I'm able to write one dgram with 1972 bytes + 1 dgram with 1520 bytes = 3492 bytes (plus 2 * 16 bytes overhead = 3524 bytes). There'd still have been space for 572 more bytes in the second dgram. Liekwise, trying to write a single 1973 dgram fills the queue and no additional bytes can be written in a second dgram, but I can write a single 2048 byte dgram. Still confused... -Jan