Date: Wed, 21 Jun 2023 01:29:48 +0100 From: Graham Perrin <grahamperrin@freebsd.org> To: freebsd-current@freebsd.org Subject: Re: kernel: sonewconn: pcb 0xfffff8002b255a00 (local:/var/run/devd.seqpacket.pipe): Listen queue overflow: 1 already in queue awaiting acceptance (60 occurrences), ? Message-ID: <b815622c-7a64-bb7b-74f9-72087c8ac461@freebsd.org> In-Reply-To: <20230620164141.209ea8bc@ernst.home> References: <3567b4c0-8143-67c8-fde7-5ae3923f53f4@freebsd.org> <20230620094108.351849f5@ernst.home> <20230620120413.Horde.E7_jFO0w9ui-Nbkh9PONJjn@webmail.leidinger.net> <20230620164141.209ea8bc@ernst.home>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--------------w6eFlrm2O743bbKV0VCXZJuk
Content-Type: multipart/mixed; boundary="------------ukgM1NkREF79h3ml1nw0XaRg";
protected-headers="v1"
From: Graham Perrin <grahamperrin@freebsd.org>
To: freebsd-current@freebsd.org
Message-ID: <b815622c-7a64-bb7b-74f9-72087c8ac461@freebsd.org>
Subject: Re: kernel: sonewconn: pcb 0xfffff8002b255a00
(local:/var/run/devd.seqpacket.pipe): Listen queue overflow: 1 already in
queue awaiting acceptance (60 occurrences), ?
References: <3567b4c0-8143-67c8-fde7-5ae3923f53f4@freebsd.org>
<20230620094108.351849f5@ernst.home>
<20230620120413.Horde.E7_jFO0w9ui-Nbkh9PONJjn@webmail.leidinger.net>
<20230620164141.209ea8bc@ernst.home>
In-Reply-To: <20230620164141.209ea8bc@ernst.home>
--------------ukgM1NkREF79h3ml1nw0XaRg
Content-Type: multipart/alternative;
boundary="------------MdmP9XZQXIv5dXomZoKU8CqM"
--------------MdmP9XZQXIv5dXomZoKU8CqM
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: base64
T24gMjAvMDYvMjAyMyAxNTo0MSwgR2FyeSBKZW5uZWpvaG4gd3JvdGU6DQo+IE9uIFR1ZSwg
MjAgSnVuIDIwMjMgMTI6MDQ6MTMgKzAyMDANCj4gQWxleGFuZGVyIExlaWRpbmdlcjxBbGV4
YW5kZXJAbGVpZGluZ2VyLm5ldD4gIHdyb3RlOg0KPg0KPj4gUXVvdGluZyBHYXJ5IEplbm5l
am9objxnYXJ5akBnbXguZGU+ICAoZnJvbSBUdWUsIDIwIEp1biAyMDIzIDA3OjQxOjA4ICsw
MDAwKToNCj4+DQo+Pj4g4oCmDQo+PiBJbiBvdGhlciB3b3JkcyB0aGUgc29mdHdhcmUgbGlz
dGVuaW5nIG9uIGl0IGRpZG4ndCBwcm9jZXNzIHRoZSByZXF1ZXN0DQo+PiBmYXN0IGVub3Vn
aCBhbmQgYSBiYWNrbG9nIHBpbGVkIHVwIChlLmcgYXBhY2hlIExpc3RlbkJhY2tsb2cgb3Ig
bmdpbngNCj4+ICJsaXN0ZW4gWCBiYWNrbG9nPXkiIGFuZCAic3lzY3RsIGtlcm4uaXB4LnNv
bWF4Y29ubj1YIiBmb3IgRnJlZUJTRA0KPj4gaXRzZWxmKS4gWW91IG1heSBuZWVkIGZhc3Rl
ciBoYXJkd2FyZSwgbW9yZSBwcm9jZXNzZXMvdGhyZWFkcyB0bw0KPj4gaGFuZGxlIHRoZSB0
cmFmZmljLCBvciBjb25maWd1cmUgeW91ciBzb2Z0d2FyZSB0byBkbyBsZXNzIHRvIHByb2R1
Y2UNCj4+IHRoZSBzYW1lIHJlc3VsdCAoZS5nLiBubyByZWFsLXRpbWUgRE5TIHJlc29sdXRp
b24gaW4gdGhlIGxvZ2dpbmcgb2YgYQ0KPj4gd2Vic2VydmVyIG9yIGluY3JlYXNpbmcgdGhl
IGFtb3VudCBvZiBhbGxvd2VkIGl0ZW1zIGluIHRoZSBiYWNrbG9nKS4NCj4+IElmIHlvdSBj
YW4gY2hhbmdlIHRoZSBzb2Z0d2FyZSwgdGhlcmUncyBhbHNvIHRoZSBwb3NzaWJpbGl0eSB0
byBzd2l0Y2gNCj4+IGZyb20gYmxvY2tpbmcgc29ja2V0cyB0byBub24tYmxvY2tpbmcgc29j
a2V0cyAodG8gbm90IGhhdmUgdGhlDQo+PiBzZWxlY3QvYWNjZXB0IGxvb3AgYmxvY2sgLyBy
dW4gaW50byBjb250ZW50aW9uKSBvciBrcXVldWUuDQo+Pg0KPiBPbiBteSBGcmVlQlNEMTQg
c3lzdGVtIHRoZXNlIHRoaW5ncyBhcmUgYWxsIHVuZGVyIGtlcm4uaXBjLg0KPg0KPiBtYXhj
b25uIHNlZW1zIHRvIGJlIGFuIGFsaWFzIGZvciBzb2FjY2VwdHF1ZXVlLCB3aGljaCBpcyBz
ZXQgdG8gMTI4IG9uDQo+IG15IG1hY2hpbmUuDQo+DQo+IEhvd2V2ZXIsIHRoZSBzb2Z0d2Fy
ZSBoZSdzIHVzaW5nIG1heSBoYXZlIHNldCBiYWNrbG9nIHRvIDEuICBIYXJkIHRvIHNheQ0K
PiBiYXNlZCBvbiB0aGUgdHJhY2UgaGUgcHJvdmlkZWQuDQoNClRoYW5rcywgcGVvcGxlLg0K
DQpBIGZldyBob3VycyBhZ28gSSB0b29rIGEgaGludCBmcm9tIG9uZSBvZiB0aGUgcGFnZXMg
bGlua2VkIGZyb20gDQo8aHR0cHM6Ly9kYW4ubGFuZ2lsbGUub3JnLzIwMjAvMDEvMDEvbGlz
dGVuLXF1ZXVlLW92ZXJmbG93Lz4sIGFkZGVkIGEgDQpsaW5lIHRvIG15IHN5c2N0bC5jb25m
KDUpLg0KDQoNCiUgc3lzY3RsIGtlcm4uaXBjLnNvYWNjZXB0cXVldWUNCmtlcm4uaXBjLnNv
YWNjZXB0cXVldWU6IDI1Ng0KJSBzeXNjdGwga2Vybi5pcGMubWF4Y29ubg0Kc3lzY3RsOiB1
bmtub3duIG9pZCAna2Vybi5pcGMubWF4Y29ubicNCiUgc3lzY3RsIGtlcm4uaXBjLnNvbWF4
Y29ubg0Ka2Vybi5pcGMuc29tYXhjb25uOiAyNTYNCiUgZ3JlcCBpcGMgL2V0Yy9zeXNjdGwu
Y29uZg0Ka2Vybi5pcGMuc29hY2NlcHRxdWV1ZT0yNTYNCiUNCg0KDQpJIGJlZ2FuIHdvbmRl
cmluZyBhYm91dCB0aGUgc29uZXdjb25uIGxpbmVzIGluIHJlbGF0aW9uIHRvIA0KPGh0dHBz
Oi8vYnVncy5mcmVlYnNkLm9yZy9idWd6aWxsYS9zaG93X2J1Zy5jZ2k/aWQ9MjcwOTk3PiBm
b3IgDQpuZXQtbWdtdC9uZXRkYXRhLCBhbHRob3VnaCBJIGRvbid0IGtub3cgd2hldGhlciB0
aGVyZSdzIGEgZGlyZWN0IA0KcmVsYXRpb25zaGlwLg0KDQolIG5ldHN0YXQgLUxhbiB8IGdy
ZXAgbmV0ZGF0YQ0KdW5peMKgIDAvMC8xMjjCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg
wqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAvdG1wL25ldGRhdGEtaXBjDQolDQoNCk15IHVzZSBv
ZiBOZXRkYXRhIGlzIGxvY2FsLW9ubHkuDQoNCg==
--------------MdmP9XZQXIv5dXomZoKU8CqM
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<html data-lt-installed=3D"true">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DUTF=
-8">
</head>
<body style=3D"padding-bottom: 1px;">
<div class=3D"moz-cite-prefix">On 20/06/2023 15:41, Gary Jennejohn
wrote:<br>
</div>
<blockquote type=3D"cite"
cite=3D"mid:20230620164141.209ea8bc@ernst.home">
<pre class=3D"moz-quote-pre" wrap=3D"">On Tue, 20 Jun 2023 12:04:13=
+0200
Alexander Leidinger <a class=3D"moz-txt-link-rfc2396E" href=3D"mailto:Ale=
xander@leidinger.net"><Alexander@leidinger.net></a> wrote:
</pre>
<blockquote type=3D"cite">
<pre class=3D"moz-quote-pre" wrap=3D"">Quoting Gary Jennejohn <a =
class=3D"moz-txt-link-rfc2396E" href=3D"mailto:garyj@gmx.de"><garyj@gm=
x.de></a> (from Tue, 20 Jun 2023 07:41:08 +0000):
</pre>
<blockquote type=3D"cite">=E2=80=A6<br>
</blockquote>
<pre class=3D"moz-quote-pre" wrap=3D"">
In other words the software listening on it didn't process the request
fast enough and a backlog piled up (e.g apache ListenBacklog or nginx
"listen X backlog=3Dy" and "sysctl kern.ipx.somaxconn=3DX" for FreeBSD
itself). You may need faster hardware, more processes/threads to
handle the traffic, or configure your software to do less to produce
the same result (e.g. no real-time DNS resolution in the logging of a
webserver or increasing the amount of allowed items in the backlog).
If you can change the software, there's also the possibility to switch
from blocking sockets to non-blocking sockets (to not have the
select/accept loop block / run into contention) or kqueue.
</pre>
</blockquote>
<pre class=3D"moz-quote-pre" wrap=3D"">
On my FreeBSD14 system these things are all under kern.ipc.
maxconn seems to be an alias for soacceptqueue, which is set to 128 on
my machine.
However, the software he's using may have set backlog to 1. Hard to say
based on the trace he provided.</pre>
</blockquote>
<br>
<p>Thanks, people. <br>
</p>
<p>A few hours ago I took a hint from one of the pages linked from
<a class=3D"moz-txt-link-rfc2396E" href=3D"https://dan.langille.org=
/2020/01/01/listen-queue-overflow/"><https://dan.langille.org/2020/01/=
01/listen-queue-overflow/></a>,
added a line to my sysctl.conf(5). <br>
</p>
<p><br>
</p>
<p>% sysctl kern.ipc.soacceptqueue<br>
kern.ipc.soacceptqueue: 256<br>
% sysctl kern.ipc.maxconn<br>
sysctl: unknown oid 'kern.ipc.maxconn'<br>
% sysctl kern.ipc.somaxconn<br>
kern.ipc.somaxconn: 256<br>
% grep ipc /etc/sysctl.conf<br>
kern.ipc.soacceptqueue=3D256<br>
% <br>
</p>
<p><br>
</p>
<p>I began wondering about the sonewconn lines in relation to
<a class=3D"moz-txt-link-rfc2396E" href=3D"https://bugs.freebsd.org=
/bugzilla/show_bug.cgi?id=3D270997"><https://bugs.freebsd.org/bugzilla=
/show_bug.cgi?id=3D270997></a>
for <span id=3D"summary_container"><span
id=3D"short_desc_nonedit_display">net-mgmt/netdata, although I
don't know whether there's a direct relationship. <br>
</span></span></p>
<p><span id=3D"summary_container"><span
id=3D"short_desc_nonedit_display">% netstat -Lan | grep netdata=
<br>
unix=C2=A0 0/0/128=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /tmp/netdata-ipc<br>
% <br>
</span></span></p>
<p><span id=3D"summary_container"><span
id=3D"short_desc_nonedit_display">My use of Netdata is
local-only. <br>
</span></span></p>
</body>
<lt-container></lt-container>
</html>
--------------MdmP9XZQXIv5dXomZoKU8CqM--
--------------ukgM1NkREF79h3ml1nw0XaRg--
--------------w6eFlrm2O743bbKV0VCXZJuk
Content-Type: application/pgp-signature; name="OpenPGP_signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="OpenPGP_signature"
-----BEGIN PGP SIGNATURE-----
wsF5BAABCAAjFiEEWT/lssMHB+28ly8Kt2dIb0oY1AsFAmSSRHwFAwAAAAAACgkQt2dIb0oY1At7
Pw//bWvCWVpcUGzF3vnrgfDN4YYGStCV8rhqMIMfT7ZWDhl/GFB+DnrbACNJI6vO3PZzSwzqZacP
NKKoqBNumDi6F4AOIezIENf2rUtTzjdt593gdjKOFaCTKEGHmGesL6VlgYMufuOxl0MDHCQnMZjM
kBmTekfpBJtnPRLoRBVLvyCSalXCU/zTMQsc0rk5T3VfkfWFLUn9J1peo5Tn7M8lIhqskltOstvC
zSmEIIS5bafs8VX9LzYEbdmYU9DEd5rdIwEPWxvX0EVyP1WeGMuBpe786YtXCRIqJaugBoJdWyqX
TV1xecFFGWVX2jyUdl3/9KF0QN6kBO7oVDgpxaqndofcznyDm7qOIeAgZrnJKugsK0wyW1b8vpa2
h2KfKtDQocmbg50mXufQ+NRNyNc5BnxQpMsmn3ONedpdsQZ1SDprkbSHUR4XFg1KgxOymXozMSCv
VGv3XDKBl0v+9yr8V5hqWV0DoFxspk5xbE3Ol47sFB0KR33OsshtAdsG6dENKKLgSIwZiU1/dVv2
9J5u2nEivayBgoFNzn+yb84OzqoswtshBBZvKE2PdNSJDj2XfwqNE0wD3hesqyM40BrRjn91DJCj
6OUDWwKJozo3WjMns0ys1wp3mFE9OG13GHY2VuJs+wFhtDeIz3KRyvbsJaLxGTgyUMYhfKueKvr4
wOk=
=DxYW
-----END PGP SIGNATURE-----
--------------w6eFlrm2O743bbKV0VCXZJuk--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?b815622c-7a64-bb7b-74f9-72087c8ac461>