From nobody Fri Jun 27 05:53:02 2025 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4bT4Tw65vbz60KSy for ; Fri, 27 Jun 2025 05:53:08 +0000 (UTC) (envelope-from ben@benhutton.com.au) Received: from mail.myuniquemail.com (mail.myuniquemail.com [115.70.107.139]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4bT4Tv6HRtz3KP3 for ; Fri, 27 Jun 2025 05:53:07 +0000 (UTC) (envelope-from ben@benhutton.com.au) Authentication-Results: mx1.freebsd.org; none Received: from [10.128.2.107] (unknown [10.128.10.1]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by mail.myuniquemail.com (Postfix) with ESMTPSA id 10E2B1F9B34; Fri, 27 Jun 2025 13:53:03 +0800 (AWST) Content-Type: multipart/alternative; boundary="------------l50OztuS80EnHgmo2Y705YJE" Message-ID: <9fe64741-9f42-4e7e-9671-345221676136@benhutton.com.au> Date: Fri, 27 Jun 2025 13:53:02 +0800 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Network Tuning - mbuf To: Michael Tuexen Cc: freebsd-net@freebsd.org References: <8255b0b9-c9df-4af9-bbb2-94140edf189c@benhutton.com.au> <1B2AEE29-C71B-4EF7-9DDC-F45A13B0DC5F@lurchi.franken.de> Content-Language: en-US From: Ben Hutton In-Reply-To: <1B2AEE29-C71B-4EF7-9DDC-F45A13B0DC5F@lurchi.franken.de> X-Rspamd-Queue-Id: 4bT4Tv6HRtz3KP3 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:10143, ipnet:115.70.104.0/21, country:AU] This is a multi-part message in MIME format. --------------l50OztuS80EnHgmo2Y705YJE Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hi Michael, 8G $ netstat -m 1590074/4231/1594305 mbufs in use (current/cache/total) 797974/2592/800566/1800796 mbuf clusters in use (current/cache/total/max) 797974/790 mbuf+clusters out of packet secondary zone in use (current/cache) 644657/1542/646199/1550398 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/74192 9k jumbo clusters in use (current/cache/total/max) 0/0/0/41733 16k jumbo clusters in use (current/cache/total/max) 4572094K/12409K/4584504K bytes allocated to network (current/cache/total) 0/8507/8489 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/30432/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 485354407/0/0 requests for jumbo clusters denied (4k/9k/16k) 2 sendfile syscalls 2 sendfile syscalls completed without I/O request 0 requests for I/O initiated by sendfile 0 pages read by sendfile as part of a request 2 pages were valid at time of a sendfile request 0 pages were valid and substituted to bogus page 0 pages were requested for read ahead by applications 0 pages were read ahead by sendfile 0 times sendfile encountered an already busy page 0 requests for sfbufs denied 0 requests for sfbufs delayed Kind regards Ben On 27/06/2025 12:52, Michael Tuexen wrote: >> On 27. Jun 2025, at 04:17, Ben Hutton wrote: >> >> Hi, >> I'm currently having an issue with a spring-boot application (with nginx in front on the same instance) running on FreeBSD 14.1 in AWS. Two of our instances at present have had the application go offline with the following appearing in the /var/log/messages: >> Jun 26 07:57:47 freebsd kernel: [zone: mbuf_jumbo_page] kern.ipc.nmbjumbop limit reached >> Jun 26 07:57:47 freebsd kernel: [zone: mbuf_cluster] kern.ipc.nmbclusters limit reached >> Jun 26 07:59:34 freebsd kernel: sonewconn: pcb 0xfffff8021bd74000 (0.0.0.0:443 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (104 occurrences), euid 0, rgid 0, jail 0 >> Jun 26 08:01:51 freebsd kernel: sonewconn: pcb 0xfffff8021bd74000 (0.0.0.0:443 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (13 occurrences), euid 0, rgid 0, jail 0 >> >> Each time this has occurred I have increased the nmbjumbop and nmbclusters values. The last time by a huge amount to see if we can mitigate the issue. Once I adjust the values the application starts responding to requests again. >> My question is, is just increasing this the correct course of action or should I be investigating something else, or adjusting other settings accordingly? Also if this is due to an underlying issue and not just network load how would I get to the root cause? Note the application streams allot of files in rapid succession which I'm suspecting is what is causing the issue. > Hi Ben, > > how much memory does your VM have? What is the output of > netstat -m > when the system is in operation? > > Best regards > Michael >> Thanks >> Ben >> --------------l50OztuS80EnHgmo2Y705YJE Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit

Hi Michael,

8G

$ netstat -m
1590074/4231/1594305 mbufs in use (current/cache/total)
797974/2592/800566/1800796 mbuf clusters in use (current/cache/total/max)
797974/790 mbuf+clusters out of packet secondary zone in use (current/cache)
644657/1542/646199/1550398 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/74192 9k jumbo clusters in use (current/cache/total/max)
0/0/0/41733 16k jumbo clusters in use (current/cache/total/max)
4572094K/12409K/4584504K bytes allocated to network (current/cache/total)
0/8507/8489 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/30432/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
485354407/0/0 requests for jumbo clusters denied (4k/9k/16k)
2 sendfile syscalls
2 sendfile syscalls completed without I/O request
0 requests for I/O initiated by sendfile
0 pages read by sendfile as part of a request
2 pages were valid at time of a sendfile request
0 pages were valid and substituted to bogus page
0 pages were requested for read ahead by applications
0 pages were read ahead by sendfile
0 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed

Kind regards
Ben

On 27/06/2025 12:52, Michael Tuexen wrote:
On 27. Jun 2025, at 04:17, Ben Hutton <ben@benhutton.com.au> wrote:

Hi,
I'm currently having an issue with a spring-boot application (with nginx  in front on the same instance) running on FreeBSD 14.1 in AWS. Two of our instances at present have had the application go offline with the following appearing in the /var/log/messages:
Jun 26 07:57:47 freebsd kernel: [zone: mbuf_jumbo_page] kern.ipc.nmbjumbop limit reached 
Jun 26 07:57:47 freebsd kernel: [zone: mbuf_cluster] kern.ipc.nmbclusters limit reached 
Jun 26 07:59:34 freebsd kernel: sonewconn: pcb 0xfffff8021bd74000 (0.0.0.0:443 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (104 occurrences), euid 0, rgid 0, jail 0 
Jun 26 08:01:51 freebsd kernel: sonewconn: pcb 0xfffff8021bd74000 (0.0.0.0:443 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (13 occurrences), euid 0, rgid 0, jail 0

Each time this has occurred I have increased the nmbjumbop and nmbclusters values. The last time by a huge amount to see if we can mitigate the issue. Once I adjust the values the application starts responding to requests again.
My question is, is just increasing this the correct course of action or should I be investigating something else, or adjusting other settings accordingly? Also if this is due to an underlying issue and not just network load how would I get to the root cause? Note the application streams allot of files in rapid succession which I'm suspecting is what is causing the issue.

Hi Ben,

how much memory does your VM have? What is the output of
netstat -m
when the system is in operation?

Best regards
Michael

Thanks
Ben



    

  


--------------l50OztuS80EnHgmo2Y705YJE--