Date: Wed, 4 Jun 2025 20:29:40 +0300 From: Christos Chatzaras <chris@cretaforce.gr> To: questions@freebsd.org Cc: freebsd-net <freebsd-net@freebsd.org> Subject: Re: Problem with net.inet.tcp.path_mtu_discovery=1 Message-ID: <D9DE01A2-96DF-4804-875C-2424BEF733F3@cretaforce.gr> In-Reply-To: <bf557c42-625f-4b8a-b5df-7a45c84e40ee@app.fastmail.com> References: <9728060D-2C02-426B-BACE-F2D2F651A62F@cretaforce.gr> <bf557c42-625f-4b8a-b5df-7a45c84e40ee@app.fastmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
> On 4 Jun 2025, at 19:36, Dave Cottlehuber <dch@skunkwerks.at> wrote:
>
> On Wed, 4 Jun 2025, at 16:36, Christos Chatzaras wrote:
>> Hello,
>>
>> I manage some servers hosting websites.
>
> What does tcpdump/wireshark show for traffic, particularly icmp? Wireshark is very helpful in explaining some issues.
>
> What is the actual MTU on the working net vs the failing one?
>
> Is there a local MTU where the failing websites start working again?
>
> see ping(8) and use -v -D -s …. together to find a working MTU and cross check with tcpdump to find where things seem to break.
>
> On a recent cloud environment I needed to add ‘ set reassemble yes no-df’ to my pf.conf to address MTU issues between VNET jails and the internet.
>
> Happy hunting
> Dave
>
First, I reverted the server settings to their defaults:
sysctl net.inet.tcp.path_mtu_discovery=1
sysctl net.inet.tcp.pmtud_blackhole_detection=0
Next, I set the MTU on my local computer to 1460 and everything worked as expected:
tcpdump: listening on en0, link-type EN10MB (Ethernet), snapshot length 524288 bytes
20:15:05.651375 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
192.168.2.18.65322 > 94.130.217.87.443: Flags [S], cksum 0x293e (correct), seq 3503095669, win 65535, options [mss 1420,nop,wscale 6,nop,nop,TS val 639376397 ecr 0,sackOK,eol], length 0
20:15:05.705913 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto TCP (6), length 60)
94.130.217.87.443 > 192.168.2.18.65322: Flags [S.], cksum 0x9c22 (correct), seq 3647364942, ack 3503095670, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1782053626 ecr 639376397], length 0
However, when I set my local computer’s MTU back to 1500 (the default), the issue reappeared:
tcpdump: listening on en0, link-type EN10MB (Ethernet), snapshot length 524288 bytes
20:17:45.662993 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
192.168.2.18.65333 > 94.130.217.87.443: Flags [S], cksum 0x4a07 (correct), seq 3674289142, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 681359835 ecr 0,sackOK,eol], length 0
20:17:45.726988 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto TCP (6), length 60)
94.130.217.87.443 > 192.168.2.18.65333: Flags [S.], cksum 0x9b1d (correct), seq 1443843488, ack 3674289143, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2890559459 ecr 681359835], length 0
So, with local computer MTU 1460, everything works, but with MTU 1500, the problem persists.
[-- Attachment #2 --]
<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><meta http-equiv="content-type" content="text/html; charset=utf-8"><div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On 4 Jun 2025, at 19:36, Dave Cottlehuber <dch@skunkwerks.at> wrote:</div><br class="Apple-interchange-newline"><div><div>On Wed, 4 Jun 2025, at 16:36, Christos Chatzaras wrote:<br><blockquote type="cite">Hello,<br><br>I manage some servers hosting websites.<br></blockquote><br>What does tcpdump/wireshark show for traffic, particularly icmp? Wireshark is very helpful in explaining some issues.<br><br>What is the actual MTU on the working net vs the failing one?<br><br>Is there a local MTU where the failing websites start working again?<br><br>see ping(8) and use -v -D -s …. together to find a working MTU and cross check with tcpdump to find where things seem to break.<br><br>On a recent cloud environment I needed to add ‘ set reassemble yes no-df’ to my pf.conf to address MTU issues between VNET jails and the internet.<br><br>Happy hunting<br>Dave<br><br></div></div></blockquote></div><div><br></div><div><p class="p1">First, I reverted the server settings to their defaults:</p></div><div><div style="caret-color: rgba(0, 0, 0, 0.847); color: rgba(0, 0, 0, 0.847); font-family: -apple-system-font;">sysctl net.inet.tcp.path_mtu_discovery=1<br></div><span style="caret-color: rgba(0, 0, 0, 0.847); color: rgba(0, 0, 0, 0.847); font-family: -apple-system-font;">sysctl net.inet.tcp.pmtud_blackhole_detection=0</span></div><div><span style="caret-color: rgba(0, 0, 0, 0.847); color: rgba(0, 0, 0, 0.847); font-family: -apple-system-font;"><br></span></div><div><div><p class="p1">Next, I set the MTU on my local computer to 1460 and everything worked as expected:</p></div><div><div>tcpdump: listening on en0, link-type EN10MB (Ethernet), snapshot length 524288 bytes</div><div>20:15:05.651375 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)</div><div> 192.168.2.18.65322 > 94.130.217.87.443: Flags [S], cksum 0x293e (correct), seq 3503095669, win 65535, options [mss 1420,nop,wscale 6,nop,nop,TS val 639376397 ecr 0,sackOK,eol], length 0</div><div>20:15:05.705913 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto TCP (6), length 60)</div><div> 94.130.217.87.443 > 192.168.2.18.65322: Flags [S.], cksum 0x9c22 (correct), seq 3647364942, ack 3503095670, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1782053626 ecr 639376397], length 0</div></div></div><div><br></div><div><p class="p1">However, when I set my local computer’s MTU back to 1500 (the default), the issue reappeared:</p></div><div><div>tcpdump: listening on en0, link-type EN10MB (Ethernet), snapshot length 524288 bytes</div><div>20:17:45.662993 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)</div><div> 192.168.2.18.65333 > 94.130.217.87.443: Flags [S], cksum 0x4a07 (correct), seq 3674289142, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 681359835 ecr 0,sackOK,eol], length 0</div><div>20:17:45.726988 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto TCP (6), length 60)</div><div> 94.130.217.87.443 > 192.168.2.18.65333: Flags [S.], cksum 0x9b1d (correct), seq 1443843488, ack 3674289143, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2890559459 ecr 681359835], length 0</div></div><div><br></div><div><p class="p1">So, with local computer MTU 1460, everything works, but with MTU 1500, the problem persists.</p></div></div></body></html>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?D9DE01A2-96DF-4804-875C-2424BEF733F3>