From nobody Wed Jun 4 17:29:40 2025 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4bCF2S1grfz5xM3W for ; Wed, 04 Jun 2025 17:29:52 +0000 (UTC) (envelope-from chris@cretaforce.gr) Received: from smtp7.cretaforce.gr (smtp7.cretaforce.gr [159.69.101.226]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.cretaforce.gr", Issuer "RapidSSL TLS RSA CA G1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4bCF2Q4N3gz3J4H for ; Wed, 04 Jun 2025 17:29:50 +0000 (UTC) (envelope-from chris@cretaforce.gr) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cretaforce.gr header.s=cretaforce header.b=SmJIZ6rg; spf=pass (mx1.freebsd.org: domain of chris@cretaforce.gr designates 159.69.101.226 as permitted sender) smtp.mailfrom=chris@cretaforce.gr; dmarc=pass (policy=none) header.from=cretaforce.gr Received: from server1.cretaforce.gr (server1.cretaforce.gr [94.130.217.104]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) client-signature RSA-PSS (2048 bits)) (Client CN "*.cretaforce.gr", Issuer "RapidSSL TLS RSA CA G1" (verified OK)) by smtp.cretaforce.gr (Postfix) with ESMTPS id 30A20208B2 for ; Wed, 4 Jun 2025 20:29:50 +0300 (EEST) Received: from smtpclient.apple (unknown [149.210.4.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: chris@cretaforce.gr) by server1.cretaforce.gr (Postfix) with ESMTPSA id 252FF11917; Wed, 04 Jun 2025 20:29:48 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cretaforce.gr; s=cretaforce; t=1749058179; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=V0RLhkHFfFK7j65shVnirBULB+Js7Nn1Upne4EfSf3c=; b=SmJIZ6rgM6L7AKt+Qfs1lop1fssp8VKyEAlNxO5Y6AUAzctdP8AJOb4uFfHHYxVklyV+tW Y9CISWjiG4DGMBo5h1BQfdX3tH1T07kzU7rPsY+X5layhbqVW3D42JpQXnNEOKmKE5it6y 4KYZNrd9IoZx6cq11zpk7lcbgQzZ510= From: Christos Chatzaras Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_AFC174C6-8041-41D5-8BB6-45B909FA2BD1" List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.600.51.1.1\)) Subject: Re: Problem with net.inet.tcp.path_mtu_discovery=1 Date: Wed, 4 Jun 2025 20:29:40 +0300 In-Reply-To: Cc: freebsd-net To: questions@freebsd.org References: <9728060D-2C02-426B-BACE-F2D2F651A62F@cretaforce.gr> X-Mailer: Apple Mail (2.3826.600.51.1.1) X-Spamd-Result: default: False [-1.38 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-0.998]; NEURAL_SPAM_LONG(0.86)[0.861]; NEURAL_HAM_MEDIUM(-0.74)[-0.737]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[cretaforce.gr,none]; R_SPF_ALLOW(-0.20)[+ip4:159.69.101.226]; R_DKIM_ALLOW(-0.20)[cretaforce.gr:s=cretaforce]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_ALL(0.00)[]; DKIM_TRACE(0.00)[cretaforce.gr:+]; TO_DN_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:24940, ipnet:159.69.0.0/16, country:DE]; DWL_DNSWL_NONE(0.00)[cretaforce.gr:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEFALL_USER(0.00)[chris]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[159.69.101.226:from] X-Rspamd-Queue-Id: 4bCF2Q4N3gz3J4H X-Spamd-Bar: - --Apple-Mail=_AFC174C6-8041-41D5-8BB6-45B909FA2BD1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 4 Jun 2025, at 19:36, Dave Cottlehuber wrote: >=20 > On Wed, 4 Jun 2025, at 16:36, Christos Chatzaras wrote: >> Hello, >>=20 >> I manage some servers hosting websites. >=20 > What does tcpdump/wireshark show for traffic, particularly icmp? = Wireshark is very helpful in explaining some issues. >=20 > What is the actual MTU on the working net vs the failing one? >=20 > Is there a local MTU where the failing websites start working again? >=20 > see ping(8) and use -v -D -s =E2=80=A6. together to find a working MTU = and cross check with tcpdump to find where things seem to break. >=20 > On a recent cloud environment I needed to add =E2=80=98 set reassemble = yes no-df=E2=80=99 to my pf.conf to address MTU issues between VNET = jails and the internet. >=20 > Happy hunting > Dave >=20 First, I reverted the server settings to their defaults: sysctl net.inet.tcp.path_mtu_discovery=3D1 sysctl net.inet.tcp.pmtud_blackhole_detection=3D0 Next, I set the MTU on my local computer to 1460 and everything worked = as expected: tcpdump: listening on en0, link-type EN10MB (Ethernet), snapshot length = 524288 bytes 20:15:05.651375 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto = TCP (6), length 64) 192.168.2.18.65322 > 94.130.217.87.443: Flags [S], cksum 0x293e = (correct), seq 3503095669, win 65535, options [mss 1420,nop,wscale = 6,nop,nop,TS val 639376397 ecr 0,sackOK,eol], length 0 20:15:05.705913 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto = TCP (6), length 60) 94.130.217.87.443 > 192.168.2.18.65322: Flags [S.], cksum 0x9c22 = (correct), seq 3647364942, ack 3503095670, win 65535, options [mss = 1460,nop,wscale 6,sackOK,TS val 1782053626 ecr 639376397], length 0 However, when I set my local computer=E2=80=99s MTU back to 1500 (the = default), the issue reappeared: tcpdump: listening on en0, link-type EN10MB (Ethernet), snapshot length = 524288 bytes 20:17:45.662993 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto = TCP (6), length 64) 192.168.2.18.65333 > 94.130.217.87.443: Flags [S], cksum 0x4a07 = (correct), seq 3674289142, win 65535, options [mss 1460,nop,wscale = 6,nop,nop,TS val 681359835 ecr 0,sackOK,eol], length 0 20:17:45.726988 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto = TCP (6), length 60) 94.130.217.87.443 > 192.168.2.18.65333: Flags [S.], cksum 0x9b1d = (correct), seq 1443843488, ack 3674289143, win 65535, options [mss = 1460,nop,wscale 6,sackOK,TS val 2890559459 ecr 681359835], length 0 So, with local computer MTU 1460, everything works, but with MTU 1500, = the problem persists.= --Apple-Mail=_AFC174C6-8041-41D5-8BB6-45B909FA2BD1 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8


On 4 Jun 2025, at 19:36, Dave Cottlehuber = <dch@skunkwerks.at> wrote:

On Wed, 4 Jun 2025, at = 16:36, Christos Chatzaras wrote:
Hello,

I manage some servers hosting = websites.

What does tcpdump/wireshark show for = traffic, particularly icmp? Wireshark is very helpful in explaining some = issues.

What is the actual MTU on the working net vs the failing = one?

Is there a local MTU where the failing websites start = working again?

see ping(8) and use -v -D -s =E2=80=A6. together = to find a working MTU and cross check with tcpdump to find where things = seem to break.

On a recent cloud environment I needed to add =E2=80= =98 set reassemble yes no-df=E2=80=99 to my pf.conf to address MTU = issues between VNET jails and the internet.

Happy = hunting
Dave


=

First, I reverted the server settings to their = defaults:

sysctl = net.inet.tcp.path_mtu_discovery=3D1
sysctl = net.inet.tcp.pmtud_blackhole_detection=3D0

Next, I set the MTU on my local computer to 1460 and = everything worked as expected:

tcpdump: listening on = en0, link-type EN10MB (Ethernet), snapshot length 524288 = bytes
20:15:05.651375 IP (tos 0x0, ttl 64, id 0, offset 0, = flags [DF], proto TCP (6), length 64)
    = 192.168.2.18.65322 > 94.130.217.87.443: Flags [S], cksum 0x293e = (correct), seq 3503095669, win 65535, options [mss 1420,nop,wscale = 6,nop,nop,TS val 639376397 ecr 0,sackOK,eol], length = 0
20:15:05.705913 IP (tos 0x0, ttl 54, id 0, offset 0, flags = [DF], proto TCP (6), length 60)
    = 94.130.217.87.443 > 192.168.2.18.65322: Flags [S.], cksum 0x9c22 = (correct), seq 3647364942, ack 3503095670, win 65535, options [mss = 1460,nop,wscale 6,sackOK,TS val 1782053626 ecr 639376397], length = 0

However, when I = set my local computer=E2=80=99s MTU back to 1500 (the default), the = issue reappeared:

tcpdump: listening on en0, = link-type EN10MB (Ethernet), snapshot length 524288 = bytes
20:17:45.662993 IP (tos 0x0, ttl 64, id 0, offset 0, = flags [DF], proto TCP (6), length 64)
    = 192.168.2.18.65333 > 94.130.217.87.443: Flags [S], cksum 0x4a07 = (correct), seq 3674289142, win 65535, options [mss 1460,nop,wscale = 6,nop,nop,TS val 681359835 ecr 0,sackOK,eol], length = 0
20:17:45.726988 IP (tos 0x0, ttl 54, id 0, offset 0, flags = [DF], proto TCP (6), length 60)
    = 94.130.217.87.443 > 192.168.2.18.65333: Flags [S.], cksum 0x9b1d = (correct), seq 1443843488, ack 3674289143, win 65535, options [mss = 1460,nop,wscale 6,sackOK,TS val 2890559459 ecr 681359835], length = 0

So, with local computer = MTU 1460, everything works, but with MTU 1500, the problem = persists.

= --Apple-Mail=_AFC174C6-8041-41D5-8BB6-45B909FA2BD1--