From nobody Tue Jul 16 20:04:27 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WNqm071fWz5RsC9 for ; Tue, 16 Jul 2024 20:04:32 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-yb1-xb29.google.com (mail-yb1-xb29.google.com [IPv6:2607:f8b0:4864:20::b29]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WNqm02FMCz46gG for ; Tue, 16 Jul 2024 20:04:32 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b="K2/p3Wjh"; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::b29 as permitted sender) smtp.mailfrom=markjdb@gmail.com Received: by mail-yb1-xb29.google.com with SMTP id 3f1490d57ef6-e03a8955ae3so5811829276.1 for ; Tue, 16 Jul 2024 13:04:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721160271; x=1721765071; darn=freebsd.org; h=content-disposition:mime-version:message-id:subject:to:from:date :sender:from:to:cc:subject:date:message-id:reply-to; bh=JUl4l1+TU33DlWcIIRhD5Y+SfrSWKqDE3scviGFgA3o=; b=K2/p3WjhDnOg/xsqTLwIv4c1pBziEcr9bwF+EUI9Tmzk4KEAN/8qSHE3/bcjcQikAR F0IcgqOtcOVPsVHTAcB03H27NXetoGm1WVme1xY9MRFK7S6tg4fUtxDQnWT6nk2S7aRY wuNg8lpIatg8UvpUuo3kM7eZPucc1Mpr5usGdQrMTw0tjIMpuoJk2ZT7/ERapvwuRKUN OoILyFB3s/oa53bAJ6ZgbPjVgDtmogTvXaww1QuuzvR8MbNTICzMhaOWMGGKinZqU3kh /LAxCQYL2mUtunv8OE9bs3pAqb6Mbw+y+8yCYTCPCU/M2JaSO70OQxMnBojIz1TWoeCw xmRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721160271; x=1721765071; h=content-disposition:mime-version:message-id:subject:to:from:date :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=JUl4l1+TU33DlWcIIRhD5Y+SfrSWKqDE3scviGFgA3o=; b=lN+6sgg+b1u6431ZQvi5YY1iHfKqDIXGHJfMrcbzRWDgYf2c2x5lOozQVAu9Amkr+N 4KoSb8KngdY6oMdqpCmlH+9LCCp7rioHdQsxQQmdL/MCf3JSR8YRGVp+GKGfbIlILkE+ h+8chqIvL3qms0N/gtwShNDK8RRnZXZ/6SLSB9cYMCndOnRezoH4wDR2KhJMSvP5kZHS fpIPca9xt1kk3s7ldl0u1nYbiLdveOq9gvhQF5dBSuprIHDHKqxS8BOXVJjFMYOvE9cn Tn73Dh5ukFRIsdDNfnXjPMUkyHt8Pb/2ymcM7JctcrBmTB8inNmzqtJjwAFTT0l2T6Sk knhw== X-Gm-Message-State: AOJu0YyQp5tcknPtlqHQTl4xX1XZKwqE70cvRQs8ihdlzUeTPqNcGclA sakhSVEH9pEft9xd0aILYJXXOuzR0k0AUbsHlclAnVAnD7On2PkQiE5Tz30i X-Google-Smtp-Source: AGHT+IHPGByrlpGpoXfg4zML8kjGPQBGeysc/v72DcqddTVDLCl2hRRY4/ecgpxxeUPgVFXBtWFcmw== X-Received: by 2002:a05:6902:2b01:b0:dfe:5a00:df5c with SMTP id 3f1490d57ef6-e05d56bcd0amr4932413276.18.1721160270801; Tue, 16 Jul 2024 13:04:30 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b76199d5a2sm33858646d6.60.2024.07.16.13.04.30 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 13:04:30 -0700 (PDT) Date: Tue, 16 Jul 2024 16:04:27 -0400 From: Mark Johnston To: freebsd-net@freebsd.org Subject: flushing default router list upon inet6 route flush Message-ID: List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.60 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), DKIM not aligned (relaxed),none]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[gmail.com:+]; ARC_NA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; TO_DN_NONE(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DOM_EQ_FROM_DOM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b29:from] X-Rspamd-Queue-Id: 4WNqm02FMCz46gG Hello, When IPv6 SLAAC is configured for an interface, the kernel will update its default router list upon receipt of a router advertisement. In so doing it may install a default route; in the kernel this happens in defrouter_addreq(). If one uses "route flush" or "service routing restart" to reset the routing tables, the default router list is not purged, so a subsequent RA from the original default router does not update the list, and so does not re-create the default route, even if one re-runs rtsol(8). This appears to be a bug, but I'm not sure where best to fix it. Should "service routing restart" invoke "ndp -R" to flush the default router list? Should route(8) handle this as part of a flush command? Or something else? From nobody Wed Jul 17 01:19:53 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WNym45RNQz5QfYL for ; Wed, 17 Jul 2024 01:20:04 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WNym44hdPz4f0J; Wed, 17 Jul 2024 01:20:04 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721179204; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=70jfBbG1vIEgtquho8IAMH/iFHFbMA/OXCci8ZZa/PU=; b=GiF7fv1YMu+p+iC+EQHorp7Fx3BSTVpQQv8OxDncT8DD7dcJxn4oFhybwI1SCpqP4lm1Fq ALupByV+JXcDlX7P3N9CmzMVyoBJxxecWhFiSamFUhdNCS7LGXvPPxLTr0erWxRZ9/pOr4 CTrK6/Uhwxjat+5Z5e9PrTuHuokBeh085YPHHAIC/ovQkbisH/PPuqLE2gimoJZ+47n2na nMyUDO0VuO1spXlQafwMmIA9Lwi2bqMUBHIc/Nnhogr2xtZjwHbNi3hSLT7vzrq6JNrpa2 DBxBHtP6BTey7UvM/pSGMTpu/QaY37qy10hyeaqsqxOLo0l2DJoM84aXn6cWOw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721179204; a=rsa-sha256; cv=none; b=sEngw0h8dXDMwWW3ELtDpjXzzm0fmCTgNLlkeoJJrgaocO6onVWS9q0fs80X89ShmMomNr aJCcF0JTxNhRJ6G3PeX4lZNLj6OniwoTVNdWQRepv0JTH9HtiB/NrPV7p74DVorL2N0Up2 vIrHf/UFDi7W0/yYGj6CNQ3bFSR6/MrKNvV/qiQpVG1tQYAk7jXlD7xWJ4/yZS3A8ZZhWN GO75w/BfA3ADVmm9oMXn0o90CwvHFU+ukP3jT61UHB5LHTze1PL62ymAQgQb+kCBYL1ZsU NWOnN4i60ZldNwEJwxyPgZbLoRTKuHkPyMJ7M66Ubd/uxmO/4WJ7NT61yYBtLQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721179204; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=70jfBbG1vIEgtquho8IAMH/iFHFbMA/OXCci8ZZa/PU=; b=QVd4tHNqT/MSK/8KTFCUopmCbha5P0pysAYmc9HNQyeoIG1lngaHtAa7v5qdr+XpukeczG vLxB6r3nGCJdfSEabxZFHZCaN1P0SWOkS8X4j953H1//+IU40eeJhYIucBJGSDeUL/Blxf ihg8jEOj83Kbhkp0qpmk27CB4NyVMsH2hMLUudOgws/STUWMks3YkbGroy2q5HjoHm6SXV H+9PoIu3klsO2MM6LLDd39AUnMTJvVTYy2UqubavvYI+f3ZIaINybvWidcTJCAklWyM9qu iaf966WG3kxiqU5n8b+s1WcKuqRH5kL22k5OzqxCMaCc/OChHxqXhcMi3lNSNw== Received: from smtpclient.apple (unknown [IPv6:2001:19f0:6001:9db:98f0:9fe0:3545:10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4WNym33xlYz144F; Wed, 17 Jul 2024 01:20:03 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Content-Type: text/plain; charset=us-ascii List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.8\)) Subject: Re: flushing default router list upon inet6 route flush From: Zhenlei Huang In-Reply-To: Date: Wed, 17 Jul 2024 09:19:53 +0800 Cc: freebsd-net@freebsd.org Content-Transfer-Encoding: 7bit Message-Id: References: To: Mark Johnston X-Mailer: Apple Mail (2.3696.120.41.1.8) > On Jul 17, 2024, at 4:04 AM, Mark Johnston wrote: > > Hello, > > When IPv6 SLAAC is configured for an interface, the kernel will update > its default router list upon receipt of a router advertisement. In so > doing it may install a default route; in the kernel this happens in > defrouter_addreq(). > > If one uses "route flush" or "service routing restart" to reset the > routing tables, the default router list is not purged, so a subsequent > RA from the original default router does not update the list, and so > does not re-create the default route, even if one re-runs rtsol(8). > > This appears to be a bug, but I'm not sure where best to fix it. Should > "service routing restart" invoke "ndp -R" to flush the default router > list? That can be a workaround, but not the ideal fix. > Should route(8) handle this as part of a flush command? No, I do not think so. route(8) should handle the routing / FIB parts. IPv6 default route list is maintained as a per AF basis. Handling the default route list via route(8), aka the userland, seems to be more a HACK. > Or > something else? I'd propose that the kernel handle this situation, so that for other cases such as `route -6 delete default`, or route change event from NETLINK socket, the IPv6 SLAAC default router feature can also work as expected. To be precise, `sys/netinet6/nd6_rtr.c` listen on route events and clear the `installed` flag on deleting the previously installed default route, or maybe purge all default route list. Then the next time the kernel receives a RA it re-installs the default route. This IMO may have side effect that user may really want to delete the default route while not providing an explicit default route. I think for that case user should disable accepting RA on the interface ( aka ifconfig em0 inet6 no_radr, or turn on net.inet6.ip6.no_radr globally). How about this proposal ? Best regards, Zhenlei From nobody Wed Jul 17 06:36:35 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WP5nN6y7fz5Qff0 for ; Wed, 17 Jul 2024 06:36:40 +0000 (UTC) (envelope-from roy@marples.name) Received: from sender-of-o57.zoho.eu (sender-of-o57.zoho.eu [136.143.169.57]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WP5nN1r2wz45xw; Wed, 17 Jul 2024 06:36:39 +0000 (UTC) (envelope-from roy@marples.name) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; t=1721198197; cv=none; d=zohomail.eu; s=zohoarc; b=evW0D8utAlO6xJ7v96JjF3z/A8T3LqgFkhOywKtjYIiqRwK97d6o9tkTe/+nFphO7XZZb1HMUnt+QRDBUWvcTQ/MCNPHAFu2rXVStrXG7J57uiSy+8E/8JybdDMSUskG1d4mRt3+r6WaeuY4hIdCxzY+SCvEpFZass101I2fMNo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.eu; s=zohoarc; t=1721198197; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To; bh=M/V8HAnEP6uVTDILppkFJhqL27CZdTP4AzgNmc1MsrQ=; b=WMZS+eTmn2icP53btS6xrtW0Iw00RHpqNz3SFtWsv8S/QgWjRrcCkRf4ZMg3uA5iGpse3IhkOZoFZo419r7BYQfecw/s/WH7cl7eadzPgwLuK5OdrKzhCNF3KtxsVdm1klWUwOCyJwoUo3cU63QgJnD62+gHBZh41t9tPAe29ho= ARC-Authentication-Results: i=1; mx.zohomail.eu; dkim=pass header.i=marples.name; spf=pass smtp.mailfrom=roy@marples.name; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1721198197; s=zmail; d=marples.name; i=roy@marples.name; h=Date:Date:From:From:To:To:Cc:Cc:Message-ID:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id:Reply-To; bh=M/V8HAnEP6uVTDILppkFJhqL27CZdTP4AzgNmc1MsrQ=; b=ML0MaFR1jQ4WVberpgsLNTN4FBVksXrXvzsjF21P6ZP7kI4Fs50TDUtPi88u8SJL t0751nZdDe5adg+Rs1D+WMp+t8dQi5wPrG6HZBD8nYcybSF4jbiHnFONi7d4WlJZWaG vgGfajhl8AuYVNuzkXBLdxTQdfravJPmXhBMrAuw= Received: from mail.zoho.eu by mx.zoho.eu with SMTP id 1721198195159961.6384912931234; Wed, 17 Jul 2024 08:36:35 +0200 (CEST) Date: Wed, 17 Jul 2024 07:36:35 +0100 From: Roy Marples To: "Mark Johnston" Cc: "freebsd-net" Message-ID: <190bf6831ac.b8c8bcf41215199.2223628794184357959@marples.name> In-Reply-To: References: Subject: Re: flushing default router list upon inet6 route flush List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:41913, ipnet:136.143.168.0/23, country:CH] X-Rspamd-Queue-Id: 4WP5nN1r2wz45xw Hi Mark ---- On Tue, 16 Jul 2024 21:04:27 +0100 Mark Johnston wrote --- > Hello, > > When IPv6 SLAAC is configured for an interface, the kernel will update > its default router list upon receipt of a router advertisement. In so > doing it may install a default route; in the kernel this happens in > defrouter_addreq(). > > If one uses "route flush" or "service routing restart" to reset the > routing tables, the default router list is not purged, so a subsequent > RA from the original default router does not update the list, and so > does not re-create the default route, even if one re-runs rtsol(8). > > This appears to be a bug, but I'm not sure where best to fix it. Should > "service routing restart" invoke "ndp -R" to flush the default router > list? Should route(8) handle this as part of a flush command? Or > something else? rc.d/rtsold should probably be fixed, but it also really the wrong solutuion. Kernel handling of RA cannot be influenced very well from userland and as you see, this affects the default route. An alternative solution would be to use dhcpcd(8) from ports to manage RA instead of the kernel. It can also handle DHCP, replacing dhclient(8) but that's up to you. You might want to add `noipv4` to `/etc/dhcpcd.conf` until you're happy changing fully over. dhcpcd will intelligently manage all the routes. You can purge everything (including addresses and DNS) with `dhcpcd -k` and restore them with `dhcpcd -n`. Good luck! Roy From nobody Wed Jul 17 20:00:31 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPRd92v3Mz5QHXF for ; Wed, 17 Jul 2024 20:00:45 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-vk1-f180.google.com (mail-vk1-f180.google.com [209.85.221.180]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPRd850tkz4ctb; Wed, 17 Jul 2024 20:00:44 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.221.180 as permitted sender) smtp.mailfrom=asomers@gmail.com Received: by mail-vk1-f180.google.com with SMTP id 71dfb90a1353d-4f2f39829a9so30115e0c.2; Wed, 17 Jul 2024 13:00:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721246443; x=1721851243; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xum+T2H/5fdMy2+AdTrPljq44v7mYxkPNqFHTnJivC8=; b=TRZ40KrMBEsuhUIasJCnjy5YJ6Z1fTiB3UjploIcDMBNlovddYwtQMgkHHbdkyM2hw tzt1+xS8svzVMY4xyh8EV7e0fN4b7mjqBVrDQgTalCqs3L55M9PL7Da7eY4FqBUzFFMi 64tVYpYjR/X14xTfm/aREmq82BIb1C9W5TxagtyjcJF8Cuht47NjpwPENNX12D/OoT3a vEhAuruX3oJVv08kdlFlcmazBDIA8rhm3xQJRCDMf/BLci9if1/PTtYhd4Q9zbkKm3Lz g32+46McEs4bM4asrc+pav5mefSL/VKZEjysS9xD5dgugo3ygESPJwmLBHairE9I6hiC bNoA== X-Forwarded-Encrypted: i=1; AJvYcCUb6Z+fAeHz0l/xt9FJhPQZMNamc/qk2nqJdV5iAiyFO4VMy76w3YwH1WbZzlKc3ZqlnzgAeax7Iq1UDFI5uQLnTtrI6od/rA== X-Gm-Message-State: AOJu0YyK8aPwUISNR0GMdWgFIu5zEwe7cKhnFUjlv6oMsWKG5CndWvc5 Kgi72VNXtYyStqfsEFLl3UL8eMyH1TnvrGuOIUpQoDN+YevEkQRja48e6fMZy87NiFjvJH6iz12 0MOauBHnpGcYXuaqG9+enBP0LVaQR6i1i X-Google-Smtp-Source: AGHT+IHZCcX0HJoMtZvDu4xwBqAbIQcN+XslNWAS4lSTIPaF/82g2t9WofmR+cliRUjIikYV5TPrwsn5+RbspI71IyY= X-Received: by 2002:a05:6122:1347:b0:4df:1a3f:2ec1 with SMTP id 71dfb90a1353d-4f4df64d2ffmr4012588e0c.1.1721246443073; Wed, 17 Jul 2024 13:00:43 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Wed, 17 Jul 2024 14:00:31 -0600 Message-ID: Subject: TCP Success Story (was Re: TCP_RACK, TCP_BBR, and firewalls) To: Michael Tuexen Cc: Alan Somers , FreeBSD Net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.81 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.98)[-0.978]; NEURAL_HAM_SHORT(-0.93)[-0.934]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; ARC_NA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEFALL_USER(0.00)[asomers]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.221.180:from]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.221.180:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; RCPT_COUNT_THREE(0.00)[3] X-Rspamd-Queue-Id: 4WPRd850tkz4ctb On Sat, Jul 13, 2024 at 1:50=E2=80=AFAM wrote: > > > On 13. Jul 2024, at 01:43, Alan Somers wrote: > > > > I've been experimenting with RACK and BBR. In my environment, they > > can dramatically improve single-stream TCP performance, which is > > awesome. But pf interferes. I have to disable pf in order for them > > to work at all. > > > > Is this a known limitation? If not, I will experiment some more to > > determine exactly what aspect of my pf configuration is responsible. > > If so, can anybody suggest what changes would have to happen to make > > the two compatible? > A problem with same symptoms was already reported and fixed in > https://reviews.freebsd.org/D43769 > > Which version are you using? > > Best regards > Michael > > > > -Alan TLDR; tcp_rack is good, cc_chd is better, and tcp_bbr is best I want to follow up with the list to post my conclusions. Firstly tuexen@ helped me solve my problem: in FreeBSD 14.0 there is a 3-way incompatibility between (tcp_bbr || tcp_rack) && lro && pf. I can confirm that tcp_bbr works for me if I either disable LRO, disable PF, or switch to a 14.1 server. Here's the real problem: on multiple production servers, downloading large files (or ZFS send/recv streams) was slow. After ruling out many possible causes, wireshark revealed that the connection was suffering about 0.05% packet loss. I don't know the source of that packet loss, but I don't believe it to be congestion-related. Along with a 54ms RTT, that's a fatal combination for the throughput of loss-based congestion control algorithms. According to the Mathis Formula [1], I could only expect 1.1 MBps over such a connection. That's actually worse than what I saw. With default settings (cc_cubic), I averaged 5.6 MBps. Probably Mathis's assumptions are outdated, but that's still pretty close for such a simple formula that's 27 years old. So I benchmarked all available congestion control algorithms for single download streams. The results are summarized in the table below. Algo Packet Loss Rate Average Throughput vegas 0.05% 2.0 MBps newreno 0.05% 3.2 MBps cubic 0.05% 5.6 MBps hd 0.05% 8.6 MBps cdg 0.05% 13.5 MBps rack 0.04% 14 MBps htcp 0.05% 15 MBps dctcp 0.05% 15 MBps chd 0.05% 17.3 MBps bbr 0.05% 29.2 MBps cubic 10% 159 kBps chd 10% 208 kBps bbr 10% 5.7 MBps RACK seemed to achieve about the same maximum bandwidth as BBR, though it took a lot longer to get there. Also, with RACK, wireshark reported about 10x as many retransmissions as dropped packets, which is suspicious. At one point, something went haywire and packet loss briefly spiked to the neighborhood of 10%. I took advantage of the chaos to repeat my measurements. As the table shows, all algorithms sucked under those conditions, but BBR sucked impressively less than the others. Disclaimer: there was significant run-to-run variation; the presented results are averages. And I did not attempt to measure packet loss exactly for most runs; 0.05% is merely an average of a few selected runs. These measurements were taken on a production server running a real workload, which introduces noise. Soon I hope to have the opportunity to repeat the experiment on an idle server in the same environment. In conclusion, while we'd like to use BBR, we really can't until we upgrade to 14.1, which hopefully will be soon. So in the meantime we've switched all relevant servers from cubic to chd, and we'll reevaluate BBR after the upgrade. [1]: https://www.slac.stanford.edu/comp/net/wan-mon/thru-vs-loss.html -Alan From nobody Wed Jul 17 20:27:03 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPSCc1VHgz5QKXc for ; Wed, 17 Jul 2024 20:27:08 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (mail-n.franken.de [193.175.24.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPSCb57Fjz4frW; Wed, 17 Jul 2024 20:27:07 +0000 (UTC) (envelope-from tuexen@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (unknown [IPv6:2a02:8109:1140:c3d:25ea:b605:4a77:64ed]) (Authenticated sender: macmic) by drew.franken.de (Postfix) with ESMTPSA id 42212721E2817; Wed, 17 Jul 2024 22:27:04 +0200 (CEST) Content-Type: text/plain; charset=utf-8 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.600.62\)) Subject: Re: TCP Success Story (was Re: TCP_RACK, TCP_BBR, and firewalls) From: tuexen@freebsd.org In-Reply-To: Date: Wed, 17 Jul 2024 22:27:03 +0200 Cc: FreeBSD Net Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Alan Somers X-Mailer: Apple Mail (2.3774.600.62) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:680, ipnet:193.174.0.0/15, country:DE] X-Rspamd-Queue-Id: 4WPSCb57Fjz4frW > On 17. Jul 2024, at 22:00, Alan Somers wrote: >=20 > On Sat, Jul 13, 2024 at 1:50=E2=80=AFAM wrote: >>=20 >>> On 13. Jul 2024, at 01:43, Alan Somers wrote: >>>=20 >>> I've been experimenting with RACK and BBR. In my environment, they >>> can dramatically improve single-stream TCP performance, which is >>> awesome. But pf interferes. I have to disable pf in order for them >>> to work at all. >>>=20 >>> Is this a known limitation? If not, I will experiment some more to >>> determine exactly what aspect of my pf configuration is responsible. >>> If so, can anybody suggest what changes would have to happen to make >>> the two compatible? >> A problem with same symptoms was already reported and fixed in >> https://reviews.freebsd.org/D43769 >>=20 >> Which version are you using? >>=20 >> Best regards >> Michael >>>=20 >>> -Alan >=20 > TLDR; tcp_rack is good, cc_chd is better, and tcp_bbr is best >=20 > I want to follow up with the list to post my conclusions. Firstly > tuexen@ helped me solve my problem: in FreeBSD 14.0 there is a 3-way > incompatibility between (tcp_bbr || tcp_rack) && lro && pf. I can > confirm that tcp_bbr works for me if I either disable LRO, disable PF, > or switch to a 14.1 server. >=20 > Here's the real problem: on multiple production servers, downloading > large files (or ZFS send/recv streams) was slow. After ruling out > many possible causes, wireshark revealed that the connection was > suffering about 0.05% packet loss. I don't know the source of that > packet loss, but I don't believe it to be congestion-related. Along > with a 54ms RTT, that's a fatal combination for the throughput of > loss-based congestion control algorithms. According to the Mathis > Formula [1], I could only expect 1.1 MBps over such a connection. > That's actually worse than what I saw. With default settings > (cc_cubic), I averaged 5.6 MBps. Probably Mathis's assumptions are > outdated, but that's still pretty close for such a simple formula > that's 27 years old. >=20 > So I benchmarked all available congestion control algorithms for > single download streams. The results are summarized in the table > below. >=20 > Algo Packet Loss Rate Average Throughput > vegas 0.05% 2.0 MBps > newreno 0.05% 3.2 MBps > cubic 0.05% 5.6 MBps > hd 0.05% 8.6 MBps > cdg 0.05% 13.5 MBps > rack 0.04% 14 MBps > htcp 0.05% 15 MBps > dctcp 0.05% 15 MBps > chd 0.05% 17.3 MBps > bbr 0.05% 29.2 MBps > cubic 10% 159 kBps > chd 10% 208 kBps > bbr 10% 5.7 MBps >=20 > RACK seemed to achieve about the same maximum bandwidth as BBR, though > it took a lot longer to get there. Also, with RACK, wireshark > reported about 10x as many retransmissions as dropped packets, which > is suspicious. >=20 > At one point, something went haywire and packet loss briefly spiked to > the neighborhood of 10%. I took advantage of the chaos to repeat my > measurements. As the table shows, all algorithms sucked under those > conditions, but BBR sucked impressively less than the others. >=20 > Disclaimer: there was significant run-to-run variation; the presented > results are averages. And I did not attempt to measure packet loss > exactly for most runs; 0.05% is merely an average of a few selected > runs. These measurements were taken on a production server running a > real workload, which introduces noise. Soon I hope to have the > opportunity to repeat the experiment on an idle server in the same > environment. >=20 > In conclusion, while we'd like to use BBR, we really can't until we > upgrade to 14.1, which hopefully will be soon. So in the meantime > we've switched all relevant servers from cubic to chd, and we'll > reevaluate BBR after the upgrade. Hi Alan, just to be clear: the version of BBR currently implemented is BBR version 1, which is known to be unfair in certain scenarios. Google is still working on BBR to address this problem and improve it in other aspects. But there is no RFC yet and the updates haven't been implemented yet in FreeBSD. Best regards Michael >=20 > [1]: https://www.slac.stanford.edu/comp/net/wan-mon/thru-vs-loss.html >=20 > -Alan From nobody Thu Jul 18 10:38:05 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPq5V3QDZz5RCqF for ; Thu, 18 Jul 2024 10:38:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPq5V2Klkz4hy5 for ; Thu, 18 Jul 2024 10:38:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721299086; a=rsa-sha256; cv=none; b=q7op03viwkDyMBt+H6VFLTbbULpEPR8Etyp0xpvqdSfq2X52vxWBvlhAD4CtgSYAWjtUGt +ds9f5mLtstgAwG7zkA3loQx9oxYe+LZhikAfLYvnXT20njTwgaIEfzrSrOYE9ZWZL8d0p iRR8xt9xNZrmQ5nUMkKXTZBEVT08/hJKjXx9+ngvnygUxrFnaU/IZ5X6AL5HSgpO1lMQ+O Q6laN1ak2uJsvJtVPaJYcgN1eoEkj3oJT80HzDy6DBkZHtIu9+eWhWAVKz4wwO9tstsJqt pvC9RxZmNWkbTXPtArj7K3oz0gfC/F/hQr3v1OsTfMLD4boDB8KXQXX2obEtiQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721299086; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qNPABO4nattrQFh7Nv32mzlFRLwJvbHsIw080GHHRg0=; b=PDXic3w3DBj3lqIXQqjo50/9vkC2XuraTgkCq3vJQmB0IV45uw8/yX2YjNQ+M6FloUMyH8 bQAw4fc2QbUaTK5CbNQWCbwhV6zgyMJCAWfYvq/oHMO+wQZRCBGHzYKPEpPOQzaGHimebR 15hHWMAd0mQBsOzls3LgxmZySqKO9ytqy8dFlg2bmbJ89QEpyk4RS/58Jxy1nY5Bae+LHq qs/L/4eIxerT8/PixkX3jXC5IAYZgc1e8LItvp2VkmGAfH9K/hzY8YXC1hQAMUWdY30cJg c9nUREsFQO7NiIe1pA5Db58Wb9fRJ2FkIFjYlxVgaJeEdTVZgTTawEMYLsr/jA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WPq5V1qlpzlBC for ; Thu, 18 Jul 2024 10:38:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 46IAc6fh032728 for ; Thu, 18 Jul 2024 10:38:06 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 46IAc64r032727 for net@FreeBSD.org; Thu, 18 Jul 2024 10:38:06 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 280037] KTLS with Intel QAT may trigger kernel panics Date: Thu, 18 Jul 2024 10:38:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: 3226388001@jcom.home.ne.jp X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D280037 --- Comment #12 from ss3bsd <3226388001@jcom.home.ne.jp> --- > I'm now running the same machine with=20 > kern.ipc.tls.cbc_enable=3D0=20 > to see if the stability changes. The machine worked for 2 weeks without panic. Changed back kern.ipc.tls.cbc_enable=3D0 -> 1, a couple of days ago. The machine today panics again at CBC related part of code (ktls_ocf_tls_cbc_decrypt). So, the cause might be something specific to CBC. (not a strong evidence though.) -- Unread portion of the kernel message buffer: trap number =3D 12 panic: page fault cpuid =3D 1 time =3D 1721297633 KDB: stack backtrace: #0 0xffffffff809d2b5d at kdb_backtrace+0x5d #1 0xffffffff809858c1 at vpanic+0x131 #2 0xffffffff80985783 at panic+0x43 #3 0xffffffff80e5f91b at trap_fatal+0x40b #4 0xffffffff80e5f966 at trap_pfault+0x46 #5 0xffffffff80e36288 at calltrap+0x8 #6 0xffffffff80e21cc8 at bounce_bus_dmamap_load_buffer+0x178 #7 0xffffffff809c8a07 at bus_dmamap_load_crp_buffer+0x237 #8 0xffffffff82ddb7ad at qat_ocf_process+0x40d #9 0xffffffff80c83fd0 at crypto_dispatch+0x60 #10 0xffffffff80c8c3cd at ktls_ocf_dispatch+0x5d #11 0xffffffff80c8d3f4 at ktls_ocf_tls_cbc_decrypt+0x344 #12 0xffffffff80a1a484 at ktls_work_thread+0x664 #13 0xffffffff8093fc7f at fork_exit+0x7f #14 0xffffffff80e372ee at fork_trampoline+0xe Uptime: 2d17h13m52s --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Jul 18 10:40:00 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPq7j6tmQz5RCc5 for ; Thu, 18 Jul 2024 10:40:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPq7j48h8z4jdp for ; Thu, 18 Jul 2024 10:40:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721299201; a=rsa-sha256; cv=none; b=sIBlUHw7+MmLM3PFeRZ9ULIOPcOTlFMUk4fm6+9VQOIzasWpuHnPjRNAkPDvyL3+OxFq16 Cqn32qZinCQQv/PsGMpgZ+ruPExdWzn+mAZ177REjlorKm+hb+ehoeeTXBxl+UFNaCPJDT 8Ql264mD/VFXmbT934HYZSFVqUCLvfY/BTlsIBzq5HtNwypHfBQU4TOOYvEtIiwtczgSVy QVNGJbG9yzYdb0WZaBF8WsEfON9yHmSAYN4hZ76lQo0m6fQeCxbnt7e52wC5gtqlHKQl98 YC6KK5amQ/hzryiPOOi5omKtYWm/Jc/fUQTQDy6htpQxBMUry7OU95qj6SCiUg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721299201; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rYch72yGm2Tkf8TL4RUE9r9IZqxyEOVATOvZdaPlwX8=; b=i4lwwoX+wFEGkwpaQLcAbgIcSCoLxZC02GAX5g3ld1aVwjl1i/B6HBCVv1hQ8pQG8TN6Zy ntpiXTnoOejPPzAUW29aoOHKU8eNeuLA011gHkuVCMSjmrtx8b7LBXl3eRrRxbxm7LMtUY ugv0u55Pu2pdUKA5XoayqPB/i+UZe+y11Ro0lYIQEaInc780zwHCZ/21NTaRAZ+Lx1EqsR KTfWebT4R+40MVai6uG9TkExl9DLVwpwgyhtZEYwPfb2daPUcwxHo1UooJlHCwyFLIXlaT neAYY/yAKZwMWvYQ6FMXcNzz1niwINDTraXn/bY+GiOMF7N0jAwlX2BuAM/UVA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WPq7j3cXPzlfQ for ; Thu, 18 Jul 2024 10:40:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 46IAe1VT038231 for ; Thu, 18 Jul 2024 10:40:01 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 46IAe1i5038229 for net@FreeBSD.org; Thu, 18 Jul 2024 10:40:01 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 279875] sockstat: segmentation fault Date: Thu, 18 Jul 2024 10:40:00 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: john@jmarshall.id.au X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D279875 John Marshall changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |john@jmarshall.id.au --- Comment #1 from John Marshall --- 'Me too' Recent 14-STABLE amd64 FreeBSD 14.1-STABLE #0 stable/14-n268159-60f78f8ed14d: Tue Jul 16 19:25:41 AEST 2024=20=20=20=20 john@rwsrv08.gfn.riverwillow.net.au:/build/obj/john/kits/src/amd64.amd64/sy= s/RWSRV08 No segfault if I specify -j to restrict dispaly to one of the jails, only i= f I specify -j0 or omit -j. This is my third build of 14-STABLE (beginning early May) and all of them have done the same. Same vintage 14-STABLE on i386 is fine. I only have the two systems running FreeBSD. rwsrv08# lldb -X sockstat (lldb) target create "sockstat" Current executable set to '/usr/bin/sockstat' (x86_64). (lldb) run Process 87548 launched: '/usr/bin/sockstat' (x86_64) USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS= =20=20=20=20=20=20 root sockstat 87554 6 stream -> [87548 8] root sockstat 87553 6 stream -> [87548 7] ... root syslogd 2948 9 dgram /var/run/logpriv root gssd 2810 3 stream /var/run/gssd.sock Process 87548 stopped * thread #1, name =3D 'sockstat', stop reason =3D signal SIGSEGV: address n= ot mapped to object (fault address: 0x18) frame #0: 0x000002c892dde507 sockstat`displaysock [inlined] file_compare(a=3D, b=3D0x0000000000000000) at sockstat.c:179:38 176 static int64_t 177 file_compare(const struct file *a, const struct file *b) 178 { -> 179 return ((int64_t)(a->xf_data/2 - b->xf_data/2)); ^ 180 } 181 RB_GENERATE_STATIC(files_t, file, file_tree, file_compare); 182=20=20 (lldb) bt * thread #1, name =3D 'sockstat', stop reason =3D signal SIGSEGV: address n= ot mapped to object (fault address: 0x18) * frame #0: 0x000002c892dde507 sockstat`displaysock [inlined] file_compare(a=3D, b=3D0x0000000000000000) at sockstat.c:179:38 frame #1: 0x000002c892dde507 sockstat`displaysock [inlined] files_t_RB_FIND(head=3D, elm=3D) at sockstat.c:18= 1:1 frame #2: 0x000002c892dde4fe sockstat`displaysock(s=3D0x00001790ce24be0= 0, pos=3D) at sockstat.c:1165:10 frame #3: 0x000002c892ddd71f sockstat`display at sockstat.c:1345:4 frame #4: 0x000002c892ddcc07 sockstat`main(argc=3D, argv=3D) at sockstat.c:1577:2 frame #5: 0x000002d0b7f008da libc.so.7`__libc_start1(argc=3D1, argv=3D0x000002d0b2e0ed10, env=3D0x000002d0b2e0ed20, cleanup=3D, mainX=3D(sockstat`main at sockstat.c:1434)) at libc_start1.c:157:7 frame #6: 0x000002c892ddb18d sockstat`_start at crt1_s.S:83 (lldb) q --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Jul 18 10:48:24 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPqKN4c8nz5RDJd for ; Thu, 18 Jul 2024 10:48:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPqKN3K3gz4kFM for ; Thu, 18 Jul 2024 10:48:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721299704; a=rsa-sha256; cv=none; b=ko7pfUNOtMbvV++APCRBXPYEcZ6PboYrkyBPvZa0ejISWIVBKV6dTjtVM+xIvua+HJDrP+ Valsnqv9IO6aEsgdS3bMCedcbgyxThIC34TJ3v88Z1qvNjO+J3OaFQCI76jpIoSNkewqo3 rX8MJOUBldktT/XKV8LwNxS0MVCZhKzgteiqtgQZKi0uLkfPcnTM0GtXtoKjzDPElDs5Lp FasoGvM5Um4+7VAnQNWpEYvl+ACOaaxaW2MYziGGCtfmAFxE0A4ZhwK26dtnxr4xH+Y6oe opmaeNEdUummkfbg1sPEctVucNVTk8bMDgUcBMhFrFryC8KDL9tPOmnGBI3GKg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721299704; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=B0cu+UiWR9dEhvXiW/p6BK6ynvDlwjU7UlCyQ6bIAtE=; b=qhA+uDezSFCUrQXsMXTgkEP1T5i4FP/rlnrV/OnS2D7Aj5QcRj7nP+GBha5EZzIWnVUj3r CSBnMwivn8+Kzzrwq+C5C1p58XXtiXBsaQCvI1pn9OxvRFagB60KRBauXx5tylnK9sh92U cbZr70hP0bNUal6qACDG+Yx5uDAbdtQVriJo00NUzFoIZ0xLcMFhqO/KR4s1voI6iQ7z2x EBcWnD3IGtCFro4nPwIEo+ZfbfZ8paQCSnL1NGFrbpCfjv9NXQEnHGPXoNB2n560I1eX2A Ra6oyoDpasF+cW/vxcXId7Wr2rhr9JlzEY25P1wVCunyiB8lPEA1+r5GMX92OQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WPqKN2q7Kzltp for ; Thu, 18 Jul 2024 10:48:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 46IAmOch074199 for ; Thu, 18 Jul 2024 10:48:24 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 46IAmOv6074194 for net@FreeBSD.org; Thu, 18 Jul 2024 10:48:24 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 280037] KTLS with Intel QAT may trigger kernel panics Date: Thu, 18 Jul 2024 10:48:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: 3226388001@jcom.home.ne.jp X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D280037 --- Comment #13 from ss3bsd <3226388001@jcom.home.ne.jp> --- By the way, is there any easy way to disable only CBC acceleration of QAT b= ut enable that of KTLS offload? I want to try that next if there is. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Jul 18 13:00:38 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPtGn0S7Bz5RPkY for ; Thu, 18 Jul 2024 13:01:21 +0000 (UTC) (envelope-from junho.choi@gmail.com) Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [IPv6:2a00:1450:4864:20::12b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPtGm4jFpz3xWN; Thu, 18 Jul 2024 13:01:20 +0000 (UTC) (envelope-from junho.choi@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x12b.google.com with SMTP id 2adb3069b0e04-52e9f788e7bso407497e87.0; Thu, 18 Jul 2024 06:01:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721307677; x=1721912477; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=zULF/gbAryHYfR6gEc0c+LdZEGoRE2j1Ibpbvpx3eCA=; b=IK2xXuSsDnMky55gASOgRMkTbDkyaK5V0wsqWKKnIpbqu5GIL62sqJ/v1x8FkGmvTF VKd6V2HYy7MWkU9Bh8zgwp9WqNY4U3l9jPHQPQGLyjYOom8xTWeICM585+r94Bo1/9Av LSaZDjnUAa5bgNYNcL1cgfSe1lwgY6ZaCxTLbENqQsouUJIwDzK2Ce14Tv16nlw6ckrU SjuldroXGCNjjDhqkDbJgPyMxfhVg0wZBdbRIJkzIVc4D8G6A6nrFub39MElWtokUwZA Py81ZayLbtvF+i6Rst/3Eaw5Un2r57ArFdQnxbpPelf9IYW83MzC1dsbALD3VU6RLzd9 bbTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721307677; x=1721912477; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=zULF/gbAryHYfR6gEc0c+LdZEGoRE2j1Ibpbvpx3eCA=; b=S9gN8K48dRJ+c+OBYLJg0FOEPrGCxe1nJCjAQPK1tU/49cNkMJ80VL8PaSjoo0OSs2 cLTTzA/vkiJplvWJ4MPgClhJ858mzB/IilKiPfgXaMjjd2esxY89Ur3bBty4isIe5mu4 19O5wEC2vEyayhiRD2NraCGeDwnxB+wYIwat0442PKTdnJUoskhwGM7SNNVycOCQgRCB /bMvyrFKrNO+iwGxGk7a0DYtdrsjt/Zngc5wy7wZpsKuq6xNeqx6MAbHhntN/7Ri35qH VczrAGhNAqm0rRH+0AAGcX/ndaEtpn/AtQ6BvJyzkoecr27uL28VWTUeBQhx0uNsljWN gPyg== X-Forwarded-Encrypted: i=1; AJvYcCWxDukFe9ueR0X9DSXJSSVbL44zH7QBHVGHZdbXHWjhsgUj4DWXEHWX+v+2uCHusB8VxmiGcw6iNh3cAE2e6F92hhLMV3XGbg== X-Gm-Message-State: AOJu0YxWA5v/YyeEFYHXjwaYmy+a0CSWBimlnkrBUH8mwc827TZQwDVC EOgafW32tC22fcJOs/LwP4PdfDkj36fgyCqGg/G7uKY27WWqFRz8Fw1wUVcCtWDAOAY6Pz5pxds IaHHi8VVg0oXUrxZOmX0vV56FVWHX5Q== X-Google-Smtp-Source: AGHT+IEN02eOpwejlEhEq0Si5kmcszm7F/atZMsxSUYTIerKAxRgi9ljMVXru1MoZ0Zju9NtczPmOD9OFRIRZQvPUh8= X-Received: by 2002:a05:6512:1243:b0:52c:df86:68c6 with SMTP id 2adb3069b0e04-52ee53b1b4amr3345566e87.16.1721307676726; Thu, 18 Jul 2024 06:01:16 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Junho Choi Date: Thu, 18 Jul 2024 22:00:38 +0900 Message-ID: Subject: Re: TCP Success Story (was Re: TCP_RACK, TCP_BBR, and firewalls) To: tuexen@freebsd.org Cc: Alan Somers , FreeBSD Net Content-Type: multipart/alternative; boundary="0000000000009edf8f061d852cb0" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4WPtGm4jFpz3xWN --0000000000009edf8f061d852cb0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Alan - this is a great result to see. Thanks for experimenting. Just curious why bbr and rack don't co-exist? Those are two separate things= . Is it a current bug or by design? BR, On Thu, Jul 18, 2024 at 5:27=E2=80=AFAM wrote: > > On 17. Jul 2024, at 22:00, Alan Somers wrote: > > > > On Sat, Jul 13, 2024 at 1:50=E2=80=AFAM wrote: > >> > >>> On 13. Jul 2024, at 01:43, Alan Somers wrote: > >>> > >>> I've been experimenting with RACK and BBR. In my environment, they > >>> can dramatically improve single-stream TCP performance, which is > >>> awesome. But pf interferes. I have to disable pf in order for them > >>> to work at all. > >>> > >>> Is this a known limitation? If not, I will experiment some more to > >>> determine exactly what aspect of my pf configuration is responsible. > >>> If so, can anybody suggest what changes would have to happen to make > >>> the two compatible? > >> A problem with same symptoms was already reported and fixed in > >> https://reviews.freebsd.org/D43769 > >> > >> Which version are you using? > >> > >> Best regards > >> Michael > >>> > >>> -Alan > > > > TLDR; tcp_rack is good, cc_chd is better, and tcp_bbr is best > > > > I want to follow up with the list to post my conclusions. Firstly > > tuexen@ helped me solve my problem: in FreeBSD 14.0 there is a 3-way > > incompatibility between (tcp_bbr || tcp_rack) && lro && pf. I can > > confirm that tcp_bbr works for me if I either disable LRO, disable PF, > > or switch to a 14.1 server. > > > > Here's the real problem: on multiple production servers, downloading > > large files (or ZFS send/recv streams) was slow. After ruling out > > many possible causes, wireshark revealed that the connection was > > suffering about 0.05% packet loss. I don't know the source of that > > packet loss, but I don't believe it to be congestion-related. Along > > with a 54ms RTT, that's a fatal combination for the throughput of > > loss-based congestion control algorithms. According to the Mathis > > Formula [1], I could only expect 1.1 MBps over such a connection. > > That's actually worse than what I saw. With default settings > > (cc_cubic), I averaged 5.6 MBps. Probably Mathis's assumptions are > > outdated, but that's still pretty close for such a simple formula > > that's 27 years old. > > > > So I benchmarked all available congestion control algorithms for > > single download streams. The results are summarized in the table > > below. > > > > Algo Packet Loss Rate Average Throughput > > vegas 0.05% 2.0 MBps > > newreno 0.05% 3.2 MBps > > cubic 0.05% 5.6 MBps > > hd 0.05% 8.6 MBps > > cdg 0.05% 13.5 MBps > > rack 0.04% 14 MBps > > htcp 0.05% 15 MBps > > dctcp 0.05% 15 MBps > > chd 0.05% 17.3 MBps > > bbr 0.05% 29.2 MBps > > cubic 10% 159 kBps > > chd 10% 208 kBps > > bbr 10% 5.7 MBps > > > > RACK seemed to achieve about the same maximum bandwidth as BBR, though > > it took a lot longer to get there. Also, with RACK, wireshark > > reported about 10x as many retransmissions as dropped packets, which > > is suspicious. > > > > At one point, something went haywire and packet loss briefly spiked to > > the neighborhood of 10%. I took advantage of the chaos to repeat my > > measurements. As the table shows, all algorithms sucked under those > > conditions, but BBR sucked impressively less than the others. > > > > Disclaimer: there was significant run-to-run variation; the presented > > results are averages. And I did not attempt to measure packet loss > > exactly for most runs; 0.05% is merely an average of a few selected > > runs. These measurements were taken on a production server running a > > real workload, which introduces noise. Soon I hope to have the > > opportunity to repeat the experiment on an idle server in the same > > environment. > > > > In conclusion, while we'd like to use BBR, we really can't until we > > upgrade to 14.1, which hopefully will be soon. So in the meantime > > we've switched all relevant servers from cubic to chd, and we'll > > reevaluate BBR after the upgrade. > Hi Alan, > > just to be clear: the version of BBR currently implemented is > BBR version 1, which is known to be unfair in certain scenarios. > Google is still working on BBR to address this problem and improve > it in other aspects. But there is no RFC yet and the updates haven't > been implemented yet in FreeBSD. > > Best regards > Michael > > > > [1]: https://www.slac.stanford.edu/comp/net/wan-mon/thru-vs-loss.html > > > > -Alan > > > --=20 Junho Choi | https://saturnsoft.net --0000000000009edf8f061d852cb0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Alan - this is a great result to see. Thanks for expe= rimenting.

Just curious why bbr and rack don&#= 39;t co-exist? Those are two separate things.
Is it a current bug= or by design?

BR,

On Thu, Jul 18, 2024= at 5:27=E2=80=AFAM <tuexen@freebs= d.org> wrote:
> On 17. Jul 2024, at 22:00, Alan Somers <asomers@freebsd.org> wrote:
>
> On Sat, Jul 13, 2024 at 1:50=E2=80=AFAM <tuexen@freebsd.org> wrote:
>>
>>> On 13. Jul 2024, at 01:43, Alan Somers <asomers@FreeBSD.org= > wrote:
>>>
>>> I've been experimenting with RACK and BBR.=C2=A0 In my env= ironment, they
>>> can dramatically improve single-stream TCP performance, which = is
>>> awesome.=C2=A0 But pf interferes.=C2=A0 I have to disable pf i= n order for them
>>> to work at all.
>>>
>>> Is this a known limitation?=C2=A0 If not, I will experiment so= me more to
>>> determine exactly what aspect of my pf configuration is respon= sible.
>>> If so, can anybody suggest what changes would have to happen t= o make
>>> the two compatible?
>> A problem with same symptoms was already reported and fixed in
>> https://reviews.freebsd.org/D43769
>>
>> Which version are you using?
>>
>> Best regards
>> Michael
>>>
>>> -Alan
>
> TLDR; tcp_rack is good, cc_chd is better, and tcp_bbr is best
>
> I want to follow up with the list to post my conclusions.=C2=A0 Firstl= y
> tuexen@ helped me solve my problem: in FreeBSD 14.0 there is a 3-way > incompatibility between (tcp_bbr || tcp_rack) && lro &&= ; pf.=C2=A0 I can
> confirm that tcp_bbr works for me if I either disable LRO, disable PF,=
> or switch to a 14.1 server.
>
> Here's the real problem: on multiple production servers, downloadi= ng
> large files (or ZFS send/recv streams) was slow.=C2=A0 After ruling ou= t
> many possible causes, wireshark revealed that the connection was
> suffering about 0.05% packet loss.=C2=A0 I don't know the source o= f that
> packet loss, but I don't believe it to be congestion-related.=C2= =A0 Along
> with a 54ms RTT, that's a fatal combination for the throughput of<= br> > loss-based congestion control algorithms.=C2=A0 According to the Mathi= s
> Formula [1], I could only expect 1.1 MBps over such a connection.
> That's actually worse than what I saw.=C2=A0 With default settings=
> (cc_cubic), I averaged 5.6 MBps.=C2=A0 Probably Mathis's assumptio= ns are
> outdated, but that's still pretty close for such a simple formula<= br> > that's 27 years old.
>
> So I benchmarked all available congestion control algorithms for
> single download streams.=C2=A0 The results are summarized in the table=
> below.
>
> Algo=C2=A0 =C2=A0 Packet Loss Rate=C2=A0 =C2=A0 Average Throughput
> vegas=C2=A0 =C2=A00.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A02.0 MBps
> newreno 0.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03.= 2 MBps
> cubic=C2=A0 =C2=A00.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A05.6 MBps
> hd=C2=A0 =C2=A0 =C2=A0 0.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A08.6 MBps
> cdg=C2=A0 =C2=A0 =C2=A00.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A013.5 MBps
> rack=C2=A0 =C2=A0 0.04%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A014 MBps
> htcp=C2=A0 =C2=A0 0.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A015 MBps
> dctcp=C2=A0 =C2=A00.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A015 MBps
> chd=C2=A0 =C2=A0 =C2=A00.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A017.3 MBps
> bbr=C2=A0 =C2=A0 =C2=A00.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A029.2 MBps
> cubic=C2=A0 =C2=A010%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0159 kBps
> chd=C2=A0 =C2=A0 =C2=A010%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0208 kBps
> bbr=C2=A0 =C2=A0 =C2=A010%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A05.7 MBps
>
> RACK seemed to achieve about the same maximum bandwidth as BBR, though=
> it took a lot longer to get there.=C2=A0 Also, with RACK, wireshark > reported about 10x as many retransmissions as dropped packets, which > is suspicious.
>
> At one point, something went haywire and packet loss briefly spiked to=
> the neighborhood of 10%.=C2=A0 I took advantage of the chaos to repeat= my
> measurements.=C2=A0 As the table shows, all algorithms sucked under th= ose
> conditions, but BBR sucked impressively less than the others.
>
> Disclaimer: there was significant run-to-run variation; the presented<= br> > results are averages.=C2=A0 And I did not attempt to measure packet lo= ss
> exactly for most runs; 0.05% is merely an average of a few selected > runs.=C2=A0 These measurements were taken on a production server runni= ng a
> real workload, which introduces noise.=C2=A0 Soon I hope to have the > opportunity to repeat the experiment on an idle server in the same
> environment.
>
> In conclusion, while we'd like to use BBR, we really can't unt= il we
> upgrade to 14.1, which hopefully will be soon.=C2=A0 So in the meantim= e
> we've switched all relevant servers from cubic to chd, and we'= ll
> reevaluate BBR after the upgrade.
Hi Alan,

just to be clear: the version of BBR currently implemented is
BBR version 1, which is known to be unfair in certain scenarios.
Google is still working on BBR to address this problem and improve
it in other aspects. But there is no RFC yet and the updates haven't been implemented yet in FreeBSD.

Best regards
Michael
>
> [1]: https://www.slac.stanford.= edu/comp/net/wan-mon/thru-vs-loss.html
>
> -Alan




--
Junho Choi <junho dot choi at gmail.com> | https://saturnsoft.net
--0000000000009edf8f061d852cb0-- From nobody Thu Jul 18 14:02:14 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPvdL0yysz5RTtR for ; Thu, 18 Jul 2024 14:02:30 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oo1-f53.google.com (mail-oo1-f53.google.com [209.85.161.53]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPvdK2Kktz44qD; Thu, 18 Jul 2024 14:02:29 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.161.53 as permitted sender) smtp.mailfrom=asomers@gmail.com Received: by mail-oo1-f53.google.com with SMTP id 006d021491bc7-5ce739c2650so437928eaf.1; Thu, 18 Jul 2024 07:02:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721311348; x=1721916148; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=biDhjVlSlGYuJF3U027WFtvxNX7E00DlTuXBjCgbZqU=; b=v5emZPz7pbTDPl+ie6++Ago4mVsKdR+nud7iYGEEGtNe62WruqAQh61OONLYtpVk7M BTbcLLIiHpvduFVYHNrqmSj5Ktgh84/VgRTyHzGFpaa97EenPpX4iVux+hX5HiLDiu+b VguHkLePBlz2cUbIVtQpjVLAqzWGaqI7hw1U3QyWGm6E6ULqp+fLSzioEEaDitpAP6mF bfPOYiIvnndT1lcVaRJfbRvRrzNRXq4axkn+pnWtIF6tXZIKfggcXzWeQF/Xx/By7Re5 nE5etlHF+O52/srPB2CAXHNondL4OhluhEiQe0OBTQD3XcsFlxn/2dEZeEuYztJ+gJxQ kKuQ== X-Forwarded-Encrypted: i=1; AJvYcCVHcLEpCVQCjogGg3C475l7AkzEq8MIY+72eeXstEHO1Kma0fY18eAOMcR4372QKU1jzE6xSBDHULVnhXrpgNrI1uTLlEcgSA== X-Gm-Message-State: AOJu0YyN6SgcyV1cP4HXFeCFCfQBSUO7E203xy6pDSKjMppMNoh9JdvO iUC86MxZvw/4es6y3CQAHlnocp+iZHqR6nBIyH7FGVrRnwsRZ4zHA5VPwFiASHEeZ6pl52899FD 83uibaUGr+r9+fZlqVGkDwNFofyWtyw== X-Google-Smtp-Source: AGHT+IE2qEWsonK/N+Ih61FK6NB+KWDoDQtqypO5tKWWZcsGYsz2cB3+ecZo+BLtUKxT4+mPpL1NQnutrhl+SGjPMj4= X-Received: by 2002:a05:6358:e48b:b0:1ab:f2b3:f018 with SMTP id e5c5f4694b2df-1aca9c3b0eemr256219555d.0.1721311346598; Thu, 18 Jul 2024 07:02:26 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Thu, 18 Jul 2024 08:02:14 -0600 Message-ID: Subject: Re: TCP Success Story (was Re: TCP_RACK, TCP_BBR, and firewalls) To: Junho Choi Cc: tuexen@freebsd.org, FreeBSD Net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.82)[-0.822]; NEURAL_HAM_MEDIUM(-0.68)[-0.680]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; RWL_MAILSPIKE_GOOD(-0.10)[209.85.161.53:from]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_TO(0.00)[gmail.com]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; MISSING_XM_UA(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; TAGGED_RCPT(0.00)[]; FREEFALL_USER(0.00)[asomers]; RCVD_IN_DNSWL_NONE(0.00)[209.85.161.53:from] X-Rspamd-Queue-Id: 4WPvdK2Kktz44qD I'm not sure what you're asking. BBR and RACK are two different algorithms that accomplish the same thing. It wouldn't make sense to use both on the same socket at the same time. On Thu, Jul 18, 2024 at 7:01=E2=80=AFAM Junho Choi w= rote: > > Alan - this is a great result to see. Thanks for experimenting. > > Just curious why bbr and rack don't co-exist? Those are two separate thin= gs. > Is it a current bug or by design? > > BR, > > On Thu, Jul 18, 2024 at 5:27=E2=80=AFAM wrote: >> >> > On 17. Jul 2024, at 22:00, Alan Somers wrote: >> > >> > On Sat, Jul 13, 2024 at 1:50=E2=80=AFAM wrote: >> >> >> >>> On 13. Jul 2024, at 01:43, Alan Somers wrote: >> >>> >> >>> I've been experimenting with RACK and BBR. In my environment, they >> >>> can dramatically improve single-stream TCP performance, which is >> >>> awesome. But pf interferes. I have to disable pf in order for them >> >>> to work at all. >> >>> >> >>> Is this a known limitation? If not, I will experiment some more to >> >>> determine exactly what aspect of my pf configuration is responsible. >> >>> If so, can anybody suggest what changes would have to happen to make >> >>> the two compatible? >> >> A problem with same symptoms was already reported and fixed in >> >> https://reviews.freebsd.org/D43769 >> >> >> >> Which version are you using? >> >> >> >> Best regards >> >> Michael >> >>> >> >>> -Alan >> > >> > TLDR; tcp_rack is good, cc_chd is better, and tcp_bbr is best >> > >> > I want to follow up with the list to post my conclusions. Firstly >> > tuexen@ helped me solve my problem: in FreeBSD 14.0 there is a 3-way >> > incompatibility between (tcp_bbr || tcp_rack) && lro && pf. I can >> > confirm that tcp_bbr works for me if I either disable LRO, disable PF, >> > or switch to a 14.1 server. >> > >> > Here's the real problem: on multiple production servers, downloading >> > large files (or ZFS send/recv streams) was slow. After ruling out >> > many possible causes, wireshark revealed that the connection was >> > suffering about 0.05% packet loss. I don't know the source of that >> > packet loss, but I don't believe it to be congestion-related. Along >> > with a 54ms RTT, that's a fatal combination for the throughput of >> > loss-based congestion control algorithms. According to the Mathis >> > Formula [1], I could only expect 1.1 MBps over such a connection. >> > That's actually worse than what I saw. With default settings >> > (cc_cubic), I averaged 5.6 MBps. Probably Mathis's assumptions are >> > outdated, but that's still pretty close for such a simple formula >> > that's 27 years old. >> > >> > So I benchmarked all available congestion control algorithms for >> > single download streams. The results are summarized in the table >> > below. >> > >> > Algo Packet Loss Rate Average Throughput >> > vegas 0.05% 2.0 MBps >> > newreno 0.05% 3.2 MBps >> > cubic 0.05% 5.6 MBps >> > hd 0.05% 8.6 MBps >> > cdg 0.05% 13.5 MBps >> > rack 0.04% 14 MBps >> > htcp 0.05% 15 MBps >> > dctcp 0.05% 15 MBps >> > chd 0.05% 17.3 MBps >> > bbr 0.05% 29.2 MBps >> > cubic 10% 159 kBps >> > chd 10% 208 kBps >> > bbr 10% 5.7 MBps >> > >> > RACK seemed to achieve about the same maximum bandwidth as BBR, though >> > it took a lot longer to get there. Also, with RACK, wireshark >> > reported about 10x as many retransmissions as dropped packets, which >> > is suspicious. >> > >> > At one point, something went haywire and packet loss briefly spiked to >> > the neighborhood of 10%. I took advantage of the chaos to repeat my >> > measurements. As the table shows, all algorithms sucked under those >> > conditions, but BBR sucked impressively less than the others. >> > >> > Disclaimer: there was significant run-to-run variation; the presented >> > results are averages. And I did not attempt to measure packet loss >> > exactly for most runs; 0.05% is merely an average of a few selected >> > runs. These measurements were taken on a production server running a >> > real workload, which introduces noise. Soon I hope to have the >> > opportunity to repeat the experiment on an idle server in the same >> > environment. >> > >> > In conclusion, while we'd like to use BBR, we really can't until we >> > upgrade to 14.1, which hopefully will be soon. So in the meantime >> > we've switched all relevant servers from cubic to chd, and we'll >> > reevaluate BBR after the upgrade. >> Hi Alan, >> >> just to be clear: the version of BBR currently implemented is >> BBR version 1, which is known to be unfair in certain scenarios. >> Google is still working on BBR to address this problem and improve >> it in other aspects. But there is no RFC yet and the updates haven't >> been implemented yet in FreeBSD. >> >> Best regards >> Michael >> > >> > [1]: https://www.slac.stanford.edu/comp/net/wan-mon/thru-vs-loss.html >> > >> > -Alan >> >> > > > -- > Junho Choi | https://saturnsoft.net From nobody Thu Jul 18 14:03:33 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPvfp3F5tz5RVWg for ; Thu, 18 Jul 2024 14:03:46 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPvfp1TT5z45V8; Thu, 18 Jul 2024 14:03:46 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qt1-f173.google.com with SMTP id d75a77b69052e-449f23df593so1733031cf.1; Thu, 18 Jul 2024 07:03:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721311425; x=1721916225; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Qyz7cdYAeGl5GeyWeUTFUEypZQsAxhGf0ioV8wHg00k=; b=nNU/Z+owltbWZZqAEA5QavioLLoK0fbX/R3yqhX9+ITgIv+KQ667NbpJeTwkXevkug 1YQ6PLo9JyEc90A5ZzRxG+tNT5Nkju/scnQQaOozKGLNYuCRV9pzw50lQ+CqTPdUZzvT x1kOclWl8dPZQxWu5NqjxVoMajq5EME+y0Oj/IkvcvPWqe2DoMVky1Ruj7M9CLlDE+Ie k+H6gkKezR6hfZYz5DTuNjHatSrnO/9HnMZ5QILzdxHc+7dbIT/nT5YKlEhiOQ2EpRX0 DegRr2eWNa6miA4wQeeGSk/qyCHl0HG+0kF0T2GUxXd8AtUAN7NG3JXvR7yFAFvYIdf8 j5gQ== X-Gm-Message-State: AOJu0Yxv2iYvsr+j0IyQSQNk9M5bQ5Ocy6NNH5zqykrQS2IMhCL5ULWO 2opQIAAIVvjRpw/2fdAcaYorLPdV5PDWOy8MSgCil1uoUPYgji97rxnleMJjOa16JHxBCVKEQKK ov3zFBh0ULHbfi5OKWEc61cTJl7eJYw== X-Google-Smtp-Source: AGHT+IHQc/3wZjn3JUkEsWNa2it2emnVnnsE6aRvmRwHhf5yYwsRi32mBoNG6toGDO8GwDWPRbLFgok4AaCAR/6UrUo= X-Received: by 2002:ac8:7e8f:0:b0:447:dc9a:1cea with SMTP id d75a77b69052e-44f96a1bc99mr10323001cf.13.1721311424900; Thu, 18 Jul 2024 07:03:44 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Thu, 18 Jul 2024 08:03:33 -0600 Message-ID: Subject: Re: TCP Success Story (was Re: TCP_RACK, TCP_BBR, and firewalls) To: tuexen@freebsd.org Cc: FreeBSD Net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Queue-Id: 4WPvfp1TT5z45V8 On Wed, Jul 17, 2024 at 2:27=E2=80=AFPM wrote: > > > On 17. Jul 2024, at 22:00, Alan Somers wrote: > > > > On Sat, Jul 13, 2024 at 1:50=E2=80=AFAM wrote: > >> > >>> On 13. Jul 2024, at 01:43, Alan Somers wrote: > >>> > >>> I've been experimenting with RACK and BBR. In my environment, they > >>> can dramatically improve single-stream TCP performance, which is > >>> awesome. But pf interferes. I have to disable pf in order for them > >>> to work at all. > >>> > >>> Is this a known limitation? If not, I will experiment some more to > >>> determine exactly what aspect of my pf configuration is responsible. > >>> If so, can anybody suggest what changes would have to happen to make > >>> the two compatible? > >> A problem with same symptoms was already reported and fixed in > >> https://reviews.freebsd.org/D43769 > >> > >> Which version are you using? > >> > >> Best regards > >> Michael > >>> > >>> -Alan > > > > TLDR; tcp_rack is good, cc_chd is better, and tcp_bbr is best > > > > I want to follow up with the list to post my conclusions. Firstly > > tuexen@ helped me solve my problem: in FreeBSD 14.0 there is a 3-way > > incompatibility between (tcp_bbr || tcp_rack) && lro && pf. I can > > confirm that tcp_bbr works for me if I either disable LRO, disable PF, > > or switch to a 14.1 server. > > > > Here's the real problem: on multiple production servers, downloading > > large files (or ZFS send/recv streams) was slow. After ruling out > > many possible causes, wireshark revealed that the connection was > > suffering about 0.05% packet loss. I don't know the source of that > > packet loss, but I don't believe it to be congestion-related. Along > > with a 54ms RTT, that's a fatal combination for the throughput of > > loss-based congestion control algorithms. According to the Mathis > > Formula [1], I could only expect 1.1 MBps over such a connection. > > That's actually worse than what I saw. With default settings > > (cc_cubic), I averaged 5.6 MBps. Probably Mathis's assumptions are > > outdated, but that's still pretty close for such a simple formula > > that's 27 years old. > > > > So I benchmarked all available congestion control algorithms for > > single download streams. The results are summarized in the table > > below. > > > > Algo Packet Loss Rate Average Throughput > > vegas 0.05% 2.0 MBps > > newreno 0.05% 3.2 MBps > > cubic 0.05% 5.6 MBps > > hd 0.05% 8.6 MBps > > cdg 0.05% 13.5 MBps > > rack 0.04% 14 MBps > > htcp 0.05% 15 MBps > > dctcp 0.05% 15 MBps > > chd 0.05% 17.3 MBps > > bbr 0.05% 29.2 MBps > > cubic 10% 159 kBps > > chd 10% 208 kBps > > bbr 10% 5.7 MBps > > > > RACK seemed to achieve about the same maximum bandwidth as BBR, though > > it took a lot longer to get there. Also, with RACK, wireshark > > reported about 10x as many retransmissions as dropped packets, which > > is suspicious. > > > > At one point, something went haywire and packet loss briefly spiked to > > the neighborhood of 10%. I took advantage of the chaos to repeat my > > measurements. As the table shows, all algorithms sucked under those > > conditions, but BBR sucked impressively less than the others. > > > > Disclaimer: there was significant run-to-run variation; the presented > > results are averages. And I did not attempt to measure packet loss > > exactly for most runs; 0.05% is merely an average of a few selected > > runs. These measurements were taken on a production server running a > > real workload, which introduces noise. Soon I hope to have the > > opportunity to repeat the experiment on an idle server in the same > > environment. > > > > In conclusion, while we'd like to use BBR, we really can't until we > > upgrade to 14.1, which hopefully will be soon. So in the meantime > > we've switched all relevant servers from cubic to chd, and we'll > > reevaluate BBR after the upgrade. > Hi Alan, > > just to be clear: the version of BBR currently implemented is > BBR version 1, which is known to be unfair in certain scenarios. > Google is still working on BBR to address this problem and improve > it in other aspects. But there is no RFC yet and the updates haven't > been implemented yet in FreeBSD. I've also heard that RACK suffers from fairness problems. Do you know how RACK and BBR compare for fairness? From nobody Thu Jul 18 15:23:27 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPxQr320sz5Rbcv for ; Thu, 18 Jul 2024 15:23:32 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qv1-xf2c.google.com (mail-qv1-xf2c.google.com [IPv6:2607:f8b0:4864:20::f2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPxQq4Gs2z4FZj; Thu, 18 Jul 2024 15:23:31 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=Lgf6yPW7; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::f2c as permitted sender) smtp.mailfrom=markjdb@gmail.com Received: by mail-qv1-xf2c.google.com with SMTP id 6a1803df08f44-6b7734055d1so4995946d6.3; Thu, 18 Jul 2024 08:23:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721316210; x=1721921010; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=Uz8/rHwkccAkXqPD4L/69CxuE7PDXhu8Rlr7d8GHQqo=; b=Lgf6yPW7j1HJFgP3woeE1fbiwO15u57MolQMpFNotRMo8AiwGj7TYeZNA63+7oJ2rJ U5SKvmw7VnvOUWbGRvkWFd1AMHkInKczRVLLjBrxgEaS80kOqDhUB/8TdNEuxel8MCoU 1j3RajA2m+1+h9/omDdX0G9QV9jriFunyIeRFX5wF5LOn1GMmy8VUyoVR6FUVNMiF+1r vJZHz7uD7SXaYUHChrosch5U9G8Gk2OIclL8RXnzfsWrNJQYgSFxPIS3BNAtRYiW3KiW aSyO1v/SttiAyslgLqG109ZZPygqTSVf6NGUAXbP8O3ddrCq9YMYbDmCHKSTxCRdyVQx eWxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721316210; x=1721921010; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Uz8/rHwkccAkXqPD4L/69CxuE7PDXhu8Rlr7d8GHQqo=; b=lksgVZ/rUZLPWY7WMyc1QKFvshdcLJG/3LyhxaVaLyDCp+mvoSZtuVBAGp3PWAlXOm DD5JVEP/XgUMFP17TwwZkFPe8XsDkQJJUjWtyAVsB8OlY30LYrT6Hajgsq/T5GyQwKmG jg/R80gZp5xsyMslRpRBoBtucw+nDd38SaFphVaay9T95fv7m8pnEf7wX/Jp/D80coVu tSGUGgZcX6fvmumY5BqfgG6WnhAXZ/GK/8auvtW6okVfHm5ZMifeyZiz7dEbyURdY36a gUDQLhZ8mfZI6bfF2AGfsoD/vl6m8oBKisfqHoaFVNUsdYrVX6MPt5/q4KIjc/9SQy8Z kDUQ== X-Gm-Message-State: AOJu0Yz3nDdQSOczzJdyeLfJuL4H6eYhOopWcPBUDUUuWv25AmIj5Lzu 2j4NrACf32Uw4tRQ8E64zAA+OfgPENXU5TRf+vWclzhwnDOJhfsjINlwpt8X X-Google-Smtp-Source: AGHT+IFwyZr8BOrRgw7vi2Kwpxcqsbjs3uGPZEAu/eqSMcp0i8LGSfQ9u6Yl+DlNZygVaA9K8WjezA== X-Received: by 2002:a05:6214:2688:b0:6b5:a4f6:daa2 with SMTP id 6a1803df08f44-6b79c5392f6mr33756576d6.17.1721316210127; Thu, 18 Jul 2024 08:23:30 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b79c50d560sm9080116d6.53.2024.07.18.08.23.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jul 2024 08:23:29 -0700 (PDT) Date: Thu, 18 Jul 2024 11:23:27 -0400 From: Mark Johnston To: Zhenlei Huang Cc: freebsd-net@freebsd.org Subject: Re: flushing default router list upon inet6 route flush Message-ID: References: List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.54 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.94)[-0.939]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), DKIM not aligned (relaxed),none]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; RCVD_TLS_LAST(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::f2c:from] X-Rspamd-Queue-Id: 4WPxQq4Gs2z4FZj On Wed, Jul 17, 2024 at 09:19:53AM +0800, Zhenlei Huang wrote: > > > > On Jul 17, 2024, at 4:04 AM, Mark Johnston wrote: > > > > Hello, > > > > When IPv6 SLAAC is configured for an interface, the kernel will update > > its default router list upon receipt of a router advertisement. In so > > doing it may install a default route; in the kernel this happens in > > defrouter_addreq(). > > > > If one uses "route flush" or "service routing restart" to reset the > > routing tables, the default router list is not purged, so a subsequent > > RA from the original default router does not update the list, and so > > does not re-create the default route, even if one re-runs rtsol(8). > > > > This appears to be a bug, but I'm not sure where best to fix it. Should > > "service routing restart" invoke "ndp -R" to flush the default router > > list? > > That can be a workaround, but not the ideal fix. > > > Should route(8) handle this as part of a flush command? > > No, I do not think so. route(8) should handle the routing / FIB parts. > IPv6 default route list is maintained as a per AF basis. Handling the > default route list via route(8), aka the userland, seems to be more a > HACK. > > > Or > > something else? > > I'd propose that the kernel handle this situation, so that for other cases > such as `route -6 delete default`, or route change event from NETLINK > socket, the IPv6 SLAAC default router feature can also work as expected. > > To be precise, `sys/netinet6/nd6_rtr.c` listen on route events and clear > the `installed` flag on deleting the previously installed default route, or > maybe purge all default route list. Then the next time the kernel receives > a RA it re-installs the default route. Thank you for the hint. It turns out that the kernel already does this, but a bug was preventing it from working correctly. https://reviews.freebsd.org/D46020 fixes the problem for me. > This IMO may have side effect that user may really want to delete the > default route while not providing an explicit default route. I think for that > case user should disable accepting RA on the interface ( aka > ifconfig em0 inet6 no_radr, or turn on net.inet6.ip6.no_radr globally). That sounds perfectly reasonable. > How about this proposal ? > > > Best regards, > Zhenlei > From nobody Thu Jul 18 15:48:22 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPxzq2j6Sz5Rd4k for ; Thu, 18 Jul 2024 15:48:39 +0000 (UTC) (envelope-from sm@codenetworks.net) Received: from relayout03-q01.dominioabsoluto.net (relayout03-q01.dominioabsoluto.net [217.116.26.243]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPxzn6BkFz4Hl2 for ; Thu, 18 Jul 2024 15:48:37 +0000 (UTC) (envelope-from sm@codenetworks.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=codenetworks.net header.s=domabs header.b=WtdE6VXN; dmarc=none; spf=pass (mx1.freebsd.org: domain of sm@codenetworks.net designates 217.116.26.243 as permitted sender) smtp.mailfrom=sm@codenetworks.net Received: from relayout03-redir.dominioabsoluto.net (relayout03-redir.dominioabsoluto.net [217.116.26.247]) by relayout03.dominioabsoluto.net (Postfix) with ESMTP id 4WPxzj29SSz1xs0 for ; Thu, 18 Jul 2024 17:48:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=codenetworks.net; s=domabs; t=1721317713; bh=zDyIFpEzAJ/HeOBvXpGxSQjAyju1+oPdSF9UXLfV824=; h=From:Subject:Date:References:In-Reply-To:To:From; b=WtdE6VXNPz5/3YUDd1Z7ri6KZorOxZCwIs0x/id7xnI5he71MDcEkzpD+Gc567VwW wglSR6g9jRAJKJpJGo+DGUFQQ4oQRdeORO+f+HD16OsFFtd2QLe1tCJO0OHB0r2zH6 bDTA4b2ov3K8ak8nOKjM2U2RkmdqVfLd7H2RT7QI= Received: from smtpclient.apple (unknown [188.241.98.123]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: sm.codenetworks.net) by relayout03-dsp.dominioabsoluto.net (Postfix) with ESMTPSA id 4WPxzh6XkRz1xs0 for ; Thu, 18 Jul 2024 17:48:32 +0200 (CEST) Content-Type: multipart/alternative; boundary=Apple-Mail-2F3EF50F-171F-4DA5-8688-F58A1AB937F5 Content-Transfer-Encoding: 7bit From: Santiago Martinez List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (1.0) Subject: Re: Multiple Fibs and INET6 Date: Thu, 18 Jul 2024 17:48:22 +0200 Message-Id: References: In-Reply-To: To: freebsd-net@freebsd.org X-Mailer: iPhone Mail (21F90) X-PostalOut-Country: IP: 188.241.98.123 | Country: ES X-PostalOut-Information: AntiSPAM and AntiVIRUS on relayout03 X-PostalOut-MsgID: 4WPxzh6XkRz1xs0.AE241 X-PostalOut-SpamCheck: no es spam, clean X-PostalOut-From: sm@codenetworks.net X-PostalOut-Watermark: 1721922513.15001@jbprNJrfWyTKoSgsgZBQPg X-Spam-Status: No X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.79 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.992]; R_SPF_ALLOW(-0.20)[+ip4:217.116.26.0/24]; RCVD_IN_DNSWL_LOW(-0.20)[217.116.26.243:from,217.116.26.247:received]; R_DKIM_ALLOW(-0.20)[codenetworks.net:s=domabs]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RWL_MAILSPIKE_GOOD(-0.10)[217.116.26.243:from]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[codenetworks.net:+]; DMARC_NA(0.00)[codenetworks.net]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:16371, ipnet:217.116.24.0/21, country:ES]; ARC_NA(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; APPLE_IOS_MAILER_COMMON(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~] X-Rspamd-Queue-Id: 4WPxzn6BkFz4Hl2 --Apple-Mail-2F3EF50F-171F-4DA5-8688-F58A1AB937F5 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi everyone, Did anyone had the chance to take a look? For me it=E2=80=99s a bug but before filling the PR want to know what=E2=80=99= s your view or if it=E2=80=99s a limitation or bug by design. Br Santi > On 12 Jul 2024, at 19:06, Santiago Martinez wrote: >=20 > =EF=BB=BF > Hi Everyone. >=20 > While adding -F ( fib as used in netstat ) to ping and ping6 I have found s= omething that from my understanding is not correct. > Please can you advise? > I have the following setup : >=20 > -- two fibs (0 and 1)=20 > -- two loop-backs (lo0 and lo1). > -- Lo1 has been assigned to fib1 > -- net.add_addr_allfibs =3D 0 > My interface output looks like this: >=20 > ifconfig lo0 | grep inet6 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >=20 > ifconfig lo1 | grep inet6 > inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3 >=20 >=20 > If I do a netstat -rn -6 -F0 I get the following which is was i expected.= >=20 > Internet6: > Destination Gateway Flags N= etif Expire > ::/96 link#2 URS = lo0 > ::1 link#2 UHS = lo0 > ::ffff:0.0.0.0/96 link#2 URS = lo0 > fe80::%lo0/10 link#2 URS = lo0 > fe80::%lo0/64 link#2 U = lo0 > fe80::1%lo0 link#2 UHS = lo0 > ff02::/16 link#2 URS = lo0 >=20 >=20 > Now, netstat -rn -6 -F1 shows "fe80::1%lo0" which should not be there a= nd "fe80::1%lo1" is missing which should be there. > Internet6: > Destination Gateway Flags N= etif Expire > fe80::%lo1/64 link#3 U = lo1 > fe80::1%lo0 link#2 UHS = lo0 >=20 >=20 > What output I was expecting was: > Internet6: > Destination Gateway Flags N= etif Expire > fe80::%lo1/64 link#3 U = lo1 > fe80::1%lo1 link#3 UHS = lo1 >=20 >=20 >=20 > This makes the ping -6 -F0 fe80::1%lo0 to work but ping -6 -F1 fe80::1%l0= 1 to fail which I wanted to use as test case. >=20 > Thanks in advance. >=20 > Santiago >=20 --Apple-Mail-2F3EF50F-171F-4DA5-8688-F58A1AB937F5 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Hi e= veryone,
Did anyone had the chance to take a look?
For me it=E2=80=99s a bug but before filling the PR want t= o know what=E2=80=99s your view or if it=E2=80=99s a limitation or bug by de= sign.
Br
Santi


On 12 Jul 2024= , at 19:06, Santiago Martinez <sm@codenetworks.net> wrote:

=EF=BB=BF =20 =20 =20

Hi Everyone.

While adding -F ( fib as used in netstat ) to ping and ping6 I have found something that from my understanding is not correct.
Please can you advise?

I have the following setup :

-- two fibs (0 and 1) 
-- two  loop-backs (lo0 and lo1).
-- Lo1 has been assigned to fib1
-- net.add_addr_allfibs =3D 0

My interface output looks like this:

ifconfig lo0 | grep inet6
       inet6 ::1 prefixlen 128
       inet6 fe80::1%lo0 prefixle= n 64 scopeid 0x2

ifconfig lo1 | grep inet6
       inet6 fe80::1%lo1 prefixle= n 64 scopeid 0x3

If I do a netstat -rn -6  -F0 I get the following which is was i= expected.

Internet6:
Destination          &n= bsp;            = Gateway            &n= bsp;          Flags  =    Netif Expire
::/96           &n= bsp;            =      link#2            &n= bsp;           URS &n= bsp;       lo0
::1           &nbs= p;            &n= bsp;      link#2            &n= bsp;           UHS &n= bsp;       lo0
::ffff:0.0.0.0/96         &n= bsp;       link#2            &n= bsp;           URS &n= bsp;       lo0
fe80::%lo0/10          =            link#2            &n= bsp;           URS &n= bsp;       lo0
fe80::%lo0/64          =            link#2            &n= bsp;           U &nbs= p;         lo0
fe80::1%lo0          &n= bsp;            = link#2            &n= bsp;           UHS &n= bsp;       lo0
ff02::/16          &nbs= p;            &n= bsp; link#2            &n= bsp;           URS &n= bsp;       lo0

Now,  netstat -rn -6  -F1 shows  "fe80::1%lo0" which shou= ld not be there and "fe80::1%lo1" is missing which should be there.

Internet6:
Destination          &n= bsp;            = Gateway            &n= bsp;          Flags  =    Netif Expire
fe80::%lo1/64          =            link#3            &n= bsp;           U &nbs= p;         lo1
fe80::1%lo0          = ;            &nb= sp;link#2            =             UHS &= nbsp;       lo0

What output I was expecting was:

Internet6:
Destination          &n= bsp;            = Gateway            &n= bsp;          Flags  =    Netif Expire
fe80::%lo1/64          =            link#3            &n= bsp;           U &nbs= p;         lo1
fe80::1%lo1          = ;             li= nk#3            =             UHS  = ;       lo1


This makes the ping -6 -F0 fe80::1%lo0  to work but ping -6 -F1 fe80::1%l01 to fail which I wanted to use as test case.

Thanks in advance.

Santiago


=20
= --Apple-Mail-2F3EF50F-171F-4DA5-8688-F58A1AB937F5-- From nobody Thu Jul 18 16:09:51 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPyST0pdVz5Rg0H for ; Thu, 18 Jul 2024 16:10:01 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPySS6QRBz4L0s; Thu, 18 Jul 2024 16:10:00 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721319000; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=gaUpuF8VhO2j/qXyMr1ljDgiRMxme/67Tb2OrtuABkE=; b=HTaTE9TnaN8bSXMGjBDAf3xRZk554Gna8fggXo4Hw+q3PV0ZU07UzPCYnhvP/0Jai4L1oo rEkT1TeiflbuO8S0dMEwtKYmyJwDRxCG0/Y9QuQJZfUBk53Bwnj0kpMGYzLKXBFqjY4bKh RlD7+GAGD8JjcFEiBj0D6hmC+IELLURBwO5g1EWhoqww1x/t9xXTRNB30/OJVhDFU4QP5V Al7JorjGMxSZy1TX7VTNkKqH4Th3cKsCDGU+wBBKlxgz92tJd0ewyUcMJTO6O6IrbwrueF gBAHMxPWmr3NhY6PJhUFHo0lGgBK4uhsHnYG555Dt2CdTzuDW1p1fdX5worJfA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721319000; a=rsa-sha256; cv=none; b=MJK51Vam15GWCmd2v1xUzCDZxL18fwvz2bQz462vqPN/TCfbt77Dm+iOntg/3w1bnnZsmt fDgdweF6S+bAdAEqwy1az7dlP3e71hTYoovXAleFZwP2ZHYWPuEuGQuNO7zEQ4wYvJfykQ BAmoqfad4KD+C//KaVvjCNkFsIrN5nAO8ER12/mAjpN+c2xuMmThb9ikCf+EJZc2dMPq1J CGA7ybspfuOX6UG/Bcn69ORZ4f59AaoccO1ZZ4YbjyL+VryT/6g248gtmfrbkUvM7J42ln uoYaIjiue08ltQNH0tE6c0D5EOyK7H+9gOp3SdqP8yNTH2rzCfqsz5Uub/Y3tA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721319000; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=gaUpuF8VhO2j/qXyMr1ljDgiRMxme/67Tb2OrtuABkE=; b=Vov/WOSirBtpZJMIY1J70QjYIoIIzD7y+T7rbvDXSO7aqU25vUKi11a7pw50i5Yjt8yVR4 6K+cBt36n/KxDrI9mEUkvixeJeeWXLldQDf5nxJywjU1uYTuph41EL+KJSLcQYbTsOj7md XJJAmi1mw5x8RftMJLAMBcOSXA8U4TZS84jY1xXMt6o8KN7mPPwWatqUus9t8Boc+OZY2C QeTZ2WXQar1vuvnIu+c7VE/+ge/by1dMgJfeRXT19e0OKbFGvQEyeV3T78JQJUot/CgefO AmWB1vgaysbTarCqJcskEqvk69M95FNVcQhOKv7GC6SWKZVfkOCsNS64unT46A== Received: from smtpclient.apple (unknown [IPv6:2001:19f0:6001:9db:98f0:9fe0:3545:10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4WPySR5lKPzdNr; Thu, 18 Jul 2024 16:09:59 +0000 (UTC) (envelope-from zlei@FreeBSD.org) From: Zhenlei Huang Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_D218F227-362B-461E-B658-BA5165D0EFE0" List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.8\)) Subject: Re: Multiple Fibs and INET6 Date: Fri, 19 Jul 2024 00:09:51 +0800 In-Reply-To: Cc: "freebsd-net@freebsd.org" To: Santiago Martinez References: X-Mailer: Apple Mail (2.3696.120.41.1.8) --Apple-Mail=_D218F227-362B-461E-B658-BA5165D0EFE0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Jul 13, 2024, at 1:06 AM, Santiago Martinez = wrote: >=20 > Hi Everyone. >=20 > While adding -F ( fib as used in netstat ) to ping and ping6 I have = found something that from my understanding is not correct. > Please can you advise? > I have the following setup : >=20 > -- two fibs (0 and 1)=20 > -- two loop-backs (lo0 and lo1). > -- Lo1 has been assigned to fib1 > -- net.add_addr_allfibs =3D 0 > My interface output looks like this: >=20 >=20 > ifconfig lo0 | grep inet6=20 > inet6 ::1 prefixlen 128=20 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2=20 >=20 > ifconfig lo1 | grep inet6=20 > inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3 >=20 >=20 > If I do a netstat -rn -6 -F0 I get the following which is was i = expected. >=20 > Internet6:=20 > Destination Gateway Flags = Netif Expire=20 > ::/96 link#2 URS = lo0=20 > ::1 link#2 UHS = lo0=20 > ::ffff:0.0.0.0/96 link#2 URS = lo0=20 > fe80::%lo0/10 link#2 URS = lo0=20 > fe80::%lo0/64 link#2 U = lo0=20 > fe80::1%lo0 link#2 UHS = lo0=20 > ff02::/16 link#2 URS = lo0 >=20 >=20 > Now, netstat -rn -6 -F1 shows "fe80::1%lo0" which should not be = there and "fe80::1%lo1" is missing which should be there. >=20 > Internet6:=20 > Destination Gateway Flags = Netif Expire=20 > fe80::%lo1/64 link#3 U = lo1=20 > fe80::1%lo0 link#2 UHS = lo0 >=20 That seems wrong from my first glance. IIRC, there's HACK ( I'd prefer = this ) for loopback route. For example ``` # sysctl net.fibs=3D3 net.fibs: 2 -> 3 # ifconfig epair create # epair0a # ifconfig epair0a fib 2 # ifconfig epair0a inet6 -ifdisabled up # netstat -6rnF 2 Routing tables (fib: 2) Internet6: Destination Gateway Flags = Netif Expire fe80::%epair0a/64 link#5 U = epair0a fe80::3b:b3ff:fe8f:9a0a%lo0 link#1 UHS = lo0 ``` The loopback route always refer the first loop interface, aka lo0. =20 >=20 >=20 > What output I was expecting was: > Internet6:=20 > Destination Gateway Flags = Netif Expire=20 > fe80::%lo1/64 link#3 U = lo1=20 > fe80::1%lo1 link#3 UHS = lo1 >=20 >=20 >=20 > This makes the ping -6 -F0 fe80::1%lo0 to work but ping -6 -F1 = fe80::1%l01 to fail which I wanted to use as test case. >=20 That is interesting. I can ping without failure. ``` # setfib 1 ping6 -c3 fe80::1%lo1 PING(56=3D40+8+8 bytes) fe80::1%lo1 --> fe80::1%lo1 16 bytes from fe80::1%lo1, icmp_seq=3D0 hlim=3D64 time=3D0.050 ms 16 bytes from fe80::1%lo1, icmp_seq=3D1 hlim=3D64 time=3D0.067 ms 16 bytes from fe80::1%lo1, icmp_seq=3D2 hlim=3D64 time=3D0.096 ms --- fe80::1%lo1 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev =3D 0.050/0.071/0.096/0.019 ms ``` Best regards, Zhenlei > Thanks in advance. >=20 > Santiago >=20 >=20 --Apple-Mail=_D218F227-362B-461E-B658-BA5165D0EFE0 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

On Jul 13, 2024, at 1:06 AM, Santiago Martinez <sm@codenetworks.net>= wrote:

=20 =20

Hi Everyone.

While adding -F ( fib as used in netstat ) to ping and ping6 I have found something that from my understanding is not correct.
Please can you advise?

I have the = following setup :

-- two fibs (0 and 1) 
-- two  loop-backs (lo0 and lo1).
-- Lo1 has been assigned to fib1
-- net.add_addr_allfibs =3D 0

My interface output looks like this:


ifconfig lo0 = | grep inet6
       inet6 ::1 prefixlen = 128
       inet6 fe80::1%lo0 = prefixlen 64 scopeid 0x2

ifconfig lo1 | grep inet6
       inet6 fe80::1%lo1 = prefixlen 64 scopeid 0x3

If I do a netstat -rn -6  -F0 I get = the following which is was i expected.

Internet6:
Destination =             &n= bsp;         Gateway =             &n= bsp;         Flags =     Netif Expire
::/96 =             &n= bsp;           &nbs= p;   link#2 =             &n= bsp;          URS =         lo0
::1 =             &n= bsp;           &nbs= p;     link#2 =             &n= bsp;          UHS =         lo0
::ffff:0.0.0.0/96 =             &n= bsp;   link#2 =             &n= bsp;          URS =         lo0
fe80::%lo0/10 =             &n= bsp;       link#2 =             &n= bsp;          URS =         lo0
fe80::%lo0/64 =             &n= bsp;       link#2 =             &n= bsp;          U =           lo0
fe80::1%lo0 =             &n= bsp;         link#2 =             &n= bsp;          UHS =         lo0
ff02::/16 =             &n= bsp;           link= #2 =             &n= bsp;          URS =         lo0

Now,  netstat -rn -6  -F1 shows  "fe80::1%lo0" which = should not be there and "fe80::1%lo1" is missing which should be there.

Internet6:
Destination =             &n= bsp;         Gateway =             &n= bsp;         Flags =     Netif Expire
fe80::%lo1/64 =             &n= bsp;       link#3 =             &n= bsp;          U =           lo1
fe80::1%lo0 =             &n= bsp;         link#2 =             &n= bsp;          UHS =         lo0

That seems wrong from my = first glance. IIRC, there's HACK ( I'd prefer this ) for loopback route. = For example
```
# sysctl = net.fibs=3D3
net.fibs: 2 -> 3
# ifconfig epair = create
# epair0a
# = ifconfig epair0a fib 2
# = ifconfig epair0a inet6 -ifdisabled = up
# netstat -6rnF 2
Routing tables (fib: 2)

Internet6:
Destination               =         Gateway           =             Flags     Netif = Expire
fe80::%epair0a/64   =               link#5     =                    U =       epair0a
fe80::3b:b3ff:fe8f:9a0a%lo0       link#1 =                     =    UHS         = lo0
```

The loopback route always refer the first loop = interface, aka lo0.  


What output I was expecting was:

Internet6:
Destination =             &n= bsp;         Gateway =             &n= bsp;         Flags =     Netif Expire
fe80::%lo1/64 =             &n= bsp;       link#3 =             &n= bsp;          U =           lo1
fe80::1%lo1 =             &n= bsp;         link#3 =             &n= bsp;          UHS =         lo1


This makes the ping -6 -F0 fe80::1%lo0  to = work but ping -6 -F1 fe80::1%l01 to fail which I wanted to use as test case.

That is interesting. = I can ping without failure.

```
# setfib 1 ping6 -c3 = fe80::1%lo1
PING(56=3D40+8+8 bytes) fe80::1%lo1 --> = fe80::1%lo1
16 bytes from fe80::1%lo1, icmp_seq=3D0 hlim=3D64 = time=3D0.050 ms
16 bytes from fe80::1%lo1, icmp_seq=3D1 = hlim=3D64 time=3D0.067 ms
16 bytes from fe80::1%lo1, = icmp_seq=3D2 hlim=3D64 time=3D0.096 ms

--- fe80::1%lo1 ping statistics ---
3 = packets transmitted, 3 packets received, 0.0% packet = loss
round-trip min/avg/max/stddev =3D 0.050/0.071/0.096/0.019 = ms
```

Best = regards,
Zhenlei

Thanks in advance.

Santiago





= --Apple-Mail=_D218F227-362B-461E-B658-BA5165D0EFE0-- From nobody Thu Jul 18 16:11:53 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPyVn02hvz5RgMx for ; Thu, 18 Jul 2024 16:12:01 +0000 (UTC) (envelope-from sm@codenetworks.net) Received: from relayout02-q04.dominioabsoluto.net (relayout02-q04.dominioabsoluto.net [217.116.26.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPyVm0R7hz4Lt7 for ; Thu, 18 Jul 2024 16:12:00 +0000 (UTC) (envelope-from sm@codenetworks.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=codenetworks.net header.s=domabs header.b=pXqqU+Is; dmarc=none; spf=pass (mx1.freebsd.org: domain of sm@codenetworks.net designates 217.116.26.80 as permitted sender) smtp.mailfrom=sm@codenetworks.net Received: from relayout02-redir.dominioabsoluto.net (relayout02-redir.dominioabsoluto.net [217.116.26.75]) by relayout02.dominioabsoluto.net (Postfix) with ESMTP id 4WPyVj32YKzlWZq for ; Thu, 18 Jul 2024 18:11:57 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=codenetworks.net; s=domabs; t=1721319117; bh=ruWW9hnNRknvq+pZJnsSQdbbluYOzW3p8tTmKjyRFag=; h=Date:Subject:To:References:From:In-Reply-To:From; b=pXqqU+IszRz0DSGQuxWnu+5y+dn2JuJVTUCJUldAz6FAKcwmYbSfME739ue8j2KUq ObFrVAKmqoQ6uXfPQr53mx5N8MmiN4HR5qLNP4/Ia7d4xewh+mYrxTwuG3DeskxEn6 Dq/bwfgqfEx20PDagC+u/jBhywzRLKEpSV0Vy48I= Received: from [192.168.3.20] (unknown [188.241.98.123]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sm.codenetworks.net) by relayout02-dsp.dominioabsoluto.net (Postfix) with ESMTPSA id 4WPyVd2VdczlWBJ for ; Thu, 18 Jul 2024 18:11:53 +0200 (CEST) Content-Type: multipart/alternative; boundary="------------KSf0CWNK2xTv00FE0ZzP8ZWu" Message-ID: <70305bf3-cfe4-4cdd-8cb8-c03c1a9c2b8f@codenetworks.net> Date: Thu, 18 Jul 2024 18:11:53 +0200 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Multiple Fibs and INET6 To: freebsd-net@freebsd.org References: Content-Language: en-US From: Santiago Martinez In-Reply-To: X-PostalOut-Country: IP: 188.241.98.123 | Country: ES X-PostalOut-Information: AntiSPAM and AntiVIRUS on relayout02 X-PostalOut-MsgID: 4WPyVd2VdczlWBJ.AC77B X-PostalOut-SpamCheck: no es spam, clean X-PostalOut-From: sm@codenetworks.net X-PostalOut-Watermark: 1721923916.6966@ey1sAjYEmlW5wXJaio+nTw X-Spam-Status: No X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.77 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.975]; RCVD_IN_DNSWL_LOW(-0.20)[217.116.26.80:from,217.116.26.75:received]; R_SPF_ALLOW(-0.20)[+ip4:217.116.26.0/24:c]; R_DKIM_ALLOW(-0.20)[codenetworks.net:s=domabs]; RWL_MAILSPIKE_GOOD(-0.10)[217.116.26.80:from]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:16371, ipnet:217.116.24.0/21, country:ES]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DMARC_NA(0.00)[codenetworks.net]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[codenetworks.net:+] X-Rspamd-Queue-Id: 4WPyVm0R7hz4Lt7 This is a multi-part message in MIME format. --------------KSf0CWNK2xTv00FE0ZzP8ZWu Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Interesting, I'm running 14.1p2. how does your routing table looks for fib1 ? Santi On 7/18/24 18:09, Zhenlei Huang wrote: > > >> On Jul 13, 2024, at 1:06 AM, Santiago Martinez >> wrote: >> >> Hi Everyone. >> >> While adding -F ( fib as used in netstat ) to ping and ping6 I have >> found something that from my understanding is not correct. >> Please can you advise? >> >> I have the following setup : >> >> -- two fibs (0 and 1) >> -- two  loop-backs (lo0 and lo1). >> -- Lo1 has been assigned to fib1 >> --net.add_addr_allfibs = 0 >> >> My interface output looks like this: >> >> >> ifconfig lo0 | grep inet6 >>        inet6 ::1 prefixlen 128 >>        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >> >> ifconfig lo1 | grep inet6 >>        inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3 >> >> If I do a netstat -rn -6  -F0 I get the following which is was i >> expected. >> >> Internet6: >> Destination                       Gateway                       Flags >>     Netif Expire >> ::/96                             link#2                        URS >>         lo0 >> ::1                               link#2                        UHS >>         lo0 >> ::ffff:0.0.0.0/96                 link#2                        URS >>         lo0 >> fe80::%lo0/10                     link#2                        URS >>         lo0 >> fe80::%lo0/64                     link#2                        U >>           lo0 >> fe80::1%lo0                       link#2                        UHS >>         lo0 >> ff02::/16                         link#2                        URS >>         lo0 >> >> Now,  netstat -rn -6  -F1 shows  "fe80::1%lo0" which should not be >> there and "fe80::1%lo1" is missing which should be there. >> >> Internet6: >> Destination                       Gateway                       Flags >>     Netif Expire >> fe80::%lo1/64                     link#3                        U >>           lo1 >> *fe80::1%lo0                       link#2                        UHS >>         lo0* >> > That seems wrong from my first glance. IIRC, there's HACK ( I'd prefer > this ) for loopback route. For example > ``` > # sysctl net.fibs=3 > net.fibs: 2 -> 3 > # ifconfig epair create > # epair0a > # ifconfig epair0a fib 2 > # ifconfig epair0a inet6 -ifdisabled up > # netstat -6rnF 2 > Routing tables (fib: 2) > > Internet6: > Destination                     Gateway                       Flags   > Netif Expire > fe80::%epair0a/64                 link#5                        U epair0a > fe80::3b:b3ff:fe8f:9a0a%lo0       link#1                        UHS   >       lo0 > ``` > > The loopback route always refer the first loop interface, aka lo0. >> >> >> What output I was expecting was: >> >> Internet6: >> Destination                       Gateway                       Flags >>     Netif Expire >> fe80::%lo1/64                     link#3                        U >>           lo1 >> *fe80::1%lo1                       link#3                        UHS >>         lo1* >> >> >> This makes the ping -6 -F0 fe80::1%lo0  to work but ping -6 -F1 >> fe80::1%l01 to fail which I wanted to use as test case. >> > That is interesting. I can ping without failure. > > ``` > # setfib 1 ping6 -c3 fe80::1%lo1 > PING(56=40+8+8 bytes) fe80::1%lo1 --> fe80::1%lo1 > 16 bytes from fe80::1%lo1, icmp_seq=0 hlim=64 time=0.050 ms > 16 bytes from fe80::1%lo1, icmp_seq=1 hlim=64 time=0.067 ms > 16 bytes from fe80::1%lo1, icmp_seq=2 hlim=64 time=0.096 ms > > --- fe80::1%lo1 ping statistics --- > 3 packets transmitted, 3 packets received, 0.0% packet loss > round-trip min/avg/max/stddev = 0.050/0.071/0.096/0.019 ms > ``` > > Best regards, > Zhenlei > >> Thanks in advance. >> >> Santiago >> >> > > > --------------KSf0CWNK2xTv00FE0ZzP8ZWu Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

Interesting, I'm running 14.1p2.

how does your routing table looks for fib1 ?

Santi


On 7/18/24 18:09, Zhenlei Huang wrote:


On Jul 13, 2024, at 1:06 AM, Santiago Martinez <sm@codenetworks.net> wrote:

Hi Everyone.

While adding -F ( fib as used in netstat ) to ping and ping6 I have found something that from my understanding is not correct.
Please can you advise?

I have the following setup :

-- two fibs (0 and 1) 
-- two  loop-backs (lo0 and lo1).
-- Lo1 has been assigned to fib1
-- net.add_addr_allfibs = 0

My interface output looks like this:


ifconfig lo0 | grep inet6
       inet6 ::1 prefixlen 128
       inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2

ifconfig lo1 | grep inet6
       inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3

If I do a netstat -rn -6  -F0 I get the following which is was i expected.

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             link#2                        URS         lo0
::1                               link#2                        UHS         lo0
::ffff:0.0.0.0/96                 link#2                        URS         lo0
fe80::%lo0/10                     link#2                        URS         lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16                         link#2                        URS         lo0

Now,  netstat -rn -6  -F1 shows  "fe80::1%lo0" which should not be there and "fe80::1%lo1" is missing which should be there.

Internet6:
Destination                       Gateway                       Flags     Netif Expire
fe80::%lo1/64                     link#3                        U           lo1
fe80::1%lo0                       link#2                        UHS         lo0

That seems wrong from my first glance. IIRC, there's HACK ( I'd prefer this ) for loopback route. For example
```
# sysctl net.fibs=3
net.fibs: 2 -> 3
# ifconfig epair create
# epair0a
# ifconfig epair0a fib 2
# ifconfig epair0a inet6 -ifdisabled up
# netstat -6rnF 2
Routing tables (fib: 2)

Internet6:
Destination                       Gateway                       Flags     Netif Expire
fe80::%epair0a/64                 link#5                        U       epair0a
fe80::3b:b3ff:fe8f:9a0a%lo0       link#1                        UHS         lo0
```

The loopback route always refer the first loop interface, aka lo0.  


What output I was expecting was:

Internet6:
Destination                       Gateway                       Flags     Netif Expire
fe80::%lo1/64                     link#3                        U           lo1
fe80::1%lo1                       link#3                        UHS         lo1


This makes the ping -6 -F0 fe80::1%lo0  to work but ping -6 -F1 fe80::1%l01 to fail which I wanted to use as test case.

That is interesting. I can ping without failure.

```
# setfib 1 ping6 -c3 fe80::1%lo1
PING(56=40+8+8 bytes) fe80::1%lo1 --> fe80::1%lo1
16 bytes from fe80::1%lo1, icmp_seq=0 hlim=64 time=0.050 ms
16 bytes from fe80::1%lo1, icmp_seq=1 hlim=64 time=0.067 ms
16 bytes from fe80::1%lo1, icmp_seq=2 hlim=64 time=0.096 ms

--- fe80::1%lo1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.050/0.071/0.096/0.019 ms
```

Best regards,
Zhenlei

Thanks in advance.

Santiago





--------------KSf0CWNK2xTv00FE0ZzP8ZWu-- From nobody Thu Jul 18 16:15:12 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPyZb6dG4z5RgL5 for ; Thu, 18 Jul 2024 16:15:19 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPyZb67Q2z4MJN; Thu, 18 Jul 2024 16:15:19 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721319319; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=sNjm+Rjv/6mnmd39H4MZC4WkZts2D959+JqVxhl9ki0=; b=AJJ/YeD11jopG1AWnE57iLOAEgznwuoyZwYSgBJLYpuowzrlJtFPLoKdgWX/BNO5z9N1Lm NgtkFv+QvDSDviOgPUz3i3S0y/UHr3CIsHrsGZFbnAuirnOApCZlpkRce4YO/pYMVWlb9/ c9Z038YMo5lAN0jA+C4waymB0HfzP4FtxTtTa5tZtXdhDHaLdspYgy74FQph3kzXz6fstU e7XM+Mhcf3ttA1eoaT96xC7ueMvRw+cEMPoiH7bnR3S6dVam277qwG7xXJman7ogLh5yQY YuAxJJ3kHd9iaLn/WTiceyG8ylRCSJkWWKUdVaaZxpedZWSfM/qVbgHq3J6wvg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721319319; a=rsa-sha256; cv=none; b=WJ4a0P/GbyuGSo4sj9EY6JMUeucEJ44r3QiYO8oqR2U2eWhwQFc9p3Q/4AkgMbbubdxLxC uIBs5QITcaiylBQbndShH2+S4wTmP/xMMhKcj5/7Ft0YCoUNSaIG1KMqA5KBit6HnFGi0d sglY8B111S9z7K0XZrEgBDjDRYvt0KUfDne9iqFMbwWT/ke9B+S7I9TF05gxuvo4Qm6isx qiEF7BBOakOm0Fkj01+/4wkPdPR06v7jq1foluH87exbJbnJiQ9+655QV9YkONRzsPksBR CrO4hp/JChSTxBwdzeTGyjSFVNNk+n0wlf3zqt4lTaxT/8gnT8l2XoGEV8TCRg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721319319; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=sNjm+Rjv/6mnmd39H4MZC4WkZts2D959+JqVxhl9ki0=; b=S+rwKQhtAQBp9OrTVizr7DXmJSCQsseeXVKtToqrWxFwj4phXxkpVYfmq2wXBjlMIgE0Gj 0jrlj6X/ZchH10Js1mSROnriEe+vVus09i4X9httQa7TEBl/GN0KcA432sTS10ysBp92GO t7q2yYyLj2iEOQ1jL8Lm+IrKYoEBz3WBwQAHjzs35ydLnmTJCzNXCOOUHeGEawy3qfXGgU 0ii/0rYvE7fsfm94lMVpxr46l3lHviMru+tcYGIS5EFk3QuO6DsG3N8OWWpVLpa8M+X/xN Vv2UGKJo+Zco3wnRj72xeam237ne27cuqX3A4W0J+4vjiMnggPvtJWL4TxrQIQ== Received: from smtpclient.apple (ns1.oxydns.net [45.32.91.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4WPyZZ5KXWzd9l; Thu, 18 Jul 2024 16:15:18 +0000 (UTC) (envelope-from zlei@FreeBSD.org) From: Zhenlei Huang Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_C338E674-F01E-4A6C-8A76-67DBE390042A" List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.8\)) Subject: Re: Multiple Fibs and INET6 Date: Fri, 19 Jul 2024 00:15:12 +0800 In-Reply-To: <70305bf3-cfe4-4cdd-8cb8-c03c1a9c2b8f@codenetworks.net> Cc: freebsd-net@freebsd.org To: Santiago Martinez References: <70305bf3-cfe4-4cdd-8cb8-c03c1a9c2b8f@codenetworks.net> X-Mailer: Apple Mail (2.3696.120.41.1.8) --Apple-Mail=_C338E674-F01E-4A6C-8A76-67DBE390042A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Jul 19, 2024, at 12:11 AM, Santiago Martinez = wrote: >=20 > Interesting, I'm running 14.1p2. >=20 >=20 Yes, I'm running exactly the same version with you. > how does your routing table looks for fib1 ? >=20 >=20 ``` # netstat -6rnF 1 Routing tables (fib: 1) Internet6: Destination Gateway Flags = Netif Expire fe80::%lo1/64 link#5 U = lo1 fe80::1%lo0 link#2 UHS = lo0 ``` > Santi >=20 >=20 >=20 > On 7/18/24 18:09, Zhenlei Huang wrote: >>=20 >>=20 >>> On Jul 13, 2024, at 1:06 AM, Santiago Martinez > wrote: >>>=20 >>> Hi Everyone. >>>=20 >>> While adding -F ( fib as used in netstat ) to ping and ping6 I have = found something that from my understanding is not correct. >>> Please can you advise? >>> I have the following setup : >>>=20 >>> -- two fibs (0 and 1)=20 >>> -- two loop-backs (lo0 and lo1). >>> -- Lo1 has been assigned to fib1 >>> -- net.add_addr_allfibs =3D 0 >>> My interface output looks like this: >>>=20 >>>=20 >>> ifconfig lo0 | grep inet6=20 >>> inet6 ::1 prefixlen 128=20 >>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2=20 >>>=20 >>> ifconfig lo1 | grep inet6=20 >>> inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3 >>>=20 >>>=20 >>> If I do a netstat -rn -6 -F0 I get the following which is was i = expected. >>>=20 >>> Internet6:=20 >>> Destination Gateway = Flags Netif Expire=20 >>> ::/96 link#2 URS = lo0=20 >>> ::1 link#2 UHS = lo0=20 >>> ::ffff:0.0.0.0/96 link#2 URS = lo0=20 >>> fe80::%lo0/10 link#2 URS = lo0=20 >>> fe80::%lo0/64 link#2 U = lo0=20 >>> fe80::1%lo0 link#2 UHS = lo0=20 >>> ff02::/16 link#2 URS = lo0 >>>=20 >>>=20 >>> Now, netstat -rn -6 -F1 shows "fe80::1%lo0" which should not be = there and "fe80::1%lo1" is missing which should be there. >>>=20 >>> Internet6:=20 >>> Destination Gateway = Flags Netif Expire=20 >>> fe80::%lo1/64 link#3 U = lo1=20 >>> fe80::1%lo0 link#2 UHS = lo0 >>>=20 >> That seems wrong from my first glance. IIRC, there's HACK ( I'd = prefer this ) for loopback route. For example >> ``` >> # sysctl net.fibs=3D3 >> net.fibs: 2 -> 3 >> # ifconfig epair create >> # epair0a >> # ifconfig epair0a fib 2 >> # ifconfig epair0a inet6 -ifdisabled up >> # netstat -6rnF 2 >> Routing tables (fib: 2) >>=20 >> Internet6: >> Destination Gateway Flags = Netif Expire >> fe80::%epair0a/64 link#5 U = epair0a >> fe80::3b:b3ff:fe8f:9a0a%lo0 link#1 UHS = lo0 >> ``` >>=20 >> The loopback route always refer the first loop interface, aka lo0. =20= >>>=20 >>>=20 >>> What output I was expecting was: >>> Internet6:=20 >>> Destination Gateway = Flags Netif Expire=20 >>> fe80::%lo1/64 link#3 U = lo1=20 >>> fe80::1%lo1 link#3 UHS = lo1 >>>=20 >>>=20 >>>=20 >>> This makes the ping -6 -F0 fe80::1%lo0 to work but ping -6 -F1 = fe80::1%l01 to fail which I wanted to use as test case. >>>=20 >> That is interesting. I can ping without failure. >>=20 >> ``` >> # setfib 1 ping6 -c3 fe80::1%lo1 >> PING(56=3D40+8+8 bytes) fe80::1%lo1 --> fe80::1%lo1 >> 16 bytes from fe80::1%lo1, icmp_seq=3D0 hlim=3D64 time=3D0.050 ms >> 16 bytes from fe80::1%lo1, icmp_seq=3D1 hlim=3D64 time=3D0.067 ms >> 16 bytes from fe80::1%lo1, icmp_seq=3D2 hlim=3D64 time=3D0.096 ms >>=20 >> --- fe80::1%lo1 ping statistics --- >> 3 packets transmitted, 3 packets received, 0.0% packet loss >> round-trip min/avg/max/stddev =3D 0.050/0.071/0.096/0.019 ms >> ``` >>=20 >> Best regards, >> Zhenlei >>=20 >>>=20 >>> Thanks in advance. >>>=20 >>> Santiago >>>=20 >>>=20 >>=20 >>=20 >>=20 --Apple-Mail=_C338E674-F01E-4A6C-8A76-67DBE390042A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

On Jul 19, 2024, at 12:11 AM, Santiago Martinez <sm@codenetworks.net>= wrote:

=20 =20

Interesting, I'm running 14.1p2.



Yes, I'm running exactly the same version with you.

how does your routing table looks for fib1 = ?



```
# netstat -6rnF 1
Routing = tables (fib: 1)

Internet6:
Destination     =                   Gateway =                     =   Flags     Netif Expire
fe80::%lo1/64   =                   link#5 =                     =    U           = lo1
fe80::1%lo0             =           link#2         =                UHS     =     lo0
```

Santi


On 7/18/24 18:09, Zhenlei Huang = wrote:


On Jul 13, 2024, at 1:06 AM, Santiago Martinez <sm@codenetworks.net> wrote:

Hi Everyone.

While adding -F ( fib as used in netstat ) to ping and ping6 I have found something that from my understanding is not correct.
Please can you advise?

I have = the following setup :

-- two fibs (0 and 1) 
-- two  loop-backs (lo0 and lo1).
-- Lo1 has been assigned to fib1
-- net.add_addr_allfibs =3D 0

My interface output looks like this:


ifconfig lo0 | grep inet6
       inet6 ::1 = prefixlen 128
       inet6 = fe80::1%lo0 prefixlen 64 scopeid 0x2

ifconfig lo1 | grep inet6
       inet6 = fe80::1%lo1 prefixlen 64 scopeid 0x3

If I do a netstat -rn -6  = -F0 I get the following which is was i expected.

Internet6:
Destination =             &n= bsp;         Gateway =             &n= bsp;         Flags =     Netif Expire
::/96 =             &n= bsp;           &nbs= p;   link#2 =             &n= bsp;          URS =         lo0
::1 =             &n= bsp;           &nbs= p;     link#2 =             &n= bsp;          UHS =         lo0
::ffff:0.0.0.0/96 =             &n= bsp;   link#2 =             &n= bsp;          URS =         lo0
fe80::%lo0/10 =             &n= bsp;       link#2 =             &n= bsp;          URS =         lo0
fe80::%lo0/64 =             &n= bsp;       link#2 =             &n= bsp;          U =           lo0
fe80::1%lo0 =             &n= bsp;         link#2 =             &n= bsp;          UHS =         lo0
ff02::/16 =             &n= bsp;           link= #2 =             &n= bsp;          URS =         lo0

Now,  netstat -rn -6  -F1 shows  = "fe80::1%lo0" which should not be there and "fe80::1%lo1" is missing which should be there.

Internet6:
Destination =             &n= bsp;         Gateway =             &n= bsp;         Flags =     Netif Expire
fe80::%lo1/64 =             &n= bsp;       link#3 =             &n= bsp;          U =           lo1
fe80::1%lo0 =             &n= bsp;         link#2 =             &n= bsp;          UHS =         lo0

That seems wrong from my first glance. IIRC, there's HACK ( I'd prefer this ) for loopback route. For example
```
# sysctl net.fibs=3D3
net.fibs: 2 -> 3
# ifconfig epair create
# epair0a
# ifconfig epair0a fib 2
# ifconfig epair0a inet6 -ifdisabled up
# netstat -6rnF 2
Routing tables (fib: 2)

Internet6:
Destination                   =     Gateway               =         Flags     Netif Expire
fe80::%epair0a/64                 = link#5                   =      U       epair0a
fe80::3b:b3ff:fe8f:9a0a%lo0       link#1         =                UHS     =     lo0
```

The loopback route always refer the first loop = interface, aka lo0.  


What output I was expecting was:

Internet6:
Destination =             &n= bsp;         Gateway =             &n= bsp;         Flags =     Netif Expire
fe80::%lo1/64 =             &n= bsp;       link#3 =             &n= bsp;          U =           lo1
fe80::1%lo1 =             &n= bsp;         link#3 =             &n= bsp;          UHS =         lo1


This makes the ping -6 -F0 = fe80::1%lo0  to work but ping -6 -F1 fe80::1%l01 to fail which I wanted to use as test case.

That is interesting. I can ping without = failure.

```
# setfib 1 ping6 -c3 fe80::1%lo1
PING(56=3D40+8+8 bytes) fe80::1%lo1 --> = fe80::1%lo1
16 bytes from fe80::1%lo1, icmp_seq=3D0 = hlim=3D64 time=3D0.050 ms
16 bytes from fe80::1%lo1, icmp_seq=3D1 = hlim=3D64 time=3D0.067 ms
16 bytes from fe80::1%lo1, icmp_seq=3D2 = hlim=3D64 time=3D0.096 ms

--- fe80::1%lo1 ping statistics ---
3 packets transmitted, 3 packets received, = 0.0% packet loss
round-trip min/avg/max/stddev =3D = 0.050/0.071/0.096/0.019 ms
```

Best regards,
Zhenlei


Thanks in advance.

Santiago








= --Apple-Mail=_C338E674-F01E-4A6C-8A76-67DBE390042A-- From nobody Thu Jul 18 16:34:12 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPz0W0TBcz5RhH2 for ; Thu, 18 Jul 2024 16:34:19 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (mail-n.franken.de [193.175.24.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPz0V1x0zz4Pl2; Thu, 18 Jul 2024 16:34:18 +0000 (UTC) (envelope-from tuexen@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=freebsd.org (policy=none); spf=softfail (mx1.freebsd.org: 193.175.24.27 is neither permitted nor denied by domain of tuexen@freebsd.org) smtp.mailfrom=tuexen@freebsd.org Received: from smtpclient.apple (unknown [IPv6:2a02:8109:1140:c3d:25ea:b605:4a77:64ed]) (Authenticated sender: macmic) by drew.franken.de (Postfix) with ESMTPSA id 04EF4721E2817; Thu, 18 Jul 2024 18:34:13 +0200 (CEST) Content-Type: text/plain; charset=utf-8 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.600.62\)) Subject: Re: TCP Success Story (was Re: TCP_RACK, TCP_BBR, and firewalls) From: tuexen@freebsd.org In-Reply-To: Date: Thu, 18 Jul 2024 18:34:12 +0200 Cc: Alan Somers , FreeBSD Net Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Junho Choi X-Mailer: Apple Mail (2.3774.600.62) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.09 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.996]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : No valid SPF, No valid DKIM,none]; RCVD_IN_DNSWL_LOW(-0.10)[193.175.24.27:from]; FROM_NO_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; RCVD_TLS_ALL(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; FREEFALL_USER(0.00)[tuexen]; ASN(0.00)[asn:680, ipnet:193.174.0.0/15, country:DE]; RCVD_VIA_SMTP_AUTH(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; ARC_NA(0.00)[]; R_DKIM_NA(0.00)[]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_RCPT(0.00)[]; MIME_TRACE(0.00)[0:+] X-Rspamd-Queue-Id: 4WPz0V1x0zz4Pl2 > On 18. Jul 2024, at 15:00, Junho Choi wrote: >=20 > Alan - this is a great result to see. Thanks for experimenting. >=20 > Just curious why bbr and rack don't co-exist? Those are two separate = things. > Is it a current bug or by design? Technically RACK and BBR can coexist. The problem was with pf and/or = LRO. But this is all fixed now in 14.1 and head. Best regards Michael >=20 > BR, >=20 > On Thu, Jul 18, 2024 at 5:27=E2=80=AFAM wrote: >> On 17. Jul 2024, at 22:00, Alan Somers wrote: >>=20 >> On Sat, Jul 13, 2024 at 1:50=E2=80=AFAM wrote: >>>=20 >>>> On 13. Jul 2024, at 01:43, Alan Somers wrote: >>>>=20 >>>> I've been experimenting with RACK and BBR. In my environment, they >>>> can dramatically improve single-stream TCP performance, which is >>>> awesome. But pf interferes. I have to disable pf in order for = them >>>> to work at all. >>>>=20 >>>> Is this a known limitation? If not, I will experiment some more to >>>> determine exactly what aspect of my pf configuration is = responsible. >>>> If so, can anybody suggest what changes would have to happen to = make >>>> the two compatible? >>> A problem with same symptoms was already reported and fixed in >>> https://reviews.freebsd.org/D43769 >>>=20 >>> Which version are you using? >>>=20 >>> Best regards >>> Michael >>>>=20 >>>> -Alan >>=20 >> TLDR; tcp_rack is good, cc_chd is better, and tcp_bbr is best >>=20 >> I want to follow up with the list to post my conclusions. Firstly >> tuexen@ helped me solve my problem: in FreeBSD 14.0 there is a 3-way >> incompatibility between (tcp_bbr || tcp_rack) && lro && pf. I can >> confirm that tcp_bbr works for me if I either disable LRO, disable = PF, >> or switch to a 14.1 server. >>=20 >> Here's the real problem: on multiple production servers, downloading >> large files (or ZFS send/recv streams) was slow. After ruling out >> many possible causes, wireshark revealed that the connection was >> suffering about 0.05% packet loss. I don't know the source of that >> packet loss, but I don't believe it to be congestion-related. Along >> with a 54ms RTT, that's a fatal combination for the throughput of >> loss-based congestion control algorithms. According to the Mathis >> Formula [1], I could only expect 1.1 MBps over such a connection. >> That's actually worse than what I saw. With default settings >> (cc_cubic), I averaged 5.6 MBps. Probably Mathis's assumptions are >> outdated, but that's still pretty close for such a simple formula >> that's 27 years old. >>=20 >> So I benchmarked all available congestion control algorithms for >> single download streams. The results are summarized in the table >> below. >>=20 >> Algo Packet Loss Rate Average Throughput >> vegas 0.05% 2.0 MBps >> newreno 0.05% 3.2 MBps >> cubic 0.05% 5.6 MBps >> hd 0.05% 8.6 MBps >> cdg 0.05% 13.5 MBps >> rack 0.04% 14 MBps >> htcp 0.05% 15 MBps >> dctcp 0.05% 15 MBps >> chd 0.05% 17.3 MBps >> bbr 0.05% 29.2 MBps >> cubic 10% 159 kBps >> chd 10% 208 kBps >> bbr 10% 5.7 MBps >>=20 >> RACK seemed to achieve about the same maximum bandwidth as BBR, = though >> it took a lot longer to get there. Also, with RACK, wireshark >> reported about 10x as many retransmissions as dropped packets, which >> is suspicious. >>=20 >> At one point, something went haywire and packet loss briefly spiked = to >> the neighborhood of 10%. I took advantage of the chaos to repeat my >> measurements. As the table shows, all algorithms sucked under those >> conditions, but BBR sucked impressively less than the others. >>=20 >> Disclaimer: there was significant run-to-run variation; the presented >> results are averages. And I did not attempt to measure packet loss >> exactly for most runs; 0.05% is merely an average of a few selected >> runs. These measurements were taken on a production server running a >> real workload, which introduces noise. Soon I hope to have the >> opportunity to repeat the experiment on an idle server in the same >> environment. >>=20 >> In conclusion, while we'd like to use BBR, we really can't until we >> upgrade to 14.1, which hopefully will be soon. So in the meantime >> we've switched all relevant servers from cubic to chd, and we'll >> reevaluate BBR after the upgrade. > Hi Alan, >=20 > just to be clear: the version of BBR currently implemented is > BBR version 1, which is known to be unfair in certain scenarios. > Google is still working on BBR to address this problem and improve > it in other aspects. But there is no RFC yet and the updates haven't > been implemented yet in FreeBSD. >=20 > Best regards > Michael >>=20 >> [1]: https://www.slac.stanford.edu/comp/net/wan-mon/thru-vs-loss.html >>=20 >> -Alan >=20 >=20 >=20 >=20 > --=20 > Junho Choi | https://saturnsoft.net From nobody Thu Jul 18 16:34:36 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPz142TXJz5Rhw6 for ; Thu, 18 Jul 2024 16:34:48 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPz1330jtz4QY9; Thu, 18 Jul 2024 16:34:47 +0000 (UTC) (envelope-from tuexen@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=freebsd.org (policy=none); spf=softfail (mx1.freebsd.org: 2001:638:a02:a001:20e:cff:fe4a:feaa is neither permitted nor denied by domain of tuexen@freebsd.org) smtp.mailfrom=tuexen@freebsd.org Received: from smtpclient.apple (unknown [IPv6:2a02:8109:1140:c3d:25ea:b605:4a77:64ed]) (Authenticated sender: micmac) by drew.franken.de (Postfix) with ESMTPSA id E4F5F721E2817; Thu, 18 Jul 2024 18:34:36 +0200 (CEST) Content-Type: text/plain; charset=utf-8 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.600.62\)) Subject: Re: TCP Success Story (was Re: TCP_RACK, TCP_BBR, and firewalls) From: tuexen@freebsd.org In-Reply-To: Date: Thu, 18 Jul 2024 18:34:36 +0200 Cc: Alan Somers , FreeBSD Net Content-Transfer-Encoding: quoted-printable Message-Id: <2CD5D69A-28FF-41E2-AF51-AFB24D4932A9@freebsd.org> References: To: Junho Choi X-Mailer: Apple Mail (2.3774.600.62) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.66 / 15.00]; NEURAL_HAM_SHORT(-0.99)[-0.995]; NEURAL_HAM_LONG(-0.99)[-0.993]; NEURAL_HAM_MEDIUM(-0.77)[-0.769]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : No valid SPF, No valid DKIM,none]; ONCE_RECEIVED(0.10)[]; FROM_NO_DN(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_TO(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; FREEFALL_USER(0.00)[tuexen]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; ASN(0.00)[asn:680, ipnet:2001:638::/32, country:DE]; R_DKIM_NA(0.00)[]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_RCPT(0.00)[]; MIME_TRACE(0.00)[0:+] X-Rspamd-Queue-Id: 4WPz1330jtz4QY9 > On 18. Jul 2024, at 15:00, Junho Choi wrote: >=20 > Alan - this is a great result to see. Thanks for experimenting. >=20 > Just curious why bbr and rack don't co-exist? Those are two separate = things. > Is it a current bug or by design? Technically RACK and BBR can coexist. The problem was with pf and/or = LRO. But this is all fixed now in 14.1 and head. Best regards Michael >=20 > BR, >=20 > On Thu, Jul 18, 2024 at 5:27=E2=80=AFAM wrote: >> On 17. Jul 2024, at 22:00, Alan Somers wrote: >>=20 >> On Sat, Jul 13, 2024 at 1:50=E2=80=AFAM wrote: >>>=20 >>>> On 13. Jul 2024, at 01:43, Alan Somers wrote: >>>>=20 >>>> I've been experimenting with RACK and BBR. In my environment, they >>>> can dramatically improve single-stream TCP performance, which is >>>> awesome. But pf interferes. I have to disable pf in order for = them >>>> to work at all. >>>>=20 >>>> Is this a known limitation? If not, I will experiment some more to >>>> determine exactly what aspect of my pf configuration is = responsible. >>>> If so, can anybody suggest what changes would have to happen to = make >>>> the two compatible? >>> A problem with same symptoms was already reported and fixed in >>> https://reviews.freebsd.org/D43769 >>>=20 >>> Which version are you using? >>>=20 >>> Best regards >>> Michael >>>>=20 >>>> -Alan >>=20 >> TLDR; tcp_rack is good, cc_chd is better, and tcp_bbr is best >>=20 >> I want to follow up with the list to post my conclusions. Firstly >> tuexen@ helped me solve my problem: in FreeBSD 14.0 there is a 3-way >> incompatibility between (tcp_bbr || tcp_rack) && lro && pf. I can >> confirm that tcp_bbr works for me if I either disable LRO, disable = PF, >> or switch to a 14.1 server. >>=20 >> Here's the real problem: on multiple production servers, downloading >> large files (or ZFS send/recv streams) was slow. After ruling out >> many possible causes, wireshark revealed that the connection was >> suffering about 0.05% packet loss. I don't know the source of that >> packet loss, but I don't believe it to be congestion-related. Along >> with a 54ms RTT, that's a fatal combination for the throughput of >> loss-based congestion control algorithms. According to the Mathis >> Formula [1], I could only expect 1.1 MBps over such a connection. >> That's actually worse than what I saw. With default settings >> (cc_cubic), I averaged 5.6 MBps. Probably Mathis's assumptions are >> outdated, but that's still pretty close for such a simple formula >> that's 27 years old. >>=20 >> So I benchmarked all available congestion control algorithms for >> single download streams. The results are summarized in the table >> below. >>=20 >> Algo Packet Loss Rate Average Throughput >> vegas 0.05% 2.0 MBps >> newreno 0.05% 3.2 MBps >> cubic 0.05% 5.6 MBps >> hd 0.05% 8.6 MBps >> cdg 0.05% 13.5 MBps >> rack 0.04% 14 MBps >> htcp 0.05% 15 MBps >> dctcp 0.05% 15 MBps >> chd 0.05% 17.3 MBps >> bbr 0.05% 29.2 MBps >> cubic 10% 159 kBps >> chd 10% 208 kBps >> bbr 10% 5.7 MBps >>=20 >> RACK seemed to achieve about the same maximum bandwidth as BBR, = though >> it took a lot longer to get there. Also, with RACK, wireshark >> reported about 10x as many retransmissions as dropped packets, which >> is suspicious. >>=20 >> At one point, something went haywire and packet loss briefly spiked = to >> the neighborhood of 10%. I took advantage of the chaos to repeat my >> measurements. As the table shows, all algorithms sucked under those >> conditions, but BBR sucked impressively less than the others. >>=20 >> Disclaimer: there was significant run-to-run variation; the presented >> results are averages. And I did not attempt to measure packet loss >> exactly for most runs; 0.05% is merely an average of a few selected >> runs. These measurements were taken on a production server running a >> real workload, which introduces noise. Soon I hope to have the >> opportunity to repeat the experiment on an idle server in the same >> environment. >>=20 >> In conclusion, while we'd like to use BBR, we really can't until we >> upgrade to 14.1, which hopefully will be soon. So in the meantime >> we've switched all relevant servers from cubic to chd, and we'll >> reevaluate BBR after the upgrade. > Hi Alan, >=20 > just to be clear: the version of BBR currently implemented is > BBR version 1, which is known to be unfair in certain scenarios. > Google is still working on BBR to address this problem and improve > it in other aspects. But there is no RFC yet and the updates haven't > been implemented yet in FreeBSD. >=20 > Best regards > Michael >>=20 >> [1]: https://www.slac.stanford.edu/comp/net/wan-mon/thru-vs-loss.html >>=20 >> -Alan >=20 >=20 >=20 >=20 > --=20 > Junho Choi | https://saturnsoft.net From nobody Thu Jul 18 16:35:46 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WPz2G0C98z5Rj1w for ; Thu, 18 Jul 2024 16:35:50 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (mail-n.franken.de [193.175.24.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WPz2F5xF9z4Qyw; Thu, 18 Jul 2024 16:35:49 +0000 (UTC) (envelope-from tuexen@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (unknown [IPv6:2a02:8109:1140:c3d:25ea:b605:4a77:64ed]) (Authenticated sender: micmac) by drew.franken.de (Postfix) with ESMTPSA id DB41E721E2817; Thu, 18 Jul 2024 18:35:46 +0200 (CEST) Content-Type: text/plain; charset=utf-8 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.600.62\)) Subject: Re: TCP Success Story (was Re: TCP_RACK, TCP_BBR, and firewalls) From: tuexen@freebsd.org In-Reply-To: Date: Thu, 18 Jul 2024 18:35:46 +0200 Cc: FreeBSD Net Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Alan Somers X-Mailer: Apple Mail (2.3774.600.62) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:680, ipnet:193.174.0.0/15, country:DE] X-Rspamd-Queue-Id: 4WPz2F5xF9z4Qyw > On 18. Jul 2024, at 16:03, Alan Somers wrote: >=20 > On Wed, Jul 17, 2024 at 2:27=E2=80=AFPM wrote: >>=20 >>> On 17. Jul 2024, at 22:00, Alan Somers wrote: >>>=20 >>> On Sat, Jul 13, 2024 at 1:50=E2=80=AFAM wrote: >>>>=20 >>>>> On 13. Jul 2024, at 01:43, Alan Somers = wrote: >>>>>=20 >>>>> I've been experimenting with RACK and BBR. In my environment, = they >>>>> can dramatically improve single-stream TCP performance, which is >>>>> awesome. But pf interferes. I have to disable pf in order for = them >>>>> to work at all. >>>>>=20 >>>>> Is this a known limitation? If not, I will experiment some more = to >>>>> determine exactly what aspect of my pf configuration is = responsible. >>>>> If so, can anybody suggest what changes would have to happen to = make >>>>> the two compatible? >>>> A problem with same symptoms was already reported and fixed in >>>> https://reviews.freebsd.org/D43769 >>>>=20 >>>> Which version are you using? >>>>=20 >>>> Best regards >>>> Michael >>>>>=20 >>>>> -Alan >>>=20 >>> TLDR; tcp_rack is good, cc_chd is better, and tcp_bbr is best >>>=20 >>> I want to follow up with the list to post my conclusions. Firstly >>> tuexen@ helped me solve my problem: in FreeBSD 14.0 there is a 3-way >>> incompatibility between (tcp_bbr || tcp_rack) && lro && pf. I can >>> confirm that tcp_bbr works for me if I either disable LRO, disable = PF, >>> or switch to a 14.1 server. >>>=20 >>> Here's the real problem: on multiple production servers, downloading >>> large files (or ZFS send/recv streams) was slow. After ruling out >>> many possible causes, wireshark revealed that the connection was >>> suffering about 0.05% packet loss. I don't know the source of that >>> packet loss, but I don't believe it to be congestion-related. Along >>> with a 54ms RTT, that's a fatal combination for the throughput of >>> loss-based congestion control algorithms. According to the Mathis >>> Formula [1], I could only expect 1.1 MBps over such a connection. >>> That's actually worse than what I saw. With default settings >>> (cc_cubic), I averaged 5.6 MBps. Probably Mathis's assumptions are >>> outdated, but that's still pretty close for such a simple formula >>> that's 27 years old. >>>=20 >>> So I benchmarked all available congestion control algorithms for >>> single download streams. The results are summarized in the table >>> below. >>>=20 >>> Algo Packet Loss Rate Average Throughput >>> vegas 0.05% 2.0 MBps >>> newreno 0.05% 3.2 MBps >>> cubic 0.05% 5.6 MBps >>> hd 0.05% 8.6 MBps >>> cdg 0.05% 13.5 MBps >>> rack 0.04% 14 MBps >>> htcp 0.05% 15 MBps >>> dctcp 0.05% 15 MBps >>> chd 0.05% 17.3 MBps >>> bbr 0.05% 29.2 MBps >>> cubic 10% 159 kBps >>> chd 10% 208 kBps >>> bbr 10% 5.7 MBps >>>=20 >>> RACK seemed to achieve about the same maximum bandwidth as BBR, = though >>> it took a lot longer to get there. Also, with RACK, wireshark >>> reported about 10x as many retransmissions as dropped packets, which >>> is suspicious. >>>=20 >>> At one point, something went haywire and packet loss briefly spiked = to >>> the neighborhood of 10%. I took advantage of the chaos to repeat my >>> measurements. As the table shows, all algorithms sucked under those >>> conditions, but BBR sucked impressively less than the others. >>>=20 >>> Disclaimer: there was significant run-to-run variation; the = presented >>> results are averages. And I did not attempt to measure packet loss >>> exactly for most runs; 0.05% is merely an average of a few selected >>> runs. These measurements were taken on a production server running = a >>> real workload, which introduces noise. Soon I hope to have the >>> opportunity to repeat the experiment on an idle server in the same >>> environment. >>>=20 >>> In conclusion, while we'd like to use BBR, we really can't until we >>> upgrade to 14.1, which hopefully will be soon. So in the meantime >>> we've switched all relevant servers from cubic to chd, and we'll >>> reevaluate BBR after the upgrade. >> Hi Alan, >>=20 >> just to be clear: the version of BBR currently implemented is >> BBR version 1, which is known to be unfair in certain scenarios. >> Google is still working on BBR to address this problem and improve >> it in other aspects. But there is no RFC yet and the updates haven't >> been implemented yet in FreeBSD. >=20 > I've also heard that RACK suffers from fairness problems. Do you know > how RACK and BBR compare for fairness? RACK should be fare, BBR (version 1) is known not be be fair... Best regards Michael From nobody Thu Jul 18 17:23:53 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQ05p0zBxz5RmdQ for ; Thu, 18 Jul 2024 17:23:58 +0000 (UTC) (envelope-from sm@codenetworks.net) Received: from relayout05-q01.dominioabsoluto.net (relayout05-q01.dominioabsoluto.net [217.116.26.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQ05n4YZvz4h5s; Thu, 18 Jul 2024 17:23:57 +0000 (UTC) (envelope-from sm@codenetworks.net) Authentication-Results: mx1.freebsd.org; none Received: from relayout05-redir.dominioabsoluto.net (relayout05-redir.dominioabsoluto.net [217.116.26.107]) by relayout05.dominioabsoluto.net (Postfix) with ESMTP id 4WQ05k6wCwzMp95; Thu, 18 Jul 2024 19:23:54 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=codenetworks.net; s=domabs; t=1721323434; bh=bHQYRWZ+X94ecM8m0Ox9dpdDv8Sg7EF7F/N+kcPqx9Y=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=Fdj6v4STZtOYrcjYKelVE0ClBc0W20Rilp26OWRzyWYqK114EzNTctQ45ANkslw9C gb9+1XCXbeHGdvl7BAFMdYe/Um1wPRpNddWSTUmlfS1vSDq9vxB4SSwyC+bWTTzbg2 bqEf02W7HPshnAnEM+OkmOVh/kkBVBIgd/Y0o53Y= Received: from [192.168.3.20] (unknown [188.241.98.123]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sm.codenetworks.net) by relayout05-dsp.dominioabsoluto.net (Postfix) with ESMTPSA id 4WQ05k4DFyzMp95; Thu, 18 Jul 2024 19:23:54 +0200 (CEST) Content-Type: multipart/alternative; boundary="------------0K0uDWZTW2U6ylXmGou9U7nS" Message-ID: <48536b16-e7f1-49d3-b0ac-37773ac686bf@codenetworks.net> Date: Thu, 18 Jul 2024 19:23:53 +0200 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Multiple Fibs and INET6 To: Zhenlei Huang Cc: freebsd-net@freebsd.org References: <70305bf3-cfe4-4cdd-8cb8-c03c1a9c2b8f@codenetworks.net> Content-Language: en-US From: Santiago Martinez In-Reply-To: X-PostalOut-Country: IP: 188.241.98.123 | Country: ES X-PostalOut-Information: AntiSPAM and AntiVIRUS on relayout05 X-PostalOut-MsgID: 4WQ05k4DFyzMp95.A1892 X-PostalOut-SpamCheck: no es spam, clean X-PostalOut-From: sm@codenetworks.net X-PostalOut-Watermark: 1721928234.80647@8zpNz9P8Pe4pW4FHSMf1hQ X-Spam-Status: No X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:16371, ipnet:217.116.24.0/21, country:ES] X-Rspamd-Queue-Id: 4WQ05n4YZvz4h5s This is a multi-part message in MIME format. --------------0K0uDWZTW2U6ylXmGou9U7nS Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Indeed, ping does work if I ping the "fe80::1%lo1" on FIB 1, which is correct. My script was getting the address from the routing table output (F1) which is returning "%lo0" instead of the correct loopback number (lo6 in my case) and as a result it was failing. The routing table should return the correct loop-back interface instead of lo0. Not sure how difficult or easy it, i will take a look ( ENOCLUE). Best regards. Santi On 7/18/24 18:15, Zhenlei Huang wrote: > > >> On Jul 19, 2024, at 12:11 AM, Santiago Martinez >> wrote: >> >> Interesting, I'm running 14.1p2. >> >> > > Yes, I'm running exactly the same version with you. >> >> how does your routing table looks for fib1 ? >> >> > > ``` > # netstat -6rnF 1 > Routing tables (fib: 1) > > Internet6: > Destination                       Gateway     Flags     Netif Expire > fe80::%lo1/64                     link#5      U           lo1 > fe80::1%lo0                       link#2      UHS         lo0 > ``` >> >> Santi >> >> >> On 7/18/24 18:09, Zhenlei Huang wrote: >>> >>> >>>> On Jul 13, 2024, at 1:06 AM, Santiago Martinez >>>> wrote: >>>> >>>> Hi Everyone. >>>> >>>> While adding -F ( fib as used in netstat ) to ping and ping6 I have >>>> found something that from my understanding is not correct. >>>> Please can you advise? >>>> >>>> I have the following setup : >>>> >>>> -- two fibs (0 and 1) >>>> -- two  loop-backs (lo0 and lo1). >>>> -- Lo1 has been assigned to fib1 >>>> --net.add_addr_allfibs = 0 >>>> >>>> My interface output looks like this: >>>> >>>> >>>> ifconfig lo0 | grep inet6 >>>>        inet6 ::1 prefixlen 128 >>>>        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>> >>>> ifconfig lo1 | grep inet6 >>>>        inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3 >>>> >>>> If I do a netstat -rn -6  -F0 I get the following which is was i >>>> expected. >>>> >>>> Internet6: >>>> Destination                       Gateway >>>>                       Flags     Netif Expire >>>> ::/96                             link#2                        URS >>>>         lo0 >>>> ::1                               link#2                        UHS >>>>         lo0 >>>> ::ffff:0.0.0.0/96                 link#2                        URS >>>>         lo0 >>>> fe80::%lo0/10                     link#2                        URS >>>>         lo0 >>>> fe80::%lo0/64                     link#2                        U >>>>           lo0 >>>> fe80::1%lo0                       link#2                        UHS >>>>         lo0 >>>> ff02::/16                         link#2                        URS >>>>         lo0 >>>> >>>> Now,  netstat -rn -6  -F1 shows  "fe80::1%lo0" which should not be >>>> there and "fe80::1%lo1" is missing which should be there. >>>> >>>> Internet6: >>>> Destination                       Gateway >>>>                       Flags     Netif Expire >>>> fe80::%lo1/64                     link#3                        U >>>>           lo1 >>>> *fe80::1%lo0                       link#2 >>>>                        UHS         lo0* >>>> >>> That seems wrong from my first glance. IIRC, there's HACK ( I'd >>> prefer this ) for loopback route. For example >>> ``` >>> # sysctl net.fibs=3 >>> net.fibs: 2 -> 3 >>> # ifconfig epair create >>> # epair0a >>> # ifconfig epair0a fib 2 >>> # ifconfig epair0a inet6 -ifdisabled up >>> # netstat -6rnF 2 >>> Routing tables (fib: 2) >>> >>> Internet6: >>> Destination                       Gateway     Flags     Netif Expire >>> fe80::%epair0a/64                 link#5                        U   >>>     epair0a >>> fe80::3b:b3ff:fe8f:9a0a%lo0       link#1                        UHS lo0 >>> ``` >>> >>> The loopback route always refer the first loop interface, aka lo0. >>>> >>>> >>>> What output I was expecting was: >>>> >>>> Internet6: >>>> Destination                       Gateway >>>>                       Flags     Netif Expire >>>> fe80::%lo1/64                     link#3                        U >>>>           lo1 >>>> *fe80::1%lo1                       link#3                        >>>> UHS         lo1* >>>> >>>> >>>> This makes the ping -6 -F0 fe80::1%lo0  to work but ping -6 -F1 >>>> fe80::1%l01 to fail which I wanted to use as test case. >>>> >>> That is interesting. I can ping without failure. >>> >>> ``` >>> # setfib 1 ping6 -c3 fe80::1%lo1 >>> PING(56=40+8+8 bytes) fe80::1%lo1 --> fe80::1%lo1 >>> 16 bytes from fe80::1%lo1, icmp_seq=0 hlim=64 time=0.050 ms >>> 16 bytes from fe80::1%lo1, icmp_seq=1 hlim=64 time=0.067 ms >>> 16 bytes from fe80::1%lo1, icmp_seq=2 hlim=64 time=0.096 ms >>> >>> --- fe80::1%lo1 ping statistics --- >>> 3 packets transmitted, 3 packets received, 0.0% packet loss >>> round-trip min/avg/max/stddev = 0.050/0.071/0.096/0.019 ms >>> ``` >>> >>> Best regards, >>> Zhenlei >>> >>>> >>>> Thanks in advance. >>>> >>>> Santiago >>>> >>>> >>> >>> >>> > > > --------------0K0uDWZTW2U6ylXmGou9U7nS Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

Indeed, ping does work if I ping the "fe80::1%lo1" on FIB 1,  which is correct.

My script was getting the address from the routing table output  (F1) which is returning "%lo0" instead of the correct loopback number (lo6 in my case) and as a result it was failing. 

The routing table should return the correct loop-back interface instead of lo0. Not sure how difficult or easy it, i will take a look ( ENOCLUE).

Best regards.

Santi


On 7/18/24 18:15, Zhenlei Huang wrote:


On Jul 19, 2024, at 12:11 AM, Santiago Martinez <sm@codenetworks.net> wrote:

Interesting, I'm running 14.1p2.



Yes, I'm running exactly the same version with you.

how does your routing table looks for fib1 ?



```
# netstat -6rnF 1
Routing tables (fib: 1)

Internet6:
Destination                       Gateway                       Flags     Netif Expire
fe80::%lo1/64                     link#5                        U           lo1
fe80::1%lo0                       link#2                        UHS         lo0
```

Santi


On 7/18/24 18:09, Zhenlei Huang wrote:


On Jul 13, 2024, at 1:06 AM, Santiago Martinez <sm@codenetworks.net> wrote:

Hi Everyone.

While adding -F ( fib as used in netstat ) to ping and ping6 I have found something that from my understanding is not correct.
Please can you advise?

I have the following setup :

-- two fibs (0 and 1) 
-- two  loop-backs (lo0 and lo1).
-- Lo1 has been assigned to fib1
-- net.add_addr_allfibs = 0

My interface output looks like this:


ifconfig lo0 | grep inet6
       inet6 ::1 prefixlen 128
       inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2

ifconfig lo1 | grep inet6
       inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3

If I do a netstat -rn -6  -F0 I get the following which is was i expected.

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             link#2                        URS         lo0
::1                               link#2                        UHS         lo0
::ffff:0.0.0.0/96                 link#2                        URS         lo0
fe80::%lo0/10                     link#2                        URS         lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16                         link#2                        URS         lo0

Now,  netstat -rn -6  -F1 shows  "fe80::1%lo0" which should not be there and "fe80::1%lo1" is missing which should be there.

Internet6:
Destination                       Gateway                       Flags     Netif Expire
fe80::%lo1/64                     link#3                        U           lo1
fe80::1%lo0                       link#2                        UHS         lo0

That seems wrong from my first glance. IIRC, there's HACK ( I'd prefer this ) for loopback route. For example
```
# sysctl net.fibs=3
net.fibs: 2 -> 3
# ifconfig epair create
# epair0a
# ifconfig epair0a fib 2
# ifconfig epair0a inet6 -ifdisabled up
# netstat -6rnF 2
Routing tables (fib: 2)

Internet6:
Destination                       Gateway                       Flags     Netif Expire
fe80::%epair0a/64                 link#5                        U       epair0a
fe80::3b:b3ff:fe8f:9a0a%lo0       link#1                        UHS         lo0
```

The loopback route always refer the first loop interface, aka lo0.  


What output I was expecting was:

Internet6:
Destination                       Gateway                       Flags     Netif Expire
fe80::%lo1/64                     link#3                        U           lo1
fe80::1%lo1                       link#3                        UHS         lo1


This makes the ping -6 -F0 fe80::1%lo0  to work but ping -6 -F1 fe80::1%l01 to fail which I wanted to use as test case.

That is interesting. I can ping without failure.

```
# setfib 1 ping6 -c3 fe80::1%lo1
PING(56=40+8+8 bytes) fe80::1%lo1 --> fe80::1%lo1
16 bytes from fe80::1%lo1, icmp_seq=0 hlim=64 time=0.050 ms
16 bytes from fe80::1%lo1, icmp_seq=1 hlim=64 time=0.067 ms
16 bytes from fe80::1%lo1, icmp_seq=2 hlim=64 time=0.096 ms

--- fe80::1%lo1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.050/0.071/0.096/0.019 ms
```

Best regards,
Zhenlei


Thanks in advance.

Santiago








--------------0K0uDWZTW2U6ylXmGou9U7nS-- From nobody Thu Jul 18 18:37:45 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQ1lB69VWz5Q8ls for ; Thu, 18 Jul 2024 18:37:58 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ua1-f44.google.com (mail-ua1-f44.google.com [209.85.222.44]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQ1lB4VJFz4q3c; Thu, 18 Jul 2024 18:37:58 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ua1-f44.google.com with SMTP id a1e0cc1a2514c-821db15a930so375474241.1; Thu, 18 Jul 2024 11:37:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721327877; x=1721932677; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9Xskhp8JiX2ePIUVLp1yUNj4v+Qc7CHphQ2kMceE+ck=; b=FV5gjCp6LfRXAJV15Zww4OCy5refSbiaaqOln8A02zVRhDnbI+lTbmZ6Kh9Mr3uYLH NQXnNvQMblEMGKtUzoXaP6o+cAmyq8ok+GJqIyzcn6QEo04L/Sunghhmb9A0dyRgGe5f 01EPAE9iIfn9sjT3ku98/1pM3U9HWpCDKz2jMIoRijpa39gl6jdrpEGbyj4UaFjREq3D Ofljss55X1ZL4XX4ASrWq7BP/1vraRj5TN+gmCLXe9B1bGDxc4F3hYErwyyeMaO1A0wH 3+B0Z4/9zpM1QoZTeqH7e0Qr7ys5iN+/k3odwwLYjsrCaPIkisDDCBid03hV1xX6Zb9R EcgA== X-Forwarded-Encrypted: i=1; AJvYcCUAWOIXOH6/j9MmbZjVtS4RpUJ6yTWv0oJ63ibAvQPcx2uWAlW78YIsoyKwEbMZ+hYAy3ZWp+nYiLcTWG99P1CZMyy+BsK5Qg== X-Gm-Message-State: AOJu0YzJ+0sYMm1bI3zbzzsvMsrMeSl35c8XSA2s2yODZnjo/HSP+NkC iK1gmJExXXJ/e6/4zuzCaoNnFZ/FyXPNHiigqbnjYJHMPiecbHiRtGjzXZC4167yqAhMF9YDlQr bIWit6Dchr6zrqUaP2SG5XvUpAA6vvw== X-Google-Smtp-Source: AGHT+IF/6mZ6bb0RWrvWOvgr53MQJlTL0f7q1qgWxooFaCWPqtzBixJN78rMpwUgbIs1TwqlFEPCjwJBDM+33bKnPRE= X-Received: by 2002:a05:6102:5487:b0:48f:e655:fad9 with SMTP id ada2fe7eead31-49159a62193mr6602561137.33.1721327877281; Thu, 18 Jul 2024 11:37:57 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Thu, 18 Jul 2024 12:37:45 -0600 Message-ID: Subject: Re: TCP Success Story (was Re: TCP_RACK, TCP_BBR, and firewalls) To: tuexen@freebsd.org Cc: Junho Choi , FreeBSD Net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Queue-Id: 4WQ1lB4VJFz4q3c Coexist how? Do you mean that one socket can use one and a different socket uses the other? That makes sense. On Thu, Jul 18, 2024 at 10:34=E2=80=AFAM wrote: > > > On 18. Jul 2024, at 15:00, Junho Choi wrote: > > > > Alan - this is a great result to see. Thanks for experimenting. > > > > Just curious why bbr and rack don't co-exist? Those are two separate th= ings. > > Is it a current bug or by design? > Technically RACK and BBR can coexist. The problem was with pf and/or LRO. > > But this is all fixed now in 14.1 and head. > > Best regards > Michael > > > > BR, > > > > On Thu, Jul 18, 2024 at 5:27=E2=80=AFAM wrote: > >> On 17. Jul 2024, at 22:00, Alan Somers wrote: > >> > >> On Sat, Jul 13, 2024 at 1:50=E2=80=AFAM wrote: > >>> > >>>> On 13. Jul 2024, at 01:43, Alan Somers wrote: > >>>> > >>>> I've been experimenting with RACK and BBR. In my environment, they > >>>> can dramatically improve single-stream TCP performance, which is > >>>> awesome. But pf interferes. I have to disable pf in order for them > >>>> to work at all. > >>>> > >>>> Is this a known limitation? If not, I will experiment some more to > >>>> determine exactly what aspect of my pf configuration is responsible. > >>>> If so, can anybody suggest what changes would have to happen to make > >>>> the two compatible? > >>> A problem with same symptoms was already reported and fixed in > >>> https://reviews.freebsd.org/D43769 > >>> > >>> Which version are you using? > >>> > >>> Best regards > >>> Michael > >>>> > >>>> -Alan > >> > >> TLDR; tcp_rack is good, cc_chd is better, and tcp_bbr is best > >> > >> I want to follow up with the list to post my conclusions. Firstly > >> tuexen@ helped me solve my problem: in FreeBSD 14.0 there is a 3-way > >> incompatibility between (tcp_bbr || tcp_rack) && lro && pf. I can > >> confirm that tcp_bbr works for me if I either disable LRO, disable PF, > >> or switch to a 14.1 server. > >> > >> Here's the real problem: on multiple production servers, downloading > >> large files (or ZFS send/recv streams) was slow. After ruling out > >> many possible causes, wireshark revealed that the connection was > >> suffering about 0.05% packet loss. I don't know the source of that > >> packet loss, but I don't believe it to be congestion-related. Along > >> with a 54ms RTT, that's a fatal combination for the throughput of > >> loss-based congestion control algorithms. According to the Mathis > >> Formula [1], I could only expect 1.1 MBps over such a connection. > >> That's actually worse than what I saw. With default settings > >> (cc_cubic), I averaged 5.6 MBps. Probably Mathis's assumptions are > >> outdated, but that's still pretty close for such a simple formula > >> that's 27 years old. > >> > >> So I benchmarked all available congestion control algorithms for > >> single download streams. The results are summarized in the table > >> below. > >> > >> Algo Packet Loss Rate Average Throughput > >> vegas 0.05% 2.0 MBps > >> newreno 0.05% 3.2 MBps > >> cubic 0.05% 5.6 MBps > >> hd 0.05% 8.6 MBps > >> cdg 0.05% 13.5 MBps > >> rack 0.04% 14 MBps > >> htcp 0.05% 15 MBps > >> dctcp 0.05% 15 MBps > >> chd 0.05% 17.3 MBps > >> bbr 0.05% 29.2 MBps > >> cubic 10% 159 kBps > >> chd 10% 208 kBps > >> bbr 10% 5.7 MBps > >> > >> RACK seemed to achieve about the same maximum bandwidth as BBR, though > >> it took a lot longer to get there. Also, with RACK, wireshark > >> reported about 10x as many retransmissions as dropped packets, which > >> is suspicious. > >> > >> At one point, something went haywire and packet loss briefly spiked to > >> the neighborhood of 10%. I took advantage of the chaos to repeat my > >> measurements. As the table shows, all algorithms sucked under those > >> conditions, but BBR sucked impressively less than the others. > >> > >> Disclaimer: there was significant run-to-run variation; the presented > >> results are averages. And I did not attempt to measure packet loss > >> exactly for most runs; 0.05% is merely an average of a few selected > >> runs. These measurements were taken on a production server running a > >> real workload, which introduces noise. Soon I hope to have the > >> opportunity to repeat the experiment on an idle server in the same > >> environment. > >> > >> In conclusion, while we'd like to use BBR, we really can't until we > >> upgrade to 14.1, which hopefully will be soon. So in the meantime > >> we've switched all relevant servers from cubic to chd, and we'll > >> reevaluate BBR after the upgrade. > > Hi Alan, > > > > just to be clear: the version of BBR currently implemented is > > BBR version 1, which is known to be unfair in certain scenarios. > > Google is still working on BBR to address this problem and improve > > it in other aspects. But there is no RFC yet and the updates haven't > > been implemented yet in FreeBSD. > > > > Best regards > > Michael > >> > >> [1]: https://www.slac.stanford.edu/comp/net/wan-mon/thru-vs-loss.html > >> > >> -Alan > > > > > > > > > > -- > > Junho Choi | https://saturnsoft.net > From nobody Thu Jul 18 19:23:02 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQ2lH2ppSz5QDMy for ; Thu, 18 Jul 2024 19:23:07 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (mail-n.franken.de [193.175.24.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQ2lG6z0Qz4tsn; Thu, 18 Jul 2024 19:23:06 +0000 (UTC) (envelope-from tuexen@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (unknown [IPv6:2a02:8109:1140:c3d:2030:6b10:c350:abba]) (Authenticated sender: micmac) by drew.franken.de (Postfix) with ESMTPSA id 2505E721E2817; Thu, 18 Jul 2024 21:23:03 +0200 (CEST) Content-Type: text/plain; charset=utf-8 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.600.62\)) Subject: Re: TCP Success Story (was Re: TCP_RACK, TCP_BBR, and firewalls) From: tuexen@freebsd.org In-Reply-To: Date: Thu, 18 Jul 2024 21:23:02 +0200 Cc: Junho Choi , FreeBSD Net Content-Transfer-Encoding: quoted-printable Message-Id: <400A46A2-E75F-4BE3-BFFF-340CF4557322@freebsd.org> References: To: Alan Somers X-Mailer: Apple Mail (2.3774.600.62) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[]; ASN(0.00)[asn:680, ipnet:193.174.0.0/15, country:DE] X-Rspamd-Queue-Id: 4WQ2lG6z0Qz4tsn > On 18. Jul 2024, at 20:37, Alan Somers wrote: >=20 > Coexist how? Do you mean that one socket can use one and a different > socket uses the other? That makes sense. Correct. Best regards Michael >=20 > On Thu, Jul 18, 2024 at 10:34=E2=80=AFAM wrote: >>=20 >>> On 18. Jul 2024, at 15:00, Junho Choi wrote: >>>=20 >>> Alan - this is a great result to see. Thanks for experimenting. >>>=20 >>> Just curious why bbr and rack don't co-exist? Those are two separate = things. >>> Is it a current bug or by design? >> Technically RACK and BBR can coexist. The problem was with pf and/or = LRO. >>=20 >> But this is all fixed now in 14.1 and head. >>=20 >> Best regards >> Michael >>>=20 >>> BR, >>>=20 >>> On Thu, Jul 18, 2024 at 5:27=E2=80=AFAM wrote: >>>> On 17. Jul 2024, at 22:00, Alan Somers wrote: >>>>=20 >>>> On Sat, Jul 13, 2024 at 1:50=E2=80=AFAM wrote: >>>>>=20 >>>>>> On 13. Jul 2024, at 01:43, Alan Somers = wrote: >>>>>>=20 >>>>>> I've been experimenting with RACK and BBR. In my environment, = they >>>>>> can dramatically improve single-stream TCP performance, which is >>>>>> awesome. But pf interferes. I have to disable pf in order for = them >>>>>> to work at all. >>>>>>=20 >>>>>> Is this a known limitation? If not, I will experiment some more = to >>>>>> determine exactly what aspect of my pf configuration is = responsible. >>>>>> If so, can anybody suggest what changes would have to happen to = make >>>>>> the two compatible? >>>>> A problem with same symptoms was already reported and fixed in >>>>> https://reviews.freebsd.org/D43769 >>>>>=20 >>>>> Which version are you using? >>>>>=20 >>>>> Best regards >>>>> Michael >>>>>>=20 >>>>>> -Alan >>>>=20 >>>> TLDR; tcp_rack is good, cc_chd is better, and tcp_bbr is best >>>>=20 >>>> I want to follow up with the list to post my conclusions. Firstly >>>> tuexen@ helped me solve my problem: in FreeBSD 14.0 there is a = 3-way >>>> incompatibility between (tcp_bbr || tcp_rack) && lro && pf. I can >>>> confirm that tcp_bbr works for me if I either disable LRO, disable = PF, >>>> or switch to a 14.1 server. >>>>=20 >>>> Here's the real problem: on multiple production servers, = downloading >>>> large files (or ZFS send/recv streams) was slow. After ruling out >>>> many possible causes, wireshark revealed that the connection was >>>> suffering about 0.05% packet loss. I don't know the source of that >>>> packet loss, but I don't believe it to be congestion-related. = Along >>>> with a 54ms RTT, that's a fatal combination for the throughput of >>>> loss-based congestion control algorithms. According to the Mathis >>>> Formula [1], I could only expect 1.1 MBps over such a connection. >>>> That's actually worse than what I saw. With default settings >>>> (cc_cubic), I averaged 5.6 MBps. Probably Mathis's assumptions are >>>> outdated, but that's still pretty close for such a simple formula >>>> that's 27 years old. >>>>=20 >>>> So I benchmarked all available congestion control algorithms for >>>> single download streams. The results are summarized in the table >>>> below. >>>>=20 >>>> Algo Packet Loss Rate Average Throughput >>>> vegas 0.05% 2.0 MBps >>>> newreno 0.05% 3.2 MBps >>>> cubic 0.05% 5.6 MBps >>>> hd 0.05% 8.6 MBps >>>> cdg 0.05% 13.5 MBps >>>> rack 0.04% 14 MBps >>>> htcp 0.05% 15 MBps >>>> dctcp 0.05% 15 MBps >>>> chd 0.05% 17.3 MBps >>>> bbr 0.05% 29.2 MBps >>>> cubic 10% 159 kBps >>>> chd 10% 208 kBps >>>> bbr 10% 5.7 MBps >>>>=20 >>>> RACK seemed to achieve about the same maximum bandwidth as BBR, = though >>>> it took a lot longer to get there. Also, with RACK, wireshark >>>> reported about 10x as many retransmissions as dropped packets, = which >>>> is suspicious. >>>>=20 >>>> At one point, something went haywire and packet loss briefly spiked = to >>>> the neighborhood of 10%. I took advantage of the chaos to repeat = my >>>> measurements. As the table shows, all algorithms sucked under = those >>>> conditions, but BBR sucked impressively less than the others. >>>>=20 >>>> Disclaimer: there was significant run-to-run variation; the = presented >>>> results are averages. And I did not attempt to measure packet loss >>>> exactly for most runs; 0.05% is merely an average of a few selected >>>> runs. These measurements were taken on a production server running = a >>>> real workload, which introduces noise. Soon I hope to have the >>>> opportunity to repeat the experiment on an idle server in the same >>>> environment. >>>>=20 >>>> In conclusion, while we'd like to use BBR, we really can't until we >>>> upgrade to 14.1, which hopefully will be soon. So in the meantime >>>> we've switched all relevant servers from cubic to chd, and we'll >>>> reevaluate BBR after the upgrade. >>> Hi Alan, >>>=20 >>> just to be clear: the version of BBR currently implemented is >>> BBR version 1, which is known to be unfair in certain scenarios. >>> Google is still working on BBR to address this problem and improve >>> it in other aspects. But there is no RFC yet and the updates haven't >>> been implemented yet in FreeBSD. >>>=20 >>> Best regards >>> Michael >>>>=20 >>>> [1]: = https://www.slac.stanford.edu/comp/net/wan-mon/thru-vs-loss.html >>>>=20 >>>> -Alan >>>=20 >>>=20 >>>=20 >>>=20 >>> -- >>> Junho Choi | https://saturnsoft.net >>=20 >=20 From nobody Fri Jul 19 03:07:15 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQF3c5D5Nz5QxZj for ; Fri, 19 Jul 2024 03:07:56 +0000 (UTC) (envelope-from junho.choi@gmail.com) Received: from mail-lj1-x231.google.com (mail-lj1-x231.google.com [IPv6:2a00:1450:4864:20::231]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQF3c2h4fz4cxc; Fri, 19 Jul 2024 03:07:56 +0000 (UTC) (envelope-from junho.choi@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lj1-x231.google.com with SMTP id 38308e7fff4ca-2eeb2d60efbso22936401fa.1; Thu, 18 Jul 2024 20:07:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721358473; x=1721963273; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=MT2G3BtEhcqel7vj1rWb+BdogIQ79JXaiPiWYDfQFo0=; b=mVprtj0vDD/J0PTFn2xXUEKhh7DRLue/e3TZoKvdnuVd0ScFWAHZHLAK/boYSXB1jm XZDDQScZ8GZBtSC7S8U9Ex7GZ4tjpit/ko0FFA6FfE0XaraINDL4xyvJNVCZH+Jxs1vF rCbKHw0nG1WkM88jPTCNn8/BsrP8PfrL14kKEqzDBWhi8Ax6KhepgexYDGVB3krgqPZI CiWc00GZCP3+J+TuTjvu90V/+a3Dfhhu53DNVQ+cgRkTkl/0OlehskChkNoY8id2gm4a GN8IkzYs94+YErxxaXyxjh7zF2ez2ab0TvZBjhiu8AM4D7Bt34mFixSKpwuBAtAlfo/r wH5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721358473; x=1721963273; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=MT2G3BtEhcqel7vj1rWb+BdogIQ79JXaiPiWYDfQFo0=; b=f7X+qigN9jPvPjq3ySReIsatvG6yLifVVBZjG5sjZYb2EkQOyDiN7GycC6YjwWFqhd TvS0cuuR/qW3ftbNM8WopDhiirxefpVv7XObxQ6Kw4zY06QUHmDZ78bObf9iQhrQZnzn YLhTixPzuhgeiU0Y/6tS/Et0J7mD20c2Cc52Q6ArSbCaKK7K/4DWZIqtPFt1kDK0MqWG JQN1EWuRrFUDUmQ9LwTN0AhEKflgpZ4UzGDA+SkYM/Ph3HWuWqpINNIoVy2bwF5gTE7V 3SzoifhCnbxdyym80/KjImfxRRjuxtDrnHFolplAU+7FM6AalJ1bKVom4gZtXNcWCrC3 QJuQ== X-Forwarded-Encrypted: i=1; AJvYcCX+MAqfPalU0249daRhkZzq8wAWODyw+WZC57/br9YfTkryYzczLRdVGPi30M+r7oBnBrSAB8HlKsG6ypTFO8fBmR2VSN6hkA== X-Gm-Message-State: AOJu0YxiBrpVM9W7yFwYJJ6WWFPi4RF2Ks3CrFd9cAU2Jv3mK+oE3Q/g X9RuUhEC2kOMxBUqnHEvzFVFw90HGIonoGeXE9D0+MzDk7ik0eYFCcky8BeLRKe03wJ3xj0b8Tb 54kOXzCCdPFvrKf1ZBa7X/C5GCsrx2mJb X-Google-Smtp-Source: AGHT+IGa3TjIdY3AmBgQCpzhIwdf8g4zBxhOQUTC0XDCbvxEfvX3KX3zBLd68izJQSwRJ+48S+7oovCzEMPCrbouW2w= X-Received: by 2002:a05:6512:3a91:b0:52e:95fc:3937 with SMTP id 2adb3069b0e04-52ee53b1494mr4829728e87.15.1721358472871; Thu, 18 Jul 2024 20:07:52 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 References: <400A46A2-E75F-4BE3-BFFF-340CF4557322@freebsd.org> In-Reply-To: <400A46A2-E75F-4BE3-BFFF-340CF4557322@freebsd.org> From: Junho Choi Date: Fri, 19 Jul 2024 12:07:15 +0900 Message-ID: Subject: Re: TCP Success Story (was Re: TCP_RACK, TCP_BBR, and firewalls) To: tuexen@freebsd.org Cc: Alan Somers , FreeBSD Net Content-Type: multipart/alternative; boundary="0000000000004e85a0061d9100d9" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4WQF3c2h4fz4cxc --0000000000004e85a0061d9100d9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable RACK is a loss detection algorithm and BBR is a congestion control algorithm so it's on a different layer. e.g. linux can configure them independently. However in FreeBSD it looks like it is using the same configuration sysctl (net.inet.tcp.functions_default=3Dtcp_rack|tcp_bbr), so not able to set it both. Is there any plan to improve it? or does tcp_bbr include tcp_rack's loss probe behavior? A little confused. Best, On Fri, Jul 19, 2024 at 4:23=E2=80=AFAM wrote: > > On 18. Jul 2024, at 20:37, Alan Somers wrote: > > > > Coexist how? Do you mean that one socket can use one and a different > > socket uses the other? That makes sense. > Correct. > > Best regards > Michael > > > > On Thu, Jul 18, 2024 at 10:34=E2=80=AFAM wrote: > >> > >>> On 18. Jul 2024, at 15:00, Junho Choi wrote: > >>> > >>> Alan - this is a great result to see. Thanks for experimenting. > >>> > >>> Just curious why bbr and rack don't co-exist? Those are two separate > things. > >>> Is it a current bug or by design? > >> Technically RACK and BBR can coexist. The problem was with pf and/or > LRO. > >> > >> But this is all fixed now in 14.1 and head. > >> > >> Best regards > >> Michael > >>> > >>> BR, > >>> > >>> On Thu, Jul 18, 2024 at 5:27=E2=80=AFAM wrote: > >>>> On 17. Jul 2024, at 22:00, Alan Somers wrote: > >>>> > >>>> On Sat, Jul 13, 2024 at 1:50=E2=80=AFAM wrote: > >>>>> > >>>>>> On 13. Jul 2024, at 01:43, Alan Somers wrote= : > >>>>>> > >>>>>> I've been experimenting with RACK and BBR. In my environment, the= y > >>>>>> can dramatically improve single-stream TCP performance, which is > >>>>>> awesome. But pf interferes. I have to disable pf in order for th= em > >>>>>> to work at all. > >>>>>> > >>>>>> Is this a known limitation? If not, I will experiment some more t= o > >>>>>> determine exactly what aspect of my pf configuration is responsibl= e. > >>>>>> If so, can anybody suggest what changes would have to happen to ma= ke > >>>>>> the two compatible? > >>>>> A problem with same symptoms was already reported and fixed in > >>>>> https://reviews.freebsd.org/D43769 > >>>>> > >>>>> Which version are you using? > >>>>> > >>>>> Best regards > >>>>> Michael > >>>>>> > >>>>>> -Alan > >>>> > >>>> TLDR; tcp_rack is good, cc_chd is better, and tcp_bbr is best > >>>> > >>>> I want to follow up with the list to post my conclusions. Firstly > >>>> tuexen@ helped me solve my problem: in FreeBSD 14.0 there is a 3-way > >>>> incompatibility between (tcp_bbr || tcp_rack) && lro && pf. I can > >>>> confirm that tcp_bbr works for me if I either disable LRO, disable P= F, > >>>> or switch to a 14.1 server. > >>>> > >>>> Here's the real problem: on multiple production servers, downloading > >>>> large files (or ZFS send/recv streams) was slow. After ruling out > >>>> many possible causes, wireshark revealed that the connection was > >>>> suffering about 0.05% packet loss. I don't know the source of that > >>>> packet loss, but I don't believe it to be congestion-related. Along > >>>> with a 54ms RTT, that's a fatal combination for the throughput of > >>>> loss-based congestion control algorithms. According to the Mathis > >>>> Formula [1], I could only expect 1.1 MBps over such a connection. > >>>> That's actually worse than what I saw. With default settings > >>>> (cc_cubic), I averaged 5.6 MBps. Probably Mathis's assumptions are > >>>> outdated, but that's still pretty close for such a simple formula > >>>> that's 27 years old. > >>>> > >>>> So I benchmarked all available congestion control algorithms for > >>>> single download streams. The results are summarized in the table > >>>> below. > >>>> > >>>> Algo Packet Loss Rate Average Throughput > >>>> vegas 0.05% 2.0 MBps > >>>> newreno 0.05% 3.2 MBps > >>>> cubic 0.05% 5.6 MBps > >>>> hd 0.05% 8.6 MBps > >>>> cdg 0.05% 13.5 MBps > >>>> rack 0.04% 14 MBps > >>>> htcp 0.05% 15 MBps > >>>> dctcp 0.05% 15 MBps > >>>> chd 0.05% 17.3 MBps > >>>> bbr 0.05% 29.2 MBps > >>>> cubic 10% 159 kBps > >>>> chd 10% 208 kBps > >>>> bbr 10% 5.7 MBps > >>>> > >>>> RACK seemed to achieve about the same maximum bandwidth as BBR, thou= gh > >>>> it took a lot longer to get there. Also, with RACK, wireshark > >>>> reported about 10x as many retransmissions as dropped packets, which > >>>> is suspicious. > >>>> > >>>> At one point, something went haywire and packet loss briefly spiked = to > >>>> the neighborhood of 10%. I took advantage of the chaos to repeat my > >>>> measurements. As the table shows, all algorithms sucked under those > >>>> conditions, but BBR sucked impressively less than the others. > >>>> > >>>> Disclaimer: there was significant run-to-run variation; the presente= d > >>>> results are averages. And I did not attempt to measure packet loss > >>>> exactly for most runs; 0.05% is merely an average of a few selected > >>>> runs. These measurements were taken on a production server running = a > >>>> real workload, which introduces noise. Soon I hope to have the > >>>> opportunity to repeat the experiment on an idle server in the same > >>>> environment. > >>>> > >>>> In conclusion, while we'd like to use BBR, we really can't until we > >>>> upgrade to 14.1, which hopefully will be soon. So in the meantime > >>>> we've switched all relevant servers from cubic to chd, and we'll > >>>> reevaluate BBR after the upgrade. > >>> Hi Alan, > >>> > >>> just to be clear: the version of BBR currently implemented is > >>> BBR version 1, which is known to be unfair in certain scenarios. > >>> Google is still working on BBR to address this problem and improve > >>> it in other aspects. But there is no RFC yet and the updates haven't > >>> been implemented yet in FreeBSD. > >>> > >>> Best regards > >>> Michael > >>>> > >>>> [1]: https://www.slac.stanford.edu/comp/net/wan-mon/thru-vs-loss.htm= l > >>>> > >>>> -Alan > >>> > >>> > >>> > >>> > >>> -- > >>> Junho Choi | https://saturnsoft.net > >> > > > > > --=20 Junho Choi | https://saturnsoft.net --0000000000004e85a0061d9100d9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
RACK is a loss detection algorithm and BBR is a conge= stion control algorithm so it's on a different layer.
e.g. li= nux can configure them independently.

However = in FreeBSD it looks like it is using the same configuration sysctl (net.ine= t.tcp.functions_default=3Dtcp_rack|tcp_bbr),
so not able to set i= t both.

Is there any plan to improve it? or does t= cp_bbr include tcp_rack's loss probe behavior?

A little confused.

Best,

=

= On Fri, Jul 19, 2024 at 4:23=E2=80=AFAM <tuexen@freebsd.org> wrote:
> On 18. Jul 2024, at 20:37, Alan Somers <asomers@freebsd.org<= /a>> wrote:
>
> Coexist how?=C2=A0 Do you mean that one socket can use one and a diffe= rent
> socket uses the other?=C2=A0 That makes sense.
Correct.

Best regards
Michael
>
> On Thu, Jul 18, 2024 at 10:34=E2=80=AFAM <tuexen@freebsd.org> wrote:
>>
>>> On 18. Jul 2024, at 15:00, Junho Choi <junho.choi@gmail.com> wrote: >>>
>>> Alan - this is a great result to see. Thanks for experimenting= .
>>>
>>> Just curious why bbr and rack don't co-exist? Those are tw= o separate things.
>>> Is it a current bug or by design?
>> Technically RACK and BBR can coexist. The problem was with pf and/= or LRO.
>>
>> But this is all fixed now in 14.1 and head.
>>
>> Best regards
>> Michael
>>>
>>> BR,
>>>
>>> On Thu, Jul 18, 2024 at 5:27=E2=80=AFAM <tuexen@freebsd.org> wrote:
>>>> On 17. Jul 2024, at 22:00, Alan Somers <asomers@freebsd.org> wrote= :
>>>>
>>>> On Sat, Jul 13, 2024 at 1:50=E2=80=AFAM <tuexen@freebsd.org> wrote:=
>>>>>
>>>>>> On 13. Jul 2024, at 01:43, Alan Somers <asomers= @FreeBSD.org> wrote:
>>>>>>
>>>>>> I've been experimenting with RACK and BBR.=C2= =A0 In my environment, they
>>>>>> can dramatically improve single-stream TCP perform= ance, which is
>>>>>> awesome.=C2=A0 But pf interferes.=C2=A0 I have to = disable pf in order for them
>>>>>> to work at all.
>>>>>>
>>>>>> Is this a known limitation?=C2=A0 If not, I will e= xperiment some more to
>>>>>> determine exactly what aspect of my pf configurati= on is responsible.
>>>>>> If so, can anybody suggest what changes would have= to happen to make
>>>>>> the two compatible?
>>>>> A problem with same symptoms was already reported and = fixed in
>>>>> https://reviews.freebsd.org/D43769
>>>>>
>>>>> Which version are you using?
>>>>>
>>>>> Best regards
>>>>> Michael
>>>>>>
>>>>>> -Alan
>>>>
>>>> TLDR; tcp_rack is good, cc_chd is better, and tcp_bbr is b= est
>>>>
>>>> I want to follow up with the list to post my conclusions.= =C2=A0 Firstly
>>>> tuexen@ helped me solve my problem: in FreeBSD 14.0 there = is a 3-way
>>>> incompatibility between (tcp_bbr || tcp_rack) && l= ro && pf.=C2=A0 I can
>>>> confirm that tcp_bbr works for me if I either disable LRO,= disable PF,
>>>> or switch to a 14.1 server.
>>>>
>>>> Here's the real problem: on multiple production server= s, downloading
>>>> large files (or ZFS send/recv streams) was slow.=C2=A0 Aft= er ruling out
>>>> many possible causes, wireshark revealed that the connecti= on was
>>>> suffering about 0.05% packet loss.=C2=A0 I don't know = the source of that
>>>> packet loss, but I don't believe it to be congestion-r= elated.=C2=A0 Along
>>>> with a 54ms RTT, that's a fatal combination for the th= roughput of
>>>> loss-based congestion control algorithms.=C2=A0 According = to the Mathis
>>>> Formula [1], I could only expect 1.1 MBps over such a conn= ection.
>>>> That's actually worse than what I saw.=C2=A0 With defa= ult settings
>>>> (cc_cubic), I averaged 5.6 MBps.=C2=A0 Probably Mathis'= ;s assumptions are
>>>> outdated, but that's still pretty close for such a sim= ple formula
>>>> that's 27 years old.
>>>>
>>>> So I benchmarked all available congestion control algorith= ms for
>>>> single download streams.=C2=A0 The results are summarized = in the table
>>>> below.
>>>>
>>>> Algo=C2=A0 =C2=A0 Packet Loss Rate=C2=A0 =C2=A0 Average Th= roughput
>>>> vegas=C2=A0 =C2=A00.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A02.0 MBps
>>>> newreno 0.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A03.2 MBps
>>>> cubic=C2=A0 =C2=A00.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A05.6 MBps
>>>> hd=C2=A0 =C2=A0 =C2=A0 0.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A08.6 MBps
>>>> cdg=C2=A0 =C2=A0 =C2=A00.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A013.5 MBps
>>>> rack=C2=A0 =C2=A0 0.04%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A014 MBps
>>>> htcp=C2=A0 =C2=A0 0.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A015 MBps
>>>> dctcp=C2=A0 =C2=A00.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A015 MBps
>>>> chd=C2=A0 =C2=A0 =C2=A00.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A017.3 MBps
>>>> bbr=C2=A0 =C2=A0 =C2=A00.05%=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A029.2 MBps
>>>> cubic=C2=A0 =C2=A010%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0159 kBps
>>>> chd=C2=A0 =C2=A0 =C2=A010%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0208 kBps
>>>> bbr=C2=A0 =C2=A0 =C2=A010%=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A05.7 MBps
>>>>
>>>> RACK seemed to achieve about the same maximum bandwidth as= BBR, though
>>>> it took a lot longer to get there.=C2=A0 Also, with RACK, = wireshark
>>>> reported about 10x as many retransmissions as dropped pack= ets, which
>>>> is suspicious.
>>>>
>>>> At one point, something went haywire and packet loss brief= ly spiked to
>>>> the neighborhood of 10%.=C2=A0 I took advantage of the cha= os to repeat my
>>>> measurements.=C2=A0 As the table shows, all algorithms suc= ked under those
>>>> conditions, but BBR sucked impressively less than the othe= rs.
>>>>
>>>> Disclaimer: there was significant run-to-run variation; th= e presented
>>>> results are averages.=C2=A0 And I did not attempt to measu= re packet loss
>>>> exactly for most runs; 0.05% is merely an average of a few= selected
>>>> runs.=C2=A0 These measurements were taken on a production = server running a
>>>> real workload, which introduces noise.=C2=A0 Soon I hope t= o have the
>>>> opportunity to repeat the experiment on an idle server in = the same
>>>> environment.
>>>>
>>>> In conclusion, while we'd like to use BBR, we really c= an't until we
>>>> upgrade to 14.1, which hopefully will be soon.=C2=A0 So in= the meantime
>>>> we've switched all relevant servers from cubic to chd,= and we'll
>>>> reevaluate BBR after the upgrade.
>>> Hi Alan,
>>>
>>> just to be clear: the version of BBR currently implemented is<= br> >>> BBR version 1, which is known to be unfair in certain scenario= s.
>>> Google is still working on BBR to address this problem and imp= rove
>>> it in other aspects. But there is no RFC yet and the updates h= aven't
>>> been implemented yet in FreeBSD.
>>>
>>> Best regards
>>> Michael
>>>>
>>>> [1]: https://www.sl= ac.stanford.edu/comp/net/wan-mon/thru-vs-loss.html
>>>>
>>>> -Alan
>>>
>>>
>>>
>>>
>>> --
>>> Junho Choi <junho dot choi at gmail.com> | https://saturnsoft.ne= t
>>
>




--
Junho Choi <junho dot choi at gmail.com> | https://saturnsoft.net
--0000000000004e85a0061d9100d9-- From nobody Fri Jul 19 05:40:18 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQJRS3hRPz5QxmD for ; Fri, 19 Jul 2024 05:40:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQJRS2dR9z4t72 for ; Fri, 19 Jul 2024 05:40:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721367620; a=rsa-sha256; cv=none; b=GhDlzyqCAofpm1PRlWBe9y8SXEPiZxXxh8O0rGFngW9hqX5+KVMW8OvxRabXuCFtUrtGeW 06RA0aIq1/ly+9izkckucSs/JDxH7PFViRaLXgOX/BQyE0BO+8hMmIQUYsMzmqVpDi58e8 13DaUlU8+z09ukl08wQv8u2Mm1uHW/qSIzstp64GeJ6FCgS1t2/xbdOKFz+n/w+6jnxBzl eV6ykJVUTkW69ZGsLbCa0lTaI/78ere25CIVxDIbR9HiHI2vfqWulyqJXRe2wGFZDbHMzD kkVI69uyEyhtDieGXONFk7ouXIbynZQ/VAGMcYrnEN0T2tDqSUlHUpxvR1KGFw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721367620; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j5dOjcuw165QR54NqSdtgfE824FyO/pJmnV4aPDblAk=; b=CaAz9gCueqPWKPaLAFx0ThWRsoa+mut/5nOWGt6zbP4NjItxhXTWIKkqdV7vnIS0ZUzL8w afmna5SH/GXlZTWrsIInEh9dM9tJaijE+gBEGPhghWAtfBLBRLb7BH9VDrIHrNmItvjznr pLPb+SS0PkIpjFeLQrruAIno7W9CtJjwaokqu4BjnUDAFbPFrPI0tKrEVVEP11VRBkCRZ4 QgVwqgoJLC7jKwBeDlSLH6ZdQVzMrBlaw4kAGnA0QMWlLkpAxsrO/VcOukVVA0aNPuqfJw O8QMHgagTGCJzWNYoU+rIU00Kx16jTScN+SRYw4xPtQasiZ7hAWSwEe6jsT3qw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WQJRS2Bm7zKMv for ; Fri, 19 Jul 2024 05:40:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 46J5eKhe035569 for ; Fri, 19 Jul 2024 05:40:20 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 46J5eKqP035568 for net@FreeBSD.org; Fri, 19 Jul 2024 05:40:20 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 279875] sockstat: segmentation fault Date: Fri, 19 Jul 2024 05:40:18 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D279875 Konstantin Belousov changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kib@FreeBSD.org --- Comment #2 from Konstantin Belousov --- Rebuild sockstat without optimization, like this: # make -C /usr/src/usr.bin/sockstat DEBUG_FLAGS=3D-g clean all install then get the backtrace from gdb. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Jul 19 06:05:19 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQK0Q2DtVz5R0sk for ; Fri, 19 Jul 2024 06:05:26 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQK0P2SPPz3x5R; Fri, 19 Jul 2024 06:05:25 +0000 (UTC) (envelope-from tuexen@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=freebsd.org (policy=none); spf=softfail (mx1.freebsd.org: 2001:638:a02:a001:20e:cff:fe4a:feaa is neither permitted nor denied by domain of tuexen@freebsd.org) smtp.mailfrom=tuexen@freebsd.org Received: from smtpclient.apple (unknown [193.203.3.54]) (Authenticated sender: micmac) by mail-n.franken.de (Postfix) with ESMTPSA id 9C3B7721E2817; Fri, 19 Jul 2024 08:05:19 +0200 (CEST) Content-Type: text/plain; charset=utf-8 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.600.62\)) Subject: Re: TCP Success Story (was Re: TCP_RACK, TCP_BBR, and firewalls) From: tuexen@freebsd.org In-Reply-To: Date: Fri, 19 Jul 2024 08:05:19 +0200 Cc: Alan Somers , FreeBSD Net Content-Transfer-Encoding: quoted-printable Message-Id: <9B43971A-1E12-44C4-930B-51BB1F3ABDC1@freebsd.org> References: <400A46A2-E75F-4BE3-BFFF-340CF4557322@freebsd.org> To: Junho Choi X-Mailer: Apple Mail (2.3774.600.62) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.66 / 15.00]; NEURAL_HAM_LONG(-0.99)[-0.993]; NEURAL_HAM_SHORT(-0.99)[-0.993]; NEURAL_HAM_MEDIUM(-0.78)[-0.778]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : No valid SPF, No valid DKIM,none]; ONCE_RECEIVED(0.10)[]; FROM_NO_DN(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_TO(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; FREEFALL_USER(0.00)[tuexen]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; ASN(0.00)[asn:680, ipnet:2001:638::/32, country:DE]; R_DKIM_NA(0.00)[]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_RCPT(0.00)[]; MIME_TRACE(0.00)[0:+] X-Rspamd-Queue-Id: 4WQK0P2SPPz3x5R > On 19. Jul 2024, at 05:07, Junho Choi wrote: >=20 > RACK is a loss detection algorithm and BBR is a congestion control = algorithm so it's on a different layer. > e.g. linux can configure them independently. >=20 > However in FreeBSD it looks like it is using the same configuration = sysctl (net.inet.tcp.functions_default=3Dtcp_rack|tcp_bbr), > so not able to set it both. FreeBSD supports multiple TCP stacks. One stack has the name RACK and = does RACK loss discovery, but much more like pacing, advanced LRO and so on. You can configure CC modules like newreno, cubic or so to be used. Another TCP stack is called BBR. It does BBRv1 congestion control and other things. >=20 > Is there any plan to improve it? or does tcp_bbr include tcp_rack's = loss probe behavior? I am not aware of a plan to improve the BBR stack right now. Best regards Michael >=20 > A little confused. >=20 > Best, >=20 >=20 > On Fri, Jul 19, 2024 at 4:23=E2=80=AFAM wrote: >> On 18. Jul 2024, at 20:37, Alan Somers wrote: >>=20 >> Coexist how? Do you mean that one socket can use one and a different >> socket uses the other? That makes sense. > Correct. >=20 > Best regards > Michael >>=20 >> On Thu, Jul 18, 2024 at 10:34=E2=80=AFAM wrote: >>>=20 >>>> On 18. Jul 2024, at 15:00, Junho Choi wrote: >>>>=20 >>>> Alan - this is a great result to see. Thanks for experimenting. >>>>=20 >>>> Just curious why bbr and rack don't co-exist? Those are two = separate things. >>>> Is it a current bug or by design? >>> Technically RACK and BBR can coexist. The problem was with pf and/or = LRO. >>>=20 >>> But this is all fixed now in 14.1 and head. >>>=20 >>> Best regards >>> Michael >>>>=20 >>>> BR, >>>>=20 >>>> On Thu, Jul 18, 2024 at 5:27=E2=80=AFAM wrote: >>>>> On 17. Jul 2024, at 22:00, Alan Somers = wrote: >>>>>=20 >>>>> On Sat, Jul 13, 2024 at 1:50=E2=80=AFAM = wrote: >>>>>>=20 >>>>>>> On 13. Jul 2024, at 01:43, Alan Somers = wrote: >>>>>>>=20 >>>>>>> I've been experimenting with RACK and BBR. In my environment, = they >>>>>>> can dramatically improve single-stream TCP performance, which is >>>>>>> awesome. But pf interferes. I have to disable pf in order for = them >>>>>>> to work at all. >>>>>>>=20 >>>>>>> Is this a known limitation? If not, I will experiment some more = to >>>>>>> determine exactly what aspect of my pf configuration is = responsible. >>>>>>> If so, can anybody suggest what changes would have to happen to = make >>>>>>> the two compatible? >>>>>> A problem with same symptoms was already reported and fixed in >>>>>> https://reviews.freebsd.org/D43769 >>>>>>=20 >>>>>> Which version are you using? >>>>>>=20 >>>>>> Best regards >>>>>> Michael >>>>>>>=20 >>>>>>> -Alan >>>>>=20 >>>>> TLDR; tcp_rack is good, cc_chd is better, and tcp_bbr is best >>>>>=20 >>>>> I want to follow up with the list to post my conclusions. Firstly >>>>> tuexen@ helped me solve my problem: in FreeBSD 14.0 there is a = 3-way >>>>> incompatibility between (tcp_bbr || tcp_rack) && lro && pf. I can >>>>> confirm that tcp_bbr works for me if I either disable LRO, disable = PF, >>>>> or switch to a 14.1 server. >>>>>=20 >>>>> Here's the real problem: on multiple production servers, = downloading >>>>> large files (or ZFS send/recv streams) was slow. After ruling out >>>>> many possible causes, wireshark revealed that the connection was >>>>> suffering about 0.05% packet loss. I don't know the source of = that >>>>> packet loss, but I don't believe it to be congestion-related. = Along >>>>> with a 54ms RTT, that's a fatal combination for the throughput of >>>>> loss-based congestion control algorithms. According to the Mathis >>>>> Formula [1], I could only expect 1.1 MBps over such a connection. >>>>> That's actually worse than what I saw. With default settings >>>>> (cc_cubic), I averaged 5.6 MBps. Probably Mathis's assumptions = are >>>>> outdated, but that's still pretty close for such a simple formula >>>>> that's 27 years old. >>>>>=20 >>>>> So I benchmarked all available congestion control algorithms for >>>>> single download streams. The results are summarized in the table >>>>> below. >>>>>=20 >>>>> Algo Packet Loss Rate Average Throughput >>>>> vegas 0.05% 2.0 MBps >>>>> newreno 0.05% 3.2 MBps >>>>> cubic 0.05% 5.6 MBps >>>>> hd 0.05% 8.6 MBps >>>>> cdg 0.05% 13.5 MBps >>>>> rack 0.04% 14 MBps >>>>> htcp 0.05% 15 MBps >>>>> dctcp 0.05% 15 MBps >>>>> chd 0.05% 17.3 MBps >>>>> bbr 0.05% 29.2 MBps >>>>> cubic 10% 159 kBps >>>>> chd 10% 208 kBps >>>>> bbr 10% 5.7 MBps >>>>>=20 >>>>> RACK seemed to achieve about the same maximum bandwidth as BBR, = though >>>>> it took a lot longer to get there. Also, with RACK, wireshark >>>>> reported about 10x as many retransmissions as dropped packets, = which >>>>> is suspicious. >>>>>=20 >>>>> At one point, something went haywire and packet loss briefly = spiked to >>>>> the neighborhood of 10%. I took advantage of the chaos to repeat = my >>>>> measurements. As the table shows, all algorithms sucked under = those >>>>> conditions, but BBR sucked impressively less than the others. >>>>>=20 >>>>> Disclaimer: there was significant run-to-run variation; the = presented >>>>> results are averages. And I did not attempt to measure packet = loss >>>>> exactly for most runs; 0.05% is merely an average of a few = selected >>>>> runs. These measurements were taken on a production server = running a >>>>> real workload, which introduces noise. Soon I hope to have the >>>>> opportunity to repeat the experiment on an idle server in the same >>>>> environment. >>>>>=20 >>>>> In conclusion, while we'd like to use BBR, we really can't until = we >>>>> upgrade to 14.1, which hopefully will be soon. So in the meantime >>>>> we've switched all relevant servers from cubic to chd, and we'll >>>>> reevaluate BBR after the upgrade. >>>> Hi Alan, >>>>=20 >>>> just to be clear: the version of BBR currently implemented is >>>> BBR version 1, which is known to be unfair in certain scenarios. >>>> Google is still working on BBR to address this problem and improve >>>> it in other aspects. But there is no RFC yet and the updates = haven't >>>> been implemented yet in FreeBSD. >>>>=20 >>>> Best regards >>>> Michael >>>>>=20 >>>>> [1]: = https://www.slac.stanford.edu/comp/net/wan-mon/thru-vs-loss.html >>>>>=20 >>>>> -Alan >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>> -- >>>> Junho Choi | https://saturnsoft.net >>>=20 >>=20 >=20 >=20 >=20 >=20 > --=20 > Junho Choi | https://saturnsoft.net From nobody Fri Jul 19 08:42:06 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQNTC0Mcsz5RFW9 for ; Fri, 19 Jul 2024 08:42:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQNTB4vgRz4Fkn for ; Fri, 19 Jul 2024 08:42:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721378526; a=rsa-sha256; cv=none; b=g2dlI7L6iueD+Hn0c2cgzvMqF58d1ykYlu4I7sQmlQhGemjYA+TwjMi7dCxJ7h1wyLKpYW m9j9UifXCdDjmdZhcwcLYH1lHgk7gBXsZUSaD9xvBo6nts0OTgQYVNc5L8GdbNKQjB5FGG /ET7QIe6wFX2wd4BWmjD7TP2hPh/0BvAS3kXOb14Mm9LI2Kcr23scpbXDtEkh88BB0qJF6 uOIVVskuSvbFMKlcA/czOQB77kv6CIhOVy4Zxv2IeUrknQwLzcFpBsGZhvDVJI2xcSYIps FOougkTPtWmyjyhYb2YEXN6nayTqNDQwXf9r8hNAZIt2ZbX6m/NvSAibHOjQHw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721378526; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QJovI7GLSnM7gLeCuktwG/Z3ihwV7B0i/qLstQfK6No=; b=nTdp37LX/Rgai9i7etrx/wW9eanS03n3r07AryL0eXeLqTLURpUeolW39jyRp7wZCXkhKc 41S9spE3JQcZy0CQ7zFkDatYAfbKPwIdzcEteKRxVRgfDvDxAfZJvRhRBDo01nWeUM/FJb QC3ffQZSv0Rn5LwlNhbZtxyD82TVGGCEH7EaaEdCOV48QZZEx8YNqS4HtZeoMXPz9u3QoO yOrvbxvSF1Uywxw+M8cey/zKpAY8DZf7EB7N0C+tynn9heNJt0GfICYdhuX6VNxWSD/DWI yDCLA5ley/I9KyBKfCRVaogjzu1JGwNTp/usxyy2Gc8Yy1Blnm2hADJP6/QbUw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WQNTB4WGMzPR9 for ; Fri, 19 Jul 2024 08:42:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 46J8g6qx095823 for ; Fri, 19 Jul 2024 08:42:06 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 46J8g6ve095822 for net@FreeBSD.org; Fri, 19 Jul 2024 08:42:06 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 279875] sockstat: segmentation fault Date: Fri, 19 Jul 2024 08:42:06 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: john@jmarshall.id.au X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D279875 --- Comment #3 from John Marshall --- (In reply to Konstantin Belousov from comment #2) Thank you. - Re-built and installed sockstat as per your instruction. - Built and installed devel/gdb rwsrv08# gdb sockstat GNU gdb (GDB) 14.1 [GDB v14.1 for FreeBSD] ... Reading symbols from sockstat... Reading symbols from /build/usr/lib/debug//usr/bin/sockstat.debug... (gdb) r Starting program: /usr/bin/sockstat=20 [Detaching after fork from child process 85890] USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS= =20=20=20=20=20=20 root sockstat 85895 6 stream -> [85889 8] root sockstat 85894 6 stream -> [85889 7] ... root syslogd 2948 9 dgram /var/run/logpriv root gssd 2810 3 stream /var/run/gssd.sock Program received signal SIGSEGV, Segmentation fault. Address not mapped to object. files_t_RB_FIND (head=3D, elm=3D) at /kits/src/usr.bin/sockstat/sockstat.c:181 181 RB_GENERATE_STATIC(files_t, file, file_tree, file_compare); (gdb) bt #0 files_t_RB_FIND (head=3D, elm=3D) at /kits/src/usr.bin/sockstat/sockstat.c:181 #1 displaysock (s=3Ds@entry=3D0x80184b440, pos=3D, pos@entr= y=3D30) at /kits/src/usr.bin/sockstat/sockstat.c:1165 #2 0x000000000102671f in display () at /kits/src/usr.bin/sockstat/sockstat.c:1345 #3 0x0000000001025c07 in main (argc=3D, argv=3D) at /kits/src/usr.bin/sockstat/sockstat.c:1577 (gdb) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Jul 19 10:22:21 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQQht6LF3z5RP0x for ; Fri, 19 Jul 2024 10:22:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQQht5KrBz4Pdy for ; Fri, 19 Jul 2024 10:22:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721384542; a=rsa-sha256; cv=none; b=PO7ga9nzwB8PMIsO6vFo5FHywnVaaSZt2tqBgHTJ18M5P4KoJAXt1B6nnI5ErBwEJJusiZ oehvrXpKxTSWJGlI6lM+TpTMUb0Dik5lRmqk6THr0FfTsam8m6X9qgVkO9aAroNZJQRBJp NfHZ5r8hS8Hs+Oa1jdsH0bfNN1UgbRAzSALuiCH9QGF80bmpztSCI+rZt+ntAk7H651DI2 s5mHxOZImI3KZyzTJcs5nvbhd39RApjumA8amEc8XPTmwx4s+m7rZoloRSjXyGWjP0oWDh oQjsNWVeeHbDCdlW1SdipvTP8EiKzGr061RPpABHXtLvAXaQn3pumpDMHDRwuQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721384542; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TpapyzKCTG3mVZr2zB4+uvwy3fAu2Jfr921vU+rJRYg=; b=l9c1AaWdAEl7gK9oqwFSZK6K8B8MrSPrlFbKDx+Vu59FEF7ydWOCIXLRv8kxTlXgfPMA8e wPlysHvuWqPSFUArqykhwatcVPujxd+yZgbxYbgwUEClo2eeKRY8eaa/4TNVq82ap+J4A8 6N81XieQNg5EYUiMo2FotOpWoN9juG8Fo8mae100eoBeczGyZRN+F+EYVU5JhAJyuEoU0M lUM+crcRYwUW2GX0PipB5vdkNeNlq6sS6fKrvMoCUZrkGNYjmA+u1sxTMcQMjhdmdBK/Cg pQYB6bRQNJP86aErQQR+Spmh4cajdxsX8u0p0jpXXDJi88Wh1zWatFr5n0ODxA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WQQht4xh7zShh for ; Fri, 19 Jul 2024 10:22:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 46JAMMNf005784 for ; Fri, 19 Jul 2024 10:22:22 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 46JAMMmG005783 for net@FreeBSD.org; Fri, 19 Jul 2024 10:22:22 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 279875] sockstat: segmentation fault Date: Fri, 19 Jul 2024 10:22:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: john@jmarshall.id.au X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D279875 --- Comment #4 from John Marshall --- (In reply to Konstantin Belousov from comment #2) Sorry. That re-compile still had -O2. I found that I was able to get it to compile with -O0 by adding 'CFLAGS=3D-O0' to /etc/make.conf. Is there a wa= y to tell make(1) to use -O0 just for 'this' Makefile? Here is a more useful backtrace I hope. rwsrv08#=20 rwsrv08# gdb sockstat GNU gdb (GDB) 14.1 [GDB v14.1 for FreeBSD] ... Reading symbols from sockstat... Reading symbols from /build/usr/lib/debug//usr/bin/sockstat.debug... (gdb) r Starting program: /usr/bin/sockstat=20 [Detaching after fork from child process 88560] USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS= =20=20=20=20=20=20 root sockstat 88565 6 stream -> [88559 8] root sockstat 88564 6 stream -> [88559 7] ... root syslogd 2948 9 dgram /var/run/logpriv root gssd 2810 3 stream /var/run/gssd.sock Program received signal SIGSEGV, Segmentation fault. Address not mapped to object. 0x0000000001029669 in displaysock (s=3D0x80184b200, pos=3D59) at /kits/src/usr.bin/sockstat/sockstat.c:1169 1169 (u_long)f->xf_pid, f->xf_fd); (gdb) bt #0 0x0000000001029669 in displaysock (s=3D0x80184b200, pos=3D59) at /kits/src/usr.bin/sockstat/sockstat.c:1169 #1 0x000000000102776b in display () at /kits/src/usr.bin/sockstat/sockstat.c:1345 #2 0x0000000001024eab in main (argc=3D0, argv=3D0x7fffffffeb60) at /kits/src/usr.bin/sockstat/sockstat.c:1577 (gdb) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Jul 19 23:04:08 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQlbw2N1Mz5QYQD for ; Fri, 19 Jul 2024 23:04:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQlbw0xwFz4b6t for ; Fri, 19 Jul 2024 23:04:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721430252; a=rsa-sha256; cv=none; b=IwWP5nfxoJzspXa1a7+yi8DBqZkVW/0cd+VjwsgMW0t3brIX5U53ytDok02iClKwHMLKoc MuhoTeoZZTVI+4eNqykyGS8TX1BixbdHUiAYRyv0yE2mIBxYt8j6AfWZ3hJAPqpANp5PvZ /WUYQqy+ijFXocKGIl03v3BhU6qBREWA27RfbSkZ2uDWNFB2s0LjJV9lYnzwxpYboZA/Rr xnv+bT7VzPRBq0PgQqtBIqkSFZAPRkU/JQxxZv3ORbzWPSliZbG0hwGA4ygU5KvnPgFW+x UO5kbtI+BEdyqDeSZWgF01dwUhbv46yQ5ANfuBeVYQytGVepFaF7dfY9IFtheg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721430252; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9atJJaOOhhn0icC6ZoOEjAGEf1CTJjKtbl1pOrn2bLI=; b=kiuXphiqMIGiDCSk/jYmEvVRChe0d1fWhXQsBWqVqiPWqIgRQTnjAgzvurLQPMlfoOz9Hz 7jf3uC63KR73mu3DzoTYhKpOB5+C+tLgJ1Ne+V1gWn4OOTJRaEaHpv424nddpMhawB78ht w9lB8nfrqmYWNPkOLhYneOdTbv7pikOK8qLlZnDO37Q+72IwM2Bw+CFCRv72TMULal2C3J +vs+3UyJWGuRG+X5xvc/qu2EodH68ClX8X7gzBmAkwJ+JBxL6/ftEtUkqmPxJevdq0scGn P+coaT1bngKukkRCoeXxFAXaxcqCj94EAj6Uvkkip2bUdDCZ4n/8Tr0UVB15Jg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WQlbw0YpSzrsN for ; Fri, 19 Jul 2024 23:04:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 46JN4Cwk005549 for ; Fri, 19 Jul 2024 23:04:12 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 46JN4CYD005548 for net@FreeBSD.org; Fri, 19 Jul 2024 23:04:12 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 279875] sockstat: segmentation fault Date: Fri, 19 Jul 2024 23:04:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D279875 --- Comment #5 from Konstantin Belousov --- Do you have set something that prevents visibility of processes/jails? Like see_other_uids etc? Anyway, please apply this debugging patch and report if the program still SIGSEGVs with it. diff --git a/usr.bin/sockstat/sockstat.c b/usr.bin/sockstat/sockstat.c index 73b1f00a4481..20a0a5e65e0a 100644 --- a/usr.bin/sockstat/sockstat.c +++ b/usr.bin/sockstat/sockstat.c @@ -1164,6 +1164,7 @@ displaysock(struct sock *s, int pos) f =3D RB_FIND(files_t, &ftree, &(struct file){ .xf_data =3D p->socket }); + if (f !=3D NULL) pos +=3D xprintf("[%lu %d]", (u_long)f->xf_pid, f->xf_fd); } else @@ -1183,6 +1184,7 @@ displaysock(struct sock *s, int pos) f =3D RB_FIND(files_t, &ftree, &(struct file){ .xf_data =3D p->socket }); + if (f !=3D NULL); pos +=3D xprintf("%s[%lu %d]", fref ? "" : ",", (u_long)f->xf_pid, f->xf_fd); --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Jul 20 00:26:38 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQnR42MLDz5QhSl for ; Sat, 20 Jul 2024 00:26:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQnR371fFz4jjF for ; Sat, 20 Jul 2024 00:26:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721435200; a=rsa-sha256; cv=none; b=YZwIF1BrhFdCBVqnIMdiSoZXjnzsZePt+E4V12My1V2oLtPWW/Px9LHReaqRxb2r7LRHm0 7g+DmSqTTiE/AWyTb7/lOHsVE3p7cQCJRxACAIs680BRxUOedLXHAcolhHhKOXcWfXsn8F ok7GGYmhqwk5NX5gH1HUlA63hQhfnrJp++jvQAgWnUePfacb0SzqUI+MbT9iC3gPmf7/0C l9Yadi4gQOvfz2ydlBbLvhHt4+1PLYFJd8Byu/g4a0aXW9iALrufiI+W84aTVzToDI0T2x D7QqxSKitbxc4A5TEVqSIygkXE9msST3zc4Fu+/PiRAtLJX76Up8Im0DUv1cxQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721435200; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YTFevXiN8o/Reoz7xygY/wLHqTt96yd2QXfONEYE75A=; b=s+BImaF6/KgbLM0Lwf3O8twk2S0nLOlXbmD9/HxlMX83ZVOnS7+WrLMU/xHch7j8R28/02 SBVdhgEewKPEUob0hSwhwBTxLzWlfL4NT+KdtzG/7b8LbGZAx70xWToM4kV5ca3qFrtvag GMwyQgMMN9WrIFOc9InK1ZU0ftSCwEP2aN+9ZPNPluPYoExgVKLJhi9STHk0isOLrjo7GD 1pYAZLVHsxlMbpFhuyH38noAcLaSUcDPnWDex4iWkTN7zLVCdrpPtnw/h6DsJxOYxZBlUI iAwsUob+uPq2p2u/Novxwr2c67lvdBZ0sC48pJFrobXT35rcE0/lwlpOzHYZMg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WQnR36bwCztSj for ; Sat, 20 Jul 2024 00:26:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 46K0QdwP043077 for ; Sat, 20 Jul 2024 00:26:39 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 46K0QdWr043074 for net@FreeBSD.org; Sat, 20 Jul 2024 00:26:39 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 279875] sockstat: segmentation fault Date: Sat, 20 Jul 2024 00:26:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: john@jmarshall.id.au X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D279875 --- Comment #6 from John Marshall --- (In reply to Konstantin Belousov from comment #5) That debugging patch stops the segfault in my case. Program runs to complet= ion. Yes, I have security restrictions in place. My sysctl.conf includes the following. -- kern.securelevel=3D0 # init(8) will raise this to 1 going multi-user. security.bsd.see_other_uids=3D0 security.bsd.see_other_gids=3D0 security.bsd.unprivileged_proc_debug=3D0 security.bsd.unprivileged_read_msgbuf=3D0 -- Thank you. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Jul 20 00:39:50 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQnkH50mLz5QjDH for ; Sat, 20 Jul 2024 00:39:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQnkH3yVgz4ksC for ; Sat, 20 Jul 2024 00:39:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721435991; a=rsa-sha256; cv=none; b=A1VNtOpzepqVCWvelC6u6S5rgtRnDQvV4pw92U2gxGxSq+lKh2WeBcCL4lKke0+S368wNW 78QHyqvO7dcjYUQSV1eQum20zRGTo9l6QBnlaRlzQRA7rMyfWOZN6Boiv4ats4NzXUt2e5 VOzi0dTi2aKxC6YiTW1h1zcLyPM6+PvFnuAuKBUpnjzvjsFPrpdbUGt4JLQ2p5AlVy3cLc rojgoz126EDQcMS6zYLjmEKprVJYmAOblYEeX2yKcOneQ5xqt829Fa/S29KQ91oemu+nHm R5y0jFC7LlTC7/8a8OB3aDsb9EgT49OvpXa/ldS1fAP+uKTlShJjlnj9EwOMNA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721435991; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hZr20vJUdIZYWfS3hTkUO40rf92s1uGrvwWC20pHkDk=; b=d5fAlvj3NvNOQ0nVkKyYuH2h2zZv0QWl+UWzixIQmGduSkAyzVDccAygWvajjkUw4tX6BW /lpNfufO8sCKr4wrze0adZuq0D/CgV2azV4zyKdGB5PdnKBtONHm+5UIyRDmgEQopJ38Ty spPHX6Zz4ggwcIi9mN2XB3R1F2ZBXIglDmgaFNDuu9y27462+uJjGsPAMRvovvtkG7OLXY WVh4IbbkQgubWAU1xRV2Zu1HzyOEnJ2/PaF7R7B0ZPK4Z/1BmlvGkcVoT/yIK2bRpR/wjx TOJIyVkGEPcEsielnYAt+mMK41JL57N+g/EqJeqS0y5EaziHTp+zSy0zXt7Qrg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WQnkH3XTNzvFf for ; Sat, 20 Jul 2024 00:39:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 46K0dpVf091584 for ; Sat, 20 Jul 2024 00:39:51 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 46K0dp81091583 for net@FreeBSD.org; Sat, 20 Jul 2024 00:39:51 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 279875] sockstat: segmentation fault Date: Sat, 20 Jul 2024 00:39:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D279875 --- Comment #7 from Konstantin Belousov --- https://reviews.freebsd.org/D46050 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sat Jul 20 07:22:56 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQygN4P1Nz5R7cj for ; Sat, 20 Jul 2024 07:22:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQygN3Gggz4V2f for ; Sat, 20 Jul 2024 07:22:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721460176; a=rsa-sha256; cv=none; b=dB8/gs7ik7bNoVzI4wa+vdeqVEnHM1PWP5BNvXzgagbJolZznxbQm/AE8ggE88EcdoKYSQ HBrV8UeWkLUIqPt+3o/+onGH9IHeL70kkQ5xvee9jSoJWYVoR57/BJoXp8RiW8LAj3J13N 6Urm1vn2g+uHCuJxvZgwg0kraL3qy2lgwtAqgUXC4+86Ty3VQHSZUoElmok2RTfef+9wCW 7VZXxXIAWvTN+ixNJQBJ8uW/tOJSkqY8d3mFdKd0eCsYHk1HPSzU7B9bOiwgRqIBkRMKMH Xpe1n2ByUCko4jhk3WHdDBybZlzjIEJAUXnVC36n/k0Jhyu8xrDqMVyczCltAA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721460176; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bqzQwpMU79Omffmk+JJTmWG42Px/QWR+1gO1N+4B3eU=; b=Xra+RdGYSAEfjXWWhWYiVG+eq3SmWpZTjvbRxfdHcHxlD8Dj1j3T7veY7M6F0xyQOKJRNX i73v+udTkhNPt8DvOajXgXzloQGfUnv9Q8UhXZ+mtPpKJz2wOScjniq553IMbW2JC+sxwk KQQUdgcVs7wsaqvnvCaPlIwaEAzpFPo7sOZginRhdT0+J0J3NTuo8scp2WnaFxnJJY4qw1 q3O5kayp1yh5ZbSlERrTZZqZiJqe+LFf8R23tXWubvo7GMU4feOYfQg2qcjhZ3+SC93QIG y5fCkjIjPh7GwT+RG9QF+Tpb76KfU2tKFWjSwsJ4jxj0uolsl7whqyquHkvNVQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WQygN0fRzz16Br for ; Sat, 20 Jul 2024 07:22:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 46K7MtC6012510 for ; Sat, 20 Jul 2024 07:22:55 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 46K7Mt5f012509 for net@FreeBSD.org; Sat, 20 Jul 2024 07:22:55 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 279653] Page fault in in6_selecthlim Date: Sat, 20 Jul 2024 07:22:56 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: takahiro.kurosawa@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D279653 --- Comment #16 from takahiro.kurosawa@gmail.com --- I've retracted review D45690 because it does not completely fix the problem. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Jul 21 07:35:41 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WRZvd6mt2z5QxVB for ; Sun, 21 Jul 2024 07:35:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WRZvd4fDPz4dts for ; Sun, 21 Jul 2024 07:35:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721547341; a=rsa-sha256; cv=none; b=Pi1IUVfujMjUJ0i0Ee938ZvMPMmwLhdvZG+YyS8d4w3eQ4iuKyyIneMXBrwrnLKPqf24ij AErp4jeoY0ZvIVfKvHwHt3R5ko0HWJhPvl/BpYk+JxfCxLHT1V+PqvUxBeIuErBTUbFXdn Y1KSYDDYyE6CHNDpqTr5UJz8+B+NwtSc6FdIJr+0OryGWU5SuBhq+vYHwSPj6Pusx1Jdyb 9awH6G8KzqE3dOCi5m6a05kVh7Z2gx+znfkbeGxnG4EbCzLGwe6F1MRXWKZwTaHEVSarIL Ohw8BGkSCatBBP5ytLqOGwpsZ6UztblNqKc2CtqBDhmsNCyDYT55npTvYaSITw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721547341; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cYKekaBLZ//sRjcamCWi1mpfwn6oL0m9avnvk9V5AKc=; b=XhHmU4bff5XTuLPmWapJeX/3kVTWhHE+Cq/KXCGZR2D51oFwphC3KFT4a4hoG8qk1clgLk 2ZZ9wwpvuBNM9LgGKtICCQR1vDwOCewSsFkAduWr2R1togA2Mh6A0+Dzanhehhzgk0mex8 PtNmGuYrCwodh+OaXpbuAm3KISn+LiY6sjKkqqr7zqTCetVeLcidv6XlO4hZpNuMbDvVDp SKL11322+BmMQQDagApZLusFM0J101j7ILf/dEcV7Z3CQqjIym/mH+e9FK2TAsAOMHvxlm nXiG3+xK/v+PGLZbxDRRu/n1ntQM0Ca3hQBRG/conkDc1UZkjD42vgnav86Vwg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WRZvd4GPszrm8 for ; Sun, 21 Jul 2024 07:35:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 46L7Zf09083192 for ; Sun, 21 Jul 2024 07:35:41 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 46L7ZfE5083187 for net@FreeBSD.org; Sun, 21 Jul 2024 07:35:41 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 280386] if_bridge throws output errors under load Date: Sun, 21 Jul 2024 07:35:41 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D280386 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |net@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Jul 21 08:52:02 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WRcbl4J03z5R4hs for ; Sun, 21 Jul 2024 08:52:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WRcbl2CvCz4mrZ for ; Sun, 21 Jul 2024 08:52:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721551923; a=rsa-sha256; cv=none; b=XRUa604oktD/i1xCVKDqn4lUuP1+I5wxFiHwMJ8H7YDrAvWlbgkJ1kPmmYMn1vUYm8eaBL G2OOo3GDpomMjkXYor7yXmZGDiDdIX8KzDAzckm4p1Pugq+ZoyWHBwbDwUsuZfaYlKdKTA 8ff64ezpAJ2rx9pqsIDrQ8WqA3oTNUB2e4XtXhY8DfZ0xeEDaMMSJYDq3A2rFFcmmkcXkp x9J+iNQrBcZnrk0GgArsqcWfxgcX7N5HwJwmPAJjTepbY5pg3oKEuzzW73QSLFkgAh1zTh 1rBjLvGvcKC4oEP289sIcjWXYlcxZGfvNNUZjpVZM+WyjSowQ9PRuqxKJiMf+A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721551923; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L2X6vaP5RAguDEZYTuIXdfnPnjuzojlWPq6r5swyd3c=; b=WPSPhX1Tvh8rotlqQaZccgQp+TSpIYA+GdVFVv/M22tklnGeamGYvXxonJa6Mtk0zIY1yQ XdFWUZx899CAaaKvn/NltPf1ll4TzNmvCa9nLCWxor38PCxbUcboVutkly0LU+o1UFKmty XbZi1qolpKHaY9EFLUUB0fAG9J4cZ+m98crdwrrWyiC744gyE5Nym5611gTEzClPWpT1yg Kr7KvxtJuWUs+fyv864DqJ2yFsw/WqtvWGSg88dWZW0YEuoYlUOs1oj0y2QUejRLe2Meo9 spRpVzkYLTX07uZyVTeeB3dxxrgyegifiEKmsknj7JEUQH0nMsC1zeiVH6h/6g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WRcbl1Qz2ztVB for ; Sun, 21 Jul 2024 08:52:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 46L8q3aw003467 for ; Sun, 21 Jul 2024 08:52:03 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 46L8q3pP003463 for net@FreeBSD.org; Sun, 21 Jul 2024 08:52:03 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 279875] sockstat: segmentation fault Date: Sun, 21 Jul 2024 08:52:02 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D279875 --- Comment #8 from commit-hook@FreeBSD.org --- A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3D35f4984343229545881a324a00cdbb398= 0d675ce commit 35f4984343229545881a324a00cdbb3980d675ce Author: Konstantin Belousov AuthorDate: 2024-07-20 00:30:55 +0000 Commit: Konstantin Belousov CommitDate: 2024-07-21 08:51:42 +0000 sockstat(1): tolerate situation where file info cannot be fetched Either due to a race, or to the privilege restrictions, it is not guaranteed that kern.files returned file information for all pcbs read from net.inet..pcblist. In this case the file rbtree does not return the matching file by data address, and code must avoid dereferencing NULL. PR: 279875 Reviewed by: asomers Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D46050 usr.bin/sockstat/sockstat.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Jul 21 18:44:12 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WRsl20Kjpz5QGXR for ; Sun, 21 Jul 2024 18:44:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WRsl16NlSz42xx for ; Sun, 21 Jul 2024 18:44:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721587453; a=rsa-sha256; cv=none; b=qYPJvZn2AKL8hreZ7UQhLDA/g+pKE2GWsKexWkiLKvWmnA0S65ra+P1GNvVPkKuTkmKg1D QyuWk1282HO3ZvRdcU1pSkuHrVpi7R8kjDbDsMW4iAErA3KhcBOMgm4OvPxxXcj34hP3NO NPurlqxxxY/Or+ebW9tRHOatWAtQ8WZlHXugzbRBusygH6N7DMS4eyZsPDK2Hv8OsQNHUQ tUY6ZgDM7NL6V6ftCq5c16b+k87q1l0F1mDQJLspW3TcQgQJjqzb4EEMpn3eSBJlgki1Xt nSTA2GeLdvscbx4GzCFsazwZB/biRXNbD0TE1VJ+ZQcmFH0DB7uJhIqG8LlPCA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721587453; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tVbANujM/cym6BPD0b7GrTVhaolYMSortddoHCw+Oak=; b=o3IgLlGgkESuXsP2gRXxcY/G9RBAPGOXq00rwzrUoOGhzn+WnP1MvUuNkMSRAtnHFhTxP0 Jx5awc9vWJsbVqZB9UjUXfSLwgUjhSHAQdnAhhptB9YfRaTu+VCTQV4O5YAoKr03KNZR7j +Eo4yazFmhgjKjDhhCEjd8lEHkKGcy3m0v855ClVva1Y6NuZqK94NsRL+rxoga4R8/4VFW OiOE9mwN8miO82Zxu2UU5OVMJYKbooNvXxOyJilH+NJ0hM+VTJvh8NdMG9EE0lTL5t9xQe AVTeIFrB3dzkA7pjQdix61ru/fW/dVP6YyPQn3kuGAKi7AEw4KpYqwhEeA1u0g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WRsl160XFz1BSW for ; Sun, 21 Jul 2024 18:44:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 46LIiDvq061231 for ; Sun, 21 Jul 2024 18:44:13 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 46LIiDSs061230 for net@FreeBSD.org; Sun, 21 Jul 2024 18:44:13 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 250357] [tcp] RFC 5961 is not implemented completely Date: Sun, 21 Jul 2024 18:44:12 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: Unspecified X-Bugzilla-Keywords: needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: tuexen@freebsd.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D250357 --- Comment #5 from commit-hook@FreeBSD.org --- A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3D646c28ea80cb0f9258386626297495b5a= 0e56db5 commit 646c28ea80cb0f9258386626297495b5a0e56db5 Author: Michael Tuexen AuthorDate: 2024-07-21 09:37:35 +0000 Commit: Michael Tuexen CommitDate: 2024-07-21 09:37:35 +0000 tcp: improve SEG.ACK validation Implement the improved SEG.ACK validation described in RFC 5961. In addition to that, also detect ghost ACKs, which are ACKs for data that has never been sent. The additional checks are enabled by default, but can be disabled by setting the sysctl-variable net.inet.tcp.insecure_ack to a non-zero value. PR: 250357 Reviewed by: Peter Lei, rscheff (older version) MFC after: 1 week Sponsored by: Netflix, Inc. Differential Revision: https://reviews.freebsd.org/D45894 share/man/man4/tcp.4 | 5 ++++- sys/netinet/in_kdtrace.c | 2 ++ sys/netinet/in_kdtrace.h | 3 +++ sys/netinet/tcp_input.c | 44 +++++++++++++++++++++++++++++++++++++++= ++++ sys/netinet/tcp_stacks/bbr.c | 37 ++++++++++++++++++++++++++++++++++++ sys/netinet/tcp_stacks/rack.c | 39 ++++++++++++++++++++++++++++++++++++++ sys/netinet/tcp_var.h | 9 ++++++++- usr.bin/netstat/inet.c | 8 ++++++-- 8 files changed, 143 insertions(+), 4 deletions(-) --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Sun Jul 21 21:00:50 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WRwmg0Xrlz5QV5R for ; Sun, 21 Jul 2024 21:00:51 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WRwmf2R0pz4KK9 for ; Sun, 21 Jul 2024 21:00:50 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721595650; a=rsa-sha256; cv=none; b=DKvfCWc+GyLfPYGQq3B7qmwpFvxlXyARYHe9TR2t7cCvzgDznVxAJcdLu1+IZ+HJQ0wFgk VoiUhS40ClH/BaUDvUGnEjPyH++DC1LuB0JvAcMjzSV/xhiVXauvH3bc1atcr8CpgHP6ny UMHygyjpbwFESONBsudL0dLSe+q4fj6GB+kV05yq4O4YKSCGYYwsOyB7Aw4tgO4PrTFP+p uViBYDO0LYPA2ePQKG6FyXw1dIrDL2ga3hR4gXaaC7WhJ6MneKZvgn5pRosRKQvZ4HbyWk xN4XbyQzVmxk/99LIRLXNYoQyVoLpAReV+Mtjrvv1s5imHtal4nk7SahM0VLEA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721595650; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=qmpqbSEz/rWXH62OxJ903Y1NY7FRpbYzNhP3hh0HE20=; b=EThnKdaVd/aJb03clyOoWVrhq6WLCLCAeplkAPq6w3Tga0gDFMGuBaCowQ4mwOpvwjRmno ZJIkf79koBo8HbL1rm1N7gXNzuQln+i7hfb1AQ8bC5K14rBsRU2Os1k80z/ax88m1lK705 wl94IGv5nKp8QzpIQgQRmNBu1ghjQtzTatRXD+P8FbhVwNGbv1Bwg0+xZbpH2/lKsSYFO7 fGW/c2O02Yittxg1eiU+kT6HhONLKGxxbnu5Rf2s5lN1t8/OWEQR3PNGxzQNkDMtkUkhtu nk6EGnisPUhZH9P2rstnBaLOc3tJM8pRujVLE0uNAoQrVXmikx0wCmPexZ+4oQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4WRwmf1zf9zGFL for ; Sun, 21 Jul 2024 21:00:50 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 46LL0oMK020799 for ; Sun, 21 Jul 2024 21:00:50 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 46LL0oEf020797 for net@FreeBSD.org; Sun, 21 Jul 2024 21:00:50 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202407212100.46LL0oEf020797@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: net@FreeBSD.org Subject: Problem reports for net@FreeBSD.org that need special attention Date: Sun, 21 Jul 2024 21:00:50 +0000 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="17215956501.Adca1f442.12228" Content-Transfer-Encoding: 7bit --17215956501.Adca1f442.12228 Date: Sun, 21 Jul 2024 21:00:50 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 254445 | cloned_interfaces="bridge0" does not respect net. Open | 166724 | if_re(4): watchdog timeout Open | 200836 | iovctl(8): Return descriptions in the returned sc Open | 223824 | Panic in ng_base.c (netgraph) Open | 232472 | ixgbe(4): SR-IOV passthru not working on Hyper-V Open | 234073 | ixl(4): Host X710-DA2 drops connect starting bhyv Open | 241106 | tun/ppp: panic: vm_fault: fault on nofault entry Open | 245981 | bnxt(4): BCM57414 / BCM57416 not initializing: bn Open | 257038 | em(4): Panic on HTTP traffic to or from jail thro Open | 257286 | gateway with `ping -6 -e` is ignored Open | 258623 | cxgbe(4): Slow routing performance: 2 numa domain Open | 258850 | lagg(4): interface vanishes when both member inte Open | 261866 | ixgbe(4): Resets media type -> autoselect after s Open | 262024 | em(4): iflib handles bad packets incorrectly Open | 262093 | ixl(4): RX packet errors on Intel X710 after 12.2 Open | 263568 | ix(4): SR-IOV connection lost after loading VM wi In Progress | 118111 | rc: network.subr Add MAC address based interface 17 problems total for which you should take action. --17215956501.Adca1f442.12228 Date: Sun, 21 Jul 2024 21:00:50 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
New         |    254445 | cloned_interfaces="bridge0" does not respect net.
Open        |    166724 | if_re(4): watchdog timeout
Open        |    200836 | iovctl(8): Return descriptions in the returned sc
Open        |    223824 | Panic in ng_base.c (netgraph)
Open        |    232472 | ixgbe(4): SR-IOV passthru not working on Hyper-V 
Open        |    234073 | ixl(4): Host X710-DA2 drops connect starting bhyv
Open        |    241106 | tun/ppp: panic: vm_fault: fault on nofault entry 
Open        |    245981 | bnxt(4): BCM57414 / BCM57416 not initializing: bn
Open        |    257038 | em(4): Panic on HTTP traffic to or from jail thro
Open        |    257286 | gateway with `ping -6 -e` is ignored
Open        |    258623 | cxgbe(4): Slow routing performance: 2 numa domain
Open        |    258850 | lagg(4): interface vanishes when both member inte
Open        |    261866 | ixgbe(4): Resets media type -> autoselect after s
Open        |    262024 | em(4): iflib handles bad packets incorrectly
Open        |    262093 | ixl(4): RX packet errors on Intel X710 after 12.2
Open        |    263568 | ix(4): SR-IOV connection lost after loading VM wi
In Progress |    118111 | rc: network.subr Add MAC address based interface 

17 problems total for which you should take action.
--17215956501.Adca1f442.12228--