From nobody Mon Apr 8 03:16:47 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VCZ590TgBz5Fxw4 for ; Mon, 8 Apr 2024 03:17:01 +0000 (UTC) (envelope-from antonyyudin@gmail.com) Received: from mail-il1-f179.google.com (mail-il1-f179.google.com [209.85.166.179]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VCZ575Dhdz4Ttp for ; Mon, 8 Apr 2024 03:16:59 +0000 (UTC) (envelope-from antonyyudin@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of antonyyudin@gmail.com designates 209.85.166.179 as permitted sender) smtp.mailfrom=antonyyudin@gmail.com Received: by mail-il1-f179.google.com with SMTP id e9e14a558f8ab-36a0f64f5e0so10298065ab.3 for ; Sun, 07 Apr 2024 20:16:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712546218; x=1713151018; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=V5jlJENhq8EQ3BboFp/qviqiOAB4EiMCY8LMz5/+MzE=; b=SNII8CNLJdhyiJwj4rUPBEPrSmxfFBapm/+4U+SjCSpKNMA82CDtIyRQqhNRdWbodw a9BtEY6sZI7LG5Y7BHg0G3gM040l8MK9ydzi/eZ89tIkzhNM9Z0vN2j7L+tkhfXH5gDC FKQQCZ71YVrPUkhtovmqtrgDeuGYaVWvpXcQlF2KsU0wtQNoo2k2qbXCmqfXczHFdqJ9 3KgNf+tzmAykkvGrEQy9Vxwcm2FvDY+YP79oHQ3sEpKsk/LbO2h7JklsKRTaifDCoNgx H3tXrXh5LW/3D46t0PHGsKSxxTkQfqGDkdlVa2MuWt9J9EITLjg77MpSYArB45orveP3 XeEA== X-Gm-Message-State: AOJu0Yx79O/TLSuqCzuqkqVPtCavOyI88WAN0vrDUjeZCuCIpeXQXpFQ lupfGCJgn/hQ1oPpi9dMizBnH9dWKmQHJMpsh7vPH/kg18FoJywkY8eo/x67 X-Google-Smtp-Source: AGHT+IHZuGDuKFdfcmnnf4xMQtd2uVRjMwkUmAyxi+weDCRRTxy3+Mn7DV9UJ6QMiBl+vXsCfeLbgA== X-Received: by 2002:a05:6e02:2487:b0:36a:16ba:a0b5 with SMTP id bt7-20020a056e02248700b0036a16baa0b5mr8394296ilb.17.1712546218376; Sun, 07 Apr 2024 20:16:58 -0700 (PDT) Received: from mail-io1-f44.google.com (mail-io1-f44.google.com. [209.85.166.44]) by smtp.gmail.com with ESMTPSA id b30-20020a026f5e000000b0047f13b719besm2279570jae.88.2024.04.07.20.16.58 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 07 Apr 2024 20:16:58 -0700 (PDT) Received: by mail-io1-f44.google.com with SMTP id ca18e2360f4ac-7d0486cf91aso224874139f.1 for ; Sun, 07 Apr 2024 20:16:58 -0700 (PDT) X-Received: by 2002:a92:ca48:0:b0:36a:29b9:cfb5 with SMTP id q8-20020a92ca48000000b0036a29b9cfb5mr71571ilo.24.1712546218068; Sun, 07 Apr 2024 20:16:58 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 From: Anton Yudin Date: Sun, 7 Apr 2024 23:16:47 -0400 X-Gmail-Original-Message-ID: Message-ID: Subject: How to ignore a default route for one of the dhclient-ed interface? To: freebsd-net@freebsd.org Content-Type: multipart/alternative; boundary="000000000000fd5e3d06158d3cbc" X-Spamd-Bar: - X-Spamd-Result: default: False [-1.93 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.93)[-0.935]; FORGED_SENDER(0.30)[contact@antonyudin.com,antonyyudin@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[antonyudin.com]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_NEQ_ENVFROM(0.00)[contact@antonyudin.com,antonyyudin@gmail.com]; MISSING_XM_UA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.166.179:from]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[209.85.166.179:from,209.85.166.44:received] X-Rspamd-Queue-Id: 4VCZ575Dhdz4Ttp --000000000000fd5e3d06158d3cbc Content-Type: text/plain; charset="UTF-8" Hello. I'm running a FreeBSD 14 with two interfaces that use DHCP. I would like to make one of the interfaces to never set the default route. Right now the first interface to be fully up sets the default route. I tried to set the following in /etc/dhclient.conf ---------------8<------------------------ interface "wan1" { ignore routers; } ---------------8<------------------------ but the default route still gets set. I ended up creating a /etc/dhclient-enter-hooks with a very hacky code that overrides the "route" command: ---------------8<------------------------ route() { if [ "X$interface" = "Xwan1" -a "X$2" = "Xdefault" ]; then echo "ignore route $1 $2 $3 $4" | logger -t "enter-hooks" else /sbin/route $1 $2 $3 $4 fi } ---------------8<------------------------ Is there a better way of doing this? Thanks. --000000000000fd5e3d06158d3cbc Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello.

=C2=A0 I'm runnin= g a FreeBSD 14 with two interfaces that use DHCP.
=C2=A0 I would = like to make one of the interfaces to never set the default route.
=C2=A0 Right now the first interface to be fully up sets the default rout= e.

=C2=A0 I tried to set the following in /etc= /dhclient.conf
---------------8<------------------------
=C2=A0 interface "wan1" {
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 ignore routers;
=C2=A0 }
---------------8&= lt;------------------------
=C2=A0 but the default route still ge= ts set.

=C2=A0 I ended up creating a /etc/dhclient= -enter-hooks with a very hacky code that overrides the "route" co= mmand:
---------------8<------------------------
route() {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if [ "X$interface" =3D = "Xwan1" -a "X$2" =3D "Xdefault" =C2=A0]; then=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "igno= re route $1 $2 $3 $4" | logger -t "enter-hooks"
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 else
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 /sbin/route $1 $2 $3 $4
=C2=A0 =C2=A0 =C2=A0 =C2=A0 fi
}
---------------8<------------------------

<= div>Is there a better way of doing this?

Thanks.
--000000000000fd5e3d06158d3cbc-- From nobody Mon Apr 8 07:18:06 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VCgRX6tRxz5GPpZ for ; Mon, 8 Apr 2024 07:18:16 +0000 (UTC) (envelope-from list_freebsd@bluerosetech.com) Received: from echo.brtsvcs.net (echo.brtsvcs.net [IPv6:2607:f740:c::4ae]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4VCgRX4gvxz4DHV for ; Mon, 8 Apr 2024 07:18:16 +0000 (UTC) (envelope-from list_freebsd@bluerosetech.com) Authentication-Results: mx1.freebsd.org; none Received: from chombo.houseloki.net (65-100-43-2.dia.static.qwest.net [65.100.43.2]) by echo.brtsvcs.net (Postfix) with ESMTPS id 0B87F38D00; Mon, 08 Apr 2024 07:18:08 +0000 (UTC) Received: from [10.26.25.100] (ivy.pas.ds.pilgrimaccounting.com [10.26.25.100]) by chombo.houseloki.net (Postfix) with ESMTPSA id D27073F099; Mon, 8 Apr 2024 00:18:06 -0700 (PDT) Message-ID: <8bd004ff-f4a1-8cf4-dc90-a95328ff5ec9@bluerosetech.com> Date: Mon, 8 Apr 2024 00:18:06 -0700 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.15.1 Subject: Re: How to ignore a default route for one of the dhclient-ed interface? To: Anton Yudin , freebsd-net@freebsd.org References: Content-Language: en-US From: list_freebsd@bluerosetech.com In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36236, ipnet:2607:f740:c::/48, country:US] X-Rspamd-Queue-Id: 4VCgRX4gvxz4DHV On 2024-04-07 20:16, Anton Yudin wrote: >   I'm running a FreeBSD 14 with two interfaces that use DHCP. >   I would like to make one of the interfaces to never set the default > route. >   Right now the first interface to be fully up sets the default route. > >   I tried to set the following in /etc/dhclient.conf > ---------------8<------------------------ >   interface "wan1" { >       ignore routers; >   } > ---------------8<------------------------ >   but the default route still gets set. [...] > Is there a better way of doing this? What happens if you explicitly set the request statement for each of the interfaces, including routers in only the one you want as your gateway? "By default, the DHCPv4 client requests the subnet-mask, broadcast-address, time-offset, routers, domain-name, domain-name-servers and host-name options" --dhclient.conf(5) From nobody Mon Apr 8 08:23:21 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VChtl4nlDz5GWd4 for ; Mon, 8 Apr 2024 08:23:27 +0000 (UTC) (envelope-from roy@marples.name) Received: from sender-of-o58.zoho.eu (sender-of-o58.zoho.eu [136.143.169.58]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4VChtl16BJz4L44 for ; Mon, 8 Apr 2024 08:23:26 +0000 (UTC) (envelope-from roy@marples.name) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; t=1712564603; cv=none; d=zohomail.eu; s=zohoarc; b=fZdM9X+FLudS167JH/RvbQP4RL3sz08rMV/ZVSW/p06Dn6Uh1b81KLBn/1FsiUzCSMvM7OnePji+BqQIOHfG3cCMcTaU7XPeMIwiV86UrYU+6uMY43U4hytRlKdIEZAHqyV5cdWRPBiYipSLStd481r8rwSKq7Gcn7SdmUaB4Es= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.eu; s=zohoarc; t=1712564603; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To; bh=5R1w0LInmTp1SRasaFDYFeIPSxFjZMheLh3Ae6Dsn6U=; b=FgC4oHRXA7mJz+iFrrT93ONRzufIvszW7GV+aDDdTd8d7RoNERKr1ODhIvJDjkFQ+X1O/0zjX1yDY+FvGmGpjvs5WxQIzibuE8g4gMBg5mddmd2YlmzSdJ2dinxN/nzt8H0gVMt9BhwFZ+U5De+1+DfQ1QNax2p143+00W155iA= ARC-Authentication-Results: i=1; mx.zohomail.eu; dkim=pass header.i=marples.name; spf=pass smtp.mailfrom=roy@marples.name; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1712564603; s=zmail; d=marples.name; i=roy@marples.name; h=Date:Date:From:From:To:To:Cc:Cc:Message-ID:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id:Reply-To; bh=5R1w0LInmTp1SRasaFDYFeIPSxFjZMheLh3Ae6Dsn6U=; b=aDMUU4Dln8wxAWs2Us9ULTuilnPM5DFG+Q0SeTDF9jflKDO4tNDiIH3speXm0D5l bn1PyIL774WqSBv6TOdNmonZG8KAt1dWlD5VubjSc5QasKhWNSQ+cnkO/rckpSl8zbL QLW+X5eyvPcotz4eC/XzrGPMWL9Gyo/st+mu33O0= Received: from mail.zoho.eu by mx.zoho.eu with SMTP id 171256460135357.460271678040385; Mon, 8 Apr 2024 10:23:21 +0200 (CEST) Date: Mon, 08 Apr 2024 09:23:21 +0100 From: Roy Marples To: "Anton Yudin" Cc: "freebsd-net" Message-ID: <18ebcce010c.c4bc5178125082.198975928787124826@marples.name> In-Reply-To: References: Subject: Re: How to ignore a default route for one of the dhclient-ed interface? List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:41913, ipnet:136.143.168.0/23, country:CH] X-Rspamd-Queue-Id: 4VChtl16BJz4L44 ---- On Mon, 08 Apr 2024 04:16:47 +0100 Anton Yudin wrote ---=20 > =C2=A0 I'm running a FreeBSD 14 with two interfaces that use DHCP.=C2=A0= I would like to make one of the interfaces to never set the default route.= =C2=A0 Right now the first interface to be fully up sets the default route. >=20 > =C2=A0 I tried to set the following in /etc/dhclient.conf > ---------------8=C2=A0 interface "wan1" {=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ignore routers;=C2=A0 }---------------8=C2=A0 but the default route still g= ets set. > =C2=A0 I ended up creating a /etc/dhclient-enter-hooks with a very hacky= code that overrides the "route" command:---------------8<-----------------= ------- > route() { > =C2=A0 =C2=A0 =C2=A0 =C2=A0 if [ "X$interface" =3D "Xwan1" -a "X$2" =3D = "Xdefault" =C2=A0]; then > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 echo "ignore rou= te $1 $2 $3 $4" | logger -t "enter-hooks" > =C2=A0 =C2=A0 =C2=A0 =C2=A0 else > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /sbin/route $1 $= 2 $3 $4 > =C2=A0 =C2=A0 =C2=A0 =C2=A0 fi > }---------------8 > Is there a better way of doing this? Yes there is. You can use dhcpcd (available in ports) instead where you can prefer an int= erface by metric. When both interfaces have a DHCP lease, dhcpcd will prefer the lowest metri= c and adjust routing and everything else accordingly. Assuming you have wan0 and wan1 and are ignoring wan1 the chances are that = dhcpcd will just work for you out of the box. Otherwise you can tell dhcpcd your preference like so: interface eth0 metric 1 Here, eth0 will take preference other any other interface. OR You can also remove the option from the DHCP packet before processing interface wan1 nooption routers Hope this helps Roy From nobody Mon Apr 8 17:31:27 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VCx331Qkhz5HPdL for ; Mon, 8 Apr 2024 17:31:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VCx326gPdz4T5v for ; Mon, 8 Apr 2024 17:31:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1712597486; a=rsa-sha256; cv=none; b=tjjbeyFL8pGZiR8sXzkYS6LifXmFVxf+CH+dvttlW2EuMjzORhVxhSFSABf/xEaYzXKJSs 5l4I0ieLs/bLxmsCjvUTwN50l3EhMyXmZAEm7n6QBxRRDM6Khv2o/A6MTW5909YiJuwU3D xHFta9kBCaM9WwKzi3LyronYMDqRBQeUYdF3rLdY13W+Y+vthzpXCiAz0s56bJ8jQBoS9G YqSbG/OjcGLupWariN20KYHpAIxnd1zJzgcrEeRsx5Af9PhHZH4Z5/gcmRSQ5umhCEZK4r g7OAdaregQieydBE+wAiwsO0us9Fm1mRkRpZyTKH12nUemIgs6ZR5yUbBIFiAQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1712597486; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JLrk/fwawpa5PePK0iuBNoPUWVXFbZQfjO8WY540W/o=; b=q4DDdhVkFefmtoy0YaaTEivKqxMj6hoFcjeCBu2ly0pWXS03u7FXFlOwRCyR884t/50NrE klZe9kSD9fnwuy2oDYiOeQI67BN9UQJkrUjQkGPfj/2ze47rDepzgLNO0nclflTksPbrR6 iIgTPkx1sIiMvLcvG6yKDycXPEemuORw0m/KEHYdkfayI4dSiKBwkigk1GsmRrYme2IEdI 2noX3Iff4NozKkXVoYZkOImKK7pavQGX4OmcXZjbFH87Pdl+3CKf6kCc+EXvOqV2nd+4d9 Zw1lEMMQFdL2Z4XTej9Zn7Q/2mZk30oaHZQoZb2qumcRy6ViB32B/Nl1WIuoRA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4VCx326GPXzLHm for ; Mon, 8 Apr 2024 17:31:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 438HVQtw069642 for ; Mon, 8 Apr 2024 17:31:26 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 438HVQMP069641 for net@FreeBSD.org; Mon, 8 Apr 2024 17:31:26 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 278220] gre: route add -iface invalid argument Date: Mon, 08 Apr 2024 17:31:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D278220 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |net@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Apr 10 11:40:16 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VF1986KPBz5GxMl; Wed, 10 Apr 2024 11:40:28 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VF1985JrBz4JTh; Wed, 10 Apr 2024 11:40:28 +0000 (UTC) (envelope-from eduardo@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1712749228; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=l8vh42v6fmv7XhH8SXV5MwwLBBKllEko3xorlPVnaYw=; b=MDOBjzYfktSQk+MMYo2jokuffnXVLtbqgRQcES8s45ZMOcwvzRpZavXfBunXZXOOQrtbZL 4Hc95bCv9vSP5T4T12Ui4wG1GQfPYaWsjkxweFqYyDOj1PcnEgcS6582b8v7jAgzUHqtZN fixgZLSBkBRpjIE+zZ+fuWvpH8JwyS1v1SrP8xRAc+jB4W4g474TViRROzC6A8GFfUuvmZ a7gaCUl0kKqU8c3wRkXVpvXTmYsEWYMl/OKMiiZMDVlkjeP9pbxFrCpLVFBxX7KfbtMvFy h+wBobI1K7ImLdU7pIwAxDrkUMc60awUtQHhoKo8oBI5I8Vec/V+ShnV+hN8sA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1712749228; a=rsa-sha256; cv=none; b=h4t2wQB586gYwqpt9duoGHK8t7x1neb6IewtBZUwOhMaI/yOOnAHmIlsGjWinFt6nM74V0 dMbJgvaHPhJa2UVUneUG9Zq3eh5wD0LL7ZVSvhcLiDb1U2kswtOXCh3magOkMf19feL6vr wQ8kI/QUpbmOTCIWCj1kwVa6Um4+UNuWnHQluSJf6ZvAxf994I6JMam/kQwEnXfVJiGITI qH0+FhtonCweyfXO13EbtJa5C/xSdFU6hMTxt2v6PN1NU4iYzZoj3hBmnICY70whHhgRmD UUrA2uiBjQJ8L6yIBhZCKNXub23gBw9tkLRmEp5E+HIW7PpT6tVB7HqWnhOhMg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1712749228; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=l8vh42v6fmv7XhH8SXV5MwwLBBKllEko3xorlPVnaYw=; b=VVz9aRM8f3ZHdVbFR/l8ETNeZdhJ4SeDNaAbINRYIMm+A4nbWqOhxgjDj7S+9NYM2Ai/m2 FmivSJfLOkg31ypgw4dd2ct2VkkHajPl+wuzvZWa93t+qgnAxkvblDfeGt5oyUNcMWk9c6 bi60GJyX+DVXKoPPlAoB0zMlbX78BU7ut6XH3jdUhm1OhZNvGsK7TyEZgmW9BVh1Zo9eqx CddWeGNvLPKSfMub+K44qabYCir9j5J/3aphg8lmebct2M4nuNu/5886wGfQuLkEZHUl2K 5KrifwBC4QVtdN/Eq8uWGarlcyhkJAcjNwNEN8IN72e98RbfNDDNb6sMiAodnQ== Received: from mail-yb1-f181.google.com (mail-yb1-f181.google.com [209.85.219.181]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: eduardo) by smtp.freebsd.org (Postfix) with ESMTPSA id 4VF1984kkTzLQn; Wed, 10 Apr 2024 11:40:28 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: by mail-yb1-f181.google.com with SMTP id 3f1490d57ef6-dc236729a2bso6308233276.0; Wed, 10 Apr 2024 04:40:28 -0700 (PDT) X-Forwarded-Encrypted: i=1; AJvYcCUpEzH1bSXatb9OyFDiw2rmdOiCyQKZ3dap3T9svyfGL/ivBCLS183bJQZJfrnqdM+3CGyZd6TpMhWOavbM1mcWCoL4ftWmMTJYmTwhEqq/td+9dviUhQdDP7ii/noZMbGMBJ2z8SmcV8ELkkt1NBe/qrf0tzkn X-Gm-Message-State: AOJu0Yw8nglrE/rdqV/6XvLzStCInUG+FqpwODverstCQRlIESB7tRF7 5xZo1upotbQ8ueW8PdTbLa11djf3U7A0zAHI0+EvV4Hi0rjkaXMsyaMjsT+W42MlCBGvhObyARV 8QZSv09uzuvcCNxYEgFlD+t2gyZk= X-Google-Smtp-Source: AGHT+IE9OgPBrk3nu4n44r3WDLgcsQdARQYYW6dMCmekc+eLpEAQo51FYbRxDz80TYeEtiOgK5hh+52bpEh64hqiEpc= X-Received: by 2002:a25:d842:0:b0:dcf:bf94:bc30 with SMTP id p63-20020a25d842000000b00dcfbf94bc30mr2503007ybg.34.1712749227563; Wed, 10 Apr 2024 04:40:27 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 References: <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> <6047C8EF-B1B0-4286-93FA-AA38F8A18656@karels.net> <8031cd99-ded8-4b06-93b3-11cc729a8b2c@app.fastmail.com> <38c54399-6c96-44d8-a3a2-3cc1bfbe50c2@app.fastmail.com> <27d8144f-0658-46f6-b8f3-35eb60061644@lakerest.net> <5C9863F7-0F1C-4D02-9F6D-9DDC5FBEB368@freebsd.org> In-Reply-To: <5C9863F7-0F1C-4D02-9F6D-9DDC5FBEB368@freebsd.org> From: Nuno Teixeira Date: Wed, 10 Apr 2024 12:40:16 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Request for Testing: TCP RACK To: tuexen@freebsd.org Cc: Drew Gallatin , Konstantin Belousov , rrs , Mike Karels , garyj@gmx.de, current@freebsd.org, net@freebsd.org, Randall Stewart Content-Type: multipart/alternative; boundary="0000000000004c6d020615bc81cf" --0000000000004c6d020615bc81cf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello all, @ current 1500018 and fetching torrents with net-p2p/qbittorrent finished ~2GB download and connection UP until the end: --- Apr 10 11:26:46 leg kernel: re0: watchdog timeout Apr 10 11:26:46 leg kernel: re0: link state changed to DOWN Apr 10 11:26:49 leg dhclient[58810]: New IP Address (re0): 192.168.1.67 Apr 10 11:26:49 leg dhclient[58814]: New Subnet Mask (re0): 255.255.255.0 Apr 10 11:26:49 leg dhclient[58818]: New Broadcast Address (re0): 192.168.1.255 Apr 10 11:26:49 leg kernel: re0: link state changed to UP Apr 10 11:26:49 leg dhclient[58822]: New Routers (re0): 192.168.1.1 --- In the past tests, I've got more watchdog timeouts, connection goes down and a reboot needed to put it back (`service netif restart` didn't work). Other way to reproduce this is using sysutils/restic (backup program) to read/check all files from a remote server via sftp: `restic -r sftp:user@remote:restic-repo check --read-data` from a 60GB compressed backup. --- watchdog timeout x3 as above --- restic check fail log @ 15% progress: --- Load(, 17310001, 0) returned error, retrying after 1.7670599s: connection lost Load(, 17456892, 0) returned error, retrying after 4.619104908s: connection lost Load(, 17310001, 0) returned error, retrying after 5.477648517s: connection lost List(lock) returned error, retrying after 293.057766ms: connection lost List(lock) returned error, retrying after 385.206693ms: connection lost List(lock) returned error, retrying after 1.577594281s: connection lost Connection continues UP. Cheers, escreveu (quinta, 28/03/2024 =C3=A0(s) 15:53): > > On 28. Mar 2024, at 15:00, Nuno Teixeira wrote: > > > > Hello all! > > > > Running rack @b7b78c1c169 "Optimize HPTS..." very happy on my laptop > (amd64)! > > > > Thanks all! > Thanks for the feedback! > > Best regards > Michael > > > > Drew Gallatin escreveu (quinta, 21/03/2024 =C3= =A0(s) > 12:58): > > The entire point is to *NOT* go through the overhead of scheduling > something asynchronously, but to take advantage of the fact that a > user/kernel transition is going to trash the cache anyway. > > > > In the common case of a system which has less than the threshold numbe= r > of connections , we access the tcp_hpts_softclock function pointer, make > one function call, and access hpts_that_need_softclock, and then return. > So that's 2 variables and a function call. > > > > I think it would be preferable to avoid that call, and to move the > declaration of tcp_hpts_softclock and hpts_that_need_softclock so that th= ey > are in the same cacheline. Then we'd be hitting just a single line in th= e > common case. (I've made comments on the review to that effect). > > > > Also, I wonder if the threshold could get higher by default, so that > hpts is never called in this context unless we're to the point where we'r= e > scheduling thousands of runs of the hpts thread (and taking all those clo= ck > interrupts). > > > > Drew > > > > On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: > >> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: > >>> Ok I have created > >>> > >>> https://reviews.freebsd.org/D44420 > >>> > >>> > >>> To address the issue. I also attach a short version of the patch that > Nuno > >>> can try and validate > >>> > >>> it works. Drew you may want to try this and validate the optimization > does > >>> kick in since I can > >>> > >>> only now test that it does not on my local box :) > >> The patch still causes access to all cpu's cachelines on each userret. > >> It would be much better to inc/check the threshold and only schedule t= he > >> call when exceeded. Then the call can occur in some dedicated context= , > >> like per-CPU thread, instead of userret. > >> > >>> > >>> > >>> R > >>> > >>> > >>> > >>> On 3/18/24 3:42 PM, Drew Gallatin wrote: > >>>> No. The goal is to run on every return to userspace for every threa= d. > >>>> > >>>> Drew > >>>> > >>>> On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: > >>>>> On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: > >>>>>> I got the idea from > >>>>>> > https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf > >>>>>> The gist is that the TCP pacing stuff needs to run frequently, and > >>>>>> rather than run it out of a clock interrupt, its more efficient to > run > >>>>>> it out of a system call context at just the point where we return = to > >>>>>> userspace and the cache is trashed anyway. The current > implementation > >>>>>> is fine for our workload, but probably not idea for a generic > system. > >>>>>> Especially one where something is banging on system calls. > >>>>>> > >>>>>> Ast's could be the right tool for this, but I'm super unfamiliar > with > >>>>>> them, and I can't find any docs on them. > >>>>>> > >>>>>> Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent = to > >>>>>> what's happening here? > >>>>> This call would need some AST number added, and then it registers t= he > >>>>> ast to run on next return to userspace, for the current thread. > >>>>> > >>>>> Is it enough? > >>>>>> > >>>>>> Drew > >>>>> > >>>>>> > >>>>>> On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: > >>>>>>> On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > >>>>>>>> On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote: > >>>>>>>> > >>>>>>>>>> On 18. Mar 2024, at 12:42, Nuno Teixeira > >>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Hello all! > >>>>>>>>>> > >>>>>>>>>> It works just fine! > >>>>>>>>>> System performance is OK. > >>>>>>>>>> Using patch on main-n268841-b0aaf8beb126(-dirty). > >>>>>>>>>> > >>>>>>>>>> --- > >>>>>>>>>> net.inet.tcp.functions_available: > >>>>>>>>>> Stack D > >>>>> Alias PCB count > >>>>>>>>>> freebsd freebsd 0 > >>>>>>>>>> rack * > >>>>> rack 38 > >>>>>>>>>> --- > >>>>>>>>>> > >>>>>>>>>> It would be so nice that we can have a sysctl tunnable for > >>>>> this patch > >>>>>>>>>> so we could do more tests without recompiling kernel. > >>>>>>>>> Thanks for testing! > >>>>>>>>> > >>>>>>>>> @gallatin: can you come up with a patch that is acceptable > >>>>> for Netflix > >>>>>>>>> and allows to mitigate the performance regression. > >>>>>>>> > >>>>>>>> Ideally, tcphpts could enable this automatically when it > >>>>> starts to be > >>>>>>>> used (enough?), but a sysctl could select auto/on/off. > >>>>>>> There is already a well-known mechanism to request execution of t= he > >>>>>>> specific function on return to userspace, namely AST. The > difference > >>>>>>> with the current hack is that the execution is requested for one > >>>>> callback > >>>>>>> in the context of the specific thread. > >>>>>>> > >>>>>>> Still, it might be worth a try to use it; what is the reason to > >>>>> hit a thread > >>>>>>> that does not do networking, with TCP processing? > >>>>>>> > >>>>>>>> > >>>>>>>> Mike > >>>>>>>> > >>>>>>>>> Best regards > >>>>>>>>> Michael > >>>>>>>>>> > >>>>>>>>>> Thanks all! > >>>>>>>>>> Really happy here :) > >>>>>>>>>> > >>>>>>>>>> Cheers, > >>>>>>>>>> > >>>>>>>>>> Nuno Teixeira escreveu (domingo, > >>>>> 17/03/2024 =C3=A0(s) 20:26): > >>>>>>>>>>> > >>>>>>>>>>> Hello, > >>>>>>>>>>> > >>>>>>>>>>>> I don't have the full context, but it seems like the > >>>>> complaint is a performance regression in bonnie++ and perhaps other > >>>>> things when tcp_hpts is loaded, even when it is not used. Is that > >>>>> correct? > >>>>>>>>>>>> > >>>>>>>>>>>> If so, I suspect its because we drive the > >>>>> tcp_hpts_softclock() routine from userret(), in order to avoid tons > >>>>> of timer interrupts and context switches. To test this theory, yo= u > >>>>> could apply a patch like: > >>>>>>>>>>> > >>>>>>>>>>> It's affecting overall system performance, bonnie was just > >>>>> a way to > >>>>>>>>>>> get some numbers to compare. > >>>>>>>>>>> > >>>>>>>>>>> Tomorrow I will test patch. > >>>>>>>>>>> > >>>>>>>>>>> Thanks! > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Nuno Teixeira > >>>>>>>>>>> FreeBSD Committer (ports) > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Nuno Teixeira > >>>>>>>>>> FreeBSD Committer (ports) > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > >>> diff --git a/sys/netinet/tcp_hpts.c b/sys/netinet/tcp_hpts.c > >>> index 8c4d2d41a3eb..eadbee19f69c 100644 > >>> --- a/sys/netinet/tcp_hpts.c > >>> +++ b/sys/netinet/tcp_hpts.c > >>> @@ -216,6 +216,7 @@ struct tcp_hpts_entry { > >>> void *ie_cookie; > >>> uint16_t p_num; /* The hpts number one per cpu */ > >>> uint16_t p_cpu; /* The hpts CPU */ > >>> + uint8_t hit_callout_thresh; > >>> /* There is extra space in here */ > >>> /* Cache line 0x100 */ > >>> struct callout co __aligned(CACHE_LINE_SIZE); > >>> @@ -269,6 +270,11 @@ static struct hpts_domain_info { > >>> int cpu[MAXCPU]; > >>> } hpts_domains[MAXMEMDOM]; > >>> > >>> +counter_u64_t hpts_that_need_softclock; > >>> +SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, needsoftclock= , > CTLFLAG_RD, > >>> + &hpts_that_need_softclock, > >>> + "Number of hpts threads that need softclock"); > >>> + > >>> counter_u64_t hpts_hopelessly_behind; > >>> > >>> SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, hopeless, > CTLFLAG_RD, > >>> @@ -334,7 +340,7 @@ SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, > precision, CTLFLAG_RW, > >>> &tcp_hpts_precision, 120, > >>> "Value for PRE() precision of callout"); > >>> SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, cnt_thresh, CTLFLAG_RW, > >>> - &conn_cnt_thresh, 0, > >>> + &conn_cnt_thresh, DEFAULT_CONNECTION_THESHOLD, > >>> "How many connections (below) make us use the callout based > mechanism"); > >>> SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, logging, CTLFLAG_RW, > >>> &hpts_does_tp_logging, 0, > >>> @@ -1548,6 +1554,9 @@ __tcp_run_hpts(void) > >>> struct tcp_hpts_entry *hpts; > >>> int ticks_ran; > >>> > >>> + if (counter_u64_fetch(hpts_that_need_softclock) =3D=3D 0) > >>> + return; > >>> + > >>> hpts =3D tcp_choose_hpts_to_run(); > >>> > >>> if (hpts->p_hpts_active) { > >>> @@ -1683,6 +1692,13 @@ tcp_hpts_thread(void *ctx) > >>> ticks_ran =3D tcp_hptsi(hpts, 1); > >>> tv.tv_sec =3D 0; > >>> tv.tv_usec =3D hpts->p_hpts_sleep_time * HPTS_TICKS_PER_SLOT; > >>> + if ((hpts->p_on_queue_cnt > conn_cnt_thresh) && > (hpts->hit_callout_thresh =3D=3D 0)) { > >>> + hpts->hit_callout_thresh =3D 1; > >>> + counter_u64_add(hpts_that_need_softclock, 1); > >>> + } else if ((hpts->p_on_queue_cnt <=3D conn_cnt_thresh) && > (hpts->hit_callout_thresh =3D=3D 1)) { > >>> + hpts->hit_callout_thresh =3D 0; > >>> + counter_u64_add(hpts_that_need_softclock, -1); > >>> + } > >>> if (hpts->p_on_queue_cnt >=3D conn_cnt_thresh) { > >>> if(hpts->p_direct_wake =3D=3D 0) { > >>> /* > >>> @@ -1818,6 +1834,7 @@ tcp_hpts_mod_load(void) > >>> cpu_top =3D NULL; > >>> #endif > >>> tcp_pace.rp_num_hptss =3D ncpus; > >>> + hpts_that_need_softclock =3D counter_u64_alloc(M_WAITOK); > >>> hpts_hopelessly_behind =3D counter_u64_alloc(M_WAITOK); > >>> hpts_loops =3D counter_u64_alloc(M_WAITOK); > >>> back_tosleep =3D counter_u64_alloc(M_WAITOK); > >>> @@ -2042,6 +2059,7 @@ tcp_hpts_mod_unload(void) > >>> free(tcp_pace.grps, M_TCPHPTS); > >>> #endif > >>> > >>> + counter_u64_free(hpts_that_need_softclock); > >>> counter_u64_free(hpts_hopelessly_behind); > >>> counter_u64_free(hpts_loops); > >>> counter_u64_free(back_tosleep); > >> > >> > > > > > > > > -- > > Nuno Teixeira > > FreeBSD Committer (ports) > > --=20 Nuno Teixeira FreeBSD Committer (ports) --0000000000004c6d020615bc81cf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello all,

@ current 1500018= and fetching torrents with net-p2p/qbittorrent finished ~2GB download and = connection UP until the end:

---
Apr 10 11:26:46 leg kernel: re0: watchdog timeout
Apr 10 11:26:46 leg = kernel: re0: link state changed to DOWN
Apr 10 11:26:49 leg dhclient[588= 10]: New IP Address (re0): 192.168.1.67
Apr 10 11:26:49 leg dhclient[588= 14]: New Subnet Mask (re0): 255.255.255.0
Apr 10 11:26:49 leg dhclient[5= 8818]: New Broadcast Address (re0): 192.168.1.255
Apr 10 11:26:49 leg ke= rnel: re0: link state changed to UP
Apr 10 11:26:49 leg dhclient[58822]:= New Routers (re0): 192.168.1.1
---

In t= he past tests, I've got more watchdog timeouts, connection goes down an= d a reboot needed to put it back (`service netif restart` didn't work).=

Other way to reproduce this is using sysutils/res= tic (backup program) to read/check all files from a remote server via sftp:=

`restic -r sftp:user@remote:restic-repo check --r= ead-data` from a 60GB compressed backup.

---
=
watchdog timeout x3 as above
---

re= stic check fail log @ 15% progress:
---
<snip>= ;
Load(<data/52e2923dd6>, 17310001, 0) returned error, = retrying after 1.7670599s: connection lost
Load(<data/d27a0abe0f>,= 17456892, 0) returned error, retrying after 4.619104908s: connection lost<= br>Load(<data/52e2923dd6>, 17310001, 0) returned error, retrying afte= r 5.477648517s: connection lost
List(lock) returned error, retrying afte= r 293.057766ms: connection lost
List(lock) returned error, retrying afte= r 385.206693ms: connection lost
List(lock) returned error, retrying afte= r 1.577594281s: connection lost
<snip>

=
Connection continues UP.

Cheers,

<= ;tuexen@freebsd.org= > escreveu (quinta, 28/03/2024 =C3=A0(s) 15:53):
> On 28. Mar 2024, at 15:00, Nun= o Teixeira <edu= ardo@freebsd.org> wrote:
>
> Hello all!
>
> Running rack @b7b78c1c169 "Optimize HPTS..." very happy on m= y laptop (amd64)!
>
> Thanks all!
Thanks for the feedback!

Best regards
Michael
>
> Drew Gallatin <gallatin@freebsd.org> escreveu (quinta, 21/03/2024 =C3=A0(s) 1= 2:58):
> The entire point is to *NOT* go through the overhead of scheduling som= ething asynchronously, but to take advantage of the fact that a user/kernel= transition is going to trash the cache anyway.
>
> In the common case of a system which has less than the threshold=C2=A0= number of connections , we access the tcp_hpts_softclock function pointer,= make one function call, and access hpts_that_need_softclock, and then retu= rn.=C2=A0 So that's 2 variables and a function call.
>
> I think it would be preferable to avoid that call, and to move the dec= laration of tcp_hpts_softclock and hpts_that_need_softclock so that they ar= e in the same cacheline.=C2=A0 Then we'd be hitting just a single line = in the common case.=C2=A0 (I've made comments on the review to that eff= ect).
>
> Also, I wonder if the threshold could get higher by default, so that h= pts is never called in this context unless we're to the point where we&= #39;re scheduling thousands of runs of the hpts thread (and taking all thos= e clock interrupts).
>
> Drew
>
> On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote:
>> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote:
>>> Ok I have created
>>>
>>> https://reviews.freebsd.org/D44420
>>>
>>>
>>> To address the issue. I also attach a short version of the pat= ch that Nuno
>>> can try and validate
>>>
>>> it works. Drew you may want to try this and validate the optim= ization does
>>> kick in since I can
>>>
>>> only now test that it does not on my local box :)
>> The patch still causes access to all cpu's cachelines on each = userret.
>> It would be much better to inc/check the threshold and only schedu= le the
>> call when exceeded.=C2=A0 Then the call can occur in some dedicate= d context,
>> like per-CPU thread, instead of userret.
>>
>>>
>>>
>>> R
>>>
>>>
>>>
>>> On 3/18/24 3:42 PM, Drew Gallatin wrote:
>>>> No.=C2=A0 The goal is to run on every return to userspace = for every thread.
>>>>
>>>> Drew
>>>>
>>>> On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrot= e:
>>>>> On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallati= n wrote:
>>>>>> I got the idea from
>>>>>> https= ://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf
>>>>>> The gist is that the TCP pacing stuff needs to run= frequently, and
>>>>>> rather than run it out of a clock interrupt, its m= ore efficient to run
>>>>>> it out of a system call context at just the point = where we return to
>>>>>> userspace and the cache is trashed anyway. The cur= rent implementation
>>>>>> is fine for our workload, but probably not idea fo= r a generic system.
>>>>>> Especially one where something is banging on syste= m calls.
>>>>>>
>>>>>> Ast's could be the right tool for this, but I&= #39;m super unfamiliar with
>>>>>> them, and I can't find any docs on them.
>>>>>>
>>>>>> Would ast_register(0, ASTR_UNCOND, 0, func) be rou= ghly equivalent to
>>>>>> what's happening here?
>>>>> This call would need some AST number added, and then i= t registers the
>>>>> ast to run on next return to userspace, for the curren= t thread.
>>>>>
>>>>> Is it enough?
>>>>>>
>>>>>> Drew
>>>>>
>>>>>>
>>>>>> On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belou= sov wrote:
>>>>>>> On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike= Karels wrote:
>>>>>>>> On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote:
>>>>>>>>
>>>>>>>>>> On 18. Mar 2024, at 12:42, Nuno Te= ixeira
>>>>> <eduardo@freebsd.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Hello all!
>>>>>>>>>>
>>>>>>>>>> It works just fine!
>>>>>>>>>> System performance is OK.
>>>>>>>>>> Using patch on main-n268841-b0aaf8= beb126(-dirty).
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> net.inet.tcp.functions_available:<= br> >>>>>>>>>> Stack=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0D
>>>>> Alias=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 PCB count
>>>>>>>>>> freebsd freebsd=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=
>>>>>>>>>> rack=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 *
>>>>> rack=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A038
>>>>>>>>>> ---
>>>>>>>>>>
>>>>>>>>>> It would be so nice that we can ha= ve a sysctl tunnable for
>>>>> this patch
>>>>>>>>>> so we could do more tests without = recompiling kernel.
>>>>>>>>> Thanks for testing!
>>>>>>>>>
>>>>>>>>> @gallatin: can you come up with a patc= h that is acceptable
>>>>> for Netflix
>>>>>>>>> and allows to mitigate the performance= regression.
>>>>>>>>
>>>>>>>> Ideally, tcphpts could enable this automat= ically when it
>>>>> starts to be
>>>>>>>> used (enough?), but a sysctl could select = auto/on/off.
>>>>>>> There is already a well-known mechanism to req= uest execution of the
>>>>>>> specific function on return to userspace, name= ly AST.=C2=A0 The difference
>>>>>>> with the current hack is that the execution is= requested for one
>>>>> callback
>>>>>>> in the context of the specific thread.
>>>>>>>
>>>>>>> Still, it might be worth a try to use it; what= is the reason to
>>>>> hit a thread
>>>>>>> that does not do networking, with TCP processi= ng?
>>>>>>>
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>>> Best regards
>>>>>>>>> Michael
>>>>>>>>>>
>>>>>>>>>> Thanks all!
>>>>>>>>>> Really happy here :)
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Nuno Teixeira <eduardo@freebsd.org> escrev= eu (domingo,
>>>>> 17/03/2024 =C3=A0(s) 20:26):
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>>> I don't have the full = context, but it seems like the
>>>>> complaint is a performance regression in bonnie++ and = perhaps other
>>>>> things when tcp_hpts is loaded, even when it is not us= ed.=C2=A0 Is that
>>>>> correct?
>>>>>>>>>>>>
>>>>>>>>>>>> If so, I suspect its becau= se we drive the
>>>>> tcp_hpts_softclock() routine from userret(), in order = to avoid tons
>>>>> of timer interrupts and context switches.=C2=A0 To tes= t this theory,=C2=A0 you
>>>>> could apply a patch like:
>>>>>>>>>>>
>>>>>>>>>>> It's affecting overall sys= tem performance, bonnie was just
>>>>> a way to
>>>>>>>>>>> get some numbers to compare. >>>>>>>>>>>
>>>>>>>>>>> Tomorrow I will test patch. >>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Nuno Teixeira
>>>>>>>>>>> FreeBSD Committer (ports)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Nuno Teixeira
>>>>>>>>>> FreeBSD Committer (ports)
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>
>>> diff --git a/sys/netinet/tcp_hpts.c b/sys/netinet/tcp_hpts.c >>> index 8c4d2d41a3eb..eadbee19f69c 100644
>>> --- a/sys/netinet/tcp_hpts.c
>>> +++ b/sys/netinet/tcp_hpts.c
>>> @@ -216,6 +216,7 @@ struct tcp_hpts_entry {
>>> void *ie_cookie;
>>> uint16_t p_num; /* The hpts number one per cpu */
>>> uint16_t p_cpu; /* The hpts CPU */
>>> + uint8_t hit_callout_thresh;
>>> /* There is extra space in here */
>>> /* Cache line 0x100 */
>>> struct callout co __aligned(CACHE_LINE_SIZE);
>>> @@ -269,6 +270,11 @@ static struct hpts_domain_info {
>>> int cpu[MAXCPU];
>>> } hpts_domains[MAXMEMDOM];
>>>
>>> +counter_u64_t hpts_that_need_softclock;
>>> +SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, needso= ftclock, CTLFLAG_RD,
>>> +=C2=A0 =C2=A0 &hpts_that_need_softclock,
>>> +=C2=A0 =C2=A0 "Number of hpts threads that need softcloc= k");
>>> +
>>> counter_u64_t hpts_hopelessly_behind;
>>>
>>> SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, hopeles= s, CTLFLAG_RD,
>>> @@ -334,7 +340,7 @@ SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, p= recision, CTLFLAG_RW,
>>>=C2=A0 =C2=A0 =C2=A0&tcp_hpts_precision, 120,
>>>=C2=A0 =C2=A0 =C2=A0"Value for PRE() precision of callout&= quot;);
>>> SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, cnt_thresh, CTLFLAG_R= W,
>>> -=C2=A0 =C2=A0 &conn_cnt_thresh, 0,
>>> +=C2=A0 =C2=A0 &conn_cnt_thresh, DEFAULT_CONNECTION_THESHO= LD,
>>>=C2=A0 =C2=A0 =C2=A0"How many connections (below) make us = use the callout based mechanism");
>>> SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, logging, CTLFLAG_RW,<= br> >>>=C2=A0 =C2=A0 =C2=A0&hpts_does_tp_logging, 0,
>>> @@ -1548,6 +1554,9 @@ __tcp_run_hpts(void)
>>> struct tcp_hpts_entry *hpts;
>>> int ticks_ran;
>>>
>>> + if (counter_u64_fetch(hpts_that_need_softclock) =3D=3D 0) >>> + return;
>>> +
>>> hpts =3D tcp_choose_hpts_to_run();
>>>
>>> if (hpts->p_hpts_active) {
>>> @@ -1683,6 +1692,13 @@ tcp_hpts_thread(void *ctx)
>>> ticks_ran =3D tcp_hptsi(hpts, 1);
>>> tv.tv_sec =3D 0;
>>> tv.tv_usec =3D hpts->p_hpts_sleep_time * HPTS_TICKS_PER_SLO= T;
>>> + if ((hpts->p_on_queue_cnt > conn_cnt_thresh) &&= ; (hpts->hit_callout_thresh =3D=3D 0)) {
>>> + hpts->hit_callout_thresh =3D 1;
>>> + counter_u64_add(hpts_that_need_softclock, 1);
>>> + } else if ((hpts->p_on_queue_cnt <=3D conn_cnt_thresh)= && (hpts->hit_callout_thresh =3D=3D 1)) {
>>> + hpts->hit_callout_thresh =3D 0;
>>> + counter_u64_add(hpts_that_need_softclock, -1);
>>> + }
>>> if (hpts->p_on_queue_cnt >=3D conn_cnt_thresh) {
>>> if(hpts->p_direct_wake =3D=3D 0) {
>>> /*
>>> @@ -1818,6 +1834,7 @@ tcp_hpts_mod_load(void)
>>> cpu_top =3D NULL;
>>> #endif
>>> tcp_pace.rp_num_hptss =3D ncpus;
>>> + hpts_that_need_softclock =3D counter_u64_alloc(M_WAITOK); >>> hpts_hopelessly_behind =3D counter_u64_alloc(M_WAITOK);
>>> hpts_loops =3D counter_u64_alloc(M_WAITOK);
>>> back_tosleep =3D counter_u64_alloc(M_WAITOK);
>>> @@ -2042,6 +2059,7 @@ tcp_hpts_mod_unload(void)
>>> free(tcp_pace.grps, M_TCPHPTS);
>>> #endif
>>>
>>> + counter_u64_free(hpts_that_need_softclock);
>>> counter_u64_free(hpts_hopelessly_behind);
>>> counter_u64_free(hpts_loops);
>>> counter_u64_free(back_tosleep);
>>
>>
>
>
>
> --
> Nuno Teixeira
> FreeBSD Committer (ports)



--
Nuno Teixeira
FreeBSD Committ= er (ports)
--0000000000004c6d020615bc81cf-- From nobody Wed Apr 10 12:11:54 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VF1sd6DPgz5H1cg; Wed, 10 Apr 2024 12:12:05 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VF1sd378Cz4MhP; Wed, 10 Apr 2024 12:12:05 +0000 (UTC) (envelope-from tuexen@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (unknown [IPv6:2a02:8109:1140:c3d:fdb8:ab7b:81de:24b7]) (Authenticated sender: micmac) by drew.franken.de (Postfix) with ESMTPSA id B916C7220B801; Wed, 10 Apr 2024 14:11:54 +0200 (CEST) Content-Type: text/plain; charset=utf-8 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.500.171.1.1\)) Subject: Re: Request for Testing: TCP RACK From: tuexen@freebsd.org In-Reply-To: Date: Wed, 10 Apr 2024 14:11:54 +0200 Cc: Drew Gallatin , Konstantin Belousov , rrs , Mike Karels , garyj@gmx.de, current@freebsd.org, net@freebsd.org, Randall Stewart Content-Transfer-Encoding: quoted-printable Message-Id: <52479AA6-04F6-4D4A-ABE0-7142B47E28DF@freebsd.org> References: <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> <6047C8EF-B1B0-4286-93FA-AA38F8A18656@karels.net> <8031cd99-ded8-4b06-93b3-11cc729a8b2c@app.fastmail.com> <38c54399-6c96-44d8-a3a2-3cc1bfbe50c2@app.fastmail.com> <27d8144f-0658-46f6-b8f3-35eb60061644@lakerest.net> <5C9863F7-0F1C-4D02-9F6D-9DDC5FBEB368@freebsd.org> To: Nuno Teixeira X-Mailer: Apple Mail (2.3774.500.171.1.1) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:680, ipnet:2001:638::/32, country:DE] X-Rspamd-Queue-Id: 4VF1sd378Cz4MhP > On 10. Apr 2024, at 13:40, Nuno Teixeira wrote: >=20 > Hello all, >=20 > @ current 1500018 and fetching torrents with net-p2p/qbittorrent = finished ~2GB download and connection UP until the end:=20 >=20 > --- > Apr 10 11:26:46 leg kernel: re0: watchdog timeout > Apr 10 11:26:46 leg kernel: re0: link state changed to DOWN > Apr 10 11:26:49 leg dhclient[58810]: New IP Address (re0): = 192.168.1.67 > Apr 10 11:26:49 leg dhclient[58814]: New Subnet Mask (re0): = 255.255.255.0 > Apr 10 11:26:49 leg dhclient[58818]: New Broadcast Address (re0): = 192.168.1.255 > Apr 10 11:26:49 leg kernel: re0: link state changed to UP > Apr 10 11:26:49 leg dhclient[58822]: New Routers (re0): 192.168.1.1 > --- >=20 > In the past tests, I've got more watchdog timeouts, connection goes = down and a reboot needed to put it back (`service netif restart` didn't = work). >=20 > Other way to reproduce this is using sysutils/restic (backup program) = to read/check all files from a remote server via sftp: >=20 > `restic -r sftp:user@remote:restic-repo check --read-data` from a 60GB = compressed backup. >=20 > --- > watchdog timeout x3 as above > --- >=20 > restic check fail log @ 15% progress: > --- > > Load(, 17310001, 0) returned error, retrying after = 1.7670599s: connection lost > Load(, 17456892, 0) returned error, retrying after = 4.619104908s: connection lost > Load(, 17310001, 0) returned error, retrying after = 5.477648517s: connection lost > List(lock) returned error, retrying after 293.057766ms: connection = lost > List(lock) returned error, retrying after 385.206693ms: connection = lost > List(lock) returned error, retrying after 1.577594281s: connection = lost > >=20 > Connection continues UP. Hi, I'm not sure what the issue is you are reporting. Could you state what behavior you are experiencing with the base stack and with the RACK stack. In particular, what the difference is? Best regards Michael >=20 > Cheers, >=20 > escreveu (quinta, 28/03/2024 =C3=A0(s) 15:53): >> On 28. Mar 2024, at 15:00, Nuno Teixeira wrote: >>=20 >> Hello all! >>=20 >> Running rack @b7b78c1c169 "Optimize HPTS..." very happy on my laptop = (amd64)! >>=20 >> Thanks all! > Thanks for the feedback! >=20 > Best regards > Michael >>=20 >> Drew Gallatin escreveu (quinta, 21/03/2024 = =C3=A0(s) 12:58): >> The entire point is to *NOT* go through the overhead of scheduling = something asynchronously, but to take advantage of the fact that a = user/kernel transition is going to trash the cache anyway. >>=20 >> In the common case of a system which has less than the threshold = number of connections , we access the tcp_hpts_softclock function = pointer, make one function call, and access hpts_that_need_softclock, = and then return. So that's 2 variables and a function call. >>=20 >> I think it would be preferable to avoid that call, and to move the = declaration of tcp_hpts_softclock and hpts_that_need_softclock so that = they are in the same cacheline. Then we'd be hitting just a single line = in the common case. (I've made comments on the review to that effect). >>=20 >> Also, I wonder if the threshold could get higher by default, so that = hpts is never called in this context unless we're to the point where = we're scheduling thousands of runs of the hpts thread (and taking all = those clock interrupts). >>=20 >> Drew >>=20 >> On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: >>> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: >>>> Ok I have created >>>>=20 >>>> https://reviews.freebsd.org/D44420 >>>>=20 >>>>=20 >>>> To address the issue. I also attach a short version of the patch = that Nuno >>>> can try and validate >>>>=20 >>>> it works. Drew you may want to try this and validate the = optimization does >>>> kick in since I can >>>>=20 >>>> only now test that it does not on my local box :) >>> The patch still causes access to all cpu's cachelines on each = userret. >>> It would be much better to inc/check the threshold and only schedule = the >>> call when exceeded. Then the call can occur in some dedicated = context, >>> like per-CPU thread, instead of userret. >>>=20 >>>>=20 >>>>=20 >>>> R >>>>=20 >>>>=20 >>>>=20 >>>> On 3/18/24 3:42 PM, Drew Gallatin wrote: >>>>> No. The goal is to run on every return to userspace for every = thread. >>>>>=20 >>>>> Drew >>>>>=20 >>>>> On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: >>>>>> On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: >>>>>>> I got the idea from >>>>>>> = https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf >>>>>>> The gist is that the TCP pacing stuff needs to run frequently, = and >>>>>>> rather than run it out of a clock interrupt, its more efficient = to run >>>>>>> it out of a system call context at just the point where we = return to >>>>>>> userspace and the cache is trashed anyway. The current = implementation >>>>>>> is fine for our workload, but probably not idea for a generic = system. >>>>>>> Especially one where something is banging on system calls. >>>>>>>=20 >>>>>>> Ast's could be the right tool for this, but I'm super unfamiliar = with >>>>>>> them, and I can't find any docs on them. >>>>>>>=20 >>>>>>> Would ast_register(0, ASTR_UNCOND, 0, func) be roughly = equivalent to >>>>>>> what's happening here? >>>>>> This call would need some AST number added, and then it registers = the >>>>>> ast to run on next return to userspace, for the current thread. >>>>>>=20 >>>>>> Is it enough? >>>>>>>=20 >>>>>>> Drew >>>>>>=20 >>>>>>>=20 >>>>>>> On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: >>>>>>>> On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: >>>>>>>>> On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote: >>>>>>>>>=20 >>>>>>>>>>> On 18. Mar 2024, at 12:42, Nuno Teixeira >>>>>> wrote: >>>>>>>>>>>=20 >>>>>>>>>>> Hello all! >>>>>>>>>>>=20 >>>>>>>>>>> It works just fine! >>>>>>>>>>> System performance is OK. >>>>>>>>>>> Using patch on main-n268841-b0aaf8beb126(-dirty). >>>>>>>>>>>=20 >>>>>>>>>>> --- >>>>>>>>>>> net.inet.tcp.functions_available: >>>>>>>>>>> Stack D >>>>>> Alias PCB count >>>>>>>>>>> freebsd freebsd 0 >>>>>>>>>>> rack * >>>>>> rack 38 >>>>>>>>>>> --- >>>>>>>>>>>=20 >>>>>>>>>>> It would be so nice that we can have a sysctl tunnable for >>>>>> this patch >>>>>>>>>>> so we could do more tests without recompiling kernel. >>>>>>>>>> Thanks for testing! >>>>>>>>>>=20 >>>>>>>>>> @gallatin: can you come up with a patch that is acceptable >>>>>> for Netflix >>>>>>>>>> and allows to mitigate the performance regression. >>>>>>>>>=20 >>>>>>>>> Ideally, tcphpts could enable this automatically when it >>>>>> starts to be >>>>>>>>> used (enough?), but a sysctl could select auto/on/off. >>>>>>>> There is already a well-known mechanism to request execution of = the >>>>>>>> specific function on return to userspace, namely AST. The = difference >>>>>>>> with the current hack is that the execution is requested for = one >>>>>> callback >>>>>>>> in the context of the specific thread. >>>>>>>>=20 >>>>>>>> Still, it might be worth a try to use it; what is the reason to >>>>>> hit a thread >>>>>>>> that does not do networking, with TCP processing? >>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> Mike >>>>>>>>>=20 >>>>>>>>>> Best regards >>>>>>>>>> Michael >>>>>>>>>>>=20 >>>>>>>>>>> Thanks all! >>>>>>>>>>> Really happy here :) >>>>>>>>>>>=20 >>>>>>>>>>> Cheers, >>>>>>>>>>>=20 >>>>>>>>>>> Nuno Teixeira escreveu (domingo, >>>>>> 17/03/2024 =C3=A0(s) 20:26): >>>>>>>>>>>>=20 >>>>>>>>>>>> Hello, >>>>>>>>>>>>=20 >>>>>>>>>>>>> I don't have the full context, but it seems like the >>>>>> complaint is a performance regression in bonnie++ and perhaps = other >>>>>> things when tcp_hpts is loaded, even when it is not used. Is = that >>>>>> correct? >>>>>>>>>>>>>=20 >>>>>>>>>>>>> If so, I suspect its because we drive the >>>>>> tcp_hpts_softclock() routine from userret(), in order to avoid = tons >>>>>> of timer interrupts and context switches. To test this theory, = you >>>>>> could apply a patch like: >>>>>>>>>>>>=20 >>>>>>>>>>>> It's affecting overall system performance, bonnie was just >>>>>> a way to >>>>>>>>>>>> get some numbers to compare. >>>>>>>>>>>>=20 >>>>>>>>>>>> Tomorrow I will test patch. >>>>>>>>>>>>=20 >>>>>>>>>>>> Thanks! >>>>>>>>>>>>=20 >>>>>>>>>>>> -- >>>>>>>>>>>> Nuno Teixeira >>>>>>>>>>>> FreeBSD Committer (ports) >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>> -- >>>>>>>>>>> Nuno Teixeira >>>>>>>>>>> FreeBSD Committer (ports) >>>>>>>>>=20 >>>>>>>>=20 >>>>>>=20 >>>>>=20 >>>=20 >>>> diff --git a/sys/netinet/tcp_hpts.c b/sys/netinet/tcp_hpts.c >>>> index 8c4d2d41a3eb..eadbee19f69c 100644 >>>> --- a/sys/netinet/tcp_hpts.c >>>> +++ b/sys/netinet/tcp_hpts.c >>>> @@ -216,6 +216,7 @@ struct tcp_hpts_entry { >>>> void *ie_cookie; >>>> uint16_t p_num; /* The hpts number one per cpu */ >>>> uint16_t p_cpu; /* The hpts CPU */ >>>> + uint8_t hit_callout_thresh; >>>> /* There is extra space in here */ >>>> /* Cache line 0x100 */ >>>> struct callout co __aligned(CACHE_LINE_SIZE); >>>> @@ -269,6 +270,11 @@ static struct hpts_domain_info { >>>> int cpu[MAXCPU]; >>>> } hpts_domains[MAXMEMDOM]; >>>>=20 >>>> +counter_u64_t hpts_that_need_softclock; >>>> +SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, = needsoftclock, CTLFLAG_RD, >>>> + &hpts_that_need_softclock, >>>> + "Number of hpts threads that need softclock"); >>>> + >>>> counter_u64_t hpts_hopelessly_behind; >>>>=20 >>>> SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, hopeless, = CTLFLAG_RD, >>>> @@ -334,7 +340,7 @@ SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, = precision, CTLFLAG_RW, >>>> &tcp_hpts_precision, 120, >>>> "Value for PRE() precision of callout"); >>>> SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, cnt_thresh, CTLFLAG_RW, >>>> - &conn_cnt_thresh, 0, >>>> + &conn_cnt_thresh, DEFAULT_CONNECTION_THESHOLD, >>>> "How many connections (below) make us use the callout based = mechanism"); >>>> SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, logging, CTLFLAG_RW, >>>> &hpts_does_tp_logging, 0, >>>> @@ -1548,6 +1554,9 @@ __tcp_run_hpts(void) >>>> struct tcp_hpts_entry *hpts; >>>> int ticks_ran; >>>>=20 >>>> + if (counter_u64_fetch(hpts_that_need_softclock) =3D=3D 0) >>>> + return; >>>> + >>>> hpts =3D tcp_choose_hpts_to_run(); >>>>=20 >>>> if (hpts->p_hpts_active) { >>>> @@ -1683,6 +1692,13 @@ tcp_hpts_thread(void *ctx) >>>> ticks_ran =3D tcp_hptsi(hpts, 1); >>>> tv.tv_sec =3D 0; >>>> tv.tv_usec =3D hpts->p_hpts_sleep_time * HPTS_TICKS_PER_SLOT; >>>> + if ((hpts->p_on_queue_cnt > conn_cnt_thresh) && = (hpts->hit_callout_thresh =3D=3D 0)) { >>>> + hpts->hit_callout_thresh =3D 1; >>>> + counter_u64_add(hpts_that_need_softclock, 1); >>>> + } else if ((hpts->p_on_queue_cnt <=3D conn_cnt_thresh) && = (hpts->hit_callout_thresh =3D=3D 1)) { >>>> + hpts->hit_callout_thresh =3D 0; >>>> + counter_u64_add(hpts_that_need_softclock, -1); >>>> + } >>>> if (hpts->p_on_queue_cnt >=3D conn_cnt_thresh) { >>>> if(hpts->p_direct_wake =3D=3D 0) { >>>> /* >>>> @@ -1818,6 +1834,7 @@ tcp_hpts_mod_load(void) >>>> cpu_top =3D NULL; >>>> #endif >>>> tcp_pace.rp_num_hptss =3D ncpus; >>>> + hpts_that_need_softclock =3D counter_u64_alloc(M_WAITOK); >>>> hpts_hopelessly_behind =3D counter_u64_alloc(M_WAITOK); >>>> hpts_loops =3D counter_u64_alloc(M_WAITOK); >>>> back_tosleep =3D counter_u64_alloc(M_WAITOK); >>>> @@ -2042,6 +2059,7 @@ tcp_hpts_mod_unload(void) >>>> free(tcp_pace.grps, M_TCPHPTS); >>>> #endif >>>>=20 >>>> + counter_u64_free(hpts_that_need_softclock); >>>> counter_u64_free(hpts_hopelessly_behind); >>>> counter_u64_free(hpts_loops); >>>> counter_u64_free(back_tosleep); >>>=20 >>>=20 >>=20 >>=20 >>=20 >> --=20 >> Nuno Teixeira >> FreeBSD Committer (ports) >=20 >=20 >=20 > --=20 > Nuno Teixeira > FreeBSD Committer (ports) From nobody Wed Apr 10 12:39:17 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VF2TG09Dlz5H380; Wed, 10 Apr 2024 12:39:30 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VF2TF6pbSz4R7d; Wed, 10 Apr 2024 12:39:29 +0000 (UTC) (envelope-from eduardo@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1712752769; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=DpkPlNoF0Dic+HFl8IAcIYjH1s/pGnxXnG0kc5ttbZM=; b=VXO6sAPz5owi9bvRhnUIsoBeGkaCJWcoUvmZ+Pq8+3/8VVfZli9MhctJZQPlLI9krg97z8 4g2MZqO1ATKgp8ETVsqPEEdJ7z7kxbPk9//fgghq9u/X0Xxoxd0ECGskb+03pPH0yTwP/z kSiq5luCPO+p/ujm53CGjfkKAfNg8U8cOTvgYo/Jtbao7DEeqrSPXzE6dlpp9IugxtAq+l f7cIeCwW6FGahDnX5P/dLowLgKIza6Xj4gD345Z/yLmfOAMlRbS5KQhKclQLhw7XEpHaJa KZ17KalDZOdP4XkhnY4VVy3Pi8/DTBD8Q0v+Ak/3fZadxlzXbp+GFMDp6hF+Vg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1712752769; a=rsa-sha256; cv=none; b=TyiRyIsxByXPlIVp5WISYZzdaOZCSbPZTEy8ilqeJ7WfqptPQwNsgrFkficXRxtSCBo4cd av+zLwmXEm/DdvZJUa+vf2U9RwzeB5Bca65AgCL2+OIVFEawS1BaASX3YE1EWWi89FnU5Y fGVcetIOJ75NTNqXNAD+10dEKDZgiO5n/ZocIXSx8nrV5eJwiRpYwq9IW8DqzgpMpomI2A gYSp/Q1CHhp+evK7UA3rjmLdALu02dBOOLnq2HNOh6t35sR2iMeFYAoa2Fz3i/R+URV/Ra Fz261QLiS1TxU8jM0yFojJGFdpYUPixMjBbo9x5Gn0F6dXI5yHtRxYo+FNzqiA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1712752769; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=DpkPlNoF0Dic+HFl8IAcIYjH1s/pGnxXnG0kc5ttbZM=; b=IC72YzBOQOtfrJH1iu9Aw1NX9NXw/HqUCQ/yqQhZmtpJks15GRcvyDT2dzWqwtiKHYbwBH ft46y18etBkr+1TbGUmJpNU1Q652PsmCV3z4owBTIcEdk9K1v1VGsOrXEsrlnXY1uRsmhL oNY5i6bXc3dReJxjwSizVUQuO5T5wYSFLPfmYJ0tT17SFGYCRxmAeIXya3EYYEvdGo5ZiY gzlmL1Y39Ffgn9ZII2ZnwCJ2ZQ3OAPH4yok5ZbxVP8Ngl2UaWRnFsrUgf/ib6yOYM3SmJA w+v/sMukFLH5dmh2w76+B7TYx6BLqO6Jbnq6absbM5/gY8f7n/r478Hjrhvz+g== Received: from mail-qt1-f180.google.com (mail-qt1-f180.google.com [209.85.160.180]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: eduardo) by smtp.freebsd.org (Postfix) with ESMTPSA id 4VF2TF6C2mzM4K; Wed, 10 Apr 2024 12:39:29 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: by mail-qt1-f180.google.com with SMTP id d75a77b69052e-434899d6d2bso13270921cf.0; Wed, 10 Apr 2024 05:39:29 -0700 (PDT) X-Forwarded-Encrypted: i=1; AJvYcCV5+GMBASN1pIuTI+qiCXyeMWT46xWeBCPVyYb5FVWoDOM35OMBb4e1aN/UjZ+lPzZqzaqYjP71JZYjUJUHGzA/Dai7FJqaEZcSk/CdymQfhd+MNA9SE0Th3RiIUIHqjpWEkBHmQgIZUNQ0u+vptbA2PGoLet17 X-Gm-Message-State: AOJu0Yz4dRLEi1tzX8cvrm6KR7iqEtd8TWYKZxpbCGdsS45xJDNK3X/1 cGP+StQxJB+iCwprNukSOmxqMeL78B6Y3wamLbibtveAyrfk7gEHbjyoaAMStlLG4MHSKaTV8jk RABPJzRt9hvSpALIH0k/8sFVLTu0= X-Google-Smtp-Source: AGHT+IELruYdl2HXz5eJ5u2RBZt4dvlQU22q2uwUByxBPplP6lm08HPZaprphMaSnqC7DujvKztmckYLSruG29p5yrc= X-Received: by 2002:ac8:7c52:0:b0:436:4e4c:3cab with SMTP id o18-20020ac87c52000000b004364e4c3cabmr170207qtv.33.1712752769307; Wed, 10 Apr 2024 05:39:29 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 References: <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> <6047C8EF-B1B0-4286-93FA-AA38F8A18656@karels.net> <8031cd99-ded8-4b06-93b3-11cc729a8b2c@app.fastmail.com> <38c54399-6c96-44d8-a3a2-3cc1bfbe50c2@app.fastmail.com> <27d8144f-0658-46f6-b8f3-35eb60061644@lakerest.net> <5C9863F7-0F1C-4D02-9F6D-9DDC5FBEB368@freebsd.org> <52479AA6-04F6-4D4A-ABE0-7142B47E28DF@freebsd.org> In-Reply-To: <52479AA6-04F6-4D4A-ABE0-7142B47E28DF@freebsd.org> From: Nuno Teixeira Date: Wed, 10 Apr 2024 13:39:17 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Request for Testing: TCP RACK To: tuexen@freebsd.org Cc: Drew Gallatin , Konstantin Belousov , rrs , Mike Karels , garyj@gmx.de, current@freebsd.org, net@freebsd.org, Randall Stewart Content-Type: multipart/alternative; boundary="0000000000006723920615bd54db" --0000000000006723920615bd54db Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable With base stack I can complete restic check successfully downloading/reading/checking all files from a "big" remote compressed backup. Changing it to RACK stack, it fails. I run this command often because in the past, compression corruption occured and this is the equivalent of restoring backup to check its integrity. Maybe someone could do a restic test to check if this is reproducible. Thanks, escreveu (quarta, 10/04/2024 =C3=A0(s) 13:12): > > > > On 10. Apr 2024, at 13:40, Nuno Teixeira wrote: > > > > Hello all, > > > > @ current 1500018 and fetching torrents with net-p2p/qbittorrent > finished ~2GB download and connection UP until the end: > > > > --- > > Apr 10 11:26:46 leg kernel: re0: watchdog timeout > > Apr 10 11:26:46 leg kernel: re0: link state changed to DOWN > > Apr 10 11:26:49 leg dhclient[58810]: New IP Address (re0): 192.168.1.67 > > Apr 10 11:26:49 leg dhclient[58814]: New Subnet Mask (re0): 255.255.255= .0 > > Apr 10 11:26:49 leg dhclient[58818]: New Broadcast Address (re0): > 192.168.1.255 > > Apr 10 11:26:49 leg kernel: re0: link state changed to UP > > Apr 10 11:26:49 leg dhclient[58822]: New Routers (re0): 192.168.1.1 > > --- > > > > In the past tests, I've got more watchdog timeouts, connection goes dow= n > and a reboot needed to put it back (`service netif restart` didn't work). > > > > Other way to reproduce this is using sysutils/restic (backup program) t= o > read/check all files from a remote server via sftp: > > > > `restic -r sftp:user@remote:restic-repo check --read-data` from a 60GB > compressed backup. > > > > --- > > watchdog timeout x3 as above > > --- > > > > restic check fail log @ 15% progress: > > --- > > > > Load(, 17310001, 0) returned error, retrying after > 1.7670599s: connection lost > > Load(, 17456892, 0) returned error, retrying after > 4.619104908s: connection lost > > Load(, 17310001, 0) returned error, retrying after > 5.477648517s: connection lost > > List(lock) returned error, retrying after 293.057766ms: connection lost > > List(lock) returned error, retrying after 385.206693ms: connection lost > > List(lock) returned error, retrying after 1.577594281s: connection lost > > > > > > Connection continues UP. > Hi, > > I'm not sure what the issue is you are reporting. Could you state > what behavior you are experiencing with the base stack and with > the RACK stack. In particular, what the difference is? > > Best regards > Michael > > > > Cheers, > > > > escreveu (quinta, 28/03/2024 =C3=A0(s) 15:53): > >> On 28. Mar 2024, at 15:00, Nuno Teixeira wrote: > >> > >> Hello all! > >> > >> Running rack @b7b78c1c169 "Optimize HPTS..." very happy on my laptop > (amd64)! > >> > >> Thanks all! > > Thanks for the feedback! > > > > Best regards > > Michael > >> > >> Drew Gallatin escreveu (quinta, 21/03/2024 =C3= =A0(s) > 12:58): > >> The entire point is to *NOT* go through the overhead of scheduling > something asynchronously, but to take advantage of the fact that a > user/kernel transition is going to trash the cache anyway. > >> > >> In the common case of a system which has less than the threshold > number of connections , we access the tcp_hpts_softclock function pointer= , > make one function call, and access hpts_that_need_softclock, and then > return. So that's 2 variables and a function call. > >> > >> I think it would be preferable to avoid that call, and to move the > declaration of tcp_hpts_softclock and hpts_that_need_softclock so that th= ey > are in the same cacheline. Then we'd be hitting just a single line in th= e > common case. (I've made comments on the review to that effect). > >> > >> Also, I wonder if the threshold could get higher by default, so that > hpts is never called in this context unless we're to the point where we'r= e > scheduling thousands of runs of the hpts thread (and taking all those clo= ck > interrupts). > >> > >> Drew > >> > >> On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: > >>> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: > >>>> Ok I have created > >>>> > >>>> https://reviews.freebsd.org/D44420 > >>>> > >>>> > >>>> To address the issue. I also attach a short version of the patch tha= t > Nuno > >>>> can try and validate > >>>> > >>>> it works. Drew you may want to try this and validate the optimizatio= n > does > >>>> kick in since I can > >>>> > >>>> only now test that it does not on my local box :) > >>> The patch still causes access to all cpu's cachelines on each userret= . > >>> It would be much better to inc/check the threshold and only schedule > the > >>> call when exceeded. Then the call can occur in some dedicated contex= t, > >>> like per-CPU thread, instead of userret. > >>> > >>>> > >>>> > >>>> R > >>>> > >>>> > >>>> > >>>> On 3/18/24 3:42 PM, Drew Gallatin wrote: > >>>>> No. The goal is to run on every return to userspace for every > thread. > >>>>> > >>>>> Drew > >>>>> > >>>>> On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: > >>>>>> On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: > >>>>>>> I got the idea from > >>>>>>> > https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf > >>>>>>> The gist is that the TCP pacing stuff needs to run frequently, an= d > >>>>>>> rather than run it out of a clock interrupt, its more efficient t= o > run > >>>>>>> it out of a system call context at just the point where we return > to > >>>>>>> userspace and the cache is trashed anyway. The current > implementation > >>>>>>> is fine for our workload, but probably not idea for a generic > system. > >>>>>>> Especially one where something is banging on system calls. > >>>>>>> > >>>>>>> Ast's could be the right tool for this, but I'm super unfamiliar > with > >>>>>>> them, and I can't find any docs on them. > >>>>>>> > >>>>>>> Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent > to > >>>>>>> what's happening here? > >>>>>> This call would need some AST number added, and then it registers > the > >>>>>> ast to run on next return to userspace, for the current thread. > >>>>>> > >>>>>> Is it enough? > >>>>>>> > >>>>>>> Drew > >>>>>> > >>>>>>> > >>>>>>> On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: > >>>>>>>> On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > >>>>>>>>> On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote: > >>>>>>>>> > >>>>>>>>>>> On 18. Mar 2024, at 12:42, Nuno Teixeira > >>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Hello all! > >>>>>>>>>>> > >>>>>>>>>>> It works just fine! > >>>>>>>>>>> System performance is OK. > >>>>>>>>>>> Using patch on main-n268841-b0aaf8beb126(-dirty). > >>>>>>>>>>> > >>>>>>>>>>> --- > >>>>>>>>>>> net.inet.tcp.functions_available: > >>>>>>>>>>> Stack D > >>>>>> Alias PCB count > >>>>>>>>>>> freebsd freebsd 0 > >>>>>>>>>>> rack * > >>>>>> rack 38 > >>>>>>>>>>> --- > >>>>>>>>>>> > >>>>>>>>>>> It would be so nice that we can have a sysctl tunnable for > >>>>>> this patch > >>>>>>>>>>> so we could do more tests without recompiling kernel. > >>>>>>>>>> Thanks for testing! > >>>>>>>>>> > >>>>>>>>>> @gallatin: can you come up with a patch that is acceptable > >>>>>> for Netflix > >>>>>>>>>> and allows to mitigate the performance regression. > >>>>>>>>> > >>>>>>>>> Ideally, tcphpts could enable this automatically when it > >>>>>> starts to be > >>>>>>>>> used (enough?), but a sysctl could select auto/on/off. > >>>>>>>> There is already a well-known mechanism to request execution of > the > >>>>>>>> specific function on return to userspace, namely AST. The > difference > >>>>>>>> with the current hack is that the execution is requested for one > >>>>>> callback > >>>>>>>> in the context of the specific thread. > >>>>>>>> > >>>>>>>> Still, it might be worth a try to use it; what is the reason to > >>>>>> hit a thread > >>>>>>>> that does not do networking, with TCP processing? > >>>>>>>> > >>>>>>>>> > >>>>>>>>> Mike > >>>>>>>>> > >>>>>>>>>> Best regards > >>>>>>>>>> Michael > >>>>>>>>>>> > >>>>>>>>>>> Thanks all! > >>>>>>>>>>> Really happy here :) > >>>>>>>>>>> > >>>>>>>>>>> Cheers, > >>>>>>>>>>> > >>>>>>>>>>> Nuno Teixeira escreveu (domingo, > >>>>>> 17/03/2024 =C3=A0(s) 20:26): > >>>>>>>>>>>> > >>>>>>>>>>>> Hello, > >>>>>>>>>>>> > >>>>>>>>>>>>> I don't have the full context, but it seems like the > >>>>>> complaint is a performance regression in bonnie++ and perhaps othe= r > >>>>>> things when tcp_hpts is loaded, even when it is not used. Is that > >>>>>> correct? > >>>>>>>>>>>>> > >>>>>>>>>>>>> If so, I suspect its because we drive the > >>>>>> tcp_hpts_softclock() routine from userret(), in order to avoid ton= s > >>>>>> of timer interrupts and context switches. To test this theory, y= ou > >>>>>> could apply a patch like: > >>>>>>>>>>>> > >>>>>>>>>>>> It's affecting overall system performance, bonnie was just > >>>>>> a way to > >>>>>>>>>>>> get some numbers to compare. > >>>>>>>>>>>> > >>>>>>>>>>>> Tomorrow I will test patch. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks! > >>>>>>>>>>>> > >>>>>>>>>>>> -- > >>>>>>>>>>>> Nuno Teixeira > >>>>>>>>>>>> FreeBSD Committer (ports) > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Nuno Teixeira > >>>>>>>>>>> FreeBSD Committer (ports) > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > >>> > >>>> diff --git a/sys/netinet/tcp_hpts.c b/sys/netinet/tcp_hpts.c > >>>> index 8c4d2d41a3eb..eadbee19f69c 100644 > >>>> --- a/sys/netinet/tcp_hpts.c > >>>> +++ b/sys/netinet/tcp_hpts.c > >>>> @@ -216,6 +216,7 @@ struct tcp_hpts_entry { > >>>> void *ie_cookie; > >>>> uint16_t p_num; /* The hpts number one per cpu */ > >>>> uint16_t p_cpu; /* The hpts CPU */ > >>>> + uint8_t hit_callout_thresh; > >>>> /* There is extra space in here */ > >>>> /* Cache line 0x100 */ > >>>> struct callout co __aligned(CACHE_LINE_SIZE); > >>>> @@ -269,6 +270,11 @@ static struct hpts_domain_info { > >>>> int cpu[MAXCPU]; > >>>> } hpts_domains[MAXMEMDOM]; > >>>> > >>>> +counter_u64_t hpts_that_need_softclock; > >>>> +SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, > needsoftclock, CTLFLAG_RD, > >>>> + &hpts_that_need_softclock, > >>>> + "Number of hpts threads that need softclock"); > >>>> + > >>>> counter_u64_t hpts_hopelessly_behind; > >>>> > >>>> SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, hopeless, > CTLFLAG_RD, > >>>> @@ -334,7 +340,7 @@ SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, > precision, CTLFLAG_RW, > >>>> &tcp_hpts_precision, 120, > >>>> "Value for PRE() precision of callout"); > >>>> SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, cnt_thresh, CTLFLAG_RW, > >>>> - &conn_cnt_thresh, 0, > >>>> + &conn_cnt_thresh, DEFAULT_CONNECTION_THESHOLD, > >>>> "How many connections (below) make us use the callout based > mechanism"); > >>>> SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, logging, CTLFLAG_RW, > >>>> &hpts_does_tp_logging, 0, > >>>> @@ -1548,6 +1554,9 @@ __tcp_run_hpts(void) > >>>> struct tcp_hpts_entry *hpts; > >>>> int ticks_ran; > >>>> > >>>> + if (counter_u64_fetch(hpts_that_need_softclock) =3D=3D 0) > >>>> + return; > >>>> + > >>>> hpts =3D tcp_choose_hpts_to_run(); > >>>> > >>>> if (hpts->p_hpts_active) { > >>>> @@ -1683,6 +1692,13 @@ tcp_hpts_thread(void *ctx) > >>>> ticks_ran =3D tcp_hptsi(hpts, 1); > >>>> tv.tv_sec =3D 0; > >>>> tv.tv_usec =3D hpts->p_hpts_sleep_time * HPTS_TICKS_PER_SLOT; > >>>> + if ((hpts->p_on_queue_cnt > conn_cnt_thresh) && > (hpts->hit_callout_thresh =3D=3D 0)) { > >>>> + hpts->hit_callout_thresh =3D 1; > >>>> + counter_u64_add(hpts_that_need_softclock, 1); > >>>> + } else if ((hpts->p_on_queue_cnt <=3D conn_cnt_thresh) && > (hpts->hit_callout_thresh =3D=3D 1)) { > >>>> + hpts->hit_callout_thresh =3D 0; > >>>> + counter_u64_add(hpts_that_need_softclock, -1); > >>>> + } > >>>> if (hpts->p_on_queue_cnt >=3D conn_cnt_thresh) { > >>>> if(hpts->p_direct_wake =3D=3D 0) { > >>>> /* > >>>> @@ -1818,6 +1834,7 @@ tcp_hpts_mod_load(void) > >>>> cpu_top =3D NULL; > >>>> #endif > >>>> tcp_pace.rp_num_hptss =3D ncpus; > >>>> + hpts_that_need_softclock =3D counter_u64_alloc(M_WAITOK); > >>>> hpts_hopelessly_behind =3D counter_u64_alloc(M_WAITOK); > >>>> hpts_loops =3D counter_u64_alloc(M_WAITOK); > >>>> back_tosleep =3D counter_u64_alloc(M_WAITOK); > >>>> @@ -2042,6 +2059,7 @@ tcp_hpts_mod_unload(void) > >>>> free(tcp_pace.grps, M_TCPHPTS); > >>>> #endif > >>>> > >>>> + counter_u64_free(hpts_that_need_softclock); > >>>> counter_u64_free(hpts_hopelessly_behind); > >>>> counter_u64_free(hpts_loops); > >>>> counter_u64_free(back_tosleep); > >>> > >>> > >> > >> > >> > >> -- > >> Nuno Teixeira > >> FreeBSD Committer (ports) > > > > > > > > -- > > Nuno Teixeira > > FreeBSD Committer (ports) > > --=20 Nuno Teixeira FreeBSD Committer (ports) --0000000000006723920615bd54db Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
With base stack I can complete restic check successfu= lly downloading/reading/checking all files from a "big" remote co= mpressed backup.
Changing it to RACK stack, it fails.
<= br>
I run this command often because in the past, compression cor= ruption occured and this is the equivalent of restoring backup to check its= integrity.

Maybe someone could do a restic test t= o check if this is reproducible.

Thanks,
=



<tue= xen@freebsd.org> escreveu (quarta, 10/04/2024 =C3=A0(s) 13:12):
<= /div>


> On 10. Apr 2024, at 13:40, Nuno Teixeira <eduardo@freebsd.org> wrote:
>
> Hello all,
>
> @ current 1500018 and fetching torrents with net-p2p/qbittorrent finis= hed ~2GB download and connection UP until the end:
>
> ---
> Apr 10 11:26:46 leg kernel: re0: watchdog timeout
> Apr 10 11:26:46 leg kernel: re0: link state changed to DOWN
> Apr 10 11:26:49 leg dhclient[58810]: New IP Address (re0): 192.168.1.6= 7
> Apr 10 11:26:49 leg dhclient[58814]: New Subnet Mask (re0): 255.255.25= 5.0
> Apr 10 11:26:49 leg dhclient[58818]: New Broadcast Address (re0): 192.= 168.1.255
> Apr 10 11:26:49 leg kernel: re0: link state changed to UP
> Apr 10 11:26:49 leg dhclient[58822]: New Routers (re0): 192.168.1.1 > ---
>
> In the past tests, I've got more watchdog timeouts, connection goe= s down and a reboot needed to put it back (`service netif restart` didn'= ;t work).
>
> Other way to reproduce this is using sysutils/restic (backup program) = to read/check all files from a remote server via sftp:
>
> `restic -r sftp:user@remote:restic-repo check --read-data` from a 60GB= compressed backup.
>
> ---
> watchdog timeout x3 as above
> ---
>
> restic check fail log @ 15% progress:
> ---
> <snip>
> Load(<data/52e2923dd6>, 17310001, 0) returned error, retrying af= ter 1.7670599s: connection lost
> Load(<data/d27a0abe0f>, 17456892, 0) returned error, retrying af= ter 4.619104908s: connection lost
> Load(<data/52e2923dd6>, 17310001, 0) returned error, retrying af= ter 5.477648517s: connection lost
> List(lock) returned error, retrying after 293.057766ms: connection los= t
> List(lock) returned error, retrying after 385.206693ms: connection los= t
> List(lock) returned error, retrying after 1.577594281s: connection los= t
> <snip>
>
> Connection continues UP.
Hi,

I'm not sure what the issue is you are reporting. Could you state
what behavior you are experiencing with the base stack and with
the RACK stack. In particular, what the difference is?

Best regards
Michael
>
> Cheers,
>
> <tuexen@fre= ebsd.org> escreveu (quinta, 28/03/2024 =C3=A0(s) 15:53):
>> On 28. Mar 2024, at 15:00, Nuno Teixeira <eduardo@freebsd.org> wrote:
>>
>> Hello all!
>>
>> Running rack @b7b78c1c169 "Optimize HPTS..." very happy = on my laptop (amd64)!
>>
>> Thanks all!
> Thanks for the feedback!
>
> Best regards
> Michael
>>
>> Drew Gallatin <gallatin@freebsd.org> escreveu (quinta, 21/03/2024 =C3= =A0(s) 12:58):
>> The entire point is to *NOT* go through the overhead of scheduling= something asynchronously, but to take advantage of the fact that a user/ke= rnel transition is going to trash the cache anyway.
>>
>> In the common case of a system which has less than the threshold= =C2=A0 number of connections , we access the tcp_hpts_softclock function po= inter, make one function call, and access hpts_that_need_softclock, and the= n return.=C2=A0 So that's 2 variables and a function call.
>>
>> I think it would be preferable to avoid that call, and to move the= declaration of tcp_hpts_softclock and hpts_that_need_softclock so that the= y are in the same cacheline.=C2=A0 Then we'd be hitting just a single l= ine in the common case.=C2=A0 (I've made comments on the review to that= effect).
>>
>> Also, I wonder if the threshold could get higher by default, so th= at hpts is never called in this context unless we're to the point where= we're scheduling thousands of runs of the hpts thread (and taking all = those clock interrupts).
>>
>> Drew
>>
>> On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote:
>>> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote:
>>>> Ok I have created
>>>>
>>>> https://reviews.freebsd.org/D44420
>>>>
>>>>
>>>> To address the issue. I also attach a short version of the= patch that Nuno
>>>> can try and validate
>>>>
>>>> it works. Drew you may want to try this and validate the o= ptimization does
>>>> kick in since I can
>>>>
>>>> only now test that it does not on my local box :)
>>> The patch still causes access to all cpu's cachelines on e= ach userret.
>>> It would be much better to inc/check the threshold and only sc= hedule the
>>> call when exceeded.=C2=A0 Then the call can occur in some dedi= cated context,
>>> like per-CPU thread, instead of userret.
>>>
>>>>
>>>>
>>>> R
>>>>
>>>>
>>>>
>>>> On 3/18/24 3:42 PM, Drew Gallatin wrote:
>>>>> No.=C2=A0 The goal is to run on every return to usersp= ace for every thread.
>>>>>
>>>>> Drew
>>>>>
>>>>> On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov = wrote:
>>>>>> On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gal= latin wrote:
>>>>>>> I got the idea from
>>>>>>> h= ttps://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf >>>>>>> The gist is that the TCP pacing stuff needs to= run frequently, and
>>>>>>> rather than run it out of a clock interrupt, i= ts more efficient to run
>>>>>>> it out of a system call context at just the po= int where we return to
>>>>>>> userspace and the cache is trashed anyway. The= current implementation
>>>>>>> is fine for our workload, but probably not ide= a for a generic system.
>>>>>>> Especially one where something is banging on s= ystem calls.
>>>>>>>
>>>>>>> Ast's could be the right tool for this, bu= t I'm super unfamiliar with
>>>>>>> them, and I can't find any docs on them. >>>>>>>
>>>>>>> Would ast_register(0, ASTR_UNCOND, 0, func) be= roughly equivalent to
>>>>>>> what's happening here?
>>>>>> This call would need some AST number added, and th= en it registers the
>>>>>> ast to run on next return to userspace, for the cu= rrent thread.
>>>>>>
>>>>>> Is it enough?
>>>>>>>
>>>>>>> Drew
>>>>>>
>>>>>>>
>>>>>>> On Mon, Mar 18, 2024, at 2:33 PM, Konstantin B= elousov wrote:
>>>>>>>> On Mon, Mar 18, 2024 at 07:26:10AM -0500, = Mike Karels wrote:
>>>>>>>>> On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote: >>>>>>>>>
>>>>>>>>>>> On 18. Mar 2024, at 12:42, Nun= o Teixeira
>>>>>> <eduardo@freebsd.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello all!
>>>>>>>>>>>
>>>>>>>>>>> It works just fine!
>>>>>>>>>>> System performance is OK.
>>>>>>>>>>> Using patch on main-n268841-b0= aaf8beb126(-dirty).
>>>>>>>>>>>
>>>>>>>>>>> ---
>>>>>>>>>>> net.inet.tcp.functions_availab= le:
>>>>>>>>>>> Stack=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0D<= br> >>>>>> Alias=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 PCB count
>>>>>>>>>>> freebsd freebsd=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 0
>>>>>>>>>>> rack=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 *=
>>>>>> rack=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A038
>>>>>>>>>>> ---
>>>>>>>>>>>
>>>>>>>>>>> It would be so nice that we ca= n have a sysctl tunnable for
>>>>>> this patch
>>>>>>>>>>> so we could do more tests with= out recompiling kernel.
>>>>>>>>>> Thanks for testing!
>>>>>>>>>>
>>>>>>>>>> @gallatin: can you come up with a = patch that is acceptable
>>>>>> for Netflix
>>>>>>>>>> and allows to mitigate the perform= ance regression.
>>>>>>>>>
>>>>>>>>> Ideally, tcphpts could enable this aut= omatically when it
>>>>>> starts to be
>>>>>>>>> used (enough?), but a sysctl could sel= ect auto/on/off.
>>>>>>>> There is already a well-known mechanism to= request execution of the
>>>>>>>> specific function on return to userspace, = namely AST.=C2=A0 The difference
>>>>>>>> with the current hack is that the executio= n is requested for one
>>>>>> callback
>>>>>>>> in the context of the specific thread.
>>>>>>>>
>>>>>>>> Still, it might be worth a try to use it; = what is the reason to
>>>>>> hit a thread
>>>>>>>> that does not do networking, with TCP proc= essing?
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>>> Best regards
>>>>>>>>>> Michael
>>>>>>>>>>>
>>>>>>>>>>> Thanks all!
>>>>>>>>>>> Really happy here :)
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> Nuno Teixeira <eduardo@freebsd.org> es= creveu (domingo,
>>>>>> 17/03/2024 =C3=A0(s) 20:26):
>>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>>> I don't have the f= ull context, but it seems like the
>>>>>> complaint is a performance regression in bonnie++ = and perhaps other
>>>>>> things when tcp_hpts is loaded, even when it is no= t used.=C2=A0 Is that
>>>>>> correct?
>>>>>>>>>>>>>
>>>>>>>>>>>>> If so, I suspect its b= ecause we drive the
>>>>>> tcp_hpts_softclock() routine from userret(), in or= der to avoid tons
>>>>>> of timer interrupts and context switches.=C2=A0 To= test this theory,=C2=A0 you
>>>>>> could apply a patch like:
>>>>>>>>>>>>
>>>>>>>>>>>> It's affecting overall= system performance, bonnie was just
>>>>>> a way to
>>>>>>>>>>>> get some numbers to compar= e.
>>>>>>>>>>>>
>>>>>>>>>>>> Tomorrow I will test patch= .
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Nuno Teixeira
>>>>>>>>>>>> FreeBSD Committer (ports)<= br> >>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Nuno Teixeira
>>>>>>>>>>> FreeBSD Committer (ports)
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>
>>>> diff --git a/sys/netinet/tcp_hpts.c b/sys/netinet/tcp_hpts= .c
>>>> index 8c4d2d41a3eb..eadbee19f69c 100644
>>>> --- a/sys/netinet/tcp_hpts.c
>>>> +++ b/sys/netinet/tcp_hpts.c
>>>> @@ -216,6 +216,7 @@ struct tcp_hpts_entry {
>>>> void *ie_cookie;
>>>> uint16_t p_num; /* The hpts number one per cpu */
>>>> uint16_t p_cpu; /* The hpts CPU */
>>>> + uint8_t hit_callout_thresh;
>>>> /* There is extra space in here */
>>>> /* Cache line 0x100 */
>>>> struct callout co __aligned(CACHE_LINE_SIZE);
>>>> @@ -269,6 +270,11 @@ static struct hpts_domain_info {
>>>> int cpu[MAXCPU];
>>>> } hpts_domains[MAXMEMDOM];
>>>>
>>>> +counter_u64_t hpts_that_need_softclock;
>>>> +SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, ne= edsoftclock, CTLFLAG_RD,
>>>> +=C2=A0 =C2=A0 &hpts_that_need_softclock,
>>>> +=C2=A0 =C2=A0 "Number of hpts threads that need soft= clock");
>>>> +
>>>> counter_u64_t hpts_hopelessly_behind;
>>>>
>>>> SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, hop= eless, CTLFLAG_RD,
>>>> @@ -334,7 +340,7 @@ SYSCTL_INT(_net_inet_tcp_hpts, OID_AUT= O, precision, CTLFLAG_RW,
>>>>=C2=A0 =C2=A0 &tcp_hpts_precision, 120,
>>>>=C2=A0 =C2=A0 "Value for PRE() precision of callout&qu= ot;);
>>>> SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, cnt_thresh, CTLFL= AG_RW,
>>>> -=C2=A0 =C2=A0 &conn_cnt_thresh, 0,
>>>> +=C2=A0 =C2=A0 &conn_cnt_thresh, DEFAULT_CONNECTION_TH= ESHOLD,
>>>>=C2=A0 =C2=A0 "How many connections (below) make us us= e the callout based mechanism");
>>>> SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, logging, CTLFLAG_= RW,
>>>>=C2=A0 =C2=A0 &hpts_does_tp_logging, 0,
>>>> @@ -1548,6 +1554,9 @@ __tcp_run_hpts(void)
>>>> struct tcp_hpts_entry *hpts;
>>>> int ticks_ran;
>>>>
>>>> + if (counter_u64_fetch(hpts_that_need_softclock) =3D=3D 0= )
>>>> + return;
>>>> +
>>>> hpts =3D tcp_choose_hpts_to_run();
>>>>
>>>> if (hpts->p_hpts_active) {
>>>> @@ -1683,6 +1692,13 @@ tcp_hpts_thread(void *ctx)
>>>> ticks_ran =3D tcp_hptsi(hpts, 1);
>>>> tv.tv_sec =3D 0;
>>>> tv.tv_usec =3D hpts->p_hpts_sleep_time * HPTS_TICKS_PER= _SLOT;
>>>> + if ((hpts->p_on_queue_cnt > conn_cnt_thresh) &= & (hpts->hit_callout_thresh =3D=3D 0)) {
>>>> + hpts->hit_callout_thresh =3D 1;
>>>> + counter_u64_add(hpts_that_need_softclock, 1);
>>>> + } else if ((hpts->p_on_queue_cnt <=3D conn_cnt_thr= esh) && (hpts->hit_callout_thresh =3D=3D 1)) {
>>>> + hpts->hit_callout_thresh =3D 0;
>>>> + counter_u64_add(hpts_that_need_softclock, -1);
>>>> + }
>>>> if (hpts->p_on_queue_cnt >=3D conn_cnt_thresh) {
>>>> if(hpts->p_direct_wake =3D=3D 0) {
>>>> /*
>>>> @@ -1818,6 +1834,7 @@ tcp_hpts_mod_load(void)
>>>> cpu_top =3D NULL;
>>>> #endif
>>>> tcp_pace.rp_num_hptss =3D ncpus;
>>>> + hpts_that_need_softclock =3D counter_u64_alloc(M_WAITOK)= ;
>>>> hpts_hopelessly_behind =3D counter_u64_alloc(M_WAITOK); >>>> hpts_loops =3D counter_u64_alloc(M_WAITOK);
>>>> back_tosleep =3D counter_u64_alloc(M_WAITOK);
>>>> @@ -2042,6 +2059,7 @@ tcp_hpts_mod_unload(void)
>>>> free(tcp_pace.grps, M_TCPHPTS);
>>>> #endif
>>>>
>>>> + counter_u64_free(hpts_that_need_softclock);
>>>> counter_u64_free(hpts_hopelessly_behind);
>>>> counter_u64_free(hpts_loops);
>>>> counter_u64_free(back_tosleep);
>>>
>>>
>>
>>
>>
>> --
>> Nuno Teixeira
>> FreeBSD Committer (ports)
>
>
>
> --
> Nuno Teixeira
> FreeBSD Committer (ports)



--
Nuno Teixeira
FreeBSD Committ= er (ports)
--0000000000006723920615bd54db-- From nobody Wed Apr 10 12:44:59 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VF2br1PBdz5H3fW; Wed, 10 Apr 2024 12:45:12 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VF2br16Sjz4THK; Wed, 10 Apr 2024 12:45:12 +0000 (UTC) (envelope-from eduardo@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1712753112; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=doNA/KZxMAy9H2qRyYQFZaDJlTs3p0PtQwPZnbswvE0=; b=JphAbMSEVC8ZZZFhwP6xtwOfEOROF/BHSi/D+Om7Mux0qwl/BtHW5ywz9tBvvPOWp3rtlV JI1sLkL1KgaGsU33X2t1PRl7Is3s7KlJXkxlrbgd1M5cr+iRtkAFPTjWGnNvtm2g1RxYvU mpv3pkxLNXvKZRCpt9uhue5GVEkbeNn15oCFp4BCMai6GvTokhQyd5KfXMbV1FUFtz3tQn 4JMqNrC71okwFHyTXFbmkyVnkVz4j/eUzDPqL+bFVjzyF0HbrKW96MxhrK2lEthya65lnk vOlrntCHD/6WFuIAH6yan7crn38WHFb8JUUiH8ER0eNHjIV08kAXfk8N549k6g== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1712753112; a=rsa-sha256; cv=none; b=E8w3Mtm8iYk8UsPO3ue9/8qfISFftmmrL2mJOrDn9vcYUobCxa2k9E2sBIpXnNVg+H6Fju u+uDE/SXIJRegF/R5eOl3aw4paJGcZEx02Ffu5rNZ0cpkOOag2jPpb5F42WTcGoa3teeo9 vbgwv9YSkCGOZ2NcTZrDnb7SIgkkruJB1pK1touIWWi6CiE5jY3ppHuvTCLHu9hV9UeeVO nQIgNugZUsKD2fdFD4ZpJ+OmOH7JgEVOsuY2p4Hb1VF/HtlKOOMX7NAdY7PMo+QyIZQPS6 RIVkTQL+/bdEstA+kGLxqmaaIhp+fAbxbFD/kxlRDrPZsEgNt6YUKVK9ecK1HQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1712753112; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=doNA/KZxMAy9H2qRyYQFZaDJlTs3p0PtQwPZnbswvE0=; b=ZZXIQOIVdwGa83vXPnjexa5JqOD9ckzyoSCXE0NZRAq4gOb7Hy97RIA4wbjLi+DyUcgQam MoiCcjQbOvHHWDpDOqY5xWLHCBMl8geC8kLs8d742RV3K5P0y6oBBABq6D+f2/iZWjvCHa 7XDt69SjXbzHILEr8VBlbDe1TrCAjGjYXapcN+mk9kQldEuQa18u0nPDzShXnS0WI31ZJb jNYN9OifIJaaqmZ/3vYhVtNq6rCfxPWbUbw2zZK5Mn+jcGwelO8VZk7YUZlKJx2NY+Lr6R ocjmOueFquNoZjwEwhoQXJZ/io2YC8J77yYerCoLTLlpWy0gGWAJt/ytm/DnSg== Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: eduardo) by smtp.freebsd.org (Postfix) with ESMTPSA id 4VF2br0bCqzM4T; Wed, 10 Apr 2024 12:45:12 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: by mail-qt1-f176.google.com with SMTP id d75a77b69052e-434a5c9b998so11729521cf.0; Wed, 10 Apr 2024 05:45:12 -0700 (PDT) X-Forwarded-Encrypted: i=1; AJvYcCWMb+n1GCBadWwfygFWv59jGFdvPUf4JARMy9xJoF3sVk8UMMzoFie93KCO9xbmeIl/oClhQldrxllYrp+2uMXfAfkz0zfl1oLkbAO6zjAwW+aR28xXaet9E8kbhh67iqp2wTPcSFjpNE/QxPHXLmvqic3Dikmo X-Gm-Message-State: AOJu0YyvBSdAVP7G7MDgkYda5WTrCpjrnKM7LsUSnRA1+9dSFhEb5Ilh rQy8JjWMGO1pk/jyhxV+USmxL97KZMx7Gr4v4GFvf7f5ff0ByQUD8nyQhyDlT99k/pip05nAYIt Ky2HJ/nziQIkuaN5ga3Gp4hfGyfU= X-Google-Smtp-Source: AGHT+IG28PXFtdY1WZ5r6onFzj1ABQIX6rYXb2ZGBE10CiwM7XZOsZlIo+IKvqixNfAvKXVIMzj+/YQu6J19LaTKwJU= X-Received: by 2002:ac8:7fca:0:b0:434:cebd:9551 with SMTP id b10-20020ac87fca000000b00434cebd9551mr2490104qtk.27.1712753111088; Wed, 10 Apr 2024 05:45:11 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 References: <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> <6047C8EF-B1B0-4286-93FA-AA38F8A18656@karels.net> <8031cd99-ded8-4b06-93b3-11cc729a8b2c@app.fastmail.com> <38c54399-6c96-44d8-a3a2-3cc1bfbe50c2@app.fastmail.com> <27d8144f-0658-46f6-b8f3-35eb60061644@lakerest.net> <5C9863F7-0F1C-4D02-9F6D-9DDC5FBEB368@freebsd.org> <52479AA6-04F6-4D4A-ABE0-7142B47E28DF@freebsd.org> In-Reply-To: From: Nuno Teixeira Date: Wed, 10 Apr 2024 13:44:59 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Request for Testing: TCP RACK To: tuexen@freebsd.org Cc: Drew Gallatin , Konstantin Belousov , rrs , Mike Karels , garyj@gmx.de, current@freebsd.org, net@freebsd.org, Randall Stewart Content-Type: multipart/alternative; boundary="000000000000c64a570615bd6820" --000000000000c64a570615bd6820 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable (...) Backup server is https://www.rsync.net/ (free 500GB for FreeBSD developers). Nuno Teixeira escreveu (quarta, 10/04/2024 =C3=A0(s) 13:39): > With base stack I can complete restic check successfully > downloading/reading/checking all files from a "big" remote compressed > backup. > Changing it to RACK stack, it fails. > > I run this command often because in the past, compression corruption > occured and this is the equivalent of restoring backup to check its > integrity. > > Maybe someone could do a restic test to check if this is reproducible. > > Thanks, > > > > escreveu (quarta, 10/04/2024 =C3=A0(s) 13:12): > >> >> >> > On 10. Apr 2024, at 13:40, Nuno Teixeira wrote: >> > >> > Hello all, >> > >> > @ current 1500018 and fetching torrents with net-p2p/qbittorrent >> finished ~2GB download and connection UP until the end: >> > >> > --- >> > Apr 10 11:26:46 leg kernel: re0: watchdog timeout >> > Apr 10 11:26:46 leg kernel: re0: link state changed to DOWN >> > Apr 10 11:26:49 leg dhclient[58810]: New IP Address (re0): 192.168.1.6= 7 >> > Apr 10 11:26:49 leg dhclient[58814]: New Subnet Mask (re0): >> 255.255.255.0 >> > Apr 10 11:26:49 leg dhclient[58818]: New Broadcast Address (re0): >> 192.168.1.255 >> > Apr 10 11:26:49 leg kernel: re0: link state changed to UP >> > Apr 10 11:26:49 leg dhclient[58822]: New Routers (re0): 192.168.1.1 >> > --- >> > >> > In the past tests, I've got more watchdog timeouts, connection goes >> down and a reboot needed to put it back (`service netif restart` didn't >> work). >> > >> > Other way to reproduce this is using sysutils/restic (backup program) >> to read/check all files from a remote server via sftp: >> > >> > `restic -r sftp:user@remote:restic-repo check --read-data` from a 60GB >> compressed backup. >> > >> > --- >> > watchdog timeout x3 as above >> > --- >> > >> > restic check fail log @ 15% progress: >> > --- >> > >> > Load(, 17310001, 0) returned error, retrying after >> 1.7670599s: connection lost >> > Load(, 17456892, 0) returned error, retrying after >> 4.619104908s: connection lost >> > Load(, 17310001, 0) returned error, retrying after >> 5.477648517s: connection lost >> > List(lock) returned error, retrying after 293.057766ms: connection los= t >> > List(lock) returned error, retrying after 385.206693ms: connection los= t >> > List(lock) returned error, retrying after 1.577594281s: connection los= t >> > >> > >> > Connection continues UP. >> Hi, >> >> I'm not sure what the issue is you are reporting. Could you state >> what behavior you are experiencing with the base stack and with >> the RACK stack. In particular, what the difference is? >> >> Best regards >> Michael >> > >> > Cheers, >> > >> > escreveu (quinta, 28/03/2024 =C3=A0(s) 15:53): >> >> On 28. Mar 2024, at 15:00, Nuno Teixeira wrote: >> >> >> >> Hello all! >> >> >> >> Running rack @b7b78c1c169 "Optimize HPTS..." very happy on my laptop >> (amd64)! >> >> >> >> Thanks all! >> > Thanks for the feedback! >> > >> > Best regards >> > Michael >> >> >> >> Drew Gallatin escreveu (quinta, 21/03/2024 >> =C3=A0(s) 12:58): >> >> The entire point is to *NOT* go through the overhead of scheduling >> something asynchronously, but to take advantage of the fact that a >> user/kernel transition is going to trash the cache anyway. >> >> >> >> In the common case of a system which has less than the threshold >> number of connections , we access the tcp_hpts_softclock function pointe= r, >> make one function call, and access hpts_that_need_softclock, and then >> return. So that's 2 variables and a function call. >> >> >> >> I think it would be preferable to avoid that call, and to move the >> declaration of tcp_hpts_softclock and hpts_that_need_softclock so that t= hey >> are in the same cacheline. Then we'd be hitting just a single line in t= he >> common case. (I've made comments on the review to that effect). >> >> >> >> Also, I wonder if the threshold could get higher by default, so that >> hpts is never called in this context unless we're to the point where we'= re >> scheduling thousands of runs of the hpts thread (and taking all those cl= ock >> interrupts). >> >> >> >> Drew >> >> >> >> On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: >> >>> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: >> >>>> Ok I have created >> >>>> >> >>>> https://reviews.freebsd.org/D44420 >> >>>> >> >>>> >> >>>> To address the issue. I also attach a short version of the patch >> that Nuno >> >>>> can try and validate >> >>>> >> >>>> it works. Drew you may want to try this and validate the >> optimization does >> >>>> kick in since I can >> >>>> >> >>>> only now test that it does not on my local box :) >> >>> The patch still causes access to all cpu's cachelines on each userre= t. >> >>> It would be much better to inc/check the threshold and only schedule >> the >> >>> call when exceeded. Then the call can occur in some dedicated >> context, >> >>> like per-CPU thread, instead of userret. >> >>> >> >>>> >> >>>> >> >>>> R >> >>>> >> >>>> >> >>>> >> >>>> On 3/18/24 3:42 PM, Drew Gallatin wrote: >> >>>>> No. The goal is to run on every return to userspace for every >> thread. >> >>>>> >> >>>>> Drew >> >>>>> >> >>>>> On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: >> >>>>>> On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: >> >>>>>>> I got the idea from >> >>>>>>> >> https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf >> >>>>>>> The gist is that the TCP pacing stuff needs to run frequently, a= nd >> >>>>>>> rather than run it out of a clock interrupt, its more efficient >> to run >> >>>>>>> it out of a system call context at just the point where we retur= n >> to >> >>>>>>> userspace and the cache is trashed anyway. The current >> implementation >> >>>>>>> is fine for our workload, but probably not idea for a generic >> system. >> >>>>>>> Especially one where something is banging on system calls. >> >>>>>>> >> >>>>>>> Ast's could be the right tool for this, but I'm super unfamiliar >> with >> >>>>>>> them, and I can't find any docs on them. >> >>>>>>> >> >>>>>>> Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalen= t >> to >> >>>>>>> what's happening here? >> >>>>>> This call would need some AST number added, and then it registers >> the >> >>>>>> ast to run on next return to userspace, for the current thread. >> >>>>>> >> >>>>>> Is it enough? >> >>>>>>> >> >>>>>>> Drew >> >>>>>> >> >>>>>>> >> >>>>>>> On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: >> >>>>>>>> On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: >> >>>>>>>>> On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote: >> >>>>>>>>> >> >>>>>>>>>>> On 18. Mar 2024, at 12:42, Nuno Teixeira >> >>>>>> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>> Hello all! >> >>>>>>>>>>> >> >>>>>>>>>>> It works just fine! >> >>>>>>>>>>> System performance is OK. >> >>>>>>>>>>> Using patch on main-n268841-b0aaf8beb126(-dirty). >> >>>>>>>>>>> >> >>>>>>>>>>> --- >> >>>>>>>>>>> net.inet.tcp.functions_available: >> >>>>>>>>>>> Stack D >> >>>>>> Alias PCB count >> >>>>>>>>>>> freebsd freebsd 0 >> >>>>>>>>>>> rack * >> >>>>>> rack 38 >> >>>>>>>>>>> --- >> >>>>>>>>>>> >> >>>>>>>>>>> It would be so nice that we can have a sysctl tunnable for >> >>>>>> this patch >> >>>>>>>>>>> so we could do more tests without recompiling kernel. >> >>>>>>>>>> Thanks for testing! >> >>>>>>>>>> >> >>>>>>>>>> @gallatin: can you come up with a patch that is acceptable >> >>>>>> for Netflix >> >>>>>>>>>> and allows to mitigate the performance regression. >> >>>>>>>>> >> >>>>>>>>> Ideally, tcphpts could enable this automatically when it >> >>>>>> starts to be >> >>>>>>>>> used (enough?), but a sysctl could select auto/on/off. >> >>>>>>>> There is already a well-known mechanism to request execution of >> the >> >>>>>>>> specific function on return to userspace, namely AST. The >> difference >> >>>>>>>> with the current hack is that the execution is requested for on= e >> >>>>>> callback >> >>>>>>>> in the context of the specific thread. >> >>>>>>>> >> >>>>>>>> Still, it might be worth a try to use it; what is the reason to >> >>>>>> hit a thread >> >>>>>>>> that does not do networking, with TCP processing? >> >>>>>>>> >> >>>>>>>>> >> >>>>>>>>> Mike >> >>>>>>>>> >> >>>>>>>>>> Best regards >> >>>>>>>>>> Michael >> >>>>>>>>>>> >> >>>>>>>>>>> Thanks all! >> >>>>>>>>>>> Really happy here :) >> >>>>>>>>>>> >> >>>>>>>>>>> Cheers, >> >>>>>>>>>>> >> >>>>>>>>>>> Nuno Teixeira escreveu (domingo, >> >>>>>> 17/03/2024 =C3=A0(s) 20:26): >> >>>>>>>>>>>> >> >>>>>>>>>>>> Hello, >> >>>>>>>>>>>> >> >>>>>>>>>>>>> I don't have the full context, but it seems like the >> >>>>>> complaint is a performance regression in bonnie++ and perhaps oth= er >> >>>>>> things when tcp_hpts is loaded, even when it is not used. Is tha= t >> >>>>>> correct? >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> If so, I suspect its because we drive the >> >>>>>> tcp_hpts_softclock() routine from userret(), in order to avoid to= ns >> >>>>>> of timer interrupts and context switches. To test this theory, >> you >> >>>>>> could apply a patch like: >> >>>>>>>>>>>> >> >>>>>>>>>>>> It's affecting overall system performance, bonnie was just >> >>>>>> a way to >> >>>>>>>>>>>> get some numbers to compare. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Tomorrow I will test patch. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Thanks! >> >>>>>>>>>>>> >> >>>>>>>>>>>> -- >> >>>>>>>>>>>> Nuno Teixeira >> >>>>>>>>>>>> FreeBSD Committer (ports) >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> -- >> >>>>>>>>>>> Nuno Teixeira >> >>>>>>>>>>> FreeBSD Committer (ports) >> >>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>>> >> >>> >> >>>> diff --git a/sys/netinet/tcp_hpts.c b/sys/netinet/tcp_hpts.c >> >>>> index 8c4d2d41a3eb..eadbee19f69c 100644 >> >>>> --- a/sys/netinet/tcp_hpts.c >> >>>> +++ b/sys/netinet/tcp_hpts.c >> >>>> @@ -216,6 +216,7 @@ struct tcp_hpts_entry { >> >>>> void *ie_cookie; >> >>>> uint16_t p_num; /* The hpts number one per cpu */ >> >>>> uint16_t p_cpu; /* The hpts CPU */ >> >>>> + uint8_t hit_callout_thresh; >> >>>> /* There is extra space in here */ >> >>>> /* Cache line 0x100 */ >> >>>> struct callout co __aligned(CACHE_LINE_SIZE); >> >>>> @@ -269,6 +270,11 @@ static struct hpts_domain_info { >> >>>> int cpu[MAXCPU]; >> >>>> } hpts_domains[MAXMEMDOM]; >> >>>> >> >>>> +counter_u64_t hpts_that_need_softclock; >> >>>> +SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, >> needsoftclock, CTLFLAG_RD, >> >>>> + &hpts_that_need_softclock, >> >>>> + "Number of hpts threads that need softclock"); >> >>>> + >> >>>> counter_u64_t hpts_hopelessly_behind; >> >>>> >> >>>> SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, hopeless, >> CTLFLAG_RD, >> >>>> @@ -334,7 +340,7 @@ SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, >> precision, CTLFLAG_RW, >> >>>> &tcp_hpts_precision, 120, >> >>>> "Value for PRE() precision of callout"); >> >>>> SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, cnt_thresh, CTLFLAG_RW, >> >>>> - &conn_cnt_thresh, 0, >> >>>> + &conn_cnt_thresh, DEFAULT_CONNECTION_THESHOLD, >> >>>> "How many connections (below) make us use the callout based >> mechanism"); >> >>>> SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, logging, CTLFLAG_RW, >> >>>> &hpts_does_tp_logging, 0, >> >>>> @@ -1548,6 +1554,9 @@ __tcp_run_hpts(void) >> >>>> struct tcp_hpts_entry *hpts; >> >>>> int ticks_ran; >> >>>> >> >>>> + if (counter_u64_fetch(hpts_that_need_softclock) =3D=3D 0) >> >>>> + return; >> >>>> + >> >>>> hpts =3D tcp_choose_hpts_to_run(); >> >>>> >> >>>> if (hpts->p_hpts_active) { >> >>>> @@ -1683,6 +1692,13 @@ tcp_hpts_thread(void *ctx) >> >>>> ticks_ran =3D tcp_hptsi(hpts, 1); >> >>>> tv.tv_sec =3D 0; >> >>>> tv.tv_usec =3D hpts->p_hpts_sleep_time * HPTS_TICKS_PER_SLOT; >> >>>> + if ((hpts->p_on_queue_cnt > conn_cnt_thresh) && >> (hpts->hit_callout_thresh =3D=3D 0)) { >> >>>> + hpts->hit_callout_thresh =3D 1; >> >>>> + counter_u64_add(hpts_that_need_softclock, 1); >> >>>> + } else if ((hpts->p_on_queue_cnt <=3D conn_cnt_thresh) && >> (hpts->hit_callout_thresh =3D=3D 1)) { >> >>>> + hpts->hit_callout_thresh =3D 0; >> >>>> + counter_u64_add(hpts_that_need_softclock, -1); >> >>>> + } >> >>>> if (hpts->p_on_queue_cnt >=3D conn_cnt_thresh) { >> >>>> if(hpts->p_direct_wake =3D=3D 0) { >> >>>> /* >> >>>> @@ -1818,6 +1834,7 @@ tcp_hpts_mod_load(void) >> >>>> cpu_top =3D NULL; >> >>>> #endif >> >>>> tcp_pace.rp_num_hptss =3D ncpus; >> >>>> + hpts_that_need_softclock =3D counter_u64_alloc(M_WAITOK); >> >>>> hpts_hopelessly_behind =3D counter_u64_alloc(M_WAITOK); >> >>>> hpts_loops =3D counter_u64_alloc(M_WAITOK); >> >>>> back_tosleep =3D counter_u64_alloc(M_WAITOK); >> >>>> @@ -2042,6 +2059,7 @@ tcp_hpts_mod_unload(void) >> >>>> free(tcp_pace.grps, M_TCPHPTS); >> >>>> #endif >> >>>> >> >>>> + counter_u64_free(hpts_that_need_softclock); >> >>>> counter_u64_free(hpts_hopelessly_behind); >> >>>> counter_u64_free(hpts_loops); >> >>>> counter_u64_free(back_tosleep); >> >>> >> >>> >> >> >> >> >> >> >> >> -- >> >> Nuno Teixeira >> >> FreeBSD Committer (ports) >> > >> > >> > >> > -- >> > Nuno Teixeira >> > FreeBSD Committer (ports) >> >> > > -- > Nuno Teixeira > FreeBSD Committer (ports) > --=20 Nuno Teixeira FreeBSD Committer (ports) --000000000000c64a570615bd6820 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
(...)

Backup server is https://www.rsync.net/ (free 500GB for Fr= eeBSD developers).

Nuno Teixeira <eduardo@freebsd.org> escreveu (quarta, 10/04/2024 =C3=A0= (s) 13:39):
With base stack I can complete restic check successfully = downloading/reading/checking all files from a "big" remote compre= ssed backup.
Changing it to RACK stack, it fails.

<= /div>
I run this command often because in the past, compression corrupt= ion occured and this is the equivalent of restoring backup to check its int= egrity.

Maybe someone could do a restic test to ch= eck if this is reproducible.

Thanks,



<tuexen@freebsd.org> escreveu (quarta, 10/04/2024 =C3=A0(= s) 13:12):


> On 10. Apr 2024, at 13:40, Nuno Teixeira <eduardo@freebsd.org> wrote:
>
> Hello all,
>
> @ current 1500018 and fetching torrents with net-p2p/qbittorrent finis= hed ~2GB download and connection UP until the end:
>
> ---
> Apr 10 11:26:46 leg kernel: re0: watchdog timeout
> Apr 10 11:26:46 leg kernel: re0: link state changed to DOWN
> Apr 10 11:26:49 leg dhclient[58810]: New IP Address (re0): 192.168.1.6= 7
> Apr 10 11:26:49 leg dhclient[58814]: New Subnet Mask (re0): 255.255.25= 5.0
> Apr 10 11:26:49 leg dhclient[58818]: New Broadcast Address (re0): 192.= 168.1.255
> Apr 10 11:26:49 leg kernel: re0: link state changed to UP
> Apr 10 11:26:49 leg dhclient[58822]: New Routers (re0): 192.168.1.1 > ---
>
> In the past tests, I've got more watchdog timeouts, connection goe= s down and a reboot needed to put it back (`service netif restart` didn'= ;t work).
>
> Other way to reproduce this is using sysutils/restic (backup program) = to read/check all files from a remote server via sftp:
>
> `restic -r sftp:user@remote:restic-repo check --read-data` from a 60GB= compressed backup.
>
> ---
> watchdog timeout x3 as above
> ---
>
> restic check fail log @ 15% progress:
> ---
> <snip>
> Load(<data/52e2923dd6>, 17310001, 0) returned error, retrying af= ter 1.7670599s: connection lost
> Load(<data/d27a0abe0f>, 17456892, 0) returned error, retrying af= ter 4.619104908s: connection lost
> Load(<data/52e2923dd6>, 17310001, 0) returned error, retrying af= ter 5.477648517s: connection lost
> List(lock) returned error, retrying after 293.057766ms: connection los= t
> List(lock) returned error, retrying after 385.206693ms: connection los= t
> List(lock) returned error, retrying after 1.577594281s: connection los= t
> <snip>
>
> Connection continues UP.
Hi,

I'm not sure what the issue is you are reporting. Could you state
what behavior you are experiencing with the base stack and with
the RACK stack. In particular, what the difference is?

Best regards
Michael
>
> Cheers,
>
> <tuexen@fre= ebsd.org> escreveu (quinta, 28/03/2024 =C3=A0(s) 15:53):
>> On 28. Mar 2024, at 15:00, Nuno Teixeira <eduardo@freebsd.org> wrote:
>>
>> Hello all!
>>
>> Running rack @b7b78c1c169 "Optimize HPTS..." very happy = on my laptop (amd64)!
>>
>> Thanks all!
> Thanks for the feedback!
>
> Best regards
> Michael
>>
>> Drew Gallatin <gallatin@freebsd.org> escreveu (quinta, 21/03/2024 =C3= =A0(s) 12:58):
>> The entire point is to *NOT* go through the overhead of scheduling= something asynchronously, but to take advantage of the fact that a user/ke= rnel transition is going to trash the cache anyway.
>>
>> In the common case of a system which has less than the threshold= =C2=A0 number of connections , we access the tcp_hpts_softclock function po= inter, make one function call, and access hpts_that_need_softclock, and the= n return.=C2=A0 So that's 2 variables and a function call.
>>
>> I think it would be preferable to avoid that call, and to move the= declaration of tcp_hpts_softclock and hpts_that_need_softclock so that the= y are in the same cacheline.=C2=A0 Then we'd be hitting just a single l= ine in the common case.=C2=A0 (I've made comments on the review to that= effect).
>>
>> Also, I wonder if the threshold could get higher by default, so th= at hpts is never called in this context unless we're to the point where= we're scheduling thousands of runs of the hpts thread (and taking all = those clock interrupts).
>>
>> Drew
>>
>> On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote:
>>> On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote:
>>>> Ok I have created
>>>>
>>>> https://reviews.freebsd.org/D44420
>>>>
>>>>
>>>> To address the issue. I also attach a short version of the= patch that Nuno
>>>> can try and validate
>>>>
>>>> it works. Drew you may want to try this and validate the o= ptimization does
>>>> kick in since I can
>>>>
>>>> only now test that it does not on my local box :)
>>> The patch still causes access to all cpu's cachelines on e= ach userret.
>>> It would be much better to inc/check the threshold and only sc= hedule the
>>> call when exceeded.=C2=A0 Then the call can occur in some dedi= cated context,
>>> like per-CPU thread, instead of userret.
>>>
>>>>
>>>>
>>>> R
>>>>
>>>>
>>>>
>>>> On 3/18/24 3:42 PM, Drew Gallatin wrote:
>>>>> No.=C2=A0 The goal is to run on every return to usersp= ace for every thread.
>>>>>
>>>>> Drew
>>>>>
>>>>> On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov = wrote:
>>>>>> On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gal= latin wrote:
>>>>>>> I got the idea from
>>>>>>> h= ttps://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf >>>>>>> The gist is that the TCP pacing stuff needs to= run frequently, and
>>>>>>> rather than run it out of a clock interrupt, i= ts more efficient to run
>>>>>>> it out of a system call context at just the po= int where we return to
>>>>>>> userspace and the cache is trashed anyway. The= current implementation
>>>>>>> is fine for our workload, but probably not ide= a for a generic system.
>>>>>>> Especially one where something is banging on s= ystem calls.
>>>>>>>
>>>>>>> Ast's could be the right tool for this, bu= t I'm super unfamiliar with
>>>>>>> them, and I can't find any docs on them. >>>>>>>
>>>>>>> Would ast_register(0, ASTR_UNCOND, 0, func) be= roughly equivalent to
>>>>>>> what's happening here?
>>>>>> This call would need some AST number added, and th= en it registers the
>>>>>> ast to run on next return to userspace, for the cu= rrent thread.
>>>>>>
>>>>>> Is it enough?
>>>>>>>
>>>>>>> Drew
>>>>>>
>>>>>>>
>>>>>>> On Mon, Mar 18, 2024, at 2:33 PM, Konstantin B= elousov wrote:
>>>>>>>> On Mon, Mar 18, 2024 at 07:26:10AM -0500, = Mike Karels wrote:
>>>>>>>>> On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote: >>>>>>>>>
>>>>>>>>>>> On 18. Mar 2024, at 12:42, Nun= o Teixeira
>>>>>> <eduardo@freebsd.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello all!
>>>>>>>>>>>
>>>>>>>>>>> It works just fine!
>>>>>>>>>>> System performance is OK.
>>>>>>>>>>> Using patch on main-n268841-b0= aaf8beb126(-dirty).
>>>>>>>>>>>
>>>>>>>>>>> ---
>>>>>>>>>>> net.inet.tcp.functions_availab= le:
>>>>>>>>>>> Stack=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0D<= br> >>>>>> Alias=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 PCB count
>>>>>>>>>>> freebsd freebsd=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 0
>>>>>>>>>>> rack=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 *=
>>>>>> rack=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A038
>>>>>>>>>>> ---
>>>>>>>>>>>
>>>>>>>>>>> It would be so nice that we ca= n have a sysctl tunnable for
>>>>>> this patch
>>>>>>>>>>> so we could do more tests with= out recompiling kernel.
>>>>>>>>>> Thanks for testing!
>>>>>>>>>>
>>>>>>>>>> @gallatin: can you come up with a = patch that is acceptable
>>>>>> for Netflix
>>>>>>>>>> and allows to mitigate the perform= ance regression.
>>>>>>>>>
>>>>>>>>> Ideally, tcphpts could enable this aut= omatically when it
>>>>>> starts to be
>>>>>>>>> used (enough?), but a sysctl could sel= ect auto/on/off.
>>>>>>>> There is already a well-known mechanism to= request execution of the
>>>>>>>> specific function on return to userspace, = namely AST.=C2=A0 The difference
>>>>>>>> with the current hack is that the executio= n is requested for one
>>>>>> callback
>>>>>>>> in the context of the specific thread.
>>>>>>>>
>>>>>>>> Still, it might be worth a try to use it; = what is the reason to
>>>>>> hit a thread
>>>>>>>> that does not do networking, with TCP proc= essing?
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>>> Best regards
>>>>>>>>>> Michael
>>>>>>>>>>>
>>>>>>>>>>> Thanks all!
>>>>>>>>>>> Really happy here :)
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> Nuno Teixeira <eduardo@freebsd.org> es= creveu (domingo,
>>>>>> 17/03/2024 =C3=A0(s) 20:26):
>>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>>> I don't have the f= ull context, but it seems like the
>>>>>> complaint is a performance regression in bonnie++ = and perhaps other
>>>>>> things when tcp_hpts is loaded, even when it is no= t used.=C2=A0 Is that
>>>>>> correct?
>>>>>>>>>>>>>
>>>>>>>>>>>>> If so, I suspect its b= ecause we drive the
>>>>>> tcp_hpts_softclock() routine from userret(), in or= der to avoid tons
>>>>>> of timer interrupts and context switches.=C2=A0 To= test this theory,=C2=A0 you
>>>>>> could apply a patch like:
>>>>>>>>>>>>
>>>>>>>>>>>> It's affecting overall= system performance, bonnie was just
>>>>>> a way to
>>>>>>>>>>>> get some numbers to compar= e.
>>>>>>>>>>>>
>>>>>>>>>>>> Tomorrow I will test patch= .
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Nuno Teixeira
>>>>>>>>>>>> FreeBSD Committer (ports)<= br> >>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Nuno Teixeira
>>>>>>>>>>> FreeBSD Committer (ports)
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>
>>>> diff --git a/sys/netinet/tcp_hpts.c b/sys/netinet/tcp_hpts= .c
>>>> index 8c4d2d41a3eb..eadbee19f69c 100644
>>>> --- a/sys/netinet/tcp_hpts.c
>>>> +++ b/sys/netinet/tcp_hpts.c
>>>> @@ -216,6 +216,7 @@ struct tcp_hpts_entry {
>>>> void *ie_cookie;
>>>> uint16_t p_num; /* The hpts number one per cpu */
>>>> uint16_t p_cpu; /* The hpts CPU */
>>>> + uint8_t hit_callout_thresh;
>>>> /* There is extra space in here */
>>>> /* Cache line 0x100 */
>>>> struct callout co __aligned(CACHE_LINE_SIZE);
>>>> @@ -269,6 +270,11 @@ static struct hpts_domain_info {
>>>> int cpu[MAXCPU];
>>>> } hpts_domains[MAXMEMDOM];
>>>>
>>>> +counter_u64_t hpts_that_need_softclock;
>>>> +SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, ne= edsoftclock, CTLFLAG_RD,
>>>> +=C2=A0 =C2=A0 &hpts_that_need_softclock,
>>>> +=C2=A0 =C2=A0 "Number of hpts threads that need soft= clock");
>>>> +
>>>> counter_u64_t hpts_hopelessly_behind;
>>>>
>>>> SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, hop= eless, CTLFLAG_RD,
>>>> @@ -334,7 +340,7 @@ SYSCTL_INT(_net_inet_tcp_hpts, OID_AUT= O, precision, CTLFLAG_RW,
>>>>=C2=A0 =C2=A0 &tcp_hpts_precision, 120,
>>>>=C2=A0 =C2=A0 "Value for PRE() precision of callout&qu= ot;);
>>>> SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, cnt_thresh, CTLFL= AG_RW,
>>>> -=C2=A0 =C2=A0 &conn_cnt_thresh, 0,
>>>> +=C2=A0 =C2=A0 &conn_cnt_thresh, DEFAULT_CONNECTION_TH= ESHOLD,
>>>>=C2=A0 =C2=A0 "How many connections (below) make us us= e the callout based mechanism");
>>>> SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, logging, CTLFLAG_= RW,
>>>>=C2=A0 =C2=A0 &hpts_does_tp_logging, 0,
>>>> @@ -1548,6 +1554,9 @@ __tcp_run_hpts(void)
>>>> struct tcp_hpts_entry *hpts;
>>>> int ticks_ran;
>>>>
>>>> + if (counter_u64_fetch(hpts_that_need_softclock) =3D=3D 0= )
>>>> + return;
>>>> +
>>>> hpts =3D tcp_choose_hpts_to_run();
>>>>
>>>> if (hpts->p_hpts_active) {
>>>> @@ -1683,6 +1692,13 @@ tcp_hpts_thread(void *ctx)
>>>> ticks_ran =3D tcp_hptsi(hpts, 1);
>>>> tv.tv_sec =3D 0;
>>>> tv.tv_usec =3D hpts->p_hpts_sleep_time * HPTS_TICKS_PER= _SLOT;
>>>> + if ((hpts->p_on_queue_cnt > conn_cnt_thresh) &= & (hpts->hit_callout_thresh =3D=3D 0)) {
>>>> + hpts->hit_callout_thresh =3D 1;
>>>> + counter_u64_add(hpts_that_need_softclock, 1);
>>>> + } else if ((hpts->p_on_queue_cnt <=3D conn_cnt_thr= esh) && (hpts->hit_callout_thresh =3D=3D 1)) {
>>>> + hpts->hit_callout_thresh =3D 0;
>>>> + counter_u64_add(hpts_that_need_softclock, -1);
>>>> + }
>>>> if (hpts->p_on_queue_cnt >=3D conn_cnt_thresh) {
>>>> if(hpts->p_direct_wake =3D=3D 0) {
>>>> /*
>>>> @@ -1818,6 +1834,7 @@ tcp_hpts_mod_load(void)
>>>> cpu_top =3D NULL;
>>>> #endif
>>>> tcp_pace.rp_num_hptss =3D ncpus;
>>>> + hpts_that_need_softclock =3D counter_u64_alloc(M_WAITOK)= ;
>>>> hpts_hopelessly_behind =3D counter_u64_alloc(M_WAITOK); >>>> hpts_loops =3D counter_u64_alloc(M_WAITOK);
>>>> back_tosleep =3D counter_u64_alloc(M_WAITOK);
>>>> @@ -2042,6 +2059,7 @@ tcp_hpts_mod_unload(void)
>>>> free(tcp_pace.grps, M_TCPHPTS);
>>>> #endif
>>>>
>>>> + counter_u64_free(hpts_that_need_softclock);
>>>> counter_u64_free(hpts_hopelessly_behind);
>>>> counter_u64_free(hpts_loops);
>>>> counter_u64_free(back_tosleep);
>>>
>>>
>>
>>
>>
>> --
>> Nuno Teixeira
>> FreeBSD Committer (ports)
>
>
>
> --
> Nuno Teixeira
> FreeBSD Committer (ports)



--
Nuno Teixeira
FreeBSD Committ= er (ports)


--
Nuno Teixeira
FreeBSD Committ= er (ports)
--000000000000c64a570615bd6820-- From nobody Fri Apr 12 19:01:00 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VGQrc59Xyz5HvNk for ; Fri, 12 Apr 2024 19:01:04 +0000 (UTC) (envelope-from lexi@le-fay.org) Received: from fuchsia.eden.le-Fay.ORG (fuchsia.eden.le-fay.org [81.187.47.195]) by mx1.freebsd.org (Postfix) with ESMTP id 4VGQrb6qs3z4LT1 for ; Fri, 12 Apr 2024 19:01:03 +0000 (UTC) (envelope-from lexi@le-fay.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=le-fay.org header.s=fuchsia header.b=Z0DnGhOv; dmarc=none; spf=softfail (mx1.freebsd.org: 81.187.47.195 is neither permitted nor denied by domain of lexi@le-fay.org) smtp.mailfrom=lexi@le-fay.org Received: from iris.eden.le-Fay.ORG (iris.eden.le-fay.org [IPv6:2001:8b0:aab5:106:3::6]) by fuchsia.eden.le-Fay.ORG (Postfix) with ESMTP id 2B77E88E5 for ; Fri, 12 Apr 2024 19:01:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=le-fay.org; s=fuchsia; t=1712948461; bh=zFLHMv9zMWF/gumZ0aYiwq2JUJurQEbGU8xsTKqT+BA=; h=Date:From:To:Subject; b=Z0DnGhOvH9paA7bcrYvJbcLHbl3sWToCLPREtPF1DKzCGd7XRxtOTsl/Yh76FPK8k ynWk27ddb/WPW97FQpbaGm8fyQFvwsMXmsVnN4N3zn/xGoRcPFWK+rrePH05rksFqb g/PbU8pOmr2sPg17TL5YlKk0it1wIVIUfY9hkutE= Received: from ilythia.eden.le-fay.org (ilythia.eden.le-fay.org [IPv6:2001:8b0:aab5:106:3::10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by iris.eden.le-Fay.ORG (Postfix) with ESMTPSA id ABC752C0421 for ; Fri, 12 Apr 2024 20:01:00 +0100 (BST) Date: Fri, 12 Apr 2024 20:01:00 +0100 From: Lexi Winter To: net@freebsd.org Subject: netpfil/pf -- review request Message-ID: Mail-Followup-To: net@freebsd.org List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="+LAt56m3yBAKfS5e" Content-Disposition: inline X-Spamd-Bar: - X-Spamd-Result: default: False [-1.80 / 15.00]; VIOLATED_DIRECT_SPF(3.50)[]; SIGNED_PGP(-2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; R_DKIM_ALLOW(-0.20)[le-fay.org:s=fuchsia]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; RCVD_NO_TLS_LAST(0.10)[]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DMARC_NA(0.00)[le-fay.org]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[le-fay.org:+]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:20712, ipnet:81.187.0.0/16, country:GB]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MISSING_XM_UA(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[net@freebsd.org]; MLMMJ_DEST(0.00)[net@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; DWL_DNSWL_NONE(0.00)[le-fay.org:dkim] X-Rspamd-Queue-Id: 4VGQrb6qs3z4LT1 --+LAt56m3yBAKfS5e Content-Type: text/plain; charset=us-ascii Content-Disposition: inline hello, if someone had a chance to review this change to netpfil/pf: https://github.com/freebsd/freebsd-src/pull/1157 i would appreciate that. thanks, lexi. --+LAt56m3yBAKfS5e Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCAAdFiEEuwt6MaPcv/+Mo+ftDHqbqZ41x5kFAmYZhOkACgkQDHqbqZ41 x5mTDAv+Lt7OqEFM6WdKRHc0nBkZ43C9PcG8cEujCIpExP78WuyaWG7EBvcPtr0v b0UTyFHF+Dnwc3CpJdA4kpWexk/iy+pWOIKb3wtzNXKyaFmqzTB5v5QiLQ12l24G jSH7xIeAvCK4CztwMJv4+zCpKBduEoVVKYo6mHsO/L39E6QGbMOrJY5UXpT5yJ6m XfFcfU/9JpkrMYsTiKU2XQbNO/yndKSpcb1JP4fj/+cFm8JbvtBZYdrKBDWHHCWK R5UF6NM49msOYDMoCjpgwPrLjZD0n+BuKH5F69ct0ZiyKNGuc43+2KDf4JDLLklv bFhusYObrpTGStBb9dqRZA3EpxxAnz13Gdvx5O4Q1SDGjZ4/GQw31WvyahfPO+ao bsWmb6MLiA46FaOkAJsqGs/vpZ2BAWr2WHM0HyElWkhfxhGSt0bBfcfXAUG7+bAC QU3RgP9DpTW8iwwKjBNO+JYuea0TZUktpYK4ejRclh5KfDz8vCayy98dG3z0LpK4 UbURYref =6s9H -----END PGP SIGNATURE----- --+LAt56m3yBAKfS5e-- From nobody Sun Apr 14 21:00:22 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VHjPM3NbWz5H3gl for ; Sun, 14 Apr 2024 21:00:23 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VHjPL6SZhz4tPt for ; Sun, 14 Apr 2024 21:00:22 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1713128422; a=rsa-sha256; cv=none; b=WxMdUO/nLf+gf7wSOKFUlpsDV89KnCxIMPjEOMG/uphsmNyl/nrgkDFSZPxvckn3642fFt Mio+ZSlWAe7A3Ax54ryIgkhqUtDYtU6tiN/KAtQeuLXT2TZvTVy1AbMB6cmhieAtij17Nn ne3dSpxPIvh40B8LvJ9LwE/Ge+RqrW28kshswq1uW+iAwVb3CfnStBBHSHC1c3CYnaJ3hM 6vb0BXQq34YU/s6PJbe3H2kn5soAq1GeiNsILH4/dg3gwkdo4nVXVRvILophfcL7gK2ZPp xXxS4CP94btDtxwEebF+g0PQZlENgx0d6gxbvxHugVKy55RlB95P7GQQxEs0KA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1713128422; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=NZPpq3cbc5ZEPQscZmRRnNMRs4EC678kI1aVkAe1PdQ=; b=Thh34LR+I3f0YL6nzjq+eAxZuZTEqVTos7uEvOUtMhMMwp2KABXpihUpx9R2CrBsemUj7s L2yWDzbzzyUh9Ea6WhkpHm53q1M7asuM42lY58JU7n/hVUIUiJU3sYBlLJ1Ovv7Ht1YhRk g3hP5Ce1gjqzfS+2M94Hhx9mSfV5bMfvWGf01QPL7a9ns2qptU7Xs5sfQHJiPHbtF4blP7 qMpbxv0L7cDhBz+ygYC/VbNkgIqSELSFdo+LoKAHMixt1brSTAD6YC7ECD0fKejGbk3ETo rwLFbu4q1ucSvIVk43Z2fO1nxTVrg1wPoQEIFPqkobHVAloQmzBQaUS7zxL6/Q== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4VHjPL64lTzXX2 for ; Sun, 14 Apr 2024 21:00:22 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 43EL0McR079116 for ; Sun, 14 Apr 2024 21:00:22 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 43EL0M2m079104 for net@FreeBSD.org; Sun, 14 Apr 2024 21:00:22 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202404142100.43EL0M2m079104@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: net@FreeBSD.org Subject: Problem reports for net@FreeBSD.org that need special attention Date: Sun, 14 Apr 2024 21:00:22 +0000 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="17131284226.096c8c685.74329" Content-Transfer-Encoding: 7bit --17131284226.096c8c685.74329 Date: Sun, 14 Apr 2024 21:00:22 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 254445 | cloned_interfaces="bridge0" does not respect net. Open | 166724 | if_re(4): watchdog timeout Open | 200836 | iovctl(8): Return descriptions in the returned sc Open | 223824 | Panic in ng_base.c (netgraph) Open | 230807 | if_alc(4): Driver not working for Killer Networki Open | 232472 | ixgbe(4): SR-IOV passthru not working on Hyper-V Open | 234073 | ixl(4): Host X710-DA2 drops connect starting bhyv Open | 241106 | tun/ppp: panic: vm_fault: fault on nofault entry Open | 245981 | bnxt(4): BCM57414 / BCM57416 not initializing: bn Open | 256217 | [tcp] High system load because of interrupts with Open | 257038 | em(4): Panic on HTTP traffic to or from jail thro Open | 257286 | gateway with `ping -6 -e` is ignored Open | 258623 | cxgbe(4): Slow routing performance: 2 numa domain Open | 258850 | lagg(4): interface vanishes when both member inte Open | 261866 | ixgbe(4): Resets media type -> autoselect after s Open | 262024 | em(4): iflib handles bad packets incorrectly Open | 262093 | ixl(4): RX packet errors on Intel X710 after 12.2 Open | 263568 | ix(4): SR-IOV connection lost after loading VM wi In Progress | 118111 | rc: network.subr Add MAC address based interface 19 problems total for which you should take action. --17131284226.096c8c685.74329 Date: Sun, 14 Apr 2024 21:00:22 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
New         |    254445 | cloned_interfaces="bridge0" does not respect net.
Open        |    166724 | if_re(4): watchdog timeout
Open        |    200836 | iovctl(8): Return descriptions in the returned sc
Open        |    223824 | Panic in ng_base.c (netgraph)
Open        |    230807 | if_alc(4): Driver not working for Killer Networki
Open        |    232472 | ixgbe(4): SR-IOV passthru not working on Hyper-V 
Open        |    234073 | ixl(4): Host X710-DA2 drops connect starting bhyv
Open        |    241106 | tun/ppp: panic: vm_fault: fault on nofault entry 
Open        |    245981 | bnxt(4): BCM57414 / BCM57416 not initializing: bn
Open        |    256217 | [tcp] High system load because of interrupts with
Open        |    257038 | em(4): Panic on HTTP traffic to or from jail thro
Open        |    257286 | gateway with `ping -6 -e` is ignored
Open        |    258623 | cxgbe(4): Slow routing performance: 2 numa domain
Open        |    258850 | lagg(4): interface vanishes when both member inte
Open        |    261866 | ixgbe(4): Resets media type -> autoselect after s
Open        |    262024 | em(4): iflib handles bad packets incorrectly
Open        |    262093 | ixl(4): RX packet errors on Intel X710 after 12.2
Open        |    263568 | ix(4): SR-IOV connection lost after loading VM wi
In Progress |    118111 | rc: network.subr Add MAC address based interface 

19 problems total for which you should take action.
--17131284226.096c8c685.74329--