From nobody Mon Mar 18 11:42:48 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TytJh2x8Lz5DYVH; Mon, 18 Mar 2024 11:43:00 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TytJh2WPWz41V2; Mon, 18 Mar 2024 11:43:00 +0000 (UTC) (envelope-from eduardo@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1710762180; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nRW1zKiuHdfkRBaVz8VhI700iA151KX5T6T7h+C7OmI=; b=VplVI3hsHi/DAQ9CC+Az/nlItLz3U0KGpRO27aVAuz6/SB3mh9cKKCbEC36smoNFW2KoaU TFquuXzGjOtYMztqtPVNDwBAf5XfbHz/4Lsr8fBKJxVgCrZ1APyPztytc2u3ThbzQaPYcS 7zIrSeh2ZRybSJqnIN7n2+m6BqTKfL4wPkiB7MMtGasIz7+10iYzbfT8EvV6b2HfF1FmPk AFJWkjwaijwM7yhgqO/enyyykHcgWrIxTrWkOBtns4KQgZCMJYXUwVNS7X+YNyNLNDv9q3 z6FKeGv0Jr5909JA9pQrD8Jy5XbGfROHftN02DnaoZPcaNH+QTmXtdNZ/fiECQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1710762180; a=rsa-sha256; cv=none; b=LqUEI6u1lQiGBldQlundAvbdBcePuAR2V7262iwhZ6XNqP7F5gP6fSET31gwcHcaj83UEH jKtTXl9clEKSLo3f8mG7+qPb3r8Gm83bQ5CxcM28SW8bpCjBXffWhRU/oh/4ysgBrlXUdl E5eqbxTxT8JaT5Fxy+fnpFfRVbIEhAbXZD13D8IP1NdG+A4nUkGM4QQkNtwkmMl6maJnM6 Agz3/6/3PK3KiT59/7/5BsDPE+k90jsQuZnsYjasoJaH7IcLmr1ObOJN84YXxfVv5ojh8h BoPl7ywvgRsIrqalP2Jh3V2/Kc2yM9xU1eSucKc4dNh2E9ER6nJhTW+UNm02wg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1710762180; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nRW1zKiuHdfkRBaVz8VhI700iA151KX5T6T7h+C7OmI=; b=ToFI5GV7ky7ztM2bGkvR8ZF7KyXakTW0HzT6i51uG6XIOJYzCiYgrz04C/Xq7newcUt2I8 HmPGOy3ysCxk3TO46MKxuL+9CUr8HU2N1gnhJcSbWYQHgKdD1OKE+LdjRUTU52iEz1tTUN allbzkayumN2dII/yxIvPQxi6LDtahATjtemT5uVjenDnWSTFJ8lR1G1bXzaLw+zxF6IbV /qw1Co3gEqDt7N7brKsh9XmL8cyXPGrTuclgKLvilPFvbGVC+VHZAbCi3U0+F/bOISofQJ VTIHFPY6bWKHU1wuVKNX0Zl1S75heaBb2wFQK27SJ+meJZnxCtCgWPtPC56KYg== Received: from mail-yb1-f170.google.com (mail-yb1-f170.google.com [209.85.219.170]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: eduardo) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TytJh2045zFcY; Mon, 18 Mar 2024 11:43:00 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: by mail-yb1-f170.google.com with SMTP id 3f1490d57ef6-dd161eb03afso3508443276.0; Mon, 18 Mar 2024 04:43:00 -0700 (PDT) X-Forwarded-Encrypted: i=1; AJvYcCWI+9L6oc6rkCRNCb2F5fT/iFW29yVs5SwDG2ZQbk2TmR2bWP+KK619nQZz+0X3VRaDrOw32x8Pp7TtAV6/Z4KQssnXRu69s1UMB5aENTO9BWAfOpyheblBVAetk4CC3NJyBfculA== X-Gm-Message-State: AOJu0Yz8o6MY5hoO0xu67Pd8Ld2CG01nxu0rvBmaiFPv9N2kJGxMxuhg Vfsa2lJ5H3oW8zlOldF74JnlEk3IJ83acymQp7SYQYUE6VMGZxZWe+dVsd+ftFmLD3ymV1pn0nx naOelOEusoycdVqXLY7YsME0mhMA= X-Google-Smtp-Source: AGHT+IEVGcLQaZvLj2F2msTlyRw8wgpGp7EpaTC93rNd7cTVEkp6hphPXf+/wSRuTrGZlmLIeMSrEpvJ3QNahO+lP8Q= X-Received: by 2002:a5b:cc2:0:b0:dd1:37ff:696 with SMTP id e2-20020a5b0cc2000000b00dd137ff0696mr8777489ybr.59.1710762179517; Mon, 18 Mar 2024 04:42:59 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 References: <42C327BD-6CE4-43AA-A1AE-3BEC08D623DB@freebsd.org> <486915F0-456B-4B09-A8BC-93BBA79C4CA1@freebsd.org> <20240313080624.6c73908c@ernst.home> <508E3B47-8E1B-469F-97B1-2171A3098888@freebsd.org> <86a5n1i0xg.fsf@ltc.des.dev> <78D1FF09-71A3-4486-B934-D8332F54B237@freebsd.org> <20240316104053.20bef8c2@ernst.home> <20240316115128.33d11f7b@ernst.home> <7367F29A-D52B-4828-B79A-AA2667E81E7D@freebsd.org> <4FF534F6-B35D-4596-8D1E-226AD1347AC8@freebsd.org> <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> In-Reply-To: From: Nuno Teixeira Date: Mon, 18 Mar 2024 11:42:48 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Request for Testing: TCP RACK To: Drew Gallatin Cc: tuexen@freebsd.org, garyj@gmx.de, current@freebsd.org, net@freebsd.org, Randall Stewart Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello all! It works just fine! System performance is OK. Using patch on main-n268841-b0aaf8beb126(-dirty). --- net.inet.tcp.functions_available: Stack D Alias PCB coun= t freebsd freebsd 0 rack * rack 38 --- It would be so nice that we can have a sysctl tunnable for this patch so we could do more tests without recompiling kernel. Thanks all! Really happy here :) Cheers, Nuno Teixeira escreveu (domingo, 17/03/2024 =C3=A0(s)= 20:26): > > Hello, > > > I don't have the full context, but it seems like the complaint is a per= formance regression in bonnie++ and perhaps other things when tcp_hpts is l= oaded, even when it is not used. Is that correct? > > > > If so, I suspect its because we drive the tcp_hpts_softclock() routine = from userret(), in order to avoid tons of timer interrupts and context swit= ches. To test this theory, you could apply a patch like: > > It's affecting overall system performance, bonnie was just a way to > get some numbers to compare. > > Tomorrow I will test patch. > > Thanks! > > -- > Nuno Teixeira > FreeBSD Committer (ports) --=20 Nuno Teixeira FreeBSD Committer (ports) From nobody Mon Mar 18 12:04:40 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tytns5zP5z5Dbj5; Mon, 18 Mar 2024 12:04:49 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tytns1Q0pz44Xs; Mon, 18 Mar 2024 12:04:49 +0000 (UTC) (envelope-from tuexen@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (unknown [IPv6:2a02:c6a0:3052:202:550a:8b4c:ffef:3f6f]) (Authenticated sender: micmac) by drew.franken.de (Postfix) with ESMTPSA id 6DD94721E2806; Mon, 18 Mar 2024 13:04:41 +0100 (CET) Content-Type: text/plain; charset=utf-8 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.500.171.1.1\)) Subject: Re: Request for Testing: TCP RACK From: tuexen@freebsd.org In-Reply-To: Date: Mon, 18 Mar 2024 13:04:40 +0100 Cc: garyj@gmx.de, current@freebsd.org, net@freebsd.org, Randall Stewart Content-Transfer-Encoding: quoted-printable Message-Id: References: <42C327BD-6CE4-43AA-A1AE-3BEC08D623DB@freebsd.org> <486915F0-456B-4B09-A8BC-93BBA79C4CA1@freebsd.org> <20240313080624.6c73908c@ernst.home> <508E3B47-8E1B-469F-97B1-2171A3098888@freebsd.org> <86a5n1i0xg.fsf@ltc.des.dev> <78D1FF09-71A3-4486-B934-D8332F54B237@freebsd.org> <20240316104053.20bef8c2@ernst.home> <20240316115128.33d11f7b@ernst.home> <7367F29A-D52B-4828-B79A-AA2667E81E7D@freebsd.org> <4FF534F6-B35D-4596-8D1E-226AD1347AC8@freebsd.org> <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> To: Nuno Teixeira , Drew Gallatin X-Mailer: Apple Mail (2.3774.500.171.1.1) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, T_SCC_BODY_TEXT_LINE autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:680, ipnet:2001:638::/32, country:DE] X-Rspamd-Queue-Id: 4Tytns1Q0pz44Xs > On 18. Mar 2024, at 12:42, Nuno Teixeira wrote: >=20 > Hello all! >=20 > It works just fine! > System performance is OK. > Using patch on main-n268841-b0aaf8beb126(-dirty). >=20 > --- > net.inet.tcp.functions_available: > Stack D Alias PCB = count > freebsd freebsd 0 > rack * rack 38 > --- >=20 > It would be so nice that we can have a sysctl tunnable for this patch > so we could do more tests without recompiling kernel. Thanks for testing! @gallatin: can you come up with a patch that is acceptable for Netflix and allows to mitigate the performance regression. Best regards Michael >=20 > Thanks all! > Really happy here :) >=20 > Cheers, >=20 > Nuno Teixeira escreveu (domingo, 17/03/2024 = =C3=A0(s) 20:26): >>=20 >> Hello, >>=20 >>> I don't have the full context, but it seems like the complaint is a = performance regression in bonnie++ and perhaps other things when = tcp_hpts is loaded, even when it is not used. Is that correct? >>>=20 >>> If so, I suspect its because we drive the tcp_hpts_softclock() = routine from userret(), in order to avoid tons of timer interrupts and = context switches. To test this theory, you could apply a patch like: >>=20 >> It's affecting overall system performance, bonnie was just a way to >> get some numbers to compare. >>=20 >> Tomorrow I will test patch. >>=20 >> Thanks! >>=20 >> -- >> Nuno Teixeira >> FreeBSD Committer (ports) >=20 >=20 >=20 > --=20 > Nuno Teixeira > FreeBSD Committer (ports) From nobody Mon Mar 18 12:26:10 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TyvGg3NcDz5DdcB; Mon, 18 Mar 2024 12:26:19 +0000 (UTC) (envelope-from mike@karels.net) Received: from mail2.karels.net (mail2.karels.net [3.19.118.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "freebsd", Issuer "freebsd" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TyvGf65Zgz47fH; Mon, 18 Mar 2024 12:26:18 +0000 (UTC) (envelope-from mike@karels.net) Authentication-Results: mx1.freebsd.org; none Received: from mail2.karels.net (localhost [IPv6:0:0:0:0:0:0:0:1]) by mail2.karels.net (8.18.1/8.18.1) with ESMTP id 42ICQBBv054697; Mon, 18 Mar 2024 07:26:11 -0500 (CDT) (envelope-from mike@karels.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=karels.net; s=mail2; t=1710764772; bh=ewZYGbxI4QWGlXbwewAilCFxYmXLkdBwORFYqLEWzNw=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=ap7WZBuT+KXXggGCehJWapm7X7UfcU/kFTSJVhf/RurKXnmp+WBJ8yTl/TiymFtv/ ++p8/jmHeGEhduWXotDAOnZkRZ9iok3d4UGEUxWTJ5jc/3U8X0qAnndHEuByEp7tfk Uub65E/PjBb0eqfAvqn1ioGJI1JourWHg41XD1OJotfdDNxAQMmNw731cGiCjkcRP0 P6GdaV1nkWGjwGTfgoYurljaqSfuDt3V6Ot8YSQlzEyQ0UM0Q+qKMVqcqZs71jRJIR GKwE2yY4VWX7e/a75qeCaZiVvVGAQ4O9AHk+6flLT0l6LFzkXMqswnbGpCgkV1QgPA wUQHc309o9frg== Received: from [10.0.2.130] ([73.62.165.147]) by mail2.karels.net with ESMTPSA id jWyrL+My+GWn1QAAs/W3XQ (envelope-from ); Mon, 18 Mar 2024 07:26:11 -0500 From: Mike Karels To: tuexen@freebsd.org Cc: Nuno Teixeira , Drew Gallatin , garyj@gmx.de, current@freebsd.org, net@freebsd.org, Randall Stewart Subject: Re: Request for Testing: TCP RACK Date: Mon, 18 Mar 2024 07:26:10 -0500 X-Mailer: MailMate (1.14r6015) Message-ID: <6047C8EF-B1B0-4286-93FA-AA38F8A18656@karels.net> In-Reply-To: References: <42C327BD-6CE4-43AA-A1AE-3BEC08D623DB@freebsd.org> <486915F0-456B-4B09-A8BC-93BBA79C4CA1@freebsd.org> <20240313080624.6c73908c@ernst.home> <508E3B47-8E1B-469F-97B1-2171A3098888@freebsd.org> <86a5n1i0xg.fsf@ltc.des.dev> <78D1FF09-71A3-4486-B934-D8332F54B237@freebsd.org> <20240316104053.20bef8c2@ernst.home> <20240316115128.33d11f7b@ernst.home> <7367F29A-D52B-4828-B79A-AA2667E81E7D@freebsd.org> <4FF534F6-B35D-4596-8D1E-226AD1347AC8@freebsd.org> <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:16509, ipnet:3.16.0.0/14, country:US] X-Rspamd-Queue-Id: 4TyvGf65Zgz47fH On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote: >> On 18. Mar 2024, at 12:42, Nuno Teixeira wrote: >> >> Hello all! >> >> It works just fine! >> System performance is OK. >> Using patch on main-n268841-b0aaf8beb126(-dirty). >> >> --- >> net.inet.tcp.functions_available: >> Stack D Alias PCB= count >> freebsd freebsd 0 >> rack * rack 38 >> --- >> >> It would be so nice that we can have a sysctl tunnable for this patch >> so we could do more tests without recompiling kernel. > Thanks for testing! > > @gallatin: can you come up with a patch that is acceptable for Netflix > and allows to mitigate the performance regression. Ideally, tcphpts could enable this automatically when it starts to be used (enough?), but a sysctl could select auto/on/off. Mike > Best regards > Michael >> >> Thanks all! >> Really happy here :) >> >> Cheers, >> >> Nuno Teixeira escreveu (domingo, 17/03/2024 =C3=A0= (s) 20:26): >>> >>> Hello, >>> >>>> I don't have the full context, but it seems like the complaint is a = performance regression in bonnie++ and perhaps other things when tcp_hpts= is loaded, even when it is not used. Is that correct? >>>> >>>> If so, I suspect its because we drive the tcp_hpts_softclock() routi= ne from userret(), in order to avoid tons of timer interrupts and context= switches. To test this theory, you could apply a patch like: >>> >>> It's affecting overall system performance, bonnie was just a way to >>> get some numbers to compare. >>> >>> Tomorrow I will test patch. >>> >>> Thanks! >>> >>> -- >>> Nuno Teixeira >>> FreeBSD Committer (ports) >> >> >> >> -- = >> Nuno Teixeira >> FreeBSD Committer (ports) From nobody Mon Mar 18 18:33:16 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tz3QS2HNjz5FDfn for ; Mon, 18 Mar 2024 18:33:36 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tz3QS0B95z3yfW for ; Mon, 18 Mar 2024 18:33:35 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1] (may be forged)) by kib.kiev.ua (8.18.1/8.18.1) with ESMTP id 42IIXHW2082887; Mon, 18 Mar 2024 20:33:20 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 42IIXHW2082887 Received: (from kostik@localhost) by tom.home (8.18.1/8.18.1/Submit) id 42IIXH90082886; Mon, 18 Mar 2024 20:33:17 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 18 Mar 2024 20:33:16 +0200 From: Konstantin Belousov To: Mike Karels Cc: tuexen@freebsd.org, Nuno Teixeira , Drew Gallatin , garyj@gmx.de, current@freebsd.org, net@freebsd.org, Randall Stewart Subject: Re: Request for Testing: TCP RACK Message-ID: References: <4FF534F6-B35D-4596-8D1E-226AD1347AC8@freebsd.org> <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> <6047C8EF-B1B0-4286-93FA-AA38F8A18656@karels.net> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <6047C8EF-B1B0-4286-93FA-AA38F8A18656@karels.net> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Queue-Id: 4Tz3QS0B95z3yfW On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote: > > >> On 18. Mar 2024, at 12:42, Nuno Teixeira wrote: > >> > >> Hello all! > >> > >> It works just fine! > >> System performance is OK. > >> Using patch on main-n268841-b0aaf8beb126(-dirty). > >> > >> --- > >> net.inet.tcp.functions_available: > >> Stack D Alias PCB count > >> freebsd freebsd 0 > >> rack * rack 38 > >> --- > >> > >> It would be so nice that we can have a sysctl tunnable for this patch > >> so we could do more tests without recompiling kernel. > > Thanks for testing! > > > > @gallatin: can you come up with a patch that is acceptable for Netflix > > and allows to mitigate the performance regression. > > Ideally, tcphpts could enable this automatically when it starts to be > used (enough?), but a sysctl could select auto/on/off. There is already a well-known mechanism to request execution of the specific function on return to userspace, namely AST. The difference with the current hack is that the execution is requested for one callback in the context of the specific thread. Still, it might be worth a try to use it; what is the reason to hit a thread that does not do networking, with TCP processing? > > Mike > > > Best regards > > Michael > >> > >> Thanks all! > >> Really happy here :) > >> > >> Cheers, > >> > >> Nuno Teixeira escreveu (domingo, 17/03/2024 à(s) 20:26): > >>> > >>> Hello, > >>> > >>>> I don't have the full context, but it seems like the complaint is a performance regression in bonnie++ and perhaps other things when tcp_hpts is loaded, even when it is not used. Is that correct? > >>>> > >>>> If so, I suspect its because we drive the tcp_hpts_softclock() routine from userret(), in order to avoid tons of timer interrupts and context switches. To test this theory, you could apply a patch like: > >>> > >>> It's affecting overall system performance, bonnie was just a way to > >>> get some numbers to compare. > >>> > >>> Tomorrow I will test patch. > >>> > >>> Thanks! > >>> > >>> -- > >>> Nuno Teixeira > >>> FreeBSD Committer (ports) > >> > >> > >> > >> -- > >> Nuno Teixeira > >> FreeBSD Committer (ports) > From nobody Mon Mar 18 19:13:11 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tz4JY0z7Mz5Dnb8; Mon, 18 Mar 2024 19:13:33 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tz4JY0PwGz44Jt; Mon, 18 Mar 2024 19:13:33 +0000 (UTC) (envelope-from gallatin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1710789213; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=LBPHA0DV2V4AA/r8GnMiwoXOA7ru9W10xJUHU6dWmyY=; b=xkXtsWbWGBgK/RI4ZpWEISyJ0SYG1afTzfwLf8Xakj9LWXPsQT8EXUybqhx2BFxjZqukEY RiCWlenfTqDhQdbNIybGUldF7VceMrwgOSPZt/nAr2wg2mKtoArbzKFNV171QoJACp7T0X ERVWXY5xCxaPWThDS/CpTu2XRoU9Cu4TmyWse4uSopKDjGymnoAT7+kcP3m5gLjlscIvwL EA1eF8wX8ohqwhm5YVx41eDQwwiftbkIKD7mlBBVahj0DxFYkTmCtq3pEaQzsOOhBo4Gss 2aH5+Et8yYCD/oeHGz4UnEfsKj/sIN2nwI27eNclrGzFT54rNOdrodPvqOpL7A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1710789213; a=rsa-sha256; cv=none; b=VU2CMXmlKcoQJJUz5zJr0RorQiuNX8f3v3ZMBO7M+vrL+6yYnjzT80f28dUu7zA2tZN025 TA5bSGAm2mu9Sc7N0LvSGh+hg4F6FuNFgtjTCjQhVopFWPiymczMNCtvOgPRF3FPDlrllx mDhfnx1b4knaJcmSBo3SwjJXACoUV8fa2AU4nnk3yiE+rDL9pRB0c3uNb7fFmOC9TkwbDn Y3npys97HLxtv2YNYRVGQXFYPsQLAqXuT4f6Uf7FKpcYYdh0NhwKmr+5Vpkua6qiPnovuk Av7Jf0Ijzb3syEPvvAlQGbE3D87+dGUc6y6ELf48T/OSo0Zbo4LH6R1awTBHkg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1710789213; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=LBPHA0DV2V4AA/r8GnMiwoXOA7ru9W10xJUHU6dWmyY=; b=XBW4GC6kX1lbQ+rQfFGVWzDBDTBds4bQUgSAuwffCu3TnnHKtqc3hkoF06k3JwM3W0N2La oBE09Kcf+2mkaIioKVSInuubXfhbpdfw7YSLrMqSk+0o6OyAiNRuKmwwhvWk/SQE/Tvepi Om3llYeP4WdLR78NwUalGGjauCz4y6vsa19I5msuZj1DO9+D8ioAHRx4V/pXg7m5aqNVoO A7KnyQR/GTFwwmjVeNDw3W9oa6iyTwPvy10sMeiDNuAjLO4tEmeOlrVbnWXFMUDvnZ6AyZ BTn+gxKrhdYywZtxQt0rTp55kEiZpNg2+gDetu2XbINgWhQYfNNCMjJc1V+Xyw== Received: from fauth2-smtp.messagingengine.com (fauth2-smtp.messagingengine.com [103.168.172.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: gallatin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Tz4JX4pn9zPyv; Mon, 18 Mar 2024 19:13:32 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfauth.nyi.internal (Postfix) with ESMTP id 8E7E71200066; Mon, 18 Mar 2024 15:13:31 -0400 (EDT) Received: from imap53 ([10.202.2.103]) by compute5.internal (MEProxy); Mon, 18 Mar 2024 15:13:31 -0400 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgdduvddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvfevufgtsegrtderreerreejnecuhfhrohhmpedfffhr vgifucfirghllhgrthhinhdfuceoghgrlhhlrghtihhnsehfrhgvvggsshgurdhorhhgqe enucggtffrrghtthgvrhhnpefgteeluefggeevheefheejgedtvdelheejfffhgeeuhfel keevheeiieeuhfefieenucffohhmrghinhepmhhpihdqshifshdrohhrghenucevlhhush htvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehgrghllhgrthhinhdo mhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqudeffeehledvvdduiedqvdelhe dtgedukeegqdhgrghllhgrthhinheppehfrhgvvggsshgurdhorhhgsehfrghsthhmrghi lhdrtghomh X-ME-Proxy: Feedback-ID: i41414658:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 52F5C36400BD; Mon, 18 Mar 2024 15:13:31 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-300-gdee1775a43-fm-20240315.001-gdee1775a List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Message-Id: <8031cd99-ded8-4b06-93b3-11cc729a8b2c@app.fastmail.com> In-Reply-To: References: <4FF534F6-B35D-4596-8D1E-226AD1347AC8@freebsd.org> <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> <6047C8EF-B1B0-4286-93FA-AA38F8A18656@karels.net> Date: Mon, 18 Mar 2024 15:13:11 -0400 From: "Drew Gallatin" To: "Konstantin Belousov" , "Mike Karels" Cc: tuexen , "Nuno Teixeira" , garyj@gmx.de, current@freebsd.org, net@freebsd.org, "Randall Stewart" Subject: Re: Request for Testing: TCP RACK Content-Type: multipart/alternative; boundary=01b96c257b37417295d61c17eb06343b --01b96c257b37417295d61c17eb06343b Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable I got the idea from https://people.mpi-sws.org/~druschel/publications/so= ft-timers-tocs.pdf The gist is that the TCP pacing stuff needs to run f= requently, and rather than run it out of a clock interrupt, its more eff= icient to run it out of a system call context at just the point where we= return to userspace and the cache is trashed anyway. The current impl= ementation is fine for our workload, but probably not idea for a generic= system. Especially one where something is banging on system calls. =20 Ast's could be the right tool for this, but I'm super unfamiliar with th= em, and I can't find any docs on them.=20 Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent to wha= t's happening here? Drew On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: > On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > > On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote: > >=20 > > >> On 18. Mar 2024, at 12:42, Nuno Teixeira wr= ote: > > >> > > >> Hello all! > > >> > > >> It works just fine! > > >> System performance is OK. > > >> Using patch on main-n268841-b0aaf8beb126(-dirty). > > >> > > >> --- > > >> net.inet.tcp.functions_available: > > >> Stack D Alias = PCB count > > >> freebsd freebsd = 0 > > >> rack * rack = 38 > > >> --- > > >> > > >> It would be so nice that we can have a sysctl tunnable for this p= atch > > >> so we could do more tests without recompiling kernel. > > > Thanks for testing! > > > > > > @gallatin: can you come up with a patch that is acceptable for Net= flix > > > and allows to mitigate the performance regression. > >=20 > > Ideally, tcphpts could enable this automatically when it starts to be > > used (enough?), but a sysctl could select auto/on/off. > There is already a well-known mechanism to request execution of the > specific function on return to userspace, namely AST. The difference > with the current hack is that the execution is requested for one callb= ack > in the context of the specific thread. >=20 > Still, it might be worth a try to use it; what is the reason to hit a = thread > that does not do networking, with TCP processing? >=20 > >=20 > > Mike > >=20 > > > Best regards > > > Michael > > >> > > >> Thanks all! > > >> Really happy here :) > > >> > > >> Cheers, > > >> > > >> Nuno Teixeira escreveu (domingo, 17/03/2024= =C3=A0(s) 20:26): > > >>> > > >>> Hello, > > >>> > > >>>> I don't have the full context, but it seems like the complaint = is a performance regression in bonnie++ and perhaps other things when tc= p_hpts is loaded, even when it is not used. Is that correct? > > >>>> > > >>>> If so, I suspect its because we drive the tcp_hpts_softclock() = routine from userret(), in order to avoid tons of timer interrupts and c= ontext switches. To test this theory, you could apply a patch like: > > >>> > > >>> It's affecting overall system performance, bonnie was just a way= to > > >>> get some numbers to compare. > > >>> > > >>> Tomorrow I will test patch. > > >>> > > >>> Thanks! > > >>> > > >>> -- > > >>> Nuno Teixeira > > >>> FreeBSD Committer (ports) > > >> > > >> > > >> > > >> --=20 > > >> Nuno Teixeira > > >> FreeBSD Committer (ports) > >=20 >=20 --01b96c257b37417295d61c17eb06343b Content-Type: text/html;charset=utf-8 Content-Transfer-Encoding: quoted-printable
I got the idea = from https://people.mpi-sws.org/~druschel/publications/s= oft-timers-tocs.pdf  The gist is that the TCP pacing stuff need= s to run frequently, and rather than run it out of a clock interrupt, it= s more efficient to run it out of a system call context at just the poin= t where we return to userspace and the cache is trashed anyway. &nb= sp; The current implementation is fine for our workload, but probably no= t idea for a generic system.  Especially one where something is ban= ging on system calls. 

Ast's could be= the right tool for this, but I'm super unfamiliar with them, and I can'= t find any docs on them.

Would ast_registe= r(0, ASTR_UNCOND, 0, func) be roughly equivalent to what's happening her= e?

Drew

On Mon= , Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote:
On Mon, Mar 18, 2024 at 07:2= 6:10AM -0500, Mike Karels wrote:
> On 18 Mar 2024, at 7= :04, tuexen@freebsd.org w= rote:
> 
> >> On 18. Mar 20= 24, at 12:42, Nuno Teixeira <e= duardo@freebsd.org> wrote:
> >>
<= div>> >> Hello all!
> >>
&= gt; >> It works just fine!
> >> System perf= ormance is OK.
> >> Using patch on main-n268841-b= 0aaf8beb126(-dirty).
> >>
> >= > ---
> >> net.inet.tcp.functions_available:
> >> Stack      &nbs= p;           &nbs= p;        D Alias   &n= bsp;           &n= bsp;            P= CB count
> >> freebsd    &nbs= p;           &nbs= p;          freebsd &n= bsp;           &n= bsp;            0=
> >> rack      &nb= sp;           &nb= sp;         * rack  &n= bsp;           &n= bsp;           &n= bsp;  38
> >> ---
> >>= ;
> >> It would be so nice that we can have a sys= ctl tunnable for this patch
> >> so we could do m= ore tests without recompiling kernel.
> > Thanks for= testing!
> >
> > @gallatin: can= you come up with a patch that is acceptable for Netflix
&= gt; > and allows to mitigate the performance regression.
> 
> Ideally, tcphpts could enable this autom= atically when it starts to be
> used (enough?), but a s= ysctl could select auto/on/off.
There is already a well-kn= own mechanism to request execution of the
specific functio= n on return to userspace, namely AST.  The difference
with the current hack is that the execution is requested for one callba= ck
in the context of the specific thread.
Still, it might be worth a try to use it; what is the reaso= n to hit a thread
that does not do networking, with TCP pr= ocessing?

> 
> M= ike
> 
> > Best regards
> > Michael
> >>
>= >> Thanks all!
> >> Really happy here :)
> >>
> >> Cheers,
=
> >>
> >> Nuno Teixeira <eduardo@freebsd.org> escreveu (do= mingo, 17/03/2024 =C3=A0(s) 20:26):
> >>>
<= /div>
> >>> Hello,
> >>>
> >>>> I don't have the full context, but it see= ms like the complaint is a performance regression in bonnie++ and perhap= s other things when tcp_hpts is loaded, even when it is not used.  = Is that correct?
> >>>>
> = >>>> If so, I suspect its because we drive the tcp_hpts_soft= clock() routine from userret(), in order to avoid tons of timer interrup= ts and context switches.  To test this theory,  you could appl= y a patch like:
> >>>
> >&= gt;> It's affecting overall system performance, bonnie was just a way= to
> >>> get some numbers to compare.
> >>>
> >>> Tomorrow I will= test patch.
> >>>
> >>= > Thanks!
> >>>
> >>= > --
> >>> Nuno Teixeira
>= >>> FreeBSD Committer (ports)
> >>
<= /div>
> >>
> >>
> &= gt;> -- 
> >> Nuno Teixeira
> >> FreeBSD Committer (ports)
> 


--01b96c257b37417295d61c17eb06343b-- From nobody Mon Mar 18 19:41:47 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tz4xM0w25z5DrFT; Mon, 18 Mar 2024 19:41:59 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tz4xL64Xyz47lj; Mon, 18 Mar 2024 19:41:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1] (may be forged)) by kib.kiev.ua (8.18.1/8.18.1) with ESMTP id 42IJflGB000230; Mon, 18 Mar 2024 21:41:50 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 42IJflGB000230 Received: (from kostik@localhost) by tom.home (8.18.1/8.18.1/Submit) id 42IJflUr000229; Mon, 18 Mar 2024 21:41:47 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 18 Mar 2024 21:41:47 +0200 From: Konstantin Belousov To: Drew Gallatin Cc: Mike Karels , tuexen , Nuno Teixeira , garyj@gmx.de, current@freebsd.org, net@freebsd.org, Randall Stewart Subject: Re: Request for Testing: TCP RACK Message-ID: References: <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> <6047C8EF-B1B0-4286-93FA-AA38F8A18656@karels.net> <8031cd99-ded8-4b06-93b3-11cc729a8b2c@app.fastmail.com> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <8031cd99-ded8-4b06-93b3-11cc729a8b2c@app.fastmail.com> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Queue-Id: 4Tz4xL64Xyz47lj On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: > I got the idea from > https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf > The gist is that the TCP pacing stuff needs to run frequently, and > rather than run it out of a clock interrupt, its more efficient to run > it out of a system call context at just the point where we return to > userspace and the cache is trashed anyway. The current implementation > is fine for our workload, but probably not idea for a generic system. > Especially one where something is banging on system calls. > > Ast's could be the right tool for this, but I'm super unfamiliar with > them, and I can't find any docs on them. > > Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent to > what's happening here? This call would need some AST number added, and then it registers the ast to run on next return to userspace, for the current thread. Is it enough? > > Drew > > On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: > > On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > > > On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote: > > > > > > >> On 18. Mar 2024, at 12:42, Nuno Teixeira wrote: > > > >> > > > >> Hello all! > > > >> > > > >> It works just fine! > > > >> System performance is OK. > > > >> Using patch on main-n268841-b0aaf8beb126(-dirty). > > > >> > > > >> --- > > > >> net.inet.tcp.functions_available: > > > >> Stack D Alias PCB count > > > >> freebsd freebsd 0 > > > >> rack * rack 38 > > > >> --- > > > >> > > > >> It would be so nice that we can have a sysctl tunnable for this patch > > > >> so we could do more tests without recompiling kernel. > > > > Thanks for testing! > > > > > > > > @gallatin: can you come up with a patch that is acceptable for Netflix > > > > and allows to mitigate the performance regression. > > > > > > Ideally, tcphpts could enable this automatically when it starts to be > > > used (enough?), but a sysctl could select auto/on/off. > > There is already a well-known mechanism to request execution of the > > specific function on return to userspace, namely AST. The difference > > with the current hack is that the execution is requested for one callback > > in the context of the specific thread. > > > > Still, it might be worth a try to use it; what is the reason to hit a thread > > that does not do networking, with TCP processing? > > > > > > > > Mike > > > > > > > Best regards > > > > Michael > > > >> > > > >> Thanks all! > > > >> Really happy here :) > > > >> > > > >> Cheers, > > > >> > > > >> Nuno Teixeira escreveu (domingo, 17/03/2024 à(s) 20:26): > > > >>> > > > >>> Hello, > > > >>> > > > >>>> I don't have the full context, but it seems like the complaint is a performance regression in bonnie++ and perhaps other things when tcp_hpts is loaded, even when it is not used. Is that correct? > > > >>>> > > > >>>> If so, I suspect its because we drive the tcp_hpts_softclock() routine from userret(), in order to avoid tons of timer interrupts and context switches. To test this theory, you could apply a patch like: > > > >>> > > > >>> It's affecting overall system performance, bonnie was just a way to > > > >>> get some numbers to compare. > > > >>> > > > >>> Tomorrow I will test patch. > > > >>> > > > >>> Thanks! > > > >>> > > > >>> -- > > > >>> Nuno Teixeira > > > >>> FreeBSD Committer (ports) > > > >> > > > >> > > > >> > > > >> -- > > > >> Nuno Teixeira > > > >> FreeBSD Committer (ports) > > > > > From nobody Mon Mar 18 19:42:42 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tz4yc6SHbz5DrRX; Mon, 18 Mar 2024 19:43:04 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tz4yc5mjGz4CNN; Mon, 18 Mar 2024 19:43:04 +0000 (UTC) (envelope-from gallatin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1710790984; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Q09t9S3BPqCd2J+f0CbsV0jK2W3DmzGtvs+ffQMpKac=; b=I8/m2ozO2d3OtmpIjXgFPfpAXa1bybZ/KDOMR3BDiQ0s0+SO1lPyi+Nis4yMOCFY4uV/6/ YB9ZVyw9kTqwLGDE6lKFkKS0l+i+8b1rsNAn044Q83chS6epxhITsHCeyVAUAixKA2fTfR OMt49rBtpQb/cKjL2KSdt/obCVsNWNS7rt9RntxRuKBftXgDNTkB2bvqnXqld9H/ZOK0E/ j231SyP7PnySE/vgZIB9q7BU/JU34dw8Un6n6+x0En28KGMw4m6M6cCRv5q4mRWFHmiexN ObZHQk+3rvZZyC9LN2vxj2ugGGfV78QcRt7Wt+neFcSfdUWQNmBUo5+GsdDgRg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1710790984; a=rsa-sha256; cv=none; b=DBJIJL55XWpi2xUKQfZYFCS4khZ5/b4tol0FnKSbOQbsYEzC2vCJ06GJLVRiD0itRMINh7 rTsTkk1zHjlAoBnVsQ4e1AY35PEvB9BzU/hiqzRyzy614rfOH0DrEel81kj7/wd32R8wWX BxVYUmW6N0qQmgdVNfB8yqlU/42LJA/AO5fn9e9HKl/3z7R0hDtRQN/CMYlZP+5XpLC8T7 XlkKa8bMAmJQ8G6DMUi99N7cyGF0ogv20Ib9fXoWgroyFga9Cd9LsY/lMxIdceJ0Ub4jgH 9HjG/Q+CVv/trSLsW27BPsU0QWCX2o7SIEBumaObp651kpCyDICJ6C08z51oeQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1710790984; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Q09t9S3BPqCd2J+f0CbsV0jK2W3DmzGtvs+ffQMpKac=; b=SmUyykqshjgTYeZ6APoFuG3Y8GOC7pYqtNFBA5cQkXUZh1RoOiJ+Qy2iQiYOaRb1lHAZr8 Ouf5N5FwtRbS2FeCEDZwETwJpe0I/Z2q697w2gmGeUIFMgQ8FSoByuvmOvMgaRy/EwYQzu rYmoj2YpErBUieEQ1DsXquNevI5V4vB3KBJLpYw4Job7gXOepvVJLfS/78r8Ka4PgpO/II 5Kp0TUVHX6e2PdFVB2Mc6kJdKDDz16rPDufea3ZgdlmT6sLMMsKzP/OKyZGduCuUn2Rpd1 Qr1vSSmc7tP/8PZjcP/LsW5jVp5u5SeY+6VL2iI9nhRDQ8k5dKDeYfAIIYLE1w== Received: from fauth1-smtp.messagingengine.com (fauth1-smtp.messagingengine.com [103.168.172.200]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: gallatin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Tz4yc3kdxzPj4; Mon, 18 Mar 2024 19:43:04 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfauth.nyi.internal (Postfix) with ESMTP id 027CB120006B; Mon, 18 Mar 2024 15:43:03 -0400 (EDT) Received: from imap53 ([10.202.2.103]) by compute5.internal (MEProxy); Mon, 18 Mar 2024 15:43:04 -0400 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgdduvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvfevufgtsegrtderreerreejnecuhfhrohhmpedfffhr vgifucfirghllhgrthhinhdfuceoghgrlhhlrghtihhnsehfrhgvvggsshgurdhorhhgqe enucggtffrrghtthgvrhhnpefgteeluefggeevheefheejgedtvdelheejfffhgeeuhfel keevheeiieeuhfefieenucffohhmrghinhepmhhpihdqshifshdrohhrghenucevlhhush htvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehgrghllhgrthhinhdo mhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqudeffeehledvvdduiedqvdelhe dtgedukeegqdhgrghllhgrthhinheppehfrhgvvggsshgurdhorhhgsehfrghsthhmrghi lhdrtghomh X-ME-Proxy: Feedback-ID: i41414658:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id B879D36400BB; Mon, 18 Mar 2024 15:43:03 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-300-gdee1775a43-fm-20240315.001-gdee1775a List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Message-Id: <38c54399-6c96-44d8-a3a2-3cc1bfbe50c2@app.fastmail.com> In-Reply-To: References: <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> <6047C8EF-B1B0-4286-93FA-AA38F8A18656@karels.net> <8031cd99-ded8-4b06-93b3-11cc729a8b2c@app.fastmail.com> Date: Mon, 18 Mar 2024 15:42:42 -0400 From: "Drew Gallatin" To: "Konstantin Belousov" Cc: "Mike Karels" , tuexen , "Nuno Teixeira" , garyj@gmx.de, current@freebsd.org, net@freebsd.org, "Randall Stewart" Subject: Re: Request for Testing: TCP RACK Content-Type: multipart/alternative; boundary=49341a7599d5444a8bbcc6b7abbe0677 --49341a7599d5444a8bbcc6b7abbe0677 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable No. The goal is to run on every return to userspace for every thread. Drew On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: > On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: > > I got the idea from > > https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.p= df > > The gist is that the TCP pacing stuff needs to run frequently, and > > rather than run it out of a clock interrupt, its more efficient to r= un > > it out of a system call context at just the point where we return to > > userspace and the cache is trashed anyway. The current implementation > > is fine for our workload, but probably not idea for a generic system. > > Especially one where something is banging on system calls. > > > > Ast's could be the right tool for this, but I'm super unfamiliar with > > them, and I can't find any docs on them. > > > > Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent to > > what's happening here? > This call would need some AST number added, and then it registers the > ast to run on next return to userspace, for the current thread. >=20 > Is it enough? > > > > Drew >=20 > >=20 > > On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: > > > On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > > > > On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote: > > > >=20 > > > > >> On 18. Mar 2024, at 12:42, Nuno Teixeira wrote: > > > > >> > > > > >> Hello all! > > > > >> > > > > >> It works just fine! > > > > >> System performance is OK. > > > > >> Using patch on main-n268841-b0aaf8beb126(-dirty). > > > > >> > > > > >> --- > > > > >> net.inet.tcp.functions_available: > > > > >> Stack D Alias = PCB count > > > > >> freebsd freebsd = 0 > > > > >> rack * rack = 38 > > > > >> --- > > > > >> > > > > >> It would be so nice that we can have a sysctl tunnable for th= is patch > > > > >> so we could do more tests without recompiling kernel. > > > > > Thanks for testing! > > > > > > > > > > @gallatin: can you come up with a patch that is acceptable for= Netflix > > > > > and allows to mitigate the performance regression. > > > >=20 > > > > Ideally, tcphpts could enable this automatically when it starts = to be > > > > used (enough?), but a sysctl could select auto/on/off. > > > There is already a well-known mechanism to request execution of the > > > specific function on return to userspace, namely AST. The differe= nce > > > with the current hack is that the execution is requested for one c= allback > > > in the context of the specific thread. > > >=20 > > > Still, it might be worth a try to use it; what is the reason to hi= t a thread > > > that does not do networking, with TCP processing? > > >=20 > > > >=20 > > > > Mike > > > >=20 > > > > > Best regards > > > > > Michael > > > > >> > > > > >> Thanks all! > > > > >> Really happy here :) > > > > >> > > > > >> Cheers, > > > > >> > > > > >> Nuno Teixeira escreveu (domingo, 17/03/= 2024 =C3=A0(s) 20:26): > > > > >>> > > > > >>> Hello, > > > > >>> > > > > >>>> I don't have the full context, but it seems like the compla= int is a performance regression in bonnie++ and perhaps other things whe= n tcp_hpts is loaded, even when it is not used. Is that correct? > > > > >>>> > > > > >>>> If so, I suspect its because we drive the tcp_hpts_softcloc= k() routine from userret(), in order to avoid tons of timer interrupts a= nd context switches. To test this theory, you could apply a patch like: > > > > >>> > > > > >>> It's affecting overall system performance, bonnie was just a= way to > > > > >>> get some numbers to compare. > > > > >>> > > > > >>> Tomorrow I will test patch. > > > > >>> > > > > >>> Thanks! > > > > >>> > > > > >>> -- > > > > >>> Nuno Teixeira > > > > >>> FreeBSD Committer (ports) > > > > >> > > > > >> > > > > >> > > > > >> --=20 > > > > >> Nuno Teixeira > > > > >> FreeBSD Committer (ports) > > > >=20 > > >=20 >=20 --49341a7599d5444a8bbcc6b7abbe0677 Content-Type: text/html;charset=utf-8 Content-Transfer-Encoding: quoted-printable
No.  The g= oal is to run on every return to userspace for every thread.

Drew

On Mon, Mar 18, 2024= , at 3:41 PM, Konstantin Belousov wrote:
On Mon, Mar 18, 2024 at 03:13:11PM -0400, = Drew Gallatin wrote:
> I got the idea from
> https://people.mpi-sws.org/~druschel/publications= /soft-timers-tocs.pdf
> The gist is that the TCP pa= cing stuff needs to run frequently, and
> rather than r= un it out of a clock interrupt, its more efficient to run
= > it out of a system call context at just the point where we return t= o
> userspace and the cache is trashed anyway. The curr= ent implementation
> is fine for our workload, but prob= ably not idea for a generic system.
> Especially one wh= ere something is banging on system calls.
>
> Ast's could be the right tool for this, but I'm super unfamiliar= with
> them, and I can't find any docs on them.
>
> Would ast_register(0, ASTR_UNCOND, 0, fu= nc) be roughly equivalent to
> what's happening here?
This call would need some AST number added, and then it reg= isters the
ast to run on next return to userspace, for the= current thread.

Is it enough?
>
> Drew

> 
> On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov = wrote:
> > On Mon, Mar 18, 2024 at 07:26:10AM -0500,= Mike Karels wrote:
> > > On 18 Mar 2024, at 7:04= , tuexen@freebsd.org wrot= e:
> > > 
> > > >&= gt; On 18. Mar 2024, at 12:42, Nuno Teixeira <eduardo@freebsd.org> wrote:
> &g= t; > >>
> > > >> Hello all!
> > > >>
> > > >> It= works just fine!
> > > >> System performan= ce is OK.
> > > >> Using patch on main-n268= 841-b0aaf8beb126(-dirty).
> > > >>
> > > >> ---
> > > >> = net.inet.tcp.functions_available:
> > > >> = Stack           &= nbsp;           &= nbsp;   D Alias        = ;            = ;        PCB count
>= > > >> freebsd       &nb= sp;           &nb= sp;       freebsd    &= nbsp;           &= nbsp;         0
&g= t; > > >> rack       &nbs= p;           &nbs= p;        * rack   &nb= sp;           &nb= sp;           &nb= sp; 38
> > > >> ---
> >= > >>
> > > >> It would be so nice= that we can have a sysctl tunnable for this patch
> &g= t; > >> so we could do more tests without recompiling kernel.
> > > > Thanks for testing!
> = > > >
> > > > @gallatin: can you come= up with a patch that is acceptable for Netflix
> > = > > and allows to mitigate the performance regression.
> > > 
> > > Ideally, tcphpts co= uld enable this automatically when it starts to be
> &g= t; > used (enough?), but a sysctl could select auto/on/off.
=
> > There is already a well-known mechanism to request execut= ion of the
> > specific function on return to usersp= ace, namely AST.  The difference
> > with the c= urrent hack is that the execution is requested for one callback
> > in the context of the specific thread.
>= ; > 
> > Still, it might be worth a try to u= se it; what is the reason to hit a thread
> > that d= oes not do networking, with TCP processing?
> > = ;
> > > 
> > > Mike
> > > 
> > > > Best= regards
> > > > Michael
> &g= t; > >>
> > > >> Thanks all!
> > > >> Really happy here :)
>= > > >>
> > > >> Cheers,
> > > >>
> > > >> Nu= no Teixeira <eduardo@freebsd.o= rg> escreveu (domingo, 17/03/2024 =C3=A0(s) 20:26):
> > > >>>
> > > >>> H= ello,
> > > >>>
> > = > >>>> I don't have the full context, but it seems like t= he complaint is a performance regression in bonnie++ and perhaps other t= hings when tcp_hpts is loaded, even when it is not used.  Is that c= orrect?
> > > >>>>
>= > > >>>> If so, I suspect its because we drive the tc= p_hpts_softclock() routine from userret(), in order to avoid tons of tim= er interrupts and context switches.  To test this theory,  you= could apply a patch like:
> > > >>>
=
> > > >>> It's affecting overall system per= formance, bonnie was just a way to
> > > >>= > get some numbers to compare.
> > > >>&= gt;
> > > >>> Tomorrow I will test patch= .
> > > >>>
> > >= >>> Thanks!
> > > >>>
> > > >>> --
> > > >&g= t;> Nuno Teixeira
> > > >>> FreeBSD C= ommitter (ports)
> > > >>
>= ; > > >>
> > > >>
> > > >> -- 
> > > >> = Nuno Teixeira
> > > >> FreeBSD Committer (p= orts)
> > > 
> > 


--49341a7599d5444a8bbcc6b7abbe0677-- From nobody Mon Mar 18 22:16:14 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tz8MN57dlz5F6rf for ; Mon, 18 Mar 2024 22:16:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tz8MN43z4z4ZKp for ; Mon, 18 Mar 2024 22:16:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1710800176; a=rsa-sha256; cv=none; b=PHSw+kjtNYwbLP9FsHPLUpxyn3JN7w9NPbFSgt5PHrH9ka5elbtBJ7xAn/9LVX1LoZ60yG 7RL54lI7kU2NyOiKqO1Splj+d9C1Zbnz/8SsONg6zc3U2NWKdadghLDD3qgHuIJ9z1WUu1 fdTbwQdZYGSXOAGl4MnVHCv196nltBVsy3MALHP0maK7oD2cAAkuHhoDFm+HWk1WYtfuQF zXhb8YCC4JaHg88F6Rh3Zr/4DMwtRFtpOr1Nq0We8NlEF4CrnZP1bVjWSNqGqbgG5VlMab q0X3JTuHz9RN7x/NbSEOnb/I832DTJVOWVXnqNZDPWPR9fyWg8hxt+osVZ5MzQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1710800176; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pEwSe5IxF/9fBGZr7HEoVjQgFv/dJAXw+cwhuZ97R4U=; b=dX2Z6eNp3H5IX+f4vGDYKrd5Yc1ff6dkYIh4FNuRbBTUe+09FbPKiBCNcmW18oJlLf2DxE crK8E09SWZOZ5pZCkfaSYoxiGgHQ5mb5EJh/JE4MrOGQIIbXZoqh160jj2C3nuI3P8TZol G03uuxt4qF4VFPJrkl9BGrT31Nbo4EORRQy2jo7kuRhgPX7ALzAF3AzZH80Fc5WXiZaCqG 16aEBzHw7Xg6YHY2ED3xQX20Dl9VOSjqwhI1Ifok1TYNFUErWEks4lNky7RmyBSTxwfKGo QQw+lyorzceOJ09nkOmkNy+LgdSWaNuHOwtwi54QdUBWVuxa1iRxt4hXR4BTjg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Tz8MN3YrgzhST for ; Mon, 18 Mar 2024 22:16:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42IMGGPM018952 for ; Mon, 18 Mar 2024 22:16:16 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42IMGGjX018951 for net@FreeBSD.org; Mon, 18 Mar 2024 22:16:16 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 277771] ice driver report I2C error messages of E822 copper LAN ports Date: Mon, 18 Mar 2024 22:16:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277771 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |net@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Mar 19 18:01:33 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tzfg341wzz5F252 for ; Tue, 19 Mar 2024 18:01:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tzfg255gRz4gl6 for ; Tue, 19 Mar 2024 18:01:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1710871294; a=rsa-sha256; cv=none; b=T40Z38CADjcgNnVy0W1jwfTeXbyC97wJT5DGH2G60faI3I4mG5QXVxNROn7axCA52TF5Zj f8mYdr/EBAF9FLy4po1/LJZmlkjkW7TzeLC6T8m6CwVve9Jh7PNS2snZhAMQ/u0kdHH0AL svJC9s3SWwgw24e08J9b6/7qas77tYQnRZS2W/jesQtqXziza6RMJU81eTfozV4JTqSW3q ZgVK8IUvvtFN62keTJ2m2v9PI1hkNaQuB8lBPdHIW8XUIDoy3qXQweUV1DozETEanInb/A 3ELJ2mm1PiPNy6UIwNp+Rtp9+YlkzDInZMPeLs5W3exjREYWgjiNTupshwL/dQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1710871294; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SC/LMgI5/zjaN+VrlNUdgxxKrsl8ATrDVmkNuJ9dx3s=; b=WJhYdKW4j/Cj74S5Jil+7ER7vACBL4TAzujHrpp9f9lpmBUV4pzBQKdcC44BlBwFXW8mCM ZYHDOjba0o1Uv4N1BbLtu/Jf4z5XQYgrfwrfMai9jdEtq8SEVj6RBaJFPkO92WU4exNhZv AXa+lWy96HoeG38xzN4KqExW2R4Uni+wRlh1ClwwDF6UKyjpmVFNEb5ir4sD8FpBg5dP6x xJLDN3b7wH4hfqqrlc3GIzzF0pCLI3HyOIzMGadjyr1LvZsHsXi4c0xms8dyA0UHcawc1X aBISRlJx0pNUdK9+kcv3RDcxffYwFTYP4yMXBOL/ohf+1Z142T6j4rQa0JXkZA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Tzfg24RwKzHfL for ; Tue, 19 Mar 2024 18:01:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42JI1YW0023366 for ; Tue, 19 Mar 2024 18:01:34 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42JI1YCr023361 for net@FreeBSD.org; Tue, 19 Mar 2024 18:01:34 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 231366] problems with CARP setup on two servers Date: Tue, 19 Mar 2024 18:01:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: arm X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: glebius@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Overcome By Events X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D231366 Gleb Smirnoff changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |Overcome By Events Status|New |Closed --- Comment #1 from Gleb Smirnoff --- FreeBSD 11.2 reached its End of Life October 31, 2019 and FreeBSD 11 branch reached EoL September 30, 2021. Closing bug as obsolete. To submitter: if you still exprience the problem on a modern FreeBSD release, please submit a new bug. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Mar 19 18:04:11 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tzfk40L3Gz5F2H8 for ; Tue, 19 Mar 2024 18:04:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tzfk36M7Qz4gwj for ; Tue, 19 Mar 2024 18:04:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1710871451; a=rsa-sha256; cv=none; b=M9o1iuLCsOZD4GLC+3y69jJd/WSO7MsDJ6u3hN4A4dCdl45eycnmCAhBcgw6rdToUpLdkU biS7G3cGYlb8/OPUg81PQYBFzRwOOqLa9QyNnPWRSgoEp2ai+YcjMyvIPgg50iAe4pzX9+ HJBityKfRMkYQfQq3pdZuESccm+r30Jkhq8HQtgZHbXPv5yk6bRJiE3kj0l7xKEa4Hrq4s fDK9MXkYxsyQmhNDhRb0HvL0I3dBfjdB+gbTcwOUXks7skvmTTNNFEj7XVvEUWwrdB9tft ljw8Y6X4eRgau1UMhigSzjNO2dxjtG94ih5OqDhVdE5E2i5TvDIrsyYBadt1LQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1710871451; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/7FaTE0SnSQCqJboaoD20yYh7AK6Vc869VEDbodVXRo=; b=XADoLNi7CAXywHgfUNU3ipVcfwIztE6s2lXYi4/QkefVEtsf2/ufOZ5Fg7knF9v8Pjhdxr s8hsP0BIvaHQ7TINM7blyyrg5a8vigx2vZ+IyM1koRqu+J96bJd93fUPeVtaLPbtoYmEzN hnXLQWBO/qWSna9sfCQBwrbOvuAmwFt1y2jXB5b4xGF1HBEHnmIIGnKgKbrc7Ppc2WzsuB WpoauVex5gNNqjD6WFd8/LNMda+dPbuLQwT3pum6vvgdrB9zewDHwZ8kbOzehyS7A594oR r6WrJcy2dtX1ksBLuVAPB7uylyVZPtaCT0p2wRZShvb8JlQ/fxF8Vps//zGALA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Tzfk35ymWzHhp for ; Tue, 19 Mar 2024 18:04:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42JI4BxK040187 for ; Tue, 19 Mar 2024 18:04:11 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42JI4BBF040185 for net@FreeBSD.org; Tue, 19 Mar 2024 18:04:11 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 229547] CARP preempt didn't work Date: Tue, 19 Mar 2024 18:04:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: glebius@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Overcome By Events X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D229547 Gleb Smirnoff changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |Overcome By Events Status|New |Closed --- Comment #1 from Gleb Smirnoff --- FreeBSD 11.2 reached its End of Life October 31, 2019 and FreeBSD 11 branch reached EoL September 30, 2021. Closing bug as obsolete. To submitter: if you still exprience the problem on a modern FreeBSD release, please submit a new bug. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Mar 19 18:58:12 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TzgwP48zSz5F8nL for ; Tue, 19 Mar 2024 18:58:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TzgwP3489z4pLV for ; Tue, 19 Mar 2024 18:58:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1710874693; a=rsa-sha256; cv=none; b=aLpsIm9MXk8dnvYPKoBpX9DXYt8SrWt4zk5H4sgL7fxU9JVPfEGP7oLKgaFaTP/T+TBke4 Nxo4Asvozx8Dg1Rh27udlsoGzD877a76dVCgN58OeXDaN7XdG7YdBO6W9aXv+c6wZOZtwW TTNDX3m4lYxdEt52FDcvUrMMATEeF0hAOblS+sIesnAE8GoXoTn9AHq4MlwOVZ6rha94IL g4iE9rzpMDP3idbV1GWjNK9GikztRjI5hLYCr23pABMMRmSYErG8+iQ1AiMwj6RTfwy/JF k/v7IRqmaqc/xXvqz+7aR4ueGAFeAg+Vn68knt8PjdCX3S02wIjXrBxH/oHBLw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1710874693; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fdm4k8p8H+i/rjT95DltfMzqETwbm3Mk0vS0EgXtQ6Y=; b=xM4xsFQj454u0+7eLy+ajRxwTysGF7oIyPDe9SdbVf+a+rhGY48sO91SrZXZGX47vonNH+ QhkV+wNKJxtczjVCMRMxJgQ/+l+gTH6g5Q3lhEVSn1HKxMADaIIF5gpTteya/gjpXfoXkj 5vC/ugC0L9eF/gP0ibaIkNEAEwGzh8nw7Z2OybPxoSGLd4VJTccJadAZLEp8g74mFPEfCx DA0j1wSfRs4dbo8Qv9mau+AZuwNUDbGVulTgwVcxkvjmASNzIGCiVQAkkNr27ILUAqDm1j /RA+aEyWn6egAmJXSEx8FXmU3XkWOy4xo030lSiyFupJ1+qS3/2K4J/4ufmIMg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TzgwP2fG0zK75 for ; Tue, 19 Mar 2024 18:58:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42JIwD1K020654 for ; Tue, 19 Mar 2024 18:58:13 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42JIwDSu020649 for net@FreeBSD.org; Tue, 19 Mar 2024 18:58:13 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 277349] The net.inet.ip.source_address_validation should ignore CARP IP in backup state Date: Tue, 19 Mar 2024 18:58:12 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277349 --- Comment #7 from commit-hook@FreeBSD.org --- A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3D56f7860087eec14b4a65310b70bd704e7= 9e1b48c commit 56f7860087eec14b4a65310b70bd704e79e1b48c Author: Gleb Smirnoff AuthorDate: 2024-03-19 18:48:59 +0000 Commit: Gleb Smirnoff CommitDate: 2024-03-19 18:48:59 +0000 carp: check CARP status in in_localip_fib(), in6_localip_fib() Don't report a BACKUP CARP address as local. These two functions are u= sed only by source address validation for input packets, controlled by sysc= tls net.inet.ip.source_address_validation and net.inet6.ip6.source_address_validation. For this purpose we definitely want to treat BACKUP addresses as non local. This change is conservative and doesn't modify compat in_localip() and in6_localip(). They are used more widely than the FIB-aware versions. The change would modify the notion of ipfw(4) 'me' keyword. There might be other consequences as in_localip() is used by various tunneling protocols. PR: 277349 sys/netinet/in.c | 4 +++- sys/netinet6/in6.c | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Mar 20 03:54:41 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TzvqQ4141z5DrMH for ; Wed, 20 Mar 2024 03:54:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TzvqQ2z4yz4pm4 for ; Wed, 20 Mar 2024 03:54:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1710906882; a=rsa-sha256; cv=none; b=CE6zcRU8GGP6gNBwrNdOloZNbUIaUioN8eEg34yM3PzBAlrYn5VxsS4U+CwcStxzpvcDFG By3Hi10S+sYRUtVswNGtBM2qHOA8bN/TqbkLO7SrV+9otZ/e06nSQGyL9qC8xbWyeLNmEi 1ZckAJ3QISZgmUP+ChuXS0ogUR9LdHUT7K2dERRjfCeje6u3omXJjqiKA1YhGFTdI96sL4 D8xwRuzX5dFRNAS6OAGRNp+HTh95bA1cMMxjvDh9lEHx2AIwBgAUWs01XGjvok2zZJPuzj 3aLck+ALI9RXo65R36tCq6ObJCb2nH3I4l7BwZ8Z6YjOx5MgNBR/BkAWKx+UbA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1710906882; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mMFdrqCOCKTNrJjCYXU7ihl0W7xhU12VwHiVUtuqNlI=; b=Z0NAKW9fl8GS2+dSmbzUtIcKPamdT43RDjzMD5JUExW0kQPrtbSaEKC2I7y0Rb4MC8zmDD Jvj35XeXJdJeXr3o/pD601ztW7vNX43B0/q3WGN0YUtSq3ZEsxEte+Q27eT2b+vtCbiId5 3Te72psGCpwdewaw19doGUUmv6/PA5WJbCAEqF9NhmB45ln9+IeVzQ9WO4bdpq+9KzskUZ StjWbNjOo8lrGA8YGKbTqKM7A3ZEXX9mN1cm6BHp7kdn/AKFl0S8k+049luf7pAXplmZKc HSdxzijsT6mJneXYwOYgIAuDVZoSQh3+RiOHySdPm0FJp4NVSh5DFxCC8WmZVw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TzvqQ2ZsTzbYp for ; Wed, 20 Mar 2024 03:54:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42K3sgn3089758 for ; Wed, 20 Mar 2024 03:54:42 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42K3sg5i089753 for net@FreeBSD.org; Wed, 20 Mar 2024 03:54:42 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 275251] Realtek PCIe 2.5GbE Family Controller not being detected Date: Wed, 20 Mar 2024 03:54:41 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mariah9xx@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275251 Mariah Carey changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mariah9xx@gmail.com --- Comment #20 from Mariah Carey --- It's possible that the driver you're currently using (realtek-re-kmod) may = not be fully compatible with your hardware or system configuration, leading to = the SSL decryption errors you're encountering. https://wordlewebsite.com --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Mar 20 16:52:37 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V0F5H5dCyz5FvDB for ; Wed, 20 Mar 2024 16:52:51 +0000 (UTC) (envelope-from pavel.popa@edu.unife.it) Received: from mail-pg1-x531.google.com (mail-pg1-x531.google.com [IPv6:2607:f8b0:4864:20::531]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V0F5F6xJqz3yjV for ; Wed, 20 Mar 2024 16:52:49 +0000 (UTC) (envelope-from pavel.popa@edu.unife.it) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=edu.unife.it header.s=google header.b=mZ2PJw3R; dmarc=pass (policy=reject) header.from=edu.unife.it; spf=pass (mx1.freebsd.org: domain of pavel.popa@edu.unife.it designates 2607:f8b0:4864:20::531 as permitted sender) smtp.mailfrom=pavel.popa@edu.unife.it Received: by mail-pg1-x531.google.com with SMTP id 41be03b00d2f7-5cfd95130c6so11918a12.1 for ; Wed, 20 Mar 2024 09:52:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=edu.unife.it; s=google; t=1710953568; x=1711558368; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=6x/WhwrBxQntRYRIIHnvZ1bw3JS3lYUoKq0g9mfjolU=; b=mZ2PJw3Rbj14UJutF0AryD4jwIxgO/Fr5Zapvv4ot3qO4NmdM4oxi/v8FDRfpniPvq 26yxfPS/nuxcUbZlgmZcKtrCm+6+bMD+yT9FdRlXrt+s464W1UX6x4to3TGiYyyy+R4K l2h7qckYMImQvHAVY6kV0Ta9E7B5+ajmus0LM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710953568; x=1711558368; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=6x/WhwrBxQntRYRIIHnvZ1bw3JS3lYUoKq0g9mfjolU=; b=ZuFrnxx0sZHFTMW5hJndccwfkBq5UL2NeuYcIailbCB/8Ewogcg6PbJJdmNqejWP+m fEsz1ruwm2CnDlj3LUPPminK6fnoc1pL9r8VVQBk2Uhi71QdbG/s0j+JwgTHvz1hf6sI OnC4RhrC3y1TqJPWGuWRA9kF2jDleb/ySblK1RL7SG2B2IGYbbcZmrN2jS2DKrpUpfk0 vQcGlDLP06NIulCJThsdmI7rfmlcirWpGZAxK0wnZCDKCOULNYuxNN02FVjYyx7sIiok 31exQ5xNz+ykjqgLNl0UVgCJKXV8g7Q3usvqhghwP2KdLkNZRicfmTCvJxy7mSVqxQjy fFcQ== X-Gm-Message-State: AOJu0YxBM5fedXh+pVXEWHSKos/kU40yHXcS5WBVYuZjgo91O369uN9+ anHaca3o3Mwopuy0a3u2f6E++WKsObxjhScEfgLLslm6m/h4mmlZqwKEx+1VkNvu2Epgu1990MP Z9JgKc8m1iZ+mPVf+AmKvZ40HNaszrXOGR4fGSR84f1pDjHEg/H97bw== X-Google-Smtp-Source: AGHT+IFisYEQ2e5hvVDhh5S/A6LB37OcN3SaoWWfNHxJq0N3rp/9vhnUiggJrumVqFpWm3o50IRhJwdsYkDeFdhXClE= X-Received: by 2002:a05:6a20:29a1:b0:1a3:4785:c8be with SMTP id f33-20020a056a2029a100b001a34785c8bemr5299950pzh.16.1710953567761; Wed, 20 Mar 2024 09:52:47 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 From: PAVEL POPA Date: Wed, 20 Mar 2024 17:52:37 +0100 Message-ID: Subject: IFLIB msi-x initialization extension/patch request To: freebsd-net@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.97 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[unife.it:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.972]; DMARC_POLICY_ALLOW(-0.50)[edu.unife.it,reject]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[edu.unife.it:s=google]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MISSING_XM_UA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_DN_NONE(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::531:from]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; DKIM_TRACE(0.00)[edu.unife.it:+] X-Rspamd-Queue-Id: 4V0F5F6xJqz3yjV Hi everyone, Don't know where I should ask this, so I'll just try with this mailing list as a first attempt. I have implemented an IFLIB based network device driver for a Xilinx FPGA card with a custom firmware, everything seems to work fine except for the number of RX/TX queues selected by IFLIB during 'iflib_msix_init()', which seems to be based on the number of MSI-X vectors actually available on the HW, which in my specific case that's 8. This FPGA does implement a solution that bypasses the fact of dedicating an MSI-X vector per queue, by providing an "interrupt aggregation ring" for which the driver receives interrupts to read from this interrupt ring whose descriptors tell which specific queue and queue type (if RX or TX) needs servicing. In this way the number of RX/TX queues that can be allocated is independent of the MSI-X vectors available on the HW. I have implemented this successfully in the driver already, but with the limitation of not being able to allocate more than 8 RX/TX queues due to the way IFLIB decides that. Finally to my question, it seems to me that a pretty simple patch on the IFLIB side could do the trick, i.e. allowing to have this "flexibility" in the number of RX/TX queues decided, when the related interrupts are to be processed via such "interrupt ring" mechanism/design. Who should I ask for this if such an extension/patch is even possible and acceptable? Should I propose the patch myself or it can be handled by some maintainer? Thanks From nobody Wed Mar 20 22:47:23 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V0NyM50y0z5F88J for ; Wed, 20 Mar 2024 22:47:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V0NyM3wWSz4jk5 for ; Wed, 20 Mar 2024 22:47:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1710974843; a=rsa-sha256; cv=none; b=cEZovrBbxZ94I9LT8GxE5ceVrsAmgKQSL6/wUDUSOwsOrK4B0EJTyP0Ro2xmvtpG1s8d9s e8cq4a86Sr9P3EHHxeXJXFFtttuVQNmuyPJmVEljEOxQPGb46bFKioEaIK/zP/QMGqEpPq GCwcm4gn/StBmZMMPht5jny72Ux3sgOUzgefqjDLFctlOuYfAuwSK3WaCfJUTWwWeFLH5T PgOIHIl/fGRZ+SJZ8usXE4znvNXTJd1/y/EawPab8r5QO1gC89sKepp612MMf5xu6rmoPr m6yFjDTvuV58yDp8eAktA1IYdqx/Q0lrxgvTBPgCJk6/ihUza9fQvmW5pA7l5g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1710974843; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s3IsvjaycBabseHVPyCOqSaw0KIzT4TGGdItDjPSSTg=; b=hKWcw/zy82PmFQA3KC6ENTZIwMhKKvz6oiCU5m4JGlxY0R4b5szsq0x5s/EN/7Lz/2dmdu eQZ2dXOLHcj4vuBfrJizyLX05iy1vLWAdOjU5j3K+lvuPfcheoaJ9HyqnVRmiCfanwafGC DkHxOuxkHGXhW1uBmXhP3a0gzjKwfyjDuyT5xr1Wjl3goEkb3bcs49HWUmbY6OLfGvrmxw Pqf1Hj1aa/9XBzuVURMdvjgjTypAi0qkZncwbli7Xmc1qduzxq3XccSfE2iom8fZioJTAc IyH+B5BNDqwh4TAkyCuEtIQTM12ktrMZ1vY8RVoV+flDlE8vi2uDQHNtHKpJ8w== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4V0NyM3X4dz18vw for ; Wed, 20 Mar 2024 22:47:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42KMlNcp087367 for ; Wed, 20 Mar 2024 22:47:23 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42KMlN8D087366 for net@FreeBSD.org; Wed, 20 Mar 2024 22:47:23 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 277849] [bge] panic when attaching if_bge on Supermicro H13SSL-N Date: Wed, 20 Mar 2024 22:47:23 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277849 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |net@FreeBSD.org Keywords| |crash --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Mar 21 00:17:37 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V0Qyr38Lnz5FJrg for ; Thu, 21 Mar 2024 00:17:56 +0000 (UTC) (envelope-from kib@freebsd.org) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4V0Qyq4lvTz4vPN for ; Thu, 21 Mar 2024 00:17:55 +0000 (UTC) (envelope-from kib@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=freebsd.org (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kib@freebsd.org) smtp.mailfrom=kib@freebsd.org Received: from tom.home (kib@localhost [127.0.0.1] (may be forged)) by kib.kiev.ua (8.18.1/8.18.1) with ESMTP id 42L0HcYZ082994; Thu, 21 Mar 2024 02:17:41 +0200 (EET) (envelope-from kib@freebsd.org) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 42L0HcYZ082994 Received: (from kostik@localhost) by tom.home (8.18.1/8.18.1/Submit) id 42L0Hb4s082987; Thu, 21 Mar 2024 02:17:37 +0200 (EET) (envelope-from kib@freebsd.org) X-Authentication-Warning: tom.home: kostik set sender to kib@freebsd.org using -f Date: Thu, 21 Mar 2024 02:17:37 +0200 From: Konstantin Belousov To: rrs Cc: Drew Gallatin , Mike Karels , tuexen , Nuno Teixeira , garyj@gmx.de, current@freebsd.org, net@freebsd.org, Randall Stewart Subject: Re: Request for Testing: TCP RACK Message-ID: References: <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> <6047C8EF-B1B0-4286-93FA-AA38F8A18656@karels.net> <8031cd99-ded8-4b06-93b3-11cc729a8b2c@app.fastmail.com> <38c54399-6c96-44d8-a3a2-3cc1bfbe50c2@app.fastmail.com> <27d8144f-0658-46f6-b8f3-35eb60061644@lakerest.net> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <27d8144f-0658-46f6-b8f3-35eb60061644@lakerest.net> X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.98 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.979]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : No valid SPF, No valid DKIM,none]; HAS_XAW(0.00)[]; RCPT_COUNT_SEVEN(0.00)[9]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[kib]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_CC(0.00)[freebsd.org,karels.net,gmx.de]; MLMMJ_DEST(0.00)[net@freebsd.org]; R_SPF_SOFTFAIL(0.00)[~all:c]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MISSING_XM_UA(0.00)[] X-Rspamd-Queue-Id: 4V0Qyq4lvTz4vPN On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: > Ok I have created > > https://reviews.freebsd.org/D44420 > > > To address the issue. I also attach a short version of the patch that Nuno > can try and validate > > it works. Drew you may want to try this and validate the optimization does > kick in since I can > > only now test that it does not on my local box :) The patch still causes access to all cpu's cachelines on each userret. It would be much better to inc/check the threshold and only schedule the call when exceeded. Then the call can occur in some dedicated context, like per-CPU thread, instead of userret. > > > R > > > > On 3/18/24 3:42 PM, Drew Gallatin wrote: > > No.  The goal is to run on every return to userspace for every thread. > > > > Drew > > > > On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: > > > On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: > > > > I got the idea from > > > > https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf > > > > The gist is that the TCP pacing stuff needs to run frequently, and > > > > rather than run it out of a clock interrupt, its more efficient to run > > > > it out of a system call context at just the point where we return to > > > > userspace and the cache is trashed anyway. The current implementation > > > > is fine for our workload, but probably not idea for a generic system. > > > > Especially one where something is banging on system calls. > > > > > > > > Ast's could be the right tool for this, but I'm super unfamiliar with > > > > them, and I can't find any docs on them. > > > > > > > > Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent to > > > > what's happening here? > > > This call would need some AST number added, and then it registers the > > > ast to run on next return to userspace, for the current thread. > > > > > > Is it enough? > > > > > > > > Drew > > > > > > > > > > > On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: > > > > > On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > > > > > > On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote: > > > > > > > > > > > > >> On 18. Mar 2024, at 12:42, Nuno Teixeira > > > wrote: > > > > > > >> > > > > > > >> Hello all! > > > > > > >> > > > > > > >> It works just fine! > > > > > > >> System performance is OK. > > > > > > >> Using patch on main-n268841-b0aaf8beb126(-dirty). > > > > > > >> > > > > > > >> --- > > > > > > >> net.inet.tcp.functions_available: > > > > > > >> Stack                           D > > > Alias                            PCB count > > > > > > >> freebsd freebsd                          0 > > > > > > >> rack                            * > > > rack                             38 > > > > > > >> --- > > > > > > >> > > > > > > >> It would be so nice that we can have a sysctl tunnable for > > > this patch > > > > > > >> so we could do more tests without recompiling kernel. > > > > > > > Thanks for testing! > > > > > > > > > > > > > > @gallatin: can you come up with a patch that is acceptable > > > for Netflix > > > > > > > and allows to mitigate the performance regression. > > > > > > > > > > > > Ideally, tcphpts could enable this automatically when it > > > starts to be > > > > > > used (enough?), but a sysctl could select auto/on/off. > > > > > There is already a well-known mechanism to request execution of the > > > > > specific function on return to userspace, namely AST.  The difference > > > > > with the current hack is that the execution is requested for one > > > callback > > > > > in the context of the specific thread. > > > > > > > > > > Still, it might be worth a try to use it; what is the reason to > > > hit a thread > > > > > that does not do networking, with TCP processing? > > > > > > > > > > > > > > > > > Mike > > > > > > > > > > > > > Best regards > > > > > > > Michael > > > > > > >> > > > > > > >> Thanks all! > > > > > > >> Really happy here :) > > > > > > >> > > > > > > >> Cheers, > > > > > > >> > > > > > > >> Nuno Teixeira escreveu (domingo, > > > 17/03/2024 à(s) 20:26): > > > > > > >>> > > > > > > >>> Hello, > > > > > > >>> > > > > > > >>>> I don't have the full context, but it seems like the > > > complaint is a performance regression in bonnie++ and perhaps other > > > things when tcp_hpts is loaded, even when it is not used.  Is that > > > correct? > > > > > > >>>> > > > > > > >>>> If so, I suspect its because we drive the > > > tcp_hpts_softclock() routine from userret(), in order to avoid tons > > > of timer interrupts and context switches.  To test this theory,  you > > > could apply a patch like: > > > > > > >>> > > > > > > >>> It's affecting overall system performance, bonnie was just > > > a way to > > > > > > >>> get some numbers to compare. > > > > > > >>> > > > > > > >>> Tomorrow I will test patch. > > > > > > >>> > > > > > > >>> Thanks! > > > > > > >>> > > > > > > >>> -- > > > > > > >>> Nuno Teixeira > > > > > > >>> FreeBSD Committer (ports) > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> -- > > > > > > >> Nuno Teixeira > > > > > > >> FreeBSD Committer (ports) > > > > > > > > > > > > > > > > > diff --git a/sys/netinet/tcp_hpts.c b/sys/netinet/tcp_hpts.c > index 8c4d2d41a3eb..eadbee19f69c 100644 > --- a/sys/netinet/tcp_hpts.c > +++ b/sys/netinet/tcp_hpts.c > @@ -216,6 +216,7 @@ struct tcp_hpts_entry { > void *ie_cookie; > uint16_t p_num; /* The hpts number one per cpu */ > uint16_t p_cpu; /* The hpts CPU */ > + uint8_t hit_callout_thresh; > /* There is extra space in here */ > /* Cache line 0x100 */ > struct callout co __aligned(CACHE_LINE_SIZE); > @@ -269,6 +270,11 @@ static struct hpts_domain_info { > int cpu[MAXCPU]; > } hpts_domains[MAXMEMDOM]; > > +counter_u64_t hpts_that_need_softclock; > +SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, needsoftclock, CTLFLAG_RD, > + &hpts_that_need_softclock, > + "Number of hpts threads that need softclock"); > + > counter_u64_t hpts_hopelessly_behind; > > SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, hopeless, CTLFLAG_RD, > @@ -334,7 +340,7 @@ SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, precision, CTLFLAG_RW, > &tcp_hpts_precision, 120, > "Value for PRE() precision of callout"); > SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, cnt_thresh, CTLFLAG_RW, > - &conn_cnt_thresh, 0, > + &conn_cnt_thresh, DEFAULT_CONNECTION_THESHOLD, > "How many connections (below) make us use the callout based mechanism"); > SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, logging, CTLFLAG_RW, > &hpts_does_tp_logging, 0, > @@ -1548,6 +1554,9 @@ __tcp_run_hpts(void) > struct tcp_hpts_entry *hpts; > int ticks_ran; > > + if (counter_u64_fetch(hpts_that_need_softclock) == 0) > + return; > + > hpts = tcp_choose_hpts_to_run(); > > if (hpts->p_hpts_active) { > @@ -1683,6 +1692,13 @@ tcp_hpts_thread(void *ctx) > ticks_ran = tcp_hptsi(hpts, 1); > tv.tv_sec = 0; > tv.tv_usec = hpts->p_hpts_sleep_time * HPTS_TICKS_PER_SLOT; > + if ((hpts->p_on_queue_cnt > conn_cnt_thresh) && (hpts->hit_callout_thresh == 0)) { > + hpts->hit_callout_thresh = 1; > + counter_u64_add(hpts_that_need_softclock, 1); > + } else if ((hpts->p_on_queue_cnt <= conn_cnt_thresh) && (hpts->hit_callout_thresh == 1)) { > + hpts->hit_callout_thresh = 0; > + counter_u64_add(hpts_that_need_softclock, -1); > + } > if (hpts->p_on_queue_cnt >= conn_cnt_thresh) { > if(hpts->p_direct_wake == 0) { > /* > @@ -1818,6 +1834,7 @@ tcp_hpts_mod_load(void) > cpu_top = NULL; > #endif > tcp_pace.rp_num_hptss = ncpus; > + hpts_that_need_softclock = counter_u64_alloc(M_WAITOK); > hpts_hopelessly_behind = counter_u64_alloc(M_WAITOK); > hpts_loops = counter_u64_alloc(M_WAITOK); > back_tosleep = counter_u64_alloc(M_WAITOK); > @@ -2042,6 +2059,7 @@ tcp_hpts_mod_unload(void) > free(tcp_pace.grps, M_TCPHPTS); > #endif > > + counter_u64_free(hpts_that_need_softclock); > counter_u64_free(hpts_hopelessly_behind); > counter_u64_free(hpts_loops); > counter_u64_free(back_tosleep); From nobody Thu Mar 21 07:28:36 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V0cWp5XZpz5FP6h for ; Thu, 21 Mar 2024 07:28:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V0cWp4WrBz4jMF for ; Thu, 21 Mar 2024 07:28:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711006118; a=rsa-sha256; cv=none; b=w4qgmmha/MNoX1+hAdpB8Et8D1VlNlgT5WseMGoDYFsa4CUgoludogkSP/XX4XMVahR2Sb VVLo4H0kqCAECrp9CMiqXvZcqzUHNo0Sv/pgHfE410rN/79lvi8XULPe0ccYOX26c93b7g sPJlJPebEPYD735wzNAbs74gCBwIveQxn2Y7RUC06ng1Szx6kgQmjj+GhZqZ+8P/k7Eyic T89lB11YCf0P6ppMw1MgrScaSyzqxoFQWxP0Y68MYO64m69I4SyNrVv0ESCRbqdWSjcYPC ogMqZdO++Rp9JELNaBNNakeu3RQbFLYa76E53RcbKyB6Br4H6NBlBvMZjKMyEw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711006118; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0LmBsh4tzoLWQPO+v82Ph95EImf1T0hUaDjYIiq3/r0=; b=qAn8YLJPgSGsAEFgGnF+VjhTSpTlsA77TlsMB11m289OPf0k+ZmiiVqDSlXH57zPb9h+v7 KyprRRkfW702l8Zo8bz7lmuQibAG/9ibbP+XtZfvwDGi2Tf1XJL3HB4lTq6PmSUqe2jtBw q16b44nd0RajsYZfRoZG/YQR9WnkyJBZj/1kHwxRY2pyWFh9DrzktNQehMopFaW3thPua2 R/2mgEkpQBFJqIDHvOIojz8baCLNtEjN97zs0vG7rB6ohQJrMkysPH9fBtnnXg4biqHT7Q xU6lw+2m9DTddTCfB8yHqpHcgmGkvE/0YJtm4fCWc72WzXao5WhvTfr0w0cmSQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4V0cWp41NdzQvY for ; Thu, 21 Mar 2024 07:28:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42L7ScvD003116 for ; Thu, 21 Mar 2024 07:28:38 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42L7Sccg003115 for net@FreeBSD.org; Thu, 21 Mar 2024 07:28:38 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 276838] ovpn(4) - problems with large TCP segments over IPv6 tunnel when DCO module is used at both ends Date: Thu, 21 Mar 2024 07:28:36 +0000 X-Bugzilla-Reason: CC AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kp@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276838 --- Comment #3 from Kristof Provost --- I am unable to reproduce any such issue testing locally. Over an IPv6 tunne= l I manage to transmit arbitrary sized ping requests, and I also see no issues transmitting a large (1MB) file. Patch for reference: diff --git a/tests/sys/net/if_ovpn/if_ovpn.sh b/tests/sys/net/if_ovpn/if_ovpn.sh index bbaffa0bce73..0ec2563cf355 100644 --- a/tests/sys/net/if_ovpn/if_ovpn.sh +++ b/tests/sys/net/if_ovpn/if_ovpn.sh @@ -308,10 +308,28 @@ atf_test_case "4in6" "cleanup" keepalive 100 600 " + dd if=3D/dev/random of=3Dtest.img bs=3D1024 count=3D1024 + cat test.img | jexec a nc -N -l 1234 & + # Give the tunnel time to come up sleep 10 atf_check -s exit:0 -o ignore jexec b ping -c 3 198.51.100.1 + + # MTU sweep + for i in `seq 1000 1500` + do + atf_check -s exit:0 -o ignore jexec b \ + ping -c 1 -s $i 198.51.100.1 + done + + rcvmd5=3D$(jexec b nc -N -w 3 198.51.100.1 1234 | md5) + md5=3D$(md5 test.img) + + if [ $md5 !=3D $rcvmd5 ]; + then + atf_fail "Transmit corruption!" + fi } 4in6_cleanup() --=20 You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.= From nobody Thu Mar 21 11:13:05 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V0jVp6F2yz5Fqqc for ; Thu, 21 Mar 2024 11:13:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V0jVp5C6bz4DPc for ; Thu, 21 Mar 2024 11:13:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711019586; a=rsa-sha256; cv=none; b=rHOqbh3YlfL1XQ8a9kz52hdGNS2NBsR9E2EP6qh8tslcqDLf5gEXoXgXuer11Jqd8ZYfCd CJhd6ZvR3LHzyzR1m/S82DYXwCchVvRnabpEUk6wzR9Y3/X08+dR80BWnrPSXb9iHbd+9x AgWOgECa4Rs5JB6xOjAfDbwDjbynFeIKoGbhpLhVPY6WFB+NtEjxVP8566fAKYN7Qhl6/W rI8pwf0DvNGXvtdKy5j+tq3hB7DM0brpBeBqJQf1P4CSoDdKAoAkJVY9GhoxWbQcLcQPD7 nUEEN/uJetRCBRPoaUrTs/WeYCL4QGjYQh/NVIvKP2+3ugb8kLjPamKer6kPTg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711019586; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jSaMYXgiPPzjEZoaiNfvOTbfrFPf6gJt9WpWlN3yydQ=; b=qsVNGV65vp4i0LDjYaGbskd3N8EUyzm/EmoHSnaet+DrcFvtpyejxjiSjNj2+n1r128Aq7 IEFNA5PTpCSDWyTEcAxoVDmZYzS5WDRYXSgO4Kmjn70hNhBjAXVFZUHfhQOqqur4q+i6Rd 9Op6s5r+pR8BpwbQnLwW+0Zo3W25p1d1J4XSQ4SgrVmHM7sceQEtFWtBghUQfLSGxTP1+J 1O5eTIo9IZs/roT9RkfVWdRYVjAy4XSstA3r8a20crUR4A/TUERCtm+TxGDPV/cX6uqHOK H/35JbpC5MaWEYFwEUyKoXTHTAc3QUyxRVVnc8YS0N360CYF3SUTxciAMD5ZaQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4V0jVp4prqzXjW for ; Thu, 21 Mar 2024 11:13:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42LBD6du078538 for ; Thu, 21 Mar 2024 11:13:06 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42LBD6kT078537 for net@FreeBSD.org; Thu, 21 Mar 2024 11:13:06 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 276838] ovpn(4) - problems with large TCP segments over IPv6 tunnel when DCO module is used at both ends Date: Thu, 21 Mar 2024 11:13:05 +0000 X-Bugzilla-Reason: AssignedTo CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: zarychtam@plan-b.pwste.edu.pl X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276838 --- Comment #4 from Marek Zarychta --- In my case, the limits for the successful "ping -4 -s $size remote_IP" are: - in IPv6 tunnel, $size is limited to 1400 - in IPv4 tunnel $size limit is 1412 It was tested between FreeBSD stable/14 running OpenVPN 2.6.10 with DCO and FreeBSD CURRENT running OpenVPN 2.6.9 as a client.=20 It's not the case though. The problem I faced was too large TCP segments th= at couldn't pass the tunnel thus preventing SSH sessions from being established over tunnel. --=20 You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.= From nobody Thu Mar 21 12:57:44 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V0lqy484gz5DYjn; Thu, 21 Mar 2024 12:58:06 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V0lqy14d6z4Pvg; Thu, 21 Mar 2024 12:58:06 +0000 (UTC) (envelope-from gallatin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711025886; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=VDngckJsj6YwsGVPKf4TVi02lPK/AhReZS1mJGoCgQY=; b=SLdD4qCxkcNyxCM641BomckfhnCmHmOUf+Qu2DB2ox5hdqhn006Jg5xDp+OmAN6cf4B1vg DHfBECa/vScysiMHAk67BMmJbY7ru8En7Xmqz5bYPd/mJ20S/uJJyJNYKfCBCaIacNY56D VQbLsyqPYxhHZW5DWIVHx8+HD90w2FN5Ai9Xg/qRrfLLDBSP5j1t7vpWeOvQJTbRssMZje 5GYSyXxZc9Uudl9CWFqFPvz59+oaSfQMj9wVYSW0UzwEj6oXItzxQZBFSv90oTe7fTrnwq 5LOqAP82hTK13Tk6ErDrE/Akwi1HfWlMnYpZgtRpqfx+elIO9HpWcPqoBwCAug== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711025886; a=rsa-sha256; cv=none; b=jXlRKPpla4YXlUMnXqETuVj9PTgL8DlGh4CZX3T4eTYDYFpruyw30kRynwTW8/1l1qZ6hK EefV0LDY87QJUk0sPITzUNvKI4QgfZHVnHbvHaR8NEpCW0jJwgpt4Q2vCTE+vXBns0KBQs dboTuZ5z6H8hxl59JVk3xSkTnctgjM6OlfnJ1lG/+yspZWd1cTkSN/sc6cXHUPjvoWdOPo 5yJScXooAMXP8Qn8qlvXRvzIj5fa0Cyd3ML0EyQx/5ZrGhl8sUYPG0a3RFVZOVkJs/PF2V rfsA2gVO/kv51UXX19UZYVP5MFQoK3AQInyPG8vCuzFl89PMFU4tshfP6Uqgzw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711025886; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=VDngckJsj6YwsGVPKf4TVi02lPK/AhReZS1mJGoCgQY=; b=m0bnOnL3qhWofFV4sboanFG6WwMDYk/7Oh3aJxcVj6qYWLKLwCFXZPs5kEbtbjdhv+TgLf dz1vJfRUjcu5cvzEyzfBr/5MdMLNGcV5hAlQmT2RNzgGmAYDyVE+2XkeBT59l7+pGpibce 1nEka2r0AqOY9KlZn49nvAB1oM6Qpjh/hwIL/+87PlxjEZ/IsvVW0jYMGlAOwEeJ/8Stsd vNm58qTR0ikCLfwQfCHlpxNNPM3e9dves0fM92Wfh3PIh25OmYB28PO0hUjNtMRBTctAcA vt5ujcjSpCw8YeIaqOUX4CUpJ4LMhzStafluAGS+Y/UGd8WhauZwHyojEvL9og== Received: from fauth2-smtp.messagingengine.com (fauth2-smtp.messagingengine.com [103.168.172.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: gallatin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4V0lqx6pz5zjjD; Thu, 21 Mar 2024 12:58:05 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfauth.nyi.internal (Postfix) with ESMTP id 448621200066; Thu, 21 Mar 2024 08:58:05 -0400 (EDT) Received: from imap53 ([10.202.2.103]) by compute5.internal (MEProxy); Thu, 21 Mar 2024 08:58:05 -0400 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrleeigdegiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefofgggkfgjfhffhffvvefutgesrgdtreerreerjeenucfhrhhomhepfdffrhgv ficuifgrlhhlrghtihhnfdcuoehgrghllhgrthhinhesfhhrvggvsghsugdrohhrgheqne cuggftrfgrthhtvghrnhepudehheeiffefudeuteeluddtgeeijedtffehjeeufeeiteei vdegvdeiiefgtddvnecuffhomhgrihhnpehfrhgvvggsshgurdhorhhgpdhmphhiqdhsfi hsrdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhho mhepghgrlhhlrghtihhnodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqddufe efheelvddvudeiqddvleehtdegudekgedqghgrlhhlrghtihhnpeepfhhrvggvsghsugdr ohhrghesfhgrshhtmhgrihhlrdgtohhm X-ME-Proxy: Feedback-ID: i41414658:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id E0755364006F; Thu, 21 Mar 2024 08:58:04 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-332-gdeb4194079-fm-20240319.002-gdeb41940 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Message-Id: In-Reply-To: References: <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> <6047C8EF-B1B0-4286-93FA-AA38F8A18656@karels.net> <8031cd99-ded8-4b06-93b3-11cc729a8b2c@app.fastmail.com> <38c54399-6c96-44d8-a3a2-3cc1bfbe50c2@app.fastmail.com> <27d8144f-0658-46f6-b8f3-35eb60061644@lakerest.net> Date: Thu, 21 Mar 2024 08:57:44 -0400 From: "Drew Gallatin" To: "Konstantin Belousov" , rrs Cc: "Mike Karels" , tuexen , "Nuno Teixeira" , garyj@gmx.de, current@freebsd.org, net@freebsd.org, "Randall Stewart" Subject: Re: Request for Testing: TCP RACK Content-Type: multipart/alternative; boundary=3b39556bddae44fcbfa6d30004956a6c --3b39556bddae44fcbfa6d30004956a6c Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable The entire point is to *NOT* go through the overhead of scheduling somet= hing asynchronously, but to take advantage of the fact that a user/kerne= l transition is going to trash the cache anyway. In the common case of a system which has less than the threshold number= of connections , we access the tcp_hpts_softclock function pointer, mak= e one function call, and access hpts_that_need_softclock, and then retur= n. So that's 2 variables and a function call. I think it would be preferable to avoid that call, and to move the decla= ration of tcp_hpts_softclock and hpts_that_need_softclock so that they a= re in the same cacheline. Then we'd be hitting just a single line in th= e common case. (I've made comments on the review to that effect). Also, I wonder if the threshold could get higher by default, so that hpt= s is never called in this context unless we're to the point where we're = scheduling thousands of runs of the hpts thread (and taking all those cl= ock interrupts). Drew On Wed, Mar 20, 2024, at 8:17 PM, Konstantin Belousov wrote: > On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs wrote: > > Ok I have created > >=20 > > https://reviews.freebsd.org/D44420 > >=20 > >=20 > > To address the issue. I also attach a short version of the patch tha= t Nuno > > can try and validate > >=20 > > it works. Drew you may want to try this and validate the optimizatio= n does > > kick in since I can > >=20 > > only now test that it does not on my local box :) > The patch still causes access to all cpu's cachelines on each userret. > It would be much better to inc/check the threshold and only schedule t= he > call when exceeded. Then the call can occur in some dedicated context, > like per-CPU thread, instead of userret. >=20 > >=20 > >=20 > > R > >=20 > >=20 > >=20 > > On 3/18/24 3:42 PM, Drew Gallatin wrote: > > > No. The goal is to run on every return to userspace for every thr= ead. > > >=20 > > > Drew > > >=20 > > > On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote: > > > > On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote: > > > > > I got the idea from > > > > > https://people.mpi-sws.org/~druschel/publications/soft-timers-= tocs.pdf > > > > > The gist is that the TCP pacing stuff needs to run frequently,= and > > > > > rather than run it out of a clock interrupt, its more efficien= t to run > > > > > it out of a system call context at just the point where we ret= urn to > > > > > userspace and the cache is trashed anyway. The current impleme= ntation > > > > > is fine for our workload, but probably not idea for a generic = system. > > > > > Especially one where something is banging on system calls. > > > > > > > > > > Ast's could be the right tool for this, but I'm super unfamili= ar with > > > > > them, and I can't find any docs on them. > > > > > > > > > > Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equival= ent to > > > > > what's happening here? > > > > This call would need some AST number added, and then it register= s the > > > > ast to run on next return to userspace, for the current thread. > > > >=20 > > > > Is it enough? > > > > > > > > > > Drew > > > >=20 > > > > > > > > > > On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote: > > > > > > On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote: > > > > > > > On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote: > > > > > > > > > > > > > > >> On 18. Mar 2024, at 12:42, Nuno Teixeira > > > > wrote: > > > > > > > >> > > > > > > > >> Hello all! > > > > > > > >> > > > > > > > >> It works just fine! > > > > > > > >> System performance is OK. > > > > > > > >> Using patch on main-n268841-b0aaf8beb126(-dirty). > > > > > > > >> > > > > > > > >> --- > > > > > > > >> net.inet.tcp.functions_available: > > > > > > > >> Stack D > > > > Alias PCB count > > > > > > > >> freebsd freebsd 0 > > > > > > > >> rack * > > > > rack 38 > > > > > > > >> --- > > > > > > > >> > > > > > > > >> It would be so nice that we can have a sysctl tunnable = for > > > > this patch > > > > > > > >> so we could do more tests without recompiling kernel. > > > > > > > > Thanks for testing! > > > > > > > > > > > > > > > > @gallatin: can you come up with a patch that is acceptab= le > > > > for Netflix > > > > > > > > and allows to mitigate the performance regression. > > > > > > > > > > > > > > Ideally, tcphpts could enable this automatically when it > > > > starts to be > > > > > > > used (enough?), but a sysctl could select auto/on/off. > > > > > > There is already a well-known mechanism to request execution= of the > > > > > > specific function on return to userspace, namely AST. The d= ifference > > > > > > with the current hack is that the execution is requested for= one > > > > callback > > > > > > in the context of the specific thread. > > > > > > > > > > > > Still, it might be worth a try to use it; what is the reason= to > > > > hit a thread > > > > > > that does not do networking, with TCP processing? > > > > > > > > > > > > > > > > > > > > Mike > > > > > > > > > > > > > > > Best regards > > > > > > > > Michael > > > > > > > >> > > > > > > > >> Thanks all! > > > > > > > >> Really happy here :) > > > > > > > >> > > > > > > > >> Cheers, > > > > > > > >> > > > > > > > >> Nuno Teixeira escreveu (domingo, > > > > 17/03/2024 =C3=A0(s) 20:26): > > > > > > > >>> > > > > > > > >>> Hello, > > > > > > > >>> > > > > > > > >>>> I don't have the full context, but it seems like the > > > > complaint is a performance regression in bonnie++ and perhaps ot= her > > > > things when tcp_hpts is loaded, even when it is not used. Is th= at > > > > correct? > > > > > > > >>>> > > > > > > > >>>> If so, I suspect its because we drive the > > > > tcp_hpts_softclock() routine from userret(), in order to avoid t= ons > > > > of timer interrupts and context switches. To test this theory, = you > > > > could apply a patch like: > > > > > > > >>> > > > > > > > >>> It's affecting overall system performance, bonnie was = just > > > > a way to > > > > > > > >>> get some numbers to compare. > > > > > > > >>> > > > > > > > >>> Tomorrow I will test patch. > > > > > > > >>> > > > > > > > >>> Thanks! > > > > > > > >>> > > > > > > > >>> -- > > > > > > > >>> Nuno Teixeira > > > > > > > >>> FreeBSD Committer (ports) > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> -- > > > > > > > >> Nuno Teixeira > > > > > > > >> FreeBSD Committer (ports) > > > > > > > > > > > > > > > > >=20 > > >=20 >=20 > > diff --git a/sys/netinet/tcp_hpts.c b/sys/netinet/tcp_hpts.c > > index 8c4d2d41a3eb..eadbee19f69c 100644 > > --- a/sys/netinet/tcp_hpts.c > > +++ b/sys/netinet/tcp_hpts.c > > @@ -216,6 +216,7 @@ struct tcp_hpts_entry { > > void *ie_cookie; > > uint16_t p_num; /* The hpts number one per cpu */ > > uint16_t p_cpu; /* The hpts CPU */ > > + uint8_t hit_callout_thresh; > > /* There is extra space in here */ > > /* Cache line 0x100 */ > > struct callout co __aligned(CACHE_LINE_SIZE); > > @@ -269,6 +270,11 @@ static struct hpts_domain_info { > > int cpu[MAXCPU]; > > } hpts_domains[MAXMEMDOM]; > > =20 > > +counter_u64_t hpts_that_need_softclock; > > +SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, needsoftcloc= k, CTLFLAG_RD, > > + &hpts_that_need_softclock, > > + "Number of hpts threads that need softclock"); > > + > > counter_u64_t hpts_hopelessly_behind; > > =20 > > SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, hopeless, CT= LFLAG_RD, > > @@ -334,7 +340,7 @@ SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, precisi= on, CTLFLAG_RW, > > &tcp_hpts_precision, 120, > > "Value for PRE() precision of callout"); > > SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, cnt_thresh, CTLFLAG_RW, > > - &conn_cnt_thresh, 0, > > + &conn_cnt_thresh, DEFAULT_CONNECTION_THESHOLD, > > "How many connections (below) make us use the callout based mec= hanism"); > > SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO, logging, CTLFLAG_RW, > > &hpts_does_tp_logging, 0, > > @@ -1548,6 +1554,9 @@ __tcp_run_hpts(void) > > struct tcp_hpts_entry *hpts; > > int ticks_ran; > > =20 > > + if (counter_u64_fetch(hpts_that_need_softclock) =3D=3D 0) > > + return; > > + > > hpts =3D tcp_choose_hpts_to_run(); > > =20 > > if (hpts->p_hpts_active) { > > @@ -1683,6 +1692,13 @@ tcp_hpts_thread(void *ctx) > > ticks_ran =3D tcp_hptsi(hpts, 1); > > tv.tv_sec =3D 0; > > tv.tv_usec =3D hpts->p_hpts_sleep_time * HPTS_TICKS_PER_SLOT; > > + if ((hpts->p_on_queue_cnt > conn_cnt_thresh) && (hpts->hit_callout= _thresh =3D=3D 0)) { > > + hpts->hit_callout_thresh =3D 1; > > + counter_u64_add(hpts_that_need_softclock, 1); > > + } else if ((hpts->p_on_queue_cnt <=3D conn_cnt_thresh) && (hpts->h= it_callout_thresh =3D=3D 1)) { > > + hpts->hit_callout_thresh =3D 0; > > + counter_u64_add(hpts_that_need_softclock, -1); > > + } > > if (hpts->p_on_queue_cnt >=3D conn_cnt_thresh) { > > if(hpts->p_direct_wake =3D=3D 0) { > > /* > > @@ -1818,6 +1834,7 @@ tcp_hpts_mod_load(void) > > cpu_top =3D NULL; > > #endif > > tcp_pace.rp_num_hptss =3D ncpus; > > + hpts_that_need_softclock =3D counter_u64_alloc(M_WAITOK); > > hpts_hopelessly_behind =3D counter_u64_alloc(M_WAITOK); > > hpts_loops =3D counter_u64_alloc(M_WAITOK); > > back_tosleep =3D counter_u64_alloc(M_WAITOK); > > @@ -2042,6 +2059,7 @@ tcp_hpts_mod_unload(void) > > free(tcp_pace.grps, M_TCPHPTS); > > #endif > > =20 > > + counter_u64_free(hpts_that_need_softclock); > > counter_u64_free(hpts_hopelessly_behind); > > counter_u64_free(hpts_loops); > > counter_u64_free(back_tosleep); >=20 >=20 --3b39556bddae44fcbfa6d30004956a6c Content-Type: text/html;charset=utf-8 Content-Transfer-Encoding: quoted-printable
The entire poin= t is to *NOT* go through the overhead of scheduling something asynchrono= usly, but to take advantage of the fact that a user/kernel transition is= going to trash the cache anyway.

In the co= mmon case of a system which has less than the threshold  number of = connections , we access the tcp_hpts_softclock function pointer, make on= e function call, and access hpts_that_need_softclock, and then return.&n= bsp; So that's 2 variables and a function call.

=
I think it would be preferable to avoid that call, and to move the = declaration of tcp_hpts_softclock and hpts_that_need_softclock so that t= hey are in the same cacheline.  Then we'd be hitting just a single = line in the common case.  (I've made comments on the review to that= effect).

Also, I wonder if the threshold c= ould get higher by default, so that hpts is never called in this context= unless we're to the point where we're scheduling thousands of runs of t= he hpts thread (and taking all those clock interrupts).
Drew

On Wed, Mar 20, 2024, at = 8:17 PM, Konstantin Belousov wrote:
On Tue, Mar 19, 2024 at 06:19:52AM -0400, rrs w= rote:
> Ok I have created
> To address the issue. I also attach a sho= rt version of the patch that Nuno
> can try and validat= e
> 
> it works. Drew you may wan= t to try this and validate the optimization does
> kick= in since I can
> 
> only now tes= t that it does not on my local box :)
The patch still caus= es access to all cpu's cachelines on each userret.
It woul= d be much better to inc/check the threshold and only schedule the
call when exceeded.  Then the call can occur in some dedica= ted context,
like per-CPU thread, instead of userret.
<= /div>

> 
> 
> R
> 
> 
> 
> On 3/18/24 3:42 PM, Drew Gallatin wrote:=
> > No.  The goal is to run on every return to= userspace for every thread.
> > 
> > Drew
> > 
> > On= Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote:
> > > On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin = wrote:
> > > > I got the idea from
> > > > https://people.mpi-sws.org/~drusc= hel/publications/soft-timers-tocs.pdf
> > > &= gt; The gist is that the TCP pacing stuff needs to run frequently, and
> > > > rather than run it out of a clock inter= rupt, its more efficient to run
> > > > it out= of a system call context at just the point where we return to
=
> > > > userspace and the cache is trashed anyway. The = current implementation
> > > > is fine for our= workload, but probably not idea for a generic system.
>= ; > > > Especially one where something is banging on system cal= ls.
> > > >
> > > > = Ast's could be the right tool for this, but I'm super unfamiliar with
> > > > them, and I can't find any docs on them.=
> > > >
> > > > Wou= ld ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent to
> > > > what's happening here?
> &g= t; > This call would need some AST number added, and then it register= s the
> > > ast to run on next return to userspac= e, for the current thread.
> > > 
<= div>> > > Is it enough?
> > > >
> > > > Drew
> > > 
=
> > > >
> > > > On Mon,= Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote:
>= > > > > On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Kare= ls wrote:
> > > > > > On 18 Mar 2024, at= 7:04, tuexen@freebsd.org= wrote:
> > > > > >
> &= gt; > > > > >> On 18. Mar 2024, at 12:42, Nuno Teixeir= a
> > > <eduardo@freebsd.org> wrote:
> > > > &= gt; > >>
> > > > > > >> H= ello all!
> > > > > > >>
=
> > > > > > >> It works just fine!
> > > > > > >> System performance is OK.
> > > > > > >> Using patch on main-= n268841-b0aaf8beb126(-dirty).
> > > > > >= ; >>
> > > > > > >> ---
<= /div>
> > > > > > >> net.inet.tcp.functions_= available:
> > > > > > >> Stack&nb= sp;           &nb= sp;           &nb= sp;  D
> > > Alias    &n= bsp;           &n= bsp;           PCB cou= nt
> > > > > > >> freebsd freebsd&= nbsp;           &= nbsp;           &= nbsp; 0
> > > > > > >> rack &= nbsp;           &= nbsp;           &= nbsp;  *
> > > rack    &= nbsp;           &= nbsp;            = 38
> > > > > > >> ---
> > > > > > >>
> > > &g= t; > > >> It would be so nice that we can have a sysctl tunn= able for
> > > this patch
> >= > > > > >> so we could do more tests without recompil= ing kernel.
> > > > > > > Thanks for = testing!
> > > > > > >
= > > > > > > > @gallatin: can you come up with a pat= ch that is acceptable
> > > for Netflix
=
> > > > > > > and allows to mitigate the perfo= rmance regression.
> > > > > >
=
> > > > > > Ideally, tcphpts could enable this au= tomatically when it
> > > starts to be
<= div>> > > > > > used (enough?), but a sysctl could sel= ect auto/on/off.
> > > > > There is already= a well-known mechanism to request execution of the
> &= gt; > > > specific function on return to userspace, namely AST.=   The difference
> > > > > with the cu= rrent hack is that the execution is requested for one
>= > > callback
> > > > > in the contex= t of the specific thread.
> > > > >
> > > > > Still, it might be worth a try to use it= ; what is the reason to
> > > hit a thread
> > > > > that does not do networking, with TCP p= rocessing?
> > > > >
> >= ; > > > >
> > > > > > Mike
> > > > > >
> > > = > > > > Best regards
> > > > > = > > Michael
> > > > > > >>
> > > > > > >> Thanks all!
> > > > > > >> Really happy here :)
> > > > > > >>
> > &= gt; > > > >> Cheers,
> > > > &g= t; > >>
> > > > > > >> Nu= no Teixeira <eduardo@freebsd.o= rg> escreveu (domingo,
> > > 17/03/2024 =C3= =A0(s) 20:26):
> > > > > > >>><= br>
> > > > > > >>> Hello,
> > > > > > >>>
> >= > > > > >>>> I don't have the full context, but= it seems like the
> > > complaint is a performan= ce regression in bonnie++ and perhaps other
> > >= things when tcp_hpts is loaded, even when it is not used.  Is that=
> > > correct?
> > > >= > > >>>>
> > > > > > = >>>> If so, I suspect its because we drive the
> > > tcp_hpts_softclock() routine from userret(), in order to= avoid tons
> > > of timer interrupts and context= switches.  To test this theory,  you
> > = > could apply a patch like:
> > > > > &g= t; >>>
> > > > > > >>>= It's affecting overall system performance, bonnie was just
> > > a way to
> > > > > > &g= t;>> get some numbers to compare.
> > > >= ; > > >>>
> > > > > > >= ;>> Tomorrow I will test patch.
> > > > = > > >>>
> > > > > > >&= gt;> Thanks!
> > > > > > >>>=
> > > > > > >>> --
> > > > > > >>> Nuno Teixeira
> > > > > > >>> FreeBSD Committer (ports)
> > > > > > >>
> &= gt; > > > > >>
> > > > > = > >>
> > > > > > >> --
> > > > > > >> Nuno Teixeira
> > > > > > >> FreeBSD Committer (ports)<= br>
> > > > > >
> > >= > >
> > > 
> >&nb= sp;

> diff --git a/sys/netinet/tcp_hpts.= c b/sys/netinet/tcp_hpts.c
> index 8c4d2d41a3eb..eadbee= 19f69c 100644
> --- a/sys/netinet/tcp_hpts.c
<= div>> +++ b/sys/netinet/tcp_hpts.c
> @@ -216,6 +216,= 7 @@ struct tcp_hpts_entry {
>  void *ie_cookie;<= br>
>  uint16_t p_num; /* The hpts number one per cp= u */
>  uint16_t p_cpu; /* The hpts CPU */
> + uint8_t hit_callout_thresh;
>  /*= There is extra space in here */
>  /* Cache line= 0x100 */
>  struct callout co __aligned(CACHE_LI= NE_SIZE);
> @@ -269,6 +270,11 @@ static struct hpts_dom= ain_info {
>  int cpu[MAXCPU];
>=   } hpts_domains[MAXMEMDOM];
>  
> +counter_u64_t hpts_that_need_softclock;
> = +SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO, needsoftclock, C= TLFLAG_RD,
> +    &hpts_that_need_so= ftclock,
> +    "Number of hpts threads = that need softclock");
> +
>  cou= nter_u64_t hpts_hopelessly_behind;
>  
>  SYSCTL_COUNTER_U64(_net_inet_tcp_hpts_stats, OID_AUTO= , hopeless, CTLFLAG_RD,
> @@ -334,7 +340,7 @@ SYSCTL_IN= T(_net_inet_tcp_hpts, OID_AUTO, precision, CTLFLAG_RW,
>= ;      &tcp_hpts_precision, 120,
<= div>>      "Value for PRE() precision of cal= lout");
>  SYSCTL_INT(_net_inet_tcp_hpts, OID_AUTO= , cnt_thresh, CTLFLAG_RW,
> -    &co= nn_cnt_thresh, 0,
> +    &conn_cnt_t= hresh, DEFAULT_CONNECTION_THESHOLD,
>   =    "How many connections (below) make us use the callout based= mechanism");
>  SYSCTL_INT(_net_inet_tcp_hpts, OI= D_AUTO, logging, CTLFLAG_RW,
>    &= nbsp; &hpts_does_tp_logging, 0,
> @@ -1548,6 +1554,= 9 @@ __tcp_run_hpts(void)
>  struct tcp_hpts_entr= y *hpts;
>  int ticks_ran;
>&nbs= p; 
> + if (counter_u64_fetch(hpts_that_need_softc= lock) =3D=3D 0)
> + return;
> +
>  hpts =3D tcp_choose_hpts_to_run();
&g= t;  
>  if (hpts->p_hpts_active) {
> @@ -1683,6 +1692,13 @@ tcp_hpts_thread(void *ctx)
<= /div>
>  ticks_ran =3D tcp_hptsi(hpts, 1);
&g= t;  tv.tv_sec =3D 0;
>  tv.tv_usec =3D hpts= ->p_hpts_sleep_time * HPTS_TICKS_PER_SLOT;
> + if ((= hpts->p_on_queue_cnt > conn_cnt_thresh) && (hpts->hit_c= allout_thresh =3D=3D 0)) {
> + hpts->hit_callout_th= resh =3D 1;
> + counter_u64_add(hpts_that_need_softclo= ck, 1);
> + } else if ((hpts->p_on_queue_cnt <=3D= conn_cnt_thresh) && (hpts->hit_callout_thresh =3D=3D 1)) {
> + hpts->hit_callout_thresh =3D 0;
&g= t; + counter_u64_add(hpts_that_need_softclock, -1);
> = + }
>  if (hpts->p_on_queue_cnt >=3D conn_c= nt_thresh) {
>  if(hpts->p_direct_wake =3D=3D= 0) {
>  /*
> @@ -1818,6 +1834= ,7 @@ tcp_hpts_mod_load(void)
>  cpu_top =3D NULL= ;
>  #endif
>  tcp_pace.rp_= num_hptss =3D ncpus;
> + hpts_that_need_softclock =3D c= ounter_u64_alloc(M_WAITOK);
>  hpts_hopelessly_be= hind =3D counter_u64_alloc(M_WAITOK);
>  hpts_loo= ps =3D counter_u64_alloc(M_WAITOK);
>  back_tosle= ep =3D counter_u64_alloc(M_WAITOK);
> @@ -2042,6 +2059,= 7 @@ tcp_hpts_mod_unload(void)
>  free(tcp_pace.g= rps, M_TCPHPTS);
>  #endif
> = ; 
> + counter_u64_free(hpts_that_need_softclock);=
>  counter_u64_free(hpts_hopelessly_behind);
=
>  counter_u64_free(hpts_loops);
>&= nbsp; counter_u64_free(back_tosleep);


=

--3b39556bddae44fcbfa6d30004956a6c-- From nobody Thu Mar 21 13:07:55 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V0m3J3dfJz5DZfr for ; Thu, 21 Mar 2024 13:07:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V0m3J2cbzz4S0P for ; Thu, 21 Mar 2024 13:07:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711026476; a=rsa-sha256; cv=none; b=gbmxjX0xilrut/JB0MmR1Fo3sTsXEbJS8qaS0AcY1PwzrxNus7QX/skhkwWE3j4YBvfN4h H473cKrxPKeQG0+r6qNsgR1rV8c0yJp99VF9jmI50CaUtIbPxTAxuTIA8DAS61uupTv9Z8 MCDROBfQRgJUEo8ERzpCSzkF0u7AwhFT9BdS2Hl4tuxACqN/KdXp+YluVDORBJNe2mrqxF 5NSbjXIx+yGSN5iUQMFVX9qJGLUrRI0+By/WWlVxXpP2ZbFh01LHXghtAyZuPZWSj0Lg8e g4XRngRnafqzV8f+Akvc+V7mIm25I8wcJCg3uGMgB6CuEaCshTACvtS6wIiFHg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711026476; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QPwUA/jHRsRXJv5hIXasPoI1jxYPnQEbUfsPBylvphw=; b=A02Rhu7vh82ac186X8MrMmtObvnu3cdqwv6SdOOauCj0H7mL/b+0oEt64eeBt1kM75yWRc qWoe95Jg34R+t/l+wV19p6o/RwxgR1WPt4H0djQ6pN5Luo0TVAxGR/a4TmlEAZi+UUSn+k 0FjHKNdMS3NoejsmNaRCIMFuLJZwaHAydwBxosJbFqzvN1F2/jNdOLJeN7b2abXt+vDax8 PnZ6Z3RQ2j2zMbToyGoxJot541JHpyXDXJnOBvhXbrbkjFqIu39n22Qt41hdw1LMuxEZNO f2hs9TcEh5GgdgTDDOeGq4yc/vkeFVH9hV5fc4zrv1Ua8XcnhvOA7s5kSAHatw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4V0m3J2DHYzcCX for ; Thu, 21 Mar 2024 13:07:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42LD7uas060935 for ; Thu, 21 Mar 2024 13:07:56 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42LD7uP0060934 for net@FreeBSD.org; Thu, 21 Mar 2024 13:07:56 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 277349] The net.inet.ip.source_address_validation should ignore CARP IP in backup state Date: Thu, 21 Mar 2024 13:07:55 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mickael.maillot@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277349 mickael.maillot@gmail.com changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mickael.maillot@gmail.com --- Comment #8 from mickael.maillot@gmail.com --- this fix will be merged in stable/14 ? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Mar 21 17:11:17 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V0sS82P76z5F4bN for ; Thu, 21 Mar 2024 17:11:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V0sS632vdz3xSC for ; Thu, 21 Mar 2024 17:11:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711041078; a=rsa-sha256; cv=none; b=MRMqCfC36IGaKqTJTmxoIJcdocg8AheQ4arZkIOPpGGCPHAuuDokcLRH9eHzeN5bASmStS v79eVcgdi+aOALFn08YjFhmlQgONwNmt1tgUN3tRq6KoBv/3XwXWHiX0fbSNxcb6060gav 780rlBiErrRHadJawHpURHsJ4x79WTWl1XFZZITO0tKagQax5zkadA6j22GyBifeeWXi0G yR8fQUiU6ci56t38ppWUJMRJ1WmdydXfBCY+oQBKQyIg/Kh2iihKeD9Ux5uGH0MGQKwp45 pYxJiCQv18qku9vxLequNfD+vpkBCS8a1oL9gKArO8LZFnz4YEda0ikFbyANYQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711041078; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5pIx7dJlkb+dsRqamyVH50p/wqw4v7f9ZKlyrgQHU4k=; b=QJOL8KAPFuEG0RHcIq02yycpsk85OBO/OeQoBkLvLr+0rTnWVCNZw342Rm24mwiMlhCY9e coxB6WcDZC/IpxJ2d9eFG6G5fcYAlvNp0JWE9OakGX0SSyWh6ai8LZtHjm5ju1f4ncBWe6 u3NYXnfLmsZ0WYKwLZANKYY3PxkKyArqT8zU8GkX2t5N9JoQ0ffUt1zPixsqMreDMGmaTQ rNKCOLdI+cmStkC2rwvuLiERPt/NIAVQmV5ySRJpyLRDeYVuJJKEtWJ0R66eTbN8MEHmph kNzkpozs1ds93NH32diErwFfuD3d4xIyOvRJqXdJhQ6gGcGC8vbgdgESnimqjQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4V0sS62dwtzkVL for ; Thu, 21 Mar 2024 17:11:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42LHBIgj001639 for ; Thu, 21 Mar 2024 17:11:18 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42LHBI1d001638 for net@FreeBSD.org; Thu, 21 Mar 2024 17:11:18 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 277349] The net.inet.ip.source_address_validation should ignore CARP IP in backup state Date: Thu, 21 Mar 2024 17:11:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: glebius@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277349 --- Comment #9 from Gleb Smirnoff --- On Thu Mar 21 13:07:55 2024 UTC, mickael.maillot@gmail.com wrote: > this fix will be merged in stable/14 ? I plan to merge next week. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Mar 21 20:42:15 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V0y7k2QZ2z5FTQC for ; Thu, 21 Mar 2024 20:42:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V0y7k1QTgz4Yk5 for ; Thu, 21 Mar 2024 20:42:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711053746; a=rsa-sha256; cv=none; b=tu8/2st3g7E2HN+c6dtOungiG0qPM3XkCcPIowbiwzFJ1Ye74jtiSEDlMFJS7rCsBGTBwJ iiqLb1GXru7Tcul4qgNtxZJAPOnYuCZggjlJH/K7iEFuU/1JN9US+Dh580hwmRP4OHSjss X8qECi5RLJOaAjmB6FDtwCU2jlpR8Nt5AfgTtIdALMxLa2On++58yh0wGgPTaS49NSmm68 yBbjdQOEoX1ZSx7P8al+522PfgX6hJJiftmLVxBERE9agtVkiDcC8OXqPPRtNl2YPvY0Ik W/HqlmcHYfgc8gD4yv1R6sqcMRoOMMvzHj1xDGUImD/MoW1QL5GyrMr+IlgqyQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711053746; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/b6fOJwgPbcnvjzwm4bB7wvlZaJvJklDPd4983CIM+g=; b=ga0jFMXeT/MFUPr90+xJefyjamOyvhCfHzr95bx7jHYQ7v95lNzpynn86fFgTQpBEF6l66 TxI1/+VUPclRc2UQFb+OsoYx5d0r+eMQDjPZhsWvbXU1wdGRzVZAccDbX3+gIo6TsXCO93 jEGlIjk/pD4IkcjxL3rSGaGv4Yv0bgp+I4JfH0QarXv/GPxvmeXv/4a4xGBTMO+H32+8cY cx3l3VqtPupZ9qjdEGe2bMI+wG1VAY1q6sxx5O4kPGaSh7r8ZiJ0Zu8liBq3MY6FCY1vm+ rOh9YXeLnqRPWbqEpjhuQdRmu/oC4A0PPS2YOCEVG3VqptKyvDtyQ2o3GlEPMA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4V0y7k11hVzqvF for ; Thu, 21 Mar 2024 20:42:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42LKgQw2010676 for ; Thu, 21 Mar 2024 20:42:26 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42LKgQQG010671 for net@FreeBSD.org; Thu, 21 Mar 2024 20:42:26 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 166724] if_re(4): watchdog timeout Date: Thu, 21 Mar 2024 20:42:15 +0000 X-Bugzilla-Reason: AssignedTo CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: needs-patch, needs-qa X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: vova@fbsd.ru X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: mfc-stable13? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D166724 --- Comment #116 from vova@fbsd.ru --- (In reply to vova from comment #115) Well ... with if_re from net/realtek-re-kmod also I am loosing connectivity= ... not that often as with in-tree driver, with quite funny results: 1. system timer gots so driffted that ntpd dies with panic: Mar 20 15:17:55 srv ntpd[1583]: Clock offset exceeds panic threshold. Mar 20 15:17:55 srv ntpd[1583]: Set system clock by hand. and also massive 'Limiting icmp unreach' (looks like just after restoring) Mar 20 15:28:33 srv kernel: Limiting icmp unreach response from 419 to 195 packets/sec Mar 20 15:45:02 srv kernel: Limiting icmp unreach response from 407 to 216 packets/sec Mar 20 15:51:33 srv kernel: Limiting icmp unreach response from 392 to 214 packets/sec going to change network card, as not likely this one can be trusted --=20 You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.= From nobody Thu Mar 21 23:19:53 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V11dP2qJgz5F2Y1 for ; Thu, 21 Mar 2024 23:19:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V11dP0xcZz4n53 for ; Thu, 21 Mar 2024 23:19:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711063193; a=rsa-sha256; cv=none; b=BDQfbMJ/kwJTToyk3PKHEVdqIqw6fz/z0ViF9QylQh6L3NoPG0mRDmddFGKr0fJzcZGU6w Ly/4KeflJ99XsQJzxxdcA0+GyDgn6l9jSNwIWRPEA40Gx/11zena+ukqo5INXhl1qJ3lq6 lmyUUxCuOKFXVI8Wr7eUn4r+LTsW4apb6ViN1l2DRrb3U4gx8yo6IrRXOD26dcBdRQRZwY yDruHbeMAH/6QfkEOtKhW2jNlPza+Ey+MIQDQht/POCOgLahAhG3cxYWF4NjyH6jMspq+7 vvcD5YyJVb/sPO/nW3Q9XXow4ey2Ac55DEJkIfgGJpQudOHrMMa//I4OBUs6SQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711063193; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wbLoQ+y17vXbpYb5EqGa1WhGlVQU9sFEojgf0CYUn6g=; b=lBJlYCbdDs/N4vd5grnSnFsRCIand7DERm0XiOgOrQ0o6ImLWIrffUCtJR2btbKJnR6L4g f+5PrNTy2/NTOLcILRID2Z6z+hNfopr62rGgLYInUEzkVQNuqcHVWdB9sEfBUywcyNjkuQ 4Fd+eAS6dfi5UaL7ilzTDtYYZurwVS0pgC4xXH/SPVvIQQbaknkg1VXM+TzdCDWXRiOBFW jPw4pNXBSd2cPtZGeY9JWLFSJ+Q4aJ/Wj/iDznNYV7V2M6oucdwglOKWeLQKjRmX9NKHvU WzXsW6FjnhzAV5faygldWRvvuVs9S09Iv8KyAsc2Z2EuvgJb0IUsHqelE6DWKg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4V11dP0Ys4zvq1 for ; Thu, 21 Mar 2024 23:19:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42LNJqHA007509 for ; Thu, 21 Mar 2024 23:19:52 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42LNJqTD007508 for net@FreeBSD.org; Thu, 21 Mar 2024 23:19:52 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 277875] pfctl cowardly refuses to load rules, broken between 8c94ed992702 & f29af8618bf9 Date: Thu, 21 Mar 2024 23:19:53 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: freebsd@igalic.co X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277875 Mina Gali=C4=87 changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |freebsd@igalic.co Assignee|bugs@FreeBSD.org |net@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Mar 22 02:31:00 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V15sx0sMzz5FQ6X for ; Fri, 22 Mar 2024 02:31:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V15sw45jDz55tJ for ; Fri, 22 Mar 2024 02:31:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711074660; a=rsa-sha256; cv=none; b=xT0MLDw4xR8gV/+zI1T71htGDMSNm2FZCAz8tMm4Xe2zgI6ZM88OCmxL6tUB4Jtf3xt8c+ Vz/LHHjHJ+Q/LcadAlihru+irToBcggBPBaYY9CS1q+GBSXFrkWcDS2VZOI3K1OujGU0ZS ZqWLr/VeH5e4wBS8v+BDSxcIcwEUDwtEWz66W7g0q2Gtzx6/AUiz3GJerpfSlSq1ho//aQ RG9zvQhJ/Uc47psxg2kFQ2lrKw/xybqx+l2c3gPUTxFnIE74laQhBAiS9eOgIK0CeMWnc7 X7Dfzq+B9y92mAIpGMVQa4H5efoIRJs9/x77/L5AwYjVxNW/iuroWCFd8NoE5A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711074660; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mKdVKqS2QFW42HlDDKTvIZ6jozOxesThsNbVwl7AUJI=; b=ivdJ2Ensr/86yNgg+XQv3/yNhHpQ2NLLuUwPrMsvVkjj5c1gTHOULTXCgu6uJRsk1YAv67 TUrdGu2oydceOqcO7bYYc8awgcFE8bm5TPbxYyH0GrBYH6U+B1VMgexafGcaf+5wCZ/A5W 9sv0UTpk7qfIcN/g+ubrF0y7bLxR0TAGYP23jbMrVelFs27EZ0hLIjhZdcgp0ihr/PcDfF M3OKsmyajyHyHc/ooUneYflItOuNMEfwYq80Qdr2P5xWCrNo33fMq+pXKSmV4sCv33lkcu 2Y0lhw735lHST+2t7nUsVHb2+1+/MWYejtPDiuINAvj7hUk0+3pFT32/opu7yQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4V15sw3gy1z10ym for ; Fri, 22 Mar 2024 02:31:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42M2V0tM002611 for ; Fri, 22 Mar 2024 02:31:00 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42M2V0wW002599 for net@FreeBSD.org; Fri, 22 Mar 2024 02:31:00 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 277875] pfctl cowardly refuses to load rules, broken between 8c94ed992702 & f29af8618bf9 Date: Fri, 22 Mar 2024 02:31:00 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kp@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277875 Kristof Provost changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kp@freebsd.org --- Comment #2 from Kristof Provost --- This is interesting. I can load the abridged pf.conf, but it seems to hallucinate ridentifier settings which are not in the file. That may point towards some memory corruption during the parsing, but valgr= ind doesn't immediately seem to detect anything. I see the above behaviour on both amd64 and arm64. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Mar 22 08:37:33 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V1G0t1mzDz5G5Vd for ; Fri, 22 Mar 2024 08:37:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V1G0s6Pqwz4Rf7 for ; Fri, 22 Mar 2024 08:37:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711096653; a=rsa-sha256; cv=none; b=mYYqyOo8ElNRRi7uLjOHDKDgb42mQ05XYzmhfM/nEv+7K1tczhdqzvKJdX5iEVJ+dLs72Y K55fVko0ycwv5donVo5wLErMshblTfvUnb6lvbvP+vcLxW/uVfCPC7yIWOUn9gIiQFXSa5 mpdFMs/oGrSZHyD5YoiJyf74uT6doas23BV1dxkuMdOX4fHHP5iDCbjV5QLnTBbWyb35Vt JU3DcqKYl9YF3yJeES9MnA7Wir9kQBtNSL872DAGOatRnd8IcBQ0wC2veWnRQ86+OySj3C yScFFyvwIqFffkppDcvXJKodwNdyCvA6M0Uhu/M8kcfW7SttMe61G/2Sa5b5+A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711096653; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WvSiklIXwms53HtvpB0T1KbRzE+x2hBi5h+VH+0ep+c=; b=qPn8OiBysY1+z/MpEBOzUPM2L1N0QvNRgiI44zJg94vUn2CxYIxYRIKidjFkvjpo2O/Y+4 JQlY7ApMTOB6E2sE58xLL6GP7fhAAyFfyd+Y3QlV9s8WUmYq0QJV+VO7OBgY4oztEyvoYg X9QxRU8GQrUXKvpU4Ij3I5Riq5ZeuB5ECkF42lM2qxv8lXrmGm0HsRhTwHvN9fzMQWSINB ourKpaqCXxU+nnYRjEZje70Y7J4ItyY7cHV+S8jN/AvuxugAzkz7aBsetiN6jp5a2p3qae Nk4J3CJMc1wQcpnzg9hUEisekuqPzRHU9F2Nu2MPfJSKShIr6vUzQ179TYtYJg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4V1G0s62Qfz1BDw for ; Fri, 22 Mar 2024 08:37:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42M8bXAM023349 for ; Fri, 22 Mar 2024 08:37:33 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42M8bXaJ023347 for net@FreeBSD.org; Fri, 22 Mar 2024 08:37:33 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 277875] pfctl cowardly refuses to load rules, broken between 8c94ed992702 & f29af8618bf9 Date: Fri, 22 Mar 2024 08:37:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277875 --- Comment #3 from commit-hook@FreeBSD.org --- A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3D88f557a2a9c31aaa954841b06d38eec84= a1702fc commit 88f557a2a9c31aaa954841b06d38eec84a1702fc Author: Kristof Provost AuthorDate: 2024-03-22 03:21:50 +0000 Commit: Kristof Provost CommitDate: 2024-03-22 08:00:05 +0000 libpfctl: fix incorrect labels copy We copied the entire parsed_labels struct, including the counter to a field that was only big enough for the labels (so not the counter). PR: 277875 MFC after: 1 week lib/libpfctl/libpfctl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Mar 22 08:37:35 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V1G0w276Gz5G5MT for ; Fri, 22 Mar 2024 08:37:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V1G0v4vb7z4RTN for ; Fri, 22 Mar 2024 08:37:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711096655; a=rsa-sha256; cv=none; b=LYDzbs2caF2nPuBzadhjDn+DE1Rjql4TXxtjy42hP1shR7IrMU3gJP4VKv+mfH1U3Eb9OQ AxVaMl8bUGFqZI14ho1pbTJhpXYiX5tOSpkZR8ih36FGMJLJqLa0Soi19oip8lRn0rEJ6g kHwQyrO6Hken/2HpuHIa2LHTuLnm6tbvVmc5+4IOji00GP6yWG7QJQSB4bnbVaoC+OigIr N0csRnpA+Mlg3GHsay0u2RTUnBgQgdGjrvncSqFxv9pjOaZ3nl61CBbL3+4ApuzncrnGP4 LILEMZn0IGYI08sf+0enwOEP3i1IZHCzUmPLSIZI8LguESrjkU0VBUCW7mbFLg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711096655; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=osXXBB3kw5lA9IBzr8ZKotJqkgeLIZ60Xd0A640mi44=; b=oAUg+SecWitlVF7U0VekQctW2jh6H5sTEBT85wk89F9TwJZ/Epp7esmAEBxocAah1XUpU+ yzYgB8meoIdG5YbJTJ5pSqPBDeH36vavXeV8AE9Imz8pR7OsEllU29ApRCGQbzEaXPlgKK lfIY/ocHPeFs3VcFJQSR8dq2YojTHkttAeSpztrlT8jHeSOC3vMhlqlpchBqYr0Ab7SsS5 dmw3rpChdY4NWzhoW92cY9oZRAXTV73bWkCc3fH1FFuPZWgo1Je5imBxy817OH/ChYa/j0 52hG1xIVNV+JZiHOxKY9G+zKZQnmNNMXVIM47FwC8jq6ipKgP+qQN1gikg6idA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4V1G0v4X9Fz1Bg7 for ; Fri, 22 Mar 2024 08:37:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42M8bZfK023485 for ; Fri, 22 Mar 2024 08:37:35 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42M8bZ3o023484 for net@FreeBSD.org; Fri, 22 Mar 2024 08:37:35 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 276838] ovpn(4) - problems with large TCP segments over IPv6 tunnel when DCO module is used at both ends Date: Fri, 22 Mar 2024 08:37:35 +0000 X-Bugzilla-Reason: AssignedTo CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276838 --- Comment #5 from commit-hook@FreeBSD.org --- A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3De08b44339b65a6aa7df4a58d0b1f471ba= 16da410 commit e08b44339b65a6aa7df4a58d0b1f471ba16da410 Author: Kristof Provost AuthorDate: 2024-03-21 07:38:45 +0000 Commit: Kristof Provost CommitDate: 2024-03-22 08:00:05 +0000 if_ovpn tests: test large packets in IPv6 tunnel There's a report of MTU issues over IPv6 DCO tunnels. Extend the 4in6 test to send a series of pings with different sizes, as well as transfer a large file. No issues were found, but we may as well extend the test case. PR: 276838 tests/sys/net/if_ovpn/if_ovpn.sh | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) --=20 You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.= From nobody Fri Mar 22 09:05:12 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V1Gd62VjDz5G8HT for ; Fri, 22 Mar 2024 09:05:30 +0000 (UTC) (envelope-from benoitc@enki-multimedia.eu) Received: from mail-4022.proton.ch (mail-4022.proton.ch [185.70.40.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V1Gd44hynz4Vr9 for ; Fri, 22 Mar 2024 09:05:28 +0000 (UTC) (envelope-from benoitc@enki-multimedia.eu) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=enki-multimedia.eu header.s=protonmail2 header.b=kgIzDaPU; dmarc=pass (policy=none) header.from=enki-multimedia.eu; spf=pass (mx1.freebsd.org: domain of benoitc@enki-multimedia.eu designates 185.70.40.22 as permitted sender) smtp.mailfrom=benoitc@enki-multimedia.eu DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enki-multimedia.eu; s=protonmail2; t=1711098324; x=1711357524; bh=cBjHVPdhiHRbITarKkKtEjwVlWwyebWMN+jRxzXwpN8=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=kgIzDaPUDvgGe+wlgGTh6gqrSfan2mwoMmcKl19/QYyrFGgLRlQuUH4bJFmWSHXG8 9pCpmKXytLe1FTcc3FMmB7G/TChaHn4F+/MGY8XT/vnjpvfSDRTTPf0jYB1p531kE7 ztmuC8p0hUd1an9Q1paTKi/i70yKfm/k2V4SuqBsOKCtdL3nlfWB9SdAT0/nYnblJH Ji+bt/Bi11l23vFNKF+G4PGlLEViph/vlTdRzUMFLGnFf/6L9ymTqBHFLGFiC6xrjX 6p8E2Fh/HSY3vUJyODL1Hc07wPFPtkHYD/3npRxkqn2Zh7OEF5q37tDEmDqghI3PT2 clhndvPeYii6A== Date: Fri, 22 Mar 2024 09:05:12 +0000 To: Zhenlei Huang From: Benoit Chesneau Cc: Marek Zarychta , FreeBSD Net Subject: Re: ipv4 route with ipv6 local link nexthop ? Message-ID: <_dTkes6xnAAQDFSPOPVSOT7GfMjJzaajrkovO456NI6vBSNn7afe4S3p9UCvw35j_sMnvF_CYIDtmT5JcKm7nHhTSVV7VeW3NfdZCR2iTcA=@enki-multimedia.eu> In-Reply-To: <1A5D703E-37B1-407C-8B72-0F4B62DC4219@FreeBSD.org> References: <367504DC-48DA-4DFD-9DB6-CC571F0D26B8@FreeBSD.org> <764E12BF-5D31-4905-98AE-6D745BFD1DC2@FreeBSD.org> <4380f799-b961-4daf-8514-679c06214d55@plan-b.pwste.edu.pl> <24620735-923d-4603-8c92-1d9b23d3ce80@plan-b.pwste.edu.pl> <323D6B49-EC5C-4011-8BBA-1EAB9DFC4BC2@FreeBSD.org> <1A5D703E-37B1-407C-8B72-0F4B62DC4219@FreeBSD.org> Feedback-ID: 9066678:user:proton List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="b1_LE2649YWVsmFw4KGLN3gQqtInOkXHQEyfYdVKyNtzc" X-Spamd-Bar: - X-Spamd-Result: default: False [-1.09 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; AUTOGEN_PHP_SPAMMY(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; URI_COUNT_ODD(1.00)[7]; NEURAL_HAM_SHORT(-0.99)[-0.988]; DMARC_POLICY_ALLOW(-0.50)[enki-multimedia.eu,none]; RWL_MAILSPIKE_VERYGOOD(-0.20)[185.70.40.22:from]; R_DKIM_ALLOW(-0.20)[enki-multimedia.eu:s=protonmail2]; R_SPF_ALLOW(-0.20)[+ip4:185.70.40.0/24]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MIME_BASE64_TEXT(0.10)[]; TO_DN_ALL(0.00)[]; HAS_PHPMAILER_SIG(0.00)[]; DKIM_TRACE(0.00)[enki-multimedia.eu:+]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:62371, ipnet:185.70.40.0/24, country:CH]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MISSING_XM_UA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; RCVD_COUNT_ZERO(0.00)[0]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~] X-Rspamd-Queue-Id: 4V1Gd44hynz4Vr9 This is a multi-part message in MIME format. --b1_LE2649YWVsmFw4KGLN3gQqtInOkXHQEyfYdVKyNtzc Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 QXdlc29tZSEgRG8gd2UgaGF2ZSBhIGNoYW5jZSBpdCBsYW5kIGluIGEgcGF0Y2ggcmVsZWFzZSBz b29uID8gT3IgYmV0dGVyIHRvIHVzZSBhIFNUQUJMRSB1bnRpbCB0aGUgMTQuMSBpcyByZWxlYXNl ZD8KCkJlbm/DrnQKT24gVGh1cnNkYXksIE1hcmNoIDE0dGgsIDIwMjQgYXQgMTA6NTYsIFpoZW5s ZWkgSHVhbmcgPHpsZWlARnJlZUJTRC5vcmc+IHdyb3RlOgoKPj4gT24gTWFyIDE0LCAyMDI0LCBh dCA5OjA0IEFNLCBaaGVubGVpIEh1YW5nIDx6bGVpQEZyZWVCU0Qub3JnPiB3cm90ZToKPj4KPj4+ IE9uIE1hciAxNCwgMjAyNCwgYXQgMzowNyBBTSwgTWFyZWsgWmFyeWNodGEgPHphcnljaHRhbUBw bGFuLWIucHdzdGUuZWR1LnBsPiB3cm90ZToKPj4+Cj4+PiBXIGRuaXUgMTMuMDMuMjAyNCBvIDE4 OjU5LCBNYXJlayBaYXJ5Y2h0YSBwaXN6ZToKPj4+Cj4+Pj4gVyBkbml1IDEzLjAzLjIwMjQgbyAx NjozMSwgQmVub2l0IENoZXNuZWF1IHBpc3plOgo+Pj4+Cj4+Pj4+IEhybSBJIHRob3VnaHQgaXQg d2FzIGltcGxlbWVudGVkIHZpYWh0dHBzOi8vcmV2aWV3cy5mcmVlYnNkLm9yZy9yRzYyZTFhNDM3 ZjMyODVlNzg1ZDliMzVhNDc2ZDM2YTQ2OWE5MDAyOGQKPj4+Pj4KPj4+Pj4gV2Fzbid0IGl0IG1l cmdlZCA/IChhbHNvIHByZXR0eSBzdXJlIEkgZGlkIHRlc3QgaXQgaW4gZnJlZWJzZCAxMykuCj4+ Pj4KPj4+PiBGV0lXOiBpdCB3b3JrcyBmaW5lIG9uIENVUlJFTlQKPj4+Pgo+Pj4+ICMgaWZjb25m aWcgdmxhbjggZGVzdHJveQo+Pj4+ICMgaWZjb25maWcgdmxhbjggY3JlYXRlIHZsYW5kZXYgYmdl MCB2bGFuIDggdXAKPj4+PiAjIGlmY29uZmlnIHZsYW44IGluZXQ2IC1pZmRpc2FibGVkIGF1dG9f bGlua2xvY2FsCj4+Pj4gIyByb3V0ZSBhZGQgLW5ldCAxMC4xMS4xMy4wLzI0IC1pbmV0NiBmZTgw OjozNjBhOjExZmY6ZmUxYjo0MDRlJXZsYW44Cj4+Pj4gYWRkIG5ldCAxMC4xMS4xMy4wOiBnYXRl d2F5IGZlODA6OjM2MGE6MTFmZjpmZTFiOjQwNGUldmxhbjggZmliIDAKPj4+Cj4+PiBJdCBsb29r cyBsaWtlIHRoZSBmaXggaXMgaW4gZjgxODU1OTc3NGNiMGMxNTE2MzY0YzRiZWNhMzYxNDgwZmQ2 OGI1YiAuIFpoZW5sZWksIGNvdWxkIHlvdSBwbGVhc2UgTUZDIHRoaXMgb25lWzFdID8KPj4+Cj4+ PiBDaGVycnktcGlja2luZyBpdCB0byBzdGFibGUvMTQgbWFrZXMgcm91dGUgZnVsbHkgZnVuY3Rp b25hbC4gSSBoYXZlIHRlc3RlZCBpdCBiZXR3ZWVuIHN0YWJsZS8xNCB3aXRoIHRoaXMgZml4IGFw cGxpZWQgYW5kIENVUlJFTlQuCj4+Cj4+IFRoYW5rcyBmb3IgZmluZGluZyB0aGUgZml4IGFuZCB0 aGUgY29uZmlybWF0aW9uICwgSSdsbCB0YWtlIGNhcmUgb2YgdGhhdCA6KQo+Cj4gRG9uZS4gVGhl IGZpeCBhbmQgdGVzdHMgYXJlIGFsbCBNRkNlZCBpbnRvIHN0YWJsZS8xNCBicmFuY2guCj4KPj4+ IEhvc3QgQToKPj4+Cj4+PiAjIGlmY29uZmlnIGxvMTAgZGVzdHJveQo+Pj4gIyBpZmNvbmZpZyBs bzEwIGNyZWF0ZQo+Pj4gIyBpZmNvbmZpZyBsbzEwIDEwLjExLjEzLjEvMjQKPj4+ICMgaWZjb25m aWcgdmxhbjggZGVzdHJveQo+Pj4gIyBpZmNvbmZpZyB2bGFuOCBjcmVhdGUgdmxhbmRldiBiZ2Uw IHZsYW4gOCB1cAo+Pj4gIyBpZmNvbmZpZyB2bGFuOCBpbmV0NiAtaWZkaXNhYmxlZCBhdXRvX2xp bmtsb2NhbAo+Pj4gIyByb3V0ZSBhZGQgLW5ldCAxMC4xMS4xMi4wLzI0IC1pbmV0NiBmZTgwOjo2 YWI1Ojk5ZmY6ZmViZDo4MTA4JXZsYW44Cj4+PiBhZGQgbmV0IDEwLjExLjEyLjA6IGdhdGV3YXkg ZmU4MDo6NmFiNTo5OWZmOmZlYmQ6ODEwOCV2bGFuOCBmaWIgMAo+Pj4gIyBwaW5nIC1jNSAtUyAx MC4xMS4xMy4xIDEwLjExLjEyLjEKPj4+IFBJTkcgMTAuMTEuMTIuMSAoMTAuMTEuMTIuMSkgZnJv bSAxMC4xMS4xMy4xOiA1NiBkYXRhIGJ5dGVzCj4+PiA2NCBieXRlcyBmcm9tIDEwLjExLjEyLjE6 IGljbXBfc2VxPTAgdHRsPTY0IHRpbWU9MjAwMi4zMDMgbXMKPj4+IDY0IGJ5dGVzIGZyb20gMTAu MTEuMTIuMTogaWNtcF9zZXE9MSB0dGw9NjQgdGltZT0xMDAwLjQ2MSBtcwo+Pj4gNjQgYnl0ZXMg ZnJvbSAxMC4xMS4xMi4xOiBpY21wX3NlcT0yIHR0bD02NCB0aW1lPTAuMTY3IG1zCj4+PiA2NCBi eXRlcyBmcm9tIDEwLjExLjEyLjE6IGljbXBfc2VxPTMgdHRsPTY0IHRpbWU9MC4yMjIgbXMKPj4+ IDY0IGJ5dGVzIGZyb20gMTAuMTEuMTIuMTogaWNtcF9zZXE9NCB0dGw9NjQgdGltZT0wLjIwNyBt cwo+Pj4KPj4+IC0tLSAxMC4xMS4xMi4xIHBpbmcgc3RhdGlzdGljcyAtLS0KPj4+IDUgcGFja2V0 cyB0cmFuc21pdHRlZCwgNSBwYWNrZXRzIHJlY2VpdmVkLCAwLjAlIHBhY2tldCBsb3NzCj4+PiBy b3VuZC10cmlwIG1pbi9hdmcvbWF4L3N0ZGRldiA9IDAuMTY3LzYwMC42NzIvMjAwMi4zMDMvODAw Ljc2MyBtcwo+Pj4KPj4+IEhvc3QgQjoKPj4+Cj4+PiAjIGlmY29uZmlnIGxvMTAgZGVzdHJveQo+ Pj4gIyBpZmNvbmZpZyBsbzEwIGNyZWF0ZQo+Pj4gIyBpZmNvbmZpZyBsbzEwIDEwLjExLjEyLjEv MjQKPj4+ICMgaWZjb25maWcgdmxhbjggZGVzdHJveQo+Pj4gaWZjb25maWc6IGludGVyZmFjZSB2 bGFuOCBkb2VzIG5vdCBleGlzdAo+Pj4gIyBpZmNvbmZpZyB2bGFuOCBjcmVhdGUgdmxhbmRldiBi Y2UwIHZsYW4gOCB1cAo+Pj4gIyBpZmNvbmZpZyB2bGFuOCBpbmV0NiAtaWZkaXNhYmxlZCBhdXRv X2xpbmtsb2NhbAo+Pj4gIyByb3V0ZSBhZGQgLW5ldCAxMC4xMS4xMy4wLzI0IC1pbmV0NiBmZTgw OjoyNmJlOjVmZjpmZTEwOmM5MDAldmxhbjgKPj4+IGFkZCBuZXQgMTAuMTEuMTMuMDogZ2F0ZXdh eSBmZTgwOjoyNmJlOjVmZjpmZTEwOmM5MDAldmxhbjggZmliIDAKPj4+ICMgcGluZyAtYzUgLVMg MTAuMTEuMTIuMSAxMC4xMS4xMy4xCj4+PiBQSU5HIDEwLjExLjEzLjEgKDEwLjExLjEzLjEpIGZy b20gMTAuMTEuMTIuMTogNTYgZGF0YSBieXRlcwo+Pj4gNjQgYnl0ZXMgZnJvbSAxMC4xMS4xMy4x OiBpY21wX3NlcT0wIHR0bD02NCB0aW1lPTEwMDAuMjg1IG1zCj4+PiA2NCBieXRlcyBmcm9tIDEw LjExLjEzLjE6IGljbXBfc2VxPTEgdHRsPTY0IHRpbWU9MC4xNDEgbXMKPj4+IDY0IGJ5dGVzIGZy b20gMTAuMTEuMTMuMTogaWNtcF9zZXE9MiB0dGw9NjQgdGltZT0wLjIzMSBtcwo+Pj4gNjQgYnl0 ZXMgZnJvbSAxMC4xMS4xMy4xOiBpY21wX3NlcT0zIHR0bD02NCB0aW1lPTAuMjM1IG1zCj4+PiA2 NCBieXRlcyBmcm9tIDEwLjExLjEzLjE6IGljbXBfc2VxPTQgdHRsPTY0IHRpbWU9MC4xNzQgbXMK Pj4+Cj4+PiAtLS0gMTAuMTEuMTMuMSBwaW5nIHN0YXRpc3RpY3MgLS0tCj4+PiA1IHBhY2tldHMg dHJhbnNtaXR0ZWQsIDUgcGFja2V0cyByZWNlaXZlZCwgMC4wJSBwYWNrZXQgbG9zcwo+Pj4gcm91 bmQtdHJpcCBtaW4vYXZnL21heC9zdGRkZXYgPSAwLjE0MS8yMDAuMjEzLzEwMDAuMjg1LzQwMC4w MzYgbXMKPj4+Cj4+PiAxLmh0dHBzOi8vY2dpdC5mcmVlYnNkLm9yZy9zcmMvY29tbWl0Lz9pZD1m ODE4NTU5Nzc0Y2IwYzE1MTYzNjRjNGJlY2EzNjE0ODBmZDY4YjViCj4+Pgo+Pj4gQ2hlZXJzCj4+ Pgo+Pj4gLS0KPj4+IE1hcmVrIFphcnljaHRh --b1_LE2649YWVsmFw4KGLN3gQqtInOkXHQEyfYdVKyNtzc Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDE0 cHg7Ij5Bd2Vzb21lISBEbyB3ZSBoYXZlIGEgY2hhbmNlIGl0IGxhbmQgaW4gYSBwYXRjaCByZWxl YXNlIHNvb24gPyBPciBiZXR0ZXIgdG8gdXNlIGEgU1RBQkxFIHVudGlsIHRoZSAxNC4xIGlzIHJl bGVhc2VkPyZuYnNwOzwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1z ZXJpZjsgZm9udC1zaXplOiAxNHB4OyI+PGJyPjwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5 OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4OyI+PGJyPjwvZGl2Pg0KPGRpdiBj bGFzcz0icHJvdG9ubWFpbF9zaWduYXR1cmVfYmxvY2siIHN0eWxlPSJmb250LWZhbWlseTogQXJp YWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweDsiPg0KICAgIDxkaXYgY2xhc3M9InByb3Rv bm1haWxfc2lnbmF0dXJlX2Jsb2NrLXVzZXIiPg0KICAgICAgICA8ZGl2IHN0eWxlPSJmb250LXN0 eWxlOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiBub3JtYWw7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7 IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9y bWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyBmb250LWZhbWls eTogSGVsdmV0aWNhOyBmb250LXNpemU6IDEycHg7IGNvbG9yOiByZ2IoMCwgMCwgMCk7Ij5CZW5v w650Jm5ic3A7PC9kaXY+PC9kaXY+PC9kaXY+PGRpdiBjbGFzcz0icHJvdG9ubWFpbF9xdW90ZSI+ DQogICAgICAgIE9uIFRodXJzZGF5LCBNYXJjaCAxNHRoLCAyMDI0IGF0IDEwOjU2LCBaaGVubGVp IEh1YW5nICZsdDt6bGVpQEZyZWVCU0Qub3JnJmd0OyB3cm90ZTo8YnI+DQogICAgICAgIDxibG9j a3F1b3RlIGNsYXNzPSJwcm90b25tYWlsX3F1b3RlIiB0eXBlPSJjaXRlIj4NCiAgICAgICAgICAg IDxiciBjbGFzcz0iIj48ZGl2PjxiciBjbGFzcz0iIj48YmxvY2txdW90ZSBjbGFzcz0iIiB0eXBl PSJjaXRlIj48ZGl2IGNsYXNzPSIiPk9uIE1hciAxNCwgMjAyNCwgYXQgOTowNCBBTSwgWmhlbmxl aSBIdWFuZyAmbHQ7PGEgY2xhc3M9IiIgaHJlZj0ibWFpbHRvOnpsZWlARnJlZUJTRC5vcmciIHJl bD0ibm9yZWZlcnJlciBub2ZvbGxvdyBub29wZW5lciIgdGFyZ2V0PSJfYmxhbmsiPnpsZWlARnJl ZUJTRC5vcmc8L2E+Jmd0OyB3cm90ZTo8L2Rpdj48YnIgY2xhc3M9IkFwcGxlLWludGVyY2hhbmdl LW5ld2xpbmUiPjxkaXYgY2xhc3M9IiI+PGJyIGNsYXNzPSJBcHBsZS1pbnRlcmNoYW5nZS1uZXds aW5lIj48YnIgc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBI ZWx2ZXRpY2E7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlh bnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFs OyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5v bmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQt c3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIiPjxibG9j a3F1b3RlIHN0eWxlPSJmb250LWZhbWlseTogSGVsdmV0aWNhOyBmb250LXNpemU6IDEzcHg7IGZv bnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6 IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgb3JwaGFuczogYXV0bzsgdGV4dC1hbGlnbjog c3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFj ZTogbm9ybWFsOyB3aWRvd3M6IGF1dG87IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQt c2l6ZS1hZGp1c3Q6IGF1dG87IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1k ZWNvcmF0aW9uOiBub25lOyIgY2xhc3M9IiIgdHlwZT0iY2l0ZSI+PGRpdiBjbGFzcz0iIj5PbiBN YXIgMTQsIDIwMjQsIGF0IDM6MDcgQU0sIE1hcmVrIFphcnljaHRhICZsdDs8YSBjbGFzcz0iIiBo cmVmPSJtYWlsdG86emFyeWNodGFtQHBsYW4tYi5wd3N0ZS5lZHUucGwiIHJlbD0ibm9yZWZlcnJl ciBub2ZvbGxvdyBub29wZW5lciIgdGFyZ2V0PSJfYmxhbmsiPnphcnljaHRhbUBwbGFuLWIucHdz dGUuZWR1LnBsPC9hPiZndDsgd3JvdGU6PC9kaXY+PGJyIGNsYXNzPSJBcHBsZS1pbnRlcmNoYW5n ZS1uZXdsaW5lIj48ZGl2IGNsYXNzPSIiPjxzcGFuIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAs IDAsIDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250 LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0 MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVu dDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1z cGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0 aW9uOiBub25lOyBmbG9hdDogbm9uZTsgZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7IiBjbGFz cz0iIj5XIGRuaXUgMTMuMDMuMjAyNCBvJm5ic3A7MTg6NTksIE1hcmVrIFphcnljaHRhIHBpc3pl Ojwvc3Bhbj48YnIgc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5 OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9u dC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6 IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNm b3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtp dC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IiBjbGFzcz0i Ij48YmxvY2txdW90ZSBzdHlsZT0iZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6 ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBm b250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFy dDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBu b3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7 IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIiIHR5cGU9ImNpdGUiPlcgZG5pdSAxMy4w My4yMDI0IG8mbmJzcDsxNjozMSwgQmVub2l0IENoZXNuZWF1IHBpc3plOjxiciBjbGFzcz0iIj48 YmxvY2txdW90ZSBjbGFzcz0iIiB0eXBlPSJjaXRlIj5Icm0gSSB0aG91Z2h0IGl0IHdhcyBpbXBs ZW1lbnRlZCB2aWE8c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3Nw YW4+PGEgY2xhc3M9IiIgaHJlZj0iaHR0cHM6Ly9yZXZpZXdzLmZyZWVic2Qub3JnL3JHNjJlMWE0 MzdmMzI4NWU3ODVkOWIzNWE0NzZkMzZhNDY5YTkwMDI4ZCIgcmVsPSJub3JlZmVycmVyIG5vZm9s bG93IG5vb3BlbmVyIiB0YXJnZXQ9Il9ibGFuayI+aHR0cHM6Ly9yZXZpZXdzLmZyZWVic2Qub3Jn L3JHNjJlMWE0MzdmMzI4NWU3ODVkOWIzNWE0NzZkMzZhNDY5YTkwMDI4ZDwvYT48YnIgY2xhc3M9 IiI+PGJyIGNsYXNzPSIiPldhc24ndCBpdCBtZXJnZWQgPyAoYWxzbyBwcmV0dHkgc3VyZSBJIGRp ZCB0ZXN0IGl0IGluIGZyZWVic2QgMTMpLjxiciBjbGFzcz0iIj48YnIgY2xhc3M9IiI+PC9ibG9j a3F1b3RlPkZXSVc6IGl0IHdvcmtzIGZpbmUgb24gQ1VSUkVOVDxiciBjbGFzcz0iIj48YnIgY2xh c3M9IiI+IyBpZmNvbmZpZyB2bGFuOCBkZXN0cm95PGJyIGNsYXNzPSIiPiMgaWZjb25maWcgdmxh bjggY3JlYXRlIHZsYW5kZXYgYmdlMCB2bGFuIDggdXA8YnIgY2xhc3M9IiI+IyBpZmNvbmZpZyB2 bGFuOCBpbmV0NiAtaWZkaXNhYmxlZCBhdXRvX2xpbmtsb2NhbDxiciBjbGFzcz0iIj4jIHJvdXRl IGFkZCAtbmV0IDEwLjExLjEzLjAvMjQgLWluZXQ2IGZlODA6OjM2MGE6MTFmZjpmZTFiOjQwNGUl dmxhbjg8YnIgY2xhc3M9IiI+YWRkIG5ldCAxMC4xMS4xMy4wOiBnYXRld2F5IGZlODA6OjM2MGE6 MTFmZjpmZTFiOjQwNGUldmxhbjggZmliIDA8YnIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPjwvYmxv Y2txdW90ZT48c3BhbiBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1p bHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBm b250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2lu Zzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFu c2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Vi a2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsgZmxvYXQ6 IG5vbmU7IGRpc3BsYXk6IGlubGluZSAhaW1wb3J0YW50OyIgY2xhc3M9IiI+SXQgbG9va3MgbGlr ZSB0aGUgZml4IGlzIGluIGY4MTg1NTk3NzRjYjBjMTUxNjM2NGM0YmVjYTM2MTQ4MGZkNjhiNWIg LiBaaGVubGVpLCBjb3VsZCB5b3UgcGxlYXNlIE1GQyB0aGlzIG9uZVsxXSA/PC9zcGFuPjxiciBz dHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3Vs YXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fw czogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0 LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdo aXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tl LXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIiPjxiciBzdHlsZT0i Y2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZv bnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9y bWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWdu OiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNw YWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRo OiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIiPjxzcGFuIHN0eWxlPSJjYXJl dC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1z aXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7 IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0 YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6 IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBw eDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyBmbG9hdDogbm9uZTsgZGlzcGxheTogaW5saW5lICFp bXBvcnRhbnQ7IiBjbGFzcz0iIj5DaGVycnktcGlja2luZyBpdCB0byBzdGFibGUvMTQgbWFrZXMg cm91dGUgZnVsbHkgZnVuY3Rpb25hbC4gSSBoYXZlIHRlc3RlZCBpdCBiZXR3ZWVuIHN0YWJsZS8x NCB3aXRoIHRoaXMgZml4IGFwcGxpZWQgYW5kIENVUlJFTlQuPC9zcGFuPjxiciBzdHlsZT0iY2Fy ZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQt c2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFs OyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBz dGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNl OiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAw cHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIiPjwvZGl2PjwvYmxvY2txdW90ZT48 ZGl2IGNsYXNzPSIiIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWls eTogSGVsdmV0aWNhOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12 YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5v cm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3Jt OiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10 ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7Ij48YnIgY2xhc3M9 IiI+PC9kaXY+PGRpdiBjbGFzcz0iIiBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsg Zm9udC1mYW1pbHk6IEhlbHZldGljYTsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3Jt YWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1z cGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0 LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7 IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyI+ VGhhbmtzIGZvciBmaW5kaW5nIHRoZSBmaXggYW5kIHRoZSBjb25maXJtYXRpb24gLCBJJ2xsIHRh a2UgY2FyZSBvZiB0aGF0IDopPC9kaXY+PC9kaXY+PC9ibG9ja3F1b3RlPjxkaXY+PGJyIGNsYXNz PSIiPjwvZGl2PjxkaXY+RG9uZS4gVGhlIGZpeCBhbmQgdGVzdHMgYXJlIGFsbCBNRkNlZCBpbnRv IHN0YWJsZS8xNCBicmFuY2guPC9kaXY+PGJyIGNsYXNzPSIiPjxibG9ja3F1b3RlIGNsYXNzPSIi IHR5cGU9ImNpdGUiPjxkaXYgY2xhc3M9IiI+PGJyIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAs IDAsIDApOyBmb250LWZhbWlseTogSGVsdmV0aWNhOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5 bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsg bGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAw cHg7IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNp bmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246 IG5vbmU7IiBjbGFzcz0iIj48YmxvY2txdW90ZSBzdHlsZT0iZm9udC1mYW1pbHk6IEhlbHZldGlj YTsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBz OiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IG9ycGhh bnM6IGF1dG87IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5z Zm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd2lkb3dzOiBhdXRvOyB3b3JkLXNwYWNp bmc6IDBweDsgLXdlYmtpdC10ZXh0LXNpemUtYWRqdXN0OiBhdXRvOyAtd2Via2l0LXRleHQtc3Ry b2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIiIHR5cGU9ImNp dGUiPjxkaXYgY2xhc3M9IiI+PGJyIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBm b250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBu b3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRl ci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0 ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAw cHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25l OyIgY2xhc3M9IiI+PHNwYW4gc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQt ZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1h bDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNw YWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQt dHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBweDsg LXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IGZs b2F0OiBub25lOyBkaXNwbGF5OiBpbmxpbmUgIWltcG9ydGFudDsiIGNsYXNzPSIiPkhvc3QgQTo8 L3NwYW4+PGJyIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTog TWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQt dmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBu b3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9y bTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQt dGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyIgY2xhc3M9IiI+ PGJyIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8t UmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFu dC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7 IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9u ZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1z dHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyIgY2xhc3M9IiI+PHNwYW4g c3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBNZW5sby1SZWd1 bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNh cHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4 dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25lOyB3 aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9r ZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IGZsb2F0OiBub25lOyBkaXNwbGF5 OiBpbmxpbmUgIWltcG9ydGFudDsiIGNsYXNzPSIiPiMgaWZjb25maWcgbG8xMCBkZXN0cm95PC9z cGFuPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1l bmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZh cmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9y bWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06 IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRl eHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIiPjxz cGFuIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8t UmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFu dC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7 IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9u ZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1z dHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyBmbG9hdDogbm9uZTsgZGlz cGxheTogaW5saW5lICFpbXBvcnRhbnQ7IiBjbGFzcz0iIj4jIGlmY29uZmlnIGxvMTAgY3JlYXRl PC9zcGFuPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6 IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250 LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzog bm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zv cm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0 LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIi PjxzcGFuIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVu bG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFy aWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3Jt YWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTog bm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4 dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyBmbG9hdDogbm9uZTsg ZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7IiBjbGFzcz0iIj4jIGlmY29uZmlnIGxvMTAgMTAu MTEuMTMuMS8yNDwvc3Bhbj48YnIgc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZv bnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5v cm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVy LXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRl eHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBw eDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7 IiBjbGFzcz0iIj48c3BhbiBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1m YW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFs OyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3Bh Y2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10 cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAt d2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsgZmxv YXQ6IG5vbmU7IGRpc3BsYXk6IGlubGluZSAhaW1wb3J0YW50OyIgY2xhc3M9IiI+IyBpZmNvbmZp ZyB2bGFuOCBkZXN0cm95PC9zcGFuPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAw KTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHls ZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBs ZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBw eDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2lu ZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjog bm9uZTsiIGNsYXNzPSIiPjxzcGFuIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBm b250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBu b3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRl ci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0 ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAw cHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25l OyBmbG9hdDogbm9uZTsgZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7IiBjbGFzcz0iIj4jIGlm Y29uZmlnIHZsYW44IGNyZWF0ZSB2bGFuZGV2IGJnZTAgdmxhbiA4IHVwPC9zcGFuPjxiciBzdHls ZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7 IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczog bm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFs aWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRl LXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdp ZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIiPjxzcGFuIHN0eWxlPSJj YXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9u dC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3Jt YWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246 IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3Bh Y2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6 IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyBmbG9hdDogbm9uZTsgZGlzcGxheTogaW5saW5l ICFpbXBvcnRhbnQ7IiBjbGFzcz0iIj4jIGlmY29uZmlnIHZsYW44IGluZXQ2IC1pZmRpc2FibGVk IGF1dG9fbGlua2xvY2FsPC9zcGFuPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAw KTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHls ZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBs ZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBw eDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2lu ZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjog bm9uZTsiIGNsYXNzPSIiPjxzcGFuIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBm b250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBu b3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRl ci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0 ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAw cHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25l OyBmbG9hdDogbm9uZTsgZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7IiBjbGFzcz0iIj4jIHJv dXRlIGFkZCAtbmV0IDEwLjExLjEyLjAvMjQgLWluZXQ2IGZlODA6OjZhYjU6OTlmZjpmZWJkOjgx MDgldmxhbjg8L3NwYW4+PGJyIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250 LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3Jt YWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1z cGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0 LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7 IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyIg Y2xhc3M9IiI+PHNwYW4gc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFt aWx5OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsg Zm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNp bmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJh bnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdl YmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IGZsb2F0 OiBub25lOyBkaXNwbGF5OiBpbmxpbmUgIWltcG9ydGFudDsiIGNsYXNzPSIiPmFkZCBuZXQgMTAu MTEuMTIuMDogZ2F0ZXdheSBmZTgwOjo2YWI1Ojk5ZmY6ZmViZDo4MTA4JXZsYW44IGZpYiAwPC9z cGFuPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1l bmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZh cmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9y bWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06 IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRl eHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIiPjxz cGFuIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8t UmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFu dC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7 IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9u ZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1z dHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyBmbG9hdDogbm9uZTsgZGlz cGxheTogaW5saW5lICFpbXBvcnRhbnQ7IiBjbGFzcz0iIj4jIHBpbmcgLWM1IC1TIDEwLjExLjEz LjEgMTAuMTEuMTIuMTwvc3Bhbj48YnIgc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7 IGZvbnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6 IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0 dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7 IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6 IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5v bmU7IiBjbGFzcz0iIj48c3BhbiBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9u dC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9y bWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXIt c3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4 dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4 OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsg ZmxvYXQ6IG5vbmU7IGRpc3BsYXk6IGlubGluZSAhaW1wb3J0YW50OyIgY2xhc3M9IiI+UElORyAx MC4xMS4xMi4xICgxMC4xMS4xMi4xKSBmcm9tIDEwLjExLjEzLjE6IDU2IGRhdGEgYnl0ZXM8L3Nw YW4+PGJyIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVu bG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFy aWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3Jt YWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTog bm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4 dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyIgY2xhc3M9IiI+PHNw YW4gc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBNZW5sby1S ZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50 LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsg dGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25l OyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0 cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IGZsb2F0OiBub25lOyBkaXNw bGF5OiBpbmxpbmUgIWltcG9ydGFudDsiIGNsYXNzPSIiPjY0IGJ5dGVzIGZyb20gMTAuMTEuMTIu MTogaWNtcF9zZXE9MCB0dGw9NjQgdGltZT0yMDAyLjMwMyBtczwvc3Bhbj48YnIgc3R5bGU9ImNh cmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250 LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1h bDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjog c3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFj ZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDog MHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IiBjbGFzcz0iIj48c3BhbiBzdHlsZT0iY2FyZXQt Y29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6 ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBm b250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFy dDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBu b3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7 IHRleHQtZGVjb3JhdGlvbjogbm9uZTsgZmxvYXQ6IG5vbmU7IGRpc3BsYXk6IGlubGluZSAhaW1w b3J0YW50OyIgY2xhc3M9IiI+NjQgYnl0ZXMgZnJvbSAxMC4xMS4xMi4xOiBpY21wX3NlcT0xIHR0 bD02NCB0aW1lPTEwMDAuNDYxIG1zPC9zcGFuPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigw LCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9u dC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDog NDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRl bnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQt c3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3Jh dGlvbjogbm9uZTsiIGNsYXNzPSIiPjxzcGFuIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAs IDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0 eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7 IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDog MHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFj aW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9u OiBub25lOyBmbG9hdDogbm9uZTsgZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7IiBjbGFzcz0i Ij42NCBieXRlcyBmcm9tIDEwLjExLjEyLjE6IGljbXBfc2VxPTIgdHRsPTY0IHRpbWU9MC4xNjcg bXM8L3NwYW4+PGJyIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWls eTogTWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZv bnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5n OiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5z Zm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJr aXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyIgY2xhc3M9 IiI+PHNwYW4gc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBN ZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12 YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5v cm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3Jt OiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10 ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IGZsb2F0OiBub25l OyBkaXNwbGF5OiBpbmxpbmUgIWltcG9ydGFudDsiIGNsYXNzPSIiPjY0IGJ5dGVzIGZyb20gMTAu MTEuMTIuMTogaWNtcF9zZXE9MyB0dGw9NjQgdGltZT0wLjIyMiBtczwvc3Bhbj48YnIgc3R5bGU9 ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBm b250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5v cm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGln bjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1z cGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0 aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IiBjbGFzcz0iIj48c3BhbiBzdHlsZT0iY2Fy ZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQt c2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFs OyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBz dGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNl OiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAw cHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsgZmxvYXQ6IG5vbmU7IGRpc3BsYXk6IGlubGluZSAh aW1wb3J0YW50OyIgY2xhc3M9IiI+NjQgYnl0ZXMgZnJvbSAxMC4xMS4xMi4xOiBpY21wX3NlcT00 IHR0bD02NCB0aW1lPTAuMjA3IG1zPC9zcGFuPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigw LCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9u dC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDog NDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRl bnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQt c3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3Jh dGlvbjogbm9uZTsiIGNsYXNzPSIiPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAw KTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHls ZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBs ZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBw eDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2lu ZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjog bm9uZTsiIGNsYXNzPSIiPjxzcGFuIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBm b250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBu b3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRl ci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0 ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAw cHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25l OyBmbG9hdDogbm9uZTsgZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7IiBjbGFzcz0iIj4tLS0g MTAuMTEuMTIuMSBwaW5nIHN0YXRpc3RpY3MgLS0tPC9zcGFuPjxiciBzdHlsZT0iY2FyZXQtY29s b3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTog MTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250 LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsg dGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3Jt YWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRl eHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIiPjxzcGFuIHN0eWxlPSJjYXJldC1jb2xvcjog cmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4 OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2Vp Z2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0 LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsg d29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1k ZWNvcmF0aW9uOiBub25lOyBmbG9hdDogbm9uZTsgZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7 IiBjbGFzcz0iIj41IHBhY2tldHMgdHJhbnNtaXR0ZWQsIDUgcGFja2V0cyByZWNlaXZlZCwgMC4w JSBwYWNrZXQgbG9zczwvc3Bhbj48YnIgc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7 IGZvbnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6 IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0 dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7 IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6 IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5v bmU7IiBjbGFzcz0iIj48c3BhbiBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9u dC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9y bWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXIt c3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4 dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4 OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsg ZmxvYXQ6IG5vbmU7IGRpc3BsYXk6IGlubGluZSAhaW1wb3J0YW50OyIgY2xhc3M9IiI+cm91bmQt dHJpcCBtaW4vYXZnL21heC9zdGRkZXYgPSAwLjE2Ny82MDAuNjcyLzIwMDIuMzAzLzgwMC43NjMg bXM8L3NwYW4+PGJyIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWls eTogTWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZv bnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5n OiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5z Zm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJr aXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyIgY2xhc3M9 IiI+PGJyIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVu bG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFy aWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3Jt YWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTog bm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4 dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyIgY2xhc3M9IiI+PHNw YW4gc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBNZW5sby1S ZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50 LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsg dGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25l OyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0 cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IGZsb2F0OiBub25lOyBkaXNw bGF5OiBpbmxpbmUgIWltcG9ydGFudDsiIGNsYXNzPSIiPkhvc3QgQjo8L3NwYW4+PGJyIHN0eWxl PSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsg Zm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBu b3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxp Z246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUt c3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lk dGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyIgY2xhc3M9IiI+PGJyIHN0eWxlPSJjYXJl dC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1z aXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7 IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0 YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6 IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBw eDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyIgY2xhc3M9IiI+PHNwYW4gc3R5bGU9ImNhcmV0LWNv bG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6 IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9u dC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7 IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9y bWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0 ZXh0LWRlY29yYXRpb246IG5vbmU7IGZsb2F0OiBub25lOyBkaXNwbGF5OiBpbmxpbmUgIWltcG9y dGFudDsiIGNsYXNzPSIiPiMgaWZjb25maWcgbG8xMCBkZXN0cm95PC9zcGFuPjxiciBzdHlsZT0i Y2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZv bnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9y bWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWdu OiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNw YWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRo OiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIiPjxzcGFuIHN0eWxlPSJjYXJl dC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1z aXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7 IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0 YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6 IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBw eDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyBmbG9hdDogbm9uZTsgZGlzcGxheTogaW5saW5lICFp bXBvcnRhbnQ7IiBjbGFzcz0iIj4jIGlmY29uZmlnIGxvMTAgY3JlYXRlPC9zcGFuPjxiciBzdHls ZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7 IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczog bm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFs aWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRl LXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdp ZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIiPjxzcGFuIHN0eWxlPSJj YXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9u dC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3Jt YWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246 IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3Bh Y2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6 IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyBmbG9hdDogbm9uZTsgZGlzcGxheTogaW5saW5l ICFpbXBvcnRhbnQ7IiBjbGFzcz0iIj4jIGlmY29uZmlnIGxvMTAgMTAuMTEuMTIuMS8yNDwvc3Bh bj48YnIgc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBNZW5s by1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJp YW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1h bDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBu b25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0 LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IiBjbGFzcz0iIj48c3Bh biBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJl Z3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQt Y2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0 ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7 IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ry b2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsgZmxvYXQ6IG5vbmU7IGRpc3Bs YXk6IGlubGluZSAhaW1wb3J0YW50OyIgY2xhc3M9IiI+IyBpZmNvbmZpZyB2bGFuOCBkZXN0cm95 PC9zcGFuPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6 IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250 LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzog bm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zv cm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0 LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIi PjxzcGFuIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVu bG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFy aWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3Jt YWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTog bm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4 dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyBmbG9hdDogbm9uZTsg ZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7IiBjbGFzcz0iIj5pZmNvbmZpZzogaW50ZXJmYWNl IHZsYW44IGRvZXMgbm90IGV4aXN0PC9zcGFuPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigw LCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9u dC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDog NDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRl bnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQt c3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3Jh dGlvbjogbm9uZTsiIGNsYXNzPSIiPjxzcGFuIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAs IDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0 eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7 IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDog MHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFj aW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9u OiBub25lOyBmbG9hdDogbm9uZTsgZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7IiBjbGFzcz0i Ij4jIGlmY29uZmlnIHZsYW44IGNyZWF0ZSB2bGFuZGV2IGJjZTAgdmxhbiA4IHVwPC9zcGFuPjxi ciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJl Z3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQt Y2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0 ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7 IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ry b2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIiPjxzcGFuIHN0 eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxh cjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBz OiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQt YWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hp dGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Ut d2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyBmbG9hdDogbm9uZTsgZGlzcGxheTog aW5saW5lICFpbXBvcnRhbnQ7IiBjbGFzcz0iIj4jIGlmY29uZmlnIHZsYW44IGluZXQ2IC1pZmRp c2FibGVkIGF1dG9fbGlua2xvY2FsPC9zcGFuPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigw LCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9u dC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDog NDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRl bnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQt c3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3Jh dGlvbjogbm9uZTsiIGNsYXNzPSIiPjxzcGFuIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAs IDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0 eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7 IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDog MHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFj aW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9u OiBub25lOyBmbG9hdDogbm9uZTsgZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7IiBjbGFzcz0i Ij4jIHJvdXRlIGFkZCAtbmV0IDEwLjExLjEzLjAvMjQgLWluZXQ2IGZlODA6OjI2YmU6NWZmOmZl MTA6YzkwMCV2bGFuODwvc3Bhbj48YnIgc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7 IGZvbnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6 IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0 dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7 IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6 IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5v bmU7IiBjbGFzcz0iIj48c3BhbiBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9u dC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9y bWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXIt c3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4 dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4 OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsg ZmxvYXQ6IG5vbmU7IGRpc3BsYXk6IGlubGluZSAhaW1wb3J0YW50OyIgY2xhc3M9IiI+YWRkIG5l dCAxMC4xMS4xMy4wOiBnYXRld2F5IGZlODA6OjI2YmU6NWZmOmZlMTA6YzkwMCV2bGFuOCBmaWIg MDwvc3Bhbj48YnIgc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5 OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9u dC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6 IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNm b3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtp dC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IiBjbGFzcz0i Ij48c3BhbiBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1l bmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZh cmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9y bWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06 IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRl eHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsgZmxvYXQ6IG5vbmU7 IGRpc3BsYXk6IGlubGluZSAhaW1wb3J0YW50OyIgY2xhc3M9IiI+IyBwaW5nIC1jNSAtUyAxMC4x MS4xMi4xIDEwLjExLjEzLjE8L3NwYW4+PGJyIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAs IDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0 eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7 IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDog MHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFj aW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9u OiBub25lOyIgY2xhc3M9IiI+PHNwYW4gc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7 IGZvbnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6 IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0 dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7 IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6 IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5v bmU7IGZsb2F0OiBub25lOyBkaXNwbGF5OiBpbmxpbmUgIWltcG9ydGFudDsiIGNsYXNzPSIiPlBJ TkcgMTAuMTEuMTMuMSAoMTAuMTEuMTMuMSkgZnJvbSAxMC4xMS4xMi4xOiA1NiBkYXRhIGJ5dGVz PC9zcGFuPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6 IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250 LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzog bm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zv cm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0 LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIi PjxzcGFuIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVu bG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFy aWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3Jt YWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTog bm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4 dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyBmbG9hdDogbm9uZTsg ZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7IiBjbGFzcz0iIj42NCBieXRlcyBmcm9tIDEwLjEx LjEzLjE6IGljbXBfc2VxPTAgdHRsPTY0IHRpbWU9MTAwMC4yODUgbXM8L3NwYW4+PGJyIHN0eWxl PSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsg Zm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBu b3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxp Z246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUt c3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lk dGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyIgY2xhc3M9IiI+PHNwYW4gc3R5bGU9ImNh cmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250 LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1h bDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjog c3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFj ZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDog MHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IGZsb2F0OiBub25lOyBkaXNwbGF5OiBpbmxpbmUg IWltcG9ydGFudDsiIGNsYXNzPSIiPjY0IGJ5dGVzIGZyb20gMTAuMTEuMTMuMTogaWNtcF9zZXE9 MSB0dGw9NjQgdGltZT0wLjE0MSBtczwvc3Bhbj48YnIgc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2Io MCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZv bnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6 IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5k ZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3Jk LXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29y YXRpb246IG5vbmU7IiBjbGFzcz0iIj48c3BhbiBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAw LCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1z dHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAw OyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6 IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3Bh Y2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlv bjogbm9uZTsgZmxvYXQ6IG5vbmU7IGRpc3BsYXk6IGlubGluZSAhaW1wb3J0YW50OyIgY2xhc3M9 IiI+NjQgYnl0ZXMgZnJvbSAxMC4xMS4xMy4xOiBpY21wX3NlcT0yIHR0bD02NCB0aW1lPTAuMjMx IG1zPC9zcGFuPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1p bHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBm b250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2lu Zzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFu c2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Vi a2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNz PSIiPjxzcGFuIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTog TWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQt dmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBu b3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9y bTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQt dGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyBmbG9hdDogbm9u ZTsgZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7IiBjbGFzcz0iIj42NCBieXRlcyBmcm9tIDEw LjExLjEzLjE6IGljbXBfc2VxPTMgdHRsPTY0IHRpbWU9MC4yMzUgbXM8L3NwYW4+PGJyIHN0eWxl PSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsg Zm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBu b3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxp Z246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUt c3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lk dGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyIgY2xhc3M9IiI+PHNwYW4gc3R5bGU9ImNh cmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250 LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1h bDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjog c3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFj ZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDog MHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IGZsb2F0OiBub25lOyBkaXNwbGF5OiBpbmxpbmUg IWltcG9ydGFudDsiIGNsYXNzPSIiPjY0IGJ5dGVzIGZyb20gMTAuMTEuMTMuMTogaWNtcF9zZXE9 NCB0dGw9NjQgdGltZT0wLjE3NCBtczwvc3Bhbj48YnIgc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2Io MCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZv bnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6 IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5k ZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3Jk LXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29y YXRpb246IG5vbmU7IiBjbGFzcz0iIj48YnIgc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwg MCk7IGZvbnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5 bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsg bGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAw cHg7IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNp bmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246 IG5vbmU7IiBjbGFzcz0iIj48c3BhbiBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsg Zm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTog bm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0 ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsg dGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzog MHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9u ZTsgZmxvYXQ6IG5vbmU7IGRpc3BsYXk6IGlubGluZSAhaW1wb3J0YW50OyIgY2xhc3M9IiI+LS0t IDEwLjExLjEzLjEgcGluZyBzdGF0aXN0aWNzIC0tLTwvc3Bhbj48YnIgc3R5bGU9ImNhcmV0LWNv bG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6 IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9u dC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7 IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9y bWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0 ZXh0LWRlY29yYXRpb246IG5vbmU7IiBjbGFzcz0iIj48c3BhbiBzdHlsZT0iY2FyZXQtY29sb3I6 IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNw eDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdl aWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4 dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7 IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQt ZGVjb3JhdGlvbjogbm9uZTsgZmxvYXQ6IG5vbmU7IGRpc3BsYXk6IGlubGluZSAhaW1wb3J0YW50 OyIgY2xhc3M9IiI+NSBwYWNrZXRzIHRyYW5zbWl0dGVkLCA1IHBhY2tldHMgcmVjZWl2ZWQsIDAu MCUgcGFja2V0IGxvc3M8L3NwYW4+PGJyIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDAp OyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxl OiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxl dHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4 OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5n OiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBu b25lOyIgY2xhc3M9IiI+PHNwYW4gc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZv bnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5v cm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVy LXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRl eHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBw eDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7 IGZsb2F0OiBub25lOyBkaXNwbGF5OiBpbmxpbmUgIWltcG9ydGFudDsiIGNsYXNzPSIiPnJvdW5k LXRyaXAgbWluL2F2Zy9tYXgvc3RkZGV2ID0gMC4xNDEvMjAwLjIxMy8xMDAwLjI4NS80MDAuMDM2 IG1zPC9zcGFuPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1p bHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBm b250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2lu Zzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFu c2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Vi a2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNz PSIiPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1l bmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZh cmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9y bWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06 IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRl eHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIiPjxz cGFuIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8t UmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFu dC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7 IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9u ZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1z dHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyBmbG9hdDogbm9uZTsgZGlz cGxheTogaW5saW5lICFpbXBvcnRhbnQ7IiBjbGFzcz0iIj4xLjxzcGFuIGNsYXNzPSJBcHBsZS1j b252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48L3NwYW4+PGEgc3R5bGU9ImZvbnQtZmFtaWx5 OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9u dC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6 IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNm b3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtp dC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyIgY2xhc3M9IiIgaHJlZj0iaHR0cHM6Ly9jZ2l0LmZy ZWVic2Qub3JnL3NyYy9jb21taXQvP2lkPWY4MTg1NTk3NzRjYjBjMTUxNjM2NGM0YmVjYTM2MTQ4 MGZkNjhiNWIiIHJlbD0ibm9yZWZlcnJlciBub2ZvbGxvdyBub29wZW5lciIgdGFyZ2V0PSJfYmxh bmsiPmh0dHBzOi8vY2dpdC5mcmVlYnNkLm9yZy9zcmMvY29tbWl0Lz9pZD1mODE4NTU5Nzc0Y2Iw YzE1MTYzNjRjNGJlY2EzNjE0ODBmZDY4YjViPC9hPjxzcGFuIHN0eWxlPSJjYXJldC1jb2xvcjog cmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1zaXplOiAxM3B4 OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2Vp Z2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0 LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsg d29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgdGV4dC1k ZWNvcmF0aW9uOiBub25lOyBmbG9hdDogbm9uZTsgZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7 IiBjbGFzcz0iIj48c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3Nw YW4+PC9zcGFuPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1p bHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBm b250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2lu Zzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFu c2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Vi a2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNz PSIiPjxiciBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6IE1l bmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZh cmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9y bWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06 IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRl eHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsiIGNsYXNzPSIiPjxz cGFuIHN0eWxlPSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8t UmVndWxhcjsgZm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFu dC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7 IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9u ZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1z dHJva2Utd2lkdGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyBmbG9hdDogbm9uZTsgZGlz cGxheTogaW5saW5lICFpbXBvcnRhbnQ7IiBjbGFzcz0iIj5DaGVlcnM8L3NwYW4+PGJyIHN0eWxl PSJjYXJldC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsg Zm9udC1zaXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBu b3JtYWw7IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxp Z246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUt c3BhY2U6IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lk dGg6IDBweDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyIgY2xhc3M9IiI+PGJyIHN0eWxlPSJjYXJl dC1jb2xvcjogcmdiKDAsIDAsIDApOyBmb250LWZhbWlseTogTWVubG8tUmVndWxhcjsgZm9udC1z aXplOiAxM3B4OyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7 IGZvbnQtd2VpZ2h0OiA0MDA7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IHRleHQtYWxpZ246IHN0 YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6 IG5vcm1hbDsgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBw eDsgdGV4dC1kZWNvcmF0aW9uOiBub25lOyIgY2xhc3M9IiI+PHNwYW4gc3R5bGU9ImNhcmV0LWNv bG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6 IDEzcHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9u dC13ZWlnaHQ6IDQwMDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7 IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9y bWFsOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0 ZXh0LWRlY29yYXRpb246IG5vbmU7IGZsb2F0OiBub25lOyBkaXNwbGF5OiBpbmxpbmUgIWltcG9y dGFudDsiIGNsYXNzPSIiPi0tPHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5i c3A7PC9zcGFuPjwvc3Bhbj48YnIgc3R5bGU9ImNhcmV0LWNvbG9yOiByZ2IoMCwgMCwgMCk7IGZv bnQtZmFtaWx5OiBNZW5sby1SZWd1bGFyOyBmb250LXNpemU6IDEzcHg7IGZvbnQtc3R5bGU6IG5v cm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IDQwMDsgbGV0dGVy LXNwYWNpbmc6IG5vcm1hbDsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRl eHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3b3JkLXNwYWNpbmc6IDBw eDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRpb246IG5vbmU7 IiBjbGFzcz0iIj48c3BhbiBzdHlsZT0iY2FyZXQtY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1m YW1pbHk6IE1lbmxvLVJlZ3VsYXI7IGZvbnQtc2l6ZTogMTNweDsgZm9udC1zdHlsZTogbm9ybWFs OyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3Bh Y2luZzogbm9ybWFsOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10 cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdvcmQtc3BhY2luZzogMHB4OyAt d2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsgZmxv YXQ6IG5vbmU7IGRpc3BsYXk6IGlubGluZSAhaW1wb3J0YW50OyIgY2xhc3M9IiI+TWFyZWsgWmFy eWNodGE8L3NwYW4+PC9kaXY+PC9ibG9ja3F1b3RlPjwvZGl2PjwvYmxvY2txdW90ZT48L2Rpdj48 YnIgY2xhc3M9IiI+PGRpdiBjbGFzcz0iIj4NCjxkaXY+PGJyIGNsYXNzPSIiPjwvZGl2Pg0KDQo8 L2Rpdj4NCjxiciBjbGFzcz0iIj4NCiAgICAgICAgPC9ibG9ja3F1b3RlPjxicj4NCiAgICA8L2Rp dj4= --b1_LE2649YWVsmFw4KGLN3gQqtInOkXHQEyfYdVKyNtzc-- From nobody Fri Mar 22 13:41:14 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V1NlQ0Qjzz5DxvH for ; Fri, 22 Mar 2024 13:41:22 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V1NlQ098Tz4CTn; Fri, 22 Mar 2024 13:41:22 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711114882; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=p51x+ittN3UBS8qg0sUpsrMkWWjc9/v7qOc80ALiCJU=; b=weGYsZndLXIRtgDUQYmXcyF7Eejs/rYbVjCLpeGn8LgiALH4wRKAb25cW28tKVQaTVXStd E9frl8SI40nUCQjnzCZm+jjB2CxFqZnmdk2Zb6sKkewo7raBmiS3uAIG9+S0l3PGNLwE3H ctuADfIlUDYy6kogMupiiKKjOIOkTW6Mr7FJL4LkX9W/pJPWpdhjd6q1IvQKMOyZgoGTOy sARUmjvwxvg3H7DwQKbcA+yM3kLJGK+Y6/xOLoQy3erbZ4yCLJWLT4SIQ5sOLmvrQCVX64 UFGh8v7A/I6CVg1VO+ackR+z4AU4PQfxk7xQ3aGy20SvBvwwL8GQaYxY9J/fgg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711114882; a=rsa-sha256; cv=none; b=TV+0Avow49Jqu4pv4h6ZIJ/4OPQsTB4KLIx+VUuIKJzR/3//K6ajT368WDKCjtQftaF57P SQCVhEkb6nEaGjUDBqWPYNwbXO2bdxQL9f6n4Hwn+dHIzNM8BAc5ZFCPICTzmIbFJ66jap X/vm6bxKInPVmGRsFA1ljuaGB9QSZqbDB76oc+o4yy+DmtWq8CRT9DTK/jyE7OQvLMc176 tHHrne7D1rAzWpcv5QDpaf/WRdPQtixn/qYSCcccqGUCTsjJTVzdBPT8adh51A4vAsEJpe inSUaFxCcsKzv1bpfVFrXAKzzeQl4ZtSQANITvtXigfD0lCVi9aggrpanfyYzA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711114882; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=p51x+ittN3UBS8qg0sUpsrMkWWjc9/v7qOc80ALiCJU=; b=U5oF0u/ygnDTaz4IncIujCL8pJEAbJUCqi1Ukol7hMvk+JUC7CBZjB1aoxeab7byVUdWWL GF4y2Oj9wuCtocm+/s/k23tDwPDxZWT++V0/sYHQXbpenec4ETDcperrpP6ut9llXOBEnm 2Xftph2+PVIeyL+cC6OAYCfiB3XolgiXbA2o+SkuoLXpmMaT5Npgn1q+Sn6Ia+mK/plfaF e90XLa3CIYbLsKi0fY98s1p3zlM/o4QWkByCrIvPb2cyJJ/FWzaUJtUXfSF9+Wa8oUgrOs T1BjjEE0OWq5Kiwj0sRuZc+lIJTm7l2jDXMXOLcUWgL7P3RZcglnBqHoXpIGKw== Received: from smtpclient.apple (ns1.oxydns.net [45.32.91.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4V1NlN0V2tz1ThH; Fri, 22 Mar 2024 13:41:19 +0000 (UTC) (envelope-from zlei@FreeBSD.org) From: Zhenlei Huang Message-Id: Content-Type: multipart/signed; boundary="Apple-Mail=_138E9550-2C3B-4A0C-B695-45F6695B4844"; protocol="application/pgp-signature"; micalg=pgp-sha512 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.8\)) Subject: Re: ipv4 route with ipv6 local link nexthop ? Date: Fri, 22 Mar 2024 21:41:14 +0800 In-Reply-To: <_dTkes6xnAAQDFSPOPVSOT7GfMjJzaajrkovO456NI6vBSNn7afe4S3p9UCvw35j_sMnvF_CYIDtmT5JcKm7nHhTSVV7VeW3NfdZCR2iTcA=@enki-multimedia.eu> Cc: Marek Zarychta , FreeBSD Net , FreeBSD Security Team To: Benoit Chesneau References: <367504DC-48DA-4DFD-9DB6-CC571F0D26B8@FreeBSD.org> <764E12BF-5D31-4905-98AE-6D745BFD1DC2@FreeBSD.org> <4380f799-b961-4daf-8514-679c06214d55@plan-b.pwste.edu.pl> <24620735-923d-4603-8c92-1d9b23d3ce80@plan-b.pwste.edu.pl> <323D6B49-EC5C-4011-8BBA-1EAB9DFC4BC2@FreeBSD.org> <1A5D703E-37B1-407C-8B72-0F4B62DC4219@FreeBSD.org> <_dTkes6xnAAQDFSPOPVSOT7GfMjJzaajrkovO456NI6vBSNn7afe4S3p9UCvw35j_sMnvF_CYIDtmT5JcKm7nHhTSVV7VeW3NfdZCR2iTcA=@enki-multimedia.eu> X-Mailer: Apple Mail (2.3696.120.41.1.8) --Apple-Mail=_138E9550-2C3B-4A0C-B695-45F6695B4844 Content-Type: multipart/alternative; boundary="Apple-Mail=_A305FDFA-DFEF-455B-9C05-5EFBCD190554" --Apple-Mail=_A305FDFA-DFEF-455B-9C05-5EFBCD190554 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Mar 22, 2024, at 5:05 PM, Benoit Chesneau = wrote: >=20 > Awesome! Do we have a chance it land in a patch release soon ? Or = better to use a STABLE until the 14.1 is released? >=20 Or you can stay on 14.0 If the workaround can fulfill. 14.1 is about to = be released at 18 June as per the schedule [1] , that is about 3 and half months. CCing secteam, I do not think there is any security impact so probably = it does not deserve an EN. See also the bug report = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275341 = . 1. https://www.freebsd.org/releases/14.1R/schedule/ Best regards, Zhenlei >=20 > Beno=C3=AEt > On Thursday, March 14th, 2024 at 10:56, Zhenlei Huang = wrote: >>=20 >>=20 >>> On Mar 14, 2024, at 9:04 AM, Zhenlei Huang > wrote: >>>=20 >>>=20 >>>=20 >>>> On Mar 14, 2024, at 3:07 AM, Marek Zarychta = > = wrote: >>>>=20 >>>> W dniu 13.03.2024 o 18:59, Marek Zarychta pisze: >>>>> W dniu 13.03.2024 o 16:31, Benoit Chesneau pisze: >>>>>> Hrm I thought it was implemented via = https://reviews.freebsd.org/rG62e1a437f3285e785d9b35a476d36a469a90028d = >>>>>>=20 >>>>>> Wasn't it merged ? (also pretty sure I did test it in freebsd = 13). >>>>>>=20 >>>>> FWIW: it works fine on CURRENT >>>>>=20 >>>>> # ifconfig vlan8 destroy >>>>> # ifconfig vlan8 create vlandev bge0 vlan 8 up >>>>> # ifconfig vlan8 inet6 -ifdisabled auto_linklocal >>>>> # route add -net 10.11.13.0/24 -inet6 = fe80::360a:11ff:fe1b:404e%vlan8 >>>>> add net 10.11.13.0: gateway fe80::360a:11ff:fe1b:404e%vlan8 fib 0 >>>>>=20 >>>> It looks like the fix is in = f818559774cb0c1516364c4beca361480fd68b5b . Zhenlei, could you please MFC = this one[1] ? >>>>=20 >>>> Cherry-picking it to stable/14 makes route fully functional. I have = tested it between stable/14 with this fix applied and CURRENT. >>>=20 >>> Thanks for finding the fix and the confirmation , I'll take care of = that :) >>=20 >> Done. The fix and tests are all MFCed into stable/14 branch. >>=20 >>>=20 >>>>=20 >>>> Host A: >>>>=20 >>>> # ifconfig lo10 destroy >>>> # ifconfig lo10 create >>>> # ifconfig lo10 10.11.13.1/24 >>>> # ifconfig vlan8 destroy >>>> # ifconfig vlan8 create vlandev bge0 vlan 8 up >>>> # ifconfig vlan8 inet6 -ifdisabled auto_linklocal >>>> # route add -net 10.11.12.0/24 -inet6 = fe80::6ab5:99ff:febd:8108%vlan8 >>>> add net 10.11.12.0: gateway fe80::6ab5:99ff:febd:8108%vlan8 fib 0 >>>> # ping -c5 -S 10.11.13.1 10.11.12.1 >>>> PING 10.11.12.1 (10.11.12.1) from 10.11.13.1: 56 data bytes >>>> 64 bytes from 10.11.12.1: icmp_seq=3D0 ttl=3D64 time=3D2002.303 ms >>>> 64 bytes from 10.11.12.1: icmp_seq=3D1 ttl=3D64 time=3D1000.461 ms >>>> 64 bytes from 10.11.12.1: icmp_seq=3D2 ttl=3D64 time=3D0.167 ms >>>> 64 bytes from 10.11.12.1: icmp_seq=3D3 ttl=3D64 time=3D0.222 ms >>>> 64 bytes from 10.11.12.1: icmp_seq=3D4 ttl=3D64 time=3D0.207 ms >>>>=20 >>>> --- 10.11.12.1 ping statistics --- >>>> 5 packets transmitted, 5 packets received, 0.0% packet loss >>>> round-trip min/avg/max/stddev =3D 0.167/600.672/2002.303/800.763 ms >>>>=20 >>>> Host B: >>>>=20 >>>> # ifconfig lo10 destroy >>>> # ifconfig lo10 create >>>> # ifconfig lo10 10.11.12.1/24 >>>> # ifconfig vlan8 destroy >>>> ifconfig: interface vlan8 does not exist >>>> # ifconfig vlan8 create vlandev bce0 vlan 8 up >>>> # ifconfig vlan8 inet6 -ifdisabled auto_linklocal >>>> # route add -net 10.11.13.0/24 -inet6 = fe80::26be:5ff:fe10:c900%vlan8 >>>> add net 10.11.13.0: gateway fe80::26be:5ff:fe10:c900%vlan8 fib 0 >>>> # ping -c5 -S 10.11.12.1 10.11.13.1 >>>> PING 10.11.13.1 (10.11.13.1) from 10.11.12.1: 56 data bytes >>>> 64 bytes from 10.11.13.1: icmp_seq=3D0 ttl=3D64 time=3D1000.285 ms >>>> 64 bytes from 10.11.13.1: icmp_seq=3D1 ttl=3D64 time=3D0.141 ms >>>> 64 bytes from 10.11.13.1: icmp_seq=3D2 ttl=3D64 time=3D0.231 ms >>>> 64 bytes from 10.11.13.1: icmp_seq=3D3 ttl=3D64 time=3D0.235 ms >>>> 64 bytes from 10.11.13.1: icmp_seq=3D4 ttl=3D64 time=3D0.174 ms >>>>=20 >>>> --- 10.11.13.1 ping statistics --- >>>> 5 packets transmitted, 5 packets received, 0.0% packet loss >>>> round-trip min/avg/max/stddev =3D 0.141/200.213/1000.285/400.036 ms >>>>=20 >>>> 1. = https://cgit.freebsd.org/src/commit/?id=3Df818559774cb0c1516364c4beca36148= 0fd68b5b = >>>>=20 >>>> Cheers >>>>=20 >>>> -- >>>> Marek Zarychta >>=20 >>=20 >>=20 >=20 --Apple-Mail=_A305FDFA-DFEF-455B-9C05-5EFBCD190554 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On Mar 22, 2024, at 5:05 PM, Benoit Chesneau <benoitc@enki-multimedia.eu> wrote:

Awesome! Do we have a chance it = land in a patch release soon ? Or better to use a STABLE until the 14.1 = is released? 


Or you can stay on 14.0 If the workaround can = fulfill. 14.1 is about to be released at 18 June as per = the
schedule [1] , that is about 3 and half = months.

CCing secteam, I do not = think there is any security impact so probably it does not deserve an = EN.


=
Best regards,
Zhenlei


Beno=C3=AEt 
On Thursday, March 14th, 2024 at 10:56, Zhenlei Huang <zlei@FreeBSD.org> = wrote:


On Mar 14, 2024, at 9:04 AM, = Zhenlei Huang <zlei@FreeBSD.org> wrote:



On Mar 14, 2024, at 3:07 AM, Marek = Zarychta <zarychtam@plan-b.pwste.edu.pl> wrote:

W dniu = 13.03.2024 o 18:59, Marek Zarychta pisze:
W dniu 13.03.2024 o 16:31, Benoit = Chesneau pisze:
Hrm I = thought it was implemented via https://reviews.freebsd.org/rG62e1a437f3285e785d9b35a476= d36a469a90028d

Wasn't it merged ? (also = pretty sure I did test it in freebsd 13).

FWIW: it works fine on CURRENT

# ifconfig vlan8 destroy
# ifconfig vlan8 = create vlandev bge0 vlan 8 up
# ifconfig vlan8 inet6 = -ifdisabled auto_linklocal
# route add -net 10.11.13.0/24 = -inet6 fe80::360a:11ff:fe1b:404e%vlan8
add net 10.11.13.0: = gateway fe80::360a:11ff:fe1b:404e%vlan8 fib 0

It looks like the fix is in = f818559774cb0c1516364c4beca361480fd68b5b . Zhenlei, could you please MFC = this one[1] ?

Cherry-picking = it to stable/14 makes route fully functional. I have tested it between = stable/14 with this fix applied and CURRENT.

Thanks for = finding the fix and the confirmation , I'll take care of that = :)

Done. The fix and tests are all MFCed into stable/14 = branch.



Host A:

# ifconfig lo10 destroy
# ifconfig lo10 create
# ifconfig lo10 10.11.13.1/24
# ifconfig vlan8 destroy
# ifconfig vlan8 create vlandev bge0 vlan 8 up
# ifconfig = vlan8 inet6 -ifdisabled auto_linklocal
# route add -net 10.11.12.0/24 -inet6 = fe80::6ab5:99ff:febd:8108%vlan8
add net 10.11.12.0: gateway fe80::6ab5:99ff:febd:8108%vlan8 = fib 0
# ping -c5 -S = 10.11.13.1 10.11.12.1
PING 10.11.12.1 (10.11.12.1) from 10.11.13.1: 56 data = bytes
64 bytes from = 10.11.12.1: icmp_seq=3D0 ttl=3D64 time=3D2002.303 ms
64 bytes from = 10.11.12.1: icmp_seq=3D1 ttl=3D64 time=3D1000.461 ms
64 bytes from = 10.11.12.1: icmp_seq=3D2 ttl=3D64 time=3D0.167 ms
64 bytes from = 10.11.12.1: icmp_seq=3D3 ttl=3D64 time=3D0.222 ms
64 bytes from = 10.11.12.1: icmp_seq=3D4 ttl=3D64 time=3D0.207 ms

--- = 10.11.12.1 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet = loss
round-trip = min/avg/max/stddev =3D 0.167/600.672/2002.303/800.763 ms

Host = B:

# ifconfig = lo10 destroy
# ifconfig = lo10 create
# ifconfig = lo10 10.11.12.1/24
# ifconfig vlan8 destroy
ifconfig: interface vlan8 does not exist
# ifconfig = vlan8 create vlandev bce0 vlan 8 up
# ifconfig vlan8 inet6 -ifdisabled auto_linklocal
# route add = -net 10.11.13.0/24 -inet6 fe80::26be:5ff:fe10:c900%vlan8
add net = 10.11.13.0: gateway fe80::26be:5ff:fe10:c900%vlan8 fib 0
# ping -c5 -S = 10.11.12.1 10.11.13.1
PING 10.11.13.1 (10.11.13.1) from 10.11.12.1: 56 data = bytes
64 bytes from = 10.11.13.1: icmp_seq=3D0 ttl=3D64 time=3D1000.285 ms
64 bytes from = 10.11.13.1: icmp_seq=3D1 ttl=3D64 time=3D0.141 ms
64 bytes from = 10.11.13.1: icmp_seq=3D2 ttl=3D64 time=3D0.231 ms
64 bytes from = 10.11.13.1: icmp_seq=3D3 ttl=3D64 time=3D0.235 ms
64 bytes from = 10.11.13.1: icmp_seq=3D4 ttl=3D64 time=3D0.174 ms

--- = 10.11.13.1 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet = loss
round-trip = min/avg/max/stddev =3D 0.141/200.213/1000.285/400.036 ms

1. https://cgit.freebsd.org/src/commit/?id=3Df818559774cb0c= 1516364c4beca361480fd68b5b 

Cheers

-- Marek = Zarychta







= --Apple-Mail=_A305FDFA-DFEF-455B-9C05-5EFBCD190554-- --Apple-Mail=_138E9550-2C3B-4A0C-B695-45F6695B4844 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iNUEARYKAH0WIQRj28YmNowGX1isJg7GJJ6Jgbd0XwUCZf2Kel8UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0NjNE QkM2MjYzNjhDMDY1RjU4QUMyNjBFQzYyNDlFODk4MUI3NzQ1RgAKCRDGJJ6Jgbd0 XxjgAPwKIIHrrxlRRGKKoNRBExxt6gHizfvAOx8nVPZgRXjOkwEA2lKVo/kMQLb6 cnX4W3Nesu7NeKj85ImXLUClOAh3LAE= =gQlo -----END PGP SIGNATURE----- --Apple-Mail=_138E9550-2C3B-4A0C-B695-45F6695B4844-- From nobody Sat Mar 23 22:23:54 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V2DHt652zz5G1Cj for ; Sat, 23 Mar 2024 22:23:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V2DHt50Mtz4SL7 for ; Sat, 23 Mar 2024 22:23:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711232634; a=rsa-sha256; cv=none; b=vIxkWJx6B4Rl/ab/6w/fDigiMuoqsCumxDGLUSm+KleyktzZNZbl7aGjAjgStGQ2pVVk4K oMmSPwkca9dKrVpvtzuoEFFBS3lvKJ5pVWKglUB5ELdINm5gi6QvXeg6EWTgQG+eix8uRv AlUhu6kOCyOSJ0uusAP8NbRScBAd92p+taotXBASkRPxByE5Vq7JjX3ifF3aWakG2fQaOZ zCjTLrA1u8Iu4bbu/67yF6TQowLjI7g1MlBaqee8J8hL4hqjsz3kdW260Jyshgm7kf5CZA HOiJDpjR9HYladuAAuHay3eHUC/+x8lz6tO9yzLUSIv8WNHUh4dCb89tygwbUQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711232634; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FS54+OQPpcrA4Wn1XGVdC2j0RKY3zTlMzZNP7CtCUrc=; b=ygn1H9os2KvYZDQSutm4huDDUNelZ/EzpIM31l2MjjP9A9Mt8UoiPlj2n9yg0FGQgauGhU l7gj93uMDW/Nr21DMsuX4Y/oK8lrKjggD1uIs+79aiSEe2cNTPn6TL4NPPLElNPxSGCM0U RFGuxFY296jpANZS1c6++usDIt5jGkGriFzGJcdcZ5mm8tHWoX94eoYsz6R9Iu0M+jVtzW /PtaeEPArHWvfK8eTYBE2G4lZlPetZU1/Dim9jg15bZ9y9jPtQABe1YVe0DPPUEUB3JmLH NNgYGY8vesR++RucbGPu7XDW3iSrIEMg04E0avicSJJ9YrDShYdD53qG6upixg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4V2DHt4bWsz16WS for ; Sat, 23 Mar 2024 22:23:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42NMNsG7047354 for ; Sat, 23 Mar 2024 22:23:54 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42NMNs6M047353 for net@FreeBSD.org; Sat, 23 Mar 2024 22:23:54 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 277875] pfctl cowardly refuses to load rules, broken between 8c94ed992702 & f29af8618bf9 Date: Sat, 23 Mar 2024 22:23:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 15.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dch@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.isobsolete attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277875 Dave Cottlehuber changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #249387|0 |1 is obsolete| | Attachment #249388|0 |1 is obsolete| | --- Comment #4 from Dave Cottlehuber --- Created attachment 249438 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D249438&action= =3Dedit truss log Thanks, rebuilt with that patch included. I reduced the failing ruleset to this minimal example: ``` # pfctl -s Running Enabled # pfctl -F all Ethernet rules cleared rules cleared nat cleared 0 tables deleted. 0 states cleared source tracking entries cleared pf: statistics cleared pf: interface flags reset root@# echo 'pass in quick on ng0 proto tcp to port 2200' | pfctl -vgf - No ALTQ support in kernel ALTQ related functions disabled pass in quick on ng0 proto tcp from any to any port =3D 2200 flags S/SA keep state # echo $status 1 # pfctl -s rules # ``` Evidently its not a ruleset parsing issue. I swapped ng0 for lo0 and the same situation occurs. running under truss, final lines from attached full log: ioctl(3,DIOCSETTIMEOUT,0x621da911a368) =3D 0 (0x0) ioctl(3,DIOCSETTIMEOUT,0x621da911a368) =3D 0 (0x0) ioctl(3,DIOCSETDEBUG,0x621da911a368) =3D 0 (0x0) sendto(5," \0\0\0\^P\0\^E\0\^A\0\0\0\0\0\0"...,32,0,NULL,0) =3D 32 (0x20) recvmsg(5,{0x621da911a26c,12,[{"\M-x\0\0\0\^P\0\^E\0\^A\0\0\0\0"...,65536}]= ,1,{},0,0},0) =3D 284 (0x11c) sendto(5,"\^\\0\0\0\^Q\0\^E\0\^B\0\0\0\0\0"...,28,0,NULL,0) =3D 28 (0x1c) recvmsg(5,{0x621da911a26c,12,[{"0\0\0\0\^B\0\0\0\^B\0\0\0\0\0\0"...,65536}]= ,1,{},0,0},0) =3D 48 (0x30) ioctl(3,DIOCSETHOSTID,0x621da911a368) =3D 0 (0x0) ioctl(3,DIOCSETREASS,0x621da911a368) =3D 0 (0x0) ioctl(3,DIOCKEEPCOUNTERS,0x621da911a310) =3D 0 (0x0) ioctl(3,DIOCGETLIMIT,0x621da911a300) =3D 0 (0x0) ioctl(3,DIOCSETSYNCOOKIES,0x621da911a300) =3D 0 (0x0) ioctl(3,DIOCXROLLBACK,0x621da911a398) =3D 0 (0x0) extl_if =3D "ng0" pass in quick on ng0 proto tcp from any to any port =3D 2200 flags S/SA keep state write(1,"extl_if =3D "ng0"\npass in quick o"...,97) =3D 97 (0x61) exit(0x1)=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 process exit, rval =3D 1 trying the same ruleset on a different arm64 box with same from-source build, it works as expected - rules loaded, and output displayed. I'll do a full re-install into an empty BE next. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Mar 24 21:01:03 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V2pPr0528z5GRCd for ; Sun, 24 Mar 2024 21:01:04 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V2pPq4MFXz4mh0 for ; Sun, 24 Mar 2024 21:01:03 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711314063; a=rsa-sha256; cv=none; b=YF87B6PnD+U8954qJ7m2egg30mrZuKquMbrkpj6+oJrncVAIU7C11y2izjR3rBRYjU7ItB behPO+kqHHR0I+c7Yi3u2tDA/MB86KZgCFdbLlGxVdwagbnRLFEQgwR1VC6XwyBMj3vhkE ypli/pTqL4J8Sq4Pqn1bsuDDnBP5E1CvVZZf9qq/n5idQ5SdHiLn6qrY6zOJwWizpB84sA bKNW3CMB4fRMDJyzvXekgoc//yZXleb5ASMO3/rFdzGyzMt7DF/0nK01WodPaeAhVfkjuQ x5fnG0GDR6UqszBJOjyog9gdqhAkhjwuSUebn0Vvm8YxBC1zJU8hC1vgv5eYcQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711314063; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=5baVbJ6nmKsmrOw17UE11EQt8rrOmJ9BhI7auTSo66Q=; b=LsQMX9t9FJ1+aHiERA/qbyem57yeB4G18nmD7yG3rcTh0EM7gV4UWtDVCAoFQ0LyR3FyJd JV+XpxExsM7FnWdLTlKBGARatcH8DQKEmfawhm35/DVSEp2HmwxMqw+Pfmabq7ZKKU+//d 4hcbuHZP/9Yh4E+jUgz1VmZS8MS/RHwZOz4S5MhuWSz3/LvYRiOVcIgu5ZFC3YDCIeJKPU LfsVDPVAF7KJE9cglafyUjKAOla7fHoaTbWgBbE7EadJyPEMdyRDunyNT8h6/r7bQF6R/I hzxZfEtrweq/YeYdmPzkxOIhEqSfvV1wrtMtLvScEmJemp7Tp8VApkoAnYMrQQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4V2pPq3Rsyzp4K for ; Sun, 24 Mar 2024 21:01:03 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42OL13CE068776 for ; Sun, 24 Mar 2024 21:01:03 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42OL134T068775 for net@FreeBSD.org; Sun, 24 Mar 2024 21:01:03 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202403242101.42OL134T068775@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: net@FreeBSD.org Subject: Problem reports for net@FreeBSD.org that need special attention Date: Sun, 24 Mar 2024 21:01:03 +0000 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="17113140634.17d5bd2C9.57478" Content-Transfer-Encoding: 7bit --17113140634.17d5bd2C9.57478 Date: Sun, 24 Mar 2024 21:01:03 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 254445 | cloned_interfaces="bridge0" does not respect net. Open | 200836 | iovctl(8): Return descriptions in the returned sc Open | 223824 | Panic in ng_base.c (netgraph) Open | 230807 | if_alc(4): Driver not working for Killer Networki Open | 232472 | ixgbe(4): SR-IOV passthru not working on Hyper-V Open | 234073 | ixl(4): Host X710-DA2 drops connect starting bhyv Open | 241106 | tun/ppp: panic: vm_fault: fault on nofault entry Open | 245981 | bnxt(4): BCM57414 / BCM57416 not initializing: bn Open | 256217 | [tcp] High system load because of interrupts with Open | 257038 | em(4): Panic on HTTP traffic to or from jail thro Open | 257286 | gateway with `ping -6 -e` is ignored Open | 258623 | cxgbe(4): Slow routing performance: 2 numa domain Open | 258850 | lagg(4): interface vanishes when both member inte Open | 261866 | ixgbe(4): Resets media type -> autoselect after s Open | 262024 | em(4): iflib handles bad packets incorrectly Open | 262093 | ixl(4): RX packet errors on Intel X710 after 12.2 Open | 263568 | ix(4): SR-IOV connection lost after loading VM wi In Progress | 118111 | rc: network.subr Add MAC address based interface 18 problems total for which you should take action. --17113140634.17d5bd2C9.57478 Date: Sun, 24 Mar 2024 21:01:03 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
New         |    254445 | cloned_interfaces="bridge0" does not respect net.
Open        |    200836 | iovctl(8): Return descriptions in the returned sc
Open        |    223824 | Panic in ng_base.c (netgraph)
Open        |    230807 | if_alc(4): Driver not working for Killer Networki
Open        |    232472 | ixgbe(4): SR-IOV passthru not working on Hyper-V 
Open        |    234073 | ixl(4): Host X710-DA2 drops connect starting bhyv
Open        |    241106 | tun/ppp: panic: vm_fault: fault on nofault entry 
Open        |    245981 | bnxt(4): BCM57414 / BCM57416 not initializing: bn
Open        |    256217 | [tcp] High system load because of interrupts with
Open        |    257038 | em(4): Panic on HTTP traffic to or from jail thro
Open        |    257286 | gateway with `ping -6 -e` is ignored
Open        |    258623 | cxgbe(4): Slow routing performance: 2 numa domain
Open        |    258850 | lagg(4): interface vanishes when both member inte
Open        |    261866 | ixgbe(4): Resets media type -> autoselect after s
Open        |    262024 | em(4): iflib handles bad packets incorrectly
Open        |    262093 | ixl(4): RX packet errors on Intel X710 after 12.2
Open        |    263568 | ix(4): SR-IOV connection lost after loading VM wi
In Progress |    118111 | rc: network.subr Add MAC address based interface 

18 problems total for which you should take action.
--17113140634.17d5bd2C9.57478--