From nobody Thu Mar 31 03:59:24 2022 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3F8E21A4F9B3 for ; Thu, 31 Mar 2022 04:00:20 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-oa1-x2f.google.com (mail-oa1-x2f.google.com [IPv6:2001:4860:4864:20::2f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KTV2C4TQfz3MBT for ; Thu, 31 Mar 2022 04:00:19 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by mail-oa1-x2f.google.com with SMTP id 586e51a60fabf-d6e29fb3d7so24144293fac.7 for ; Wed, 30 Mar 2022 21:00:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=+YMJ+oG8a73mTPlGz6TV8evXDgRLm1M4DweyKalW8P8=; b=IBVIQ9Kos5bKS1Y4W5uzhHVyJmoPI0RE9K/MYjdkR7/gSQNSUs74dyEh+vM4zID+GE rcMgnZSLJ53grhWQxKX2v73ALb2aSZi59PvC6Q/azJol7ejFpI/tNUoHfbZdVvsDSglg Qxg+Fpg/bTK4BV1PL9bjaPJl67clbIL+SnaLzGCWFw7wuXEOoknIVzRxZ6odov/kfC7l oK/WQKcBPzaH+e7EQ7DVfuQ4e1yE2VHSruFZ8XeWvGVrvskcQQckKK4+dGX6M4tJKMrv /kF8lX/6guCvKJqqLH6eN5h8Zhg8a5x1+w00Mdq2MzlrcFOA8vRX1zbiEH0lxaWScUFN G/UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=+YMJ+oG8a73mTPlGz6TV8evXDgRLm1M4DweyKalW8P8=; b=x7gbmynWrfNqe++TNBwUeO1aFlWanEGxb+MONy3PPxWBFOV6Y2r72fVUNApbaqi6dI bvMPek0hkObo1W269S5h498lInnVf6lvVesxk7uZUfNvDjwGBPGlI2YAoNtb5TSP2iNc dZ6eb16jNkvZycTceRFtcqYJRLdZMI5VgsHy2jZFjKyuyt2cmCYp4qVXbLdfEAzBlFfi 6Uts318PmFn0xvqgY+qxKLVFL/PH6w5FQspMT10oiMBs7Fwhqur9WLCVGvfJxhn+sSC2 i7fiHSUAWcE0B/biKELK1NKQYdfxBy4d/Gj6xe2rC19CNq/UkePM8yn1GM4tAqlAtBOe vSeA== X-Gm-Message-State: AOAM530I0Xai+ue0dxC3V7KlATTRfj/ozBB8Jk5XJg8Bnw5LJc0qVkEX QLL2PxSxKIVpWOL3XDSj6Aakr48K86XZKQQnnrvcK7Jr X-Google-Smtp-Source: ABdhPJzkflqU9Ue6trQMNe9o+pGbciCwN6S/pJQCi3M2a02dwqYm3ezhi4h09uD5rzqNK6a8iJLHDV7FIwTk1EghKrg= X-Received: by 2002:a05:6871:822:b0:dd:b9a5:f5cf with SMTP id q34-20020a056871082200b000ddb9a5f5cfmr1777465oap.230.1648699218778; Wed, 30 Mar 2022 21:00:18 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Kevin Oberman Date: Wed, 30 Mar 2022 20:59:24 -0700 Message-ID: Subject: Re: Slow startup from D19488 (rtsol: sendmsg: Permission denied) To: Peter Cc: "Bjoern A. Zeeb" , FreeBSD-STABLE Mailing List Content-Type: multipart/alternative; boundary="00000000000046dd3e05db7bb37a" X-Rspamd-Queue-Id: 4KTV2C4TQfz3MBT X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=IBVIQ9Ko; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of kob6558@gmail.com designates 2001:4860:4864:20::2f as permitted sender) smtp.mailfrom=kob6558@gmail.com X-Spamd-Result: default: False [0.71 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2001:4860:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[3]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FORGED_SENDER(0.30)[rkoberman@gmail.com,kob6558@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; FROM_NEQ_ENVFROM(0.00)[rkoberman@gmail.com,kob6558@gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.57)[-0.566]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(0.98)[0.975]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2001:4860:4864:20::2f:from]; HTTP_TO_IP(1.00)[]; MLMMJ_DEST(0.00)[freebsd-stable]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --00000000000046dd3e05db7bb37a Content-Type: text/plain; charset="UTF-8" On Tue, Mar 29, 2022 at 5:10 PM Peter wrote: > > Hello Bjoern, > > thanks much for the quick reply! > > On Tue, Mar 29, 2022 at 10:04:11PM +0000, Bjoern A. Zeeb wrote: > ! On Tue, 29 Mar 2022, Peter wrote: > ! > ! Hi, > ! > ! I am a bit puzzled as after two years you are the first one to report > ! that problem to my knowledge for either base system or jails. > > This is what greatly wonders me, too. So I was stronly thinking > that I am doing something wrong or unusual. But I cannot figure > it out, it just seems that the detrimental effect of the change > cannot be avoided (e.g. "service jail start" takes quite long now - > there's a lot of them). > > ! > after upgrading 12.3 to stable/13, I am seeing these > ! > errors in all my jails: > ! > > ! > > Additional TCP/IP options: log_in_vain=1. > ! > > ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib > ! > /usr/local/lib/c cmpat/pkg /usr/local/lib/compat/pkg > ! > > 32-bit compatibility ldconfig path: > ! > > rtsol: sendmsg on nrail1l: Permission denied > ! > > rtsol: sendmsg on nrail1l: Permission denied > ! > > rtsol: sendmsg on nrail1l: Permission denied > ! > > Starting Network: lo0 nrail1l. > ! > ! Can you give us a full startup log? > > It's the above, right from the beginning, and then follows: > > > lo0: flags=8049 metric 0 mtu 16384 > > options=680003 > > inet 127.0.0.1 netmask 0xff000000 > > inet6 ::1 prefixlen 128 > > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > > groups: lo > > nd6 options=21 > > nrail1l: flags=8843 metric 0 mtu > 1500 > > options=28 > > ether 06:1d:92:01:01:0a > > hwaddr 58:9c:fc:10:28:71 > > inet ************* netmask ********** broadcast ************* > > inet6 fe80::41d:92ff:fe01:10a%nrail1l prefixlen 64 scopeid 0x2 > > inet6 fd00:************ prefixlen 120 > > media: Ethernet autoselect (1000baseT ) > > status: active > > nd6 options=23 > > Starting rtsold. > > add host 127.0.0.1: gateway lo0 fib 0: route already in table > > add net default: gateway ************* > > Additional inet routing options: log ICMP redirect=YES. > > add host ::1: gateway lo0 fib 0: route already in table > > add net fe80::: gateway ::1 > > add net ff02::: gateway ::1 > > add net ::ffff:0.0.0.0: gateway ::1 > > add net ::0.0.0.0: gateway ::1 > > add net default: gateway fd00:************* > > Flushed all rules. > > Firewall rules loaded. > > Firewall logging pseudo-interface (ipfw0) created. > > Creating and/or trimming log files. > > Updating /var/run/os-release done. > > Clearing /tmp (X related). > > Updating motd:. > > Starting syslogd. > > Starting rapp. > > Starting cron. > > Starting sendmail. > > Starting sendmail_msp_queue. > > Performing sanity check on sshd configuration. > > Starting sshd. > > > > Wed Mar 30 00:52:15 CEST 2022 > > ! > Searching the cause I find change 1b5be7204eaeeaf aka D19488 > ! > > ! > This doesn't work, because the firewall is not yet present. This is > ! > ! Given you are talking firewall, I assume you are using vnet jails? > > Yes. > > ! And given you are talking ipfw I assume your default policy is deny > ! and not accept? > > Yes. > > ! And given rtsol runs I assume you have IPv6 configured and in use? > > Yes. Here is how I do it: > https://daemon.contact/ankh/articles/X3OyjgTpuv > > ! The same issue then should also happen in your base system on boot? > > No. The base system does (second level) prefix delegation and has > ipv6_gateway_enable="YES" and rtsold_enable="NO" and is not affected. > > There is one vnet jail intended as VPN server, which also has these > parameters in rc.conf and is also not affected. > > (I did not yet bother to figure out why, The shell code run from > rc.d/netif is a bit lenghty...) > > ! > happening in rc.d/netif, and that must run before rc.d/ipfw in any > ! > case, because the firewall needs to see the netifs. > ! > ! I thought ipfw could log deal with interfaces coming and going? > > Maybe it can, but then modifying the rc.d logic so to get "ipfw" run > before "netif" - that does likely open a box of worms. > > Furthermore, I do use ipfw as a genuine rerouting+filtering > framework, and that logic is entirely based on the interfaces; all > rules belong to exactly two interfaces. Here is a short abstract > of the idea: > https://forums.freebsd.org/threads/ipfw-or-pf.46706/post-561760 > > > cheerio, > PMc > > This may be irrelevant, but updating to the stable branch is not recommended as it is not regularly tested. Updating to 13.0-Release and then to stable is less likely to be problematic. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 --00000000000046dd3e05db7bb37a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Tue, Mar 29, 2022 at 5:10 PM= Peter <pmc@citylink.dino= ex.sub.org> wrote:

Hello Bjoern,

=C2=A0 thanks much for the quick reply!

On Tue, Mar 29, 2022 at 10:04:11PM +0000, Bjoern A. Zeeb wrote:
! On Tue, 29 Mar 2022, Peter wrote:
!
! Hi,
!
! I am a bit puzzled as after two years you are the first one to report
! that problem to my knowledge for either base system or jails.

This is what greatly wonders me, too. So I was stronly thinking
that I am doing something wrong or unusual. But I cannot figure
it out, it just seems that the detrimental effect of the change
cannot be avoided (e.g. "service jail start" takes quite long now= -
there's a lot of them).

! >=C2=A0 after upgrading 12.3 to stable/13, I am seeing these
! > errors in all my jails:
! >
! > > Additional TCP/IP options: log_in_vain=3D1.
! > > ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib=
! >=C2=A0 =C2=A0 =C2=A0/usr/local/lib/c cmpat/pkg /usr/local/lib/compat/= pkg
! > > 32-bit compatibility ldconfig path:
! > > rtsol: sendmsg on nrail1l: Permission denied
! > > rtsol: sendmsg on nrail1l: Permission denied
! > > rtsol: sendmsg on nrail1l: Permission denied
! > > Starting Network: lo0 nrail1l.
!
! Can you give us a full startup log?

It's the above, right from the beginning, and then follows:

> lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16= 384
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0options=3D680003<RXCSUM,TXCSUM,LIN= KSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0inet 127.0.0.1 netmask 0xff000000
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0inet6 ::1 prefixlen 128
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0inet6 fe80::1%lo0 prefixlen 64 scopei= d 0x1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0groups: lo
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0nd6 options=3D21<PERFORMNUD,AUTO_L= INKLOCAL>
> nrail1l: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> me= tric 0 mtu 1500
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0options=3D28<VLAN_MTU,JUMBO_MTU>= ;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ether 06:1d:92:01:01:0a
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0hwaddr 58:9c:fc:10:28:71
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0inet ************* netmask **********= broadcast *************
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0inet6 fe80::41d:92ff:fe01:10a%nrail1l= prefixlen 64 scopeid 0x2
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0inet6 fd00:************ prefixlen 120=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0media: Ethernet autoselect (1000baseT= <full-duplex>)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0status: active
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0nd6 options=3D23<PERFORMNUD,ACCEPT= _RTADV,AUTO_LINKLOCAL>
> Starting rtsold.
> add host 127.0.0.1: gateway lo0 fib 0: route already in table
> add net default: gateway *************
> Additional inet routing options: log ICMP redirect=3DYES.
> add host ::1: gateway lo0 fib 0: route already in table
> add net fe80::: gateway ::1
> add net ff02::: gateway ::1
> add net ::ffff:0.0.0.0: gateway ::1
> add net ::0.0.0.0: gateway ::1
> add net default: gateway fd00:*************
> Flushed all rules.
> Firewall rules loaded.
> Firewall logging pseudo-interface (ipfw0) created.
> Creating and/or trimming log files.
> Updating /var/run/os-release done.
> Clearing /tmp (X related).
> Updating motd:.
> Starting syslogd.
> Starting rapp.
> Starting cron.
> Starting sendmail.
> Starting sendmail_msp_queue.
> Performing sanity check on sshd configuration.
> Starting sshd.
>
> Wed Mar 30 00:52:15 CEST 2022

! > Searching the cause I find change=C2=A0 1b5be7204eaeeaf=C2=A0 aka=C2= =A0 D19488
! >
! > This doesn't work, because the firewall is not yet present. This= is
!
! Given you are talking firewall, I assume you are using vnet jails?

Yes.

! And given you are talking ipfw I assume your default policy is deny
! and not accept?

Yes.

! And given rtsol runs I assume you have IPv6 configured and in use?

Yes. Here is how I do it:
https://daemon.contact/ankh/articles/X3OyjgTpuv
! The same issue then should also happen in your base system on boot?

No. The base system does (second level) prefix delegation and has
ipv6_gateway_enable=3D"YES" and rtsold_enable=3D"NO" an= d is not affected.

There is one vnet jail intended as VPN server, which also has these
parameters in rc.conf and is also not affected.

(I did not yet bother to figure out why, The shell code run from
rc.d/netif is a bit lenghty...)

! > happening in rc.d/netif, and that must run before rc.d/ipfw in any ! > case, because the firewall needs to see the netifs.
!
! I thought ipfw could log deal with interfaces coming and going?

Maybe it can, but then modifying the rc.d logic so to get "ipfw" = run
before "netif" - that does likely open a box of worms.

Furthermore, I do use ipfw as a genuine rerouting+filtering
framework, and that logic is entirely based on the interfaces; all
rules belong to exactly two interfaces. Here is a short abstract
of the idea:
https://forums.freebsd.org/threads/ip= fw-or-pf.46706/post-561760


cheerio,
PMc

This may be irrelevant, but updating to the st= able branch is not recommended as it is not regularly tested. Updating to 1= 3.0-Release and then to stable is less likely to be problematic.
= --
Kevin Oberman, Pa= rt time kid herder and retired Network Engineer
E-mail: rkoberman@gmail.com
<= div>PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
--00000000000046dd3e05db7bb37a--