From nobody Fri Jul 21 09:11:21 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R6kM053t7z4dYrT for ; Fri, 21 Jul 2023 09:11:24 +0000 (UTC) (envelope-from peo@nethead.se) Received: from ns1.nethead.se (ns1.nethead.se [5.150.237.139]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) client-signature ECDSA (P-384)) (Client CN "ns1.nethead.se", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R6kLz5PHvz4NVk for ; Fri, 21 Jul 2023 09:11:23 +0000 (UTC) (envelope-from peo@nethead.se) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=nethead.se header.s=NETHEADSE header.b=jAqZ+7uI; spf=pass (mx1.freebsd.org: domain of peo@nethead.se designates 5.150.237.139 as permitted sender) smtp.mailfrom=peo@nethead.se; dmarc=pass (policy=none) header.from=nethead.se X-Virus-Scanned: amavisd-new at Nethead AB DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=nethead.se; s=NETHEADSE; t=1689930682; bh=9tagTOYm3CNsZ9huWscVpJjcYOotjrV0orQa1Qyar04=; h=Date:To:From:Subject; b=jAqZ+7uI97Q0mS2ZxhtUM/2DpMd5x00awvAfrCcTDOl+/Y8IACZIQiQ95iHB2TPvB w6JKoA6TxAOdIu+B8V2M2pz7+hLWLUqK8h1MaE8VfxN7eLge39ou910MReOz+kyl2/ zeUQrag6pT9/JHvz0fR/C/B7BpEFBx8SQONG7xuk= Message-ID: Date: Fri, 21 Jul 2023 11:11:21 +0200 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.10.1 To: stable@freebsd.org Content-Language: en-US From: Per olof Ljungmark Subject: Source upgrade 12.4 - 13.2 installworld jails Function not implemented Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.996]; DMARC_POLICY_ALLOW(-0.50)[nethead.se,none]; R_SPF_ALLOW(-0.20)[+ip4:5.150.237.139:c]; R_DKIM_ALLOW(-0.20)[nethead.se:s=NETHEADSE]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[stable@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ZERO(0.00)[0]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:8473, ipnet:5.150.192.0/18, country:SE]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[nethead.se:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4R6kLz5PHvz4NVk X-Spamd-Bar: --- Hi all, Looking for advice what to do next. This is a production server and we can only afford short downtimes. Kernel is (was) FreeBSD 12.4-STABLE stable/12-c3bbba00e GENERIC then Fetched 13.2-STABLE sources yesterday morning. Did, make buildworld make kernel make installworld etcupdate applied in between Did not reboot between kernel and world because wanted to save time, perhaps stupid. Now that I want to upgrade the jails, make installworld DESTDIR=$jail It stumbles on the first cp from the temporary dir mkdir -p /tmp/install.12dw9HKg progs=$(for prog in [ awk cap_mkdb cat chflags chmod chown cmp cp date echo egrep find grep id install ln make mkdir mtree mv pwd_mkdb rm sed services_mkdb sh sort strip sysctl test true uname wc tzsetup makewhatis ; do if progpath=`env PATH=/usr/obj/usr/src/amd64.amd64/tmp/bin:/usr/obj/usr/src/amd64.amd64/tmp/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin which $prog`; then echo $progpath; else echo "Required tool $prog not found in PATH ("/usr/obj/usr/src/amd64.amd64/tmp/bin:/usr/obj/usr/src/amd64.amd64/tmp/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin")." >&2; exit 1; fi; done); if [ -z "" ] ; then libs=$(ldd -f "%o %p\n" -f "%o %p\n" $progs 2>/dev/null | sort -u | grep -Ev '\[.*]' | while read line; do set -- $line; if [ "$2 $3" != "not found" ]; then echo $2; else echo "Required library $1 not found." >&2; exit 1; fi; done); fi; cp $libs $progs /tmp/install.12dw9HKg cp: /usr/lib/libbz2.so.4: Function not implemented cp: /lib/libc.so.7: Function not implemented ... Goes on for every item. So, any recommendation on what to do next? The host and jails still run as they should with the 12.4 kernel. Reboot? Fetch new 12.4 sources, build and reboot and start over? Other suggestions? Thank you in advance, Per From eugen@grosbein.net Fri Jul 21 09:51:56 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R6lG30sDVz4ntsc for ; Fri, 21 Jul 2023 09:52:11 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R6lG25K3Tz3NnK for ; Fri, 21 Jul 2023 09:52:10 +0000 (UTC) (envelope-from eugen@grosbein.net) Authentication-Results: mx1.freebsd.org; none Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.17.1/8.17.1) with ESMTPS id 36L9q0jw057903 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Fri, 21 Jul 2023 09:52:01 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: peo@nethead.se Received: from [10.58.0.11] (dadvw [10.58.0.11] (may be forged)) by eg.sd.rdtc.ru (8.17.1/8.17.1) with ESMTPS id 36L9q0Sr079184 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 21 Jul 2023 16:52:00 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Source upgrade 12.4 - 13.2 installworld jails Function not implemented To: Per olof Ljungmark , stable@freebsd.org References: From: Eugene Grosbein Message-ID: Date: Fri, 21 Jul 2023 16:51:56 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,SHORTCIRCUIT autolearn=disabled version=3.4.6 X-Spam-Report: * -0.0 SHORTCIRCUIT No description available. * -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on hz.grosbein.net X-Rspamd-Queue-Id: 4R6lG25K3Tz3NnK X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/32, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated 21.07.2023 16:11, Per olof Ljungmark wrote: > Hi all, > > Looking for advice what to do next. This is a production server and we can only afford short downtimes. > > Kernel is (was) > FreeBSD 12.4-STABLE stable/12-c3bbba00e GENERIC > then > Fetched 13.2-STABLE sources yesterday morning. > > Did, > make buildworld > make kernel > make installworld > > etcupdate applied in between > > Did not reboot between kernel and world because wanted to save time, perhaps stupid. You installed world on host, now you have new binaries that NEED new kernel. Reboot the host ASAP. > Now that I want to upgrade the jails, > make installworld DESTDIR=$jail That should be done after new kernel booted. From nobody Fri Jul 21 11:55:36 2023 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R6p0X6BRkz4p7BZ for ; Fri, 21 Jul 2023 11:55:40 +0000 (UTC) (envelope-from peo@nethead.se) Received: from ns1.nethead.se (ns1.nethead.se [5.150.237.139]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "ns1.nethead.se", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R6p0X44CFz4HXS for ; Fri, 21 Jul 2023 11:55:40 +0000 (UTC) (envelope-from peo@nethead.se) Authentication-Results: mx1.freebsd.org; none X-Virus-Scanned: amavisd-new at Nethead AB DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=nethead.se; s=NETHEADSE; t=1689940538; bh=ze/Qe565+NF9KmQDdkZnNXiRQ9FWSRCD02EwuHT9cG8=; h=Date:Subject:To:References:From:In-Reply-To; b=jhGryHVdvvqseEoS9ljgGNfF0Td1b1h9QetvTWnAEq6Ufh2pUi/H5jjrf7yOAnu5p ABbsaV5vcvf1BMpPItMdUiwhE8qo4LwF1goa+oCZPi1IwmYA45sxAIZGsyjjQcEXtT VMeBbEHFNM+dqbC2ChQ9kG2Y2Eq+B7llZNHzsvZY= Message-ID: <43783795-93bb-14c0-7a43-69a3c1a2f42f@nethead.se> Date: Fri, 21 Jul 2023 13:55:36 +0200 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.10.1 Subject: Re: Source upgrade 12.4 - 13.2 installworld jails Function not implemented Content-Language: en-US To: Eugene Grosbein , stable@freebsd.org References: From: Per olof Ljungmark In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4R6p0X44CFz4HXS X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8473, ipnet:5.150.192.0/18, country:SE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated On 2023-07-21 11:51, Eugene Grosbein wrote: > 21.07.2023 16:11, Per olof Ljungmark wrote: > >> Hi all, >> >> Looking for advice what to do next. This is a production server and we can only afford short downtimes. >> >> Kernel is (was) >> FreeBSD 12.4-STABLE stable/12-c3bbba00e GENERIC >> then >> Fetched 13.2-STABLE sources yesterday morning. >> >> Did, >> make buildworld >> make kernel >> make installworld >> >> etcupdate applied in between >> >> Did not reboot between kernel and world because wanted to save time, perhaps stupid. > > You installed world on host, now you have new binaries that NEED new kernel. Reboot the host ASAP. > >> Now that I want to upgrade the jails, >> make installworld DESTDIR=$jail > > That should be done after new kernel booted. > Thank you Eugene! From nobody Fri Jul 21 13:40:05 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R6rKR4BXqz4njV7 for ; Fri, 21 Jul 2023 13:40:27 +0000 (UTC) (envelope-from ruben@verweg.com) Received: from erg.verweg.com (erg.verweg.com [IPv6:2a02:898:96::5e8e:f508]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4R6rKQ4d0bz3lmy for ; Fri, 21 Jul 2023 13:40:26 +0000 (UTC) (envelope-from ruben@verweg.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=verweg.com header.s=verweg header.b=r16Rt4Oz; spf=pass (mx1.freebsd.org: domain of ruben@verweg.com designates 2a02:898:96::5e8e:f508 as permitted sender) smtp.mailfrom=ruben@verweg.com; dmarc=pass (policy=reject) header.from=verweg.com DKIM-Filter: OpenDKIM Filter v2.10.3 erg.verweg.com (unknown-jobid) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=verweg.com; s=verweg; t=1689946817; bh=+BotzxzmQP+fYcAhI3YEqMCA19sCf4ePTN5nd2s+xeg=; h=From:Date:Subject:To:From; b=r16Rt4OzSq6UaNmjGvyFmOs2x6gjaBP6aeyRYgyaWjUWzQ0bfCu9XFOGrggjhzY40 htaUuDfJVkIVC6bpkxjqFAgjBSBVd521obirE7lgQcIur5g5EPKD7qYiHHtOaKgFVJ X+wKBSlG6IVJDRwQAPdwSqhCl08OuEJhLjNE8uzr4HZZHrPKAGKjf1AGS6bN/Y/cpR DuWTT2HlwZQ7Vl470nmyl8x7MVFFMXthphpX24O5zvwE9NQBbPSOc/hD//dTDSBHYQ Vi49cTlNeiMH73NzZkonOpv2FZzvLEn/I31/azHwjggEG+N6rYlRep/Qb2riMc+GeU OJt1qDvfeRWtA== Received: from smtpclient.apple (2a10-3781-03e9.connected.by.freedominter.net [IPv6:2a10:3781:3e9:1:7889:5cf6:1265:398d] (may be forged)) (authenticated bits=0) by erg.verweg.com (8.17.1/8.17.1) with ESMTPSA id 36LDeFnl094785 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT) for ; Fri, 21 Jul 2023 13:40:16 GMT (envelope-from ruben@verweg.com) X-Authentication-Warning: erg.verweg.com: Host 2a10-3781-03e9.connected.by.freedominter.net [IPv6:2a10:3781:3e9:1:7889:5cf6:1265:398d] (may be forged) claimed to be smtpclient.apple Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Ruben van Staveren List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (1.0) Date: Fri, 21 Jul 2023 15:40:05 +0200 Subject: Random panics (page fault). ktls_decrypt might be involved? Message-Id: <8E891B5E-3DAD-45A7-9208-C07FB32A807E@verweg.com> To: freebsd-stable@freebsd.org X-Mailer: iPad Mail (20F75) X-Spamd-Result: default: False [-1.98 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.982]; MV_CASE(0.50)[]; SUBJECT_ENDS_SPACES(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[verweg.com,reject]; R_DKIM_ALLOW(-0.20)[verweg.com:s=verweg]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_DN_NONE(0.00)[]; HAS_XAW(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[verweg.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:8283, ipnet:2a02:898::/32, country:NL]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4R6rKQ4d0bz3lmy X-Spamd-Bar: - Hi list, I=E2=80=99m getting some page fault panics recently, but if I look in the cr= ashinfo dumps there seems always be a ktls_decrypt present. I=E2=80=99m usin= g kernel TLS with dnsdist. Just checking if this is the right assumption before I=E2=80=99m going to fi= le a pr Best Regards, Ruben Unread portion of the kernel message buffer: processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 2 (thr_9) trap number =3D 12 panic: page fault cpuid =3D 4 time =3D 1689945150 KDB: stack backtrace: #0 0xffffffff80c53d95 at kdb_backtrace+0x65 #1 0xffffffff80c06711 at vpanic+0x151 #2 0xffffffff80c065b3 at panic+0x43 #3 0xffffffff810b1fa7 at trap_fatal+0x387 #4 0xffffffff810b1fff at trap_pfault+0x4f #5 0xffffffff81088e48 at calltrap+0x8 #6 0xffffffff80bc2f9e at fork_exit+0x7e #7 0xffffffff81089ebe at fork_trampoline+0xe Uptime: 6h29m8s Dumping 18598 out of 130332 MB: (CTRL-C to abort) ..1%..11%..21%..31%..41%..= 51%..61%..71%..81%..91% =E2=80=A6. __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(struc= t pcpu, (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdown.= c:396 #2 0xffffffff80c062da in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:484 #3 0xffffffff80c0677e in vpanic (fmt=3D,=20 ap=3Dap@entry=3D0xfffffe017d99cc50) at /usr/src/sys/kern/kern_shutdown.c= :923 #4 0xffffffff80c065b3 in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:847 #5 0xffffffff810b1fa7 in trap_fatal (frame=3D0xfffffe017d99cd40, eva=3D81) at /usr/src/sys/amd64/amd64/trap.c:942 #6 0xffffffff810b1fff in trap_pfault (frame=3D0xfffffe017d99cd40,=20 usermode=3Dfalse, signo=3D, ucode=3D) at /usr/src/sys/amd64/amd64/trap.c:761 #7 #8 0xffffffff80c97b88 in ktls_decrypt (so=3D0xfffff801e8313000) at /usr/src/sys/kern/uipc_ktls.c:1913 #9 ktls_work_thread (ctx=3Dctx@entry=3D0xfffff801037ebc80) at /usr/src/sys/kern/uipc_ktls.c:2407 #10 0xffffffff80bc2f9e in fork_exit ( callout=3D0xffffffff80c97950 , arg=3D0xfffff801037ebc8= 0,=20 frame=3D0xfffffe017d99cf40) at /usr/src/sys/kern/kern_fork.c:1093 #11 #12 0x171e5d4cc9e9e0a3 in ?? () Backtrace stopped: Cannot access memory at address 0x430c6a4638d29dfb (kgdb)=20 ------------------------------------------------------------------------ =E2=80=A6. vmstat -z ITEM SIZE LIMIT USED FREE REQ FAILSLEEP X= DOMAIN =E2=80=A6 ktls_session: 192, 0, 1177, 497, 2352, 0, 0, 0= =E2=80=A6=