From owner-freebsd-net@freebsd.org Mon May 3 18:04:34 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4A54B639523 for ; Mon, 3 May 2021 18:04:34 +0000 (UTC) (envelope-from schmiedgen@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FYrTY18YNz3Kql for ; Mon, 3 May 2021 18:04:32 +0000 (UTC) (envelope-from schmiedgen@gmx.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1620065071; bh=i8IaWUxmG/lHMVQkGfWGPvPkLQgVkroGdD66mezdtlQ=; h=X-UI-Sender-Class:To:From:Subject:Date; b=g4PeOhSiDOj2GtHEK164MBASh65ODtcKJrSae65ZiAg4mGnYaEvBSes9qN00ZxZCk sVnZ9+j/x8oJZgqkKy3y0QzqG9iEkrQvGKeHkuq2ovl6MniSeW20LGLSkrFRi3kQ/o 4boORLOxdNrxO4V5Hf5GKScK6x2tPayfFICEO2FE= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from [192.168.10.5] ([62.246.110.10]) by mail.gmx.net (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MqJmF-1l9jkz4BZj-00nND8 for ; Mon, 03 May 2021 20:04:31 +0200 To: freebsd-net@freebsd.org From: Michael Schmiedgen Subject: page fault while in kernel mode - after upgrade from 12.2 to 13.0 Message-ID: Date: Mon, 3 May 2021 20:04:30 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.10.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:8IKsBsXjsOIWm+Ae7B4KqzFYhC873ALiDBszXxzwwjlAYpWpPIq LuHlw5oNYHhW5kkekdO0oiFfwndBkrAUJHaFPkYO+IEnG9AkpFjIvjkiv3QB581Kc6W9F5z HUKXWPO/3LjPlxqmuzkbVrq8vjZ4pGB0HvZUs6QQ2yOo91xBj00o4YQnZsHFl/BiWurQw2d gQJgDfdv+nA+Sfg3wrgIQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:vx/rXRuyRvs=:bAqlH6aEs7dfnejaaISkNp YvmH8FQb+IROQou/qOzttLmB1FndIBEIMj8kjO4UmJNPP5nhdbKco2+QzzQA7OzpT6JtHtrko OLMeDP/cXnTGpW7+BURSNJmW94AtCh32sotrAbAS31JUzsVqsgcYNf1v5wmcalB4kh8loLvm+ j/6wVtYsfbfzmYS3VSsuYk85gQQ2Td86UaJr5wtmgWFcRBthu3xkdthJSv1RJCTaaX0dUTDlJ 9XFzBTdXPGsKPzs14WWz5RKPDkQHmxBFTfYljU1U/TfXso9cSOUGVy8lg4I4ls2O+Vqj3mZ3l GRpgFUcHMs8WNokfCF5zuzB4xuGkpmOfTdEtmKSZFdJSORXtSpHmAOKVzg9Apx9g/+cnFvurB AGF0dZ+pXZfZ0FJMN+I8NNZiQw7GmL6A6wlaaGd9KdG3y1Q4NNNAMyMBc1qRcS8e881m14vOL l75DyLap7vU/GV8UXqamicKDc5PUqGnDDDO1qd3y3GDm0RQD1tEjIAXULEiX4+ogd0fZ6YMlC TGrowpR71ZQYD08fdiCYf6BfNW29OjPmQcJa741vny5bkKuhltc2Bes7omfkeWf47+AxKJfU9 wl+SsvW18XPLTMB3oKxIZFX95tuztxtAxeC5sqjquPNAnKMizjKfsNRTd364WeSZ+Wue4m3lj lelUIpsjOUSFKqU0wbBpGPkstvfd36lX2xv2J9pJ92by6r88zuSX/09N7Y/FGIURX2cU2BwMi 8LKG3eEyi6RVlvbPBuT5zNp+mXz0ebj+DGisvRnmKfzAyl4fWDZR2OF3TXoIUsiLY3ZpF7npU a9z4xghOn0Nxl8pqNkReufThNm5ZUrFEm8ZQWGwd+gdw/2PukZYWGm8hcnBiLCTvwCm+iG+7s N2/vtPbzKNOerk6QN7E9NkDvMwkCjXbFdLhG5VJ3oa2RKzj00HU+jqJb4HTx02v8LnVeKYDJ1 gaQpLJiZBaFYO9RXgc308XIEtqOgaKSFWsAskoj5F1KtwRYiodDBjHIyw2gQKaWtnIxIT7ROO fKPcAif+w8SUqoj3wXPIn2TtOeFdMvmrvu5uRA9I5XjyikBUpk89YZIJ/yAl4MCbDQle2zJZ6 eVrCCMHmu20JPjjH7X/SFBluJHvXI6vybMSRd4PD3ttzzVrWLtYbUOeL/zQTyWCOesBDgNdH6 Du7PCRca0fJW1e2cl8r1MWrYf9Mq4i2KvhIHCWzWvZGxlC9T10jVW4K6uF1MR8gsQ+hq4= X-Rspamd-Queue-Id: 4FYrTY18YNz3Kql X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmx.net header.s=badeba3b8450 header.b=g4PeOhSi; dmarc=pass (policy=none) header.from=gmx.net; spf=pass (mx1.freebsd.org: domain of schmiedgen@gmx.net designates 212.227.17.22 as permitted sender) smtp.mailfrom=schmiedgen@gmx.net X-Spamd-Result: default: False [-4.10 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_FROM(0.00)[gmx.net]; R_SPF_ALLOW(-0.20)[+ip4:212.227.17.0/27]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[gmx.net:+]; DMARC_POLICY_ALLOW(-0.50)[gmx.net,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_IN_DNSWL_LOW(-0.10)[212.227.17.22:from]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmx.net]; MID_RHS_MATCH_FROM(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[212.227.17.22:from]; DWL_DNSWL_NONE(0.00)[gmx.net:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmx.net:s=badeba3b8450]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[212.227.17.22:from:127.0.2.255]; RWL_MAILSPIKE_POSSIBLE(0.00)[212.227.17.22:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 May 2021 18:04:34 -0000 Hi List, if I start a Samba jail, after a few seconds the system crashes. Very repr= oducible. System has ~10 jails and 3 bhyve VMs. Dell server, Xeon E3-1240, 64GB RAM,= 3 way mirror ZFS. It also occurs a few seconds after I start a phone call using the SIP VM o= f that machine, very strange. I got some log messages suggesting raising somaxconn, so I did kern.ipc.somaxconn=3D4096 in sysctl.conf Below some debug information, please let me know if I should provide furth= er information. Should I open a bug or something? Thank you very much! Michael Fatal trap 12: page fault while in kernel mode cpuid =3D 0; apic id =3D 00 fault virtual address =3D 0x0 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff80ca52c0 stack pointer =3D 0x28:0xfffffe019d039650 frame pointer =3D 0x28:0xfffffe019d039690 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 649 (devd) trap number =3D 12 panic: page fault cpuid =3D 0 time =3D 1620061253 KDB: stack backtrace: #0 0xffffffff80c57345 at kdb_backtrace+0x65 #1 0xffffffff80c09d21 at vpanic+0x181 #2 0xffffffff80c09b93 at panic+0x43 #3 0xffffffff8108b187 at trap_fatal+0x387 #4 0xffffffff8108b1df at trap_pfault+0x4f #5 0xffffffff8108a83d at trap+0x27d #6 0xffffffff810617a8 at calltrap+0x8 #7 0xffffffff80ca51c3 at sbappendaddr_locked+0x93 #8 0xffffffff80cb437a at uipc_send+0x73a #9 0xffffffff80ca9053 at sosend_generic+0x633 #10 0xffffffff80ca94e0 at sosend+0x50 #11 0xffffffff80caff2e at kern_sendit+0x20e #12 0xffffffff80cb032b at sendit+0x1db #13 0xffffffff80cb013d at sys_sendto+0x4d #14 0xffffffff8108ba8c at amd64_syscall+0x10c #15 0xffffffff810620ce at fast_syscall_common+0xf8 Uptime: 2m2s Dumping 2373 out of 65454 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%.= .91% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(str= uct pcpu, (kgdb) list *0xffffffff80ca52c0 0xffffffff80ca52c0 is in sbappendaddr_locked_internal (/usr/src/sys/kern/u= ipc_sockbuf.c:1169). 1164 if (ctrl_last) 1165 ctrl_last->m_next =3D m0; /* concatenate data to c= ontrol */ 1166 else 1167 control =3D m0; 1168 m->m_next =3D control; 1169 for (n =3D m; n->m_next !=3D NULL; n =3D n->m_next) 1170 sballoc(sb, n); 1171 sballoc(sb, n); 1172 nlast =3D n; 1173 SBLINKRECORD(sb, m); (kgdb) backtrace #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdow= n.c:399 #2 0xffffffff80c09916 in kern_reboot (howto=3D260) at /usr/src/sys/kern/k= ern_shutdown.c:486 #3 0xffffffff80c09d90 in vpanic (fmt=3D, ap=3D) at /usr/src/sys/kern/kern_shutdown.c:919 #4 0xffffffff80c09b93 in panic (fmt=3D) at /usr/src/sys/kern= /kern_shutdown.c:843 #5 0xffffffff8108b187 in trap_fatal (frame=3D0xfffffe019d039590, eva=3D0)= at /usr/src/sys/amd64/amd64/trap.c:915 #6 0xffffffff8108b1df in trap_pfault (frame=3Dframe@entry=3D0xfffffe019d0= 39590, usermode=3Dfalse, signo=3D, signo@entry=3D0x0, ucode= =3D, ucode@entry=3D0x0) at /usr/src/sys/amd64/amd64/trap.c:732 #7 0xffffffff8108a83d in trap (frame=3D0xfffffe019d039590) at /usr/src/sy= s/amd64/amd64/trap.c:398 #8 #9 sbappendaddr_locked_internal (sb=3Dsb@entry=3D0xfffff800447ef4f8, asa= =3Dasa@entry=3D0xffffffff815cde60 , m0=3D, m0@entry=3D0xfffff8008b186500, control=3D0xfffff8008b186500, control@entry=3D0x0, ctrl_last=3D) at /usr/src/sys/ker= n/uipc_sockbuf.c:1169 #10 0xffffffff80ca51c3 in sbappendaddr_locked (sb=3Dsb@entry=3D0xfffff8004= 47ef4f8, asa=3Dasa@entry=3D0xffffffff815cde60 , m0=3Dm0@entry=3D0xfffff8008b186500, control=3D0x0) at /usr/src/sys/kern/uipc_sockbuf.c:1205 #11 0xffffffff80cb437a in uipc_send (so=3D, flags=3D0, m=3D= 0xfffff8008b186500, nam=3D, control=3D0x10, td=3D) at /usr/src/sys/kern/uipc_usrreq.c:1056 #12 0xffffffff80ca9053 in sosend_generic (so=3D0xfffff800444abb10, addr=3D= 0x0, uio=3D, top=3D0xfffff8008b186500, control=3D0x0, flags= =3D0, td=3D0xfffffe0165ddc500) at /usr/src/sys/kern/uipc_socket.c:1755 #13 0xffffffff80ca94e0 in sosend (so=3D0x100, so@entry=3D0xfffff800444abb1= 0, addr=3D0xb5ea5000, uio=3D0xfffff8008b186500, uio@entry=3D0xfffffe019d03= 9898, top=3D0x10, top@entry=3D0x0, control=3Dcontrol@entry=3D0x0, flags=3D272, flags@entry=3D0, td=3D0xf= ffffe0165ddc500) at /usr/src/sys/kern/uipc_socket.c:1810 #14 0xffffffff80caff2e in kern_sendit (td=3D, td@entry=3D0x= fffffe0165ddc500, s=3D8, mp=3D, mp@entry=3D0xfffffe019d0399= 80, flags=3D0, control=3D0x0, segflg=3Dsegflg@entry=3DUIO_USERSPACE) at /usr/src/sys/kern/uipc_sysc= alls.c:798 #15 0xffffffff80cb032b in sendit (td=3D0xfffffe0165ddc500, s=3D-1242935296= , mp=3Dmp@entry=3D0xfffffe019d039980, flags=3D16) at /usr/src/sys/kern/uip= c_syscalls.c:723 #16 0xffffffff80cb013d in sys_sendto (td=3D0x100, uap=3D) a= t /usr/src/sys/kern/uipc_syscalls.c:841 #17 0xffffffff8108ba8c in syscallenter (td=3D0xfffffe0165ddc500) at /usr/s= rc/sys/amd64/amd64/../../kern/subr_syscall.c:189 #18 amd64_syscall (td=3D0xfffffe0165ddc500, traced=3D0) at /usr/src/sys/am= d64/amd64/trap.c:1156 #19 #20 0x00000000002858ea in ?? ()