From nobody Fri Dec 17 19:36:33 2021 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A46F01900F85 for ; Fri, 17 Dec 2021 19:36:37 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JFzkX6gWjz3Ftv; Fri, 17 Dec 2021 19:36:36 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qk1-x736.google.com with SMTP id t83so3194315qke.8; Fri, 17 Dec 2021 11:36:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=Z7nVJyGqdZJC4dn4UHM7SsY2UMzAWxgifJRZ0mUQBuY=; b=kChppqAEvnrLJkCv18YeaOudvmQtV0HWolcVuR5s5ctHougmGq56suu4ySZuw8vMyV 4cYOU7tXFSsI0y2YC4EfwU6LJ5SIKl27OMIx78HFwEFouUIJ9Ggvart5gCDAXytmaiAU 8Icvuz1MzsyvV9CAufJOMghvJKiSFqJpc5pGHsH7nwjqlrfcL4p2JoY3UWE+Zx9BNw7g N8nN/qGah8ppGwScv9lNxAQfp3nUn3UPsbr/0xj6845teyz5pQ2Jsrri0I2lp2Ily2lY vjqcFcgPlcDAOLGXtPqHkicuhUI3kDPL1YV3Q63F/kBwPn0r+Y2NLi3cxSUtiIJEUoZK 1RZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=Z7nVJyGqdZJC4dn4UHM7SsY2UMzAWxgifJRZ0mUQBuY=; b=XNwcyePOtxkLk9RAkOvx97unNgRDjW2Xj6/FS3Cdx6MfcO4t86ra2kSltc+RMffzEj ets2eHw1yjfkN7mpz9R87JqSYaryah0wL7cREheDyhVZ/JBAHNeRmGk8m7L3kNDlNZvf bo4yWr4CMtTQw72AsKN2j9eJNC4/CUlIJ6HiD7Mzo3ty+Dvq6zy7IbQNrucSCuawtmBc Xi+rXyPXIa+z521ViRi++aJ1MuifVqqAr3LoSp6wo7K/ue+efVQ1HAnKoTO79DTTj70m uQmPeyc+v4OlF8cMd667yNLCoKUIUCm9CLtpZihMLYshVRWxnpGIew4jESXiqLW41ih2 6SWw== X-Gm-Message-State: AOAM531lKDOPWysrrCXDuaKd6fQmziEvBNs6Oys7dmxqhR5p6i6921AL sWioFW/eyT4XtfFsvyMs561rlID+rtw= X-Google-Smtp-Source: ABdhPJyAplGpWae/lfTxRAHSppIrWW+871/vfPNUjxVw+B9uFoJjRX7XPjib6vgJiDKw2tKhxrMnyA== X-Received: by 2002:a37:745:: with SMTP id 66mr2796859qkh.11.1639769796407; Fri, 17 Dec 2021 11:36:36 -0800 (PST) Received: from nuc ([142.126.186.191]) by smtp.gmail.com with ESMTPSA id x4sm7931807qtw.44.2021.12.17.11.36.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Dec 2021 11:36:35 -0800 (PST) Date: Fri, 17 Dec 2021 14:36:33 -0500 From: Mark Johnston To: Larry Rosenman Cc: Alexander Motin , Freebsd current Subject: Re: Panic: Page Fault in Kernel: Yesterday's CURRENT Message-ID: References: <3d1b5249a2c51670de496fad9e8b054c@lerctr.org> <9852ae04-6dd0-1cd4-13fe-e97c68e71b37@FreeBSD.org> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 4JFzkX6gWjz3Ftv X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=kChppqAE; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::736 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-0.08 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_NA(0.00)[freebsd.org]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; NEURAL_HAM_SHORT(-0.38)[-0.383]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::736:from]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N On Fri, Dec 10, 2021 at 10:43:19AM -0600, Larry Rosenman wrote: > 14-2021_12_07-1217 - - 1.87G 2021-12-07 12:17 > 14-2021_12_09-1957 NR / 121G 2021-12-09 19:57 > > If that's any help I can't tell what this is saying. A kernel built on the 7th does not crash, or...? Which revision did you update from before you started seeing crashes? >From a kgdb session it'd be useful to see output from (kgdb) frame 8 (kgdb) p/x *tmp to start. > On 12/10/2021 10:36 am, Alexander Motin wrote: > > Hi Larry, > > > > This looks like some use-after-free or otherwise corrupted callout > > structure. Unfortunately the backtrace does not tell what was the > > callout. When was the previous update to look what could change? > > > > On 10.12.2021 11:24, Larry Rosenman wrote: > >> FreeBSD borg.lerctr.org 14.0-CURRENT FreeBSD 14.0-CURRENT #15 > >> main-n251537-ab639f2398b: Thu Dec  9 19:45:37 CST 2021     > >> root@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL  > >> amd64 > >> > >> VMCORE *IS* available. > >> > >> > >> > >> > >> Unread portion of the kernel message buffer: > >> kernel trap 12 with interrupts disabled > >> > >> > >> Fatal trap 12: page fault while in kernel mode > >> cpuid = 0; apic id = 20 > >> fault virtual address   = 0x0 > >> fault code              = supervisor write data, page not present > >> instruction pointer     = 0x20:0xffffffff804e0db4 > >> stack pointer           = 0x0:0xfffffe0434de4e10 > >> frame pointer           = 0x0:0xfffffe0434de4e70 > >> code segment            = base 0x0, limit 0xfffff, type 0x1b > >>                         = DPL 0, pres 1, long 1, def32 0, gran 1 > >> processor eflags        = resume, IOPL = 0 > >> current process         = 82990 (c++) > >> trap number             = 12 > >> panic: page fault > >> cpuid = 0 > >> time = 1639111198 > >> KDB: stack backtrace: > >> #0 0xffffffff8050fc95 at kdb_backtrace+0x65 > >> #1 0xffffffff804c468f at vpanic+0x17f > >> #2 0xffffffff804c4503 at panic+0x43 > >> #3 0xffffffff807a2195 at trap_fatal+0x385 > >> #4 0xffffffff807a21ef at trap_pfault+0x4f > >> #5 0xffffffff80779c78 at calltrap+0x8 > >> #6 0xffffffff8045ddb8 at handleevents+0x188 > >> #7 0xffffffff8045ea3e at timercb+0x24e > >> #8 0xffffffff807ca9eb at lapic_handle_timer+0x9b > >> #9 0xffffffff8077b9b1 at Xtimerint+0xb1 > >> Uptime: 2h28m57s > >> Dumping 12829 out of 131023 > >> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > >> > >> __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > >> 55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" > >> (offsetof(struct pcpu, > >> (kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > >> #1  doadump (textdump=) > >>     at /usr/src/sys/kern/kern_shutdown.c:399 > >> #2  0xffffffff804c428c in kern_reboot (howto=260) > >>     at /usr/src/sys/kern/kern_shutdown.c:487 > >> #3  0xffffffff804c46fe in vpanic (fmt=0xffffffff807e1276 "%s", > >>     ap=) at /usr/src/sys/kern/kern_shutdown.c:920 > >> #4  0xffffffff804c4503 in panic (fmt=) > >>     at /usr/src/sys/kern/kern_shutdown.c:844 > >> #5  0xffffffff807a2195 in trap_fatal (frame=0xfffffe0434de4d50, eva=0) > >>     at /usr/src/sys/amd64/amd64/trap.c:946 > >> #6  0xffffffff807a21ef in trap_pfault (frame=0xfffffe0434de4d50, > >>     usermode=false, signo=, ucode=) > >>     at /usr/src/sys/amd64/amd64/trap.c:765 > >> #7  > >> #8  0xffffffff804e0db4 in callout_process > >> (now=now@entry=38385536922300) > >>     at /usr/src/sys/kern/kern_timeout.c:488 > >> #9  0xffffffff8045ddb8 in handleevents (now=now@entry=38385536922300, > >>     fake=fake@entry=0) at /usr/src/sys/kern/kern_clocksource.c:213 > >> #10 0xffffffff8045ea3e in timercb (et=0xffffffff80d475e0 , > >>     arg=) at /usr/src/sys/kern/kern_clocksource.c:357 > >> #11 0xffffffff807ca9eb in lapic_handle_timer > >> (frame=0xfffffe0434de4f40) > >>     at /usr/src/sys/x86/x86/local_apic.c:1364 > >> #12 > >> #13 0x000000080df42bb6 in ?? () > >> Backtrace stopped: Cannot access memory at address 0x7ffffdef2c90 > >> (kgdb)