From owner-freebsd-stable@freebsd.org Tue Dec 22 15:07:10 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 36F5B4C049A for ; Tue, 22 Dec 2020 15:07:10 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x72f.google.com (mail-qk1-x72f.google.com [IPv6:2607:f8b0:4864:20::72f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D0fnm4Qgxz4WjF for ; Tue, 22 Dec 2020 15:07:07 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qk1-x72f.google.com with SMTP id f26so10460446qka.0 for ; Tue, 22 Dec 2020 07:07:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=7reyerwyXxi6D/O4ovzww68ZJrYmL2cvn0m27U6EcB8=; b=kijI+RCY4a0pAcKI3hPlC7illr9iXhDnN9aSCnXYwSXaQHVXgDYbc47kqKo1rxMxcj oENqX2AjHW5uia49mTIUxfQbrHroUQc8RBQK5EnsTrE4Sl7ZnR/A1EIvaPleM3/Fs95i hDWhVNsr/1pyUQJe9wMXRyTbvOL3ZDgBVYA27vFX2LPHZaHK5gIThPjUE7SDewa0tpf5 X7mobS1XL/TkJLdc4hMuHwuTXJZx1MI/aZEBGWurkZ6jBAYdHxTrPDJqudLmkuFcaxu4 8oFlhsZ8K6npcAQInj+pksF0G2pt40qCB/cNQw/pSH2hu+XlZfes0HSsIJWNBLl8IUD2 odIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=7reyerwyXxi6D/O4ovzww68ZJrYmL2cvn0m27U6EcB8=; b=WsiTxMthkAZXjdC50/GiN2OKAjbb2sBbh+hC5LjC0JrdO32n7qlVHvEThpXpmxKgqC UxITlMgvm/WCxMQ01kgZ8KjcCdSwcxnPNZ1SZ4BCxw8dIzP7zOvykk2stnCkdio7EdlT Z28ecZk5WjStQKaFIleiBVIv6kSQHi2mqo96Ic9XSG9KRPpcSbR4kT3EKVJmVVUFR+16 hKfTphti1Y46fDm8WV4YOP6sZI/Dh1Y/ZBhWoXnvFPS5qdCKf17ilULmgCSwmioAcs5S WqHV1kpJMs+ov7zptst6DrPVW4nDsBkjSJ0NbM36eC2Pj3m9Ju52bJgA59xFwIjo/PW9 QVCw== X-Gm-Message-State: AOAM531OLWrw3V5QsxaScmq2pBK6NgtUKuagn0l9kns2LQ4BrWHQ8OPO VIYH9+lFpYICfS9frbe6xw5YbAYF+PYxdg== X-Google-Smtp-Source: ABdhPJzepm7mEF5LjVLTF3Wy199hdmcrrtd4v+FfOAWqnTB8bmnqImOMpXWIZhU6Q8If0oC4bKLEqg== X-Received: by 2002:ae9:e648:: with SMTP id x8mr21909987qkl.226.1608649626854; Tue, 22 Dec 2020 07:07:06 -0800 (PST) Received: from raichu ([142.126.164.150]) by smtp.gmail.com with ESMTPSA id j124sm13439030qkf.113.2020.12.22.07.07.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Dec 2020 07:07:06 -0800 (PST) Sender: Mark Johnston Date: Tue, 22 Dec 2020 10:07:04 -0500 From: Mark Johnston To: mike tancsa Cc: FreeBSD-STABLE Mailing List Subject: Re: zfs panic RELENG_12 Message-ID: References: <878824fe-dde2-b551-4685-e8bd27371275@sentex.net> <5b3415cb-2176-895e-9d22-4f4f0f359d85@sentex.net> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <5b3415cb-2176-895e-9d22-4f4f0f359d85@sentex.net> X-Rspamd-Queue-Id: 4D0fnm4Qgxz4WjF X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=kijI+RCY; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::72f as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-2.68 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.98)[-0.985]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::72f:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::72f:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::72f:from]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Dec 2020 15:07:10 -0000 On Tue, Dec 22, 2020 at 09:05:01AM -0500, mike tancsa wrote: > Hmmm, another one. Not sure if this is hardware as it seems different ? > > > > Fatal trap 12: page fault while in kernel mode > cpuid = 11; apic id = 0b > fault virtual address   = 0x0 > fault code              = supervisor write data, page not present > instruction pointer     = 0x20:0xffffffff80ca0826 > stack pointer           = 0x28:0xfffffe00bc0f8540 > frame pointer           = 0x28:0xfffffe00bc0f8590 > code segment            = base 0x0, limit 0xfffff, type 0x1b >                         = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags        = interrupt enabled, resume, IOPL = 0 > current process         = 33 (dom0) > trap number             = 12 > panic: page fault > cpuid = 11 > time = 1608641071 > KDB: stack backtrace: > #0 0xffffffff80a3fe85 at kdb_backtrace+0x65 > #1 0xffffffff809f406b at vpanic+0x17b > #2 0xffffffff809f3ee3 at panic+0x43 > #3 0xffffffff80e3fe71 at trap_fatal+0x391 > #4 0xffffffff80e3fecf at trap_pfault+0x4f > #5 0xffffffff80e3f516 at trap+0x286 > #6 0xffffffff80e19318 at calltrap+0x8 > #7 0xffffffff80ca47d4 at bucket_cache_drain+0x134 > #8 0xffffffff80c9e302 at zone_drain_wait+0xa2 > #9 0xffffffff80ca2bbd at uma_reclaim_locked+0x6d > #10 0xffffffff80ca2af4 at uma_reclaim+0x34 > #11 0xffffffff80cc5321 at vm_pageout_worker+0x421 > #12 0xffffffff80cc4ee3 at vm_pageout+0x193 > #13 0xffffffff809b55be at fork_exit+0x7e > #14 0xffffffff80e1a34e at fork_trampoline+0xe > Uptime: 5d20h37m16s > Dumping 16057 out of 65398 > MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > > __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > 55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" > (offsetof(struct pcpu, > (kgdb) bt Could you go to frame 11 and print zone->uz_name and bucket->ub_bucket[18]? I'm wondering if the item pointer was mangled somehow.