From owner-freebsd-fs@freebsd.org Wed Oct 28 13:41:51 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4903B4474A8 for ; Wed, 28 Oct 2020 13:41:51 +0000 (UTC) (envelope-from ck-lists@cksoft.de) Received: from mx1.cksoft.de (mx1.cksoft.de [IPv6:2001:67c:24f8:1::25:1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mx1.cksoft.de", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CLqVk39NYz47N0 for ; Wed, 28 Oct 2020 13:41:50 +0000 (UTC) (envelope-from ck-lists@cksoft.de) Received: from m.cksoft.de (m.cksoft.de [IPv6:2001:67c:24f8:2003::25:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx1.cksoft.de (Postfix) with ESMTPS id 950201EB6C7 for ; Wed, 28 Oct 2020 14:41:42 +0100 (CET) Received: from amavisfra2 (unknown [IPv6:2001:67c:24f8:2003::25:a2]) by m.cksoft.de (Postfix) with ESMTP id 7233D315924 for ; Wed, 28 Oct 2020 14:41:42 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from m.cksoft.de ([IPv6:2001:67c:24f8:2003::25:3]) by amavisfra2 (amavisfra2.cksoft.de [IPv6:2001:67c:24f8:2003::25:a2]) (amavisd-new, port 10051) with ESMTP id 6DG-G_SmnDiN for ; Wed, 28 Oct 2020 14:41:40 +0100 (CET) Received: from nocfra1.cksoft.de (nocfra1.cksoft.de [IPv6:2001:67c:24f8:2001::53:1]) by m.cksoft.de (Postfix) with ESMTP id 9F32F30FC3A for ; Wed, 28 Oct 2020 14:41:40 +0100 (CET) Received: by nocfra1.cksoft.de (Postfix, from userid 1000) id 8699513EE7; Wed, 28 Oct 2020 14:41:40 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by nocfra1.cksoft.de (Postfix) with ESMTP id 81E5F13E88 for ; Wed, 28 Oct 2020 14:41:40 +0100 (CET) Date: Wed, 28 Oct 2020 14:41:40 +0100 (CET) From: Christian Kratzer Reply-To: Christian Kratzer To: freebsd-fs@freebsd.org Subject: 12.1-RELEASE-p7 panic in zio_free_issue_4_6 Message-ID: X-NCC-RegID: de.cksoft X-Spammer-Kill-Ratio: 75% MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-Rspamd-Queue-Id: 4CLqVk39NYz47N0 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ck-lists@cksoft.de designates 2001:67c:24f8:1::25:1 as permitted sender) smtp.mailfrom=ck-lists@cksoft.de X-Spamd-Result: default: False [-2.86 / 15.00]; HAS_REPLYTO(0.00)[ck@cksoft.de]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; REPLYTO_DN_EQ_FROM_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; REPLYTO_DOM_EQ_FROM_DOM(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.04)[-1.044]; TO_DN_NONE(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_SHORT(-0.49)[-0.488]; DMARC_NA(0.00)[cksoft.de]; NEURAL_HAM_MEDIUM(-1.03)[-1.031]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:57407, ipnet:2001:67c:24f8::/48, country:DE]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Oct 2020 13:41:51 -0000 Hi, one of my servers with 12.1-RELEASE-p7 started crashing with following Fatal trap 12: page fault while in kernel mode cpuid = 19; apic id = 31 fault virtual address = 0x30 fault code = supervisor write data, page not present instruction pointer = 0x20:0xffffffff826877f4 stack pointer = 0x28:0xfffffe011cefeaa0 frame pointer = 0x28:0xfffffe011cefeaa0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (zio_free_issue_2_3) trap number = 12 panic: page fault cpuid = 19 time = 1603797129 KDB: stack backtrace: #0 0xffffffff80c1d2f7 at kdb_backtrace+0x67 #1 0xffffffff80bd062d at vpanic+0x19d #2 0xffffffff80bd0483 at panic+0x43 #3 0xffffffff810a8dcc at trap_fatal+0x39c #4 0xffffffff810a8e19 at trap_pfault+0x49 #5 0xffffffff810a840f at trap+0x29f #6 0xffffffff81081c9c at calltrap+0x8 #7 0xffffffff8272a903 at zio_ddt_free+0x53 #8 0xffffffff82727b7c at zio_execute+0xac #9 0xffffffff80c2fad4 at taskqueue_run_locked+0x154 #10 0xffffffff80c30e08 at taskqueue_thread_loop+0x98 #11 0xffffffff80b90c43 at fork_exit+0x83 #12 0xffffffff81082cde at fork_trampoline+0xe Uptime: 1m12s Automatic reboot in 15 seconds - press a key on the console to abort I traced thigs down to importing one of the zpools. Ths machine has a 3 zpools The first two are ok: pool: zroot state: ONLINE scan: scrub repaired 0 in 0 days 00:00:22 with 0 errors on Fri Jul 17 17:24:17 2020 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/root0 ONLINE 0 0 0 gpt/root1 ONLINE 0 0 0 root@zfsfra1:/var/crash # zpool status -v pool: zpfra1 state: ONLINE scan: scrub repaired 0 in 0 days 00:48:16 with 0 errors on Fri Jul 17 18:12:04 2020 config: NAME STATE READ WRITE CKSUM zpfra1 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/zfsfra1d01.eli ONLINE 0 0 0 gpt/zfsfra1d09.eli ONLINE 0 0 0 logs mirror-1 ONLINE 0 0 0 gpt/log0d0 ONLINE 0 0 0 gpt/log0d1 ONLINE 0 0 0 The last one has two sets of 7 disks in a raid-z2. I have removed the geli keys for the disks so that it currently cannot be imported pool: zpfra2 state: UNAVAIL status: One or more devices could not be opened. There are insufficient replicas for the pool to continue functioning. action: Attach the missing device and online it using 'zpool online'. see: http://illumos.org/msg/ZFS-8000-3C scan: none requested config: NAME STATE READ WRITE CKSUM zpfra2 UNAVAIL 0 0 0 raidz2-0 UNAVAIL 0 0 0 7179798941412063472 UNAVAIL 0 0 0 was /dev/gpt/zfsfra1d02.eli 17119114611556833764 UNAVAIL 0 0 0 was /dev/gpt/zfsfra1d03.eli 8321725234410067709 UNAVAIL 0 0 0 was /dev/gpt/zfsfra1d04.eli 7897191132634569755 UNAVAIL 0 0 0 was /dev/gpt/zfsfra1d05.eli 16873755985119583929 UNAVAIL 0 0 0 was /dev/gpt/zfsfra1d06.eli 9644713294010671122 UNAVAIL 0 0 0 was /dev/gpt/zfsfra1d07.eli 1480177385910791788 UNAVAIL 0 0 0 was /dev/gpt/zfsfra1d08.eli raidz2-1 UNAVAIL 0 0 0 1498696212334632055 UNAVAIL 0 0 0 was /dev/gpt/zfsfra1d10.eli 5551216295452602020 UNAVAIL 0 0 0 was /dev/gpt/zfsfra1d11.eli 17197173774607757750 UNAVAIL 0 0 0 was /dev/gpt/zfsfra1d12.eli 12543220242988729823 UNAVAIL 0 0 0 was /dev/gpt/zfsfra1d13.eli 711115555895092704 UNAVAIL 0 0 0 was /dev/gpt/zfsfra1d14.eli 15806058868994893097 UNAVAIL 0 0 0 was /dev/gpt/zfsfra1d15.eli 7273134084268794449 UNAVAIL 0 0 0 was /dev/gpt/zfsfra1d16.eli logs mirror-2 ONLINE 0 0 0 gpt/log1d0 ONLINE 0 0 0 gpt/log1d1 ONLINE 0 0 0 If I put the keys back the system will crash with above error after importing the pool. I also tried importing the pool readonly but it also crashed. Any ideas how to get this back into a sane state ? Because of the zio_free_issue_2_3 error I am suspecting this to be something inconsistent in the log devices. How could I remove those log devices and force import the pool. Greetings Christian -- Christian Kratzer CK Software GmbH Email: ck@cksoft.de Wildberger Weg 24/2 Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart Mobile: +49 171 1947 843 Geschaeftsfuehrer: Christian Kratzer Web: http://www.cksoft.de/ From owner-freebsd-fs@freebsd.org Thu Oct 29 07:18:45 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DA2614436AE for ; Thu, 29 Oct 2020 07:18:45 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CMGyD5pTFz4Hcj for ; Thu, 29 Oct 2020 07:18:44 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f177.google.com with SMTP id c21so2022142ljj.0 for ; Thu, 29 Oct 2020 00:18:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=v4o9fYFp7GObr+uCQI921z7saQnVBVWVDF7+RqT50e4=; b=tWtq0GbDTzpXeYjLYr2DMrELtEKZv2oA6ANlyCsBU29guHRvlZ/Nl5X7fQjfebhsGA Rib0mujHJUza7BbH2KkUKNO1tv/aa8mb5EMncSCv5TfbWOKadqJ0WuiHyqWn37577aa7 KS99qoJm3xgqNECQ2DtZUsT43u9cRzkYPrY0+YpITriL6DC94vmMvdjFc3DZ6zTt+pBS lYJwBpcJX4g0hxIu/WFLaIEBOrLWPpyrZIVTaZtdUNBCXo1NJdmhNSg5EOqtI/45GtPR 72GvmGRaVgPzoFmcKTtxIvseNldWKxQUr8KKc8vHwC5AJsLphJKl0eneNsPkiXJgmmp+ j8MA== X-Gm-Message-State: AOAM533a9vuzdoA6rqTi0KuqKvDlBvtnzQg7PJSje+n5ffRqn+NN5lkP Jwxo1ey9d6qs36uarIc9AWJNzx3wjXl0rw== X-Google-Smtp-Source: ABdhPJzWDe/X7eNZxQDaKChShwhJFKUQBX3kvLb6ZZJDD73x53+/KQhA1Au7FFVBpxaFhJgSzwENiQ== X-Received: by 2002:a2e:6e10:: with SMTP id j16mr365201ljc.320.1603955922281; Thu, 29 Oct 2020 00:18:42 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id b18sm187176lfp.89.2020.10.29.00.18.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 29 Oct 2020 00:18:41 -0700 (PDT) Subject: Re: 12.1-RELEASE-p7 panic in zio_free_issue_4_6 To: Christian Kratzer , freebsd-fs@freebsd.org References: From: Andriy Gapon Message-ID: <474d086c-5a36-0db5-974f-ccfa0acbd871@FreeBSD.org> Date: Thu, 29 Oct 2020 09:18:40 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4CMGyD5pTFz4Hcj X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.177 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-2.70 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_SHORT(-0.72)[-0.719]; RCPT_COUNT_TWO(0.00)[2]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[93.72.151.96:received]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.975]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.004]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.208.177:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.208.177:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2020 07:18:45 -0000 On 28/10/2020 15:41, Christian Kratzer wrote: > Hi, > > one of my servers with 12.1-RELEASE-p7 started crashing with following > > Fatal trap 12: page fault while in kernel mode > cpuid = 19; apic id = 31 > fault virtual address   = 0x30 > fault code              = supervisor write data, page not present > instruction pointer     = 0x20:0xffffffff826877f4 > stack pointer           = 0x28:0xfffffe011cefeaa0 > frame pointer           = 0x28:0xfffffe011cefeaa0 > code segment            = base 0x0, limit 0xfffff, type 0x1b >                         = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags        = interrupt enabled, resume, IOPL = 0 > current process         = 0 (zio_free_issue_2_3) > trap number             = 12 > panic: page fault > cpuid = 19 > time = 1603797129 > KDB: stack backtrace: > #0 0xffffffff80c1d2f7 at kdb_backtrace+0x67 > #1 0xffffffff80bd062d at vpanic+0x19d > #2 0xffffffff80bd0483 at panic+0x43 > #3 0xffffffff810a8dcc at trap_fatal+0x39c > #4 0xffffffff810a8e19 at trap_pfault+0x49 > #5 0xffffffff810a840f at trap+0x29f > #6 0xffffffff81081c9c at calltrap+0x8 > #7 0xffffffff8272a903 at zio_ddt_free+0x53 > #8 0xffffffff82727b7c at zio_execute+0xac > #9 0xffffffff80c2fad4 at taskqueue_run_locked+0x154 > #10 0xffffffff80c30e08 at taskqueue_thread_loop+0x98 > #11 0xffffffff80b90c43 at fork_exit+0x83 > #12 0xffffffff81082cde at fork_trampoline+0xe > Uptime: 1m12s > Automatic reboot in 15 seconds - press a key on the console to abort > > > I traced thigs down to importing one of the zpools. I suspect that you have a silent corruption on that pool (perhaps because of non-ECC RAM?). What you see can happen if a block pointer has a deduplication bit set, but the block is not actually deduplicated or deduplication has never been enabled at all. It would help -- with analysis -- to get a vmcore (kernel crash dump) and to install the corresponding kernel debug symbols (if not already). As to recovery, I think that the best solution is to import the pool read-only and to copy important data elsewhere. Then re-create the pool. -- Andriy From owner-freebsd-fs@freebsd.org Thu Oct 29 07:34:12 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EC8C1444301 for ; Thu, 29 Oct 2020 07:34:12 +0000 (UTC) (envelope-from ck-lists@cksoft.de) Received: from mx1.cksoft.de (mx1.cksoft.de [IPv6:2001:67c:24f8:1::25:1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mx1.cksoft.de", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CMHJ42Jg4z4J3d for ; Thu, 29 Oct 2020 07:34:12 +0000 (UTC) (envelope-from ck-lists@cksoft.de) Received: from m.cksoft.de (m.cksoft.de [IPv6:2001:67c:24f8:2003::25:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx1.cksoft.de (Postfix) with ESMTPS id 4C8271EB6C7 for ; Thu, 29 Oct 2020 08:34:11 +0100 (CET) Received: from amavisfra2 (unknown [IPv6:2001:67c:24f8:2003::25:a2]) by m.cksoft.de (Postfix) with ESMTP id 351C2315909 for ; Thu, 29 Oct 2020 08:34:11 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from m.cksoft.de ([IPv6:2001:67c:24f8:2003::25:3]) by amavisfra2 (amavisfra2.cksoft.de [IPv6:2001:67c:24f8:2003::25:a2]) (amavisd-new, port 10051) with ESMTP id NLX_XH_lqf0O for ; Thu, 29 Oct 2020 08:34:08 +0100 (CET) Received: from nocfra1.cksoft.de (nocfra1.cksoft.de [IPv6:2001:67c:24f8:2001::53:1]) by m.cksoft.de (Postfix) with ESMTP id 5093230FC3A for ; Thu, 29 Oct 2020 08:34:08 +0100 (CET) Received: by nocfra1.cksoft.de (Postfix, from userid 1000) id 4115C14B0D; Thu, 29 Oct 2020 08:34:08 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by nocfra1.cksoft.de (Postfix) with ESMTP id 3FA4814AFA for ; Thu, 29 Oct 2020 08:34:08 +0100 (CET) Date: Thu, 29 Oct 2020 08:34:08 +0100 (CET) From: Christian Kratzer Reply-To: Christian Kratzer To: freebsd-fs@freebsd.org Subject: Re: 12.1-RELEASE-p7 panic in zio_free_issue_4_6 (fwd) Message-ID: <37c67834-204f-96b6-d37-70bfd832acee@cksoft.de> X-NCC-RegID: de.cksoft X-Spammer-Kill-Ratio: 75% MIME-Version: 1.0 Content-ID: <7cb3a87d-2bc3-2567-cecc-1338ebfb745@cksoft.de> X-Rspamd-Queue-Id: 4CMHJ42Jg4z4J3d X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ck-lists@cksoft.de designates 2001:67c:24f8:1::25:1 as permitted sender) smtp.mailfrom=ck-lists@cksoft.de X-Spamd-Result: default: False [-0.28 / 15.00]; ARC_NA(0.00)[]; HAS_REPLYTO(0.00)[ck@cksoft.de]; RCVD_COUNT_FIVE(0.00)[6]; FAKE_REPLY(1.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mx1.cksoft.de:c]; REPLYTO_DN_EQ_FROM_DN(0.00)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; REPLYTO_DOM_EQ_FROM_DOM(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.02)[-1.020]; NEURAL_SPAM_SHORT(0.01)[0.009]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; CTYPE_MIXED_BOGUS(1.00)[]; DMARC_NA(0.00)[cksoft.de]; MID_RHS_MATCH_FROM(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.973]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:57407, ipnet:2001:67c:24f8::/48, country:DE]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] Content-Type: text/plain; CHARSET=UTF-8; format=flowed Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2020 07:34:13 -0000 Hi, On Thu, 29 Oct 2020, Andriy Gapon wrote: > On 28/10/2020 15:41, Christian Kratzer wrote: >> Hi, >> >> one of my servers with 12.1-RELEASE-p7 started crashing with following >> >> Fatal trap 12: page fault while in kernel mode >> cpuid = 19; apic id = 31 >> fault virtual address   = 0x30 >> fault code              = supervisor write data, page not present >> instruction pointer     = 0x20:0xffffffff826877f4 >> stack pointer           = 0x28:0xfffffe011cefeaa0 >> frame pointer           = 0x28:0xfffffe011cefeaa0 >> code segment            = base 0x0, limit 0xfffff, type 0x1b >>                         = DPL 0, pres 1, long 1, def32 0, gran 1 >> processor eflags        = interrupt enabled, resume, IOPL = 0 >> current process         = 0 (zio_free_issue_2_3) >> trap number             = 12 >> panic: page fault >> cpuid = 19 >> time = 1603797129 >> KDB: stack backtrace: >> #0 0xffffffff80c1d2f7 at kdb_backtrace+0x67 >> #1 0xffffffff80bd062d at vpanic+0x19d >> #2 0xffffffff80bd0483 at panic+0x43 >> #3 0xffffffff810a8dcc at trap_fatal+0x39c >> #4 0xffffffff810a8e19 at trap_pfault+0x49 >> #5 0xffffffff810a840f at trap+0x29f >> #6 0xffffffff81081c9c at calltrap+0x8 >> #7 0xffffffff8272a903 at zio_ddt_free+0x53 >> #8 0xffffffff82727b7c at zio_execute+0xac >> #9 0xffffffff80c2fad4 at taskqueue_run_locked+0x154 >> #10 0xffffffff80c30e08 at taskqueue_thread_loop+0x98 >> #11 0xffffffff80b90c43 at fork_exit+0x83 >> #12 0xffffffff81082cde at fork_trampoline+0xe >> Uptime: 1m12s >> Automatic reboot in 15 seconds - press a key on the console to abort >> >> >> I traced thigs down to importing one of the zpools. > > I suspect that you have a silent corruption on that pool (perhaps because of > non-ECC RAM?). This is on a DL380 G7 with 128GB of ECC ram. I have ran memtest on this server before without any defects being found. The sas disks are on an LSI hba. They also do not have defects according to smartctl. This of course does not rule out that there might be an issue with ram and I will need to recheck. Also I suspect the server might not have enough RAM for doing dedup on this 2 x 7 disk raid-z2 of 1.2GB drives. The pool was mostly in use for storing backups rsynced over night from two other servers. > What you see can happen if a block pointer has a deduplication bit set, but > the > block is not actually deduplicated or deduplication has never been enabled at > all. Could I have ran into an issue and bug by trying to do too much dedup on this pool ? > It would help -- with analysis -- to get a vmcore (kernel crash dump) and to > install the corresponding kernel debug symbols (if not already). I need to see why this server is not producing kernel crash dumps. My other setup does so I should be able to get this done. > As to recovery, I think that the best solution is to import the pool > read-only > and to copy important data elsewhere. Then re-create the pool. I was about to do that but the crash also happens when trying to import read-only. I will investigate if I can import based on an older snapshot or checkpoint but I am not sure if that will do what I want. I will keep this pool around for a couple of days and will try to get a crash dump from the system. After that I will have delete and recreate the pool and just wait for backups to roll back in. Greetings Christian -- Christian Kratzer CK Software GmbH Email: ck@cksoft.de Wildberger Weg 24/2 Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart Mobile: +49 171 1947 843 Geschaeftsfuehrer: Christian Kratzer Web: http://www.cksoft.de/ From owner-freebsd-fs@freebsd.org Thu Oct 29 07:46:38 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AD4DB44479F for ; Thu, 29 Oct 2020 07:46:38 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CMHZQ1l0Cz4KLF for ; Thu, 29 Oct 2020 07:46:37 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from [192.168.0.88] (east.meadow.volia.net [93.72.151.96]) (Authenticated sender: andriy.gapon@uabsd.com) by relay12.mail.gandi.net (Postfix) with ESMTPSA id C18CB200004; Thu, 29 Oct 2020 07:46:35 +0000 (UTC) Subject: Re: 12.1-RELEASE-p7 panic in zio_free_issue_4_6 To: Christian Kratzer Cc: freebsd-fs@freebsd.org References: <474d086c-5a36-0db5-974f-ccfa0acbd871@FreeBSD.org> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; keydata= mDMEX1iFDhYJKwYBBAHaRw8BAQdAiu8JG/oLFkVkOAJqJc7Dx5KI/Q6C3SBI20EQm+DXnAu0 HkFuZHJpeSBHYXBvbiA8YXZnQEZyZWVCU0Qub3JnPoiWBBMWCAA+FiEEyCHHZM09l0OE3Ir/ 1A1+Gq8+L1EFAl9YhQ4CGwMFCQeEzgAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ1A1+ Gq8+L1Fc0wD/ZjmhHfbCJywZU3aOxXIPjcz73FYEGMvqMCCLAWyLbSABALFL+1ZNrjV3BGjq 889cOYFuboA/Yn3eWezS+tfqYBsGuDgEX1iFDhIKKwYBBAGXVQEFAQEHQL6B20Xi600TrkpG P9fWjl7JtHNxqrHKhX6Kg7kgb4ILAwEIB4h+BBgWCAAmFiEEyCHHZM09l0OE3Ir/1A1+Gq8+ L1EFAl9YhQ4CGwwFCQeEzgAACgkQ1A1+Gq8+L1F3cgEAktp4h+IJUJxL1vn6zMOt//znni/J TanKfQuA8wGXcGkBAKpZJhqMkg+pKk7MGvJhgJ6nCpTZ+rMK6vZVZLUWc3QF Message-ID: <24b9cc11-0681-2f17-b634-d68878bc67ac@FreeBSD.org> Date: Thu, 29 Oct 2020 09:46:33 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4CMHZQ1l0Cz4KLF X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; ASN(0.00)[asn:29169, ipnet:217.70.176.0/20, country:FR] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2020 07:46:38 -0000 On 29/10/2020 09:33, Christian Kratzer wrote: > Hi, > > On Thu, 29 Oct 2020, Andriy Gapon wrote: >> On 28/10/2020 15:41, Christian Kratzer wrote: >>> I traced thigs down to importing one of the zpools. >> >> I suspect that you have a silent corruption on that pool (perhaps because of >> non-ECC RAM?). > > This is on a DL380 G7 with 128GB of ECC ram.  I have ran memtest on this server > before without any defects being found. > > The sas disks are on an LSI hba. They also do not have defects according to > smartctl. > > This of course does not rule out that there might be an issue with ram and > I will need to recheck. > > Also I suspect the server might not have enough RAM for doing dedup on this > 2 x 7 disk raid-z2 of 1.2GB drives. > > The pool was mostly in use for storing backups rsynced over night from two > other servers. > >> What you see can happen if a block pointer has a deduplication bit set, but the >> block is not actually deduplicated or deduplication has never been enabled at >> all. > > Could I have ran into an issue and bug by trying to do too much dedup on this > pool ? > >> It would help -- with analysis -- to get a vmcore (kernel crash dump) and to >> install the corresponding kernel debug symbols (if not already). > > I need to see why this server is not producing kernel crash dumps. My other setup > does so I should be able to get this done. > >> As to recovery, I think that the best solution is to import the pool read-only >> and to copy important data elsewhere.  Then re-create the pool. > > I was about to do that but the crash also happens when trying to import read-only. > > I will investigate if I can import based on an older snapshot or checkpoint but > I am > not sure if that will do what I want. > > I will keep this pool around for a couple of days and will try to get a crash dump > from the system.  After that I will have delete and recreate the pool and just > wait for backups to roll back in. Okay, let's see if we can get a vmcore. Otherwise, this is just a guess-work on my part. The problem could be very different from my initial impression. -- Andriy From owner-freebsd-fs@freebsd.org Thu Oct 29 15:48:49 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A719D455C7E for ; Thu, 29 Oct 2020 15:48:49 +0000 (UTC) (envelope-from ck-lists@cksoft.de) Received: from mx1.cksoft.de (mx1.cksoft.de [IPv6:2001:67c:24f8:1::25:1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mx1.cksoft.de", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CMVGm5hmDz3xmZ; Thu, 29 Oct 2020 15:48:48 +0000 (UTC) (envelope-from ck-lists@cksoft.de) Received: from m.cksoft.de (m.cksoft.de [195.88.109.48]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx1.cksoft.de (Postfix) with ESMTPS id C21951EB6C7; Thu, 29 Oct 2020 16:48:39 +0100 (CET) Received: from amavisfra2 (unknown [IPv6:2001:67c:24f8:2003::25:a2]) by m.cksoft.de (Postfix) with ESMTP id 84E51315909; Thu, 29 Oct 2020 16:48:39 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from m.cksoft.de ([192.168.35.42]) by amavisfra2 (amavisfra2.cksoft.de [192.168.35.44]) (amavisd-new, port 10051) with ESMTP id eXNaQt_0KB3w; Thu, 29 Oct 2020 16:48:37 +0100 (CET) Received: from nocfra1.cksoft.de (nocfra1.cksoft.de [IPv6:2001:67c:24f8:2001::53:1]) by m.cksoft.de (Postfix) with ESMTP id 2007930FC3A; Thu, 29 Oct 2020 16:48:37 +0100 (CET) Received: by nocfra1.cksoft.de (Postfix, from userid 1000) id EDDB213E65; Thu, 29 Oct 2020 16:48:36 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by nocfra1.cksoft.de (Postfix) with ESMTP id E6D0513E4A; Thu, 29 Oct 2020 16:48:36 +0100 (CET) Date: Thu, 29 Oct 2020 16:48:36 +0100 (CET) From: Christian Kratzer Reply-To: Christian Kratzer To: Andriy Gapon cc: freebsd-fs@freebsd.org Subject: Re: 12.1-RELEASE-p7 panic in zio_free_issue_4_6 In-Reply-To: <24b9cc11-0681-2f17-b634-d68878bc67ac@FreeBSD.org> Message-ID: <9ca55ded-1f91-5118-917e-3266946020@cksoft.de> References: <474d086c-5a36-0db5-974f-ccfa0acbd871@FreeBSD.org> <24b9cc11-0681-2f17-b634-d68878bc67ac@FreeBSD.org> X-NCC-RegID: de.cksoft X-Spammer-Kill-Ratio: 75% MIME-Version: 1.0 X-Rspamd-Queue-Id: 4CMVGm5hmDz3xmZ X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ck-lists@cksoft.de designates 2001:67c:24f8:1::25:1 as permitted sender) smtp.mailfrom=ck-lists@cksoft.de X-Spamd-Result: default: False [-1.95 / 15.00]; HAS_REPLYTO(0.00)[ck@cksoft.de]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; REPLYTO_DN_EQ_FROM_DN(0.00)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; REPLYTO_DOM_EQ_FROM_DOM(0.00)[]; DMARC_NA(0.00)[cksoft.de]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.003]; NEURAL_HAM_MEDIUM(-1.00)[-1.003]; NEURAL_HAM_SHORT(-0.64)[-0.640]; CTYPE_MIXED_BOGUS(1.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:57407, ipnet:2001:67c:24f8::/48, country:DE]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2020 15:48:49 -0000 Hi, On Thu, 29 Oct 2020, Andriy Gapon wrote: >> I will keep this pool around for a couple of days and will try to get a crash dump >> from the system.  After that I will have delete and recreate the pool and just >> wait for backups to roll back in. > > > Okay, let's see if we can get a vmcore. > Otherwise, this is just a guess-work on my part. > The problem could be very different from my initial impression. so I added a swap device which for some reason was missing and was about to induce the crash again. But in order to get consistent versions of the kernel debug symbols I upgraded the system to 12.2-RELEASE. Now that everything was in place I was able to import the pool in readonly mode using zpool import -m o -readonly=on zpfra2 without the system crashing. It sits there happily now pool: zpfra2 state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://illumos.org/msg/ZFS-8000-2Q scan: scrub repaired 0 in 0 days 01:31:43 with 0 errors on Fri Jul 17 18:55:35 2020 config: NAME STATE READ WRITE CKSUM zpfra2 DEGRADED 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/zfsfra1d02.eli ONLINE 0 0 0 gpt/zfsfra1d03.eli ONLINE 0 0 0 gpt/zfsfra1d04.eli ONLINE 0 0 0 gpt/zfsfra1d05.eli ONLINE 0 0 0 gpt/zfsfra1d06.eli ONLINE 0 0 0 gpt/zfsfra1d07.eli ONLINE 0 0 0 gpt/zfsfra1d08.eli ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 gpt/zfsfra1d10.eli ONLINE 0 0 0 gpt/zfsfra1d11.eli ONLINE 0 0 0 gpt/zfsfra1d12.eli ONLINE 0 0 0 gpt/zfsfra1d13.eli ONLINE 0 0 0 gpt/zfsfra1d14.eli ONLINE 0 0 0 gpt/zfsfra1d15.eli ONLINE 0 0 0 gpt/zfsfra1d16.eli ONLINE 0 0 0 logs mirror-2 UNAVAIL 0 0 0 3980362133776709100 UNAVAIL 0 0 0 was /dev/gpt/log1d0 6670731949941654186 UNAVAIL 0 0 0 was /dev/gpt/log1d1 The pool is degraded as I had already removed the log devices. I am pulling data of the pool as we speak and will recreate it. In case this happens again I will be prepared with a crash dump. I will also not enable dedup for this 15TB pool as long as the machine has only 128GB ram. Greetings Christian -- Christian Kratzer CK Software GmbH Email: ck@cksoft.de Wildberger Weg 24/2 Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart Mobile: +49 171 1947 843 Geschaeftsfuehrer: Christian Kratzer Web: http://www.cksoft.de/ From owner-freebsd-fs@freebsd.org Thu Oct 29 16:44:35 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 53452457E3C for ; Thu, 29 Oct 2020 16:44:35 +0000 (UTC) (envelope-from ck-lists@cksoft.de) Received: from mx1.cksoft.de (mx1.cksoft.de [IPv6:2001:67c:24f8:1::25:1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mx1.cksoft.de", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CMWW61kPvz42bT; Thu, 29 Oct 2020 16:44:33 +0000 (UTC) (envelope-from ck-lists@cksoft.de) Received: from m.cksoft.de (m.cksoft.de [IPv6:2001:67c:24f8:2003::25:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx1.cksoft.de (Postfix) with ESMTPS id B8D6A1EB6C7; Thu, 29 Oct 2020 17:44:32 +0100 (CET) Received: from amavisfra2 (unknown [IPv6:2001:67c:24f8:2003::25:a2]) by m.cksoft.de (Postfix) with ESMTP id 6FA44315909; Thu, 29 Oct 2020 17:44:32 +0100 (CET) X-Virus-Scanned: amavisd-new at cksoft.de Received: from m.cksoft.de ([192.168.35.42]) by amavisfra2 (amavisfra2.cksoft.de [192.168.35.44]) (amavisd-new, port 10051) with ESMTP id fNoDeWkmLIYt; Thu, 29 Oct 2020 17:44:31 +0100 (CET) Received: from nocfra1.cksoft.de (nocfra1.cksoft.de [IPv6:2001:67c:24f8:2001::53:1]) by m.cksoft.de (Postfix) with ESMTP id 0735330FC3A; Thu, 29 Oct 2020 17:44:30 +0100 (CET) Received: by nocfra1.cksoft.de (Postfix, from userid 1000) id D9CEA13E65; Thu, 29 Oct 2020 17:44:30 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by nocfra1.cksoft.de (Postfix) with ESMTP id D53A613E4A; Thu, 29 Oct 2020 17:44:30 +0100 (CET) Date: Thu, 29 Oct 2020 17:44:30 +0100 (CET) From: Christian Kratzer Reply-To: Christian Kratzer To: Andriy Gapon cc: freebsd-fs@freebsd.org Subject: Re: 12.1-RELEASE-p7 panic in zio_free_issue_4_6 In-Reply-To: <24b9cc11-0681-2f17-b634-d68878bc67ac@FreeBSD.org> Message-ID: References: <474d086c-5a36-0db5-974f-ccfa0acbd871@FreeBSD.org> <24b9cc11-0681-2f17-b634-d68878bc67ac@FreeBSD.org> X-NCC-RegID: de.cksoft X-Spammer-Kill-Ratio: 75% MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 4CMWW61kPvz42bT X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ck-lists@cksoft.de designates 2001:67c:24f8:1::25:1 as permitted sender) smtp.mailfrom=ck-lists@cksoft.de X-Spamd-Result: default: False [-2.80 / 15.00]; HAS_REPLYTO(0.00)[ck@cksoft.de]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; REPLYTO_DN_EQ_FROM_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; REPLYTO_DOM_EQ_FROM_DOM(0.00)[]; DMARC_NA(0.00)[cksoft.de]; R_SPF_ALLOW(-0.20)[+a:mail.cksoft.de]; NEURAL_HAM_LONG(-1.00)[-1.002]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.49)[-0.494]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:57407, ipnet:2001:67c:24f8::/48, country:DE]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2020 16:44:35 -0000 Hi, On Thu, 29 Oct 2020, Andriy Gapon wrote: > Okay, let's see if we can get a vmcore. > Otherwise, this is just a guess-work on my part. > The problem could be very different from my initial impression. got the data off the pool in ro and mounted again RW Fatal trap 12: page fault while in kernel mode cpuid = 7; apic id = 11 fault virtual address = 0x30 fault code = supervisor write data, page not present instruction pointer = 0x20:0xffffffff824b89e4 stack pointer = 0x28:0xfffffe012d6c7be0 frame pointer = 0x28:0xfffffe012d6c7be0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 3185 (zio_free_issue_6_3) trap number = 12 panic: page fault cpuid = 7 time = 1603989616 KDB: stack backtrace: #0 0xffffffff80c0a8f5 at kdb_backtrace+0x65 #1 0xffffffff80bbeb1b at vpanic+0x17b #2 0xffffffff80bbe993 at panic+0x43 #3 0xffffffff8108f911 at trap_fatal+0x391 #4 0xffffffff8108f96f at trap_pfault+0x4f #5 0xffffffff8108efb6 at trap+0x286 #6 0xffffffff81066f38 at calltrap+0x8 #7 0xffffffff8255e672 at zio_ddt_free+0x52 #8 0xffffffff8255ba2c at zio_execute+0xac #9 0xffffffff80c1cee4 at taskqueue_run_locked+0x144 #10 0xffffffff80c1e2d6 at taskqueue_thread_loop+0xb6 #11 0xffffffff80b8044e at fork_exit+0x7e #12 0xffffffff81067f6e at fork_trampoline+0xe Uptime: 1h13m32s Dumping 8797 out of 131020 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Dump complete Automatic reboot in 15 seconds - press a key on the console to abort Rebooting... cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 7 This is on 12.2-RELEASE now and debug symbols are in place. I will be archiving the artefacts and will get back with proper tracebacks shortly. Let me know what you need and how you would like to receive it. Greetings Christian -- Christian Kratzer CK Software GmbH Email: ck@cksoft.de Wildberger Weg 24/2 Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart Mobile: +49 171 1947 843 Geschaeftsfuehrer: Christian Kratzer Web: http://www.cksoft.de/