From owner-freebsd-fs@freebsd.org Tue Oct 8 18:02:51 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 663CA132718 for ; Tue, 8 Oct 2019 18:02:51 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 46nlZ31f6pz4KyV; Tue, 8 Oct 2019 18:02:51 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (c-73-225-95-104.hsd1.wa.comcast.net [73.225.95.104]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id x98I2gs4005817 (version=TLSv1.2 cipher=AES128-SHA bits=128 verify=NO); Tue, 8 Oct 2019 11:02:43 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: panic: Memory modified after free To: Andriy Gapon , Yuri Pankov , freebsd-fs@FreeBSD.org References: <68853b35-e202-40b9-80ad-b29bc49bb5b4@www.fastmail.com> From: Julian Elischer Message-ID: <069ede9f-1098-e809-50a2-55844dd6b57c@freebsd.org> Date: Tue, 8 Oct 2019 11:02:37 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 46nlZ31f6pz4KyV X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.997,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Oct 2019 18:02:51 -0000 On 10/8/19 12:12 AM, Andriy Gapon wrote: > On 08/10/2019 03:39, Yuri Pankov wrote: >> (posting to fs@ as backtrace suggests zfs) >> >> Started seeing the following panic (sometimes reproducible, so can't point specific revision or time range) untaring an archive over ssh into my home directory on zfs: >> >> panic: Memory modified after free 0xfffff80404cff000(4096) val=dead40de @ 0xfffff80404cff694 >> >> cpuid = 4 >> time = 1570493241 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00df912650 >> vpanic() at vpanic+0x19d/frame 0xfffffe00df9126a0 >> panic() at panic+0x43/frame 0xfffffe00df912700 >> trash_ctor() at trash_ctor+0x49/frame 0xfffffe00df912710 >> uma_zalloc_arg() at uma_zalloc_arg+0xa24/frame 0xfffffe00df9127a0 >> abd_alloc() at abd_alloc+0x152/frame 0xfffffe00df9127f0 >> arc_hdr_alloc_pabd() at arc_hdr_alloc_pabd+0x166/frame 0xfffffe00df912820 >> arc_write_ready() at arc_write_ready+0x463/frame 0xfffffe00df912860 >> zio_ready() at zio_ready+0xde/frame 0xfffffe00df9128a0 >> zio_execute() at zio_execute+0x17c/frame 0xfffffe00df9128e0 >> taskqueue_run_locked() at taskqueue_run_locked+0x10c/frame 0xfffffe00df912940 >> taskqueue_thread_loop() at taskqueue_thread_loop+0x88/frame 0xfffffe00df912970 >> fork_exit() at fork_exit+0x84/frame 0xfffffe00df9129b0 >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00df9129b0 >> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- >> >> Full text dump is at https://people.freebsd.org/~yuripv/core.txt.0. >> >> Real problem or failing hardware (memory?)? > Given a single bit difference between 0xdead40de and 0xdeadc0de, it can be > either. If the memory is not ECC, then I would assume a hardware problem first. > > so assuming that we know what type of object it was, if it is being reused, is that bit something that is or'd or anded?