From nobody Fri Jun 9 13:59:58 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qd2lN5zYrz4c86t for ; Fri, 9 Jun 2023 14:00:00 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-yw1-x1134.google.com (mail-yw1-x1134.google.com [IPv6:2607:f8b0:4864:20::1134]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qd2lN4Dgbz4ZM4; Fri, 9 Jun 2023 14:00:00 +0000 (UTC) (envelope-from mavbsd@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x1134.google.com with SMTP id 00721157ae682-565bd368e19so15728067b3.1; Fri, 09 Jun 2023 07:00:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686319199; x=1688911199; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=+LjFrVpVUv0jij90JIoEZW3iuV2frgBjVWUKPFWxh/E=; b=FEbsHNPugL0+w0Jq6vaLuYs+F+PttJDDJBaOKc43yI8nuZI2ygsCeQ0e/5i5Dru/fB wJiJjKOE5jEmlXhI7e+FJxfrnWBeEs5uvAAOYKfXBm39MgaAO1x6xqpb19ovxKd0puT5 ViKdloCBoA1jH0Grb0cdsoLZtSwffPgRQhE/VTCEw9yA1eg1wONFcQyPhz1MWQtNY8Jp Dz5MRtGLcPGlUzt19ZqDO8bH1Qb0yO1xCUcO62Jc+EtYBfiVOT0674iCiOfpXPQE313y rApKJEA5QSeK/iY65x5UcvgEWzUKTqZbAsn2+CnEStK3oNCRfVNDcm6x8LpaSvwBJlvH Jong== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686319199; x=1688911199; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+LjFrVpVUv0jij90JIoEZW3iuV2frgBjVWUKPFWxh/E=; b=NBXzUGbqp7/bkBUcCYljIAky1CL0foXx2fbabvmk+hLa4XQfORaqQeu8jeIQXeGMGY dgadKVjr1YCVZcBEITb3TvcNGFXWMddv6jZfxyHpjQ3jiypBRAjgezOJQxMY0ell6lpm sC/lF7HNNt//fWdseZ7FXbBj46kotxlB9AKV0ovO47tPc6KSkhg096+DQIoT03Jzht52 5lbKonjs84EiL3LShtLhIFMZQX/HwMbjGDvLrgzZaECK7q2BUHvPw5Db6oQYT3LbavBL BK+jINDn1IPeM4348wBA9kk2SEgi9Mz3ncEkUr8IbNYMDX1U4nz3MW58J5oRwB+FTl7c NkvA== X-Gm-Message-State: AC+VfDzQAizmOtt5tRYK1No2QguyXubhgZCBINH0zEMT5O5RwHG4Rouw z2c+FQijUaBTcChNWUAlf8hPiaVYaeE= X-Google-Smtp-Source: ACHHUZ5TOAs3mMLLEQu0UpnyWHOvqiwqeEWNC3HgUryoE3tzMs5rMJoSQ6DE6Do1Yh6MX87KnUV1Vg== X-Received: by 2002:a81:4946:0:b0:55a:ad7d:3f38 with SMTP id w67-20020a814946000000b0055aad7d3f38mr1220317ywa.26.1686319199430; Fri, 09 Jun 2023 06:59:59 -0700 (PDT) Received: from [10.230.45.5] ([38.32.73.2]) by smtp.gmail.com with ESMTPSA id j77-20020a819250000000b00565862c5e90sm615460ywg.83.2023.06.09.06.59.58 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 09 Jun 2023 06:59:59 -0700 (PDT) Message-ID: <07c3ec8e-5089-d171-0c8d-1916249f9fd7@FreeBSD.org> Date: Fri, 9 Jun 2023 09:59:58 -0400 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: panic(s) in ZFS on CURRENT Content-Language: en-US To: Gleb Smirnoff , current@freebsd.org Cc: mm@freebsd.org References: From: Alexander Motin In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4Qd2lN4Dgbz4ZM4 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N Hi Gleb, There are two probably related PRs upstream: https://github.com/openzfs/zfs/pull/14939 https://github.com/openzfs/zfs/pull/14954 On 09.06.2023 00:57, Gleb Smirnoff wrote: > On Thu, Jun 08, 2023 at 07:56:07PM -0700, Gleb Smirnoff wrote: > T> I'm switching to INVARIANTS kernel right now and will see if that panics earlier. > > This is what I got with INVARIANTS: > > panic: VERIFY3(dev->l2ad_hand <= dev->l2ad_evict) failed (225142071296 <= 225142063104) > > cpuid = 17 > time = 1686286015 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2c/frame 0xfffffe0160dcea90 > kdb_backtrace() at kdb_backtrace+0x46/frame 0xfffffe0160dceb40 > vpanic() at vpanic+0x21f/frame 0xfffffe0160dcebe0 > spl_panic() at spl_panic+0x4d/frame 0xfffffe0160dcec60 > l2arc_write_buffers() at l2arc_write_buffers+0xcda/frame 0xfffffe0160dcedf0 > l2arc_feed_thread() at l2arc_feed_thread+0x547/frame 0xfffffe0160dceec0 > fork_exit() at fork_exit+0x122/frame 0xfffffe0160dcef30 > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0160dcef30 > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > Uptime: 1m4s > Dumping 5473 out of 65308 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > > (kgdb) frame 4 > #4 0xffffffff804342ea in l2arc_write_buffers (spa=0xfffffe022e942000, dev=0xfffffe023116a000, target_sz=16777216) > at /usr/src/FreeBSD/sys/contrib/openzfs/module/zfs/arc.c:9445 > 9445 ASSERT3U(dev->l2ad_hand, <=, dev->l2ad_evict); > (kgdb) p dev > $1 = (l2arc_dev_t *) 0xfffffe023116a000 > (kgdb) p dev->l2ad_hand > $2 = 225142071296 > (kgdb) p dev->l2ad_evict > $3 = 225142063104 > (kgdb) p *dev > value of type `l2arc_dev_t' requires 66136 bytes, which is more than max-value-size > > Never seen kgdb not being able to print a structure that reported to be too big. > -- Alexander Motin