From nobody Sun Apr 9 22:15:57 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pvmds1hy4z44M0y; Sun, 9 Apr 2023 22:16:01 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oi1-x236.google.com (mail-oi1-x236.google.com [IPv6:2607:f8b0:4864:20::236]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pvmdr0m5Mz4NXm; Sun, 9 Apr 2023 22:16:00 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=ITe0UASO; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2607:f8b0:4864:20::236 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oi1-x236.google.com with SMTP id e9so2175460oig.7; Sun, 09 Apr 2023 15:16:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1681078558; x=1683670558; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=OoWj940RdtoeSQEx4KHZMdisRHXku0JQPs4Z2xGawL8=; b=ITe0UASOGDvlI/KidV2zTVugzd9M8Z1qg2MbqMBnW+UTQr55yYrOKRvqcxChn/QsJj lkSQdlOzfE6n9moIarEsQyr1JGF1sR1Gr+1TJ1RhoRg90pv/KuS1TrcZ/nKDWwzQqTPk 8EwbWsON6ocHHDhtuSB3S6htMIkYorc1rvAwXP8Fb48LgzUCHssKoulTLKNOmHxfI3aL Dowg2sHh+blJxekw+/lws0w2u5llnvmsCG01r16sedQikaNLYdwTkt11fNjw7HnOQ3wC ywNB+7lzHi8NB0ff9wUAPtslvDSPZRz5vIBf/CByi+J7fcUDiyVdG0wJsZvyFi5DACJV Pc7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681078558; x=1683670558; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=OoWj940RdtoeSQEx4KHZMdisRHXku0JQPs4Z2xGawL8=; b=IefI3kS1ZCFc8gE16TTp0f8ip0mhITUV2PgPizhXtf/k4jKt2ya6Zt+sUnriFT36Ox Z5GlYDGP0dIzjD84RCMZbWxO3hOo/Z2/oMwXAH5rynv/ICDtEp9eQnJMbLOAaqD+Jedg NBbfsQ3GHty1WD7DCh5Mj0GWzT69L4pLoaqoyWRb2B7NYCmMZSEX9H5dd36x9QS70ukR PoIcV1fgdBP+BlNGmfrFc1wrgP3DM2my/0PEeEDIOfPo29J0e75EgU6e+/m8g0kePELN bebexlQ+MaeiQE1hjW3iweqOLCeCC5rev35dEOEMAdU8mw/yrsj+OKfUgS1OhhYfGC0U y1Qg== X-Gm-Message-State: AAQBX9cUXPaSTR+piWCNZbOCWjzPz4yGI+A4yy6w6X8RQOz8KAy1jOXU j5v4tDZBahUmmhQV0AePSb+EEmjyZaBe7ih8ymY= X-Google-Smtp-Source: AKy350bmsRQECNb0w2kU3JiMTQ7sUDtJq87orkDnMuBNsW4MKT8K8bgDlFvQYjbzfH9j1F/Lho8KrYM9t5mTXQoKDZo= X-Received: by 2002:a05:6808:8c4:b0:387:82c4:6425 with SMTP id k4-20020a05680808c400b0038782c46425mr1026139oij.4.1681078558073; Sun, 09 Apr 2023 15:15:58 -0700 (PDT) List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:798d:0:b0:49c:b071:b1e3 with HTTP; Sun, 9 Apr 2023 15:15:57 -0700 (PDT) In-Reply-To: References: <202304031513.333FD6qw014903@gitrepo.freebsd.org> <20230403231444.CF48911F@slippy.cwsent.com> <20230403232549.73E331A2@slippy.cwsent.com> <20230403235851.84C0467@slippy.cwsent.com> <20230404052811.DA2172C1@slippy.cwsent.com> <7c75b934-cb0a-b32e-bc19-b1e15e8cf3aa@freebsd.org> <20230409202650.49130b92@thor.intern.walstatt.dynvpn.de> From: Mateusz Guzik Date: Mon, 10 Apr 2023 00:15:57 +0200 Message-ID: Subject: Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75 To: FreeBSD User Cc: Charlie Li , Cy Schubert , Rick Macklem , Martin Matuska , src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-2.49 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.988]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[dev-commits-src-all@freebsd.org,dev-commits-src-main@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::236:from]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; FREEMAIL_CC(0.00)[freebsd.org,cschubert.com,gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_SEVEN(0.00)[8]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; TO_DN_SOME(0.00)[] X-Rspamd-Queue-Id: 4Pvmdr0m5Mz4NXm X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On 4/9/23, Mateusz Guzik wrote: > On 4/9/23, FreeBSD User wrote: >> Am Sun, 9 Apr 2023 13:23:05 -0400 >> Charlie Li schrieb: >> >>> Mateusz Guzik wrote: >>> > On 4/9/23, Charlie Li wrote: >>> >> I've also started noticing random artefacts and malformed files >>> >> whilst >>> >> building packages with poudriere, causing all sorts of "exec format >>> >> error"s, missing .so files due to corruption, data file corruption >>> >> causing unintended failure modes, etc. All without block_cloning; >>> >> enabling such causes a panic of its own when starting multiple >>> >> builder >>> >> jails at once. >>> >> >>> > >>> > what's the panic? >>> > >>> manually typed out: >>> >>> panic: VERIFY(!zil_replaying(zilog, tx)) failed >>> >>> cpuid = 7 >>> time = 1681060472 >>> KDB: stack backtrace: >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >>> 0xfffffe02a05b28a0 >>> vpanic() at vpanic+0x152/frame 0xfffffe02a05b28f0 >>> spl_panic() at spl_panic+0x3a/frame 0xfffffe02a05b2950 >>> zfs_log_clone_range() at zfs_log_clone_range+0x1db/frame >>> 0xfffffe02a05b29e0 >>> zfs_clone_range() at zfs_clone_range+0xae2/frame 0xfffffe02a05b2bc0 >>> zfs_freebsd_copy_file_range() at zfs_freebsd_copy_file_range+0xff/frame >>> 0xfffffe02a05b2c40 >>> vn_copy_file_range() at vn_copy_file_range+0x115/frame >>> 0xfffffe02a05b2ce0 >>> kern_copy_file_range() at kern_copy_file_range+0x34e/frame >>> 0xfffffe02a05b2db0 >>> sys_copy_file_range() at sys_copy_file_range+0x78/frame >>> 0xfffffe02a05b2e00 >>> amd64_syscall() at amd64_syscall+0x148/frame 0xfffffe02a05b2f30 >>> fast_syscall_common() at fast_syscall_common+0xf8/frame >>> 0xfffffe02a05b2f30 >>> --- syscall (569, FreeBSD ELF64, copy_file_range), rip = 0x908d2a, rsp = >>> 0x820c28e68, rbp = 0x820c292b0 --- >>> KDB: enter: panic >>> [ thread pid 1856 tid 102129 ] >>> Stopped at kdb_enter+0x32: movq $0,0x12760f3(%rip) >>> db> >>> >> >> I have the same issue (crash on access of several, but random datasets). >> >> It started with /usr/ports build failures when performing updates or >> rebuilding ports, >> poudriere host doesn't work anymore, as soon as started building ports, >> the >> hosts (several of >> them, same OS revision, new ZFS option enabled) crash. >> Also when building binaries for an pkg OS distribution. >> >> That host also reports a ZFS RAIDZ pool as corrupted, out of the blue! >> Some >> files from a >> poudriere build and /usr/ports build seem to have issues with some >> temporarily created files >> in work directory. >> >> On another host /usr/ports is residing on ZFS and it crashes also when >> building/updating ports >> (/usr/ports residing on ZFS) - but on the same host /home is also >> residing >> on ZFS, but even >> downloading large amounts of emails, the host seem to be stable. Have not >> found out yet what >> kind of file access triggers the crash. >> > > I reproduced the VERIFY(!zil_replaying(zilog, tx)) panic. As the > backtrace shows it triggers when using copy_file_range, I temporarily > patched the kernel to never do block cloning. So far the only package > which failed to build was sqlite and it was for a legitimate reason > (compiler errored out due to a problem in the code). > ... and got an illegitimate failure: strip: file format not recognized the port builds after retrying iow there is more breakage. i don't know if the merge can be easily reverted now, will have to see about that -- Mateusz Guzik