From nobody Mon Mar 7 15:13:37 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DB13619EA01A; Mon, 7 Mar 2022 15:13:41 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KC26F0Lm9z3ltp; Mon, 7 Mar 2022 15:13:41 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qt1-x835.google.com with SMTP id b23so13476645qtt.6; Mon, 07 Mar 2022 07:13:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ke+WAIsIlx2cs1GtXexL6lDfk0mefkBVKXgofAu6iss=; b=BpTPXuU3R9MN/OHmXFHiq0A5tXnaua8zmaMgzkGmYm0a6E3OPCfSu8hQc4t+4W3bZN HM3Vtl4FW7/YPIKjMKRGqgGLC0fmdXp+uWInKBlKalpeywsEk4fSSi04m72MijRILcNW tIKsnM5yXpVMkurPP9M/YlTp1gyqkM6nBP8fgv7Cg8vw84OjIxPoTIm468P/oQKlnD+s kl/4/RQOQerohsWrs+76faqcxTQWSYuU1v7086barR+q3b8hKgsQ44zMCRuSNb4QGZWa U2sCXak5Z0KA49umbPPOwljEgI9bKp/RmJVje2OdQpnum6oz/jzK1e3/jQORdEngTqjc 87Ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to; bh=ke+WAIsIlx2cs1GtXexL6lDfk0mefkBVKXgofAu6iss=; b=WdGEQ2LTJIVfwnHvOy7/OAGuStOm5L6PmA7dHv3q14K8oR3qE5Wb+TZgDaVlwH1Ns9 xgFTT4/xeHEsKS8AR6PNMj39MicuS+kPPfQTrBHiHiHLEon2d0l5UfCF6cfZpgX1vR3C WqvL6XnqKPNl5fFP1s/oDWqZgcuoNwqxjEZ8w8BEqnK374auFJH+1BSePRsgqCqKY+Ji 0RzZDRgaYIfY0FKfydUq5zV8DUVloDLYGupsO5BEwQEj0FEPTIsYYqYRh5JgkHGhe9AW JxDWEoXaRGTD8SjONjuyRmKf7IHxgefsTUCI7oTHVJ9+mqGnIXY9xAfN86au4jUuD0yv nlVQ== X-Gm-Message-State: AOAM533qSuWTaPmtn8fKbBOIhp8k6So6lbj01tBN999nYwa52UhhykzQ MKrbnlQhgGnZ8dghkDGOxgY= X-Google-Smtp-Source: ABdhPJwHRV9IVngDb8Or+gaomMiu8lsMa9kNgVNzqo6DDq2FQqkpa/WosWGUb0VO6LLFGxHDylQVYQ== X-Received: by 2002:ac8:5b0f:0:b0:2e0:5a1f:9910 with SMTP id m15-20020ac85b0f000000b002e05a1f9910mr8892131qtw.356.1646666020161; Mon, 07 Mar 2022 07:13:40 -0800 (PST) Received: from nuc (198-84-189-58.cpe.teksavvy.com. [198.84.189.58]) by smtp.gmail.com with ESMTPSA id w9-20020a05620a148900b005f188f755adsm6304214qkj.32.2022.03.07.07.13.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Mar 2022 07:13:39 -0800 (PST) Date: Mon, 7 Mar 2022 10:13:37 -0500 From: Mark Johnston To: Ronald Klop Cc: bob prohaska , Mark Millard , freebsd-arm@freebsd.org, freebsd-current Subject: Re: panic: data abort in critical section or under mutex (was: Re: panic: Unknown kernel exception 0 esr_el1 2000000 (on 14-CURRENT/aarch64 Feb 28)) Message-ID: References: <1800459695.1.1646649539521@mailrelay> <132978150.92.1646660769467@mailrelay> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <132978150.92.1646660769467@mailrelay> X-Rspamd-Queue-Id: 4KC26F0Lm9z3ltp X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=BpTPXuU3; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::835 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-2.70 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::835:from]; MLMMJ_DEST(0.00)[freebsd-arm,freebsd-current]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_CC(0.00)[www.zefox.net,yahoo.com,freebsd.org]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On Mon, Mar 07, 2022 at 02:46:09PM +0100, Ronald Klop wrote: > Dear Mark Johnston, > > I did some binary search in the kernels and came to the conclusion that https://cgit.freebsd.org/src/commit/?id=1517b8d5a7f58897200497811de1b18809c07d3e still works and https://cgit.freebsd.org/src/commit/?id=407c34e735b5d17e2be574808a09e6d729b0a45a panics. > > I suspect your commit in https://cgit.freebsd.org/src/commit/?id=c84bb8cd771ce4bed58152e47a32dda470bef23a. > > Last panic: > > panic: vm_fault failed: ffff00000046e708 error 1 > cpuid = 1 > time = 1646660058 > KDB: stack backtrace: > db_trace_self() at db_trace_self > db_trace_self_wrapper() at db_trace_self_wrapper+0x30 > vpanic() at vpanic+0x174 > panic() at panic+0x44 > data_abort() at data_abort+0x2e8 > handle_el1h_sync() at handle_el1h_sync+0x10 > --- exception, esr 0x96000004 > _rm_rlock_debug() at _rm_rlock_debug+0x8c > osd_get() at osd_get+0x5c > zio_execute() at zio_execute+0xf8 > taskqueue_run_locked() at taskqueue_run_locked+0x178 > taskqueue_thread_loop() at taskqueue_thread_loop+0xc8 > fork_exit() at fork_exit+0x74 > fork_trampoline() at fork_trampoline+0x14 > KDB: enter: panic > [ thread pid 0 tid 100129 ] > Stopped at kdb_enter+0x44: undefined f902011f > db> > > A more recent kernel (912df91) still panics. See below. > > Do you have time to look into this? What can I provide in information to help? Hmm. So after my rmlock commits, we have the following disassembly for _rm_rlock() (with a few annotations added by me). Note that the pcpu pointer is stored in register x18 by convention. 0xffff00000046e304 <+0>: stp x29, x30, [sp, #-16]! 0xffff00000046e308 <+4>: mov x29, sp 0xffff00000046e30c <+8>: ldr x8, [x18] 0xffff00000046e310 <+12>: ldr x9, [x18] 0xffff00000046e314 <+16>: ldr x10, [x18] 0xffff00000046e318 <+20>: cmp x9, x10 0xffff00000046e31c <+24>: b.ne 0xffff00000046e3cc <_rm_rlock+200> // b.any 0xffff00000046e320 <+28>: ldr x9, [x18] 0xffff00000046e324 <+32>: ldrh w9, [x9, #314] 0xffff00000046e328 <+36>: cbnz w9, 0xffff00000046e3c0 <_rm_rlock+188> 0xffff00000046e32c <+40>: str wzr, [x1, #32] 0xffff00000046e330 <+44>: stp x0, x8, [x1, #16] 0xffff00000046e334 <+48>: ldrb w9, [x0, #10] 0xffff00000046e338 <+52>: tbz w9, #4, 0xffff00000046e358 <_rm_rlock+84> 0xffff00000046e33c <+56>: ldr x9, [x18] 0xffff00000046e340 <+60>: ldr w10, [x9, #888] 0xffff00000046e344 <+64>: add w10, w10, #0x1 0xffff00000046e348 <+68>: str w10, [x9, #888] 0xffff00000046e34c <+72>: ldr x9, [x18] 0xffff00000046e350 <+76>: ldr w9, [x9, #888] 0xffff00000046e354 <+80>: cbz w9, 0xffff00000046e3f4 <_rm_rlock+240> 0xffff00000046e358 <+84>: ldr w9, [x8, #1212] 0xffff00000046e35c <+88>: add x10, x18, #0x90 0xffff00000046e360 <+92>: add w9, w9, #0x1 0xffff00000046e364 <+96>: str w9, [x8, #1212] <------- critical_enter 0xffff00000046e368 <+100>: str x10, [x1, #8] 0xffff00000046e36c <+104>: ldr x9, [x18, #144] 0xffff00000046e370 <+108>: str x9, [x1] 0xffff00000046e374 <+112>: str x1, [x9, #8] 0xffff00000046e378 <+116>: str x1, [x18, #144] 0xffff00000046e37c <+120>: ldr x9, [x18] 0xffff00000046e380 <+124>: ldr w10, [x9, #356] 0xffff00000046e384 <+128>: add w10, w10, #0x1 0xffff00000046e388 <+132>: str w10, [x9, #356] 0xffff00000046e38c <+136>: ldr w9, [x8, #1212] 0xffff00000046e390 <+140>: sub w9, w9, #0x1 0xffff00000046e394 <+144>: str w9, [x8, #1212] <------- critical_exit 0xffff00000046e398 <+148>: ldrb w8, [x8, #304] 0xffff00000046e39c <+152>: ldr w9, [x18, #60] <------- loading &pc->pc_cpuid ... A (the?) problem is that the compiler is treating "pc" as an alias for x18, but the rmlock code assumes that the pcpu pointer is loaded once, as it dereferences "pc" outside of the critical section. On arm64, if a context switch occurs between the store at _rm_rlock+144 and the load at +152, and the thread is migrated to another CPU, then we'll end up using the wrong CPU ID in the rm->rm_writecpus test. I suspect the problem is unique to arm64 as its get_pcpu() implementation is different from the others in that it doesn't use volatile-qualified inline assembly. This has been the case since https://cgit.freebsd.org/src/commit/?id=63c858a04d56529eddbddf85ad04fc8e99e73762 . I haven't been able to reproduce any crashes running poudriere in an arm64 AWS instance, though. Could you please try the patch below and confirm whether it fixes your panics? I verified that the apparent problem described above is gone with the patch. diff --git a/sys/kern/kern_rmlock.c b/sys/kern/kern_rmlock.c index 0cdcfb8fec62..e51c25136ae0 100644 --- a/sys/kern/kern_rmlock.c +++ b/sys/kern/kern_rmlock.c @@ -437,6 +437,7 @@ _rm_rlock(struct rmlock *rm, struct rm_priotracker *tracker, int trylock) { struct thread *td = curthread; struct pcpu *pc; + int cpuid; if (SCHEDULER_STOPPED()) return (1); @@ -452,6 +453,7 @@ _rm_rlock(struct rmlock *rm, struct rm_priotracker *tracker, int trylock) atomic_interrupt_fence(); pc = get_pcpu(); + cpuid = pc->pc_cpuid; rm_tracker_add(pc, tracker); sched_pin(); @@ -463,7 +465,7 @@ _rm_rlock(struct rmlock *rm, struct rm_priotracker *tracker, int trylock) * conditional jump. */ if (__predict_true(0 == (td->td_owepreempt | - CPU_ISSET(pc->pc_cpuid, &rm->rm_writecpus)))) + CPU_ISSET(cpuid, &rm->rm_writecpus)))) return (1); /* We do not have a read token and need to acquire one. */