From nobody Thu Jul 21 15:31:38 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Lpc4R4LvHz4WtDP for ; Thu, 21 Jul 2022 15:31:51 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Lpc4R0987z4FCt for ; Thu, 21 Jul 2022 15:31:51 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: by mail-ed1-x535.google.com with SMTP id t3so2673542edd.0 for ; Thu, 21 Jul 2022 08:31:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=TGGGHsWrtYhxbjhIhTUfKgCII098u+xtS1Ii6kZJblI=; b=P9BHwHF5/O9cC0iNqKFTZRE4fydjOPMe9TAf0TpfSgmsIwiin9ExVWxSUlea5Rj0mk AHPTtHqVj2qUvYHY1btP4Y8K8Y9Zxp1ctya9gatQvwI3QDcMNRKqellIf+UtIjRLSDqj GhqCEJQLM7LkvZ/f7JSDRYlPQ///bk1gMA/abl6KL8lFZnQ8ec58xlWVK0Ff7N02f0aZ 4NYHQdyS3DwHPdvTiHo8Q9pADTvoMWFMQinHZ0n/NgWJBoW6/vc7IF/AEHjcqM0BxPoz ZNDLjKPz8t26FZIz8yiIRfHtDVh+d9aoR+0ZU/+jJo5nHXr5G/JvvUdN9CE83Widb4HO Jkog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=TGGGHsWrtYhxbjhIhTUfKgCII098u+xtS1Ii6kZJblI=; b=OdPyRq8QI3a0TxdentnA6rn2BAy6MKyQfz5XckoKwq5OmTX45u0cKmFOS2XcZvD1VN RC/chIPLBX778Jb59GD4IWjYeguuQJmkt6222sm8Fz1OPxW3MAEXT8p8uPvagfOgD+X9 RWloXc2w43KrK550nrHLgVjN+KRgTcBgZSt0GHMm7dpfJHBiSQ4jVWkFNJC2JHFBzi7s UvzK4vctjSOs3kbmeSh826N3skT97L5KtjFo/AunbAH+A8ROadKBQZDqusLGQ1GBl1ID SjuGV4UEt6O1qQ05rtMerNNSL1IZJVsWHs0upKAcgw42b/jWpefPQir1D0+zgpNXmBxQ PiCA== X-Gm-Message-State: AJIora8oG3nLEgRS01Sc1BRhRH/mMJ61B86vD0HRWWHxdAccvT/Dwg9a GZMIY8q9aa5YXmCrPzk9Qu7nmXC9bsqlb0j2n+OsM0SUI9sp3Q== X-Google-Smtp-Source: AGRyM1ty1hWhWQQZHHNZvSJWZXNFMA88hx7sMUwhqttJTXf0S42ahDh97sIm1zHH4DadipEuRhdp1jURuwdFebqhGGY= X-Received: by 2002:a05:6402:3807:b0:435:20fb:318d with SMTP id es7-20020a056402380700b0043520fb318dmr55863071edb.272.1658417509616; Thu, 21 Jul 2022 08:31:49 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 From: Chuck Tuffli Date: Thu, 21 Jul 2022 08:31:38 -0700 Message-ID: Subject: bhyve core dump related to llvm 14 To: FreeBSD-Current Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4Lpc4R0987z4FCt X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=P9BHwHF5; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of ctuffli@gmail.com designates 2a00:1450:4864:20::535 as permitted sender) smtp.mailfrom=ctuffli@gmail.com X-Spamd-Result: default: False [-3.94 / 15.00]; NEURAL_HAM_LONG(-1.00)[-0.998]; NEURAL_HAM_SHORT(-1.00)[-0.996]; NEURAL_HAM_MEDIUM(-0.94)[-0.944]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::535:from]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N I have a virtual machine used to test the NVMe emulation in bhyve. All of the tests in the VM pass running under FreeBSD 13.1-R, but the same VM running under -current causes bhyve(8) to dump core because of a segmentation fault. git bisect identified the last "good" commit on main as cb2ae6163174 sysvsem: Fix a typo After this commit, there are a half-dozen commits related to merging the llvm project release/14.x The core dump is repeatable and consistent. Back traces under lldb look similar to this: * thread #22, name = 'vcpu 2', stop reason = signal SIGSEGV: invalid address (fault address: 0xb8) * frame #0: 0x0000383eb9fc916b bhyve`pci_nvme_read(ctx=0x000038483ad2d700, vcpu=0, pi=0x0000000000000000, baridx=-188391150, offset=0, size=0) at pci_nvme.c:3035:34 frame #1: 0x0000384834616280 frame #2: 0x0000383eb9fc1f7a bhyve`pci_emul_mem_handler(ctx=, vcpu=, dir=, addr=, size=, val=, arg1=0x00003846e5b71600, arg2=0) at pci_emul.c:498:4 In frame 0, pi being NULL causes the core dump, but most of the arguments are invalid / garbage. Looking earlier in the stack, the vcpu value should be 2, the ctx pointer doesn't match, and the value passed to pi isn't NULL. Poking around in frame 2, I can see that the "direction" is a memory write (dir == MEM_F_WRITE) and the statement being executed is this: (*pe->pe_barwrite)(ctx, vcpu, pdi, bidx, offset, size, *val); Confusingly, the function pointer pe_barwrite is pci_nvme_write() and not pci_nvme_read() where the crash occurs. I've confirmed the fault is in pci_nvme_read() by adding an assert for pi != NULL. This is especially odd because pci_emul_mem_handler() directly calls pci_nvme_read() and pci_nvme_write(). So why does frame 1 exist at all? Using gdb, the back traces either don't decode at all or look similar to this: (gdb) bt #0 pci_nvme_read (ctx=0x944c1168700, vcpu=0, pi=0x0, baridx=-1835053270, offset=0, size=0) at /poudriere/jails/14-current-amd64/usr/src/usr.sbin/bhyve/pci_nvme.c:3035 #1 0x000009436891d8e8 in _CurrentRuneLocale () from /lib/libc.so.7 #2 0x000009436a73ca28 in ?? () #3 0x000009436a73e1c0 in ?? () ... #34 0x000009436a747600 in ?? () #35 0x0000093b3e76b088 in pci_de_lpc () #36 0x000009436a716500 in ?? () #37 0x00000944c3196d10 in ?? () #38 0x0000093b3e74501a in pci_emul_mem_handler (ctx=0x9436a7bd670, vcpu=0, dir=, addr=, size=0, val=0x646165725f657469, arg1=0x1, arg2=10185153275136) at /poudriere/jails/14-current-amd64/usr/src/usr.sbin/bhyve/pci_emul.c:498 Other random tidbits: - disabling compiler optimization (i.e. -O0) for the two files in question (pci_nvme.c and pci_emul.c) makes the core dump go away - using the default optimization level but generously sprinkling debug printf everywhere makes the core dump go away. I'm not sure where to go from here and could use some help. --chuck