From nobody Mon Oct 16 13:17:22 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S8Hhx2nthz4xMTc for ; Mon, 16 Oct 2023 13:17:37 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: from mail-yw1-f178.google.com (mail-yw1-f178.google.com [209.85.128.178]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S8Hhw3KfQz4HP5 for ; Mon, 16 Oct 2023 13:17:36 +0000 (UTC) (envelope-from ctuffli@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of ctuffli@gmail.com designates 209.85.128.178 as permitted sender) smtp.mailfrom=ctuffli@gmail.com; dmarc=none Received: by mail-yw1-f178.google.com with SMTP id 00721157ae682-5a7b92cd0ccso56656007b3.1 for ; Mon, 16 Oct 2023 06:17:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697462254; x=1698067054; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nhCeUhq9kGupkRPvL1Ai+/Km9VAUD2jMQ1Psdw+4HRU=; b=Butm3pxcGk5WF8ogx051HaIcIeRuJY8TLF8eN3A2e7UJFVhq8iTnMp7985ia4bvnit ln+vpwkR3PUci2nrx/RNeD6+OPH1aZJ9Yi9bZpDvCTP2X5xhBJnWdp5c2zGNn15WYdHp w80rNcGmm9Q5/PeJtSrR/XmJO/h892IUm3p/X94/Y4JSxgElIhIsU/0M4SxcxvzlH+LS /mwhHSxHuS0SA6Lpa7NJ1OMFfJOICnlTLthryBLB6Dob4uluBNweUING3nMxpwzfmeSB tsCmDVxa98bamYmsbrpVRcddLDVZsBebgsipZ4Hx218alI8hX0AjsVdgUodhk/sEf8U0 h8SQ== X-Gm-Message-State: AOJu0Yy1nMLRk4uCNX6qqCzUpGRNTY3qhg1JeHNfQ+2GPRvUiR6s94VM zzL06snGoiX7hMDMqwCBPqwA/dxUA7k= X-Google-Smtp-Source: AGHT+IFm/IJKTXFsiPm20/5Wp8Q9xhLQbj8yLF5DTu7/FOMO+sfH2pXefEL7hcpyVDGmu36CeKjMSw== X-Received: by 2002:a81:a014:0:b0:5a7:bc66:949f with SMTP id x20-20020a81a014000000b005a7bc66949fmr18654043ywg.26.1697462254289; Mon, 16 Oct 2023 06:17:34 -0700 (PDT) Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com. [209.85.160.175]) by smtp.gmail.com with ESMTPSA id lg17-20020a056214549100b0064f4e0b2089sm3412283qvb.33.2023.10.16.06.17.34 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 16 Oct 2023 06:17:34 -0700 (PDT) Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-41b406523fcso31864161cf.2 for ; Mon, 16 Oct 2023 06:17:34 -0700 (PDT) X-Received: by 2002:a05:622a:6dc3:b0:412:be9:87fa with SMTP id ir3-20020a05622a6dc300b004120be987famr27427182qtb.48.1697462253817; Mon, 16 Oct 2023 06:17:33 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <90d3e532-8ea7-4eea-8e31-8c363285a156@nomadlogic.org> <0ad493d5-1c1e-4370-977a-118f46ebd677@nomadlogic.org> <0c4f8149-89dd-4635-a5ed-4766fffd2553@nomadlogic.org> In-Reply-To: From: Chuck Tuffli Date: Mon, 16 Oct 2023 06:17:22 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: nvme timeout issues with hardware and bhyve vm's To: Warner Losh Cc: Pete Wright , FreeBSD Current Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.31 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.91)[-0.913]; NEURAL_HAM_SHORT(-0.39)[-0.395]; FORGED_SENDER(0.30)[chuck@freebsd.org,ctuffli@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; MIME_GOOD(-0.10)[text/plain]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.128.178:from]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.128.178:from,209.85.160.175:received]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; FROM_NEQ_ENVFROM(0.00)[chuck@freebsd.org,ctuffli@gmail.com]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; BLOCKLISTDE_FAIL(0.00)[209.85.160.175:server fail,209.85.128.178:server fail]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[freebsd.org]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4S8Hhw3KfQz4HP5 On Fri, Oct 13, 2023 at 7:34=E2=80=AFPM Warner Losh wrote: ... > Let me now if you see similar messages in stable/14. I think I've fixed a= ll the > issues with timeouts, though you shouldn't ever seem them in a vm setup > unless something else weird is going on. I'd be interested in a repo case too as I haven't seen the NVMe emulation in bhyve do this before. Were there any error messages from bhyve? The guest log messages seem to suggest that the backing storage for the emulated device is timing out. If your comment > I had similar issues on my workstation as well. Scrubbing the NVMe > device on my real-hardware workstation hasn't turned up any issues, but > the system has locked up a handful of times. means the host is seeing NVMe errors on the drives backing the zpool/zvol used by the emulated device, this might explain it. Although, it is curious the emulated controller had trouble resetting (i.e., the error message "controller ready did not become 1 within 30500 ms"). --chuck