From owner-freebsd-virtualization@freebsd.org Sun Sep 6 21:00:09 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AA19A3DD6B3 for ; Sun, 6 Sep 2020 21:00:09 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4Bl3hT47MVz4T0M for ; Sun, 6 Sep 2020 21:00:09 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id 8DC603DD4F8; Sun, 6 Sep 2020 21:00:09 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8D96D3DD7A3 for ; Sun, 6 Sep 2020 21:00:09 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bl3hT3Jh5z4T2D for ; Sun, 6 Sep 2020 21:00:09 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 569FF14A1C for ; Sun, 6 Sep 2020 21:00:09 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 086L09es021679 for ; Sun, 6 Sep 2020 21:00:09 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 086L09j4021678 for virtualization@FreeBSD.org; Sun, 6 Sep 2020 21:00:09 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202009062100.086L09j4021678@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: virtualization@FreeBSD.org Subject: Problem reports for virtualization@FreeBSD.org that need special attention Date: Sun, 6 Sep 2020 21:00:09 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Sep 2020 21:00:09 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 240945 | [hyper-v] [netvsc] hn network driver incorrectly 1 problems total for which you should take action. From owner-freebsd-virtualization@freebsd.org Sun Sep 6 22:31:27 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 16A153E08B0 for ; Sun, 6 Sep 2020 22:31:27 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bl5jn6sQZz4YTg for ; Sun, 6 Sep 2020 22:31:25 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: by mail-wr1-f44.google.com with SMTP id o5so13607471wrn.13 for ; Sun, 06 Sep 2020 15:31:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=eZ6GoQ2RudvhDx42rR+j1g/XQ8RsjBO6JQ5MsI2ScqE=; b=On2mkQv4/hlJO19VumsBOWn0WzgtMKsU+N4GkwTe5Mjg0zt0idwRc8PWa/ohwauiwk ef8KrhLNQQ0VUuyx16By0lzEJlYx51n+kOm112oVOb22SEeR2jXEgC40B2ieJdQRGmAw nrRZU4G64I9/TOjoFZUZRXk+KkCbOf/eIxUP+0/2W2vZeDCRPiKwnLIqwKgRZdEZXQVN kW/SfWvu4yQ0Rz7+FPJbfdchfjnFS9wZSK0J0JtRpzcsEA2Yd1rvzCuNmJJpclhLshjq 57zt0CkM17ggzw3Poq7qOwMl+5LzsufGoWOiVlyaCSjL7qbHhywbSuITJL1+jwrFtYcZ uVLA== X-Gm-Message-State: AOAM533f56ApVYOfh2PhIZjmoAL3zBMJCDHr6vc3ZF2Lti4d7pf0K3Ar GzGBeZDdj7yCrroXP2QKyiyz1HkrWZghRI/I7MRJEaH/ X-Google-Smtp-Source: ABdhPJykG8VsAbox2xtydnnAwzyGYWuug3c/+3GAdiHk5B+U+OXTsW4jSo44wSh5NMfji34QO+Z39wX/FKgwCEfDpk0= X-Received: by 2002:adf:f808:: with SMTP id s8mr18246634wrp.268.1599431484063; Sun, 06 Sep 2020 15:31:24 -0700 (PDT) MIME-Version: 1.0 References: <53835548-cd6e-6859-c19d-8dd96d3993f3@asusa.net> <0b2304fd-4799-f56e-2271-169a5348d654@asusa.net> <55041e52-8784-8336-3c41-1d203d6d2f98@asusa.net> <9bee075a-3ad6-76f6-1eb5-d9772c101564@asusa.net> In-Reply-To: <9bee075a-3ad6-76f6-1eb5-d9772c101564@asusa.net> From: Jason Tubnor Date: Mon, 7 Sep 2020 08:32:28 +1000 Message-ID: Subject: Re: IPv6 Problem with Bhyve To: Hiroshi Nishida Cc: "freebsd-virtualization@freebsd.org" X-Rspamd-Queue-Id: 4Bl5jn6sQZz4YTg X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of jtubnor@gmail.com designates 209.85.221.44 as permitted sender) smtp.mailfrom=jtubnor@gmail.com X-Spamd-Result: default: False [-2.85 / 15.00]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_DN_EQ_ADDR_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; RCVD_TLS_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; DMARC_NA(0.00)[tubnor.net]; NEURAL_HAM_LONG(-1.01)[-1.013]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.84)[-0.843]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.221.44:from]; NEURAL_HAM_MEDIUM(-0.99)[-0.994]; FORGED_SENDER(0.30)[jason@tubnor.net,jtubnor@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.221.44:from]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[jason@tubnor.net,jtubnor@gmail.com]; MAILMAN_DEST(0.00)[freebsd-virtualization] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Sep 2020 22:31:27 -0000 On Thu, 3 Sep 2020 at 01:56, Hiroshi Nishida wrote: > Problem solved. > > It was just vertio driver... > Upgrading to 0.1.189 solved everything. > Good to hear. > I think https://wiki.freebsd.org/bhyve/Windows should be updated, or I > can post how to run Win Server 2019 on bhyve to one of FreeBSD forums. > I'll take a look at it and update it where required. We use UEFI only across all operating systems. > > By the way, how stable is Windows Server 2019 on bhyve? > Very stable. We have not had any issues or unexpected crashes with Windows Server 2016 over the last 3+ years, nor Windows Server 2019 since we changed our SOE to that. Cheers. From owner-freebsd-virtualization@freebsd.org Sun Sep 6 23:32:27 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A44B73E2ACD for ; Sun, 6 Sep 2020 23:32:27 +0000 (UTC) (envelope-from nishida@asusa.net) Received: from asusam.asj-hosting.net (asusa.asj-hosting.net [219.118.222.245]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.asj-hosting.net", Issuer "JPRS Domain Validation Authority - G3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bl7496fRhz4cgJ for ; Sun, 6 Sep 2020 23:32:24 +0000 (UTC) (envelope-from nishida@asusa.net) Received: (qmail 15425 invoked by uid 89); 7 Sep 2020 08:32:14 +0900 X-ASJ-Track-ID: <20200906233214.15425.qmail@asusam.asj-hosting.net> X-Spam-Checker-Version: ASJ KMsrv Spam Check Process Internal X-Spam-Status: No, hits=0.0 X-Spam-Flag: No X-Virus-Scanned: ASJ KMsrv Virus Check Process 08041001 X-ASJ-SMTP-Authentication: nishida@asusa.net X-ASJ-Arrival-IP: 73.190.120.36 X-ASJ-SPF-Info: auth X-ASJ-Scan-ID: <1599435134.2114.15419@asusam.asj-hosting.net> X-ASJ-Received-SPF: pass (send with smtp authentication by nishida@asusa.net@73.190.120.36) Received: from c-73-190-120-36.hsd1.or.comcast.net (HELO rnci003.ddns.net) (nishida@asusa.net@73.190.120.36) by asusams.asj-hosting.net with ESMTPS (AES128-SHA encrypted); 7 Sep 2020 08:32:14 +0900 Subject: Re: IPv6 Problem with Bhyve To: "freebsd-virtualization@freebsd.org" References: <53835548-cd6e-6859-c19d-8dd96d3993f3@asusa.net> <0b2304fd-4799-f56e-2271-169a5348d654@asusa.net> <55041e52-8784-8336-3c41-1d203d6d2f98@asusa.net> <9bee075a-3ad6-76f6-1eb5-d9772c101564@asusa.net> From: Hiroshi Nishida Message-ID: <20dea3f3-9476-4a2d-64a9-6b28fa003ebe@asusa.net> Date: Sun, 6 Sep 2020 16:32:11 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4Bl7496fRhz4cgJ X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of nishida@asusa.net has no SPF policy when checking 219.118.222.245) smtp.mailfrom=nishida@asusa.net X-Spamd-Result: default: False [3.12 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.51)[0.507]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[asusa.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.93)[0.928]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_SPAM_LONG(0.79)[0.790]; TO_DN_EQ_ADDR_ALL(0.00)[]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:55388, ipnet:219.118.192.0/19, country:JP]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Sep 2020 23:32:27 -0000 Thanks for your reply. On 2020/09/06 15:32, Jason Tubnor wrote: > > By the way, how stable is Windows Server 2019 on bhyve? > > > Very stable.  We have not had any issues or unexpected crashes with > Windows Server 2016 over the last 3+ years, nor Windows Server 2019 > since we changed our SOE to that. > > That's good to hear. I'll push our parent company to use FreeBSD to virtualize Windows Server 2019. -- Hiroshi Nishida, PhD President ASUSA Corporation From owner-freebsd-virtualization@freebsd.org Wed Sep 9 14:31:27 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A91D63D223D for ; Wed, 9 Sep 2020 14:31:27 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [174.136.98.114]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.nomadlogic.org", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmkwZ172Pz4KZL for ; Wed, 9 Sep 2020 14:31:25 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from [192.168.1.159] (cpe-23-243-161-111.socal.res.rr.com [23.243.161.111]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id bbd1fd39 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Wed, 9 Sep 2020 14:31:24 +0000 (UTC) To: FreeBSD virtualization From: Pete Wright Subject: Nested Virtualization Under KVM Hypervisor Message-ID: Date: Wed, 9 Sep 2020 07:31:23 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4BmkwZ172Pz4KZL X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pete@nomadlogic.org designates 174.136.98.114 as permitted sender) smtp.mailfrom=pete@nomadlogic.org X-Spamd-Result: default: False [-1.90 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_SPAM_SHORT(0.05)[0.046]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.75)[-0.750]; DMARC_NA(0.00)[nomadlogic.org]; TO_DN_ALL(0.00)[]; NEURAL_HAM_MEDIUM(-0.89)[-0.894]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:25795, ipnet:174.136.96.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RECEIVED_SPAMHAUS_PBL(0.00)[23.243.161.111:received] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2020 14:31:27 -0000 Hey there, I have a host that is running via hardware virtualization under KVM and am having issues with virtualization.  My provider investigated and ensured that the hypervisor was configured to pass through VT-x to guests, and reports that Linux guests are detecting VT-x and are able to run nested virtualization on their systems.  For FreeBSD this is not working - dmesg is reporting this: CPU: Intel Xeon E312xx (Sandy Bridge) (2600.16-MHz K8-class CPU)   Origin="GenuineIntel"  Id=0x206a1  Family=0x6  Model=0x2a Stepping=1 Features=0x783fbff Features2=0x9fb82223   AMD Features=0x28100800   AMD Features2=0x1   XSAVE Features=0x1   VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr Hypervisor: Origin = "KVMKVMKVM" real memory  = 34359738368 (32768 MB) avail memory = 33365536768 (31819 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: They did verify that VT-x pass through *is* enabled as mentioned above (they shared with me the config file snippet on their end to verify), and they did extensive testing on their end and were not able to get FreeBSD to detect VT-x as being enabled. I pinged them asking for more info on which Linux kernel their hypervisors are running to see if I can help debug this on my end.  I have two questions: 1) is anyone else doing nested virtualization under KVM hypervisors.  i am confident that this is possible, but am asking for confirmation on this :) 2) Are there any other requirements that FreeBSD requires to run Bhyve nested under KVM? Thanks in advance! -pete -- Pete Wright pete@nomadlogic.org @nomadlogicLA From owner-freebsd-virtualization@freebsd.org Thu Sep 10 00:36:18 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 94F2C3E1C37 for ; Thu, 10 Sep 2020 00:36:18 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from vito-vars.onthenet.com.au (vito-vars.OntheNet.com.au [203.13.68.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bn0LV0Tgbz47Wm for ; Thu, 10 Sep 2020 00:36:17 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by vito-vars.onthenet.com.au (Postfix) with ESMTPS id A756B2083923 for ; Thu, 10 Sep 2020 10:36:07 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 9033420AE983 for ; Thu, 10 Sep 2020 10:36:07 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 8963B20F4D29 for ; Thu, 10 Sep 2020 10:36:07 +1000 (AEST) Received: from iredmail.onthenet.com.au ([127.0.0.1]) by iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id jqRq-8q7EzIB for ; Thu, 10 Sep 2020 10:36:07 +1000 (AEST) Received: from MacBook-Air-4.local (CPE-120-29-42-110.dsl.OntheNet.net [120.29.42.110]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 30D3720F4D27; Thu, 10 Sep 2020 10:36:06 +1000 (AEST) Subject: Re: Nested Virtualization Under KVM Hypervisor To: Pete Wright References: Cc: FreeBSD virtualization From: Peter Grehan Message-ID: <5eacd0a5-89bd-cc98-6c5a-d5490e33b100@freebsd.org> Date: Thu, 10 Sep 2020 10:36:06 +1000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=bM+N3M2Z c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=jXL/WSj5nX36MkVNSbrI6Q==:17 a=IkcTkHD0fZMA:10 a=reM5J-MqmosA:10 a=6I5d2MoRAAAA:8 a=y9LJmbqIxtuG6_hiBngA:9 a=QEXdDO2ut3YA:10 a=IjZwj45LgO3ly-622nXo:22 X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=LtAJNkVc c=1 sm=1 tr=0 a=eF5wA05EhuPcU0j7PNT2Vg==:117 a=jXL/WSj5nX36MkVNSbrI6Q==:17 a=IkcTkHD0fZMA:10 a=reM5J-MqmosA:10 a=6I5d2MoRAAAA:8 a=y9LJmbqIxtuG6_hiBngA:9 a=QEXdDO2ut3YA:10 a=IjZwj45LgO3ly-622nXo:22 X-Rspamd-Queue-Id: 4Bn0LV0Tgbz47Wm X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; ASN(0.00)[asn:9313, ipnet:203.13.68.0/23, country:AU]; local_wl_from(0.00)[freebsd.org] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Sep 2020 00:36:18 -0000 Hi Pete, > =C2=A0 VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,Post= Intr That's just a warning: bhyve will attempt to enable VT-x on module=20 load, and will only fail if BIOS has locked the disable. Do you know if the vmm module fails to load ? (there will be an error=20 in syslog "vmx_init: VMX operation disabled by BIOS") > Hypervisor: Origin =3D "KVMKVMKVM" > real memory=C2=A0 =3D 34359738368 (32768 MB) > avail memory =3D 33365536768 (31819 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: >=20 > They did verify that VT-x pass through *is* enabled as mentioned above=20 > (they shared with me the config file snippet on their end to verify),=20 > and they did extensive testing on their end and were not able to get=20 > FreeBSD to detect VT-x as being enabled. >=20 > I pinged them asking for more info on which Linux kernel their=20 > hypervisors are running to see if I can help debug this on my end.=C2=A0= I=20 > have two questions: >=20 > 1) is anyone else doing nested virtualization under KVM hypervisors.=C2= =A0 i=20 > am confident that this is possible, but am asking for confirmation on=20 > this :) Yes: A bug report=20 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203994 ... had the resolution that a kernel > 4.10 has to be used, and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D246168 ... requires posted interrupts to be disabled. > 2) Are there any other requirements that FreeBSD requires to run Bhyve=20 > nested under KVM? No - nested has been enabled by default with KVM for quite a while now. later, Peter. From owner-freebsd-virtualization@freebsd.org Thu Sep 10 03:34:09 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D9AB03E7AA5 for ; Thu, 10 Sep 2020 03:34:09 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [174.136.98.114]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.nomadlogic.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bn4Hg4mtcz4MJg; Thu, 10 Sep 2020 03:34:07 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from [192.168.1.160] (cpe-23-243-161-111.socal.res.rr.com [23.243.161.111]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id 2c242fb7 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 10 Sep 2020 03:34:00 +0000 (UTC) Subject: Re: Nested Virtualization Under KVM Hypervisor To: Peter Grehan Cc: FreeBSD virtualization References: <5eacd0a5-89bd-cc98-6c5a-d5490e33b100@freebsd.org> From: Pete Wright Message-ID: <5e62480f-14cb-5963-f753-8562aab381e3@nomadlogic.org> Date: Wed, 9 Sep 2020 20:33:55 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: <5eacd0a5-89bd-cc98-6c5a-d5490e33b100@freebsd.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4Bn4Hg4mtcz4MJg X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pete@nomadlogic.org designates 174.136.98.114 as permitted sender) smtp.mailfrom=pete@nomadlogic.org X-Spamd-Result: default: False [-2.53 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[nomadlogic.org]; NEURAL_HAM_LONG(-0.95)[-0.949]; RECEIVED_SPAMHAUS_PBL(0.00)[23.243.161.111:received]; NEURAL_HAM_MEDIUM(-1.03)[-1.030]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.25)[-0.249]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:25795, ipnet:174.136.96.0/20, country:US]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Sep 2020 03:34:09 -0000 On 9/9/20 5:36 PM, Peter Grehan wrote: > Hi Pete, > >>    VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr >  That's just a warning: bhyve will attempt to enable VT-x on module > load, and will only fail if BIOS has locked the disable. > >  Do you know if the vmm module fails to load ? (there will be an error > in syslog "vmx_init: VMX operation disabled by BIOS") thanks for getting back to me Peter.  The vmm.ko does load: $ kldstat|grep vmm 19    1 0xffffffff82ce1000   537420 vmm.ko $ Sep 10 03:25:01 bells kernel: vmx_init: processor does not support desired basic capabilities Sep 10 03:25:01 bells kernel: module_register_init: MOD_LOAD (vmm, 0xffffffff82ce4d20, 0) error 22 Sep 10 03:25:01 bells kernel: driver bug: Unable to set devclass (class: fdc devname: (unknown)) I also received this input from the ISP, they are running this kernel:  4.15.0-47-generic #50~16.04.1-Ubuntu SMP > >  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246168 >   ... requires posted interrupts to be disabled. > reading through that ticket i set this in loader.conf: hw.vmm.vmx.use_apic_vid=0 and verified via sysctl: $ sysctl hw.vmm.vmx.cap.virtual_interrupt_delivery hw.vmm.vmx.cap.virtual_interrupt_delivery: 0 $ since you mention this is a possible bug with KVM is there some useful information i can relay to my ISP (they are a BSD friendly vendor) that they can try on their end as a workaround, or perhaps something they ping the linux/kvm community about? thanks in advance! -pete -- Pete Wright pete@nomadlogic.org @nomadlogicLA From owner-freebsd-virtualization@freebsd.org Thu Sep 10 07:30:43 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AD9653EDBAB for ; Thu, 10 Sep 2020 07:30:43 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from vito-vars.onthenet.com.au (vito-vars.OntheNet.com.au [203.13.68.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bn9Xf6rtQz4ZpX for ; Thu, 10 Sep 2020 07:30:42 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by vito-vars.onthenet.com.au (Postfix) with ESMTPS id CF1062083928 for ; Thu, 10 Sep 2020 17:30:37 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id B77D220AE980 for ; Thu, 10 Sep 2020 17:30:37 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id ACC05209C021 for ; Thu, 10 Sep 2020 17:30:37 +1000 (AEST) Received: from iredmail.onthenet.com.au ([127.0.0.1]) by iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 5DItxeJT2tyj for ; Thu, 10 Sep 2020 17:30:37 +1000 (AEST) Received: from MacBook-Air-4.local (CPE-120-29-42-110.dsl.OntheNet.net [120.29.42.110]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 0A41B209C01A; Thu, 10 Sep 2020 17:30:37 +1000 (AEST) Subject: Re: Nested Virtualization Under KVM Hypervisor To: Pete Wright Cc: FreeBSD virtualization References: <5eacd0a5-89bd-cc98-6c5a-d5490e33b100@freebsd.org> <5e62480f-14cb-5963-f753-8562aab381e3@nomadlogic.org> From: Peter Grehan Message-ID: Date: Thu, 10 Sep 2020 17:30:36 +1000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: <5e62480f-14cb-5963-f753-8562aab381e3@nomadlogic.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=bM+N3M2Z c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=jXL/WSj5nX36MkVNSbrI6Q==:17 a=IkcTkHD0fZMA:10 a=reM5J-MqmosA:10 a=6I5d2MoRAAAA:8 a=DYosF9_QlMOk_I5_GGcA:9 a=QEXdDO2ut3YA:10 a=IjZwj45LgO3ly-622nXo:22 X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=LtAJNkVc c=1 sm=1 tr=0 a=eF5wA05EhuPcU0j7PNT2Vg==:117 a=jXL/WSj5nX36MkVNSbrI6Q==:17 a=IkcTkHD0fZMA:10 a=reM5J-MqmosA:10 a=6I5d2MoRAAAA:8 a=DYosF9_QlMOk_I5_GGcA:9 a=QEXdDO2ut3YA:10 a=IjZwj45LgO3ly-622nXo:22 X-Rspamd-Queue-Id: 4Bn9Xf6rtQz4ZpX X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; ASN(0.00)[asn:9313, ipnet:203.13.68.0/23, country:AU]; local_wl_from(0.00)[freebsd.org] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Sep 2020 07:30:43 -0000 Hi Pete, > Sep 10 03:25:01 bells kernel: vmx_init: processor does not support=20 > desired basic capabilities ^^^ This is what's causing the issues, and is covered in the first=20 bug report I mentioned=20 (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203994) Would you be able to dump the MSRs for the FreeBSD guest as mentioned=20 in the bug report ? > I also received this input from the ISP, they are running this kernel: > =C2=A04.15.0-47-generic #50~16.04.1-Ubuntu SMP I'm now wondering if this is too old a kernel :( > reading through that ticket i set this in loader.conf: > hw.vmm.vmx.use_apic_vid=3D0 >=20 > and verified via sysctl: > $ sysctl hw.vmm.vmx.cap.virtual_interrupt_delivery > hw.vmm.vmx.cap.virtual_interrupt_delivery: 0 > $ >=20 > since you mention this is a possible bug with KVM is there some useful=20 > information i can relay to my ISP (they are a BSD friendly vendor) that= =20 > they can try on their end as a workaround, or perhaps something they=20 > ping the linux/kvm community about? As above, this isn't the issue for this particular case: it manifests=20 as a hypervisor guest immediately exiting when started. later, Peter. From owner-freebsd-virtualization@freebsd.org Thu Sep 10 14:46:34 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6D3AE3D960D for ; Thu, 10 Sep 2020 14:46:34 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [174.136.98.114]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.nomadlogic.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BnMCY2YqJz460d; Thu, 10 Sep 2020 14:46:32 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from [192.168.1.159] (cpe-23-243-161-111.socal.res.rr.com [23.243.161.111]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id 2476281e (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 10 Sep 2020 14:46:31 +0000 (UTC) Subject: Re: Nested Virtualization Under KVM Hypervisor To: Peter Grehan Cc: FreeBSD virtualization References: <5eacd0a5-89bd-cc98-6c5a-d5490e33b100@freebsd.org> <5e62480f-14cb-5963-f753-8562aab381e3@nomadlogic.org> From: Pete Wright Message-ID: Date: Thu, 10 Sep 2020 07:46:31 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4BnMCY2YqJz460d X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pete@nomadlogic.org designates 174.136.98.114 as permitted sender) smtp.mailfrom=pete@nomadlogic.org X-Spamd-Result: default: False [-2.55 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[nomadlogic.org]; ARC_NA(0.00)[]; NEURAL_HAM_LONG(-1.02)[-1.018]; NEURAL_HAM_MEDIUM(-0.98)[-0.984]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.25)[-0.252]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:25795, ipnet:174.136.96.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RECEIVED_SPAMHAUS_PBL(0.00)[23.243.161.111:received] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Sep 2020 14:46:34 -0000 u On 9/10/20 12:30 AM, Peter Grehan wrote: > Hi Pete, > >> Sep 10 03:25:01 bells kernel: vmx_init: processor does not support >> desired basic capabilities > >  ^^^  This is what's causing the issues, and is covered in the first > bug report I mentioned > (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203994) > >  Would you be able to dump the MSRs for the FreeBSD guest as mentioned > in the bug report ? > Ah I see - once I realized we had a newer kernel I didn't focus too much on the first bug - here's the MSR's MSR 0x480: 0x00981000 0x11e57ed0 MSR 0x482: 0xfff9fffe 0x0401e172 MSR 0x48b: 0x000003fb 0x00000000 MSR 0x48c: 0x00000f01 0x04114040 >> since you mention this is a possible bug with KVM is there some >> useful information i can relay to my ISP (they are a BSD friendly >> vendor) that they can try on their end as a workaround, or perhaps >> something they ping the linux/kvm community about? > >  As above, this isn't the issue for this particular case: it manifests > as a hypervisor guest immediately exiting when started. > ok perfect this aligns with what i'm seeing as well when i run bhyvload or bhyvectl and it just bails with a "vm_create: Device not configured" error. Cheers, -p -- Pete Wright pete@nomadlogic.org @nomadlogicLA From owner-freebsd-virtualization@freebsd.org Fri Sep 11 13:38:40 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 33DA93DAD0E for ; Fri, 11 Sep 2020 13:38:40 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from forward104o.mail.yandex.net (forward104o.mail.yandex.net [37.140.190.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bnxfk5jM3z4M8h for ; Fri, 11 Sep 2020 13:38:38 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from mxback1o.mail.yandex.net (mxback1o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::1b]) by forward104o.mail.yandex.net (Yandex) with ESMTP id 522A4942247; Fri, 11 Sep 2020 16:38:35 +0300 (MSK) Received: from myt4-ee976ce519ac.qloud-c.yandex.net (myt4-ee976ce519ac.qloud-c.yandex.net [2a02:6b8:c00:1da4:0:640:ee97:6ce5]) by mxback1o.mail.yandex.net (mxback/Yandex) with ESMTP id duC119YemT-cYGSMstX; Fri, 11 Sep 2020 16:38:35 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1599831515; bh=3kPBM3eqaTWPGJOPsbt+U3uws3835VUSnPnJ9fmE3tU=; h=In-Reply-To:From:Date:References:To:Subject:Message-ID; b=t44PJd5ylSPSqoK6jvKUYh/4JL+RZeE0VdFXCZH3Xw2jKT9glIdRB1QVbA/ZAsb8f mBoFC8XgHk3RdXG6Ot81oA/fiFvJ6Sl6ejGVXR+fzhA6hVvh1qzDZbLKSc2BreiHFx rDmc+dh+h5Lrfj+clgPxEqlRRUxTN4p9WDIoHIU0= Received: by myt4-ee976ce519ac.qloud-c.yandex.net (smtp/Yandex) with ESMTPSA id WuZuQ5VFUZ-cYmmvxvN; Fri, 11 Sep 2020 16:38:34 +0300 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client certificate not present) Subject: Re: IPv6 Problem with Bhyve To: Hiroshi Nishida , freebsd-virtualization@freebsd.org References: <53835548-cd6e-6859-c19d-8dd96d3993f3@asusa.net> <0b2304fd-4799-f56e-2271-169a5348d654@asusa.net> From: "Andrey V. Elsukov" Autocrypt: addr=bu7cher@yandex.ru; prefer-encrypt=mutual; keydata= mQENBEwBF1kBCADB9sXFhBEUy8qQ4X63Y8eBatYMHGEFWN9ypS5lI3RE6qQW2EYbxNk7qUC5 21YIIS1mMFVBEfvR7J9uc7yaYgFCEb6Sce1RSO4ULN2mRKGHP3/Sl0ijZEjWHV91hY1YTHEF ZW/0GYinDf56sYpDDehaBF5wkWIo1+QK5nmj3vl0DIDCMNd7QEiWpyLVwECgLX2eOAXByT8B bCqVhJGcG6iFP7/B9Ll6uX5gb8thM9LM+ibwErDBVDGiOgvfxqidab7fdkh893IBCXa82H9N CNwnEtcgzh+BSKK5BgvPohFMgRwjti37TSxwLu63QejRGbZWSz3OK3jMOoF63tCgn7FvABEB AAG0JUFuZHJleSBWLiBFbHN1a292IDxidTdjaGVyQHlhbmRleC5ydT6JATgEEwECACIFAkwB F1kCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEAHF6gQQyKF6qmYIAI6ekfm1VA4T vqankI1ISE6ku4jV7UlpIQlEbE7/8n3Zd6teJ+pGOQhN5qk8QE7utdPdbktAzi+x7LIJVzUw 4TywZLXGrkP7VKYkfg6oyCGyzITghefQeJtr2TN4hYCkzPWpylkue8MtmqfZv/6royqwTbN+ +E09FQNvTgRUYJYTeQ1qOsxNRycwvw3dr2rOfuxShbzaHBB1pBIjGrMg8fC5pd65ACH5zuFV A0CoTNGMDrEZSfBkTW604UUHFFXeCoC3dwDZRKOWJ3GmMXns65Ai5YkA63BSHEE1Qle3VBhd cG1w0CB5FBV3pB27UVnf0jEbysrDqW4qN7XMRFSWNAy5AQ0ETAEXWQEIAJ2p6l9LBoqdH/0J PEFDY2t2gTvAuzz+8zs3R03dFuHcNbOwjvWCG0aOmVpAzkRa8egn5JB4sZaFUtKPYJEQ1Iu+ LUBwgvtXf4vWpzC67zs2dDuiW4LamH5p6xkTD61aHR7mCB3bg2TUjrDWn2Jt44cvoYxj3dz4 S49U1rc9ZPgD5axCNv45j72tggWlZvpefThP7xT1OlNTUqye2gAwQravXpZkl5JG4eOqJVIU X316iE3qso0iXRUtO7OseBf0PiVmk+wCahdreHOeOxK5jMhYkPKVn7z1sZiB7W2H2TojbmcK HZC22sz7Z/H36Lhg1+/RCnGzdEcjGc8oFHXHCxUAEQEAAYkBHwQYAQIACQUCTAEXWQIbDAAK CRABxeoEEMihegkYCAC3ivGYNe2taNm/4Nx5GPdzuaAJGKWksV+w9mo7dQvU+NmI2az5w8vw 98OmX7G0OV9snxMW+6cyNqBrVFTu33VVNzz9pnqNCHxGvj5dL5ltP160JV2zw2bUwJBYsgYQ WfyJJIM7l3gv5ZS3DGqaGIm9gOK1ANxfrR5PgPzvI9VxDhlr2juEVMZYAqPLEJe+SSxbwLoz BcFCNdDAyXcaAzXsx/E02YWm1hIWNRxanAe7Vlg7OL+gvLpdtrYCMg28PNqKNyrQ87LQ49O9 50IIZDOtNFeR0FGucjcLPdS9PiEqCoH7/waJxWp6ydJ+g4OYRBYNM0EmMgy1N85JJrV1mi5i Message-ID: Date: Fri, 11 Sep 2020 16:36:24 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <0b2304fd-4799-f56e-2271-169a5348d654@asusa.net> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="rji4YAvnWXZKXpFfkLnChep3tQCbWm7ZT" X-Rspamd-Queue-Id: 4Bnxfk5jM3z4M8h X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yandex.ru header.s=mail header.b=t44PJd5y; dmarc=pass (policy=none) header.from=yandex.ru; spf=pass (mx1.freebsd.org: domain of bu7cher@yandex.ru designates 37.140.190.179 as permitted sender) smtp.mailfrom=bu7cher@yandex.ru X-Spamd-Result: default: False [-5.79 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yandex.ru]; R_SPF_ALLOW(-0.20)[+ip4:37.140.128.0/18]; HAS_ATTACHMENT(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[yandex.ru:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yandex.ru,none]; NEURAL_HAM_SHORT(-0.77)[-0.769]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; FREEMAIL_ENVFROM(0.00)[yandex.ru]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:13238, ipnet:37.140.128.0/18, country:RU]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.94)[-0.939]; R_DKIM_ALLOW(-0.20)[yandex.ru:s=mail]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yandex.ru:dkim]; NEURAL_HAM_LONG(-0.99)[-0.986]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[37.140.190.179:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[37.140.190.179:from]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Sep 2020 13:38:40 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --rji4YAvnWXZKXpFfkLnChep3tQCbWm7ZT Content-Type: multipart/mixed; boundary="CucweL4zjhUVEert28QUBLCbnbygkITGt" --CucweL4zjhUVEert28QUBLCbnbygkITGt Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 01.09.2020 17:20, Hiroshi Nishida wrote: > Interestingly, I have exactly the same problem also with CentOS 8 + > KVM/QEMU. >=20 > I would appreciate it if anybody could tell me how to fix this. > Thank you. Hi, I think you just need to allow IPv6 on you bridge0 interface, try this: ifconfig bridge0 inet6 -ifdisabled --=20 WBR, Andrey V. Elsukov --CucweL4zjhUVEert28QUBLCbnbygkITGt-- --rji4YAvnWXZKXpFfkLnChep3tQCbWm7ZT Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQEzBAEBCAAdFiEE5lkeG0HaFRbwybwAAcXqBBDIoXoFAl9bfV0ACgkQAcXqBBDI oXpnRAf/dBIyPqOBatJwdv5Eit7Z8V8thPKerT93u0fVl0nJ83znirBR2iKGFOwF 1MA+gSbm9gsRgOQOe7Dp36F0Ja+rGLWiIH1PV2GPh1PNbq7hTSQTM+xmW6d70SnV 8hdFZicsT0/AcgxqqHRPElasjK+IKlw5rR2WmJbLzNWIDeKnaZIvevyU/JCwtudG wVQJae+6i54xfsUOgne5i1PzMPoBPM1R1ZBfSXNgdTiKxv3NORmsBjU7LgOCJUZm yIV8AV1RG1iwokObZbExLi6CIi8Rdyr3AbE3RosyIVNZ4sWZMv79OBTAeL7W7weP ROIPoK1Fb2T7rt5WDmkLk4Nouvtaeg== =t98q -----END PGP SIGNATURE----- --rji4YAvnWXZKXpFfkLnChep3tQCbWm7ZT-- From owner-freebsd-virtualization@freebsd.org Fri Sep 11 14:23:25 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9DD7A3DC3D8 for ; Fri, 11 Sep 2020 14:23:25 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (cross.sbone.de [195.201.62.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BnyfN3kckz4QSD for ; Fri, 11 Sep 2020 14:23:24 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 178948D4A21E; Fri, 11 Sep 2020 14:23:16 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 660D9E707D9; Fri, 11 Sep 2020 14:23:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id t7OJKKmW7upS; Fri, 11 Sep 2020 14:23:15 +0000 (UTC) Received: from [127.0.0.1] (unknown [IPv6:fde9:577b:c1a9:4902:2585:28cd:7660:77ea]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 1038BE707B5; Fri, 11 Sep 2020 14:23:13 +0000 (UTC) From: "Bjoern A. Zeeb" To: "Andrey V. Elsukov" Cc: "Hiroshi Nishida" , freebsd-virtualization@freebsd.org Subject: Re: IPv6 Problem with Bhyve Date: Fri, 11 Sep 2020 14:23:13 +0000 X-Mailer: MailMate (2.0BETAr6151) Message-ID: <32D43CAB-C5D4-424D-BE4B-536D668124D3@lists.zabbadoz.net> In-Reply-To: References: <53835548-cd6e-6859-c19d-8dd96d3993f3@asusa.net> <0b2304fd-4799-f56e-2271-169a5348d654@asusa.net> MIME-Version: 1.0 Content-Type: text/plain X-Rspamd-Queue-Id: 4BnyfN3kckz4QSD X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of bzeeb-lists@lists.zabbadoz.net designates 195.201.62.131 as permitted sender) smtp.mailfrom=bzeeb-lists@lists.zabbadoz.net X-Spamd-Result: default: False [-2.50 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:195.201.62.131]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[zabbadoz.net]; NEURAL_HAM_LONG(-1.00)[-0.999]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.21)[-0.207]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; FREEMAIL_TO(0.00)[yandex.ru]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:195.201.0.0/16, country:DE]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Sep 2020 14:23:25 -0000 On 11 Sep 2020, at 13:36, Andrey V. Elsukov wrote: > On 01.09.2020 17:20, Hiroshi Nishida wrote: >> Interestingly, I have exactly the same problem also with CentOS 8 + >> KVM/QEMU. >> >> I would appreciate it if anybody could tell me how to fix this. >> Thank you. > > Hi, > > I think you just need to allow IPv6 on you bridge0 interface, try this: > > ifconfig bridge0 inet6 -ifdisabled Do you? I think mine is running with IFDISABLED. I believe we still bridge the packets, bridge itself however will not get IPv6 config (which is rarely needed). Or do we not? I may be running on local patches as well.. /bz From owner-freebsd-virtualization@freebsd.org Fri Sep 11 14:49:07 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 800B43DD548 for ; Fri, 11 Sep 2020 14:49:07 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from forward102j.mail.yandex.net (forward102j.mail.yandex.net [IPv6:2a02:6b8:0:801:2::102]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BnzD22tPXz4SN5 for ; Fri, 11 Sep 2020 14:49:06 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from mxback13g.mail.yandex.net (mxback13g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:92]) by forward102j.mail.yandex.net (Yandex) with ESMTP id 06AFBF22457; Fri, 11 Sep 2020 17:49:03 +0300 (MSK) Received: from myt6-efff10c3476a.qloud-c.yandex.net (myt6-efff10c3476a.qloud-c.yandex.net [2a02:6b8:c12:13a3:0:640:efff:10c3]) by mxback13g.mail.yandex.net (mxback/Yandex) with ESMTP id 2BW3qnNXgl-n2Ue4DCJ; Fri, 11 Sep 2020 17:49:02 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1599835742; bh=JyofiHh+994KcyMhO/vHo04FKGXIgyW0a4XEA3lXbjw=; h=In-Reply-To:From:To:Subject:Cc:Date:References:Message-ID; b=WqlGVROavNZUphxFZKdGrCx0UooHYf22P9syEWGzi+IFDvIjfIng2j7WOBlKIEFED z5tsIs6U7JR5lU3E7t1xgubIRD2wMCv8Xd79ln/UgF5EITi2VjqYmN2cQEHc1T+DMQ VQdIrg69NuQNYy9tCPFdk7eGYAw6txK8cNptnaE0= Received: by myt6-efff10c3476a.qloud-c.yandex.net (smtp/Yandex) with ESMTPSA id RsQ4WzOiOF-n2m85lLZ; Fri, 11 Sep 2020 17:49:02 +0300 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client certificate not present) Subject: Re: IPv6 Problem with Bhyve To: "Bjoern A. Zeeb" Cc: Hiroshi Nishida , freebsd-virtualization@freebsd.org References: <53835548-cd6e-6859-c19d-8dd96d3993f3@asusa.net> <0b2304fd-4799-f56e-2271-169a5348d654@asusa.net> <32D43CAB-C5D4-424D-BE4B-536D668124D3@lists.zabbadoz.net> From: "Andrey V. Elsukov" Autocrypt: addr=bu7cher@yandex.ru; prefer-encrypt=mutual; keydata= mQENBEwBF1kBCADB9sXFhBEUy8qQ4X63Y8eBatYMHGEFWN9ypS5lI3RE6qQW2EYbxNk7qUC5 21YIIS1mMFVBEfvR7J9uc7yaYgFCEb6Sce1RSO4ULN2mRKGHP3/Sl0ijZEjWHV91hY1YTHEF ZW/0GYinDf56sYpDDehaBF5wkWIo1+QK5nmj3vl0DIDCMNd7QEiWpyLVwECgLX2eOAXByT8B bCqVhJGcG6iFP7/B9Ll6uX5gb8thM9LM+ibwErDBVDGiOgvfxqidab7fdkh893IBCXa82H9N CNwnEtcgzh+BSKK5BgvPohFMgRwjti37TSxwLu63QejRGbZWSz3OK3jMOoF63tCgn7FvABEB AAG0JUFuZHJleSBWLiBFbHN1a292IDxidTdjaGVyQHlhbmRleC5ydT6JATgEEwECACIFAkwB F1kCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEAHF6gQQyKF6qmYIAI6ekfm1VA4T vqankI1ISE6ku4jV7UlpIQlEbE7/8n3Zd6teJ+pGOQhN5qk8QE7utdPdbktAzi+x7LIJVzUw 4TywZLXGrkP7VKYkfg6oyCGyzITghefQeJtr2TN4hYCkzPWpylkue8MtmqfZv/6royqwTbN+ +E09FQNvTgRUYJYTeQ1qOsxNRycwvw3dr2rOfuxShbzaHBB1pBIjGrMg8fC5pd65ACH5zuFV A0CoTNGMDrEZSfBkTW604UUHFFXeCoC3dwDZRKOWJ3GmMXns65Ai5YkA63BSHEE1Qle3VBhd cG1w0CB5FBV3pB27UVnf0jEbysrDqW4qN7XMRFSWNAy5AQ0ETAEXWQEIAJ2p6l9LBoqdH/0J PEFDY2t2gTvAuzz+8zs3R03dFuHcNbOwjvWCG0aOmVpAzkRa8egn5JB4sZaFUtKPYJEQ1Iu+ LUBwgvtXf4vWpzC67zs2dDuiW4LamH5p6xkTD61aHR7mCB3bg2TUjrDWn2Jt44cvoYxj3dz4 S49U1rc9ZPgD5axCNv45j72tggWlZvpefThP7xT1OlNTUqye2gAwQravXpZkl5JG4eOqJVIU X316iE3qso0iXRUtO7OseBf0PiVmk+wCahdreHOeOxK5jMhYkPKVn7z1sZiB7W2H2TojbmcK HZC22sz7Z/H36Lhg1+/RCnGzdEcjGc8oFHXHCxUAEQEAAYkBHwQYAQIACQUCTAEXWQIbDAAK CRABxeoEEMihegkYCAC3ivGYNe2taNm/4Nx5GPdzuaAJGKWksV+w9mo7dQvU+NmI2az5w8vw 98OmX7G0OV9snxMW+6cyNqBrVFTu33VVNzz9pnqNCHxGvj5dL5ltP160JV2zw2bUwJBYsgYQ WfyJJIM7l3gv5ZS3DGqaGIm9gOK1ANxfrR5PgPzvI9VxDhlr2juEVMZYAqPLEJe+SSxbwLoz BcFCNdDAyXcaAzXsx/E02YWm1hIWNRxanAe7Vlg7OL+gvLpdtrYCMg28PNqKNyrQ87LQ49O9 50IIZDOtNFeR0FGucjcLPdS9PiEqCoH7/waJxWp6ydJ+g4OYRBYNM0EmMgy1N85JJrV1mi5i Message-ID: <094d603a-8520-e958-6b58-9f31074426f7@yandex.ru> Date: Fri, 11 Sep 2020 17:46:52 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <32D43CAB-C5D4-424D-BE4B-536D668124D3@lists.zabbadoz.net> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="cmrbgNLP2P37AXeR3k0tAiqLmhvcFbq9d" X-Rspamd-Queue-Id: 4BnzD22tPXz4SN5 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yandex.ru header.s=mail header.b=WqlGVROa; dmarc=pass (policy=none) header.from=yandex.ru; spf=pass (mx1.freebsd.org: domain of bu7cher@yandex.ru designates 2a02:6b8:0:801:2::102 as permitted sender) smtp.mailfrom=bu7cher@yandex.ru X-Spamd-Result: default: False [-6.06 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yandex.ru]; R_SPF_ALLOW(-0.20)[+ip6:2a02:6b8:0::/52]; HAS_ATTACHMENT(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[yandex.ru:+]; DMARC_POLICY_ALLOW(-0.50)[yandex.ru,none]; NEURAL_HAM_SHORT(-1.04)[-1.036]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; FREEMAIL_ENVFROM(0.00)[yandex.ru]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:13238, ipnet:2a02:6b8::/32, country:RU]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.94)[-0.941]; R_DKIM_ALLOW(-0.20)[yandex.ru:s=mail]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yandex.ru:dkim]; RCPT_COUNT_THREE(0.00)[3]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; NEURAL_HAM_LONG(-0.98)[-0.981]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Sep 2020 14:49:07 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --cmrbgNLP2P37AXeR3k0tAiqLmhvcFbq9d Content-Type: multipart/mixed; boundary="RWDaBOZzPr0Y7akwOo9R44T92TYMGjNa0" --RWDaBOZzPr0Y7akwOo9R44T92TYMGjNa0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 11.09.2020 17:23, Bjoern A. Zeeb wrote: > On 11 Sep 2020, at 13:36, Andrey V. Elsukov wrote: >=20 >> On 01.09.2020 17:20, Hiroshi Nishida wrote: >>> Interestingly, I have exactly the same problem also with CentOS 8 + >>> KVM/QEMU. >>> >>> I would appreciate it if anybody could tell me how to fix this. >>> Thank you. >> >> Hi, >> >> I think you just need to allow IPv6 on you bridge0 interface, try this= : >> >> ifconfig bridge0 inet6 -ifdisabled >=20 > Do you? I think mine is running with IFDISABLED. >=20 > I believe we still bridge the packets, bridge itself however will no= t > get IPv6 config (which is rarely needed). Or do we not? I may be runn= ing > on local patches as well.. I'm not 100% sure, but I have some memories that after some time of debugging of some problem with IPv6 I was realized that IFDISABLED was turned on and it silently drops the inbound IPv6 traffic at IP layer. So, you can see it in the tcpdump, but it doesn't reach applications. --=20 WBR, Andrey V. Elsukov --RWDaBOZzPr0Y7akwOo9R44T92TYMGjNa0-- --cmrbgNLP2P37AXeR3k0tAiqLmhvcFbq9d Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQEzBAEBCAAdFiEE5lkeG0HaFRbwybwAAcXqBBDIoXoFAl9bjdwACgkQAcXqBBDI oXpvYwf/QhnPuNBpangnhKNCGmNETJoW2LjqoYKKf8MgFLxowB2oMWZGWCtpd9ij sY5IuZwWjH1Kf+gqbYoHmfDUwQcAH3iafpQBTn9ERhO7FTrwkA3cL5Twt7zff2rR oiRbG2n9D2nnZFYyrNizCp9oooZJeaXhi0OwQrxIGlnf3mTeJQhTn+YXuDExmHxo ko3/nsDlyLQOWioFhiBCQC88q3Ejfgy273GAhX1HygiLdJD84WW8yVv99/CoLzGA 7zLxNgahjZc3CMdsti6HMZ3F99v99jJUvnqBj9/P5uw+9u+6vU1WQ5isq9YDkBVA 4VFBN59tQPg4tJYdepllj2fq5K6gfw== =m6Kg -----END PGP SIGNATURE----- --cmrbgNLP2P37AXeR3k0tAiqLmhvcFbq9d-- From owner-freebsd-virtualization@freebsd.org Fri Sep 11 14:51:09 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 546CA3DD4D7 for ; Fri, 11 Sep 2020 14:51:09 +0000 (UTC) (envelope-from nishida@asusa.net) Received: from asusam.asj-hosting.net (asusa.asj-hosting.net [219.118.222.245]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.asj-hosting.net", Issuer "JPRS Domain Validation Authority - G3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BnzGM6238z4SJG for ; Fri, 11 Sep 2020 14:51:06 +0000 (UTC) (envelope-from nishida@asusa.net) Received: (qmail 14429 invoked by uid 89); 11 Sep 2020 23:50:56 +0900 X-ASJ-Track-ID: <20200911145056.14429.qmail@asusam.asj-hosting.net> X-Spam-Checker-Version: ASJ KMsrv Spam Check Process Internal X-Spam-Status: No, hits=0.0 X-Spam-Flag: No X-Virus-Scanned: ASJ KMsrv Virus Check Process 08041001 X-ASJ-SMTP-Authentication: nishida@asusa.net X-ASJ-Arrival-IP: 73.190.120.36 X-ASJ-SPF-Info: auth X-ASJ-Scan-ID: <1599835855.531228.14423@asusam.asj-hosting.net> X-ASJ-Received-SPF: pass (send with smtp authentication by nishida@asusa.net@73.190.120.36) Received: from c-73-190-120-36.hsd1.or.comcast.net (HELO rnci003.ddns.net) (nishida@asusa.net@73.190.120.36) by asusams.asj-hosting.net with ESMTPS (AES128-SHA encrypted); 11 Sep 2020 23:50:55 +0900 Subject: Re: IPv6 Problem with Bhyve To: freebsd-virtualization@freebsd.org References: <53835548-cd6e-6859-c19d-8dd96d3993f3@asusa.net> <0b2304fd-4799-f56e-2271-169a5348d654@asusa.net> From: Hiroshi Nishida Message-ID: <397aeb99-9a2a-c709-c705-806a5cd8fc44@asusa.net> Date: Fri, 11 Sep 2020 07:50:53 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4BnzGM6238z4SJG X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of nishida@asusa.net has no SPF policy when checking 219.118.222.245) smtp.mailfrom=nishida@asusa.net X-Spamd-Result: default: False [-0.67 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.990]; NEURAL_HAM_LONG(-0.57)[-0.572]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[asusa.net]; AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.01)[-0.009]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:55388, ipnet:219.118.192.0/19, country:JP]; MAILMAN_DEST(0.00)[freebsd-virtualization]; FREEMAIL_CC(0.00)[yandex.ru] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Sep 2020 14:51:09 -0000 Hi, It's about CentOS and will not be suitable for this ML but ICMPv6 works with the current settings. Therefore I don't think the problem is that easy. Anyway, bhyve + Windows is running very stable and I don't see any reasons to switch to Linux right now. Also, as a new bhyve user, I'm thinking to post how to install Windows Server 2019 on bhyve to a forum as there's little information on it. On 2020/09/11 6:36, Andrey V. Elsukov wrote: > On 01.09.2020 17:20, Hiroshi Nishida wrote: >> Interestingly, I have exactly the same problem also with CentOS 8 + >> KVM/QEMU. >> >> I would appreciate it if anybody could tell me how to fix this. >> Thank you. > Hi, > > I think you just need to allow IPv6 on you bridge0 interface, try this: > > ifconfig bridge0 inet6 -ifdisabled > -- Hiroshi Nishida, PhD President ASUSA Corporation