From nobody Thu Jul 14 20:05:17 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4LkQT96NqQz4T34y for ; Thu, 14 Jul 2022 20:05:17 +0000 (UTC) (envelope-from jwd@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LkQT95t7cz3RkG for ; Thu, 14 Jul 2022 20:05:17 +0000 (UTC) (envelope-from jwd@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1657829117; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=q9CjjJWceBsQMRv9UldDRrzflSCUFffdZ4N3Od2Aw8c=; b=gPE6fWxzPXzB4YokWLsDkwpHT3Jhl9fu+AjzO14Gzo1ZHuwyEFx2AHB9jha9epGdUunknF tpsUHbED2mTi3MY6SwGQTQPwdnNqjcKdF8I2OYxXBmuNZ3B112c+364SphrKo8D/RnqYI+ bFWEoL2uNuj+Ev4ldOTOOq5dVTkJKtAUe2WjATJsIIU+8ImgYYdh4nCdLQkmuXXrxcqkDU 15S5NVUnipHEGOQ9oQwf0tlf5FhkG6/x0RgrGQ/otwppvD6a6N3AXYy+Y7IekNHzgKsW3h cmrLPw81U+hsc+JrrcUQdxXtRTNfoiWIAPQZlh5Vr3+YYYWDnfo46vAhMcZNqw== Received: by freefall.freebsd.org (Postfix, from userid 821) id B976A1732; Thu, 14 Jul 2022 20:05:17 +0000 (UTC) Date: Thu, 14 Jul 2022 20:05:17 +0000 From: John To: FreeBSD-Hackers Subject: Dynamic VF mac address manipulation / Bhyve Message-ID: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1657829117; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=q9CjjJWceBsQMRv9UldDRrzflSCUFffdZ4N3Od2Aw8c=; b=ObhItjlF1q06CrL36gjyeHR5PaBgEwa6uIFypSaD6fTQVAuCP0nxm8cVdtha8p9TNJQwU6 TKIRd/1+hYd3fbTenCzUMC9SE83DXguSdimzA3D4GJvy/YkbRbRI/BEyPFGvOc3Uq7kAbm J9DYQ5r3SAJ/k9Up7RvOqsSfLFmeaNMWDFuIGUAHxDXV7j4HF1qV09Vbz9hJEJY1CMZ7V0 i8E1Yhf+A9Aa0267nBb3Mmo5VqTknzlzVcP6bTCXXwJPenw9jT7t5Tu5aP7R9KJ0HEcdB9 WAYeZ0z5ki6cHrt6T9GsFBCAl5Arj1ohHZwrDpd2HasjA9tVr5ddLI2QWrMIwA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1657829117; a=rsa-sha256; cv=none; b=v9Xo5+tHbBwMnHMdW9I4aV4sJkVWweGaNTMaS2MCMdPKQ49sV8x68rkHM51q4d7HPcucik GAKrGVrVYWQD84XIjfOU8u+t5IGsNq9uIMMhIFf+jOhBssbd70h/SkzVi5QIvjCgUE0G3E Yx3CXDRt2FnLGy+jyEjMcnU70duQ1u5V87pAi6N4B+HztwL1y0xSJuL5YM0Zu+rJUD21tR X/r9WyjpSVeNklGS1MtO2ROxvQdMpfoWUq4EjT/wfUNE1wkrIcnbamrQvw62APIUiFo6QX OFFkhZujtYrW517H3qbRNwLkuhmPK/XZ21/EF/VEwzTrbvBsxAmhWmoGlmZZEA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N Hi - I am slowly converting a Linux based qemu/kvm solution to FreeBSD Bhyve where VMs have mac addresses assigned to them. Linux allows individual VF entries per $iface to be manipulated: ip link set ${iface} vf $vface mac $vfmac The VF is then passed into qemu/kvm and the VM sees the mac address as assigned above ($vfmac): -device vfio-pci,host=81:02.1,bus=pci.1,id=vfnet0 iovctl allows for mac address control of all VFs under an interface at one time - and requires the destruction and re-creation of all VF entries in order to modify a single entry - which breaks existing running VMs. Is there a way I have missed to modify the mac address of a single VF which is presented to a VM? I have tried the following steps. # iovctl -C -f /root/iov-cc0.conf # Create, for instance, 4 VFs agains cc0. # pciconf -l | grep vf t6vf0@pci0:65:0:8: class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x6808 subvendor=0x1425 subdevice=0x0000 t6vf1@pci0:65:0:12: class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x6808 subvendor=0x1425 subdevice=0x0000 t6vf2@pci0:65:0:16: class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x6808 subvendor=0x1425 subdevice=0x0000 t6vf3@pci0:65:0:20: class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x6808 subvendor=0x1425 subdevice=0x0000 ccv0: flags=8802 metric 0 mtu 1500 options=66ec07bb (100GBase-SR4 ) status: active nd6 options=29 ccv1: flags=8802 metric 0 mtu 1500 options=66ec07bb (100GBase-SR4 ) status: active nd6 options=29 ccv2: flags=8802 metric 0 mtu 1500 options=66ec07bb (100GBase-SR4 ) status: active nd6 options=29 ccv3: flags=8802 metric 0 mtu 1500 options=66ec07bb (100GBase-SR4 ) status: active nd6 options=29 Note: initial mac 02:00:00:00:00:00 indicating not-in-use and/or available. # ifconfig ccv3 ether 02:22:33:44:55:66 # ifconfig ccv3 ccv3: flags=8802 metric 0 mtu 1500 options=66ec07bb ether 02:22:33:44:55:66 hwaddr 02:00:00:00:00:00 media: Ethernet 100GBase-SR4 (100GBase-SR4 ) status: active nd6 options=29 Ether now shows with new mac - hwaddr the original as assigned by iovctl. Now re-assign the VF device to the pass-thru driver: devctl set driver -f pci0:65:0:20 ppt The VF device can now be passed into bhyve (65/0/20) but the new value 02:22:33:44:55:66 is lost and the prior value 02:00:00:00:00:00 is seen by the vm - but we want 02:22:33:44:55:66. So the new mac is a property of the driver, not the VF. At this point I'm wondering if adding a -U (update) option to iovctl or a new standalone program to target an individual iface/vf pair for update might be the answer. Comments welcome. Thanks, John