From owner-freebsd-virtualization@freebsd.org Fri Feb 5 12:55:35 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E86CCA77EA4 for ; Fri, 5 Feb 2016 12:55:34 +0000 (UTC) (envelope-from decui@microsoft.com) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2on0145.outbound.protection.outlook.com [207.46.100.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 300CAD4 for ; Fri, 5 Feb 2016 12:55:33 +0000 (UTC) (envelope-from decui@microsoft.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector1; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=HRyHMgtSXMGtZVS5leHrKXRAr06CzDHVgfNj1DNutC4=; b=Fq25VxqlMy+xKZHgAD+47zLBtz1KFe0ieyR2aN5PbvXwg8WL0JclbQf+KthzsI2f9Zsrg8rCCcV4POyAgVUDyjUC7hfM9rzQgIBkyd7Wywax4ZeU6f6Hx5/I8AJZngpMLbbaYfR4pYMYERMTAyPlsl5GJ2duwqRtDH1hO4zmid0= Received: from BN3PR0301CA0025.namprd03.prod.outlook.com (10.160.180.163) by CY1PR0301MB0842.namprd03.prod.outlook.com (10.160.163.148) with Microsoft SMTP Server (TLS) id 15.1.403.16; Fri, 5 Feb 2016 08:23:03 +0000 Received: from BY2FFO11FD015.protection.gbl (2a01:111:f400:7c0c::111) by BN3PR0301CA0025.outlook.office365.com (2a01:111:e400:4000::35) with Microsoft SMTP Server (TLS) id 15.1.403.16 via Frontend Transport; Fri, 5 Feb 2016 08:23:03 +0000 Authentication-Results: spf=pass (sender IP is 23.103.249.84) smtp.mailfrom=microsoft.com; bugworks.com; dkim=none (message not signed) header.d=none; bugworks.com; dmarc=pass action=none header.from=microsoft.com; Received-SPF: Pass (protection.outlook.com: domain of microsoft.com designates 23.103.249.84 as permitted sender) receiver=protection.outlook.com; client-ip=23.103.249.84; helo=064-smtp-out.microsoft.com; Received: from 064-smtp-out.microsoft.com (23.103.249.84) by BY2FFO11FD015.mail.protection.outlook.com (10.1.14.131) with Microsoft SMTP Server (TLS) id 15.1.409.7 via Frontend Transport; Fri, 5 Feb 2016 08:23:02 +0000 Received: from HKXPR3004MB0088.064d.mgd.msft.net (141.251.197.88) by HKXPR3004MB0088.064d.mgd.msft.net (141.251.197.88) with Microsoft SMTP Server (TLS) id 15.1.409.10; Fri, 5 Feb 2016 08:22:57 +0000 Received: from HKXPR3004MB0088.064d.mgd.msft.net ([141.251.197.88]) by HKXPR3004MB0088.064d.mgd.msft.net ([141.251.197.88]) with mapi id 15.01.0409.010; Fri, 5 Feb 2016 08:22:57 +0000 From: Dexuan Cui To: Jac Backus , "Sephe Qiao (Wicresoft)" , Kylie Liang , "'freebsd-virtualization@freebsd.org'" , BSD Integration Components for Hyper-V Subject: RE: Hyper-V networking: problem after upgrade to 10.2 Thread-Topic: Hyper-V networking: problem after upgrade to 10.2 Thread-Index: AdFXuefm77fqXshWSUq/xyLaKrOi2AAEaGnQAA5R0WAAAFPEIAAHGGIQAB849pAAABp5MAAD39gwAAxCoKAALasVEAAz7EpgAAH/D1AASYVK4AAUkXXgABTfDnAAFm9KUAAPGOaQAADXxAAARLesMAAMBcNAABCbgZAAANu3cAAenrMwAA2JiMAANmckAAABL75w Date: Fri, 5 Feb 2016 08:22:56 +0000 Message-ID: <2cdde8e104d04875a3e2614686426375@HKXPR3004MB0088.064d.mgd.msft.net> References: <8572369fcc3b408891fc1a5a7d11e892@SG2PR3002MB0107.064d.mgd.msft.net> <1430812ff38c4e08a4b91ea25fdb5a7b@HKXPR3004MB0088.064d.mgd.msft.net> <1fdbb9b939c54e31ac00329f61bf6f41@ORTELIUS.internal.bugworks.com> <2323532a95934cdfae0142a9d6f88fd8@SG2PR3004MB0090.064d.mgd.msft.net> <2b22a9544c6c4ff3b017133a346e75e2@SG2PR3004MB0090.064d.mgd.msft.net> <7df41cc958ee408297487d4ffbb91903@HKXPR3004MB0088.064d.mgd.msft.net> <4e85ad234a0b4618ae5862fbcd266e3d@ORTELIUS.internal.bugworks.com> <0f761afc10864ad3aeb89ee7c9b6e8ac@HKXPR3004MB0088.064d.mgd.msft.net> <4105829efb1e4b3a91bc17f7fbdf8ac8@HKXPR3004MB0088.064d.mgd.msft.net> <0ce04bf413204478b1e7cc71ac28ecac@ORTELIUS.internal.bugworks.com> <1aba0d827b9041b79b85a09cf70e52b1@HKXPR3004MB0088.064d.mgd.msft.net> <4a98b41221ed4c3b84a8c733aa23f24d@HKXPR3004MB0088.064d.mgd.msft.net> <5a09277abe094f5989fb145c12a511df@ORTELIUS.internal.bugworks.com> <17b639f383df4f42b1e050f6e40d1ad2@HKXPR3004MB0088.064d.mgd.msft.net> <53238096372f4362b436382a430bde57@ORTELIUS.internal.bugworks.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [141.251.56.4] X-MS-Office365-Filtering-Correlation-Id: 223a8e23-752a-47fa-94dc-08d32e0590c0 MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD015; 1:4Tkbs9+gEv/c9GhVOgOTrHXu6KhSE2P7AOEVPQTbGiGXtr+IOb21s5gCQAjpeegyddYX1UxoG1hGV1RIitZ5IKWZVME+03nVxQ6liqc3u6TlBfwE6MNYivF2mgwtmc35oyamhk+PY2wFhDay30BbwM7mM9SsWGohFz84inSt1pFSB9ZhrOWXqsFpWfBM9SPJcnx2yWsOLzR2FTmh9aeE2ImI6xtn3VIJv8JCUYmCCgsJH2CvAdiKEhaqBSe09HvBciqlw+YoZl5c7ZFrwyR3NI4uRBp+LLEPTtHpJbuof9MoJ3yoCzUOcpU9SgtOHsQQi9lvfcFhJ1f6xDe4WbhLEQ== X-Forefront-Antispam-Report: =?us-ascii?Q?CIP:23.103.249.84; CTRY:; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(100190?= =?us-ascii?Q?20)(2980300002)(438002)(52604005)(199003)(3905003)(45984002)?= =?us-ascii?Q?(164054003)(51914003)(189002)(71364002)(13464003)(377424004)?= =?us-ascii?Q?(377454003)(5008740100001)(76176999)(24736003)(50986999)(589?= =?us-ascii?Q?0100001)(15975445007)(19627595001)(5001770100001)(2561002)(9?= =?us-ascii?Q?9936001)(2900100001)(18206015028)(3900700001)(106466001)(108?= =?us-ascii?Q?616004)(5003600100002)(5001960100002)(54356999)(107886002)(8?= =?us-ascii?Q?6146001)(6806005)(5005710100001)(11100500001)(189998001)(512?= =?us-ascii?Q?954002)(2906002)(19300405004)(19617315012)(2950100001)(10400?= =?us-ascii?Q?500002)(16236675004)(87936001)(5004730100002)(92566002)(1776?= =?us-ascii?Q?0045003)(66066001)(67866002)(300700001)(33646002)(1958040500?= =?us-ascii?Q?1)(19580395003)(1511001)(3846002)(102836003)(86362001)(58600?= =?us-ascii?Q?3)(10290500002)(2421001)(1096002)(66926002)(575784001)(12207?= =?us-ascii?Q?00001)(6116002)(19625215002)(16796002)(84326002)(40014501000?= =?us-ascii?Q?02)(10090500001)(790700001)(93886004)(260700001)(7099028)(49?= =?us-ascii?Q?1001)(559001)(579004)(562774006);DIR:OUT;SFP:1102;SCL:1;SRVR?= =?us-ascii?Q?:CY1PR0301MB0842;H:064-smtp-out.microsoft.com;FPR:;SPF:Pass;?= =?us-ascii?Q?MLV:sfv;A:1;MX:1;LANG:en;?= X-Microsoft-Exchange-Diagnostics: 1; CY1PR0301MB0842; 2:gvxgFq7rP0IvcbHh/zMDn28qNrqy84zkNt9XFywSe329zNcdZbt5nRY52lWAjq9/UlUBzqIlwM32fOksOXbhcMy/DOjSSs6WhXBiVci7oNjOoYGDKfMDIvTGuiLHBqKLKvbloOEhJBweA+LPJFLzJHJVysiUKkcGXM6TFbpTSuRoTJuh/oPmqbsG27evNwcw; 3:IhiyLqrlbuK69Tlh3EG7pprP5aayzI03y3bmk3PPngVbSPOrAM66TqHRRm43mwAjSmlXW4R5rVNafx8FCPJ3PGIPgD0kMshZ00D9ovA7MRtIzV7g811AfzV8/I8rz6N5JqmyxDsuAGo94d+VHw41L/tWFtVy9l1Jwa6uk3WCwnBfP1rCx186ISfz9rDNTmg8P14C1nUcae2udV64eAYwROFvOjTw7I8G5GK0bOD36ID3E9f56rociBKI/zzc+tCT43R4v/90ip/yDGRpLTzofQ==; 25:DbQQqkC2RzK5a0LpkFDpyUp1fTzgDIhRdR2ZXrhrOxQwk7b74g/syIIVTfnDd8c4SSZDSp6y0F0YrAVwMuP7fiVRCB/pH6V6CZUyP5lIm7SAJB6HnXcc9m3rnRI1LjkRXj+UPex2N45d217gEEEfW5jw/Mo5VtHO7R2h/79oonxFpKw7UhWg1fSz5xCNVSpBUduQAkM3UF2UI5GmJnpWRD0QDFngMSvlYge/mlgO0ouUFld0mkG+163I88GBsKiqA4XIz5GG4C3CnBsWWaAeogC5/shKQcMgeeRtPbiwho+e8iKWg4D2mEUCkiQXdl04izEJcB8lzk79HCAYuWZkMbfkjBtZM6wbBMhSx7DWCK8= X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(8251501002); SRVR:CY1PR0301MB0842; X-O365EOP-Header: O365_EOP: Allow for Unauthenticated Relay X-Microsoft-Exchange-Diagnostics: 1; CY1PR0301MB0842; 20:75VMNnMSgdn27o1NHjg/I4QXHPYCckckIDd02M5Fg1OSxXP3qQVx4hH/3QTFAFIvExmZgMlIOZdMCKutA5+UDnajcCK+mMK7U+LHy//AfCsjs9Pen9JxULV1DPB+Rpb1Sh2oWe+XQ6HocCrBELH1l2sSQNPNO+HT6Y9oWaN73TZSchhDRt/E52Ec5nsPyxHa9osKtcHNF/2v7V2fpZBsHl8hnuTsLreF1eVxY24wDxGjhxuQnJyYPtkt1s8ElHM9r+mPeiN1w/iWZYT3fkPBPzvZgExuPwSwTi7E05ACdSQ/9+1cfwgi5RdXkbCqU0rrCt2L0Z6xFb4G1WOgOhjoyboqa/kV/VpVMrPlZbOajZP3vPYX+8H9mw96Lf2jQePc4yYcL2azi7N3C9KOmZ8IG4JOmNMqLcljjjEns4/73ZJ+K+TtR0BLG1qBFB3EexIZDoOkpzqzz1qXdnhHAD4g1uUBgP8JgscPbgUDnwjawHoWatmFyMkBku+QTJCnxreBA1n+9kuCUKUKHifQZV9tULRMJqXbQOeEG+gUZmom1r0JhNhl6LwvKdwqo/hLpOsZLU/hdoPp7+/PWJaLRLxqYgddE90cMAId5rtbizWPB5c= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(102415267)(61425038)(102615245)(601004)(2401047)(8121501046)(13018025)(5005006)(13016025)(3002001)(10201501046)(61426038)(61427038); SRVR:CY1PR0301MB0842; BCL:0; PCL:0; RULEID:; SRVR:CY1PR0301MB0842; X-Microsoft-Exchange-Diagnostics: 1; CY1PR0301MB0842; 4:w0R4L+fGyRVGHx5GjQ/tk4liRSDZzoa1L2BNNwaIw8FoKtiycnTG9O5dyh+17gQBJW4FCAIiw48cbwOSP2eygkP54/cHuRpBIMEom36eSKOWZyMRBBFno0ZnrH+6yjSTfDLBIQcnI7tzaE1Ml9FCyXikbO3txerWfWhLjTZTq4tbvgie30GLHyiv9Hno2eb76R1fCttf58I4Vm05ELAahhxHHeaztuP9JL98RHTqgqjvAJ/6jKS3POa5z9RCCKR6wEaH0WPcYHFauCRfqo89+nvrPlOAuuC8aVEdFjzqj7PECAQB1lDSizJ7T8aJBzdKv6p832oN78ahTC56FNITnwcfSsi8oK0TQTZWrIStE78h7kSWRNG/7CR0mJZ/ebF6k/fdXtZExobHlTVSm8BNCHmlxEKgk1Bko9Ai0EKyme4+HV8qYMbY/xoWufKeM3stvtT5jRPdhv1KoPdlU4a1fMtEHOr7bJQUWIwz32rTyZuEUHdKlYn4WosAGo2so7ZC X-Forefront-PRVS: 0843C17679 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY1PR0301MB0842; 23:i6cEpr242sHVjyp96k1+M03fsiKQZUemigTtKyj?= =?us-ascii?Q?LnW7K7N30T2V8OVrDlSm0nKucP3h/lyK9VIBTcju6VGMWjR1n4uGVETIV8Xx?= =?us-ascii?Q?nw5N/h+r3u6Ys4xaGUrq9JrUsLUx98ZnPso+7L4hmYFHao3BLelFTKpQQW/r?= =?us-ascii?Q?8dbJgvyAq/i0bQQWeqkdS874QSlXa5uH9+rL0of9FXglMs3z2X3dsKVN/41l?= =?us-ascii?Q?YO5BsaSvdtBg58E3/mrdWKAeIZBEYhSNK+bengLaCUNAQSeRnOVn/9rK5rNs?= =?us-ascii?Q?j2XdPjlMhKex1RqcfTJp5E/sgLm4FLDu3SCjr9PLd6/S2o0s+kKjhg5UUfGN?= =?us-ascii?Q?XwVgNEc6rbb3glW0jXtFl8erMh7nZkY0jK/3g1oXPh8m64Nav7Wz42KaWeR5?= =?us-ascii?Q?CgKuunRxSWV4fpUaT8Fr0nm/CD1C4BgyvP386wxc+lBhu0pweC3HHOFBhSQS?= =?us-ascii?Q?uC4MWrsFWcZMmYVhhKwCUz+ryPAo4NV2ozjOie6ZbgS7OWthOPb/hBXOEFYH?= =?us-ascii?Q?sOJC5rtKUZTLsehDsNe9nS+Do9zsnbNv4u0FxcI/sFlZNPEA/ns1FBvOyYt6?= =?us-ascii?Q?+4eTd4GCw8dU2tqtEAsSAfOILvoT5JgGk63yNb3Lbjx7ArEeVcSXkebiNxB/?= =?us-ascii?Q?lXLhR9KSVVCW+402tbIby0yw2CSufRS7YhTiBA/taKPuxvNb5VJNBU1R/PhB?= =?us-ascii?Q?/W5d+H9aqGaey39P/iniaavNuQS1Ma2VVWMG35tFoWYjF29kxDcl+3h8KHD+?= =?us-ascii?Q?v7igeEPLipBnkjWf7XpX56rTqsCtFacWEIDiT9ZayBfzvR3aLdqlHYceHtTH?= =?us-ascii?Q?xuvheqWdlpVReb+5x5KVSXC+Ok81pjcCrogr8jXW874zRaBb2X5qQJklIvjw?= =?us-ascii?Q?3VnXqHHqUvdmcXYkvl6lr7N17nqm33deLu81CfxOk6G4t4+6uadsz+LZtdRu?= =?us-ascii?Q?VmNP1UWBkd/H2fN+IuIECVEpRQj/vgDCl/LhVR9YpyrfeRN8Ix7Ul0KS4+Zb?= =?us-ascii?Q?aclSD2eTFviNeU6DQrjdwavaNJoXn46Ho0Izu/OmrqlNHcTD7yDZt+wHE/Xx?= =?us-ascii?Q?FuqO7yv8H6IGW9BBQQHDAQIro3Q7i82L+1Rfjw9VJce/TUa2hZYsBTsC9Xmd?= =?us-ascii?Q?PP2lAz3o64B+KeCUmUqLkGwPDcDOlPYRTn3nZa0G0oARJ5HRMRbMAjRHmNDi?= =?us-ascii?Q?Ghm/E2dXHKgzJba/kqYVoq3BSDijbzNUpNSvnnb3D2OjOgDRjoX1QDtY7eTh?= =?us-ascii?Q?ZzdSvylUw6BTA2dDJwcsc008AA3uPiz/k4KQ+vH1o1Qf7yigWjm16vqxDxg3?= =?us-ascii?Q?tyj/B/tAwcuKYSPfWYWqmToX8PvzCIXjCN+Xiys/gwtNtCdi8Rx3KMOtVpCe?= =?us-ascii?Q?IG9DrtTHHOL0zGNTS7hac0l2nUXK4a+hyE8Md62G7ZolDP5rY+puLngwQVEl?= =?us-ascii?Q?D9w0EH4aQTtvb/8FgvldmXSg8P1Dxqy0QS9x2UCU8XqVZzSdz+uOlcLDORpe?= =?us-ascii?Q?+dZqQzMI4rU7Tx+tP1WDCK/18nhJwhpqVyzl/GeBUJqk2JbYhOCDtKMWlRPr?= =?us-ascii?Q?VLQj1yO7rKZm2LnnBCiICEK0yMyBlqUgNGwa4EJWKvk/3FTZFHTl3A9bj7Tn?= =?us-ascii?Q?KPFJ0Xxcfvu6ePnXwc07nBtXxhJPIINxOGPGUlt7S5Xqi3s0OyMzgq82JM0o?= =?us-ascii?Q?cs8W3JRX3mXPFbw4NqJanBGOTayuFZL/4CXtWnO3AjqquNNsW7hkem+Za+n2?= =?us-ascii?Q?arHvF+CGxdZgXU9ZYdrlijciBmv+llZkYTYilknqZRUXDIZKLZbRypPkYiBn?= =?us-ascii?Q?GBzFgzn4MKuHM0gY/C8PDzC65UK4EG8KEZGPaBy/kHEuD3Pn6ioCNp72yWaI?= =?us-ascii?Q?Uz6ywcUdj1cYnO9VyvW2f1Ep8dkIQSRFY7IUzsInt9mWTGqOImeTM3pVHUs+?= =?us-ascii?Q?IA7B3P6/4PbnR0JRmDkYGTSW+/9obLhUEKnq77bHdFhpb0AXPYpsOrqQEZxy?= =?us-ascii?Q?XwtGpaS0LEeaQimW82yFJEf+i52DlLeiOpn/P+8/KnOGrOQ=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; CY1PR0301MB0842; 5:/RcXa+qev/XqrXTgJWwPWueQTUwOTofpaujf/wrqFzuMIUuZSk12Uk6WqEZc0M2Zq3NCeCjzYc+f/icD0R01RJH4+3NXbjGIirIelOAg4pYkuWdmENWVK/3GLtpy3T0PH3nkwV/gZM978r79qY4ulg==; 24:13KsOXEnOPB8PfpcgHaT7q0IBqwkR2/KAvRxGraoaptUjBwnivVVroaFicJwLzoJUXWCnrywA8B96HQabp4MICdOYkCqXKFnZxKRGdJQiK4= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Feb 2016 08:23:02.4506 (UTC) X-MS-Exchange-CrossTenant-Id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=72f988bf-86f1-41af-91ab-2d7cd011db47; Ip=[23.103.249.84]; Helo=[064-smtp-out.microsoft.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0301MB0842 X-Mailman-Approved-At: Fri, 05 Feb 2016 13:04:43 +0000 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2016 12:55:35 -0000 Hi Jac, Thanks for confirming the fixes! Please let me explain the issue a little more. The issue exists in the VMBu= s driver. Since the network driver depends on the VMBus driver, we have the= issue. The issue is: the VMBus doesn't guarantee the serialization of registering = multiple NICs, e.g., supposing the VM has 2 NICs (NIC1 with MAC1 and NIC2 w= ith MAC2), 2 kernel threads (FreeBSD's taskqueues) are created to register = the NICs concurrently; if sometimes NIC1 is registered first, NIC1 will be = "hn0"and sometimes if NIC2 is registered first, NIC2 will be "hn0"... My fixes make sure the proper serialization, so now we can make sure NIC1 i= s always registered before NIC2. About another issue ("calcru: runtime went backwards"), I made a fix recent= ly: https://reviews.freebsd.org/D5174. Please try it. I'm trying to push this fix to the Head branch. Thanks, -- Dexuan From: Jac Backus [mailto:j.backus@bugworks.com] Sent: Friday, February 5, 2016 15:55 To: Dexuan Cui ; Sephe Qiao (Wicresoft) ; Kylie Liang ; 'freebsd-virtualization@fre= ebsd.org' ; BSD Integration Components = for Hyper-V Subject: RE: Hyper-V networking: problem after upgrade to 10.2 Hello Dexuan, I build a new kernel and the problem is gone. I tried to understand your description of the patch. What I find remarkable= is, that there is apparently no fixed relation between the virtual nics de= fined in Hyper-V (with a fixed mac address) and the devices, hn0, hn1, etc = in the guest, and is the order determined at boot time. Something completely different: maybe you can point me to an explanation o= f these messages when I boot: calcru: runtime went backwards from 1672 usec to 845 usec for pid 144 (adjk= erntz) calcru: runtime went backwards from 5 usec to 2 usec for pid 7 (pagezero) calcru: runtime went backwards from 126 usec to 63 usec for pid 4 (sctp_ite= rator) calcru: runtime went backwards from 6920 usec to 3743 usec for pid 3 (fdc0) calcru: runtime went backwards from 33080 usec to 16732 usec for pid 13 (ge= om) calcru: runtime went backwards from 12117 usec to 6318 usec for pid 1 (init= ) calcru: runtime went backwards from 155335 usec to 104098 usec for pid 0 (k= ernel) I happens since I virtualized the server. Again, I thank you very much for the help. And, of course, especially for t= he fix! With kind regards, Jac Van: Dexuan Cui [mailto:decui@microsoft.com] Verzonden: donderdag 4 februari 2016 7:01 Aan: Jac Backus; Sephe Qiao (Wicresoft); Kylie Liang; 'freebsd-virtualizati= on@freebsd.org'; BSD Integration Components for Hyper-V Onderwerp: RE: Hyper-V networking: problem after upgrade to 10.2 Hi Jac, The easiest way to get the fixes is just using the latest stable/10 branch = as I mentioned in another mail. Alternatively, if you want to use your local 10.2 source code in /usr/src/ = + the fixes only, please read the below: I put the 10.2-specific fixes onto my github branch "decui/10.2/fix_mac_ord= er" (please see the top 2 patches) : https://github.com/dcui/freebsd/commits/decui/10.2/fix_mac_order (I resolve= d a small code conflict) I verified the 2 patches could be cleanly applied to a clean installation o= f 10.2 VM: [root@decui-bsd102 /usr/src]# pwd /usr/src [root@decui-bsd102 /usr/src]# wget https://github.com/dcui/freebsd/commit/b= 706b383da285376554bcb69f44c4cc10270de24.patch --2016-02-04 13:37:04-- https://github.com/dcui/freebsd/commit/b706b383da2= 85376554bcb69f44c4cc10270de24.patch Resolving github.com (github.com)... 192.30.252.131 Connecting to github.com (github.com)|192.30.252.131|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/plain] Saving to: 'b706b383da285376554bcb69f44c4cc10270de24.patch.1' b706b383da285376554bcb69f44c4cc10270de24.pa [ <=3D> = ] 7.13= K --.-KB/s in 0.006s 2016-02-04 13:37:05 (1.22 MB/s) - 'b706b383da285376554bcb69f44c4cc10270de24= .patch.1' saved [7297] [root@decui-bsd102 /usr/src]# patch -sp1 < b706b383da285376554bcb69f44c4cc1= 0270de24.patch [root@decui-bsd102 /usr/src]# wget https://github.com/dcui/freebsd/commit/2= bff041dbed26b5da88c4be72b4701bbf6c460cd.patch --2016-02-04 13:37:26-- https://github.com/dcui/freebsd/commit/2bff041dbed= 26b5da88c4be72b4701bbf6c460cd.patch Resolving github.com (github.com)... 192.30.252.129 Connecting to github.com (github.com)|192.30.252.129|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/plain] Saving to: '2bff041dbed26b5da88c4be72b4701bbf6c460cd.patch.1' 2bff041dbed26b5da88c4be72b4701bbf6c460cd.pa [ <=3D> = ] 4.22= K --.-KB/s in 0.003s 2016-02-04 13:37:27 (1.31 MB/s) - '2bff041dbed26b5da88c4be72b4701bbf6c460cd= .patch.1' saved [4323] [root@decui-bsd102 /usr/src]# patch -sp1 < 2bff041dbed26b5da88c4be72b4701bb= f6c460cd.patch [root@decui-bsd102 /usr/src]# If the related files in your /usr/src/sys/dev/hyperv/ were messed up by the= previous failed patch commands, you can replace the files with the version= here (https://github.com/dcui/freebsd/tree/decui/10.2/fix_mac_order/sys/de= v/hyperv. You can use the "Raw" format functionality to get the related URL= s of the files and use 'wget' to get them): sys/dev/hyperv/include/hyperv.h sys/dev/hyperv/vmbus/hv_channel_mgmt.c sys/dev/hyperv/vmbus/hv_vmbus_drv_freebsd.c sys/dev/hyperv/vmbus/hv_vmbus_priv.h Please let me know if this will work for you. Thanks, -- Dexuan From: Jac Backus [mailto:j.backus@bugworks.com] Sent: Thursday, February 4, 2016 7:18 To: Dexuan Cui >; Sephe Qia= o (Wicresoft) >; Kyli= e Liang >; 'freebsd-virtu= alization@freebsd.org' >; BSD Integration Components for Hyper-V > Subject: RE: Hyper-V networking: problem after upgrade to 10.2 Hello Dexuan, The first patch gives no messages. When trying the second: 112 # patch -sp1 < 1e469c559048fe6ec3641da3bb21ab87215c6506.patch 1 out of 7 hunks failed--saving rejects to sys/dev/hyperv/vmbus/hv_channel_= mgmt.c.rej Attached you find the patched file (as a session log). With kind regards, Jac Van: Dexuan Cui [mailto:decui@microsoft.com] Verzonden: woensdag 3 februari 2016 10:05 Aan: Jac Backus; Sephe Qiao (Wicresoft); Kylie Liang; 'freebsd-virtualizati= on@freebsd.org'; BSD Integration Components for Hyper-V Onderwerp: RE: Hyper-V networking: problem after upgrade to 10.2 Thanks for the confirmation, Jac. I might be wrong with 10.1 - it may not have the issue. In 10.2 we made a lot of changes and I think the race condition was introdu= ced. To test the 2 patches, you can do something like cd /usr/src (supposing the 10.2 kernel code is in the sys/ sub-directory) wget https://github.com/freebsd/freebsd/commit/850d0994e48b0ef68d33875e2632= 6d44931fcf1e.patch patch -sp1 < 850d0994e48b0ef68d33875e26326d44931fcf1e.patch wget https://github.com/freebsd/freebsd/commit/1e469c559048fe6ec3641da3bb21= ab87215c6506.patch patch -sp1 < 1e469c559048fe6ec3641da3bb21ab87215c6506.patch make buildkernel KERNCONF=3DGENERIC -j8 make installkernel reboot You may get a small issue when applying the second patch as I did: 1 out of 8 hunks failed--saving rejects to sys/dev/hyperv/vmbus/hv_channel_= mgmt.c.rej You can fix this by checking the .patch/.c files and manually editing the .= c file. Thanks, -- Dexuan From: Jac Backus [mailto:j.backus@bugworks.com] Sent: Wednesday, February 3, 2016 16:10 To: Dexuan Cui >; Sephe Qia= o (Wicresoft) >; Kyli= e Liang >; 'freebsd-virtu= alization@freebsd.org' >; BSD Integration Components for Hyper-V > Subject: RE: Hyper-V networking: problem after upgrade to 10.2 Good day Dexuan, I think it is. I should like to test. Are there some instructions for patching the 10.2 ke= rnel source? You mention 10.1 too, but I never had the problem with 10.1. Thanks very for your kind help! With kind regards, Jac Van: Dexuan Cui [mailto:decui@microsoft.com] Verzonden: woensdag 3 februari 2016 1:50 Aan: Jac Backus; Sephe Qiao (Wicresoft); Kylie Liang; 'freebsd-virtualizati= on@freebsd.org'; BSD Integration Components for Hyper-V Onderwerp: RE: Hyper-V networking: problem after upgrade to 10.2 Jac, really great news! So, can I think the whole issue in your side is caused by Bug 205156? The fix to the bug has been in the 10/stable branch and should be in the co= ming 10.3. For 10.1 and 10.2 , I'm afraid you'll have to manually apply the patches an= d build a new kernel. BTW, the bug is actually a race condition when the netvsc driver registers = multiple NIC devices, so sometimes we can easily repro the issue and someti= mes we can't. Thanks, -- Dexuan From: Jac Backus [mailto:j.backus@bugworks.com] Sent: Wednesday, February 3, 2016 2:28 To: Dexuan Cui >; Sephe Qia= o (Wicresoft) >; Kyli= e Liang >; 'freebsd-virtu= alization@freebsd.org' >; BSD Integration Components for Hyper-V > Subject: RE: Hyper-V networking: problem after upgrade to 10.2 Dexuan, you are briljant! That is the problem: Hn0 has the mac address of hn1, hn1 of hn2 and hn2 of = hn0. So they have shifted one position to the left. With kind regards, Jac Van: Dexuan Cui [mailto:decui@microsoft.com] Verzonden: maandag 1 februari 2016 10:41 Aan: Jac Backus; Sephe Qiao (Wicresoft); Kylie Liang; 'freebsd-virtualizati= on@freebsd.org'; BSD Integration Components for Hyper-V Onderwerp: RE: Hyper-V networking: problem after upgrade to 10.2 Hmm, it's really strange... what's the difference between your existing 10.= 1 VM and a fresh 10.1 VM... :( BTW, please check if you are seeing this bug (it looks in your side the ne= twork can stop working after a VM reboot): Bug 205156 - [Hyper-V] NICs' (hn0, hn1) MAC addresses can appear in an unce= rtain way across reboot (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id= =3D205156)? Thanks, -- Dexuan From: Jac Backus [mailto:j.backus@bugworks.com] Sent: Monday, February 1, 2016 17:17 To: Dexuan Cui >; Sephe Qia= o (Wicresoft) >; Kyli= e Liang >; 'freebsd-virtu= alization@freebsd.org' >; BSD Integration Components for Hyper-V > Subject: RE: Hyper-V networking: problem after upgrade to 10.2 Good day Dexuan, I did. Unfortunately, no difference. And at the moment the server is running on the 10.2 kernel: uname -a FreeBSD roadrunner.acme.inc 10.2-RELEASE-p9 FreeBSD 10.2-RELEASE-p9 #0: Thu= Jan 14 01:32:46 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/u= sr/src/sys/GENERIC amd64 But there is a big chance, that after a reboot, the network is gone again. I will see if, when it works, it keeps working. I suppose it does. With kind regards, Jac Van: Dexuan Cui [mailto:decui@microsoft.com] Verzonden: maandag 1 februari 2016 3:07 Aan: Jac Backus; Sephe Qiao (Wicresoft); Kylie Liang; 'freebsd-virtualizati= on@freebsd.org'; BSD Integration Components for Hyper-V Onderwerp: RE: Hyper-V networking: problem after upgrade to 10.2 Hi Jac, Good to know this! It looks to me something in the VM or in the host might be causing the issu= e??? Can you please do another quick test: shut down the "buggy" VM and remove i= t in Hyper-V Manager (this will keep the .vhdx image) and then re-create th= e VM with the .vhdx image? Thanks, -- Dexuan From: Jac Backus [mailto:j.backus@bugworks.com] Sent: Sunday, January 31, 2016 23:21 To: Dexuan Cui >; Sephe Qia= o (Wicresoft) >; Kyli= e Liang >; 'freebsd-virtu= alization@freebsd.org' >; BSD Integration Components for Hyper-V > Subject: RE: Hyper-V networking: problem after upgrade to 10.2 Hello Dexuan, I did a fresh install of a 10.1 VM and upgraded it to 10.2. Is looks like i= t works well. With kind regards, Jac Van: Dexuan Cui [mailto:decui@microsoft.com] Verzonden: zondag 31 januari 2016 7:07 Aan: Jac Backus; Sephe Qiao (Wicresoft); Kylie Liang; 'freebsd-virtualizati= on@freebsd.org'; BSD Integration Components for Hyper-V Onderwerp: RE: Hyper-V networking: problem after upgrade to 10.2 Hi Jac, Good to know the information. Since I can't repro the issue, it's difficult for me to debug it. :( I'm guessing if it would help if you use a permanent ARP entry in the VM ("= arp -s hostname ether_addr") for the other end - surely this is only for de= bug purpose. During the VM boot-up, can you keep pinging the VM from the other host. I m= ean: it looks the NIC never works since it becomes UP in the VM? BTW, I'm not sure if it's easy for you to do the same test as mine, i.e., d= o a fresh installation of 10.1 VM and upgrade it to 10.2. @Sephe, any idea? Thanks, -- Dexuan From: Jac Backus [mailto:j.backus@bugworks.com] Sent: Sunday, January 31, 2016 3:42 To: Dexuan Cui >; Sephe Qia= o (Wicresoft) >; Kyli= e Liang >; 'freebsd-virtu= alization@freebsd.org' >; BSD Integration Components for Hyper-V > Subject: RE: Hyper-V networking: problem after upgrade to 10.2 Good day Dexuan, There is something wrong with getting mac addresses for host on the lan, it= seems. When I ping the 10.2 server from a host on the net, I see on that host arp= requests (Wireshark: who has ... Tell ...) for the 10.2 server. Arp -a on the 10.2 server itself says for the non-server entries ?
at (imcomplete) on hn0 expired [ethernet] Tcpdump on the 10.2 server only shows arp requests: ARP, Request who-has ... tell ... Does this help? With kind regards, Jac Van: Dexuan Cui [mailto:decui@microsoft.com] Verzonden: vrijdag 29 januari 2016 9:59 Aan: Jac Backus; Sephe Qiao (Wicresoft); Kylie Liang; 'freebsd-virtualizati= on@freebsd.org'; BSD Integration Components for Hyper-V Onderwerp: RE: Hyper-V networking: problem after upgrade to 10.2 Hmm, it's strange we can't repro. I suppose you can't ping the netgate VM (or machine) 's IP address either? = When this happens, can you check the arp table in both sides? Can you please run tcpdump in the VM and in the gateway to diagnose the iss= ue? Thanks, -- Dexuan From: Jac Backus [mailto:j.backus@bugworks.com] Sent: Friday, January 29, 2016 15:36 To: Dexuan Cui >; Sephe Qia= o (Wicresoft) >; Kyli= e Liang >; 'freebsd-virtu= alization@freebsd.org' >; BSD Integration Components for Hyper-V > Subject: RE: Hyper-V networking: problem after upgrade to 10.2 Hello Dexuan, That remarkable. My uname -a is indentical. I can ping local interfaces. Ping to other addresses in local subnet gives:= Ping: sendto: Host is down. Ping to other addresses gives: Ping: sendto: N= o route to host. Routing tables (netstat -rn) for both versions look the same. Is there something I can test? With kind regards, Jac Van: Dexuan Cui [mailto:decui@microsoft.com] Verzonden: vrijdag 29 januari 2016 4:25 Aan: Jac Backus; Sephe Qiao (Wicresoft); Kylie Liang; 'freebsd-virtualizati= on@freebsd.org'; BSD Integration Components for Hyper-V Onderwerp: RE: Hyper-V networking: problem after upgrade to 10.2 Hi Jac, I installed a 10.1 VM with FreeBSD-10.1-RELEASE-amd64-dvd1.iso and upgraded= it to 10.2 by running "freebsd-update upgrade -r 10.2-RELEASE". Everything worked just fine. With the new kernel (see the below), ssh and s= cp still works fine for me. # uname -a FreeBSD bsd101 10.2-RELEASE-p9 FreeBSD 10.2-RELEASE-p9 #0: Thu Jan 14 01:32= :46 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GE= NERIC amd64 What's the specific symptom for "networking does not work anymore" in your = side(upgrading from 10.1 to 10.2)? Thanks, -- Dexuan From: Jac Backus [mailto:j.backus@bugworks.com] Sent: Wednesday, January 27, 2016 17:35 To: Dexuan Cui >; Sephe Qia= o (Wicresoft) >; Kyli= e Liang >; 'freebsd-virtu= alization@freebsd.org' >; BSD Integration Components for Hyper-V > Subject: RE: Hyper-V networking: problem after upgrade to 10.2 Hello Dexuan, Unfortunetely, no OACTIVE flag: hn0: flags=3D8843 metric 0 mtu 1500 options=3D31b With kind regards, Jac Van: Dexuan Cui [mailto:decui@microsoft.com] Verzonden: woensdag 27 januari 2016 4:09 Aan: Sephe Qiao (Wicresoft); Jac Backus; Kylie Liang; 'freebsd-virtualizati= on@freebsd.org'; BSD Integration Components for Hyper-V Onderwerp: RE: Hyper-V networking: problem after upgrade to 10.2 Hi Jac, Please show 'ifconfig -a' when the issue happens (when you upgrade 10.1 fro= m 10.2). We suspect it may be a known OACTIVE issue and "ifconfig -a' can confirm th= is, the output has the string "OACTIVE". It looks somehow the issue doesn't happen when we use a 10.2 fresh installa= tion. Thanks, -- Dexuan From: Sephe Qiao (Wicresoft) Sent: Wednesday, January 27, 2016 9:13 To: Jac Backus >; Dexua= n Cui >; Kylie Liang >; 'freebsd-virtualization@free= bsd.org' >; BSD Integration Components for Hyper-V > Subject: RE: Hyper-V networking: problem after upgrade to 10.2 Oh, please ignore this, I think its solved :) From: Sephe Qiao (Wicresoft) Sent: Wednesday, January 27, 2016 9:10 AM To: Jac Backus >; Dexua= n Cui >; Kylie Liang >; 'freebsd-virtualization@free= bsd.org' >; BSD Integration Components for Hyper-V > Subject: RE: Hyper-V networking: problem after upgrade to 10.2 Hi Jac, What's the output of 'ifconfig -a' when this happened? Thanks, sephe From: Jac Backus [mailto:j.backus@bugworks.com] Sent: Tuesday, January 26, 2016 6:37 PM To: Dexuan Cui >; Kylie Lia= ng >; 'freebsd-virtualiza= tion@freebsd.org' >; BSD Integration Components for Hyper-V > Subject: RE: Hyper-V networking: problem after upgrade to 10.2 Hello Dexuan, It seems, it is not completely correct, although the effect is as if it is = not working. Systat -ifstat 1 shows this: [cid:image001.png@01D159DA.65A3A0E0] So something is happening. But I can not reach anything. And the server can= not be reached from the lan (hn0) or internet (hn1 and hn2). I get a firewall message in /var/log/messages (first message from 11:18:55)= : [cid:image002.png@01D159DA.65A3A0E0] But this is just caused by the problem? If I can help with further information, please let me know. Regarding Bug 187006, all interfaces have fixed addresses. With kind regards, Jac -----Oorspronkelijk bericht----- Van: Dexuan Cui [mailto:decui@microsoft.com] Verzonden: dinsdag 26 januari 2016 7:55 Aan: Kylie Liang; Jac Backus; 'freebsd-virtualization@freebsd.org'; BSD Int= egration Components for Hyper-V Onderwerp: RE: Hyper-V networking: problem after upgrade to 10.2 Hi Jac, BTW, what do you mean by saying "networking does not work anymore" -- can = you please check if your issue is the same as Bug 187006 - [hyper-v] dynamic address (dhcp) obtaining doesn't work on HYP= ER-V OS 2012 R2 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D187006 ? Thanks, -- Dexuan > -----Original Message----- > From: Dexuan Cui > Sent: Tuesday, January 26, 2016 14:49 > To: Kylie Liang >; Jac = Backus > >; 'freebsd-virtualiz= ation@freebsd.org' > >; BSD Integration Components for > Hyper-V > > Subject: RE: Hyper-V networking: problem after upgrade to 10.2 > > Hi Jac, > Kylie meant disabling TSO. Please try this ("ifconfig hn0 -tso"). > > The message " hn0: unknown status 1073872902 received" should be an > unnecessary warning only. > My 10.2 VM can work fine even if I see the message too. > > Can you please install a 10.2 VM from the 10.2 .ISO file directly as I > did and see if it works for you? > > I guess we never tried upgrading 10.1 from 10.2. > Can you please list the steps how you did the upgrading? We'll try the > same steps. > > Thanks, > -- Dexuan > > > -----Original Message----- > > From: Kylie Liang > > Sent: Tuesday, January 26, 2016 8:01 > > To: Jac Backus >; '= freebsd- > virtualization@freebsd.org' > > >; BSD Integration Components for > > Hyper- > V > > > > > Subject: RE: Hyper-V networking: problem after upgrade to 10.2 > > > > Hi Jac, > > > > Thank you for asking. To isolate your issue, could you please try > > disabling SO > on > > your 10.2 system first? Thank you. > > > > And I would like to confirm with you > > 1) You met issue for 10.2 kernel + 10.2 system > > 2) No issue for 10.1 kernel + 10.1 system > > 3) No issue for 10.1 kernel + 10.2 system > > > > Right? And add our engineers in the list. > > > > Thanks, > > Kylie Liang > > > > -----Original Message----- > > From: owner-freebsd-virtualization@freebsd.org > > [mailto:owner-freebsd- virtualization@freebsd.org] On Behalf Of Jac > > Backus > > Sent: Tuesday, January 26, 2016 5:56 AM > > To: 'freebsd-virtualization@freebsd.org' > > > > > Subject: Hyper-V networking: problem after upgrade to 10.2 > > > > Dear reader, > > > > Today, I did upgrade FreeBSD 10.1 to 10.2 running on Hyper-V on a > > full > patched > > Windows Server 2012 R2 x64 version. > > > > After the update, networking does not work anymore. > > > > In /var/log/messages is this: > > > > Jan 25 21:02:01 mercurius kernel: hn0: > > on > > vmbus0 Jan 25 21:02:01 mercurius kernel: hn0: unknown status > > 1073872902 received Jan 25 21:02:01 mercurius kernel: hn0: unknown > > status 1073872902 received Jan 25 21:02:01 mercurius kernel: hn0: hv > > send offload request succeeded Jan 25 21:02:01 mercurius kernel: hn0: U= sing defaults for TSO: > > 65518/35/2048 Jan 25 21:02:01 mercurius kernel: hn0: Ethernet address: > > 00:15:5d:ac:11:08 Jan 25 21:02:01 mercurius kernel: hn1: > Network > > Interface> on vmbus0 Jan 25 21:02:01 mercurius kernel: hn1: unknown > > Interface> status > > 1073872902 received Jan 25 21:02:01 mercurius kernel: hn1: unknown > > status > > 1073872902 received Jan 25 21:02:01 mercurius kernel: hn1: hv send > > offload request succeeded Jan 25 21:02:01 mercurius kernel: hn1: > > Using defaults for > TSO: > > 65518/35/2048 Jan 25 21:02:01 mercurius kernel: hn1: Ethernet address: > > 00:15:5d:ac:11:09 Jan 25 21:02:01 mercurius kernel: hn2: > Network > > Interface> on vmbus0 Jan 25 21:02:01 mercurius kernel: hn2: unknown > > Interface> status > > 1073872902 received Jan 25 21:02:01 mercurius kernel: hn2: unknown > > status > > 1073872902 received Jan 25 21:02:01 mercurius kernel: hn2: hv send > > offload request succeeded Jan 25 21:02:01 mercurius kernel: hn2: > > Using defaults for > TSO: > > 65518/35/2048 Jan 25 21:02:01 mercurius kernel: hn2: Ethernet address: > > 00:15:5d:ac:11:07 > > > > It worked fine with the 10.1 kernel, and when I boot this kernel, it wo= rks again: > > > > Jan 25 22:20:02 mercurius kernel: hn0: > > on > > vmbus0 Jan 25 22:20:02 mercurius kernel: hn0: Ethernet address: > > 00:15:5d:ac:11:07 Jan 25 22:20:02 mercurius kernel: hn1: > Network > > Interface> on vmbus0 Jan 25 22:20:02 mercurius kernel: hn1: Ethernet ad= dress: > > 00:15:5d:ac:11:08 Jan 25 22:20:02 mercurius kernel: hn2: > Network > > Interface> on vmbus0 Jan 25 22:20:02 mercurius kernel: hn2: Ethernet ad= dress: > > 00:15:5d:ac:11:09 > > > > So I am running a 10.2 system on a 10.1 kernel at the moment. > > > > I found nothing in /usr/src/UPDATING and not really anything on the net= . > > > > So, could you tell why does this happen, and how can I solve this? > > > > Thanks for the help! > > > > With kind regards, > > > > Jac Backus > > > > > > > > _______________________________________________ > > freebsd-virtualization@freebsd.org mailing list > > > https://na01.safelinks.protection.outlook.com/?url=3Dhttps%3a%2f%2flists > .freebs > > d.org%2fmailman%2flistinfo%2ffreebsd- > > > virtualization&data=3D01%7c01%7ckyliel%40064d.mgd.microsoft.com%7cc9ca2e > > > 0d0fef482b553f08d325d3aefb%7c72f988bf86f141af91ab2d7cd011db47%7c1&s > > data=3Do%2bMZGuBW0frrQhjAPkhrWlLgNEH8LJ7BiLUyiO4tvR0%3d > > To unsubscribe, send any mail to "freebsd-virtualization- > > unsubscribe@freebsd.org"