From owner-freebsd-virtualization@FreeBSD.ORG Mon Jan 26 07:00:57 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 90ED470E; Mon, 26 Jan 2015 07:00:57 +0000 (UTC) Received: from ustc.edu.cn (email6.ustc.edu.cn [IPv6:2001:da8:d800::8]) by mx1.freebsd.org (Postfix) with ESMTP id C712A3EB; Mon, 26 Jan 2015 07:00:56 +0000 (UTC) Received: from freebsd (unknown [58.211.218.74]) by newmailweb.ustc.edu.cn (Coremail) with SMTP id LkAmygCnVzwc5sVUXUm7Ag--.39613S2; Mon, 26 Jan 2015 15:00:52 +0800 (CST) Date: Mon, 26 Jan 2015 15:00:44 +0800 From: Tiwei Bie To: Peter Grehan Subject: Re: [PATCH] Add netmap support to bhyve Message-ID: <20150126070044.GA54990@freebsd> References: <1422045440-9410-1-git-send-email-btw@mail.ustc.edu.cn> <54C5B59B.2070100@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <54C5B59B.2070100@freebsd.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-CM-TRANSID: LkAmygCnVzwc5sVUXUm7Ag--.39613S2 X-Coremail-Antispam: 1UD129KBjvJXoWxtFWkWF1kWr1rXFWfXFyUtrb_yoWxurykpF W2gF15tr4kZwnxAa9xK3Z0gryfKF1rXFZFg3y5Kry5Kw1UW3yY9ay0yFy5A3y3urZ7AF1U JwsIqry8KrWktF7anT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUk0b7Iv0xC_Kw4lb4IE77IF4wAFF20E14v26r1j6r4UM7CY07I2 0VC2zVCF04k26cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rw A2F7IY1VAKz4vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_tr0E3s1l84ACjcxK6xII jxv20xvEc7CjxVAFwI0_Gr1j6F4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwV C2z280aVCY1x0267AKxVW0oVCq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC 0VAKzVAqx4xG6I80ewAv7VC0I7IYx2IY67AKxVWUXVWUAwAv7VC2z280aVAFwI0_Gr0_Cr 1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0Y48IcVAKI48JMxkIecxEwVAFwVW8CwCF04k20xvY 0x0EwIxGrwCFx2IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E14v26r1j6r18MI8I3I 0E7480Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_Jrv_JF1lIxkGc2Ij64vIr41lIxAI cVC0I7IYx2IY67AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Jr0_Gr1lIxAIcV CF04k26cxKx2IYs7xG6rW3Jr0E3s1lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280 aVCY1x0267AKxVWUJVW8JbIYCTnIWIevJa73UjIFyTuYvjxUy3kuDUUUU X-CM-SenderInfo: xewzqzxdloh3xvwfhvlgxou0/1tbiAQUBAVQhl+sX7QABs2 Cc: freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jan 2015 07:00:57 -0000 On Sun, Jan 25, 2015 at 07:33:47PM -0800, Peter Grehan wrote: > Hi, > > > I saw that there is an open task of adding Netmap support to bhyve in > > the latest status report of FreeBSD [1]. So I implement the netmap > > support for bhyve, and following is the patch. Hope this can be helpful! > > Nice work ! Did you manage to get any throughput numbers ? Thanks! Yeah, I have done some throughput tests. Here are the details: #1. The parameters I used to launch the VMs: $ sudo /home/btw/src/usr.sbin/bhyve/bhyve \ -A -H -P \ -s 0:0,hostbridge \ -s 1:0,lpc \ -s 2,virtio-blk,./disk-vm1 \ -s 3:0,virtio-net,vale0:vm1 \ -s 3:1,virtio-net,tap1 \ -l com1,stdio \ -m 2G -c 2 vm1 $ sudo /home/btw/src/usr.sbin/bhyve/bhyve \ -A -H -P \ -s 0:0,hostbridge \ -s 1:0,lpc \ -s 2,virtio-blk,./disk-vm2 \ -s 3:0,virtio-net,vale0:vm2 \ -s 3:1,virtio-net,tap2 \ -l com1,stdio \ -m 2G -c 1 vm2 #2. The outputs of pkt-gen as the receiver when using tap+bridge: $ ./pkt-gen -i vtnet1 -f rx 075.499873 main [1686] interface is vtnet1 075.500516 extract_ip_range [289] range is 10.0.0.1:0 to 10.0.0.1:0 075.500677 extract_ip_range075.501408 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512 [289] range is 075.503540 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512 10.1.0.1:0 to 10.1.0.1:0 075.505840 [ 79] vtnet_netmap_free_bufs freed 1024 mbufs, 0 netmap bufs on 1 queues 075.507838 main [1877] mapped 334980KB at 0x801dff000 Receiving from netmap:vtnet1: 1 queues, 1 threads and 1 cpus. 075.509298 main [1963] Wait 2 secs for phy reset 077.553231 main [1965] Ready... 077.553787 nm_open [456] overriding ifname vtnet1 ringid 0x0 flags 0x1 077.554511 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512 077.556006 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512 077.557663 receiver_body [1219] reading from netmap:vtnet1 fd 4 main_fd 3 078.563897 main_thread [1483] 311989 pps (313942 pkts in 1006261 usec) 079.574224 main_thread [1483] 333345 pps (336771 pkts in 1010278 usec) 080.584718 main_thread [1483] 307455 pps (310688 pkts in 1010514 usec) 081.600552 main_thread [1483] 332357 pps (337629 pkts in 1015862 usec) 082.604157 main_thread [1483] 342985 pps (344222 pkts in 1003606 usec) 083.614341 main_thread [1483] 327806 pps (331144 pkts in 1010184 usec) 084.634725 main_thread [1483] 320947 pps (327489 pkts in 1020385 usec) 085.645380 main_thread [1483] 328959 pps (332456 pkts in 1010630 usec) 086.654096 main_thread [1483] 313482 pps (316222 pkts in 1008740 usec) 087.671256 main_thread [1483] 307234 pps (312506 pkts in 1017161 usec) 088.673871 main_thread [1483] 311188 pps (312002 pkts in 1002615 usec) 089.686453 main_thread [1483] 322132 pps (326185 pkts in 1012582 usec) 090.694224 main_thread [1483] 331368 pps (333922 pkts in 1007708 usec) 091.714343 main_thread [1483] 290061 pps (295915 pkts in 1020181 usec) 092.724065 main_thread [1483] 316366 pps (319442 pkts in 1009722 usec) 093.749938 main_thread [1483] 322501 pps (330845 pkts in 1025874 usec) 094.758012 main_thread [1483] 292125 pps (294484 pkts in 1008074 usec) 095.765606 main_thread [1483] 338889 pps (341463 pkts in 1007594 usec) 096.781560 main_thread [1483] 327785 pps (333013 pkts in 1015949 usec) 097.789245 main_thread [1483] 330064 pps (332602 pkts in 1007690 usec) 098.794141 main_thread [1483] 328757 pps (330367 pkts in 1004896 usec) 099.804160 main_thread [1483] 302752 pps (305785 pkts in 1010019 usec) ^C100.400690 sigint_h [326] received control-C on thread 0x801806800 100.820895 main_thread [1483] 178717 pps (181708 pkts in 1016735 usec) Received 7300802 packets, in 22.84 seconds. Speed: 319.61 Kpps 100.822469 [ 79] vtnet_netmap_free_bufs freed 0 mbufs, 506 netmap bufs on 1 queues #3. The outputs of pkt-gen as the receiver when using netmap+vale: $ ./pkt-gen -i vtnet0 -f rx 139.084350 main [1686] interface is vtnet0 139.085422 extract_ip_range [289] range is 10.0.0.1:0 to 10.0.0.139.086441 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512 1:0 139.085705 139.090250 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512 extract_ip_range [289] range is 10.1.0.1:0 to 10.1.0.1:0 139.094911 [ 79] vtnet_netmap_free_bufs freed 1024 mbufs, 0 netmap bufs on 1 queues 139.098783 main [1877] mapped 334980KB at 0x801dff000 Receiving from netmap:vtnet0: 1 queues, 1 threads and 1 cpus. 139.099642 main [1963] Wait 2 secs for phy reset 141.142968 main [1965] Ready... 141.143439 nm_op141.143596 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512 en [456] overrid141.145487 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512 ing ifname vtnet0 ringid 0x0 flags 0x1 141.147778 receiver_body [1219] reading from netmap:vtnet0 fd 4 main_fd 3 142.165927 main_thread [1483] 616128 pps (627330 pkts in 1018181 usec) 143.195891 main_thread [1483] 617845 pps (636357 pkts in 1029963 usec) 144.226279 main_thread [1483] 640422 pps (659884 pkts in 1030389 usec) 145.234087 main_thread [1483] 642099 pps (647113 pkts in 1007808 usec) 146.244540 main_thread [1483] 642227 pps (648839 pkts in 1010296 usec) 147.286503 main_thread [1483] 642392 pps (669410 pkts in 1042058 usec) 148.301086 main_thread [1483] 616898 pps (625932 pkts in 1014645 usec) 149.314164 main_thread [1483] 643629 pps (652046 pkts in 1013078 usec) 150.324461 main_thread [1483] 644681 pps (651319 pkts in 1010297 usec) 151.341187 main_thread [1483] 615727 pps (626025 pkts in 1016725 usec) 152.350815 main_thread [1483] 617538 pps (623484 pkts in 1009629 usec) 153.382287 main_thread [1483] 641915 pps (662117 pkts in 1031472 usec) 154.384370 main_thread [1483] 642324 pps (643662 pkts in 1002083 usec) 155.395499 main_thread [1483] 590924 pps (597500 pkts in 1011129 usec) 156.412232 main_thread [1483] 616879 pps (627201 pkts in 1016733 usec) 157.415682 main_thread [1483] 627524 pps (629689 pkts in 1003450 usec) 158.453857 main_thread [1483] 617421 pps (640991 pkts in 1038175 usec) 159.465732 main_thread [1483] 617130 pps (624458 pkts in 1011875 usec) 160.478797 main_thread [1483] 643278 pps (651682 pkts in 1013065 usec) 161.494513 main_thread [1483] 642237 pps (652330 pkts in 1015716 usec) 162.504104 main_thread [1483] 642578 pps (648740 pkts in 1009589 usec) 163.526906 main_thread [1483] 616442 pps (630460 pkts in 1022741 usec) 164.542778 main_thread [1483] 644121 pps (654385 pkts in 1015935 usec) ^C165.214479 sigint_h [326] received control-C on thread 0x801806800 165.580379 main_thread [1483] 415359 pps (430977 pkts in 1037601 usec) Received 15161931 packets, in 24.07 seconds. Speed: 630.00 Kpps 165.581943 [ 79] vtnet_netmap_free_bufs freed 0 mbufs, 488 netmap bufs on 1 queues Tiwei Bie