Date: Mon, 26 Jan 2015 15:00:44 +0800 From: Tiwei Bie <btw@mail.ustc.edu.cn> To: Peter Grehan <grehan@freebsd.org> Cc: freebsd-virtualization@freebsd.org Subject: Re: [PATCH] Add netmap support to bhyve Message-ID: <20150126070044.GA54990@freebsd> In-Reply-To: <54C5B59B.2070100@freebsd.org> References: <1422045440-9410-1-git-send-email-btw@mail.ustc.edu.cn> <54C5B59B.2070100@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Jan 25, 2015 at 07:33:47PM -0800, Peter Grehan wrote:
> Hi,
>
> > I saw that there is an open task of adding Netmap support to bhyve in
> > the latest status report of FreeBSD [1]. So I implement the netmap
> > support for bhyve, and following is the patch. Hope this can be helpful!
>
> Nice work ! Did you manage to get any throughput numbers ?
Thanks! Yeah, I have done some throughput tests. Here are the details:
#1. The parameters I used to launch the VMs:
$ sudo /home/btw/src/usr.sbin/bhyve/bhyve \
-A -H -P \
-s 0:0,hostbridge \
-s 1:0,lpc \
-s 2,virtio-blk,./disk-vm1 \
-s 3:0,virtio-net,vale0:vm1 \
-s 3:1,virtio-net,tap1 \
-l com1,stdio \
-m 2G -c 2 vm1
$ sudo /home/btw/src/usr.sbin/bhyve/bhyve \
-A -H -P \
-s 0:0,hostbridge \
-s 1:0,lpc \
-s 2,virtio-blk,./disk-vm2 \
-s 3:0,virtio-net,vale0:vm2 \
-s 3:1,virtio-net,tap2 \
-l com1,stdio \
-m 2G -c 1 vm2
#2. The outputs of pkt-gen as the receiver when using tap+bridge:
$ ./pkt-gen -i vtnet1 -f rx
075.499873 main [1686] interface is vtnet1
075.500516 extract_ip_range [289] range is 10.0.0.1:0 to 10.0.0.1:0
075.500677 extract_ip_range075.501408 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512
[289] range is 075.503540 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512
10.1.0.1:0 to 10.1.0.1:0
075.505840 [ 79] vtnet_netmap_free_bufs freed 1024 mbufs, 0 netmap bufs on 1 queues
075.507838 main [1877] mapped 334980KB at 0x801dff000
Receiving from netmap:vtnet1: 1 queues, 1 threads and 1 cpus.
075.509298 main [1963] Wait 2 secs for phy reset
077.553231 main [1965] Ready...
077.553787 nm_open [456] overriding ifname vtnet1 ringid 0x0 flags 0x1
077.554511 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512
077.556006 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512
077.557663 receiver_body [1219] reading from netmap:vtnet1 fd 4 main_fd 3
078.563897 main_thread [1483] 311989 pps (313942 pkts in 1006261 usec)
079.574224 main_thread [1483] 333345 pps (336771 pkts in 1010278 usec)
080.584718 main_thread [1483] 307455 pps (310688 pkts in 1010514 usec)
081.600552 main_thread [1483] 332357 pps (337629 pkts in 1015862 usec)
082.604157 main_thread [1483] 342985 pps (344222 pkts in 1003606 usec)
083.614341 main_thread [1483] 327806 pps (331144 pkts in 1010184 usec)
084.634725 main_thread [1483] 320947 pps (327489 pkts in 1020385 usec)
085.645380 main_thread [1483] 328959 pps (332456 pkts in 1010630 usec)
086.654096 main_thread [1483] 313482 pps (316222 pkts in 1008740 usec)
087.671256 main_thread [1483] 307234 pps (312506 pkts in 1017161 usec)
088.673871 main_thread [1483] 311188 pps (312002 pkts in 1002615 usec)
089.686453 main_thread [1483] 322132 pps (326185 pkts in 1012582 usec)
090.694224 main_thread [1483] 331368 pps (333922 pkts in 1007708 usec)
091.714343 main_thread [1483] 290061 pps (295915 pkts in 1020181 usec)
092.724065 main_thread [1483] 316366 pps (319442 pkts in 1009722 usec)
093.749938 main_thread [1483] 322501 pps (330845 pkts in 1025874 usec)
094.758012 main_thread [1483] 292125 pps (294484 pkts in 1008074 usec)
095.765606 main_thread [1483] 338889 pps (341463 pkts in 1007594 usec)
096.781560 main_thread [1483] 327785 pps (333013 pkts in 1015949 usec)
097.789245 main_thread [1483] 330064 pps (332602 pkts in 1007690 usec)
098.794141 main_thread [1483] 328757 pps (330367 pkts in 1004896 usec)
099.804160 main_thread [1483] 302752 pps (305785 pkts in 1010019 usec)
^C100.400690 sigint_h [326] received control-C on thread 0x801806800
100.820895 main_thread [1483] 178717 pps (181708 pkts in 1016735 usec)
Received 7300802 packets, in 22.84 seconds.
Speed: 319.61 Kpps
100.822469 [ 79] vtnet_netmap_free_bufs freed 0 mbufs, 506 netmap bufs on 1 queues
#3. The outputs of pkt-gen as the receiver when using netmap+vale:
$ ./pkt-gen -i vtnet0 -f rx
139.084350 main [1686] interface is vtnet0
139.085422 extract_ip_range [289] range is 10.0.0.1:0 to 10.0.0.139.086441 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512
1:0
139.085705 139.090250 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512
extract_ip_range [289] range is 10.1.0.1:0 to 10.1.0.1:0
139.094911 [ 79] vtnet_netmap_free_bufs freed 1024 mbufs, 0 netmap bufs on 1 queues
139.098783 main [1877] mapped 334980KB at 0x801dff000
Receiving from netmap:vtnet0: 1 queues, 1 threads and 1 cpus.
139.099642 main [1963] Wait 2 secs for phy reset
141.142968 main [1965] Ready...
141.143439 nm_op141.143596 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512
en [456] overrid141.145487 [ 407] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512
ing ifname vtnet0 ringid 0x0 flags 0x1
141.147778 receiver_body [1219] reading from netmap:vtnet0 fd 4 main_fd 3
142.165927 main_thread [1483] 616128 pps (627330 pkts in 1018181 usec)
143.195891 main_thread [1483] 617845 pps (636357 pkts in 1029963 usec)
144.226279 main_thread [1483] 640422 pps (659884 pkts in 1030389 usec)
145.234087 main_thread [1483] 642099 pps (647113 pkts in 1007808 usec)
146.244540 main_thread [1483] 642227 pps (648839 pkts in 1010296 usec)
147.286503 main_thread [1483] 642392 pps (669410 pkts in 1042058 usec)
148.301086 main_thread [1483] 616898 pps (625932 pkts in 1014645 usec)
149.314164 main_thread [1483] 643629 pps (652046 pkts in 1013078 usec)
150.324461 main_thread [1483] 644681 pps (651319 pkts in 1010297 usec)
151.341187 main_thread [1483] 615727 pps (626025 pkts in 1016725 usec)
152.350815 main_thread [1483] 617538 pps (623484 pkts in 1009629 usec)
153.382287 main_thread [1483] 641915 pps (662117 pkts in 1031472 usec)
154.384370 main_thread [1483] 642324 pps (643662 pkts in 1002083 usec)
155.395499 main_thread [1483] 590924 pps (597500 pkts in 1011129 usec)
156.412232 main_thread [1483] 616879 pps (627201 pkts in 1016733 usec)
157.415682 main_thread [1483] 627524 pps (629689 pkts in 1003450 usec)
158.453857 main_thread [1483] 617421 pps (640991 pkts in 1038175 usec)
159.465732 main_thread [1483] 617130 pps (624458 pkts in 1011875 usec)
160.478797 main_thread [1483] 643278 pps (651682 pkts in 1013065 usec)
161.494513 main_thread [1483] 642237 pps (652330 pkts in 1015716 usec)
162.504104 main_thread [1483] 642578 pps (648740 pkts in 1009589 usec)
163.526906 main_thread [1483] 616442 pps (630460 pkts in 1022741 usec)
164.542778 main_thread [1483] 644121 pps (654385 pkts in 1015935 usec)
^C165.214479 sigint_h [326] received control-C on thread 0x801806800
165.580379 main_thread [1483] 415359 pps (430977 pkts in 1037601 usec)
Received 15161931 packets, in 24.07 seconds.
Speed: 630.00 Kpps
165.581943 [ 79] vtnet_netmap_free_bufs freed 0 mbufs, 488 netmap bufs on 1 queues
Tiwei Bie
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20150126070044.GA54990>