Date: Mon, 14 Mar 2016 13:52:24 +0100 From: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= <olivier@cochard.me> To: "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Limited number of vale switch useable with bhyve ? Message-ID: <CA%2Bq%2BTcqQ=ZBqyhyzUBVMxwMaOfp1YJk48M-%2BxrxcFz_7jB0qNw@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi, Thanks to btw's patch I can now use bhyve and vale. But I'm hitting a limitation: I want to start 5 VM connected by a full-meshed network. This mean: - VM 1 has 4 interfaces toward each other 4 VMs - and so one for all others 4 VMs This give these 5 command lines: bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc \ -s 2:0,virtio-net,vale12:BSDRP_1,mac=58:9c:fc:01:02:01 \ -s 2:1,virtio-net,vale13:BSDRP_1,mac=58:9c:fc:01:03:01 \ -s 2:2,virtio-net,vale14:BSDRP_1,mac=58:9c:fc:01:04:01 \ -s 2:3,virtio-net,vale15:BSDRP_1,mac=58:9c:fc:01:05:01 \ -s 1:0,virtio-blk,/tmp/BSDRP/BSDRP_1 \ -l com1,/dev/nmdm1A BSDRP_1 bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc \ -s 2:0,virtio-net,vale15:BSDRP_5,mac=58:9c:fc:01:05:05 \ -s 2:1,virtio-net,vale25:BSDRP_5,mac=58:9c:fc:02:05:05 \ -s 2:2,virtio-net,vale35:BSDRP_5,mac=58:9c:fc:03:05:05 \ -s 2:3,virtio-net,vale45:BSDRP_5,mac=58:9c:fc:04:05:05 \ -s 1:0,virtio-blk,/tmp/BSDRP/BSDRP_5 \ -l com1,/dev/nmdm5A BSDRP_5 bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc \ -s 2:0,virtio-net,vale12:BSDRP_2,mac=58:9c:fc:01:02:02 \ -s 2:1,virtio-net,vale23:BSDRP_2,mac=58:9c:fc:02:03:02 \ -s 2:2,virtio-net,vale24:BSDRP_2,mac=58:9c:fc:02:04:02 \ -s 2:3,virtio-net,vale25:BSDRP_2,mac=58:9c:fc:02:05:02 \ -s 1:0,virtio-blk,/tmp/BSDRP/BSDRP_2 \ -l com1,/dev/nmdm2A BSDRP_2 bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc -s 2:0,virtio-net,vale13:BSDRP_3,mac=58:9c:fc:01:03:03 \ -s 2:1,virtio-net,vale23:BSDRP_3,mac=58:9c:fc:02:03:03 \ -s 2:2,virtio-net,vale34:BSDRP_3,mac=58:9c:fc:03:04:03 \ -s 2:3,virtio-net,vale35:BSDRP_3,mac=58:9c:fc:03:05:03 \ -s 1:0,virtio-blk,/tmp/BSDRP/BSDRP_3 \ -l com1,/dev/nmdm3A BSDRP_3 bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc -s 2:0,virtio-net,vale14:BSDRP_4,mac=58:9c:fc:01:04:04 \ -s 2:1,virtio-net,vale24:BSDRP_4,mac=58:9c:fc:02:04:04 \ -s 2:2,virtio-net,vale34:BSDRP_4,mac=58:9c:fc:03:04:04 \ -s 2:3,virtio-net,vale45:BSDRP_4,mac=58:9c:fc:04:05:04 \ -s 1:0,virtio-blk,/tmp/BSDRP/BSDRP_4 -l com1,/dev/nmdm4A BSDRP_4 But host system complains it didn't reach to allocate memory,: 743.527464 nm_open [608] NIOCREGIF failed: Cannot allocate memory vale35:BSDRP_5 743.527470 nm_open [608] NIOCREGIF failed: Cannot allocate memory vale45:BSDRP_4 open of netmap device vale35:BSDRP_5 failed open of netmap device vale45:BSDRP_4 failed 743.561324 nm_open [608] NIOCREGIF failed: Cannot allocate memory vale45:BSDRP_5 open of netmap device vale45:BSDRP_5 failed 743.561359 nm_open [608] NIOCREGIF failed: Cannot allocate memory vale35:BSDRP_3 open of netmap device vale35:BSDRP_3 failed And the vtnet interfaces that failed to allocate memory are correctly in "status: no carrier mode" in the guests. But the host system has plenty of free RAM: root@lame5:~ # top -b last pid: 2242; load averages: 0.09, 0.24, 0.22 up 0+04:24:44 13:23:20 64 processes: 1 running, 63 sleeping Mem: 556M Active, 94M Inact, 5645M Wired, 243G Free ARC: 1273M Total, 461M MFU, 805M MRU, 16K Anon, 3117K Header, 3332K Other Swap: 4096M Total, 4096M Free Can I increase some sysctl dev.netmap for allowing more memory to be used by netmap/vale ? I've tried to multiply per 10 dev.netmap.buf_num, but no changes. Thanks
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2Bq%2BTcqQ=ZBqyhyzUBVMxwMaOfp1YJk48M-%2BxrxcFz_7jB0qNw>