Date: Thu, 28 May 2015 00:47:20 -0600 From: Blake Caldwell <caldweba@colorado.edu> To: Luigi Rizzo <rizzo@iet.unipi.it>, Hans Petter Selasky <hps@selasky.org>, Oded Shanoon <odeds@mellanox.com> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: netmap and mlx4 driver status (linux) Message-ID: <C5A3C8CB-C3FE-46B7-A1BF-2C48D978358B@colorado.edu> In-Reply-To: <CA%2BhQ2%2BjM9bvSQ8rjB=8ikZ-DtuVqYzj84MAGRTd75UJX0_Ur0g@mail.gmail.com> References: <3010CFE2-66B7-416B-92DE-C1B669CC33BE@colorado.edu> <555C9F30.8070405@selasky.org> <CA%2BhQ2%2BjM9bvSQ8rjB=8ikZ-DtuVqYzj84MAGRTd75UJX0_Ur0g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello, I have made necessary tweaks to the mlx4 patches for a successful build = on Linux 3.13.11 (Ubuntu 14.04) and enabled the driver in the Linux = build system. See: https://github.com/caldweba/netmap.git = <https://github.com/caldweba/netmap.git>; for my additional commits. Without any core modifications to the mlx4 netmap driver, I am actually = getting reduced performance, 2.5 Mpps on a 40G port. I=92m interested in = improving the performance of this driver, but as I=92m new to netmap and = even these drivers, some assistance would be welcome. As Luigi = mentioned, the Mellanox developer documentation seems to be a stumbling = point. Would anyone from Mellanox be able to lend some expertise? It would appear mlx4_netmap_txsync() is the place to focus optimization, = and the comments Luigi put in will be helpful. Although I=92m a little = confused the on the remaining work for mlx4_netmap_tx_config (marked = TODO). See = https://github.com/caldweba/netmap/blob/master/LINUX/mlx4_netmap_linux.h = <https://github.com/caldweba/netmap/blob/master/LINUX/mlx4_netmap_linux.h>= for Luigi=92s current mlx4_netmap_txsync() code. Below is my output from pkt-gen and from ethtool on the device. Best regards, Blake =97=97=97=97=97 $ sudo build-apps/pkt-gen -i p2p1 -f tx -n 500111222 -l 60 -w 5 060.428278 main [1649] interface is p2p1 060.428770 extract_ip_range [287] range is 10.0.0.1:0 to 10.0.0.1:0 060.428782 extract_ip_range [287] range is 10.1.0.1:0 to 10.1.0.1:0 060.875064 main [1840] mapped 334980KB at 0x7fd1f04d5000 Sending on netmap:p2p1: 8 queues, 1 threads and 1 cpus. 10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff) 060.875151 main [1924] Sending 512 packets every 0.000000000 s 060.875158 main [1926] Wait 5 secs for phy reset 065.875244 main [1928] Ready... 065.875276 nm_open [456] overriding ifname p2p1 ringid 0x0 flags 0x1 065.914805 sender_body [1014] start, fd 4 main_fd 3 065.958284 sender_body [1083] drop copy 066.915788 main_thread [1446] 2468560 pps (2471088 pkts in 1001024 usec) 067.916827 main_thread [1446] 2476292 pps (2478865 pkts in 1001039 usec) 068.917815 main_thread [1446] 2476261 pps (2478708 pkts in 1000988 usec) 069.918864 main_thread [1446] 2476232 pps (2478827 pkts in 1001048 usec) 070.919902 main_thread [1446] 2476031 pps (2478604 pkts in 1001039 usec) 071.920920 main_thread [1446] 2476304 pps (2478825 pkts in 1001018 usec) 072.921896 main_thread [1446] 2476349 pps (2478766 pkts in 1000976 usec) 073.922948 main_thread [1446] 2476327 pps (2478932 pkts in 1001052 usec) 074.923924 main_thread [1446] 2476301 pps (2478715 pkts in 1000975 usec) 075.924903 main_thread [1446] 2476257 pps (2478681 pkts in 1000979 usec) 076.925918 main_thread [1446] 2476195 pps (2478708 pkts in 1001015 usec) 077.926970 main_thread [1446] 2476242 pps (2478849 pkts in 1001053 usec) dmesg: [52591.017469] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 = (Feb 2014) [52591.017621] mlx4_en 0000:04:00.0: registered PHC clock [52591.017780] mlx4_en 0000:04:00.0: Activating port:1 [52591.023552] mlx4_en: eth0: Using 192 TX rings [52591.023554] mlx4_en: eth0: Using 8 RX rings [52591.023556] mlx4_en: eth0: frag:0 - size:1526 prefix:0 align:0 = stride:1536 [52591.040585] mlx4_en: eth0: Initializing port [52591.040732] 779.121252 [2720] netmap_attach success for = eth0 tx 8/512 rx 8/1024 queues/slots [52591.060580] mlx4_en 0000:04:00.0: Activating port:2 [52591.068337] mlx4_en: eth1: Using 192 TX rings [52591.068340] mlx4_en: eth1: Using 8 RX rings [52591.068342] mlx4_en: eth1: frag:0 - size:1526 prefix:0 align:0 = stride:1536 [52591.085696] mlx4_en: eth1: Initializing port [52591.085867] 779.166352 [2720] netmap_attach success for = eth1 tx 8/512 rx 8/1024 queues/slots [52591.960730] mlx4_en: eth0: Link Up [52593.029536] systemd-udevd[50993]: renamed network interface eth0 to = p2p1 [52593.061736] systemd-udevd[50996]: renamed network interface eth1 to = rename28 [52624.680481] mlx4_en: p2p1: frag:0 - size:1526 prefix:0 align:0 = stride:1536 [52624.834109] 812.888289 [ 473] mlx4_en_tx_irq XXXXXXXXX = tx_irq 0 unexpected, ignoring [55436.322304] 622.179577 [ 665] mlx4_netmap_config using only 8 = out of 192 tx queues [55436.339688] 622.196947 [ 672] mlx4_netmap_config txr 8 txd 512 = bufsize 32768 -- rxr 8 rxd 1024 act 1024 bufsize 16384 [55436.361877] 622.219119 [ 124] mlx4_netmap_reg setting = netmap mode for eth0 to ON [55436.379345] 622.236575 [ 127] mlx4_netmap_reg unloading = eth0 [55436.485781] 622.342926 [ 163] mlx4_netmap_reg loading eth0 [55436.501124] mlx4_en: p2p1: frag:0 - size:1526 prefix:0 align:0 = stride:1536 [55436.517514] 622.374635 [ 628] mlx4_netmap_rx_config stride 16 = possible frags 1 descsize 0 DS_SIZE 16 [55436.536462] 622.393570 [ 648] mlx4_netmap_rx_config ring 0 done [55436.551746] 622.408842 [ 648] mlx4_netmap_rx_config ring 1 done [55436.567111] 622.424194 [ 628] mlx4_netmap_rx_config stride 16 = possible frags 1 descsize 0 DS_SIZE 16 [55436.586261] 622.443330 [ 648] mlx4_netmap_rx_config ring 2 done [55436.601844] 622.458900 [ 648] mlx4_netmap_rx_config ring 3 done [55436.617525] 622.474569 [ 648] mlx4_netmap_rx_config ring 4 done [55436.633057] 622.490089 [ 648] mlx4_netmap_rx_config ring 5 done [55436.648376] 622.505396 [ 648] mlx4_netmap_rx_config ring 6 done [55436.780501] 622.637414 [ 165] mlx4_netmap_reg start_port = returns 0 [55436.796403] mlx4_en: p2p1: Link Down [55437.755281] mlx4_en: p2p1: Link Up $ ethtool p2p1 Settings for p2p1: Supported ports: [ TP ] Supported link modes: 10000baseT/Full=20 Supported pause frame use: No Supports auto-negotiation: No Advertised link modes: 10000baseT/Full=20 Advertised pause frame use: No Advertised auto-negotiation: No Speed: 40000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 0 Transceiver: internal Auto-negotiation: off MDI-X: Unknown Cannot get wake-on-lan settings: Operation not permitted Current message level: 0x00000014 (20) link ifdown Link detected: yes $ ethtool -i p2p1 driver: mlx4_en version: 2.2-1 (Feb 2014) firmware-version: 2.30.3200 bus-info: 0000:04:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: no supports-priv-flags: yes ethtool -g p2p1 Ring parameters for p2p1: Pre-set maximums: RX: 8192 RX Mini: 0 RX Jumbo: 0 TX: 8192 Current hardware settings: RX: 1024 RX Mini: 0 RX Jumbo: 0 TX: 512 Coalesce parameters for p2p1: Adaptive RX: on TX: off stats-block-usecs: 0 sample-interval: 0 pkt-rate-low: 400000 pkt-rate-high: 450000 rx-usecs: 16 rx-frames: 44 rx-usecs-irq: 0 rx-frames-irq: 0 tx-usecs: 16 tx-frames: 16 tx-usecs-irq: 0 tx-frames-irq: 0 rx-usecs-low: 0 rx-frame-low: 0 tx-usecs-low: 0 tx-frame-low: 0 rx-usecs-high: 128 rx-frame-high: 0 tx-usecs-high: 0 tx-frame-high: 0 $ sudo ethtool -k p2p1 Features for p2p1: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: on tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: on tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: on udp-fragmentation-offload: off [fixed] generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off [fixed] receive-hashing: on highdma: on [fixed] rx-vlan-filter: on [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] tx-mpls-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: on loopback: off rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] > On May 20, 2015, at 9:18 AM, Luigi Rizzo <rizzo@iet.unipi.it> wrote: >=20 > hi all, >=20 > the mlx4 netmap patch (for linux only) was something i did long > ago when i had some mellanox hardware available, but no documentation > so i had to resort to interpreting what the linux driver did. >=20 > At the time i had the following performance (on PCIe v2 bus): >=20 > 10G ports: tx/rx at about 7 Mpps with 64 byte packets > could saturate the link with 192 or 256 byte packets >=20 > 40G ports: tx/rx at about 11 Mpps with 64 byte packets > max 28 Gbit/s even with 1500 byte frames >=20 > I don't know if the limited performance was due to bus, > firmware or lack of documentation, anyways this is not > something i can or want to deal with. >=20 > My understanding is that Mellanox does not release programming > documentation, so the only way to have native netmap support > for that card would be to have Mellanox work on that and > provide a suitable patch. >=20 > I do not expect more than a week's work (the typical extra > code in each driver is about 500 lines, and very simple) > for someone with access to documentation. Also, the patch > for FreeBSD and Linux is typically very similar so once we > have a driver for one, the other would be trivial. >=20 > It would be of course great to add Mellanox to the list of > devices with native netmap support, together with Chelsio > and Intel. >=20 > Perhaps Hans (who may have contacts) can talk to the right > people and figure out. On my side, I am happy to give directions > on what needs to be done and import any patch that should > be made available. >=20 > cheers > luigi >=20 > On Wed, May 20, 2015 at 4:50 PM, Hans Petter Selasky <hps@selasky.org = <mailto:hps@selasky.org>> wrote: > On 05/20/15 16:13, Blake Caldwell wrote: > Hello, >=20 > I noticed that the mlx4_en patch for netmap is LINUX/wip-patches, so = they are not enabled in the normal build process. I=92m curious about = the status of mlx4 support? >=20 > If additional work to the patches is needed, any details as to what = the issues were. >=20 > Any info would be great! Thanks in advance! >=20 >=20 > Hi Blake, >=20 > The MLX4 driver is being actively maintained in -stable and -current. = Regarding netmap support for the FreeBSD MLX4 en driver, I'm not sure. = Maybe Oded knows, CC'ed? Do you have a link for the patch you are = referring? >=20 > This there any particular use-case you are interested in? >=20 > --HPS >=20 >=20 > _______________________________________________ > freebsd-net@freebsd.org <mailto:freebsd-net@freebsd.org> mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net = <http://lists.freebsd.org/mailman/listinfo/freebsd-net>; > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org = <mailto:freebsd-net-unsubscribe@freebsd.org>" >=20 >=20 >=20 > --=20 > = -----------------------------------------+------------------------------- > Prof. Luigi RIZZO, rizzo@iet.unipi.it <mailto:rizzo@iet.unipi.it> . = Dip. di Ing. dell'Informazione > http://www.iet.unipi.it/~luigi/ <http://www.iet.unipi.it/~luigi/>; = . Universita` di Pisa > TEL +39-050-2217533 . via Diotisalvi 2 > Mobile +39-338-6809875 . 56122 PISA (Italy) > = -----------------------------------------+-------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C5A3C8CB-C3FE-46B7-A1BF-2C48D978358B>