Date: Fri, 12 Apr 2019 07:39:28 -0500 From: Jason Bacon <bacon4000@gmail.com> To: Hans Petter Selasky <hps@selasky.org>, "freebsd-infiniband@freebsd.org" <freebsd-infiniband@freebsd.org> Subject: Re: Kernel modules Message-ID: <d82f3a60-6ad4-dba8-a15b-355a536a9a83@gmail.com> In-Reply-To: <6673df26-8bba-ebd3-b2c5-d7e9c97db557@gmail.com> References: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> <f3f94452-155f-79f4-72d8-bf65760ae5b0@selasky.org> <598a58f0-89b8-d00d-5ed7-74dd7005950f@gmail.com> <73ce0738-4d63-2f25-2ff6-00f0092de136@selasky.org> <2090dd24-db43-b689-4289-f50bd70090ea@gmail.com> <6673df26-8bba-ebd3-b2c5-d7e9c97db557@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2019-03-18 10:54, Jason Bacon wrote: > On 3/18/19 10:08 AM, Jason Bacon wrote: >> On 3/18/19 9:10 AM, Hans Petter Selasky wrote: >>> On 3/18/19 2:56 PM, Jason Bacon wrote: >>>> On 3/18/19 7:55 AM, Hans Petter Selasky wrote: >>>>> On 3/18/19 1:32 PM, Jason Bacon wrote: >>>>>> >>>>>> Anyone using the kernel modules lately? >>>>>> >>>>>> After building per the instructions on the wiki >>>>>> (https://wiki.freebsd.org/InfiniBand): >>>>>> >>>>>> <<<ROOT@unixdev.ceas>>> /home/bacon 1005 # kldload ibcore >>>>>> kldload: an error occurred while loading module ibcore. Please >>>>>> check dmesg(8) for more details. >>>>>> >>>>>> dmesg: >>>>>> >>>>>> link_elf_obj: symbol if_index undefined >>>>>> linker_load_file: /boot/modules/ibcore.ko - unsupported file type >>>>> >>>>> Hi, >>>>> >>>>> You built the KMODs out of the kernel? Try adding >>>>> DEBUG_FLAGS="-DVIMAGE=1" to your out of tree build. >>>>> >>>>> --HPS >>>> >>>> I used the script for building kernel modules (which I wrote and >>>> posted at https://wiki.freebsd.org/InfiniBand a few years ago, >>>> based on advice from a Mellanox developer at the time). This will >>>> show exactly what I did. >>>> >>> >>> Can you try: >>> >>> make DEBUG_FLAGS="-DVIMAGE=1" >>> >>> instead of >>> >>> make >>> >>> in the script. In FreeBSD-12 VIMAGE was turned on by default unlike >>> FreeBSD-11. >>> >>> --HPS >>> >> >> That worked, thanks. I'll update the wiki after some testing. >> >> ... >> objcopy --only-keep-debug ipoib.ko.full ipoib.ko.debug >> objcopy --strip-debug --add-gnu-debuglink=ipoib.ko.debug >> ipoib.ko.full ipoib.ko >> install -T release -o root -g wheel -m 555 ipoib.ko /boot/modules/ >> kldxref /boot/modules >> Id Refs Address Size Name >> 1 72 0xffffffff80200000 243cd00 kernel >> 2 1 0xffffffff8263e000 8b50 ng_ubt.ko >> 3 2 0xffffffff82647000 12dd0 ng_hci.ko >> 4 2 0xffffffff8265a000 2d00 ng_bluetooth.ko >> 5 3 0xffffffff8265d000 175d8 netgraph.ko >> 7 5 0xffffffff82688000 279b0 linuxkpi.ko >> 9 1 0xffffffff8274d000 3a9a10 zfs.ko >> 10 2 0xffffffff82af7000 a4f0 opensolaris.ko >> 11 1 0xffffffff82b02000 20af0 firewire.ko >> 12 1 0xffffffff82b23000 bf70 cuse.ko >> 13 2 0xffffffff82b2f000 665e0 mlx4.ko >> 15 1 0xffffffff82bd0000 5fb8 uplcom.ko >> 16 2 0xffffffff82bd6000 ab88 ucom.ko >> 17 1 0xffffffff82e21000 1a20 fdescfs.ko >> 18 1 0xffffffff82e23000 490c linprocfs.ko >> 19 3 0xffffffff82e28000 2e28 linux_common.ko >> 20 1 0xffffffff82e2b000 81f0 tmpfs.ko >> 21 1 0xffffffff82e34000 1800 uhid.ko >> 22 1 0xffffffff82e36000 23a8 ums.ko >> 23 1 0xffffffff82e39000 39960 linux.ko >> 24 1 0xffffffff82e73000 33c50 linux64.ko >> 25 1 0xffffffff82ea7000 acf mac_ntpd.ko >> 26 1 0xffffffff82ea8000 27b0 nullfs.ko >> 33 3 0xffffffff82eab000 4f2d8 ibcore.ko >> 34 1 0xffffffff82efb000 1fd39 mlx4ib.ko >> 35 1 0xffffffff82f1b000 85a0 ipoib.ko >> > > So ib0 came up immediately after configuring in rc.conf: > > ifconfig_ib0="inet 192.168.129.17 netmask 255.255.128.0" > > Next issue is poor performance shown by iperf: > > root@zfs-01:~ # iperf -c raid-01-hpc > ------------------------------------------------------------ > Client connecting to raid-01-hpc, TCP port 5001 > TCP window size: 33.3 KByte (default) > ------------------------------------------------------------ > [ 3] local 192.168.129.17 port 28021 connected with 192.168.129.1 > port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.1 sec 127 MBytes 106 Mbits/sec > > raid-01 is a CentOS 7 server. > > Performance between two CentOS servers is more reasonable: > > [root@raid-02.mortimer ~] 133: iperf -c raid-01-hpc > ------------------------------------------------------------ > Client connecting to raid-01-hpc, TCP port 5001 > TCP window size: 2.50 MByte (default) > ------------------------------------------------------------ > [ 3] local 192.168.129.2 port 57952 connected with 192.168.129.1 port > 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 24.1 GBytes 20.7 Gbits/sec > > Ultimately I aim to use NFS over IB here, but I think iperf is telling > me that ipoib needs to be tuned. > > Might this be connected vs datagram issue? If so, how to change it > when using loadable modules? The wiki only mentions building IPOIB_CM > into the kernel. > > If we can figure this out, I'll fully document everything on the wiki, > of course. > > Thanks, > > JB > Finally found a moment to look at this again. I added CFLAGS+=-DIPOIB_CM to /etc/make.conf and rebuilt the modules. This seems to have worked, because ifconfig now allows me to set an MTU over 4096, which was not possible without IPOIB_CM. I'm still seeing abysmal performance with iperf, though: [root@raid-01.mortimer bacon] 221: ip link show ib0 6: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast state UP mode DEFAULT group default qlen 256 link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:f4:52:14:03:00:98:37:81 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff [root@raid-01.mortimer bacon] 222: iperf -s -B raid-01-hpc ------------------------------------------------------------ Server listening on TCP port 5001 Binding to local address raid-01-hpc TCP window size: 85.3 KByte (default) ------------------------------------------------------------ root@zfs-01:~ # ifconfig ib0 ib0: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520 options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE> lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.f4.52.14.3.0.92.88.d1 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> root@zfs-01:~ # iperf -c raid-01-hpc ------------------------------------------------------------ Client connecting to raid-01-hpc, TCP port 5001 TCP window size: 40.8 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.17 port 39939 connected with 192.168.129.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-15.1 sec 106 KBytes 57.5 Kbits/sec Any suggestions? Thanks, JB -- Earth is a beta site.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?d82f3a60-6ad4-dba8-a15b-355a536a9a83>