From owner-freebsd-infiniband@freebsd.org Fri Apr 12 12:39:33 2019 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 211AB157BCCA for ; Fri, 12 Apr 2019 12:39:33 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: from mail-oi1-x243.google.com (mail-oi1-x243.google.com [IPv6:2607:f8b0:4864:20::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3690071E3D for ; Fri, 12 Apr 2019 12:39:32 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: by mail-oi1-x243.google.com with SMTP id j132so7819301oib.2 for ; Fri, 12 Apr 2019 05:39:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:references:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=V4mOSH785n3E3x/JheaYFoILYk79esC40XC9uaDy01w=; b=ZWn6DQpihOzly1VctRmRBWnApu6VakrsyAqBHCknyEemuLRyCxymezlz1WfzcVpo77 jeyHlerINyA1QjOChU4y3JtXCi1f/yMsZ9QTGzqIUG/c0qHHdWjP1XCcuPPHASrqYmy9 u1ojm6Vnze9O8EG9JN+V2Cj9l/XkB/NdkE4OL7yk3qYa7JqSUDgVCn/lPjVgHg/NLJz+ yBPQOSbapFyF5O2CpjNqV8/FqPEXxdIcr5YDU1T0MyWOAomadccHiCYtalBGXiNp0zMt Bbf0TDqan7iz1h1WTASkjOheg9Mp/MvIEyz4oWAr+NeOe0A/b9m2VcJRTEiwAVMnyG3n RzPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=V4mOSH785n3E3x/JheaYFoILYk79esC40XC9uaDy01w=; b=jajUP8tFmzpL7Ya7mFCui7aD5bqx7cKCN3KHsXJIFbhRKaBhxlg/3PsPBRW0Q8Lqfm e9+aFuFGEP5HJOQUA50gc4sp7+R7UjXVxlXjXxKArNJhAjdC0KVVsrLcezutw1LMCOEa TB9Yk02aWo661Pg7C4r9yhODvQ0tdsJ0Utd/XIBUdRGMpMczHB3zRWvHFdqNVqD6thH4 KqDNwEY7iaD+qVSpdZI5USwa7ZKHDjGJP4zggEHP3FLD/6N2+nDEwDa5sLwYtPVGL9zW 6wmBX68G3dMRd6ph1zi56FaGjJWYFKziywPQKpzSBBqd4hUe1+3W3pHa6J9bXt3eAZ1f kdBQ== X-Gm-Message-State: APjAAAX5HVvh+8rruVh8iDJO31coB/KPDCz+mYZix8lXpDne/Iuj9vgB cvSHeIX2PACpjXRwmXMbU8Cf+ScX X-Google-Smtp-Source: APXvYqwBbgm8x0DvtHF0/2cRI5XDmlkH2CFed6hBK3Go5z5CdHb+XVITOLJnVh5Wr0eBKyxl3RQGgQ== X-Received: by 2002:aca:3e83:: with SMTP id l125mr9286175oia.146.1555072770705; Fri, 12 Apr 2019 05:39:30 -0700 (PDT) Received: from manatee.acadix.biz (cpe-174-102-163-140.wi.res.rr.com. [174.102.163.140]) by smtp.gmail.com with ESMTPSA id p205sm15679533oia.49.2019.04.12.05.39.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 Apr 2019 05:39:29 -0700 (PDT) Subject: Re: Kernel modules From: Jason Bacon To: Hans Petter Selasky , "freebsd-infiniband@freebsd.org" References: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> <598a58f0-89b8-d00d-5ed7-74dd7005950f@gmail.com> <73ce0738-4d63-2f25-2ff6-00f0092de136@selasky.org> <2090dd24-db43-b689-4289-f50bd70090ea@gmail.com> <6673df26-8bba-ebd3-b2c5-d7e9c97db557@gmail.com> Message-ID: Date: Fri, 12 Apr 2019 07:39:28 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <6673df26-8bba-ebd3-b2c5-d7e9c97db557@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 3690071E3D X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=ZWn6DQpi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of bacon4000@gmail.com designates 2607:f8b0:4864:20::243 as permitted sender) smtp.mailfrom=bacon4000@gmail.com X-Spamd-Result: default: False [-4.63 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.98)[-0.984,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-infiniband@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[3.4.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.64)[ip: (2.01), ipnet: 2607:f8b0::/32(-2.94), asn: 15169(-2.18), country: US(-0.06)] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 12:39:33 -0000 On 2019-03-18 10:54, Jason Bacon wrote: > On 3/18/19 10:08 AM, Jason Bacon wrote: >> On 3/18/19 9:10 AM, Hans Petter Selasky wrote: >>> On 3/18/19 2:56 PM, Jason Bacon wrote: >>>> On 3/18/19 7:55 AM, Hans Petter Selasky wrote: >>>>> On 3/18/19 1:32 PM, Jason Bacon wrote: >>>>>> >>>>>> Anyone using the kernel modules lately? >>>>>> >>>>>> After building per the instructions on the wiki >>>>>> (https://wiki.freebsd.org/InfiniBand): >>>>>> >>>>>> <<>> /home/bacon 1005 # kldload ibcore >>>>>> kldload: an error occurred while loading module ibcore. Please >>>>>> check dmesg(8) for more details. >>>>>> >>>>>> dmesg: >>>>>> >>>>>> link_elf_obj: symbol if_index undefined >>>>>> linker_load_file: /boot/modules/ibcore.ko - unsupported file type >>>>> >>>>> Hi, >>>>> >>>>> You built the KMODs out of the kernel? Try adding >>>>> DEBUG_FLAGS="-DVIMAGE=1" to your out of tree build. >>>>> >>>>> --HPS >>>> >>>> I used the script for building kernel modules (which I wrote and >>>> posted at https://wiki.freebsd.org/InfiniBand a few years ago, >>>> based on advice from a Mellanox developer at the time). This will >>>> show exactly what I did. >>>> >>> >>> Can you try: >>> >>> make DEBUG_FLAGS="-DVIMAGE=1" >>> >>> instead of >>> >>> make >>> >>> in the script. In FreeBSD-12 VIMAGE was turned on by default unlike >>> FreeBSD-11. >>> >>> --HPS >>> >> >> That worked, thanks.  I'll update the wiki after some testing. >> >> ... >> objcopy --only-keep-debug ipoib.ko.full ipoib.ko.debug >> objcopy --strip-debug --add-gnu-debuglink=ipoib.ko.debug >> ipoib.ko.full ipoib.ko >> install -T release -o root -g wheel -m 555   ipoib.ko /boot/modules/ >> kldxref /boot/modules >> Id Refs Address                Size Name >>  1   72 0xffffffff80200000  243cd00 kernel >>  2    1 0xffffffff8263e000     8b50 ng_ubt.ko >>  3    2 0xffffffff82647000    12dd0 ng_hci.ko >>  4    2 0xffffffff8265a000     2d00 ng_bluetooth.ko >>  5    3 0xffffffff8265d000    175d8 netgraph.ko >>  7    5 0xffffffff82688000    279b0 linuxkpi.ko >>  9    1 0xffffffff8274d000   3a9a10 zfs.ko >> 10    2 0xffffffff82af7000     a4f0 opensolaris.ko >> 11    1 0xffffffff82b02000    20af0 firewire.ko >> 12    1 0xffffffff82b23000     bf70 cuse.ko >> 13    2 0xffffffff82b2f000    665e0 mlx4.ko >> 15    1 0xffffffff82bd0000     5fb8 uplcom.ko >> 16    2 0xffffffff82bd6000     ab88 ucom.ko >> 17    1 0xffffffff82e21000     1a20 fdescfs.ko >> 18    1 0xffffffff82e23000     490c linprocfs.ko >> 19    3 0xffffffff82e28000     2e28 linux_common.ko >> 20    1 0xffffffff82e2b000     81f0 tmpfs.ko >> 21    1 0xffffffff82e34000     1800 uhid.ko >> 22    1 0xffffffff82e36000     23a8 ums.ko >> 23    1 0xffffffff82e39000    39960 linux.ko >> 24    1 0xffffffff82e73000    33c50 linux64.ko >> 25    1 0xffffffff82ea7000      acf mac_ntpd.ko >> 26    1 0xffffffff82ea8000     27b0 nullfs.ko >> 33    3 0xffffffff82eab000    4f2d8 ibcore.ko >> 34    1 0xffffffff82efb000    1fd39 mlx4ib.ko >> 35    1 0xffffffff82f1b000     85a0 ipoib.ko >> > > So ib0 came up immediately after configuring in rc.conf: > > ifconfig_ib0="inet 192.168.129.17 netmask 255.255.128.0" > > Next issue is poor performance shown by iperf: > > root@zfs-01:~ # iperf -c raid-01-hpc > ------------------------------------------------------------ > Client connecting to raid-01-hpc, TCP port 5001 > TCP window size: 33.3 KByte (default) > ------------------------------------------------------------ > [  3] local 192.168.129.17 port 28021 connected with 192.168.129.1 > port 5001 > [ ID] Interval       Transfer     Bandwidth > [  3]  0.0-10.1 sec   127 MBytes   106 Mbits/sec > > raid-01 is a CentOS 7 server. > > Performance between two CentOS servers is more reasonable: > > [root@raid-02.mortimer ~] 133: iperf -c raid-01-hpc > ------------------------------------------------------------ > Client connecting to raid-01-hpc, TCP port 5001 > TCP window size: 2.50 MByte (default) > ------------------------------------------------------------ > [  3] local 192.168.129.2 port 57952 connected with 192.168.129.1 port > 5001 > [ ID] Interval       Transfer     Bandwidth > [  3]  0.0-10.0 sec  24.1 GBytes  20.7 Gbits/sec > > Ultimately I aim to use NFS over IB here, but I think iperf is telling > me that ipoib needs to be tuned. > > Might this be connected vs datagram issue?  If so, how to change it > when using loadable modules? The wiki only mentions building IPOIB_CM > into the kernel. > > If we can figure this out, I'll fully document everything on the wiki, > of course. > > Thanks, > >     JB > Finally found a moment to look at this again. I added CFLAGS+=-DIPOIB_CM to /etc/make.conf and rebuilt the modules. This seems to have worked, because ifconfig now allows me to set an MTU over 4096, which was not possible without IPOIB_CM. I'm still seeing abysmal performance with iperf, though: [root@raid-01.mortimer bacon] 221: ip link show ib0 6: ib0: mtu 65520 qdisc pfifo_fast state UP mode DEFAULT group default qlen 256     link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:f4:52:14:03:00:98:37:81 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff [root@raid-01.mortimer bacon] 222: iperf -s -B raid-01-hpc ------------------------------------------------------------ Server listening on TCP port 5001 Binding to local address raid-01-hpc TCP window size: 85.3 KByte (default) ------------------------------------------------------------ root@zfs-01:~ # ifconfig ib0 ib0: flags=8002 metric 0 mtu 65520     options=80018     lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.f4.52.14.3.0.92.88.d1     nd6 options=29 root@zfs-01:~ # iperf -c raid-01-hpc ------------------------------------------------------------ Client connecting to raid-01-hpc, TCP port 5001 TCP window size: 40.8 KByte (default) ------------------------------------------------------------ [  3] local 192.168.1.17 port 39939 connected with 192.168.129.1 port 5001 [ ID] Interval       Transfer     Bandwidth [  3]  0.0-15.1 sec   106 KBytes  57.5 Kbits/sec Any suggestions? Thanks,     JB -- Earth is a beta site. From owner-freebsd-infiniband@freebsd.org Fri Apr 12 12:52:54 2019 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C16E157C207 for ; Fri, 12 Apr 2019 12:52:54 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: from mail-oi1-x241.google.com (mail-oi1-x241.google.com [IPv6:2607:f8b0:4864:20::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 149F272782 for ; Fri, 12 Apr 2019 12:52:53 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: by mail-oi1-x241.google.com with SMTP id i21so7804853oib.11 for ; Fri, 12 Apr 2019 05:52:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:references:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=bXgecLhdZbMO0l+7Fyk0NTjEVad31Pk2auWOnShKzlA=; b=cGoIY7ojE30KL0P4Q1Z5D/Z+hZ7oChvLf2Dopps7e+D2GG934jtt72XcfF4K5oE5Sc TtiP1JtqTl1/WI4CGTOqhJEL/M0JQxJknkw6Fztb2fE5MUxadMW0alZb1kJ1MqY3YZoH apfeNNPtai+C2hirNVBZN3h5+BGHzLQLI6WJ3L84kcJOdV8bzDL6JhIZKEZNigWm4no4 NeHgyuTcqyPvxXmbrcIqr1wDjf8lHpvKv+rBoBZePqx7+VyD1R7WUrgCTeOqgd6FfT9X GUh5bQOiV0DWs30M7e51sTiY4SI0wBI0AaTYKNt9StAtcv4Aj+4Dkaz2WRpx5IlP9Y86 7PjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=bXgecLhdZbMO0l+7Fyk0NTjEVad31Pk2auWOnShKzlA=; b=ViKo/L2dmI8HIoiYkiMz4mMcc909TSjpLSyG8OYv8o7Di9MCevUwD/Zn7JcS7hyPAK FMwvX+AI/04f/zDluIsq/3mBSTxVAMshBOnJ6y68DRLuqpB0XZb9euSCtFKChSifSq7n jdIhe8e3mlEgD+tA12rrPxkh3DuQ90GBHum8Agdc1icNLhgKpM4HlmF11dKL2IpDROOZ nBEgWq7m08/zQuaRw6xqW5oHTHzdKmKEuZKg1aA2QP2Jv6DWQP1QeArnpCdxj0fuz2AQ gBGjfiwi/YinRCGkmhfKS2/YubH75pPMSOurIQPYf+tSS3K7i3aNsyDhB2zVXslXg7hW UotQ== X-Gm-Message-State: APjAAAVNYhpJIES9wi2jzRq084WeN6U5PIGUXu16ZktkmjQfCyUspmge g0z7aUJ+NGtRBjQmBp7mtK8G0N9i X-Google-Smtp-Source: APXvYqyBDvIhxr4vjYc52iMFvhu2J62TtYszSbrXyWkrGcFK2Tub/1y62SoFf0I2r7oimVpyqifIXw== X-Received: by 2002:aca:f515:: with SMTP id t21mr9941078oih.175.1555073571488; Fri, 12 Apr 2019 05:52:51 -0700 (PDT) Received: from manatee.acadix.biz (cpe-174-102-163-140.wi.res.rr.com. [174.102.163.140]) by smtp.gmail.com with ESMTPSA id q124sm10927767oia.13.2019.04.12.05.52.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 Apr 2019 05:52:50 -0700 (PDT) Subject: Re: Kernel modules From: Jason Bacon To: Hans Petter Selasky , "freebsd-infiniband@freebsd.org" References: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> <598a58f0-89b8-d00d-5ed7-74dd7005950f@gmail.com> <73ce0738-4d63-2f25-2ff6-00f0092de136@selasky.org> <2090dd24-db43-b689-4289-f50bd70090ea@gmail.com> <6673df26-8bba-ebd3-b2c5-d7e9c97db557@gmail.com> Message-ID: <31fb116e-d6a8-c2dd-3f75-e35d4ad4b5cd@gmail.com> Date: Fri, 12 Apr 2019 07:52:49 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 149F272782 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=cGoIY7oj; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of bacon4000@gmail.com designates 2607:f8b0:4864:20::241 as permitted sender) smtp.mailfrom=bacon4000@gmail.com X-Spamd-Result: default: False [-4.69 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.99)[-0.987,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-infiniband@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[1.4.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.69)[ip: (1.75), ipnet: 2607:f8b0::/32(-2.94), asn: 15169(-2.18), country: US(-0.06)] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 12:52:54 -0000 On 2019-04-12 07:39, Jason Bacon wrote: > On 2019-03-18 10:54, Jason Bacon wrote: >> On 3/18/19 10:08 AM, Jason Bacon wrote: >>> On 3/18/19 9:10 AM, Hans Petter Selasky wrote: >>>> On 3/18/19 2:56 PM, Jason Bacon wrote: >>>>> On 3/18/19 7:55 AM, Hans Petter Selasky wrote: >>>>>> On 3/18/19 1:32 PM, Jason Bacon wrote: >>>>>>> >>>>>>> Anyone using the kernel modules lately? >>>>>>> >>>>>>> After building per the instructions on the wiki >>>>>>> (https://wiki.freebsd.org/InfiniBand): >>>>>>> >>>>>>> <<>> /home/bacon 1005 # kldload ibcore >>>>>>> kldload: an error occurred while loading module ibcore. Please >>>>>>> check dmesg(8) for more details. >>>>>>> >>>>>>> dmesg: >>>>>>> >>>>>>> link_elf_obj: symbol if_index undefined >>>>>>> linker_load_file: /boot/modules/ibcore.ko - unsupported file type >>>>>> >>>>>> Hi, >>>>>> >>>>>> You built the KMODs out of the kernel? Try adding >>>>>> DEBUG_FLAGS="-DVIMAGE=1" to your out of tree build. >>>>>> >>>>>> --HPS >>>>> >>>>> I used the script for building kernel modules (which I wrote and >>>>> posted at https://wiki.freebsd.org/InfiniBand a few years ago, >>>>> based on advice from a Mellanox developer at the time). This will >>>>> show exactly what I did. >>>>> >>>> >>>> Can you try: >>>> >>>> make DEBUG_FLAGS="-DVIMAGE=1" >>>> >>>> instead of >>>> >>>> make >>>> >>>> in the script. In FreeBSD-12 VIMAGE was turned on by default unlike >>>> FreeBSD-11. >>>> >>>> --HPS >>>> >>> >>> That worked, thanks.  I'll update the wiki after some testing. >>> >>> ... >>> objcopy --only-keep-debug ipoib.ko.full ipoib.ko.debug >>> objcopy --strip-debug --add-gnu-debuglink=ipoib.ko.debug >>> ipoib.ko.full ipoib.ko >>> install -T release -o root -g wheel -m 555   ipoib.ko /boot/modules/ >>> kldxref /boot/modules >>> Id Refs Address                Size Name >>>  1   72 0xffffffff80200000  243cd00 kernel >>>  2    1 0xffffffff8263e000     8b50 ng_ubt.ko >>>  3    2 0xffffffff82647000    12dd0 ng_hci.ko >>>  4    2 0xffffffff8265a000     2d00 ng_bluetooth.ko >>>  5    3 0xffffffff8265d000    175d8 netgraph.ko >>>  7    5 0xffffffff82688000    279b0 linuxkpi.ko >>>  9    1 0xffffffff8274d000   3a9a10 zfs.ko >>> 10    2 0xffffffff82af7000     a4f0 opensolaris.ko >>> 11    1 0xffffffff82b02000    20af0 firewire.ko >>> 12    1 0xffffffff82b23000     bf70 cuse.ko >>> 13    2 0xffffffff82b2f000    665e0 mlx4.ko >>> 15    1 0xffffffff82bd0000     5fb8 uplcom.ko >>> 16    2 0xffffffff82bd6000     ab88 ucom.ko >>> 17    1 0xffffffff82e21000     1a20 fdescfs.ko >>> 18    1 0xffffffff82e23000     490c linprocfs.ko >>> 19    3 0xffffffff82e28000     2e28 linux_common.ko >>> 20    1 0xffffffff82e2b000     81f0 tmpfs.ko >>> 21    1 0xffffffff82e34000     1800 uhid.ko >>> 22    1 0xffffffff82e36000     23a8 ums.ko >>> 23    1 0xffffffff82e39000    39960 linux.ko >>> 24    1 0xffffffff82e73000    33c50 linux64.ko >>> 25    1 0xffffffff82ea7000      acf mac_ntpd.ko >>> 26    1 0xffffffff82ea8000     27b0 nullfs.ko >>> 33    3 0xffffffff82eab000    4f2d8 ibcore.ko >>> 34    1 0xffffffff82efb000    1fd39 mlx4ib.ko >>> 35    1 0xffffffff82f1b000     85a0 ipoib.ko >>> >> >> So ib0 came up immediately after configuring in rc.conf: >> >> ifconfig_ib0="inet 192.168.129.17 netmask 255.255.128.0" >> >> Next issue is poor performance shown by iperf: >> >> root@zfs-01:~ # iperf -c raid-01-hpc >> ------------------------------------------------------------ >> Client connecting to raid-01-hpc, TCP port 5001 >> TCP window size: 33.3 KByte (default) >> ------------------------------------------------------------ >> [  3] local 192.168.129.17 port 28021 connected with 192.168.129.1 >> port 5001 >> [ ID] Interval       Transfer     Bandwidth >> [  3]  0.0-10.1 sec   127 MBytes   106 Mbits/sec >> >> raid-01 is a CentOS 7 server. >> >> Performance between two CentOS servers is more reasonable: >> >> [root@raid-02.mortimer ~] 133: iperf -c raid-01-hpc >> ------------------------------------------------------------ >> Client connecting to raid-01-hpc, TCP port 5001 >> TCP window size: 2.50 MByte (default) >> ------------------------------------------------------------ >> [  3] local 192.168.129.2 port 57952 connected with 192.168.129.1 >> port 5001 >> [ ID] Interval       Transfer     Bandwidth >> [  3]  0.0-10.0 sec  24.1 GBytes  20.7 Gbits/sec >> >> Ultimately I aim to use NFS over IB here, but I think iperf is >> telling me that ipoib needs to be tuned. >> >> Might this be connected vs datagram issue?  If so, how to change it >> when using loadable modules? The wiki only mentions building IPOIB_CM >> into the kernel. >> >> If we can figure this out, I'll fully document everything on the >> wiki, of course. >> >> Thanks, >> >>     JB >> > > Finally found a moment to look at this again. > > I added > > CFLAGS+=-DIPOIB_CM > > to > > /etc/make.conf > > and rebuilt the modules. > > This seems to have worked, because ifconfig now allows me to set an > MTU over 4096, which was not possible without IPOIB_CM. > > I'm still seeing abysmal performance with iperf, though: > > [root@raid-01.mortimer bacon] 221: ip link show ib0 > 6: ib0: mtu 65520 qdisc pfifo_fast > state UP mode DEFAULT group default qlen 256 >     link/infiniband > 80:00:02:08:fe:80:00:00:00:00:00:00:f4:52:14:03:00:98:37:81 brd > 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff > > [root@raid-01.mortimer bacon] 222: iperf -s -B raid-01-hpc > ------------------------------------------------------------ > Server listening on TCP port 5001 > Binding to local address raid-01-hpc > TCP window size: 85.3 KByte (default) > ------------------------------------------------------------ > > root@zfs-01:~ # ifconfig ib0 > ib0: flags=8002 metric 0 mtu 65520 >     options=80018 >     lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.f4.52.14.3.0.92.88.d1 >     nd6 options=29 > > root@zfs-01:~ # iperf -c raid-01-hpc > ------------------------------------------------------------ > Client connecting to raid-01-hpc, TCP port 5001 > TCP window size: 40.8 KByte (default) > ------------------------------------------------------------ > [  3] local 192.168.1.17 port 39939 connected with 192.168.129.1 port > 5001 > [ ID] Interval       Transfer     Bandwidth > [  3]  0.0-15.1 sec   106 KBytes  57.5 Kbits/sec > > Any suggestions? > > Thanks, > >     JB > Hold on... I just noticed that ifconfig wasn't showing an IP address for ib0, so how was it even connecting to the Linux server???? After restarting netif, performance is close to what I'm seeing CentOS to CentOS: root@zfs-01:~ # ifconfig ib0 ib0: flags=8043 metric 0 mtu 65520     options=80018     lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.f4.52.14.3.0.92.88.d1     inet 192.168.129.17 netmask 0xffff8000 broadcast 192.168.255.255     nd6 options=29 root@zfs-01:~ # iperf -c raid-01-hpc ------------------------------------------------------------ Client connecting to raid-01-hpc, TCP port 5001 TCP window size:  968 KByte (default) ------------------------------------------------------------ [  3] local 192.168.129.17 port 57686 connected with 192.168.129.1 port 5001 [ ID] Interval       Transfer     Bandwidth [  3]  0.0-10.0 sec  18.8 GBytes  16.2 Gbits/sec Pending stability and NFS testing, it appears that a stock FreeBSD server may be viable in an IB HPC cluster at this point.  At least enabling and maintaining the IB modules can be done with minimal down time now that world is IB-enabled and the stock kernel modules appear to work reasonably well. Regards,     JB -- Earth is a beta site. From owner-freebsd-infiniband@freebsd.org Fri Apr 12 12:57:37 2019 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 45A49157C2FF for ; Fri, 12 Apr 2019 12:57:37 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EEF94728D2 for ; Fri, 12 Apr 2019 12:57:35 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2016.home.selasky.org (unknown [176.74.212.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 328D52601C5; Fri, 12 Apr 2019 14:57:28 +0200 (CEST) Subject: Re: Kernel modules To: Jason Bacon , "freebsd-infiniband@freebsd.org" References: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> <598a58f0-89b8-d00d-5ed7-74dd7005950f@gmail.com> <73ce0738-4d63-2f25-2ff6-00f0092de136@selasky.org> <2090dd24-db43-b689-4289-f50bd70090ea@gmail.com> <6673df26-8bba-ebd3-b2c5-d7e9c97db557@gmail.com> From: Hans Petter Selasky Message-ID: Date: Fri, 12 Apr 2019 14:57:01 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: EEF94728D2 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 88.99.82.50 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-6.51 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[selasky.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mail.turbocat.net]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.96)[-0.962,0]; IP_SCORE(-3.24)[ip: (-9.44), ipnet: 88.99.0.0/16(-4.71), asn: 24940(-2.03), country: DE(-0.01)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:88.99.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 12:57:37 -0000 On 4/12/19 2:39 PM, Jason Bacon wrote: > root@zfs-01:~ # ifconfig ib0 > ib0: flags=8002 metric 0 mtu 65520 >     options=80018 >     lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.f4.52.14.3.0.92.88.d1 >     nd6 options=29 Can you try setting an mtu of 4000 bytes on both sides and re-run the test? This large mtu I think is not supported. --HPS From owner-freebsd-infiniband@freebsd.org Fri Apr 12 13:04:28 2019 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6DC72157C4D0 for ; Fri, 12 Apr 2019 13:04:28 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: from mail-ot1-x32f.google.com (mail-ot1-x32f.google.com [IPv6:2607:f8b0:4864:20::32f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BD34272BF3 for ; Fri, 12 Apr 2019 13:04:26 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: by mail-ot1-x32f.google.com with SMTP id k21so8316598otf.1 for ; Fri, 12 Apr 2019 06:04:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=Khb7mUcnFa6issuEFskLY8A7mFWk+DU64WvHTwl47ZE=; b=sSUJuLZUBJSgRXu3rmMC+bthAKtVVvZKVtlOU5KR7DKsj8xAPZCf01SDzD3alzbQjo t7QouNYpZ5FFrcqjq+P/jcJabOcrENyPyorQkiu4tpWLLkToZf0Waobf/r1p+wsseT08 IeEdq4DRwU9pOD6o7AWeox2wOddeVwQ6RVhvbwEGTEL4/F+wHD8QxrdUWEAL925YPtzj 0onj4Ykx0MulBJ082jr+v39afzb6ZcUREKwrv3gxA8l20QDfM5bHg5Cw1Y6Mnt8yyMXA ZW6N88S/jwAG9KcFoBYNOfTDNiwiqyIpLvmaM7+k078BR/IhNlBNfECtiWD99l1soy6L eKTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=Khb7mUcnFa6issuEFskLY8A7mFWk+DU64WvHTwl47ZE=; b=ps2KuJ5XvCyDHP1EddOZFI+r3oNw1Sm7EMDZ8kPNH6Y5G+syN3eaRGgZQzKFeDACJp /ldq+45B1IoXkACoLe+8ge8/cluf+XxHzaxvX5GdBkLfVxcWdVOGLLORZdzo7tc4mSHn 8TN9G+0WJ7QgPxiR44JZEPZsdDSEmqUauV/PV5D2CS9rs6M2t2Nat5+wbQiVVcPKKwNe LzlC+ixwCAh+3+g4JfRO53kiPKN9Xa/c6ubCYBOWCKToUKMsoSMG1WzjCWhlFcnoSH0q cBKeb01mO7zZRZa4N32zVlUdiw+Ucx6koRcrmLfil+71uRrmeZB3+BsLOB+ed3VFeCKO KUFg== X-Gm-Message-State: APjAAAUjJ0UfmR6b0phO2lKmtid5ZnMCqDxxSb8xjQYb3gybLYJyJYpN XaPZw/iUdC5YnIEHwHdSU5qP4eL2 X-Google-Smtp-Source: APXvYqx0a+kfAA1tej2UC5YukdwPio7+KA+hm8y1mo5baCCG27wQmlOYySkuTrBzUNWI0CoLNSVoTw== X-Received: by 2002:a9d:7306:: with SMTP id e6mr2959316otk.79.1555074265628; Fri, 12 Apr 2019 06:04:25 -0700 (PDT) Received: from manatee.acadix.biz (cpe-174-102-163-140.wi.res.rr.com. [174.102.163.140]) by smtp.gmail.com with ESMTPSA id d7sm16183307oig.33.2019.04.12.06.04.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 Apr 2019 06:04:24 -0700 (PDT) Subject: Re: Kernel modules To: Hans Petter Selasky , "freebsd-infiniband@freebsd.org" References: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> <598a58f0-89b8-d00d-5ed7-74dd7005950f@gmail.com> <73ce0738-4d63-2f25-2ff6-00f0092de136@selasky.org> <2090dd24-db43-b689-4289-f50bd70090ea@gmail.com> <6673df26-8bba-ebd3-b2c5-d7e9c97db557@gmail.com> From: Jason Bacon Message-ID: <2f4d9a14-4ff6-0d34-06f0-bbb4ac76c6bd@gmail.com> Date: Fri, 12 Apr 2019 08:04:23 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: BD34272BF3 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=sSUJuLZU; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of bacon4000@gmail.com designates 2607:f8b0:4864:20::32f as permitted sender) smtp.mailfrom=bacon4000@gmail.com X-Spamd-Result: default: False [-6.81 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.99)[-0.989,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-infiniband@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[f.2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-2.81)[ip: (-8.87), ipnet: 2607:f8b0::/32(-2.94), asn: 15169(-2.18), country: US(-0.06)] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 13:04:28 -0000 On 2019-04-12 07:57, Hans Petter Selasky wrote: > On 4/12/19 2:39 PM, Jason Bacon wrote: >> root@zfs-01:~ # ifconfig ib0 >> ib0: flags=8002 metric 0 mtu 65520 >>      options=80018 >>      lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.f4.52.14.3.0.92.88.d1 >>      nd6 options=29 > > Can you try setting an mtu of 4000 bytes on both sides and re-run the > test? This large mtu I think is not supported. > > --HPS I assume you saw my followup showing 16 Gb/s... I'll try playing with MTU anyway.  Maybe that will improve performance a bit more? I'm going to do a bunch of tuning and test NFS.  Will report results back here when I have some substantial info. Thanks,     JB -- Earth is a beta site. From owner-freebsd-infiniband@freebsd.org Sat Apr 13 13:52:26 2019 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9617B15794AB for ; Sat, 13 Apr 2019 13:52:26 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: from mail-ot1-x334.google.com (mail-ot1-x334.google.com [IPv6:2607:f8b0:4864:20::334]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D355A84BC9 for ; Sat, 13 Apr 2019 13:52:24 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: by mail-ot1-x334.google.com with SMTP id e80so10824916ote.5 for ; Sat, 13 Apr 2019 06:52:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:references:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=jrSnzP6islWzOYRgirRH4o6+xLWx+9PbI/SHA1AI+T8=; b=BtBSpoawlqcbUOEMjfvvjKMMDFCwobHdcNvCj9fPWIp7Mj2fSl4vmnkjppwGI+bnW3 5uSxFdh/OngSUrfZMVtqDOPoDgMS7ZNPjbRh2h5eavBYBaq55bznj1oNSC1usY6e4Ov+ i8TP6EYRMzCIrsR0GaBo1r9CBgaG3njlPCa6zKYyDwAq9liJBXHOXKqFyIfEdoHWIp0d WauD17/yM7/9HXDgnHFZTF0rQBc5sLccyllYbfzqFhSdwUvSqdeAk8bQvoh+0oTf7Pj7 iSXFs/YDJmaZbzTMtc5Ob0q5wx9YT3ZJpC2P0yyBxcMjWi9LIqXsOw3FKuxJLPc74wnj soHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=jrSnzP6islWzOYRgirRH4o6+xLWx+9PbI/SHA1AI+T8=; b=S3HZhN64HEBfup+j+e9A/yGlfN+gCFnr2neLMBz0oRdDFpZMhed1HwCryEnLsIwHHj 4YcvUyjuvHDYBMBRCPx4lwDSW5OWrldP0vyKK7kIVaVlV3pJFcKrIW4MTWusvY7leuba JK1A+9AyXM11ptXkItq7doCiD7setCErCtLgtbtV48YehCoiOURPURZknTakR/wGKnUH wQNrFlaN5clsWZFncup9hiZ/mnKtzqMspVrWou4vfBdQW1msYEaXt32VhEDVPGjIrtxq UqQtExPwxM1sPSA56nqHKm3db57LEOD6XcCL1nQiEI6hgi+1fSwC51xdQoQh4C4yMKTY gq5A== X-Gm-Message-State: APjAAAUBndxXXEGuT5iMCci7xWgRoAd6OwA7gv4M5vDzJc0dU6HhvIez TIqrWbzlpYxJiOy2bQwyavqu6Yya X-Google-Smtp-Source: APXvYqwl/uEMpeut/kUpw6CmMMBrp86i4GXumI3z2czbYJai+fyxUZUHbXIVsfHDmPLWQ85vFYcJ3A== X-Received: by 2002:a05:6830:1398:: with SMTP id d24mr41293302otq.104.1555163543440; Sat, 13 Apr 2019 06:52:23 -0700 (PDT) Received: from cray.acadix.biz ([2605:a000:bc45:5f00:223:24ff:fe37:c4d7]) by smtp.gmail.com with ESMTPSA id b17sm17475199otq.26.2019.04.13.06.52.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 13 Apr 2019 06:52:22 -0700 (PDT) Subject: Re: Kernel modules From: Jason Bacon To: Hans Petter Selasky , "freebsd-infiniband@freebsd.org" References: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> <598a58f0-89b8-d00d-5ed7-74dd7005950f@gmail.com> <73ce0738-4d63-2f25-2ff6-00f0092de136@selasky.org> <2090dd24-db43-b689-4289-f50bd70090ea@gmail.com> <6673df26-8bba-ebd3-b2c5-d7e9c97db557@gmail.com> <2f4d9a14-4ff6-0d34-06f0-bbb4ac76c6bd@gmail.com> Message-ID: <5166ec29-876b-0bd3-8a84-8a222647e87a@gmail.com> Date: Sat, 13 Apr 2019 08:52:21 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <2f4d9a14-4ff6-0d34-06f0-bbb4ac76c6bd@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: D355A84BC9 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=BtBSpoaw; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of bacon4000@gmail.com designates 2607:f8b0:4864:20::334 as permitted sender) smtp.mailfrom=bacon4000@gmail.com X-Spamd-Result: default: False [-6.79 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.98)[-0.981,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-infiniband@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[4.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.80)[ip: (-8.82), ipnet: 2607:f8b0::/32(-2.95), asn: 15169(-2.19), country: US(-0.06)] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Apr 2019 13:52:26 -0000 On 2019-04-12 08:04, Jason Bacon wrote: > On 2019-04-12 07:57, Hans Petter Selasky wrote: >> On 4/12/19 2:39 PM, Jason Bacon wrote: >>> root@zfs-01:~ # ifconfig ib0 >>> ib0: flags=8002 metric 0 mtu 65520 >>>      options=80018 >>>      lladdr 80.0.2.8.fe.80.0.0.0.0.0.0.f4.52.14.3.0.92.88.d1 >>>      nd6 options=29 >> >> Can you try setting an mtu of 4000 bytes on both sides and re-run the >> test? This large mtu I think is not supported. >> >> --HPS > I assume you saw my followup showing 16 Gb/s... > > I'll try playing with MTU anyway.  Maybe that will improve performance > a bit more? > > I'm going to do a bunch of tuning and test NFS.  Will report results > back here when I have some substantial info. > > Thanks, > >     JB > Some data for comparison. Regarding MTU, the bigger the better, up to a point.  At 65520, my server became unresponsive to the point of an ssh session timing out.  It recovered after a minute or two and there did not seem to be any permanent harm.  Lower MTUs provide more stable performance (monitoring with "iostat 1") and lower throughput. For now I'm using 16380, 1/4 of 65520 which is the default on CentOS 7. I haven't yet seen any stability issues at this level. Explanation of data: data-05 is a CentOS 7 RAID server, XFS filesystem. zfs-01 is a FreeBSD 12 RAID server. Hardware is identical - PowerEdge R720xd, 12 ST2000NM0023 SAS drives, RAID-6, PERC H710 mini (MegaRAID). "-local" means benchmark run on the server, testing the local RAID. "-nfs4" means benchmark run on a compute node, testing NFS over FDR Infiniband. Benchmarked with and without ZFS lz4 compression enabled on the server. All results are the average of 3 trials. Highlights: raid-05-nfs4 vs zfs-01-nfs4: o The FreeBSD server outperformed the CentOS server on random and sequential reads. o The FreeBSD server fell short on fresh write and way short on overwrite. zfs-01-local vs FreeBSD 10 results: o FreeBSD is seeing performance limits on the local array for some reason. o Local disk performance was much better on FreeBSD 10 a couple years ago.  ZFS or mrsas regression?  As I recall, it was overall about 5% faster than CentOS 6 + XFS at that time. o Would resolving this push FreeBSD's NFS write performance over CentOS? Overall, I'd say we're looking pretty good at this point. Performance is way more than adequate for most HPC jobs.  I suspect some tuning and/or minor improvements to the IB code will improve it further. Stability will take a long time to test properly.  I'm going to start by rerunning some of our most I/O-intensive jobs on it - jobs that actually broke our CentOS RAID servers until I switched them to NFS over RDMA. ==> bench-raid-05-local <==    93.92 GiB write       4.00 MiB blocks     71378.00 ms      1347.42 MiB/s         1024 seek        4.00 MiB blocks        18.06 ms       227.56 MiB/s    93.92 GiB read        4.00 MiB blocks     67519.00 ms      1424.43 MiB/s    93.92 GiB rewrite     4.00 MiB blocks     73635.00 ms      1306.12 MiB/s ==> bench-raid-05-nfs4 <==   125.03 GiB write       4.00 MiB blocks    145903.00 ms       877.53 MiB/s         1024 seek        4.00 MiB blocks        23.98 ms       170.67 MiB/s   125.03 GiB read        4.00 MiB blocks    236010.00 ms       542.49 MiB/s   125.03 GiB rewrite     4.00 MiB blocks    158151.00 ms       809.57 MiB/s ==> bench-zfs-01-local <==   127.74 GiB write       4.00 MiB blocks    157977.00 ms       828.00 MiB/s         1024 seek        4.00 MiB blocks        18.39 ms       227.56 MiB/s   127.74 GiB read        4.00 MiB blocks    165471.00 ms       790.50 MiB/s   127.74 GiB rewrite     4.00 MiB blocks    116542.00 ms      1122.38 MiB/s ==> bench-zfs-01-lz4-nfs4 <==   125.03 GiB write       4.00 MiB blocks    185550.00 ms       690.03 MiB/s         1024 seek        4.00 MiB blocks        24.32 ms       170.67 MiB/s   125.03 GiB read        4.00 MiB blocks    234103.00 ms       546.91 MiB/s   125.03 GiB rewrite     4.00 MiB blocks    423833.00 ms       302.09 MiB/s ==> bench-zfs-01-nfs4 <==   125.03 GiB write       4.00 MiB blocks    174645.00 ms       733.11 MiB/s         1024 seek        4.00 MiB blocks        14.67 ms       273.07 MiB/s   125.03 GiB read        4.00 MiB blocks    225402.00 ms       568.03 MiB/s   125.03 GiB rewrite     4.00 MiB blocks    413798.00 ms       309.41 MiB/s FreeBSD 10.3 local disk results from a couple years ago on the same machine:   127.76 GiB write       4.00 MiB blocks    101323.00 ms 1291.13 MiB/s         1024 seek        4.00 MiB blocks        18.57 ms 215.58 MiB/s   127.76 GiB read        4.00 MiB blocks     95363.00 ms 1371.83 MiB/s   127.76 GiB rewrite     4.00 MiB blocks    108186.00 ms 1209.23 MiB/s -- Earth is a beta site. From owner-freebsd-infiniband@freebsd.org Sat Apr 13 18:29:42 2019 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DCC5157FC92 for ; Sat, 13 Apr 2019 18:29:42 +0000 (UTC) (envelope-from justin@postgresql.org) Received: from meldrar.postgresql.org (meldrar.postgresql.org [IPv6:2a02:c0:301:0:ffff::31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "meldrar.postgresql.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 327608D02F for ; Sat, 13 Apr 2019 18:29:40 +0000 (UTC) (envelope-from justin@postgresql.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=postgresql.org; s=20171124; h=Message-ID:References:In-Reply-To:Subject:Cc: To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version:Sender: Reply-To:Content-ID:Content-Description; bh=qWDQNeFamd5cHvin9qhhaiCuPMuysaXitvd9uwyZlsc=; b=3cIJuVk838irQaJTLwnksj9rIE uMl3lYxrTRXjLrSahX/7lAwe+yLugja0nfVSexHYVWIUyKM4AVo58uDtZ9YjzQiKUtvalcBq/kbQQ Z4XTJhyy4ljulbhZxZpHtdvEgoOOtUuD9MzXXACPb6YPvi0wKwKezb1PvHyI79S8ilW3YiMwfnPLh mmXIPvbjYZw9VWFlsHT3U7zkNUkWTdc0EQB+xs9nqsnEjXuji6kXm+iGO1pFUrJzMtsGaq62ixklY esNrkAymKuB0l0tJbkDG5aNIKoyqHzagsMyK+bbuAN/9pKYaaql3CkljtXWLoggcRVL5HX48ElwH9 GKNKicxg==; Received: from meldrar.postgresql.org ([87.238.57.231] helo=webmail.postgresql.org) by meldrar.postgresql.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hFNPB-00065T-Ts; Sat, 13 Apr 2019 18:29:36 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Sun, 14 Apr 2019 04:29:33 +1000 From: Justin Clift To: Jason Bacon Cc: Hans Petter Selasky , freebsd-infiniband@freebsd.org Subject: Re: Kernel modules In-Reply-To: <5166ec29-876b-0bd3-8a84-8a222647e87a@gmail.com> References: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> <598a58f0-89b8-d00d-5ed7-74dd7005950f@gmail.com> <73ce0738-4d63-2f25-2ff6-00f0092de136@selasky.org> <2090dd24-db43-b689-4289-f50bd70090ea@gmail.com> <6673df26-8bba-ebd3-b2c5-d7e9c97db557@gmail.com> <2f4d9a14-4ff6-0d34-06f0-bbb4ac76c6bd@gmail.com> <5166ec29-876b-0bd3-8a84-8a222647e87a@gmail.com> Message-ID: X-Sender: justin@postgresql.org User-Agent: Roundcube Webmail/1.2.3 X-Pg-Spam-Score: -2.9 (--) X-Rspamd-Queue-Id: 327608D02F X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=postgresql.org header.s=20171124 header.b=3cIJuVk8; dmarc=pass (policy=none) header.from=postgresql.org X-Spamd-Result: default: False [-3.02 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DKIM_TRACE(0.00)[postgresql.org:+]; RCVD_IN_DNSWL_MED(-0.20)[1.3.0.0.0.0.0.0.0.0.0.0.f.f.f.f.0.0.0.0.1.0.3.0.0.c.0.0.2.0.a.2.list.dnswl.org : 127.0.9.2]; DMARC_POLICY_ALLOW(-0.50)[postgresql.org,none]; MX_GOOD(-0.01)[makus.postgresql.org,magus.postgresql.org]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-0.00)[country: NO(-0.01)]; ASN(0.00)[asn:39029, ipnet:2a02:c0::/32, country:NO]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; R_DKIM_ALLOW(-0.20)[postgresql.org:s=20171124]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; DWL_DNSWL_MED(0.00)[postgresql.org.dwl.dnswl.org : 127.0.9.2]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_SHORT(0.49)[0.487,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_NA(0.00)[]; RBL_COMPOSITE_RCVD_IN_DNSWL_MED_DWL_DNSWL_MED(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Apr 2019 18:29:42 -0000 On 2019-04-13 23:52, Jason Bacon wrote: > Stability will take a long time to test properly.  I'm going to start > by rerunning some of our most I/O-intensive jobs on it - jobs that > actually broke our CentOS RAID servers until I switched them to NFS > over RDMA. That's got to be the first time anyone's ever mentioned "NFS over RDMA" as increasing a systems' stability. :) + Justin From owner-freebsd-infiniband@freebsd.org Sat Apr 13 18:41:16 2019 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BBBAC1580042 for ; Sat, 13 Apr 2019 18:41:16 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: from mail-oi1-x22f.google.com (mail-oi1-x22f.google.com [IPv6:2607:f8b0:4864:20::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E07228D5F4 for ; Sat, 13 Apr 2019 18:41:15 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: by mail-oi1-x22f.google.com with SMTP id i21so10635628oib.11 for ; Sat, 13 Apr 2019 11:41:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=Y2qCStCcF880Q+sazEyYtbqOfiq5TgIn1yP2ccAjRsg=; b=AjyF2yxHvm8xzVTqbFnYSJfpuq2lPvHKH6AsLsKGCLg7hJDIQcuqZMXRcJ54KxoxK2 BmLmPDEJFPpvJjqZm/N91ZeXL2HC8cVM8UJa98CH68bPp1QP+7xzTB6IchGrcozll7o3 Gp73cKNtPTEDHSPRj+4Mwj9/zvaAnq0h5PRGbzf8tFkD9jjXtnlN8vG/032TEPy7PJxu lbEGnuFjlbgpVqVS/NL/IldXLuOPx7ouUuEnuu6zw80D7jnXxGgRhR2lFPOcoNrquUU9 yxmJ6eN4hoco4sul+SsdUzuPHMflZoA5pEpLEjRGAIwqn07JiyUnob67wCJBEnwOc84j Eqhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=Y2qCStCcF880Q+sazEyYtbqOfiq5TgIn1yP2ccAjRsg=; b=Mbyixxe8197RppWV/Z3Ub0THSUXiJv+XBcCULvOH8cSOVePjBYVZsnCgCk3og3CAAv d3d2rQZzbf/IqsooowAlRECuoAsZYgIyRv9YWJQbNuNbeJeDGmF6qlwWgzl253NC7Q2p stpelikbSx7u6OSmSLHtxBXF4bh5sW2vczf59Pqx5lqnsB9023hfbnoB0LoE1xnJ1x/y 40fzUkUlxMuve77cW9Rj5eS1XbRpaNaCFl6ce4DPQWdghi7dm24VZDozRsa28hkJMiMC cgO3gTo/eAfPlAjHgFLScm1fh1yFFsobiPaPqMTyhHEoRgE+KbTCs27/wGOMeh7/vX4L 7WWQ== X-Gm-Message-State: APjAAAXN05lDMzsiuT26yCfzwmyGFxRzbaEQg8m3Ohe8HW1kpQwYb7b/ R6TrJwmmyb3Lu1+25cr+/a7qvxej X-Google-Smtp-Source: APXvYqzwYm9xLNr/8lVMH4dhBIZ5Wjzd+AoLfa7JI/lVzrQ9nbbS2gASpVa5bdPu8fGjnMWoFYR+/Q== X-Received: by 2002:aca:eb55:: with SMTP id j82mr13981923oih.178.1555180874411; Sat, 13 Apr 2019 11:41:14 -0700 (PDT) Received: from cray.acadix.biz ([2605:a000:bc45:5f00:223:24ff:fe37:c4d7]) by smtp.gmail.com with ESMTPSA id n77sm22298561ota.77.2019.04.13.11.41.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 13 Apr 2019 11:41:13 -0700 (PDT) Subject: Re: Kernel modules To: Justin Clift Cc: Hans Petter Selasky , freebsd-infiniband@freebsd.org References: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> <598a58f0-89b8-d00d-5ed7-74dd7005950f@gmail.com> <73ce0738-4d63-2f25-2ff6-00f0092de136@selasky.org> <2090dd24-db43-b689-4289-f50bd70090ea@gmail.com> <6673df26-8bba-ebd3-b2c5-d7e9c97db557@gmail.com> <2f4d9a14-4ff6-0d34-06f0-bbb4ac76c6bd@gmail.com> <5166ec29-876b-0bd3-8a84-8a222647e87a@gmail.com> From: Jason Bacon Message-ID: <236a3839-e880-ab17-146a-4521d1894813@gmail.com> Date: Sat, 13 Apr 2019 13:41:12 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: E07228D5F4 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=AjyF2yxH; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of bacon4000@gmail.com designates 2607:f8b0:4864:20::22f as permitted sender) smtp.mailfrom=bacon4000@gmail.com X-Spamd-Result: default: False [-6.73 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.97)[-0.969,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-infiniband@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[f.2.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.75)[ip: (-8.56), ipnet: 2607:f8b0::/32(-2.95), asn: 15169(-2.19), country: US(-0.06)] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Apr 2019 18:41:17 -0000 On 2019-04-13 13:29, Justin Clift wrote: > On 2019-04-13 23:52, Jason Bacon wrote: > >> Stability will take a long time to test properly.  I'm going to start >> by rerunning some of our most I/O-intensive jobs on it - jobs that >> actually broke our CentOS RAID servers until I switched them to NFS >> over RDMA. > > That's got to be the first time anyone's ever mentioned "NFS over > RDMA" as > increasing a systems' stability. :) > > + Justin Believe it or not...  ;-) After my upgrade from CentOS 6 to CentOS 7, NFS over TCP started falling apart under heavy load; servers and compute nodes becoming unresponsive and requiring a reboot to restore stability. If it's due to problems in the CentOS TCP stack, NFS over RDMA would help by eliminating the TCP stack from the pathway. One one cluster (old qlogic HCAs), setting net.core.netdev_budget=2000 seems to have solved the issue.  On the other (newer Mellanox FDR HCAs), it did not seem to help, so I tried RDMA and it's been stable ever since.  Down side is we can no longer monitor traffic with iftop... -- Earth is a beta site.