From owner-freebsd-infiniband@freebsd.org Mon Mar 18 12:32:28 2019 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 586BF153DC99 for ; Mon, 18 Mar 2019 12:32:28 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: from mail-ot1-x330.google.com (mail-ot1-x330.google.com [IPv6:2607:f8b0:4864:20::330]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 60A7C87598 for ; Mon, 18 Mar 2019 12:32:27 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: by mail-ot1-x330.google.com with SMTP id t7so14209530otk.8 for ; Mon, 18 Mar 2019 05:32:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:from:subject:message-id:date:user-agent:mime-version :content-transfer-encoding:content-language; bh=64t9aplc4pqNnZqOCBdlmnWxTnhoy+R1pzQfxlx1gE8=; b=g5QPnfscHR3NvKm/0coOcFgd25vMXgYFmt3q12uO5wNtSrny+ss+Sb3y4gJYam+MyY jEHOS4mjDDopXrFyhGVAuvDf3HB6LJQqdyYgzrntNvQ13gUJsRVp8JRQHV3nLPd6ZYru nzAc4QTfyudqL03aCHYqnBsFgTr/f1WEgMbZZD9DKDWgw3Zb1bejtKJuPxYgYTdtzE3Y dVyMgblpAsHBfcizWYlDSC5Vh5T8i4dAvEke8hz6TRJ6TPZv8z1/IHAxAI20toEV7R63 nMnrNKFLtloCxkelHbdHffPMqgPAmm0yGiiEteuSTpPMPwejBRlD5l8ihCwIOM7kDqqw VReg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-transfer-encoding:content-language; bh=64t9aplc4pqNnZqOCBdlmnWxTnhoy+R1pzQfxlx1gE8=; b=jHk55HtY4wL2sPHVGXRoWvf1mn624sJogniRHyhxS9qxu0oX2+6oz6KHZGpv7jVVhV ank51c0pXRiqD6kViueDvx8pkeoufaeF2aHnnVYMU3GKrBYZc20OWUTRTqQdO0dv4FGI 5jZigzWHJ0OMt/IXd4Lb8nAqZQMEZfXowk2hyeX6ZXda4lvBCebUhU+Re2SvfDYqY0Bv VIaduQcSRwd3MkpdXv1zqqXkCHzyk4Oe8ylVPgKX/Sxl7QtRzkSqbV9egY4ylyFBySLa 69OrSnfga/nk4ER3UxBjMYMbWEBXDOv8MTH0MosOrrZQ2iJHln2+n2WiSEtThf7smuj5 mtpA== X-Gm-Message-State: APjAAAUSueb2/z3iR8a3vc+HtVH4GheBwnpueMhtnx3s+2VRpHC07kQ9 V3a/Eh7MSbMGeLprrdeTlddDbUd4 X-Google-Smtp-Source: APXvYqw/WU3tUq51bjFis4xPfnYutB6nu3+cs/9s1TlKLwpnbv9VsntB/pLh/CtuaDjGOsMwP+hJRg== X-Received: by 2002:a9d:2c28:: with SMTP id f37mr7403794otb.126.1552912345887; Mon, 18 Mar 2019 05:32:25 -0700 (PDT) Received: from cray.acadix.biz (cpe-174-102-163-140.wi.res.rr.com. [174.102.163.140]) by smtp.gmail.com with ESMTPSA id d21sm884107otl.33.2019.03.18.05.32.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 18 Mar 2019 05:32:24 -0700 (PDT) To: "freebsd-infiniband@freebsd.org" From: Jason Bacon Subject: Kernel modules Message-ID: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> Date: Mon, 18 Mar 2019 07:32:23 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.5.3 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 60A7C87598 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=g5QPnfsc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of bacon4000@gmail.com designates 2607:f8b0:4864:20::330 as permitted sender) smtp.mailfrom=bacon4000@gmail.com X-Spamd-Result: default: False [-6.63 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.97)[-0.973,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-infiniband@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-2.65)[ip: (-8.32), ipnet: 2607:f8b0::/32(-2.75), asn: 15169(-2.09), country: US(-0.07)]; RCVD_IN_DNSWL_NONE(0.00)[0.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; TO_DN_EQ_ADDR_ALL(0.00)[] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2019 12:32:28 -0000 Anyone using the kernel modules lately? After building per the instructions on the wiki (https://wiki.freebsd.org/InfiniBand): <<>> /home/bacon 1005 # kldload ibcore kldload: an error occurred while loading module ibcore. Please check dmesg(8) for more details. dmesg: link_elf_obj: symbol if_index undefined linker_load_file: /boot/modules/ibcore.ko - unsupported file type Thanks,     JB -- Earth is a beta site. From owner-freebsd-infiniband@freebsd.org Mon Mar 18 12:55:43 2019 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 848F2153E7A1 for ; Mon, 18 Mar 2019 12:55:43 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C3DA08847B for ; Mon, 18 Mar 2019 12:55:42 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2016.home.selasky.org (unknown [176.74.212.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 2711D260348; Mon, 18 Mar 2019 13:55:40 +0100 (CET) Subject: Re: Kernel modules To: Jason Bacon , "freebsd-infiniband@freebsd.org" References: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> From: Hans Petter Selasky Message-ID: Date: Mon, 18 Mar 2019 13:55:15 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: C3DA08847B X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 88.99.82.50 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-6.46 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[selasky.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: mail.turbocat.net]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.93)[-0.929,0]; IP_SCORE(-3.22)[ip: (-9.46), ipnet: 88.99.0.0/16(-4.65), asn: 24940(-1.96), country: DE(-0.01)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:88.99.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2019 12:55:43 -0000 On 3/18/19 1:32 PM, Jason Bacon wrote: > > Anyone using the kernel modules lately? > > After building per the instructions on the wiki > (https://wiki.freebsd.org/InfiniBand): > > <<>> /home/bacon 1005 # kldload ibcore > kldload: an error occurred while loading module ibcore. Please check > dmesg(8) for more details. > > dmesg: > > link_elf_obj: symbol if_index undefined > linker_load_file: /boot/modules/ibcore.ko - unsupported file type Hi, You built the KMODs out of the kernel? Try adding DEBUG_FLAGS="-DVIMAGE=1" to your out of tree build. --HPS From owner-freebsd-infiniband@freebsd.org Mon Mar 18 13:56:50 2019 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D30EC1540211 for ; Mon, 18 Mar 2019 13:56:49 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: from mail-ot1-x344.google.com (mail-ot1-x344.google.com [IPv6:2607:f8b0:4864:20::344]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DAEB18A5F7 for ; Mon, 18 Mar 2019 13:56:48 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: by mail-ot1-x344.google.com with SMTP id r19so14506294otn.1 for ; Mon, 18 Mar 2019 06:56:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=NfdQ6rxM9RXttL3IvG+EF59LW/KIyu2oj0dZZBArP9U=; b=UWdlVsX+lhh5SEeywmp/HuT250qE94tooZ+XcdZtOCNWPR89Vkvg9tsHPLUbcUhKiM oVDL5KykweoZdAnHWeGgufEmCPcgg+cQSyjW88jAjOGWdiwGRyHtwV5revj0qtesKJ+4 jGd4xzH2z9q2BdbP7dL/0Angtrcq1pQEO85aPZS5+RAEqGTbIzXkKhYLiDuTQNOpVG+C 6j8lDIiVt7GWj7YAUO0bUNuOZGth+/Pmf4LQuUmyhRBeQUUhSkFsFAfwMVeFeCuDlORF EzzAt0S2uszTQ6gT40m7gU5DFDQGTmdQK9xs06yqjMzXjIvVD1r1Z1pf1tJQ17WFkzpG AAwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=NfdQ6rxM9RXttL3IvG+EF59LW/KIyu2oj0dZZBArP9U=; b=mUkVqGainIvxeQltx5NGGPuwgrlLbzM/ENG0vRxomlyX492mbKas98e34PW2efmysN HNvXVDHEIQQqrfK2IavIOCKkICVoqqEs5IdmTuryCPmm3H8W10F8M1rxG9al6qulXyh3 GyfptL8Y+nKR0RKFBxaubA1TeeLA+ySFzEGrrqopVuYzzIR+lxP5+WA/xTx7eFl/JXf9 W6DugL83NZk2Q3h6jx4RvDKprvdwka4YEbOrMQYGPXdAsqpCBMvZ7Ot+2nIiyk6u/oXt /ek31700PExUQ4m/UDr1S7BquA2u9hLW0bRJkMPy8ziGiFvwFnK7Pf41fa1DbXNXEsP8 g+gA== X-Gm-Message-State: APjAAAVAlYjr+DryYXa65lFcAtQGdIb7agYk5GH7X+XYNNKMzGpiHIUl 7UU11C0lv16h9a6mMtxBxHt11YTz X-Google-Smtp-Source: APXvYqwSUKS2Iq3rcjz/gEpv2TttyVdUJKYiCfyjP/aT2fZWaF4kpFyzCWy8UCLFXcy5L0mjSkTgjA== X-Received: by 2002:a05:6830:2002:: with SMTP id e2mr11070908otp.180.1552917407804; Mon, 18 Mar 2019 06:56:47 -0700 (PDT) Received: from cray.acadix.biz (cpe-174-102-163-140.wi.res.rr.com. [174.102.163.140]) by smtp.gmail.com with ESMTPSA id u198sm1221593oie.26.2019.03.18.06.56.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 18 Mar 2019 06:56:46 -0700 (PDT) Subject: Re: Kernel modules To: Hans Petter Selasky , "freebsd-infiniband@freebsd.org" References: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> From: Jason Bacon Message-ID: <598a58f0-89b8-d00d-5ed7-74dd7005950f@gmail.com> Date: Mon, 18 Mar 2019 08:56:43 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.5.3 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: DAEB18A5F7 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=UWdlVsX+; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of bacon4000@gmail.com designates 2607:f8b0:4864:20::344 as permitted sender) smtp.mailfrom=bacon4000@gmail.com X-Spamd-Result: default: False [-4.38 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.90)[-0.896,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-infiniband@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[4.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.47)[ip: (2.53), ipnet: 2607:f8b0::/32(-2.75), asn: 15169(-2.09), country: US(-0.07)] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2019 13:56:50 -0000 On 3/18/19 7:55 AM, Hans Petter Selasky wrote: > On 3/18/19 1:32 PM, Jason Bacon wrote: >> >> Anyone using the kernel modules lately? >> >> After building per the instructions on the wiki >> (https://wiki.freebsd.org/InfiniBand): >> >> <<>> /home/bacon 1005 # kldload ibcore >> kldload: an error occurred while loading module ibcore. Please check >> dmesg(8) for more details. >> >> dmesg: >> >> link_elf_obj: symbol if_index undefined >> linker_load_file: /boot/modules/ibcore.ko - unsupported file type > > Hi, > > You built the KMODs out of the kernel? Try adding > DEBUG_FLAGS="-DVIMAGE=1" to your out of tree build. > > --HPS I used the script for building kernel modules (which I wrote and posted at https://wiki.freebsd.org/InfiniBand a few years ago, based on advice from a Mellanox developer at the time).  This will show exactly what I did. Hoping there's a simple fix for this. In theory, it should now be possible to enable IB support on a stock FreeBSD system in a few minutes, since we should be able to use loadable modules instead of doing a buildkernel, and IB support is now included in the base, making buildworld unnecessary: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211761 This is a big step toward enabling the use of stock FreeBSD systems in HPC. I had successfully used the IB stack before by loading the modules and doing a buildworld. Regards,     JB -- Earth is a beta site. From owner-freebsd-infiniband@freebsd.org Mon Mar 18 14:11:23 2019 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 93E69154086C for ; Mon, 18 Mar 2019 14:11:23 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A76E88ADEC for ; Mon, 18 Mar 2019 14:11:22 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2016.home.selasky.org (unknown [176.74.212.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 6F216260348; Mon, 18 Mar 2019 15:11:14 +0100 (CET) Subject: Re: Kernel modules To: Jason Bacon , "freebsd-infiniband@freebsd.org" References: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> <598a58f0-89b8-d00d-5ed7-74dd7005950f@gmail.com> From: Hans Petter Selasky Message-ID: <73ce0738-4d63-2f25-2ff6-00f0092de136@selasky.org> Date: Mon, 18 Mar 2019 15:10:49 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <598a58f0-89b8-d00d-5ed7-74dd7005950f@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: A76E88ADEC X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 88.99.82.50 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-6.48 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[selasky.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mail.turbocat.net]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.96)[-0.956,0]; IP_SCORE(-3.22)[ip: (-9.46), ipnet: 88.99.0.0/16(-4.65), asn: 24940(-1.96), country: DE(-0.01)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:88.99.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2019 14:11:23 -0000 On 3/18/19 2:56 PM, Jason Bacon wrote: > On 3/18/19 7:55 AM, Hans Petter Selasky wrote: >> On 3/18/19 1:32 PM, Jason Bacon wrote: >>> >>> Anyone using the kernel modules lately? >>> >>> After building per the instructions on the wiki >>> (https://wiki.freebsd.org/InfiniBand): >>> >>> <<>> /home/bacon 1005 # kldload ibcore >>> kldload: an error occurred while loading module ibcore. Please check >>> dmesg(8) for more details. >>> >>> dmesg: >>> >>> link_elf_obj: symbol if_index undefined >>> linker_load_file: /boot/modules/ibcore.ko - unsupported file type >> >> Hi, >> >> You built the KMODs out of the kernel? Try adding >> DEBUG_FLAGS="-DVIMAGE=1" to your out of tree build. >> >> --HPS > > I used the script for building kernel modules (which I wrote and posted > at https://wiki.freebsd.org/InfiniBand a few years ago, based on advice > from a Mellanox developer at the time).  This will show exactly what I did. > Can you try: make DEBUG_FLAGS="-DVIMAGE=1" instead of make in the script. In FreeBSD-12 VIMAGE was turned on by default unlike FreeBSD-11. --HPS > Hoping there's a simple fix for this. > > In theory, it should now be possible to enable IB support on a stock > FreeBSD system in a few minutes, since we should be able to use loadable > modules instead of doing a buildkernel, and IB support is now included > in the base, making buildworld unnecessary: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211761 > > This is a big step toward enabling the use of stock FreeBSD systems in HPC. > > I had successfully used the IB stack before by loading the modules and > doing a buildworld. > > Regards, > >     JB > From owner-freebsd-infiniband@freebsd.org Mon Mar 18 14:13:14 2019 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 50DD615408B9 for ; Mon, 18 Mar 2019 14:13:14 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AB7DF8AF07 for ; Mon, 18 Mar 2019 14:13:13 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2016.home.selasky.org (unknown [176.74.212.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 405A2260348; Mon, 18 Mar 2019 15:13:10 +0100 (CET) Subject: Re: Kernel modules From: Hans Petter Selasky To: Jason Bacon , "freebsd-infiniband@freebsd.org" References: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> <598a58f0-89b8-d00d-5ed7-74dd7005950f@gmail.com> <73ce0738-4d63-2f25-2ff6-00f0092de136@selasky.org> Message-ID: <6eba1af5-947f-0167-18f1-9bc14ab3b39b@selasky.org> Date: Mon, 18 Mar 2019 15:12:46 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <73ce0738-4d63-2f25-2ff6-00f0092de136@selasky.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: AB7DF8AF07 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 88.99.82.50 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-6.45 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net]; MX_GOOD(-0.01)[cached: mail.turbocat.net]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.92)[-0.921,0]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:88.99.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[selasky.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-3.22)[ip: (-9.46), ipnet: 88.99.0.0/16(-4.65), asn: 24940(-1.96), country: DE(-0.01)]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2019 14:13:14 -0000 On 3/18/19 3:10 PM, Hans Petter Selasky wrote: > > Can you try: > > make DEBUG_FLAGS="-DVIMAGE=1" > > instead of > > make > > in the script. In FreeBSD-12 VIMAGE was turned on by default unlike > FreeBSD-11. > > --HPS And don't forget: make clean first. From owner-freebsd-infiniband@freebsd.org Mon Mar 18 15:08:30 2019 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CA42C154341B for ; Mon, 18 Mar 2019 15:08:30 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: from mail-ot1-x331.google.com (mail-ot1-x331.google.com [IPv6:2607:f8b0:4864:20::331]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B48F18CDB7 for ; Mon, 18 Mar 2019 15:08:29 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: by mail-ot1-x331.google.com with SMTP id e80so9592274ote.5 for ; Mon, 18 Mar 2019 08:08:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=rDtAyzSKlbs42u73t+TZ3mZkdajpGmMY7Fx5sEm1ZCw=; b=WU/DfTtnok8dkUK+g7qkNvXYvXobvuoOsKapfdbW6yLxGTLg9CoOWevOG6Auoj1+Is bYlFjPJ6CNKHLYU54+pvMfj8pnZOrS8P7r+muQptTL488xh/13k8pHCKft9slsLkEzip 5AbIvpmioqdupMEVtEpLMmCAIb9ufsAZdSTT/pihUwvwUtZT9jrOpkrqqwkEmgjkAIc5 hZ4GOTYeKyYxQNcWRlF5KwwDm35G/vA26ejZ8cpmdhLRohKCTF4nNSJ9baUhE3fqkeNa swxtEyqsThChIpZ3qVmtlwnlP6jBEPNQKcPiO1IwHOvCe03/ZqhAGkv9fMXVisgVTuTR 4+TA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=rDtAyzSKlbs42u73t+TZ3mZkdajpGmMY7Fx5sEm1ZCw=; b=kyQvYURdEYCflFpBVCvy73jzx5ttHa25zm20Wwu7a76Cchs6cjt3AoRgGUZZhcH82u 6V6A+tzvDGVBt/yzk/NWa2f7OPxNAeGl9bFFjIN1za4ot+sxmVnUlwQx/L2Sk2QnBLTU QFa8Yv5H18yddrN4gyKzujrv0hRECab0n1chDTC0nTK93LqWmWd8l7lXo+p5DbTbfGkZ Y/YeOXNU/taUPHdnAS9lCwtvtfDMWaSQYkAd5Lk2EuUq739mcJJWJgz6jWOiuwd0ilqQ pLjf6S+U2ElkXcmKySf9r5Zkp7Ja4zv0bvHjewSSI20S5HZZCANgGWFKhIqSev5NOen5 1nAQ== X-Gm-Message-State: APjAAAWHXl3re+jwii39auVYGgcFHj8Xz+TGByhLQLbrn4gexa8qsYhD kPlLd24kSgNyMU2zgPCHEs0vVwGJ X-Google-Smtp-Source: APXvYqw7Bv43XVwKz38EQxSmSWslH7+ozmNb0y22mmG2BFTXsJpFn9lrFxNYELWWJ58x3OS9xhnSXA== X-Received: by 2002:a9d:5f15:: with SMTP id f21mr10275914oti.212.1552921707098; Mon, 18 Mar 2019 08:08:27 -0700 (PDT) Received: from cray.acadix.biz (cpe-174-102-163-140.wi.res.rr.com. [174.102.163.140]) by smtp.gmail.com with ESMTPSA id a20sm4678728oih.57.2019.03.18.08.08.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 18 Mar 2019 08:08:26 -0700 (PDT) Subject: Re: Kernel modules To: Hans Petter Selasky , "freebsd-infiniband@freebsd.org" References: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> <598a58f0-89b8-d00d-5ed7-74dd7005950f@gmail.com> <73ce0738-4d63-2f25-2ff6-00f0092de136@selasky.org> From: Jason Bacon Message-ID: <2090dd24-db43-b689-4289-f50bd70090ea@gmail.com> Date: Mon, 18 Mar 2019 10:08:25 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.5.3 MIME-Version: 1.0 In-Reply-To: <73ce0738-4d63-2f25-2ff6-00f0092de136@selasky.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: B48F18CDB7 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=WU/DfTtn; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of bacon4000@gmail.com designates 2607:f8b0:4864:20::331 as permitted sender) smtp.mailfrom=bacon4000@gmail.com X-Spamd-Result: default: False [-6.70 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.99)[-0.991,0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-infiniband@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[1.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.70)[ip: (-8.59), ipnet: 2607:f8b0::/32(-2.75), asn: 15169(-2.09), country: US(-0.07)] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2019 15:08:31 -0000 On 3/18/19 9:10 AM, Hans Petter Selasky wrote: > On 3/18/19 2:56 PM, Jason Bacon wrote: >> On 3/18/19 7:55 AM, Hans Petter Selasky wrote: >>> On 3/18/19 1:32 PM, Jason Bacon wrote: >>>> >>>> Anyone using the kernel modules lately? >>>> >>>> After building per the instructions on the wiki >>>> (https://wiki.freebsd.org/InfiniBand): >>>> >>>> <<>> /home/bacon 1005 # kldload ibcore >>>> kldload: an error occurred while loading module ibcore. Please >>>> check dmesg(8) for more details. >>>> >>>> dmesg: >>>> >>>> link_elf_obj: symbol if_index undefined >>>> linker_load_file: /boot/modules/ibcore.ko - unsupported file type >>> >>> Hi, >>> >>> You built the KMODs out of the kernel? Try adding >>> DEBUG_FLAGS="-DVIMAGE=1" to your out of tree build. >>> >>> --HPS >> >> I used the script for building kernel modules (which I wrote and >> posted at https://wiki.freebsd.org/InfiniBand a few years ago, based >> on advice from a Mellanox developer at the time).  This will show >> exactly what I did. >> > > Can you try: > > make DEBUG_FLAGS="-DVIMAGE=1" > > instead of > > make > > in the script. In FreeBSD-12 VIMAGE was turned on by default unlike > FreeBSD-11. > > --HPS > That worked, thanks.  I'll update the wiki after some testing. ... objcopy --only-keep-debug ipoib.ko.full ipoib.ko.debug objcopy --strip-debug --add-gnu-debuglink=ipoib.ko.debug ipoib.ko.full ipoib.ko install -T release -o root -g wheel -m 555   ipoib.ko /boot/modules/ kldxref /boot/modules Id Refs Address                Size Name  1   72 0xffffffff80200000  243cd00 kernel  2    1 0xffffffff8263e000     8b50 ng_ubt.ko  3    2 0xffffffff82647000    12dd0 ng_hci.ko  4    2 0xffffffff8265a000     2d00 ng_bluetooth.ko  5    3 0xffffffff8265d000    175d8 netgraph.ko  7    5 0xffffffff82688000    279b0 linuxkpi.ko  9    1 0xffffffff8274d000   3a9a10 zfs.ko 10    2 0xffffffff82af7000     a4f0 opensolaris.ko 11    1 0xffffffff82b02000    20af0 firewire.ko 12    1 0xffffffff82b23000     bf70 cuse.ko 13    2 0xffffffff82b2f000    665e0 mlx4.ko 15    1 0xffffffff82bd0000     5fb8 uplcom.ko 16    2 0xffffffff82bd6000     ab88 ucom.ko 17    1 0xffffffff82e21000     1a20 fdescfs.ko 18    1 0xffffffff82e23000     490c linprocfs.ko 19    3 0xffffffff82e28000     2e28 linux_common.ko 20    1 0xffffffff82e2b000     81f0 tmpfs.ko 21    1 0xffffffff82e34000     1800 uhid.ko 22    1 0xffffffff82e36000     23a8 ums.ko 23    1 0xffffffff82e39000    39960 linux.ko 24    1 0xffffffff82e73000    33c50 linux64.ko 25    1 0xffffffff82ea7000      acf mac_ntpd.ko 26    1 0xffffffff82ea8000     27b0 nullfs.ko 33    3 0xffffffff82eab000    4f2d8 ibcore.ko 34    1 0xffffffff82efb000    1fd39 mlx4ib.ko 35    1 0xffffffff82f1b000     85a0 ipoib.ko -- Earth is a beta site. From owner-freebsd-infiniband@freebsd.org Mon Mar 18 15:54:21 2019 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 35FC81544C27 for ; Mon, 18 Mar 2019 15:54:21 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: from mail-ot1-x344.google.com (mail-ot1-x344.google.com [IPv6:2607:f8b0:4864:20::344]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4112C8EED1 for ; Mon, 18 Mar 2019 15:54:20 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: by mail-ot1-x344.google.com with SMTP id q24so14864795otk.0 for ; Mon, 18 Mar 2019 08:54:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:references:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=hwpp7LIfVA+Zcfgvl3v803ZhblZvUrnaT6CeDbsEObw=; b=mwkvM7bw59+MUGyFEVtJV16VLjh0AIcAZUFhEPNqf4M7Ff+X1W1Tz5+9D+HFYrglDk FrTWK+RY5S8mOfyNoZEqrxSN6TthbzjNzbF0MWIszae9QUaQW4JlWPCxZbuBwBG9LwUW Je/KTdmERvFzZea4X1r9gcgZ1GAlRJOSliU+sfcbdpry42xVXUZdHgozQjnTkGkL4V/p kfIvl4CoA2kBfSQxI9EBhMMUTkMtx84ZWGxPDJaPjNGkpt3+IG2LBZsoEHge6VBza87i xoAHUlCfxsgPcSVCwczrchTL2iDEiQVbvPfHEM+0M54iGHoOrr/XM6oHIk0wG6v5/XlN esng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=hwpp7LIfVA+Zcfgvl3v803ZhblZvUrnaT6CeDbsEObw=; b=RsQeLDo29xDu3skHiSMevrEzof5okj4Z1hAXcJAH0j6mOMHkS0bn3uuQOcU8r12lXa y06nZtzRpFk9v2J8mGwH+IYyL1iKlz3eX8EP+hSd19oE+keIHwwPbYcdLeR8gR51r/9N uPt5mtR1u1cqnvfHIMWWwwNvB6V8bop85YhX7rl4pkbd5wJizEIoK8j5HjgalsrxbaEM 8i48HogzeP78VrdMlRSax81r7IA9/NJAkdq8Ag2pgZYkN1XuVrrFVHAXBqfu6e5KbQmo 8OFANnTllxEvLDqmki3HeYn7naJ5jLF6znRaFXH8lMcuKFeF+F6auAZto+SF3dKMB6+h G5kg== X-Gm-Message-State: APjAAAUXbqsKiXadEajoSMOz29ogdHvTn0N8TGCfLjO8vnRT37z48nnn 1CzIPLXPf+v1JChnIH3kbJSwXx8j X-Google-Smtp-Source: APXvYqzBpFs01rs3NOyYkcM4blqsTsDixOou1iKZkQGw8mLPiQjA6KV5Qr9Di/E6MAVgQ8MFHV3IzA== X-Received: by 2002:a9d:6c54:: with SMTP id g20mr11755815otq.4.1552924459119; Mon, 18 Mar 2019 08:54:19 -0700 (PDT) Received: from cray.acadix.biz (cpe-174-102-163-140.wi.res.rr.com. [174.102.163.140]) by smtp.gmail.com with ESMTPSA id c16sm4028737otl.70.2019.03.18.08.54.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 18 Mar 2019 08:54:17 -0700 (PDT) Subject: Re: Kernel modules From: Jason Bacon To: Hans Petter Selasky , "freebsd-infiniband@freebsd.org" References: <0eba9ec9-692f-7677-2b10-4e67a232821c@gmail.com> <598a58f0-89b8-d00d-5ed7-74dd7005950f@gmail.com> <73ce0738-4d63-2f25-2ff6-00f0092de136@selasky.org> <2090dd24-db43-b689-4289-f50bd70090ea@gmail.com> Message-ID: <6673df26-8bba-ebd3-b2c5-d7e9c97db557@gmail.com> Date: Mon, 18 Mar 2019 10:54:16 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.5.3 MIME-Version: 1.0 In-Reply-To: <2090dd24-db43-b689-4289-f50bd70090ea@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4112C8EED1 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=mwkvM7bw; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of bacon4000@gmail.com designates 2607:f8b0:4864:20::344 as permitted sender) smtp.mailfrom=bacon4000@gmail.com X-Spamd-Result: default: False [-4.43 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.93)[-0.925,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-infiniband@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[4.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.49)[ip: (2.45), ipnet: 2607:f8b0::/32(-2.75), asn: 15169(-2.09), country: US(-0.07)] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2019 15:54:21 -0000 On 3/18/19 10:08 AM, Jason Bacon wrote: > On 3/18/19 9:10 AM, Hans Petter Selasky wrote: >> On 3/18/19 2:56 PM, Jason Bacon wrote: >>> On 3/18/19 7:55 AM, Hans Petter Selasky wrote: >>>> On 3/18/19 1:32 PM, Jason Bacon wrote: >>>>> >>>>> Anyone using the kernel modules lately? >>>>> >>>>> After building per the instructions on the wiki >>>>> (https://wiki.freebsd.org/InfiniBand): >>>>> >>>>> <<>> /home/bacon 1005 # kldload ibcore >>>>> kldload: an error occurred while loading module ibcore. Please >>>>> check dmesg(8) for more details. >>>>> >>>>> dmesg: >>>>> >>>>> link_elf_obj: symbol if_index undefined >>>>> linker_load_file: /boot/modules/ibcore.ko - unsupported file type >>>> >>>> Hi, >>>> >>>> You built the KMODs out of the kernel? Try adding >>>> DEBUG_FLAGS="-DVIMAGE=1" to your out of tree build. >>>> >>>> --HPS >>> >>> I used the script for building kernel modules (which I wrote and >>> posted at https://wiki.freebsd.org/InfiniBand a few years ago, based >>> on advice from a Mellanox developer at the time). This will show >>> exactly what I did. >>> >> >> Can you try: >> >> make DEBUG_FLAGS="-DVIMAGE=1" >> >> instead of >> >> make >> >> in the script. In FreeBSD-12 VIMAGE was turned on by default unlike >> FreeBSD-11. >> >> --HPS >> > > That worked, thanks.  I'll update the wiki after some testing. > > ... > objcopy --only-keep-debug ipoib.ko.full ipoib.ko.debug > objcopy --strip-debug --add-gnu-debuglink=ipoib.ko.debug ipoib.ko.full > ipoib.ko > install -T release -o root -g wheel -m 555   ipoib.ko /boot/modules/ > kldxref /boot/modules > Id Refs Address                Size Name >  1   72 0xffffffff80200000  243cd00 kernel >  2    1 0xffffffff8263e000     8b50 ng_ubt.ko >  3    2 0xffffffff82647000    12dd0 ng_hci.ko >  4    2 0xffffffff8265a000     2d00 ng_bluetooth.ko >  5    3 0xffffffff8265d000    175d8 netgraph.ko >  7    5 0xffffffff82688000    279b0 linuxkpi.ko >  9    1 0xffffffff8274d000   3a9a10 zfs.ko > 10    2 0xffffffff82af7000     a4f0 opensolaris.ko > 11    1 0xffffffff82b02000    20af0 firewire.ko > 12    1 0xffffffff82b23000     bf70 cuse.ko > 13    2 0xffffffff82b2f000    665e0 mlx4.ko > 15    1 0xffffffff82bd0000     5fb8 uplcom.ko > 16    2 0xffffffff82bd6000     ab88 ucom.ko > 17    1 0xffffffff82e21000     1a20 fdescfs.ko > 18    1 0xffffffff82e23000     490c linprocfs.ko > 19    3 0xffffffff82e28000     2e28 linux_common.ko > 20    1 0xffffffff82e2b000     81f0 tmpfs.ko > 21    1 0xffffffff82e34000     1800 uhid.ko > 22    1 0xffffffff82e36000     23a8 ums.ko > 23    1 0xffffffff82e39000    39960 linux.ko > 24    1 0xffffffff82e73000    33c50 linux64.ko > 25    1 0xffffffff82ea7000      acf mac_ntpd.ko > 26    1 0xffffffff82ea8000     27b0 nullfs.ko > 33    3 0xffffffff82eab000    4f2d8 ibcore.ko > 34    1 0xffffffff82efb000    1fd39 mlx4ib.ko > 35    1 0xffffffff82f1b000     85a0 ipoib.ko > So ib0 came up immediately after configuring in rc.conf: ifconfig_ib0="inet 192.168.129.17 netmask 255.255.128.0" Next issue is poor performance shown by iperf: root@zfs-01:~ # iperf -c raid-01-hpc ------------------------------------------------------------ Client connecting to raid-01-hpc, TCP port 5001 TCP window size: 33.3 KByte (default) ------------------------------------------------------------ [  3] local 192.168.129.17 port 28021 connected with 192.168.129.1 port 5001 [ ID] Interval       Transfer     Bandwidth [  3]  0.0-10.1 sec   127 MBytes   106 Mbits/sec raid-01 is a CentOS 7 server. Performance between two CentOS servers is more reasonable: [root@raid-02.mortimer ~] 133: iperf -c raid-01-hpc ------------------------------------------------------------ Client connecting to raid-01-hpc, TCP port 5001 TCP window size: 2.50 MByte (default) ------------------------------------------------------------ [  3] local 192.168.129.2 port 57952 connected with 192.168.129.1 port 5001 [ ID] Interval       Transfer     Bandwidth [  3]  0.0-10.0 sec  24.1 GBytes  20.7 Gbits/sec Ultimately I aim to use NFS over IB here, but I think iperf is telling me that ipoib needs to be tuned. Might this be connected vs datagram issue?  If so, how to change it when using loadable modules? The wiki only mentions building IPOIB_CM into the kernel. If we can figure this out, I'll fully document everything on the wiki, of course. Thanks,     JB -- Earth is a beta site.