From owner-freebsd-infiniband@freebsd.org Thu Mar 5 16:58:18 2020 Return-Path: Delivered-To: freebsd-infiniband@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EB3D326E5B1 for ; Thu, 5 Mar 2020 16:58:18 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48YH4n3BdTz4FCY; Thu, 5 Mar 2020 16:58:16 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2020.home.selasky.org (unknown [62.141.129.235]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 1A1DA260152; Thu, 5 Mar 2020 17:58:06 +0100 (CET) Subject: Re: [PATCH]: ipoib with mlx4 initialisation ordering To: Andreas Kempe Cc: Konstantin Belousov , Meny Yossefi , freebsd-infiniband@freebsd.org References: <20200222004838.GA22659@moira.hest-guild.se> <9d76992b-6ba4-2419-61ff-5035aa45e597@selasky.org> <20200224194608.GC22659@moira.hest-guild.se> <16883d49-3cc0-d9cc-0877-46f811eeb8f1@selasky.org> <20200226210554.GE22659@moira.hest-guild.se> <20200226213022.GG22659@moira.hest-guild.se> <2226834e-4184-a581-87bb-3b8ce6c184da@selasky.org> <20200226230012.GA6559@moira.hest-guild.se> From: Hans Petter Selasky Message-ID: Date: Thu, 5 Mar 2020 17:56:24 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.4.2 MIME-Version: 1.0 In-Reply-To: <20200226230012.GA6559@moira.hest-guild.se> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 48YH4n3BdTz4FCY X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 88.99.82.50 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-5.37 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.95)[-0.949,0]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[selasky.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-3.12)[ip: (-9.31), ipnet: 88.99.0.0/16(-4.71), asn: 24940(-1.56), country: DE(-0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:88.99.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Mar 2020 16:58:19 -0000 Hi Andreas, Can you try this patch instead: https://reviews.freebsd.org/D23973 The issue is a bit deeper than what you found. --HPS From owner-freebsd-infiniband@freebsd.org Fri Mar 6 11:13:50 2020 Return-Path: Delivered-To: freebsd-infiniband@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BEFEC265809 for ; Fri, 6 Mar 2020 11:13:50 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [IPv6:2a01:4f8:c17:6c4b::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48YlNr72tRz44fT; Fri, 6 Mar 2020 11:13:48 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2020.home.selasky.org (unknown [62.141.129.235]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id C9D6C260108; Fri, 6 Mar 2020 12:13:39 +0100 (CET) From: Hans Petter Selasky Subject: Re: [PATCH]: ipoib with mlx4 initialisation ordering To: Andreas Kempe Cc: Konstantin Belousov , Meny Yossefi , freebsd-infiniband@freebsd.org References: <20200222004838.GA22659@moira.hest-guild.se> <9d76992b-6ba4-2419-61ff-5035aa45e597@selasky.org> <20200224194608.GC22659@moira.hest-guild.se> <16883d49-3cc0-d9cc-0877-46f811eeb8f1@selasky.org> <20200226210554.GE22659@moira.hest-guild.se> <20200226213022.GG22659@moira.hest-guild.se> <2226834e-4184-a581-87bb-3b8ce6c184da@selasky.org> <20200226230012.GA6559@moira.hest-guild.se> Message-ID: Date: Fri, 6 Mar 2020 12:13:06 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.4.2 MIME-Version: 1.0 In-Reply-To: <20200226230012.GA6559@moira.hest-guild.se> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 48YlNr72tRz44fT X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 2a01:4f8:c17:6c4b::2 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-4.92 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.95)[-0.954,0]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[selasky.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-2.67)[ip: (-9.21), ipnet: 2a01:4f8::/29(-2.55), asn: 24940(-1.56), country: DE(-0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Mar 2020 11:13:50 -0000 Hi Andreas, I've just updated https://reviews.freebsd.org/D23973 with some more changes after an internal discussion. Can you give the latest version of the patch a spin? Thank you! --HPS From owner-freebsd-infiniband@freebsd.org Fri Mar 6 20:12:26 2020 Return-Path: Delivered-To: freebsd-infiniband@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 150B726FF21 for ; Fri, 6 Mar 2020 20:12:26 +0000 (UTC) (envelope-from SRS0+HUnA=4X=moira.hest-guild.se=andkem@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48YzLG4NYTz3NGR for ; Fri, 6 Mar 2020 20:12:22 +0000 (UTC) (envelope-from SRS0+HUnA=4X=moira.hest-guild.se=andkem@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id DFDD140017 for ; Fri, 6 Mar 2020 21:12:17 +0100 (CET) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id CD4F840019; Fri, 6 Mar 2020 21:12:17 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=disabled version=3.4.2 X-Spam-Score: 0.0 Received: from moira.hest-guild.se (tunnel534205-pt.tunnel.tserv24.sto1.ipv6.he.net [IPv6:2001:470:27:5eb::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 564DC40017 for ; Fri, 6 Mar 2020 21:12:17 +0100 (CET) Received: from andkem (uid 1000) (envelope-from andkem@moira.hest-guild.se) id 1878f0a0 by moira.hest-guild.se (DragonFly Mail Agent v0.13); Fri, 06 Mar 2020 21:12:16 +0100 Date: Fri, 6 Mar 2020 21:12:16 +0100 From: Andreas Kempe To: Hans Petter Selasky Cc: Konstantin Belousov , Meny Yossefi , freebsd-infiniband@freebsd.org Subject: Re: [PATCH]: ipoib with mlx4 initialisation ordering Message-ID: <20200306201216.GA13373@moira.hest-guild.se> References: <20200222004838.GA22659@moira.hest-guild.se> <9d76992b-6ba4-2419-61ff-5035aa45e597@selasky.org> <20200224194608.GC22659@moira.hest-guild.se> <16883d49-3cc0-d9cc-0877-46f811eeb8f1@selasky.org> <20200226210554.GE22659@moira.hest-guild.se> <20200226213022.GG22659@moira.hest-guild.se> <2226834e-4184-a581-87bb-3b8ce6c184da@selasky.org> <20200226230012.GA6559@moira.hest-guild.se> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="AqsLC8rIMeq19msA" Content-Disposition: inline In-Reply-To: X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 48YzLG4NYTz3NGR X-Spamd-Bar: -------- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of SRS0@lysator.liu.se designates 130.236.254.3 as permitted sender) smtp.mailfrom=SRS0@lysator.liu.se X-Spamd-Result: default: False [-8.00 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; RCVD_TLS_LAST(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-infiniband@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[3.254.236.130.list.dnswl.org : 127.0.11.2]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; IP_SCORE(-3.20)[ip: (-8.32), ipnet: 130.236.0.0/16(-4.26), asn: 2843(-3.41), country: SE(-0.03)]; SIGNED_PGP(-2.00)[]; FORGED_SENDER(0.30)[kempe@lysator.liu.se,SRS0@lysator.liu.se]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:2843, ipnet:130.236.0.0/16, country:SE]; TAGGED_FROM(0.00)[HUnA=4X=moira.hest-guild.se=andkem]; FROM_NEQ_ENVFROM(0.00)[kempe@lysator.liu.se,SRS0@lysator.liu.se] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Mar 2020 20:12:26 -0000 --AqsLC8rIMeq19msA Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Mar 06, 2020 at 12:13:06PM +0100, Hans Petter Selasky wrote: > Hi Andreas, >=20 > I've just updated https://reviews.freebsd.org/D23973 with some more chang= es > after an internal discussion. >=20 > Can you give the latest version of the patch a spin? >=20 Hello Hans Petter, I've tried this latest version of the patch and it solves the issue on our machine. Thank you for taking the time! Cordially, Andreas Kempe --AqsLC8rIMeq19msA Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEETci4cPcl+ZcyiACiCkqKrhcKSD0FAl5irpcACgkQCkqKrhcK SD2cWg/+IzkY3QUJdMBSCXRcbbCmGzdV1PGohfWp4/QznG1BSsBr0+OG/rYOZr// vzMAwGjWTCz10cB1/QFQ/68Z6dbW4TdKOTmCrIYH491gab3ND4+Nh5bqbGH0YY2A Qdjn/kdR1oHFAQU2E4jmliXdk+zSMeeOYKgjCtxO/FsyeT3nKZCeLvqm03W1s6D6 NwBFEn++BRl7ndlqQs4G5BDWHdGZcjmRcD9UKI9BaWaSDw+MoYOfuan6osIl3oh1 dsRjrOSuiEtUOb0Xt6EpHBGc93oVteLeM7Ihp+gJ9IxxErY2n0z3OYjiTDlC+90t Fr0R5U42qUaYNpIO9BmGBjyYKVJwZ84UKdqQeT1V/xvol9cr1v9K6W7FHSE6j/ry mbv0GTeeAtcvho5WI8b10RGsbGdmQERrWkzum7aEM8smKN/zXjDF90PSjUj5rrWm wvyIEa6FbCCH6SQMDeWoFFDixKf/mr9hO9DWVVP23TeX8DcDcmFi0n/QIQVPtz2G dlDUNTwWOCxUaO6xxJrnn+jCgADbxLwYAIkwbD15R5kRHMSbvPNqxrXrztZj71dy 8GHp0mVp5GtB/MiZKv2KVky+ACklyMmI5Pe80v4uWUG37mLnAysDnJ1Qwz9NEebV R76X+4N1dacCVUWuGrWeWIIVtaLsDrAIogb+u6norp23DAEBmnU= =qn6o -----END PGP SIGNATURE----- --AqsLC8rIMeq19msA--