From owner-freebsd-infiniband@FreeBSD.ORG Sun Jun 9 11:11:37 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A03F334A for ; Sun, 9 Jun 2013 11:11:37 +0000 (UTC) (envelope-from alexl@mellanox.com) Received: from eu1sys200aog125.obsmtp.com (eu1sys200aog125.obsmtp.com [207.126.144.159]) by mx1.freebsd.org (Postfix) with ESMTP id 9FE8C1183 for ; Sun, 9 Jun 2013 11:11:36 +0000 (UTC) Received: from MTLCAS02.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob125.postini.com ([207.126.147.11]) with SMTP ID DSNKUbRiyiZhg82wr5Y2XtSKd6yAVsz80/oq@postini.com; Sun, 09 Jun 2013 11:11:36 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS02.mtl.com ([10.0.8.72]) with mapi id 14.03.0123.003; Sun, 9 Jun 2013 14:11:04 +0300 From: Alex Liptsin To: "freebsd-infiniband@freebsd.org" Subject: How to switch Connected and Datagram IPoIB modes - FreeBSD 9.1 Thread-Topic: How to switch Connected and Datagram IPoIB modes - FreeBSD 9.1 Thread-Index: Ac5lAgVeFGRv6msgSj2jIIfNtWo2BQ== Date: Sun, 9 Jun 2013 11:11:02 +0000 Message-ID: <64DAB3164E410447932305F50F896D8D6AF6D25C@MTLDAG01.mtl.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jun 2013 11:11:37 -0000 Hello. Can I edit IPOIB_CM option manually, during the runtime or I need to compil= e the Kernel each time I what to switch UD an CM mode? options IPOIB_CM # Use connect mode ipoib Regards, Alex Liptsin Software Quality Assurance Engineer | Mellanox Technologies Ltd. Office: +972 (74) 7236141 Mobile: +972(54) 7833986 Fax: +972(74) 7236161 Email: alexl@mellanox.com Mellanox, Tel-Hai Industrial Park. Building 7, M.P. Upper Galilee 12100 Isr= ael From owner-freebsd-infiniband@FreeBSD.ORG Sun Jun 9 12:37:10 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 904F25E1; Sun, 9 Jun 2013 12:37:10 +0000 (UTC) (envelope-from alexl@mellanox.com) Received: from eu1sys200aog120.obsmtp.com (eu1sys200aog120.obsmtp.com [207.126.144.149]) by mx1.freebsd.org (Postfix) with ESMTP id 18AD01703; Sun, 9 Jun 2013 12:37:08 +0000 (UTC) Received: from MTLCAS01.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob120.postini.com ([207.126.147.11]) with SMTP ID DSNKUbR20hCLTG9M1dfNz2Vdk3nUEuY5eO+T@postini.com; Sun, 09 Jun 2013 12:37:09 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS01.mtl.com ([10.0.8.71]) with mapi id 14.03.0123.003; Sun, 9 Jun 2013 15:35:33 +0300 From: Alex Liptsin To: "freebsd-infiniband@freebsd.org" , "freebsd-net@freebsd.org" , "freebsd-questions@freebsd.org" Subject: Mellanox NIC names changed, each kldunload/kldload mlx4ib module Thread-Topic: Mellanox NIC names changed, each kldunload/kldload mlx4ib module Thread-Index: Ac5lDdVKtrqzpa1jQyWp4SCSJfBUVA== Date: Sun, 9 Jun 2013 12:35:32 +0000 Message-ID: <64DAB3164E410447932305F50F896D8D6AF6D2E6@MTLDAG01.mtl.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Regev Lev X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jun 2013 12:37:10 -0000 Hi. I work with FreeBSD9.1 and Mellanox devices. Every time I unload / load mlx4ib module, NIC names of mellanox devices (ib= X) are renamed. Can I prevent it? [root@h-qa-032 mlx4]# ifconfig ib8: flags=3D8002 metric 0 mtu 65520 options=3D80018 lladdr 80.28.0.48.fe.80.0.0.0.0.0.0.0.2.c9.3.0.2e.48.31 nd6 options=3D29 ib9: flags=3D8002 metric 0 mtu 65520 options=3D80018 lladdr 80.28.0.49.fe.80.0.0.0.0.0.0.0.2.c9.3.0.2e.48.32 nd6 options=3D29 [root@h-qa-032 mlx4]# kldunload mlx4ib [root@h-qa-032 mlx4]# kldload -v mlx4ib Loaded mlx4ib, id=3D9 [root@h-qa-032 mlx4]# ifconfig ib10: flags=3D8002 metric 0 mtu 65520 options=3D80018 lladdr 80.30.0.48.fe.80.0.0.0.0.0.0.0.2.c9.3.0.2e.48.31 nd6 options=3D29 ib11: flags=3D8002 metric 0 mtu 65520 options=3D80018 lladdr 80.30.0.49.fe.80.0.0.0.0.0.0.0.2.c9.3.0.2e.48.32 nd6 options=3D29 Regards, Alex Liptsin Software Quality Assurance Engineer | Mellanox Technologies Ltd. Office: +972 (74) 7236141 Mobile: +972(54) 7833986 Fax: +972(74) 7236161 Email: alexl@mellanox.com Mellanox, Tel-Hai Industrial Park. Building 7, M.P. Upper Galilee 12100 Isr= ael From owner-freebsd-infiniband@FreeBSD.ORG Sun Jun 9 18:46:08 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6F98E64C; Sun, 9 Jun 2013 18:46:08 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pb0-x234.google.com (mail-pb0-x234.google.com [IPv6:2607:f8b0:400e:c01::234]) by mx1.freebsd.org (Postfix) with ESMTP id 3FD1F19BC; Sun, 9 Jun 2013 18:46:08 +0000 (UTC) Received: by mail-pb0-f52.google.com with SMTP id xa12so6502298pbc.11 for ; Sun, 09 Jun 2013 11:46:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=nNjUAJmFn79l1m0FsW6u8rOTczMdqUPfMsNzPEsOZmE=; b=vhhia6nYBPKBUbnU1uZqfO04huYgVMAe/oCCtzU4TaU4qxf+d0ofsVmhcGI4Zq39C9 TyOyLFMk1QaYeeKZa8OBNWKaKKVwUgqBGhsBUK+EUpf7UbPpZKQMzvyohvdCev7o45xQ qdCX33x96OqMBjczSD5p94RsZh1hvjExvu95kIADMg2trD505e9cNKxi3xhkgFtob7A7 5EmRAun0eVO+m1cw0GtPfKnqZFnu3kTaJ4wPYCxU+2mz1WA7ZQp8NlpF3sb8bC3SYAq3 lXU8VmYDnC5YNdxIjTdErdzHtZTtBnLTWDlZsSB1nS4fKxf4oRyAqzYEYzs4/WpFqHbE g56A== X-Received: by 10.66.17.137 with SMTP id o9mr11129956pad.142.1370803568035; Sun, 09 Jun 2013 11:46:08 -0700 (PDT) Received: from [192.168.20.5] (c-98-203-241-95.hsd1.wa.comcast.net. [98.203.241.95]) by mx.google.com with ESMTPSA id p2sm12252558pag.22.2013.06.09.11.46.06 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 09 Jun 2013 11:46:07 -0700 (PDT) Subject: Re: Mellanox NIC names changed, each kldunload/kldload mlx4ib module Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Garrett Cooper In-Reply-To: <64DAB3164E410447932305F50F896D8D6AF6D2E6@MTLDAG01.mtl.com> Date: Sun, 9 Jun 2013 11:46:04 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <03D4440E-2D1D-497C-B084-CAB47DE5F880@gmail.com> References: <64DAB3164E410447932305F50F896D8D6AF6D2E6@MTLDAG01.mtl.com> To: Alex Liptsin X-Mailer: Apple Mail (2.1283) Cc: "freebsd-infiniband@freebsd.org" , Regev Lev , "freebsd-questions@freebsd.org" , "freebsd-net@freebsd.org" X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jun 2013 18:46:08 -0000 On Jun 9, 2013, at 5:35 AM, Alex Liptsin wrote: > Hi. >=20 > I work with FreeBSD9.1 and Mellanox devices. > Every time I unload / load mlx4ib module, NIC names of mellanox = devices (ibX) are renamed. > Can I prevent it? >=20 > [root@h-qa-032 mlx4]# ifconfig > ib8: flags=3D8002 metric 0 mtu 65520 > options=3D80018 > lladdr 80.28.0.48.fe.80.0.0.0.0.0.0.0.2.c9.3.0.2e.48.31 > nd6 options=3D29 > ib9: flags=3D8002 metric 0 mtu 65520 > options=3D80018 > lladdr 80.28.0.49.fe.80.0.0.0.0.0.0.0.2.c9.3.0.2e.48.32 > nd6 options=3D29 >=20 > [root@h-qa-032 mlx4]# kldunload mlx4ib >=20 > [root@h-qa-032 mlx4]# kldload -v mlx4ib > Loaded mlx4ib, id=3D9 >=20 > [root@h-qa-032 mlx4]# ifconfig > ib10: flags=3D8002 metric 0 mtu 65520 > options=3D80018 > lladdr 80.30.0.48.fe.80.0.0.0.0.0.0.0.2.c9.3.0.2e.48.31 > nd6 options=3D29 > ib11: flags=3D8002 metric 0 mtu 65520 > options=3D80018 > lladdr 80.30.0.49.fe.80.0.0.0.0.0.0.0.2.c9.3.0.2e.48.32 > nd6 options=3D29 You're probably running into a driver bug because OFED/ipoib was = never meant to be unloaded (assuming you're using my sources). Check the = detach/destroy routines to make sure that it's properly detaching = everything and updating indexes in the network stack before it unloads = the driver. Cheers, -Garrett From owner-freebsd-infiniband@FreeBSD.ORG Sun Jun 9 18:51:01 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 66E7C24B for ; Sun, 9 Jun 2013 18:51:01 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-bk0-x235.google.com (mail-bk0-x235.google.com [IPv6:2a00:1450:4008:c01::235]) by mx1.freebsd.org (Postfix) with ESMTP id C716B1A31 for ; Sun, 9 Jun 2013 18:50:59 +0000 (UTC) Received: by mail-bk0-f53.google.com with SMTP id e11so2382512bkh.12 for ; Sun, 09 Jun 2013 11:50:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=W8tTRXpc07s9hi+zFkfO6rXxjtQhO+JPwkxj1pc40ds=; b=yIFwcUJHpGGuwxIbnjuJ0g1r3y88jq88/TN0sptbGpUgog4hCFaXe/IsUJqs5MLIm9 cHXf+/wSoa/R/zKjOhPwQv7G8qktPARhi1cp6hjxaMiF7fbRvWEIu6g2APnaoeJTnOaD OX4oaMBY9sTgeMgxaMjQgG92PaiS/BR8XI3D41knqMH94P1OYBUIp2/abnJU1EKGJC0l PsmH8CHFdo5wvZBKn/R4lvIi63yNFBSKcbptwEZteFkw8cs9aNpphjsxS/GybHrnG1li c74ufQfYd0SEpFDL4AGEQNg0ppn4c6bQ2HRd7WyFpLiU0Rjn4mTFKpnP/IkivdxUsWZk /Xiw== MIME-Version: 1.0 X-Received: by 10.204.173.9 with SMTP id n9mr1049252bkz.47.1370803858974; Sun, 09 Jun 2013 11:50:58 -0700 (PDT) Received: by 10.204.240.144 with HTTP; Sun, 9 Jun 2013 11:50:58 -0700 (PDT) In-Reply-To: <64DAB3164E410447932305F50F896D8D6AF6D25C@MTLDAG01.mtl.com> References: <64DAB3164E410447932305F50F896D8D6AF6D25C@MTLDAG01.mtl.com> Date: Sun, 9 Jun 2013 11:50:58 -0700 Message-ID: Subject: Re: How to switch Connected and Datagram IPoIB modes - FreeBSD 9.1 From: Garrett Cooper To: Alex Liptsin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-infiniband@freebsd.org" X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Jun 2013 18:51:01 -0000 On Sun, Jun 9, 2013 at 4:11 AM, Alex Liptsin wrote: > Hello. > > Can I edit IPOIB_CM option manually, during the runtime or I need to > compile the Kernel each time I what to switch UD an CM mode? > > options IPOIB_CM # Use connect mode ipoib > The way to solve this would be to add either a tunable/sysctl (if you can modify this at runtime), or just a tunable (if you can modify this only at load time). That way you could switch between modes without having to recompile the kernel (and optionally without a reboot if you unload the module and reload it). This ASSUMES that no driver structures or code paths that are being modified in such a way that you _HAVE_ to have this option compiled into the kernel. And yeah.. I'm not an expert in Infiniband -- other people on my team are -- so my recommendation is generic because I haven't read and fully digested the IB spec. Cheers, -Garrett From owner-freebsd-infiniband@FreeBSD.ORG Mon Jun 10 13:41:47 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 544666B3 for ; Mon, 10 Jun 2013 13:41:47 +0000 (UTC) (envelope-from jsteckli@os.inf.tu-dresden.de) Received: from os.inf.tu-dresden.de (os.inf.tu-dresden.de [IPv6:2002:8d4c:3001:48::99]) by mx1.freebsd.org (Postfix) with ESMTP id 19C8418F0 for ; Mon, 10 Jun 2013 13:41:47 +0000 (UTC) Received: from [2002:8d4c:3001:48:ea40:f2ff:fee2:6328] by os.inf.tu-dresden.de with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.80.1) id 1Um2Lx-0001Mo-Ih for freebsd-infiniband@freebsd.org; Mon, 10 Jun 2013 15:41:46 +0200 Message-ID: <51B5D798.5090008@os.inf.tu-dresden.de> Date: Mon, 10 Jun 2013 15:41:44 +0200 From: Julian Stecklina User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130514 Thunderbird/17.0.6 MIME-Version: 1.0 To: "freebsd-infiniband@freebsd.org" Subject: ib1: timing out; N sends not completed X-Enigmail-Version: 1.5.1 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="----enig2XELEWWATAANBQTANMPNP" X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jun 2013 13:41:47 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) ------enig2XELEWWATAANBQTANMPNP Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hello, I have two machines connected back-to-back via Infiniband with Mellanox Infinihost III adapters. One machine runs Linux (Fedora 19) and the other 9-STABLE. I sometimes get: ib1: timing out; 47 sends not completed ib1: timing out; 1 sends not completed ib1: timing out; 56 sends not completed or similar and TCP connections will be stuck after each timeout for a while. It is relatively easy to reproduce this behavior with NetPIPE. Any advice? Julian ------enig2XELEWWATAANBQTANMPNP Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) iEYEARECAAYFAlG115kACgkQ2EtjUdW3H9nLqgCdHu76PgVQQKPPXT9v+c5tR9kS eDMAn3raiW+HU2hudIYDn1eec0lh8520 =rwlV -----END PGP SIGNATURE----- ------enig2XELEWWATAANBQTANMPNP-- From owner-freebsd-infiniband@FreeBSD.ORG Mon Jun 10 16:03:20 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 5DA445CD for ; Mon, 10 Jun 2013 16:03:20 +0000 (UTC) (envelope-from accornehl@gmail.com) Received: from mail-ee0-x232.google.com (mail-ee0-x232.google.com [IPv6:2a00:1450:4013:c00::232]) by mx1.freebsd.org (Postfix) with ESMTP id EC0701081 for ; Mon, 10 Jun 2013 16:03:19 +0000 (UTC) Received: by mail-ee0-f50.google.com with SMTP id d49so3147035eek.9 for ; Mon, 10 Jun 2013 09:03:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=fzMnzuoQNeeQ2wKDs0C0yH0oCz0x2zCP3Q5XyFy/ZLc=; b=lBXbsn/QaCsPBGvpYSIgJczCCkZCEhANnFFW8P7fKwPOnt1C16Qj6JCTGfWRaAvXWf 1Bz3Msfec5Re60Q/lXPC/k9CuKsGzjV7qjsz/75UdAqRBhoDJyfkVcfGHslxaI++qga8 0t0L/TNBO1ykqOaJuSmdz0mk85Wf+JFyvLgkedAMrgkRM8g87MUt42QqEXuTcfsewvK2 1LorlkSKPSo+ysfoisJcFsX+4MAXADVIEEn8i9oWQrjJJ+LlHUzroTQ/8CsZUxGDNZ0y BQ341RNCchgUbBZp96Iy+5olPRrK5aVNJGpIOJPqUL6w60ncMpQyXtOxz4vqU/Loh+93 5NGg== MIME-Version: 1.0 X-Received: by 10.14.69.199 with SMTP id n47mr11915357eed.11.1370880198850; Mon, 10 Jun 2013 09:03:18 -0700 (PDT) Received: by 10.223.77.92 with HTTP; Mon, 10 Jun 2013 09:03:18 -0700 (PDT) Received: by 10.223.77.92 with HTTP; Mon, 10 Jun 2013 09:03:18 -0700 (PDT) In-Reply-To: <51B5D798.5090008@os.inf.tu-dresden.de> References: <51B5D798.5090008@os.inf.tu-dresden.de> Date: Mon, 10 Jun 2013 16:03:18 +0000 Message-ID: Subject: Re: ib1: timing out; N sends not completed From: Anthony Cornehl To: Julian Stecklina Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-infiniband@freebsd.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jun 2013 16:03:20 -0000 On Jun 10, 2013 6:41 AM, "Julian Stecklina" wrote: > > Hello, > > I have two machines connected back-to-back via Infiniband with Mellanox > Infinihost III adapters. One machine runs Linux (Fedora 19) and the > other 9-STABLE. > > I sometimes get: > > ib1: timing out; 47 sends not completed > ib1: timing out; 1 sends not completed > ib1: timing out; 56 sends not completed > > or similar and TCP connections will be stuck after each timeout for a > while. It is relatively easy to reproduce this behavior with NetPIPE. > > Any advice? > > Julian > Hey Julian, Just some questions to try and clarify the issue... - which machine is the OpenSM master running on? - what does your qkey violation count look like when you run a portinfo on the ports? - does the issue persist when a switch is added between the hosts? Cheers! From owner-freebsd-infiniband@FreeBSD.ORG Mon Jun 10 17:05:21 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id C98B42CF for ; Mon, 10 Jun 2013 17:05:21 +0000 (UTC) (envelope-from jsteckli@os.inf.tu-dresden.de) Received: from os.inf.tu-dresden.de (os.inf.tu-dresden.de [IPv6:2002:8d4c:3001:48::99]) by mx1.freebsd.org (Postfix) with ESMTP id 893031620 for ; Mon, 10 Jun 2013 17:05:21 +0000 (UTC) Received: from [2002:8d4c:3001:48:ea40:f2ff:fee2:6328] by os.inf.tu-dresden.de with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.80.1) id 1Um5Wy-0007tp-MF; Mon, 10 Jun 2013 19:05:20 +0200 Message-ID: <51B6074F.3080202@os.inf.tu-dresden.de> Date: Mon, 10 Jun 2013 19:05:19 +0200 From: Julian Stecklina User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130514 Thunderbird/17.0.6 MIME-Version: 1.0 To: Anthony Cornehl Subject: Re: ib1: timing out; N sends not completed References: <51B5D798.5090008@os.inf.tu-dresden.de> In-Reply-To: X-Enigmail-Version: 1.5.1 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="----enig2MPIPBTOGJSXTHFCISFLB" Cc: freebsd-infiniband@freebsd.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Jun 2013 17:05:21 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) ------enig2MPIPBTOGJSXTHFCISFLB Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 06/10/2013 06:03 PM, Anthony Cornehl wrote: >=20 > On Jun 10, 2013 6:41 AM, "Julian Stecklina" > > > wrote: >> >> Hello, >> >> I have two machines connected back-to-back via Infiniband with Mellano= x >> Infinihost III adapters. One machine runs Linux (Fedora 19) and the >> other 9-STABLE. >> >> I sometimes get: >> >> ib1: timing out; 47 sends not completed >> ib1: timing out; 1 sends not completed >> ib1: timing out; 56 sends not completed >> >> or similar and TCP connections will be stuck after each timeout for a >> while. It is relatively easy to reproduce this behavior with NetPIPE. >> >> Any advice? >> >> Julian >> >=20 > Hey Julian, >=20 > Just some questions to try and clarify the issue... >=20 > - which machine is the OpenSM master running on? The Linux box: opensm-3.3.15 > - what does your qkey violation count look like when you run a portinfo= > on the ports? Is there a way to do this from the command line? I can only find the corresponding C function. > - does the issue persist when a switch is added between the hosts? I can't tell you, because I don't have one available right now. Julian ------enig2MPIPBTOGJSXTHFCISFLB Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) iEYEARECAAYFAlG2B08ACgkQ2EtjUdW3H9lHVwCfTRtDs7zS5qYom+mZsnmBG1JV sGkAoM0AGTmjb3MHGmPdqF09eDz5TJ1W =gr5x -----END PGP SIGNATURE----- ------enig2MPIPBTOGJSXTHFCISFLB-- From owner-freebsd-infiniband@FreeBSD.ORG Wed Jun 12 07:06:56 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DFDE46B6; Wed, 12 Jun 2013 07:06:56 +0000 (UTC) (envelope-from alexl@mellanox.com) Received: from eu1sys200aog116.obsmtp.com (eu1sys200aog116.obsmtp.com [207.126.144.141]) by mx1.freebsd.org (Postfix) with ESMTP id 448D6147F; Wed, 12 Jun 2013 07:06:54 +0000 (UTC) Received: from MTLCAS02.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob116.postini.com ([207.126.147.11]) with SMTP ID DSNKUbgd9Hy0VzjaM3Vz/+xju2KgOwOWJ2ql@postini.com; Wed, 12 Jun 2013 07:06:55 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS02.mtl.com ([10.0.8.72]) with mapi id 14.03.0123.003; Wed, 12 Jun 2013 10:06:26 +0300 From: Alex Liptsin To: "freebsd-infiniband@freebsd.org" , "freebsd-net@freebsd.org" , "freebsd-questions@freebsd.org" Subject: Failed to allocate receive buffer problem Thread-Topic: Failed to allocate receive buffer problem Thread-Index: Ac5nO1pW2LKE00ohRUCwnHjcKFM9Mw== Date: Wed, 12 Jun 2013 07:06:26 +0000 Message-ID: <64DAB3164E410447932305F50F896D8D6AF6E2C3@MTLDAG01.mtl.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Regev Lev X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jun 2013 07:06:57 -0000 Hi. I have a problem that when running a ping (or any other traffic) over IPoIB= port, Traffic fails after some time. At destination server DMESG I see that errors: Jun 11 14:42:11 h-qa-033 kernel: ib1: failed to allocate receive buffer 253 Jun 11 14:42:12 h-qa-033 kernel: ib1: failed to allocate receive buffer 254 Jun 11 14:42:13 h-qa-033 kernel: ib1: failed to allocate receive buffer 255 Jun 11 14:42:14 h-qa-033 kernel: ib1: failed to allocate receive buffer 0 Jun 11 14:42:15 h-qa-033 kernel: ib1: failed to allocate receive buffer 1 Jun 11 14:42:16 h-qa-033 kernel: ib1: failed to allocate receive buffer 2 Jun 11 14:42:17 h-qa-033 kernel: ib1: failed to allocate receive buffer 3 Jun 11 14:42:18 h-qa-033 kernel: ib1: failed to allocate receive buffer 4 Jun 11 14:42:19 h-qa-033 kernel: ib1: failed to allocate receive buffer 5 Jun 11 14:42:20 h-qa-033 kernel: ib1: failed to allocate receive buffer 6 Jun 11 14:42:21 h-qa-033 kernel: ib1: failed to allocate receive buffer 7 I work with FreeBSD 9.1. Is it a bug or some configuration issues? Thanks. Regards, Alex Liptsin Software Quality Assurance Engineer | Mellanox Technologies Ltd. Office: +972 (74) 7236141 Mobile: +972(54) 7833986 Fax: +972(74) 7236161 Email: alexl@mellanox.com Mellanox, Tel-Hai Industrial Park. Building 7, M.P. Upper Galilee 12100 Isr= ael