From owner-freebsd-infiniband@FreeBSD.ORG Sun Dec 28 00:43:28 2014 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8017E63B for ; Sun, 28 Dec 2014 00:43:28 +0000 (UTC) Received: from mail-pa0-x22b.google.com (mail-pa0-x22b.google.com [IPv6:2607:f8b0:400e:c03::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B0E4199D for ; Sun, 28 Dec 2014 00:43:28 +0000 (UTC) Received: by mail-pa0-f43.google.com with SMTP id kx10so15184842pab.16 for ; Sat, 27 Dec 2014 16:43:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dpdtech.com; s=google; h=from:content-type:subject:date:message-id:to:mime-version; bh=IjEJ9Du8w/opEZmumTiGJkst0f44uL0HwV9x7SZ1f0I=; b=embGBrq3OjQi0xx6wVXJDie1o573b/KVWmU2MWNnxVJzkNeEppAeiTVxj6qEFQrtwy IjEo0k5jzJYQd2dQ3tL6/1pA8ebOrT3kY4r0HapKJtxGXmDBOl2JuMqFfVvLmYGfEg/0 LR0tgN/FJnbl+dh2jO+zVLLrI71RbK+eyrXm0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:subject:date:message-id:to :mime-version; bh=IjEJ9Du8w/opEZmumTiGJkst0f44uL0HwV9x7SZ1f0I=; b=ld2RDtkEHrXchAieV4DNReVrck4wo572Pd9rht+tUapbfKeKmopbTsnMSvMOS563hT G6wGiuzV3Fz4AEYoKnoMFiqWeNM9SkOljcFzeOSeU+fWyGGHHNdR+4BpqNf61RbC0HMn ziZ3O466Ha5gddXjXVDgh/Fitvtsm7tEqpUVvvXQnDYbgtBQXJ82OT48aqzORhzQQfI6 kwCqmALZM9fyuoXqhmqh5yKdUj4rC9xsRIpHptJbfKq3/UGQ2Joq/MOcnAJiqRGZVzmb gTwovuDgh9XnjBN3Ubt/Ai2fbf/UwxQbkStKI2fy3EcVXAMBEiw9BgVUHW7C0byjs8WZ jjKg== X-Gm-Message-State: ALoCoQlYWZWxLymPwEiuRUj1JrwVfF3mfH5qaBPJdqrMJmZ54nYtlQwdI8ZtMYqwN7kVBsOtlH0D X-Received: by 10.66.184.48 with SMTP id er16mr77988521pac.61.1419727407656; Sat, 27 Dec 2014 16:43:27 -0800 (PST) Received: from 173-13-188-41-sfba.hfc.comcastbusiness.net (173-13-188-41-sfba.hfc.comcastbusiness.net. [173.13.188.41]) by mx.google.com with ESMTPSA id jf2sm14284516pbb.96.2014.12.27.16.43.26 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 27 Dec 2014 16:43:26 -0800 (PST) From: "David P. Discher" X-Pgp-Agent: GPGMail 2.5b3 Content-Type: multipart/signed; boundary="Apple-Mail=_EE4F8816-B7C3-4E84-B92D-B8D826AF8202"; protocol="application/pgp-signature"; micalg=pgp-sha512 Subject: FreeBSD/mthca (Mellanox MT25218 InfiniHostEx ) [poor] Performance, Firmware Upgrade, and SDP Date: Sat, 27 Dec 2014 16:43:24 -0800 Message-Id: To: freebsd-infiniband@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) X-Mailer: Apple Mail (2.1993) X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Dec 2014 00:43:28 -0000 --Apple-Mail=_EE4F8816-B7C3-4E84-B92D-B8D826AF8202 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 On the performance side for Oded=E2=80=99s reply might be the answer to = this : - = https://lists.freebsd.org/pipermail/freebsd-infiniband/2014-December/00012= 4.html This hardware, especially some of the lower end cards like the MT25218 - = are trickling into the used marked, and are really attractively priced = for some small clusters (or in my case storage + Xen/Hypervisor) setups. = So, I=E2=80=99d love to see this code updated, even if its not = upstreamed right away, I=E2=80=99d be happy to do some testing with some = patches to -head or 10-stable. Using both 10-stable and -head, with IPoIB - I can=E2=80=99t get more = than 1.3 Gbps (in one direction). Even bringing up the second port, = doesn=E2=80=99t increase bandwidth. Even though the IPoIB is old, what = I=E2=80=99ve notice is that I=E2=80=99m maxing the CPU for one of the = three MSI-X IRQs (IRQ 260, the first one, which I think is = MTHCA_EQ_COMP, if they are allocated in order ). dmesg + boot verbose: ib_mthca0: HCA FW version 5.2.916 is old (5.3.000 is current). ib_mthca0: If you have problems, try updating your HCA FW. ib_mthca0: attempting to allocate 3 MSI-X vectors (32 supported) msi: routing MSI-X IRQ 260 to local APIC 0 vector 62 msi: routing MSI-X IRQ 261 to local APIC 0 vector 63 msi: routing MSI-X IRQ 262 to local APIC 0 vector 64 ib_mthca0: using IRQs 260-262 for MSI-X I assume this has nothing to really with the IPoIB part, as much as = interrupt handling in the hardware driver. Looking at the code, it = looks like the MSI-X vectors are hard-coded and can=E2=80=99t be = increased, event thought it seems like the card can support up to 32. = So, I think I know the answer, which is no, but can the number of MSI-X = vectors be increased, so the interrupt load can be spread over more CPUs = ? 2) Firmware upgrade for the HCA. I got all the tools from Mellanox, = however the syntax/directions for using flint says to point it to a = non-exisiting /dev path in FreeBSD. Can one use FreeBSD to flash the = firmware ? I could use some help getting the tools to talk to the card. and 3) SDP - Sockets Direct Protocol - while I=E2=80=99m waiting for = other things, I=E2=80=99d like to play with SDP. Quick googling : - http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html > env LD_PRELOAD=3D/usr/lib/libibsdp.so iperf -s /usr/lib/libibsdp.so: Undefined symbol = =E2=80=9Cprogram_invocation_short_name=E2=80=9D However, doing this in FreeBSD, gives me a missing symbol. =46rom = searching around, program_invocation_short_name, is a gcc-libc thing. = I=E2=80=99ve searched around, can=E2=80=99t seem to find an easy way to = work around this. Any suggestions ? Thanks ! - David P. Discher http://davidpdischer.com/ AIM: DavidDPD | Y!M: daviddpdz --Apple-Mail=_EE4F8816-B7C3-4E84-B92D-B8D826AF8202 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUn1ItAAoJEEmwU6XuhYWOyVYIALA1ETjN4F1inbQz2h/LvyQN ZsPKbH8VLgbAG9r3iKos7aGDy7tIIpWOK6myaQ2nI6adyYI2VXMegKLrUzCg1ess J/l13jlqCbKRtrAFiKKQS4p/mL1OxnKWK8wbLHnbbt6X+CNH/AnUenbj+rqMYp4t 6aMFXPYHj4piFjT1qCH1872vNm0DWF888S0r+164oBcUJgLeTL/jIZTL0Mfv2p2V /pZYgxsVnefyC3k0lBJ1JlSE7teaL40MQNbD+3geWIYlAfW72phzvUzUk91mW191 GBAPJa45YlN+XzV+U2U5jJtPxSlZBrDCTNZyUkRyM0rVeniwGsgj3q8w2hcicZM= =Ns9v -----END PGP SIGNATURE----- --Apple-Mail=_EE4F8816-B7C3-4E84-B92D-B8D826AF8202--