Date: Sat, 27 Dec 2014 16:43:24 -0800 From: "David P. Discher" <dpd@dpdtech.com> To: freebsd-infiniband@freebsd.org Subject: FreeBSD/mthca (Mellanox MT25218 InfiniHostEx ) [poor] Performance, Firmware Upgrade, and SDP Message-ID: <C06A80D7-AE40-4987-9BB3-FC1D20100F9C@dpdtech.com>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On the performance side for Oded’s reply might be the answer to this : - https://lists.freebsd.org/pipermail/freebsd-infiniband/2014-December/000124.html This hardware, especially some of the lower end cards like the MT25218 - are trickling into the used marked, and are really attractively priced for some small clusters (or in my case storage + Xen/Hypervisor) setups. So, I’d love to see this code updated, even if its not upstreamed right away, I’d be happy to do some testing with some patches to -head or 10-stable. Using both 10-stable and -head, with IPoIB - I can’t get more than 1.3 Gbps (in one direction). Even bringing up the second port, doesn’t increase bandwidth. Even though the IPoIB is old, what I’ve notice is that I’m maxing the CPU for one of the three MSI-X IRQs (IRQ 260, the first one, which I think is MTHCA_EQ_COMP, if they are allocated in order ). dmesg + boot verbose: ib_mthca0: HCA FW version 5.2.916 is old (5.3.000 is current). ib_mthca0: If you have problems, try updating your HCA FW. ib_mthca0: attempting to allocate 3 MSI-X vectors (32 supported) msi: routing MSI-X IRQ 260 to local APIC 0 vector 62 msi: routing MSI-X IRQ 261 to local APIC 0 vector 63 msi: routing MSI-X IRQ 262 to local APIC 0 vector 64 ib_mthca0: using IRQs 260-262 for MSI-X I assume this has nothing to really with the IPoIB part, as much as interrupt handling in the hardware driver. Looking at the code, it looks like the MSI-X vectors are hard-coded and can’t be increased, event thought it seems like the card can support up to 32. So, I think I know the answer, which is no, but can the number of MSI-X vectors be increased, so the interrupt load can be spread over more CPUs ? 2) Firmware upgrade for the HCA. I got all the tools from Mellanox, however the syntax/directions for using flint says to point it to a non-exisiting /dev path in FreeBSD. Can one use FreeBSD to flash the firmware ? I could use some help getting the tools to talk to the card. and 3) SDP - Sockets Direct Protocol - while I’m waiting for other things, I’d like to play with SDP. Quick googling : - http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html > env LD_PRELOAD=/usr/lib/libibsdp.so iperf -s /usr/lib/libibsdp.so: Undefined symbol “program_invocation_short_name” However, doing this in FreeBSD, gives me a missing symbol. From searching around, program_invocation_short_name, is a gcc-libc thing. I’ve searched around, can’t seem to find an easy way to work around this. Any suggestions ? Thanks ! - David P. Discher http://davidpdischer.com/ AIM: DavidDPD | Y!M: daviddpdz [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUn1ItAAoJEEmwU6XuhYWOyVYIALA1ETjN4F1inbQz2h/LvyQN ZsPKbH8VLgbAG9r3iKos7aGDy7tIIpWOK6myaQ2nI6adyYI2VXMegKLrUzCg1ess J/l13jlqCbKRtrAFiKKQS4p/mL1OxnKWK8wbLHnbbt6X+CNH/AnUenbj+rqMYp4t 6aMFXPYHj4piFjT1qCH1872vNm0DWF888S0r+164oBcUJgLeTL/jIZTL0Mfv2p2V /pZYgxsVnefyC3k0lBJ1JlSE7teaL40MQNbD+3geWIYlAfW72phzvUzUk91mW191 GBAPJa45YlN+XzV+U2U5jJtPxSlZBrDCTNZyUkRyM0rVeniwGsgj3q8w2hcicZM= =Ns9v -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C06A80D7-AE40-4987-9BB3-FC1D20100F9C>