From owner-freebsd-infiniband@FreeBSD.ORG Tue Jun 4 09:22:52 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C6DBAB5; Tue, 4 Jun 2013 09:22:52 +0000 (UTC) (envelope-from alexl@mellanox.com) Received: from eu1sys200aog107.obsmtp.com (eu1sys200aog107.obsmtp.com [207.126.144.123]) by mx1.freebsd.org (Postfix) with ESMTP id 539CE1463; Tue, 4 Jun 2013 09:22:46 +0000 (UTC) Received: from MTLCAS01.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob107.postini.com ([207.126.147.11]) with SMTP ID DSNKUa2x5Wz95+hbRBQTxFrrl1iFkHSbU5BX@postini.com; Tue, 04 Jun 2013 09:22:48 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS01.mtl.com ([10.0.8.71]) with mapi id 14.03.0123.003; Tue, 4 Jun 2013 12:18:46 +0300 From: Alex Liptsin To: "freebsd-infiniband@freebsd.org" , "freebsd-net@freebsd.org" , "freebsd-questions@freebsd.org" Subject: How to compile ipoib module manually? Thread-Topic: How to compile ipoib module manually? Thread-Index: Ac5hBINaGaDVjqC1T4OcCXIOoLbpUQ== Date: Tue, 4 Jun 2013 09:18:46 +0000 Message-ID: <64DAB3164E410447932305F50F896D8D6AF6B412@MTLDAG01.mtl.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Meny Yossefi , Regev Lev X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 09:22:52 -0000 Hi. I work with FreeBSD9.1 and Mellanox devices. The kernel was configured with OFED support but without IB support: MYKERNEL file: # # GENERIC -- Generic kernel configuration file for FreeBSD/amd64 # # For more information on this file, please read the config(5) manual page, # and/or the handbook section on Kernel Configuration Files: # # http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig= -config.html # # The handbook is also available locally in /usr/share/doc/handbook # if you've installed the doc distribution, otherwise always see the # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the # latest information. # # An exhaustive list of options and more detailed explanations of the # device lines is also present in the ../../conf/NOTES and NOTES files. # If you are in doubt as to the purpose or necessity of a line, check first # in NOTES. # # $FreeBSD: release/9.1.0/sys/amd64/conf/GENERIC 238090 2012-07-04 00:54:16= Z delphij $ cpu HAMMER include GENERIC ident MYKERNEL makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug= symbols options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemptio= n options INET # InterNETworking options INET6 # IPv6 communications pr= otocols options SCTP # Stream Control Transmi= ssion Protocol options FFS # Berkeley Fast Filesy= stem options SOFTUPDATES # Enable FFS soft updates suppor= t options UFS_ACL # Support for a= ccess control lists options UFS_DIRHASH # Improve performance on big dir= ectories options UFS_GJOURNAL # Enable gjournal-base= d UFS journaling options MD_ROOT # MD is a potential= root device options NFSCL # New Network Filesystem = Client options NFSD # New Network Filesystem= Server options NFSLOCKD # Network Lock Manager options NFS_ROOT # NFS usable as /, requires NF= SCL options MSDOSFS # MSDOS Filesyste= m options CD9660 # ISO 9660 Fil= esystem options PROCFS # Process file= system (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_RAID # Soft RAID functionality. options GEOM_LABEL # Provides labelization options COMPAT_FREEBSD32 # Compatible with i386 binaries options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_FREEBSD5 # Compatible with FreeBSD5 options COMPAT_FREEBSD6 # Compatible with FreeBSD6 options COMPAT_FREEBSD7 # Compatible with FreeBSD7 options SCSI_DELAY=3D5000 # Delay (in ms) befo= re probing SCSI options KTRACE # ktrace(1) su= pport options STACK # stack(9) support options SYSVSHM # SYSV-style sha= red memory options SYSVMSG # SYSV-style mes= sage queues options SYSVSEM # SYSV-style sem= aphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time e= xtensions options PRINTF_BUFR_SIZE=3D128 # Prevent printf output b= eing interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks= for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framewor= k #options KDTRACE_FRAME # Ensure frames are compi= led in #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel options KDB # Kernel debugger relate= d code options KDB_TRACE # Print a stack trace for a pa= nic # Make an SMP-capable kernel by default options SMP # Symmetric MultiProcess= or Kernel # CPU frequency control device cpufreq # Bus support. device acpi device pci # Floppy drives device fdc # ATA controllers device ahci # AHCI-compatible SATA controllers device ata # Legacy ATA/SATA controllers options ATA_CAM # Handle legacy controllers w= ith CAM options ATA_STATIC_ID # Static device numbering device mvs # Marvell 88SX50XX/88SX60XX/88SX70= XX/SoC SATA device siis # SiliconImage SiI3124/SiI3132/S= iI3531 SATA # SCSI Controllers device ahc # AHA2940 and onboard AIC7xxx devi= ces options AHC_REG_PRETTY_PRINT # Print register bitfields in= debug # output. Adds= ~128k to driver. device ahd # AHA39320/29320 and onboard AIC79= xx devices options AHD_REG_PRETTY_PRINT # Print register bitfields in = debug # output. Adds= ~215k to driver. device esp # AMD Am53C974 (Tekram DC-390(T)) device hptiop # Highpoint RocketRaid 3xxx series device isp # Qlogic family #device ispfw # Firmware for QLogic H= BAs- normally a module device mpt # LSI-Logic MPT-Fusion device mps # LSI-Logic MPT-Fusion 2 #device ncr # NCR/Symbios Logic device sym # NCR/Symbios Logic (newer chipset= s + those of `ncr') device trm # Tekram DC395U/UW/F DC315U adapte= rs device adv # Advansys SCSI adapters device adw # Advansys wide SCSI adapters device aic # Adaptec 15[012]x SCSI adapters,= AIC-6[23]60. device bt # Buslogic/Mylex MultiMaster SCS= I adapters device isci # Intel C600 SAS controller # ATA/SCSI peripherals device scbus # SCSI bus (required for ATA/SCSI) device ch # SCSI media changers device da # Direct Access (disks) device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct ATA/S= CSI access) device ses # Enclosure Services (SES and SAF= -TE) device ctl # CAM Target Layer # RAID controllers interfaced to the SCSI subsystem device amr # AMI MegaRAID device arcmsr # Areca SATA II RAID #XXX it is not 64-bit clean, -scottl #device asr # DPT SmartRAID V, VI = and Adaptec SCSI RAID device ciss # Compaq Smart RAID 5* device dpt # DPT Smartcache III, IV - See NO= TES for options device hptmv # Highpoint RocketRAID 182x device hptrr # Highpoint RocketRAID 17xx, 22xx,= 23xx, 25xx device iir # Intel Integrated RAID device ips # IBM (Adaptec) ServeRAID device mly # Mylex AcceleRAID/eXtremeRAID device twa # 3ware 9000 series PATA/SATA RAID device tws # LSI 3ware 9750 SATA+SAS 6Gb/s R= AID controller # RAID controllers device aac # Adaptec FSA RAID device aacp # SCSI passthrough for aac (require= s CAM) device ida # Compaq Smart RAID device mfi # LSI MegaRAID SAS device mlx # Mylex DAC960 family #XXX pointer/int warnings #device pst # Promise Supertrak SX= 6000 device twe # 3ware ATA RAID # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer device vga # VGA video card driver options VESA # Add support for VESA BIOS Extensio= ns (VBE) device splash # Splash screen and screen saver su= pport # syscons is the default console driver, resembling an SCO console device sc options SC_PIXEL_MODE # add support for the raster text = mode device agp # support several AGP chipsets # PCCARD (PCMCIA) support # PCMCIA and cardbus bridge support device cbb # cardbus (yenta) bridge device pccard # PC Card (16-bit) bus device cardbus # CardBus (32-bit) bus # Serial (COM) ports device uart # Generic UART driver # Parallel port device ppc device ppbus # Parallel port bus (required) device lpt # Printer device plip # TCP/IP over parallel device ppi # Parallel port interface device #device vpo # Requires scbus and da device puc # Multi I/O cards and multi-channe= l UARTs # PCI Ethernet NICs. device bxe # Broadcom BCM57710/BCM57711/BCM57= 711E 10Gb Ethernet device de # DEC/Intel DC21x4x (``Tulip'') device em # Intel PRO/1000 Gigabit Ethernet = Family device igb # Intel PRO/1000 PCIE Server Giga= bit Family device ixgbe # Intel PRO/10GbE PCIE Ethernet Fam= ily device le # AMD Am7900 LANCE and Am79C9xx = PCnet device ti # Alteon Networks Tigon I/II gi= gabit Ethernet device txp # 3Com 3cR990 (``Typhoon'') device vx # 3Com 3c590, 3c595 (``Vortex'') # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs= ! device miibus # MII bus support device ae # Attansic/Atheros L2 FastEtherne= t device age # Attansic/Atheros L1 Gigabit Ethe= rnet device alc # Atheros AR8131/AR8132 Ethernet device ale # Atheros AR8121/AR8113/AR8114 Et= hernet device bce # Broadcom BCM5706/BCM5708 Gigabit= Ethernet device bfe # Broadcom BCM440x 10/100 Etherne= t device bge # Broadcom BCM570xx Gigabit Ethern= et device cas # Sun Cassini/Cassini+ and NS DP8= 3065 Saturn device dc # DEC/Intel 21143 and various wor= kalikes device et # Agere ET1310 10/100/Gigabit Et= hernet device fxp # Intel EtherExpress PRO/100B (82= 557, 82558) device gem # Sun GEM/Sun ERI/Apple GMAC device hme # Sun HME (Happy Meal Ethernet) device jme # JMicron JMC250 Gigabit/JMC260 Fa= st Ethernet device lge # Level 1 LXT1001 gigabit Etherne= t device msk # Marvell/SysKonnect Yukon II Giga= bit Ethernet device nfe # nVidia nForce MCP on-board Ethe= rnet device nge # NatSemi DP83820 gigabit Ethernet #device nve # nVidia nForce MCP on-= board Ethernet Networking device pcn # AMD Am79C97x PCI 10/100 (precede= nce over 'le') device re # RealTek 8139C+/8169/8169S/8110= S device rl # RealTek 8129/8139 device sf # Adaptec AIC-6915 (``Starfire''= ) device sge # Silicon Integrated Systems SiS1= 90/191 device sis # Silicon Integrated Systems SiS= 900/SiS 7016 device sk # SysKonnect SK-984x & SK-982x g= igabit Ethernet device ste # Sundance ST201 (D-Link DFE-550T= X) device stge # Sundance/Tamarack TC9021 gigabit= Ethernet device tl # Texas Instruments ThunderLAN device tx # SMC EtherPower II (83c170 ``EP= IC'') device vge # VIA VT612x gigabit Ethernet device vr # VIA Rhine, Rhine II device wb # Winbond W89C840F device xl # 3Com 3c90x (``Boomerang'', ``C= yclone'') # ISA Ethernet NICs. pccard NICs included. device cs # Crystal Semiconductor CS89x0 N= IC # 'device ed' requires 'device miibus' device ed # NE[12]000, SMC Ultra, 3c503, DS= 8390 cards device ex # Intel EtherExpress Pro/10 and P= ro/10+ device ep # Etherlink III based cards device fe # Fujitsu MB8696x based cards device sn # SMC's 9000 series of Ethernet = chips device xe # Xircom pccard Ethernet # Wireless NIC cards device wlan # 802.11 support options IEEE80211_DEBUG # enable debug msgs options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's options IEEE80211_SUPPORT_MESH # enable 802.11s draft support device wlan_wep # 802.11 WEP support device wlan_ccmp # 802.11 CCMP support device wlan_tkip # 802.11 TKIP support device wlan_amrr # AMRR transmit rate control algorithm device an # Aironet 4500/4800 802.11 wirele= ss NICs. device ath # Atheros NIC's device ath_pci # Atheros pci/cardbus gl= ue device ath_hal # pci/cardbus chip suppo= rt options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descr= iptors device ath_rate_sample # SampleRate tx rate control for= ath #device bwi # Broadcom BCM430x/BCM4= 31x wireless NICs. #device bwn # Broadcom BCM43xx wirel= ess NICs. device ipw # Intel 2100 wireless NICs. device iwi # Intel 2200BG/2225BG/2915ABG wir= eless NICs. device iwn # Intel 4965/1000/5000/6000 wirele= ss NICs. device malo # Marvell Libertas wireless NICs. device mwl # Marvell 88W8363 802.11n wireless = NICs. device ral # Ralink Technology RT2500 wirel= ess NICs. device wi # WaveLAN/Intersil/Symbol 802.11 = wireless NICs. device wpi # Intel 3945ABG wireless NICs. # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device vlan # 802.1Q VLAN support device tun # Packet tunnel. device pty # BSD-style compatibility pseudo = ttys device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translat= ion) device firmware # firmware assist module # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter # USB support options USB_DEBUG # enable debug msgs device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0= ) device xhci # XHCI PCI->USB interface (USB 3.= 0) device usb # USB Bus (required) #device udbp # USB Double Bulk Pipe d= evices (needs netgraph) device uhid # "Human Interface Devices" device ukbd # Keyboard device ulpt # Printer device umass # Disks/Mass storage - Requires scbu= s and da device ums # Mouse device urio # Diamond Rio 500 MP3 player # USB Serial devices device u3g # USB-based 3G modems (Option, Hua= wei, Sierra) device uark # Technologies ARK3116 based seria= l adapters device ubsa # Belkin F5U103 and compatible ser= ial adapters device uftdi # For FTDI usb serial adapters device uipaq # Some WinCE based devices device uplcom # Prolific PL-2303 serial= adapters device uslcom # SI Labs CP2101/CP2102 serial adapt= ers device uvisor # Visor and Palm devices device uvscom # USB serial support for = DDI pocket's PHS # USB Ethernet, requires miibus device aue # ADMtek USB Ethernet device axe # ASIX Electronics USB Ethernet device cdce # Generic USB over Ethernet device cue # CATC USB Ethernet device kue # Kawasaki LSI USB Ethernet device rue # RealTek RTL8150 USB Ethernet device udav # Davicom DM9601E USB # USB Wireless device rum # Ralink Technology RT2501USB wirel= ess NICs device run # Ralink Technology RT2700/RT2800= /RT3000 NICs. device uath # Atheros AR5523 wireless NICs device upgt # Conexant/Intersil PrismGT wirele= ss NICs. device ural # Ralink Technology RT2500USB wir= eless NICs device urtw # Realtek RTL8187B/L wireless NICs device zyd # ZyDAS zd1211/zd1211b wireless N= ICs # FireWire support device firewire # FireWire bus code # sbp(4) works for some systems but causes boot failure on others #device sbp # SCSI over FireWire (= Requires scbus and da) device fwe # Ethernet over FireWire (non-stan= dard!) device fwip # IP over FireWire (RFC 2734,3146) device dcons # Dumb console driver device dcons_crom # Configuration ROM for dcons # Sound support device sound # Generic sound driver (required) device snd_cmi # CMedia CMI8338/CMI8738 device snd_csa # Crystal Semiconductor C= S461x/428x device snd_emu10kx # Creative SoundBlaster Live! and Audigy device snd_es137x # Ensoniq AudioPCI ES137x device snd_hda # Intel High Definition A= udio device snd_ich # Intel, NVidia and othe= r ICH AC'97 Audio device snd_uaudio # USB Audio device snd_via8233 # VIA VT8233x Audio # Mellanox and IB support options OFED # Infiniband protocol options OFED_DEBUG_INIT options SDP # Sockets Direct Protocol for infiniband options SDP_DEBUG # options IPOIB_DEBUG # options IPOIB_CM # Use connect mode ipoib device mthca # Infinihost cards # device mlx4ib # ConnectX Infiniband support #device mlxen # ConnectX Ethernet support # device ipoib # IP over IB devices I commented on that lines, because I want to compile and load that modules = manually. I had succeed to compile and load mlx4, mlx4ib and mlxen from /sys/modules: [root@h-qa-033 mlxen]# kldstat Id Refs Address Size Name 1 14 0xffffffff80200000 13acbd8 kernel 2 1 0xffffffff81612000 21e5 if_mos.ko 3 3 0xffffffff81615000 124eb mlx4.ko 4 1 0xffffffff81628000 e225 mlx4ib.ko 5 1 0xffffffff81637000 ec60 mlxen.ko The problem is that IPOIB module is missing in /sys/modules. 1. Where can I find it? 2. How can I compile ipoib support? Regards, Alex Liptsin Software Quality Assurance Engineer | Mellanox Technologies Ltd. Office: +972 (74) 7236141 Mobile: +972(54) 7833986 Fax: +972(74) 7236161 Email: alexl@mellanox.com Mellanox, Tel-Hai Industrial Park. Building 7, M.P. Upper Galilee 12100 Isr= ael From owner-freebsd-infiniband@FreeBSD.ORG Wed Jun 5 15:13:41 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id BCB1B495; Wed, 5 Jun 2013 15:13:41 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) by mx1.freebsd.org (Postfix) with ESMTP id 997B81E8F; Wed, 5 Jun 2013 15:13:41 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 024B4B941; Wed, 5 Jun 2013 11:13:41 -0400 (EDT) From: John Baldwin To: freebsd-net@freebsd.org Subject: Re: How to compile ipoib module manually? Date: Wed, 5 Jun 2013 11:09:56 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p25; KDE/4.5.5; amd64; ; ) References: <64DAB3164E410447932305F50F896D8D6AF6B412@MTLDAG01.mtl.com> In-Reply-To: <64DAB3164E410447932305F50F896D8D6AF6B412@MTLDAG01.mtl.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201306051109.57160.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 05 Jun 2013 11:13:41 -0400 (EDT) Cc: Regev Lev , Alex Liptsin , "freebsd-infiniband@freebsd.org" , Meny Yossefi , "freebsd-questions@freebsd.org" X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 15:13:41 -0000 On Tuesday, June 04, 2013 5:18:46 am Alex Liptsin wrote: > I commented on that lines, because I want to compile and load that modules manually. > I had succeed to compile and load mlx4, mlx4ib and mlxen from /sys/modules: > > [root@h-qa-033 mlxen]# kldstat > Id Refs Address Size Name > 1 14 0xffffffff80200000 13acbd8 kernel > 2 1 0xffffffff81612000 21e5 if_mos.ko > 3 3 0xffffffff81615000 124eb mlx4.ko > 4 1 0xffffffff81628000 e225 mlx4ib.ko > 5 1 0xffffffff81637000 ec60 mlxen.ko > > The problem is that IPOIB module is missing in /sys/modules. > > 1. Where can I find it? > > 2. How can I compile ipoib support? You will have to create one. You should be able to use the existing module Makefiles as a guide. -- John Baldwin From owner-freebsd-infiniband@FreeBSD.ORG Wed Jun 5 17:25:36 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B504CF4E; Wed, 5 Jun 2013 17:25:36 +0000 (UTC) (envelope-from alexl@mellanox.com) Received: from eu1sys200aog105.obsmtp.com (eu1sys200aog105.obsmtp.com [207.126.144.119]) by mx1.freebsd.org (Postfix) with ESMTP id 17A7B1638; Wed, 5 Jun 2013 17:25:34 +0000 (UTC) Received: from MTLCAS02.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob105.postini.com ([207.126.147.11]) with SMTP ID DSNKUa90hz8OlSTV5819r9UgteaN7dNl4Ucj@postini.com; Wed, 05 Jun 2013 17:25:36 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS02.mtl.com ([10.0.8.72]) with mapi id 14.03.0123.003; Wed, 5 Jun 2013 20:22:01 +0300 From: Alex Liptsin To: John Baldwin Subject: Re: How to compile ipoib module manually? Thread-Topic: How to compile ipoib module manually? Thread-Index: Ac5hBINaGaDVjqC1T4OcCXIOoLbpUQA4RRMAAArmUzs= Date: Wed, 5 Jun 2013 17:22:01 +0000 Message-ID: <2243E588-1BF8-4935-ACA2-48BA173335FB@Mellanox.Com> References: <64DAB3164E410447932305F50F896D8D6AF6B412@MTLDAG01.mtl.com>, <201306051109.57160.jhb@freebsd.org> In-Reply-To: <201306051109.57160.jhb@freebsd.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-8-i" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" , Regev Lev , "freebsd-infiniband@freebsd.org" , Meny Yossefi , "freebsd-questions@freebsd.org" X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 17:25:36 -0000 Thanks a lot. Alex L. Sent from my iPhone On 5 =E1=E9=E5=F0 2013, at 18:13, "John Baldwin" > wrote: On Tuesday, June 04, 2013 5:18:46 am Alex Liptsin wrote: I commented on that lines, because I want to compile and load that modules manually. I had succeed to compile and load mlx4, mlx4ib and mlxen from /sys/modules: [root@h-qa-033 mlxen]# kldstat Id Refs Address Size Name 1 14 0xffffffff80200000 13acbd8 kernel 2 1 0xffffffff81612000 21e5 if_mos.ko 3 3 0xffffffff81615000 124eb mlx4.ko 4 1 0xffffffff81628000 e225 mlx4ib.ko 5 1 0xffffffff81637000 ec60 mlxen.ko The problem is that IPOIB module is missing in /sys/modules. 1. Where can I find it? 2. How can I compile ipoib support? You will have to create one. You should be able to use the existing module Makefiles as a guide. -- John Baldwin From owner-freebsd-infiniband@FreeBSD.ORG Wed Jun 5 17:37:15 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id EA3C637C; Wed, 5 Jun 2013 17:37:15 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pd0-f178.google.com (mail-pd0-f178.google.com [209.85.192.178]) by mx1.freebsd.org (Postfix) with ESMTP id B521516BB; Wed, 5 Jun 2013 17:37:15 +0000 (UTC) Received: by mail-pd0-f178.google.com with SMTP id w16so2139879pde.9 for ; Wed, 05 Jun 2013 10:37:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=Elfjc8XLk8ZNXjpKJp4DQkktAwsvHwx4vvtjeOU+F6I=; b=qS8flcG+yFBcJyiDxWaWSGj3K9OJQGmTasmH6UXQ6hIfYjlaUsSfsPJW+C2CfiQwPN cjBjXwzzU86Xty8qNzuVLYLF/cqqjALsJ44O4FCcILlTAT3lMmZm/Tzn9c11BKLfbKAE ziVg88Yck+8VYAo1YtgINsgnmVuOwH4WEEo3FY5gffla49C+8ZvbBkQluR0VQUrNVa0L pva/eTPeYpKD5WHBKok2OBn8fUvdaICkZtS5TBYVmOCVSZVDZm5pTRVxHdpG4VAYh0D7 W5iBMsTuQWAndbYb+h65JyMKhVtkwzqtswHe0E+wsjw3BAILhIyfKo0d18/NQSPFTreJ hqVQ== X-Received: by 10.66.126.11 with SMTP id mu11mr34181878pab.81.1370453834995; Wed, 05 Jun 2013 10:37:14 -0700 (PDT) Received: from [192.168.20.5] (c-98-203-241-95.hsd1.wa.comcast.net. [98.203.241.95]) by mx.google.com with ESMTPSA id p2sm5008986pag.22.2013.06.05.10.37.13 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 05 Jun 2013 10:37:14 -0700 (PDT) Subject: Re: How to compile ipoib module manually? Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=ISO-8859-8-i From: Garrett Cooper In-Reply-To: <2243E588-1BF8-4935-ACA2-48BA173335FB@Mellanox.Com> Date: Wed, 5 Jun 2013 10:34:51 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <64DAB3164E410447932305F50F896D8D6AF6B412@MTLDAG01.mtl.com>, <201306051109.57160.jhb@freebsd.org> <2243E588-1BF8-4935-ACA2-48BA173335FB@Mellanox.Com> To: Alex Liptsin X-Mailer: Apple Mail (2.1283) Cc: John Baldwin , freebsd-net@freebsd.org, Regev Lev , freebsd-infiniband@freebsd.org, Meny Yossefi , freebsd-questions@freebsd.org X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 17:37:16 -0000 On Jun 5, 2013, at 10:22 AM, Alex Liptsin wrote: > Thanks a lot. >=20 > Alex L. >=20 >=20 > Sent from my iPhone >=20 > On 5 =E1=E9=E5=F0 2013, at 18:13, "John Baldwin" = > wrote: >=20 > On Tuesday, June 04, 2013 5:18:46 am Alex Liptsin wrote: > I commented on that lines, because I want to compile and load that = modules > manually. > I had succeed to compile and load mlx4, mlx4ib and mlxen from = /sys/modules: >=20 > [root@h-qa-033 mlxen]# kldstat > Id Refs Address Size Name > 1 14 0xffffffff80200000 13acbd8 kernel > 2 1 0xffffffff81612000 21e5 if_mos.ko > 3 3 0xffffffff81615000 124eb mlx4.ko > 4 1 0xffffffff81628000 e225 mlx4ib.ko > 5 1 0xffffffff81637000 ec60 mlxen.ko >=20 > The problem is that IPOIB module is missing in /sys/modules. >=20 > 1. Where can I find it? >=20 > 2. How can I compile ipoib support? >=20 > You will have to create one. You should be able to use the existing = module > Makefiles as a guide. (Please don't cross-post multiple mailing lists) See also: https://github.com/yaneurabeya/freebsd/tree/ib-modules = . Diff that branch and master to see what I've done so far to make the = IB stuff into modules (example: = https://github.com/yaneurabeya/freebsd/blob/ib-modules/sys/modules/ipoib/M= akefile ). It's not perfect, but it's a start. Something that I've = wanted to push into the Isilon IB branch for some time, but it's going = to take a while before that's committed back to FreeBSD proper I think. Please credit me and add sponsored-by: EMC Isilon for the = initial work if you push this elsewhere. Otherwise, it's all your's to = play with :). Cheers, -Garrett= From owner-freebsd-infiniband@FreeBSD.ORG Thu Jun 6 11:48:22 2013 Return-Path: Delivered-To: freebsd-infiniband@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E0895F9E for ; Thu, 6 Jun 2013 11:48:22 +0000 (UTC) (envelope-from jsteckli@os.inf.tu-dresden.de) Received: from os.inf.tu-dresden.de (os.inf.tu-dresden.de [IPv6:2002:8d4c:3001:48::99]) by mx1.freebsd.org (Postfix) with ESMTP id A70FD1B60 for ; Thu, 6 Jun 2013 11:48:22 +0000 (UTC) Received: from [2002:8d4c:3001:48:ea40:f2ff:fee2:6328] by os.inf.tu-dresden.de with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.80.1) id 1UkYg1-0002cJ-UR for freebsd-infiniband@freebsd.org; Thu, 06 Jun 2013 13:48:21 +0200 Message-ID: <51B07705.207@os.inf.tu-dresden.de> Date: Thu, 06 Jun 2013 13:48:21 +0200 From: Julian Stecklina User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130514 Thunderbird/17.0.6 MIME-Version: 1.0 Newsgroups: gmane.os.freebsd.stable To: freebsd-infiniband@freebsd.org Subject: Reproducable Infiniband panic Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jun 2013 11:48:22 -0000 Hello, I see a reproducable panic when doing ibping and aborting it with ^C. My setup is two machines with Mellanox Infinihost III HCAs (one Linux one FreeBSD) connected back-to-back. Details below. I can upload 2 crash dumps, if this is useful. For some reason the port doesn't become ACTIVE, so no packets arrive, but that is probably unrelated. % uname -a FreeBSD cosel.inf.tu-dresden.de 9.1-STABLE FreeBSD 9.1-STABLE #0 r+b6547e3: Wed Jun 5 18:29:51 CEST 2013 julian@cosel.inf.tu-dresden.de:/usr/obj/usr/home/julian/src/freebsd/sys/COSEL amd64 % sudo ibping 2 ^C --- (Lid 2) ibping statistics --- 6 packets transmitted, 0 received, 100% packet loss, time 5161 ms rtt min/avg/max = 0.000/0.000/0.000 ms Fatal trap 12: page fault while in kernel mode cpuid = 6; apic id = 06 fault virtual address = 0x18 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff807a3d83 stack pointer = 0x28:0xffffff8092c97890 frame pointer = 0x28:0xffffff8092c978b0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 1489 (ibping) trap number = 12 panic: page fault cpuid = 6 KDB: stack backtrace: #0 0xffffffff80632a96 at kdb_backtrace+0x66 #1 0xffffffff805f9fce at panic+0x1ce #2 0xffffffff808a7380 at trap_fatal+0x290 #3 0xffffffff808a76e1 at trap_pfault+0x211 #4 0xffffffff808a7c94 at trap+0x344 #5 0xffffffff80891043 at calltrap+0x8 #6 0xffffffff80513c39 at devfs_destroy_cdevpriv+0x69 #7 0xffffffff80513e47 at devfs_close_f+0x57 #8 0xffffffff805b4b23 at _fdrop+0x23 #9 0xffffffff805b65ec at closef+0x4c #10 0xffffffff805b76cc at fdfree+0x23c #11 0xffffffff805c4945 at exit1+0x305 #12 0xffffffff805c5d0e at sys_sys_exit+0xe #13 0xffffffff808a6b56 at amd64_syscall+0x5d6 #14 0xffffffff80891327 at Xfast_syscall+0xf7 Full backtrace from kgdb: #0 doadump (textdump=) at pcpu.h:234 No locals. #1 0xffffffff805f9aa4 in kern_reboot (howto=260) at /usr/home/julian/src/freebsd/sys/kern/kern_shutdown.c:449 _ep = (struct eventhandler_entry *) 0x0 _el = first_buf_printf = 1 #2 0xffffffff805f9fa7 in panic (fmt=0x1
) at /usr/home/julian/src/freebsd/sys/kern/kern_shutdown.c:637 td = (struct thread *) 0x1 bootopt = newpanic = ap = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 0xffffff8092b934f0, reg_save_area = 0xffffff8092b93410}} panic_cpu = 7 buf = "page fault", '\0' #3 0xffffffff808a7380 in trap_fatal (frame=0xc, eva=) at /usr/home/julian/src/freebsd/sys/amd64/amd64/trap.c:878 code = ss = 40 type = 12 esp = softseg = {ssd_base = 0, ssd_limit = 1048575, ssd_type = 27, ssd_dpl = 0, ssd_p = 1, ssd_long = 1, ssd_def32 = 0, ssd_gran = 1} msg = #4 0xffffffff808a76e1 in trap_pfault (frame=0xffffff8092b937e0, usermode=0) at /usr/home/julian/src/freebsd/sys/amd64/amd64/trap.c:794 id = va = 0 vm = map = 0xfffffe000b0a3498 rv = 0 ftype = 255 'ΓΏ' td = (struct thread *) 0xfffffe000b0af000 p = (struct proc *) 0xfffffe000b181950 eva = 24 #5 0xffffffff808a7c94 in trap (frame=0xffffff8092b937e0) at /usr/home/julian/src/freebsd/sys/amd64/amd64/trap.c:463 regs = {r_r15 = -2136840320, r_r14 = -547294202144, r_r13 = -547294202208, r_r12 = -2140660471, r_r11 = -2136840320, r_r10 = 594, r_r9 = -547294202160, r_r8 = -2198830683992, r_rdi = 0, r_rsi = -2136780531, r_rbp = 219043332096, r_rbx = -2198837989376, r_rdx = -547294202048, r_rcx = 2154444695, r_rax = -2133265824, r_trapno = 192571360, r_fs = 65024, r_gs = 65535, r_err = 525312, r_es = 65408, r_ds = 65535, r_rip = -2136840320, r_cs = -547294202064, r_rflags = -2133515200, r_rsp = -547294201968, r_ss = 0} td = (struct thread *) 0xfffffe000b0af000 p = i = ucode = code = 0 type = 12 addr = ksi = {ksi_link = {tqe_next = 0xfffffe000553ac00, tqe_prev = 0xfffffe000b0af000}, ksi_info = {si_signo = -1833355440, si_errno = -128, si_code = -2140661293, si_pid = -1, si_uid = 0, si_status = 0, si_addr = 0xfffffe0000000000, si_value = {sival_int = -1833355392, sival_ptr = 0xffffff8092b93780, sigval_int = -1833355392, sigval_ptr = 0xffffff8092b93780}, _reason = {_fault = {_trapno = -2138032854}, _timer = {_timerid = -2138032854, _overrun = -1}, _mesgq = {_mqd = -2138032854}, _poll = {_band = -2138032854}, __spare__ = { __spare1__ = -2138032854, __spare2__ = {192571360, -512, 192571360, -512, 2, 0, 0}}}}, ksi_flags = -1833355296, ksi_sigq = 0xffffffff806955f3} #6 0xffffffff80891043 in calltrap () at /usr/home/julian/src/freebsd/sys/amd64/amd64/exception.S:228 No locals. #7 0xffffffff807a3d83 in linux_file_dtor (cdp=0xfffffe000aeabb80) at /usr/home/julian/src/freebsd/sys/ofed/include/linux/linux_compat.c:214 filp = (struct linux_file *) 0xfffffe000aeabb80 #8 0xffffffff80513c39 in devfs_destroy_cdevpriv (p=0xfffffe0005772980) at /usr/home/julian/src/freebsd/sys/fs/devfs/devfs_vnops.c:159 No locals. #9 0xffffffff80513e47 in devfs_close_f (fp=0xfffffe000b0e9aa0, td=) at /usr/home/julian/src/freebsd/sys/fs/devfs/devfs_vnops.c:619 error = 0 fpop = (struct file *) 0x0 #10 0xffffffff805b4b23 in _fdrop (fp=0xfffffe000b0e9aa0, td=) at file.h:334 error = 0 #11 0xffffffff805b65ec in closef (fp=0xfffffe000b0e9aa0, td=0xfffffe000b0af000) at /usr/home/julian/src/freebsd/sys/kern/kern_descrip.c:2272 vp = lf = {l_start = 0, l_len = -2198837126832, l_pid = 4, l_type = 0, l_whence = 0, l_sysid = 185266176} fdtol = (struct filedesc_to_leader *) 0x0 fdp = fp_object = (struct file *) 0xfffffe000b0e9aa0 #12 0xffffffff805b76cc in fdfree (td=0xfffffe000b0af000) at /usr/home/julian/src/freebsd/sys/kern/kern_descrip.c:1976 fdp = (struct filedesc *) 0xfffffe000b1a8000 fpp = (struct file **) 0xfffffe000b1a8098 i = 3 fdtol = fp = (struct file *) 0xfffffe000b0e9aa0 cdir = jdir = rdir = vp = lf = {l_start = -547294201248, l_len = -2140995476, l_pid = 4, l_type = 0, l_whence = 0, l_sysid = 0} #13 0xffffffff805c4945 in exit1 (td=0xfffffe000b0af000, rv=0) at /usr/home/julian/src/freebsd/sys/kern/kern_exit.c:301 id = p = (struct proc *) 0xfffffe000b181950 nq = q = (struct proc *) 0x4 vtmp = ttyvp = plim = reason = #14 0xffffffff805c5d0e in sys_sys_exit (td=, uap=) at /usr/home/julian/src/freebsd/sys/kern/kern_exit.c:122 No locals. #15 0xffffffff808a6b56 in amd64_syscall (td=0xfffffe000b0af000, traced=0) at subr_syscall.c:135 sa = {code = 1, callp = 0xffffffff80d31330, args = {0, 0, 10, 0, 0, 0, 133124, -547294200768}, narg = 1} error = 0 ksi = {ksi_link = {tqe_next = 0x0, tqe_prev = 0x0}, ksi_info = {si_signo = 2, si_errno = 0, si_code = 65542, si_pid = 0, si_uid = 0, si_status = 0, si_addr = 0x0, si_value = {sival_int = 0, sival_ptr = 0x0, sigval_int = 0, sigval_ptr = 0x0}, _reason = {_fault = { _trapno = 0}, _timer = {_timerid = 0, _overrun = 0}, _mesgq = {_mqd = 0}, _poll = {_band = 0}, __spare__ = {__spare1__ = 0, __spare2__ = {0, 0, 0, 0, 0, 0, 0}}}}, ksi_flags = 0, ksi_sigq = 0x0} #16 0xffffffff80891327 in Xfast_syscall () at /usr/home/julian/src/freebsd/sys/amd64/amd64/exception.S:387 No locals. #17 0x0000000800eda82c in ?? () Julian