From owner-freebsd-questions@FreeBSD.ORG Sun Aug 18 03:19:10 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 546DAD3A for ; Sun, 18 Aug 2013 03:19:10 +0000 (UTC) (envelope-from aurfalien@gmail.com) Received: from mail-ob0-x230.google.com (mail-ob0-x230.google.com [IPv6:2607:f8b0:4003:c01::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1A8EB2B4D for ; Sun, 18 Aug 2013 03:19:10 +0000 (UTC) Received: by mail-ob0-f176.google.com with SMTP id uz19so3465215obc.7 for ; Sat, 17 Aug 2013 20:19:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to; bh=S9osmcQUEKrTuASD8sXI2hkPSDj0KNyNfd7r37Tya6k=; b=mD+zTfg5jWsHY5hvVv2fgxh22+2y/NEbDk4O7SXg/xYr+l/N6b6j4RzzlkdbIOgkAx OrywgHk1baGenR+NE8w1Y5pybiX0OZJnj04/YPFMErS9q5M3SHTjAMOCi9Bdde0aPQep L4ySG0n/8sJC0qVdhYXqheEVBksaY2iQK97eLwTeDLQosqJ7tRdxJe11bzHEjb1yEfQ0 D8VnWj4TKazywBlmWaWsHuByscfyIuBPW7+90SOA7v1ROr+paIlr1fY64z8HKevahUP3 LedLOQ6Osz15xJUIKUCzW2ZV7pJphkivblnU6r38cNDYO9hLC7za46f868Wbp20GVVSq fC9Q== X-Received: by 10.60.84.147 with SMTP id z19mr9339990oey.21.1376795949092; Sat, 17 Aug 2013 20:19:09 -0700 (PDT) Received: from [192.168.1.74] (75-63-29-182.lightspeed.irvnca.sbcglobal.net. [75.63.29.182]) by mx.google.com with ESMTPSA id a18sm5340860obf.7.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 17 Aug 2013 20:19:07 -0700 (PDT) Subject: Re: Myrinet 10Gb odd behavior - SOLVED Mime-Version: 1.0 (Apple Message framework v1085) From: aurfalien In-Reply-To: Date: Sat, 17 Aug 2013 20:19:04 -0700 Message-Id: References: <297A8244-3756-4126-9F23-B772B81127C7@gmail.com> <52F83CC6-2D87-43E7-9F9C-9D16ED637064@gmail.com> To: iamatt X-Mailer: Apple Mail (2.1085) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Mailing List X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Aug 2013 03:19:10 -0000 Spoke to soon. Fine for a while (doing a 5 day rsync of 38TB) but = getting those errors every 7 min. And I'm only getting 1.24Gb/s over a = 10Gb jumbo link. Definitely causing connection issues. Using it for ethernet. Gonna go in tomorrow and give my Solarflare another shot as it was = giving me issues but the rel notes say to try this, so I will; - The driver uses mbufs to store packet data which come from a set of = pools of limted size. See man 7 tuning for more details. The following = command can display the number of used and free mbufs within the pools the = Solarflare driver uses # vmstat -z | head -n 1; vmstat -z | grep mbuf ITEM SIZE LIMIT USED FREE REQUESTS = FAILURES mbuf_cluster: 2048, 25600, 1408, 658, 31604, = 0 mbuf_jumbo_page: 4096, 12800, 0, 76, 2063, = 0 mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, = 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, = 0 If a pool is exhausted (i.e. the failure count in the right hand = column is non-zero, networking applications may hang or received packets may be = dropped. Hence you may need to increase these limits using the following = sysctls: kern.ipc.nmbclusters (for mbuf_cluster) kern.ipc.nmbjumbop (for mbuf_jumbo_page) kern.ipc.nmbjumbo9 (for mbuf_jumbo_9k) kern.ipc.nmbjumbo16 (for mbuf_jumbo_16k) - aurf On Aug 17, 2013, at 8:14 PM, iamatt wrote: > Wow myricom still around... used to use the lanai stuff never on bsd = though. All FDR Infiniband these days. Are you using the myrinet = protocol or ethernet, just curious. Glad you got it working! >=20 > On Aug 16, 2013 8:12 PM, "aurfalien" wrote: >=20 > On Aug 16, 2013, at 8:47 AM, aurfalien wrote: >=20 > > Forgot to mention my loader.conf; > > > > if_mxge_load=3D"YES" > > mxge_ethp_z8e_load=3D"YES" > > mxge_eth_z8e_load=3D"YES" > > mxge_rss_ethp_z8e_load=3D"YES" > > mxge_rss_eth_z8e_load=3D"YES" > > > > > > I blindly added these w/o thinking what they do. > > > > Should I simply only load the first line? > > > > - aurf > > > > > > On Aug 16, 2013, at 8:18 AM, aurfalien wrote: > > > >> Hi, > >> > >> I've been suspecting my NIC is not up to par and notice this in the = logs every few minutes; > >> > >> Aug 16 08:05:06 prometheus kernel: mxge0: slice 0 struck? ring = state: > >> Aug 16 08:05:06 prometheus kernel: mxge0: tx.req=3D1914503981 = tx.done=3D1914503810, tx.queue_active=3D0 > >> Aug 16 08:05:06 prometheus kernel: mxge0: tx.activate=3D0 = tx.deactivate=3D0 > >> Aug 16 08:05:06 prometheus kernel: mxge0: pkt_done=3D1824019832 = fw=3D1824019931 > >> Aug 16 08:05:06 prometheus kernel: mxge0: Watchdog reset! > >> Aug 16 08:05:06 prometheus kernel: mxge0: NIC did not reboot, not = resetting > >> > >> Could tis be effecting throughput? > >> > >> My card is a Myri-10G-PCIE-8A > >> > >> I did install the Myrinet dev tools for FreeBSD and ran myri_info = which yields; > >> > >> pci-dev at 05:00.0 vendor:product(rev)=3D14c1:0008(00) > >> behind bridge root-port: 00:03.0 8086:3c08 (x8.1/x16.3) > >> Myri-10G-PCIE-8A -- Link x8 > >> EEPROM String-spec: > >> MAC=3D00:60:dd:45:73:23 > >> SN=3D413665 > >> PWR=3D100 > >> PC=3D10G-PCIE-8A-R > >> PN=3D09-03852 > >> XFI=3DAEL1010 > >> TAG=3Dze_tools-1_4_45 > >> > >> EEPROM MCP, PRESENT, length =3D 103384, crc=3D0x119daf46 > >> ETHZ::1.4.45 2009/08/22 18:57:06 self extracting firmware > >> Bundle: exec_len=3D72144, PCI-ROM-len =3D 31232 > >> Running MCP: > >> ETH ::1.4.55 -P- 2012/04/21 01:48:34 myri10ge firmware > >> > >> Any insights are appreciated. > >> > >> - aurf >=20 >=20 > Did the ole RTFM and re programmed the firmware, all good now. >=20 > - aurf > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to = "freebsd-questions-unsubscribe@freebsd.org"