From owner-freebsd-net@FreeBSD.ORG Wed Dec 18 01:03:55 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 032B86A6 for ; Wed, 18 Dec 2013 01:03:55 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AEAD51FA9 for ; Wed, 18 Dec 2013 01:03:54 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id rBI13hOX003080 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 17 Dec 2013 17:03:43 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id rBI13hRF003079; Tue, 17 Dec 2013 17:03:43 -0800 (PST) (envelope-from jmg) Date: Tue, 17 Dec 2013 17:03:42 -0800 From: John-Mark Gurney To: Claude Marinier Subject: Re: em3 no carrier Message-ID: <20131218010342.GG99167@funkthat.com> Mail-Followup-To: Claude Marinier , Michael Tuexen , "freebsd-net@freebsd.org" References: <34E38D657499684D9EBAA6D73CD2D1E21958EE4A62@CAEMEX81.caecorp.cae.com> <34E38D657499684D9EBAA6D73CD2D1E21958EE4AD3@CAEMEX81.caecorp.cae.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <34E38D657499684D9EBAA6D73CD2D1E21958EE4AD3@CAEMEX81.caecorp.cae.com> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Tue, 17 Dec 2013 17:03:43 -0800 (PST) Cc: Michael Tuexen , "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Dec 2013 01:03:55 -0000 Claude Marinier wrote this message on Tue, Dec 17, 2013 at 19:43 -0500: > -----Original Message----- > From: Michael Tuexen [mailto:Michael.Tuexen@lurchi.franken.de] > Sent: Tuesday, December 17, 2013 5:24 PM > To: Claude Marinier > Cc: freebsd-net@freebsd.org > Subject: Re: em3 no carrier > > On Dec 17, 2013, at 10:51 PM, Claude Marinier wrote: > > > > >>> -----Original Message----- > >>> From: Rodrigo Osorio [mailto:rodrigo at bebik.net] > >>> Sent: Tuesday, December 17, 2013 11:40 AM > >>> To: Claude Marinier > >>> Cc: freebsd-net at freebsd.org > >>> Subject: Re: em3 no carrier > >>> > >>> Hi, > >>> > >>> Can you give us more informations about the network card / chips ? > >>> Did you found any error in /var/log/ ? > >>> > >>> - rodrigo > >> > >> My apologies, I missed that bit of information. > >> > >> Yes, there is an error in /var/log/messages. > >> > >> Dec 17 11:33:07 WANemu bird: OSPF: Socket error on em3: No buffer > >> space available Dec 17 11:33:47 WANemu last message repeated 4 times > >> Dec 17 11:35:57 WANemu last message repeated 13 times Dec 17 11:45:57 > >> WANemu last message repeated 60 times Dec 17 11:55:47 WANemu last > >> message repeated 59 times > >> > >> > >> Also note that the HP NC364T uses an Intel 82571EB chipset. > >> > >>> On 17/12/13 11:27 -0500, Claude Marinier wrote: > >>>> Hi, > >>>> > >>>> This is a WAN emulator (ipfw, DummyNet, and BIRD). FreeBSD 9.2 > >>>> Release (amd64) is running on DL360 G5 with a new NC364T quad-port > >>>> Ethernet NIC. I installed the new NIC yesterday. The four ports are > >>>> connected to routers in a lab. > >>>> > >>>> FreeBSD WAMemu 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Thu Sep > >>>> 26 22:50:31 UTC 2013 root at > >>>> bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > >>>> > >>>> Dec 16 17:13:19 WANemu kernel: em3: >>>> Connection 7.3.8> port 0x6020-0x603f mem > >>>> 0xfdee0000-0xfdefffff,0xfde00000-0xfde7ffff irq 17 at device 0.1 on > >>>> pci22 Dec 16 17:13:19 WANemu kernel: em3: Using an MSI interrupt > >>>> Dec 16 17:13:19 WANemu kernel: em3: Ethernet address: > >>>> e8:39:35:13:21:6e > >>>> > >>>> The four em interfaces are configured the same way (lines in > >>>> rc.conf are identical except for IP address. > >>>> > >>>> ifconfig_em3="inet x.y.113.197/29 media 100baseTX mediaopt full-duplex" > >>>> ifconfig_em0="inet x.y.113.14/29 media 100baseTX mediaopt full-duplex" > >>>> ifconfig_em2="inet x.y.113.109/29 media 100baseTX mediaopt full-duplex" > >>>> ifconfig_em1="inet x.y.113.189/29 media 100baseTX mediaopt full-duplex" > >>>> > >>>> The GE 0/1 port on the Cisco 2821 shows no link and ifconfig shows > >>>> "no carrier" for em3. Oddly, it also shows autoselect. > >>>> > >>>> em0: flags=8843 metric 0 mtu 1500 > >>>> options=4019b > >>>> ether e8:39:35:13:21:6d > >>>> inet x.y.113.14 netmask 0xfffffff8 broadcast x.y.113.15 > >>>> inet6 fe80::ea39:35ff:fe13:216d%em0 prefixlen 64 scopeid 0x1 > >>>> nd6 options=29 > >>>> media: Ethernet 100baseTX > >>>> status: active > >>>> em1: flags=8843 metric 0 mtu 1500 > >>>> options=4019b > >>>> ether e8:39:35:13:21:6c > >>>> inet x.y.113.189 netmask 0xfffffff8 broadcast x.y.113.191 > >>>> inet6 fe80::ea39:35ff:fe13:216c%em1 prefixlen 64 scopeid 0x2 > >>>> nd6 options=29 > >>>> media: Ethernet 100baseTX > >>>> status: active > >>>> em2: flags=8843 metric 0 mtu 1500 > >>>> options=4019b > >>>> ether e8:39:35:13:21:6f > >>>> inet 131.140.113.109 netmask 0xfffffff8 broadcast 131.140.113.111 > >>>> inet6 fe80::ea39:35ff:fe13:216f%em2 prefixlen 64 scopeid 0x3 > >>>> nd6 options=29 > >>>> media: Ethernet 100baseTX > >>>> status: active > >>>> em3: flags=8843 metric 0 mtu 1500 > >>>> options=4019b > >>>> ether e8:39:35:13:21:6e > >>>> inet 131.140.113.197 netmask 0xfffffff8 broadcast 131.140.113.199 > >>>> inet6 fe80::ea39:35ff:fe13:216e%em3 prefixlen 64 scopeid 0x4 > >>>> nd6 options=29 > >>>> media: Ethernet 100baseTX (autoselect) > >>>> status: no carrier > >>>> > >>>> I have tried three Ethernet cables with no difference. This > >>>> morning, I briefly booted Puppy Linux from USB and manually > >>>> configured the four Ethernet interfaces. They all work (ping good > >>>> to all four routers); the Cisco on em3 shows happy lights. I then > >>>> booted back into FreeBSD, the problem with em3 remains. I have not > >>>> had any success searching with Google; I may not have used suitable search terms. > >>>> > >>>> One more odd symptom: bmon has trouble with these interfaces. > >>>> > >>>> em0è95^S!m on WANemu > >>>> Name > >>>> qqqqqqqqqqqqqqqqqqqqqqqq > >>>> WANemu (local) > >>>> 0 em0è95^S!m > >>>> 1 em1è95^S!l > >>>> 2 em2è95^S!o > >>>> 3 em3è95^S!n > >>>> 4 usbus0 > >>>> 5 usbus1 > >>>> 6 usbus2 > >>>> 7 usbus3 > >>>> 8 usbus4 > >>>> 9 usbus5 > >>>> 10 ipfw0 > >>>> 11 lo0 > >>>> > >>>> It looks like FreeBSD is somehow mishandling these interfaces. > >>>> Before purchasing it, I searched with Google and found reports of > >>>> people using this device without problems. I do not know how to proceed. > >>>> > >>>> Thank you for your help. > > > > Oddly, replacing em3 with bce1 left the system with only two > > functionning interfaces. The order of the interface definitions in > > rc.local makes a difference. The worst order is bce1, em0, em2, em1 > > which left only > > em0 and em2 working. Changing the order to bce1, em0, em1, em2 allows > > the three 'em' interfaces to work. In both, bce1 is not working (no > > carrier). Changing the order to em0, em1, em2, bce1 did not improve > > the situation. > > > > This is a WAN emulator and the rc.conf definitions are generated by > > the configuration program from site names, hence the odd order. I am > > changing the order manually. > > > > I have reconfigured the Cisco router to use G0/0 instead of G0/1. This > > has made no noticeable difference. > > Did you configure it to 100MBit, full-duplex? > > ------- > > Yes, I configured everything (all eight interfaces: four on FreeBSD and one on each of the routers) to be 100baseTX and full-duplex. > > As a test, I left out the 'media' and 'mediaopt' portions of the interface definitions in rc.conf and all the interfaces came up but three were in half-duplex mode, I think em0 was full-duplex (I am not in the lab to check). > > I have heard horror stories about Ethernet auto sensing, so I make sure the Cisco routers and FreeBSD specify 100 Mbps and full-duplex. I begin to wonder if even that is safe. :-( > > [ why does Outlook not quote properly ] It could be that there is a race condition when the media opts are set at the same time the interface is brought up... I have you tried to down the interface, set the media opts, wait a second or two, and then bring up the interface? I would go back to autoneg... Even though the 2821 is old, the autoneg issue was mostly when both sides were playing bad, and your nics are modern... At a previous company, we had an issue w/ our box at a client site... The box would work fine for months, and then for some reason it would suffer packet loss... After a long bit of puzzling, (I don't even remeber how we figured it out, but I think we got a dump of the switch configuration), it turned out that someone had configured our box to be 100mbit/full duplex, but not the switch... It turned out that this was "ok" for fxp as it would autoneg when brought up, and then it would switch to 100mbit/full... Both sides would be happy till the switch rebooted and then tried to autoneg again, but now the fxp had disabled it, and the switch would fall back to 100mbit/half... I fixed fxp to down/up the phy when setting fixed modes so that the switch would loose it's autoneg and then the customer would immediately see the issue instead of weeks/months/years later when the switch was power cycled... after the above, I considered anything not autoneg to be bad... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."