From owner-freebsd-net@FreeBSD.ORG Wed Oct 12 17:05:37 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 731CB106566B for ; Wed, 12 Oct 2011 17:05:37 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id F18CB8FC12 for ; Wed, 12 Oct 2011 17:05:36 +0000 (UTC) Received: by wyj26 with SMTP id 26so1389403wyj.13 for ; Wed, 12 Oct 2011 10:05:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=Zi2slF/EAeE4kvVPdIUMWBMreOnll02m42oksI+SRBc=; b=DBna38dpVpnr/I/+H8aXmjfHQHDL8+c2mmVPpTVWGO5reHjraRRxDS/9PGkn9hld52 nR78Y05d98r7rXIH8NqkxPWzubN23rkIyzUxe4fsolxCMl688FC7voGYLhDoC7a0l1QO V6crDtiBJNH9IISZn7PZA3lVBsq64gK4MT3/o= Received: by 10.227.154.2 with SMTP id m2mr3219311wbw.39.1318439135473; Wed, 12 Oct 2011 10:05:35 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id p2sm1043684wbo.17.2011.10.12.10.05.30 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 12 Oct 2011 10:05:33 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Wed, 12 Oct 2011 10:03:47 -0700 From: YongHyeon PYUN Date: Wed, 12 Oct 2011 10:03:47 -0700 To: Karim Message-ID: <20111012170347.GA9138@michelle.cdnetworks.com> References: <4E94637A.5090607@gmail.com> <20111011171029.GA5661@michelle.cdnetworks.com> <4E959F06.6040906@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4E959F06.6040906@gmail.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org, Kevin Oberman Subject: Re: if_msk.c link negotiation / packet drops X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2011 17:05:37 -0000 On Wed, Oct 12, 2011 at 10:07:02AM -0400, Karim wrote: > On 11-10-11 01:31 PM, Kevin Oberman wrote: > >On Tue, Oct 11, 2011 at 10:10 AM, YongHyeon PYUN wrote: > >>On Tue, Oct 11, 2011 at 11:40:42AM -0400, Karim wrote: > >>>Hi List, > >>> > >>>Using a Marvell NIC plugged into a CISCO switch I see the > >>>auto-negotiation failing and even when forcing the device to full-duplex > >>>we sometimes see packet drops. > >>> > >>>Here is the device description from dmesg: > >>> > >>>mskc0: port 0xbe00-0xbeff mem > >>>0xfdefc000-0xfdefffff irq 16 at device 0.0 on pci1 > >>>msk0: on mskc0 > >>>msk0: Ethernet address: 00:03:2d:09:94:52 > >>>miibus0: on msk0 > >>>e1000phy0: PHY 0 on miibus0 > >>>e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > >>>1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow > >>>mskc0: [ITHREAD] > >>> > >>>The switch its plugged in (Cisco) is configured for 100baseTX > >>>full-duplex. > >>> > >>>ifconfig reports: > >>> > >>>msk0: > >>>flags=608843 > >>>metric 0 mtu 1500 > >>> options=40018 > >>The flags and options show that you're using very customized > >>driver, right? > >> > >>> ether 00:03:2d:09:94:52 > >>> inet 192.168.122.7 netmask 0xffffff00 broadcast 192.168.122.255 > >>> media: Ethernet autoselect (100baseTX) > >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > >> > >>Resolved duplex is half so I guess it would be normal to see > >>dropped frames which may be triggered by collision. > >You have a duplex mis-match. If you are hard setting the remote end to > >full, the local end must also be configured o full. Auto-configuration > >of duplex requires that both ends run auto-config. When one end is set > >to not do auto-config, the other end SHOULD always set to half-duplex. > >This is part of 802.3 that is a carry-over from the days when hubs and > >coax dominated, so the default was declared to be half. Since so much > >hardware now exists with that default, changing it ill never happen. > > > >Either set your computer to full duplex or turn on auto-configuration > >on the Cisco. > > > >Very little hardware now in service fails to auto-config correctly, > >but the practice of lacking down the duplex setting became common in > >the early days of full-duplex when it was not yet a standard and many > >Ethernet chip-sets didn't play nice with others. Things would be much > >better if people would just stop hard-setting the duplex, but old, old > >habits and memes die hard. > > > >Also, contrary to common belief, collisions are NOT errors. They are a > >normal part of half-dulpex Ethernet operation and do NOT result in > >packets being dropped. Only "excessive collisions do and they ARE a > >real error and a clear indication that something is wrong. > Hi, > > Thanks for the feedback and detailed information. > > I have to clarify here; I get the issue even with forcing full duplex. > > The driver modifications are minor and shouldn't affect link negotiation > or link state. Its also interesting that this problem only shows up with > Cisco's switches and faced with other types of switches the issue does > not come up. > Right. It would be also interesting to see MAC statistics of Cisco switch and check both parties agree on resolved speed/duplex/flow-control. > Also, nothing like collisions or missed/dropped packets can be found in > msk_stats (see below) that somewhat relate to the issue I'm seeing. One > thing interesting is the ifm_status value from msk_mediastatus. It often > changes from (IFM_AVALID | IFM_ACTIVE) which is 0x3 to 0x1 (IFM_AVALID) > for no reason I can tell. > Hmm, that indicates driver lost established link. msk(4) will detect this condition and stop RX/TX MACs until it knows PHY re-established a link. This may be the reason why you see occasional packet drops. However I don't know why PHY loses established link in the middle of working. > From the code in e1000phy_status: > > static void > e1000phy_status(struct mii_softc *sc) > { > struct mii_data *mii = sc->mii_pdata; > int bmcr, bmsr, ssr; > > mii->mii_media_status = IFM_AVALID; > mii->mii_media_active = IFM_ETHER; > > bmsr = PHY_READ(sc, E1000_SR) | PHY_READ(sc, E1000_SR); > bmcr = PHY_READ(sc, E1000_CR); > ssr = PHY_READ(sc, E1000_SSR); > > if (bmsr & E1000_SR_LINK_STATUS) > mii->mii_media_status |= IFM_ACTIVE; > > > I can see the bmsr & E1000_SR_LINK_STATUS check failing when the problem > occurs. As a side note why are we ORing the same call twice isn't the > same thing as calling it once: > > bmsr = PHY_READ(sc, E1000_SR) | PHY_READ(sc, E1000_SR); > The E1000_SR_LINK_STATUS bit is latched low so it should be read twice. If you want to read once use E1000_SSR_LINK bit of E1000_SSR register but I remember that bit was not reliable on some PHY models. By chance, does your back-ported driver include r222219? If yes, did you cold boot after applying the change? Warm boot does have effect. > As requested here is my msk0 stats output right after the problem showed up: > > dev.msk.0.stats.rx.ucast_frames: 58103886 > dev.msk.0.stats.rx.bcast_frames: 0 > dev.msk.0.stats.rx.pause_frames: 0 > dev.msk.0.stats.rx.mcast_frames: 0 > dev.msk.0.stats.rx.crc_errs: 0 > dev.msk.0.stats.rx.good_octets: 5927739395 > dev.msk.0.stats.rx.bad_octets: 0 > dev.msk.0.stats.rx.frames_64: 53 > dev.msk.0.stats.rx.frames_65_127: 58091128 > dev.msk.0.stats.rx.frames_128_255: 12545 > dev.msk.0.stats.rx.frames_256_511: 40 > dev.msk.0.stats.rx.frames_512_1023: 89 > dev.msk.0.stats.rx.frames_1024_1518: 31 > dev.msk.0.stats.rx.frames_1519_max: 0 > dev.msk.0.stats.rx.frames_too_long: 0 > dev.msk.0.stats.rx.jabbers: 0 > dev.msk.0.stats.rx.overflows: 0 > dev.msk.0.stats.tx.ucast_frames: 58104799 > dev.msk.0.stats.tx.bcast_frames: 53 > dev.msk.0.stats.tx.pause_frames: 0 > dev.msk.0.stats.tx.mcast_frames: 0 > dev.msk.0.stats.tx.octets: 5927439760 > dev.msk.0.stats.tx.frames_64: 547 > dev.msk.0.stats.tx.frames_65_127: 58091680 > dev.msk.0.stats.tx.frames_128_255: 12576 > dev.msk.0.stats.tx.frames_256_511: 17 > dev.msk.0.stats.tx.frames_512_1023: 1 > dev.msk.0.stats.tx.frames_1024_1518: 32 > dev.msk.0.stats.tx.frames_1519_max: 0 > dev.msk.0.stats.tx.colls: 2 > dev.msk.0.stats.tx.late_colls: 2 > dev.msk.0.stats.tx.excess_colls: 0 > dev.msk.0.stats.tx.multi_colls: 0 > dev.msk.0.stats.tx.single_colls: 2 > dev.msk.0.stats.tx.underflows: 0 > > The 2 collisions occurred before I forced the interface to full-duplex. > > Karim.