Date: Wed, 12 Oct 2011 10:03:47 -0700 From: YongHyeon PYUN <pyunyh@gmail.com> To: Karim <fodillemlinkarim@gmail.com> Cc: freebsd-net@freebsd.org, Kevin Oberman <kob6558@gmail.com> Subject: Re: if_msk.c link negotiation / packet drops Message-ID: <20111012170347.GA9138@michelle.cdnetworks.com> In-Reply-To: <4E959F06.6040906@gmail.com> References: <4E94637A.5090607@gmail.com> <20111011171029.GA5661@michelle.cdnetworks.com> <CAN6yY1tWQZwdqgYdN3uBBdXiGQ2OFDMYbSjhEUeTimHjBnR9iA@mail.gmail.com> <4E959F06.6040906@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Oct 12, 2011 at 10:07:02AM -0400, Karim wrote:
> On 11-10-11 01:31 PM, Kevin Oberman wrote:
> >On Tue, Oct 11, 2011 at 10:10 AM, YongHyeon PYUN<pyunyh@gmail.com> wrote:
> >>On Tue, Oct 11, 2011 at 11:40:42AM -0400, Karim wrote:
> >>>Hi List,
> >>>
> >>>Using a Marvell NIC plugged into a CISCO switch I see the
> >>>auto-negotiation failing and even when forcing the device to full-duplex
> >>>we sometimes see packet drops.
> >>>
> >>>Here is the device description from dmesg:
> >>>
> >>>mskc0:<Marvell Yukon 88E8053 Gigabit Ethernet> port 0xbe00-0xbeff mem
> >>>0xfdefc000-0xfdefffff irq 16 at device 0.0 on pci1
> >>>msk0:<Marvell Technology Group Ltd. Yukon EC Id 0xb6 Rev 0x02> on mskc0
> >>>msk0: Ethernet address: 00:03:2d:09:94:52
> >>>miibus0:<MII bus> on msk0
> >>>e1000phy0:<Marvell 88E1111 Gigabit PHY> PHY 0 on miibus0
> >>>e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
> >>>1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
> >>>mskc0: [ITHREAD]
> >>>
> >>>The switch its plugged in (Cisco) is configured for 100baseTX
> >>>full-duplex.
> >>>
> >>>ifconfig reports:
> >>>
> >>>msk0:
> >>>flags=608843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,SATELLITE,LAN_NET>
> >>>metric 0 mtu 1500
> >>> options=40018<VLAN_MTU,VLAN_HWTAGGING>
> >>The flags and options show that you're using very customized
> >>driver, right?
> >>
> >>> ether 00:03:2d:09:94:52
> >>> inet 192.168.122.7 netmask 0xffffff00 broadcast 192.168.122.255
> >>> media: Ethernet autoselect (100baseTX<half-duplex>)
> >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >>
> >>Resolved duplex is half so I guess it would be normal to see
> >>dropped frames which may be triggered by collision.
> >You have a duplex mis-match. If you are hard setting the remote end to
> >full, the local end must also be configured o full. Auto-configuration
> >of duplex requires that both ends run auto-config. When one end is set
> >to not do auto-config, the other end SHOULD always set to half-duplex.
> >This is part of 802.3 that is a carry-over from the days when hubs and
> >coax dominated, so the default was declared to be half. Since so much
> >hardware now exists with that default, changing it ill never happen.
> >
> >Either set your computer to full duplex or turn on auto-configuration
> >on the Cisco.
> >
> >Very little hardware now in service fails to auto-config correctly,
> >but the practice of lacking down the duplex setting became common in
> >the early days of full-duplex when it was not yet a standard and many
> >Ethernet chip-sets didn't play nice with others. Things would be much
> >better if people would just stop hard-setting the duplex, but old, old
> >habits and memes die hard.
> >
> >Also, contrary to common belief, collisions are NOT errors. They are a
> >normal part of half-dulpex Ethernet operation and do NOT result in
> >packets being dropped. Only "excessive collisions do and they ARE a
> >real error and a clear indication that something is wrong.
> Hi,
>
> Thanks for the feedback and detailed information.
>
> I have to clarify here; I get the issue even with forcing full duplex.
>
> The driver modifications are minor and shouldn't affect link negotiation
> or link state. Its also interesting that this problem only shows up with
> Cisco's switches and faced with other types of switches the issue does
> not come up.
>
Right. It would be also interesting to see MAC statistics of Cisco
switch and check both parties agree on resolved
speed/duplex/flow-control.
> Also, nothing like collisions or missed/dropped packets can be found in
> msk_stats (see below) that somewhat relate to the issue I'm seeing. One
> thing interesting is the ifm_status value from msk_mediastatus. It often
> changes from (IFM_AVALID | IFM_ACTIVE) which is 0x3 to 0x1 (IFM_AVALID)
> for no reason I can tell.
>
Hmm, that indicates driver lost established link. msk(4) will
detect this condition and stop RX/TX MACs until it knows PHY
re-established a link. This may be the reason why you see occasional
packet drops. However I don't know why PHY loses established link
in the middle of working.
> From the code in e1000phy_status:
>
> static void
> e1000phy_status(struct mii_softc *sc)
> {
> struct mii_data *mii = sc->mii_pdata;
> int bmcr, bmsr, ssr;
>
> mii->mii_media_status = IFM_AVALID;
> mii->mii_media_active = IFM_ETHER;
>
> bmsr = PHY_READ(sc, E1000_SR) | PHY_READ(sc, E1000_SR);
> bmcr = PHY_READ(sc, E1000_CR);
> ssr = PHY_READ(sc, E1000_SSR);
>
> if (bmsr & E1000_SR_LINK_STATUS)
> mii->mii_media_status |= IFM_ACTIVE;
>
>
> I can see the bmsr & E1000_SR_LINK_STATUS check failing when the problem
> occurs. As a side note why are we ORing the same call twice isn't the
> same thing as calling it once:
>
> bmsr = PHY_READ(sc, E1000_SR) | PHY_READ(sc, E1000_SR);
>
The E1000_SR_LINK_STATUS bit is latched low so it should be read
twice. If you want to read once use E1000_SSR_LINK bit of
E1000_SSR register but I remember that bit was not reliable on some
PHY models.
By chance, does your back-ported driver include r222219?
If yes, did you cold boot after applying the change?
Warm boot does have effect.
> As requested here is my msk0 stats output right after the problem showed up:
>
> dev.msk.0.stats.rx.ucast_frames: 58103886
> dev.msk.0.stats.rx.bcast_frames: 0
> dev.msk.0.stats.rx.pause_frames: 0
> dev.msk.0.stats.rx.mcast_frames: 0
> dev.msk.0.stats.rx.crc_errs: 0
> dev.msk.0.stats.rx.good_octets: 5927739395
> dev.msk.0.stats.rx.bad_octets: 0
> dev.msk.0.stats.rx.frames_64: 53
> dev.msk.0.stats.rx.frames_65_127: 58091128
> dev.msk.0.stats.rx.frames_128_255: 12545
> dev.msk.0.stats.rx.frames_256_511: 40
> dev.msk.0.stats.rx.frames_512_1023: 89
> dev.msk.0.stats.rx.frames_1024_1518: 31
> dev.msk.0.stats.rx.frames_1519_max: 0
> dev.msk.0.stats.rx.frames_too_long: 0
> dev.msk.0.stats.rx.jabbers: 0
> dev.msk.0.stats.rx.overflows: 0
> dev.msk.0.stats.tx.ucast_frames: 58104799
> dev.msk.0.stats.tx.bcast_frames: 53
> dev.msk.0.stats.tx.pause_frames: 0
> dev.msk.0.stats.tx.mcast_frames: 0
> dev.msk.0.stats.tx.octets: 5927439760
> dev.msk.0.stats.tx.frames_64: 547
> dev.msk.0.stats.tx.frames_65_127: 58091680
> dev.msk.0.stats.tx.frames_128_255: 12576
> dev.msk.0.stats.tx.frames_256_511: 17
> dev.msk.0.stats.tx.frames_512_1023: 1
> dev.msk.0.stats.tx.frames_1024_1518: 32
> dev.msk.0.stats.tx.frames_1519_max: 0
> dev.msk.0.stats.tx.colls: 2
> dev.msk.0.stats.tx.late_colls: 2
> dev.msk.0.stats.tx.excess_colls: 0
> dev.msk.0.stats.tx.multi_colls: 0
> dev.msk.0.stats.tx.single_colls: 2
> dev.msk.0.stats.tx.underflows: 0
>
> The 2 collisions occurred before I forced the interface to full-duplex.
>
> Karim.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20111012170347.GA9138>