Date: Thu, 16 Oct 2008 08:58:36 +0900 From: Pyun YongHyeon <pyunyh@gmail.com> To: Koen Martens <gmc@sonologic.nl> Cc: freebsd-current@freebsd.org, Kudo Chien <ckchien@gmail.com> Subject: Re: msk watchdog timeout Message-ID: <20081015235836.GB22992@cdnetworks.co.kr> In-Reply-To: <20081015135651.GC12650@latitude.dh2.sono> References: <20071001080058.GE18658@cdnetworks.co.kr> <8bebfc960710010707o4b7e9b2cif086e35b4f8f736d@mail.gmail.com> <20071002004521.GH18658@cdnetworks.co.kr> <8bebfc960710012041h205216ecl75d44e0c9ba69068@mail.gmail.com> <20071002035814.GJ18658@cdnetworks.co.kr> <8bebfc960710022231m4b26a07ahdaef145e07862fb6@mail.gmail.com> <20071004011348.GC30781@cdnetworks.co.kr> <20081015122233.GA11699@latitude.dh2.sono> <20081015123440.GJ14769@cdnetworks.co.kr> <20081015135651.GC12650@latitude.dh2.sono>
next in thread | previous in thread | raw e-mail | index | archive | help
--VrqPEDrXMn8OVzN4
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
On Wed, Oct 15, 2008 at 03:56:52PM +0200, Koen Martens wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Wed, Oct 15, 2008 at 09:34:40PM +0900, Pyun YongHyeon wrote:
> > On Wed, Oct 15, 2008 at 02:22:34PM +0200, Koen Martens wrote:
> > > The problem occurs under load (rsyncing tens of gigabytes over
> > > gigabit link for example). I tried configuring the switch port
> > > down to 100MB, in the hopes that msk0 would be more stable. It
> > > is, but it still goes down after a while with watchdog timeouts.
> > >
> > > I am now running it with msi disabled, it appears it lasts longer
> > > than before now. But judging by what others said on this subject
> > > already, it might still go wrong after as much as a month.
> > >
> > > Also, I've never had these problems when the machine was still
> > > on 6.x with the myk driver. Only after I upgraded it this tuesday
> > > to RELENG_7, trouble started.
> > >
> > > This is a server that I need to put back into production. I could
> > > give you some time on it before I do that, but that'd have to be
> > > *right now* so i guess that won't work out really.
> > >
> > > I'll probably install a nic to be used instead of the built-in
> > > yukon interface, to get back the required stability.
> > >
> >
> > I'm not sure whether 88E8050 also has RAM buffer. Youkon
> > controllers seems to have silicon bugs for hardwares with RAM
> > buffer. msk(4) in HEAD has workaround code for the silicon bug.
> > Would you try latest msk(4) from HEAD?(Just copy
> > if_msk.c/if_mskreg.h from HEAD to your box and rebuild kernel.)
> > Also show me verbosed boot message(msk(4) related one would be
> > enough).
>
> No dice, compile bails out:
>
> o-pointer-sign -fformat-extensions -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -mno-align-long-strings -mpreferred-stack-boundary=2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding -Werror /usr/src/sys/dev/msk/if_msk.c
> /usr/src/sys/dev/msk/if_msk.c:845:50: error: macro "MEXTADD" passed 8 arguments, but takes just 7
> /usr/src/sys/dev/msk/if_msk.c: In function 'msk_jumbo_newbuf':
> /usr/src/sys/dev/msk/if_msk.c:844: error: 'MEXTADD' undeclared (first use in this function)
> /usr/src/sys/dev/msk/if_msk.c:844: error: (Each undeclared identifier is reported only once
> /usr/src/sys/dev/msk/if_msk.c:844: error: for each function it appears in.)
> *** Error code 1
>
> Stop in /usr/obj/usr/src/sys/GENERIC.
> *** Error code 1
>
> Stop in /usr/src.
> *** Error code 1
>
> Stop in /usr/src.
> postel#
>
> I'll have to get back to you about the verbose boot, as i have no remote
> console on that machine.
>
Oops! Sorry, I forgot there is difference in jumbo frame buffer
handling. Try attached patch and show me verbosed boot message.
--
Regards,
Pyun YongHyeon
--VrqPEDrXMn8OVzN4
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="msk.watchdog.diff"
Index: if_msk.c
===================================================================
--- if_msk.c (revision 183165)
+++ if_msk.c (working copy)
@@ -244,6 +244,9 @@
static int msk_handle_events(struct msk_softc *);
static void msk_handle_hwerr(struct msk_if_softc *, uint32_t);
static void msk_intr_hwerr(struct msk_softc *);
+#ifndef __NO_STRICT_ALIGNMENT
+static __inline void msk_fixup_rx(struct mbuf *);
+#endif
static void msk_rxeof(struct msk_if_softc *, uint32_t, int);
static void msk_jumbo_rxeof(struct msk_if_softc *, uint32_t, int);
static void msk_txeof(struct msk_if_softc *, int);
@@ -783,7 +786,12 @@
return (ENOBUFS);
m->m_len = m->m_pkthdr.len = MCLBYTES;
- m_adj(m, ETHER_ALIGN);
+ if ((sc_if->msk_flags & MSK_FLAG_RAMBUF) == 0)
+ m_adj(m, ETHER_ALIGN);
+#ifndef __NO_STRICT_ALIGNMENT
+ else
+ m_adj(m, MSK_RX_BUF_ALIGN);
+#endif
if (bus_dmamap_load_mbuf_sg(sc_if->msk_cdata.msk_rx_tag,
sc_if->msk_cdata.msk_rx_sparemap, m, segs, &nsegs,
@@ -840,7 +848,12 @@
return (ENOBUFS);
}
m->m_pkthdr.len = m->m_len = MSK_JLEN;
- m_adj(m, ETHER_ALIGN);
+ if ((sc_if->msk_flags & MSK_FLAG_RAMBUF) == 0)
+ m_adj(m, ETHER_ALIGN);
+#ifndef __NO_STRICT_ALIGNMENT
+ else
+ m_adj(m, MSK_RX_BUF_ALIGN);
+#endif
if (bus_dmamap_load_mbuf_sg(sc_if->msk_cdata.msk_jumbo_rx_tag,
sc_if->msk_cdata.msk_jumbo_rx_sparemap, m, segs, &nsegs,
@@ -1041,14 +1054,16 @@
{
int next;
int i;
- uint8_t val;
/* Get adapter SRAM size. */
- val = CSR_READ_1(sc, B2_E_0);
- sc->msk_ramsize = (val == 0) ? 128 : val * 4;
+ sc->msk_ramsize = CSR_READ_1(sc, B2_E_0) * 4;
if (bootverbose)
device_printf(sc->msk_dev,
"RAM buffer size : %dKB\n", sc->msk_ramsize);
+ if (sc->msk_ramsize == 0)
+ return (0);
+
+ sc->msk_pflags |= MSK_FLAG_RAMBUF;
/*
* Give receiver 2/3 of memory and round down to the multiple
* of 1024. Tx/Rx RAM buffer size of Yukon II shoud be multiple
@@ -1412,6 +1427,7 @@
sc_if->msk_if_dev = dev;
sc_if->msk_port = port;
sc_if->msk_softc = sc;
+ sc_if->msk_flags = sc->msk_pflags;
sc->msk_if[port] = sc_if;
/* Setup Tx/Rx queue register offsets. */
if (port == MSK_PORT_A) {
@@ -1976,6 +1992,7 @@
struct msk_rxdesc *jrxd;
struct msk_jpool_entry *entry;
uint8_t *ptr;
+ bus_size_t rxalign;
int error, i;
mtx_init(&sc_if->msk_jlist_mtx, "msk_jlist_mtx", NULL, MTX_DEF);
@@ -2107,9 +2124,16 @@
goto fail;
}
+ rxalign = 1;
+ /*
+ * Workaround hardware hang which seems to happen when Rx buffer
+ * is not aligned on multiple of FIFO word(8 bytes).
+ */
+ if ((sc_if->msk_flags & MSK_FLAG_RAMBUF) != 0)
+ rxalign = MSK_RX_BUF_ALIGN;
/* Create tag for Rx buffers. */
error = bus_dma_tag_create(sc_if->msk_cdata.msk_parent_tag,/* parent */
- 1, 0, /* alignment, boundary */
+ rxalign, 0, /* alignment, boundary */
BUS_SPACE_MAXADDR, /* lowaddr */
BUS_SPACE_MAXADDR, /* highaddr */
NULL, NULL, /* filter, filterarg */
@@ -2918,6 +2942,23 @@
return (0);
}
+#ifndef __NO_STRICT_ALIGNMENT
+static __inline void
+msk_fixup_rx(struct mbuf *m)
+{
+ int i;
+ uint16_t *src, *dst;
+
+ src = mtod(m, uint16_t *);
+ dst = src - 3;
+
+ for (i = 0; i < (m->m_len / sizeof(uint16_t) + 1); i++)
+ *dst++ = *src++;
+
+ m->m_data -= (MSK_RX_BUF_ALIGN - ETHER_ALIGN);
+}
+#endif
+
static void
msk_rxeof(struct msk_if_softc *sc_if, uint32_t status, int len)
{
@@ -2955,6 +2996,10 @@
}
m->m_pkthdr.rcvif = ifp;
m->m_pkthdr.len = m->m_len = len;
+#ifndef __NO_STRICT_ALIGNMENT
+ if ((sc_if->msk_flags & MSK_FLAG_RAMBUF) != 0)
+ msk_fixup_rx(m);
+#endif
ifp->if_ipackets++;
/* Check for VLAN tagged packets. */
if ((status & GMR_FS_VLAN) != 0 &&
@@ -3008,6 +3053,10 @@
}
m->m_pkthdr.rcvif = ifp;
m->m_pkthdr.len = m->m_len = len;
+#ifndef __NO_STRICT_ALIGNMENT
+ if ((sc_if->msk_flags & MSK_FLAG_RAMBUF) != 0)
+ msk_fixup_rx(m);
+#endif
ifp->if_ipackets++;
/* Check for VLAN tagged packets. */
if ((status & GMR_FS_VLAN) != 0 &&
@@ -3677,7 +3726,7 @@
/* Configure hardware VLAN tag insertion/stripping. */
msk_setvlan(sc_if, ifp);
- if (sc->msk_hw_id == CHIP_ID_YUKON_EC_U) {
+ if ((sc_if->msk_flags & MSK_FLAG_RAMBUF) == 0) {
/* Set Rx Pause threshould. */
CSR_WRITE_1(sc, MR_ADDR(sc_if->msk_port, RX_GMF_LP_THR),
MSK_ECU_LLPP);
@@ -3790,6 +3839,8 @@
int ltpp, utpp;
sc = sc_if->msk_softc;
+ if ((sc_if->msk_flags & MSK_FLAG_RAMBUF) == 0)
+ return;
/* Setup Rx Queue. */
CSR_WRITE_1(sc, RB_ADDR(sc_if->msk_rxq, RB_CTRL), RB_RST_CLR);
Index: if_mskreg.h
===================================================================
--- if_mskreg.h (revision 183165)
+++ if_mskreg.h (working copy)
@@ -2158,6 +2158,7 @@
#define MSK_TX_RING_CNT 256
#define MSK_RX_RING_CNT 256
+#define MSK_RX_BUF_ALIGN 8
#define MSK_JUMBO_RX_RING_CNT MSK_RX_RING_CNT
#define MSK_STAT_RING_CNT ((1 + 3) * (MSK_TX_RING_CNT + MSK_RX_RING_CNT))
#define MSK_MAXTXSEGS 32
@@ -2307,6 +2308,7 @@
uint32_t msk_coppertype;
uint32_t msk_intrmask;
uint32_t msk_intrhwemask;
+ uint32_t msk_pflags;
int msk_suspended;
int msk_clock;
int msk_msi;
@@ -2348,6 +2350,8 @@
int msk_phytype;
int msk_phyaddr;
int msk_link;
+ uint32_t msk_flags;
+#define MSK_FLAG_RAMBUF 0x0010
struct callout msk_tick_ch;
int msk_watchdog_timer;
uint32_t msk_txq; /* Tx. Async Queue offset */
--VrqPEDrXMn8OVzN4--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20081015235836.GB22992>