Date: Wed, 12 Sep 2007 17:21:13 -0400 From: "Robert Wojciechowski" <robertw@expressyard.com> To: <pyunyh@gmail.com> Cc: shigeaki@f.csce.kyushu-u.ac.jp, freebsd-net@freebsd.org, Josh Mouch <jmouch@expressyard.com> Subject: RE: FreeBSD nfe driver and IPMI cards Message-ID: <85D4F2C294E8434CA0AF775741532686623694@server1.ssgi.local> In-Reply-To: <20070912004554.GA8992@cdnetworks.co.kr> References: <85D4F2C294E8434CA0AF775741532686623679@server1.ssgi.local> <20070912004554.GA8992@cdnetworks.co.kr>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] > > I'm the FreeBSD nfe driver from > > http://www.f.csce.kyushu-u.ac.jp/~shigeaki/software/freebsd-nfe.html > > with FreeBSD 6-stable with good results for the most part. The only > > issue I've experienced is that during a detach/shutdown of if_nfe, > the > > IPMI IP address I have set on my servers ceases to respond as well > as > > the ability to manage the servers. > > > > > > > > I traced the problem down to nfe_stop() and the fact that it > completely > > disables the Rx and Tx on the NIC. I have patched the driver to not > > disable the Rx/Tx and IPMI continues to work after a 'ifconfig nfe0 > > down', 'shutdown -p now', etc. > > > > > > > > Does anyone have any comments on this change I've made and any > possible > > side effects? Can this be included in the mainstream distribution of > the > > Because MAC is still alive if's possible to recieve a packet. All DMA > maps are unloaded and buffers are already freed in nfe_stop so it > would cause panic I guess. But I'm not familiar with IPMI so I'm not > sure. Interesting, that is an issue that was also resolved by the forcedeth driver in Linux by resetting in nv_close to prevent DMA into freed memory. > > > nfe drivers (and updated in 7-CURRENT) without causing any adverse > > problems? > > > > I have no experience on IPMI but the change you've made would not > completely solve the issue. I guess supporting IPMI needs lots of > more work including: > o Autodetect IPMI capability. > o Autodetect active IPMI session in device attach and don't blindly > reset MAC/PHY. > o Don't blindly stop Tx/Rx on device detach. > Given that lack of publicly available datasheet for the hardware > supporing IPMI would be severly limited. Fortunately Linux seems to > have basic IPMI support in their forcedeth driver. Their code doesn't > easy to read but you may see what should be done in driver. However > I have no idea what we can do when active IPMI session is present in > driver attach phase. Normally PHY driver would reset PHY hardware > itself in driver attach which in turn would result in losing the IPMI > connection. > Since we have no idea how to auto-detect IPMI, I added a device sysctl (enable_ipmi) to control the behavior I'm testing in-house right now. If the enable_ipmi sysctl is enabled, it'll: * resets after disabling the Rx/Tx and before disabling interrupts * re-enables the Rx/Tx after the ring buffers are freed, just like forcedeth but also enabling the Tx for IPMI (Linux just enabled the Rx for WOL) I have attached my new patch to this email that does the above, guarded by the sysctl. I have no idea how to handle the second case you mentioned, during attach. It does indeed cause a disruption in IPMI, but only for a few seconds. Where is the MAC and PHY reset and if it wasn't reset when using IPMI, what problems could it cause? -- Robert [-- Attachment #2 --] diff -ur nfe-20070816/if_nfe.c nfe-20070816-ssgi/if_nfe.c --- nfe-20070816/if_nfe.c Tue Sep 11 07:36:24 2007 +++ nfe-20070816-ssgi/if_nfe.c Wed Sep 12 13:02:37 2007 @@ -24,7 +24,7 @@ __FBSDID("$FreeBSD: src/sys/dev/nfe/if_nfe.c,v 1.19 2007/07/24 01:11:00 yongari Exp $"); /* Uncomment the following line to enable polling. */ -#define DEVICE_POLLING +/* #define DEVICE_POLLING */ #ifdef HAVE_KERNEL_OPTION_HEADERS #include "opt_device_polling.h" @@ -81,6 +81,7 @@ static int nfe_resume(device_t); static void nfe_shutdown(device_t); static void nfe_power(struct nfe_softc *); +static void nfe_reset(struct nfe_softc *); static int nfe_miibus_readreg(device_t, int, int); static int nfe_miibus_writereg(device_t, int, int, int); static void nfe_miibus_statchg(device_t); @@ -127,6 +128,7 @@ static void nfe_dma_map_segs(void *, bus_dma_segment_t *, int, int); static int sysctl_int_range(SYSCTL_HANDLER_ARGS, int, int); +static int sysctl_hw_nfe_enable_ipmi(SYSCTL_HANDLER_ARGS); static int sysctl_hw_nfe_proc_limit(SYSCTL_HANDLER_ARGS); #ifdef NFE_DEBUG @@ -534,6 +536,23 @@ SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), + OID_AUTO, "enable_ipmi", CTLTYPE_INT | CTLFLAG_RW, + &sc->nfe_enable_ipmi, 0, sysctl_hw_nfe_enable_ipmi, "I", + "enable IPMI"); + + sc->nfe_enable_ipmi = NFE_ENABLE_IPMI_DEFAULT; + error = resource_int_value(device_get_name(dev), device_get_unit(dev), + "enable_ipmi", &sc->nfe_enable_ipmi); + if (error == 0) { + if (sc->nfe_process_limit < 0 || sc->nfe_process_limit > 1) { + device_printf(dev, "enable_ipmi value out of range; " + "using default: %d\n", NFE_ENABLE_IPMI_DEFAULT); + sc->nfe_enable_ipmi = NFE_ENABLE_IPMI_DEFAULT; + } + } + + SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), + SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "process_limit", CTLTYPE_INT | CTLFLAG_RW, &sc->nfe_process_limit, 0, sysctl_hw_nfe_proc_limit, "I", "max number of Rx events to process"); @@ -779,12 +798,7 @@ if ((sc->nfe_flags & NFE_PWR_MGMT) == 0) return; - NFE_WRITE(sc, NFE_RXTX_CTL, NFE_RXTX_RESET | NFE_RXTX_BIT2); - NFE_WRITE(sc, NFE_MAC_RESET, NFE_MAC_RESET_MAGIC); - DELAY(100); - NFE_WRITE(sc, NFE_MAC_RESET, 0); - DELAY(100); - NFE_WRITE(sc, NFE_RXTX_CTL, NFE_RXTX_BIT2); + nfe_reset(sc); pwr = NFE_READ(sc, NFE_PWR2_CTL); pwr &= ~NFE_PWR2_WAKEUP_MASK; if (sc->nfe_revid >= 0xa3 && @@ -794,6 +808,15 @@ NFE_WRITE(sc, NFE_PWR2_CTL, pwr); } +static void +nfe_reset(struct nfe_softc *sc) +{ + NFE_WRITE(sc, NFE_RXTX_CTL, NFE_RXTX_RESET | NFE_RXTX_BIT2); + NFE_WRITE(sc, NFE_MAC_RESET, NFE_MAC_RESET_MAGIC); + DELAY(100); + NFE_WRITE(sc, NFE_MAC_RESET, 0); + DELAY(100); +} static void nfe_miibus_statchg(device_t dev) @@ -3009,6 +3032,11 @@ /* disable Rx */ NFE_WRITE(sc, NFE_RX_CTL, 0); + /* reset the Tx/Rx so the NIC does not DMA into freed memory */ + if (&sc->nfe_enable_ipmi) { + nfe_reset(sc); + } + /* disable interrupts */ nfe_disable_intr(sc); @@ -3055,6 +3083,12 @@ tdata->m = NULL; } } + + /* enable the Tx/Rx for IPMI support */ + if (&sc->nfe_enable_ipmi) { + NFE_WRITE(sc, NFE_TX_CTL, NFE_TX_START); + NFE_WRITE(sc, NFE_RX_CTL, NFE_RX_START); + } } @@ -3210,4 +3244,11 @@ return (sysctl_int_range(oidp, arg1, arg2, req, NFE_PROC_MIN, NFE_PROC_MAX)); +} + + +static int +sysctl_hw_nfe_enable_ipmi(SYSCTL_HANDLER_ARGS) +{ + return (sysctl_int_range(oidp, arg1, arg2, req, 0, 1)); } Only in nfe-20070816-ssgi: if_nfe.c.orig Only in nfe-20070816-ssgi: if_nfe.c.rej Binary files nfe-20070816/if_nfe.ko and nfe-20070816-ssgi/if_nfe.ko differ Binary files nfe-20070816/if_nfe.o and nfe-20070816-ssgi/if_nfe.o differ diff -ur nfe-20070816/if_nfereg.h nfe-20070816-ssgi/if_nfereg.h --- nfe-20070816/if_nfereg.h Wed Aug 15 20:51:18 2007 +++ nfe-20070816-ssgi/if_nfereg.h Wed Sep 12 12:45:37 2007 @@ -22,6 +22,7 @@ #define NFE_JUMBO_RX_RING_COUNT NFE_RX_RING_COUNT #define NFE_TX_RING_COUNT 256 +#define NFE_ENABLE_IPMI_DEFAULT 0 #define NFE_PROC_DEFAULT ((NFE_RX_RING_COUNT * 3) / 4) #define NFE_PROC_MIN 50 #define NFE_PROC_MAX (NFE_RX_RING_COUNT - 1) diff -ur nfe-20070816/if_nfevar.h nfe-20070816-ssgi/if_nfevar.h --- nfe-20070816/if_nfevar.h Wed Aug 15 20:51:18 2007 +++ nfe-20070816-ssgi/if_nfevar.h Wed Sep 12 12:35:47 2007 @@ -115,6 +115,7 @@ int nfe_suspended; int nfe_framesize; int nfe_process_limit; + int nfe_enable_ipmi; int nfe_force_tx; uint32_t nfe_irq_status; uint32_t nfe_irq_mask;
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?85D4F2C294E8434CA0AF775741532686623694>