Date: Fri, 8 Feb 2002 22:00:01 -0800 (PST) From: David S Madole <david@madole.net> To: freebsd-bugs@FreeBSD.org Subject: Re: kern/32338: Network to disk write performance low under ATA with DMA Message-ID: <200202090600.g19601t52652@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR kern/32338; it has been noted by GNATS.
From: David S Madole <david@madole.net>
To: Luigi Rizzo <luigi@FreeBSD.org>
Cc: wpaul@FreeBSD.org, freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/32338: Network to disk write performance low under ATA with DMA
Date: Sat, 9 Feb 2002 05:55:07 +0000
Luigi,
Actually, in my case, it's the PCI latency that made all the
difference. With the latency timer set to 32 -- the way my
BIOS sets it up -- overflows occur many times a second. Removing
sis_init helped a little because it caused only one or two
packets to get dropped, instead of 10's or 100's while the card
was begin reset. That change took me from 50Kbyte/sec to
500KByte/sec on FTP on 100MB full-duplex.
Putting in the latency time patch causes two overflows to occur
after boot-up, until the timer reaches 96. Then I never get an
overflow again. From disk-to-disk between two machines with sis0
cards, I get 9.8Mbytes/second on FTP - fast! If I go from disk
to /dev/null (ie GET test /dev/null), I get 11.8MByte/second,
pretty much full wire-rate.
I'm surprised you get a lot of overflows even on the slow
machine, since the card empties through DMA. I can see mbufs
might overflow, but not the card fifo at 33Mhz PCI DMA. You may
need the latency timer increased on that machine, too. You can
always bump it up by hand just to try (ie pciconf -w pci0:4:1
0xc 96).
Thanks for looking into this,
David
On Saturday 09 February 2002 01:57 am, you wrote:
> Hi,
> sorry for the delay with which I reply but i only had a chance
> to test this now.
> It seems that the removal of sis_init from sis_rxeoc() is
> definitely an important part of the fix, as in presence of
> bidirectional traffic an overflow on the receive side will
> cause a reset of the chip and drop all packets queued on
> the transmit side.
>
> Bill, do you think the removal of sis_init() has ill side
> effects otherwise ? I am bombing the interface with 148kpps
> (on a 486-133 so there are plenty of overflows)
> and things seem to work fine, much better indeed than
> when sis_init() was included in sis_rxeoc().
>
> Not sure about the PCI latency register, i would like
> to test this a bit more, as there are many reasons why
> rxeoc is called.
>
> cheers
> luigi
>
> On Mon, Jan 07, 2002 at 08:13:19PM -0500, David S. Madole wrote:
> > >===== Original Message From sos@freebsd.dk =====
> > >
> > >It seems David S Madole wrote:
> > >> >Description:
> > >>
> > >> Data transfers from network to ATA hard drive are very slow.
> > >> Visible with Samba, HTTP PUT, etc., but most easily demonstrated
> > >> with FTP. Only occurs when DMA is enabled on the ATA controller.
> > >>
> > >> Interestingly, only seems to happen when network data is being
> > >> written to the drive. Doing a large write to the drive while
> > >> simultaneously doing a 'ping -f -s 1400' to another node from
> > >> another session does not slow the disk writes significantly.
> > >>
> > >> The drive is a Maxtor 60GB (model number in included dmesg). NIC
> > >> is NetGear with sis driver, although same problem occurs with
> > >> LinkSys card on dc driver. Also occurs with older Maxtor 6GB
> > >> drive.
> > >
> > >Hmm, this sounds like the SiS network card may be using DMA too
> > >and badly at that, making the net card and the ATA driver
> > >compete for the bus...
> >
> > ( my apologies if anyone has received this already, but I don't
> > think the copy I sent previously went out correctly since it never
> > showed up on gnats
> > and I was having mail trouble for a few days )
> >
> > Bill, I copied you since this is your driver.
> >
> > Luigi, I apologize if inappropriate to copy you on this, but I see
> > you've been
> > active in the sis driver lately and thought you might be
> > interested.
> >
> > Søren, thanks for the response. As I mentioned before, I did have
> > this problem with the dc driver as well, on an ADMtek AN985.
> >
> > It turns out that the NIC was unable to DMA it's FIFO into memory
> > fast enough,
> > causing it to overflow and packets to get lost, throttling back
> > TCP. It seems
> > that my BIOS initializes the PCI latency timer on the card to 32,
> > which doesn't let it move enough data at a time to keep up with
> > 100Mbs while competing with the ATA controller.
> >
> > I am heistant to second-quess every BIOS, so I decided to try an
> > adaptive approach and bump the timer up a little bit (32) each time
> > an overflow occurs.
> > On my machine, once it reaches 96 the card is stable and never
> > overflows again. This increased my FTP-to-disk throughput to a
> > little over 9MBps, pretty
> > good. I also changed the driver to not reinitialize the card when
> > an overflow
> > or receive error occurs -- it seems unnecessary and takes a
> > relatively long time, causing many packets to get lost.
> >
> > I also adjusted some NIC settings particular to the National chip
> > to make them
> > conditional on the silicon version, as the latest datasheet
> > specifies. I'm guessing it doesn't really matter much since these
> > are probably just defaults
> > on the later chips.
> >
> > I think some other DMA-based NIC drivers, like dc could probably
> > benefit from
> > this as well. I can fix up the dc driver as well, if this seems
> > like the best
> > way to go, and see if it fixes the issue there, too.
> >
> > By the way, the attached diff is against 4.4-RELEASE.
> >
> > Thanks,
> > Dave
> >
> >
> >
> > *** if_sisreg.h.orig Wed Feb 21 22:17:51 2001
> > --- if_sisreg.h Thu Nov 29 19:28:35 2001
> > ***************
> > *** 76,81 ****
> > --- 76,82 ----
> >
> > /* NS DP83815 registers */
> > #define NS_CLKRUN 0x3C
> > + #define NS_SRR 0x58
> > #define NS_BMCR 0x80
> > #define NS_BMSR 0x84
> > #define NS_PHYIDR1 0x88
> > ***************
> > *** 401,406 ****
> > --- 402,408 ----
> > struct sis_list_data *sis_ldata;
> > struct sis_ring_data sis_cdata;
> > struct callout_handle sis_stat_ch;
> > + device_t sis_dev;
> > };
> >
> > /*
> > *** if_sis.c.orig Wed Feb 21 22:17:51 2001
> > --- if_sis.c Thu Nov 29 19:41:31 2001
> > ***************
> > *** 723,728 ****
> > --- 723,730 ----
> > unit = device_get_unit(dev);
> > bzero(sc, sizeof(struct sis_softc));
> >
> > + sc->sis_dev = dev;
> > +
> > if (pci_get_device(dev) == SIS_DEVICEID_900)
> > sc->sis_type = SIS_TYPE_900;
> > if (pci_get_device(dev) == SIS_DEVICEID_7016)
> > ***************
> > *** 1159,1166 ****
> > void sis_rxeoc(sc)
> > struct sis_softc *sc;
> > {
> > sis_rxeof(sc);
> > - sis_init(sc);
> > return;
> > }
> >
> > --- 1161,1183 ----
> > void sis_rxeoc(sc)
> > struct sis_softc *sc;
> > {
> > + int latency;
> > +
> > + /*
> > + * The BIOS may have initialized the maximum latency timer
> > + * too low to be able to keep up with a 100Mbs stream when
> > + * heavy disk or other DMA is taking place. Try to correct
> > + * for this adaptively by bumping it up a little each time
> > + * the receive FIFO overflows.
> > + */
> > +
> > + latency = pci_read_config(sc->sis_dev, PCIR_LATTIMER, 1);
> > + if (latency < 255) {
> > + if ((latency += 32) > 255) latency = 255;
> > + pci_write_config(sc->sis_dev, PCIR_LATTIMER, latency, 1);
> > + }
> > +
> > sis_rxeof(sc);
> > return;
> > }
> >
> > ***************
> > *** 1293,1305 ****
> > sis_txeof(sc);
> >
> > if ((status & SIS_ISR_RX_DESC_OK) ||
> > ! (status & SIS_ISR_RX_OK))
> > sis_rxeof(sc);
> >
> > ! if ((status & SIS_ISR_RX_ERR) ||
> > ! (status & SIS_ISR_RX_OFLOW)) {
> > sis_rxeoc(sc);
> > - }
> >
> > if (status & SIS_ISR_SYSERR) {
> > sis_reset(sc);
> > --- 1310,1321 ----
> > sis_txeof(sc);
> >
> > if ((status & SIS_ISR_RX_DESC_OK) ||
> > ! (status & SIS_ISR_RX_OK) ||
> > ! (status & SIS_ISR_RX_ERR))
> > sis_rxeof(sc);
> >
> > ! if (status & SIS_ISR_RX_OFLOW)
> > sis_rxeoc(sc);
> >
> > if (status & SIS_ISR_SYSERR) {
> > sis_reset(sc);
> > ***************
> > *** 1562,1569 ****
> > * performance." Note however that at least three
> > * of the registers are listed as "reserved" in
> > * the register map, so who knows what they do.
> > */
> > ! if (sc->sis_type == SIS_TYPE_83815) {
> > CSR_WRITE_4(sc, NS_PHY_PAGE, 0x0001);
> > CSR_WRITE_4(sc, NS_PHY_CR, 0x189C);
> > CSR_WRITE_4(sc, NS_PHY_TDATA, 0x0000);
> > --- 1578,1593 ----
> > * performance." Note however that at least three
> > * of the registers are listed as "reserved" in
> > * the register map, so who knows what they do.
> > + *
> > + * An appararently later version (December 2000)
> > + * has this data on page 78 and qualifies it as
> > + * applying only to silicon version 203h, which
> > + * is aparently a typo'd reference to version 302h
> > + * as referred to on page 64.
> > */
> > ! if (sc->sis_type == SIS_TYPE_83815 &&
> > ! CSR_READ_4(sc, NS_SRR) == 0x0302) {
> > !
> > CSR_WRITE_4(sc, NS_PHY_PAGE, 0x0001);
> > CSR_WRITE_4(sc, NS_PHY_CR, 0x189C);
> > CSR_WRITE_4(sc, NS_PHY_TDATA, 0x0000);
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-bugs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200202090600.g19601t52652>