Date: Sun, 10 Nov 2002 11:21:23 -0500 From: Don Bowman <don@sandvine.com> To: 'Kevin Day' <toasty@dragondata.com>, freebsd-net@freebsd.org Subject: RE: Packet forwarding overhead - with ipfw counting Message-ID: <FE045D4D9F7AED4CBFF1B3B813C8533701022EC3@mail.sandvine.com>
next in thread | raw e-mail | index | archive | help
From: Kevin Day [mailto:toasty@dragondata.com] > When we're pushing 250-300mbits through, we're using about 15% of its > 2.4Ghz P4 Xeon CPU. All of it is in "interrupt" time... that > seems a bit > high, but that'll still let us max things out at 1gbit so we're ok. Try applying these diff to your bge driver, it should reduce your interrupt time substantially in this configuration. I also increased net.inet.ip.intr_queue_maxlen to 500 from 50 since I was seeing drops. Out of curiousity, which motherboard is this? I've been doing some modelling using the e7500 vs serverworks, and the serverworks is significantly better, but no one seems to make a 1U one with 2 PCI-X slots. The e7500 has a 1GB/s half-duplex hublink versus the 3.2GB/s full-duplex one on the GC-LE. Index: if_bge.c =================================================================== RCS file: /cvs/src/sys/dev/bge/if_bge.c,v retrieving revision 1.3.2.18 diff -C5 -r1.3.2.18 if_bge.c *** if_bge.c 2 Nov 2002 18:22:23 -0000 1.3.2.18 --- if_bge.c 10 Nov 2002 16:12:03 -0000 *************** *** 1654,1668 **** error = ENXIO; goto fail; } /* Set default tuneable values. */ sc->bge_stat_ticks = BGE_TICKS_PER_SEC; ! sc->bge_rx_coal_ticks = 150; ! sc->bge_tx_coal_ticks = 150; ! sc->bge_rx_max_coal_bds = 64; ! sc->bge_tx_max_coal_bds = 128; /* Set up ifnet structure */ ifp = &sc->arpcom.ac_if; ifp->if_softc = sc; ifp->if_unit = sc->bge_unit; --- 1654,1692 ---- error = ENXIO; goto fail; } /* Set default tuneable values. */ + /* How often should we update the statistics in host memory? */ sc->bge_stat_ticks = BGE_TICKS_PER_SEC; ! /* The coalescing works as follows: for each of Rx|Tx, there ! * are two tunables: ticks, and packets. The first one to trip ! * will cause an interrupt. For exampple, if the ticks is set to ! * 1us, an interrupt will be generated no more than 1us after ! * a packet has come in. If the bds is set to 10, then the ! * interrupt would be after 10 packets had been received. ! * If ticks=1 and bds=10, then the interrupt will come in ! * min(1us, 10packets time), likely 1us. ! * Tuning these to larger values reduces interrupts at the ! * expense of latency to interactive applications. If you ! * are serving files, make these large. If you are running ! * telnet sessions, make them small. ! * ! * The settings below, 500us means a max interrupt rate ! * of 2000/s due to the ticks elapsing, and 120 means ! * a peak interrupt rate of ~2000/s due to avg packets (512) arriving ! * (for min sized packets this would be 870, for max ! * sized packets it would be 41: 1Gps / ((8*size)+96)) ! */ ! /* RX Interrupt no more than every 500 us */ ! sc->bge_rx_coal_ticks = 500; ! /* TX Interrupt no more than every 500 us */ ! sc->bge_tx_coal_ticks = 500; ! /* RX Interrupt no more than every 120 packets */ ! sc->bge_rx_max_coal_bds = 120; ! /* TX Interrupt no more than every 120 packets */ ! sc->bge_tx_max_coal_bds = 120; /* Set up ifnet structure */ ifp = &sc->arpcom.ac_if; ifp->if_softc = sc; ifp->if_unit = sc->bge_unit; Index: if_bgereg.h =================================================================== RCS file: /cvs/src/sys/dev/bge/if_bgereg.h,v retrieving revision 1.1.2.7 diff -C5 -r1.1.2.7 if_bgereg.h *** if_bgereg.h 2 Nov 2002 18:17:55 -0000 1.1.2.7 --- if_bgereg.h 10 Nov 2002 16:12:21 -0000 *************** *** 2057,2068 **** * Memory management stuff. Note: the SSLOTS, MSLOTS and JSLOTS * values are tuneable. They control the actual amount of buffers * allocated for the standard, mini and jumbo receive rings. */ ! #define BGE_SSLOTS 256 ! #define BGE_MSLOTS 256 #define BGE_JSLOTS 384 #define BGE_JRAWLEN (BGE_JUMBO_FRAMELEN + ETHER_ALIGN + sizeof(u_int64_t)) #define BGE_JLEN (BGE_JRAWLEN + (sizeof(u_int64_t) - \ (BGE_JRAWLEN % sizeof(u_int64_t)))) --- 2057,2068 ---- * Memory management stuff. Note: the SSLOTS, MSLOTS and JSLOTS * values are tuneable. They control the actual amount of buffers * allocated for the standard, mini and jumbo receive rings. */ ! #define BGE_SSLOTS 384 ! #define BGE_MSLOTS 384 #define BGE_JSLOTS 384 #define BGE_JRAWLEN (BGE_JUMBO_FRAMELEN + ETHER_ALIGN + sizeof(u_int64_t)) #define BGE_JLEN (BGE_JRAWLEN + (sizeof(u_int64_t) - \ (BGE_JRAWLEN % sizeof(u_int64_t)))) --don (don@sandvine.com www.sandvine.com) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?FE045D4D9F7AED4CBFF1B3B813C8533701022EC3>