Date: Thu, 23 Jan 2003 12:13:58 +0100 (CET) From: Hartmut Brandt <brandt@fokus.fraunhofer.de> To: FreeBSD-gnats-submit@FreeBSD.org Subject: kern/47393: netgraph can run out of queue items Message-ID: <200301231113.h0NBDwiU001087@catssrv.fokus.gmd.de>
next in thread | raw e-mail | index | archive | help
>Number: 47393
>Category: kern
>Synopsis: netgraph can run out of queue items
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Thu Jan 23 03:20:04 PST 2003
>Closed-Date:
>Last-Modified:
>Originator: Hartmut Brandt
>Release: FreeBSD 5.0-CURRENT sparc64
>Organization:
FhI Fokus
>Environment:
System: FreeBSD catssrv.fokus.gmd.de 5.0-CURRENT FreeBSD 5.0-CURRENT #8: Wed Jan 22 18:21:10 CET 2003 hbb@catssrv.fokus.gmd.de:/opt/obj/usr/src/sys/CATSSRV sparc64
>Description:
netgraph has an upper limit on queue items. Queue items are used to hold
various things in the input queues of nodes. If there is a high number of
messages flowing through the system, especially if some of the nodes have
put hooks into forced queuing mode, netgraph very fast runs out of queue items
and blocks.
>How-To-Repeat:
Build a very simple node, that forwards messages between two hooks. When
attaching the hooks let the node put its hooks into queing mode. Then insert
a large number of message (several 10k per second) into the system and observe
that the netgraph system will block.
>Fix:
The attached patch make loader tuneables out of two of the key variables
that control this behaviour: the maximum number of queue items ever allocated
by netgraph and the maximum number of free queue items to be cached on a local
queue. This allows one to tune the system for very high netgraph througput and
to trade memory vs. throughput (with the second variable). Both variables are
also available as read-only sysctls.
Index: sys/boot/forth/loader.conf
===================================================================
RCS file: /home/cvs/freebsd/src/sys/boot/forth/loader.conf,v
retrieving revision 1.68
diff -c -r1.68 loader.conf
*** sys/boot/forth/loader.conf 26 Nov 2002 13:55:50 -0000 1.68
--- sys/boot/forth/loader.conf 15 Jan 2003 14:50:05 -0000
***************
*** 102,107 ****
--- 102,109 ----
#debug.ktr.cpumask="0xf" # Bitmask of CPUs to enable KTR on
#debug.ktr.mask="0x1200" # Bitmask of KTR events to enable
#debug.ktr.verbose="1" # Enable console dump of KTR events
+ #net.graph.maxalloc="128" # Maximum number of queue items to allocate
+ #net.graph.ngqfreemax="64" # Maximum number of free queue items to cache
##############################################################
Index: sys/netgraph/ng_base.c
===================================================================
RCS file: /home/cvs/freebsd/src/sys/netgraph/ng_base.c,v
retrieving revision 1.65
diff -c -r1.65 ng_base.c
*** sys/netgraph/ng_base.c 1 Jan 2003 18:48:55 -0000 1.65
--- sys/netgraph/ng_base.c 15 Jan 2003 14:50:05 -0000
***************
*** 3017,3024 ****
static int allocated; /* number of items malloc'd */
static int maxalloc = 128; /* limit the damage of a leak */
! static const int ngqfreemax = 64;/* cache at most this many */
static const int ngqfreelow = 4; /* try malloc if free < this */
static volatile int ngqfreesize; /* number of cached entries */
#ifdef NETGRAPH_DEBUG
--- 3017,3034 ----
static int allocated; /* number of items malloc'd */
+
static int maxalloc = 128; /* limit the damage of a leak */
! static int ngqfreemax = 64;/* cache at most this many */
!
! TUNABLE_INT("net.graph.maxalloc", &maxalloc);
! SYSCTL_INT(_net_graph, OID_AUTO, maxalloc, CTLFLAG_RD, &maxalloc,
! 0, "Maximum number of queue items to allocate");
!
! TUNABLE_INT("net.graph.ngqfreemax", &ngqfreemax);
! SYSCTL_INT(_net_graph, OID_AUTO, ngqfreemax, CTLFLAG_RD, &ngqfreemax,
! 0, "Maximum number of free queue items to cache");
!
static const int ngqfreelow = 4; /* try malloc if free < this */
static volatile int ngqfreesize; /* number of cached entries */
#ifdef NETGRAPH_DEBUG
>Release-Note:
>Audit-Trail:
>Unformatted:
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-bugs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200301231113.h0NBDwiU001087>