Date: Tue, 13 May 2003 17:43:13 +0200 From: Shaun Jurrens <shaun.jurrens@skoleetaten.oslo.no> To: freebsd-net@freebsd.org Subject: KVM exhaustion from routing table "leaks" Message-ID: <20030513154313.GR547@nevada.skoleetaten.oslo.no>
next in thread | raw e-mail | index | archive | help
--ZuvoxmZMh0nHqhEq
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Hi all,
I've been fighting with a long term problem with a box that does a good deal
of packet pushing for a /17 and a little ipf work as well. One nic does ip=
nat
for a /24. The box does some static routing between 2 cisco routers and the=
=20
routes are added via the rc.conf mechanism for static routes, e.g.:
static_routes=3D"bla0 bla1 ...."
route_bla0=3D" -net 193.xxx.3.0 -netmask 255.255.255.0 193.xxx.19x.1x"
route_bla1=3D" -net 193.xxx.4.0 -netmask 255.255.255.192 193.xxx.19x.1x"
The problem is in the continuous growth of cloned routes in the routing tab=
le.
I've managed to allocate enough kva to keep the box up for an extended amou=
nt=20
of time but, eventually, it chews up every bit of kva that it can and=20
allocating new routes fails and it has even taken the box down on occassion.
This happens on other boxes as well with lesser traffic.
The box is running 4.7-RELEASE-p7, with 5 fxp nic's, (4 in use)
=2E..
Timecounter "i8254" frequency 1193182 Hz
Timecounter "TSC" frequency 863678217 Hz
CPU: Pentium III/Pentium III Xeon/Celeron (863.68-MHz 686-class CPU)
Origin =3D "GenuineIntel" Id =3D 0x686 Stepping =3D 6
Features=3D0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE=
,MCA,CM
OV,PAT,PSE36,MMX,FXSR,SSE>
real memory =3D 671023104 (655296K bytes)
avail memory =3D 648257536 (633064K bytes)
Preloaded elf kernel "kernel" at 0xc031f000.
Preloaded elf module "ipl.ko" at 0xc031f09c.
=2E..
So, now a little information over the current state of things:
nol33n0x:/#> netstat -arn | wc -l
696714
Number of static routes:
nol33n0x:/#> netstat -arn | grep S | wc -l
34
Number of static routes with -cloning set:
nol33n0x:/#> netstat -arn | grep Sc | wc -l
34
Number of cloned routes (box is still running, so number has grown):
nol33n0x:/#> netstat -arn | grep W | wc -l
696830
Use of KVA by routing table:
nol33n0x:/#> vmstat -m | grep routetbl
...
routetbl1394589196107K 196107K262144K 1465571 0 0 16,32,64,12=
8,256
and a little more:
Memory Totals: In Use Free Requests
204934K 4103K 264929580
Observations:=20
Number of routes with 'Use' =3D=3D 0 on fxp0 (nic to "default" router):
nol33n0x:/#> netstat -arn | awk '/fxp0/ { print $5 }' | grep -e '^0$' | wc =
-l
294790
Number of routes with 'Ref' =3D=3D 0 on fxp0:
nol33n0x:/#> netstat -arn | awk '/fxp0/ { print $4 }' | grep -e '^0$' | wc =
-l
3
Number of routes with 'Use' =3D=3D 0 on fxp1 (small /24):
nol33n0x:/#> netstat -arn | awk '/fxp1/ { print $5 }' | grep -e '^0$' | wc =
-l
1
Number of routes with 'Use' =3D=3D 0 on fxp2 (most of the rest of our /17):
nol33n0x:/#> netstat -arn | awk '/fxp2/ { print $5 }' | grep -e '^0$' | wc =
-l 49
Number of routes with 'Ref' =3D=3D 0 on fxp2:
nol33n0x:/#> netstat -arn | awk '/fxp2/ { print $4 }' | grep -e '^0$' | wc =
-l
7
How icmp redirect is handled:
nol33n0x:/#> sysctl -a | grep redir
net.inet.ip.redirect: 1
net.inet.icmp.drop_redirect: 1
net.inet.icmp.log_redirect: 0
Sysctl's on routing:
net.inet.ip.rtexpire: 2
net.inet.ip.rtminexpire: 2
net.inet.ip.rtmaxcache: 512 (these seem to have no effect whatsoever...)
Specific questions:
1. Why do statically added routes assume -cloning?
2. Forgive my ignorance, but why is -cloning necessary for the default rout=
e?
3. Although I haven't done an exhaustive comparison of the content of the=
=20
routing table, why don't cloned routes with Use=3D=3D0 time out?
4. There was a security advisory about a possible DoS dealing with -cloning=
=20
and KVA exhaustion on an earlier -release, was the fix part of the breakage?
5. Manual removal of routes with 'Use'=3D=3D0 does not free up kernel memor=
y, why?
I'm starting to think the next hack I'm going to have to try is running rou=
ted
or zebra to manipulate the routing table more actively, even though this wo=
uld
seem to be sort of giving in to the problem, instead of fixing (assuming I'm
not just imagining all of this).
Perhaps I'm just ignorant of how routing is supposed to work, if so, I'll t=
ake
my cluebat like a man. I haven't dug through the code because I _know_ I'm=
=20
ignorant there. Guess I could use a little help. Comments and/or suggestio=
ns=20
welcome.
--=20
Med vennlig hilsen/Sincerely,
Shaun D. Jurrens
Drift og Sikkerhetskonsulent
IKT-Avdeling
Oslo Skoleetaten
Tel: +47 2208 7394
Mobil: +47 9820 8826
gpg key fingerprint: 007A B6BD 8B1B BAB9 C583 2D19 3A7F 4A3E F83E 84AE
--ZuvoxmZMh0nHqhEq
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (FreeBSD)
iD8DBQE+wRKROn9KPvg+hK4RAo6GAJ91yI96ShFkhLGvVR2Aj0HRXWOFPQCg1oSZ
ea3U5W6DjhW+ofRzy0pEGcs=
=YgYr
-----END PGP SIGNATURE-----
--ZuvoxmZMh0nHqhEq--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030513154313.GR547>