Date: Tue, 30 Sep 2014 16:56:00 -0700 From: Rumen Telbizov <telbizov@gmail.com> To: glebius@freebsd.org Cc: "Alexander V. Chernikov" <melifaro@freebsd.org>, "freebsd-stable@freebsd.org" <freebsd-stable@freebsd.org>, brian@freebsd.org Subject: Re: 10.1-BETA2 possible kernel memory leak in routing table Message-ID: <CAENR%2B_WbntqjE4b=iZS8z30AK7gSpur00HsWP9-T_UJ7OosU8Q@mail.gmail.com> In-Reply-To: <542AFAE3.9030705@FreeBSD.org> References: <CAENR%2B_UVLDDrsef2W4CXCFX65EYaxeKN4MNWbgoyaZ5qDGe1Pg@mail.gmail.com> <542AAA3C.1080803@ipfw.ru> <CAENR%2B_X5KTdeb00f9NShN1YK%2BT2aY1vG5YcTCgu4aXZO=%2Bpa=g@mail.gmail.com> <542AE376.6000003@FreeBSD.org> <CAENR%2B_XX4jnD6SBi8S1dGfWM68tmcm0aE2iMVA3LDR3R8ygQYw@mail.gmail.com> <542AFAE3.9030705@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello everyone,
I think I got one step closer. At this point I am almost certain that the *leak
is related to PF reloading tables.*
Adding Gleb Smirnoff here since it might be something related to the
rewrite of PF in FreeBSD 10.
Here are the additional details.
The following dtrace script shows kernel memory leaks per probe:
#!/usr/sbin/dtrace -s
#pragma D option quiet
BEGIN {
printf("%-20s %20s %20s %20s\n", "FUNCTION", "ALLOCATED", "FREE",
"TOTAL");
}
dtmalloc::$1:malloc {
@malloc[probefunc] = sum(args[3]);
@total[probefunc] = sum(args[3]);
}
dtmalloc::$1:free {
@free[probefunc] = sum(args[3]);
@total[probefunc] = sum(-args[3]);
}
tick-1s {
printa("%-20.20s %20@d %20@d %20@d\n", @malloc, @free, @total);
}
Running it like this:
# ./script2.d routetbl
FUNCTION ALLOCATED
FREE TOTAL
routetbl 512
512 0
...
routetbl 46592
46592 0
routetbl 808960
444416 364544
...
routetbl 861184
496640 364544
routetbl 1623552
894464 729088
...
routetbl 1641984
912896 729088
The interval between the increase of 356KB is almost exactly 30 seconds.
This made me think about what I run on this machine every 30 seconds. I do
in fact have a service which every 30 seconds processes data, compiles PF
tables and reloads the PF tables like this: /sbin/pfctl -T load -f
/etc/firewall/pf.conf
I stopped this service and the amount of memory remained static! No more
leaks. I checked vmstat -m | grep routetbl and it had stopped growing. A
manual pfctl -f /etc/firewall/pf.conf causes a leak of 356KB of memory
every single time I run it.
Brian Somers and I are currently looking into the source of PF in latest
10-STABLE and trying to figure out what is going on. We were able to
replicate this problem on a 11-CURRENT (Sep 12th) machine as well. A simple
PF ruleset with 1 rule and 1 table. Every few reloads of the firewall
and vmstat
-m | grep routetbl shows increased memory usage.
For completeness - Alexander's dtrace script, with an aggregate twist,
shows this (and only this) after about a minute:
#!/usr/sbin/dtrace -s
fbt:kernel:rt_msg2:entry {
@rt_msg2[stack()] = count();
}
fbt:kernel:rn_addroute:entry {
@rn_addroute[stack()] = count();
}
Output:
kernel`sysctl_rtsock+0x274
kernel`sysctl_root+0x214
kernel`userland_sysctl+0x1d8
kernel`sys___sysctl+0x74
kernel`amd64_syscall+0x334
kernel`0xffffffff80900e0b
2340
kernel`sysctl_rtsock+0x64c
kernel`sysctl_root+0x214
kernel`userland_sysctl+0x1d8
kernel`sys___sysctl+0x74
kernel`amd64_syscall+0x334
kernel`0xffffffff80900e0b
4680
Getting pretty close ...
Regards,
--
Rumen Telbizov
Unix Systems Administrator <http://telbizov.com>;
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAENR%2B_WbntqjE4b=iZS8z30AK7gSpur00HsWP9-T_UJ7OosU8Q>