Date: Sun, 28 Feb 2010 13:30:59 +0200 From: Mikolaj Golub <to.my.trociny@gmail.com> To: Kostik Belousov <kostikbel@gmail.com> Cc: freebsd-net@freebsd.org Subject: Re: kmem leakage on tun/tap device removal Message-ID: <86fx4loavg.fsf@kopusha.onet> In-Reply-To: <20100227151626.GJ2489@deviant.kiev.zoral.com.ua> (Kostik Belousov's message of "Sat\, 27 Feb 2010 17\:16\:26 %2B0200") References: <86hbp3jape.fsf@kopusha.onet> <20100227151626.GJ2489@deviant.kiev.zoral.com.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 27 Feb 2010 17:16:26 +0200 Kostik Belousov wrote:
> Please test the patch below.
Tested. It works as expected: running in loop
ifconfig tun0 create
ifconfig tun0 destroy
(and in parallel the same for tun1) no kmem leakage is observed:
Time Type InUse MemUse HighUse Requests Size(s)
20:20 DEVFS1 104 26K - 12481 256
20:25 DEVFS1 104 26K - 110695 256
20:30 DEVFS1 103 26K - 228557 256
20:35 DEVFS1 102 26K - 345257 256
20:40 DEVFS1 113 29K - 460939 256
20:45 DEVFS1 102 26K - 575485 256
20:50 DEVFS1 109 28K - 687967 256
20:55 DEVFS1 103 26K - 800969 256
21:00 DEVFS1 102 26K - 909753 256
21:05 DEVFS1 101 26K - 1022029 256
21:10 DEVFS1 103 26K - 1132231 256
21:15 DEVFS1 105 27K - 1239151 256
21:20 DEVFS1 106 27K - 1349211 256
But I have faced with another issue (not related to your patch, as it is
observed with unpatched kernel too). When I try to run concurrently two
create/destroy scripts with the same interface the system panics:
Bad link elm <ifp address> next->prev != elm
Here are details about panic:
Unread portion of the kernel message buffer:
panic: Bad link elm 0xc5f1a800 next->prev != elm
cpuid = 2
KDB: enter: panic
exclusive sleep mutex if_clone lock (if_clone lock) r = 0 (0xc0da1cf0) locked @ /usr/src/sys/net/if_clone.c:248
exclusive sleep mutex if_clone lock (if_clone lock) r = 0 (0xc0da1cf0) locked @ /usr/src/sys/net/if_clone.c:248
exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xc6cd3560) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148
exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xc6b4dbd0) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148
Physical memory: 2019 MB
Dumping 160 MB: 145 129 113 97 81 65 49 33 17 1
#0 doadump () at pcpu.h:246
246 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0 doadump () at pcpu.h:246
#1 0xc04e8bb9 in db_fncall (dummy1=-1064515926, dummy2=0, dummy3=-1, dummy4=0xe83f4834 "HH?è")
at /usr/src/sys/ddb/db_command.c:548
#2 0xc04e8fef in db_command (last_cmdp=0xc0de14dc, cmd_table=0x0, dopager=0)
at /usr/src/sys/ddb/db_command.c:445
#3 0xc04e90a4 in db_command_script (command=0xc0de2404 "call doadump")
at /usr/src/sys/ddb/db_command.c:516
#4 0xc04ed1d0 in db_script_exec (scriptname=0xe83f4940 "kdb.enter.panic", warnifnotfound=Variable "warnifnotfound" is not available.
)
at /usr/src/sys/ddb/db_script.c:302
#5 0xc04ed2b7 in db_script_kdbenter (eventname=0xc0ca1948 "panic") at /usr/src/sys/ddb/db_script.c:324
#6 0xc04eaf98 in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_main.c:228
#7 0xc08cc526 in kdb_trap (type=3, code=0, tf=0xe83f4a7c) at /usr/src/sys/kern/subr_kdb.c:535
#8 0xc0bdd38b in trap (frame=0xe83f4a7c) at /usr/src/sys/i386/i386/trap.c:690
#9 0xc0bbef1b in calltrap () at /usr/src/sys/i386/i386/exception.s:165
#10 0xc08cc6aa in kdb_enter (why=0xc0ca1948 "panic", msg=0xc0ca1948 "panic") at cpufunc.h:71
#11 0xc089d716 in panic (fmt=0xc0c3c80c "Bad link elm %p next->prev != elm")
at /usr/src/sys/kern/kern_shutdown.c:562
#12 0xc094e7fb in if_clone_destroyif (ifc=0xc0da1cc0, ifp=0xc5f1a800) at /usr/src/sys/net/if_clone.c:249
#13 0xc094eb52 in if_clone_destroy (name=0xc664ac20 "tun0") at /usr/src/sys/net/if_clone.c:227
#14 0xc094c8a6 in ifioctl (so=0xc6e0a9a8, cmd=2149607801, data=0xc664ac20 "tun0", td=0xc66c0d80)
at /usr/src/sys/net/if.c:2412
#15 0xc08e8b25 in soo_ioctl (fp=0xc6d46af0, cmd=2149607801, data=0xc664ac20, active_cred=0xc5f62280,
td=0xc66c0d80) at /usr/src/sys/kern/sys_socket.c:212
#16 0xc08e31bd in kern_ioctl (td=0xc66c0d80, fd=3, com=2149607801, data=0xc664ac20 "tun0") at file.h:262
#17 0xc08e3344 in ioctl (td=0xc66c0d80, uap=0xe83f4cf8) at /usr/src/sys/kern/sys_generic.c:678
#18 0xc0bdca33 in syscall (frame=0xe83f4d38) at /usr/src/sys/i386/i386/trap.c:1078
#19 0xc0bbefb0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:261
#20 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) fr 12
#12 0xc094e7fb in if_clone_destroyif (ifc=0xc0da1cc0, ifp=0xc5f1a800) at /usr/src/sys/net/if_clone.c:249
249 IFC_IFLIST_REMOVE(ifc, ifp);
(kgdb) list
244 * switch to the vnet context of the target vnet.
245 */
246 CURVNET_SET_QUIET(ifp->if_vnet);
247
248 IF_CLONE_LOCK(ifc);
249 IFC_IFLIST_REMOVE(ifc, ifp);
250 IF_CLONE_UNLOCK(ifc);
251
252 if_delgroup(ifp, ifc->ifc_name);
253
The concurrent ifconfig destroy thread:
#0 sched_switch (td=0xc6c51900, newtd=0xc597b6c0, flags=1547) at /usr/src/sys/kern/sched_ule.c:1864
#1 0xc08a65a0 in mi_switch (flags=1547, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:449
#2 0xc08bf165 in sched_preempt (td=0xc6c51900) at /usr/src/sys/kern/sched_ule.c:2078
#3 0xc0bce954 in ipi_bitmap_handler (frame=
{tf_fs = 8, tf_es = -1060437976, tf_ds = 40, tf_edi = -956607168, tf_esi = -956607232, tf_ebp = -397423980, tf_isp = -397424072, tf_ebx = -959068160, tf_edx = -974018560, tf_ecx = 1, tf_eax = -960161535, tf_trapno = 0, tf_err = 0, tf_eip = -1064007759, tf_cs = 32, tf_eflags = 658, tf_esp = -397423980, tf_ss = -1063994479}) at /usr/src/sys/i386/i386/mp_machdep.c:1266
#4 0xc0bbf63e in Xipi_intr_bitmap_handler () at apic_vector.s:285
#5 0xc09487b1 in if_purgemaddrs (ifp=0xc5f1a800) at /usr/src/sys/net/if.c:780
#6 0xc09545b9 in tun_destroy (tp=0xc6fb5500) at /usr/src/sys/net/if_tun.c:259
#7 0xc09548e8 in tun_clone_destroy (ifp=0xc5f1a800) at /usr/src/sys/net/if_tun.c:277
#8 0xc094e207 in ifc_simple_destroy (ifc=0xc0da1cc0, ifp=0xc5f1a800) at /usr/src/sys/net/if_clone.c:595
#9 0xc094e871 in if_clone_destroyif (ifc=0xc0da1cc0, ifp=0xc5f1a800) at /usr/src/sys/net/if_clone.c:254
#10 0xc094eb52 in if_clone_destroy (name=0xc6f855a0 "tun0") at /usr/src/sys/net/if_clone.c:227
#11 0xc094c8a6 in ifioctl (so=0xc6e04670, cmd=2149607801, data=0xc6f855a0 "tun0", td=0xc6c51900)
at /usr/src/sys/net/if.c:2412
#12 0xc08e8b25 in soo_ioctl (fp=0xc6d3b0a8, cmd=2149607801, data=0xc6f855a0, active_cred=0xc6cce800,
td=0xc6c51900) at /usr/src/sys/kern/sys_socket.c:212
#13 0xc08e31bd in kern_ioctl (td=0xc6c51900, fd=3, com=2149607801, data=0xc6f855a0 "tun0") at file.h:262
#14 0xc08e3344 in ioctl (td=0xc6c51900, uap=0xe84fccf8) at /usr/src/sys/kern/sys_generic.c:678
#15 0xc0bdca33 in syscall (frame=0xe84fcd38) at /usr/src/sys/i386/i386/trap.c:1078
#16 0xc0bbefb0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:261
#17 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) fr 9
#9 0xc094e871 in if_clone_destroyif (ifc=0xc0da1cc0, ifp=0xc5f1a800) at /usr/src/sys/net/if_clone.c:254
254 err = (*ifc->ifc_destroy)(ifc, ifp);
(kgdb) list
249 IFC_IFLIST_REMOVE(ifc, ifp);
250 IF_CLONE_UNLOCK(ifc);
251
252 if_delgroup(ifp, ifc->ifc_name);
253
254 err = (*ifc->ifc_destroy)(ifc, ifp);
255
256 if (err != 0) {
257 if_addgroup(ifp, ifc->ifc_name);
258
So, it is a race but for me some things look a bit strange (may be because my
knowledge in this field is rather casual).
We have:
#define IFC_IFLIST_REMOVE(_ifc, _ifp) \
LIST_REMOVE(_ifp, if_clones)
and the message "Bad link elm 0xc5f1a800 next->prev != elm" should mean that
*(0xc5f1a800)->if_clones.le_prev != (0xc5f1a800)
But actually it is not observed:
(kgdb) p ifp
$1 = (struct ifnet *) 0xc5f1a800
(kgdb) p *(ifp)->if_clones.le_prev
$2 = (struct ifnet *) 0xc5f1c800
Also it looks like IFC_IFLIST_REMOVE() was properly locked. At the first
glance the code below IF_CLONE_UNLOCK() does not work with if_clones list.
And doesn't this look strange that we have two messages about the same lock?
exclusive sleep mutex if_clone lock (if_clone lock) r = 0 (0xc0da1cf0) locked @ /usr/src/sys/net/if_clone.c:248
exclusive sleep mutex if_clone lock (if_clone lock) r = 0 (0xc0da1cf0) locked @ /usr/src/sys/net/if_clone.c:248
My kernel config is:
include GENERIC
ident DTRACE
options INVARIANTS
options INVARIANT_SUPPORT
options WITNESS
options DEBUG_LOCKS
options DEBUG_VFS_LOCKS
options DIAGNOSTIC
options KDB
options DDB
options KDTRACE_HOOKS
options DDB_CTF
I am running i386 8.0-STABLE (but rather old, from Dec 1, I can run tests on
newer sources if this makes difference).
Hardware:
CPU: Intel(R) Core(TM)2 Quad CPU Q8200 @ 2.33GHz (2335.63-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x10677 Stepping = 7
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x8e39d<SSE3,DTES64,MON,DS_CPL,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1>
AMD Features=0x20100000<NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 2147483648 (2048 MB)
avail memory = 2074243072 (1978 MB)
ACPI APIC Table: <INTEL DG31GL>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
I tried running the test on GENERIC (without debugging options) and got "page
fault" panic.
Unread portion of the kernel message buffer:
bpfdetach: tun0 was not attached
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x1a0
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc087233f
stack pointer = 0x28:0xe8495a2c
frame pointer = 0x28:0xe8495a44
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 53523 (ifconfig)
trap number = 12
panic: page fault
cpuid = 0
Uptime: 1m15s
Physical memory: 2019 MB
Dumping 109 MB: 94 78 62 46 30 14
137 Thread 100216 (PID=53523: ifconfig) doadump () at pcpu.h:246
#0 doadump () at pcpu.h:246
#1 0xc0881c97 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416
#2 0xc0881f89 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:579
#3 0xc0bb39ec in trap_fatal (frame=0xe84959ec, eva=416) at /usr/src/sys/i386/i386/trap.c:938
#4 0xc0bb3c70 in trap_pfault (frame=0xe84959ec, usermode=0, eva=416) at /usr/src/sys/i386/i386/trap.c:851
#5 0xc0bb4675 in trap (frame=0xe84959ec) at /usr/src/sys/i386/i386/trap.c:533
#6 0xc0b96e0b in calltrap () at /usr/src/sys/i386/i386/exception.s:165
#7 0xc087233f in _mtx_lock_sleep (m=0xc5c4b22c, tid=3330476288, opts=0, file=0x0, line=0)
at /usr/src/sys/kern/kern_mutex.c:369
#8 0xc0926471 in if_detach (ifp=0xc5c4b000) at /usr/src/sys/net/if.c:1188
#9 0xc0930879 in tun_destroy (tp=0xc6860d80) at /usr/src/sys/net/if_tun.c:259
#10 0xc0931927 in tun_clone_destroy (ifp=0xc5c4b000) at /usr/src/sys/net/if_tun.c:277
#11 0xc092a407 in ifc_simple_destroy (ifc=0xc0d496e0, ifp=0xc5c4b000) at /usr/src/sys/net/if_clone.c:595
#12 0xc092a62c in if_clone_destroyif (ifc=0xc0d496e0, ifp=0xc5c4b000) at /usr/src/sys/net/if_clone.c:254
#13 0xc092a9e2 in if_clone_destroy (name=0xc5f201e0 "tun0") at /usr/src/sys/net/if_clone.c:227
#14 0xc0928a26 in ifioctl (so=0xc6806000, cmd=2149607801, data=0xc5f201e0 "tun0", td=0xc6830900)
at /usr/src/sys/net/if.c:2412
#15 0xc08c4c32 in soo_ioctl (fp=0xc6743968, cmd=2149607801, data=0xc5f201e0, active_cred=0xc66fa680,
td=0xc6830900) at /usr/src/sys/kern/sys_socket.c:212
#16 0xc08bdb00 in kern_ioctl (td=0xc6830900, fd=3, com=2149607801, data=0xc5f201e0 "tun0") at file.h:262
#17 0xc08bdc74 in ioctl (td=0xc6830900, uap=0xe8495cf8) at /usr/src/sys/kern/sys_generic.c:678
#18 0xc0bb3fb5 in syscall (frame=0xe8495d38) at /usr/src/sys/i386/i386/trap.c:1078
#19 0xc0b96e70 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:261
#20 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
138 Thread 100211 (PID=53526: ifconfig) sched_switch (td=0xc6832480, newtd=0xc5951b40, flags=259)
at /usr/src/sys/kern/sched_ule.c:1864
#0 sched_switch (td=0xc6832480, newtd=0xc5951b40, flags=259) at /usr/src/sys/kern/sched_ule.c:1864
#1 0xc088a15a in mi_switch (flags=259, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:449
#2 0xc08bc02b in turnstile_wait (ts=0xc5ec1700, owner=0xc6830900, queue=Variable "queue" is not available.
)
at /usr/src/sys/kern/subr_turnstile.c:745
#3 0xc088073f in _rw_rlock (rw=0xc0db6024, file=0x0, line=0) at /usr/src/sys/kern/kern_rwlock.c:460
#4 0xc0924867 in ifunit_ref (name=0xc685e200 "tun0") at /usr/src/sys/net/if.c:2017
#5 0xc0928d10 in ifioctl (so=0xc6820338, cmd=3223349536, data=0xc685e200 "tun0", td=0xc6832480)
at /usr/src/sys/net/if.c:2420
#6 0xc08c4c32 in soo_ioctl (fp=0xc6808c78, cmd=3223349536, data=0xc685e200, active_cred=0xc6802500,
td=0xc6832480) at /usr/src/sys/kern/sys_socket.c:212
#7 0xc08bdb00 in kern_ioctl (td=0xc6832480, fd=3, com=3223349536, data=0xc685e200 "tun0") at file.h:262
#8 0xc08bdc74 in ioctl (td=0xc6832480, uap=0xe846ccf8) at /usr/src/sys/kern/sys_generic.c:678
#9 0xc0bb3fb5 in syscall (frame=0xe846cd38) at /usr/src/sys/i386/i386/trap.c:1078
#10 0xc0b96e70 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:261
#11 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
--
Mikolaj Golub
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?86fx4loavg.fsf>