Date: Wed, 25 Apr 2012 23:57:36 +0200 From: Marko Zec <zec@fer.hr> To: <freebsd-virtualization@freebsd.org> Subject: Re: vimage tool crash when deleting a jail @netisr_process_workstream_proto Message-ID: <201204252357.36459.zec@fer.hr> In-Reply-To: <CA%2BsBSoLs6sbyJr2%2BVLwAbTY%2BKs2gB0orEpRXK-KWVr1Z543jYg@mail.gmail.com> References: <CA%2BsBSoLs6sbyJr2%2BVLwAbTY%2BKs2gB0orEpRXK-KWVr1Z543jYg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wednesday 25 April 2012 14:08:48 Monthadar Al Jaberi wrote:
> Hi,
>
> Not sure if I should post this on virt or jail.
>
> I am not sure about this, but I thought it was an amd64 specific thing
> (had posted about it some time ago), but now I also get a panic on
> i386. What is weird is that if I add options VNET_DEBUG to the kernel
> config I dont get the panic! Debug output after vimage -c jid=0:
> hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
> hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required
>
>
> Also There seem to be a LOR after running vimage -c jid=0
> lock order reversal:
> 1st 0xc1037dac allprison (allprison) @ /usr/src/sys/kern/kern_jail.c:970
> 2nd 0xc11b23f4 vnet_sysinit_sxlock (vnet_sysinit_sxlock) @
> /usr/src/sys/net/vnet.c:615
> KDB: stack backtrace:
> db_trace_self_wrapper(c0e95cbb,6b636f6c,20402029,7273752f,6372732f,...)
> at db_trace_self_wrapper+0x26
> kdb_backtrace(c09e2ecb,c0e9974c,c1183ed0,267,e1b409f8,...) at
> kdb_backtrace+0x2a
> _witness_debugger(c0e9974c,c11b23f4,c0ea92c5,c7d64fc8,c0ea9418,...) at
> _witness_debugger+0x25
> witness_checkorder(c11b23f4,1,c0ea9418,267,0,...) at
> witness_checkorder+0x86f _sx_slock(c11b23f4,0,c0ea9418,267,cad672e0,...) at
> _sx_slock+0x9a
> vnet_sysinit(cad7f000,c0ff9700,5560,cad7e028,c0fb3588,...) at
> vnet_sysinit+0x2b vnet_alloc(cad7e028,c0e8c936,0,10,0,...) at
> vnet_alloc+0x168
> kern_jail_set(cad672e0,c9470d00,1,c9470d00,0,...) at kern_jail_set+0x1bb4
> sys_jail_set(cad672e0,e1b40cec,c0edada8,c0e9a9a6,c1047f40,...) at
> sys_jail_set+0x50
> syscall(e1b40d28) at syscall+0x2de
> Xint0x80_syscall() at Xint0x80_syscall+0x21
> --- syscall (507, FreeBSD ELF32, sys_jail_set), eip = 0x280bfd5b, esp
> = 0xbfbfe23c, ebp = 0xbfbfe328 ---
>
>
>
> This is my setup:
> Host PC: Ubuntu 11.04 (Linux bane 2.6.38-12-generic)
> VirtualBox: 4.1.6 r74713
> FreeBSD gues: i386 head@234636 (attaching kernel config)
>
> I compile and install /usr/src/tools/tools/vimage
>
> running:
> vimage -c jid=0
> vimage -d jid=0
>
> crashes the kernel (attaching core.txt.2)
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address = 0xdeadc0e6
> fault code = supervisor read, page not present
> instruction pointer = 0x20:0xc0a78d20
> stack pointer = 0x28:0xc7980c48
> frame pointer = 0x28:0xc7980c90
> code segment = base 0x0, limit 0xfffff, type 0x1b
> = DPL 0, pres 1, def32 1, gran 1
> processor eflags = interrupt enabled, resume, IOPL = 0
> current process = 12 (swi1: netisr 0)
>
> #0 doadump (textdump=0) at pcpu.h:244
> 244 pcpu.h: No such file or directory.
> in pcpu.h
> (kgdb) #0 doadump (textdump=0) at pcpu.h:244
> #1 0xc05104b3 in db_dump (dummy=-1062761184, dummy2=0, dummy3=-1,
> dummy4=0xc798096c "") at /usr/src/sys/ddb/db_command.c:538
> #2 0xc050fbd1 in db_command (last_cmdp=0xc10000dc, cmd_table=0x0,
> dopager=1) at /usr/src/sys/ddb/db_command.c:449
> #3 0xc050fd2a in db_command_loop () at /usr/src/sys/ddb/db_command.c:502
> #4 0xc0511d1d in db_trap (type=12, code=0)
at /usr/src/sys/ddb/db_main.c:231
> #5 0xc09de976 in kdb_trap (type=12, code=0,
> tf=0xc7980c08)
> at /usr/src/sys/kern/subr_kdb.c:654
> #6 0xc0cf2eff in trap_fatal (frame=0xc7980c08, eva=3735929062)
> at /usr/src/sys/i386/i386/trap.c:1013
> #7 0xc0cf32ee in trap_pfault (frame=0xc7980c08, usermode=0,
> eva=3735929062) at /usr/src/sys/i386/i386/trap.c:936
> #8 0xc0cf40b1 in trap (frame=0xc7980c08) at
> /usr/src/sys/i386/i386/trap.c:546 #9 0xc0cdd8ec in calltrap () at
> /usr/src/sys/i386/i386/exception.s:169 #10 0xc0a78d20 in swi_net
> (arg=0xc1825880) at /usr/src/sys/net/netisr.c:805
The backtrace says it's most probably this line in netisr.c:
805 CURVNET_SET(m->m_pkthdr.rcvif->if_vnet);
> #11 0xc0979d75 in
> intr_event_execute_handlers (p=0xc7dc6598, ie=0xc7e95300) at
> /usr/src/sys/kern/kern_intr.c:1260
> #12 0xc097ac49 in ithread_loop (arg=0xc7e0f7a0)
> at /usr/src/sys/kern/kern_intr.c:1273
> #13 0xc0976fa8 in fork_exit (callout=0xc097aba0 <ithread_loop>,
> arg=0xc7e0f7a0, frame=0xc7980d28) at /usr/src/sys/kern/kern_fork.c:992
> #14 0xc0cdd994 in fork_trampoline () at
> /usr/src/sys/i386/i386/exception.s:276 (kgdb)
>
>
> It seems that it crashes on netisr_process_workstream_proto, why are
> we even in this function? Is someone sending a packet?
When netisr_process_workstream_proto() processes queued mbufs, in options
VIMAGE builds it expects all mbufs to have m->m_pkthdr.rcvif set to a valid
interface, so that the curvnet context could be harvested from there.
Perhaps you've created a new mbuf and queued it for netisr dispatching, but
have left m->m_pkthdr.rcvif uninitialized?
Marko
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201204252357.36459.zec>