Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 26 Apr 2012 18:27:00 +0200
From:      Monthadar Al Jaberi <monthadar@gmail.com>
To:        Marko Zec <zec@fer.hr>
Cc:        freebsd-virtualization@freebsd.org
Subject:   Re: vimage tool crash when deleting a jail @netisr_process_workstream_proto
Message-ID:  <CA%2BsBSo%2BSYuLUqxerTnFKRTAZTT7TYY_2fWHqRqYSR4kRMQ6tJw@mail.gmail.com>
In-Reply-To: <201204252357.36459.zec@fer.hr>
References:  <CA%2BsBSoLs6sbyJr2%2BVLwAbTY%2BKs2gB0orEpRXK-KWVr1Z543jYg@mail.gmail.com> <201204252357.36459.zec@fer.hr>

next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Apr 25, 2012 at 11:57 PM, Marko Zec <zec@fer.hr> wrote:
> On Wednesday 25 April 2012 14:08:48 Monthadar Al Jaberi wrote:
>> Hi,
>>
>> Not sure if I should post this on virt or jail.
>>
>> I am not sure about this, but I thought it was an amd64 specific thing
>> (had posted about it some time ago), but =A0now I also get a panic on
>> i386. What is weird is that if I add options VNET_DEBUG to the kernel
>> config I dont get the panic! Debug output after vimage -c jid=3D0:
>> hhook_vnet_uninit: hhook_head type=3D1, id=3D1 cleanup required
>> hhook_vnet_uninit: hhook_head type=3D1, id=3D0 cleanup required
>>
>>
>> Also There seem to be a LOR after running vimage -c jid=3D0
>> lock order reversal:
>> =A01st 0xc1037dac allprison (allprison) @ /usr/src/sys/kern/kern_jail.c:=
970
>> =A02nd 0xc11b23f4 vnet_sysinit_sxlock (vnet_sysinit_sxlock) @
>> /usr/src/sys/net/vnet.c:615
>> KDB: stack backtrace:
>> db_trace_self_wrapper(c0e95cbb,6b636f6c,20402029,7273752f,6372732f,...)
>> at db_trace_self_wrapper+0x26
>> kdb_backtrace(c09e2ecb,c0e9974c,c1183ed0,267,e1b409f8,...) at
>> kdb_backtrace+0x2a
>> _witness_debugger(c0e9974c,c11b23f4,c0ea92c5,c7d64fc8,c0ea9418,...) at
>> _witness_debugger+0x25
>> witness_checkorder(c11b23f4,1,c0ea9418,267,0,...) at
>> witness_checkorder+0x86f _sx_slock(c11b23f4,0,c0ea9418,267,cad672e0,...)=
 at
>> _sx_slock+0x9a
>> vnet_sysinit(cad7f000,c0ff9700,5560,cad7e028,c0fb3588,...) at
>> vnet_sysinit+0x2b vnet_alloc(cad7e028,c0e8c936,0,10,0,...) at
>> vnet_alloc+0x168
>> kern_jail_set(cad672e0,c9470d00,1,c9470d00,0,...) at kern_jail_set+0x1bb=
4
>> sys_jail_set(cad672e0,e1b40cec,c0edada8,c0e9a9a6,c1047f40,...) at
>> sys_jail_set+0x50
>> syscall(e1b40d28) at syscall+0x2de
>> Xint0x80_syscall() at Xint0x80_syscall+0x21
>> --- syscall (507, FreeBSD ELF32, sys_jail_set), eip =3D 0x280bfd5b, esp
>> =3D 0xbfbfe23c, ebp =3D 0xbfbfe328 ---
>>
>>
>>
>> This is my setup:
>> Host PC: Ubuntu 11.04 (Linux bane 2.6.38-12-generic)
>> VirtualBox: 4.1.6 r74713
>> FreeBSD gues: i386 head@234636 (attaching kernel config)
>>
>> I compile and install /usr/src/tools/tools/vimage
>>
>> running:
>> vimage -c jid=3D0
>> vimage -d jid=3D0
>>
>> crashes the kernel (attaching core.txt.2)
>> Fatal trap 12: page fault while in kernel mode
>> cpuid =3D 0; apic id =3D 00
>> fault virtual address =3D 0xdeadc0e6
>> fault code =A0 =A0 =A0 =A0 =A0 =A0=3D supervisor read, page not present
>> instruction pointer =A0 =3D 0x20:0xc0a78d20
>> stack pointer =A0 =A0 =A0 =A0 =3D 0x28:0xc7980c48
>> frame pointer =A0 =A0 =A0 =A0 =3D 0x28:0xc7980c90
>> code segment =A0 =A0 =A0 =A0 =A0=3D base 0x0, limit 0xfffff, type 0x1b
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D DPL 0, pres 1, def32 1, =
gran 1
>> processor eflags =A0 =A0 =A0=3D interrupt enabled, resume, IOPL =3D 0
>> current process =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 12 (swi1: netisr 0)
>>
>> #0 =A0doadump (textdump=3D0) at pcpu.h:244
>> 244 =A0 pcpu.h: No such file or directory.
>> =A0 =A0 =A0 in pcpu.h
>> (kgdb) #0 =A0doadump (textdump=3D0) at pcpu.h:244
>> #1 =A00xc05104b3 in db_dump (dummy=3D-1062761184, dummy2=3D0, dummy3=3D-=
1,
>> =A0 =A0 dummy4=3D0xc798096c "") at /usr/src/sys/ddb/db_command.c:538
>> #2 =A00xc050fbd1 in db_command (last_cmdp=3D0xc10000dc, cmd_table=3D0x0,
>> dopager=3D1) at /usr/src/sys/ddb/db_command.c:449
>> #3 =A00xc050fd2a in db_command_loop () at /usr/src/sys/ddb/db_command.c:=
502
>> #4 =A00xc0511d1d in db_trap (type=3D12, code=3D0)
> at /usr/src/sys/ddb/db_main.c:231
>> #5 =A00xc09de976 in kdb_trap (type=3D12, code=3D0,
>> tf=3D0xc7980c08)
>> =A0 =A0 at /usr/src/sys/kern/subr_kdb.c:654
>> #6 =A00xc0cf2eff in trap_fatal (frame=3D0xc7980c08, eva=3D3735929062)
>> =A0 =A0 at /usr/src/sys/i386/i386/trap.c:1013
>> #7 =A00xc0cf32ee in trap_pfault (frame=3D0xc7980c08, usermode=3D0,
>> eva=3D3735929062) at /usr/src/sys/i386/i386/trap.c:936
>> #8 =A00xc0cf40b1 in trap (frame=3D0xc7980c08) at
>> /usr/src/sys/i386/i386/trap.c:546 #9 =A00xc0cdd8ec in calltrap () at
>> /usr/src/sys/i386/i386/exception.s:169 #10 0xc0a78d20 in swi_net
>> (arg=3D0xc1825880) at /usr/src/sys/net/netisr.c:805
>
> The backtrace says it's most probably this line in netisr.c:
>
> =A0 =A0805 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 CURVNET_SET(m->m_pkthdr.rcvif-=
>if_vnet);
>
>> #11 0xc0979d75 in
>> intr_event_execute_handlers (p=3D0xc7dc6598, ie=3D0xc7e95300) at
>> /usr/src/sys/kern/kern_intr.c:1260
>> #12 0xc097ac49 in ithread_loop (arg=3D0xc7e0f7a0)
>> =A0 =A0 at /usr/src/sys/kern/kern_intr.c:1273
>> #13 0xc0976fa8 in fork_exit (callout=3D0xc097aba0 <ithread_loop>,
>> =A0 =A0 arg=3D0xc7e0f7a0, frame=3D0xc7980d28) at /usr/src/sys/kern/kern_=
fork.c:992
>> #14 0xc0cdd994 in fork_trampoline () at
>> /usr/src/sys/i386/i386/exception.s:276 (kgdb)
>>
>>
>> It seems that it crashes on netisr_process_workstream_proto, why are
>> we even in this function? Is someone sending a packet?
>
> When netisr_process_workstream_proto() processes queued mbufs, in options
> VIMAGE builds it expects all mbufs to have m->m_pkthdr.rcvif set to a val=
id
> interface, so that the curvnet context could be harvested from there.
>
> Perhaps you've created a new mbuf and queued it for netisr dispatching, b=
ut
> have left m->m_pkthdr.rcvif uninitialized?

I have not done anything I am aware of, basically I called vimage the
first thing I log in freebsd. so I have not assigned it any interface
or anything :/

When I do this in single user mode I dont get this problem but I guess
that is because no interfaces are up yet :)

But I think this is weird:
VNET_ASSERT(m->m_pkthdr.rcvif !=3D NULL,
		    ("%s:%d rcvif =3D=3D NULL: m=3D%p", __func__, __LINE__, m));

this panic should have showed up if no one have set rcvif, but when I
check its value it is pointing to 0xdeadc0de, why?

>
> Marko



--=20
Monthadar Al Jaberi



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BsBSo%2BSYuLUqxerTnFKRTAZTT7TYY_2fWHqRqYSR4kRMQ6tJw>