From owner-freebsd-virtualization@FreeBSD.ORG Wed Apr 25 21:58:37 2012 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A85D81065670 for ; Wed, 25 Apr 2012 21:58:37 +0000 (UTC) (envelope-from zec@fer.hr) Received: from mail.zvne.fer.hr (mail.zvne.fer.hr [161.53.66.5]) by mx1.freebsd.org (Postfix) with ESMTP id 0239F8FC16 for ; Wed, 25 Apr 2012 21:58:36 +0000 (UTC) Received: from munja.zvne.fer.hr (161.53.66.248) by mail.zvne.fer.hr (161.53.66.5) with Microsoft SMTP Server id 14.1.355.2; Wed, 25 Apr 2012 23:58:29 +0200 Received: from sluga.fer.hr ([161.53.66.244]) by munja.zvne.fer.hr with Microsoft SMTPSVC(6.0.3790.4675); Wed, 25 Apr 2012 23:58:29 +0200 Received: from localhost ([161.53.19.8]) by sluga.fer.hr with Microsoft SMTPSVC(6.0.3790.4675); Wed, 25 Apr 2012 23:58:28 +0200 From: Marko Zec To: Date: Wed, 25 Apr 2012 23:57:36 +0200 User-Agent: KMail/1.9.10 References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-ID: <201204252357.36459.zec@fer.hr> X-OriginalArrivalTime: 25 Apr 2012 21:58:29.0042 (UTC) FILETIME=[8C269920:01CD232E] Cc: Subject: Re: vimage tool crash when deleting a jail @netisr_process_workstream_proto X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2012 21:58:37 -0000 On Wednesday 25 April 2012 14:08:48 Monthadar Al Jaberi wrote: > Hi, > > Not sure if I should post this on virt or jail. > > I am not sure about this, but I thought it was an amd64 specific thing > (had posted about it some time ago), but now I also get a panic on > i386. What is weird is that if I add options VNET_DEBUG to the kernel > config I dont get the panic! Debug output after vimage -c jid=0: > hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required > hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required > > > Also There seem to be a LOR after running vimage -c jid=0 > lock order reversal: > 1st 0xc1037dac allprison (allprison) @ /usr/src/sys/kern/kern_jail.c:970 > 2nd 0xc11b23f4 vnet_sysinit_sxlock (vnet_sysinit_sxlock) @ > /usr/src/sys/net/vnet.c:615 > KDB: stack backtrace: > db_trace_self_wrapper(c0e95cbb,6b636f6c,20402029,7273752f,6372732f,...) > at db_trace_self_wrapper+0x26 > kdb_backtrace(c09e2ecb,c0e9974c,c1183ed0,267,e1b409f8,...) at > kdb_backtrace+0x2a > _witness_debugger(c0e9974c,c11b23f4,c0ea92c5,c7d64fc8,c0ea9418,...) at > _witness_debugger+0x25 > witness_checkorder(c11b23f4,1,c0ea9418,267,0,...) at > witness_checkorder+0x86f _sx_slock(c11b23f4,0,c0ea9418,267,cad672e0,...) at > _sx_slock+0x9a > vnet_sysinit(cad7f000,c0ff9700,5560,cad7e028,c0fb3588,...) at > vnet_sysinit+0x2b vnet_alloc(cad7e028,c0e8c936,0,10,0,...) at > vnet_alloc+0x168 > kern_jail_set(cad672e0,c9470d00,1,c9470d00,0,...) at kern_jail_set+0x1bb4 > sys_jail_set(cad672e0,e1b40cec,c0edada8,c0e9a9a6,c1047f40,...) at > sys_jail_set+0x50 > syscall(e1b40d28) at syscall+0x2de > Xint0x80_syscall() at Xint0x80_syscall+0x21 > --- syscall (507, FreeBSD ELF32, sys_jail_set), eip = 0x280bfd5b, esp > = 0xbfbfe23c, ebp = 0xbfbfe328 --- > > > > This is my setup: > Host PC: Ubuntu 11.04 (Linux bane 2.6.38-12-generic) > VirtualBox: 4.1.6 r74713 > FreeBSD gues: i386 head@234636 (attaching kernel config) > > I compile and install /usr/src/tools/tools/vimage > > running: > vimage -c jid=0 > vimage -d jid=0 > > crashes the kernel (attaching core.txt.2) > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0xdeadc0e6 > fault code = supervisor read, page not present > instruction pointer = 0x20:0xc0a78d20 > stack pointer = 0x28:0xc7980c48 > frame pointer = 0x28:0xc7980c90 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 12 (swi1: netisr 0) > > #0 doadump (textdump=0) at pcpu.h:244 > 244 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) #0 doadump (textdump=0) at pcpu.h:244 > #1 0xc05104b3 in db_dump (dummy=-1062761184, dummy2=0, dummy3=-1, > dummy4=0xc798096c "") at /usr/src/sys/ddb/db_command.c:538 > #2 0xc050fbd1 in db_command (last_cmdp=0xc10000dc, cmd_table=0x0, > dopager=1) at /usr/src/sys/ddb/db_command.c:449 > #3 0xc050fd2a in db_command_loop () at /usr/src/sys/ddb/db_command.c:502 > #4 0xc0511d1d in db_trap (type=12, code=0) at /usr/src/sys/ddb/db_main.c:231 > #5 0xc09de976 in kdb_trap (type=12, code=0, > tf=0xc7980c08) > at /usr/src/sys/kern/subr_kdb.c:654 > #6 0xc0cf2eff in trap_fatal (frame=0xc7980c08, eva=3735929062) > at /usr/src/sys/i386/i386/trap.c:1013 > #7 0xc0cf32ee in trap_pfault (frame=0xc7980c08, usermode=0, > eva=3735929062) at /usr/src/sys/i386/i386/trap.c:936 > #8 0xc0cf40b1 in trap (frame=0xc7980c08) at > /usr/src/sys/i386/i386/trap.c:546 #9 0xc0cdd8ec in calltrap () at > /usr/src/sys/i386/i386/exception.s:169 #10 0xc0a78d20 in swi_net > (arg=0xc1825880) at /usr/src/sys/net/netisr.c:805 The backtrace says it's most probably this line in netisr.c: 805 CURVNET_SET(m->m_pkthdr.rcvif->if_vnet); > #11 0xc0979d75 in > intr_event_execute_handlers (p=0xc7dc6598, ie=0xc7e95300) at > /usr/src/sys/kern/kern_intr.c:1260 > #12 0xc097ac49 in ithread_loop (arg=0xc7e0f7a0) > at /usr/src/sys/kern/kern_intr.c:1273 > #13 0xc0976fa8 in fork_exit (callout=0xc097aba0 , > arg=0xc7e0f7a0, frame=0xc7980d28) at /usr/src/sys/kern/kern_fork.c:992 > #14 0xc0cdd994 in fork_trampoline () at > /usr/src/sys/i386/i386/exception.s:276 (kgdb) > > > It seems that it crashes on netisr_process_workstream_proto, why are > we even in this function? Is someone sending a packet? When netisr_process_workstream_proto() processes queued mbufs, in options VIMAGE builds it expects all mbufs to have m->m_pkthdr.rcvif set to a valid interface, so that the curvnet context could be harvested from there. Perhaps you've created a new mbuf and queued it for netisr dispatching, but have left m->m_pkthdr.rcvif uninitialized? Marko