From owner-freebsd-virtualization@FreeBSD.ORG Thu Apr 26 16:27:02 2012 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2045F1065675 for ; Thu, 26 Apr 2012 16:27:02 +0000 (UTC) (envelope-from monthadar@gmail.com) Received: from mail-wg0-f50.google.com (mail-wg0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 9D5CF8FC0C for ; Thu, 26 Apr 2012 16:27:01 +0000 (UTC) Received: by wgbds12 with SMTP id ds12so1330953wgb.31 for ; Thu, 26 Apr 2012 09:27:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=DYtlUiHP43vLY4aNTzFn3HIlbFV7mPWiUQTaTR73yek=; b=NJej9pAYTCCDdWHYHcIa5ytFxGrTihctYYOqN7ltjOWTjBW16l1jAbDs3Dwsp8Bn5I xsY1vAqORx7/RTKCoSlJQcv0uLn/f25qQdxiOaLB1cCnhunR2eD6HIzooNKENOUogq7x rA+ZpmccXD7pb6MQnagAdZe96l6C/KvL1vgSthcX4a1K6LIwH+2t9b2RLqDg1DHJGGrr ImTvTFK2YyPEvolC7uZCCGEZeXpsM8Kux0b+ijhxdMDdxi9gxxK1mWfB9FNFCqHchMY2 6BJv8lElaAPhDbwB3BbVwB3l1jwl4KiERiZod/YMr6yHSOLnYK4X8fdbAOCMoBI9HZly ic/Q== MIME-Version: 1.0 Received: by 10.180.101.8 with SMTP id fc8mr18206562wib.12.1335457620583; Thu, 26 Apr 2012 09:27:00 -0700 (PDT) Received: by 10.223.155.74 with HTTP; Thu, 26 Apr 2012 09:27:00 -0700 (PDT) In-Reply-To: <201204252357.36459.zec@fer.hr> References: <201204252357.36459.zec@fer.hr> Date: Thu, 26 Apr 2012 18:27:00 +0200 Message-ID: From: Monthadar Al Jaberi To: Marko Zec Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-virtualization@freebsd.org Subject: Re: vimage tool crash when deleting a jail @netisr_process_workstream_proto X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2012 16:27:02 -0000 On Wed, Apr 25, 2012 at 11:57 PM, Marko Zec wrote: > On Wednesday 25 April 2012 14:08:48 Monthadar Al Jaberi wrote: >> Hi, >> >> Not sure if I should post this on virt or jail. >> >> I am not sure about this, but I thought it was an amd64 specific thing >> (had posted about it some time ago), but =A0now I also get a panic on >> i386. What is weird is that if I add options VNET_DEBUG to the kernel >> config I dont get the panic! Debug output after vimage -c jid=3D0: >> hhook_vnet_uninit: hhook_head type=3D1, id=3D1 cleanup required >> hhook_vnet_uninit: hhook_head type=3D1, id=3D0 cleanup required >> >> >> Also There seem to be a LOR after running vimage -c jid=3D0 >> lock order reversal: >> =A01st 0xc1037dac allprison (allprison) @ /usr/src/sys/kern/kern_jail.c:= 970 >> =A02nd 0xc11b23f4 vnet_sysinit_sxlock (vnet_sysinit_sxlock) @ >> /usr/src/sys/net/vnet.c:615 >> KDB: stack backtrace: >> db_trace_self_wrapper(c0e95cbb,6b636f6c,20402029,7273752f,6372732f,...) >> at db_trace_self_wrapper+0x26 >> kdb_backtrace(c09e2ecb,c0e9974c,c1183ed0,267,e1b409f8,...) at >> kdb_backtrace+0x2a >> _witness_debugger(c0e9974c,c11b23f4,c0ea92c5,c7d64fc8,c0ea9418,...) at >> _witness_debugger+0x25 >> witness_checkorder(c11b23f4,1,c0ea9418,267,0,...) at >> witness_checkorder+0x86f _sx_slock(c11b23f4,0,c0ea9418,267,cad672e0,...)= at >> _sx_slock+0x9a >> vnet_sysinit(cad7f000,c0ff9700,5560,cad7e028,c0fb3588,...) at >> vnet_sysinit+0x2b vnet_alloc(cad7e028,c0e8c936,0,10,0,...) at >> vnet_alloc+0x168 >> kern_jail_set(cad672e0,c9470d00,1,c9470d00,0,...) at kern_jail_set+0x1bb= 4 >> sys_jail_set(cad672e0,e1b40cec,c0edada8,c0e9a9a6,c1047f40,...) at >> sys_jail_set+0x50 >> syscall(e1b40d28) at syscall+0x2de >> Xint0x80_syscall() at Xint0x80_syscall+0x21 >> --- syscall (507, FreeBSD ELF32, sys_jail_set), eip =3D 0x280bfd5b, esp >> =3D 0xbfbfe23c, ebp =3D 0xbfbfe328 --- >> >> >> >> This is my setup: >> Host PC: Ubuntu 11.04 (Linux bane 2.6.38-12-generic) >> VirtualBox: 4.1.6 r74713 >> FreeBSD gues: i386 head@234636 (attaching kernel config) >> >> I compile and install /usr/src/tools/tools/vimage >> >> running: >> vimage -c jid=3D0 >> vimage -d jid=3D0 >> >> crashes the kernel (attaching core.txt.2) >> Fatal trap 12: page fault while in kernel mode >> cpuid =3D 0; apic id =3D 00 >> fault virtual address =3D 0xdeadc0e6 >> fault code =A0 =A0 =A0 =A0 =A0 =A0=3D supervisor read, page not present >> instruction pointer =A0 =3D 0x20:0xc0a78d20 >> stack pointer =A0 =A0 =A0 =A0 =3D 0x28:0xc7980c48 >> frame pointer =A0 =A0 =A0 =A0 =3D 0x28:0xc7980c90 >> code segment =A0 =A0 =A0 =A0 =A0=3D base 0x0, limit 0xfffff, type 0x1b >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D DPL 0, pres 1, def32 1, = gran 1 >> processor eflags =A0 =A0 =A0=3D interrupt enabled, resume, IOPL =3D 0 >> current process =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 12 (swi1: netisr 0) >> >> #0 =A0doadump (textdump=3D0) at pcpu.h:244 >> 244 =A0 pcpu.h: No such file or directory. >> =A0 =A0 =A0 in pcpu.h >> (kgdb) #0 =A0doadump (textdump=3D0) at pcpu.h:244 >> #1 =A00xc05104b3 in db_dump (dummy=3D-1062761184, dummy2=3D0, dummy3=3D-= 1, >> =A0 =A0 dummy4=3D0xc798096c "") at /usr/src/sys/ddb/db_command.c:538 >> #2 =A00xc050fbd1 in db_command (last_cmdp=3D0xc10000dc, cmd_table=3D0x0, >> dopager=3D1) at /usr/src/sys/ddb/db_command.c:449 >> #3 =A00xc050fd2a in db_command_loop () at /usr/src/sys/ddb/db_command.c:= 502 >> #4 =A00xc0511d1d in db_trap (type=3D12, code=3D0) > at /usr/src/sys/ddb/db_main.c:231 >> #5 =A00xc09de976 in kdb_trap (type=3D12, code=3D0, >> tf=3D0xc7980c08) >> =A0 =A0 at /usr/src/sys/kern/subr_kdb.c:654 >> #6 =A00xc0cf2eff in trap_fatal (frame=3D0xc7980c08, eva=3D3735929062) >> =A0 =A0 at /usr/src/sys/i386/i386/trap.c:1013 >> #7 =A00xc0cf32ee in trap_pfault (frame=3D0xc7980c08, usermode=3D0, >> eva=3D3735929062) at /usr/src/sys/i386/i386/trap.c:936 >> #8 =A00xc0cf40b1 in trap (frame=3D0xc7980c08) at >> /usr/src/sys/i386/i386/trap.c:546 #9 =A00xc0cdd8ec in calltrap () at >> /usr/src/sys/i386/i386/exception.s:169 #10 0xc0a78d20 in swi_net >> (arg=3D0xc1825880) at /usr/src/sys/net/netisr.c:805 > > The backtrace says it's most probably this line in netisr.c: > > =A0 =A0805 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 CURVNET_SET(m->m_pkthdr.rcvif-= >if_vnet); > >> #11 0xc0979d75 in >> intr_event_execute_handlers (p=3D0xc7dc6598, ie=3D0xc7e95300) at >> /usr/src/sys/kern/kern_intr.c:1260 >> #12 0xc097ac49 in ithread_loop (arg=3D0xc7e0f7a0) >> =A0 =A0 at /usr/src/sys/kern/kern_intr.c:1273 >> #13 0xc0976fa8 in fork_exit (callout=3D0xc097aba0 , >> =A0 =A0 arg=3D0xc7e0f7a0, frame=3D0xc7980d28) at /usr/src/sys/kern/kern_= fork.c:992 >> #14 0xc0cdd994 in fork_trampoline () at >> /usr/src/sys/i386/i386/exception.s:276 (kgdb) >> >> >> It seems that it crashes on netisr_process_workstream_proto, why are >> we even in this function? Is someone sending a packet? > > When netisr_process_workstream_proto() processes queued mbufs, in options > VIMAGE builds it expects all mbufs to have m->m_pkthdr.rcvif set to a val= id > interface, so that the curvnet context could be harvested from there. > > Perhaps you've created a new mbuf and queued it for netisr dispatching, b= ut > have left m->m_pkthdr.rcvif uninitialized? I have not done anything I am aware of, basically I called vimage the first thing I log in freebsd. so I have not assigned it any interface or anything :/ When I do this in single user mode I dont get this problem but I guess that is because no interfaces are up yet :) But I think this is weird: VNET_ASSERT(m->m_pkthdr.rcvif !=3D NULL, ("%s:%d rcvif =3D=3D NULL: m=3D%p", __func__, __LINE__, m)); this panic should have showed up if no one have set rcvif, but when I check its value it is pointing to 0xdeadc0de, why? > > Marko --=20 Monthadar Al Jaberi