From owner-freebsd-virtualization@FreeBSD.ORG Mon May 9 14:33:41 2011 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10BD0106566C; Mon, 9 May 2011 14:33:41 +0000 (UTC) (envelope-from zec@fer.hr) Received: from munja.zvne.fer.hr (munja.zvne.fer.hr [161.53.66.248]) by mx1.freebsd.org (Postfix) with ESMTP id 936B48FC08; Mon, 9 May 2011 14:33:40 +0000 (UTC) Received: from sluga.fer.hr ([161.53.66.244]) by munja.zvne.fer.hr with Microsoft SMTPSVC(6.0.3790.4675); Mon, 9 May 2011 16:21:35 +0200 Received: from localhost ([161.53.19.8]) by sluga.fer.hr with Microsoft SMTPSVC(6.0.3790.4675); Mon, 9 May 2011 16:21:35 +0200 From: Marko Zec To: freebsd-virtualization@freebsd.org Date: Mon, 9 May 2011 16:21:15 +0200 User-Agent: KMail/1.9.10 References: <86aaewdopy.fsf@kopusha.home.net> In-Reply-To: <86aaewdopy.fsf@kopusha.home.net> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <201105091621.16414.zec@fer.hr> X-OriginalArrivalTime: 09 May 2011 14:21:35.0404 (UTC) FILETIME=[66F166C0:01CC0E54] Cc: Mikolaj Golub , Kostik Belousov Subject: Re: vnet: acessing module's virtualized global variables from another module X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 May 2011 14:33:41 -0000 On Monday 09 May 2011 14:48:25 Mikolaj Golub wrote: > Hi, > > Trying ipfw_nat under VIMAGE kernel I got this panic on the module load: Hi, I think the problem here is that curvnet context is not set properly on entry to ipfw_nat_modevent(). The canonical way to initialize VNET-enabled subsystems is to trigger them using VNET_SYSINIT() macros (instead of using modevent mechanisms), which in turn ensure that: a) that the initializer function gets invoked for each existing vnet b) curvnet context is set properly on entry to initializer functions and Cheers, Marko > Fatal trap 12: page fault while in kernel mode > cpuid = 1; apic id = 01 > fault virtual address = 0x4 > fault code = supervisor read, page not present > instruction pointer = 0x20:0xc09f098e > stack pointer = 0x28:0xf563b944 > frame pointer = 0x28:0xf563b998 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 4264 (kldload) > > witness_checkorder(c6d5e91c,9,ca0ac2e3,223,0,...) at > witness_checkorder+0x6e _rw_wlock(c6d5e91c,ca0ac2e3,223,0,c0e8f795,...) at > _rw_wlock+0x82 > ipfw_nat_modevent(c98a48c0,0,0,75,0,...) at ipfw_nat_modevent+0x41 > module_register_init(ca0ad508,0,c0e8d834,e6,0,...) at > module_register_init+0xa7 > linker_load_module(0,f563bc18,c0e8d834,3fc,f563bc28,...) at > linker_load_module+0xa05 > kern_kldload(c86835c0,c72d3400,f563bc40,0,c8d0d000,...) at > kern_kldload+0x133 kldload(c86835c0,f563bcec,c09e8940,c86835c0,0,...) at > kldload+0x74 syscallenter(c86835c0,f563bce4,c0ce05dd,c1022150,0,...) at > syscallenter+0x263 syscall(f563bd28) at syscall+0x34 > Xint0x80_syscall() at Xint0x80_syscall+0x21 > --- syscall (304, FreeBSD ELF32, kldload), eip = 0x280da00b, esp = > 0xbfbfe79c, ebp = 0xbfbfec88 - > > It crashed on acessing data from virtualized global variable V_layer3_chain > in ipfw_nat_modevent(). V_layer3_chain is defined in ipfw module and it > turns out that &V_layer3_chain returns wrong location from anywhere but > ipfw.ko. > > May be this is a known issue, but I have not found info about this, so > below are details of investigation why this happens. > > Virtualized global variables are defined using the VNET_DEFINE() macro, > which places them in the 'set_vnet' linker set (in the base kernel or in > module). This is used to > > 1) copy these "default" values to each virtual network stack instance when > created; > > 2) act as unique global names by which the variable can be referred to. The > location of a per-virtual instance variable is calculated at run-time like > in the example below for layer3_chain variable in the default vnet (vnet0): > > vnet0->vnet_data_base + (uintptr_t) & vnet_entry_layer3_chain (1) > > For modules the thing is more complicated. When a module is loaded its > global variables from 'set_vnet' linker set are copied to the kernel > 'set_vnet', and for module to be able to access them the linker reallocates > all references accordingly (kern/link_elf.c:elf_relocaddr()): > > if (x >= ef->vnet_start && x < ef->vnet_stop) > return ((x - ef->vnet_start) + ef->vnet_base); > > So from inside the module the access to its virtualized variables works, > but from the outside we get wrong location using calculation like above > (1), because &vnet_entry_layer3_chain returns address of the variable in > the module's 'set_vnet'. > > The workaround is to compile such modules into the kernel or use a hack I > have done for ipfw_nat -- add the function to ipfw module which returns the > location of virtualized layer3_chain variable and use this location instead > of V_layer3_chain macro (see the attached patch). > > But I suppose the problem is not a new and there might be better approach > already invented to deal with this?