From owner-freebsd-hackers@FreeBSD.ORG Thu Jul 15 18:13:56 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 992891065673 for ; Thu, 15 Jul 2010 18:13:56 +0000 (UTC) (envelope-from krivenok.dmitry@gmail.com) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 1F4778FC08 for ; Thu, 15 Jul 2010 18:13:55 +0000 (UTC) Received: by ewy26 with SMTP id 26so394224ewy.13 for ; Thu, 15 Jul 2010 11:13:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=2g0uAjNjNZNxFq56f+NdclQ4GU3VnvQ6nq6607Q4IZU=; b=HKoIT8ZD+WfkReTX7cZwReCjkQEhfGIpoc/DLO83B1v5SaQr2iLjJj4zkUwVfhw1zv cxcPOCz/37aEvNcrgWcoljHCwYhz+jwVEQerA66uMWpGaqbpetB73e24VBwrWbcEzK5H 2YCbxGJtL8Q87qDn1xXPjtJabFlDCKcW7bsqU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=rox56dXuO/cU9/dUFbiy8jz45FNpHWIsIcr4CWQKCV3Tss4h75Kb0u4mpvWTWhJh/v S2YPxEtx1q5UBisIDn2YdUQtKWeYBgNRWnyECo2a0GI2hz/AGu2X4a2qwIpMqiwJ6La2 VS6gUkobc7LcYVuxzkJSQ8eBfkKSduUJEfNKc= MIME-Version: 1.0 Received: by 10.213.20.142 with SMTP id f14mr50953ebb.30.1279217634885; Thu, 15 Jul 2010 11:13:54 -0700 (PDT) Received: by 10.213.23.14 with HTTP; Thu, 15 Jul 2010 11:13:54 -0700 (PDT) In-Reply-To: <20100715143235.GU2381@deviant.kiev.zoral.com.ua> References: <20100715143235.GU2381@deviant.kiev.zoral.com.ua> Date: Thu, 15 Jul 2010 22:13:54 +0400 Message-ID: From: Dmitry Krivenok To: Kostik Belousov Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: Kernel linker and undefined references in KLD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jul 2010 18:13:56 -0000 Unfortunately, one can easily miss such problems during build of the module= . I'm working on a system which consists of lots of user-space programs and kernel modules and uses it's own complicated build system. gcc option -Werror is not used by the build system (I believe it should), that's why we just missed a compiler warning when called undeclared function (actually declared as static deep in kernel source). Then we got kernel panic and started investigating it. I disassembled seltdinit function and found that it takes pointer to thread from %eax register and global kernel functions (e.g. kern_select) calling seltdinit copy the pointer to %eax just before "call" instruction. Then I disassembled my own function calling seltdinit and found that it passes the pointer via stack and doesn't do anything with %eax register. I'm not OS/compilers/ASM expert, but I guess that gcc is free to optimize passing of parameters to static functions because it has all the information about callers of these static functions (all stuff is inside one translation unit). So kernel functions call static functions in a right way, but modules compiled separately may use wrong calling convention. On Thu, Jul 15, 2010 at 6:32 PM, Kostik Belousov wrot= e: > On Thu, Jul 15, 2010 at 06:07:36PM +0400, Dmitry Krivenok wrote: >> Hello Hackers, >> >> I have a question about kernel linker. >> Please take a look at an example of simple module: >> >> ########################################################################= #### >> #include >> #include >> #include >> #include >> #include >> #include >> #include >> >> /* DECLARING A FUNCTION JUST TO AVOID WARNING WHICH IS TREAT AS ERROR >> BY DEFAULT */ >> void seltdinit(struct thread* t); >> >> static int event_handler(struct module *module, int event, void *arg) >> { >> =A0 int e =3D 0; >> =A0 switch (event) >> =A0 =A0 { >> =A0 =A0 =A0 case MOD_LOAD: >> =A0 =A0 =A0 =A0 /* CALLING A FUNCTION DECLARED AS STATIC IN kern/sys_gen= eric.c */ >> =A0 =A0 =A0 =A0 seltdinit(curthread); >> =A0 =A0 =A0 =A0 break; >> =A0 =A0 =A0 case MOD_QUIESCE: >> =A0 =A0 =A0 =A0 break; >> =A0 =A0 =A0 case MOD_UNLOAD: >> =A0 =A0 =A0 =A0 break; >> =A0 =A0 =A0 case MOD_SHUTDOWN: >> =A0 =A0 =A0 =A0 break; >> =A0 =A0 =A0 default: >> =A0 =A0 =A0 =A0 e =3D EOPNOTSUPP; >> =A0 =A0 =A0 =A0 break; >> =A0 =A0 } >> >> =A0 return(e); >> }; >> ########################################################################= #### >> >> As you can see, this module calls seltdinit function declared as >> _static_ in kern/sys_generic.c file. >> I added a declaration of seltdinit function w/o static just to avoid >> compilation error since -Werror is used >> by default in FreeBSD. >> >> Then I successfully built the module: >> >> $ make >> Warning: Object directory not changed from original >> /usr/home/krived/work/freebsd/trouble >> cc -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE >> -nostdinc =A0 -I. -I@ -I@/contrib/altq -finline-limit=3D8000 --param >> inline-unit-growth=3D100 --param large-function-growth=3D1000 -fno-commo= n >> -fno-omit-frame-pointer =A0-mcmodel=3Dkernel -mno-red-zone =A0-mfpmath= =3D387 >> -mno-sse -mno-sse2 -mno-sse3 -mno-mmx -mno-3dnow =A0-msoft-float >> -fno-asynchronous-unwind-tables -ffreestanding -fstack-protector >> -std=3Diso9899:1999 -fstack-protector -Wall -Wredundant-decls >> -Wnested-externs -Wstrict-prototypes =A0-Wmissing-prototypes >> -Wpointer-arith -Winline -Wcast-qual =A0-Wundef -Wno-pointer-sign >> -fformat-extensions -c trouble.c >> ld =A0-d -warn-common -r -d -o trouble.ko trouble.o >> :> export_syms >> awk -f /sys/conf/kmod_syms.awk trouble.ko =A0export_syms | xargs -J% >> objcopy % trouble.ko >> objcopy --strip-debug trouble.ko >> $ >> >> As expected, seltdinit symbol is undefined in trouble.ko file: >> $ nm trouble.ko | grep seltdinit >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0U seltdinit >> $ >> and is local in kernel binary file: >> $ nm /boot/kernel/kernel | grep seltdinit >> ffffffff805fda50 t seltdinit >> $ >> >> I expected to get something like "undefined reference to seltdinit" >> error, but the module was loaded w/o any errors: >> >> $ sudo make load >> /sbin/kldload -v /usr/home/krived/work/freebsd/trouble/trouble.ko >> Loaded /usr/home/krived/work/freebsd/trouble/trouble.ko, id=3D2 >> $ kldstat | grep trouble >> =A02 =A0 =A01 0xffffffff81212000 126 =A0 =A0 =A0trouble.ko >> $ sudo make unload >> /sbin/kldunload -v trouble.ko >> Unloading trouble.ko, id=3D2 >> $ >> >> Could you please explain why the linker (or whatever is loading >> modules in FreeBSD) doesn't complain? >> >> For example, ld tool complains for the similar user-space example: >> >> ########################################################################= #### >> $ cat seltdinit.c >> static void seltdinit(void* td) >> { >> =A0 return; >> } >> $ cat main.c >> void seltdinit(void* td); >> >> int main() >> { >> =A0 seltdinit(0); >> =A0 return 0; >> } >> $ gcc -Wall -c seltdinit.c >> seltdinit.c:2: warning: 'seltdinit' defined but not used >> $ gcc -Wall -c main.c >> $ gcc seltdinit.o main.o -o sel >> main.o: In function `main': >> main.c:(.text+0xa): undefined reference to `seltdinit' >> collect2: ld returned 1 exit status >> $ >> ########################################################################= #### >> >> Thanks in advance! > The kernel linker ignores weak attribute of the symbol, as you see. > There is more bugs in this department, in regard of the list of > exported symbols from the modules. > > I have a patch that fixes the issues, but I am leery to commit it, since > the fix effectively breaks significant set of the modules. > --=20 Sincerely yours, Dmitry V. Krivenok e-mail: krivenok.dmitry@gmail.com skype: krivenok_dmitry jabber: krivenok_dmitry@jabber.ru icq: 242-526-443