Date: Tue, 22 May 2007 18:45:00 +1000 From: Lawrence Stewart <lstewart@room52.net> To: =?UTF-8?B?RGFnLUVybGluZyBTbcO4cmdyYXY=?= <des@des.no> Cc: freebsd-hackers@freebsd.org Subject: Re: Writing a plain text file to disk from kernel space Message-ID: <4652AD8C.7000605@room52.net> In-Reply-To: <86sl9qtpd1.fsf@dwp.des.no> References: <4649349D.4060101@room52.net> <200705150847.38838.marc.loerner@hob.de> <46499491.2010205@room52.net> <f2j5hf$hap$2@sea.gmane.org> <46515DE0.20209@room52.net> <86sl9qtpd1.fsf@dwp.des.no>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] OK well that's cleared up. Thanks DES. So I went back to trying the kernio code I found here (http://people.freebsd.org/~pjd/misc/kernio/) which does operate directly on vnodes (first time I tried it I was getting kernel panics, so I assumed the code was a bit dated and carried on down the path of trying to get file descriptors working). Anyways, I modified the filewriter kernel module that I attached to my previous post so that it used the kio code, and verified that the code does work when called across different threads (in the module's init and deinit functions). However, when I ported it into my other module that I'm actually working on that uses the pfil hooks, I started getting hard resets again. After further investigation, it turns out that the pfil input hook I'm using, which catches packets as they traverse up the network stack, has no problems, and will happily write to the file using the kio_write function. However, in the pfil output hook, a call to kio_write causes a hard reset, with the following text shown on tty0: Sleeping thread (tid 100069, pid 613) owns a non-sleepable lock panic: sleeping thread If I comment out the kio_write code and put a printf instead, there are no such problems, so it seems the kio_write function is doing something that is upsetting the kernel, but only when called from a function that is acting as a pfil output hook? Strikes me as odd behaviour. I don't understand which thread the error is in relation to, why that thread is sleeping or which lock it is referring to. I tried wrapping the call to kio_write in a mutex, in case there was a race condition caused by multiple threads trying to write to the file at the one time, but that hasn't made a difference at all. I've attached the code that demonstrates the problem, but be warned: I've intentionally left it in a state that demonstrates the problem, and it will therefore hard reset any machine you run it on. Any thoughts on what's going on and how I can make the kio code not break the kernel when called from within a function that is acting as a pfil output hook? Cheers, Lawrence Dag-Erling Smørgrav wrote: > Lawrence Stewart <lstewart@room52.net> writes: > >> I suspect that you can't use a file descriptor that was opened in one >> thread in a completely different thread, but I'm not sure if this is >> true, and if it is true, how to get around it. >> > > A file descriptor is an index into a file table. Different threads have > different file tables. > > If you want to read from or write to files within the kernel, you need > to operate directly on vnodes, not on file descriptors. > > DES > [-- Attachment #2 --] ELF 4 4 ( @ @ @ x x % . $ - ) " & # + , * ! ' % ( @ @ / 7 C S [ d v h P $ % , 5 ? _DYNAMIC _GLOBAL_OFFSET_TABLE_ vn_start_write vn_lock mtx_destroy _mtx_lock_flags uprintf mtx_init vn_finished_write module_register_init pfil_add_hook bzero vn_open_cred _mtx_unlock_flags VOP_WRITE_APV vop_lease_desc vop_write_desc pfil_head_get vop_unlock_desc VOP_UNLOCK_APV VOP_LEASE_APV NDFREE vn_close rootvnode pfil_remove_hook __start_set_sysinit_set __stop_set_sysinit_set __start_set_modmetadata_set __stop_set_modmetadata_set _edata __bss_start _end ' . r $ ) . A e n z $ ( 8 < + + 5 ? ) Y # # k $ $ * 6 F % h ! z 3 H " w - - 3 s UWVS`d= P$Z u B P$Ju BEjE EE E E } uE PuVj߉V 1u(UEUPE E }rXZEe[^_Ud P juUWVS|]d= j|Vj EPE j|xPu]uԉ]}E E E E E Wh u U(EEU}E E Pr YU[UEUԍEUPE E EpXZEUU}E E Pr^_xe[^1_ÐUj.h j h j h 5 j0h j h 1ÐUj:h j h j h 5 tj<h j h 1ÐUVSjjÅXZ t tNt01e[^Ív Sjj h Sjj h ѐSjj h Sjj h 뭐UE t Ht á 1҅tÐ_ t j j h h h h h h h h 1҃ 땡 1҅t p h Y5 $ h h filewriter filewriter.c packet in packet out filewriter_mutex /var/log/test23.log 1.0 Loaded %s %s Unloaded %s %s 0 $ $ 0 o' @ $FreeBSD$ GCC: (GNU) 3.4.4 [FreeBSD] 20050518 GCC: (GNU) 3.4.4 [FreeBSD] 20050518 .symtab .strtab .shstrtab .hash .dynsym .dynstr .rel.dyn .text .rodata set_sysinit_set set_modmetadata_set .data .dynamic .got .bss .comment T ! ) 1 0 : @ 2 v H X l , r @ @ x { D T 4 l < 1 K $ Z g 0 ? $ ? < w ) 2 " < | @ F U q @ z . F U a p ~ $ __set_sysinit_set_sym_filewritermodule_sys_init filewritermodule_sys_init filewriter_mod load_handler __set_modmetadata_set_sym__mod_metadata_md_filewriter _mod_metadata_md_filewriter filewriter_hooked filewriter_chkinput filewriter_mtx filewriter_vnode filewriter_chkoutput filewriter_hook_pfil kio_open kio_write kio_close vn_start_write __start_set_modmetadata_set _DYNAMIC vn_lock mtx_destroy _mtx_lock_flags uprintf mtx_init vn_finished_write module_register_init pfil_add_hook bzero __stop_set_sysinit_set vn_open_cred _mtx_unlock_flags VOP_WRITE_APV __start_set_sysinit_set vop_lease_desc __bss_start vop_write_desc pfil_head_get vop_unlock_desc VOP_UNLOCK_APV VOP_LEASE_APV _edata _GLOBAL_OFFSET_TABLE_ _end NDFREE vn_close rootvnode __stop_set_modmetadata_set pfil_remove_hook [-- Attachment #3 --] /*- * Copyright (c) 2006 Pawel Jakub Dawidek <p.dawidek@wheel.pl> * All rights reserved. * * Downloaded from http://people.freebsd.org/~pjd/misc/kernio/ */ #ifndef _SYS_KERNIO_H_ #define _SYS_KERNIO_H_ struct vnode *kio_open(const char *file, int flags, int cmode); void kio_close(struct vnode *vp); int kio_write(struct vnode *vp, void *buf, size_t size); #endif /* _SYS_KERNIO_H_ */ [-- Attachment #4 --] SRCS=vnode_if.h subr_kernio.c filewriter.c KMOD=filewriter .include <bsd.kmod.mk> [-- Attachment #5 --] /*- * Copyright (c) 2006 Pawel Jakub Dawidek <p.dawidek@wheel.pl> * All rights reserved. * * Downloaded from http://people.freebsd.org/~pjd/misc/kernio/ */ #include <sys/cdefs.h> __FBSDID("$FreeBSD$"); #include <sys/param.h> #include <sys/systm.h> #include <sys/kernel.h> #include <sys/kthread.h> #include <sys/namei.h> #include <sys/proc.h> #include <sys/filedesc.h> #include <sys/vnode.h> #include <sys/malloc.h> #include <sys/unistd.h> #include <sys/fcntl.h> #include "kernio.h" struct vnode * kio_open(const char *file, int flags, int cmode) { struct thread *td = curthread; struct nameidata nd; int error; if (td->td_proc->p_fd->fd_rdir == NULL) td->td_proc->p_fd->fd_rdir = rootvnode; if (td->td_proc->p_fd->fd_cdir == NULL) td->td_proc->p_fd->fd_cdir = rootvnode; flags = FFLAGS(flags); NDINIT(&nd, LOOKUP, NOFOLLOW, UIO_SYSSPACE, file, td); error = vn_open_cred(&nd, &flags, cmode, td->td_ucred, -1); NDFREE(&nd, NDF_ONLY_PNBUF); if (error != 0) return (NULL); /* We just unlock so we hold a reference. */ VOP_UNLOCK(nd.ni_vp, 0, td); return (nd.ni_vp); } void kio_close(struct vnode *vp) { struct thread *td = curthread; vn_close(vp, FWRITE, td->td_ucred, td); } int kio_write(struct vnode *vp, void *buf, size_t size) { struct thread *td = curthread; struct mount *mp; struct uio auio; struct iovec aiov; bzero(&aiov, sizeof(aiov)); bzero(&auio, sizeof(auio)); aiov.iov_base = buf; aiov.iov_len = size; auio.uio_iov = &aiov; auio.uio_offset = 0; auio.uio_segflg = UIO_SYSSPACE; auio.uio_rw = UIO_WRITE; auio.uio_iovcnt = 1; auio.uio_resid = size; auio.uio_td = td; /* * Do all of the junk required to write now. */ vn_start_write(vp, &mp, V_WAIT); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY, td); VOP_LEASE(vp, td, td->td_ucred, LEASE_WRITE); // TODO: Ideally, the flags passed into VOP_WRITE on the next line shouldn't // be hardcoded. Is there a way to calculate them based on the flags // passed into kio_open? VOP_WRITE(vp, &auio, IO_UNIT | IO_SYNC | IO_APPEND, td->td_ucred); VOP_UNLOCK(vp, 0, td); vn_finished_write(mp); return (0); }
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4652AD8C.7000605>