From owner-freebsd-hackers@FreeBSD.ORG Mon Jun 13 18:11:29 2005 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DE25516A41C for ; Mon, 13 Jun 2005 18:11:29 +0000 (GMT) (envelope-from julian@elischer.org) Received: from bigwoop.vicor-nb.com (bigwoop.vicor-nb.com [208.206.78.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id A570443D53 for ; Mon, 13 Jun 2005 18:11:27 +0000 (GMT) (envelope-from julian@elischer.org) Received: from [208.206.78.97] (julian.vicor-nb.com [208.206.78.97]) by bigwoop.vicor-nb.com (Postfix) with ESMTP id 1C76E7A403; Mon, 13 Jun 2005 11:11:27 -0700 (PDT) Message-ID: <42ADCC4F.4090201@elischer.org> Date: Mon, 13 Jun 2005 11:11:27 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.7) Gecko/20050423 X-Accept-Language: en, hu MIME-Version: 1.0 To: Apache Xie References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: contigmalloc() and mmap() X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jun 2005 18:11:30 -0000 2nd try to answer this.. Apache Xie wrote: > I have some experiences in writing Linux device driver, > but I am new to FreeBSD kernel, although from the first > glimpse, there seems no big differences between the kernel > operations a char device driver can use, but I met some > problems when the driver is running in FreeBSD. > > Our device is an experimental communication card, which can > do remote DMA between two cards, which means the card in one > node can DMA memory data to network, and when data are transfered > to the card on another node, it will be DMAed to memory too. > > > Because the card can only use contiguous physical memory for > DMA operations, so data in user process will be copied to an > contiguous memory buffer, then DMA will tranfer data in this > buffer, and this buffer is allocated by driver using __get_free_pages() > in Linux. The buffer is mmaped to user process space, so user > process can do the copy directly in user space, it can directly > orgnize data in this mmaped memory too. ok, so far so good.. machine to machine DMA capability with contiguous buffers.. nothing too complicated so far.. I once did a simiar system on BSD4.3 in 1992. > > When I am porting my driver to FreeBSD, I use /dev/bktr driver as the > example, seems easily, just using contigmalloc() to allocate the > buffer, and in driver's _mmap() function, return the physical > address for each page to be mmaped. > > The problem is, in Linux, I allocate buffer in driver's ioctl() > function, and free it in a timer function, many processes may use the > driver > at the same time, each process use a different kernel buffer, when > the process first use the driver, it calls __get_free_pages() to allocate > kernel buffer, and when it exit, it trigger timer function, the timer > function will can free_pages() to free the buffer, so these two kernel > interfaces will be called frequently, but this usage pattern works > correctly in Linux. I'm unsure about this usage of the timer (callout(?) ) how does the timer know which buffer pages to remove? and if each userspace process does an ioctl to allocate a different buffer, are the new pages also visible to other processes? in FreeBSD your driver can register with at_exit() in 4.x and with eventhandler_register() in 5.x or 6.x to have stuff done when the process exits. (there are also at_fork and at_exec handlers too.) see man at_exit (etc.) in 4.x and man 9 EVENTHANDLER in 5.x and 6.x You could also make a hash table on the PID so that you keep different information available for different processes and look them up using curproc->p_pid (for 4.x) and curthhread->td_proc->p_pid for 5.x and 6.x to look up the Using these primatives you should be able to simulate what you need I believe. > In FreeBSD, the driver works in the same pattern, but when a user process > mmap driver's buffer (allocated by contigmalloc()) and is killed, then > when > another process mmap the same buffer again, sometimes it cannot get > correct > data from the mmaped pages, which means the user space virtual aderess > may > not point to the correct physical page of driver's buffer, sometimes > the OS > even panic with some information such as "Trap 12, Page not present" etc. sounds like you are getting the buffers confused.. are you doing correc treference counting on the buffers? > > I browsed kernel tree, I found those drivers which use contigmalloc() and > contigfree() always call these two kernel interfaces in _attach() and > _detach(), but in my driver, I call contigmalloc() in ioctl(), and call > contigfree() in a callout function which is set by callout_reset(). why by callout_reset()? > What I want to know is if contigmalloc() and contigfree() can only be > used > under some conditions? not that I know of.. > > And recently, I modified my driver, I allocated a big chunk of contiguous > physical memory using contigmalloc() in the driver's _attach() function, > and I use a simple first-fit algorithm to manage this memory myself, > which > mean in ioctl() I use my allocate/deallocate functions instead of > contigmalloc(), > in _detach() function contigfree() is called to free the big chunk of > memory, > no panic again, but sometimes, process cannot get the correct data from > the mmaped memory. I don't know why. it does sound a bit like you are not keeping the information separated between processes enough. > > Any help is welcomed. > > Thanks. > > _________________________________________________________________ > Express yourself instantly with MSN Messenger! Download today it's > FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/ > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org"