From owner-freebsd-scsi@FreeBSD.ORG Sun Jan 18 15:37:16 2004 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 16FA116A4CE; Sun, 18 Jan 2004 15:37:16 -0800 (PST) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 753FF43D2F; Sun, 18 Jan 2004 15:37:14 -0800 (PST) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) i0INbD82097879; Sun, 18 Jan 2004 15:37:13 -0800 (PST) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.12.9p2/8.12.9/Submit) id i0INbDtj097878; Sun, 18 Jan 2004 15:37:13 -0800 (PST) (envelope-from dillon) Date: Sun, 18 Jan 2004 15:37:13 -0800 (PST) From: Matthew Dillon Message-Id: <200401182337.i0INbDtj097878@apollo.backplane.com> To: Scott Long , freebsd-hackers@freebsd.org, Paul Twohey , scsi@freebsd.org References: <20040118160802.GC32115@FreeBSD.org.ua> <200401181844.i0IIivlQ096389@apollo.backplane.com> <400AE3AB.1070102@freebsd.org> <200401181957.i0IJvFTe096883@apollo.backplane.com> <200401182257.i0IMv63i097663@apollo.backplane.com> Subject: Re3: [CHECKER] bugs in FreeBSD X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Jan 2004 23:37:16 -0000 More research... correct me if I am wrong but it appears that the 5.x kmem_malloc() code may have some issues. If you look down at around line 349 there is a comment: /* * Note: if M_NOWAIT specified alone, allocate from * interrupt-safe queues only (just the free list). If * M_USE_RESERVE is also specified, we can also * allocate from the cache. Neither of the latter two * flags may be specified from an interrupt since interrupts * are not allowed to mess with the cache queue. */ if ((flags & (M_NOWAIT|M_USE_RESERVE)) == M_NOWAIT) pflags = VM_ALLOC_INTERRUPT | VM_ALLOC_WIRED; else pflags = VM_ALLOC_SYSTEM | VM_ALLOC_WIRED; Here's the problem... the problem is that malloc(...M_NOWAIT) is used by interrupts not only to avoid blocking, but also to avoid messing with the VM Page 'cache' queue. But in 5.x it is possible for non-interrupt threads to preempt other non-interrupt threads indirectly (due to an interrupt trying to get a mutex that a non-interrupt thread currently holds). Am I correct? But the non-interrupt thread will almost certainly be making memory allocations with M_WAITOK, which means that a preempting thread *CAN* wind up pulling pages out of the 'cache' queue. Now, my understanding is that 5.x's mutexes around the VM system means that this, in fact, will work just fine. So, that means that the above comment is no longer correct, right? In fact, interrupts *should* be able to allocate pages from the VM page 'cache' queue in 5.x now. This leads to the obvious conclusion that 'critical' code, such as the CAM code, which cannot afford to block but which also does terrible things when an M_NOWAIT allocation fails should be able to use (M_WAITOK|M_USE_RESERVE|M_USE_INTERRUPT_RESERVE) and this would result in far safer operation then the current M_NOWAIT use results in. (M_USE_INTERRUPT_RESERVE would be a new M_* flag that allows the system to exhaust the entire free page reserve if necessary and has the same effect as M_NOWAIT had before, but the combination of flags would now allow interrupt-time allocations to also allocate from the cache queue making it virtually impossible for such allocations to fail and that, combined with M_WAITOK, would allow all NULL checks to be removed. It could actually be considered a critical error for the above flags combination to deadlock. -Matt