From owner-freebsd-bugs@FreeBSD.ORG Sat Dec 10 18:30:04 2005 Return-Path: X-Original-To: freebsd-bugs@hub.freebsd.org Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E751B16A420 for ; Sat, 10 Dec 2005 18:30:04 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6F47843D67 for ; Sat, 10 Dec 2005 18:30:03 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id jBAIU3k4001286 for ; Sat, 10 Dec 2005 18:30:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id jBAIU3r5001279; Sat, 10 Dec 2005 18:30:03 GMT (envelope-from gnats) Resent-Date: Sat, 10 Dec 2005 18:30:03 GMT Resent-Message-Id: <200512101830.jBAIU3r5001279@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Simon Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BDB9516A41F for ; Sat, 10 Dec 2005 18:26:00 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [216.136.204.117]) by mx1.FreeBSD.org (Postfix) with ESMTP id 559D143D4C for ; Sat, 10 Dec 2005 18:26:00 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.13.1/8.13.1) with ESMTP id jBAIQ0Jw049892 for ; Sat, 10 Dec 2005 18:26:00 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.13.1/8.13.1/Submit) id jBAIQ0Rl049891; Sat, 10 Dec 2005 18:26:00 GMT (envelope-from nobody) Message-Id: <200512101826.jBAIQ0Rl049891@www.freebsd.org> Date: Sat, 10 Dec 2005 18:26:00 GMT From: Simon To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-2.3 Cc: Subject: kern/90206: Server reboot after "FAILURE - out of memory in ata_raid_init_request" X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Dec 2005 18:30:05 -0000 >Number: 90206 >Category: kern >Synopsis: Server reboot after "FAILURE - out of memory in ata_raid_init_request" >Confidential: no >Severity: critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Dec 10 18:30:02 GMT 2005 >Closed-Date: >Last-Modified: >Originator: Simon >Release: 6.0-RELEASE >Organization: KSD >Environment: FreeBSD bnet 6.0-RELEASE FreeBSD 6.0-RELEASE #0: Sat Dec 10 02:13:22 UTC 2005 ender@bnet:/usr/obj/usr/src/sys/BNET i386 >Description: A server deamon I run keeps its database in memory, and after a schedualed time flushes the dirty database files from memory to harddrives. The database files are stored as millions of tiny CDB binary files in 1 directory. I use RAID 0+1 on a promise raid card. When the server kicks off a flush and starts doing a large amount of IO, i get the following errors in dmesg: Dec 10 13:51:39 bnet kernel: DOH! ata_alloc_request failed! Dec 10 13:51:39 bnet kernel: FAILURE - out of memory in ata_raid_init_request Dec 10 13:51:39 bnet last message repeated 4 times Dec 10 13:51:39 bnet kernel: FAILURE g- out of memory i_vn fsat_da_one(raid):_iarni0st_1dre[WquRIestTE(offset=2030 Dec 10 13:51:39 bnet kernel: g_vfs Dec 10 13:51:39 bnet kernel: _done()FA:aILr0UREs1 -d[ oWRIut oTEf (omeffmoseryt= 2in ata0329_rai03d_94init_request Dec 10 13:51:39 bnet kernel: FAILURE - out of memory in ata_raid_init_request Dec 10 13:51:39 bnet kernel: FAILURE - out of memory in ata_rai62d_4, inleit_rengquthest=16384)]error = 5 Dec 10 13:51:39 bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203290411008, length=16384)]error = 5 Dec 10 13:51:39 bnet kernel: Dec 10 13:51:39 bnet kernel: g_vfs_dFAoInLUe():RE - arou0st 1dof[ mWRemITory E(inoffs aet=t2a_raid0329_init0427392 Dec 10 13:51:39 bnet kernel: g_vfs_done():ar0s1d[WRITE_re(oquffseest=t203290460160, length=16384)]error = 5 Dec 10 13:51:39 bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203290509312, length=16384)]error = 5 Dec 10 13:51:39 bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=20329057 Dec 10 13:51:39 bnet kernel: 4848, lenFAILgtURh=E - 163ou84t )]of error = 5 Dec 10 13:51:39 bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203290787840, length=16384)]error = 5 Dec 10 13:51:39 bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203675779072, length=16384)]error me= mory5 in ata_r Dec 10 13:51:39 bnet kernel: FAILURE - out of memory in ata_raid_init_request Dec 10 13:51:39 bnet kernel: F Dec 10 13:51:39 bnet kernel: AILURE - out go_fv fmse_mdoornye (i)n: aart0as_1rda[iWdR_IiTnEi(to_frfesqeute=st Dec 10 13:51:39 bnet kernel: FAILURE - out of memory i2n0 3a8t6a8_5r2a0i4d4_8i,n ilte_nrgetqhu=est Dec 10 13:51:39 bnet kernel: 16384)]error = 5 Dec 10 13:51:39 bnet kernel: FAILURE - ougt_ ovff mse_modroy nien (a)ta:_arra0isd1_di[nWiRtI_TrEe(qofufesestt= Dec 10 13:51:39 bnet kernel: 203868536832, length=F16A3I84L)U]ReErr o-r =o u5t Dec 10 13:51:39 bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203 8o6f8 5m5e3m2o1r6y, ilne nagttah_=raid_1i6n3i8t Dec 10 13:51:39 bnet kernel: FAILURE - out of memorry i=n 5a Dec 10 13:51:39 bnet kernel: ta_raid_init_request Dec 10 13:51:39 bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=F2A04I0L61U2R78E2 0-8 ,o ulte nogft hm=emory 1in ata Dec 10 13:51:39 bnet kernel: Dec 10 13:52:02 bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=12288)]error = 5 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ Dec 10 13:52:02 bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=14336)]error = 5 Dec 10 13:52:02 bnet kernel: FAILURE - out of memory in ata_raid_init_request Dec 10 13:52:02 bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=14336)]error = 5 Dec 10 13:52:02 bnet kernel: FAILURE - out of memory in ata_raid_init_request Dec 10 13:52:02 bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=14336)]error = 5 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ Dec 10 13:53:24 bnet kernel: Copyright (c) 1992-2005 The FreeBSD Project. Dec 10 13:53:24 bnet kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Dec 10 13:53:24 bnet kernel: The Regents of the University of California. All rights reserved. Dec 10 13:53:24 bnet kernel: FreeBSD 6.0-RELEASE #0: Sat Dec 10 02:13:22 UTC 2005 Kernel: http://ender.tog.net/crash/BNET dmesg: http://ender.tog.net/crash/dmesg.today loader.conf: http://ender.tog.net/crash/loader.conf messages: http://ender.tog.net/crash/messages.0 This problem was also repeated on just RAID 0 array. Unfortunatally i do not have a kernel dump file at the moment due to the bnet# dumpon /dev/ar0 dumpon: ioctl(DIOCSKERNELDUMP): Operation not supported by device Limitation, but hopefully will have another disk in this server to get a proper kernel dump soon. >How-To-Repeat: Run a deamon that uses alot of disk IO on a raid array. >Fix: >Release-Note: >Audit-Trail: >Unformatted: