Date: Sat, 10 Dec 2005 18:26:00 GMT From: Simon <ender@tog.net> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/90206: Server reboot after "FAILURE - out of memory in ata_raid_init_request" Message-ID: <200512101826.jBAIQ0Rl049891@www.freebsd.org> Resent-Message-ID: <200512101830.jBAIU3r5001279@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 90206 >Category: kern >Synopsis: Server reboot after "FAILURE - out of memory in ata_raid_init_request" >Confidential: no >Severity: critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Dec 10 18:30:02 GMT 2005 >Closed-Date: >Last-Modified: >Originator: Simon >Release: 6.0-RELEASE >Organization: KSD >Environment: FreeBSD bnet 6.0-RELEASE FreeBSD 6.0-RELEASE #0: Sat Dec 10 02:13:22 UTC 2005 ender@bnet:/usr/obj/usr/src/sys/BNET i386 >Description: A server deamon I run keeps its database in memory, and after a schedualed time flushes the dirty database files from memory to harddrives. The database files are stored as millions of tiny CDB binary files in 1 directory. I use RAID 0+1 on a promise raid card. When the server kicks off a flush and starts doing a large amount of IO, i get the following errors in dmesg: Dec 10 13:51:39 <kern.crit> bnet kernel: DOH! ata_alloc_request failed! Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request Dec 10 13:51:39 <kern.crit> bnet last message repeated 4 times Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE g- out of memory i_vn fsat_da_one(raid):_iarni0st_1dre[WquRIestTE(offset=2030 Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs Dec 10 13:51:39 <kern.crit> bnet kernel: _done()FA:aILr0UREs1 -d[ oWRIut oTEf (omeffmoseryt= 2in ata0329_rai03d_94init_request Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_rai62d_4, inleit_rengquthest=16384)]error = 5 Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203290411008, length=16384)]error = 5 Dec 10 13:51:39 <kern.crit> bnet kernel: Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_dFAoInLUe():RE - arou0st 1dof[ mWRemITory E(inoffs aet=t2a_raid0329_init0427392 Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE_re(oquffseest=t203290460160, length=16384)]error = 5 Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203290509312, length=16384)]error = 5 Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=20329057 Dec 10 13:51:39 <kern.crit> bnet kernel: 4848, lenFAILgtURh=E - 163ou84t )]of error = 5 Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203290787840, length=16384)]error = 5 Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203675779072, length=16384)]error me= mory5 in ata_r Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request Dec 10 13:51:39 <kern.crit> bnet kernel: F Dec 10 13:51:39 <kern.crit> bnet kernel: AILURE - out go_fv fmse_mdoornye (i)n: aart0as_1rda[iWdR_IiTnEi(to_frfesqeute=st Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory i2n0 3a8t6a8_5r2a0i4d4_8i,n ilte_nrgetqhu=est Dec 10 13:51:39 <kern.crit> bnet kernel: 16384)]error = 5 Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - ougt_ ovff mse_modroy nien (a)ta:_arra0isd1_di[nWiRtI_TrEe(qofufesestt= Dec 10 13:51:39 <kern.crit> bnet kernel: 203868536832, length=F16A3I84L)U]ReErr o-r =o u5t Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203 8o6f8 5m5e3m2o1r6y, ilne nagttah_=raid_1i6n3i8t Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memorry i=n 5a Dec 10 13:51:39 <kern.crit> bnet kernel: ta_raid_init_request Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=F2A04I0L61U2R78E2 0-8 ,o ulte nogft hm=emory 1in ata Dec 10 13:51:39 <kern.crit> bnet kernel: <snip> Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=12288)]error = 5 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=14336)]error = 5 Dec 10 13:52:02 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=14336)]error = 5 Dec 10 13:52:02 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=14336)]error = 5 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ <crash> Dec 10 13:53:24 <kern.crit> bnet kernel: Copyright (c) 1992-2005 The FreeBSD Project. Dec 10 13:53:24 <kern.crit> bnet kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Dec 10 13:53:24 <kern.crit> bnet kernel: The Regents of the University of California. All rights reserved. Dec 10 13:53:24 <kern.crit> bnet kernel: FreeBSD 6.0-RELEASE #0: Sat Dec 10 02:13:22 UTC 2005 Kernel: http://ender.tog.net/crash/BNET dmesg: http://ender.tog.net/crash/dmesg.today loader.conf: http://ender.tog.net/crash/loader.conf messages: http://ender.tog.net/crash/messages.0 This problem was also repeated on just RAID 0 array. Unfortunatally i do not have a kernel dump file at the moment due to the bnet# dumpon /dev/ar0 dumpon: ioctl(DIOCSKERNELDUMP): Operation not supported by device Limitation, but hopefully will have another disk in this server to get a proper kernel dump soon. >How-To-Repeat: Run a deamon that uses alot of disk IO on a raid array. >Fix: >Release-Note: >Audit-Trail: >Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200512101826.jBAIQ0Rl049891>