From owner-freebsd-amd64@FreeBSD.ORG Tue Oct 21 08:10:01 2008 Return-Path: Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A242A106567E for ; Tue, 21 Oct 2008 08:10:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7CFBB8FC1C for ; Tue, 21 Oct 2008 08:10:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id m9L8A1VW027198 for ; Tue, 21 Oct 2008 08:10:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id m9L8A1q5027197; Tue, 21 Oct 2008 08:10:01 GMT (envelope-from gnats) Resent-Date: Tue, 21 Oct 2008 08:10:01 GMT Resent-Message-Id: <200810210810.m9L8A1q5027197@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-amd64@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Martin W Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A4BC4106566B for ; Tue, 21 Oct 2008 08:09:33 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 91BDC8FC18 for ; Tue, 21 Oct 2008 08:09:33 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id m9L89W9m033908 for ; Tue, 21 Oct 2008 08:09:32 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id m9L89W2c033907; Tue, 21 Oct 2008 08:09:32 GMT (envelope-from nobody) Message-Id: <200810210809.m9L89W2c033907@www.freebsd.org> Date: Tue, 21 Oct 2008 08:09:32 GMT From: Martin W To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 X-Mailman-Approved-At: Tue, 21 Oct 2008 11:23:42 +0000 Cc: Subject: amd64/128263: 2 amd64 dl380 g5 with dual quadcore xeons, 8 and 16gb ram, crash and dump mem X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Oct 2008 08:10:01 -0000 >Number: 128263 >Category: amd64 >Synopsis: 2 amd64 dl380 g5 with dual quadcore xeons, 8 and 16gb ram, crash and dump mem >Confidential: no >Severity: serious >Priority: high >Responsible: freebsd-amd64 >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Oct 21 08:10:01 UTC 2008 >Closed-Date: >Last-Modified: >Originator: Martin W >Release: 7.0 amd64 >Organization: Cypoint >Environment: FreeBSD db04 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sun Feb 24 10:35:36 UTC 2008 root@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 >Description: Two HP DL 380 G5's with dual quadcore xeons, 8GB ram on one and 16GB ram on the other. They have recently been crashing and dumping mem. The 16GB one has crashed 3 times the last 1 1/2 month, twice yesterday. The 8GB one had its first crash yesterday. They are db-servers which run postgresql or mysql. They are both Running 7.0 amd64 RELEASE with GENERIC kernel, completely default installs except for some sysctl tweaks: kern.ipc.maxsockets=16424 kern.maxfiles=65536 kern.maxfilesperproc=32768 net.inet.tcp.recvspace=32768 net.inet.tcp.sendspace=65536 Below is the output from kgdb, which looks about the same on both of them. >How-To-Repeat: Random, so don't know. >Fix: >Release-Note: >Audit-Trail: >Unformatted: >kgdb /usr/obj/usr/src/sys/GENERIC/kernel.debug /var/crash/vmcore.0 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd". Unread portion of the kernel message buffer: <2>NMI ISA b0, EISA ff <2>NMaINNr MMNNiNNItMMIIM M yIII ISeSr AIr ISS IA IbASIAbS Ao b0r0, I0Sb,0Ab ,0l,i k,A Ee ,IEE ElIybS IhAIaSS SE A0rI b,AAf 0dfwfS f E,aff< f 2< > R2 <>p <<><22<><22>>a2rA>> ifRI t2ARSM>yEAa fIff A Mi< 2p< <>S2 < e r<> R222rflof>>uAfa Mr ,prr >e f<

>2il>yk ree2rlrMor,i 2 tlyiRp>k>eaARl yh AryhaMa rirtadd wewyaapM rrarrer rei eoirf aftialrriupor,lye .ula triyek.elrr yi h,at rdyl eweierrarrrrkere ofaoiollrur,r e,y. r ,llh aliirkdikkeeelllwyyya rhhehaaa rrfdrdawdawirlaurrweaere e .fa i fflaauiirleu.lreu.re. F ata FFFaaa ttlaFl t alt arltt atrra apa tFprp aFtat a aa 1Flaa ttall ltrrt aapt p11prr99aa1p9::9 :n :o nn-o1nm9np- 11 o9n:m -nmaasaosnk-sakkabmlbeal baei snitkea1n99rblrlnee utpeitr r::inu o: tnttrenraprnno-puet prt m ansn rturwooahtpnk-ambtl-naimlper atpr a-im pwea sass kkiah wnhwihlkei bnatlaeki leelrei rn n er iekbilnetle brmeonuirpd nuterkn ecle rinpltept tlmrtereruo aian inptreuddpm e k=eo d ecr2t pn;tp euwilc da prr awrmpuiho=idiuheipltpl cee d ti wrni ak ehr=c1ipd u0i;= i;a pni 0 cd 2 = inlepl kw amioee 3dnps t=;i rhua pidnicertc c0nk i0iele oi enc priidi du inndellp o==i nnts0t 1r0 ei3nru kcet s =t rm=moonddee iior0nnx8 :u0cxpe l 5;sociftnm o itfetrrauofcf tfif=cdppuepf o0fnnxui i88 ppo0:o0iiddc =i7nx0nftfeftref 8f2=f fsft a0f =6ixrc8 :=0fcdp 8 0x7f f04f0fkxf ffp88f=; 2:fof08xi0nt70u;i0e rfs faa5piff cdp i=tf8a cf 2ff k i cn is it7sr uddc ;tf fa8 cp 0 o==t i ki n=t7 0ep0ofxrif nao t1 0e 8r2 : s0 xtp faf f c k =004i p o6nfi0fxfnc f1ftae=b0r : 0 x7f0fx 1 0 pi nisnt:9f 0fx isdr f=u f f5 fff=0 ff00ofxtircuntct7eiro ni nt=i p0oxs88 f0f1rfn:afaf7tof0a:mirn0tu cxeptrof0bex7f3f 98f0pffoffifnif oiff0tf 0enntfrr a=m e f a bp7 9 aoff irnfat0m0efxr e8 ep:=o r p=0fof r 0i anxf1 t 0m: efifn8e0 xt0 700xff8f:f0fxeffffrx pf 8f2f=f ff forf fi sff=f n0t exr0 010 ftf001 : 2 afc8k800 0 2xdf7a 0 0 0f = cf7xf8of 0d0 fxfpf:o08f18x f0 0e0=0: 01xs2 fef2f0fxgdf2f022 i0ms ne1000ft n:ct0otsa1cftfafcfke fk 20d rpx=e 2bfdasse f f 8poo0 7 ie6 g00x0finn0 ttmefefrr80f,e nlc i tmo d ef= i=t sf f0e0xgfb00fma0f1fs2 x21 6ee3 0 :=s0fx 0ftx =f1an,0 0cf0kf : fxt0f000pxfffxfafbf1f00t7:ofy p c=e,o 0lax0ifnxi1 bbamdiste e0 x f s fetfff gfm, ef=0anxfebftrf0 t ,y D=P Ll ibp04 feff7fa 9ff0f a m0,sifat ex0 b 1 pfr 7 a=m e0fxr190 :br es00x x af01f0f,,fm fex ffpl iofmflfoi t,= n ptg 0yx pfDefo0fifnf1f f,f PiLd0 xftnfearb 7 a 1fe0fb,e f 3,2 p t0yfft0 e pr , fr ee s g r= rra mae m 1 0aepDxPnL, o 1 bl1o0ng 1 ,, dp perrpo=f 3ceDsPeLs oi=in n 0tt xe1er=0 : 0 r 0 xx1 2s, loo0 f1 f 0 f f:n0,gr 0 1,p,r ege fd l fx = f=fraesagnfs f03 0x0f01x0f12, = 1lo n pir gonc0t0 ,1f1:02:e e1rgrsurpa,nts o r1 edeef f0flpnaarbgo3l2cx6003xff0 eess0d,8ff0ff0f 0f1= ,s i ogrfc0fnI rtaee2o6r03dcOfnl00e0 sr 1faf 1cugPsp tp Le n= r2ooe0c0g603a=e b0 slsemidonr,t e rre fIlOaPgLu s= =0 pitcn d4cut rere1e0netn0re 2ruarnrbte =s ecgbm6lu4epnt aosdeen2pter 0 0 od ,tc ep reosIncOe0s=Psasbsl e Le g=mc= oxbean=d ,1 186I O0(diP(Li c=udr drlel0ets0ee,n:t l0ix=0 cs ,m elibpceuu r2:r)pe raocscpe sunst0t)t r p r otcr=e isasa 1 7e g m0epp =nxn0t(miui,t mn1dub l i0 m=emrlb eer5 :=( i d=c ix1199pl e :fp uac1p)nui0 3ct)r:a p ft xnn ub mtobnr0e-ra mfaffxf pa,s =nks u1a9 bl temeb eirn t fyf0f=e pf1r9rxfef0f f u,pf, t t0lytx,ip1em i b0txrt 1aybpp0ex f f 0c xfp1fu f ib ,= =dt y=Dp P e2 L =0 D0P DLx,1U P 0pbp,Lrt 0iem ,psre :1p r ,7 e dl=so 1n51Dg,e hPs1L 2 ,0 ml 04o1,n ,gd1 pelsr1o,f eng 3d s2 e11 ,f,0 ,l og ndr3ega2 n f01 3,,1 2 dpegfr r30a2o ,n0c,e s ggs orrr1a ae nfpn rl 1ao1cge ps spr sro=ooc rcei nseetfsslseooarrgrr u esepfft l= la egianngatssbe l== er dirinun,t epIrttO ePreLunrarbup lptt=e eed0n,n a baIlbcluOeerPddr,,Le n tII= OO pPP0LLr == o c c00ue s rcsrcu eu nr=trr re 1e2nnt p t(p rirpodorclcoeeessscs e: s =c p s1u=63 ) ( 1=i d4t 1lr(1iae dpl(:e in d:uclm pbeuec:5p r)u c 4p =) tu1r 97)t arp at prn aupnm unbmuembrbee rr = = = 11 991P9 h y sical memory: 8178 MB Dumping 577 MB: 562 546 530 514 498 482 466 450 434 418 402 386 370 354 338 322 306 290 274 258 242 226 210 194 178 162 146 130 114 98 82 66 50 34 18 2 #0 doadump () at pcpu.h:194 194 pcpu.h: No such file or directory. in pcpu.h (kgdb) backtrace #0 doadump () at pcpu.h:194 #1 0x0000000000000004 in ?? () #2 0xffffffff80477699 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #3 0xffffffff80477a9d in panic (fmt=0x104

) at /usr/src/sys/kern/kern_shutdown.c:563 #4 0xffffffff8072ec94 in trap_fatal (frame=0xffffff00010f3000, eva=18446742974215641296) at /usr/src/sys/amd64/amd64/trap.c:724 #5 0xffffffff8072f7e5 in trap (frame=0xffffffffab795f40) at /usr/src/sys/amd64/amd64/trap.c:526 #6 0xffffffff8071596b in nmi_calltrap () at /usr/src/sys/amd64/amd64/exception.S:387 #7 0xffffffff8070ff82 in acpi_cpu_c1 () at /usr/src/sys/amd64/acpica/acpi_machdep.c:68 #8 0xffffffff801ca7ec in acpi_cpu_idle () at /usr/src/sys/dev/acpica/acpi_cpu.c:939 #9 0xffffffff80495028 in sched_idletd (dummy=Variable "dummy" is not available. ) at /usr/src/sys/kern/sched_4bsd.c:1377 #10 0xffffffff80458db3 in fork_exit (callout=0xffffffff80494f90 , arg=0x0, frame=0xffffffffac284c80) at /usr/src/sys/kern/kern_fork.c:781 #11 0xffffffff807159de in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:415 #12 0x0000000000000000 in ?? () #13 0x0000000000000000 in ?? () #14 0x0000000000000001 in ?? () #15 0x0000000000000000 in ?? () #16 0x0000000000000000 in ?? () #17 0x0000000000000000 in ?? () #18 0x0000000000000000 in ?? () #19 0x0000000000000000 in ?? () #20 0x0000000000000000 in ?? () #21 0x0000000000000000 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000000000000 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000cbd000 in ?? () #37 0xffffffff80494f90 in critical_enter () at kern_switch.c:167 #38 0x00000000fffffffb in ?? () #39 0xffffff00010e58d0 in ?? () #40 0xffffff0001105680 in ?? () #41 0xffffff00010f3000 in ?? () #42 0xffffffffac284b98 in ?? () #43 0xffffff00010f3000 in ?? () #44 0xffffffff80495359 in sched_switch (td=0x0, newtd=0xffffffff80494f90, flags=0) at /usr/src/sys/kern/sched_4bsd.c:905 #45 0x0000000000000000 in ?? () #46 0x0000000000000000 in ?? () #47 0x0000000000000000 in ?? () #48 0x0000000000000000 in ?? () #49 0x0000000000000000 in ?? () #50 0x0000000000000000 in ?? () #51 0x0000000000000000 in ?? () #52 0x0000000000000000 in ?? () #53 0x0000000000000000 in ?? () ---Type to continue, or q to quit--- #54 0x0000000000000000 in ?? () #55 0x0000000000000000 in ?? () #56 0x0000000000000000 in ?? () #57 0x0000000000000000 in ?? () #58 0x0000000000000000 in ?? () #59 0x0000000000000000 in ?? () #60 0x0000000000000000 in ?? () #61 0x0000000000000000 in ?? () #62 0x0000000000000000 in ?? () #63 0x0000000000000000 in ?? () #64 0x0000000000000000 in ?? () #65 0x0000000000000000 in ?? () #66 0x0000000000000000 in ?? () #67 0x0000000000000000 in ?? () #68 0x0000000000000000 in ?? () #69 0x0000000000000000 in ?? () #70 0x0000000000000000 in ?? () #71 0x0000000000000000 in ?? () #72 0x0000000000000000 in ?? () #73 0x0000000000000000 in ?? () #74 0x0000000000000000 in ?? () #75 0x0000000000000000 in ?? () #76 0x0000000000000000 in ?? () #77 0x0000000000000000 in ?? () #78 0x0000000000000000 in ?? () #79 0x0000000000000000 in ?? () #80 0x0000000000000000 in ?? () #81 0x0000000000000000 in ?? () #82 0x0000000000000000 in ?? () #83 0x0000000000000000 in ?? () #84 0x0000000000000000 in ?? () #85 0x0000000000000000 in ?? () #86 0x0000000000000000 in ?? () #87 0x0000000000000000 in ?? () #88 0x0000000000000000 in ?? () #89 0x0000000000000000 in ?? () #90 0x0000000000000000 in ?? () #91 0x0000000000000000 in ?? () #92 0x0000000000000000 in ?? () #93 0x0000000000000000 in ?? () #94 0x0000000000000000 in ?? () #95 0x0000000000000000 in ?? () #96 0x0000000000000000 in ?? () #97 0x0000000000000000 in ?? () #98 0x0000000000000000 in ?? () #99 0x0000000000000000 in ?? () #100 0x0000000000000000 in ?? () #101 0x0000000000000000 in ?? () #102 0x0000000000000000 in ?? () #103 0x0000000000000000 in ?? () #104 0x0000000000000000 in ?? () #105 0x0000000000000000 in ?? () #106 0x0000000000000000 in ?? () #107 0x0000000000000000 in ?? () ---Type to continue, or q to quit--- #108 0x0000000000000000 in ?? () #109 0x0000000000000000 in ?? () #110 0x0000000000000000 in ?? () #111 0x0000000000000000 in ?? () #112 0x0000000000000000 in ?? () #113 0x0000000000000000 in ?? () #114 0x0000000000000000 in ?? () #115 0x0000000000000000 in ?? () #116 0x0000000000000000 in ?? () Cannot access memory at address 0xffffffffac285000