Date: Thu, 29 Mar 2012 11:43:45 -0500 From: Jim Bryant <kc5vdj.freebsd@gmail.com> To: Mark Felder <feld@feld.me> Cc: freebsd-hackers@freebsd.org, freebsd-questions@freebsd.org Subject: Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash Message-ID: <4F749141.8010109@gmail.com> In-Reply-To: <op.wbwe9s0k34t2sn@tech304> References: <op.wbwe9s0k34t2sn@tech304>
next in thread | previous in thread | raw e-mail | index | archive | help
This sounds just like a race condition that happens under Windows 7 on this laptop. The race condition, as far as I can tell involves heavy disk access and heavy network access, and usually leaves the drive light on, while all activity monitors (alldisk, allcpu, allnetwork) are still active, although on this laptop disk takes priority, and network slows to a crawl. occasionally, the mouse will stop working, along with everything else, but usually not. keyboard is lower priority, and doesn't do anything. You might want to check with mickeysoft, this might just be their problem. This sounds so freaking similar to the issue I get, and I think it's a race condition (shared interrupts??). This laptop is a Compaq Presario C300 series, with the 945GM chipset and a T7600 Core2 Duo CPU, with 3G of RAM. Mark Felder wrote: > Alright guys, I'm at the end of my rope here. For those that haven't > seen my previous emails here's the (not so) quick breakdown: > > Overview: > > FreeBSD ?? - 7.4 never crash > FreeBSD 8.0 - 8.2 crashes > FreeBSD 8-STABLE, 8.3, and 9.0 are untested (Sorry, not possible in > our production at this time, and we were hoping we could base some > stuff on 8.3 for long term stability...) > ESXi: Confirmed ESXi 4.0 - 5.0 has this problem. Haven't tested on > others. > > > History: > > Over the course of the last 2 years we've been banging our heads on > the wall. VMWare is done debugging this. They claim it's not a VMWare > issue. They can't identify what the heck happens. We had a glimmer of > hope with ESXi 5.0 fixing it because we never saw any crashes in the > handful of deployments, but our dreams were crushed today -- two days > before an outage to begin migration to ESXi 5.0 -- when a customer's > ESXi 5.0 server and FreeBSD 8.2 guest crashed. > > > Crash Details: > > The keyboard/mouse usually stops responding for input on the console; > normally we can't type in a username or password. However, we can > switch VTs. > > If there's a shell on the console and we can type, we can only run > things in memory. Any time we try to access the disk it will hang > indefinitely. > > The server still has network access. We can ping it without issue. SSH > of course kicks you out because it can't do any I/O. > > If we were to serve a lightweight http server off a memory backed > filesystem I'm confident it would run just fine as long as it wasn't > logging or anything. > > On ESXi you see that there is a CPU spike of 100% that goes on > indefinitely. No idea what the FreeBSD OS itself thinks it is doing > because we can't run top during the crash. > > This crash can affect a server and happen multiple times a week. It > can also not show up for 180 days or more. But it does happen. The > server can be 100% idle and crash. We have servers that do more I/O > than the ones that crash could ever attempt to do and these don't > crash at all. Completely inexplicable. > > > Things we've looked into: > > Nothing about the installed software matters. We've tried cross > referencing the crashed servers by the programs they run but the base > OS is the only common denominator due to the wide variety of servers > it has affected. > > Storage doesn't matter. We've tried different iSCSI SANs, we've tried > different switches, we've tried local datastores on the ESXi servers > themselves. > > HP servers, Dell servers -- doesn't seem to matter either. (All with > latest firmwares, BIOSes, etc) > > VMWare gave us a ton of debugging tasks, and we've given them > gigabytes of debugging info and data; they can't find anything. > > VMWare tools -- with, without, using open-vm-tools makes no > difference. I think we've done a fair job ruling out VMWare. > > > I think we've finally found enough data that this is definitely > something in the FreeBSD world. I'm going to begin prepping some of > the known crashy servers with more debugging. Any suggestions on what > I should build the kernel with? They never do a proper panic, but I > definitely want to at least *try* to get into the debugger the next > time it crashes. And when it crashes, what the heck should I be > running? I've never played with the KDB before... > > > Thank you for any suggestions and help you can give me.... > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F749141.8010109>