Date: Tue, 2 Dec 2003 13:39:50 -0800 (PST) From: Nick Johnson <freebsd@spatula.net> To: Brian Behlendorf <brian@hyperreal.org> Cc: java@freebsd.org Subject: Re: Running server-side Java on FreeBSD in production environments Message-ID: <20031202131725.S24955@turing.morons.org> In-Reply-To: <20031202130629.U56021@fez.hyperreal.org> References: <20031202130629.U56021@fez.hyperreal.org>
next in thread | previous in thread | raw e-mail | index | archive | help
I've actually been running my site with FreeBSD Java since July of 2002, first with the green-threads JDK 1.3 with OpenJIT, then with the native threads 1.3 with Hotspot, and now with 1.4.2-p5. My observation is that for server-side work (which is the only thing I've tested) I run into more problems because of MY code than because of the JVM. I haven't experienced any kind of crash of the JVM since 1.4.1-p4. The fork fixes for p5 have improved stability quite a bit in my environment, or at least I get the impression they have. I used to experience Resin occasionally going apeshit; that hasn't happened since I went to 1.4.2-p5. I cannot stress enough the need of making sure your code is sound before you run it in production. You CAN bring down a servlet container with the code you run in it if you're not careful. The easiest way to do that is to keep too many things in session or application scope or otherwise manage to allocate large objects with long lifespans that use up all your available heap. (voice of experience) I also suggest running the JVM for a servlet container with these options: -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC The -server flag tends to make server startup and class reloading take a bit longer as I think it makes Hotspot a bit more aggressive about what it compiles ahead of time. The other flags I got from a Sun paper on garbage collection: http://java.sun.com/docs/hotspot/gc1.4.2/ Using incremental/low-pause garbage collection is pretty important for a servlet container, imo; I'd generally rather add a few ms to each request than have requests take 10-15 seconds occasionally when the JVM needs to do GC. One sorta hokey/kludgy thing I do as well is to perform a nightly restart of all my JVMs. I don't know if I still need to do that with 1.4.2-p5. The main reason I've been doing it at all is that memory seems to get fragmented or possibly some objects are just never getting garbage collected, so the JVM processes were using more CPU and memory as time went by. I haven't verified whether this is still problematic in p5. I definitely recommend arranging servlet containers into a primary/backup JVM situation if your container allows it. I also suggest load balancing across two containers for performance (it seems like 2 containers handle load better under high-load condition than one container with twice as many threads, which makes sense if you think about how threads and processes differ). My arrangement has 3 containers; two load-balanced primary JVMs and one backup JVM. When restarting your containers and when starting them the first time, I suggest staggering the loading because of the amount of CPU the -server switch causes the JVM to use as it starts. I start and restart mine with a 60-second sleep interval between them. Otherwise, you end up thrashing during startup, which can actually cause startup to fail because, in the case of Resin anyway, the minder process can't contact the servlet container because it's still loading because the CPU is pegged trying to start up many containers at once. Another thing I recommend is writing a "ping" process for your containers. I have one I wrote for Resin that checks each minute to see that Resin's "srun" service is working and is capable of loading a page (including configurable timeouts, retries, etc). If there's no answer, it automatically restarts the container and sends me an email. Though it's quite rare that this actually does any work, it does happen periodically and it's nice to have things restarted for me in the middle of the night ;) That's all I can think of at the moment. Let me know if you'd like more detail about anything in particular. I'm currently running morons.org, ln-s.net, and go2rider.org with the FreeBSD JVM. Nick On Tue, 2 Dec 2003, Brian Behlendorf wrote: > > So, I've seen a few mentions here of folks using Java server apps on > FreeBSD in production. I've been very happy to watch the progress by the > Java team on the native JDKs from the sidelines, as well as the > improvements to libc_r and such that have been MFC'd, but I'm curious as > to whether the consensus on this list is that it's ready to be used for > production. Anyone want to share good success or horror stories? We're > trying to determine whether the apache.org box can start to run our own > software. :) Speed is definitely less important than proper functioning. > > Brian > _______________________________________________ > freebsd-java@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-java > To unsubscribe, send any mail to "freebsd-java-unsubscribe@freebsd.org" > -- "The aptly-named morons.org is an obscenity-laced screed..." -- Robert P. Lockwood, Catholic League director of research Nick Johnson, version 2.1 http://web.morons.org/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20031202131725.S24955>