Date: Sat, 19 Mar 2005 02:13:48 +0000 From: "Youlin Feng" <youlinfeng@hotmail.com> To: freebsd-net@freebsd.org Subject: Lesson learned from working with slow HW crypto card Message-ID: <BAY101-F18F7CA6F160E49261AD8A2CD4B0@phx.gbl>
next in thread | raw e-mail | index | archive | help
This is a lesson I learned from working with a slow HW crypto card in the presence of the GbE interface. Hope it is helpful to others when they run into the same issue. We are using a 200Mpps HW crypto card on a FreeBSD 5.3 system as a GbE gateway. With ~700Mbps traffic inbound for encryption, the system dropped into a livelock state and the throughput is barely ~50Mbps. The console doesn't respond to key strokes and the system complains about the lack of memory. The problem turns out to be related to the fact that the crypto request queue doesn't have a limit on the queue length. Due to the async nature of HW crypto processing and the speed mismatch between the crypto card and the GbE interface, the request queue becomes so long that CPU will be pretty much monopolized by the crypto_proc thread running at SWI_NET pri level, leaving no cycles for lower pri kernel tasks and applications. Because the crypto callback thread can only run as fast as the HW crypto speed, memory resources are freed at a much slower rate than when they were allocated, leading to out-of-memory condition. The workaround I used is to cap the crypto request queue, ie crp_q. Capping it to 64 works fine with the 200Mbps crypto card, increasing the throughput to ~200Mbps. And livelock condition is also gone. The asymmetric op queue, ie crp_kq, can be left as unlimited because there are much fewer such requests. The limited request queue shouldn't have any impact on the SW crypto since it is a synchronous operation.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BAY101-F18F7CA6F160E49261AD8A2CD4B0>