From owner-freebsd-net@FreeBSD.ORG Sat Mar 19 02:13:49 2005 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AE76D16A4CE for ; Sat, 19 Mar 2005 02:13:49 +0000 (GMT) Received: from hotmail.com (bay101-f18.bay101.hotmail.com [64.4.56.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5AEA543D2F for ; Sat, 19 Mar 2005 02:13:49 +0000 (GMT) (envelope-from youlinfeng@hotmail.com) Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Fri, 18 Mar 2005 18:13:49 -0800 Message-ID: Received: from 64.4.56.201 by by101fd.bay101.hotmail.msn.com with HTTP; Sat, 19 Mar 2005 02:13:48 GMT X-Originating-IP: [64.4.56.201] X-Originating-Email: [youlinfeng@hotmail.com] X-Sender: youlinfeng@hotmail.com From: "Youlin Feng" To: freebsd-net@freebsd.org Date: Sat, 19 Mar 2005 02:13:48 +0000 X-OriginalArrivalTime: 19 Mar 2005 02:13:49.0282 (UTC) FILETIME=[49C47420:01C52C29] MIME-Version: 1.0 Content-Type: text/plain; format="flowed" X-Content-Filtered-By: Mailman/MimeDel 2.1.1 Subject: Lesson learned from working with slow HW crypto card X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Mar 2005 02:13:49 -0000 This is a lesson I learned from working with a slow HW crypto card in the presence of the GbE interface. Hope it is helpful to others when they run into the same issue. We are using a 200Mpps HW crypto card on a FreeBSD 5.3 system as a GbE gateway. With ~700Mbps traffic inbound for encryption, the system dropped into a livelock state and the throughput is barely ~50Mbps. The console doesn't respond to key strokes and the system complains about the lack of memory. The problem turns out to be related to the fact that the crypto request queue doesn't have a limit on the queue length. Due to the async nature of HW crypto processing and the speed mismatch between the crypto card and the GbE interface, the request queue becomes so long that CPU will be pretty much monopolized by the crypto_proc thread running at SWI_NET pri level, leaving no cycles for lower pri kernel tasks and applications. Because the crypto callback thread can only run as fast as the HW crypto speed, memory resources are freed at a much slower rate than when they were allocated, leading to out-of-memory condition. The workaround I used is to cap the crypto request queue, ie crp_q. Capping it to 64 works fine with the 200Mbps crypto card, increasing the throughput to ~200Mbps. And livelock condition is also gone. The asymmetric op queue, ie crp_kq, can be left as unlimited because there are much fewer such requests. The limited request queue shouldn't have any impact on the SW crypto since it is a synchronous operation.