From owner-svn-src-stable@FreeBSD.ORG Sun Apr 10 15:40:57 2011 Return-Path: Delivered-To: svn-src-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CB84A1065678; Sun, 10 Apr 2011 15:40:57 +0000 (UTC) (envelope-from trociny@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 9CC708FC0A; Sun, 10 Apr 2011 15:40:57 +0000 (UTC) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id p3AFev5L052543; Sun, 10 Apr 2011 15:40:57 GMT (envelope-from trociny@svn.freebsd.org) Received: (from trociny@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id p3AFevAW052541; Sun, 10 Apr 2011 15:40:57 GMT (envelope-from trociny@svn.freebsd.org) Message-Id: <201104101540.p3AFevAW052541@svn.freebsd.org> From: Mikolaj Golub Date: Sun, 10 Apr 2011 15:40:57 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-8@freebsd.org X-SVN-Group: stable-8 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r220524 - stable/8/sys/geom/gate X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Apr 2011 15:40:57 -0000 Author: trociny Date: Sun Apr 10 15:40:57 2011 New Revision: 220524 URL: http://svn.freebsd.org/changeset/base/220524 Log: MFC r220264: r220264 (pjd): GEOM has an internal mechanism to deal with ENOMEM errors returned via g_io_deliver(). In such case it increases 'pace' counter on each ENOMEM and reschedules the request. The 'pace' counter is decreased for each request going down, but until 'pace' is greater than zero, GEOM will handle at most 10 requests per second. For GEOM GATE users that are proxy to local GEOM providers (like ggatel(8) and HAST) we can end up with almost permanent slow down of GEOM down queue. This is because once we reach GEOM GATE queue limit, we return ENOMEM to the GEOM. This means that we have, eg. 1024 I/O requests in the GEOM GATE queue. To make room in the queue and stop returning ENOMEM we need to proceed the requests of course, but those requests are handled by userland daemons that handle them by reading/writing also from/to local GEOM providers. For example with HAST, a new requests comes to /dev/hast/data, which is GEOM GATE provider. GEOM GATE passes the request to hastd(8) and hastd(8) reads/writes from/to /dev/da0. Once we reach GEOM GATE queue limit, to free up a slot in GEOM GATE queue, hastd(8) has to read/write from/to /dev/da0, but this request will also be very slow, because GEOM now slows down all the requests. We end up with full queue that we can unload at the speed of 10 requests per second. This simply looks like a deadlock. Fix it by allowing userland daemons that work with both GEOM GATE and local GEOM providers to specify unlimited queue size, so GEOM GATE will never return ENOMEM to the GEOM. Approved by: pjd (mentor) Modified: stable/8/sys/geom/gate/g_gate.c Directory Properties: stable/8/sys/ (props changed) stable/8/sys/amd64/include/xen/ (props changed) stable/8/sys/cddl/contrib/opensolaris/ (props changed) stable/8/sys/contrib/dev/acpica/ (props changed) stable/8/sys/contrib/pf/ (props changed) Modified: stable/8/sys/geom/gate/g_gate.c ============================================================================== --- stable/8/sys/geom/gate/g_gate.c Sun Apr 10 15:28:37 2011 (r220523) +++ stable/8/sys/geom/gate/g_gate.c Sun Apr 10 15:40:57 2011 (r220524) @@ -192,7 +192,7 @@ g_gate_start(struct bio *bp) } mtx_lock(&sc->sc_queue_mtx); - if (sc->sc_queue_count > sc->sc_queue_size) { + if (sc->sc_queue_size > 0 && sc->sc_queue_count > sc->sc_queue_size) { mtx_unlock(&sc->sc_queue_mtx); G_GATE_LOGREQ(1, bp, "Queue full, request canceled."); g_io_deliver(bp, ENOMEM);