From owner-freebsd-hackers@freebsd.org Mon Nov 28 17:50:58 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C03CFC5A7F8 for ; Mon, 28 Nov 2016 17:50:58 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-yw0-x233.google.com (mail-yw0-x233.google.com [IPv6:2607:f8b0:4002:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7C5A8145B for ; Mon, 28 Nov 2016 17:50:58 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: by mail-yw0-x233.google.com with SMTP id a10so121313748ywa.3 for ; Mon, 28 Nov 2016 09:50:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=AQHvmY3P9wdjIO6cxnaA209bXcBsC0TCyQb0xwJAdKE=; b=m86M5m7ncNeKLbWWVidJsoO2JQObfZasFF0kTehWLIB4RQzvPiphadz3f0z4NCUO7K V+jaJBJwZCKP3kdW9imCLFeAIF9/ik7kx3AdBexCbacIEoaWqpd4td8VawvOOIhwf+W4 WETJtGGKknjHkewX364m5GIhyqQDrVb0zqudWj7rVOP2drYUkYLxVo89QmxI3Hf4TiTV DvbVEYRyHezFGIj2CR3AnK48X5ncKn7Ehwd4y1jTs4bU+NqOf6f6iNkWBLOIszgXvXO3 qxs3Mdg56dW8ShqCOMblvBwF7t/L8ToIIZboYK5cCTIvd4OTTax+DBRXNo4VMBod006S lFSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=AQHvmY3P9wdjIO6cxnaA209bXcBsC0TCyQb0xwJAdKE=; b=aZzsXYGwsaqAWuv5yCMRnzZTZaFzoBBIHAH30X2oh1PirCSV7CPlrOFkz06eIOriVH 0th7U65Z/vy7T11x8XUuO1r3Fv5mOBBUe+GcY7r6heQ1IeeVzejbuyfLEviOIFDEZraI ySuce+Ep2XsNzZHMutFFTFRfJt0JXoDHCGPslQrR7bgimWt0rqgAfeboBO2jDl0d82uo ADp4KHJR9W21gzcUUGiD3jErbuDiCjNqPp7TNecnKLaais6+3QoGUH2esJgD/ULjlO49 QNbVtCVdK9ije+113r18H6ZKz+duJNkEaky/sBZCTMEeuUQdJEDs97FWNCG3NZunoPJS M9mQ== X-Gm-Message-State: AKaTC03Leobzxdk+rinLIEHp1Pn6Q+TLQMeyNDm1HegD7RPdPm/pq0J5blHEBLSUc6fY1XLw5IcRX6rA5zjr3Q== X-Received: by 10.129.46.22 with SMTP id u22mr26670997ywu.199.1480355457624; Mon, 28 Nov 2016 09:50:57 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.201.197 with HTTP; Mon, 28 Nov 2016 09:50:57 -0800 (PST) In-Reply-To: <20161128162240.GM99742@zxy.spb.ru> References: <20161128041847.GA65249@charmander> <20161128120046.GP54029@kib.kiev.ua> <20161128144135.10f93205@fabiankeil.de> <20161128160311.GQ54029@kib.kiev.ua> <20161128162240.GM99742@zxy.spb.ru> From: David Cross Date: Mon, 28 Nov 2016 12:50:57 -0500 Message-ID: Subject: Re: FreeBSD 11 i386 disk deadlock (I think) (now with reproduction steps!) To: Slawa Olhovchenkov Cc: Konstantin Belousov , Fabian Keil , freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Nov 2016 17:50:58 -0000 I wouldn't call this a 'workaround', but the right answer. Something in the disk io path shouldn't be allocating memory out of the pool that can cause paging (since any of that could be IN the path for paging). It was what I assumed Fabian's proposed patch was. >From looking at the process list on my machine, it seems that geli allocates a process per core per provider, is there a reason to not have each of these on startup allocate themselves a single buffer of sector-size, and just put all operations through that? You're not (realistically) going to get more concurrency than that. I guess another approach would be to pre-allocate a ring buffer of the desired operational depth.. but that seems overkill. On Mon, Nov 28, 2016 at 11:22 AM, Slawa Olhovchenkov wrote: > On Mon, Nov 28, 2016 at 06:03:11PM +0200, Konstantin Belousov wrote: > > > On Mon, Nov 28, 2016 at 02:43:30PM +0100, Fabian Keil wrote: > > > David Cross wrote: > > > > > > > This is certainly new behavior, or a new manifestation. > > > > > > Recently a couple of uma consumers were changed to share uma zones > > > instead of using a dedicated zone. As a result geli competes with > > > more uma consumers and is more likely to deadlock. The bug isn't > > > new, it's just triggered more often now. > > The problem happens on layer much lower than UMA, it is whole reusable > > page pool which is depleted and cannot be re-filled without allocating > > more memory. If you think about it, the deadlock is obviously trivial: > > pagedaemon is the main source of the free pages, but if producing free > > page requires allocating one, low memory condition is equal to deadlock. > > > > It was always there, in the sense that for all versions of freebsd, if > > file/disk write path requires memory allocation, there is the trouble. > > > > For geom, some special unique measures were taken so that bio allocations > > do not cause the issue in typical situations. > > Typical workaround for this is pre-allocate some memory for this > operation. >