From owner-freebsd-hackers@freebsd.org Sat Mar 7 11:42:25 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2AE5E262958 for ; Sat, 7 Mar 2020 11:42:25 +0000 (UTC) (envelope-from kib@freebsd.org) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48ZMzM5vjqz455W for ; Sat, 7 Mar 2020 11:42:23 +0000 (UTC) (envelope-from kib@freebsd.org) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id 027BgEo3045292 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 7 Mar 2020 13:42:17 +0200 (EET) (envelope-from kib@freebsd.org) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 027BgEo3045292 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id 027BgERS045291; Sat, 7 Mar 2020 13:42:14 +0200 (EET) (envelope-from kib@freebsd.org) X-Authentication-Warning: tom.home: kostik set sender to kib@freebsd.org using -f Date: Sat, 7 Mar 2020 13:42:14 +0200 From: Konstantin Belousov To: Keno Fischer Cc: freebsd-hackers@freebsd.org, Elliot Saba Subject: Re: FreeBSD Pipe behavior in pipe OOM situations Message-ID: <20200307114214.GH98340@kib.kiev.ua> References: <20200304233906.GB98340@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 48ZMzM5vjqz455W X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_SPAM_MEDIUM(0.97)[0.968,0]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; NEURAL_HAM_LONG(-0.97)[-0.971,0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Mar 2020 11:42:25 -0000 On Sat, Mar 07, 2020 at 12:33:02AM -0500, Keno Fischer wrote: > Hi Konstantin, > > thanks for getting back to me, > > First, there is a requirement that an atomic write size exists, i.e. writes > > less than SC_PIPE_BUF are guaranteed to not interleave if succeeded. Our > > PIPE_BUF is 512 bytes. > > > > Useful to know, thanks. > > > > I think that unexplained blocking (it is very hard to track down such > > state) is worse then ENOMEM outcome. > > > > This is probably true, but the ENOMEM behavior is by no means benign. > If a process exhausts all pipe kva, the next process trying to allocate and > write to a pipe will probably crash. That could basically be anything in the > system. From the ssh server to (in our case) the infrastructure that runs > jobs. > Of course arguable the same thing happens in a regular ENOMEM situation > also, but between paging and user space monitoring of memory usage, such > situations seem easier to manage. I guess there may also be a concern about > dos-style attacks. It seems pretty easy to allocate enough pipes to exhaust > that limit. I do not disagree, but we do not have a good way out if the pipe KVA is exhausted. I can propose a different approach to it: allocate at least the minimal sized buffer on pipe creation, and return ENOMEM if allocation failed. Then userspace can get ENOMEM on e.g. pipe(2) or open of the named fifo, but if the pipe creation failed, then write(2) is guaranteed to not return with ENOMEM. Prototype of the change is at https://reviews.freebsd.org/D23993, I did not yet even booted with it. > > Regardless, I just wanted to raise this, since I considered the behavior > odd and > we didn't see it elsewhere. We have since found the culprit for the OOM > condition: > One of our tests tries to provoke an EMFILE condition to test our handling > of this > corner case, so it just fills every unallocated fd with pipes. However, > since it doesn't > write to them, we never actually see a failure there. Instead some random > other process > in the system will crash. Sometimes another test run (where we saw the > error), but > occasionally also the CI system itself. I suspect this is responsible for a > fair number > of mysterious failures we observed. From our perspective this issue should > be resolved - > I guess I'll leave it to you to decide whether there's anything to be done > about the denial > of service concern. I don't know what guarantees FreeBSD makes for kernel > resource > usage particularly in the context of jails, so I don't know if this is of > concern at all. In > regular usage, without a malicious program (or > well-it's-malicious-but-it's-a-test-script), > the admin probably would just bump the sysctl and everything would keep > running nicely. With that change in place you are more likely to get ENOMEM than EMFILE. > > Thanks again for your detailed reply. > > Keno