From owner-freebsd-hackers@freebsd.org Sat Mar 7 05:33:42 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5152325BD29 for ; Sat, 7 Mar 2020 05:33:42 +0000 (UTC) (envelope-from keno@juliacomputing.com) Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48ZCnw5Ysbz3P8c for ; Sat, 7 Mar 2020 05:33:40 +0000 (UTC) (envelope-from keno@juliacomputing.com) Received: by mail-io1-xd36.google.com with SMTP id r15so4260859iog.0 for ; Fri, 06 Mar 2020 21:33:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=juliacomputing-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=xzb6DzfVJJ/BlLl2W/P+AjQD+rB0jCaZeEszsIY29Ro=; b=oxPGlfwkQPOaE+vmaz/e/DiEapR0RYG5A6Njms11f+6IAWKvZdLoe5hamyGHz+BORf Nh+JVGgu9V48u0RehIrNSiFghkXol4shWxLrZeZ3BcbZfo5y4vzbwtaASzhXoTTPMXZ7 cUvx8PXMKAoD+bScFVyzkVlV2QqHJ1cPJUCpR2Lotk998nEojyzlOFM+4D7rwU25KSzF nWxZNPsDmttzYuO7E/JqLIrj+v/NTX2w/PRJ59x0T+aPuif4sJW8FBxR4AloRI3bO7gf L2yY9hLGOQXaldhf9zSaimswNRyW/nU9k4ziF9jE2WVGnEh+0WG9UW4yiFsvFbXm1QJU xqRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=xzb6DzfVJJ/BlLl2W/P+AjQD+rB0jCaZeEszsIY29Ro=; b=piCC2AD7rNfQ50c5Lz1PNBs6UVj6y6EVPiJnhk3O1gPLG4scGSUg9yEiiVHIeScngq 9LvtcU7E3oWfAOBO7qDxywcJgiAm3NATtwM01yF4gcPiV5YsKaq6SUORscx5cHNN5SWk D1SMzCwmtQ0NN6e3gC+5esKvy3ATSF6i9oFtmOMnLXuXYhuzwVheBKBsI0CvKPNgjjeo /AYBI/w+cwicDxCyzn4s4/MmQ1sc4z4xsng/xEhk9i0/g2PByRn01XILVV8TicOIp4BD lk7Jm4IEidfJYcHJVocRdvpOspO6439xRKJC4N1ITu+dXpYku8Gq4pfXvIwVJK2c3781 rEwQ== X-Gm-Message-State: ANhLgQ2xQlzusD1pnu06RYMba3qBUyTRk3LsIXzT+wkxWGuWrcZDrNO3 XtdQ8ndgm5f7WkR+OiNIsYPWFwJ29TnzwkrmnrPi/A== X-Google-Smtp-Source: ADFU+vsYR77FxUR39DLZYJUI6Fr+liDFCIQ7Q0AyJ8X55uowBZ2/TrN2617Li6f+h7WMRXu5lSrNwgbEceiRvZkEauU= X-Received: by 2002:a5e:8f45:: with SMTP id x5mr5743663iop.19.1583559218192; Fri, 06 Mar 2020 21:33:38 -0800 (PST) MIME-Version: 1.0 References: <20200304233906.GB98340@kib.kiev.ua> In-Reply-To: <20200304233906.GB98340@kib.kiev.ua> From: Keno Fischer Date: Sat, 7 Mar 2020 00:33:02 -0500 Message-ID: Subject: Re: FreeBSD Pipe behavior in pipe OOM situations To: Konstantin Belousov Cc: freebsd-hackers@freebsd.org, Elliot Saba X-Rspamd-Queue-Id: 48ZCnw5Ysbz3P8c X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=juliacomputing-com.20150623.gappssmtp.com header.s=20150623 header.b=oxPGlfwk; dmarc=none; spf=none (mx1.freebsd.org: domain of keno@juliacomputing.com has no SPF policy when checking 2607:f8b0:4864:20::d36) smtp.mailfrom=keno@juliacomputing.com X-Spamd-Result: default: False [-4.30 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[juliacomputing-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[juliacomputing.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[juliacomputing-com.20150623.gappssmtp.com:+]; RCVD_IN_DNSWL_NONE(0.00)[6.3.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.00)[ip: (-6.43), ipnet: 2607:f8b0::/32(-1.86), asn: 15169(-1.65), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Mar 2020 05:33:42 -0000 Hi Konstantin, thanks for getting back to me, First, there is a requirement that an atomic write size exists, i.e. writes > less than SC_PIPE_BUF are guaranteed to not interleave if succeeded. Our > PIPE_BUF is 512 bytes. > Useful to know, thanks. > I think that unexplained blocking (it is very hard to track down such > state) is worse then ENOMEM outcome. > This is probably true, but the ENOMEM behavior is by no means benign. If a process exhausts all pipe kva, the next process trying to allocate and write to a pipe will probably crash. That could basically be anything in the system. From the ssh server to (in our case) the infrastructure that runs jobs. Of course arguable the same thing happens in a regular ENOMEM situation also, but between paging and user space monitoring of memory usage, such situations seem easier to manage. I guess there may also be a concern about dos-style attacks. It seems pretty easy to allocate enough pipes to exhaust that limit. Regardless, I just wanted to raise this, since I considered the behavior odd and we didn't see it elsewhere. We have since found the culprit for the OOM condition: One of our tests tries to provoke an EMFILE condition to test our handling of this corner case, so it just fills every unallocated fd with pipes. However, since it doesn't write to them, we never actually see a failure there. Instead some random other process in the system will crash. Sometimes another test run (where we saw the error), but occasionally also the CI system itself. I suspect this is responsible for a fair number of mysterious failures we observed. From our perspective this issue should be resolved - I guess I'll leave it to you to decide whether there's anything to be done about the denial of service concern. I don't know what guarantees FreeBSD makes for kernel resource usage particularly in the context of jails, so I don't know if this is of concern at all. In regular usage, without a malicious program (or well-it's-malicious-but-it's-a-test-script), the admin probably would just bump the sysctl and everything would keep running nicely. Thanks again for your detailed reply. Keno