From owner-freebsd-hackers@freebsd.org Wed Mar 4 21:43:36 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4C3CD252AA2 for ; Wed, 4 Mar 2020 21:43:36 +0000 (UTC) (envelope-from keno@juliacomputing.com) Received: from mail-il1-x143.google.com (mail-il1-x143.google.com [IPv6:2607:f8b0:4864:20::143]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48XnSQ0LFbz4b4F for ; Wed, 4 Mar 2020 21:43:33 +0000 (UTC) (envelope-from keno@juliacomputing.com) Received: by mail-il1-x143.google.com with SMTP id g126so3183406ilh.2 for ; Wed, 04 Mar 2020 13:43:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=juliacomputing-com.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to:cc; bh=mP8tDChjczQfoaVT7BtcMixrtSz+HS/j/2DN2oIXxL8=; b=UwK5mv+NgTPOrG+VSnOZIOV5kAOrF6lozUMRX0jAFJ8EzEg/nD2DsgugSeeDYroGSk geAAcP4fhD8EZ4SwYob/4RmnMKrP4fIjtDtOsZWWt8owp1DLcW0/ExxXpVBSpMQ+UPaC WBf0483aN9HYVOYOJDwx9tZNR7ITZwicHcGAWTVjZYYke6HwpHE1fN9Eb3wWj/bMTLHk WjyG6xKpoj3jvyE0eQkTrfcgURUCFhZit6cainJJMAccmiTIs7l0Q9Fzo1LzzAi/aeWp X+xneZ+m0ttVCLp7e01/4Tv6Ps4mihsbotqIXk2lbfy2p5EN27gGwlZq00lT5h7369fQ +CKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=mP8tDChjczQfoaVT7BtcMixrtSz+HS/j/2DN2oIXxL8=; b=Qrd+2nNFaDfEYsBXuX5c/AByf3gbsehYEOR8fRRMND9H4pzrziCEEt0mbelhpux7Bl QEUewQpyv4QJWMgRgDIgtCDwZaZB76iTvnJ55shQxp8WdFUibBw3vS/3DUOkaI7+nDg6 tMXXxovX+RaC0poOZyVdAiBhRawazMBHm0jKv1lqgy9gS4D92fgyICd1CttzNfI+V4K5 SN8cEfN8oefX04nO6481pjyxj+GV3/VpX3ec0e2/wcSmItFYFvqco7TU9EInlIoX5NQX /VBXZLktkeBpwgJLFSxw0TVUtcILLeJi4834WHSEpbRLuBZ6v88PcUwvFXoKKKt4892X ZAnQ== X-Gm-Message-State: ANhLgQ1m1z/cjpKLhp0XSaokTNqOHRRg1dxgOvwLLV/c1SRhrL74AIcW a6gEThppMRvJYSHLKuvRz+a0CSzUZtow8rlAHTGKCnWfDIw= X-Google-Smtp-Source: ADFU+vsXFcQQQcGRvM+sK+OGB3Uk55uxON19Wki3BGl/WmjWuKil8CeWkw+WU+4oedvLhDp/z4rsFaIZN4D4VN/2AWc= X-Received: by 2002:a92:6b03:: with SMTP id g3mr4735809ilc.201.1583358212512; Wed, 04 Mar 2020 13:43:32 -0800 (PST) MIME-Version: 1.0 From: Keno Fischer Date: Wed, 4 Mar 2020 16:42:56 -0500 Message-ID: Subject: FreeBSD Pipe behavior in pipe OOM situations To: freebsd-hackers@freebsd.org Cc: Elliot Saba X-Rspamd-Queue-Id: 48XnSQ0LFbz4b4F X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=juliacomputing-com.20150623.gappssmtp.com header.s=20150623 header.b=UwK5mv+N; dmarc=none; spf=none (mx1.freebsd.org: domain of keno@juliacomputing.com has no SPF policy when checking 2607:f8b0:4864:20::143) smtp.mailfrom=keno@juliacomputing.com X-Spamd-Result: default: False [-1.60 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.993,0]; R_DKIM_ALLOW(-0.20)[juliacomputing-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[juliacomputing.com]; URI_COUNT_ODD(1.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[juliacomputing-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[3.4.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-0.30)[ip: (2.04), ipnet: 2607:f8b0::/32(-1.86), asn: 15169(-1.66), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Mar 2020 21:43:36 -0000 Greetings, I am debugging intermittent failures we see on the CI system for the Julia programming language on FreeBSD, but not elsewhere. The Julia ticket for this issue can be found at https://github.com/JuliaLang/julia/issues/23143. The symptom is an ENOMEM error on a write to a pipe, together with the following message in dmesg: kern.ipc.maxpipekva exceeded; see tuning(7) Now, as far as I understand it, what's happening here is that FreeBSD has a hard limit on the amount of kernel memory that can be used for pipe buffers, which we are exceeding by creating too many pipes (not entirely surprising, our test suites spawns many processes and uses lots of pipes). I understand that we can likely work around this issue by increasing the referenced sysctl. However, I am a bit puzzled by the ENOMEM behavior. I don't have very much experience with the FreeBSD kernel, but from my experience from working on other operating systems, I would have expected that either: 1) Some minimal buffer is allocated anyway and exempt from such pipe-specific memory limits (e.g. a few bytes of the pipe struct), or, 2) The writing process is blocked until pipe buffer space becomes available (e.g. by a different pipe draining and freeing up space), or, 3) The writing process is blocked until a reader comes along, at which point the write is performed directly without intermediate kernel buffer. I.e. I would have expected such an OOM situation for pipe buffers to degrade pipe performance, but not to have it exposed to the user. Indeed, a cursory read of the FreeBSD kernel source seems to reinforce this notion. In pipe_create, we see the following comment: ``` /* * Note that these functions can fail if pipe map is exhausted * (as a result of too many pipes created), but we ignore the * error as it is not fatal and could be provoked by * unprivileged users. The only consequence is worse performance * with given pipe. */ if (amountpipekva > maxpipekva / 2) (void)pipespace_new(pipe, SMALL_PIPE_SIZE); else (void)pipespace_new(pipe, PIPE_SIZE); ``` But then later, in pipe_write, we see: ``` if (wpipe->pipe_buffer.size == 0) { /* * This can only happen for reverse direction use of pipes * in a complete OOM situation. */ error = ENOMEM; ``` >From my (admittedly limited) understanding of the code, it doesn't seem that either comment is accurate. If the pipe buffer allocation fails, then `write`s will return `ENOMEM`, even in the forward direction (the buffer for the reverse direction isn't allocated by default, but as indicated by the first comment, the allocation for the forward direction can certainly fail). I was hoping a FreeBSD kernel developer could shed some light on whether the kernel behavior we're experiencing here is indeed expected on FreeBSD, or whether it would be expected that the kernel would try harder to service the pipe request in such a situation. Thanks, Keno