Date: Wed, 4 Mar 2020 16:42:56 -0500 From: Keno Fischer <keno@juliacomputing.com> To: freebsd-hackers@freebsd.org Cc: Elliot Saba <elliot.saba@juliacomputing.com> Subject: FreeBSD Pipe behavior in pipe OOM situations Message-ID: <CABV8kRy2Uu6fZwQR37135LvgUCxYFd6eiNt4NMQLg_jpHq42Lg@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Greetings, I am debugging intermittent failures we see on the CI system for the Julia programming language on FreeBSD, but not elsewhere. The Julia ticket for this issue can be found at https://github.com/JuliaLang/julia/issues/23143. The symptom is an ENOMEM error on a write to a pipe, together with the following message in dmesg: kern.ipc.maxpipekva exceeded; see tuning(7) Now, as far as I understand it, what's happening here is that FreeBSD has a hard limit on the amount of kernel memory that can be used for pipe buffers, which we are exceeding by creating too many pipes (not entirely surprising, our test suites spawns many processes and uses lots of pipes). I understand that we can likely work around this issue by increasing the referenced sysctl. However, I am a bit puzzled by the ENOMEM behavior. I don't have very much experience with the FreeBSD kernel, but from my experience from working on other operating systems, I would have expected that either: 1) Some minimal buffer is allocated anyway and exempt from such pipe-specific memory limits (e.g. a few bytes of the pipe struct), or, 2) The writing process is blocked until pipe buffer space becomes available (e.g. by a different pipe draining and freeing up space), or, 3) The writing process is blocked until a reader comes along, at which point the write is performed directly without intermediate kernel buffer. I.e. I would have expected such an OOM situation for pipe buffers to degrade pipe performance, but not to have it exposed to the user. Indeed, a cursory read of the FreeBSD kernel source seems to reinforce this notion. In pipe_create, we see the following comment: ``` /* * Note that these functions can fail if pipe map is exhausted * (as a result of too many pipes created), but we ignore the * error as it is not fatal and could be provoked by * unprivileged users. The only consequence is worse performance * with given pipe. */ if (amountpipekva > maxpipekva / 2) (void)pipespace_new(pipe, SMALL_PIPE_SIZE); else (void)pipespace_new(pipe, PIPE_SIZE); ``` But then later, in pipe_write, we see: ``` if (wpipe->pipe_buffer.size == 0) { /* * This can only happen for reverse direction use of pipes * in a complete OOM situation. */ error = ENOMEM; ``` >From my (admittedly limited) understanding of the code, it doesn't seem that either comment is accurate. If the pipe buffer allocation fails, then `write`s will return `ENOMEM`, even in the forward direction (the buffer for the reverse direction isn't allocated by default, but as indicated by the first comment, the allocation for the forward direction can certainly fail). I was hoping a FreeBSD kernel developer could shed some light on whether the kernel behavior we're experiencing here is indeed expected on FreeBSD, or whether it would be expected that the kernel would try harder to service the pipe request in such a situation. Thanks, Keno
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABV8kRy2Uu6fZwQR37135LvgUCxYFd6eiNt4NMQLg_jpHq42Lg>