From owner-freebsd-current@freebsd.org Tue May 14 22:03:30 2019 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 224ED159A1A5 for ; Tue, 14 May 2019 22:03:30 +0000 (UTC) (envelope-from tmunro@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BDB3486C51; Tue, 14 May 2019 22:03:29 +0000 (UTC) (envelope-from tmunro@freebsd.org) Received: from mail-oi1-f174.google.com (mail-oi1-f174.google.com [209.85.167.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: tmunro/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 77C62838A; Tue, 14 May 2019 22:03:29 +0000 (UTC) (envelope-from tmunro@freebsd.org) Received: by mail-oi1-f174.google.com with SMTP id m204so335809oib.0; Tue, 14 May 2019 15:03:29 -0700 (PDT) X-Gm-Message-State: APjAAAVbrWVsszqiZHG+pxGSbujP0w+jmhQEK+oxe74zBjuuTmkPLU6X Zt76v2Skj9osGzK1BvjMSsYCpV+DtYUjO2ofeFI= X-Google-Smtp-Source: APXvYqw7wPJmIFndYU4fxE96fSnmwF1iKUBX7lbQHy1bgUagFO2phJNA9+RWR8UTRvurBicDz2YiRsdyL3i7A5VKyb4= X-Received: by 2002:aca:330b:: with SMTP id z11mr4449661oiz.148.1557871408683; Tue, 14 May 2019 15:03:28 -0700 (PDT) MIME-Version: 1.0 References: <9c5eaa94-f55b-464a-ab0f-267e7fce4bd0@FreeBSD.org> <20190318153230.GS96870@kib.kiev.ua> <782c2d4b-6920-085b-5489-65fae462a194@FreeBSD.org> In-Reply-To: From: Thomas Munro Date: Wed, 15 May 2019 10:02:51 +1200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: leaked swap? To: Bob Bishop Cc: Andriy Gapon , Konstantin Belousov , FreeBSD Current , Alan Somers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: BDB3486C51 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.96)[-0.957,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2019 22:03:30 -0000 On Tue, May 14, 2019 at 9:24 PM Bob Bishop wrote: > > On 14 May 2019, at 07:50, Andriy Gapon wrote: > > In the end, it was POSIX shared memory. > > I put the system into the single-user to clean up the memory as much as possible > > and then I panic'ed it and went through dirty pages and their related objects in > > kgdb. As far as I can tell, the memory was leaked via POSIX shared memory > > objects that were never shm_unlink-ed. It seems that there was a misbehaving > > program that had been creating such objects and then losing track of them. (I > > was able to identify it from names it used for the objects) > > > > It seems that, unfortunately, there is no way to list / discover POSIX shared > > memory objects that are not opened by any process. > > Losing track of shared memory objects has been a problem since SysVr2 ... > > > I wrote a small gdb script to examine shm_dictionary in kgdb. It would be nice > > to have a utility (and a kernel interface) that could do the same from userland. > > ... it is indeed high time it was fixed. Hello, Newbie hacker here. Here are some things I've noticed while working (fairly heavily) with POSIX shm: 1. As mentioned, you can't list 'em (unlike Linux, where you can just ls /dev/shm). There's a TODO note, but it's not clear whether it's best to extend ipcs or create a new userspace tool, and it wasn't immediately clear to me how to feed the arbitrary sized results back to userspace. I had a scheme worked out where you'd keep calling a sysctl repeatedly to collect the data until it was done, and it'd sometimes tell you you need a bigger buffer (because you probably have to drain at least a whole hash table bucket at a time), but an experienced FreeBSD hacker told me that was BS, and maybe what's needed is a device you read. As well as the list-all-the-segments tool, you'd also want to be able to unlink to tidy up. 2. procstat -v doesn't show the paths of POSIX shm objects that are mapped in (unlike the Linux equivalent, where they're treated as mapped files). That's quite useful to me when developing. I have a draft patch somewhere that fixes that, though I hadn't got around to dealing with jail prefixes sensibly. 3. The hash table is of fixed size and has one simple lock. Maybe this is not really a problem for anyone yet, I don't know. Perhaps the locking should be made more granular to reduce contention from the list-all-the-segments thing that needs to loop over it. 4. I'm suspicious of the way jailing is implemented; I haven't tested but I suspect that jails leak all their POSIX shm when they shut down, but then if you start a new jail at the same path it can be accessed again.