From owner-svn-src-all@freebsd.org Tue Sep 15 15:20:41 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 825303D626A for ; Tue, 15 Sep 2020 15:20:41 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qv1-xf31.google.com (mail-qv1-xf31.google.com [IPv6:2607:f8b0:4864:20::f31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BrRkb4bwlz4FHY for ; Tue, 15 Sep 2020 15:20:39 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qv1-xf31.google.com with SMTP id q10so1909499qvs.1 for ; Tue, 15 Sep 2020 08:20:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=VPxu8SFchIKjiN5XSqHM3N1EkkM5BiCYjn7I7gPM8XM=; b=boQitBEdyAuS6c0XKmlRIbjQVoTGYVO7nkUUbTJwM7/yyvK5Cn1b45yw2vXH5mjnJh rvB008IcdTA8JidmhaDi+hElAMHUYSRBYQPX1lYIWJ1sTqBKchtNDa0GqVnGYeQEIrGv 9PQk8lIinTq1gbAu6vTAgGh/YHNqDhk+1DY/R/LYB8gOOXQ8iPgcQqKPy/81GQfx8Pmu HmNF4gXqShcyuKtYK6ofDPEYmLeYxCmVOmNBX5lZPmjCl06SSOnM1pQcCtrK+1nD4dom LAfHhixErwtlZNNXpn3mbJX1NmpS0PluatlTxIfZ6an3soHLazq2xmrtHi2GibocVmAy VzzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=VPxu8SFchIKjiN5XSqHM3N1EkkM5BiCYjn7I7gPM8XM=; b=Ab6OKu33j0x5enLwDt12LbSwr/QIIfjHAhVeLLRaWXPlth5GdZQIOsx4V1cXAHiVQc donzos2TZp/4ac0hyIkTY3yu7tAJnKjTgVN8Gos12CzGcIEiiL67jf6HH5KjXeTU+nzR Cd1JCYRFOQAzJzVupbxjsJ6OwpOivs0eJBXYaJX+FH1exDimxsGXlhSyamZOI4T1jGDV 9yLBcCHWi1Wz6BNg7uMcjbf5r1zasto3jX6adZMRUAk99nKkg3KmdDDrPwdtoQ3SX7rd LbM0wdNQ5GMnt/RVsD8vB2WmLl/3fKXxmVlklhcZkI/i84xQgoYEJkqN0jfVr4hqy0Ex oxdg== X-Gm-Message-State: AOAM532VSbS425+1Hls+8/FyYK/K4gA86Fqjo96k88ilF23Z3cJuIZn5 FbngyhhD+7eiajCCEO3APcHoOzyDNlkfMcLx+t69dQ== X-Google-Smtp-Source: ABdhPJx/iyiXGuXDn+GOqguExXZsb3ab/Q/oH21Oq8ykyLtEDUeHsHJxhR2K9nmUO8U7XGz9wS8dpYMRLmVJIOcZPLc= X-Received: by 2002:a0c:a162:: with SMTP id d89mr19139898qva.28.1600183238420; Tue, 15 Sep 2020 08:20:38 -0700 (PDT) MIME-Version: 1.0 References: <202008290429.07T4TrbH007764@repo.freebsd.org> <20200915141530.GM94807@kib.kiev.ua> <20200915142209.GA4145@raichu> <20200915143730.GB4145@raichu> In-Reply-To: <20200915143730.GB4145@raichu> From: Warner Losh Date: Tue, 15 Sep 2020 09:20:27 -0600 Message-ID: Subject: Re: svn commit: r364944 - head/sys/kern To: Mark Johnston Cc: Konstantin Belousov , Warner Losh , src-committers , svn-src-all , svn-src-head , Peter Holm X-Rspamd-Queue-Id: 4BrRkb4bwlz4FHY X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=boQitBEd; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::f31) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-1.86 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-0.89)[-0.889]; FROM_HAS_DN(0.00)[]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org,holm.cc]; NEURAL_HAM_LONG(-0.99)[-0.988]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[svn-src-all@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; NEURAL_SPAM_SHORT(0.01)[0.014]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::f31:from]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; MAILMAN_DEST(0.00)[svn-src-all] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Sep 2020 15:20:41 -0000 On Tue, Sep 15, 2020 at 8:37 AM Mark Johnston wrote: > On Tue, Sep 15, 2020 at 10:22:09AM -0400, Mark Johnston wrote: > > On Tue, Sep 15, 2020 at 05:15:30PM +0300, Konstantin Belousov wrote: > > > On Sat, Aug 29, 2020 at 04:29:53AM +0000, Warner Losh wrote: > > > > Author: imp > > > > Date: Sat Aug 29 04:29:53 2020 > > > > New Revision: 364944 > > > > URL: https://svnweb.freebsd.org/changeset/base/364944 > > > > > > > > Log: > > > > devctl: move to using a uma zone > > > > > > > > Convert the memory management of devctl. Rewrite if to make better > > > > use of memory. This eliminates several mallocs (5? worse case) > needed > > > > to send a message. It's now possible to always send a message, > though > > > > if things are really backed up the oldest message will be dropped > to > > > > free up space for the newest. > > > > > > > > Add a static bus_child_{location,pnpinfo}_sb to start migrating to > > > > sbuf instead of buffer + length. Use it in the new code. Other > code > > > > will be converted later (bus_child_*_str is only used inside of > > > > subr_bus.c, though implemented in ~100 places in the tree). > > > > > > > > Reviewed by: markj@ > > > > Differential Revision: https://reviews.freebsd.org/D26140 > > > > > > > > Modified: > > > > head/sys/kern/subr_bus.c > > > > > > > > > > > + } else { > > > > + /* dei can't be NULL -- we know we have at least one in > the zone */ > > > > + dei = uma_zalloc(devsoftc.zone, M_NOWAIT); > > > > + MPASS(dei != NULL); > > > This does not work. I believe you need to disable per-cpu cache for > the > > > zone, at least. But I am not sure it is enough. > > > > From the report we have: > > > > db:0:pho> show uma > > Zone Size Used Free Requests Sleeps Bucket > Total Mem XFree > > DEVCTL 1024 6415164 0 6416203 0 16 > 6569127936 0 > > > > so it looks like the primary problem is a leak. > > > > > https://people.freebsd.org/~pho/stress/log/kostik1314.txt > > devctl_queue() does not maintain the queue length bound, so if devd goes > away (due to an OOM kill in this case), we can end up with a large > backlog. This hack might be sufficient. > > diff --git a/sys/kern/subr_bus.c b/sys/kern/subr_bus.c > index 19c056ab9974..91c57cdfae5b 100644 > --- a/sys/kern/subr_bus.c > +++ b/sys/kern/subr_bus.c > @@ -635,7 +635,14 @@ devctl_queue(struct dev_event_info *dei) > { > mtx_lock(&devsoftc.mtx); > STAILQ_INSERT_TAIL(&devsoftc.devq, dei, dei_link); > - devsoftc.queued++; > + if (devctl_queue_length != 0 && > + devctl_queue_length == devsoftc.queued) { > + dei = STAILQ_FIRST(&devsoftc.devq); > + STAILQ_REMOVE_HEAD(&devsoftc.devq, dei_link); > + uma_zfree(devsoftc.zone, dei); > + } else { > + devsoftc.queued++; > + } > cv_broadcast(&devsoftc.cv); > KNOTE_LOCKED(&devsoftc.sel.si_note, 0); > mtx_unlock(&devsoftc.mtx); > I've come to a similar conclusion... You can't queue w/o allocating... and that's where we're supposed to pop off the oldest, free it so we can allocate. So this isn't the right place for this fix, and the 'queued' name may be a bad name since it's from before. It's now supposed to be closer to 'allocated'. Warner