From owner-svn-src-all@freebsd.org Tue Sep 15 14:37:36 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 459193D4C46; Tue, 15 Sep 2020 14:37:36 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BrQmv2fvjz4B1V; Tue, 15 Sep 2020 14:37:35 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qk1-x736.google.com with SMTP id f142so4360386qke.13; Tue, 15 Sep 2020 07:37:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=gPKbWKPvWRFOxn2o7DtyHRE8bInKUYbARhuwQ9P5QhA=; b=CLUunMaEU0rLPlImL/M1JI1Ae0scFUe1tzO33h/01qycjCwoK+U6o7gK6B9+IePTu6 4QHT1tW8AfFT7nA2qtk5S8SHAGjO8/8OScY/Uc2/BwvrzRy8E1K9o/17vNXq2I7GDJVd RLI4YiJwbIV5SRfqRmUvEfqjiZOM7yz0bly/ou7INHzB3k06LK8WRJzKxlFoLr+j/1zw BRp+OHjIrWUk0eY+7IOhdg+R+MoZSSMmr1ZeFdz3IXwYQgTTulzBK09MoubDo5UWfdxF bWYBxS9gbIg9Syp3MwjKfsQmrGZfKsJfBU1wO8Io6P2xPgNeiOQFsZ9S5dpaefVHOoYl gROA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to; bh=gPKbWKPvWRFOxn2o7DtyHRE8bInKUYbARhuwQ9P5QhA=; b=R13KdsPNgHZ5Eqb4V6a7mj4IOPQU1D9ITfjpAnmttNfD0kzc56+CzqmDMwFkpNU2O0 Lq3oE+18bmIE65IU9rKUOQocswY0malr4CojtOHHu5KrP06zbTG3Dh9E99yMAUmGoIs0 +0hctr/0NzuIOKBH99cf4dJSZdVDZ9oJ8yClkEbpvWrXaIAnwQoFvzgx7rTAPGVf7iAc 4WTK7L9/p+oP6XBI0i8wxKsW06P0MmWxsUk+KQZ2OZ3mRPDrH+CrvW5zQckQfJGuN73U vIZ6iQfrVh2+bdB2rVjefr7T9AYKbUkJTbdCu8j6Fh+MX1apE0sPqAuVIvqzw676E/1v qsqQ== X-Gm-Message-State: AOAM532hHcFHhud2B3t3GH7gGANDufbB77Ah2HUinTZ7kE7f//SvnAQ4 hzxOScEvFXxx+dl/j2VzHuc= X-Google-Smtp-Source: ABdhPJw9746+Inzk32wurWzUZ8Q+ukA/wIJ0sQohzOVS3Bmc5fLoX2zyoDZqtnZMpcR05gZMN0Rm2Q== X-Received: by 2002:a37:7785:: with SMTP id s127mr17682826qkc.386.1600180654362; Tue, 15 Sep 2020 07:37:34 -0700 (PDT) Received: from raichu (bras-vprn-toroon0560w-lp130-18-184-148-120-111.dsl.bell.ca. [184.148.120.111]) by smtp.gmail.com with ESMTPSA id m138sm17199553qke.99.2020.09.15.07.37.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2020 07:37:32 -0700 (PDT) Sender: Mark Johnston Date: Tue, 15 Sep 2020 10:37:30 -0400 From: Mark Johnston To: Konstantin Belousov Cc: Warner Losh , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org, peter@holm.cc Subject: Re: svn commit: r364944 - head/sys/kern Message-ID: <20200915143730.GB4145@raichu> References: <202008290429.07T4TrbH007764@repo.freebsd.org> <20200915141530.GM94807@kib.kiev.ua> <20200915142209.GA4145@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200915142209.GA4145@raichu> X-Rspamd-Queue-Id: 4BrQmv2fvjz4B1V X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=CLUunMaE; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::736 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-1.67 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; NEURAL_HAM_SHORT(-0.07)[-0.067]; FREEMAIL_TO(0.00)[gmail.com]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[184.148.120.111:received]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.86)[-0.856]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.05)[-1.048]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::736:from]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[svn-src-all,svn-src-head] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Sep 2020 14:37:36 -0000 On Tue, Sep 15, 2020 at 10:22:09AM -0400, Mark Johnston wrote: > On Tue, Sep 15, 2020 at 05:15:30PM +0300, Konstantin Belousov wrote: > > On Sat, Aug 29, 2020 at 04:29:53AM +0000, Warner Losh wrote: > > > Author: imp > > > Date: Sat Aug 29 04:29:53 2020 > > > New Revision: 364944 > > > URL: https://svnweb.freebsd.org/changeset/base/364944 > > > > > > Log: > > > devctl: move to using a uma zone > > > > > > Convert the memory management of devctl. Rewrite if to make better > > > use of memory. This eliminates several mallocs (5? worse case) needed > > > to send a message. It's now possible to always send a message, though > > > if things are really backed up the oldest message will be dropped to > > > free up space for the newest. > > > > > > Add a static bus_child_{location,pnpinfo}_sb to start migrating to > > > sbuf instead of buffer + length. Use it in the new code. Other code > > > will be converted later (bus_child_*_str is only used inside of > > > subr_bus.c, though implemented in ~100 places in the tree). > > > > > > Reviewed by: markj@ > > > Differential Revision: https://reviews.freebsd.org/D26140 > > > > > > Modified: > > > head/sys/kern/subr_bus.c > > > > > > > > + } else { > > > + /* dei can't be NULL -- we know we have at least one in the zone */ > > > + dei = uma_zalloc(devsoftc.zone, M_NOWAIT); > > > + MPASS(dei != NULL); > > This does not work. I believe you need to disable per-cpu cache for the > > zone, at least. But I am not sure it is enough. > > From the report we have: > > db:0:pho> show uma > Zone Size Used Free Requests Sleeps Bucket Total Mem XFree > DEVCTL 1024 6415164 0 6416203 0 16 6569127936 0 > > so it looks like the primary problem is a leak. > > > https://people.freebsd.org/~pho/stress/log/kostik1314.txt devctl_queue() does not maintain the queue length bound, so if devd goes away (due to an OOM kill in this case), we can end up with a large backlog. This hack might be sufficient. diff --git a/sys/kern/subr_bus.c b/sys/kern/subr_bus.c index 19c056ab9974..91c57cdfae5b 100644 --- a/sys/kern/subr_bus.c +++ b/sys/kern/subr_bus.c @@ -635,7 +635,14 @@ devctl_queue(struct dev_event_info *dei) { mtx_lock(&devsoftc.mtx); STAILQ_INSERT_TAIL(&devsoftc.devq, dei, dei_link); - devsoftc.queued++; + if (devctl_queue_length != 0 && + devctl_queue_length == devsoftc.queued) { + dei = STAILQ_FIRST(&devsoftc.devq); + STAILQ_REMOVE_HEAD(&devsoftc.devq, dei_link); + uma_zfree(devsoftc.zone, dei); + } else { + devsoftc.queued++; + } cv_broadcast(&devsoftc.cv); KNOTE_LOCKED(&devsoftc.sel.si_note, 0); mtx_unlock(&devsoftc.mtx);