From owner-freebsd-hackers@freebsd.org Tue Sep 11 15:08:55 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DEEEC1093FBF for ; Tue, 11 Sep 2018 15:08:54 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pg1-x534.google.com (mail-pg1-x534.google.com [IPv6:2607:f8b0:4864:20::534]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5649B7AFED for ; Tue, 11 Sep 2018 15:08:54 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pg1-x534.google.com with SMTP id b129-v6so12396677pga.13 for ; Tue, 11 Sep 2018 08:08:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=QGKTaEkkf1HNlxCS8dbiIESZAcEdd+aG5ShW+GZdQEU=; b=TIbDY2aM8sDCOFvd/BwfrUnfgeqS4w9CrA9cEqY1uToyW12QTcU8/L89citFFhBKio 7nSDteD2HuTk5VazupaDTjY4vJruqxCbjKm8usvDGEsPby1NKu581Elq1I0frhXTdxLD uv+tjGOIMbcmrxejxzZvOlUNTAFQr1tgWNUwEWbb8fQ3pouOs9HkpP2s3rJV+6XczfWh MMle1CatXYVcAqmPeyybqoVh2/5fixnoJT22u5InXay/m+8BEjrk2nZrBAiknh6+Jehr Aq8q92o4uo14wvaDyWcozF36TMo+IyRggFcC0eL0V+ZsZbrxyIID/TRGFA7Np6z0Tv2v 7Hew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=QGKTaEkkf1HNlxCS8dbiIESZAcEdd+aG5ShW+GZdQEU=; b=Jarpbj1ncF54P5vGtkLGONkzm9UqS0IaybvvzZ+P13XEWCOwuYNLg6lBiNMak7ay3q WclxTjNltNe5vJz57HKbeawTFo1lskE/ydQPS2xvBqijPoe1hPjHqwGGARyNLapJzP7p ZP1NzZZQ1wFZzOD6o4xPMO004SDnUvNoVkcgq1vevPhMTvsS1R0pcnMeuxM3MJRVSEoU b5Smukhqr139jcaYIz/aipu07XySQ8kCPC5RZ9V0t3bUb3Im6aV4dFcnVz5LjW7WXWLv HRUQRsR+Us7ZMXe1ekHuK2lURtKEba+Y5q4dkjXGJeEw3b9nnY9htiTo496zmkIA0LrT 6LlQ== X-Gm-Message-State: APzg51BUxMNDz/88g9VnQdVIk9O2+GKQk4jxTPBf0JcXrAu43HRIPG2W Wc3kXanWl9A9kDLgrLIaBuU= X-Google-Smtp-Source: ANB0VdbfkPFP2z9gJHzs9ZyjbPO+nTFg7S6Th5p9OTp2wT7USD/vpgoYMwGbBdRtNYv4Mbus1CMWjA== X-Received: by 2002:a63:1947:: with SMTP id 7-v6mr29025484pgz.192.1536678533262; Tue, 11 Sep 2018 08:08:53 -0700 (PDT) Received: from raichu (toroon0560w-lp130-01-174-88-78-8.dsl.bell.ca. [174.88.78.8]) by smtp.gmail.com with ESMTPSA id r64-v6sm30539956pfk.157.2018.09.11.08.08.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 08:08:52 -0700 (PDT) Sender: Mark Johnston Date: Tue, 11 Sep 2018 11:08:49 -0400 From: Mark Johnston To: Robert Cc: freebsd-hackers@freebsd.org Subject: Re: Sudden grow of memory in "Laundry" state Message-ID: <20180911150849.GD92634@raichu> References: <55b0dd7d-19a3-b566-0602-762b783e8ff3@gmail.com> <20180911005411.GF2849@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Sep 2018 15:08:55 -0000 On Mon, Sep 10, 2018 at 10:18:31PM -0700, Robert wrote: > Hi, if I understood correctly, "written back to swap device" means they > come from swap at some point, but it's not the case (see attached graph). Sorry, I didn't mean to imply that. Pages containing your application's shared memory, for instance, would simply be written to the swap device before being freed and reused for some other purpose. Your graph shows a sudden drop in free memory. Does that coincide with the sudden increase in size of the laundry queue? > Swap was 16GB, and slightly reduced when pages rapidly started to move > from free (or "Inactive"?) into "Laundry" queue. Right. Specifically, the amount of free swap space decreased right at the time that the amount of free memory dropped, so what likely happened is that the system wrote some pages in "Laundry" to the swap device so that they could be freed, as a response to the drop in free memory. > Here is vmstat output: > > vmstat -s > [...] > 12 clean page reclamation shortfalls This line basically means that at some point we were writing pages to the swap device as fast as possible in order to reclaim some memory. > What do you think? How pages could be moved into "Laundry" without being > in Swap? That's perfectly normal. Pages typically move from "Active" or "Inactive" to laundry. > > On 09/10/18 17:54, Mark Johnston wrote: > > On Mon, Sep 10, 2018 at 11:44:52AM -0700, Robert wrote: > >> Hi, I have a server with FreeBSD 11.2 and 48 Gigs of RAM where an app > >> with extensive usage of shared memory (24GB allocation) is running. > >> > >> After some random amount of time (usually few days of running), there > >> happens a sudden increase of "Laundry" memory grow (from zero to 24G in > >> a few minutes). > >> > >> Then slowly it reduces. > >> > >> Are described symptoms normal and expected? I've never noticed anything > >> like that on 11.1. > > The laundry queue contains dirty inactive pages, which need to be > > written back to the swap device or a filesystem before they can be > > reused. When the system is short for free pages, it will scan the > > inactive queue looking for clean pages, which can be freed cheaply. > > Dirty pages are moved to the laundry queue. My guess is that the > > system was running without a page shortage for a long time, and > > suddenly experienced some memory pressure. This caused lots of > > pages to move from the inactive queue to the laundry queue. Demand > > for free pages will then cause pages in the laundry queue to be > > written back and freed, or requeued if the page was referenced after > > being placed in the laundry queue. "vmstat -s" and "sysctl vm.stats" > > output might make things more clear. > > > > All this is to say that there's nothing particularly abnormal about what > > you're observing, though it's not clear what effects this behaviour has > > on your workload, if any. I can't think of any direct reason this would > > happen on 11.2 but not 11.1. >