From nobody Thu Aug 3 15:14:44 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RGspJ4Z5Kz3cK22 for ; Thu, 3 Aug 2023 15:14:48 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qv1-xf34.google.com (mail-qv1-xf34.google.com [IPv6:2607:f8b0:4864:20::f34]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RGspJ24jnz4Pk7 for ; Thu, 3 Aug 2023 15:14:48 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qv1-xf34.google.com with SMTP id 6a1803df08f44-63cf57c79b5so6687046d6.0 for ; Thu, 03 Aug 2023 08:14:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691075687; x=1691680487; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:sender :from:to:cc:subject:date:message-id:reply-to; bh=kaIWtj37fXHi8dXeaknFIboxPhSn7t3HLwpmm6rRB5Y=; b=nkUgh8odMJhxaoTx1tdstOR4i2GPpLw/r8uufloIRWrznjaxcbA5KKi1s0fZxEKBhz jiG4iOWIqsmDSEZwdB5Z4BpkWA+S/ewontVI3nE4DzjHmyfoSVuGj/wtVKSWEIXoAFMP /sL0VMWY1j/7wiNyNGnPCffAcKMbvgsCkRyQvnqDMfmBdT1D84oh9PgzRcwn9+Ww9Kga V+yn3lnqTpyfoZBX2Iv8sm8VtimxacMl/DcEpGCFOpSK3ouGe+ORZ950arl8Xf8k3zIt WO8NXpKF6WjxEKMI8AoQbWo8LJ5N0J8Eh/OSQRsojSSamWAL2aAu6a5D+YO33uhHypDp 6hyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691075687; x=1691680487; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kaIWtj37fXHi8dXeaknFIboxPhSn7t3HLwpmm6rRB5Y=; b=OAoTyeC1OCCnNapvB6kNC/YVvbtU4uU4XWz3CWaRVtHNE4kWRx4bRV/GUlIjsFHzyi kEBVo8zszX2awzvMatRS5wTlKHBFAowJISm16DFYNCfcArTZ8JXwhIaLDHJFWoUCbHRg AOy9cr4hyJL7lsWPlUZcmqgzZ+z/97PEfQvk4WhXNjaJHecfel7gGvtto1imzr4bPI+z IL9LMvx6sFMev9gv29vd/wpz5NOUyzEl7+pycVaFUQDktFeDwh5eh5cE5beVLvb8/M1x QVY0o+Kn+YCfy8+qLQAsA8mjs+/VlhWo7ASwsn9t8vM5ByZnJVj//49Pwa/m7FGNaOYt 50Fw== X-Gm-Message-State: ABy/qLaHmEwKMakE1CH0QEw6UnuwtdxRsHht9o3rkOFLBdVH3POysh1s vjwgQ2BcVc2PaAU7s9ujWNWvxQ4uNn8= X-Google-Smtp-Source: APBJJlEXvZRGUrnscggi1rB0OeIz8lu1+HcJHEGdyD2VtsVW3eaNfS50W+jXPMUTMgU8fWyL6f5hSg== X-Received: by 2002:a0c:cb09:0:b0:63c:fb1b:6bb0 with SMTP id o9-20020a0ccb09000000b0063cfb1b6bb0mr18095925qvk.52.1691075687348; Thu, 03 Aug 2023 08:14:47 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id p6-20020a0cf546000000b0063cf49c35f1sm6479878qvm.35.2023.08.03.08.14.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Aug 2023 08:14:46 -0700 (PDT) Date: Thu, 3 Aug 2023 11:14:44 -0400 From: Mark Johnston To: Shrikanth Kamath Cc: freebsd-hackers@freebsd.org Subject: Re: How to watch Active pagequeue transitions with DTrace in the vm layer Message-ID: References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 4RGspJ24jnz4Pk7 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] On Thu, Aug 03, 2023 at 12:32:22AM -0700, Shrikanth Kamath wrote: > A background on the query, > > Trying to catch a memory “spike” trigger using DTrace, refer here two “top” > snapshots captured during a 2 minute window, > > last pid: 89900; load averages: 0.75, 0.91, 0.94 up 39+00:37:30 > 20:03:14 > > Mem: 5575M Active, 2152M Inact, 4731M Laundry, 3044M Wired, 1151M Buf, 382M > Free > > Swap: 8192M Total, 1058M Used, 7134M Free, 12% Inuse > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > > 12043 root 5 35 0 11G 9747M kqread 3 > 128.8H 23.34% app1 > > 12051 root 1 20 0 3089M 2274M select 1 22:51 > 0.00% app2 > > last pid: 90442; load averages: 1.50, 1.12, 1.02 up 39+00:39:37 > 20:05:21 > > Mem: 8549M Active, 631M Inact, 3340M Laundry, 3159M Wired, 1252M Buf, 359M > Free > > Swap: 8192M Total, 1894M Used, 6298M Free, 23% Inuse > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > > 12043 root 5 24 0 11G 9445M kqread 2 > 128.8H 10.45% app1 > > 12051 root 1 20 0 3089M 2173M select 3 22:51 > 0.00% app2 > > The spike is ~3G in Active pages, the two large applications have a > combined resident size of ~12G. The resident size of the applications > hasn’t changed between these 2 readings, however there is a tar archive and > gzip on a large directory during that window likely causing a reshuffle. If > I count the page allocs and dequeue by execname with DTrace, I see > tar/vmstat which probably alloc and quickly dequeue, along with a large > dequeue being undertaken by bufdaemon and pagedaemon. > > fbt::vm_page_alloc*:entry > > { > > @cnt[execname] = count(); > > } > > fbt::vm_page_dequeue:entry > > { > > @dcnt[execname] = count(); > > } > > Page Alloc > > vmstat > 20222 > > tar 21284 > > Page Dequeue > > vmstat 20114 > > bufdaemon 21402 > > tar 21635 > > pagedaemon 360387 > > Since the tar / vmstat will not hold the pages in Active, I need to find > out what application had its pages queued in Active page queue. One possibility is that the inactive and laundry queue had previously contained many referenced pages. Then, when some memory pressure occurred, the pagedaemon scanned the queues and moved a large number of pages into the active queue. > Is it possible that the system is just moving the LRU pages of these two > large applications into the inactive queue prior to addressing memory > pressure? Do these applications need to activate those pages later and > hence it brings it back into the Active queue? How do I watch this in > action by using DTrace? Will the following probe catch this trigger? > > fbt::vm_page_activate:entry > > { > > @cnt[execname, pid] = count(); > > } > > tick-10sec > > { > > printa(@cnt); > > printf("ACTIVE[%d] pages\n", `vm_dom[0].vmd_pagequeues[1].pq_cnt); > > } > > *** This system is running only one vmdomain (# sysctl vm.ndomains –> > vm.ndomains: 1). > > *** running release 12.1, on an amd64 kernel. The physical memory installed > is 16G. In 12.1, you'd probably want something like: fbt::vm_page_enqueue:entry /args[1] == 1 /* PQ_ACTIVE *// { ... } since vm_page_unwire(m, PQ_ACTIVE) will also move a page into the active queue, but your D script above won't catch that. I would also look at the "pages reactivated by the page daemon" counter that appears in vmstat -s output. That'll tell you how many times the page daemon moved a page from PQ_INACTIVE/PQ_LAUNDRY to PQ_ACTIVE because it found that the page had been referenced.