From nobody Thu Aug 3 07:32:22 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RGgYb3DgSz4mN1Y for ; Thu, 3 Aug 2023 07:33:07 +0000 (UTC) (envelope-from shrikanth07@gmail.com) Received: from mail-ot1-x334.google.com (mail-ot1-x334.google.com [IPv6:2607:f8b0:4864:20::334]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RGgYY4d5nz3Gyq for ; Thu, 3 Aug 2023 07:33:05 +0000 (UTC) (envelope-from shrikanth07@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=dhQh0fE4; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of shrikanth07@gmail.com designates 2607:f8b0:4864:20::334 as permitted sender) smtp.mailfrom=shrikanth07@gmail.com Received: by mail-ot1-x334.google.com with SMTP id 46e09a7af769-6bc57401cb9so139727a34.0 for ; Thu, 03 Aug 2023 00:33:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691047984; x=1691652784; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=MzjS9HoPFUv2gDOcQklSS5FgsYlHNY96j4duo77g2/A=; b=dhQh0fE4C4vUl+lmEclkw3dDO+/NFKQlDvuKI7SX1CPetGTCCCkhhIrORZte2txYjm /yumEuSyB6uHCULiK1zm9xrVduTUSSdXHcHTc3XtOCUQ0ORGDEA3NQam+MqG1UDUvB1G 6PL3tuarntQTkJ5rdOicblrql4/m84c9x+5FyR6vh99LbpD+edmMFA3HGsxudnmsUp+o ASQbpt+Uv+FLSgjC646avVYv5DB/GScm44WP0Xpd0U7scalFYiQflkZ1k/PuUKx0xKoN 0YG2j2vqBTH2U/xN7PvOtwyX1xGUh3zg3MZQdJk8bU7BgtpTO26/3j0yQTqnnEDE4X2T XJtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691047984; x=1691652784; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=MzjS9HoPFUv2gDOcQklSS5FgsYlHNY96j4duo77g2/A=; b=aUxQsVg8h/VSZ++9XNGo9gGDMuNZmC/AOJpBjUZikAOA7DCGoeswX4ZmOnhg6j5r6n LBazhIjXnf/Z4jwTtQhfV6q99T9OYnLwXzVxpzg3W2wks+CQ0Y7nAHI6P8DkAH5ERUo8 Snh9RH3E6HFbw8Mp3eYuMA8CMDZ0FYvVdXiXX5FtsMi0FE1l/GtoumNb1CZ/XCsbn3LC AVKTE/UdRl2jjnfJEEtRluABHuBm3N+JbUKR9dDI/TdZuq8yQjoPkaQMxvt5KnVaSKet TGuTt9qDmsbFX59GPe01h0SztJ2wJYtvbxKL/3a06eUG3CS4kgyWAUK2Tw4ZTr+ea4eB Y4kw== X-Gm-Message-State: ABy/qLY/sVFXkm0MGliS7leS94FJl3HSWCGX7Qe8QCkhivr7/0F5Mnk1 zNagbUPAVhBaQ4Wr3oVC7a0eeLZoi11vR8DUWS+0ZwRJMog= X-Google-Smtp-Source: APBJJlGpaYG8gBhU3O5i9dFlRstnD1qc20CyDeXni2pQMoMAICSQlqNsO5M0799NPsNtCOcg9vTNPvIWr7+5p/8VmSE= X-Received: by 2002:a05:6808:200b:b0:3a7:5724:8bfd with SMTP id q11-20020a056808200b00b003a757248bfdmr3912441oiw.1.1691047983426; Thu, 03 Aug 2023 00:33:03 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 From: Shrikanth Kamath Date: Thu, 3 Aug 2023 00:32:22 -0700 Message-ID: Subject: How to watch Active pagequeue transitions with DTrace in the vm layer To: freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="00000000000059b15c0601ffcacf" X-Spamd-Result: default: False [-2.43 / 15.00]; NEURAL_HAM_LONG(-0.94)[-0.944]; NEURAL_HAM_SHORT(-0.60)[-0.603]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_SPAM_MEDIUM(0.11)[0.112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::334:from]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4RGgYY4d5nz3Gyq X-Spamd-Bar: -- --00000000000059b15c0601ffcacf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable A background on the query, Trying to catch a memory =E2=80=9Cspike=E2=80=9D trigger using DTrace, refe= r here two =E2=80=9Ctop=E2=80=9D snapshots captured during a 2 minute window, last pid: 89900; load averages: 0.75, 0.91, 0.94 up 39+00:37:30 20:03:14 Mem: 5575M Active, 2152M Inact, 4731M Laundry, 3044M Wired, 1151M Buf, 382M Free Swap: 8192M Total, 1058M Used, 7134M Free, 12% Inuse PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12043 root 5 35 0 11G 9747M kqread 3 128.8H 23.34% app1 12051 root 1 20 0 3089M 2274M select 1 22:51 0.00% app2 last pid: 90442; load averages: 1.50, 1.12, 1.02 up 39+00:39:37 20:05:21 Mem: 8549M Active, 631M Inact, 3340M Laundry, 3159M Wired, 1252M Buf, 359M Free Swap: 8192M Total, 1894M Used, 6298M Free, 23% Inuse PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12043 root 5 24 0 11G 9445M kqread 2 128.8H 10.45% app1 12051 root 1 20 0 3089M 2173M select 3 22:51 0.00% app2 The spike is ~3G in Active pages, the two large applications have a combined resident size of ~12G. The resident size of the applications hasn=E2=80=99t changed between these 2 readings, however there is a tar arc= hive and gzip on a large directory during that window likely causing a reshuffle. If I count the page allocs and dequeue by execname with DTrace, I see tar/vmstat which probably alloc and quickly dequeue, along with a large dequeue being undertaken by bufdaemon and pagedaemon. fbt::vm_page_alloc*:entry { @cnt[execname] =3D count(); } fbt::vm_page_dequeue:entry { @dcnt[execname] =3D count(); } Page Alloc vmstat 20222 tar 21284 Page Dequeue vmstat 20114 bufdaemon 21402 tar 21635 pagedaemon 360387 Since the tar / vmstat will not hold the pages in Active, I need to find out what application had its pages queued in Active page queue. Is it possible that the system is just moving the LRU pages of these two large applications into the inactive queue prior to addressing memory pressure? Do these applications need to activate those pages later and hence it brings it back into the Active queue? How do I watch this in action by using DTrace? Will the following probe catch this trigger? fbt::vm_page_activate:entry { @cnt[execname, pid] =3D count(); } tick-10sec { printa(@cnt); printf("ACTIVE[%d] pages\n", `vm_dom[0].vmd_pagequeues[1].pq_cnt); } *** This system is running only one vmdomain (# sysctl vm.ndomains =E2=80= =93> vm.ndomains: 1). *** running release 12.1, on an amd64 kernel. The physical memory installed is 16G. Regards, --=20 Shrikanth R K --00000000000059b15c0601ffcacf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
A background on the query,
=

= Trying to catch a memory =E2=80=9Cspike=E2=80=9D trigger= using DTrace, refer here two =E2=80=9Ctop=E2=80=9D snapshots captured duri= ng a 2 minute window,=C2=A0


last pid: 8990= 0;=C2=A0 load averages:=C2=A0 0.75,=C2=A0 0.91,=C2=A0 0.94=C2=A0 up 39+00:3= 7:30=C2=A0 =C2=A0 20:03:14

Mem: 5575M Active, = 2152M Inact, 4731M Laundry, 3044M Wired, 1151M Buf, 382M Free

Swap: 8192M Total, 1058M Used, 7134M Free, 12% Inuse=

PID USERNAME=C2=A0 =C2=A0 THR PRI NICE =C2=A0 SIZE= =C2=A0 RES =C2=A0 STATE =C2=A0 C TIME=C2=A0 =C2=A0 WCPU COMMAND

<= p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><= span style=3D"font-size:13pt;font-family:"Arial Narrow",sans-seri= f;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-al= ternates:normal;font-variant-east-asian:normal;vertical-align:baseline;whit= e-space:pre-wrap">12043=C2=A0 =C2=A0 root =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 5=C2=A0 =C2=A0 35 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2= =A0 11G=C2=A0 9747M kqread=C2=A0 3 128.8H=C2=A0 23.34% app1

12051=C2=A0 =C2=A0 root =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 1=C2=A0 =C2=A0 20 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 3089M=C2=A0 22= 74M select =C2=A0 1=C2=A0 22:51 =C2=A0 0.00%=C2=A0 =C2=A0 app2

last pid: 90442;=C2=A0 load averages:=C2=A0 1.50,=C2= =A0 1.12,=C2=A0 1.02=C2=A0 up 39+00:39:37=C2=A0 =C2=A0 20:05:21

<= p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><= span style=3D"font-size:13pt;font-family:"Arial Narrow",sans-seri= f;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-al= ternates:normal;font-variant-east-asian:normal;vertical-align:baseline;whit= e-space:pre-wrap">Mem: 8549M Active, 631M Inact, 3340M Laundry, 3159M Wired= , 1252M Buf, 359M Free

Swap: 8192M Total, 1894= M Used, 6298M Free, 23% Inuse

PID =C2=A0 USERN= AME =C2=A0 THR PRI NICE =C2=A0 SIZE=C2=A0 =C2=A0 RES STATE =C2=A0 C TIME = =C2=A0 WCPU=C2=A0 COMMAND

12043 =C2=A0 =C2=A0 = =C2=A0 root=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 5=C2=A0 24=C2= =A0 =C2=A0 0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 11G=C2=A0 9445M kqread=C2=A0 2 128= .8H 10.45%=C2=A0 app1

12051 =C2=A0 =C2=A0 =C2= =A0 root=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1=C2=A0 20=C2=A0 = =C2=A0 0 =C2=A0 =C2=A0 3089M=C2=A0 2173M select =C2=A0 3=C2=A0 22:51 =C2=A0= 0.00%=C2=A0 =C2=A0 app2


The spike is ~3G = in Active pages, the two large applications have a combined resident size o= f ~12G. The resident size of the applications hasn=E2=80=99t changed betwee= n these 2 readings, however there is a tar archive and gzip on a large dire= ctory during that window likely causing a reshuffle. If I count the page al= locs and dequeue by execname with DTrace, I see tar/vmstat which probably a= lloc and quickly dequeue, along with a large dequeue being undertaken by bu= fdaemon and pagedaemon.


fbt::vm_page_alloc= *:entry

{

=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0@cnt[execname] =3D count();

}

fbt::vm_page_dequeue= :entry

{

=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0@dcnt[execname] =3D count();

}


Page Alloc

=C2=A0=C2=A0vmstat=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 20222=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0

=C2=A0=C2=A0tar=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 21284

Page Dequeue=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0

=C2=A0=C2=A0vmstat=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 20114

=C2= =A0=C2=A0bufdaemon =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 21402

= =C2=A0=C2=A0tar =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 21635

=C2=A0=C2=A0pagedaemon =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 3= 60387


Since the tar / vmstat will not hold= the pages in Active, I need to find out what application had its pages que= ued in Active page queue.


Is it possible t= hat the system is just moving the LRU pages of these two large applications= into the inactive queue prior to addressing memory pressure?=C2=A0 Do thes= e applications need to activate those pages later and hence it brings it ba= ck into the Active queue? How do I watch this in action by using DTrace? Wi= ll the following probe catch this trigger?


fbt::vm_page_activate:entry

{

@cnt[execname, pid] =3D count();

<= p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><= span style=3D"font-size:13pt;font-family:"Arial Narrow",sans-seri= f;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-al= ternates:normal;font-variant-east-asian:normal;vertical-align:baseline;whit= e-space:pre-wrap">}

tick-10sec

{

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0printa(@cnt);

=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0printf("ACTIVE[%d] pages\n", `vm_dom[0].v= md_pagequeues[1].pq_cnt);

}


*** This system is running only one vmdomain (# sysctl vm.nd= omains =E2=80=93>=C2=A0 vm.ndomains: 1).

*= ** running release 12.1, on an amd64 kernel. The physical memory installed = is 16G.


Regards,
--
Shrikanth R K
--00000000000059b15c0601ffcacf--