From nobody Fri Jul 28 14:25:41 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RCl2H24LLz4pFVg; Sat, 29 Jul 2023 12:59:19 +0000 (UTC) (envelope-from owner-freebsd-hackers@freebsd.org) Received: from mail5.nevz.com (mail5.nevz.com [91.240.220.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail5.nevz.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RCl2G4dSFz3m1X; Sat, 29 Jul 2023 12:59:18 +0000 (UTC) (envelope-from owner-freebsd-hackers@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from Srv054.nevz.com (172.17.200.54) by Srv054.nevz.com (172.17.200.54) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.23; Sat, 29 Jul 2023 15:59:09 +0300 Received: from SRV032.nevz.com (172.17.200.32) by Srv054.nevz.com (172.17.200.54) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.1.2507.23 via Frontend Transport; Sat, 29 Jul 2023 15:59:09 +0300 Received: from Pickup by Srv032.nevz.com with Microsoft SMTP Server id 14.3.498.0; Sat, 29 Jul 2023 12:59:05 +0000 x-sender: freebsd-hackers+bounces-2417-kozarenkoas=nevz.com@FreeBSD.org x-receiver: spam@vSrv021.nevz.com X-Virus-Scanned: amavisd-new at nevz.com X-Authentication-Warning: mailgate.nevz.com: Host [172.16.200.14] claimed to be tmh-chpt003 X-MTA-CheckPoint: {64C3D034-0-CC810AC-51FA} DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1690554405; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=jDQ4feONKUpE2iqLM6PvGeXjFbd+1fYJjBsm1yOJSvY=; b=iAU3wmjqo2Wi2e/zDMyWC7PoKg9P3Q5d3Np18rzBhjn+e2JhD9ldot6IkoxU45HlA5PoX2 9lNfweJ/b2zjEmOZy9zVe22HVB5BdrJEnFU5qlpgUd2LGIFt65Ss7IMmZZcYu7JI8xmG/B E88MnjK/1HYpFKO10KrV7VZ7Z7to1kOPHacnzucdW0pl1ihPosBcFZk8bl9zZ67ZXQR6Ij MeFTXP2H7kPcKPaNLkoOhQjqp6v7rVGh8gj/0Yi5jWAUjpnZOLHscG3I2e+HfaDpGAl5RC TgVHIDug5WV1ijeE8jbcV42g9h7SYL7tY9TgXWTfy4NLDRGNSLXsFJeUG5OCGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1690554405; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=jDQ4feONKUpE2iqLM6PvGeXjFbd+1fYJjBsm1yOJSvY=; b=LoaqOe1p28zxkeLgk81Xtc9xaOoUDHHAZNqk1UUIThfNdFESC0HkFQ2QMyc+X6YW4ZMx6q JBbtOP/ciSL0SDLqEiRqBhtaKR8jZAdXJHi/+QndyF3FtlHHnzYZ2LyNXi2wV6PKrc1o6n CrPtPB/OwCc8qyJchKMs2E4mWVIc2uLoSOHxyaTzm6hoTj3xyncnGyB371uflzDzmy+Gho razMX2N1aghyxeCqG0aHfAudjORsgJW5wvEFf5Hq29l1FF0GBhsCucITT41p6Kz2g/5SHo iwFlCv1WB55c05tsi0ahJfxSynv5UqsooIjbBvsZZiNj+C97S7Gfmuelk3gkOA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1690554405; a=rsa-sha256; cv=none; b=BVPOqUrfQQMgHxs8oiToFrHHskx0aWmi9W5PTxfKCSszZ1OCejyqbBlJIYS8pELR7HTvpx iQoEQG6ueiX2imZoBqSHpZlmL3rTfyZjk3bPeZBwDUBFh0E7Uev0m9l0LAxUQIjZTsESjh t8Ch5doNA5An4lxjZt9RwmXUz9ox2+pS0jC6QqQ4oGLafoAOR0Kv99ICelOSkZas+TLSIz DPl340pDtAXF+0yH6axKXW5Ggg+dPf1izUr+7hlcwYEyv6rTodH4dBDojqylrXP0wcrr6m hpR2Onwhqfn24U63OTgLO9uANAJ8dK8e0zP52dD+h5eF7FoTwSPSF5W5Fz9Btw== X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1690554378; x=1691159178; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=jDQ4feONKUpE2iqLM6PvGeXjFbd+1fYJjBsm1yOJSvY=; b=OrFiUyez+agBg1MDzC1nlcRLNsjaAJS188hmfPG+vFsioanh4nsvl98XVPUBvSltNx jR0VOxXitHWNAlEMaqp4LMb4dCOmoovZWBXP0wb7sJPv6xl0BFteDhe/5W7hWAnr0WO6 4+XoHtIR8PRe1ss99ynkLD2VsxbJ6UaoxO464fqM0QeacvEOlzd8OHYRrXm805wzQ4UC NfeiGc6fW712ZaYcQwdOp8T2FMsllwd7w3IR8rQBBhdBU+cgF8tPWznIXazFRQTS+C4m 9jQUKTBsI18kQc++G5FtFt+L2O9N1nc87cnaDVrRWDvTllNoySIwGRW69ZlJwKl7iQD0 3zzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690554378; x=1691159178; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=jDQ4feONKUpE2iqLM6PvGeXjFbd+1fYJjBsm1yOJSvY=; b=QwGaEBBx2x36fyirb76YKk5MYk5pCxLHHAMaZV3vmDrNo3BfAYp6pI2GdiiF1m5sfK p7tXA13iLm80ekxTutEiMy9ha1djchdGY++cG+Qj85nchbtcYEH4BC0stN0OIXO73MIA V75u6yM6SCOjDk+gPvTnftCnQkYNO2JZX3Y65KUHZ+vmsYwChVAwVRuzDOW8atkk4MAL wIoTKCYJwLO/CDyjhdsxnoA4PrMBwYF5Vj99k905EyQN/2jqlRnzY8kmUcQo7QR0aqcN 3iM1/f0ClXxAQx6Gl/lqlch94oL43KA5pp/riN5bLGYb54WuIVT+qIVJzR3MCZn45rK/ BMMg== X-Gm-Message-State: ABy/qLYb31o/Z3CK7q0rPdRM109yFQuglWxaMlPSUin47CeoxKIEBKrM NnSA6f4eVYZ0dphi9km2VCeOCZ0PBVSgikVv1GSoJ8T/jfM= X-Google-Smtp-Source: APBJJlGYmvcIJRYGvrvIAbMLrDDoOH16D0c+esgiLUQ2RFdE2KWSMQsQFWwC6szeUaOEknOF8nLovqeDB7V3z8GoMQI= X-Received: by 2002:a17:906:3f16:b0:99b:d977:c00c with SMTP id c22-20020a1709063f1600b0099bd977c00cmr2287334ejj.45.1690554378133; Fri, 28 Jul 2023 07:26:18 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <5b3e32fa-f5cf-b965-a3f6-2788a1c6ef37@FreeBSD.org> In-Reply-To: <5b3e32fa-f5cf-b965-a3f6-2788a1c6ef37@FreeBSD.org> From: Mario Marietto Date: Fri, 28 Jul 2023 16:25:41 +0200 Message-ID: Subject: 5.7 points Re: Virtual GPU for FreeBSD as guest virtualized with qemu / kvm -- best alternative ? Cirrus ? To: Ronald Klop CC: , FreeBSD virtualization , FreeBSD Mailing List , freebsd-hackers , Content-Type: multipart/related; boundary="0000000000002ed3be06018cdddc" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-FE-Attachment-Name: unnamed.png X-FEAS-DKIM: Valid X-FE-Last-Public-Client-IP: 96.47.72.81 X-FE-Envelope-From: freebsd-hackers+bounces-2417-kozarenkoas=nevz.com@freebsd.org X-FE-Policy-ID: 1:1:1:nevz.com X-Greylist: internal networks OS Linux 3.11 and newer X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.4 (mailgate.nevz.com [172.17.206.231]); Fri, 28 Jul 2023 17:27:11 +0300 (MSK) X-Spam-Flag: YES X-Spam-Status: Yes, score=5.7 required=5.0 tests=ARC_SIGNED=0.001, ARC_VALID=0.001,BAYES_00=-1.9,DC_PNG_UNO_LARGO=0.001,DKIM_SIGNED=0.1, DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1,DKIM_VALID_EF=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.25,FREEMAIL_FROM=0.001, FSL_HELO_NON_FQDN_1=0.001,HEADER_FROM_DIFFERENT_DOMAINS=5, HTML_MESSAGE=1,MAILING_LIST_MULTI=-1,SUSPICIOUS_RECIPS=2.51, T_SCC_BODY_TEXT_LINE=-0.01 mailgate.nevz.com; whitelist Reported 0 times, welcomelisted 0 times. autolearn=no autolearn_force=no version=4.0.0 X-Spam-Orig-To: ORCPT=rfc822;kozarenkoas@nevz.com X-Spam-Report: * 0.0 ARC_VALID Message has a valid ARC signature * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's * domain * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature * 0.0 ARC_SIGNED Message has a ARC signature * -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from * envelope-from domain * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 FSL_HELO_NON_FQDN_1 No description available. * 2.5 SUSPICIOUS_RECIPS Similar addresses in recipient list * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider * [marietto2008(at)gmail.com] * 5.0 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level mail * domains are different * 1.0 HTML_MESSAGE BODY: HTML included in message * -0.0 T_SCC_BODY_TEXT_LINE No description available. * 0.2 FREEMAIL_FORGED_FROMDOMAIN 2nd level domains in From and * EnvelopeFrom freemail headers are different * 0.0 DC_PNG_UNO_LARGO Message contains a single large png image * -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list * manager X-Spam-Level: ***** X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on mailgate.nevz.com X-OriginalArrivalTime: 28 Jul 2023 14:27:15.0349 (UTC) FILETIME=[9B2B2450:01D9C15F] X-Rspamd-Queue-Id: 4RCl2G4dSFz3m1X X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:59416, ipnet:91.240.220.0/24, country:RU] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated --0000000000002ed3be06018cdddc Content-Type: multipart/alternative; boundary="0000000000002ed3bd06018cdddb" --0000000000002ed3bd06018cdddb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks. it does not work. This is the error I get : (on the attached picture,you can read "segmentation fault"... [image: unnamed.png] On Fri, Jul 28, 2023 at 4:18=E2=80=AFPM Ronald Klop wr= ote: > On 7/23/23 16:27, Mario Marietto wrote: > > Hello to everyone. > > > > > > I would like to virtualize FreeBSD on Windows 11 with qemu-kvm (for > Windows). I've end up with the following parameters which are working : > > > > > > |I:\OS\qemu\Linux\qemu\qemu-system-x86_64w.exe -accel whpx -machine q35 > \ -cpu > Westmere,vendor=3DGenuineIntel,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xs= ave,+xsaveopt,check > \ -m 8G -vga qxl -audiodev dsound,id=3Dsnd0 -device ich9-intel-hda -devic= e \ > hda-duplex,audiodev=3Dsnd0 \ -hda > "I:\OS\ISO\FreeBSD\FreeBSD-13.2-RELEASE-amd64-disc1.iso" \ -hdb > "I:\OS\qemu\Linux\FreeBSD.img" -rtc base=3Dlocaltime \ -device > nec-usb-xhci,id=3Dxhci -device usb-tablet -device usb-kbd \ -global > nec-usb-xhci.msi=3Doff -smbios type=3D2 -nodefaults -netdev user,id=3Dnet= 0 \ > -device virtio-net-pci,netdev=3Dnet0,id=3Dnet0,mac=3D52:54:00:11:22:33 \ = -device > ich9-ahci,id=3Dsata -bios "I:\OS\qemu\Linux\OSX-KVM-master\OVMF_combined.= fd"| > > > > > > Now I'm trying to configure Xorg and the xfce4 desktop environment. > Since I'm using qemu,there are a lot of display devices available to > try,but I don't know which one is good for FreeBSD. In the website below > there is a good list of all options available : > > > > > > https://www.kraxel.org/blog/2019/09/display-devices-in-qemu/ < > https://www.kraxel.org/blog/2019/09/display-devices-in-qemu/> > > > > > > On FreeBSD I have installed the package drm-kmod and it suggests to me > to add to the rc.conf file one of these parameters : > > > > > > 1. > > > > for amdgpu : kld_list=3D"amdgpu" > > > > 2. > > > > for intel : kld_list=3D"i915kms" > > > > 3. > > > > for radeonkms : kld_list=3D"radeonkms" > > > > > > qemu does not cover any of those. I don't know which option is decent > for my case. I would like to try with the cirrus display driver. Maybe th= is > one : > > > > > > https://www.freshports.org/x11-drivers/xf86-video-cirrus/ < > https://www.freshports.org/x11-drivers/xf86-video-cirrus/> > > > > > > |So,I have installed this package : pkg install xf86-video-cirrus > > > > | > > > > I've rebooted and I have changed |-vga qxl to -vga cirrus| > > > > FreeBSD recognized it at 0:1:0 > > > > > > 2023-07-23 15_21_33-QEMU.png > > > > I have created xorg.conf with this content : > > > > |Section "Device" Identifier "Device0" Driver "cirrus" BusID "PCI:0:1:0= " > Screen 0 EndSection| > > > > > > error : no screens found. > > This is the reason why it does not work : > > > > > > 2023-07-23 15 49 44.png > > > > but the module seems to be there : > > > > |cd /usr/local/lib/xorg/modules/drivers/ ls *cirrus_drv.so* ; > modesetting_drv.so ; scfb_drv.so ; vesa_drv.so ;| > > > > > > What should I do ? > > > > -- > > Mario. > > > A https://www.freshports.org/x11-drivers/xf86-video-qxl/ driver exists. > It didn't work on my qemu on Mac/ARM. Maybe it works for you. > > Ronald. > > --=20 Mario. --0000000000002ed3bd06018cdddb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thanks. it does not work. This is the error I get : (= on the attached picture,you can read "segmentation fault"...
<= /div>

3D"unnamed.png"

On Fri, Jul 28, 2023 at 4:18=E2=80= =AFPM Ronald Klop <ronald@freebsd.= org> wrote:
On 7/23/23 16:27, Mario Marietto wrote:
> Hello to everyone.
>
>
> I would like to virtualize FreeBSD on Windows 11 with qemu-kvm (for Wi= ndows). I've end up with the following parameters which are working : >
>
> |I:\OS\qemu\Linux\qemu\qemu-system-x86_64w.exe -accel whpx -machine q3= 5 \ -cpu Westmere,vendor=3DGenuineIntel,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+= aes,+xsave,+xsaveopt,check \ -m 8G -vga qxl -audiodev dsound,id=3Dsnd0 -dev= ice ich9-intel-hda -device \ hda-duplex,audiodev=3Dsnd0 \ -hda "I:\OS\= ISO\FreeBSD\FreeBSD-13.2-RELEASE-amd64-disc1.iso" \ -hdb "I:\OS\q= emu\Linux\FreeBSD.img" -rtc base=3Dlocaltime \ -device nec-usb-xhci,id= =3Dxhci -device usb-tablet -device usb-kbd \ -global nec-usb-xhci.msi=3Doff= -smbios type=3D2 -nodefaults -netdev user,id=3Dnet0 \ -device virtio-net-p= ci,netdev=3Dnet0,id=3Dnet0,mac=3D52:54:00:11:22:33 \ -device ich9-ahci,id= =3Dsata -bios "I:\OS\qemu\Linux\OSX-KVM-master\OVMF_combined.fd"|=
>
>
> Now I'm trying to configure Xorg and the xfce4 desktop environment= . Since I'm using qemu,there are a lot of display devices available to = try,but I don't know which one is good for FreeBSD. In the website belo= w there is a good list of all options available :
>
>
> https://www.kraxel.org/blog/2019/09= /display-devices-in-qemu/ <https:= //www.kraxel.org/blog/2019/09/display-devices-in-qemu/>
>
>
> On FreeBSD I have installed the package drm-kmod and it suggests to me= to add to the rc.conf file one of these parameters :
>
>
>=C2=A0 1.
>
>=C2=A0 =C2=A0 =C2=A0for amdgpu : kld_list=3D"amdgpu"
>
>=C2=A0 2.
>
>=C2=A0 =C2=A0 =C2=A0for intel : kld_list=3D"i915kms"
>
>=C2=A0 3.
>
>=C2=A0 =C2=A0 =C2=A0for radeonkms : kld_list=3D"radeonkms" >
>
> qemu does not cover any of those. I don't know which option is dec= ent for my case. I would like to try with the cirrus display driver. Maybe = this one :
>
>
> https://www.freshports.org/x11-drivers= /xf86-video-cirrus/ <https://www.fre= shports.org/x11-drivers/xf86-video-cirrus/>
>
>
> |So,I have installed this package : pkg install xf86-video-cirrus
>
> |
>
> I've rebooted and I have changed |-vga qxl to -vga cirrus|
>
> FreeBSD recognized it at 0:1:0
>
>
> 2023-07-23 15_21_33-QEMU.png
>
> I have created xorg.conf with this content :
>
> |Section "Device" Identifier "Device0" Driver &quo= t;cirrus" BusID "PCI:0:1:0" Screen 0 EndSection|
>
>
> error : no screens found.
> This is the reason why it does not work :
>
>
> 2023-07-23 15 49 44.png
>
> but the module seems to be there :
>
> |cd /usr/local/lib/xorg/modules/drivers/ ls *cirrus_drv.so* ; modesett= ing_drv.so ; scfb_drv.so ; vesa_drv.so ;|
>
>
> What should I do ?
>
> --
> Mario.


A https://www.freshports.org/x11-drivers/xf86-= video-qxl/ driver exists. It didn't work on my qemu on Mac/ARM. May= be it works for you.

Ronald.



--
Mario.
--0000000000002ed3bd06018cdddb-- --0000000000002ed3be06018cdddc Content-Type: image/png; name="unnamed.png" Content-Disposition: inline; filename="unnamed.png" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: ii_lkmodv4w0 iVBORw0KGgoAAAANSUhEUgAABPoAAAFaCAIAAAAIPKSyAAAAA3NCSVQICAjb4U/gAAAgAElEQVR4 nO3dUZajLLcA0Oq7ej4ZQ0buGGpG9yGr8+VPlACCHHHvp+4S8YioEBT/3G6339/fHzpYliWx9H6/ HxYJHO9q9f9q+1tK+QAAx/u/0QFclLYdV3a1+n+1/S2lfACAXm632+gQAAAAoDGjuwAAAExIdxcA AIAJ6e4CAAAwId1dAAAAJqS7CzDGsixvn+f5/AsA8bmeQ1i6uwAAAEzob3rx43epsR9F7BRDOtvm G41QkvP5/N30sAPa1tfwgsefdurg4zt78eZcih/OuI/74z/7zWh1czvvv6W7sGeXz35+XUd+e6DJ hlQJOJHU6G6oZzBCBUMEqgTA1ZRe+d0prsBRBhI2R3ef147hv2Dd7/dHMMuyDA+GIOLUTxjo7OMM X+N/LDpvW3Zn/LMe38dtffWenj/anx9AtbOX/0VoDwBp66O70a4dzzDO2+ihhyD1E4DejOuSoD0A bEm9uxvq2vEc4z2pUIV5QWcv/7PHz6rVUa/81c8+9HT2+Hs7e/mk418d4M3Z5dLGQHXj4ezlz8F2 Xs+Bfla6u8F7lR5pBoCrKb31ayoA8JMY3f16n8jpFX/+apuffnXp8K74VgBf55AsSpaff91a+Ur3 Nz99ReSfq+SUbdHMqKPKM3OVzKlK9x+FHse39GDl5J+/vwfPBHuAPXu0tbRf+ZdGWKdffU4kTuS/ R075xDkft9bNuac/f8IOdZad8fzSHvgaUpHhDU6gFd/dLZC49vW+LK7mP2SjiUUNy2fIbWZgeQ7Z ysDj2ySehpvm5/zl73qVsyhO/lfT+/zSHmhLJYeZfPnubsLVHitKTN+1JCeOLpqZM/Er5uo7TulV 9ijd37ryyd/ft8T54wCZ5T+wPNP5F9Wf9FYyUzY5vqUHq67+5O/vZJqPgx1c/sPjr9jf9BDW1vW/ TubjBqtpBp6PRZt45v8Y4N3a3BCZ8adPitU/djq/tAe2Vq/TuzyBgxndLbPVoX04cqNfF/XbdGk8 J/plZEh5DtzKkOPbMJ7mAVxc3fUtTvm7Xm39MWb+5/UshLeeT7qb1Pv6pj3QlqoO06gf3eXK3AbC GnJopm+RhKrwzYcy9qgII1T8q8bGdvbyiR9/Wmb893+TiSz/++5xzrr5TlGMpwgSuDLd3TJtn1iD tq5WOa+2v72VXt+uVv4eYuTVa4/39Y8N82+VFcCV1Xd3c278q+9yZKaPZvXG9rr08IhgL8330+k0 dHbY9e28Q3/HnCxF5RPw/C09vq/pHy/xnmto/f6/H4w4Y8UGmJ7R3QL37S8hDb9JA+zh+pZg6hoA OKnN7u7X9s3x8w1EaFIUzRwIwamx59J7aLT39e28Q7sPvSOvGFrsGU6xPUO7Py+zNI/ar4r6+TlV VbSDAsDKzMzBL9YBwwsYEkATrm+w6nXMf2uuZgCGS32IKNRVO1QwvH4pEaavDxF28OxDoyeKf8jh Pnv57BzafXib4vhIdfH//O/o9Nuis9hZ4SNcHoucLmBgp/XubrSrdr9v0Ke/CP/2x61L5MBS6r3p 0v2NVj6lhpTnQIl3NYtOjT3bapV/qxhmkr+/Ma9vpXpfr6IVRen5m7Pu1zTRCmGIrTZJuu109uub 9gBwUn9ut9vv7+/qsn6dzCIH93WfPjdXmn5rrcQLculM3pZ+vRbvLLGu5VOxv0UJElHlH6nMFTPV 5d+k/jSJp67+J9bdk/8B+7tzc21FO75NhvLyV0xEkkiZXqtt+q3A4py/X9eNUx+2lm6FVFoltnLu Gv/Zr29ztAfy0/duDwBHSj3MHOpk7hFMIs/VRaXpe+u90bOXT6mB5TlE6Vx0e45vzr73rj/Ryr+3 hrMJnqLoel+vohXCnrkkI5yPV3P265v2AHBit9ttdAi059WUtpTn3BxfYFaub8DFpUZ3AQAA4KR0 dyfkd9y2lOfcHF9gVq5vAH9HB0C9PVMH8Ul5zs3xBWbl+gawxejunNzb2lKec3N8gVm5vgFXZ6oq AAAA5mN0FwAAgAnp7gIAADAh3V0AAAAmpLsLAADAhHR3AYhoWZa3z6t8/gUA3C9I0N0FAABgQn/T ix+/i4z9aFunGNLZNt9ohJKchsLs5PN30MNOkLa+hhc8/rRTB3+8sxdXzq3q4Yz7uD/+Jsf37JUk YXXXdrZ/SotrT/FOfGgmk99+aLIhVYIiqdHdUM8AhAoG5uMUAyCt9E7hznIFjjLBbY7uPuvu8F9Q 7vf7I5hlWYYHA1OKc75DQ2cfB/ga/2PReduaO+M/+/Htbat8Hs2q1TZV/tME+QFUc3xPQfuB+NZH d6PV3WcY572pQ3xBzncAojGuS4L2A5Gl3t0NVXefY7wnFaowYaez1+ezx38Rq6NS+auffWjo7PH3 pnzS0uWzOsCbU6SljbHqxpvjS5Gd9wvmttLdDd6r9EgzAMCRSptemmpAEJuju1+vUzm94s9fDfPT ry4d3hXfCuDrHIZFyfLzr1srX/7+1s30WJr/1l9Kyyc/knT+pUrrT9f49xfpzqtEv/qZuUrm1KP7 j0KP49uj/ufv78Ez21fYE2Gr61Vp/cmPsE6/+plInMh/j5zyibO/x1xPtraYiPN1gDfCaft0xvM3 TnusNP1h7Yciwxv8zMR3dwskzr3ep+Vq/kM22nDTJ8p/SDzR4i81sH4O2crA49sknoabntLZy9P1 J2dR7/T9riek9T5/o7XHJj5/ocKX7+4mXO2xlue5t/XD2NZT1kUzTyZ+RVvW3rFJr7JH9f52yr/0 /aK6+N8mRXs9dgfv7874c9KXFunrWl8r2MD6mc6/6HxMbyUzZZPje0z9z9/f4JqPUx1cnsPjr9jf 9BDTzuvnap45T0sdc/3MzL/f9WQ1Teb1fFmWaCd+ZvzpQl79Y6fzN1p7LGz7IVPv8uSCjO6W2erQ Phy50a+L+m264f6Oyn9/JnVK44kWf6kh9XPgVoYc34bxNA9gMnXXqzjl6frzU35/ab6//a4n1/Es hLeeT7qb1Pt6OKo91ip9fibHUNVpqH50F2jFZT2sIYdGC2OP5kMNe1SEESr+VWNji18+CQeEfery +cmO//5vMpflf989zlk33ymK8RRBwli6u2XaPpEVX+/9vVp5ci5Xq5xX299Spderq5VntIcMD7h/ FaW/Wn3o7bXH+/rHhvm3ygoYq767m3OhX32XIDN9NKsX1telh0fUV+/9PaY8ozW/mJv6Nlynoa3D rv/nHZo7pvJXDP2tLm0VSVcVmyitP2+zVAz/Abo0/vv/frDjjCcOcADv7hZIXEmnbOb23t+rlSdw Xq5XCa/PkX4aElLX4xVwfwHYsjm6+/VHvuPfd4/QpFjdix4zTwbRe3+H5A+dqG9j9R4a7X29Ou/Q 7kPvyCuG/rYyOcX9pTT/PUO7P/9GSge2ZCrq/+dUVec9fYB+VkZ3g18sAoYXMKSuot3jAUZxvToX x2sab2Psb38EeEo9zBzqqhEqGGjr9cuHoD5UOPvQ6IniH1I/T1Q+Q+wc2n14m+L4SHXx//zv6PTb orPYeUKd7n5xuoCZwHp3N9pVo8f88m855/xx6xQdWEr9Nl23v/nx7C/PnGRbaUZdbZuUZ5CzMkfv UKMVRWl9G1L/GxZatPIvdeT1KoLe1584RXHM8Rp1fdNb2GoTptuuU9YH7QfI8ed2u/3+/q4u69fJ LHJwX/fpc3Ol6bfWSrxQlM7kbenXa8HOEmtbPp+r1JVnYt095fNWwun/1ula3yrqT1GCRFT5NT9z xUx1+Tc5H5vEM7b+F9Wf/BgSm+gq2vEqjaf6CtPqfNx/vzvX+bj/+rnn/pKja/5194utkEqr3FbO XeOPfD3M2dwc7Yf89KHuL0wg9TBzqMrUI5hEnquLStP3NvAd2tLyaZJ/6bbSaY4/ZGevb6Wu9o53 aX0bWP+bFF208i915PUqgt7Xn2iFcPb6H+3+dXYT1wftB/judruNDoH2oj3sFC0exlIf5ub4Ajy4 HsJwvrsLAADAhHR3JxTtd8Ro8TCW+jA3xxfgwfUQIvg7OgDq7Znqpodo8TCW+jA3xxfgwfUQIjO6 O6do19Zo8TCW+jA3xxfgwfUQxjNVFQAAAPMxugsAAMCEdHcBAACYkO4uAAAAE9LdBQAAYEK6uwAA AExIdxcAAIAJ/R0dQK7HJ7z7fb6sd/7X9Pnh9eAlvPWl+LZhx6lsx+wvAAAMYXSXXra6UmElAj7d vuS42v4CAHA1pxnd5Vye/aWzjBMmAn4sWpblLPuS42r7CwDABRndpaPT9ZdWAz7dXuS72v4CAHAp pxnd7d0E18QHAACYidFd+OJqL7JebX8BAJjVf6O7r7PF5s/XWjqza8VMsK+rZCbbsrp6k/xL9/fr 5jKVHql0nK9LK+rDZ7J02SYKp1X92T/zcKu+X2nhpLfee4pyAACYwMrobn57t3Rm14lngi3d334b bb7pIYemSf3ZXz5jq+Xx58vZT0MAAHj1/u5u/oS6pTO7Vs8Emx5gfEuWDjWxYmZDPzE+nL+/+ZtL OGZm3aIJlrcGh3PSf250Z/3ZXz6vOew/XqWFc/zMyW33FwAAhlt/dze/GV06s+tW+ofMjRY54Is4 pfvbe9PNy/Pgebwq6k+r9E9BPqR02MzJQfYXAAAaWpmZeab2rkb8fqcoun7jnA2zDetq+wsAwEW8 d3d7t3ebP4GZ3tbjHxM34nuX5wH1oWv+Fa5QbV5dbX8BALiOvd/dze+uPF8IXF2leVN7+kb8weXZ wwF93dJNTF9t3lxtfwEAuJS93d0iiSlw2o5SXqQRf1h59tB7Kq89ghddc1fbXwAALmJvd7e0oZw/ s3G1i/R1Hw4oz66GTOXVfJVTu9r+AgBwHeszMx/MuG5bV973na5WdFfbXwAALiVEd7cVfd3LWpYl 4KxXAADAQHu7u4l3R98WLf/kZ1IXSai+br8+WF15RusTVk8llZlJfv2sy3+PnGx7x3Pk/gIAwMH+ 3G6339/fn39t3KK+4tdm8Vtu6fRbm/5cq3pyo8wVE8nSQRbtbyK3TKXlWRRPRX343FZ69d7155j6 WSdnPu3e8Ry5vwAAcLxdo7vpBvHn0kT6KdvWA+dhWl0UrZCbzHOWWBS5fuZk2Dueq52PAABczu12 Gx3CFXnXFAAAoKuppqoCAACAB93dAYzrAgAA9PZ3dAAzMxUQAADAKEZ3x9DXBQAA6MtUVQAAAMzH 6C4AAAAT0t0FAABgQrq7AAAATEh3FwAAgAnp7gIAADAh3V0AAAAm9Hd0APCfZVne/jLZB4o/d/Ch 1W6W5n/29KW28t/aRLT4X7cS4dRQn1ulT9TMurVOVJ8BoCuju0SR0+A7tcQONtn30vzPnr5UaSbR 4o9GfW6bvpT6DAA5/txut9/f39FhcHXP9tOs4waJHWyy76X5nz19qdJMosX/meHYM0V97pE+Pcqa mX8ih37xA0BYRncJZPr20+oONtzr0vzPnr5UaVbR4o9GfW6bvpT6DABf6e4CAAAwIVNVwWBDXoQz RFPNi4tp6nNbvZ9dV58BmNt/3d30Pa/3zJOt8n/8PWet0nel6nQtn9JpOYviqUtf5DPz17+8baL0 +G5tojTlVvpWtrZ7TP08tYr633WjXRWdLIm10un3U5/rjNp3fV0Appf7MHPvmRsb5h/n/t27fErT zzEzZ/6mB5b/2Gx7uN/vJ+qNOF4/I87fE5XPWepz1x8CTnS8AKDa+8PMz9vq52DasixFMze+pT8m /ziTRu4sn5z06aGSneVZF3+R/MH2nKjyUzapn/u9RpgYEysaiC5Nnw6sU/p0VnuG6F/TN8lqdd2G xytf6clywPmb2KL6/MyqU31IZxW/PgPAYbLe3U3cDrdmbiy6IzbPf3hf96nJzJYDy7PJ8W0uvwB7 189qX/vtz41m/gzxmn4rw3Qkmem3Vt/6pSYzn+r4E1nl+xpn0fHaSlAUUhOHnb/q8+fm9teH6qHd WeszAFSYcKoqt+EKJyq0E4W6JXOM+rOHEOfhhYdEPF+by0MUPT3xlmBrddTnUdRnAPhqtu5utPt3 8ycGP/Pvl3lA0Y5vhaK25mojO9HGXV2abv5WP+yaP0CUoy7+Vs8Jp5+n3QosM1mnh4eDUJ9LcyuK p6LOqM8A8Gpvd/dq3a186dGAJg2FAwrf8W2rYlzlrfH6dQDtaw6lW1yVP6CXXj0n/VY8R9b//EdY V/8463mkPr+tnpN+YH1QnwG4oNlGd0NJNAv2/zT+9R2zPZnT1dlHRSI8DHlk/T/78ert7OUToT6/ qR7azRFnNwHgAHu7u26caXXvVu3MvyHHt7nSIn3ruQ1/hjBU3+CAGCLsZmTq87lcZDcB4Cn3u7s0 pMFxWdV9g/v9/ly3ydDlsiwV+egb8Ep9PpeL7CYAvNLd5eentrFIV59t8bY9hP3xQD71GQA43t7u buLd1Fa/2edvtFX+rXLeKoTSjeZMs7knq4r8z9I9HjXF0Zb8eLba4ukeQpPjWxRPhSbxfF3UL/+G 6ffIyXbI9VN9roin4aZPWp8BoJM/t9vt9/f352NujPR/X/+Y8Jq+ef6riYvaLkXxV6iIvyiY0lWK 4qkLqdrXw9f8+KbrT079LNWwPu8/X+riSTsynpyQQsXTdghxdVs797dJDOlg1OdE+s91M78V1Cme I+szAPSza3S37psNTfJvcq8dOM9T6aa3PgjRNZ7ex7e33vWn1MD6XHp8h5wabetn1/yHl2dOhtHq W+/8z16fm2y0YTzRrp8AUOl2u40OIaLjH9Y9y+PBAAAAp2CqKgAAACaku7vCKCsAAMDZ/R0dwEim 4gAAAJiV0d11+roAAADnZqoqAAAA5mN0FwAAgAnp7gIAADAh3V0AAAAmpLsLAADAhHR3AQAAmJDu LgAAABP6OzoA+M+yLG9/mewDyJ87+NBqN0vzP3v6Ulv5b20iWvyvW4lwaqjPbdOXmqM+A0BXRneJ ItF0m0NiB5vse2n+Z09fqjSTaPFHoz63TV9KfQaAHH9ut9vv7+/oMLi6Z/tp1nGDxA422ffS/M+e vlRpJtHi/8xw7JmiPrdNX2qm+gwAXRndJZDp20+rO9hwr0vzP3v6UqVZRYs/GvW5bfpS6jMAfKW7 CwAAwIRMVQWDDXkRzhBNtYbvTE5JfT4X9RmAuf3X3U3fw/bP3HhM/o+/56yVfv+t1dtxXcundFrO onjq0hf5zPz1L2+bKD2+W5soTbmVvpWt7R5TP0+tov533WiT9Pn5JE6Wr1tXnwO6VH0GgMPkPszc e+bGhvnHuR/3Lp/S9HPMzJm/6YHlPzbbHu73+4l6I0GO19jje/z5qz53EuR4nej4AsDT+8PMzxbA 52DasixFMze+pT8m/ziTRu4sn5z06aGSneVZF3+R/MH2nKjyUzapn/u9RpgYEysaiC5Nnw6sU/p0 VnuG6F/TN8lqdd3E8dqTPq30ZDng/E1sUX1+ZqU+A8BYWe/uJm5vWzM3Ft0Om+c/vK/71GRmy4Hl 2eT4NpdfgL3rZ7Wv/fbnRjN/hnhNv5VhOpLM9Furb/1Sk5lPdfyJrPJ9jbP0d7Qgv7sddv6qz5+b U58BIIIJp6pyP65wokI7UahbMtuOnz2EaI3ORDxfm/tDFD098ZZga/X96c9OfR5FfQaAr2br7ka7 Hzd/YvAz/36ZBxTt+FYoajuuNrITbdzVpenmb/XDrvkDejnq4m9S/9Nx6hukqc+luanPAHCkvd3d q3W38qVHA5o0HQ4ofMe3rYq241vj9esA2tccSre4Kn9AL716TvqteI6s//oGq9Tnt9Vz0qvPAHCk 2UZ3Q0k0s/aP+n59x2xP5nR19rZjhEbwkfW/dDfPfnxLnX1/1ee26QEglL3dXTfCtLp3q3bm35Dj 21xpkb61dHs/Hv9VhL7B0wEx6Bukqc8Nqc8A0Fzud3dpSAPisqr7Bvf7/bluk6GeZVkq8gnVNziA vkGa+nwu6jMAF6S7y89PbWORrj7b4m17CPvjgXzqMwBwvL3d3cS7qa1+s8/faKv8W+W8VQilG82Z ZnNPVhX5n6V7PGpKmC358Wy1xdM9hCbHtyieCk3i+bqoX/4N0++Rk+2Q66f6XBHP10X98g9SnwGg kz+32+339/fn3z3srfGx9d/XPya8pm+e/2riorZLUfwVKuIvCqZ0laJ46kKq9vXwNT++6fqTUz9L NazP+8+XunjSjownJ6RQ8bQdQlzd1s79bRJDOhj1uTp9tHgMiQNwFrtGd+u+2dAk/yb32oHzPDV5 h6rhi1gV+cdv7vSuP6UG1ufS4zvk1GhbP7vmP7w8czKMVt96568+HxZP/Is/APzndruNDiGi4x/W PcvjwQAAAKdgqioAAAAmpLu7wigrAADA2f0dHcBIpuIAAACYldHddfq6AAAA52aqKgAAAOZjdBcA AIAJ6e4CAAAwId1dAAAAJqS7CwAAwIR0dwEAAJiQ7i4AAAAT+ptevCzLz8ZHaB+L0l5XLE2fE8N+ W1H57u6rroegLv/eIR3js/odv0evMWRuPVrhR4vnU2mE8ffoGL2vz6X5S59epXc8AFAhNbqb00E9 TI9gEnmG2nempI5BQu/rc2n+0qf1jgcA6myO7j7vN+nfWUt/ha341fZ+vz+CWZal4Y++iR3ssTl4 lXl+HeARgPYlofS+PpfmL31a73gAoNr66G6ctvjDM4zmjfLVHQyy10xPTYOE3tfn0vylT+sdDwBU SD3MHOquEyoYhrvf76oEAACQsPIwc/DHGo95xklXCiCgIXcod4RqwVsUAExv893dgHf350u8AzWc 2fJtrfyUdfGU5v+Z/usUvkXTeFbkX5QsHdL+8qlQussV5ZmuJ03edVz9S6fySeQfLZ6vq+xM/3oE 99fqUTP97re1xd71nzrDb9kA4Lu7BXrPbFm66YYzW/ZulAxp9JyofEpFi6e3aPvbeybbJmtFm+l3 v2jVIMHbFj+nOl4ATOzLd3enl/9odNuZLb+OinwO5jSZ2TIRzFv6z219Laj00Mr+/F9X/NqQ6l0+ FUp3uag8e6s+XnWbeNW2/nSKp/R8r7s+5E8iGG2m3/1et5gon9IHNxoG1il9Oqui/a1Iv/r3nCMb 9ngBcDV7u7urd6bMJ1Fz0vdzf/m+0erSrbUSWeXnU6Q0/9Knvns/JT7wKfRTlE+paPH0Fm1/G57v ddeH/LWaXK8izKz7tZ9/3/he3defFUqj3XnoW8WTvn995l+aPu1rnEXHa388AJBw6dHd0l+dAca6 4HUpc0z7s8ebWPFrd6uHaPF8Vfe0S/4zCABwgL3d3WNGM/rp+vDqBYVqq03gauUZbX+jxdP7ilR6 0TvgIlnUd1rtNH4dYCxS/fB2/oBnjoEPn6fjzDxexz8MD8BlXXp0d0t61PfgYNJCxRMqmAlcrTyj 7W+0eCrk70Lpyx11L4OUqhgnfLt673k9vm6L6QzbxtPqYfVWVb30ePWOBwB+dHdppW7qHbZcrTyj 7W+0eA5Q+nLHYS+DnH2U72oP915kNwE4i83ubsBHiSI0MaOViXjmdrXyjLa/0eIp1eRlk8TDpQe8 DLJzKqnhNzJ9XQAYa+W7u8FvVw0fk4vQfwaIbOAEDdV93fv9/ly3yXW+7n6hrwsAw610d59C9QZD BQNAKJ99y7Y93v3xAADHW+/ujm0lfOrXbljdwZxpJz//XppVE6Xx9NhWaZoe6xblGaRW5ygNdchX VYLkGSGeaMcr//qw/JOZSWn6OkXxPP7x+b3Z0qhKr/9F8VRoEs8c6QGgwp/b7fb7+7u6bP9kLasf QshMnxnGHumQtj6QkL9KUfpH4rfG2dZ/9+f/Jr10a1t74qnIfytl3fSqq8FXl0+pnAy71rfEWjm7 mX+88h1Tf/rFc+T1IUfbeEqvh127eZ/5V5y/Xfe34n7Xu/zPnh4A6qQeZg51v+kRTCLPrQ8kFOXW uwDrvmnRY1v7txjh+EYz8J3JIds6Uf0Zkr5Uw+tV6fWw9yRVQ4ru7PGcPT0AVLrdbqNDuKIeDxsD AADwlBrdBQAAgJPS3R3AuC4AAEBvf0cHMDNTcQAAAIxidHcMfV0AAIC+TFUFAADAfIzuAgAAMCHd XQAAACakuwsAAMCEdHcBAACYkO4uAAAAE9LdBQAAYEJ/04uXZfnZ+EjsY1Ha64ql6XNiAAAAgFWp 0d2cDuphQgUDAABAcJuju8/uZXpYtXTQtWKQ9n6/P4JZlsUYLwAAADnWR3cz+7qHeYZhjBcAAIAc qYeZg/R1H0IFAwAAQHAr3d3gI6jBwwMAACCCzdHdgKOpAUMCAAAgJt/dBQAAYEK6uwAAAExo80NE mVbfpE08dVyaHgAAACoY3QUAAGBCe0d3SwdmDeQCAABwAKO7AAAATEh3FwAAgAltdndX55QaK2BI AAAAxLTS3Q3+em3w8AAAAIgg9TBzqNHUUMEAAAAQ3Hp39zmCGqST+QzD0C4AAAA5/txut9/f39Vl 6U5mTk/4dcXS9JlhAAAAwKfUw8yhupehggEAACC62+02OgQAAABozHd3AQAAmJDuLgAAABPS3QUA AGBCursAAABMSHcXAACACenuAgAAMCHdXQAAACakuwsAAMCEdHcBAACYkO4uAAAAE9LdBQAAYEJ/ n/9aluVr6vv93mSrj221ym0184dOmwAAACA4o7sAAABM6O/b/ycYDn3sQs5gNQAAALMyugsAAMCE dHcBAACYkO4uAAAAE3p/d7dI4v3Yz3eAPxN/nUK5KP9jbIW0FU9++tfZqnPWSs9uvbW0Xzx1jimf 0q30iCexOgAA0EOv0d3eM0UNmYkqsdHVRaXpvy7a6Zrx5KePVj4AAMAeu0Z300OLy7K8JsgfmazL v7dnr2ZroO8tntL0X9fa6Zrx5KePVj4AAMBO76O7y5rSTHs3/Qd2LVY3ff8nP33FVlbTJEaVD46n zjHx5KePVj4AAEC1XaO79BCt73T2eGL++BKtVAEAYD7v3d3SVviU70TUT34AAAkBSURBVOgm9H6C Olov6OzxxOzrAgAAB+g1M3MTofq69/v9+Q7n6tLVtbruwiOk1XeGh8RTIVo8AADANOq7u+lJevZ3 Y3rnX+HZ4/108LxZAAAApO19d/dqD4sm+t6rPd5TxD9QtHgAAIBp9Pru7qUMnyn6Oeac+YUnAACA 6enuwgB1n/gCAADy7e3u7mmy56wbp0uQ+ApxIsjEu76tdu05wJsztHtAPEVOEU+cSggAAOSrf3c3 PVNx3bqvXbXq/F/TN3y492s8b1vZUz49iCet9PgCAADB7Rrd3dMByFk3WgcjEc/qonT8x++deNJK jy8AABDa7XYbHQIAAAA0ZqoqAAAAJqS7CwAAwIR0dwEAAJiQ7i4AAAAT0t0FAABgQrq7AAAATEh3 FwAAgAnp7gIAADAh3V0AAAAmpLsLAADAhHR3AQAAmNDf9OJlWX5+fu73+9aitNcVS9PnxLDfVlSd NtfQnmLpWqR1AoaU6bUK9Yu/tHyOKc+vWznv+XWk3gerIv/zno+vPqvf8XtUcX0oKvxQR8r5Pkqo avDmstcfII7U6G5OB/UwPYJJ5Blq37m40tp4TO39uhXnFwNdqo5F2FnnOwAxbY7uPu9P6R/YSn9+ q/i57n6/P4JZlqXhr32JHeyxOWb1qCT92nPB+7pb54jzi4Ey718H6H196HR/LOV8ByCs9dHdOG2F h2cYzRsNqzsYZK/hpH3dJ+cXA12kpvW7P1ZH8vWPAHCY1MPMoe5SoYKhufv97hBvOeAZigqO18Sc j+fiYDET1x+grZWHmYf/Qpx2zDNRLrVEELCj2+T64PyCHmI+MxwwJACuY/Pd3YD3p+dLSqMktr71 wlL6zcattUq3Upeyx5TCFTNzFoWRXz6lJZleJeDpMFaQmdK71oeiePafuV9PhIrrT1H+RcnSIe0v nwqlu3zA9Tzf/uvz8PtjhdL6k34R+vN8b3X/3R9PhYr7UavrZ6vy39rW/utP2+v5180BZ+e7u200 aWc0bKwMaff0nplzYPmcrh15RqX1p3d9aFife9efs5/v0c6vaPFMqa7+DJkUMNqk003OrxPtb6ne 9wvgdL58d5dX6Z8Mdz5Flp7+p+hX29V48n/5rlM9M2fmzKW9y2crcSI9X+UXWmn92VkfPgcr8vNP 1Ieu52NR/aw+3/efj03Kp0LpLoc633tfn4/R73z/utZO0eJJ5Nnk/Oq9v72vP4kwGt4vgGnsHd1d 1jRMfwoNb3VNshrYTtrq0D50yr93Jmdsdw732rHMP+UTP4jkp68OdX88pfk3EfB835/JQNHiOYVj zvfMBD/JXlOi99UvnoYanl+n2N9Sve8XwBl5mBnoovrNK+B0jjnfo/VYosXT29X2F5jD3oeZz/5D finN9LSDnwBszvFtq+vDqxekfralPNvqfb5Hu2L0jida/YxW/gCZvLtbINq9J5T7v3lBV0vpFLdJ x/cY9+0pZKMdglDxhApmAsrzGKPO98d2V9/hTL8WG0SoYABOTXc3l6kOvko3a4L3eB1fIlM/21Ke RKZ+AjS02d0N2D+JcImPVibRnP3h1fgRBtHpaEYrf/HMbabyjHB/LBWt/MUDMKWVqaqCX2GPCW+O KaODCF6jKJL/oRFnEBxs1MU25vn+Nj9z+klmAGaVmpk51N0rVDCAUxKCKD0ZY/ZOAaCH9e7u6zf0 DgxmU/o9liY5f/3j10VD0u/RdluJTyw23FDv8glS4YPLvz60Or8OqFpN4umxrdI0PdYtyvNEJ9Ec 1+d+98dSvc/3Cs8B3pyh3SPP9xxNNlpa/p32NMJlIUIMwBB/brfb7+/v6rL9kyV8zoiYnz4zjD3S IX1u7usuvK3SNX3ODJOlH0LcWcKl5bm1VvW0mTvLp/R4ba3VsKJuhVRaRD3awV9PzFDn19sRT/93 f/5vmpyPdfUzP/+tlE2O787yKbWnb/M08PpQcbyKSq/HY71dz/c9Ab9uaM/HgVvFk6kinszEOasU nb9fNb/+HHZ/B+aQepg51Ml/8Lw4q4t6f2T4yAKPUJ4N828iVIWP72txtT2/PpeOrQ9tt56T254t nvF8722m63OEAu99vvd2rnj2Z3Xk+Ru8fgLzu91uo0PgO69apSkfXqkPAORwv4DppUZ3AQAA4KR0 d0/A745pyodX6gMAOdwv4Ar+jg6A/9RN9XQdyodX6gMAOdwv4MqM7p6Da3Ga8uGV+gBADvcLmJ+p qgAAAJiP0V0AAAAmpLsLAADAhHR3AQAAmJDuLgAAABPS3QUAAGBCursAAABM6G968ePD3KsfJUt/ s/vhdcXS9DkxAAAAwKrU6G5OB/UwoYIBAAAguM3R3Wf3Mj2sWjroWjFIe7/fH8Esy2KMFwAAgBzr o7uZfd3DPMMwxgsAAECO1MPMQfq6D6GCAQAAILiV7m7wEdTg4QEAABDB5uhuwNHUgCEBAAAQk+/u AgAAMCHdXQAAACa0+SGiTKtv0iaeOi5NDwAAABWM7gIAADChvaO7pQOzBnIBAAA4gNFdAAAAJqS7 CwAAwIQ2u7urc0qNFTAkAAAAYlrp7gZ/vTZ4eAAAAESQepg51GhqqGAAAAAIbr27+xxBDdLJfIZh aBcAAIAcf2632+/v7+qydCczpyf8umJp+swwAAAA4FPqYeZQ3ctQwQAAABDd7XYbHQIAAAA05ru7 AAAATEh3FwAAgAnp7gIAADAh3V0AAAAmpLsLAADAhHR3AQAAmJDuLgAAABPS3QUAAGBCursAAABM SHcXAACACenuAgAAMKG/6cXLsvz8/Nzv961Faa8rlqbPiQEAAABWpUZ3czqohwkVDAAAAMFtju4+ u5fpYdXSQdeKQdr7/f4IZlkWY7wAAADkWB/dzezrHuYZhjFeAAAAcqQeZg7S130IFQwAAADBrXR3 g4+gBg8PAACACDZHdwOOpgYMCQAAgJh8dxcAAIAJ6e4CAAAwoc0PEWVafZM28dRxaXoAAACoYHQX AACACe0d3S0dmDWQCwAAwAGM7gIAADAh3V0AAAAmtNndXZ1TaqyAIQEAABDTSnc3+Ou1wcMDAAAg gtTDzKFGU0MFAwAAQHD/D8mL+egiZtJsAAAAAElFTkSuQmCC --0000000000002ed3be06018cdddc-- From nobody Sun Jul 30 18:32:48 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RDVPF6FBjz4pJ18 for ; Sun, 30 Jul 2023 18:33:04 +0000 (UTC) (envelope-from ksrdhrbsd@gmail.com) Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RDVNt5qNpz3Lbv for ; Sun, 30 Jul 2023 18:33:02 +0000 (UTC) (envelope-from ksrdhrbsd@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=IEP2OrbB; spf=pass (mx1.freebsd.org: domain of ksrdhrbsd@gmail.com designates 2a00:1450:4864:20::632 as permitted sender) smtp.mailfrom=ksrdhrbsd@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-x632.google.com with SMTP id a640c23a62f3a-99bcc0adab4so601038166b.2 for ; Sun, 30 Jul 2023 11:33:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1690741979; x=1691346779; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=pAEHQ39EjReyDEUpEj8ARhT9+OPE+Uam28mWOHhAbzw=; b=IEP2OrbBZ8zF0ysXwAZAWW0qxn4rrToaP0ftMg9tq/oZYam6o41/UNMwT9++TTLGkr iOBSpoAYc7cwPNJ9uGfVXEEklWHVi10qS/E/rBlazvLUascQI+uvnovTKzTd2gM3/wND A83Umy3Vw9imClbGPCZLhvN7Mvwe2675v5lccJwp2C8iC9YSfnN+ZtNT0gvMhLHJjT2m bmFNtYlL1g1Kkl8+VcYSEm5HdvQuTaBBOOqYlv5CMzDIs97Fgs+uzreBGmBAr8DvSKfd MOrBWvSsoAs8qI4iNeehWQ5cfy49SjDUWf7o30avP0SDn5H09yxvtcS2R3u9NhKbNkWY 1+jQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690741979; x=1691346779; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pAEHQ39EjReyDEUpEj8ARhT9+OPE+Uam28mWOHhAbzw=; b=WkE5ezEwHfr6osySq+QWE98lPyDanU9mNCGGH/b4K5Ld7lWsa7t3eF2CaLznN4co+U l47q5eZ6VHhxVgb8GyM1TbefHoxpNdlPTwtZ+n0UEuigyKSIL/rmr84jNcfl5hhunwci MLtuJiIhBCimb9trUuZ8/KfgJ8qFXhakB8mct7VhC//VEErXcHrBECi1qeVEG8pA8XxA PD0QQatONgudAADJfZyQCqeGlYS5Y17rixMgsVNM0ynql9HgWH+/MixPZXulLLso7vo3 Ci2NSS6P1sGczcn7rXiwjPvHemth3Mn8vAjn4rNvWUj/gF4yQ3WiiGU8Hxm47QXoUE9K d1yg== X-Gm-Message-State: ABy/qLb6OK4YwUZQwO3HwFMNKi90aRhKzQzV9roiJrNsV5p0f2VXgtyf 4PgUzKp106jEXAKrnZoAw8+pPIaeOO5FC8n+EQ== X-Google-Smtp-Source: APBJJlGvZWQbPRW/6mC+dYHZhuzE1i/oIHyKeVi4w+dKEvT/Iqplg1ZBPCOPSRU7bQ/h25/ojpZAHrz6iGkn76nAieU= X-Received: by 2002:a17:906:114:b0:973:ff8d:2a46 with SMTP id 20-20020a170906011400b00973ff8d2a46mr4784273eje.3.1690741979209; Sun, 30 Jul 2023 11:32:59 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <5EBDAA39-3512-4AA9-A4E2-72741FAE414C@iitbombay.org> In-Reply-To: <5EBDAA39-3512-4AA9-A4E2-72741FAE414C@iitbombay.org> From: k sridhar Date: Sun, 30 Jul 2023 14:32:48 -0400 Message-ID: Subject: Re: Contributing to one of the projects in the Ideas page To: Bakul Shah Cc: imp@bsdimp.com, FreeBSD Hackers Content-Type: multipart/alternative; boundary="00000000000013e3a50601b88b65" X-Spamd-Result: default: False [-3.82 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-0.997]; NEURAL_HAM_LONG(-0.93)[-0.929]; NEURAL_HAM_MEDIUM(-0.90)[-0.899]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::632:from]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4RDVNt5qNpz3Lbv X-Spamd-Bar: --- --00000000000013e3a50601b88b65 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks a lot for your tips and pointers. I am setting up my BSD machines and will start once that is complete. Thanks. On Sun, Jul 23, 2023 at 10:44=E2=80=AFPM Bakul Shah w= rote: > Note that NetBSD seems to have the same version as FreeBSD's > ports/sysutils/isc-cron (modulo replacing sprintf, strcpy, strcat etc wit= h > safer versions). > > Paul seems to be maintaining the original at > https://github.com/vixie/cron.git with a few fixes this year. Though his > CRON_VERSION says V4.999 compared to ports/isc-cron that has V5.0! I woul= d > suggest keeping in sync with Paul's version at github and may be feeding > back improvements. > > On Jul 23, 2023, at 7:06 PM, k sridhar wrote: > > Sure, I will start by reviewing what OpenBSD and NetBSD have done. > > Thanks a lot. > > On Sun, Jul 23, 2023 at 10:04=E2=80=AFPM Warner Losh wro= te: > >> >> >> On Sun, Jul 23, 2023, 7:59 PM k sridhar wrote: >> >>> >>> Hello, >>> >>> I was browsing the Ideas page for ways to contribute to BSD. >>> I came across this (which does not have any contact, I think). >>> >>> If it is still open, I wanted to check if I could try to do this. >>> >>> Briefly - I am in IT, and have been using UNIX since grad school in the >>> early 90s (Solaris, briefly Ultrix/Aix and then different Linuxes >>> and MacOS). C/C++/Java programming, scripting, installs, etc... and var= ious >>> other stuff. >>> >>> I want to give it a shot if you all think that is ok. >>> >> >> Nobody is working on this. Might be best to see what openbsd and netbsd >> have done first and start from there. It will be the first question peop= le >> ask when you start the review process. I think yhey have some fixes and= a >> newer vixie cron import, but I'm not sure. >> >> Happy coding >> >> Warner >> >> Thanks a lot >>> k.sridhar (U.S resident in Virginia) >>> *Improve cron(8) and atrun(8)* >>> >>> *Currently, cron(8) and atrun(8) are outdated in their implementation. >>> Here are some directions for improvement:* >>> >>> - *Update cron(8) to ISC cron with security fixes from OpenBSD.* >>> - *Integrate the atrun(8) functionality into cron(8), as it was done >>> in NetBSD.* >>> >>> >>> *Requirements* >>> >>> - *Strong knowledge of the C language and Unix API.* >>> >>> > --00000000000013e3a50601b88b65 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thanks a lot for your tips and pointers.
I am setting = up my BSD machines and will start once that is complete.
Thanks.<= /div>

On Sun, Jul 23, 2023 at 10:44=E2=80=AFPM Bakul Shah <bakul@iitbombay.org> wrote:
Note that NetBSD see= ms to have the same version as FreeBSD's ports/sysutils/isc-cron (modul= o replacing sprintf, strcpy, strcat etc with safer versions).

Paul seems to be maintaining the original at https://github.com/vixie/cron.git with a few fixes this year. Though his CRON_VERSION says V4.999 compared= to ports/isc-cron that has V5.0! I would suggest keeping in sync with Paul= 's version at github and may be feeding back improvements.

=

Sure, I will start by review= ing what OpenBSD and NetBSD have done.

Thanks a lot.

On Sun, Jul 23, 2023 at 10:04=E2=80=AFPM Warner Losh <imp@bsdimp.com> wrote:
=


=
On Sun, Jul 23, 2023, 7:59 PM k sridh= ar <ksrdhrbsd@g= mail.com> wrote:

Hello,

I was browsing the Ideas page= for ways to contribute to BSD.
I came across this (which does no= t have any contact, I think).

If it is still open,= I wanted to check if I could try to do this.

= Briefly - I am in IT, and have been using UNIX since grad school in the ear= ly 90s (Solaris, briefly Ultrix/Aix and then different Linuxes and=C2=A0Mac= OS). C/C++/Java programming, scripting, installs, etc... and various other = stuff.=C2=A0

I want to give it a shot if you all thi= nk that is ok.
=

Nobody is working on this. Mi= ght be best to see what openbsd and=C2=A0 netbsd have done first and start = from there. It will be the first question people ask when you start the rev= iew=C2=A0process.=C2=A0 I think yhey have some fixes and a newer vixie cron= import, but I'm=C2=A0not sure.=C2=A0

=
Happy coding

Warner

Thanks a lot
k.sri= dhar (U.S resident in Virginia)

Improve cr= on(8) and atrun(8)

Currently, cron(8) and atrun(= 8) are outdated in their implementation. Here are some directions for impro= vement:

  • Update cron(8) to ISC cron with security fixes = from OpenBSD.=
  • Integrate the atrun(8) funct= ionality into cron(8), as it was done in NetBSD.

Requirement= s
  • Strong knowledge of the C language and Un= ix API.

--00000000000013e3a50601b88b65-- From nobody Tue Aug 1 14:47:55 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RFdJP1TSRz4qFh6 for ; Tue, 1 Aug 2023 14:48:04 +0000 (UTC) (envelope-from stephan.de.wit@deciso.com) Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-vi1eur04on2101.outbound.protection.outlook.com [40.107.8.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RFdJL4Nsdz46Dr for ; Tue, 1 Aug 2023 14:48:02 +0000 (UTC) (envelope-from stephan.de.wit@deciso.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=deciso0.onmicrosoft.com header.s=selector1-deciso0-onmicrosoft-com header.b=STJyvNfg; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (mx1.freebsd.org: domain of stephan.de.wit@deciso.com designates 40.107.8.101 as permitted sender) smtp.mailfrom=stephan.de.wit@deciso.com; dmarc=pass (policy=quarantine) header.from=deciso.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RMUnhmzQtsB9m64dbqFFhtRovLT9ABtS8p2c/jadKy8FXd8zt9TXaNf8qadnU2ctrKlQuCNbNoOPV2d3A5864s8ssIXzNtxY8Kd1QXDiHLwjoT1rD6BTdXVRZ578Rvyp5sUbufzYj5cXu7YvKyrJdB7sSELS/rDNYc7xspqBitfU9hWwYvaON7aXxfVDIii1fGylAT2FngeIm0UeViQsXhrU0ytz8h13p+38Yg/GRwznKXw+CjJupsyGSCaLNWu4SUvQbOEGv4KPjPYD8XhNCr8AIAfOYlduZD+3ZaOhrxosAQcGdNvWeWN8an4oXZH5iy4rMaBB/guNY4Py2PiCtg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=59vAbMRmktMU16rqJRwl+UXt9nl8L7ldeoeAwpZko2Q=; b=G0nmxVR4YN+qsUWre5JcG+KiyWdxXxUYk2+yeswsjXGq6pJ1w2fEt/1Zr6kxi3SAtE2b+o81kHBTCTYZYTOMO6DSq/As4e+k57+c89h38ED0Ep5eJVv3MByhdUVLbd9wj4wru0Nxp+057VvDor4cFsFPx3E9YVJrWd5DooFnVLM0+vsNCDf6nbsmvsO0WrC2IJBON6OU2ywgDvVwmFZJG8RQmTAkUJERu8qeuoscEmoIIqGW0tPlzWy2wT+g8U/T+9U/+pr3MnkpFhen2NqtDqMte3kXspGNAV5sUkKqnBbOjVZE1im1VBJu4NEeJIHaq+RguSGn0shL5Q9S/gkrUA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=deciso.com; dmarc=pass action=none header.from=deciso.com; dkim=pass header.d=deciso.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=deciso0.onmicrosoft.com; s=selector1-deciso0-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=59vAbMRmktMU16rqJRwl+UXt9nl8L7ldeoeAwpZko2Q=; b=STJyvNfgR1F8/8gXjUopy1yMO/nFYLOVTjSp8XHyOAx6jmLG0DRscMqFdtmIZ2LVM0Wlw457EXOuFUcyIwRwVccCMZRzt9T3FhytD9zay78+r0El4UYB9p9hhb9oqwAMoRib5RFkA3N/jgectbLanlO1eizML9GwhXL7BV1xtnE= Received: from AM6P193MB0311.EURP193.PROD.OUTLOOK.COM (2603:10a6:209:44::26) by DB9P193MB1772.EURP193.PROD.OUTLOOK.COM (2603:10a6:10:240::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6631.29; Tue, 1 Aug 2023 14:47:55 +0000 Received: from AM6P193MB0311.EURP193.PROD.OUTLOOK.COM ([fe80::25f1:a935:f74e:4d0]) by AM6P193MB0311.EURP193.PROD.OUTLOOK.COM ([fe80::25f1:a935:f74e:4d0%5]) with mapi id 15.20.6631.043; Tue, 1 Aug 2023 14:47:55 +0000 From: Stephan de Wit To: "freebsd-hackers@FreeBSD.org" Subject: lockmgr() in network driver context Thread-Topic: lockmgr() in network driver context Thread-Index: AdnEhMaEI8tAD3LFQqe9MbXy6ITYjg== Date: Tue, 1 Aug 2023 14:47:55 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: AM6P193MB0311:EE_|DB9P193MB1772:EE_ x-ms-office365-filtering-correlation-id: 1bd50173-1bb3-4afc-5a4d-08db929e4aa0 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: fz3/uresGqZVKdFuCd+Pb43cgLTd5Q+eiMgRLSfbGyVsR9/PMhDeTHXuPo0H8Bgn7XDauOrhEfUX3csbsBmpZiZ6hPel80DZ7ryQECPppxx+VXKxjoX0GSSkl1mOqPFroV4thAM9JlS8276geCNY7/22oPyAQENQbAm5+xBXTDM6KDkJCVEGysK9n00COOKPgGlBVGlMvbg+Un6c5mHxMWccNzTKfm07mtyp0DX3G45C4tCoCAVEAmvFcaUptmJvlTYWRS8NmK6gPxN3VmPgKBdYoYoc+NZlFeNTPqpoXx6pM+F/j4MnZyixXWbFLW9c08pcLBsh9yXhRw/QIbKHkPNoYr6EqA/sfd0n5VgI77Ni+xK0XGdSYJOaQZkABlLS8o2QJglUilQ5PTLdvmZyWD7R3xEy68GFn2Mo3ZP7Fk3Vvl1mefmL+QSjCz3XOOizk1fMNMF9xF/pJS2ogi+SHutBlSTT7g1Obbrc7LThLbP+l0UBb570XAVe4zeAEPmZ5mEt15v3Xifo95FuWg6MKi4OLCl2fl6nzO9AxLAJGrrrS28HWJDh2SaNNtWbGSehuKH3/iHfROP0KqTEbZ9WQi8HQ763PvDfp3zUKq62h9c= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM6P193MB0311.EURP193.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230028)(396003)(366004)(39830400003)(136003)(346002)(376002)(451199021)(966005)(9686003)(7696005)(478600001)(186003)(26005)(66556008)(6506007)(52536014)(2906002)(71200400001)(66446008)(5660300002)(64756008)(41300700001)(8936002)(316002)(76116006)(8676002)(66946007)(66476007)(6916009)(38100700002)(122000001)(86362001)(38070700005)(55016003)(166002)(33656002)(83380400001);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?UktIN0I4RFF4cG9FQU5GeUcrRDJodlNERmNqWXBPRVNUa24xSjNhYU5xMlFy?= =?utf-8?B?akg3TEhIN3ZlbTFZVVk3MFhjMGRIclcrSjFjOXBRckJRM3JHeVdRSG1YN1Vw?= =?utf-8?B?T2RDWVB2clhZTTUrWDkveU9Md201UStBYkNqaklWRW15UHB3alh4eG1jSXVY?= =?utf-8?B?NlNOYTd5bGs3cFY5eklrcUlHSnRqMGdJT09OUTFpSzh0bHNVNjd3cU9oOUha?= =?utf-8?B?UTMvTWV1YWo3NHhoSExUemRBcVh6cVBseHdHTG5iSCt4ZnJiZXltYjBzdVNx?= =?utf-8?B?V2xObkFCaGp0SmJzWjFaaTdsSm9uV2EwU0VRQjNlbmxBYSs5Q3FhZTVjQ01M?= =?utf-8?B?YUU4WXVudDE4cFYwMkpLckhUeWtTMHVpV0k2SEJ0WnZnWjBsMWRqYnJkWkIw?= =?utf-8?B?WDlxVWNxUXp0d1NYcE9jZXNVUXVLY2dIVUZNNjJXYlBYeHJ2cUE0bHJwVTlO?= =?utf-8?B?R2s1ditCaFE0WnNXWExMNmhud3VCSjUyT0o1OTdoYVUvczBxMWF5eFppUExY?= =?utf-8?B?WDZVVnhyYnZxdGduZWtYVnB0RnFWVG1uckpyL09EMVRWZ3l5WVdZQ1VadlFs?= =?utf-8?B?S2ZMa3I2MGlzVy9tc0Rya2ltajB0Q0h4U0NacEhrdVpCQlA2eVZ6dU5OOEtG?= =?utf-8?B?M3IyTXN6UkRhaERRVlBUdWRJdlp2NWtDSmRxRTZDeGFHSWhuVmlSTEJpSzdW?= =?utf-8?B?bTVEb1dJMW1uWlViRWxyWUFHdzUxMXdySEpyb3FlR3JpTlBHdi8zeDBSejdp?= =?utf-8?B?MGZLcytiR05jY09OT0liM0JJb25OQ1NOU0RZWVVCdGNFQ3JnazljTTIvL0Zt?= =?utf-8?B?ZUxMYjlnbUtqOHVIRGNwdmJ1K3lUSEZ0NWhOM2JQY1BjUHFhRlQ2Z1NuL1pm?= =?utf-8?B?U3RPOGw0VHBkbmRUT0JxUjl6RjYrejNvQW9ySTlNL1NsdVZtN0tpdjR6a3hh?= =?utf-8?B?ZVJWcDhaL3ozTkZiYnJ4MWhjTXBpQUlLTVFhMlRTVFIvd0U4SUI3ekoxanZn?= =?utf-8?B?M3crS2xjU0xseFE5dUxCZWFGb2R3MWphUFE2WnJsTDNLR1hyOVFiWnY0Znc5?= =?utf-8?B?bldRRmZVQXh1YS96YkQwS3Q1VStSUy9Kc05IVGduOFZ6SXl2NE9FaEt0dkp4?= =?utf-8?B?eitmcG1tdVNxdjhxV0IyZ0laTEtDQ1U5aVRFS0lKakxuaGxMVVRKbitvQ3BZ?= =?utf-8?B?Y0g2dGc2MXZUNjJsMjFxZGZhOXErZVlZQ3plZHVSeXFlNEhzbWNlaE5HRDJl?= =?utf-8?B?T3pVL3NNQUltT05zVExpdnJUSytLTmtLNEcvWDdYTENuZGNHaVpHQlh2S2tD?= =?utf-8?B?Nlpmc2JRUW9jVElxZ2RjR2EydWRVaXRXOExEM21selY3WU42VHZBa2RYYkU1?= =?utf-8?B?OVg5Z0l3R3B6YVJxajdSYUEvamExN2k0ekJhV2RteUhmY1ZFUGZBN0g2MmU0?= =?utf-8?B?enJ5MGN4bERGTldBczBYcU90bm8rV0kxWjdHaUxXbTZrdVUyZ1dROXpWMFYy?= =?utf-8?B?bmZHL24xNUFNbUkvWkZCVlF1dHdUeWZvQkRxZEVDNGU3MktucXVadXZvN2tK?= =?utf-8?B?ejMrT25QRkpyR2VmS1VZT0lWNjZsSVdCR2hWMGVxS0FqWDdSREJmZ2xjVmdp?= =?utf-8?B?V2Z3U0dlR0VmY1M3WTUrK3Mrbk1yd1hnMkFwR0xtTFlJR1JZSXlrYnJnRFBL?= =?utf-8?B?c2R3aUNlMWlEeGwyYmNPMXhJTFNpemVHMGhBUURQOTB3N1ZpaWFESUh3cnlV?= =?utf-8?B?aXJsWDBxb1d4RFg4OEZlTkN1bkl3ejJlZFhCQTN5TEpRUmxwNnIwaFVScnIv?= =?utf-8?B?RzZhak41dU5Jc3g3QldMdjhqNjEwazZqall4RnNjYVBOQk16UWtjTnB0eDVG?= =?utf-8?B?MmJmWlRsemxSTk12NnZId3U4UVRDd09mdEZpdTQyQ1JiV1RaT3NBTkdHay8w?= =?utf-8?B?cUpHK1hYazhTQ2hncG8vWlMrYWxWUmppZitKa29ZK2ltb1R5V3JDclhraXBV?= =?utf-8?B?TUNwcjFkaUpJVU5relhQTmw3QjlIM1RMYlArOFhsUDVVRjQ3QXlPVDhOeEV1?= =?utf-8?B?cmxtTjhVUDZ3RGwvRGs2RjBWcndvS1FvWXhFbS9SMkcrWmZIRXlPTnBPWGZB?= =?utf-8?Q?CPHSPwayDkaCHXUCsoXs/oGLv?= Content-Type: multipart/alternative; boundary="_000_AM6P193MB0311A026447E2B8523F4AE89BC0AAAM6P193MB0311EURP_" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: deciso.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AM6P193MB0311.EURP193.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 1bd50173-1bb3-4afc-5a4d-08db929e4aa0 X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Aug 2023 14:47:55.7017 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 0035f53c-6fda-4dca-a17f-cf54bb21f5b8 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: RMhXTqaUQZlRYDhN2LCA1N9UaI9bzJ+nsqcW+XQgGtuEQ5qx2jDP/8PQCoKNWMpMbS00HkkyAk1CvhQ0lGLO8NFzc5wYhnVvzeuKos5hlk0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9P193MB1772 X-Spamd-Result: default: False [-3.77 / 15.00]; URI_COUNT_ODD(1.00)[1]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_SHORT(-0.98)[-0.981]; NEURAL_HAM_MEDIUM(-0.97)[-0.967]; NEURAL_HAM_LONG(-0.92)[-0.922]; DMARC_POLICY_ALLOW(-0.50)[deciso.com,quarantine]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; R_DKIM_ALLOW(-0.20)[deciso0.onmicrosoft.com:s=selector1-deciso0-onmicrosoft-com]; MIME_BASE64_TEXT(0.10)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_COUNT_THREE(0.00)[3]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; FROM_HAS_DN(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@FreeBSD.org]; TO_DN_EQ_ADDR_ALL(0.00)[]; DKIM_TRACE(0.00)[deciso0.onmicrosoft.com:+]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[40.107.8.101:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.8.101:from] X-Rspamd-Queue-Id: 4RFdJL4Nsdz46Dr X-Spamd-Bar: --- --_000_AM6P193MB0311A026447E2B8523F4AE89BC0AAAM6P193MB0311EURP_ Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 SGkgYWxsLA0KDQoNCg0KSeKAmW0gZW5jb3VudGVyaW5nIGkyYyBidXMgZmFpbHVyZXMgdGhhdCBu ZWVkIG1vcmUgZ3JhY2VmdWwgZXJyb3IgaGFuZGxpbmcuDQoNClRoZSBkcml2ZXIgb3BlcmF0ZXMg aXRzIG1hbmFnZW1lbnQgdGhyZWFkcyBpbiBwb2xsaW5nIG1vZGUgd2l0aGluIGNhbGxvdXQgY29u dGV4dC4NCg0KQXMgYSBoYXJkd2FyZSBjb25zdHJhaW50LCBjb21tdW5pY2F0aW9uIHZpYSBzYWlk IGkyYyBidXMgY2FuIG9ubHkNCg0KaGFwcGVuIGZvciBhIHNpbmdsZSBuZXR3b3JrIHBvcnQgYXQg YSB0aW1lIGFuZCB0aHVzIG5lZWRzIHRvIGJlIG10eF9sb2NrKCknZC4NCg0KV2l0aCB0aGlzIGxv Y2sgaGVsZCwgaTJjIHRyYW5zZmVycyBhcmUgaW5pdGlhdGVkIGJ5IHRoZSBkcml2ZXIgYW5kIHRo ZSBkb25lDQoNCmV2ZW50IGlzIHNpZ25hbGVkIHZpYSBhbiBJU1IuIFNpbmNlIHdlIGNhbm5vdCBz bGVlcCBib3RoIGluIGNhbGxvdXQgKG5vdA0KDQphbGxvd2VkIGFjY29yZGluZyB0byBbMV0pIGFu ZCBtdXRleCBjb250ZXh0ICh3aWxsIHBhbmljKSwgdGhlIHNlY29uZCB0aHJlYWQNCg0Kd2FpdGlu ZyBvbiB0aGUgYnVzIHdpbGwgc3BpbiBpZiBpdCBkb2VzIG5vdCBsaXZlIG9uIHRoZSBzYW1lIENQ VVsxXSwgYW5kIHRoZQ0KDQppMmMgdHJhbnNmZXIgaXRzZWxmIGlzIGhhbmRsZWQgdmlhIGEgREVM QVkoKSBidXN5IGxvb3AuIElmIHRoZSBidXMgZmFpbHMsDQoNCndlIGhhdmUgMiBjb3JlcyBzcGlu bmluZyBhdCAxMDAlLCBjYXVzaW5nIGNvbm5lY3Rpb25zIHRvIGRyb3AgYW5kIGFsbA0KDQpzb3J0 cyBvZiBvdGhlciBjYXJuYWdlIHRvIGhhcHBlbi4gUmVkdWNpbmcgdGhlIHRpbWVvdXQgaXMgbm90 IGFuIG9wdGlvbiwgYXMNCg0KdGhlIHRyYW5zZmVycyBtaWdodCB0YWtlIGEgd2hpbGUuDQoNCg0K DQpTbGVlcGluZyB1bmRlciBhIE1UWF9ERUYgaXMgbm90IHBvc3NpYmxlLCBpbnN0YW50IHBhbmlj LiBIb3dldmVyLCBpZiBJIHJlcGxhY2UNCg0KdGhlIGZpcnN0IG10eF9sb2NrKCkgd2l0aCBhIGxv Y2ttZ3IoKSBjYWxsIHRoYXQgd2FzIGluaXRpYWxpemVkIHdpdGggTEtfVElNRUxPQ0sgfA0KDQpM S19DQU5SRUNVUlNFIGFuZCBhIGdpdmVuIHRpbWVvdXQsIEknbSBhYmxlIHRvIHB1dCB0aGUgdGhy ZWFkIHdhaXRpbmcgb24NCg0KdGhlIGNvbW1zIGJ1cyB0byBiZSBhdmFpbGFibGUgdG8gc2xlZXAu IElmIEkgcmVwbGFjZSB0aGUgaGFyZGNvZGVkIERFTEFZKClzDQoNCmZ1cnRoZXIgZG93biB0aGUg bGluZSBkdXJpbmcgdGhlIGkyYyB0cmFuc2ZlcnMgd2l0aCBtdHhfc2xlZXAoKSAoYW5kIHdha2V1 cF9vbmUoKSBpbiB0aGUgSVNSKSwNCg0Kbm8gQ1BVIGN5Y2xlcyBhcmUgd2FzdGVkLCB0aGlzIGlz IGFsc28gdGhlIGNhc2UgZm9yIG5vcm1hbCBvcGVyYXRpb24sDQoNCmltcHJvdmluZyB0aGUgY3Vy cmVudCBzaXR1YXRpb24gYnkgYSBtaWxlLg0KDQoNCg0KQXMgdG8gdGhlIG5hdHVyZSBvZiB0aGVz ZSBidXMgZmFpbHVyZXMsIHRoZXkgc2VlbSB0byBoYXBwZW4gdmVyeSBpbmZyZXF1ZW50bHkgYW5k IG5pZ2gNCg0Kb24gaW1wb3NzaWJsZSB0byByZXByb2R1Y2Ugb24gYSBwcm9kdWN0aW9uIHN5c3Rl bSwgYnV0IHRoZXkgYWx3YXlzIHNlZW0NCg0KdG8gcmVjb3ZlciBhZnRlciBhIGNlcnRhaW4gYW1v dW50IG9mIHRpbWUuIFRvIHJlcHJvZHVjZSB0aGUgaXNzdWUsIEkgaGF2ZQ0KDQp3aXJlZCBzb21l IHBoeXNpY2FsIHN3aXRjaGVzIHRvIHRoZSBpMmMgYnVzIHRvIHNpbXVsYXRlIHRoZSBvdXRhZ2Vz Lg0KDQoNCg0KRnJvbSBhIGRlc2lnbiBwZXJzcGVjdGl2ZSwgdGhlIGkyYyBidXMgc2hvdWxkIGJl IGEgc2luZ2xldG9uIGFuZCBzaG91bGQgYmUgdHJlYXRlZA0KDQphcyBzdWNoIHZpYSB0aGUgcXVl dWVpbmcgb2YgdHJhbnNmZXIgdGFza3MgaW5jbHVkaW5nIG1ldGFkYXRhIHRvIGxpbmsNCg0Kc3Vj aCB0cmFuc2ZlcnMgYmFjayB0byBzcGVjaWZpYyBwb3J0IGlkcywgaG93ZXZlciwgYSBjaGFuZ2Ug bGlrZSB0aGlzIHdvdWxkDQoNCnJlcXVpcmUgc2lnbmlmaWNhbnQgZWZmb3J0IHdpdGggYSBsYXJn ZSByaXNrIG9mIHJlZ3Jlc3Npb25zLCBhbmQgSSdtIG5vdCBldmVuDQoNCnN1cmUgaWYgc2xlZXBp bmcgaW4gc3VjaCBhIHNjZW5hcmlvIHdvdWxkIGJlIGNvbnNpZGVyZWQgYSB2YWxpZCB0aGluZyB0 byBkbw0KDQppbiBGcmVlQlNEIGNvbnRleHQsIHNvIEkgd291bGQgcHJlZmVyIHN0aWNraW5nIHRv IHRoZSBiYXR0bGUtaGFyZGVuZWQNCg0KY29kZS4NCg0KDQoNClRoZSBkb2N1bWVudGF0aW9uIGZv ciBsb2NrbWdyKCkgdGVsbHMgbWUgdG8gc3RheSBhd2F5IGZyb20gaXQsIGJ1dCBJDQoNCnNlZW0g dG8gaGF2ZSBsaXR0bGUgb3B0aW9ucyBsZWZ0LiBBbHNvLCB0aGUgc29sdXRpb24gc2VlbXMgdG8g dmlvbGF0ZQ0KDQp0aGUgRnJlZUJTRCBndWlkZWxpbmVzIGZvciBzbGVlcGluZyBpbiBjYWxsb3V0 IGNvbnRleHQuIEFueSB0aG91Z2h0cz8NCg0KDQoNClRoYW5rcyBpbiBhZHZhbmNlLg0KDQoNCg0K WzFdIGh0dHBzOi8vbWFuLmZyZWVic2Qub3JnL2NnaS9tYW4uY2dpP2xvY2tpbmcoOSkNCg0K --_000_AM6P193MB0311A026447E2B8523F4AE89BC0AAAM6P193MB0311EURP_ Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: base64 PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRv ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxl PjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6 IkNhbWJyaWEgTWF0aCI7DQoJcGFub3NlLTE6MiA0IDUgMyA1IDQgNiAzIDIgNDt9DQpAZm9udC1m YWNlDQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7DQoJcGFub3NlLTE6MiAxNSA1IDIgMiAyIDQgMyAy IDQ7fQ0KLyogU3R5bGUgRGVmaW5pdGlvbnMgKi8NCnAuTXNvTm9ybWFsLCBsaS5Nc29Ob3JtYWws IGRpdi5Nc29Ob3JtYWwNCgl7bWFyZ2luOjBjbTsNCglmb250LXNpemU6MTEuMHB0Ow0KCWZvbnQt ZmFtaWx5OiJDYWxpYnJpIixzYW5zLXNlcmlmOw0KCW1zby1saWdhdHVyZXM6c3RhbmRhcmRjb250 ZXh0dWFsOw0KCW1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTO30NCmE6bGluaywgc3Bhbi5Nc29I eXBlcmxpbmsNCgl7bXNvLXN0eWxlLXByaW9yaXR5Ojk5Ow0KCWNvbG9yOiMwNTYzQzE7DQoJdGV4 dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQpwLk1zb1BsYWluVGV4dCwgbGkuTXNvUGxhaW5UZXh0 LCBkaXYuTXNvUGxhaW5UZXh0DQoJe21zby1zdHlsZS1wcmlvcml0eTo5OTsNCgltc28tc3R5bGUt bGluazoiUGxhaW4gVGV4dCBDaGFyIjsNCgltYXJnaW46MGNtOw0KCWZvbnQtc2l6ZToxMS4wcHQ7 DQoJZm9udC1mYW1pbHk6IkNhbGlicmkiLHNhbnMtc2VyaWY7DQoJbXNvLWxpZ2F0dXJlczpzdGFu ZGFyZGNvbnRleHR1YWw7DQoJbXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVM7fQ0Kc3Bhbi5FbWFp bFN0eWxlMTcNCgl7bXNvLXN0eWxlLXR5cGU6cGVyc29uYWwtY29tcG9zZTsNCglmb250LWZhbWls eToiQ2FsaWJyaSIsc2Fucy1zZXJpZjsNCgljb2xvcjp3aW5kb3d0ZXh0O30NCnNwYW4uUGxhaW5U ZXh0Q2hhcg0KCXttc28tc3R5bGUtbmFtZToiUGxhaW4gVGV4dCBDaGFyIjsNCgltc28tc3R5bGUt cHJpb3JpdHk6OTk7DQoJbXNvLXN0eWxlLWxpbms6IlBsYWluIFRleHQiOw0KCWZvbnQtZmFtaWx5 OiJDYWxpYnJpIixzYW5zLXNlcmlmO30NCi5Nc29DaHBEZWZhdWx0DQoJe21zby1zdHlsZS10eXBl OmV4cG9ydC1vbmx5Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJpIixzYW5zLXNlcmlmOw0KCW1zby1m YXJlYXN0LWxhbmd1YWdlOkVOLVVTO30NCkBwYWdlIFdvcmRTZWN0aW9uMQ0KCXtzaXplOjYxMi4w cHQgNzkyLjBwdDsNCgltYXJnaW46NzIuMHB0IDcyLjBwdCA3Mi4wcHQgNzIuMHB0O30NCmRpdi5X b3JkU2VjdGlvbjENCgl7cGFnZTpXb3JkU2VjdGlvbjE7fQ0KLS0+PC9zdHlsZT48IS0tW2lmIGd0 ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlZGVmYXVsdHMgdjpleHQ9ImVkaXQiIHNwaWRtYXg9IjEw MjYiIC8+DQo8L3htbD48IVtlbmRpZl0tLT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOnNo YXBlbGF5b3V0IHY6ZXh0PSJlZGl0Ij4NCjxvOmlkbWFwIHY6ZXh0PSJlZGl0IiBkYXRhPSIxIiAv Pg0KPC9vOnNoYXBlbGF5b3V0PjwveG1sPjwhW2VuZGlmXS0tPg0KPC9oZWFkPg0KPGJvZHkgbGFu Zz0iZW4tTkwiIGxpbms9IiMwNTYzQzEiIHZsaW5rPSIjOTU0RjcyIiBzdHlsZT0id29yZC13cmFw OmJyZWFrLXdvcmQiPg0KPGRpdiBjbGFzcz0iV29yZFNlY3Rpb24xIj4NCjxwIGNsYXNzPSJNc29Q bGFpblRleHQiPjxzcGFuIGxhbmc9ImVuLU5MIj5IaSBhbGwsPG86cD48L286cD48L3NwYW4+PC9w Pg0KPHAgY2xhc3M9Ik1zb1BsYWluVGV4dCI+PHNwYW4gbGFuZz0iZW4tTkwiPjxvOnA+Jm5ic3A7 PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29QbGFpblRleHQiPjxzcGFuIGxhbmc9IkVO LVVTIj5J4oCZbSBlbmNvdW50ZXJpbmcgaTJjIGJ1cyBmYWlsdXJlcyB0aGF0IG5lZWQgbW9yZSBn cmFjZWZ1bCBlcnJvciBoYW5kbGluZy4NCjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNz PSJNc29QbGFpblRleHQiPjxzcGFuIGxhbmc9IkVOLVVTIj5UaGUgZHJpdmVyIG9wZXJhdGVzIGl0 cyBtYW5hZ2VtZW50IHRocmVhZHMgaW4gcG9sbGluZyBtb2RlIHdpdGhpbiBjYWxsb3V0IGNvbnRl eHQuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb1BsYWluVGV4dCI+PHNwYW4g bGFuZz0iZW4tTkwiPkFzIGEgaGFyZHdhcmUgY29uc3RyYWludCwgY29tbXVuaWNhdGlvbiB2aWEg c2FpZCBpMmMgYnVzIGNhbiBvbmx5PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1z b1BsYWluVGV4dCI+PHNwYW4gbGFuZz0iZW4tTkwiPmhhcHBlbiBmb3IgYSBzaW5nbGUgbmV0d29y ayBwb3J0IGF0IGEgdGltZSBhbmQgdGh1cyBuZWVkcyB0byBiZSBtdHhfbG9jaygpJ2QuPG86cD48 L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb1BsYWluVGV4dCI+PHNwYW4gbGFuZz0iZW4t TkwiPldpdGggdGhpcyBsb2NrIGhlbGQsIGkyYyB0cmFuc2ZlcnMgYXJlIGluaXRpYXRlZCBieSB0 aGUgZHJpdmVyIGFuZCB0aGUgZG9uZTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN c29QbGFpblRleHQiPjxzcGFuIGxhbmc9ImVuLU5MIj5ldmVudCBpcyBzaWduYWxlZCB2aWEgYW4g SVNSLiBTaW5jZSB3ZSBjYW5ub3Qgc2xlZXAgYm90aCBpbiBjYWxsb3V0IChub3Q8bzpwPjwvbzpw Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvUGxhaW5UZXh0Ij48c3BhbiBsYW5nPSJlbi1OTCI+ YWxsb3dlZCBhY2NvcmRpbmcgdG8gWzFdKSBhbmQgbXV0ZXggY29udGV4dCAod2lsbCBwYW5pYyks IHRoZSBzZWNvbmQgdGhyZWFkPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb1Bs YWluVGV4dCI+PHNwYW4gbGFuZz0iZW4tTkwiPndhaXRpbmcgb24gdGhlIGJ1cyB3aWxsIHNwaW4g aWYgaXQgZG9lcyBub3QgbGl2ZSBvbiB0aGUgc2FtZSBDUFVbMV0sIGFuZCB0aGU8bzpwPjwvbzpw Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvUGxhaW5UZXh0Ij48c3BhbiBsYW5nPSJlbi1OTCI+ aTJjIHRyYW5zZmVyIGl0c2VsZiBpcyBoYW5kbGVkIHZpYSBhIERFTEFZKCkgYnVzeSBsb29wLiBJ ZiB0aGUgYnVzIGZhaWxzLDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29QbGFp blRleHQiPjxzcGFuIGxhbmc9ImVuLU5MIj53ZSBoYXZlIDIgY29yZXMgc3Bpbm5pbmcgYXQgMTAw JSwgY2F1c2luZyBjb25uZWN0aW9ucyB0byBkcm9wIGFuZCBhbGw8bzpwPjwvbzpwPjwvc3Bhbj48 L3A+DQo8cCBjbGFzcz0iTXNvUGxhaW5UZXh0Ij48c3BhbiBsYW5nPSJlbi1OTCI+c29ydHMgb2Yg b3RoZXIgY2FybmFnZSB0byBoYXBwZW4uIFJlZHVjaW5nIHRoZSB0aW1lb3V0IGlzIG5vdCBhbiBv cHRpb24sIGFzPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb1BsYWluVGV4dCI+ PHNwYW4gbGFuZz0iZW4tTkwiPnRoZSB0cmFuc2ZlcnMgbWlnaHQgdGFrZSBhIHdoaWxlLiA8bzpw Pg0KPC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29QbGFpblRleHQiPjxzcGFuIGxhbmc9 ImVuLU5MIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvUGxhaW5U ZXh0Ij48c3BhbiBsYW5nPSJlbi1OTCI+U2xlZXBpbmcgdW5kZXIgYSBNVFhfREVGIGlzIG5vdDwv c3Bhbj48c3BhbiBsYW5nPSJlbi1OTCI+DQo8L3NwYW4+PHNwYW4gbGFuZz0iZW4tTkwiPnBvc3Np YmxlLCBpbnN0YW50IHBhbmljLiBIb3dldmVyLCBpZiBJIHJlcGxhY2UgPG86cD48L286cD48L3Nw YW4+PC9wPg0KPHAgY2xhc3M9Ik1zb1BsYWluVGV4dCI+PHNwYW4gbGFuZz0iZW4tTkwiPnRoZSBm aXJzdCBtdHhfbG9jazwvc3Bhbj48c3BhbiBsYW5nPSJFTi1VUyI+KCk8L3NwYW4+PHNwYW4gbGFu Zz0iZW4tTkwiPiB3aXRoIGE8L3NwYW4+PHNwYW4gbGFuZz0iZW4tTkwiPg0KPC9zcGFuPjxzcGFu IGxhbmc9ImVuLU5MIj5sb2NrbWdyKCkgY2FsbCB0aGF0IHdhcyBpbml0aWFsaXplZCB3aXRoIExL X1RJTUVMT0NLIHwgPG86cD4NCjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvUGxhaW5U ZXh0Ij48c3BhbiBsYW5nPSJlbi1OTCI+TEtfQ0FOUkVDVVJTRSBhbmQgYSBnaXZlbjwvc3Bhbj48 c3BhbiBsYW5nPSJlbi1OTCI+DQo8L3NwYW4+PHNwYW4gbGFuZz0iZW4tTkwiPnRpbWVvdXQsIEkn bSBhYmxlIHRvIHB1dCB0aGUgdGhyZWFkIHdhaXRpbmcgb248bzpwPjwvbzpwPjwvc3Bhbj48L3A+ DQo8cCBjbGFzcz0iTXNvUGxhaW5UZXh0Ij48c3BhbiBsYW5nPSJlbi1OTCI+dGhlIGNvbW1zIGJ1 cyB0byBiZSBhdmFpbGFibGUgdG88L3NwYW4+PHNwYW4gbGFuZz0iZW4tTkwiPg0KPC9zcGFuPjxz cGFuIGxhbmc9ImVuLU5MIj5zbGVlcC4gSWYgSSByZXBsYWNlIHRoZSBoYXJkY29kZWQgREVMQVko KXM8L3NwYW4+PHNwYW4gbGFuZz0iZW4tTkwiPg0KPC9zcGFuPjxzcGFuIGxhbmc9IkVOLVVTIj48 bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvUGxhaW5UZXh0Ij48c3BhbiBsYW5n PSJlbi1OTCI+ZnVydGhlciBkb3duIHRoZSBsaW5lIGR1cmluZyB0aGU8L3NwYW4+PHNwYW4gbGFu Zz0iZW4tTkwiPg0KPC9zcGFuPjxzcGFuIGxhbmc9ImVuLU5MIj5pMmMgdHJhbnNmZXJzIHdpdGgg bXR4X3NsZWVwKCkgKGFuZCB3YWtldXBfb25lKCkgaW4gdGhlIElTUiksDQo8bzpwPjwvbzpwPjwv c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvUGxhaW5UZXh0Ij48c3BhbiBsYW5nPSJlbi1OTCI+bm8g Q1BVPC9zcGFuPjxzcGFuIGxhbmc9ImVuLU5MIj4gPC9zcGFuPg0KPHNwYW4gbGFuZz0iZW4tTkwi PmN5Y2xlcyBhcmUgd2FzdGVkLCB0aGlzIGlzIGFsc28gdGhlIGNhc2UgZm9yIG5vcm1hbCBvcGVy YXRpb248L3NwYW4+PHNwYW4gbGFuZz0iRU4tVVMiPiw8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8 cCBjbGFzcz0iTXNvUGxhaW5UZXh0Ij48c3BhbiBsYW5nPSJFTi1VUyI+aW1wcm92aW5nIHRoZSBj dXJyZW50IHNpdHVhdGlvbiBieSBhIG1pbGUuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xh c3M9Ik1zb1BsYWluVGV4dCI+PHNwYW4gbGFuZz0iZW4tTkwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9z cGFuPjwvcD4NCjxwIGNsYXNzPSJNc29QbGFpblRleHQiPjxzcGFuIGxhbmc9ImVuLU5MIj5BcyB0 byB0aGU8L3NwYW4+PHNwYW4gbGFuZz0iZW4tTkwiPiA8L3NwYW4+DQo8c3BhbiBsYW5nPSJlbi1O TCI+bmF0dXJlIG9mIHRoZXNlIGJ1cyBmYWlsdXJlcywgdGhleSBzZWVtIHRvIGhhcHBlbiB2ZXJ5 IGluZnJlcXVlbnRseSBhbmQgbmlnaDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN c29QbGFpblRleHQiPjxzcGFuIGxhbmc9ImVuLU5MIj5vbiBpbXBvc3NpYmxlIHRvIHJlcHJvZHVj ZSBvbiBhIHByb2R1Y3Rpb24gc3lzdGVtLCBidXQgdGhleSBhbHdheXMgc2VlbTxvOnA+PC9vOnA+ PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29QbGFpblRleHQiPjxzcGFuIGxhbmc9ImVuLU5MIj50 byByZWNvdmVyIGFmdGVyIGEgY2VydGFpbiBhbW91bnQgb2YgdGltZS4gVG8gcmVwcm9kdWNlIHRo ZSBpc3N1ZSwgSSBoYXZlPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb1BsYWlu VGV4dCI+PHNwYW4gbGFuZz0iZW4tTkwiPndpcmVkIHNvbWUgcGh5c2ljYWwgc3dpdGNoZXMgdG8g dGhlIGkyYyBidXMgdG8gc2ltdWxhdGUgdGhlIG91dGFnZXMuJm5ic3A7DQo8bzpwPjwvbzpwPjwv c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvUGxhaW5UZXh0Ij48c3BhbiBsYW5nPSJlbi1OTCI+PG86 cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb1BsYWluVGV4dCI+PHNwYW4g bGFuZz0iZW4tTkwiPkZyb20gYTwvc3Bhbj48c3BhbiBsYW5nPSJlbi1OTCI+IDwvc3Bhbj4NCjxz cGFuIGxhbmc9ImVuLU5MIj5kZXNpZ24gcGVyc3BlY3RpdmUsIHRoZSBpMmMgYnVzIHNob3VsZCBi ZSBhIHNpbmdsZXRvbiBhbmQgc2hvdWxkIGJlIHRyZWF0ZWQ8bzpwPjwvbzpwPjwvc3Bhbj48L3A+ DQo8cCBjbGFzcz0iTXNvUGxhaW5UZXh0Ij48c3BhbiBsYW5nPSJlbi1OTCI+YXMgc3VjaCB2aWEg dGhlIHF1ZXVlaW5nIG9mIHRyYW5zZmVyIHRhc2tzIGluY2x1ZGluZyBtZXRhZGF0YSB0byBsaW5r PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb1BsYWluVGV4dCI+PHNwYW4gbGFu Zz0iZW4tTkwiPnN1Y2ggdHJhbnNmZXJzIGJhY2sgdG8gc3BlY2lmaWMgcG9ydCBpZHMsIGhvd2V2 ZXIsIGEgY2hhbmdlIGxpa2UgdGhpcyB3b3VsZDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNs YXNzPSJNc29QbGFpblRleHQiPjxzcGFuIGxhbmc9ImVuLU5MIj5yZXF1aXJlIHNpZ25pZmljYW50 IGVmZm9ydCB3aXRoIGEgbGFyZ2UgcmlzayBvZiByZWdyZXNzaW9ucywgYW5kIEknbSBub3QgZXZl bjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29QbGFpblRleHQiPjxzcGFuIGxh bmc9ImVuLU5MIj5zdXJlIGlmIHNsZWVwaW5nIGluIHN1Y2ggYSBzY2VuYXJpbyB3b3VsZCBiZSBj b25zaWRlcmVkIGEgdmFsaWQgdGhpbmcgdG8gZG88bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBj bGFzcz0iTXNvUGxhaW5UZXh0Ij48c3BhbiBsYW5nPSJlbi1OTCI+aW4gRnJlZUJTRCBjb250ZXh0 LCBzbyBJIHdvdWxkIHByZWZlciBzdGlja2luZyB0byB0aGUgYmF0dGxlLWhhcmRlbmVkPG86cD48 L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb1BsYWluVGV4dCI+PHNwYW4gbGFuZz0iZW4t TkwiPmNvZGUuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb1BsYWluVGV4dCI+ PHNwYW4gbGFuZz0iZW4tTkwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNz PSJNc29QbGFpblRleHQiPjxzcGFuIGxhbmc9ImVuLU5MIj5UaGUgZG9jdW1lbnRhdGlvbiBmb3Ig bG9ja21ncigpIHRlbGxzIG1lIHRvIHN0YXkgYXdheSBmcm9tIGl0LCBidXQgSTxvOnA+PC9vOnA+ PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29QbGFpblRleHQiPjxzcGFuIGxhbmc9ImVuLU5MIj5z ZWVtIHRvIGhhdmUgbGl0dGxlIG9wdGlvbnMgbGVmdC4gPC9zcGFuPg0KPHNwYW4gbGFuZz0iRU4t VVMiPkFsc28sIHRoZSBzb2x1dGlvbiBzZWVtcyB0byB2aW9sYXRlPG86cD48L286cD48L3NwYW4+ PC9wPg0KPHAgY2xhc3M9Ik1zb1BsYWluVGV4dCI+PHNwYW4gbGFuZz0iRU4tVVMiPnRoZSBGcmVl QlNEIGd1aWRlbGluZXMgZm9yIHNsZWVwaW5nIGluIGNhbGxvdXQgY29udGV4dC4NCjwvc3Bhbj48 c3BhbiBsYW5nPSJlbi1OTCI+QW55IHRob3VnaHRzPyA8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8 cCBjbGFzcz0iTXNvUGxhaW5UZXh0Ij48c3BhbiBsYW5nPSJlbi1OTCI+PG86cD4mbmJzcDs8L286 cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb1BsYWluVGV4dCI+PHNwYW4gbGFuZz0iRU4tVVMi PlRoYW5rcyBpbiBhZHZhbmNlLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Q bGFpblRleHQiPjxzcGFuIGxhbmc9ImVuLU5MIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+ DQo8cCBjbGFzcz0iTXNvUGxhaW5UZXh0Ij48c3BhbiBsYW5nPSJlbi1OTCI+WzFdPC9zcGFuPjxz cGFuIGxhbmc9ImVuLU5MIj4gPC9zcGFuPjxzcGFuIGxhbmc9ImVuLU5MIj48YSBocmVmPSJodHRw czovL21hbi5mcmVlYnNkLm9yZy9jZ2kvbWFuLmNnaT9sb2NraW5nKDkpIj5odHRwczovL21hbi5m cmVlYnNkLm9yZy9jZ2kvbWFuLmNnaT9sb2NraW5nKDkpPC9hPjxvOnA+PC9vOnA+PC9zcGFuPjwv cD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9ImVuLU5MIj48bzpwPiZuYnNwOzwv bzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjwvYm9keT4NCjwvaHRtbD4NCg== --_000_AM6P193MB0311A026447E2B8523F4AE89BC0AAAM6P193MB0311EURP_-- From nobody Thu Aug 3 07:32:22 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RGgYb3DgSz4mN1Y for ; Thu, 3 Aug 2023 07:33:07 +0000 (UTC) (envelope-from shrikanth07@gmail.com) Received: from mail-ot1-x334.google.com (mail-ot1-x334.google.com [IPv6:2607:f8b0:4864:20::334]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RGgYY4d5nz3Gyq for ; Thu, 3 Aug 2023 07:33:05 +0000 (UTC) (envelope-from shrikanth07@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=dhQh0fE4; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of shrikanth07@gmail.com designates 2607:f8b0:4864:20::334 as permitted sender) smtp.mailfrom=shrikanth07@gmail.com Received: by mail-ot1-x334.google.com with SMTP id 46e09a7af769-6bc57401cb9so139727a34.0 for ; Thu, 03 Aug 2023 00:33:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691047984; x=1691652784; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=MzjS9HoPFUv2gDOcQklSS5FgsYlHNY96j4duo77g2/A=; b=dhQh0fE4C4vUl+lmEclkw3dDO+/NFKQlDvuKI7SX1CPetGTCCCkhhIrORZte2txYjm /yumEuSyB6uHCULiK1zm9xrVduTUSSdXHcHTc3XtOCUQ0ORGDEA3NQam+MqG1UDUvB1G 6PL3tuarntQTkJ5rdOicblrql4/m84c9x+5FyR6vh99LbpD+edmMFA3HGsxudnmsUp+o ASQbpt+Uv+FLSgjC646avVYv5DB/GScm44WP0Xpd0U7scalFYiQflkZ1k/PuUKx0xKoN 0YG2j2vqBTH2U/xN7PvOtwyX1xGUh3zg3MZQdJk8bU7BgtpTO26/3j0yQTqnnEDE4X2T XJtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691047984; x=1691652784; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=MzjS9HoPFUv2gDOcQklSS5FgsYlHNY96j4duo77g2/A=; b=aUxQsVg8h/VSZ++9XNGo9gGDMuNZmC/AOJpBjUZikAOA7DCGoeswX4ZmOnhg6j5r6n LBazhIjXnf/Z4jwTtQhfV6q99T9OYnLwXzVxpzg3W2wks+CQ0Y7nAHI6P8DkAH5ERUo8 Snh9RH3E6HFbw8Mp3eYuMA8CMDZ0FYvVdXiXX5FtsMi0FE1l/GtoumNb1CZ/XCsbn3LC AVKTE/UdRl2jjnfJEEtRluABHuBm3N+JbUKR9dDI/TdZuq8yQjoPkaQMxvt5KnVaSKet TGuTt9qDmsbFX59GPe01h0SztJ2wJYtvbxKL/3a06eUG3CS4kgyWAUK2Tw4ZTr+ea4eB Y4kw== X-Gm-Message-State: ABy/qLY/sVFXkm0MGliS7leS94FJl3HSWCGX7Qe8QCkhivr7/0F5Mnk1 zNagbUPAVhBaQ4Wr3oVC7a0eeLZoi11vR8DUWS+0ZwRJMog= X-Google-Smtp-Source: APBJJlGpaYG8gBhU3O5i9dFlRstnD1qc20CyDeXni2pQMoMAICSQlqNsO5M0799NPsNtCOcg9vTNPvIWr7+5p/8VmSE= X-Received: by 2002:a05:6808:200b:b0:3a7:5724:8bfd with SMTP id q11-20020a056808200b00b003a757248bfdmr3912441oiw.1.1691047983426; Thu, 03 Aug 2023 00:33:03 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 From: Shrikanth Kamath Date: Thu, 3 Aug 2023 00:32:22 -0700 Message-ID: Subject: How to watch Active pagequeue transitions with DTrace in the vm layer To: freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="00000000000059b15c0601ffcacf" X-Spamd-Result: default: False [-2.43 / 15.00]; NEURAL_HAM_LONG(-0.94)[-0.944]; NEURAL_HAM_SHORT(-0.60)[-0.603]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_SPAM_MEDIUM(0.11)[0.112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::334:from]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4RGgYY4d5nz3Gyq X-Spamd-Bar: -- --00000000000059b15c0601ffcacf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable A background on the query, Trying to catch a memory =E2=80=9Cspike=E2=80=9D trigger using DTrace, refe= r here two =E2=80=9Ctop=E2=80=9D snapshots captured during a 2 minute window, last pid: 89900; load averages: 0.75, 0.91, 0.94 up 39+00:37:30 20:03:14 Mem: 5575M Active, 2152M Inact, 4731M Laundry, 3044M Wired, 1151M Buf, 382M Free Swap: 8192M Total, 1058M Used, 7134M Free, 12% Inuse PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12043 root 5 35 0 11G 9747M kqread 3 128.8H 23.34% app1 12051 root 1 20 0 3089M 2274M select 1 22:51 0.00% app2 last pid: 90442; load averages: 1.50, 1.12, 1.02 up 39+00:39:37 20:05:21 Mem: 8549M Active, 631M Inact, 3340M Laundry, 3159M Wired, 1252M Buf, 359M Free Swap: 8192M Total, 1894M Used, 6298M Free, 23% Inuse PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12043 root 5 24 0 11G 9445M kqread 2 128.8H 10.45% app1 12051 root 1 20 0 3089M 2173M select 3 22:51 0.00% app2 The spike is ~3G in Active pages, the two large applications have a combined resident size of ~12G. The resident size of the applications hasn=E2=80=99t changed between these 2 readings, however there is a tar arc= hive and gzip on a large directory during that window likely causing a reshuffle. If I count the page allocs and dequeue by execname with DTrace, I see tar/vmstat which probably alloc and quickly dequeue, along with a large dequeue being undertaken by bufdaemon and pagedaemon. fbt::vm_page_alloc*:entry { @cnt[execname] =3D count(); } fbt::vm_page_dequeue:entry { @dcnt[execname] =3D count(); } Page Alloc vmstat 20222 tar 21284 Page Dequeue vmstat 20114 bufdaemon 21402 tar 21635 pagedaemon 360387 Since the tar / vmstat will not hold the pages in Active, I need to find out what application had its pages queued in Active page queue. Is it possible that the system is just moving the LRU pages of these two large applications into the inactive queue prior to addressing memory pressure? Do these applications need to activate those pages later and hence it brings it back into the Active queue? How do I watch this in action by using DTrace? Will the following probe catch this trigger? fbt::vm_page_activate:entry { @cnt[execname, pid] =3D count(); } tick-10sec { printa(@cnt); printf("ACTIVE[%d] pages\n", `vm_dom[0].vmd_pagequeues[1].pq_cnt); } *** This system is running only one vmdomain (# sysctl vm.ndomains =E2=80= =93> vm.ndomains: 1). *** running release 12.1, on an amd64 kernel. The physical memory installed is 16G. Regards, --=20 Shrikanth R K --00000000000059b15c0601ffcacf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
A background on the query,
=

= Trying to catch a memory =E2=80=9Cspike=E2=80=9D trigger= using DTrace, refer here two =E2=80=9Ctop=E2=80=9D snapshots captured duri= ng a 2 minute window,=C2=A0


last pid: 8990= 0;=C2=A0 load averages:=C2=A0 0.75,=C2=A0 0.91,=C2=A0 0.94=C2=A0 up 39+00:3= 7:30=C2=A0 =C2=A0 20:03:14

Mem: 5575M Active, = 2152M Inact, 4731M Laundry, 3044M Wired, 1151M Buf, 382M Free

Swap: 8192M Total, 1058M Used, 7134M Free, 12% Inuse=

PID USERNAME=C2=A0 =C2=A0 THR PRI NICE =C2=A0 SIZE= =C2=A0 RES =C2=A0 STATE =C2=A0 C TIME=C2=A0 =C2=A0 WCPU COMMAND

<= p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><= span style=3D"font-size:13pt;font-family:"Arial Narrow",sans-seri= f;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-al= ternates:normal;font-variant-east-asian:normal;vertical-align:baseline;whit= e-space:pre-wrap">12043=C2=A0 =C2=A0 root =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 5=C2=A0 =C2=A0 35 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2= =A0 11G=C2=A0 9747M kqread=C2=A0 3 128.8H=C2=A0 23.34% app1

12051=C2=A0 =C2=A0 root =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 1=C2=A0 =C2=A0 20 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 3089M=C2=A0 22= 74M select =C2=A0 1=C2=A0 22:51 =C2=A0 0.00%=C2=A0 =C2=A0 app2

last pid: 90442;=C2=A0 load averages:=C2=A0 1.50,=C2= =A0 1.12,=C2=A0 1.02=C2=A0 up 39+00:39:37=C2=A0 =C2=A0 20:05:21

<= p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><= span style=3D"font-size:13pt;font-family:"Arial Narrow",sans-seri= f;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-al= ternates:normal;font-variant-east-asian:normal;vertical-align:baseline;whit= e-space:pre-wrap">Mem: 8549M Active, 631M Inact, 3340M Laundry, 3159M Wired= , 1252M Buf, 359M Free

Swap: 8192M Total, 1894= M Used, 6298M Free, 23% Inuse

PID =C2=A0 USERN= AME =C2=A0 THR PRI NICE =C2=A0 SIZE=C2=A0 =C2=A0 RES STATE =C2=A0 C TIME = =C2=A0 WCPU=C2=A0 COMMAND

12043 =C2=A0 =C2=A0 = =C2=A0 root=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 5=C2=A0 24=C2= =A0 =C2=A0 0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 11G=C2=A0 9445M kqread=C2=A0 2 128= .8H 10.45%=C2=A0 app1

12051 =C2=A0 =C2=A0 =C2= =A0 root=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1=C2=A0 20=C2=A0 = =C2=A0 0 =C2=A0 =C2=A0 3089M=C2=A0 2173M select =C2=A0 3=C2=A0 22:51 =C2=A0= 0.00%=C2=A0 =C2=A0 app2


The spike is ~3G = in Active pages, the two large applications have a combined resident size o= f ~12G. The resident size of the applications hasn=E2=80=99t changed betwee= n these 2 readings, however there is a tar archive and gzip on a large dire= ctory during that window likely causing a reshuffle. If I count the page al= locs and dequeue by execname with DTrace, I see tar/vmstat which probably a= lloc and quickly dequeue, along with a large dequeue being undertaken by bu= fdaemon and pagedaemon.


fbt::vm_page_alloc= *:entry

{

=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0@cnt[execname] =3D count();

}

fbt::vm_page_dequeue= :entry

{

=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0@dcnt[execname] =3D count();

}


Page Alloc

=C2=A0=C2=A0vmstat=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 20222=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0

=C2=A0=C2=A0tar=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 21284

Page Dequeue=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0

=C2=A0=C2=A0vmstat=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 20114

=C2= =A0=C2=A0bufdaemon =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 21402

= =C2=A0=C2=A0tar =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 21635

=C2=A0=C2=A0pagedaemon =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 3= 60387


Since the tar / vmstat will not hold= the pages in Active, I need to find out what application had its pages que= ued in Active page queue.


Is it possible t= hat the system is just moving the LRU pages of these two large applications= into the inactive queue prior to addressing memory pressure?=C2=A0 Do thes= e applications need to activate those pages later and hence it brings it ba= ck into the Active queue? How do I watch this in action by using DTrace? Wi= ll the following probe catch this trigger?


fbt::vm_page_activate:entry

{

@cnt[execname, pid] =3D count();

<= p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><= span style=3D"font-size:13pt;font-family:"Arial Narrow",sans-seri= f;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-al= ternates:normal;font-variant-east-asian:normal;vertical-align:baseline;whit= e-space:pre-wrap">}

tick-10sec

{

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0printa(@cnt);

=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0printf("ACTIVE[%d] pages\n", `vm_dom[0].v= md_pagequeues[1].pq_cnt);

}


*** This system is running only one vmdomain (# sysctl vm.nd= omains =E2=80=93>=C2=A0 vm.ndomains: 1).

*= ** running release 12.1, on an amd64 kernel. The physical memory installed = is 16G.


Regards,
--
Shrikanth R K
--00000000000059b15c0601ffcacf-- From nobody Thu Aug 3 15:14:44 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RGspJ4Z5Kz3cK22 for ; Thu, 3 Aug 2023 15:14:48 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qv1-xf34.google.com (mail-qv1-xf34.google.com [IPv6:2607:f8b0:4864:20::f34]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RGspJ24jnz4Pk7 for ; Thu, 3 Aug 2023 15:14:48 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qv1-xf34.google.com with SMTP id 6a1803df08f44-63cf57c79b5so6687046d6.0 for ; Thu, 03 Aug 2023 08:14:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691075687; x=1691680487; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:sender :from:to:cc:subject:date:message-id:reply-to; bh=kaIWtj37fXHi8dXeaknFIboxPhSn7t3HLwpmm6rRB5Y=; b=nkUgh8odMJhxaoTx1tdstOR4i2GPpLw/r8uufloIRWrznjaxcbA5KKi1s0fZxEKBhz jiG4iOWIqsmDSEZwdB5Z4BpkWA+S/ewontVI3nE4DzjHmyfoSVuGj/wtVKSWEIXoAFMP /sL0VMWY1j/7wiNyNGnPCffAcKMbvgsCkRyQvnqDMfmBdT1D84oh9PgzRcwn9+Ww9Kga V+yn3lnqTpyfoZBX2Iv8sm8VtimxacMl/DcEpGCFOpSK3ouGe+ORZ950arl8Xf8k3zIt WO8NXpKF6WjxEKMI8AoQbWo8LJ5N0J8Eh/OSQRsojSSamWAL2aAu6a5D+YO33uhHypDp 6hyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691075687; x=1691680487; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kaIWtj37fXHi8dXeaknFIboxPhSn7t3HLwpmm6rRB5Y=; b=OAoTyeC1OCCnNapvB6kNC/YVvbtU4uU4XWz3CWaRVtHNE4kWRx4bRV/GUlIjsFHzyi kEBVo8zszX2awzvMatRS5wTlKHBFAowJISm16DFYNCfcArTZ8JXwhIaLDHJFWoUCbHRg AOy9cr4hyJL7lsWPlUZcmqgzZ+z/97PEfQvk4WhXNjaJHecfel7gGvtto1imzr4bPI+z IL9LMvx6sFMev9gv29vd/wpz5NOUyzEl7+pycVaFUQDktFeDwh5eh5cE5beVLvb8/M1x QVY0o+Kn+YCfy8+qLQAsA8mjs+/VlhWo7ASwsn9t8vM5ByZnJVj//49Pwa/m7FGNaOYt 50Fw== X-Gm-Message-State: ABy/qLaHmEwKMakE1CH0QEw6UnuwtdxRsHht9o3rkOFLBdVH3POysh1s vjwgQ2BcVc2PaAU7s9ujWNWvxQ4uNn8= X-Google-Smtp-Source: APBJJlEXvZRGUrnscggi1rB0OeIz8lu1+HcJHEGdyD2VtsVW3eaNfS50W+jXPMUTMgU8fWyL6f5hSg== X-Received: by 2002:a0c:cb09:0:b0:63c:fb1b:6bb0 with SMTP id o9-20020a0ccb09000000b0063cfb1b6bb0mr18095925qvk.52.1691075687348; Thu, 03 Aug 2023 08:14:47 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id p6-20020a0cf546000000b0063cf49c35f1sm6479878qvm.35.2023.08.03.08.14.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Aug 2023 08:14:46 -0700 (PDT) Date: Thu, 3 Aug 2023 11:14:44 -0400 From: Mark Johnston To: Shrikanth Kamath Cc: freebsd-hackers@freebsd.org Subject: Re: How to watch Active pagequeue transitions with DTrace in the vm layer Message-ID: References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 4RGspJ24jnz4Pk7 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] On Thu, Aug 03, 2023 at 12:32:22AM -0700, Shrikanth Kamath wrote: > A background on the query, > > Trying to catch a memory “spike” trigger using DTrace, refer here two “top” > snapshots captured during a 2 minute window, > > last pid: 89900; load averages: 0.75, 0.91, 0.94 up 39+00:37:30 > 20:03:14 > > Mem: 5575M Active, 2152M Inact, 4731M Laundry, 3044M Wired, 1151M Buf, 382M > Free > > Swap: 8192M Total, 1058M Used, 7134M Free, 12% Inuse > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > > 12043 root 5 35 0 11G 9747M kqread 3 > 128.8H 23.34% app1 > > 12051 root 1 20 0 3089M 2274M select 1 22:51 > 0.00% app2 > > last pid: 90442; load averages: 1.50, 1.12, 1.02 up 39+00:39:37 > 20:05:21 > > Mem: 8549M Active, 631M Inact, 3340M Laundry, 3159M Wired, 1252M Buf, 359M > Free > > Swap: 8192M Total, 1894M Used, 6298M Free, 23% Inuse > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > > 12043 root 5 24 0 11G 9445M kqread 2 > 128.8H 10.45% app1 > > 12051 root 1 20 0 3089M 2173M select 3 22:51 > 0.00% app2 > > The spike is ~3G in Active pages, the two large applications have a > combined resident size of ~12G. The resident size of the applications > hasn’t changed between these 2 readings, however there is a tar archive and > gzip on a large directory during that window likely causing a reshuffle. If > I count the page allocs and dequeue by execname with DTrace, I see > tar/vmstat which probably alloc and quickly dequeue, along with a large > dequeue being undertaken by bufdaemon and pagedaemon. > > fbt::vm_page_alloc*:entry > > { > > @cnt[execname] = count(); > > } > > fbt::vm_page_dequeue:entry > > { > > @dcnt[execname] = count(); > > } > > Page Alloc > > vmstat > 20222 > > tar 21284 > > Page Dequeue > > vmstat 20114 > > bufdaemon 21402 > > tar 21635 > > pagedaemon 360387 > > Since the tar / vmstat will not hold the pages in Active, I need to find > out what application had its pages queued in Active page queue. One possibility is that the inactive and laundry queue had previously contained many referenced pages. Then, when some memory pressure occurred, the pagedaemon scanned the queues and moved a large number of pages into the active queue. > Is it possible that the system is just moving the LRU pages of these two > large applications into the inactive queue prior to addressing memory > pressure? Do these applications need to activate those pages later and > hence it brings it back into the Active queue? How do I watch this in > action by using DTrace? Will the following probe catch this trigger? > > fbt::vm_page_activate:entry > > { > > @cnt[execname, pid] = count(); > > } > > tick-10sec > > { > > printa(@cnt); > > printf("ACTIVE[%d] pages\n", `vm_dom[0].vmd_pagequeues[1].pq_cnt); > > } > > *** This system is running only one vmdomain (# sysctl vm.ndomains –> > vm.ndomains: 1). > > *** running release 12.1, on an amd64 kernel. The physical memory installed > is 16G. In 12.1, you'd probably want something like: fbt::vm_page_enqueue:entry /args[1] == 1 /* PQ_ACTIVE *// { ... } since vm_page_unwire(m, PQ_ACTIVE) will also move a page into the active queue, but your D script above won't catch that. I would also look at the "pages reactivated by the page daemon" counter that appears in vmstat -s output. That'll tell you how many times the page daemon moved a page from PQ_INACTIVE/PQ_LAUNDRY to PQ_ACTIVE because it found that the page had been referenced. From nobody Thu Aug 3 17:51:57 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RGxHh5FFhz4TlMX; Thu, 3 Aug 2023 17:52:00 +0000 (UTC) (envelope-from sanastasio@raptorengineering.com) Received: from raptorengineering.com (mail.raptorengineering.com [23.155.224.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RGxHg623Sz4ky9; Thu, 3 Aug 2023 17:51:59 +0000 (UTC) (envelope-from sanastasio@raptorengineering.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=raptorengineering.com header.s=B8E824E6-0BE2-11E6-931D-288C65937AAD header.b=M6CZ4diZ; spf=pass (mx1.freebsd.org: domain of sanastasio@raptorengineering.com designates 23.155.224.40 as permitted sender) smtp.mailfrom=sanastasio@raptorengineering.com; dmarc=pass (policy=quarantine) header.from=raptorengineering.com Received: from localhost (localhost [127.0.0.1]) by mail.rptsys.com (Postfix) with ESMTP id F2AD78285406; Thu, 3 Aug 2023 12:51:58 -0500 (CDT) Received: from mail.rptsys.com ([127.0.0.1]) by localhost (vali.starlink.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id WHesxDq3Ddyv; Thu, 3 Aug 2023 12:51:58 -0500 (CDT) Received: from localhost (localhost [127.0.0.1]) by mail.rptsys.com (Postfix) with ESMTP id 38EDC8285487; Thu, 3 Aug 2023 12:51:58 -0500 (CDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.rptsys.com 38EDC8285487 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raptorengineering.com; s=B8E824E6-0BE2-11E6-931D-288C65937AAD; t=1691085118; bh=4Vxg2XWsA+qqr2qM1uCaK1pdP62hIVl9UNC0n3PncmM=; h=Message-ID:Date:MIME-Version:From:To; b=M6CZ4diZaEnLKE2cZZP2jmAn2rYkI1hjrPXsEgGyt/VlVEWROZbHwcwJI9vVnoDwP gyWg/UV9nlzSAOSwgU2q0/jAbeMBuTjWydgiFomYSf0QOesPyR6ex53NROki6yL6Nq M4ExyFp4OUYBO0k8okMu0bGmxPtUG6XY2Pxvl1CA= X-Virus-Scanned: amavisd-new at rptsys.com Received: from mail.rptsys.com ([127.0.0.1]) by localhost (vali.starlink.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 4Bon8-AhG5I6; Thu, 3 Aug 2023 12:51:58 -0500 (CDT) Received: from [10.11.0.2] (5.edge.rptsys.com [23.155.224.38]) by mail.rptsys.com (Postfix) with ESMTPSA id C86158285406; Thu, 3 Aug 2023 12:51:57 -0500 (CDT) Message-ID: <0c24b4b7-b4c8-242d-6187-15b171c50c19@raptorengineering.com> Date: Thu, 3 Aug 2023 12:51:57 -0500 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux ppc64le; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Content-Language: en-US From: Shawn Anastasio Subject: Implementing in-kernel AES crypto acceleration on ppc (POWER8+) To: freebsd-ppc@FreeBSD.org Cc: freebsd-hackers@FreeBSD.org, Timothy Pearson Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-2.00 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[raptorengineering.com,quarantine]; R_SPF_ALLOW(-0.20)[+mx:c]; R_DKIM_ALLOW(-0.20)[raptorengineering.com:s=B8E824E6-0BE2-11E6-931D-288C65937AAD]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@FreeBSD.org,freebsd-ppc@FreeBSD.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[raptorengineering.com:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:46246, ipnet:23.155.224.0/24, country:US]; RCVD_COUNT_FIVE(0.00)[5]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Spamd-Bar: - X-Rspamd-Queue-Id: 4RGxHg623Sz4ky9 Hello all, Raptor Engineering is interested in adding support for in-kernel AES acceleration on ppc64 via the VMX crypto instructions added in ISA 2.07B, and I wanted to reach out to the community with a few questions. 1. As I understand it, FreeBSD already has support for in-kernel crypto acceleration on x86 and ARM via the aesni and armv8_crypto drivers respectively that each implement the cryptodev interface. Am I correct in understanding that adding AES acceleration for Power would just involve creating another driver here, or are there other pieces of the puzzle that I've missed? 2. I see that both the aesni and armv8 drivers make use of the fpu_kern_enter/fpu_kern_leave functions to guard access to vector registers, but it appears that these functions aren't implemented on ppc. Is that correct, or does an in-kernel facility for safely accessing vector registers on ppc already exist? 3. For the accelerated AES implementation itself, I've noticed that cryptogams[*] contains an implementation that is both widely deployed (and thus tested and likely to be correct) and also BSD licensed. Would it be acceptable to import the relevant routines to the FreeBSD kernel and have the new cryptodev driver simply call into them, or are there other considerations involved? 4. Is there a userspace test framework for the cryptodev API that could be used to validate and benchmark the new implementation, or would I have to write that myself? It appears that OpenSSL had support for /dev/crypto at one point, but I'm not sure that is the case any longer. My apologies for the large number of questions, but I look forward to hearing back and working with the FreeBSD community to get this implemented. Thanks, Shawn Anastasio [*] https://github.com/dot-asm/cryptogams From nobody Fri Aug 4 08:31:04 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RHJq14BXtz4gqT2 for ; Fri, 4 Aug 2023 08:31:57 +0000 (UTC) (envelope-from shrikanth07@gmail.com) Received: from mail-oo1-xc32.google.com (mail-oo1-xc32.google.com [IPv6:2607:f8b0:4864:20::c32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RHJq12Mxmz4CpV; Fri, 4 Aug 2023 08:31:57 +0000 (UTC) (envelope-from shrikanth07@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oo1-xc32.google.com with SMTP id 006d021491bc7-56475c1b930so309793eaf.0; Fri, 04 Aug 2023 01:31:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691137916; x=1691742716; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=OEXoWo03rm6awV3YelmRM6lxKh9eXYNAQnC4MsRhr8w=; b=jZe87ckF5jfZzlMaQH2cVOKrReDIAg+JWXNHIs8p0IpYzliqgNY9lCa6K5N48DesyN iKAlFhWzqPYJ7i0xaW6AGRPLd+7izms/+BGqKH7IkWNHups3ldK1AsHU50Ff9rHelZC+ d5Xy+9j8jLmbvierzE/Zict2cUzTYTUUYIErUyTBehJCTPPfSIuu0p6p2ehM4z44ShCB HYDpMrFWQY7GZW3PVqtza0QA8wMLd+BFD7oKsUshwX0NoITwnFv54QAg0w2KI7iRWzFS RPxKRbYzAb5ZS8Wxs7i3eXLZlbnzrQvP8g8Jpzcgz0a+iaoUVO8Y8e+u3lDVyEcFzYmz PESg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691137916; x=1691742716; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=OEXoWo03rm6awV3YelmRM6lxKh9eXYNAQnC4MsRhr8w=; b=SkH/k2bLDk4I5V24Ql4S95Vy6Q5FqFxfwl3yHCmbgEarXeMJOW36HcZiK2trP9iN2s Jv9RzBD898j7M0l1im1A8jF4IBws3TZPxtdu3CHTQGHcsw+R8Z78ZuawVBq9Lahjy7wQ /XFwjpEcltUT/y0tPQTy0GELQ1IopW+tySidST3JFPDlxq1vVYq1+C/guwaD+8UXtfTv 2knf+irO39hpwHpeNDhdhO3CkyEt2/bkAKwVA0M4mtmA1t6idQT3B8Oa8xwHxwZ7M4zM ng6MOzY6zneZX+vxlaANI7F/MUnoifmDITAfsQHBvgT7T851YVv3Z6shEY/j9qJGZaiR BbCw== X-Gm-Message-State: ABy/qLaX5rO1VfbMZKbNCH3CLGfgy1bMAYLkeB5M3tS/P/FLvfwj7DW7 UaL0btx3tPZoJ+9WpBfiuoIcp07vPGw+d549AMSL8f1ECOQ= X-Google-Smtp-Source: APBJJlHqKqzJqfN/I8wmXUYGluY0iITs6r/GgGw+wPajwHAOmJ4UZKqHgUm2xEJi/0j8lIhdsUTW1jCxo49h0HC/5Ds= X-Received: by 2002:a05:6820:2201:b0:560:b01a:653d with SMTP id cj1-20020a056820220100b00560b01a653dmr18196648oob.0.1691137915832; Fri, 04 Aug 2023 01:31:55 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Shrikanth Kamath Date: Fri, 4 Aug 2023 01:31:04 -0700 Message-ID: Subject: Re: How to watch Active pagequeue transitions with DTrace in the vm layer To: Mark Johnston Cc: freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="000000000000bd4d49060214baf9" X-Rspamd-Queue-Id: 4RHJq12Mxmz4CpV X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] --000000000000bd4d49060214baf9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks and appreciate your response Mark, a follow up query, so the system was probably at some point in the state where there were no pages in the laundry or even had pages backed by swap (refer the top snapshot below) . The two heavy applications with 12G resident + Wired + Buf already caused the Free to drop close to the minimum threshold, any further memory demand would have the pages of these applications move to laundry or swap, then would transition to Inactive or Laundry, later when these pages were referenced back the pagedaemon would move them back to the Active? Is that a correct understanding? last pid: 20494; load averages: 0.38, 0.73, 0.80 up 0+01:49:05 21:14:49 Mem: 9439M Active, 3638M Inact, 2644M Wired, 888M Buf, 413M Free Swap: 8192M Total, 8192M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12043 root 5 22 0 9069M 7752M kqread 2 49:37 6.25% app1 12051 root 1 20 0 2704M 1964M select 3 0:41 0.00% app2 So if I run DTrace probe on vm_page_enqueue I will probably see that pagedaemon might be the thread that moved all those pages to Active? Is there a way to associate these to the process which referenced these pages Regards, -- Shrikanth R K On Thu, Aug 3, 2023 at 8:14=E2=80=AFAM Mark Johnston wr= ote: > On Thu, Aug 03, 2023 at 12:32:22AM -0700, Shrikanth Kamath wrote: > > A background on the query, > > > > Trying to catch a memory =E2=80=9Cspike=E2=80=9D trigger using DTrace, = refer here two > =E2=80=9Ctop=E2=80=9D > > snapshots captured during a 2 minute window, > > > > last pid: 89900; load averages: 0.75, 0.91, 0.94 up 39+00:37:30 > > 20:03:14 > > > > Mem: 5575M Active, 2152M Inact, 4731M Laundry, 3044M Wired, 1151M Buf, > 382M > > Free > > > > Swap: 8192M Total, 1058M Used, 7134M Free, 12% Inuse > > > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMA= ND > > > > 12043 root 5 35 0 11G 9747M kqread 3 > > 128.8H 23.34% app1 > > > > 12051 root 1 20 0 3089M 2274M select 1 > 22:51 > > 0.00% app2 > > > > last pid: 90442; load averages: 1.50, 1.12, 1.02 up 39+00:39:37 > > 20:05:21 > > > > Mem: 8549M Active, 631M Inact, 3340M Laundry, 3159M Wired, 1252M Buf, > 359M > > Free > > > > Swap: 8192M Total, 1894M Used, 6298M Free, 23% Inuse > > > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > > > > 12043 root 5 24 0 11G 9445M kqread 2 > > 128.8H 10.45% app1 > > > > 12051 root 1 20 0 3089M 2173M select 3 > 22:51 > > 0.00% app2 > > > > The spike is ~3G in Active pages, the two large applications have a > > combined resident size of ~12G. The resident size of the applications > > hasn=E2=80=99t changed between these 2 readings, however there is a tar= archive > and > > gzip on a large directory during that window likely causing a reshuffle= . > If > > I count the page allocs and dequeue by execname with DTrace, I see > > tar/vmstat which probably alloc and quickly dequeue, along with a large > > dequeue being undertaken by bufdaemon and pagedaemon. > > > > fbt::vm_page_alloc*:entry > > > > { > > > > @cnt[execname] =3D count(); > > > > } > > > > fbt::vm_page_dequeue:entry > > > > { > > > > @dcnt[execname] =3D count(); > > > > } > > > > Page Alloc > > > > vmstat > > 20222 > > > > tar 2128= 4 > > > > Page Dequeue > > > > vmstat 20114 > > > > bufdaemon 21402 > > > > tar 216= 35 > > > > pagedaemon 360387 > > > > Since the tar / vmstat will not hold the pages in Active, I need to fin= d > > out what application had its pages queued in Active page queue. > > One possibility is that the inactive and laundry queue had previously > contained many referenced pages. Then, when some memory pressure > occurred, the pagedaemon scanned the queues and moved a large number of > pages into the active queue. > > > Is it possible that the system is just moving the LRU pages of these tw= o > > large applications into the inactive queue prior to addressing memory > > pressure? Do these applications need to activate those pages later and > > hence it brings it back into the Active queue? How do I watch this in > > action by using DTrace? Will the following probe catch this trigger? > > > > fbt::vm_page_activate:entry > > > > { > > > > @cnt[execname, pid] =3D count(); > > > > } > > > > tick-10sec > > > > { > > > > printa(@cnt); > > > > printf("ACTIVE[%d] pages\n", `vm_dom[0].vmd_pagequeues[1].pq_cnt= ); > > > > } > > > > *** This system is running only one vmdomain (# sysctl vm.ndomains =E2= =80=93> > > vm.ndomains: 1). > > > > *** running release 12.1, on an amd64 kernel. The physical memory > installed > > is 16G. > > In 12.1, you'd probably want something like: > > fbt::vm_page_enqueue:entry > /args[1] =3D=3D 1 /* PQ_ACTIVE *// > { > ... > } > > since vm_page_unwire(m, PQ_ACTIVE) will also move a page into the active > queue, but your D script above won't catch that. > > I would also look at the "pages reactivated by the page daemon" counter > that appears in vmstat -s output. That'll tell you how many times the > page daemon moved a page from PQ_INACTIVE/PQ_LAUNDRY to PQ_ACTIVE > because it found that the page had been referenced. > --=20 Shrikanth R K --000000000000bd4d49060214baf9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thanks and appreciate your response Mark,= a follow up query, so the system was probably at some point in the state w= here there were no pages in the laundry or even had pages backed by swap (r= efer the top snapshot below) . The two heavy applications with 12G resident= =C2=A0+ Wired=C2=A0+ Buf already caused the Free to drop close to the minim= um threshold, any further memory demand would have the pages of these appli= cations move to laundry or swap, then would transition to Inactive or Laund= ry, later when these pages were referenced back the pagedaemon would move t= hem back to the Active? Is that a correct understanding?

last pid: = 20494; =C2=A0load averages: =C2=A00.38, =C2=A00.73, =C2=A00.80 =C2=A0up 0+0= 1:49:05 =C2=A0 =C2=A021:14:49
Mem: 9439M Active, 3638M Inact, 2644M Wir= ed, 888M Buf, 413M Free =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Swap: 8192M Total, 8192M Free

PID USERNAME =C2=A0 =C2=A0THR= PRI NICE =C2=A0 SIZE =C2=A0 =C2=A0RES STATE =C2=A0 =C2=A0C =C2=A0 TIME =C2= =A0 =C2=A0WCPU COMMAND =C2=A0
12043 root =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A05 =C2=A022 =C2=A0 =C2=A00 =C2=A09069M =C2=A07752M kqread =C2=A0 2 =C2=A0= 49:37 =C2=A0 6.25% app1=C2=A0
12051 root =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A01 =C2=A020 =C2=A0 =C2=A00 =C2=A02704M =C2=A01964M select =C2=A0 3 =C2=A0= 0:41 =C2=A0 0.00% app2

So if I run DT= race probe on vm_page_enqueue I will probably see that pagedaemon might be = the thread that moved all those pages to Active? Is there a way to associat= e these to the process which referenced these pages

Regards,
--
Shrikanth R K

On Thu, Aug 3, 2023 at 8:14=E2=80=AFAM Mark Johnston <markj@freebsd.org> wrote:
On Thu, Aug 03, 2023 at 12:32:22AM -0700, Shrikanth Kama= th wrote:
> A background on the query,
>
> Trying to catch a memory =E2=80=9Cspike=E2=80=9D trigger using DTrace,= refer here two =E2=80=9Ctop=E2=80=9D
> snapshots captured during a 2 minute window,
>
> last pid: 89900;=C2=A0 load averages:=C2=A0 0.75,=C2=A0 0.91,=C2=A0 0.= 94=C2=A0 up 39+00:37:30
> 20:03:14
>
> Mem: 5575M Active, 2152M Inact, 4731M Laundry, 3044M Wired, 1151M Buf,= 382M
> Free
>
> Swap: 8192M Total, 1058M Used, 7134M Free, 12% Inuse
>
> PID USERNAME=C2=A0 =C2=A0 THR PRI NICE=C2=A0 =C2=A0SIZE=C2=A0 RES=C2= =A0 =C2=A0STATE=C2=A0 =C2=A0C TIME=C2=A0 =C2=A0 WCPU COMMAND
>
> 12043=C2=A0 =C2=A0 root=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A05=C2=A0 =C2=A0 35=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2=A0= 11G=C2=A0 9747M kqread=C2=A0 3
> 128.8H=C2=A0 23.34% app1
>
> 12051=C2=A0 =C2=A0 root=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A01=C2=A0 =C2=A0 20=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 3089M=C2=A0 2= 274M select=C2=A0 =C2=A01=C2=A0 22:51
>=C2=A0 =C2=A00.00%=C2=A0 =C2=A0 app2
>
> last pid: 90442;=C2=A0 load averages:=C2=A0 1.50,=C2=A0 1.12,=C2=A0 1.= 02=C2=A0 up 39+00:39:37
> 20:05:21
>
> Mem: 8549M Active, 631M Inact, 3340M Laundry, 3159M Wired, 1252M Buf, = 359M
> Free
>
> Swap: 8192M Total, 1894M Used, 6298M Free, 23% Inuse
>
> PID=C2=A0 =C2=A0USERNAME=C2=A0 =C2=A0THR PRI NICE=C2=A0 =C2=A0SIZE=C2= =A0 =C2=A0 RES STATE=C2=A0 =C2=A0C TIME=C2=A0 =C2=A0WCPU=C2=A0 COMMAND
>
> 12043=C2=A0 =C2=A0 =C2=A0 =C2=A0root=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 5=C2=A0 24=C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01= 1G=C2=A0 9445M kqread=C2=A0 2
> 128.8H 10.45%=C2=A0 app1
>
> 12051=C2=A0 =C2=A0 =C2=A0 =C2=A0root=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 1=C2=A0 20=C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A03089M=C2=A0 217= 3M select=C2=A0 =C2=A03=C2=A0 22:51
>=C2=A0 =C2=A00.00%=C2=A0 =C2=A0 app2
>
> The spike is ~3G in Active pages, the two large applications have a > combined resident size of ~12G. The resident size of the applications<= br> > hasn=E2=80=99t changed between these 2 readings, however there is a ta= r archive and
> gzip on a large directory during that window likely causing a reshuffl= e. If
> I count the page allocs and dequeue by execname with DTrace, I see
> tar/vmstat which probably alloc and quickly dequeue, along with a larg= e
> dequeue being undertaken by bufdaemon and pagedaemon.
>
> fbt::vm_page_alloc*:entry
>
> {
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0@cnt[execname] =3D count();
>
> }
>
> fbt::vm_page_dequeue:entry
>
> {
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0@dcnt[execname] =3D count();
>
> }
>
> Page Alloc
>
>=C2=A0 =C2=A0vmstat
> 20222
>
>=C2=A0 =C2=A0tar=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 21284
>
> Page Dequeue
>
>=C2=A0 =C2=A0vmstat=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 20114=
>
>=C2=A0 =C2=A0bufdaemon=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A021402
>
>=C2=A0 =C2=A0tar=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A021635
>
>=C2=A0 =C2=A0pagedaemon=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0360387
>
> Since the tar / vmstat will not hold the pages in Active, I need to fi= nd
> out what application had its pages queued in Active page queue.

One possibility is that the inactive and laundry queue had previously
contained many referenced pages.=C2=A0 Then, when some memory pressure
occurred, the pagedaemon scanned the queues and moved a large number of
pages into the active queue.

> Is it possible that the system is just moving the LRU pages of these t= wo
> large applications into the inactive queue prior to addressing memory<= br> > pressure?=C2=A0 Do these applications need to activate those pages lat= er and
> hence it brings it back into the Active queue? How do I watch this in<= br> > action by using DTrace? Will the following probe catch this trigger? >
> fbt::vm_page_activate:entry
>
> {
>
> @cnt[execname, pid] =3D count();
>
> }
>
> tick-10sec
>
> {
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 printa(@cnt);
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 printf("ACTIVE[%d] pages\n", `vm_= dom[0].vmd_pagequeues[1].pq_cnt);
>
> }
>
> *** This system is running only one vmdomain (# sysctl vm.ndomains =E2= =80=93>
> vm.ndomains: 1).
>
> *** running release 12.1, on an amd64 kernel. The physical memory inst= alled
> is 16G.

In 12.1, you'd probably want something like:

fbt::vm_page_enqueue:entry
/args[1] =3D=3D 1 /* PQ_ACTIVE *//
{
...
}

since vm_page_unwire(m, PQ_ACTIVE) will also move a page into the active queue, but your D script above won't catch that.

I would also look at the "pages reactivated by the page daemon" c= ounter
that appears in vmstat -s output.=C2=A0 That'll tell you how many times= the
page daemon moved a page from PQ_INACTIVE/PQ_LAUNDRY to PQ_ACTIVE
because it found that the page had been referenced.


--
= Shrikanth R K
--000000000000bd4d49060214baf9-- From nobody Fri Aug 4 13:36:05 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RHRZ019Hsz4kTK0; Fri, 4 Aug 2023 13:36:08 +0000 (UTC) (envelope-from jhibbits@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RHRZ00J0cz3CB1; Fri, 4 Aug 2023 13:36:08 +0000 (UTC) (envelope-from jhibbits@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1691156168; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0uNWYYnTDq/VsgaoM7Gz88oWckbqfTd9WXszadGHlHs=; b=CMv/0qPnN9qJ7UZLopHjtiKSpk1Nxk+DhD985KOyasZYX3xVJe/J+uRVDfhhIiNrGndnzg 3DclY+SVYy1Pm0zj5G9Xf/XWRhOwQa+MEq2QGrCKJDfeRmweDgNtznFjrM/bVBt0p9Zpwu cLa+u1ftgy6xrT80YxMbVOvJOBGiSJxnRxXKQvIcGDJO+NPgx9um8zCH9n4RhU0aR97Zty ie0to8VaWNpotbffaAUHVKr5d345lvNGoKNnuKzvGDarXLtPzsLHjOHTZb2QOvbascXlmA voKu6JJJ+Fsa0OY3fp9TF0E6suFgwAz4ahjJH5jIqwk9wWvWUFDBz/WmgR9DxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1691156168; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0uNWYYnTDq/VsgaoM7Gz88oWckbqfTd9WXszadGHlHs=; b=Ljwh13sKvW6Y48rxM6qoLL+wugY0MKmKZzL+skwfg1TCH6zKWSWkq/M/Fguaq+j2oBcgcR U6KzfTWRAE3w/waauA7XUJt30dqMdoA563HIo3cYf+usDQSxYJu35A/8hZbMLsMgkOsdPs 4oH/hLSLCqLNiidsYkNnqsqY9YegSwyIq9Bz6ymyl3arSWHxU6Xe4Wqdjg2MW02702f743 S+zfJDUbfV57yH/lGHXIV7QM5+0k9bpAHGtqg4UxXeFfVsS8SU2mk/saB9vR8ncFvNLs4g GANggh9RIykJmp5FKzqZC/iqEdPAjzqzN9fcvLZE1KedVEE7vLVLhG6hR/mpiA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1691156168; a=rsa-sha256; cv=none; b=RObrsH7YmWsI2rshvLtDh8nNtHcWeIcg4SZi1t/02ob567xW66NFL1IS73NRFWCTcAI77W bHr2y7S1VJVPjLmSQabrkCmzyAdpFyFzAi0fgHQ7SsHFasbnsPsnChfdpRO9zjJY+E4TmG jDilDYpcGplep/d0cPFFhoPLcOi7UbRgJVJTUcB1/faDECZyOYwKe740/rNYbi3SoOMfBq qPI23ZLAmp/cqYSnWa5cJfWPDXwow32UdG0RL2U+ML/Rk1A8pN9KQuOvKvhohJgXbOFdeH /EJlOqWd2XvZ7rRz//DZHmLL22gk0TN46WVmwO5yHiq0q9Wy1Bux1+MFKPD3Ig== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from ralga.knownspace (ip-163-182-7-56.dynamic.fuse.net [163.182.7.56]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: jhibbits) by smtp.freebsd.org (Postfix) with ESMTPSA id 4RHRYz4kGlz11LF; Fri, 4 Aug 2023 13:36:07 +0000 (UTC) (envelope-from jhibbits@FreeBSD.org) Date: Fri, 4 Aug 2023 09:36:05 -0400 From: Justin Hibbits To: Shawn Anastasio Cc: freebsd-ppc@FreeBSD.org, freebsd-hackers@FreeBSD.org, Timothy Pearson , John Baldwin Subject: Re: Implementing in-kernel AES crypto acceleration on ppc (POWER8+) Message-ID: <20230804093605.2a61eeed@ralga.knownspace> In-Reply-To: <0c24b4b7-b4c8-242d-6187-15b171c50c19@raptorengineering.com> References: <0c24b4b7-b4c8-242d-6187-15b171c50c19@raptorengineering.com> X-Mailer: Claws Mail 4.1.1 (GTK 3.24.38; powerpc64le-unknown-linux-gnu) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Hello, Good to see this! I'll answer inline. On Thu, 3 Aug 2023 12:51:57 -0500 Shawn Anastasio wrote: > Hello all, > > Raptor Engineering is interested in adding support for in-kernel AES > acceleration on ppc64 via the VMX crypto instructions added in ISA > 2.07B, and I wanted to reach out to the community with a few > questions. I would love to see this added. > > 1. As I understand it, FreeBSD already has support for in-kernel > crypto acceleration on x86 and ARM via the aesni and armv8_crypto > drivers respectively that each implement the cryptodev interface. Am I > correct in understanding that adding AES acceleration for Power > would just involve creating another driver here, or are there other > pieces of the puzzle that I've missed? John Baldwin can probably answer this better, but I think your understanding is correct. There might be some plumbing needed as well, but that should be minimal. > > 2. I see that both the aesni and armv8 drivers make use of the > fpu_kern_enter/fpu_kern_leave functions to guard access to vector > registers, but it appears that these functions aren't implemented > on ppc. Is that correct, or does an in-kernel facility for safely > accessing vector registers on ppc already exist? Nope, ppc doesn't have these facilities yet. It shouldn't be hard to implement, we just haven't done it yet. If you're interested in implementing them, you should be able to model it after arm64, largely. > > 3. For the accelerated AES implementation itself, I've noticed that > cryptogams[*] contains an implementation that is both widely > deployed (and thus tested and likely to be correct) and also BSD > licensed. Would it be acceptable to import the relevant routines to > the FreeBSD kernel and have the new cryptodev driver simply call into > them, or are there other considerations involved? I think the right way to do that would be to import the code as-is as third party code, and call into the routines that you need. You can #ifdef out the unneeded bits, but try to keep it as intact as possible from upstream. > > 4. Is there a userspace test framework for the cryptodev API that > could be used to validate and benchmark the new implementation, or > would I have to write that myself? It appears that OpenSSL had > support for /dev/crypto at one point, but I'm not sure that is the > case any longer. John Baldwin might have some ideas here, too. > > My apologies for the large number of questions, but I look forward to > hearing back and working with the FreeBSD community to get this > implemented. > > Thanks, > Shawn Anastasio > > [*] https://github.com/dot-asm/cryptogams > - Justin From nobody Fri Aug 4 14:44:10 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RHT4b3Wvtz4TjPt for ; Fri, 4 Aug 2023 14:44:15 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-ua1-x936.google.com (mail-ua1-x936.google.com [IPv6:2607:f8b0:4864:20::936]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RHT4b0Vzhz3LrF for ; Fri, 4 Aug 2023 14:44:15 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ua1-x936.google.com with SMTP id a1e0cc1a2514c-78f36f37e36so734766241.3 for ; Fri, 04 Aug 2023 07:44:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691160254; x=1691765054; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=F4xPbDUSjDNON73QshzmKt1RpRaL7YGoBTj79uPzFcY=; b=c315ruT25qWu+A0e+ZnCW/6BseDoVR2Esz7SJXGI3Nn3/YZmA9bYv0s6m6HZZH8A9B 5zuxThngifPPhprGcENcDaUsRmz0pbe/egBihWNDJqvOpAR0c4CYPQPXHFvbCvfINAUd 3nOql4VfYjTFkizFDDLuPjFnL9iH6VJwKGTca5U+chSfhoTOjCRK2weZNiCf/5yY2FnE aSKBOSn/Qmw8O1xSQiwVMnC8wkAwgrXnQ6AzxrbDQFlaOqhrn1Mpa7q3SuVvThvIA1c9 vF3lpSCLdq515j3NwULmNWGhghk54gNj4PFPtJFJMdWS8TMdcpNfICMcZYp7K45T/SZI no1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691160254; x=1691765054; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=F4xPbDUSjDNON73QshzmKt1RpRaL7YGoBTj79uPzFcY=; b=IUgsSPo12FCdC2XfwFdu4cjjIYkTblOSh4K2qHn2Tty0WyqJC87oh9NYcF8Fxlkpg8 sfg/keTsMKw3Tb6ob1b90aox0vuLW8J9ZZ+dzXIBe1gOJvZBp83ejgogmXqjyitDb2Z5 8O7Hqsy2/tOMa+XldX67zZNXvQTS41YEly+hJBiQtMn+RqWAgOTQWKR8T5yYotCTn62J 4SwVijEfsNJdoiT0zmMSEkkIJ9VdExNARKEPviJYlv7U2qiiT0jDLt0zm/geSA9b7qUs DNEbSWK8OBNWmqJ593TxA4wBVc3SVQ8686im96CgPoZeO4rhHd/6OgpibGMncxOE9u9s RAGQ== X-Gm-Message-State: AOJu0YymZ64kH6QroxzD4xhtf1dXRI2SciGl9cOoWrv5h/KwQvSKDhFW 3QME2oCatOn6dvRySa58k5xK3+u68rg= X-Google-Smtp-Source: AGHT+IE6vCetGL9vVG0jV4W1mq9ojRhVBozrykmV/IRJV9qldo2JyCKmBuSxTypLHgx/m6n+HeVlYQ== X-Received: by 2002:a05:6102:18a:b0:443:51a7:b63d with SMTP id r10-20020a056102018a00b0044351a7b63dmr1434965vsq.23.1691160253966; Fri, 04 Aug 2023 07:44:13 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id h6-20020a0cab06000000b0063d119034a9sm694828qvb.140.2023.08.04.07.44.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Aug 2023 07:44:12 -0700 (PDT) Date: Fri, 4 Aug 2023 10:44:10 -0400 From: Mark Johnston To: Shrikanth Kamath Cc: freebsd-hackers@freebsd.org Subject: Re: How to watch Active pagequeue transitions with DTrace in the vm layer Message-ID: References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4RHT4b0Vzhz3LrF X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] On Fri, Aug 04, 2023 at 01:31:04AM -0700, Shrikanth Kamath wrote: > Thanks and appreciate your response Mark, a follow up query, so the system > was probably at some point in the state where there were no pages in the > laundry or even had pages backed by swap (refer the top snapshot below) . > The two heavy applications with 12G resident + Wired + Buf already caused > the Free to drop close to the minimum threshold, any further memory demand > would have the pages of these applications move to laundry or swap, then > would transition to Inactive or Laundry, later when these pages were > referenced back the pagedaemon would move them back to the Active? Is that > a correct understanding? If there is a shortage of free pages, the page daemon will scan the inactive queue, trying to reclaim clean pages. Dirty pages go into the laundry; once the laundry is "large enough", the page daemon will clean pages in the laundry by writing them to swap. If, while scanning a page in the inactive or laundry queues, the pagedaemon notices that the page had been accessed since it was last visited (e.g., the "accessed" bit is set on a page table entry mapping the page), the page daemon will generally move it to the active queue. This happens lazily: if there is no demand for free pages, an accessed page can stay in the inactive/laundry queues indefinitely. > last pid: 20494; load averages: 0.38, 0.73, 0.80 up 0+01:49:05 > 21:14:49 > Mem: 9439M Active, 3638M Inact, 2644M Wired, 888M Buf, 413M Free > > Swap: 8192M Total, 8192M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 12043 root 5 22 0 9069M 7752M kqread 2 49:37 6.25% app1 > 12051 root 1 20 0 2704M 1964M select 3 0:41 0.00% app2 > > So if I run DTrace probe on vm_page_enqueue I will probably see that > pagedaemon might be the thread that moved all those pages to Active? Is > there a way to associate these to the process which referenced these pages The page daemon does not use vm_page_enqueue() to move pages back to the active queue. Instead, check the vmstat -s counter I had mentioned, "pages reactivated by the page daemon". From nobody Fri Aug 4 14:55:50 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RHTL01hdJz4TkM8; Fri, 4 Aug 2023 14:55:52 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RHTL018dGz3Mv9; Fri, 4 Aug 2023 14:55:52 +0000 (UTC) (envelope-from jhb@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1691160952; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Eikqwniskib4EfMJ7FVNqh51DaXqjA8jRpz04YBFqt4=; b=d/U29cNjbbuKpfKuFGn6x03+QwkdxkmuAxF21oc+q4mVTPr3CqYazrjScYH8JsuP3s8/Kc 16BM1C9oqFBsX85z6zWYDlHAU5iRd5FMdHgoJXncJ4Th/wp0rPHkQG2EMfmgocq5tcUnSC cR3mf4mOpbaiCQfzzC+lZhYI8M3IdFf0T+p2dKdWHxu/5y19L2ziANOk4sTyvmFYn24Fi6 Lp/kVtNCp91NaKP8XBDd4i65WPdUksT4AHzit5k3d+8fAt2iJHnuVjC79u/cPqX0p9Xr4d 8MvMm+tWyQfRZFz4QOeT9XCxXJ+3aN/o/DMcD5zixGBqWng61qVD3f0kbjduNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1691160952; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Eikqwniskib4EfMJ7FVNqh51DaXqjA8jRpz04YBFqt4=; b=J8/hyZIBcFGGiEFLLDA3pFUmGtcTzWYg0ZV8dL0BbuaqnxzVM+F9k3BD5TjFrTeRTmH9ia VgGvKipc2aUTYMtjL8silwbWVDTxfsh/v6Tvk+lBKglP8iO01MYJ+0VyJdjMu+K8XswHmM dlHNTqaevurQzvl56RIx8Kl8phmIA6ZOM+OQW5ApDqizOq08mYxTqfUG/A3aQyyY0kz0eG z+H3LSGe/g1zlIGgDk1gojr1Sln6UFKOJucA4hNSP+zJTi1filjX+Myp82tbS6jQvoBJX/ gbNySyKKYnbDJU/kUU+wIM3BhCQxfJbJbNJAFd5FVg1k5j9RoSMKGsKTlJvCEA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1691160952; a=rsa-sha256; cv=none; b=xHn3UwiuhTQ/CwVhuvWbgmdCybxAPReRXS7BrfOHewAir5np0kS3FKaivhh+Aw06tU4wL3 pnt6Q0HhumboQLspV9fs9FMNkhHtD15g2fnHozT3hTUWqms8fWje1bcVT3MyXVxLso6dDD jeWF5g78gZGccDJPiRNNfyZmRLYRr1Hb0XY10G4LB9SzvF7uiyV1cdxhsLV64zAKmC2qTk xFniAOzDRgbMPtfUd2kKidOJFyTeYDB5Bc9+si3yY+Lh7wjUg3z/Mml495hPgq7vEl3Fhp /hJShUxLnzFwQmz6Th61bgfkr4KsMd3m+WTSIClzyvw2ROGoqscOwIOgBq/01A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from [IPV6:2601:648:8680:16b0:cdd2:66b:dfd1:f731] (unknown [IPv6:2601:648:8680:16b0:cdd2:66b:dfd1:f731]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id 4RHTKz3YJqz11Nn; Fri, 4 Aug 2023 14:55:51 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Message-ID: Date: Fri, 4 Aug 2023 07:55:50 -0700 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Content-Language: en-US To: Justin Hibbits , Shawn Anastasio Cc: freebsd-ppc@FreeBSD.org, freebsd-hackers@FreeBSD.org, Timothy Pearson References: <0c24b4b7-b4c8-242d-6187-15b171c50c19@raptorengineering.com> <20230804093605.2a61eeed@ralga.knownspace> From: John Baldwin Subject: Re: Implementing in-kernel AES crypto acceleration on ppc (POWER8+) In-Reply-To: <20230804093605.2a61eeed@ralga.knownspace> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 8/4/23 6:36 AM, Justin Hibbits wrote: > Hello, > > Good to see this! I'll answer inline. > > On Thu, 3 Aug 2023 12:51:57 -0500 > Shawn Anastasio wrote: > >> Hello all, >> >> Raptor Engineering is interested in adding support for in-kernel AES >> acceleration on ppc64 via the VMX crypto instructions added in ISA >> 2.07B, and I wanted to reach out to the community with a few >> questions. > > I would love to see this added. > >> >> 1. As I understand it, FreeBSD already has support for in-kernel >> crypto acceleration on x86 and ARM via the aesni and armv8_crypto >> drivers respectively that each implement the cryptodev interface. Am I >> correct in understanding that adding AES acceleration for Power >> would just involve creating another driver here, or are there other >> pieces of the puzzle that I've missed? > > John Baldwin can probably answer this better, but I think your > understanding is correct. There might be some plumbing needed as well, > but that should be minimal. Recently for accelerated software crypto we have been using the existing assembly routines from OpenSSL (which has ppc routines IIRC) in the ossl(4) driver. For powerpc you would need to provide any ppc-specific things the OpenSSL assembly routines need (e.g. on x86 they use an array of words holding feature bits corresponding to output from cpuid), and mostly just add build glue. This driver also requires fpu_kern_*, but in general ossl(4) is preferred going forward and will eventually replace aesni and armv8crypto entirely. >> 2. I see that both the aesni and armv8 drivers make use of the >> fpu_kern_enter/fpu_kern_leave functions to guard access to vector >> registers, but it appears that these functions aren't implemented >> on ppc. Is that correct, or does an in-kernel facility for safely >> accessing vector registers on ppc already exist? > > Nope, ppc doesn't have these facilities yet. It shouldn't be hard to > implement, we just haven't done it yet. If you're interested in > implementing them, you should be able to model it after arm64, largely. Yes, this is a prerequisite. If you implement this you can also enable assembly for ZFS on powerpc as well. >> 3. For the accelerated AES implementation itself, I've noticed that >> cryptogams[*] contains an implementation that is both widely >> deployed (and thus tested and likely to be correct) and also BSD >> licensed. Would it be acceptable to import the relevant routines to >> the FreeBSD kernel and have the new cryptodev driver simply call into >> them, or are there other considerations involved? > > I think the right way to do that would be to import the code as-is as > third party code, and call into the routines that you need. You can > #ifdef out the unneeded bits, but try to keep it as intact as possible > from upstream. As mentioned above, I would prefer using the OpenSSL sources already in the tree via ossl(4). >> 4. Is there a userspace test framework for the cryptodev API that >> could be used to validate and benchmark the new implementation, or >> would I have to write that myself? It appears that OpenSSL had >> support for /dev/crypto at one point, but I'm not sure that is the >> case any longer. > > John Baldwin might have some ideas here, too. There is a cryptocheck tool in tools/tools/crypto that I tend to use. It generates "random" but deterministic (since it uses a fixed seed for libc's PRNG) tests of most of the algorithms supported by the cryptodev API with various key sizes, nonce sizes, payload, and AAD sizes. It performs each test once with OpenSSL's userland crypto and a second time with /dev/crypto and compares the results reporting any mismatches. There isn't a manpage, but there are some comments in the source. There are also some tests in the test suite that make use of the NIST known answer test vectors (which have to be installed via a nist-kat package). -- John Baldwin From nobody Fri Aug 4 16:13:42 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RHW3t19xLz4Tq2m; Fri, 4 Aug 2023 16:13:46 +0000 (UTC) (envelope-from sanastasio@raptorengineering.com) Received: from raptorengineering.com (mail.raptorengineering.com [23.155.224.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RHW3s5PS1z3b4Z; Fri, 4 Aug 2023 16:13:45 +0000 (UTC) (envelope-from sanastasio@raptorengineering.com) Authentication-Results: mx1.freebsd.org; none Received: from localhost (localhost [127.0.0.1]) by mail.rptsys.com (Postfix) with ESMTP id 3BF6A828563C; Fri, 4 Aug 2023 11:13:44 -0500 (CDT) Received: from mail.rptsys.com ([127.0.0.1]) by localhost (vali.starlink.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id RoeiEfZMSDX9; Fri, 4 Aug 2023 11:13:43 -0500 (CDT) Received: from localhost (localhost [127.0.0.1]) by mail.rptsys.com (Postfix) with ESMTP id 132348285659; Fri, 4 Aug 2023 11:13:43 -0500 (CDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.rptsys.com 132348285659 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raptorengineering.com; s=B8E824E6-0BE2-11E6-931D-288C65937AAD; t=1691165623; bh=iYMQtGDtrhMtXm8RFLhaJ9G9LL1lXuHVXruwfcBZUWU=; h=Message-ID:Date:MIME-Version:To:From; b=DAbml8koVUnmq202yDWkFhOwlVXdnxeZAdmcp6H2auGqDYmLTMUfJ+DHonVu/vFHX yuEk3+1d+6mwoEdPFO6aVagULmGHqyHGjMbG7DAgyGkzVsp5PNx6emFr6lz7exjpiE /v7+6RMqn7XjofIml/ReZt0zbv1FvNEmRM5/2688= X-Virus-Scanned: amavisd-new at rptsys.com Received: from mail.rptsys.com ([127.0.0.1]) by localhost (vali.starlink.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id j5cfwuFE8iFX; Fri, 4 Aug 2023 11:13:42 -0500 (CDT) Received: from [10.11.0.2] (5.edge.rptsys.com [23.155.224.38]) by mail.rptsys.com (Postfix) with ESMTPSA id 97737828563C; Fri, 4 Aug 2023 11:13:42 -0500 (CDT) Message-ID: Date: Fri, 4 Aug 2023 11:13:42 -0500 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux ppc64le; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: Implementing in-kernel AES crypto acceleration on ppc (POWER8+) Content-Language: en-US To: John Baldwin , Justin Hibbits Cc: freebsd-ppc@FreeBSD.org, freebsd-hackers@FreeBSD.org, Timothy Pearson References: <0c24b4b7-b4c8-242d-6187-15b171c50c19@raptorengineering.com> <20230804093605.2a61eeed@ralga.knownspace> From: Shawn Anastasio In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4RHW3s5PS1z3b4Z X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:46246, ipnet:23.155.224.0/24, country:US] On 8/4/23 9:55 AM, John Baldwin wrote: > On 8/4/23 6:36 AM, Justin Hibbits wrote: >> Hello, >> >> Good to see this! I'll answer inline. >> >> On Thu, 3 Aug 2023 12:51:57 -0500 >> Shawn Anastasio wrote: >> >>> Hello all, >>> >>> Raptor Engineering is interested in adding support for in-kernel AES >>> acceleration on ppc64 via the VMX crypto instructions added in ISA >>> 2.07B, and I wanted to reach out to the community with a few >>> questions. >> >> I would love to see this added. >> >>> >>> 1. As I understand it, FreeBSD already has support for in-kernel >>> crypto acceleration on x86 and ARM via the aesni and armv8_crypto >>> drivers respectively that each implement the cryptodev interface. Am I >>> correct in understanding that adding AES acceleration for Power >>> would just involve creating another driver here, or are there other >>> pieces of the puzzle that I've missed? >> >> John Baldwin can probably answer this better, but I think your >> understanding is correct. There might be some plumbing needed as well, >> but that should be minimal. > > Recently for accelerated software crypto we have been using the existing > assembly routines from OpenSSL (which has ppc routines IIRC) in the > ossl(4) driver. For powerpc you would need to provide any ppc-specific > things the OpenSSL assembly routines need (e.g. on x86 they use an array > of words holding feature bits corresponding to output from cpuid), and > mostly just add build glue. This driver also requires fpu_kern_*, but > in general ossl(4) is preferred going forward and will eventually > replace aesni and armv8crypto entirely. Thank you for pointing this out. It seems that sys/crypto/openssl conveniently already has the relevant ppc assembly files imported from the openssl source tree, so it looks like I'll just have to implement the ppc-specific openssl glue, in addition to the fpu_enter/leave functions. >>> 2. I see that both the aesni and armv8 drivers make use of the >>> fpu_kern_enter/fpu_kern_leave functions to guard access to vector >>> registers, but it appears that these functions aren't implemented >>> on ppc. Is that correct, or does an in-kernel facility for safely >>> accessing vector registers on ppc already exist? >> >> Nope, ppc doesn't have these facilities yet. It shouldn't be hard to >> implement, we just haven't done it yet. If you're interested in >> implementing them, you should be able to model it after arm64, largely. > > Yes, this is a prerequisite. If you implement this you can also enable > assembly for ZFS on powerpc as well. Makes sense, thank you both for the confirmation. I'll get started on mocking up an implementation of these functions. >>> 3. For the accelerated AES implementation itself, I've noticed that >>> cryptogams[*] contains an implementation that is both widely >>> deployed (and thus tested and likely to be correct) and also BSD >>> licensed. Would it be acceptable to import the relevant routines to >>> the FreeBSD kernel and have the new cryptodev driver simply call into >>> them, or are there other considerations involved? >> >> I think the right way to do that would be to import the code as-is as >> third party code, and call into the routines that you need. You can >> #ifdef out the unneeded bits, but try to keep it as intact as possible >> from upstream. > > As mentioned above, I would prefer using the OpenSSL sources already > in the tree via ossl(4). > Sounds good. >>> 4. Is there a userspace test framework for the cryptodev API that >>> could be used to validate and benchmark the new implementation, or >>> would I have to write that myself? It appears that OpenSSL had >>> support for /dev/crypto at one point, but I'm not sure that is the >>> case any longer. >> >> John Baldwin might have some ideas here, too. > > There is a cryptocheck tool in tools/tools/crypto that I tend to use. > It generates "random" but deterministic (since it uses a fixed seed > for libc's PRNG) tests of most of the algorithms supported by the > cryptodev API with various key sizes, nonce sizes, payload, and AAD > sizes. It performs each test once with OpenSSL's userland crypto > and a second time with /dev/crypto and compares the results reporting > any mismatches. There isn't a manpage, but there are some comments > in the source. > > There are also some tests in the test suite that make use of the > NIST known answer test vectors (which have to be installed via a nist-kat > package). Perfect, this seems like exactly what I was looking for. I'll begin work on the fpu_enter/leave functions and keep you all updated. Thank you both for the prompt and detailed responses! - Shawn From nobody Fri Aug 4 22:48:28 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RHgqN75VBz4mBFh for ; Fri, 4 Aug 2023 22:48:32 +0000 (UTC) (envelope-from yuri@aetern.org) Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RHgqN1Kfhz3F0w for ; Fri, 4 Aug 2023 22:48:31 +0000 (UTC) (envelope-from yuri@aetern.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=aetern.org header.s=fm3 header.b=IhVNJTCO; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=SzBj6KD4; spf=pass (mx1.freebsd.org: domain of yuri@aetern.org designates 66.111.4.29 as permitted sender) smtp.mailfrom=yuri@aetern.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 8285C5C00EA for ; Fri, 4 Aug 2023 18:48:30 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Fri, 04 Aug 2023 18:48:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aetern.org; h=cc :content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:message-id:mime-version:reply-to:sender :subject:subject:to:to; s=fm3; t=1691189310; x=1691275710; bh=cf hwTMBn4cxqWk5W5atKCTQrvqExHN3jLyHYLV7MxOw=; b=IhVNJTCOI/vG9GuGkg RecgFqeiN7WVyg96l+xABSxugorD3t45Y5QeWmBXLpDeIdAZbnzTCS4qT0zlGA9f sX/tBVLZuqw/vcgeDQ4DlLyvNQfG2QYwXIG5V5LTUlzK1AWOrWWvgynUY6S6oUut uYkU6GunDN8r5a3MbCz3gJ8ck2OlUIw/n1QClnQ1UrjBDE3J5Cuan0nlfjGOLpOB UBmNJFOc+bggpiV32nGzlx2NB6OdRg+tBDjXcQucGM3lIGQlPPx3Tw8jvNOK0S1I 9u6Tf8WJ9EcC3zIYVT9AqtHZCWmaQj5PcCnLUln28T3b9ZHAN6cADtqIwPuGBi6M bULg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:message-id:mime-version:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1691189310; x=1691275710; bh=cfhwTMBn4cxqW k5W5atKCTQrvqExHN3jLyHYLV7MxOw=; b=SzBj6KD4MI544oz3OTNdpldOujKJ2 r9oYrXkA8egN6itTTNti+IIlHZ+/lzYAfr1+hxFKaFD/HTssHYbfdSEeZtwmTz9g aoIXnyGZLaz1SQayZey8qxskJD0dj1kKqxLgpMq3PHVbcUad8QwpzQsCN//dO5/b GpMpWIhf7Fzqsq758FBxavJCSZckdYZVUbDG6HLZRX3A9cjdnJsc5Rmi0hO/NhAh AW78qLh6q+4x28LFlihg6SrALm/WfrLlBaIdmlF4NqvyS0swBmOdPt6OoYEVuc9Q ZLGOjehpF9JEXZ/xQH68+A2ACiIFl8RR422mqbWRnxAYEO4MPI01/85Tg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrkeehgdduudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepkfffgggfvffhufgtgfesthejredttd dvjeenucfhrhhomhepjghurhhiuceohihurhhisegrvghtvghrnhdrohhrgheqnecuggft rfgrthhtvghrnhepgeevkeegueefudehjeelledtvdefiedvvdefhfdvjefgfeejleeghe ehudettdeunecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhho mhephihurhhisegrvghtvghrnhdrohhrgh X-ME-Proxy: Feedback-ID: i0d79475b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 4 Aug 2023 18:48:29 -0400 (EDT) Message-ID: <6a0f626c-95b4-2398-0a6d-9428c440131b@aetern.org> Date: Sat, 5 Aug 2023 00:48:28 +0200 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: hackers@freebsd.org From: Yuri Subject: amd64-gcc12 seems to break linker sets Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4RHgqN1Kfhz3F0w X-Spamd-Bar: / X-Rspamd-Pre-Result: action=no action; module=multimap; Matched map: local_wl_from X-Spamd-Result: default: False [-0.39 / 15.00]; R_DKIM_ALLOW(-0.20)[aetern.org:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.29]; XM_UA_NO_VERSION(0.01)[]; local_wl_from(0.00)[yuri@aetern.org]; MLMMJ_DEST(0.00)[hackers@freebsd.org]; FREEFALL_USER(0.00)[yuri]; DKIM_TRACE(0.00)[aetern.org:+,messagingengine.com:+]; PREVIOUSLY_DELIVERED(0.00)[hackers@freebsd.org]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US] I noticed that kernel built using amd64-gcc12 panics on `kldload dtraceall`: panic: probe defined without a provider cpuid = 4 time = 1691186663 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x35/frame 0xfffffe00c448b8a0 vpanic() at vpanic+0x232/frame 0xfffffe00c448b950 panic() at panic+0x3c/frame 0xfffffe00c448b9b0 sdt_kld_load_probes() at sdt_kld_load_probes+0x346/frame 0xfffffe00c448bb70 sdt_load_probes_cb() at sdt_load_probes_cb+0x9/frame 0xfffffe00c448bb80 linker_file_foreach() at linker_file_foreach+0x53/frame 0xfffffe00c448bbc0 linker_load_dependencies() at linker_load_dependencies+0xe18/frame 0xfffffe00c448bc70 link_elf_load_file() at link_elf_load_file+0x10c6/frame 0xfffffe00c448bd50 sys_kldload() at sys_kldload+0x820/frame 0xfffffe00c448bdf0 amd64_syscall() at amd64_syscall+0x11b/frame 0xfffffe00c448bf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00c448bf30 --- syscall (304, FreeBSD ELF64, kldload), rip = 0x7e7e919615a, rsp = 0x7e7e70ef498, rbp = 0x7e7e70efa10 --- Looking into it, gcc12 seems to discard the __start_set_... and __stop_set_... symbols declared in sys/linker_set.h as __WEAK(), i.e. __asm__(".weak ..."). The following small test shows the problem: --- int main(void) { return (0); } __asm__(".weak __start_set_sdt_providers_set"); static void const * const __set_sdt_providers_set_sym_sdt_provider_tst __attribute__((__section__("set_" "sdt_providers_set"))) __attribute__((__used__)) = &(main); __asm__(".weak __stop_set_sdt_providers_set"); --- When built using system clang: --- $ cc -c -o t.o t.c; nm t.o 0000000000000000 r __set_sdt_providers_set_sym_sdt_provider_tst w __start_set_sdt_providers_set w __stop_set_sdt_providers_set 0000000000000000 T main --- When built using amd64-gcc12: --- $ /usr/local/bin/x86_64-unknown-freebsd14.0-gcc12 -c -o t.o t.c; nm t.o 0000000000000000 r __set_sdt_providers_set_sym_sdt_provider_tst 0000000000000000 T main --- I haven't found any options to tell gcc to stop doing that; any hints? From nobody Sat Aug 5 00:17:28 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RHjp40g90z4mJ1S for ; Sat, 5 Aug 2023 00:17:32 +0000 (UTC) (envelope-from yuri@aetern.org) Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RHjp35ZLGz3R5w for ; Sat, 5 Aug 2023 00:17:31 +0000 (UTC) (envelope-from yuri@aetern.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=aetern.org header.s=fm3 header.b=LgMqyp+s; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="d KVcgYv"; spf=pass (mx1.freebsd.org: domain of yuri@aetern.org designates 66.111.4.29 as permitted sender) smtp.mailfrom=yuri@aetern.org Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 807795C0166 for ; Fri, 4 Aug 2023 20:17:30 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Fri, 04 Aug 2023 20:17:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aetern.org; h=cc :content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm3; t= 1691194650; x=1691281050; bh=JnDNes/G/DjBKSDynQ4R9VH3q27CXvNcVXK VCsR8BpQ=; b=LgMqyp+sUNa2FwdeEipTbR3lZanoc1i/Ap8ReeleY8MC8BOQvC8 0ihLaSWRK6XDE3OhnbEVhJPH6dYAp8WaZes+2XSz3r8gq0uStet1m2GRNHOTHpUO Hy8V5Q9a5/GeqwD5LovCW3yt5IUxj9JdbHbPryIGK2Ii1zhDwz4zGvST3MIYUU+0 oD+R7AqrFtf8i+6mI2mBtVk+hnhiNcQMNvy6lVk983xGw0WEQj5cenOFKTzjNF0s fnfswdWOH6BPsEnpzxXyec3ZKkaSCellLpjty4G/TIuLTsdNgGDKVJvhzdptUU5h eejKSZ96qHvDVSXqH3N0QciptRbv4ikL0yg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1691194650; x= 1691281050; bh=JnDNes/G/DjBKSDynQ4R9VH3q27CXvNcVXKVCsR8BpQ=; b=d KVcgYvv9MKMD3Tmnzl0vb0j5c+eZn3L0RCV5UUcSTWxuDqZInp/oqiCUfzrv1YWa 7iXoj1t0pmqXJVRWcH4Nm+uM8TTOlQdRxDyiQmSo4L4aOi+YFDFabp1oViBff93q bw7CWYnnj9b+rliZ0JfXxsO1fzXs8q+k1Uu0rys3aNBPCkGmlhFUtdxb9da32iGR Jcp+Z/zteiuPpQ4jqZNWI6es1yeOmcp4NYsJ3edneYhM3Pjqb0f83yMac31QERvv P26MSCdFi5j3FUmjhM++ANHTP1jwWYJi1ChdX78XBmxy3qCXOUgicyp8qU8XFBvm buSRGJhoMZYwIQWb/InKg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrkeehgddvlecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepkfffgggfuffvfhfhjggtgfesthejre dttddvjeenucfhrhhomhepjghurhhiuceohihurhhisegrvghtvghrnhdrohhrgheqnecu ggftrfgrthhtvghrnhepgfdvhfdtjeeggfelfefggefggeffgfetffelieegfeeiheevgf dutedtuedvueevnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhf rhhomhephihurhhisegrvghtvghrnhdrohhrgh X-ME-Proxy: Feedback-ID: i0d79475b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 4 Aug 2023 20:17:29 -0400 (EDT) Message-ID: Date: Sat, 5 Aug 2023 02:17:28 +0200 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: amd64-gcc12 seems to break linker sets Content-Language: en-US To: freebsd-hackers@freebsd.org References: <6a0f626c-95b4-2398-0a6d-9428c440131b@aetern.org> From: Yuri In-Reply-To: <6a0f626c-95b4-2398-0a6d-9428c440131b@aetern.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4RHjp35ZLGz3R5w X-Spamd-Bar: / X-Rspamd-Pre-Result: action=no action; module=multimap; Matched map: local_wl_from X-Spamd-Result: default: False [-0.39 / 15.00]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.29]; R_DKIM_ALLOW(-0.20)[aetern.org:s=fm3,messagingengine.com:s=fm3]; XM_UA_NO_VERSION(0.01)[]; DKIM_TRACE(0.00)[aetern.org:+,messagingengine.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; local_wl_from(0.00)[yuri@aetern.org]; FREEFALL_USER(0.00)[yuri]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US] Yuri wrote: > I noticed that kernel built using amd64-gcc12 panics on `kldload dtraceall`: > > panic: probe defined without a provider > cpuid = 4 > time = 1691186663 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x35/frame > 0xfffffe00c448b8a0 > vpanic() at vpanic+0x232/frame 0xfffffe00c448b950 > panic() at panic+0x3c/frame 0xfffffe00c448b9b0 > sdt_kld_load_probes() at sdt_kld_load_probes+0x346/frame 0xfffffe00c448bb70 > sdt_load_probes_cb() at sdt_load_probes_cb+0x9/frame 0xfffffe00c448bb80 > linker_file_foreach() at linker_file_foreach+0x53/frame 0xfffffe00c448bbc0 > linker_load_dependencies() at linker_load_dependencies+0xe18/frame > 0xfffffe00c448bc70 > link_elf_load_file() at link_elf_load_file+0x10c6/frame 0xfffffe00c448bd50 > sys_kldload() at sys_kldload+0x820/frame 0xfffffe00c448bdf0 > amd64_syscall() at amd64_syscall+0x11b/frame 0xfffffe00c448bf30 > fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00c448bf30 > --- syscall (304, FreeBSD ELF64, kldload), rip = 0x7e7e919615a, rsp = > 0x7e7e70ef498, rbp = 0x7e7e70efa10 --- > > Looking into it, gcc12 seems to discard the __start_set_... and > __stop_set_... symbols declared in sys/linker_set.h as __WEAK(), i.e. > __asm__(".weak ..."). > > The following small test shows the problem: > --- > int > main(void) > { > return (0); > } > > __asm__(".weak __start_set_sdt_providers_set"); > static void const * const __set_sdt_providers_set_sym_sdt_provider_tst > __attribute__((__section__("set_" "sdt_providers_set"))) > __attribute__((__used__)) = &(main); > __asm__(".weak __stop_set_sdt_providers_set"); > --- > > When built using system clang: > --- > $ cc -c -o t.o t.c; nm t.o > 0000000000000000 r __set_sdt_providers_set_sym_sdt_provider_tst > w __start_set_sdt_providers_set > w __stop_set_sdt_providers_set > 0000000000000000 T main > --- > > When built using amd64-gcc12: > --- > $ /usr/local/bin/x86_64-unknown-freebsd14.0-gcc12 -c -o t.o t.c; nm t.o > 0000000000000000 r __set_sdt_providers_set_sym_sdt_provider_tst > 0000000000000000 T main > --- > > I haven't found any options to tell gcc to stop doing that; any hints? Apparently it's any gcc version (starting with oldest available in ports, 4.8) and it's something that was changed in: --- commit 32231805fbe2b9438c2de50c229b43c016207a08 Author: Greg V Date: Tue Apr 20 01:47:15 2021 +0100 linker_set: fix globl/weak symbol redefinitions to work on clang 12 --- While fixing clang, it seems to broke gcc. The following patch (reverting to previous definition for !clang) seems to work for both clang and gcc, dtraceall kld loads successfully, dtrace shows all defined std probes: diff --git a/sys/sys/cdefs.h b/sys/sys/cdefs.h index 1de042339c3..e1d3cf636dd 100644 --- a/sys/sys/cdefs.h +++ b/sys/sys/cdefs.h @@ -562,7 +562,11 @@ #endif /* __GNUC__ */ #define __GLOBL(sym) __asm__(".globl " __XSTRING(sym)) +#ifdef __clang__ #define __WEAK(sym) __asm__(".weak " __XSTRING(sym)) +#else +#define __WEAK(sym) __GLOBL(sym) +#endif #if defined(__GNUC__) #define __IDSTRING(name,string) __asm__(".ident\t\"" string "\"") From nobody Sat Aug 5 00:42:46 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RHkMD635bz4mKj3 for ; Sat, 5 Aug 2023 00:42:48 +0000 (UTC) (envelope-from yuri@aetern.org) Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RHkMD4RVkz3TX9 for ; Sat, 5 Aug 2023 00:42:48 +0000 (UTC) (envelope-from yuri@aetern.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=aetern.org header.s=fm3 header.b=vRiR5fkc; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="D JsP8O6"; spf=pass (mx1.freebsd.org: domain of yuri@aetern.org designates 66.111.4.29 as permitted sender) smtp.mailfrom=yuri@aetern.org; dmarc=none Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 3B3D85C00B4 for ; Fri, 4 Aug 2023 20:42:48 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Fri, 04 Aug 2023 20:42:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aetern.org; h=cc :content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm3; t= 1691196168; x=1691282568; bh=uv+fx7C0Wjpeg1dpCzUT5v8JlLpoZNMIsDx X4b7janA=; b=vRiR5fkcxg87icAHZBWaP62MmboHtITzMafAiVu3Mc364b9nHeZ eAGnYcQgAeW+naGS1k+QGYaRZowYvY3boQAcgQpoI2X+J/g6BH64V41rXkKowVqw 85P6ln855FuWhWJleiUrdwGFhQ3ThzM7aJ9hXJxX344ZU4doVoFhtUcfaEpDZIvJ LoE7YloS+j8fJyPbRJsmBkVptHgGzjcIUatMoywPUWzIZaMVGJKMp1ZSruZ2SfVM kHyjnm4q2jYFwOpjBz355e87v9vEuZ10lHuO/wbms6ROyITLmnJJwIhh1vrfc25Q xLghRsTBaGLy6ywBwAgVkZIvB7g3CvnzEGg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1691196168; x= 1691282568; bh=uv+fx7C0Wjpeg1dpCzUT5v8JlLpoZNMIsDxX4b7janA=; b=D JsP8O6QABjO+Ajh0RfJxw7Qw67utf42l4vYiH0WVfA57Levr/odxA15CZMiv3Io+ ZkfsX4HtyH4I7bSB+N6N5c0k1zqD3LlUaclLdjQ1CjRmfJI2JtEhzU3ZR5Ce8rey iSGE6/mBmNA9pMG8amKu0h6VnXy2PdZH7OzsvpdUZYieQv/Sq9hiRc3KaMLV2qTN Bb0+JSl9vQnN2zJB8JNy2uwsB8eb9feA1y+Z4fmIyUeblNtZuA8eBcxwV2iKKp39 EOcxEbDCXhT+Tc3v/RSNAQl0GsUrfOHcJo2LL92Yp7XF721Yh08jVlOgkjTNj0wR nUW61ksIM6c2RskmqqfCA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrkeehgdeffecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepkfffgggfuffvfhfhjggtgfesthejre dttddvjeenucfhrhhomhepjghurhhiuceohihurhhisegrvghtvghrnhdrohhrgheqnecu ggftrfgrthhtvghrnhepveekhfefffeghfehleeglefhkeegtdelgfdvledvueefhefhte ejieekfefgffelnecuffhomhgrihhnpehfrhgvvggsshgurdhorhhgnecuvehluhhsthgv rhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhephihurhhisegrvghtvghrnh drohhrgh X-ME-Proxy: Feedback-ID: i0d79475b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 4 Aug 2023 20:42:47 -0400 (EDT) Message-ID: <4daecfde-80bf-d81e-3f0b-367db1a95ef2@aetern.org> Date: Sat, 5 Aug 2023 02:42:46 +0200 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: amd64-gcc12 seems to break linker sets Content-Language: en-US To: freebsd-hackers@freebsd.org References: <6a0f626c-95b4-2398-0a6d-9428c440131b@aetern.org> From: Yuri In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4RHkMD4RVkz3TX9 X-Spamd-Bar: - X-Rspamd-Pre-Result: action=no action; module=multimap; Matched map: local_wl_from X-Spamd-Result: default: False [-1.89 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; RWL_MAILSPIKE_EXCELLENT(-0.40)[66.111.4.29:from]; R_DKIM_ALLOW(-0.20)[aetern.org:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.29:c]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.29:from]; XM_UA_NO_VERSION(0.01)[]; DKIM_TRACE(0.00)[aetern.org:+,messagingengine.com:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; FREEFALL_USER(0.00)[yuri]; DMARC_NA(0.00)[aetern.org]; local_wl_from(0.00)[yuri@aetern.org]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US] Yuri wrote: > Yuri wrote: >> I noticed that kernel built using amd64-gcc12 panics on `kldload dtraceall`: >> >> panic: probe defined without a provider >> cpuid = 4 >> time = 1691186663 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x35/frame >> 0xfffffe00c448b8a0 >> vpanic() at vpanic+0x232/frame 0xfffffe00c448b950 >> panic() at panic+0x3c/frame 0xfffffe00c448b9b0 >> sdt_kld_load_probes() at sdt_kld_load_probes+0x346/frame 0xfffffe00c448bb70 >> sdt_load_probes_cb() at sdt_load_probes_cb+0x9/frame 0xfffffe00c448bb80 >> linker_file_foreach() at linker_file_foreach+0x53/frame 0xfffffe00c448bbc0 >> linker_load_dependencies() at linker_load_dependencies+0xe18/frame >> 0xfffffe00c448bc70 >> link_elf_load_file() at link_elf_load_file+0x10c6/frame 0xfffffe00c448bd50 >> sys_kldload() at sys_kldload+0x820/frame 0xfffffe00c448bdf0 >> amd64_syscall() at amd64_syscall+0x11b/frame 0xfffffe00c448bf30 >> fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00c448bf30 >> --- syscall (304, FreeBSD ELF64, kldload), rip = 0x7e7e919615a, rsp = >> 0x7e7e70ef498, rbp = 0x7e7e70efa10 --- >> >> Looking into it, gcc12 seems to discard the __start_set_... and >> __stop_set_... symbols declared in sys/linker_set.h as __WEAK(), i.e. >> __asm__(".weak ..."). >> >> The following small test shows the problem: >> --- >> int >> main(void) >> { >> return (0); >> } >> >> __asm__(".weak __start_set_sdt_providers_set"); >> static void const * const __set_sdt_providers_set_sym_sdt_provider_tst >> __attribute__((__section__("set_" "sdt_providers_set"))) >> __attribute__((__used__)) = &(main); >> __asm__(".weak __stop_set_sdt_providers_set"); >> --- >> >> When built using system clang: >> --- >> $ cc -c -o t.o t.c; nm t.o >> 0000000000000000 r __set_sdt_providers_set_sym_sdt_provider_tst >> w __start_set_sdt_providers_set >> w __stop_set_sdt_providers_set >> 0000000000000000 T main >> --- >> >> When built using amd64-gcc12: >> --- >> $ /usr/local/bin/x86_64-unknown-freebsd14.0-gcc12 -c -o t.o t.c; nm t.o >> 0000000000000000 r __set_sdt_providers_set_sym_sdt_provider_tst >> 0000000000000000 T main >> --- >> >> I haven't found any options to tell gcc to stop doing that; any hints? > > Apparently it's any gcc version (starting with oldest available in > ports, 4.8) and it's something that was changed in: > --- > commit 32231805fbe2b9438c2de50c229b43c016207a08 > Author: Greg V > Date: Tue Apr 20 01:47:15 2021 +0100 > > linker_set: fix globl/weak symbol redefinitions to work on clang 12 > --- > > While fixing clang, it seems to broke gcc. > > The following patch (reverting to previous definition for !clang) seems > to work for both clang and gcc, dtraceall kld loads successfully, dtrace > shows all defined std probes: > > diff --git a/sys/sys/cdefs.h b/sys/sys/cdefs.h > index 1de042339c3..e1d3cf636dd 100644 > --- a/sys/sys/cdefs.h > +++ b/sys/sys/cdefs.h > @@ -562,7 +562,11 @@ > #endif /* __GNUC__ */ > > #define __GLOBL(sym) __asm__(".globl " __XSTRING(sym)) > +#ifdef __clang__ > #define __WEAK(sym) __asm__(".weak " __XSTRING(sym)) > +#else > +#define __WEAK(sym) __GLOBL(sym) > +#endif > > #if defined(__GNUC__) > #define __IDSTRING(name,string) __asm__(".ident\t\"" string "\"") > https://reviews.freebsd.org/D41329