From nobody Sun May 14 13:18:45 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QK34W6mvsz4BhRS for ; Sun, 14 May 2023 13:19:23 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-yw1-x1136.google.com (mail-yw1-x1136.google.com [IPv6:2607:f8b0:4864:20::1136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QK34V6H98z47Jv for ; Sun, 14 May 2023 13:19:22 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=T2mE2740; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::1136 as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-yw1-x1136.google.com with SMTP id 00721157ae682-55a1462f9f6so113837887b3.3 for ; Sun, 14 May 2023 06:19:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684070362; x=1686662362; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=f7NtS+pjS8OwZ9bZsD65JlGTvUZtppyLFHW3olQedmc=; b=T2mE2740hLQ6cNJLfWV75YOcjjjwN7AD+ebCYTEYXgQYD4VMpFtItzJNK9Bw0scn1Q jPtAdncGAWPsMAJnTNrIqWvPNWudCxjX3aMz1fdbdGZ2+VrqODCpAWD+7w1/5VKRmCU+ Oo660pS5huGV3KMpKmT0ojRiDeseEhvvNqy6jC+m6xevz6fVNBxb3BulzpKOjZiGgrO/ FEWi7RAPxBYUgpLjSy6yIvnCn2wnQ88uemh3l98BXnetvpv3pnAGDtUL+9GyQJmy+llL Jox2ZeGlJvCrVuxHrsB2T0oJjiCEB5agXgUi+rpUxbaQhkH83srjzoofzvxC6VZAxDzC CZHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684070362; x=1686662362; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=f7NtS+pjS8OwZ9bZsD65JlGTvUZtppyLFHW3olQedmc=; b=Ngz9kwbXGvZlfJAwgWdVSzcJr1/VMOb9F9qZ1YEiCZHOw/ok2WzF0z7wEHHDJQNIJm lAI9mCW12rZZkJ5WkyjQNidFstvtdMEZVEkgzdJL/dgqQRVzdauXdYIlWJeE+EaAI4mm qcDjS77RW5dS6cXhlVRixA4luGg2S7gbnHg1BdhOQP6G5jmo055CULMyd36SoIHqUgNf KQHILn/XalNNiCdangDw+b7MlO1dDYfX9AE2EbhMaK992WLHHFdiGC/FCw7/T7Frurq0 hJEfL9m4+TMVqJbUNUldXYKcQi9yt24JRUjnneF2vCtLEuMIjKkjaNGzY0keuFYL37pv BUVA== X-Gm-Message-State: AC+VfDzRpMA0r9Nw8bWOtAdANLUucrIaz1YccoiqSxeILP6BWFAgX659 0uSgPa6M1v+G1nPyGPMtqBS9KH8nRXpG7GlGQslRGwklVa590A== X-Google-Smtp-Source: ACHHUZ4IlWc8FIphZJiITUYHzSYldltKuEV2O3K+f7DZJAXfUBGGwGC6eL61o2siBzkXiWkuSWcAm/kccZbD5z4tcr0= X-Received: by 2002:a81:4e45:0:b0:55a:985e:8ad1 with SMTP id c66-20020a814e45000000b0055a985e8ad1mr29131073ywb.33.1684070361644; Sun, 14 May 2023 06:19:21 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 From: Mario Marietto Date: Sun, 14 May 2023 15:18:45 +0200 Message-ID: Subject: =?UTF-8?Q?Let=27s_start_a_new_project_=3A_installing_FreeBSD_on_th?= =?UTF-8?Q?e_=22PINEPHONE_=E2=80=93_Beta_Edition_with_Convergence_Package_Linux?= =?UTF-8?Q?_SmartPhone=22?= To: freebsd-hackers Content-Type: multipart/alternative; boundary="000000000000aec54b05fba72fd1" X-Spamd-Result: default: False [-3.92 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; NEURAL_HAM_SHORT(-0.92)[-0.920]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1136:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4QK34V6H98z47Jv X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --000000000000aec54b05fba72fd1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Freebsd-Hackers. I plan to buy the "PINEPHONE =E2=80=93 Beta Edition with Convergence Packag= e Linux SmartPhone". I would like to install FreeBSD instead of Linux on this device. I would like to create a team of developers. I can help those developers,but not deeply,because I'm a system administrator as a hobbyist. I can hack and configure a system if it does not require high technical skills. I'm not a developer. I'm not a software engineer. I would also know if I can participate by putting in some money into the project,because I know that you would take away time that you should devote to your job. I'm not able to quantify how much money this project would require to be completed,to have a clear idea if I can start or not the project itself. If the amount of money is very high for me,I would like to know if whether a crowdfunding project could be started to get the necessary money. Let me know your thoughts. thanks. --=20 Mario. --000000000000aec54b05fba72fd1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello Freebsd-Hackers.

I plan to buy the "PINEPHONE =E2=80=93 Beta Edition with Conv= ergence Package Linux SmartPhone". I would like to install FreeBSD ins= tead of Linux on this device. I would like to create a team of developers. = I can help those developers,but not deeply,because I'm a system adminis= trator as a hobbyist. I can hack and=20 configure a system if it does not require high technical skills. I'm no= t a developer. I'm not a software engineer. I would also know if I can p= articipate by putting in some money into the project,because I know that yo= u would take away time that you sh= ould devote to your job. I'm not able to quantify how much money this p= roject would require to be completed,to have a clear idea if I can start or= not the project itself.
If the amount of money is very high for me,I would= like to know if whether a = crowdfunding project could be started to get the necessary money. Let me kn= ow your thoughts. thanks.

--
Mario.
--000000000000aec54b05fba72fd1-- From nobody Sun May 14 14:59:29 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QK5JF4MJtz4Bn8R for ; Sun, 14 May 2023 14:59:41 +0000 (UTC) (envelope-from schakrabarti@microsoft.com) Received: from APC01-SG2-obe.outbound.protection.outlook.com (mail-sgaapc01on2071b.outbound.protection.outlook.com [IPv6:2a01:111:f400:feab::71b]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QK5JD0zCyz4H9K; Sun, 14 May 2023 14:59:40 +0000 (UTC) (envelope-from schakrabarti@microsoft.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=microsoft.com header.s=selector2 header.b=gCIqWClG; spf=pass (mx1.freebsd.org: domain of schakrabarti@microsoft.com designates 2a01:111:f400:feab::71b as permitted sender) smtp.mailfrom=schakrabarti@microsoft.com; dmarc=pass (policy=reject) header.from=microsoft.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=btk8PbaseIswuLCtFpoaJpRssZuXwOVTXw6L7B1ZMHcisZhQMmUFf0tBkd63dZnWgMhsmRW9p2crX2rBsvwbuS30pdTQcVakmGn3WLmCo2iy0BA8N69IfGwG7GhFpknq9LpBSJNTKQAE7nIrM96lihwDgLtW1MOjUEAfO60Ng0KPGdTx7E0OhGzwgFO4a99pNx+9mbJ7PAxou1EunQGvIaM83gl0K+8u10vCvSB833j202Oua7WAnxxHurIFxh5pMyZEb7qxgwvMToQgXZ02SfqT2p3FUGbZpRxyRxVVNCDWKDH/D5cdiqL6MmSV6Z5u8SsZrcCfay/lLO5MMnG6pg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6hwhh/aQkyOaK0hX0udT6VAB6C6bPVHyO9Q38AgyI3A=; b=hRo0ZCxarQhjIujes7HiM3XhbibzP8GGSIoY/CjKwIpbHAmgwaR+1tmHMI8QiSPsz41ExQ3ft44QnK0ZLYh/iGj4ERh+NlWdPjQ4iNBLX34mfp03+1A6IzdGL0lT9RzqLJcxiaT/V6Hga0TljtYix7XtPuMt3iEacYP6iqsgP29HwljAlvLy2F7gjiGj0bt02bvqO+pnJ/wRa1K7yJB+qeB+hdBB7DOddkubUG386JxByvGW9fu4p4jBvlCJmsmiot7M670XLSsorTLbM9mTHxVKohhURwC+EWhBvidueheEasoCTWsUdIM3UytbKAW8Rn4qBzIjbwysieiwUPEL2w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6hwhh/aQkyOaK0hX0udT6VAB6C6bPVHyO9Q38AgyI3A=; b=gCIqWClGJ7M/jjTeIXnA5Nb8Dp0gmqxx5MoBHCM5WIb8izsfemJymCuCCtLgb8tKNCWXKJedkuLlQDuGcOGH0cjI4DWLzmFEGzSqNgaVFDomkE5+5VgjWMaQg6EX64Px7rPFvhE0zGXPHTJBzyRTVug7Eowv1YdqaKZmj9iVcgk= Received: from PSAP153MB0536.APCP153.PROD.OUTLOOK.COM (2603:1096:301:75::14) by PUZP153MB0751.APCP153.PROD.OUTLOOK.COM (2603:1096:301:e0::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6433.3; Sun, 14 May 2023 14:59:33 +0000 Received: from PSAP153MB0536.APCP153.PROD.OUTLOOK.COM ([fe80::3636:5810:fb12:6d71]) by PSAP153MB0536.APCP153.PROD.OUTLOOK.COM ([fe80::3636:5810:fb12:6d71%7]) with mapi id 15.20.6433.001; Sun, 14 May 2023 14:59:32 +0000 From: Souradeep Chakrabarti To: Kyle Evans CC: Wei Hu , "freebsd-hackers@FreeBSD.org" Subject: RE: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 SMP Thread-Topic: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 SMP Thread-Index: AdmBrWHu4SCK2cRnRnyy6MBGajO35QDM7LrAAATdMgAAXrUlgA== Date: Sun, 14 May 2023 14:59:29 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=b230a199-7896-401d-b004-227d5ce5f029;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-05-14T14:21:26Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PSAP153MB0536:EE_|PUZP153MB0751:EE_ x-ms-office365-filtering-correlation-id: 99b460be-2fdd-483f-610e-08db548bd19b x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: SxnQfOVsRCEjx1BZVW7qq5IiKdU9624tU2VrGZSRecv/IkR9PhZ/mvNjxeXfb12y3JcdfbDeEeb8ObHQM07jAernZ6mKjSznEL5PjNhb8J00HGlN9YXbT2gNKNhuV/ZPCeaG17wYYJY5lfvSwvT+WqS2BD2AOgkh1Ly/gppZjLaq3ugTj5cdELl7wMbSt9gqnhF2n7/oGmjXlXNF918BctKxvnV19AMIIT+3/DYDGQE1yS0mYX/vJwbOTz3cLQ/keTXUKQzKvad3SaNNoa+2mpuh37NM+7arLNFMFl64+et4jBTrMLn3GiXmBh5GUyzfoU7EZkkbzmp6JCn4RqjXYrVBtcvaKBIp99vzIzHcuZdHs9wiujXVxhQreEdAZiJy4qaRd0cvxjABz7bdfZmHm5wiBrMN2vFWvVIxQWgHLTa625Q7FbYv9YaWz19Vzt4wKizxBKHCba7vlKzhDvR+yc/8luC0OoATGMnPRkoveIWm728g2odA7q99ekubrN0cqQS92Lt0L733xWLN2ZGKk5oXwYmudayI3tIjZ8Ji/PCDuPzGlvDzvU2VIyw//IfAwBMLs4QrCoX2u3zJU60qDhRC6M0ipQhcNCqH61j+6szPS836XcBzkDyfshVsuHpqL2PZ7SftJfBCkXJb0lmbIZWgKtapTK5A2p+8+vPmMa8= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PSAP153MB0536.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230028)(4636009)(366004)(376002)(346002)(396003)(136003)(39860400002)(451199021)(83380400001)(4326008)(41300700001)(316002)(6916009)(8990500004)(38070700005)(2906002)(122000001)(82960400001)(786003)(82950400001)(38100700002)(9686003)(6506007)(7696005)(26005)(478600001)(66476007)(66556008)(66946007)(66446008)(64756008)(55016003)(450100002)(76116006)(86362001)(52536014)(5660300002)(8936002)(33656002)(8676002)(10290500003)(54906003)(186003)(71200400001)(6666004)(21314003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?QXN4YURKR2gwVnRYVThDeVhNakNtdllzaDJFakJxOFFqcjJCbXlZdDR0THRP?= =?utf-8?B?ZFN3MXB3Z05yR250NmJISWxTdE15eTNYNVlNbWVURUNrZ1l0REpiMThERE1u?= =?utf-8?B?MnVhL3JNd2FoWUlJbWVYTVlOMVNBSnRuQ0dBRzdHQTlxSEpteUMwQk1nMTFK?= =?utf-8?B?Nk1ON001ODIxa09MWDdJek8yM1hMTUt0LzVUVlkzcFZnaXdLYVpyQ21aaTBT?= =?utf-8?B?YnFqYi83VWZXbFdyT2wxc2p6OFFWVFVqWk83OHZ0Nkx6UTFRSEUvY1k4bEhq?= =?utf-8?B?Uml5eTJSQWFrN2V4M1kvelZSaEozaVdTSFBzUW5JK2o0T29idStEbFhZcWVi?= =?utf-8?B?SXBodEFwVWtzT1RBdjNXVUh6Y2lrUk5GRkxTUFZDQjhJalBzTktVMWRaWXFz?= =?utf-8?B?UktUaVhzbTZQS1FrMC93VlNFMHg5UFpnc1lYUUxPbWRpWTdCKzlWdk9aVEkz?= =?utf-8?B?STVudTJMM202TWkxakhMYWp1cHFvQ1ZlaTQ4S1Y4dzY2c2tpS1BwNGxQT3hp?= =?utf-8?B?UE5PZzE5SnhpV1B6aTlKQ1hmVkJ0YzFjQno3M2hHTXhjSDU5TzRqUStCZmxu?= =?utf-8?B?YzBlVk5ObnBRUFIvV3ZrWHpVdnpESjF0UmMxU3oxMnJkeUl5emxWd2x4U2pQ?= =?utf-8?B?cEtqa2l5Q25salpyc1pObVdaYk9NekczanhXd1RaSkdFMHVQa2hBQm1KdVFk?= =?utf-8?B?RkcyMnNNd2Z2eUxNYmFOZDhLYUJsR1NGOVN2Ky9temxuNHJWdU9VSmpuZ0Ny?= =?utf-8?B?YkEvbWtFam9ad1pkY1RMVHlKOUFqSnIwMnM1WVM5NEJwaW9KRTkyUG1hMExx?= =?utf-8?B?L1FraEpETjcvcDRFNjN5Q3lmcjQ1TDA2aGx4UGM0OGlkamY3STk4T1NlSzVw?= =?utf-8?B?MUw1ZVV4cjFiZGZicno1N0hpekNHTExtSS9hY2FWVXdpcCsyNFhHYVV4YTJG?= =?utf-8?B?aUlTQ0hiTFhqSmtmUXFNcTVtUVlHVUc3bDRxMmJuZktHckNPVVhkT1F5YTBq?= =?utf-8?B?N0NaYjd3OWFJalA0Mjh0MmJGMjZpazZ3a1dHYkFXMGF6cE0wM1I5R3duVUhu?= =?utf-8?B?c05kck0rK0hhcFFDcVpQM2JQUEV5TGh5bU5IY0ZhelhHRU5JaXkwTzVnUjRI?= =?utf-8?B?N041S1BlUkFnZ2laM3NJWllTTmpNdnhsY2hWR25LOTVJZUhjSVltUGhEZEU1?= =?utf-8?B?RGk0Z0czbkI0a1NGRDZ1S0s3aEhKRGVBVHpMOG5xaHp3aUF3U0s2VEdUY2Q2?= =?utf-8?B?Vmp2dDZsTjU2NmtWZVRMRmVIMkpaOXVrbEtKTjBZb2JNb29Vb3RhM1J6ZHBM?= =?utf-8?B?K3IxL1REWFF2Tk5mdW5wQ3cvUlcrZlRkNDJrVHd0dHhFTEI1SVl4OTNFU3Fj?= =?utf-8?B?c0Uzd2lVaWI0bExxaGFYdVo2SC9YVlpmQWIwT2VUNW9YWjJMNVpoanA2eVlr?= =?utf-8?B?Y1ZoUTZFTUZlWlhldkFyNUJlUTNvRDFnNTVTZTlCUThuZjdac3E4WnlmMVBj?= =?utf-8?B?OFg5YmxxZUZVZ3JWOTB3czczaXRtdGIrNTQwREZIVlhyK21KZzJaN3hWT2Fp?= =?utf-8?B?K1g3aTNNb2d0eXdYeTJma2hsRHZ2Nm5ZWEVoWUwvYmxaYkwwZ2wwb1R6cEJY?= =?utf-8?B?L29Fd1hnM2JwSU1zSDVXN2lDM01HZkxTL1dVeTBWZHJwb09YRmtsWDhpbmg3?= =?utf-8?B?eU5DdHVvQ3VQQm1kM0pENjZVTnJwZkJpMXJXZ0o5clN6NEhHK0JTY1BJdHpQ?= =?utf-8?B?aXlrVGJBZWordkVrM01DUXN1NXFRQTlVVENNLzRnSDR5MXowOFBKUGFYbzY1?= =?utf-8?B?QUJ2b2kzbTZMa0ZaRUZ0RENPUmFIUHVsbDErYXZyVmw0bGNRVGhlSGxiZkdz?= =?utf-8?B?Q1NlYy9raEJQYUQ4Q0lBQ0k5NGU3c1BCdUQySVMyTUwybElvQnlZNkszTStS?= =?utf-8?B?cThKNCtzL0xseHRSTU8rZUNLcW5iRm92eU1EMGxOc1orMUNSV3BjL2V6Zmxi?= =?utf-8?B?a3NDVXRpYUZqa0EwRVVkL2sweFRGQWlwekFyN1NOZDZQcVFoZk4vUzlCaVZ1?= =?utf-8?Q?jupZhR?= Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PSAP153MB0536.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 99b460be-2fdd-483f-610e-08db548bd19b X-MS-Exchange-CrossTenant-originalarrivaltime: 14 May 2023 14:59:29.6748 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: pi/CJqulPQimcCsMGcJS0EZAwKm5cKT2NqhIycu8Dm81NCzV+x7ODGtfJJzEAYoLqrnqhTWAurTjyF8Erm0J+3zyxaXoN2UJUGwZMgfnOpE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PUZP153MB0751 X-Spamd-Result: default: False [-8.90 / 15.00]; WHITELIST_SPF_DKIM(-3.00)[microsoft.com:d:+,microsoft.com:s:+]; DWL_DNSWL_LOW(-1.00)[microsoft.com:dkim]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[microsoft.com,reject]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; R_DKIM_ALLOW(-0.20)[microsoft.com:s=selector2]; MIME_BASE64_TEXT(0.10)[]; MIME_GOOD(-0.10)[text/plain]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[microsoft.com:+]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[] X-Rspamd-Queue-Id: 4QK5JD0zCyz4H9K X-Spamd-Bar: -------- X-ThisMailContainsUnwantedMimeParts: N DQoNCg0KPi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+RnJvbTogS3lsZSBFdmFucyA8a2V2 YW5zQGZyZWVic2Qub3JnPg0KPlNlbnQ6IEZyaWRheSwgTWF5IDEyLCAyMDIzIDEwOjQwIFBNDQo+ VG86IFNvdXJhZGVlcCBDaGFrcmFiYXJ0aSA8c2NoYWtyYWJhcnRpQG1pY3Jvc29mdC5jb20+DQo+ Q2M6IFdlaSBIdSA8d2VoQG1pY3Jvc29mdC5jb20+OyBmcmVlYnNkLWhhY2tlcnNARnJlZUJTRC5v cmcNCj5TdWJqZWN0OiBbRVhURVJOQUxdIFJlOiBlbmFibGluZyBzYW1lIFBQSSBpbnRlcnJ1cHQg dG8gYWxsIENQVSBpbiBBUk02NCBTTVANCj4NCj5PbiBGcmksIE1heSAxMiwgMjAyMyBhdCA5OjUx 4oCvQU0gU291cmFkZWVwIENoYWtyYWJhcnRpDQo+PHNjaGFrcmFiYXJ0aUBtaWNyb3NvZnQuY29t PiB3cm90ZToNCj4+DQo+Pg0KPj4NCj4+DQo+PiA+LS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0N Cj4+ID5Gcm9tOiBTb3VyYWRlZXAgQ2hha3JhYmFydGkNCj4+ID5TZW50OiBNb25kYXksIE1heSA4 LCAyMDIzIDY6MzkgUE0NCj4+ID5UbzogS3lsZSBFdmFucyA8a2V2YW5zQGZyZWVic2Qub3JnPg0K Pj4gPkNjOiBXZWkgSHUgPHdlaEBtaWNyb3NvZnQuY29tPjsgZnJlZWJzZC1oYWNrZXJzQEZyZWVC U0Qub3JnDQo+PiA+U3ViamVjdDogZW5hYmxpbmcgc2FtZSBQUEkgaW50ZXJydXB0IHRvIGFsbCBD UFUgaW4gQVJNNjQgU01QDQo+PiA+DQo+PiA+SGkgLA0KPj4gPg0KPj4gPldoaWxlIHVzaW5nIFNN UCBpbiBBUk02NCBIeXBlci1WIHdlIGFyZSBnZXR0aW5nIHN0dWNrIGluIGJvb3QgaWYNCj4+ID50 aGVyZSBpcyBhIGludGVycnVwdCBmb3IgVk1CdXMgY29taW5nIHRvIENQVTEgYW5kIFZNQnVzIGlu dGVycnVwdA0KPj4gPmhhbmRsZXIgaXMgbm90IGdldHRpbmcgdGhhdCBpbnRlcnJ1cHQuDQo+PiA+ DQo+PiA+SW4gQVJNNjQgSHlwZXItViB3ZSBhcmUgdXNpbmcgSVJRMTggZm9yIFZNQnVzIGFuZCBp dCBpcyBhIFBQSSBpbnRlcnJ1cHQuDQo+PiA+DQo+PiA+QnV0IEh5cGV2LVYgaG9zdCBzZW5kcyBp bnRlcnJ1cHQgdG8gdGhpcyBJUlEgMTggZm9yIGJvdGggQ1BVMCBhbmQNCj4+ID5DUFUxIGluIDJD UFUgc3lzdGVtLg0KPj4gPlRoaXMgaXMgYmFzZWQgb24gdGhlIGNvcnJlc3BvbmRpbmcgVk1CdXMg Y2hhbm5lbCB3aGljaCBhc3NpZ25lZCB3aXRoIHRoZSBDUFUuDQo+PiA+DQo+PiA+Tm93IFZNQnVz IElTUiBpcyBnZXR0aW5nIHRoZSBpbnRlcnJ1cHQgaW4gQ1BVMCBidXQgbm90IGdldHRpbmcgZnJv bSBDUFUxLg0KPj4gPkFueSBpZGVhLCBob3cgd2UgY2FuIHVzZSB0aGUgc2FtZSBQUEkgMTggZm9y IGFsbCB0aGUgQ1BVIGNvcmVzPw0KPj4gPg0KPj4gPkFueSBoZWxwIHdpbGwgYmUgYXBwcmVjaWF0 ZWQsIGFzIHRoaXMgaXMgYmxvY2tpbmcgdGhlIGVuYWJsZW1lbnQgb2YNCj4+ID5GcmVlQlNEIGlu IEF6dXJlIEFSTTY0Lg0KPj4gW1NvdXJhZGVlcF0NCj4+IENhbiBzb21lb25lIHBsZWFzZSBoZWxw IG1lIGl0Lg0KPj4NCj4NCj5Mb29raW5nIGF0IGxlYXN0IGF0IHRoZSBHSUMgaW1wbGVtZW50YXRp b24sIGl0IGxvb2tzIGxpa2UgdGhpcyBpcyBhIGtub3duIGxpbWl0YXRpb246DQo+DQo+IDg3NSAg ICAgICAgIC8qDQo+IDg3NiAgICAgICAgICAqIFhYWCAtIEluIGNhc2UgdGhhdCBwZXIgQ1BVIGlu dGVycnVwdCBpcyBnb2luZyB0byBiZQ0KPmVuYWJsZWQgaW4gdGltZQ0KPiA4NzcgICAgICAgICAg KiAgICAgICB3aGVuIFNNUCBpcyBhbHJlYWR5IHN0YXJ0ZWQsIHdlIG5lZWQgc29tZSBJUEkNCj5j YWxsIHdoaWNoDQo+IDg3OCAgICAgICAgICAqICAgICAgIGVuYWJsZXMgaXQgb24gb3RoZXJzIENQ VXMuIEZ1cnRoZXIsIGl0J3MgbW9yZQ0KPmNvbXBsaWNhdGVkIGFzDQo+IDg3OSAgICAgICAgICAq ICAgICAgIHBpY19lbmFibGVfc291cmNlKCkgYW5kIHBpY19kaXNhYmxlX3NvdXJjZSgpDQo+c2hv dWxkIGFjdCBvbg0KPiA4ODAgICAgICAgICAgKiAgICAgICBwZXIgQ1BVIGJhc2lzIG9ubHkuIFRo dXMsIGl0IHNob3VsZCBiZSBzb2x2ZWQNCj5oZXJlIHNvbWVob3cuDQo+IDg4MSAgICAgICAgICAq Lw0KPiA4ODIgICAgICAgICBpZiAoaXNyYy0+aXNyY19mbGFncyAmIElOVFJfSVNSQ0ZfUFBJKQ0K PiA4ODMgICAgICAgICAgICAgICAgIENQVV9TRVQoUENQVV9HRVQoY3B1aWQpLCAmaXNyYy0+aXNy Y19jcHUpOw0KPg0KPkkgdGhpbmsgd2UgbmVlZCBzb21ldGhpbmcgL2xpa2UvIHRoaXM6DQo+aHR0 cHM6Ly9wZW9wbGUuZnIvDQo+ZWVic2Qub3JnJTJGfmtldmFucyUyRnBwaS5kaWZmJmRhdGE9MDUl N0MwMSU3Q3NjaGFrcmFiYXJ0aSU0MG1pY3Jvc29mdC5jDQo+b20lN0NjNWMzZDI1NGI5ZDg0MWU5 YWU5YjA4ZGI1MzBiYjNkMiU3QzcyZjk4OGJmODZmMTQxYWY5MWFiMmQ3Yw0KPmQwMTFkYjQ3JTdD MSU3QzAlN0M2MzgxOTUwODIwMjc3NDQ3MDYlN0NVbmtub3duJTdDVFdGcGJHWnNiMw0KPmQ4ZXlK V0lqb2lNQzR3TGpBd01EQWlMQ0pRSWpvaVYybHVNeklpTENKQlRpSTZJazFoYVd3aUxDSlhWQ0k2 TW4wJTNEDQo+JTdDMzAwMCU3QyU3QyU3QyZzZGF0YT1Td29SMnZIeGg2UVFod3BPZ0ZjUTkzNzhu RFZvdmhkRUtFWG9Fbw0KPmdQS3NjJTNEJnJlc2VydmVkPTAsIHRob3VnaCBpdCBzdGlsbCBoYXMg dGhlIGNhdmVhdCB0aGF0IFBQSXMgZWZmZWN0aXZlbHkgY2Fubm90IGJlDQo+ZnVsbHkgc2V0dXAg YmVmb3JlIFNJX1NVQl9TTVAuDQo+U28sIGl0J3MgbGlrZWx5IGFsbW9zdCBhIE5PUCBmb3IgZXhp c3RpbmcgcGxhdGZvcm1zICh3aWxsIGVtaXQgYSB3YXJuaW5nIHdpdGgNCj5ib290dmVyYm9zZSBm b3IgYXJtdjggdGltZXJzKSBidXQgbWlnaHQgZG8gdGhlIHRyaWNrIGZvciB5b3UuDQpbU291cmFk ZWVwXQ0KVGhhbmtzIGZvciB0aGUgY2hhbmdlIGJ1dCBpdCBkaWQgbm90IHNvbHZlIHRoZSBwcm9i bGVtLiBTdGlsbCB0aGUgaW50ZXJydXB0IGhhbmRsZXINCnZtYnVzX2hhbmRsZV9pbnRyKHN0cnVj dCB0cmFwZnJhbWUgKnRyYXBfZnJhbWUpLCBpcyBub3QgZ2V0dGluZyBjYWxsZWQgZm9yIHRoZSBD UFUgMS4NCkl0IGlzIG9ubHkgZ2V0dGluZyBjYWxsZWQgZm9yIENQVSAwIGFsbCB0aGUgdGltZSBp biBBUk02NCBidXQgaW4geDg2IGl0IGlzIGdldHRpbmcgY2FsbGVkDQpmb3IgYm90aCBDUFUxIGFu ZCBDUFUwLg0KRnJvbSBEREIgSSBoYXZlIGNvbGxlY3RlZCB0aGlzIGRhdGEgaW4gYXJtNjQuIGly cTE4IGlzIGZvciB2bWJ1cy4NCmRiPiBzaG93IGlycXMNCmlycTAgICA8Z2ljMCxpMD46IGNwdSAw MyAoYm91bmQpIGNudCAwDQppcnExICAgPGdpYzAsaTE+OiBjcHUgMDMgY250IDANCmlycTIgICA8 Z2ljMCxpMj46IGNwdSAwMyBjbnQgMA0KaXJxMyAgIDxnaWMwLGkzPjogY3B1IDAzIGNudCAwDQpp cnE0ICAgPGdpYzAsaTQ+OiBjcHUgMDMgY250IDANCmlycTUgICA8Z2ljMCxpNT46IGNwdSAwMyBj bnQgMA0KaXJxNiAgIDxnaWMwLGk2PjogY3B1IDAwIGNudCAwDQppcnE3ICAgPGdpYzAsaTc+OiBj cHUgMDAgY250IDANCmlycTggICA8Z2ljMCxpOD46IGNwdSAwMCBjbnQgMA0KaXJxOSAgIDxnaWMw LGk5PjogY3B1IDAwIGNudCAwDQppcnExMCAgPGdpYzAsaTEwPjogY3B1IDAwIGNudCAwDQppcnEx MSAgPGdpYzAsaTExPjogY3B1IDAwIGNudCAwDQppcnExMiAgPGdpYzAsaTEyPjogY3B1IDAwIGNu dCAwDQppcnExMyAgPGdpYzAsaTEzPjogY3B1IDAwIGNudCAwDQppcnExNCAgPGdpYzAsaTE0Pjog Y3B1IDAwIGNudCAwDQppcnExNSAgPGdpYzAsaTE1PjogY3B1IDAwIGNudCAwDQppcnExNiAgPGdp YzAscDA+OiBjcHUgMDAgY250IDANCmlycTE3ICA8Z2ljMCxwMT46IGNwdSAwMCBjbnQgMA0KaXJx MTggIDxnaWMwLHAyPjogY3B1IDAzIGNudCA0MTENCmlycTE5ICA8Z2ljMCxwMz46IGNwdSAwMCBj bnQgMA0KaXJxMjAgIDxnaWMwLHA0PjogY3B1IDAzIGNudCA4NjQzDQoNCg0KZGI+IHNob3cgaW50 cmNudA0KZ2ljMCxwMjogdm1idXMwICAgICAgICAgICAgICAgICAgICAgICAgIDQxMQ0KZ2ljMCxw NDotcmljX3RpbWVyMCAgICAgICAgICAgICAgICAgICAgIDg2NDMNCmNwdTE6YXN0ICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAxOA0KY3B1MDpwcmVlbXB0ICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDIzOQ0KY3B1MTpwcmVlbXB0ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDUy Ng0KY3B1MTpyZW5kZXp2b3VzICAgICAgICAgICAgICAgICAgICAgICAgIDINCmNwdTE6c3RvcCBo YXJkICAgICAgICAgICAgICAgICAgICAgICAgICAxDQoNCg0KZGI+IHNob3cgaW50cg0Kc3dpNjog dGFzayBxdWV1ZSAocGlkIDEyKSB7U09GVH0NCnN3aTY6IEdpYW50IHRhc2txIChwaWQgMTIpIHtT T0ZUfQ0Kc3dpNTogZmFzdCB0YXNrcSAocGlkIDEyKSB7U09GVH0NCnN3aTE6IG5ldGlzciAwIChw aWQgMTIpIHtTT0ZUfQ0KZ2ljMCxwNDotcmljX3RpbWVyMCAobm8gdGhyZWFkKQ0KZ2ljMCxpMDog cG11MCBwbXUwIChubyB0aHJlYWQpDQpnaWMwLHMzOiBhY3BpX2dlZDAgKHBpZCAxMikNCmdpYzAs czE6IHVhcnQwIChubyB0aHJlYWQpDQpzd2kwOiB1YXJ0IHVhcnQgKHBpZCAxMikge1NPRlR9DQpn aWMwLHMyOiB1YXJ0MSAobm8gdGhyZWFkKQ0KZ2ljMCxwMjogdm1idXMwIChubyB0aHJlYWQpDQo+ DQo+VGhhbmtzLA0KPg0KPkt5bGUgRXZhbnMNCg== From nobody Mon May 15 05:05:40 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QKS4c4hzwz4B8NQ for ; Mon, 15 May 2023 05:05:52 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QKS4c4GSyz3CNs for ; Mon, 15 May 2023 05:05:52 +0000 (UTC) (envelope-from kevans@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684127152; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rJS7EqRUXtx5n3iiH/x4xds+s3HOpC5iQfuEl+iCMPA=; b=K8R3vSJ+pa5AV+teXjs9tiZy1giaYzW7HJTCNx3LYwSmSni7A5xXn+yO/3Y7kY9pshAsNI mdad1Bqn2kUVB5XTN0RYfeW/j601WOUWMcRPEHOCN161MLfv33m8GRbCN6ob87CH3Ie4pC HKuKpTqIggbq73bE7UPPFszLx8KUpdj/Fv+82hENFDTypQ3oAt3COxkXIDVV6j3g5KGcPC W1J3vEl7WUfM1d1ra/974SKJuPkl+kdE5P37S7SrsQZUxbGXdBH4IA+XpCPOGBr2irUq1n etODrHbrtdcvJ0f+eu8NYlirBdQGqGx/C0fRi30bg19Dk7g67RSkmsBBh6oT4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684127152; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rJS7EqRUXtx5n3iiH/x4xds+s3HOpC5iQfuEl+iCMPA=; b=mb+yzL3FlgxUIW1CC+H8MseY7TdD+UEzZ2V2+fwnom4S/a9bkjJO9+0CD51DnlwFUivNCe ZVXu0wELs4BuEifQKpZUBa9qWQ8BVog5WrpMGJQO4xxLiuWIxnAYnVpUy0+Q3DWQd553aN GiKLKoWWF4bsFO3q5HXEXvclXcWntgGRz3Yohsxegbpa+YXGUzr212narlVqxkV7mzI1AE qSu4sW2bVXj1sTlCab4Gk3XemuqEzi1wlYpHyEwOWy9j74K75HKxAIZkUH5WEix3n476kx CTCs3MtxTX/w/j/VNAOBE0hnYyvVtNlfQl2JIeKj2Vj4nv120gvwMPR2hPVlkg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1684127152; a=rsa-sha256; cv=none; b=XLOjMLDwxq3v8+/oTUgVI/35Zt7hfKmP+HG6pBh5MDss+SY7FRPs0ZvrGji70l671AuTLO yaX4vOwcO1hlMXOk7xRU8roxQzjBc0mxsVvAo6PHmjqP+vPuDJpwjpcXPKN5mDG/JwQuM9 umW+Yh3f85AS46PtkvxZAqND7up43T2H19/Xd20wymzzVmRXIcO79R+zBfYXBasfqVjX7h JQ2+ml9O1UImhDPcI08saUbJOkhGaqPhlt4xARjYdhRhmljOEdafnDKjJFgiyowm4vF1Hn 81HzzsxK5C6o2BH765jxIcBbcLpGQAi1DDolpTvQtlMpA86S82w6l7+Ce+at6Q== Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 4QKS4c371ZzmwM for ; Mon, 15 May 2023 05:05:52 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-7577ef2fa31so2535103385a.0 for ; Sun, 14 May 2023 22:05:52 -0700 (PDT) X-Gm-Message-State: AC+VfDwSIDGVo24rCvbjRkd1mL6Z1ZPVwxkgNALhHLU2qSbLZDWYC9W5 vUwemFwS9tvAPYMgPRdKfKs/jJiZxgahiCJIjgk= X-Google-Smtp-Source: ACHHUZ4RwJ6FDGlk4rohLkSo08PxDR5Mt9qciZbBhxn8j9lmtQauKowCn/uNMtT2zKvB7B6Kfb+DC0KO7HuBVB/vces= X-Received: by 2002:a05:6214:c46:b0:621:53d1:3c69 with SMTP id r6-20020a0562140c4600b0062153d13c69mr25469061qvj.20.1684127151535; Sun, 14 May 2023 22:05:51 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Kyle Evans Date: Mon, 15 May 2023 00:05:40 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 SMP To: Souradeep Chakrabarti Cc: Wei Hu , "freebsd-hackers@FreeBSD.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-ThisMailContainsUnwantedMimeParts: N On Sun, May 14, 2023 at 9:59=E2=80=AFAM Souradeep Chakrabarti wrote: > > > > > >-----Original Message----- > >From: Kyle Evans > >Sent: Friday, May 12, 2023 10:40 PM > >To: Souradeep Chakrabarti > >Cc: Wei Hu ; freebsd-hackers@FreeBSD.org > >Subject: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 = SMP > > > >On Fri, May 12, 2023 at 9:51=E2=80=AFAM Souradeep Chakrabarti > > wrote: > >> > >> > >> > >> > >> >-----Original Message----- > >> >From: Souradeep Chakrabarti > >> >Sent: Monday, May 8, 2023 6:39 PM > >> >To: Kyle Evans > >> >Cc: Wei Hu ; freebsd-hackers@FreeBSD.org > >> >Subject: enabling same PPI interrupt to all CPU in ARM64 SMP > >> > > >> >Hi , > >> > > >> >While using SMP in ARM64 Hyper-V we are getting stuck in boot if > >> >there is a interrupt for VMBus coming to CPU1 and VMBus interrupt > >> >handler is not getting that interrupt. > >> > > >> >In ARM64 Hyper-V we are using IRQ18 for VMBus and it is a PPI interru= pt. > >> > > >> >But Hypev-V host sends interrupt to this IRQ 18 for both CPU0 and > >> >CPU1 in 2CPU system. > >> >This is based on the corresponding VMBus channel which assigned with = the CPU. > >> > > >> >Now VMBus ISR is getting the interrupt in CPU0 but not getting from C= PU1. > >> >Any idea, how we can use the same PPI 18 for all the CPU cores? > >> > > >> >Any help will be appreciated, as this is blocking the enablement of > >> >FreeBSD in Azure ARM64. > >> [Souradeep] > >> Can someone please help me it. > >> > > > >Looking at least at the GIC implementation, it looks like this is a know= n limitation: > > > > 875 /* > > 876 * XXX - In case that per CPU interrupt is going to be > >enabled in time > > 877 * when SMP is already started, we need some IPI > >call which > > 878 * enables it on others CPUs. Further, it's more > >complicated as > > 879 * pic_enable_source() and pic_disable_source() > >should act on > > 880 * per CPU basis only. Thus, it should be solved > >here somehow. > > 881 */ > > 882 if (isrc->isrc_flags & INTR_ISRCF_PPI) > > 883 CPU_SET(PCPU_GET(cpuid), &isrc->isrc_cpu); > > > >I think we need something /like/ this: > >https://people.fr/ > >eebsd.org%2F~kevans%2Fppi.diff&data=3D05%7C01%7Cschakrabarti%40microsoft= .c > >om%7Cc5c3d254b9d841e9ae9b08db530bb3d2%7C72f988bf86f141af91ab2d7c > >d011db47%7C1%7C0%7C638195082027744706%7CUnknown%7CTWFpbGZsb3 > >d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D > >%7C3000%7C%7C%7C&sdata=3DSwoR2vHxh6QQhwpOgFcQ9378nDVovhdEKEXoEo > >gPKsc%3D&reserved=3D0, though it still has the caveat that PPIs effectiv= ely cannot be > >fully setup before SI_SUB_SMP. > >So, it's likely almost a NOP for existing platforms (will emit a warning= with > >bootverbose for armv8 timers) but might do the trick for you. > [Souradeep] > Thanks for the change but it did not solve the problem. Still the interru= pt handler > vmbus_handle_intr(struct trapframe *trap_frame), is not getting called fo= r the CPU 1. > It is only getting called for CPU 0 all the time in ARM64 but in x86 it i= s getting called > for both CPU1 and CPU0. Interesting! I do see one problem with the patch (and some cosmetic issues): we really need to take the gic_mtx in gic_v3_setup_periph() right up front because CPU_SET() won't necessarily be atomic. That's not the problem, though for other reasons, but also because... > From DDB I have collected this data in arm64. irq18 is for vmbus. > db> show irqs > ... > irq18 : cpu 03 cnt 411 > .... > That would seem to indicate that both CPUs have set it up, but it occurs to me that enable_intr also needs the same treatment. Let's wipe gic_v3.c back to a clean slate and try a v2 of the patch: https://people.freebsd.org/~kevans/ppi-v2.diff For now we just pretend that we won't be disabling any PPIs as a proof-of-concept. Thanks, Kyle Evans From nobody Mon May 15 07:41:21 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QKWX810Hvz4BJ1W for ; Mon, 15 May 2023 07:41:28 +0000 (UTC) (envelope-from schakrabarti@microsoft.com) Received: from APC01-SG2-obe.outbound.protection.outlook.com (mail-sgaapc01on2113.outbound.protection.outlook.com [40.107.215.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QKWX73nZBz3hyL; Mon, 15 May 2023 07:41:27 +0000 (UTC) (envelope-from schakrabarti@microsoft.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=T1uo7jZDpaiClbEwBkfrTBoSyVC/lD2kRgGR4PSb9umtuTiVcYUxZ0WbyqP3L+/ALsDEC+K1h0Rv1ifu+LVXw17ayzO62c8KUKWi5SJ2vt3/tZn1BH86MTxHJz8P+vgilQROkWfbKjojQpKCxQv6ErIh1PGCyV0NpdDx1fsIhYJxcEuXkzK8KBIgNmjPqCT8dmRsy7iA7F+DsR3bYCkfuL3D5I4u3AVGlBtD75w7agVnYwiqX8CVbS+pjltFXj+C8q5dXx+HVBGKYqmUvYR10fqxCmFymM8OpRpEotgJF8top7wVXkG7LoYpzGHz9OEo0pEOROKrkK+wtDqJt7EbNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=o6/J3A62QD2wwJzPvdF5MVJqhuunNqdzrtsFkpdsGBQ=; b=H7o89L+4XNMzTE7NjzIF6iLw0GGhRMJGPhTiDfHcL5KZF5eZkx0H+mPjsVzZx58gDWMfH0UYzANElfyRB80bqjhurMcv6rHgn2A83H2TcHxdSovbzmhbjIwebq02LLCiF78VzR8JZ7IOGklZ4gum7BKLKr47z0XlQgwoqf0/MRzdSi1sAtO5ppDVfpNO9DQYqKT1gpb0Qc/q09P+7Bf3mq1b1EV4vNv4wWI8CTvdoL2zGJ00Np6Ewf4nadlE6sqcfNnutUrsPH3Rh+geaCGnl/tTpMpqst/7EYZ1F21UhvFoPPvTaORJ8QximEDtlio1pK6mqxGTLklx4qdnOWkE+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=o6/J3A62QD2wwJzPvdF5MVJqhuunNqdzrtsFkpdsGBQ=; b=fjdteHTFuL7sxXVXDkvIhvvMArSora1f469UpdxaXB3Y7kQVOVXdrnP+2aEuM1BcVTMld03W+G1cLb6X3ftys/6VPipSV1XE0jytHm6DM4/A693rqUE+p5bBziugyIU6x9ATnXTKglgZ04daJGhhRf1VIvqtV9kLOmdD1g/GVgk= Received: from PSAP153MB0536.APCP153.PROD.OUTLOOK.COM (2603:1096:301:75::14) by SEYP153MB0780.APCP153.PROD.OUTLOOK.COM (2603:1096:101:b1::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6433.3; Mon, 15 May 2023 07:41:21 +0000 Received: from PSAP153MB0536.APCP153.PROD.OUTLOOK.COM ([fe80::3636:5810:fb12:6d71]) by PSAP153MB0536.APCP153.PROD.OUTLOOK.COM ([fe80::3636:5810:fb12:6d71%7]) with mapi id 15.20.6433.001; Mon, 15 May 2023 07:41:21 +0000 From: Souradeep Chakrabarti To: Kyle Evans CC: Wei Hu , "freebsd-hackers@FreeBSD.org" Subject: RE: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 SMP Thread-Topic: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 SMP Thread-Index: AdmBrWHu4SCK2cRnRnyy6MBGajO35QDM7LrAAATdMgAAXrUlgAAe4ZoAAAUxoMA= Date: Mon, 15 May 2023 07:41:21 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=6e941e28-2e6e-41ff-9b6d-f8359c7376b3;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-05-15T07:34:22Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PSAP153MB0536:EE_|SEYP153MB0780:EE_ x-ms-office365-filtering-correlation-id: f0191033-8ae1-4629-5b79-08db5517c6e4 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: dAnjsRbBykjNXiXMJsvtxodxjF0qfuVW9AechOrVx21LiYGbQdeZ3EjR+zDOd5gDCBQ1RnjDEem/QMQJGZDCSl5XDuCEKGBmkjJSv3LwEWHKkcv8V1bGmrN57jN/uNaeji/ORFAYVvv8h95v2oy9cAzAtU8VZSlqGCaVZZScDB9wiKJ13LizdtREvy3uD6G/tpAg2gieZ62b7KlVh4YBH5YhHe20d/xL3gGGACn5yhp0zTfEL6Kv1pLrhikiyng6cXyR9etsDR7bZIQs9VxshVfWhz0/8DeBRAoksaapbzjnC9W+WDvdYjS3jN+E7NW1dAGjG4pioOwKAkTYovG4qak3S61MHeW/SlblO6TzDaX5QMV1Slc+D81+swiOP4pnzAxlKaX9i91pY84vF0/KOtvviL79frdJpGcNbAjLdAqJJyAP9yo6iT5KfVSZM1YnfpxhXp3fluGEql6J1iN0qWYnsdIP1R+EL5hN1YzE91kTqHR+2qt8IbhA7FNyibIaCNQRD+cGlsX+DSn947yrv64dfh3+ymkzjzEen5giZSZhTNjZ72a93de1OTotnibYu/KG53irtBlDAOrHxYoAcms6ncstqHJQJ6HZR7r4XClRvSxmh5K88ZDfDbPWZnAqJqHuuHVWWcD5E0ru8Daw+UH6+S4X7uR0/kUlliFyYYKOV2Zq18iJP400k997/H66mf1n11mCuN6YscYvp2VHag== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PSAP153MB0536.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230028)(4636009)(346002)(136003)(39860400002)(366004)(396003)(376002)(451199021)(55016003)(2906002)(8990500004)(83380400001)(450100002)(966005)(76116006)(66556008)(66476007)(66446008)(64756008)(66946007)(478600001)(10290500003)(7696005)(54906003)(8936002)(8676002)(52536014)(9686003)(26005)(186003)(6506007)(5660300002)(4326008)(786003)(316002)(41300700001)(6916009)(71200400001)(86362001)(82960400001)(82950400001)(38070700005)(38100700002)(122000001)(33656002)(10126625003)(21314003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?bDlxdlNxcjhmNjltU2NXcUdWUDQwTDNNUGgwVi9WM3pmN3dKdEdjTzdKMmlY?= =?utf-8?B?ZkJYQlBCSTlNY2Myd2xmVkJ6SlBKbDVpTkFCYkRBUG4vZ3h2Zk1NUXRrSWl3?= =?utf-8?B?bXh1SjI2cTRGMCtDUSttY0hoQXFDcnIyNmNUM1Y2cXFlUllaUVJ5aDgyYWlV?= =?utf-8?B?Z1B0bEFLbHZPUmJqejBEblNaV013Rk0zSjhPbXpic2dJUElHVVluT2xzQndt?= =?utf-8?B?eUg0eWdMOXR0YUJ2WnA5RDhuME5QaEViUjNxbER6YjBSUVl0Y3RTUHdCNU9L?= =?utf-8?B?Mks1OVF2MWVva3NZWVBldUI0RXFrRklXSDNIOExnelIrZmFSMTAwQWtWcGQr?= =?utf-8?B?YkQxcGxObEs3QWJtaEQrWlphQkhidmxvY3BLbmxEUUNRREd5NVNkd3FYcjg0?= =?utf-8?B?L0VrT093cmZWZnJDOWI0V0dBVjhja2s4MCsvWExVTis4TkRDZkdUUldwMmov?= =?utf-8?B?NGZaRnRDYVQ0dEZleWloV1lDY0NVa1NjdkJEWGRKRGFGczQ0UHVuUFFUSVBE?= =?utf-8?B?TlRnSUpFOHZRWG83R0oyeGtiT1R2RGxTYURNQkFrK2h1ZUVzM29CdTZma1Fz?= =?utf-8?B?bjQxMzVrbUNhYiswenlQcGJJb0RPd1ZZajhYUlRFZHU3R2o4MGs1V3F0SVNx?= =?utf-8?B?WUdCNm1rcUk4S3A2MkYzcEZzK01DeVhQUW9FQlk2QXBVbTRLQ2lJd2VXaWZ0?= =?utf-8?B?REhlcWNIbkZzby9BS0FoZ0JGdG9ZaXRxK1gzWS90dXZHR3N4UHJteXZaZys3?= =?utf-8?B?d2dMRWZTMmgxQ3FoQVNkeE9ETERYVE1ScmVrNHZhVWNkaWw5cU1RK0I2QWVX?= =?utf-8?B?RjhFcUZNQ29NRmdtektMSHV1WG5EMktwT2ZCOW8zcG0zd3E1T3NBMWxXUm5J?= =?utf-8?B?djNoNWFMajFibXhackdnUkdQQUdlR1ptVG9CUGFyRFRLNVdsZTM3akpiZGhI?= =?utf-8?B?WFlPVjhuSzY0VzhMV3QxcE1EQ0VaL3hTQkNZQVNOdDN0bkpRV2p2eFRUK0l0?= =?utf-8?B?ZExVV21tR25BUTJHaDdvL0svdnJybFhRR2hKRi9Ncmd6c1NKZGE5RjJiRzRv?= =?utf-8?B?WE5nSzBDTGllVjVTMWFPY1Q0VlB6K3I1OU12TG9sZTA1VHB4ck8yYlo5VWRJ?= =?utf-8?B?R0NBMGVYZUlGemJDcm9JeEFIVGNScmJMRnEvKzV5QzhvWHJuN2RQYUdGaTRm?= =?utf-8?B?L3J2TVY0VzFqREkySExNa1RDMkEvQVRQR1FjNy9ScTFiclZHTHExWkZFQnJ3?= =?utf-8?B?MWJ2cWM0d2RRZ0dXWnpFUDV2MFBYbU9aOXA5bGMwb3NaTnJBUnhIbi9URzk0?= =?utf-8?B?Y1B0eVc3dW5GREZSS3BJRjJlNkxvdUhBdFNiYUNzNkFNLzNVb3VrM1hjMlAz?= =?utf-8?B?bkhlWWdhb21jSEhmQWxDZHdqZzdGdExUZmpLV2V3aFdZMjBEZVZZYzdYdWNC?= =?utf-8?B?Mk9iQ1JqTE15YjJUbjUzOXFVOUFDRElQNDVIZnk4Zk5KcjdUdkpVSlhHTXlC?= =?utf-8?B?UFFtNDFMeHg2bWNib3RLbjZNeDJCVTZaakZyZ1ZZcENjZ2FvOG8yUnVva05Z?= =?utf-8?B?QzZEVHZ1dEZxeEtEaDE4Y1haVTAybGF6WU5XQ1g5MDM2RVFJUmFnOTRWeVhY?= =?utf-8?B?dGRaY1hQbWducnN2NmNlWk5pamljWktsZTVEZG81NDlIY1czMmlYSXMwNkJu?= =?utf-8?B?ejZkOWswM09hd2xMRXZCeTFVTFd4di9lbVB1K3JkYjZzalAvKzN5Uk5WckZR?= =?utf-8?B?M1FIOCtVc3RqNjhaVk5uclRoZXRuYWQ4NVIvemQxUHNTRWIwNjRib2R0WGtE?= =?utf-8?B?OEVMZFdId2x5N1MwbG0xNVU0VXRJc1VMU2lQa2NIa3d4NkZiTS84TTBOWDdh?= =?utf-8?B?blViMExLK0tlS0d6UzNlMFhYY3BHazRuZ3hVUm1CdWsyc2I0TDhhcnhDK2h1?= =?utf-8?B?cFBOK25QaThHTzlkb3NZU21zNE1hTXRFVlgxRnZYakhIMGYwTko1STVBSmxH?= =?utf-8?B?NUYzRkhTczJaSlQ3ZkVVMm01MkU1NTNUcVdmcVF3VWpZYi96RmZMYnNPSkpk?= =?utf-8?Q?eqav+W?= Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PSAP153MB0536.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: f0191033-8ae1-4629-5b79-08db5517c6e4 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 May 2023 07:41:21.2062 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 3Vok4FuxWX/oKlSDasRqK4KB/SS3OAyatPkDeChkZC5XR6SNPVILFR85ZIU/hA4Z52ILi4tPPnoWT2M7jANjRdMjBJxOH794+Xl9uaQHeLc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SEYP153MB0780 X-Rspamd-Queue-Id: 4QKWX73nZBz3hyL X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N DQoNCg0KPi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+RnJvbTogS3lsZSBFdmFucyA8a2V2 YW5zQGZyZWVic2Qub3JnPg0KPlNlbnQ6IE1vbmRheSwgTWF5IDE1LCAyMDIzIDEwOjM2IEFNDQo+ VG86IFNvdXJhZGVlcCBDaGFrcmFiYXJ0aSA8c2NoYWtyYWJhcnRpQG1pY3Jvc29mdC5jb20+DQo+ Q2M6IFdlaSBIdSA8d2VoQG1pY3Jvc29mdC5jb20+OyBmcmVlYnNkLWhhY2tlcnNARnJlZUJTRC5v cmcNCj5TdWJqZWN0OiBSZTogW0VYVEVSTkFMXSBSZTogZW5hYmxpbmcgc2FtZSBQUEkgaW50ZXJy dXB0IHRvIGFsbCBDUFUgaW4gQVJNNjQgU01QDQo+DQo+T24gU3VuLCBNYXkgMTQsIDIwMjMgYXQg OTo1OeKAr0FNIFNvdXJhZGVlcCBDaGFrcmFiYXJ0aQ0KPjxzY2hha3JhYmFydGlAbWljcm9zb2Z0 LmNvbT4gd3JvdGU6DQo+Pg0KPj4NCj4+DQo+Pg0KPj4gPi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0t LS0tDQo+PiA+RnJvbTogS3lsZSBFdmFucyA8a2V2YW5zQGZyZWVic2Qub3JnPg0KPj4gPlNlbnQ6 IEZyaWRheSwgTWF5IDEyLCAyMDIzIDEwOjQwIFBNDQo+PiA+VG86IFNvdXJhZGVlcCBDaGFrcmFi YXJ0aSA8c2NoYWtyYWJhcnRpQG1pY3Jvc29mdC5jb20+DQo+PiA+Q2M6IFdlaSBIdSA8d2VoQG1p Y3Jvc29mdC5jb20+OyBmcmVlYnNkLWhhY2tlcnNARnJlZUJTRC5vcmcNCj4+ID5TdWJqZWN0OiBb RVhURVJOQUxdIFJlOiBlbmFibGluZyBzYW1lIFBQSSBpbnRlcnJ1cHQgdG8gYWxsIENQVSBpbg0K Pj4gPkFSTTY0IFNNUA0KPj4gPg0KPj4gPk9uIEZyaSwgTWF5IDEyLCAyMDIzIGF0IDk6NTHigK9B TSBTb3VyYWRlZXAgQ2hha3JhYmFydGkNCj4+ID48c2NoYWtyYWJhcnRpQG1pY3Jvc29mdC5jb20+ IHdyb3RlOg0KPj4gPj4NCj4+ID4+DQo+PiA+Pg0KPj4gPj4NCj4+ID4+ID4tLS0tLU9yaWdpbmFs IE1lc3NhZ2UtLS0tLQ0KPj4gPj4gPkZyb206IFNvdXJhZGVlcCBDaGFrcmFiYXJ0aQ0KPj4gPj4g PlNlbnQ6IE1vbmRheSwgTWF5IDgsIDIwMjMgNjozOSBQTQ0KPj4gPj4gPlRvOiBLeWxlIEV2YW5z IDxrZXZhbnNAZnJlZWJzZC5vcmc+DQo+PiA+PiA+Q2M6IFdlaSBIdSA8d2VoQG1pY3Jvc29mdC5j b20+OyBmcmVlYnNkLWhhY2tlcnNARnJlZUJTRC5vcmcNCj4+ID4+ID5TdWJqZWN0OiBlbmFibGlu ZyBzYW1lIFBQSSBpbnRlcnJ1cHQgdG8gYWxsIENQVSBpbiBBUk02NCBTTVANCj4+ID4+ID4NCj4+ ID4+ID5IaSAsDQo+PiA+PiA+DQo+PiA+PiA+V2hpbGUgdXNpbmcgU01QIGluIEFSTTY0IEh5cGVy LVYgd2UgYXJlIGdldHRpbmcgc3R1Y2sgaW4gYm9vdCBpZg0KPj4gPj4gPnRoZXJlIGlzIGEgaW50 ZXJydXB0IGZvciBWTUJ1cyBjb21pbmcgdG8gQ1BVMSBhbmQgVk1CdXMgaW50ZXJydXB0DQo+PiA+ PiA+aGFuZGxlciBpcyBub3QgZ2V0dGluZyB0aGF0IGludGVycnVwdC4NCj4+ID4+ID4NCj4+ID4+ ID5JbiBBUk02NCBIeXBlci1WIHdlIGFyZSB1c2luZyBJUlExOCBmb3IgVk1CdXMgYW5kIGl0IGlz IGEgUFBJIGludGVycnVwdC4NCj4+ID4+ID4NCj4+ID4+ID5CdXQgSHlwZXYtViBob3N0IHNlbmRz IGludGVycnVwdCB0byB0aGlzIElSUSAxOCBmb3IgYm90aCBDUFUwIGFuZA0KPj4gPj4gPkNQVTEg aW4gMkNQVSBzeXN0ZW0uDQo+PiA+PiA+VGhpcyBpcyBiYXNlZCBvbiB0aGUgY29ycmVzcG9uZGlu ZyBWTUJ1cyBjaGFubmVsIHdoaWNoIGFzc2lnbmVkIHdpdGggdGhlDQo+Q1BVLg0KPj4gPj4gPg0K Pj4gPj4gPk5vdyBWTUJ1cyBJU1IgaXMgZ2V0dGluZyB0aGUgaW50ZXJydXB0IGluIENQVTAgYnV0 IG5vdCBnZXR0aW5nIGZyb20gQ1BVMS4NCj4+ID4+ID5BbnkgaWRlYSwgaG93IHdlIGNhbiB1c2Ug dGhlIHNhbWUgUFBJIDE4IGZvciBhbGwgdGhlIENQVSBjb3Jlcz8NCj4+ID4+ID4NCj4+ID4+ID5B bnkgaGVscCB3aWxsIGJlIGFwcHJlY2lhdGVkLCBhcyB0aGlzIGlzIGJsb2NraW5nIHRoZSBlbmFi bGVtZW50DQo+PiA+PiA+b2YgRnJlZUJTRCBpbiBBenVyZSBBUk02NC4NCj4+ID4+IFtTb3VyYWRl ZXBdDQo+PiA+PiBDYW4gc29tZW9uZSBwbGVhc2UgaGVscCBtZSBpdC4NCj4+ID4+DQo+PiA+DQo+ PiA+TG9va2luZyBhdCBsZWFzdCBhdCB0aGUgR0lDIGltcGxlbWVudGF0aW9uLCBpdCBsb29rcyBs aWtlIHRoaXMgaXMgYSBrbm93biBsaW1pdGF0aW9uOg0KPj4gPg0KPj4gPiA4NzUgICAgICAgICAv Kg0KPj4gPiA4NzYgICAgICAgICAgKiBYWFggLSBJbiBjYXNlIHRoYXQgcGVyIENQVSBpbnRlcnJ1 cHQgaXMgZ29pbmcgdG8gYmUNCj4+ID5lbmFibGVkIGluIHRpbWUNCj4+ID4gODc3ICAgICAgICAg ICogICAgICAgd2hlbiBTTVAgaXMgYWxyZWFkeSBzdGFydGVkLCB3ZSBuZWVkIHNvbWUgSVBJDQo+ PiA+Y2FsbCB3aGljaA0KPj4gPiA4NzggICAgICAgICAgKiAgICAgICBlbmFibGVzIGl0IG9uIG90 aGVycyBDUFVzLiBGdXJ0aGVyLCBpdCdzIG1vcmUNCj4+ID5jb21wbGljYXRlZCBhcw0KPj4gPiA4 NzkgICAgICAgICAgKiAgICAgICBwaWNfZW5hYmxlX3NvdXJjZSgpIGFuZCBwaWNfZGlzYWJsZV9z b3VyY2UoKQ0KPj4gPnNob3VsZCBhY3Qgb24NCj4+ID4gODgwICAgICAgICAgICogICAgICAgcGVy IENQVSBiYXNpcyBvbmx5LiBUaHVzLCBpdCBzaG91bGQgYmUgc29sdmVkDQo+PiA+aGVyZSBzb21l aG93Lg0KPj4gPiA4ODEgICAgICAgICAgKi8NCj4+ID4gODgyICAgICAgICAgaWYgKGlzcmMtPmlz cmNfZmxhZ3MgJiBJTlRSX0lTUkNGX1BQSSkNCj4+ID4gODgzICAgICAgICAgICAgICAgICBDUFVf U0VUKFBDUFVfR0VUKGNwdWlkKSwgJmlzcmMtPmlzcmNfY3B1KTsNCj4+ID4NCj4+ID5JIHRoaW5r IHdlIG5lZWQgc29tZXRoaW5nIC9saWtlLyB0aGlzOg0KPj4gPmh0dHBzOi8vcGVvLw0KPj4NCj4+ cGxlLmZyJTJGJmRhdGE9MDUlN0MwMSU3Q3NjaGFrcmFiYXJ0aSU0MG1pY3Jvc29mdC5jb20lN0Mz OWQ3YzUwODQ5Yg0KPjY0DQo+Pg0KPj5lYWY3MGI2MDhkYjU1MDIwZmFlJTdDNzJmOTg4YmY4NmYx NDFhZjkxYWIyZDdjZDAxMWRiNDclN0MxJTdDMCUNCj43QzYzODENCj4+DQo+Pjk3MjM5NjE1NTQ0 MTEzJTdDVW5rbm93biU3Q1RXRnBiR1pzYjNkOGV5SldJam9pTUM0d0xqQXdNREFpTEMNCj5KUUlq b2lWMg0KPj4NCj4+bHVNeklpTENKQlRpSTZJazFoYVd3aUxDSlhWQ0k2TW4wJTNEJTdDMzAwMCU3 QyU3QyU3QyZzZGF0YT1kWFZ2bkkNCj5IaE45DQo+PiA+bWR3aVBrU0pLd015RUtZaTVTeUdPdXRh NXpDWjF5c0NRJTNEJnJlc2VydmVkPTANCj4+DQo+PmVlYnNkLm9yZyUyRn5rZXZhbnMlMkZwcGku ZGlmZiZkYXRhPTA1JTdDMDElN0NzY2hha3JhYmFydGklNDBtaWNyb3NvZg0KPj4gPnQuYw0KPm9t JTdDYzVjM2QyNTRiOWQ4NDFlOWFlOWIwOGRiNTMwYmIzZDIlN0M3MmY5ODhiZjg2ZjE0MWFmOTFh YjJkN2MNCj4+DQo+PmQwMTFkYjQ3JTdDMSU3QzAlN0M2MzgxOTUwODIwMjc3NDQ3MDYlN0NVbmtu b3duJTdDVFdGcGJHWnNiDQo+Mw0KPj4NCj4+ZDhleUpXSWpvaU1DNHdMakF3TURBaUxDSlFJam9p VjJsdU16SWlMQ0pCVGlJNklrMWhhV3dpTENKWFZDSTZNbjAlM0QNCj4+DQo+PiU3QzMwMDAlN0Ml N0MlN0Mmc2RhdGE9U3dvUjJ2SHhoNlFRaHdwT2dGY1E5Mzc4bkRWb3ZoZEVLRVhvRQ0KPm8NCj4+ ID5nUEtzYyUzRCZyZXNlcnZlZD0wLCB0aG91Z2ggaXQgc3RpbGwgaGFzIHRoZSBjYXZlYXQgdGhh dCBQUElzDQo+PiA+ZWZmZWN0aXZlbHkgY2Fubm90IGJlIGZ1bGx5IHNldHVwIGJlZm9yZSBTSV9T VUJfU01QLg0KPj4gPlNvLCBpdCdzIGxpa2VseSBhbG1vc3QgYSBOT1AgZm9yIGV4aXN0aW5nIHBs YXRmb3JtcyAod2lsbCBlbWl0IGENCj4+ID53YXJuaW5nIHdpdGggYm9vdHZlcmJvc2UgZm9yIGFy bXY4IHRpbWVycykgYnV0IG1pZ2h0IGRvIHRoZSB0cmljayBmb3IgeW91Lg0KPj4gW1NvdXJhZGVl cF0NCj4+IFRoYW5rcyBmb3IgdGhlIGNoYW5nZSBidXQgaXQgZGlkIG5vdCBzb2x2ZSB0aGUgcHJv YmxlbS4gU3RpbGwgdGhlDQo+PiBpbnRlcnJ1cHQgaGFuZGxlciB2bWJ1c19oYW5kbGVfaW50cihz dHJ1Y3QgdHJhcGZyYW1lICp0cmFwX2ZyYW1lKSwgaXMgbm90DQo+Z2V0dGluZyBjYWxsZWQgZm9y IHRoZSBDUFUgMS4NCj4+IEl0IGlzIG9ubHkgZ2V0dGluZyBjYWxsZWQgZm9yIENQVSAwIGFsbCB0 aGUgdGltZSBpbiBBUk02NCBidXQgaW4geDg2DQo+PiBpdCBpcyBnZXR0aW5nIGNhbGxlZCBmb3Ig Ym90aCBDUFUxIGFuZCBDUFUwLg0KPg0KPkludGVyZXN0aW5nISBJIGRvIHNlZSBvbmUgcHJvYmxl bSB3aXRoIHRoZSBwYXRjaCAoYW5kIHNvbWUgY29zbWV0aWMNCj5pc3N1ZXMpOiB3ZSByZWFsbHkg bmVlZCB0byB0YWtlIHRoZSBnaWNfbXR4IGluIGdpY192M19zZXR1cF9wZXJpcGgoKSByaWdodCB1 cCBmcm9udA0KPmJlY2F1c2UgQ1BVX1NFVCgpIHdvbid0IG5lY2Vzc2FyaWx5IGJlIGF0b21pYy4g VGhhdCdzIG5vdCB0aGUgcHJvYmxlbSwgdGhvdWdoIGZvcg0KPm90aGVyIHJlYXNvbnMsIGJ1dCBh bHNvIGJlY2F1c2UuLi4NCj4NCj4+IEZyb20gRERCIEkgaGF2ZSBjb2xsZWN0ZWQgdGhpcyBkYXRh IGluIGFybTY0LiBpcnExOCBpcyBmb3Igdm1idXMuDQo+PiBkYj4gc2hvdyBpcnFzDQo+PiAuLi4N Cj4+IGlycTE4ICA8Z2ljMCxwMj46IGNwdSAwMyBjbnQgNDExDQo+PiAuLi4uDQo+Pg0KPg0KPlRo YXQgd291bGQgc2VlbSB0byBpbmRpY2F0ZSB0aGF0IGJvdGggQ1BVcyBoYXZlIHNldCBpdCB1cCwg YnV0IGl0IG9jY3VycyB0byBtZSB0aGF0DQo+ZW5hYmxlX2ludHIgYWxzbyBuZWVkcyB0aGUgc2Ft ZSB0cmVhdG1lbnQuIExldCdzIHdpcGUgZ2ljX3YzLmMgYmFjayB0byBhIGNsZWFuIHNsYXRlDQo+ YW5kIHRyeSBhIHYyIG9mIHRoZSBwYXRjaDoNCj5odHRwczovL3Blb3BsZS5mci8NCj5lZWJzZC5v cmclMkZ+a2V2YW5zJTJGcHBpLQ0KPnYyLmRpZmYmZGF0YT0wNSU3QzAxJTdDc2NoYWtyYWJhcnRp JTQwbWljcm9zb2Z0LmNvbSU3QzM5ZDdjNTA4NDliNjRlYQ0KPmY3MGI2MDhkYjU1MDIwZmFlJTdD NzJmOTg4YmY4NmYxNDFhZjkxYWIyZDdjZDAxMWRiNDclN0MxJTdDMCU3QzYNCj4zODE5NzIzOTYx NTU0NDExMyU3Q1Vua25vd24lN0NUV0ZwYkdac2IzZDhleUpXSWpvaU1DNHdMakF3TURBaQ0KPkxD SlFJam9pVjJsdU16SWlMQ0pCVGlJNklrMWhhV3dpTENKWFZDSTZNbjAlM0QlN0MzMDAwJTdDJTdD JTdDJnNkYXQNCj5hPVFMRG04WDF5dXVHT3kyT2VlWXhzbkYlMkZOZXVjZUFvZ3c1MTYxVTdaSDYl MkJzJTNEJnJlc2VydmVkPTANCj4NCj5Gb3Igbm93IHdlIGp1c3QgcHJldGVuZCB0aGF0IHdlIHdv bid0IGJlIGRpc2FibGluZyBhbnkgUFBJcyBhcyBhIHByb29mLW9mLWNvbmNlcHQuDQo+DQpbU291 cmFkZWVwXQ0KVGhpcyBpcyBjYXVzaW5nIHBhbmljIGR1cmluZyBib290Og0KcG11MDogTUFEVDog Y3B1IDAgKG1waWRyIDApIGlycSAwIGxldmVsLXRyaWdnZXJlZA0KcG11MDogTUFEVDogY3B1IDEg KG1waWRyIDEpIGlycSAwIGxldmVsLXRyaWdnZXJlZA0KcGFuaWM6IGdpY192M19lbmFibGVfaW50 cl9pbXBsOiBVbnN1cHBvcnRlZCBJUlEgMA0KY3B1aWQgPSAwDQp0aW1lID0gMQ0KS0RCOiBzdGFj ayBiYWNrdHJhY2U6DQpkYl90cmFjZV9zZWxmKCkgYXQgZGJfdHJhY2Vfc2VsZg0KZGJfdHJhY2Vf c2VsZl93cmFwcGVyKCkgYXQgZGJfdHJhY2Vfc2VsZl93cmFwcGVyKzB4MzANCnZwYW5pYygpIGF0 IHZwYW5pYysweDEzYw0KcGFuaWMoKSBhdCBwYW5pYysweDQ0DQpnaWNfdjNfZW5hYmxlX2ludHJf aW1wbCgpIGF0IGdpY192M19lbmFibGVfaW50cl9pbXBsKzB4ZWMNCmludHJfc2V0dXBfaXJxKCkg YXQgaW50cl9zZXR1cF9pcnErMHgzNjgNCmJ1c19zZXR1cF9pbnRyKCkgYXQgYnVzX3NldHVwX2lu dHIrMHg5NA0KcG11X2F0dGFjaCgpIGF0IHBtdV9hdHRhY2grMHg2NA0KcG11X2FjcGlfYXR0YWNo KCkgYXQgcG11X2FjcGlfYXR0YWNoKzB4OTQNCmRldmljZV9hdHRhY2goKSBhdCBkZXZpY2VfYXR0 YWNoKzB4M2Y4DQpkZXZpY2VfcHJvYmVfYW5kX2F0dGFjaCgpIGF0IGRldmljZV9wcm9iZV9hbmRf YXR0YWNoKzB4N2MNCmJ1c19nZW5lcmljX25ld19wYXNzKCkgYXQgYnVzX2dlbmVyaWNfbmV3X3Bh c3MrMHhmYw0KYnVzX2dlbmVyaWNfbmV3X3Bhc3MoKSBhdCBidXNfZ2VuZXJpY19uZXdfcGFzcysw eGFjDQpidXNfZ2VuZXJpY19uZXdfcGFzcygpIGF0IGJ1c19nZW5lcmljX25ld19wYXNzKzB4YWMN CmJ1c19zZXRfcGFzcygpIGF0IGJ1c19zZXRfcGFzcysweDRjDQptaV9zdGFydHVwKCkgYXQgbWlf c3RhcnR1cCsweDFmYw0KdmlydGRvbmUoKSBhdCB2aXJ0ZG9uZSsweDcwDQpLREI6IGVudGVyOiBw YW5pYw0KWyB0aHJlYWQgcGlkIDAgdGlkIDEwMDAwMCBdDQpTdG9wcGVkIGF0ICAgICAga2RiX2Vu dGVyKzB4NDQ6IHN0ciAgICAgeHpyLCBbeDE5LCAjMzMyOF0NCkRldGFpbHMgb2YgdGhlIGxvZyBJ IGhhdmUgcGFzdGVkIGhlcmU6IGh0dHBzOi8vcGFzdGViaW4uY29tL2VRTjVSbnREDQo+VGhhbmtz LA0KPg0KPkt5bGUgRXZhbnMNCg== From nobody Mon May 15 13:54:06 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QKgpK5s3Rz4Bdj9 for ; Mon, 15 May 2023 13:54:17 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QKgpK5Jy1z4Xqm for ; Mon, 15 May 2023 13:54:17 +0000 (UTC) (envelope-from kevans@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684158857; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CO0p/mYi5U7MIErLO8ZnwiFexF2SzApxXRtoLNpWcdw=; b=pOMwYKJZIDHj1sgAdA6yfWfTgckMxucbJ+87KU87v87lEOzjJuycFaOxRuW0i8o+u/F0Ku /Jo2yk/t5kr8+O66xfZEZGMrLoXJCxOwDnFkr6MLSEaC6w+O8qAF7O3vNKkTgA4BipSIAF MM868Qb0UyIweVFWcqAifxnuUo719P09yemPApauOb4lX6juuvxNiM0FvN+/uajTytBDKl XRHMDT1KFFVDUTewmIhM3xO93MeylWn5J7MmezQ3L0V0sGV7X6r2bg5wb2Il+j8gFFvC/1 fuXpQ6Nb0DSi7H5CLf1j6ihj3QIId04Ud/r1DXi76eoYuzmXNfVX64/M7Kqxfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684158857; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CO0p/mYi5U7MIErLO8ZnwiFexF2SzApxXRtoLNpWcdw=; b=PashcELtO2yyI3wgHl/4nS0SsfvZS/q4K81il9OrEmKAiiJ+92jjmvCyCSh/lojj6Y4DtU QWI0R+51k1GhTGq59A6OHwOAHDHVa7TEui4gE10eyRIwyUFGAowb3xYhXLBq5xXY4N5eCs TxaS517+bhy6DwmB7YKxw4hX3XLyo5f9hrU9oKHCldeuRrDaNcOOGCAL9LqAUTcmomFCqo YOfidDPRCDHCvavl/bUjdtHJBWUXXe/Xn35yjsLMlWdFn2RuK3vIfOuiIIjSqqBSx9tPyz QBbNPUI/HfUet7hqKAPqKqE1q0DoHURJ6qGnalCm/qOMKFQwF+OmSxU669qB4g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1684158857; a=rsa-sha256; cv=none; b=BER3QllgTrXA6t38Wo6slZi8bO1epPDJxuFzbJblRXLErUsk7CTJCuLJMvGudyVBbzOWNn mtFizuU+Md20J8KTtSWtQKGlKAanXagAlyqO0YiqNDXx/W0r+IPDpQ2ABQBlqI9+Otlk9K VACMMAy0UdzudnLelNk+4f3k3ZR1H7m0i1ln+emFaSldkjNmaP7pt4l0PmiA/qlcXHxqBA 6yNOYB4gUKVioFNlQnI5Hpr7HRyBnh3L6XFqox2m0ge7MPcOUvo07lxyeMi7dXBms7iYo7 avsnjNqByBlESsGTpKN7JJcrlfhS5SNmWfTZaGf58eU06QnAnep05x6fKYo10A== Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 4QKgpK4Ff9z10SL for ; Mon, 15 May 2023 13:54:17 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-759413d99afso414714285a.1 for ; Mon, 15 May 2023 06:54:17 -0700 (PDT) X-Gm-Message-State: AC+VfDyuLoCKjMKDb3nhbgEOliQb99hFOPp7NJTo4e7bi1tpYkUSY6+d /RqbmnNlN1gvXQqrXNcOy7eTWHVjhrLD4Mj99fE= X-Google-Smtp-Source: ACHHUZ78W0pq2XTuLsCBdhV0BiUarTzMRRswBhIHbNCI7+26NdFZsUF80C5Bt/J7KlWdAgAZ3Uywp433AIcKkxfmXgI= X-Received: by 2002:ad4:5bab:0:b0:621:5823:cf1c with SMTP id 11-20020ad45bab000000b006215823cf1cmr25069762qvq.14.1684158857193; Mon, 15 May 2023 06:54:17 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Kyle Evans Date: Mon, 15 May 2023 08:54:06 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 SMP To: Souradeep Chakrabarti Cc: Wei Hu , "freebsd-hackers@FreeBSD.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-ThisMailContainsUnwantedMimeParts: N On Mon, May 15, 2023 at 2:41=E2=80=AFAM Souradeep Chakrabarti wrote: > > > > > >-----Original Message----- > >From: Kyle Evans > >Sent: Monday, May 15, 2023 10:36 AM > >To: Souradeep Chakrabarti > >Cc: Wei Hu ; freebsd-hackers@FreeBSD.org > >Subject: Re: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in AR= M64 SMP > > > >On Sun, May 14, 2023 at 9:59=E2=80=AFAM Souradeep Chakrabarti > > wrote: > >> > >> > >> > >> > >> >-----Original Message----- > >> >From: Kyle Evans > >> >Sent: Friday, May 12, 2023 10:40 PM > >> >To: Souradeep Chakrabarti > >> >Cc: Wei Hu ; freebsd-hackers@FreeBSD.org > >> >Subject: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in > >> >ARM64 SMP > >> > > >> >On Fri, May 12, 2023 at 9:51=E2=80=AFAM Souradeep Chakrabarti > >> > wrote: > >> >> > >> >> > >> >> > >> >> > >> >> >-----Original Message----- > >> >> >From: Souradeep Chakrabarti > >> >> >Sent: Monday, May 8, 2023 6:39 PM > >> >> >To: Kyle Evans > >> >> >Cc: Wei Hu ; freebsd-hackers@FreeBSD.org > >> >> >Subject: enabling same PPI interrupt to all CPU in ARM64 SMP > >> >> > > >> >> >Hi , > >> >> > > >> >> >While using SMP in ARM64 Hyper-V we are getting stuck in boot if > >> >> >there is a interrupt for VMBus coming to CPU1 and VMBus interrupt > >> >> >handler is not getting that interrupt. > >> >> > > >> >> >In ARM64 Hyper-V we are using IRQ18 for VMBus and it is a PPI inte= rrupt. > >> >> > > >> >> >But Hypev-V host sends interrupt to this IRQ 18 for both CPU0 and > >> >> >CPU1 in 2CPU system. > >> >> >This is based on the corresponding VMBus channel which assigned wi= th the > >CPU. > >> >> > > >> >> >Now VMBus ISR is getting the interrupt in CPU0 but not getting fro= m CPU1. > >> >> >Any idea, how we can use the same PPI 18 for all the CPU cores? > >> >> > > >> >> >Any help will be appreciated, as this is blocking the enablement > >> >> >of FreeBSD in Azure ARM64. > >> >> [Souradeep] > >> >> Can someone please help me it. > >> >> > >> > > >> >Looking at least at the GIC implementation, it looks like this is a k= nown limitation: > >> > > >> > 875 /* > >> > 876 * XXX - In case that per CPU interrupt is going to be > >> >enabled in time > >> > 877 * when SMP is already started, we need some IPI > >> >call which > >> > 878 * enables it on others CPUs. Further, it's more > >> >complicated as > >> > 879 * pic_enable_source() and pic_disable_source() > >> >should act on > >> > 880 * per CPU basis only. Thus, it should be solved > >> >here somehow. > >> > 881 */ > >> > 882 if (isrc->isrc_flags & INTR_ISRCF_PPI) > >> > 883 CPU_SET(PCPU_GET(cpuid), &isrc->isrc_cpu); > >> > > >> >I think we need something /like/ this: > >> >https://peo/ > >> > >>ple.fr%2F&data=3D05%7C01%7Cschakrabarti%40microsoft.com%7C39d7c50849b > >64 > >> > >>eaf70b608db55020fae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0% > >7C6381 > >> > >>97239615544113%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC > >JQIjoiV2 > >> > >>luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3DdXVvnI > >HhN9 > >> >mdwiPkSJKwMyEKYi5SyGOuta5zCZ1ysCQ%3D&reserved=3D0 > >> > >>eebsd.org%2F~kevans%2Fppi.diff&data=3D05%7C01%7Cschakrabarti%40microsof > >> >t.c > >om%7Cc5c3d254b9d841e9ae9b08db530bb3d2%7C72f988bf86f141af91ab2d7c > >> > >>d011db47%7C1%7C0%7C638195082027744706%7CUnknown%7CTWFpbGZsb > >3 > >> > >>d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D > >> > >>%7C3000%7C%7C%7C&sdata=3DSwoR2vHxh6QQhwpOgFcQ9378nDVovhdEKEXoE > >o > >> >gPKsc%3D&reserved=3D0, though it still has the caveat that PPIs > >> >effectively cannot be fully setup before SI_SUB_SMP. > >> >So, it's likely almost a NOP for existing platforms (will emit a > >> >warning with bootverbose for armv8 timers) but might do the trick for= you. > >> [Souradeep] > >> Thanks for the change but it did not solve the problem. Still the > >> interrupt handler vmbus_handle_intr(struct trapframe *trap_frame), is = not > >getting called for the CPU 1. > >> It is only getting called for CPU 0 all the time in ARM64 but in x86 > >> it is getting called for both CPU1 and CPU0. > > > >Interesting! I do see one problem with the patch (and some cosmetic > >issues): we really need to take the gic_mtx in gic_v3_setup_periph() rig= ht up front > >because CPU_SET() won't necessarily be atomic. That's not the problem, t= hough for > >other reasons, but also because... > > > >> From DDB I have collected this data in arm64. irq18 is for vmbus. > >> db> show irqs > >> ... > >> irq18 : cpu 03 cnt 411 > >> .... > >> > > > >That would seem to indicate that both CPUs have set it up, but it occurs= to me that > >enable_intr also needs the same treatment. Let's wipe gic_v3.c back to a= clean slate > >and try a v2 of the patch: > >https://people.fr/ > >eebsd.org%2F~kevans%2Fppi- > >v2.diff&data=3D05%7C01%7Cschakrabarti%40microsoft.com%7C39d7c50849b64ea > >f70b608db55020fae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6 > >38197239615544113%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAi > >LCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdat > >a=3DQLDm8X1yuuGOy2OeeYxsnF%2FNeuceAogw5161U7ZH6%2Bs%3D&reserved=3D0 > > > >For now we just pretend that we won't be disabling any PPIs as a proof-o= f-concept. > > > [Souradeep] > This is causing panic during boot: > pmu0: MADT: cpu 0 (mpidr 0) irq 0 level-triggered > pmu0: MADT: cpu 1 (mpidr 1) irq 0 level-triggered > panic: gic_v3_enable_intr_impl: Unsupported IRQ 0 > cpuid =3D 0 > time =3D 1 > KDB: stack backtrace: > db_trace_self() at db_trace_self > db_trace_self_wrapper() at db_trace_self_wrapper+0x30 > vpanic() at vpanic+0x13c > panic() at panic+0x44 > gic_v3_enable_intr_impl() at gic_v3_enable_intr_impl+0xec > intr_setup_irq() at intr_setup_irq+0x368 > bus_setup_intr() at bus_setup_intr+0x94 > pmu_attach() at pmu_attach+0x64 > pmu_acpi_attach() at pmu_acpi_attach+0x94 > device_attach() at device_attach+0x3f8 > device_probe_and_attach() at device_probe_and_attach+0x7c > bus_generic_new_pass() at bus_generic_new_pass+0xfc > bus_generic_new_pass() at bus_generic_new_pass+0xac > bus_generic_new_pass() at bus_generic_new_pass+0xac > bus_set_pass() at bus_set_pass+0x4c > mi_startup() at mi_startup+0x1fc > virtdone() at virtdone+0x70 > KDB: enter: panic > [ thread pid 0 tid 100000 ] > Stopped at kdb_enter+0x44: str xzr, [x19, #3328] > Details of the log I have pasted here: https://pastebin.com/eQN5RntD I think you fetched a broken version of the patch -- there was maybe an hour where I kept replacing it because I found a problem and fixed it, but created another one to fix that I discovered on some hardware I tested on. I've re-uploaded the correct diff at https://people.freebsd.org/~kevans/ppi-v3.diff, which can't hit the panic because irq <=3D GIC_LAST_PPI all return early (the version you have, the first branch in gic_v3_enable_intr_impl was probably an incorrect `if (isrc->isrc_flags & INTR_ISRCF_PPI)` Thanks, Kyle Evans From nobody Mon May 15 17:06:48 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QKm4j0VFYz4Bqc4 for ; Mon, 15 May 2023 17:07:01 +0000 (UTC) (envelope-from schakrabarti@microsoft.com) Received: from APC01-TYZ-obe.outbound.protection.outlook.com (mail-tyzapc01on2126.outbound.protection.outlook.com [40.107.117.126]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QKm4h1xkgz3l0R; Mon, 15 May 2023 17:07:00 +0000 (UTC) (envelope-from schakrabarti@microsoft.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hbQ28ZFm3YcAvt7RNs6/Cqe2NA4dRV6o1WIrU+/NDUhM5k/fzFMoDp2LIWNq1R8MQFIKWl9DO70sbDAUOuXtr9MNLmqkVG6F8887+ZQGlIFDcpDRwUvUsI67M3+O3DMlD5C1sp9Pw0SDezSFbBI0AfhiXngrVG1+wBdmPP44j92OpApp3W+4l1oGFg69eQd80YDJjSnBdckn+XRbgy+u+/MtDPUZkjJxAfQKhAPeCoSHWO8tj69S8/br3hpkgnAZePNvEeUkU+p9nVPkRdFlIyPg8mokrz7LFGmZAHYPFpp4wt7AJ3E6mLzBqAup/MZNtMxYNSRN3LuXnqWV1A6Upw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ni5Fv/8NIAiBU55I3q6kZ3Pqob4ac8D3xFWJwk8HMgs=; b=YqeT3AIdnHXUjQdC2+7pDPRJ8Eh4U35IKfEb6HhLEC5tO5rZsaV43OH1VOf/03sVzWTr9FzBfJeS+ujvbyc6XFWi4+itnx2rDgUOH/c8bzOOtJM+JNjRbqNTihoG2s1siR8tvFXYN2xr6ki/mx10t9UGjM2qD4CKYwjpT6A7rjA2mT5ugX/iC53fIdElkoORfH1FqqLqqZWgHJ9FxrsELq2R/TuLbBR+yRXut44ahVgIRGbZxACFxUy/HY/qsTp6VrLzmr6tpqZlC1QnOXBwGZqIWVokRm4niAC1j5jxINuz1Kli3tfgUuV5tLImBF4yICJnk/RYnZYkKrbT/VTUFA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ni5Fv/8NIAiBU55I3q6kZ3Pqob4ac8D3xFWJwk8HMgs=; b=A8aATnI7nBaoMWA5xIUeQEVrKn90rec2emtuAlFbDT+sfUe1x4a20hcpoJHLDQ2BonNT0cLNPs6vjEsTINEKMA5++OeQ2lelH7o0cdWVJI+1YhtDMjfhFaaSh4MX1kF+MItS7+SKt11ttK+UIBA1v87Vx4xAnnrM1GGjKeD/Qyg= Received: from PSAP153MB0536.APCP153.PROD.OUTLOOK.COM (2603:1096:301:75::14) by PSBP153MB0373.APCP153.PROD.OUTLOOK.COM (2603:1096:301:b::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6433.4; Mon, 15 May 2023 17:06:53 +0000 Received: from PSAP153MB0536.APCP153.PROD.OUTLOOK.COM ([fe80::3636:5810:fb12:6d71]) by PSAP153MB0536.APCP153.PROD.OUTLOOK.COM ([fe80::3636:5810:fb12:6d71%7]) with mapi id 15.20.6433.001; Mon, 15 May 2023 17:06:52 +0000 From: Souradeep Chakrabarti To: Kyle Evans CC: Wei Hu , "freebsd-hackers@FreeBSD.org" Subject: RE: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 SMP Thread-Topic: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 SMP Thread-Index: AdmBrWHu4SCK2cRnRnyy6MBGajO35QDM7LrAAATdMgAAXrUlgAAe4ZoAAAUxoMAADULwAAAGs2kg Date: Mon, 15 May 2023 17:06:48 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=579a9a93-7e36-413a-9714-245856ac89e3;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-05-15T17:05:57Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PSAP153MB0536:EE_|PSBP153MB0373:EE_ x-ms-office365-filtering-correlation-id: 3fa18755-299b-45ce-b1d3-08db5566c560 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: fz3Cykqit9P4ZsFbnhMIsLIavY0/jDAQlDTW38yuWNzU+waBVYa80+j3nfdE753eI+AiuMTK+KtCYp859bNQlOAZc5Rnnw+Zrja/Qn35e8PDufsz6KuUc7fp5kXJhAFHvMETQ9V2n6MopsxhP0/3qU1v9x5v5IK3T3QuR6a8DVveN7UmzfJgZVEQYRNl9PVs3YjCx3MufVdbwI5BI4DjiAdEp5vFsd3LWzSPApA7/AY9gHKkpHKdZ2RcrFx1JkPUr+1FlJDgqlHoxAbX0kbslADIGyQLMrMSes5n//OLx7mmeAOPxzv0OAyRPfg9bhGp+3OKnPrzvo852UHsosNaCgl4fCzPHY5s0LlIwTbjis9W043O63LkdUznTouXFEve/gHkc+RbwmfIX1EXExHId8B71+3LIFZ6nOzF3VJEFWQWzGsN9ILTnqBv5XJi5N8wTB0cllaazZywJVfpFvKr6G2kzuAhzb93R0sdnfPi5CODKHB86GAwO1e0kT2oRyL/gYf1Y+a3NdE72hX+SBdC4jxZZbeLAYHshnWbgg3BnqbP0KxwcbKkJGcmZJrej5Z2y81KVkznOTVGoiRXhxt+lM62nYTeG4pGo9xMb+JpSygIdodCTTUxsLpt1eF5eatGMfC0xTqgDrIBDcl1qRHI3HpiUGpb4uaEwtiZUlrW0hMqqI9zQGXZ+6dzea+XaA6b x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PSAP153MB0536.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230028)(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(451199021)(10290500003)(6666004)(8676002)(71200400001)(478600001)(41300700001)(55016003)(4326008)(450100002)(122000001)(786003)(82950400001)(66946007)(6916009)(66556008)(82960400001)(76116006)(66446008)(316002)(64756008)(66476007)(54906003)(8936002)(38100700002)(186003)(8990500004)(9686003)(38070700005)(6506007)(2906002)(26005)(83380400001)(33656002)(966005)(86362001)(7696005)(5660300002)(52536014)(21314003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?dVlpQ0EranlOamlaZ2F1QmlDTHBJNGJkNktHL0RIVzhoS1g2cjVmb2ovZng2?= =?utf-8?B?NEN4RDJHWW9OUStGMjFOQlpkWnF3ZUNOcGRZb1BhMDJzVUxIcEtjcDM4TUdo?= =?utf-8?B?Y3pJWGFBSXRYNzd4UGV3eGFQUlFJL0ZYMmlqVFR5SmxRclk0YzlqUWl2QU1z?= =?utf-8?B?SmFNQW9KZ0JrcXp5UitiUHp0amtZTU1aR0I5ckxPWXlvVnVNUDZEcnpueXpX?= =?utf-8?B?MlB3Z0ZEbzVpMFM5a1V5S2Z4bTVGbE1zK0xndEN4U2I1VkRsNnVXeXRBMUhV?= =?utf-8?B?Y1ZXOC8wcUNzNzBCK0M0UFFpUkNaTXlVMWMrR083ZFRId1BqaTV3ekk4WFZp?= =?utf-8?B?WVZXYm1tSFhEUEZ0TXZHMzBKUXhnUGlVNmxJdzZDV0JtQVlUNHN1L2lHTFBl?= =?utf-8?B?SU84dWVCWTZkdThDOFlCQ2lNaGUzRkhsdWhQU2tBWGpTdHpsbTdvVDd3VUFG?= =?utf-8?B?OFJRakdyUnZCTFhEU0JUOFJneStQQWVQVWJvczBRNDBiRWUxZjhSdlV0UkJu?= =?utf-8?B?OXJsTTNMcklEMi9ucW1QUDdsanF5NmU2MjRNdkdqU1ErS3ZQRm9uejRxT2FU?= =?utf-8?B?Wk8rZnBkVVEvM2NoSXN1dVYwZk51dWdpenQxVG9MR1UxQjcxS20ydENOUHBp?= =?utf-8?B?UWhYQlNzNkw5ZGVpcVd6aFJvWEFmelNXVkhETW5TRTMzUDlZc1BRcFNFM3oz?= =?utf-8?B?ZTVZMVlieUEyWHBjVVYyYWZHdlVTLzV6cjBnV2RTaVAvZVhNcDYrczdLS1Rs?= =?utf-8?B?TUh2TE5vZVo1dG1Ob3haVzlCc09yd3BwaXJQYzY2bDVUckR6NERxa0Y2S3dS?= =?utf-8?B?VDk2bXF6dGh6QzZzY2xFcjFMbHpmR25oTXQrMTBjeFhKNjYvaytDWE51Mkcz?= =?utf-8?B?dGRaemdEUFFzcUdyUjdJeWJtTFlTYkp6UlNrUXR5MlhwS3ZNdmNwWEhhVnVz?= =?utf-8?B?bDBpWlhEdnhpUXgraWg4YnUrbEtOODZWbHcyRzRsMHM4KzlHSWVsakF0TEY1?= =?utf-8?B?eTJ4bXo4bnpBazh4WksyblpYY3dQeHhjQVBDcloveDVPL2d0L0IzVG4yTGFC?= =?utf-8?B?VWtXSUIzazFLM1g1Qi95Q3I0aDFidzdzOGVFY0hnNlpQK0dJK0FCVCtvdlh6?= =?utf-8?B?akYySndBeHFmM1p6dmhsQ2RZL0lSMStxUlE3RGdBakdoeXVURXd2bGY1Si9n?= =?utf-8?B?c2JXbGljbmF0RjNrbTUvZzk1bWFONjRQY3B0WjNndUhLSncybTN1WVdDVXVp?= =?utf-8?B?eVVSOGhIK0puZUlQenA0RlNIa1YwWkY4MEFKbG03ZHpSZU5OY3dZdDlGdkRP?= =?utf-8?B?T1I0dFpqcVFYa1hOcjJkYklySHJSNnBlOVl2dXFFVkU2QnFPTEVXY3ZMc3NM?= =?utf-8?B?aTB4b0pGWG1iRUZNVmpwNU4zUWFtODdDQkJIWGhmcVluWm14OUd6S1FqSU5X?= =?utf-8?B?T3hxNjF3SEhjeFdqSW1NbVRqL0tyOWljRlJiZ1RWaWpKN1cxZ0RvZkltOEFw?= =?utf-8?B?di9CYVFBeVIyVVBFMlpnU01QcURiTEtXdkkrR2YyREVXV2F6VGVJaEpWR0o0?= =?utf-8?B?VFN1T3prV1A0TEZEVGg4cWk5S091WFJ1b214SmxQSE40QjdXaW1NV20rNm1l?= =?utf-8?B?SEtKRjBpdVduaTVmeDRlQ2tHRHZRbHFaSUF5MTYzMS91Tk5vUFpvTk94OWo1?= =?utf-8?B?SjRXTXlSaDBkek5GQ3l0UkxENExXekhRbUs1WVRwWmpJaDUxcnA5MUlUVjBt?= =?utf-8?B?MTRUWnJ0NVhRQXU3bHdQWFlleTRxMGdaaE5DWHB5TzUwVWt0VW9VRWVxNDVT?= =?utf-8?B?TEFtUisyZkxtUU5hZHNURkpKb051dDhzZVZUWHVOZ2YyK1FUdkJVRFlXNlh0?= =?utf-8?B?aUtjeUF2SmZ4SnJkNU5wRjcxbC9XQkRXaklFS3hSbDUrM1R3YmxkYkdjYnFr?= =?utf-8?B?ZklSMHlNQVN5RlFDQTYyNWpLQlJQSnVnTFU3RU93VXh4QVdMOURXM2ZCZHps?= =?utf-8?B?NFJDMVhpa1FVN08zWnp2ck9DOU5GUWJXbkhYYmFReUdyV1VsbDNRRnBEZ3BE?= =?utf-8?Q?rNDbjn?= Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PSAP153MB0536.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 3fa18755-299b-45ce-b1d3-08db5566c560 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 May 2023 17:06:48.9277 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: R3FtjE3YI6tRz5n5uzFZWfEwMmORk7vBw/4OOdfM46R/RhU8UKcl9d4jil8gOTCPFxiaH73bOV1quJRO0u6Uv+2wz1De8Jf9ruLNmWuaGD4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PSBP153MB0373 X-Rspamd-Queue-Id: 4QKm4h1xkgz3l0R X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N DQoNCg0KPi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+RnJvbTogS3lsZSBFdmFucyA8a2V2 YW5zQGZyZWVic2Qub3JnPg0KPlNlbnQ6IE1vbmRheSwgTWF5IDE1LCAyMDIzIDc6MjQgUE0NCj5U bzogU291cmFkZWVwIENoYWtyYWJhcnRpIDxzY2hha3JhYmFydGlAbWljcm9zb2Z0LmNvbT4NCj5D YzogV2VpIEh1IDx3ZWhAbWljcm9zb2Z0LmNvbT47IGZyZWVic2QtaGFja2Vyc0BGcmVlQlNELm9y Zw0KPlN1YmplY3Q6IFJlOiBbRVhURVJOQUxdIFJlOiBlbmFibGluZyBzYW1lIFBQSSBpbnRlcnJ1 cHQgdG8gYWxsIENQVSBpbiBBUk02NCBTTVANCj4NCj5PbiBNb24sIE1heSAxNSwgMjAyMyBhdCAy OjQx4oCvQU0gU291cmFkZWVwIENoYWtyYWJhcnRpDQo+PHNjaGFrcmFiYXJ0aUBtaWNyb3NvZnQu Y29tPiB3cm90ZToNCj4+DQo+Pg0KPj4NCj4+DQo+PiA+LS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0t LS0NCj4+ID5Gcm9tOiBLeWxlIEV2YW5zIDxrZXZhbnNAZnJlZWJzZC5vcmc+DQo+PiA+U2VudDog TW9uZGF5LCBNYXkgMTUsIDIwMjMgMTA6MzYgQU0NCj4+ID5UbzogU291cmFkZWVwIENoYWtyYWJh cnRpIDxzY2hha3JhYmFydGlAbWljcm9zb2Z0LmNvbT4NCj4+ID5DYzogV2VpIEh1IDx3ZWhAbWlj cm9zb2Z0LmNvbT47IGZyZWVic2QtaGFja2Vyc0BGcmVlQlNELm9yZw0KPj4gPlN1YmplY3Q6IFJl OiBbRVhURVJOQUxdIFJlOiBlbmFibGluZyBzYW1lIFBQSSBpbnRlcnJ1cHQgdG8gYWxsIENQVSBp bg0KPj4gPkFSTTY0IFNNUA0KPj4gPg0KPj4gPk9uIFN1biwgTWF5IDE0LCAyMDIzIGF0IDk6NTni gK9BTSBTb3VyYWRlZXAgQ2hha3JhYmFydGkNCj4+ID48c2NoYWtyYWJhcnRpQG1pY3Jvc29mdC5j b20+IHdyb3RlOg0KPj4gPj4NCj4+ID4+DQo+PiA+Pg0KPj4gPj4NCj4+ID4+ID4tLS0tLU9yaWdp bmFsIE1lc3NhZ2UtLS0tLQ0KPj4gPj4gPkZyb206IEt5bGUgRXZhbnMgPGtldmFuc0BmcmVlYnNk Lm9yZz4NCj4+ID4+ID5TZW50OiBGcmlkYXksIE1heSAxMiwgMjAyMyAxMDo0MCBQTQ0KPj4gPj4g PlRvOiBTb3VyYWRlZXAgQ2hha3JhYmFydGkgPHNjaGFrcmFiYXJ0aUBtaWNyb3NvZnQuY29tPg0K Pj4gPj4gPkNjOiBXZWkgSHUgPHdlaEBtaWNyb3NvZnQuY29tPjsgZnJlZWJzZC1oYWNrZXJzQEZy ZWVCU0Qub3JnDQo+PiA+PiA+U3ViamVjdDogW0VYVEVSTkFMXSBSZTogZW5hYmxpbmcgc2FtZSBQ UEkgaW50ZXJydXB0IHRvIGFsbCBDUFUgaW4NCj4+ID4+ID5BUk02NCBTTVANCj4+ID4+ID4NCj4+ ID4+ID5PbiBGcmksIE1heSAxMiwgMjAyMyBhdCA5OjUx4oCvQU0gU291cmFkZWVwIENoYWtyYWJh cnRpDQo+PiA+PiA+PHNjaGFrcmFiYXJ0aUBtaWNyb3NvZnQuY29tPiB3cm90ZToNCj4+ID4+ID4+ DQo+PiA+PiA+Pg0KPj4gPj4gPj4NCj4+ID4+ID4+DQo+PiA+PiA+PiA+LS0tLS1PcmlnaW5hbCBN ZXNzYWdlLS0tLS0NCj4+ID4+ID4+ID5Gcm9tOiBTb3VyYWRlZXAgQ2hha3JhYmFydGkNCj4+ID4+ ID4+ID5TZW50OiBNb25kYXksIE1heSA4LCAyMDIzIDY6MzkgUE0NCj4+ID4+ID4+ID5UbzogS3ls ZSBFdmFucyA8a2V2YW5zQGZyZWVic2Qub3JnPg0KPj4gPj4gPj4gPkNjOiBXZWkgSHUgPHdlaEBt aWNyb3NvZnQuY29tPjsgZnJlZWJzZC1oYWNrZXJzQEZyZWVCU0Qub3JnDQo+PiA+PiA+PiA+U3Vi amVjdDogZW5hYmxpbmcgc2FtZSBQUEkgaW50ZXJydXB0IHRvIGFsbCBDUFUgaW4gQVJNNjQgU01Q DQo+PiA+PiA+PiA+DQo+PiA+PiA+PiA+SGkgLA0KPj4gPj4gPj4gPg0KPj4gPj4gPj4gPldoaWxl IHVzaW5nIFNNUCBpbiBBUk02NCBIeXBlci1WIHdlIGFyZSBnZXR0aW5nIHN0dWNrIGluIGJvb3QN Cj4+ID4+ID4+ID5pZiB0aGVyZSBpcyBhIGludGVycnVwdCBmb3IgVk1CdXMgY29taW5nIHRvIENQ VTEgYW5kIFZNQnVzDQo+PiA+PiA+PiA+aW50ZXJydXB0IGhhbmRsZXIgaXMgbm90IGdldHRpbmcg dGhhdCBpbnRlcnJ1cHQuDQo+PiA+PiA+PiA+DQo+PiA+PiA+PiA+SW4gQVJNNjQgSHlwZXItViB3 ZSBhcmUgdXNpbmcgSVJRMTggZm9yIFZNQnVzIGFuZCBpdCBpcyBhIFBQSSBpbnRlcnJ1cHQuDQo+ PiA+PiA+PiA+DQo+PiA+PiA+PiA+QnV0IEh5cGV2LVYgaG9zdCBzZW5kcyBpbnRlcnJ1cHQgdG8g dGhpcyBJUlEgMTggZm9yIGJvdGggQ1BVMA0KPj4gPj4gPj4gPmFuZA0KPj4gPj4gPj4gPkNQVTEg aW4gMkNQVSBzeXN0ZW0uDQo+PiA+PiA+PiA+VGhpcyBpcyBiYXNlZCBvbiB0aGUgY29ycmVzcG9u ZGluZyBWTUJ1cyBjaGFubmVsIHdoaWNoIGFzc2lnbmVkDQo+PiA+PiA+PiA+d2l0aCB0aGUNCj4+ ID5DUFUuDQo+PiA+PiA+PiA+DQo+PiA+PiA+PiA+Tm93IFZNQnVzIElTUiBpcyBnZXR0aW5nIHRo ZSBpbnRlcnJ1cHQgaW4gQ1BVMCBidXQgbm90IGdldHRpbmcgZnJvbQ0KPkNQVTEuDQo+PiA+PiA+ PiA+QW55IGlkZWEsIGhvdyB3ZSBjYW4gdXNlIHRoZSBzYW1lIFBQSSAxOCBmb3IgYWxsIHRoZSBD UFUgY29yZXM/DQo+PiA+PiA+PiA+DQo+PiA+PiA+PiA+QW55IGhlbHAgd2lsbCBiZSBhcHByZWNp YXRlZCwgYXMgdGhpcyBpcyBibG9ja2luZyB0aGUNCj4+ID4+ID4+ID5lbmFibGVtZW50IG9mIEZy ZWVCU0QgaW4gQXp1cmUgQVJNNjQuDQo+PiA+PiA+PiBbU291cmFkZWVwXQ0KPj4gPj4gPj4gQ2Fu IHNvbWVvbmUgcGxlYXNlIGhlbHAgbWUgaXQuDQo+PiA+PiA+Pg0KPj4gPj4gPg0KPj4gPj4gPkxv b2tpbmcgYXQgbGVhc3QgYXQgdGhlIEdJQyBpbXBsZW1lbnRhdGlvbiwgaXQgbG9va3MgbGlrZSB0 aGlzIGlzIGEga25vd24NCj5saW1pdGF0aW9uOg0KPj4gPj4gPg0KPj4gPj4gPiA4NzUgICAgICAg ICAvKg0KPj4gPj4gPiA4NzYgICAgICAgICAgKiBYWFggLSBJbiBjYXNlIHRoYXQgcGVyIENQVSBp bnRlcnJ1cHQgaXMgZ29pbmcgdG8gYmUNCj4+ID4+ID5lbmFibGVkIGluIHRpbWUNCj4+ID4+ID4g ODc3ICAgICAgICAgICogICAgICAgd2hlbiBTTVAgaXMgYWxyZWFkeSBzdGFydGVkLCB3ZSBuZWVk IHNvbWUgSVBJDQo+PiA+PiA+Y2FsbCB3aGljaA0KPj4gPj4gPiA4NzggICAgICAgICAgKiAgICAg ICBlbmFibGVzIGl0IG9uIG90aGVycyBDUFVzLiBGdXJ0aGVyLCBpdCdzIG1vcmUNCj4+ID4+ID5j b21wbGljYXRlZCBhcw0KPj4gPj4gPiA4NzkgICAgICAgICAgKiAgICAgICBwaWNfZW5hYmxlX3Nv dXJjZSgpIGFuZCBwaWNfZGlzYWJsZV9zb3VyY2UoKQ0KPj4gPj4gPnNob3VsZCBhY3Qgb24NCj4+ ID4+ID4gODgwICAgICAgICAgICogICAgICAgcGVyIENQVSBiYXNpcyBvbmx5LiBUaHVzLCBpdCBz aG91bGQgYmUgc29sdmVkDQo+PiA+PiA+aGVyZSBzb21laG93Lg0KPj4gPj4gPiA4ODEgICAgICAg ICAgKi8NCj4+ID4+ID4gODgyICAgICAgICAgaWYgKGlzcmMtPmlzcmNfZmxhZ3MgJiBJTlRSX0lT UkNGX1BQSSkNCj4+ID4+ID4gODgzICAgICAgICAgICAgICAgICBDUFVfU0VUKFBDUFVfR0VUKGNw dWlkKSwgJmlzcmMtPmlzcmNfY3B1KTsNCj4+ID4+ID4NCj4+ID4+ID5JIHRoaW5rIHdlIG5lZWQg c29tZXRoaW5nIC9saWtlLyB0aGlzOg0KPj4gPj4gPmh0dHBzOi8vcGVvLw0KPj4gPj4NCj4+DQo+ Pj5wbGUuZnIlMkYmZGF0YT0wNSU3QzAxJTdDc2NoYWtyYWJhcnRpJTQwbWljcm9zb2Z0LmNvbSU3 QzM5ZDdjNTA4NDliDQo+PiA+NjQNCj4+ID4+DQo+Pg0KPj4+ZWFmNzBiNjA4ZGI1NTAyMGZhZSU3 QzcyZjk4OGJmODZmMTQxYWY5MWFiMmQ3Y2QwMTFkYjQ3JTdDMSU3QzANCj4lDQo+PiA+N0M2Mzgx DQo+PiA+Pg0KPj4NCj4+Pjk3MjM5NjE1NTQ0MTEzJTdDVW5rbm93biU3Q1RXRnBiR1pzYjNkOGV5 SldJam9pTUM0d0xqQXdNREFpTA0KPkMNCj4+ID5KUUlqb2lWMg0KPj4gPj4NCj4+DQo+Pj5sdU16 SWlMQ0pCVGlJNklrMWhhV3dpTENKWFZDSTZNbjAlM0QlN0MzMDAwJTdDJTdDJTdDJnNkYXRhPWRY VnYNCj5uSQ0KPj4gPkhoTjkNCj4+ID4+ID5tZHdpUGtTSkt3TXlFS1lpNVN5R091dGE1ekNaMXlz Q1ElM0QmcmVzZXJ2ZWQ9MA0KPj4gPj4NCj4+DQo+Pj5lZWJzZC5vcmclMkZ+a2V2YW5zJTJGcHBp LmRpZmYmZGF0YT0wNSU3QzAxJTdDc2NoYWtyYWJhcnRpJTQwbWljcm9zbw0KPj4gPj5mDQo+PiA+ PiA+dC5jDQo+Pg0KPj5vbSU3Q2M1YzNkMjU0YjlkODQxZTlhZTliMDhkYjUzMGJiM2QyJTdDNzJm OTg4YmY4NmYxNDFhZjkxYWIyZDcNCj5jDQo+PiA+Pg0KPj4NCj4+PmQwMTFkYjQ3JTdDMSU3QzAl N0M2MzgxOTUwODIwMjc3NDQ3MDYlN0NVbmtub3duJTdDVFdGcGJHWnMNCj5iDQo+PiA+Mw0KPj4g Pj4NCj4+DQo+Pj5kOGV5SldJam9pTUM0d0xqQXdNREFpTENKUUlqb2lWMmx1TXpJaUxDSkJUaUk2 SWsxaGFXd2lMQ0pYVkNJNk1uMCUzDQo+RA0KPj4gPj4NCj4+DQo+Pj4lN0MzMDAwJTdDJTdDJTdD JnNkYXRhPVN3b1Iydkh4aDZRUWh3cE9nRmNROTM3OG5EVm92aGRFS0VYbw0KPkUNCj4+ID5vDQo+ PiA+PiA+Z1BLc2MlM0QmcmVzZXJ2ZWQ9MCwgdGhvdWdoIGl0IHN0aWxsIGhhcyB0aGUgY2F2ZWF0 IHRoYXQgUFBJcw0KPj4gPj4gPmVmZmVjdGl2ZWx5IGNhbm5vdCBiZSBmdWxseSBzZXR1cCBiZWZv cmUgU0lfU1VCX1NNUC4NCj4+ID4+ID5TbywgaXQncyBsaWtlbHkgYWxtb3N0IGEgTk9QIGZvciBl eGlzdGluZyBwbGF0Zm9ybXMgKHdpbGwgZW1pdCBhDQo+PiA+PiA+d2FybmluZyB3aXRoIGJvb3R2 ZXJib3NlIGZvciBhcm12OCB0aW1lcnMpIGJ1dCBtaWdodCBkbyB0aGUgdHJpY2sgZm9yIHlvdS4N Cj4+ID4+IFtTb3VyYWRlZXBdDQo+PiA+PiBUaGFua3MgZm9yIHRoZSBjaGFuZ2UgYnV0IGl0IGRp ZCBub3Qgc29sdmUgdGhlIHByb2JsZW0uIFN0aWxsIHRoZQ0KPj4gPj4gaW50ZXJydXB0IGhhbmRs ZXIgdm1idXNfaGFuZGxlX2ludHIoc3RydWN0IHRyYXBmcmFtZSAqdHJhcF9mcmFtZSksDQo+PiA+ PiBpcyBub3QNCj4+ID5nZXR0aW5nIGNhbGxlZCBmb3IgdGhlIENQVSAxLg0KPj4gPj4gSXQgaXMg b25seSBnZXR0aW5nIGNhbGxlZCBmb3IgQ1BVIDAgYWxsIHRoZSB0aW1lIGluIEFSTTY0IGJ1dCBp bg0KPj4gPj4geDg2IGl0IGlzIGdldHRpbmcgY2FsbGVkIGZvciBib3RoIENQVTEgYW5kIENQVTAu DQo+PiA+DQo+PiA+SW50ZXJlc3RpbmchIEkgZG8gc2VlIG9uZSBwcm9ibGVtIHdpdGggdGhlIHBh dGNoIChhbmQgc29tZSBjb3NtZXRpYw0KPj4gPmlzc3Vlcyk6IHdlIHJlYWxseSBuZWVkIHRvIHRh a2UgdGhlIGdpY19tdHggaW4gZ2ljX3YzX3NldHVwX3BlcmlwaCgpDQo+PiA+cmlnaHQgdXAgZnJv bnQgYmVjYXVzZSBDUFVfU0VUKCkgd29uJ3QgbmVjZXNzYXJpbHkgYmUgYXRvbWljLiBUaGF0J3MN Cj4+ID5ub3QgdGhlIHByb2JsZW0sIHRob3VnaCBmb3Igb3RoZXIgcmVhc29ucywgYnV0IGFsc28g YmVjYXVzZS4uLg0KPj4gPg0KPj4gPj4gRnJvbSBEREIgSSBoYXZlIGNvbGxlY3RlZCB0aGlzIGRh dGEgaW4gYXJtNjQuIGlycTE4IGlzIGZvciB2bWJ1cy4NCj4+ID4+IGRiPiBzaG93IGlycXMNCj4+ ID4+IC4uLg0KPj4gPj4gaXJxMTggIDxnaWMwLHAyPjogY3B1IDAzIGNudCA0MTENCj4+ID4+IC4u Li4NCj4+ID4+DQo+PiA+DQo+PiA+VGhhdCB3b3VsZCBzZWVtIHRvIGluZGljYXRlIHRoYXQgYm90 aCBDUFVzIGhhdmUgc2V0IGl0IHVwLCBidXQgaXQNCj4+ID5vY2N1cnMgdG8gbWUgdGhhdCBlbmFi bGVfaW50ciBhbHNvIG5lZWRzIHRoZSBzYW1lIHRyZWF0bWVudC4gTGV0J3MNCj4+ID53aXBlIGdp Y192My5jIGJhY2sgdG8gYSBjbGVhbiBzbGF0ZSBhbmQgdHJ5IGEgdjIgb2YgdGhlIHBhdGNoOg0K Pj4gPmh0dHBzOi8vcGVvLw0KPj4NCj4+cGxlLmZyJTJGJmRhdGE9MDUlN0MwMSU3Q3NjaGFrcmFi YXJ0aSU0MG1pY3Jvc29mdC5jb20lN0NiZWI3ODU1OGJlZjANCj40DQo+Pg0KPj5jNzI4MjE2MDhk YjU1NGJlMGY4JTdDNzJmOTg4YmY4NmYxNDFhZjkxYWIyZDdjZDAxMWRiNDclN0MxJTdDMCUNCj43 QzYzODENCj4+DQo+Pjk3NTU2NzQwMjcwOTcxJTdDVW5rbm93biU3Q1RXRnBiR1pzYjNkOGV5SldJ am9pTUM0d0xqQXdNREFpTEMNCj5KUUlqb2lWMg0KPj4NCj4+bHVNeklpTENKQlRpSTZJazFoYVd3 aUxDSlhWQ0k2TW4wJTNEJTdDMzAwMCU3QyU3QyU3QyZzZGF0YT1JWHpNYnYNCj5xTHFZDQo+PiA+ dTZncUxzTjBwTmJHcTNndkRSVW9YOWJZbjlVQjdOWFJVJTNEJnJlc2VydmVkPTANCj4+ID5lZWJz ZC5vcmclMkZ+a2V2YW5zJTJGcHBpLQ0KPj4NCj4+djIuZGlmZiZkYXRhPTA1JTdDMDElN0NzY2hh a3JhYmFydGklNDBtaWNyb3NvZnQuY29tJTdDMzlkN2M1MDg0OWI2NGUNCj5hDQo+Pg0KPj5mNzBi NjA4ZGI1NTAyMGZhZSU3QzcyZjk4OGJmODZmMTQxYWY5MWFiMmQ3Y2QwMTFkYjQ3JTdDMSU3QzAl N0MNCj42DQo+Pg0KPj4zODE5NzIzOTYxNTU0NDExMyU3Q1Vua25vd24lN0NUV0ZwYkdac2IzZDhl eUpXSWpvaU1DNHdMakF3TUQNCj5BaQ0KPj4NCj4+TENKUUlqb2lWMmx1TXpJaUxDSkJUaUk2SWsx aGFXd2lMQ0pYVkNJNk1uMCUzRCU3QzMwMDAlN0MlN0MlN0Mmc2RhDQo+dA0KPj4NCj4+YT1RTERt OFgxeXV1R095Mk9lZVl4c25GJTJGTmV1Y2VBb2d3NTE2MVU3Wkg2JTJCcyUzRCZyZXNlcnZlZD0N Cj4wDQo+PiA+DQo+PiA+Rm9yIG5vdyB3ZSBqdXN0IHByZXRlbmQgdGhhdCB3ZSB3b24ndCBiZSBk aXNhYmxpbmcgYW55IFBQSXMgYXMgYSBwcm9vZi1vZi0NCj5jb25jZXB0Lg0KPj4gPg0KPj4gW1Nv dXJhZGVlcF0NCj4+IFRoaXMgaXMgY2F1c2luZyBwYW5pYyBkdXJpbmcgYm9vdDoNCj4+IHBtdTA6 IE1BRFQ6IGNwdSAwIChtcGlkciAwKSBpcnEgMCBsZXZlbC10cmlnZ2VyZWQNCj4+IHBtdTA6IE1B RFQ6IGNwdSAxIChtcGlkciAxKSBpcnEgMCBsZXZlbC10cmlnZ2VyZWQNCj4+IHBhbmljOiBnaWNf djNfZW5hYmxlX2ludHJfaW1wbDogVW5zdXBwb3J0ZWQgSVJRIDAgY3B1aWQgPSAwIHRpbWUgPSAx DQo+PiBLREI6IHN0YWNrIGJhY2t0cmFjZToNCj4+IGRiX3RyYWNlX3NlbGYoKSBhdCBkYl90cmFj ZV9zZWxmDQo+PiBkYl90cmFjZV9zZWxmX3dyYXBwZXIoKSBhdCBkYl90cmFjZV9zZWxmX3dyYXBw ZXIrMHgzMA0KPj4gdnBhbmljKCkgYXQgdnBhbmljKzB4MTNjDQo+PiBwYW5pYygpIGF0IHBhbmlj KzB4NDQNCj4+IGdpY192M19lbmFibGVfaW50cl9pbXBsKCkgYXQgZ2ljX3YzX2VuYWJsZV9pbnRy X2ltcGwrMHhlYw0KPj4gaW50cl9zZXR1cF9pcnEoKSBhdCBpbnRyX3NldHVwX2lycSsweDM2OA0K Pj4gYnVzX3NldHVwX2ludHIoKSBhdCBidXNfc2V0dXBfaW50cisweDk0DQo+PiBwbXVfYXR0YWNo KCkgYXQgcG11X2F0dGFjaCsweDY0DQo+PiBwbXVfYWNwaV9hdHRhY2goKSBhdCBwbXVfYWNwaV9h dHRhY2grMHg5NA0KPj4gZGV2aWNlX2F0dGFjaCgpIGF0IGRldmljZV9hdHRhY2grMHgzZjgNCj4+ IGRldmljZV9wcm9iZV9hbmRfYXR0YWNoKCkgYXQgZGV2aWNlX3Byb2JlX2FuZF9hdHRhY2grMHg3 Yw0KPj4gYnVzX2dlbmVyaWNfbmV3X3Bhc3MoKSBhdCBidXNfZ2VuZXJpY19uZXdfcGFzcysweGZj DQo+PiBidXNfZ2VuZXJpY19uZXdfcGFzcygpIGF0IGJ1c19nZW5lcmljX25ld19wYXNzKzB4YWMN Cj4+IGJ1c19nZW5lcmljX25ld19wYXNzKCkgYXQgYnVzX2dlbmVyaWNfbmV3X3Bhc3MrMHhhYw0K Pj4gYnVzX3NldF9wYXNzKCkgYXQgYnVzX3NldF9wYXNzKzB4NGMNCj4+IG1pX3N0YXJ0dXAoKSBh dCBtaV9zdGFydHVwKzB4MWZjDQo+PiB2aXJ0ZG9uZSgpIGF0IHZpcnRkb25lKzB4NzANCj4+IEtE QjogZW50ZXI6IHBhbmljDQo+PiBbIHRocmVhZCBwaWQgMCB0aWQgMTAwMDAwIF0NCj4+IFN0b3Bw ZWQgYXQgICAgICBrZGJfZW50ZXIrMHg0NDogc3RyICAgICB4enIsIFt4MTksICMzMzI4XQ0KPj4g RGV0YWlscyBvZiB0aGUgbG9nIEkgaGF2ZSBwYXN0ZWQgaGVyZToNCj4+IGh0dHBzOi8vcGFzdC8N Cj4+DQo+ZWJpbi5jb20lMkZlUU41Um50RCZkYXRhPTA1JTdDMDElN0NzY2hha3JhYmFydGklNDBt aWNyb3NvZnQuY29tJTdDYg0KPmViNw0KPj4NCj44NTU4YmVmMDRjNzI4MjE2MDhkYjU1NGJlMGY4 JTdDNzJmOTg4YmY4NmYxNDFhZjkxYWIyZDdjZDAxMWRiNDclNw0KPkMxJTdDDQo+Pg0KPjAlN0M2 MzgxOTc1NTY3NDAyNzA5NzElN0NVbmtub3duJTdDVFdGcGJHWnNiM2Q4ZXlKV0lqb2lNQzR3TGpB DQo+d01EQWlMQ0oNCj4+DQo+UUlqb2lWMmx1TXpJaUxDSkJUaUk2SWsxaGFXd2lMQ0pYVkNJNk1u MCUzRCU3QzMwMDAlN0MlN0MlN0Mmc2RhdGE9DQo+S1lmdw0KPj4gZDhub2t3cWZWenZTTWZlbGd2 MVRyZ01QWXR6QXRHOU4lMkZQQ1JVTVElM0QmcmVzZXJ2ZWQ9MA0KPg0KPkkgdGhpbmsgeW91IGZl dGNoZWQgYSBicm9rZW4gdmVyc2lvbiBvZiB0aGUgcGF0Y2ggLS0gdGhlcmUgd2FzIG1heWJlIGFu IGhvdXIgd2hlcmUNCj5JIGtlcHQgcmVwbGFjaW5nIGl0IGJlY2F1c2UgSSBmb3VuZCBhIHByb2Js ZW0gYW5kIGZpeGVkIGl0LCBidXQgY3JlYXRlZCBhbm90aGVyIG9uZSB0bw0KPmZpeCB0aGF0IEkg ZGlzY292ZXJlZCBvbiBzb21lIGhhcmR3YXJlIEkgdGVzdGVkIG9uLiBJJ3ZlIHJlLXVwbG9hZGVk IHRoZSBjb3JyZWN0IGRpZmYNCj5hdA0KPmh0dHBzOi8vcGVvcGxlLmZyLw0KPmVlYnNkLm9yZyUy Rn5rZXZhbnMlMkZwcGktDQo+djMuZGlmZiZkYXRhPTA1JTdDMDElN0NzY2hha3JhYmFydGklNDBt aWNyb3NvZnQuY29tJTdDYmViNzg1NThiZWYwNGM3DQo+MjgyMTYwOGRiNTU0YmUwZjglN0M3MmY5 ODhiZjg2ZjE0MWFmOTFhYjJkN2NkMDExZGI0NyU3QzElN0MwJTdDDQo+NjM4MTk3NTU2NzQwMjcw OTcxJTdDVW5rbm93biU3Q1RXRnBiR1pzYjNkOGV5SldJam9pTUM0d0xqQXdNRA0KPkFpTENKUUlq b2lWMmx1TXpJaUxDSkJUaUk2SWsxaGFXd2lMQ0pYVkNJNk1uMCUzRCU3QzMwMDAlN0MlN0MlN0Mm c2QNCj5hdGE9TEglMkZWN3RYRXhHc01EUlZMbFpBRXE3MlhHUUY2dlI2WiUyQm4wVGRvVTR1ZWcl M0QmcmVzZXJ2ZWQ9MA0KPiwgd2hpY2ggY2FuJ3QgaGl0IHRoZSBwYW5pYyBiZWNhdXNlIGlycSA8 PSBHSUNfTEFTVF9QUEkgYWxsIHJldHVybiBlYXJseSAodGhlIHZlcnNpb24NCj55b3UgaGF2ZSwg dGhlIGZpcnN0IGJyYW5jaCBpbiBnaWNfdjNfZW5hYmxlX2ludHJfaW1wbCB3YXMgcHJvYmFibHkg YW4gaW5jb3JyZWN0IGBpZg0KPihpc3JjLT5pc3JjX2ZsYWdzICYgSU5UUl9JU1JDRl9QUEkpYA0K Pg0KW1NvdXJhZGVlcF0gSXQgaGFzIHdvcmtlZCEgVGhhbmtzIGEgbG90LiBXZSBzaG91bGQgZ2V0 IGl0IGNvbW1pdHRlZCBpbiBzcmMuDQpCZWZvcmUgSSBjb21taXQgdGhlIFNNUCByZWxhdGVkIGNo YW5nZXMgZm9yIEh5cGVyLVYgZHJpdmVyLg0KPlRoYW5rcywNCj4NCj5LeWxlIEV2YW5zDQo= From nobody Mon May 15 17:27:02 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QKmX30rxYz4BrDQ for ; Mon, 15 May 2023 17:27:15 +0000 (UTC) (envelope-from schakrabarti@microsoft.com) Received: from APC01-TYZ-obe.outbound.protection.outlook.com (mail-tyzapc01on2126.outbound.protection.outlook.com [40.107.117.126]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QKmX25SB7z3n8K; Mon, 15 May 2023 17:27:14 +0000 (UTC) (envelope-from schakrabarti@microsoft.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Lu1VYYjFpo1SNGZPvCyZ8g/VjaDjDPgsKcfqQlkmEIEyqoFi1I7n3r3+XYW04odurwBD0dJA5/D5U3fBWxbz96LGkEbIcXaJYfdBlamNi8LicJfNd0ntaDHuQgq6GWkKZLT9ZGKCI3j3r8x20s16gizYkgz3x4j9fjUquuL/Ch5r/Rm4S3ynn3UWl8uogb0U2ZpTF3R9vmNDw1vwwnKY8ZJN/CUUPzjq+SEOXVxW2SU/YPXJe6aQpJPfC9EjbYbTMPecTDALm5ouhfmIIDQWnhWkGj2oTL3qR/GsI2mnWvuBo4552sIUkx3UDGqi5RlsLmD+yTf4p3VVs1djh5YaTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MlBbCSUYbmQa+4WvxS25KY8otbdOFHc0yeEdI2rCnq8=; b=HH8ALWzkCkTKwfmcakJTNhsDJCDppK/zmZcLXriLumks/IJkMOM0mPySx5+AsYVmhvq/lx14LsFD5ALjbE22ajT4ZtDnI43msob8CwnNyy8q4EF1skjBTsshjaZ4ZbYaoaCFuNehZecSNV6MyNYvqC9xkG5L0WfXB3jGsmXJI+sgvkLhr852f+gFd3zovZ4sr9b2SfXTzfQkQFfYpdh60nmT1mE86Lqm0UmhkmcvoEJlnZQTDnKPfG59XfzOjw40oJ/O/0tEdBGvjGMYRRvxoQUOm6695HmgAlCOBRlAwQzzJ2bDcMI2WjkXuHMh7bPvAM/ZAU88bleLK6NeI5FVTQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MlBbCSUYbmQa+4WvxS25KY8otbdOFHc0yeEdI2rCnq8=; b=jHtF8A7WxmgbuTXKL11xRXdVctEy9fYx6YTwem9fwRJ7kQDw/9Zj4zYwsLoW2aKERAMmYKvg52oIumSEAzr2eoMk+hpvt/674JnJc6R4I/2BaKtNUfRME9buvbtoR1apTe++yRaD4UNcXCoaMdzjFedOTfHBddP8rbbyzj2DWuA= Received: from PSAP153MB0536.APCP153.PROD.OUTLOOK.COM (2603:1096:301:75::14) by TYZP153MB0966.APCP153.PROD.OUTLOOK.COM (2603:1096:400:41e::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6433.4; Mon, 15 May 2023 17:27:02 +0000 Received: from PSAP153MB0536.APCP153.PROD.OUTLOOK.COM ([fe80::3636:5810:fb12:6d71]) by PSAP153MB0536.APCP153.PROD.OUTLOOK.COM ([fe80::3636:5810:fb12:6d71%7]) with mapi id 15.20.6433.001; Mon, 15 May 2023 17:27:02 +0000 From: Souradeep Chakrabarti To: Souradeep Chakrabarti , Kyle Evans CC: Wei Hu , "freebsd-hackers@FreeBSD.org" Subject: RE: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 SMP Thread-Topic: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 SMP Thread-Index: AdmBrWHu4SCK2cRnRnyy6MBGajO35QDM7LrAAATdMgAAXrUlgAAe4ZoAAAUxoMAADULwAAAGs2kgAAB9DCA= Date: Mon, 15 May 2023 17:27:02 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=579a9a93-7e36-413a-9714-245856ac89e3;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-05-15T17:05:57Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PSAP153MB0536:EE_|TYZP153MB0966:EE_ x-ms-office365-filtering-correlation-id: e6124f34-6768-40a2-e935-08db556998be x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Q0r7X/Kl5uFvo4pj4/2J8fvh+c56p/IWhRgBYh6BrCD1dBQ+6ER5J/ycJE5ZtiHsimkHkx20TJXMNgqA9L9eo3aAzJpvP6OfIZzv4rFJ6E5cjht8cRA/NV93FymQeqWThLymobIogju/AUSxQ8WAv6Fa7wOKNxVxvMNBNS9dcGXvW5X/KkdfOYl7FM6+cZFqF4epBOyTrRRVP+B5EGV+sOphKuNo8R51s3B9Q8F6nGudEvREIwANxjjMXkR6UXc+xAp1cIkRKyq0sIZUfCijE+E27mbCJ1TkAtR1EaoC1jstVIlPSJdoc4NLcvaMxviXkFF3ejQ8Hh6Gx2Py4WKNEWu5hhYs129ygXfWEVbgZ30UZ82T0F7va+gBp9FfSH3wwAzLwLumaVtRQTXvSGAIgelCADfoe2kebxU/iZFReyJAj7il8ZoiVAn55TO3mwuYFyVTQH08LJpWF0kk+gB/qP6uU5uGvLLtHXzzyj16xo4NbwqiOHQRTfoO5MsafLjeYP+GsDqyQtzquWMffhHn4X4+kkc12RO5BRZ9XIQo5Nri7mvJ7lJcn40Cnjm/Xcy1JAEoe/GEzSSEx5WouaympTeA66dxhZ7K+w4KTUgupcfOqoCd4RKpmXPsRyzZkAAQ+xF7HNKtST30Ic13jf/5DKehiS2obFpR5b33H9GFcX6lN6ENDo+PEfaEsdeZwJSt x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PSAP153MB0536.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230028)(4636009)(39860400002)(396003)(376002)(366004)(136003)(346002)(451199021)(2940100002)(5660300002)(52536014)(41300700001)(71200400001)(2906002)(30864003)(83380400001)(8990500004)(82960400001)(122000001)(86362001)(82950400001)(38070700005)(38100700002)(33656002)(26005)(6506007)(186003)(55016003)(8676002)(8936002)(9686003)(966005)(450100002)(7696005)(66946007)(66556008)(66476007)(66446008)(64756008)(76116006)(54906003)(110136005)(478600001)(10290500003)(4326008)(786003)(316002)(21314003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?MUUvWWJVOFRPRUpmNUtGL2hyb3lTWmtHQndJck9FWU93T29mVkh3bFhCY3Bv?= =?utf-8?B?ODNxZDFZT1B0Nmc2dnF6b1l4RTJHeVZpZ01acG9RRjlUam9HZlhVVGdpSGNV?= =?utf-8?B?VllFTjBJVEh3RGRlU2d2SXorMDVmMTRkUlNCVFNmWldENE1UVVkxSFpJN0ha?= =?utf-8?B?OWcwbkpxazlkZTFxUG9FbmxON2ozYlVGZTE2WUN4eEd5Zi9uWVBoZnlRZERj?= =?utf-8?B?R2JsMnZVNE1TVjRZTXpqc1JNVi8zRi9QRnQ3UXB6U2ZkZFQzbUV1Y0VRWU1V?= =?utf-8?B?RkhFSFlCTDBDRm1UZVZYMEh2R0oxOGVDSlBkbnpueS9ZSzdXelUyVmNLWnBy?= =?utf-8?B?MGtWQnZ6WUNuQnczS2hnc3VJWi9weitwaWNmdGlWTVJVOUJFdjhLbWNia0xm?= =?utf-8?B?VDhSc0h0cGhVUTEwUGd6RDVaK3laWkthcUx5NW1UOFBnQm9xdzgxV1ptRzhB?= =?utf-8?B?d2Y3RWd3T0ZPWHluNHFiWldRY2JuVGZBRUF6bmhZVUpvdjB5bHdjbG1vMUtq?= =?utf-8?B?aTFualdYNGNEMURabGgrRFN5eVlGYmFzOUxjWWZIT3A4N3hiOXZLTUF5Nzhy?= =?utf-8?B?dHFPT2hJT0YxTUtpVXhlU1JodE9jSTV0LytkQ1JUNW5MUGI1b0U0QnJXSkps?= =?utf-8?B?M2hhMzNmVUZidFliT3dkRVE0czlTYXpTcWtvVHFmcTRqOFNYYmlVQk9ZMHpu?= =?utf-8?B?NEVzYk50K1lSZnZYd1hYYUduTXVha3ZuUnYrVmxmUVJReVNPaG0zSk9iTkZp?= =?utf-8?B?OTNEdncwZDJVbmR2MVBMNmxVTTJMemRBNTkzZnE3Rm9qUkxxaExtWkY1Qlpy?= =?utf-8?B?Z1k0ZWRqcU0yMlVSRWVLblFTQWF3aUVJbkdQbjBZZnpDWG1uQjk2VFRDdkVM?= =?utf-8?B?QlhIcktJeXZqU2VtR0F6NkhjRTV2U1VFcjRSYWtnbEllMVBCeU1pU0xJUE13?= =?utf-8?B?cG1qU1VzQWJpNU5VbEV6WWF3dEI0NllibXZoU0dFM0l2VHVVYnY5aXpPSDdB?= =?utf-8?B?RTUyamxnS1BzMFNyVWxmSUU4TENiWHNrQVNjaisvQUpBdk9hU2RoUlFDRFRZ?= =?utf-8?B?ZEo1c1M4TUt0Yk1URlM1ak40ZnIxWHdKRnJDeEFVK1JtZzVWWFB6K2hsaTJZ?= =?utf-8?B?THQ4bUpqUzRmaU5SQy8yMW5Jdi9YUFVnNVF0YnlHNW1wRGNFQ1NYcStJUEVD?= =?utf-8?B?cTljR1BqL0VLeGdDaVN4Qi85Q3VjQVVEZXMrZXBuOFNDb0JVM3BiOTRqUHdS?= =?utf-8?B?OUhreDhZa01Hc2wyeFk0TTE1SXorR213Z0x5SEdDaU1zV0d2UHpCWHBMNjFQ?= =?utf-8?B?dURlNW1JOEltbCtieVJxcUZ2Sk9aV21GYjVKR1d3ZlhhVzBvMHJKVGhUN0tW?= =?utf-8?B?bE5YZ0thakZvY2xtOXVWVWxCalJtQ2xGeGZSRTYvUnVaYWJJcDFvWGNyWFFX?= =?utf-8?B?cGQxRGtjeUtCMmNZQ2FUZTB5V0kvK1F4dFhyYk9TTE9uaVEzS2c1TkgrTHYx?= =?utf-8?B?bVEyYjV4ZThNVERsMnc5cTdRdmdZWXYxTEh0MlllRG9DZjZTQjFqaDhURGg0?= =?utf-8?B?b2lBWlBkUWZuWnVKK3R4S01aWFZFaE9jeHBQZFRwOHBXNlUrdUpnODRPUGQr?= =?utf-8?B?YllIclQ4YllwaktBMkJLUkZJVHZHR0ZqOFhSMUxSQ1NWcVc4N1cybHBHYTAx?= =?utf-8?B?YlpENDR3SlExNnZramFCWGdzRHNvT0tGa3N3VEVxaGsrRWt6QlU0VkFYWmxt?= =?utf-8?B?cWdWNnlKOFFvTkR6emI5ZkpUbG1hOGk3U3UrbEgyWmZSZXZzK25YdFNtNXln?= =?utf-8?B?WEp6MGQ0MkNYeVl5cjc1N1JNV25TODhtZzdHRG5oWWROWXVsd3czL0ZsYk1v?= =?utf-8?B?MTJCV1gvblNFZm4xa2JqQk0vS1M2KzRLcUdlMFlPSFhCS0k5cWVEejhUUGJJ?= =?utf-8?B?Ynk1ZjVxd1l0S1pOM0tzYjBvMWRScFluK3liWXB0Ti90M1lkTDZJZ0hLWmJ5?= =?utf-8?B?TjI4S1Y3RWNRV0krNlpTbFNUNy9tNmNMVklNZmVEUlJNMzlPTzhZN0RQRkdK?= =?utf-8?Q?gsHmSc?= Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PSAP153MB0536.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: e6124f34-6768-40a2-e935-08db556998be X-MS-Exchange-CrossTenant-originalarrivaltime: 15 May 2023 17:27:02.5275 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 35RbCyjXe4iQUdyaW+7LrD685H11W31kZG+wBd+fgrh6yWj3jmb78+O39Wz5ayUxrNnhJUoiAZlCMmAI6NSN55YBfsHnJGGGSI/HJriqpHY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYZP153MB0966 X-Rspamd-Queue-Id: 4QKmX25SB7z3n8K X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N DQoNCg0KPi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+RnJvbTogb3duZXItZnJlZWJzZC1o YWNrZXJzQGZyZWVic2Qub3JnIDxvd25lci1mcmVlYnNkLWhhY2tlcnNAZnJlZWJzZC5vcmc+DQo+ T24gQmVoYWxmIE9mIFNvdXJhZGVlcCBDaGFrcmFiYXJ0aQ0KPlNlbnQ6IE1vbmRheSwgTWF5IDE1 LCAyMDIzIDEwOjM3IFBNDQo+VG86IEt5bGUgRXZhbnMgPGtldmFuc0BmcmVlYnNkLm9yZz4NCj5D YzogV2VpIEh1IDx3ZWhAbWljcm9zb2Z0LmNvbT47IGZyZWVic2QtaGFja2Vyc0BGcmVlQlNELm9y Zw0KPlN1YmplY3Q6IFJFOiBbRVhURVJOQUxdIFJlOiBlbmFibGluZyBzYW1lIFBQSSBpbnRlcnJ1 cHQgdG8gYWxsIENQVSBpbiBBUk02NCBTTVANCj4NCj4NCj4NCj4NCj4+LS0tLS1PcmlnaW5hbCBN ZXNzYWdlLS0tLS0NCj4+RnJvbTogS3lsZSBFdmFucyA8a2V2YW5zQGZyZWVic2Qub3JnPg0KPj5T ZW50OiBNb25kYXksIE1heSAxNSwgMjAyMyA3OjI0IFBNDQo+PlRvOiBTb3VyYWRlZXAgQ2hha3Jh YmFydGkgPHNjaGFrcmFiYXJ0aUBtaWNyb3NvZnQuY29tPg0KPj5DYzogV2VpIEh1IDx3ZWhAbWlj cm9zb2Z0LmNvbT47IGZyZWVic2QtaGFja2Vyc0BGcmVlQlNELm9yZw0KPj5TdWJqZWN0OiBSZTog W0VYVEVSTkFMXSBSZTogZW5hYmxpbmcgc2FtZSBQUEkgaW50ZXJydXB0IHRvIGFsbCBDUFUgaW4N Cj4+QVJNNjQgU01QDQo+Pg0KPj5PbiBNb24sIE1heSAxNSwgMjAyMyBhdCAyOjQx4oCvQU0gU291 cmFkZWVwIENoYWtyYWJhcnRpDQo+PjxzY2hha3JhYmFydGlAbWljcm9zb2Z0LmNvbT4gd3JvdGU6 DQo+Pj4NCj4+Pg0KPj4+DQo+Pj4NCj4+PiA+LS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4+ PiA+RnJvbTogS3lsZSBFdmFucyA8a2V2YW5zQGZyZWVic2Qub3JnPg0KPj4+ID5TZW50OiBNb25k YXksIE1heSAxNSwgMjAyMyAxMDozNiBBTQ0KPj4+ID5UbzogU291cmFkZWVwIENoYWtyYWJhcnRp IDxzY2hha3JhYmFydGlAbWljcm9zb2Z0LmNvbT4NCj4+PiA+Q2M6IFdlaSBIdSA8d2VoQG1pY3Jv c29mdC5jb20+OyBmcmVlYnNkLWhhY2tlcnNARnJlZUJTRC5vcmcNCj4+PiA+U3ViamVjdDogUmU6 IFtFWFRFUk5BTF0gUmU6IGVuYWJsaW5nIHNhbWUgUFBJIGludGVycnVwdCB0byBhbGwgQ1BVDQo+ Pj4gPmluDQo+Pj4gPkFSTTY0IFNNUA0KPj4+ID4NCj4+PiA+T24gU3VuLCBNYXkgMTQsIDIwMjMg YXQgOTo1OeKAr0FNIFNvdXJhZGVlcCBDaGFrcmFiYXJ0aQ0KPj4+ID48c2NoYWtyYWJhcnRpQG1p Y3Jvc29mdC5jb20+IHdyb3RlOg0KPj4+ID4+DQo+Pj4gPj4NCj4+PiA+Pg0KPj4+ID4+DQo+Pj4g Pj4gPi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+Pj4gPj4gPkZyb206IEt5bGUgRXZhbnMg PGtldmFuc0BmcmVlYnNkLm9yZz4NCj4+PiA+PiA+U2VudDogRnJpZGF5LCBNYXkgMTIsIDIwMjMg MTA6NDAgUE0NCj4+PiA+PiA+VG86IFNvdXJhZGVlcCBDaGFrcmFiYXJ0aSA8c2NoYWtyYWJhcnRp QG1pY3Jvc29mdC5jb20+DQo+Pj4gPj4gPkNjOiBXZWkgSHUgPHdlaEBtaWNyb3NvZnQuY29tPjsg ZnJlZWJzZC1oYWNrZXJzQEZyZWVCU0Qub3JnDQo+Pj4gPj4gPlN1YmplY3Q6IFtFWFRFUk5BTF0g UmU6IGVuYWJsaW5nIHNhbWUgUFBJIGludGVycnVwdCB0byBhbGwgQ1BVIGluDQo+Pj4gPj4gPkFS TTY0IFNNUA0KPj4+ID4+ID4NCj4+PiA+PiA+T24gRnJpLCBNYXkgMTIsIDIwMjMgYXQgOTo1MeKA r0FNIFNvdXJhZGVlcCBDaGFrcmFiYXJ0aQ0KPj4+ID4+ID48c2NoYWtyYWJhcnRpQG1pY3Jvc29m dC5jb20+IHdyb3RlOg0KPj4+ID4+ID4+DQo+Pj4gPj4gPj4NCj4+PiA+PiA+Pg0KPj4+ID4+ID4+ DQo+Pj4gPj4gPj4gPi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+Pj4gPj4gPj4gPkZyb206 IFNvdXJhZGVlcCBDaGFrcmFiYXJ0aQ0KPj4+ID4+ID4+ID5TZW50OiBNb25kYXksIE1heSA4LCAy MDIzIDY6MzkgUE0NCj4+PiA+PiA+PiA+VG86IEt5bGUgRXZhbnMgPGtldmFuc0BmcmVlYnNkLm9y Zz4NCj4+PiA+PiA+PiA+Q2M6IFdlaSBIdSA8d2VoQG1pY3Jvc29mdC5jb20+OyBmcmVlYnNkLWhh Y2tlcnNARnJlZUJTRC5vcmcNCj4+PiA+PiA+PiA+U3ViamVjdDogZW5hYmxpbmcgc2FtZSBQUEkg aW50ZXJydXB0IHRvIGFsbCBDUFUgaW4gQVJNNjQgU01QDQo+Pj4gPj4gPj4gPg0KPj4+ID4+ID4+ ID5IaSAsDQo+Pj4gPj4gPj4gPg0KPj4+ID4+ID4+ID5XaGlsZSB1c2luZyBTTVAgaW4gQVJNNjQg SHlwZXItViB3ZSBhcmUgZ2V0dGluZyBzdHVjayBpbiBib290DQo+Pj4gPj4gPj4gPmlmIHRoZXJl IGlzIGEgaW50ZXJydXB0IGZvciBWTUJ1cyBjb21pbmcgdG8gQ1BVMSBhbmQgVk1CdXMNCj4+PiA+ PiA+PiA+aW50ZXJydXB0IGhhbmRsZXIgaXMgbm90IGdldHRpbmcgdGhhdCBpbnRlcnJ1cHQuDQo+ Pj4gPj4gPj4gPg0KPj4+ID4+ID4+ID5JbiBBUk02NCBIeXBlci1WIHdlIGFyZSB1c2luZyBJUlEx OCBmb3IgVk1CdXMgYW5kIGl0IGlzIGEgUFBJIGludGVycnVwdC4NCj4+PiA+PiA+PiA+DQo+Pj4g Pj4gPj4gPkJ1dCBIeXBldi1WIGhvc3Qgc2VuZHMgaW50ZXJydXB0IHRvIHRoaXMgSVJRIDE4IGZv ciBib3RoIENQVTANCj4+PiA+PiA+PiA+YW5kDQo+Pj4gPj4gPj4gPkNQVTEgaW4gMkNQVSBzeXN0 ZW0uDQo+Pj4gPj4gPj4gPlRoaXMgaXMgYmFzZWQgb24gdGhlIGNvcnJlc3BvbmRpbmcgVk1CdXMg Y2hhbm5lbCB3aGljaA0KPj4+ID4+ID4+ID5hc3NpZ25lZCB3aXRoIHRoZQ0KPj4+ID5DUFUuDQo+ Pj4gPj4gPj4gPg0KPj4+ID4+ID4+ID5Ob3cgVk1CdXMgSVNSIGlzIGdldHRpbmcgdGhlIGludGVy cnVwdCBpbiBDUFUwIGJ1dCBub3QgZ2V0dGluZw0KPj4+ID4+ID4+ID5mcm9tDQo+PkNQVTEuDQo+ Pj4gPj4gPj4gPkFueSBpZGVhLCBob3cgd2UgY2FuIHVzZSB0aGUgc2FtZSBQUEkgMTggZm9yIGFs bCB0aGUgQ1BVIGNvcmVzPw0KPj4+ID4+ID4+ID4NCj4+PiA+PiA+PiA+QW55IGhlbHAgd2lsbCBi ZSBhcHByZWNpYXRlZCwgYXMgdGhpcyBpcyBibG9ja2luZyB0aGUNCj4+PiA+PiA+PiA+ZW5hYmxl bWVudCBvZiBGcmVlQlNEIGluIEF6dXJlIEFSTTY0Lg0KPj4+ID4+ID4+IFtTb3VyYWRlZXBdDQo+ Pj4gPj4gPj4gQ2FuIHNvbWVvbmUgcGxlYXNlIGhlbHAgbWUgaXQuDQo+Pj4gPj4gPj4NCj4+PiA+ PiA+DQo+Pj4gPj4gPkxvb2tpbmcgYXQgbGVhc3QgYXQgdGhlIEdJQyBpbXBsZW1lbnRhdGlvbiwg aXQgbG9va3MgbGlrZSB0aGlzIGlzDQo+Pj4gPj4gPmEga25vd24NCj4+bGltaXRhdGlvbjoNCj4+ PiA+PiA+DQo+Pj4gPj4gPiA4NzUgICAgICAgICAvKg0KPj4+ID4+ID4gODc2ICAgICAgICAgICog WFhYIC0gSW4gY2FzZSB0aGF0IHBlciBDUFUgaW50ZXJydXB0IGlzIGdvaW5nIHRvIGJlDQo+Pj4g Pj4gPmVuYWJsZWQgaW4gdGltZQ0KPj4+ID4+ID4gODc3ICAgICAgICAgICogICAgICAgd2hlbiBT TVAgaXMgYWxyZWFkeSBzdGFydGVkLCB3ZSBuZWVkIHNvbWUgSVBJDQo+Pj4gPj4gPmNhbGwgd2hp Y2gNCj4+PiA+PiA+IDg3OCAgICAgICAgICAqICAgICAgIGVuYWJsZXMgaXQgb24gb3RoZXJzIENQ VXMuIEZ1cnRoZXIsIGl0J3MgbW9yZQ0KPj4+ID4+ID5jb21wbGljYXRlZCBhcw0KPj4+ID4+ID4g ODc5ICAgICAgICAgICogICAgICAgcGljX2VuYWJsZV9zb3VyY2UoKSBhbmQgcGljX2Rpc2FibGVf c291cmNlKCkNCj4+PiA+PiA+c2hvdWxkIGFjdCBvbg0KPj4+ID4+ID4gODgwICAgICAgICAgICog ICAgICAgcGVyIENQVSBiYXNpcyBvbmx5LiBUaHVzLCBpdCBzaG91bGQgYmUgc29sdmVkDQo+Pj4g Pj4gPmhlcmUgc29tZWhvdy4NCj4+PiA+PiA+IDg4MSAgICAgICAgICAqLw0KPj4+ID4+ID4gODgy ICAgICAgICAgaWYgKGlzcmMtPmlzcmNfZmxhZ3MgJiBJTlRSX0lTUkNGX1BQSSkNCj4+PiA+PiA+ IDg4MyAgICAgICAgICAgICAgICAgQ1BVX1NFVChQQ1BVX0dFVChjcHVpZCksICZpc3JjLT5pc3Jj X2NwdSk7DQo+Pj4gPj4gPg0KPj4+ID4+ID5JIHRoaW5rIHdlIG5lZWQgc29tZXRoaW5nIC9saWtl LyB0aGlzOg0KPj4+ID4+ID5odHRwczovL3Blby8NCj4+PiA+Pg0KPj4+DQo+Pj4+cGxlLmZyJTJG JmRhdGE9MDUlN0MwMSU3Q3NjaGFrcmFiYXJ0aSU0MG1pY3Jvc29mdC5jb20lN0MzOWQ3YzUwODQ5 DQo+Yg0KPj4+ID42NA0KPj4+ID4+DQo+Pj4NCj4+Pj5lYWY3MGI2MDhkYjU1MDIwZmFlJTdDNzJm OTg4YmY4NmYxNDFhZjkxYWIyZDdjZDAxMWRiNDclN0MxJTdDMA0KPj4lDQo+Pj4gPjdDNjM4MQ0K Pj4+ID4+DQo+Pj4NCj4+Pj45NzIzOTYxNTU0NDExMyU3Q1Vua25vd24lN0NUV0ZwYkdac2IzZDhl eUpXSWpvaU1DNHdMakF3TURBaQ0KPkwNCj4+Qw0KPj4+ID5KUUlqb2lWMg0KPj4+ID4+DQo+Pj4N Cj4+Pj5sdU16SWlMQ0pCVGlJNklrMWhhV3dpTENKWFZDSTZNbjAlM0QlN0MzMDAwJTdDJTdDJTdD JnNkYXRhPWRYVnYNCj4+bkkNCj4+PiA+SGhOOQ0KPj4+ID4+ID5tZHdpUGtTSkt3TXlFS1lpNVN5 R091dGE1ekNaMXlzQ1ElM0QmcmVzZXJ2ZWQ9MA0KPj4+ID4+DQo+Pj4NCj4+Pj5lZWJzZC5vcmcl MkZ+a2V2YW5zJTJGcHBpLmRpZmYmZGF0YT0wNSU3QzAxJTdDc2NoYWtyYWJhcnRpJTQwbWljcm9z DQo+bw0KPj4+ID4+Zg0KPj4+ID4+ID50LmMNCj4+Pg0KPj4+b20lN0NjNWMzZDI1NGI5ZDg0MWU5 YWU5YjA4ZGI1MzBiYjNkMiU3QzcyZjk4OGJmODZmMTQxYWY5MWFiMmQNCj43DQo+PmMNCj4+PiA+ Pg0KPj4+DQo+Pj4+ZDAxMWRiNDclN0MxJTdDMCU3QzYzODE5NTA4MjAyNzc0NDcwNiU3Q1Vua25v d24lN0NUV0ZwYkdaDQo+cw0KPj5iDQo+Pj4gPjMNCj4+PiA+Pg0KPj4+DQo+Pj4+ZDhleUpXSWpv aU1DNHdMakF3TURBaUxDSlFJam9pVjJsdU16SWlMQ0pCVGlJNklrMWhhV3dpTENKWFZDSTZNbjAl DQo+Mw0KPj5EDQo+Pj4gPj4NCj4+Pg0KPj4+PiU3QzMwMDAlN0MlN0MlN0Mmc2RhdGE9U3dvUjJ2 SHhoNlFRaHdwT2dGY1E5Mzc4bkRWb3ZoZEVLRVgNCj5vDQo+PkUNCj4+PiA+bw0KPj4+ID4+ID5n UEtzYyUzRCZyZXNlcnZlZD0wLCB0aG91Z2ggaXQgc3RpbGwgaGFzIHRoZSBjYXZlYXQgdGhhdCBQ UElzDQo+Pj4gPj4gPmVmZmVjdGl2ZWx5IGNhbm5vdCBiZSBmdWxseSBzZXR1cCBiZWZvcmUgU0lf U1VCX1NNUC4NCj4+PiA+PiA+U28sIGl0J3MgbGlrZWx5IGFsbW9zdCBhIE5PUCBmb3IgZXhpc3Rp bmcgcGxhdGZvcm1zICh3aWxsIGVtaXQgYQ0KPj4+ID4+ID53YXJuaW5nIHdpdGggYm9vdHZlcmJv c2UgZm9yIGFybXY4IHRpbWVycykgYnV0IG1pZ2h0IGRvIHRoZSB0cmljayBmb3IgeW91Lg0KPj4+ ID4+IFtTb3VyYWRlZXBdDQo+Pj4gPj4gVGhhbmtzIGZvciB0aGUgY2hhbmdlIGJ1dCBpdCBkaWQg bm90IHNvbHZlIHRoZSBwcm9ibGVtLiBTdGlsbCB0aGUNCj4+PiA+PiBpbnRlcnJ1cHQgaGFuZGxl ciB2bWJ1c19oYW5kbGVfaW50cihzdHJ1Y3QgdHJhcGZyYW1lICp0cmFwX2ZyYW1lKSwNCj4+PiA+ PiBpcyBub3QNCj4+PiA+Z2V0dGluZyBjYWxsZWQgZm9yIHRoZSBDUFUgMS4NCj4+PiA+PiBJdCBp cyBvbmx5IGdldHRpbmcgY2FsbGVkIGZvciBDUFUgMCBhbGwgdGhlIHRpbWUgaW4gQVJNNjQgYnV0 IGluDQo+Pj4gPj4geDg2IGl0IGlzIGdldHRpbmcgY2FsbGVkIGZvciBib3RoIENQVTEgYW5kIENQ VTAuDQo+Pj4gPg0KPj4+ID5JbnRlcmVzdGluZyEgSSBkbyBzZWUgb25lIHByb2JsZW0gd2l0aCB0 aGUgcGF0Y2ggKGFuZCBzb21lIGNvc21ldGljDQo+Pj4gPmlzc3Vlcyk6IHdlIHJlYWxseSBuZWVk IHRvIHRha2UgdGhlIGdpY19tdHggaW4gZ2ljX3YzX3NldHVwX3BlcmlwaCgpDQo+Pj4gPnJpZ2h0 IHVwIGZyb250IGJlY2F1c2UgQ1BVX1NFVCgpIHdvbid0IG5lY2Vzc2FyaWx5IGJlIGF0b21pYy4g VGhhdCdzDQo+Pj4gPm5vdCB0aGUgcHJvYmxlbSwgdGhvdWdoIGZvciBvdGhlciByZWFzb25zLCBi dXQgYWxzbyBiZWNhdXNlLi4uDQo+Pj4gPg0KPj4+ID4+IEZyb20gRERCIEkgaGF2ZSBjb2xsZWN0 ZWQgdGhpcyBkYXRhIGluIGFybTY0LiBpcnExOCBpcyBmb3Igdm1idXMuDQo+Pj4gPj4gZGI+IHNo b3cgaXJxcw0KPj4+ID4+IC4uLg0KPj4+ID4+IGlycTE4ICA8Z2ljMCxwMj46IGNwdSAwMyBjbnQg NDExDQo+Pj4gPj4gLi4uLg0KPj4+ID4+DQo+Pj4gPg0KPj4+ID5UaGF0IHdvdWxkIHNlZW0gdG8g aW5kaWNhdGUgdGhhdCBib3RoIENQVXMgaGF2ZSBzZXQgaXQgdXAsIGJ1dCBpdA0KPj4+ID5vY2N1 cnMgdG8gbWUgdGhhdCBlbmFibGVfaW50ciBhbHNvIG5lZWRzIHRoZSBzYW1lIHRyZWF0bWVudC4g TGV0J3MNCj4+PiA+d2lwZSBnaWNfdjMuYyBiYWNrIHRvIGEgY2xlYW4gc2xhdGUgYW5kIHRyeSBh IHYyIG9mIHRoZSBwYXRjaDoNCj4+PiA+aHR0cHM6Ly9wZW8vDQo+Pj4NCj4+PnBsZS5mciUyRiZk YXRhPTA1JTdDMDElN0NzY2hha3JhYmFydGklNDBtaWNyb3NvZnQuY29tJTdDYmViNzg1NThiZWYN Cj4wDQo+PjQNCj4+Pg0KPj4+YzcyODIxNjA4ZGI1NTRiZTBmOCU3QzcyZjk4OGJmODZmMTQxYWY5 MWFiMmQ3Y2QwMTFkYjQ3JTdDMSU3QzANCj4lDQo+PjdDNjM4MQ0KPj4+DQo+Pj45NzU1Njc0MDI3 MDk3MSU3Q1Vua25vd24lN0NUV0ZwYkdac2IzZDhleUpXSWpvaU1DNHdMakF3TURBaUwNCj5DDQo+ PkpRSWpvaVYyDQo+Pj4NCj4+Pmx1TXpJaUxDSkJUaUk2SWsxaGFXd2lMQ0pYVkNJNk1uMCUzRCU3 QzMwMDAlN0MlN0MlN0Mmc2RhdGE9SVh6TWINCj52DQo+PnFMcVkNCj4+PiA+dTZncUxzTjBwTmJH cTNndkRSVW9YOWJZbjlVQjdOWFJVJTNEJnJlc2VydmVkPTANCj4+PiA+ZWVic2Qub3JnJTJGfmtl dmFucyUyRnBwaS0NCj4+Pg0KPj4+djIuZGlmZiZkYXRhPTA1JTdDMDElN0NzY2hha3JhYmFydGkl NDBtaWNyb3NvZnQuY29tJTdDMzlkN2M1MDg0OWI2NA0KPmUNCj4+YQ0KPj4+DQo+Pj5mNzBiNjA4 ZGI1NTAyMGZhZSU3QzcyZjk4OGJmODZmMTQxYWY5MWFiMmQ3Y2QwMTFkYjQ3JTdDMSU3QzAlNw0K PkMNCj4+Ng0KPj4+DQo+Pj4zODE5NzIzOTYxNTU0NDExMyU3Q1Vua25vd24lN0NUV0ZwYkdac2Iz ZDhleUpXSWpvaU1DNHdMakF3TQ0KPkQNCj4+QWkNCj4+Pg0KPj4+TENKUUlqb2lWMmx1TXpJaUxD SkJUaUk2SWsxaGFXd2lMQ0pYVkNJNk1uMCUzRCU3QzMwMDAlN0MlN0MlN0Mmc2QNCj5hDQo+PnQN Cj4+Pg0KPj4+YT1RTERtOFgxeXV1R095Mk9lZVl4c25GJTJGTmV1Y2VBb2d3NTE2MVU3Wkg2JTJC cyUzRCZyZXNlcnZlZA0KPj0NCj4+MA0KPj4+ID4NCj4+PiA+Rm9yIG5vdyB3ZSBqdXN0IHByZXRl bmQgdGhhdCB3ZSB3b24ndCBiZSBkaXNhYmxpbmcgYW55IFBQSXMgYXMgYQ0KPj4+ID5wcm9vZi1v Zi0NCj4+Y29uY2VwdC4NCj4+PiA+DQo+Pj4gW1NvdXJhZGVlcF0NCj4+PiBUaGlzIGlzIGNhdXNp bmcgcGFuaWMgZHVyaW5nIGJvb3Q6DQo+Pj4gcG11MDogTUFEVDogY3B1IDAgKG1waWRyIDApIGly cSAwIGxldmVsLXRyaWdnZXJlZA0KPj4+IHBtdTA6IE1BRFQ6IGNwdSAxIChtcGlkciAxKSBpcnEg MCBsZXZlbC10cmlnZ2VyZWQNCj4+PiBwYW5pYzogZ2ljX3YzX2VuYWJsZV9pbnRyX2ltcGw6IFVu c3VwcG9ydGVkIElSUSAwIGNwdWlkID0gMCB0aW1lID0gMQ0KPj4+IEtEQjogc3RhY2sgYmFja3Ry YWNlOg0KPj4+IGRiX3RyYWNlX3NlbGYoKSBhdCBkYl90cmFjZV9zZWxmDQo+Pj4gZGJfdHJhY2Vf c2VsZl93cmFwcGVyKCkgYXQgZGJfdHJhY2Vfc2VsZl93cmFwcGVyKzB4MzANCj4+PiB2cGFuaWMo KSBhdCB2cGFuaWMrMHgxM2MNCj4+PiBwYW5pYygpIGF0IHBhbmljKzB4NDQNCj4+PiBnaWNfdjNf ZW5hYmxlX2ludHJfaW1wbCgpIGF0IGdpY192M19lbmFibGVfaW50cl9pbXBsKzB4ZWMNCj4+PiBp bnRyX3NldHVwX2lycSgpIGF0IGludHJfc2V0dXBfaXJxKzB4MzY4DQo+Pj4gYnVzX3NldHVwX2lu dHIoKSBhdCBidXNfc2V0dXBfaW50cisweDk0DQo+Pj4gcG11X2F0dGFjaCgpIGF0IHBtdV9hdHRh Y2grMHg2NA0KPj4+IHBtdV9hY3BpX2F0dGFjaCgpIGF0IHBtdV9hY3BpX2F0dGFjaCsweDk0DQo+ Pj4gZGV2aWNlX2F0dGFjaCgpIGF0IGRldmljZV9hdHRhY2grMHgzZjgNCj4+PiBkZXZpY2VfcHJv YmVfYW5kX2F0dGFjaCgpIGF0IGRldmljZV9wcm9iZV9hbmRfYXR0YWNoKzB4N2MNCj4+PiBidXNf Z2VuZXJpY19uZXdfcGFzcygpIGF0IGJ1c19nZW5lcmljX25ld19wYXNzKzB4ZmMNCj4+PiBidXNf Z2VuZXJpY19uZXdfcGFzcygpIGF0IGJ1c19nZW5lcmljX25ld19wYXNzKzB4YWMNCj4+PiBidXNf Z2VuZXJpY19uZXdfcGFzcygpIGF0IGJ1c19nZW5lcmljX25ld19wYXNzKzB4YWMNCj4+PiBidXNf c2V0X3Bhc3MoKSBhdCBidXNfc2V0X3Bhc3MrMHg0Yw0KPj4+IG1pX3N0YXJ0dXAoKSBhdCBtaV9z dGFydHVwKzB4MWZjDQo+Pj4gdmlydGRvbmUoKSBhdCB2aXJ0ZG9uZSsweDcwDQo+Pj4gS0RCOiBl bnRlcjogcGFuaWMNCj4+PiBbIHRocmVhZCBwaWQgMCB0aWQgMTAwMDAwIF0NCj4+PiBTdG9wcGVk IGF0ICAgICAga2RiX2VudGVyKzB4NDQ6IHN0ciAgICAgeHpyLCBbeDE5LCAjMzMyOF0NCj4+PiBE ZXRhaWxzIG9mIHRoZSBsb2cgSSBoYXZlIHBhc3RlZCBoZXJlOg0KPj4+IGh0dHBzOi8vcGFzdC8N Cj4+Pg0KPj5lYmluLmNvbSUyRmVRTjVSbnREJmRhdGE9MDUlN0MwMSU3Q3NjaGFrcmFiYXJ0aSU0 MG1pY3Jvc29mdC5jb20lN0MNCj5iDQo+PmViNw0KPj4+DQo+Pjg1NThiZWYwNGM3MjgyMTYwOGRi NTU0YmUwZjglN0M3MmY5ODhiZjg2ZjE0MWFmOTFhYjJkN2NkMDExZGI0NyUNCj43DQo+PkMxJTdD DQo+Pj4NCj4+MCU3QzYzODE5NzU1Njc0MDI3MDk3MSU3Q1Vua25vd24lN0NUV0ZwYkdac2IzZDhl eUpXSWpvaU1DNHdMag0KPkENCj4+d01EQWlMQ0oNCj4+Pg0KPj5RSWpvaVYybHVNeklpTENKQlRp STZJazFoYVd3aUxDSlhWQ0k2TW4wJTNEJTdDMzAwMCU3QyU3QyU3QyZzZGF0YT0NCj4+S1lmdw0K Pj4+IGQ4bm9rd3FmVnp2U01mZWxndjFUcmdNUFl0ekF0RzlOJTJGUENSVU1RJTNEJnJlc2VydmVk PTANCj4+DQo+PkkgdGhpbmsgeW91IGZldGNoZWQgYSBicm9rZW4gdmVyc2lvbiBvZiB0aGUgcGF0 Y2ggLS0gdGhlcmUgd2FzIG1heWJlIGFuDQo+PmhvdXIgd2hlcmUgSSBrZXB0IHJlcGxhY2luZyBp dCBiZWNhdXNlIEkgZm91bmQgYSBwcm9ibGVtIGFuZCBmaXhlZCBpdCwNCj4+YnV0IGNyZWF0ZWQg YW5vdGhlciBvbmUgdG8gZml4IHRoYXQgSSBkaXNjb3ZlcmVkIG9uIHNvbWUgaGFyZHdhcmUgSQ0K Pj50ZXN0ZWQgb24uIEkndmUgcmUtdXBsb2FkZWQgdGhlIGNvcnJlY3QgZGlmZiBhdA0KPj5odHRw czovL25hbTA2LnNhZmVsaW5rcy5wcm90ZWN0aW9uLm91dGxvb2suY29tLz91cmw9aHR0cHMlM0El MkYlMkZwZW9wbA0KPj5lLmZyJTJGJmRhdGE9MDUlN0MwMSU3Q3NjaGFrcmFiYXJ0aSU0MG1pY3Jv c29mdC5jb20lN0NjN2UzMTJkNWYxMWI0DQo+M2ViMg0KPj40NDIwOGRiNTU2NmQ2ZWMlN0M3MmY5 ODhiZjg2ZjE0MWFmOTFhYjJkN2NkMDExZGI0NyU3QzElN0MwJTdDNg0KPjM4MTk3NjcyNA0KPj42 Mzg0NjAyNSU3Q1Vua25vd24lN0NUV0ZwYkdac2IzZDhleUpXSWpvaU1DNHdMakF3TURBaUxDSlFJ am9pVjINCj5sdU16SWlMQw0KPj5KQlRpSTZJazFoYVd3aUxDSlhWQ0k2TW4wJTNEJTdDMzAwMCU3 QyU3QyU3QyZzZGF0YT1YZmdDdk83cWhuc3ANCj5WdGFqTHU1RA0KPj40NFpkc0NRRDFxR1N1ODBl bEVPMDN0ZyUzRCZyZXNlcnZlZD0wDQo+PmVlYnNkLm9yZyUyRn5rZXZhbnMlMkZwcGktDQo+PnYz LmRpZmYmZGF0YT0wNSU3QzAxJTdDc2NoYWtyYWJhcnRpJTQwbWljcm9zb2Z0LmNvbSU3Q2JlYjc4 NTU4YmVmMDRjDQo+Nw0KPj4yODIxNjA4ZGI1NTRiZTBmOCU3QzcyZjk4OGJmODZmMTQxYWY5MWFi MmQ3Y2QwMTFkYjQ3JTdDMSU3QzAlNw0KPkMNCj4+NjM4MTk3NTU2NzQwMjcwOTcxJTdDVW5rbm93 biU3Q1RXRnBiR1pzYjNkOGV5SldJam9pTUM0d0xqQXdNDQo+RA0KPj5BaUxDSlFJam9pVjJsdU16 SWlMQ0pCVGlJNklrMWhhV3dpTENKWFZDSTZNbjAlM0QlN0MzMDAwJTdDJTdDJTdDJnMNCj5kDQo+ PmF0YT1MSCUyRlY3dFhFeEdzTURSVkxsWkFFcTcyWEdRRjZ2UjZaJTJCbjBUZG9VNHVlZyUzRCZy ZXNlcnZlZD0NCj4wDQo+Piwgd2hpY2ggY2FuJ3QgaGl0IHRoZSBwYW5pYyBiZWNhdXNlIGlycSA8 PSBHSUNfTEFTVF9QUEkgYWxsIHJldHVybg0KPj5lYXJseSAodGhlIHZlcnNpb24geW91IGhhdmUs IHRoZSBmaXJzdCBicmFuY2ggaW4NCj4+Z2ljX3YzX2VuYWJsZV9pbnRyX2ltcGwgd2FzIHByb2Jh Ymx5IGFuIGluY29ycmVjdCBgaWYgKGlzcmMtPmlzcmNfZmxhZ3MNCj4+JiBJTlRSX0lTUkNGX1BQ SSlgDQo+Pg0KPltTb3VyYWRlZXBdIEl0IGhhcyB3b3JrZWQhIFRoYW5rcyBhIGxvdC4gV2Ugc2hv dWxkIGdldCBpdCBjb21taXR0ZWQgaW4gc3JjLg0KPkJlZm9yZSBJIGNvbW1pdCB0aGUgU01QIHJl bGF0ZWQgY2hhbmdlcyBmb3IgSHlwZXItViBkcml2ZXIuDQo+PlRoYW5rcywNCltTb3VyYWRlZXBd IEJ1dCBhZnRlciBpbnN0YWxsIGl0IGlzIGtlZXAgZ2V0dGluZyByZWJvb3RlZCBhZnRlciBoaXR0 aW5nIGEgcGFuaWMNCg0KDQptYnVzMDogdm1idXNfaGFuZGxlX2ludHIxIGZvciBjcHUgMA0Kdm1i dXMwOiB0aGUgaXJxIDE4DQp2bWJ1czA6IHNtcF9zdGFydGVkID0gMA0KS0R2bWJ1czA6IHZtYnVz X2hhbmRsZV9pbnRyMSBmb3IgY3B1IDANCkI6IHN0YWNrIGJhY2t0cmFjZToNCmRiX3RyYWNlX3Nl bGYoKSBhdCBkYl90cmFjZV9zZWxmDQpkYl90cmFjZV9zZWxmX3dyYXBwZXIoKSBhdCBkYl90cmFj ZV9zZWxmX3dyYXBwZXIrMHgzMA0KbWlfc3dpdGNoKCkgYXQgbWlfc3dpdGNoKzB4MjdjDQpzbGVl cHFfc3dpdGNoKCkgYXQgc2xlZXBxX3N3aXRjaCsweGZjDQpfc2xlZXAoKSBhdCBfc2xlZXArMHgy OTQNCnZtYnVzX3hhY3Rfd2FpdDEoKSBhdCB2bWJ1c194YWN0X3dhaXQxKzB4MTIwDQp2bWJ1c19z eXNpbml0KCkgYXQgdm1idXNfc3lzaW5pdCsweDZhNA0KbWlfc3RhcnR1cCgpIGF0IG1pX3N0YXJ0 dXArMHgxZmMNCnZpcnRkb25lKCkgYXQgdmlydGRvbmUrMHg3MA0KS0RCOiByZWVudGVyaW5nDQpL REI6IHN0YWNrIGJhY2t0cmFjZToNCmRiX3RyYWNlX3NlbGYoKSBhdCBkYl90cmFjZV9zZWxmDQpk Yl90cmFjZV9zZWxmX3dyYXBwZXIoKSBhdCBkYl90cmFjZV9zZWxmX3dyYXBwZXIrMHgzMA0Ka2Ri X3JlZW50ZXIoKSBhdCBrZGJfcmVlbnRlcisweDQ0DQptaV9zd2l0Y2goKSBhdCBtaV9zd2l0Y2gr MHgyODANCnNsZWVwcV9zd2l0Y2goKSBhdCBzbGVlcHFfc3dpdGNoKzB4ZmMNCl9zbGVlcCgpIGF0 IF9zbGVlcCsweDI5NA0Kdm1idXNfeGFjdF93YWl0MSgpIGF0IHZtYnVzX3hhY3Rfd2FpdDErMHgx MjANCnZtYnVzX3N5c2luaXQoKSBhdCB2bWJ1c19zeXNpbml0KzB4NmE0DQptaV9zdGFydHVwKCkg YXQgbWlfc3RhcnR1cCsweDFmYw0KdmlydGRvbmUoKSBhdCB2aXJ0ZG9uZSsweDcwDQoNCkxvb2tz IGxpa2Ugdm1idXNfc3lzaW5pdCgpIGlzIGdldHRpbmcgY2FsbGVkIGV2ZW4gYmVmb3JlIFNNUCBo YXMgc3RhcnRlZCBhZnRlciBpbnN0YWxsYXRpb24uDQpXb25kZXJpbmcgd2hhdCBjb3VsZCBiZSB0 aGUgcmVhc29uLiANCkFsc28gYW55IGlkZWEgaG93IHRvIGRpc2FibGUgdGhpcyByZWJvb3QgY3lj bGUgcG9zdCBwb3N0IHBhbmljLCBzbyB0aGF0IEkgY2FuIGdldCB0aGUgYmFja3RyYWNlIHByb3Bl cmx5Lg0KPj4NCj4+S3lsZSBFdmFucw0K From nobody Mon May 15 17:35:03 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QKmjH2D2Pz4BrhB for ; Mon, 15 May 2023 17:35:15 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QKmjH1kdlz3qT5 for ; Mon, 15 May 2023 17:35:15 +0000 (UTC) (envelope-from kevans@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684172115; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZrytbN6yuhJf19NFswIrmj5KfeDeGOSBZzh2vq+TQEo=; b=hQYJ21x15Qno0dyUSJwq0wTTcpv+iWX5QBt79+kG5J5mZmRN6NRhl8Xb3HrrsgKVSpqX6m 2Ms0aLmrVZ/zkmUU68nzAYLVGXuEZBDXD7qGzPYDxeuEEwhOYGe1m6oSOH4rZXOboZksil txxkwizbNU8zrT1kdit56g2U8I6i8b4+MvgJkeVs8nsNXC2MJkXKq1Ab1YJmnWOwxfv90U QhedtEyGn002iTgqdwfV4fCIntZTuu2TrSr2YICVhLAGYJ01b9evasVgU3YbxXK/iy4dqY IURXfjboK3hzAcnifvbKnTT1P0aaMbR4deWiGwJs101gnoDxmbqXZWgvMLCtAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684172115; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZrytbN6yuhJf19NFswIrmj5KfeDeGOSBZzh2vq+TQEo=; b=G5GzZ1O9XjwcJR3nPRD0R4AT8fWNpU1Mkb5wOCNkJyhudQlIy+tnMUXNocRedYZIHeqwTD G+Qyo49GoHyKc6oh1P4s8jggYz/CvFoedbD+1XAHZplv8K5Db+bwAaTUBZvQNKcUL4u+yh XNsvvR/U8GiJKEPRYopvqfxZ1+u9eafGdhABYN5/pp2AE0QwaR/eivoguhwHzU8GkmI5lC agTNf31IF+IEhlbFxMxwJDwpnEIByYUGhNVHPhr8F6NrMZ3Rvdk6maFPx9W2YaHnO+d1IO 0LPw22loJNGJXWKpEeLFk/KqjmOYoTBvb6MmAVhi57/UVtHaCPX/2p9rzFaRBQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1684172115; a=rsa-sha256; cv=none; b=E7pvsTF8KuQpV4MMCc2lelAhVmUr5Grmtt15A47CoZzoYzpl5b+xph4t5+VyipO2J4pOGz pNjEw5709fZu8p+szlsI4RTVebzZVuKBGOVwTPb7DDYRUpy9da6WR9YMzg1LLt33qOzLQi Tl/ofJoQCbEbYolrN8W/lRWwz4zKXl8AJ4+dAXTnQZqVu/ax2EQWwc5u+xBX8TFNTOEBCA ch+WUzdIsS2xt7vxt4NoIWvHybPXH356gMylczOXw2T8rPhvMAFVFnWlILayDK+hZrB+Pt 14PHLsb2xxnQ1X7Xk+9JKJa+9b7qPsUU84bXrJHxodpbiSA1Pf2YJbuB4uHZhA== Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 4QKmjH0ggXz13q1 for ; Mon, 15 May 2023 17:35:15 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-75774360e46so500181585a.2 for ; Mon, 15 May 2023 10:35:15 -0700 (PDT) X-Gm-Message-State: AC+VfDwwiS1gjycvYvVW0EnwxKWfDc6f5JfRAw/Ce3IABtd2n1Snpwjv 9gDHnvvUSEMD0y/E+r/p8lExwcbpoxV6dmOrlWY= X-Google-Smtp-Source: ACHHUZ5Uvbgliy5Ba9wlg2qf4ZaxZCEx9qp3CyfsQhthwbppJuo2m8pJSCip1GtH6Th6DcI+xXMEJ+uRaHT3dzp37/o= X-Received: by 2002:ad4:5ecb:0:b0:615:53c3:f32a with SMTP id jm11-20020ad45ecb000000b0061553c3f32amr60567983qvb.42.1684172114673; Mon, 15 May 2023 10:35:14 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Kyle Evans Date: Mon, 15 May 2023 12:35:03 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 SMP To: Souradeep Chakrabarti Cc: Wei Hu , "freebsd-hackers@FreeBSD.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-ThisMailContainsUnwantedMimeParts: N On Mon, May 15, 2023 at 12:27=E2=80=AFPM Souradeep Chakrabarti wrote: > > > > > >-----Original Message----- > >From: owner-freebsd-hackers@freebsd.org > >On Behalf Of Souradeep Chakrabarti > >Sent: Monday, May 15, 2023 10:37 PM > >To: Kyle Evans > >Cc: Wei Hu ; freebsd-hackers@FreeBSD.org > >Subject: RE: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in AR= M64 SMP > > > > > > > > > >>-----Original Message----- > >>From: Kyle Evans > >>Sent: Monday, May 15, 2023 7:24 PM > >>To: Souradeep Chakrabarti > >>Cc: Wei Hu ; freebsd-hackers@FreeBSD.org > >>Subject: Re: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in > >>ARM64 SMP > >> > >[Souradeep] It has worked! Thanks a lot. We should get it committed in s= rc. > >Before I commit the SMP related changes for Hyper-V driver. > >>Thanks, > Awesome- I posted the PPI patch for review here: https://reviews.freebsd.org/D40112 -- I'm travelling to Ottawa for most of today and tomorrow, but we'll see if we can get this taken care of this week. > [Souradeep] But after install it is keep getting rebooted after hitting a= panic > > > mbus0: vmbus_handle_intr1 for cpu 0 > vmbus0: the irq 18 > vmbus0: smp_started =3D 0 > KDvmbus0: vmbus_handle_intr1 for cpu 0 > B: stack backtrace: > db_trace_self() at db_trace_self > db_trace_self_wrapper() at db_trace_self_wrapper+0x30 > mi_switch() at mi_switch+0x27c > sleepq_switch() at sleepq_switch+0xfc > _sleep() at _sleep+0x294 > vmbus_xact_wait1() at vmbus_xact_wait1+0x120 > vmbus_sysinit() at vmbus_sysinit+0x6a4 > mi_startup() at mi_startup+0x1fc > virtdone() at virtdone+0x70 > KDB: reentering > KDB: stack backtrace: > db_trace_self() at db_trace_self > db_trace_self_wrapper() at db_trace_self_wrapper+0x30 > kdb_reenter() at kdb_reenter+0x44 > mi_switch() at mi_switch+0x280 > sleepq_switch() at sleepq_switch+0xfc > _sleep() at _sleep+0x294 > vmbus_xact_wait1() at vmbus_xact_wait1+0x120 > vmbus_sysinit() at vmbus_sysinit+0x6a4 > mi_startup() at mi_startup+0x1fc > virtdone() at virtdone+0x70 > > Looks like vmbus_sysinit() is getting called even before SMP has started = after installation. > Wondering what could be the reason. > Also any idea how to disable this reboot cycle post post panic, so that I= can get the backtrace properly. Odd- I thought we had pushed that back to SI_SUB_SMP late or SI_SUB_SMP + 1. The backtrace here looks reasonably complete- mi_startup() does invoke sysinits directly. Can you post your full diff against sys/dev/hyperv somewhere, please? Thanks, Kyle Evans From nobody Mon May 15 18:35:49 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QKp3P4ZHfz49wM3 for ; Mon, 15 May 2023 18:36:01 +0000 (UTC) (envelope-from schakrabarti@microsoft.com) Received: from APC01-TYZ-obe.outbound.protection.outlook.com (mail-tyzapc01on2106.outbound.protection.outlook.com [40.107.117.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QKp3P1W8Zz3wKM; Mon, 15 May 2023 18:36:01 +0000 (UTC) (envelope-from schakrabarti@microsoft.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZgR3xUTbeT74hmO9ePHjJ7cmhwpuEgzkiCk7c2mw4UXObqCOS48QiTRcoogTdVQYt0KB4p/Qzc0GZtuYnkMOXBlBzmpHZnz2CGC7VrY7hzt8qnQtVy4W+bv95qu22ZX2SsI/aRRCYeOhaQMePrYFtuayMpU/Jq31qHDk4AmV+YbPV0JUqQ+5H8NW/Vt7PJMleeOzR+vtHK38RFPslGga/WCuQDLHIJf9vcJf/E2SVnqkyKkzgaNm8Xe/JpClBAF3FYaTrzNhNPzyyOL7bR/m3BGSBNNbieAb9uzkByGAKXjM+x0+7esNujKk//HmdcYGgVRub+g+JKXLybqZ1NrJaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=62kVNRm+VBMn0M3vz4jwzXdELK3yerb1HQ4LiAvFkYM=; b=NDQj44/rFe6b+aw9qj0KvTMEP1V4kVqy4IgZf+6cLMnIl6c2BuWPZOqM+Nci43T/TKZM6ZOak9Y98blP6NpwAtie/z8Nf5dPU/vOrzafHlMjmnQz50NtgDdgy3dKKLLR5DFWZrZ8PHQixSBdj3aR2UIozXgtiKB2oODDzlpzyvX0Rr9d20AALUcHwA2gItqVoHUIo6GULgINNjyOQ8SmtzKjf9zTYKIk/F/obEjom3WE3aDGNDYX2kLksISWZq7C7VboI8KlJUfxIa3Funo+n//lrzDrsGnRXgrufZ86I89AKS2aOLQXkDhoQGhEPa1GT6YcYlxs06oz9TOdvu98cQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=62kVNRm+VBMn0M3vz4jwzXdELK3yerb1HQ4LiAvFkYM=; b=GjpOubQZRkY68kNf7DjGZjW2HuPBRtot899w9crhq+xwonRq6dI30GMjpHYPRUuhqjHbFYMgbmZuXaRD0Ezq2U0xxRDNblWadW7VG/vLZIi0L3Z401eud7SX914PzJuK2bxrcQ/4htHRi7qC8ht9+sCxogdOxWmaRRoeaEkCETw= Received: from PSAP153MB0536.APCP153.PROD.OUTLOOK.COM (2603:1096:301:75::14) by TYZP153MB0967.APCP153.PROD.OUTLOOK.COM (2603:1096:400:41b::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6433.4; Mon, 15 May 2023 18:35:51 +0000 Received: from PSAP153MB0536.APCP153.PROD.OUTLOOK.COM ([fe80::3636:5810:fb12:6d71]) by PSAP153MB0536.APCP153.PROD.OUTLOOK.COM ([fe80::3636:5810:fb12:6d71%7]) with mapi id 15.20.6433.001; Mon, 15 May 2023 18:35:49 +0000 From: Souradeep Chakrabarti To: Kyle Evans CC: Wei Hu , "freebsd-hackers@FreeBSD.org" Subject: RE: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 SMP Thread-Topic: [EXTERNAL] Re: enabling same PPI interrupt to all CPU in ARM64 SMP Thread-Index: AdmBrWHu4SCK2cRnRnyy6MBGajO35QDM7LrAAATdMgAAXrUlgAAe4ZoAAAUxoMAADULwAAAGs2kgAAB9DCAAAIb9gAAB8+qQ Date: Mon, 15 May 2023 18:35:49 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=6f60fd10-3517-43bc-8ad6-c7ac3863c296;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-05-15T18:30:57Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PSAP153MB0536:EE_|TYZP153MB0967:EE_ x-ms-office365-filtering-correlation-id: 644d085d-46d6-46a4-ebd3-08db55733478 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: xqnTxdKsx7eJk09kLxx9SmWK+nDcYTULzZ7A41SfVX8xruyZesW2fp95qB1uFKjf0Hy+qSnFSQOZ8oW8Hs3Qkbj81kZA3Xhf1WeAtUOslRXwd4xweaqRhn62/waCqj+45mfoYwfMHoGD0/Lz5irHjipCkzj1x9LkQZS/wsSZexiN1/3KRrLQ4UQIfB70A7JChSslgJRARnEMesnGvGpAPtD2WSB4+Db0S7M+VUW7A2sFPvqB4AFiRVoeAXkJDFyGs2ixHmiZK/gk6rRFX5UcTVUCu59/lNrmhwDstkp57Jvnc9BhrB4yPTMWNvCk1TMsFQ5diOP14hls1vM8cRfoKnHMSHzwP8bDnYpnY4QuHRut/YGhw8frTcm+K5B4jOPke8vkS+GSc1RApWLZFXsn4C4vLJcNW0mHM1u6iMMMQLTgOX/ki8i1BqkVo03H9EHLmMZ04XHli6dQ/6UCAvAd461IKZH3tAJh0AOF9K4ckIjKSN83FnUv7Wf3Xd7wtRUq9dYMze4QsRfrEJDNigRLEqBKlpNKmvhVquc58d7At2+OozpbsPB0Z6oix6/MO1eZoS+281VNoaPFEYHCzuSgZa/X3AHBBokNnijJs5viusU0S0v0eoQikWcyfi+c4j5E4M7jWUvjm3VuT1DrQt6EbLJj/rKAgsOOyXEHXL8BMG5lWZ8kDamIPHwlL8J35lqy x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PSAP153MB0536.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230028)(4636009)(39860400002)(396003)(376002)(366004)(136003)(346002)(451199021)(5660300002)(52536014)(41300700001)(71200400001)(2906002)(83380400001)(8990500004)(82960400001)(122000001)(86362001)(82950400001)(38070700005)(38100700002)(33656002)(26005)(6506007)(186003)(55016003)(8676002)(8936002)(9686003)(966005)(450100002)(7696005)(66946007)(66556008)(66476007)(66446008)(64756008)(76116006)(54906003)(478600001)(10290500003)(4326008)(786003)(316002)(6916009)(10126625003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?Ylpjb3RVUlowMXdlWEZDOW1ya3R1UVhXaE5KcWI5MEFNM0YzbzBLTVowMUFX?= =?utf-8?B?UnVVWTBNQmdvelBQS3BJbEhaQUxGZFFibDIvd295YUpCY0RxczlEQ0FmRHh4?= =?utf-8?B?dy9hWGgxak5HRklQRyt5djNvVGp0WWk2cXBteW4rSFExaWhHSGh2dmduV2JY?= =?utf-8?B?ekdMa2o0eXc0YVVHZjNPdjB1NHRaS1Y5N1NrbUsyTy9adE0zVlB2WlZxVDZE?= =?utf-8?B?TG9KUUJuY0ViUWxENktSWnV1RDR6cWEvQnN1SVNwczNrMGJ0RjF0TUFubkdt?= =?utf-8?B?UmZMYitYUjh2dWx3c1RhSm5JQ1BweTg0SXMwa2cwaXB6eTBHb3ZuOHNSUWFH?= =?utf-8?B?VFRPUFhRc0JmamxRNzErd0dUVnNtM0NwZHo2enlpMjE1dEhheVRMMWV4WHY4?= =?utf-8?B?YUw5bSthb0ZiOFRHd0Z0bFNqZlpQV0F0UmNINFV1MGdIVnR2ckJjK29UUFpX?= =?utf-8?B?elZlTUZ3bjdsOXpYQnZuaFBPWWIrWExhMytNbWZncTBNUEtXS1NVbklCOTZj?= =?utf-8?B?dU9aQWt4UFQyVjB3R0Z3eDd2bWRRVC9oR0Qxc0RVQzl4bjNzcnErNFJ6MXlh?= =?utf-8?B?VGMydVlTUUZhMVJCYmQzZkxDN1pFVUR0aDgrcEtvVm4rTitoOGhsTmNDRkJs?= =?utf-8?B?elhrS05xWnZrVjBKWTlSa0prYXBsY3BEZzh0KzhlWU9RYnB4dnFibXdIS0Ny?= =?utf-8?B?VlZXRG1zdVBaQUtHTkFWM3FybEpNN3VqZFRTNkNpWks5NFIxVkZtRk5CMjhY?= =?utf-8?B?Z0cwWGg2K3hMQ0s2dEtUbEhZOEtTU21hOTljekxSOUZnZ3o4VFpWbHZuZXpn?= =?utf-8?B?engwb1N6SnFUZEp0Y2dUdlo3c0VqQ3BRdjVkVEJ6QUJPTmFWQ1JPR1VXRFZX?= =?utf-8?B?OEd5ZFFSTW9ER3VqTVhXZEZ2WXBJbWhQZ1RQcTV5UzJBOGRuNmVMWWdoOUV1?= =?utf-8?B?L0RYOHRRVlR5Z1JabExnL3pOM1BINDFlSWJVYXBqRDBKSGNGd0ticUlrb3hq?= =?utf-8?B?Uk4waFdmNFdsamFzNGdEVHIvUzE4Ri8xWE9udWlkNDVQU2ZBbTdWbU8rdVYr?= =?utf-8?B?MXByTFJtaWdNc3EydTVFK0VPTWViQlBJR01rZ0wrUVpQdGJNVTlRdEdxL2cw?= =?utf-8?B?OWF4aUNmQVgzVlg4REdBaENhcjFWVUJtV2JtNVdsK0dDQmE2YllUTEk2czdj?= =?utf-8?B?QTZvSnN4Z3dnbXVuS3oyOUlyelJ1Wmd2NGZZaGhmVnM2UlJGNVB2NzdjK3Va?= =?utf-8?B?OWZOM0t0eW8zOTNUMTFqcmFzU1c4dFpCdktWS0NxK3UwN1BGZTN5cUNUYVJO?= =?utf-8?B?MUZZcGt5NEg4MU9QcTVCRk4yWnplZEZlaVJvZGNLcTVIeExBeWUxSWRld1Vx?= =?utf-8?B?YVVwRVNEcGtLVWd3aGgwVUlGdmpoclQxUFloYmx3TnBBdG5URysxRCs4b1d1?= =?utf-8?B?bEhLVzR3cTRCYWFzdC9ETkg0V1VqQkVOWmYrSUR2ZWVqMStIblI2dWlHVGFU?= =?utf-8?B?SDcwelY1WmFNZFYzY2JrZjBGcEJOcjQ0N0FFSm44MUczRnhWRUlqM0NQNGt5?= =?utf-8?B?cENWMFYyMlhyalUrMFR2ZCtBMXNxaWlwR001V1ZJbGdwTFJUckZiRncvTllY?= =?utf-8?B?R24wOVhMNEhJSnp0WWxyVTFkd2dtd2R5VzdsSWNkSXBJWnVvQzNGZTJHT1F4?= =?utf-8?B?NjFEcllzQXJYTm1Zek0yaHFtU3N3a1puUURNNjBKRUdKKzRLV2N1ZHc0WG80?= =?utf-8?B?RGdraEhaN1JwRzhReHhRSWlXdFUxM0ZEVVJzazRxcUhZcVNJZ1dDQldNeVM2?= =?utf-8?B?eHNOaGJsbVFZeGlVbnZLejBWbjlEWHU4SEQrRHZOQUVWbnpUZjcyenN6cnB4?= =?utf-8?B?RWJWWEJSK0gvU2tZMGNoVnNwZVJFTUtITEdhTkxvZ2NwSjc3b2R3b3FGdUJl?= =?utf-8?B?akFiTHdRN1BBaEtWZUFTaDh5Ti9hakRtQVNCZ210YU5UMTh3ajA2WEk5RHU5?= =?utf-8?B?c0JmdWdsQWVxUG5kUjJCVkdMbmE2YkN5OHNLRmRSU0NPMWZHWE5wUHZCeWlu?= =?utf-8?Q?gy2y3M?= Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PSAP153MB0536.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 644d085d-46d6-46a4-ebd3-08db55733478 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 May 2023 18:35:49.2498 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ykZBhbdv7bMHpiPMSFAkoj18MNYZAQrTY/sB+kNNN0az4ugZwloEsiqq4tzx4RzCl8Nydws31VP6SKpArVTj3r1NSornUwn4XyajWGlofbM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYZP153MB0967 X-Rspamd-Queue-Id: 4QKp3P1W8Zz3wKM X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N DQoNCg0KPi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+RnJvbTogS3lsZSBFdmFucyA8a2V2 YW5zQGZyZWVic2Qub3JnPg0KPlNlbnQ6IE1vbmRheSwgTWF5IDE1LCAyMDIzIDExOjA1IFBNDQo+ VG86IFNvdXJhZGVlcCBDaGFrcmFiYXJ0aSA8c2NoYWtyYWJhcnRpQG1pY3Jvc29mdC5jb20+DQo+ Q2M6IFdlaSBIdSA8d2VoQG1pY3Jvc29mdC5jb20+OyBmcmVlYnNkLWhhY2tlcnNARnJlZUJTRC5v cmcNCj5TdWJqZWN0OiBSZTogW0VYVEVSTkFMXSBSZTogZW5hYmxpbmcgc2FtZSBQUEkgaW50ZXJy dXB0IHRvIGFsbCBDUFUgaW4gQVJNNjQgU01QDQo+DQo+T24gTW9uLCBNYXkgMTUsIDIwMjMgYXQg MTI6MjfigK9QTSBTb3VyYWRlZXAgQ2hha3JhYmFydGkNCj48c2NoYWtyYWJhcnRpQG1pY3Jvc29m dC5jb20+IHdyb3RlOg0KPj4NCj4+DQo+Pg0KPj4NCj4+ID4tLS0tLU9yaWdpbmFsIE1lc3NhZ2Ut LS0tLQ0KPj4gPkZyb206IG93bmVyLWZyZWVic2QtaGFja2Vyc0BmcmVlYnNkLm9yZw0KPj4gPjxv d25lci1mcmVlYnNkLWhhY2tlcnNAZnJlZWJzZC5vcmc+DQo+PiA+T24gQmVoYWxmIE9mIFNvdXJh ZGVlcCBDaGFrcmFiYXJ0aQ0KPj4gPlNlbnQ6IE1vbmRheSwgTWF5IDE1LCAyMDIzIDEwOjM3IFBN DQo+PiA+VG86IEt5bGUgRXZhbnMgPGtldmFuc0BmcmVlYnNkLm9yZz4NCj4+ID5DYzogV2VpIEh1 IDx3ZWhAbWljcm9zb2Z0LmNvbT47IGZyZWVic2QtaGFja2Vyc0BGcmVlQlNELm9yZw0KPj4gPlN1 YmplY3Q6IFJFOiBbRVhURVJOQUxdIFJlOiBlbmFibGluZyBzYW1lIFBQSSBpbnRlcnJ1cHQgdG8g YWxsIENQVSBpbg0KPj4gPkFSTTY0IFNNUA0KPj4gPg0KPj4gPg0KPj4gPg0KPj4gPg0KPj4gPj4t LS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPj4gPj5Gcm9tOiBLeWxlIEV2YW5zIDxrZXZhbnNA ZnJlZWJzZC5vcmc+DQo+PiA+PlNlbnQ6IE1vbmRheSwgTWF5IDE1LCAyMDIzIDc6MjQgUE0NCj4+ ID4+VG86IFNvdXJhZGVlcCBDaGFrcmFiYXJ0aSA8c2NoYWtyYWJhcnRpQG1pY3Jvc29mdC5jb20+ DQo+PiA+PkNjOiBXZWkgSHUgPHdlaEBtaWNyb3NvZnQuY29tPjsgZnJlZWJzZC1oYWNrZXJzQEZy ZWVCU0Qub3JnDQo+PiA+PlN1YmplY3Q6IFJlOiBbRVhURVJOQUxdIFJlOiBlbmFibGluZyBzYW1l IFBQSSBpbnRlcnJ1cHQgdG8gYWxsIENQVQ0KPj4gPj5pbg0KPj4gPj5BUk02NCBTTVANCj4+ID4+ DQo+PiA+W1NvdXJhZGVlcF0gSXQgaGFzIHdvcmtlZCEgVGhhbmtzIGEgbG90LiBXZSBzaG91bGQg Z2V0IGl0IGNvbW1pdHRlZCBpbiBzcmMuDQo+PiA+QmVmb3JlIEkgY29tbWl0IHRoZSBTTVAgcmVs YXRlZCBjaGFuZ2VzIGZvciBIeXBlci1WIGRyaXZlci4NCj4+ID4+VGhhbmtzLA0KPj4NCj4NCj5B d2Vzb21lLSBJIHBvc3RlZCB0aGUgUFBJIHBhdGNoIGZvciByZXZpZXcgaGVyZToNCj5odHRwczov L3Jldmlld3MuZnIvDQo+ZWVic2Qub3JnJTJGRDQwMTEyJmRhdGE9MDUlN0MwMSU3Q3NjaGFrcmFi YXJ0aSU0MG1pY3Jvc29mdC5jb20lN0NmZGENCj40MzczYTNiY2M0YzYwOWQ0MDA4ZGI1NTZhYmYw OSU3QzcyZjk4OGJmODZmMTQxYWY5MWFiMmQ3Y2QwMTFkYjQ3JQ0KPjdDMSU3QzAlN0M2MzgxOTc2 ODkyMzk3MTAwODglN0NVbmtub3duJTdDVFdGcGJHWnNiM2Q4ZXlKV0lqb2kNCj5NQzR3TGpBd01E QWlMQ0pRSWpvaVYybHVNeklpTENKQlRpSTZJazFoYVd3aUxDSlhWQ0k2TW4wJTNEJTdDMzAwMCUN Cj43QyU3QyU3QyZzZGF0YT05enZLQSUyQm5tUzFqNkVOUWlQNGJMeGQxbFRDV21NaHhXWVRENHdB STkwa2MlMw0KPkQmcmVzZXJ2ZWQ9MCAtLSBJJ20gdHJhdmVsbGluZyB0byBPdHRhd2EgZm9yIG1v c3Qgb2YgdG9kYXkgYW5kIHRvbW9ycm93LCBidXQgd2UnbGwNCj5zZWUgaWYgd2UgY2FuIGdldCB0 aGlzIHRha2VuIGNhcmUgb2YgdGhpcyB3ZWVrLg0KPg0KPj4gW1NvdXJhZGVlcF0gQnV0IGFmdGVy IGluc3RhbGwgaXQgaXMga2VlcCBnZXR0aW5nIHJlYm9vdGVkIGFmdGVyDQo+PiBoaXR0aW5nIGEg cGFuaWMNCj4+DQo+Pg0KPj4gbWJ1czA6IHZtYnVzX2hhbmRsZV9pbnRyMSBmb3IgY3B1IDANCj4+ IHZtYnVzMDogdGhlIGlycSAxOA0KPj4gdm1idXMwOiBzbXBfc3RhcnRlZCA9IDANCj4+IEtEdm1i dXMwOiB2bWJ1c19oYW5kbGVfaW50cjEgZm9yIGNwdSAwDQo+PiBCOiBzdGFjayBiYWNrdHJhY2U6 DQo+PiBkYl90cmFjZV9zZWxmKCkgYXQgZGJfdHJhY2Vfc2VsZg0KPj4gZGJfdHJhY2Vfc2VsZl93 cmFwcGVyKCkgYXQgZGJfdHJhY2Vfc2VsZl93cmFwcGVyKzB4MzANCj4+IG1pX3N3aXRjaCgpIGF0 IG1pX3N3aXRjaCsweDI3Yw0KPj4gc2xlZXBxX3N3aXRjaCgpIGF0IHNsZWVwcV9zd2l0Y2grMHhm Yw0KPj4gX3NsZWVwKCkgYXQgX3NsZWVwKzB4Mjk0DQo+PiB2bWJ1c194YWN0X3dhaXQxKCkgYXQg dm1idXNfeGFjdF93YWl0MSsweDEyMA0KPj4gdm1idXNfc3lzaW5pdCgpIGF0IHZtYnVzX3N5c2lu aXQrMHg2YTQNCj4+IG1pX3N0YXJ0dXAoKSBhdCBtaV9zdGFydHVwKzB4MWZjDQo+PiB2aXJ0ZG9u ZSgpIGF0IHZpcnRkb25lKzB4NzANCj4+IEtEQjogcmVlbnRlcmluZw0KPj4gS0RCOiBzdGFjayBi YWNrdHJhY2U6DQo+PiBkYl90cmFjZV9zZWxmKCkgYXQgZGJfdHJhY2Vfc2VsZg0KPj4gZGJfdHJh Y2Vfc2VsZl93cmFwcGVyKCkgYXQgZGJfdHJhY2Vfc2VsZl93cmFwcGVyKzB4MzANCj4+IGtkYl9y ZWVudGVyKCkgYXQga2RiX3JlZW50ZXIrMHg0NA0KPj4gbWlfc3dpdGNoKCkgYXQgbWlfc3dpdGNo KzB4MjgwDQo+PiBzbGVlcHFfc3dpdGNoKCkgYXQgc2xlZXBxX3N3aXRjaCsweGZjDQo+PiBfc2xl ZXAoKSBhdCBfc2xlZXArMHgyOTQNCj4+IHZtYnVzX3hhY3Rfd2FpdDEoKSBhdCB2bWJ1c194YWN0 X3dhaXQxKzB4MTIwDQo+PiB2bWJ1c19zeXNpbml0KCkgYXQgdm1idXNfc3lzaW5pdCsweDZhNA0K Pj4gbWlfc3RhcnR1cCgpIGF0IG1pX3N0YXJ0dXArMHgxZmMNCj4+IHZpcnRkb25lKCkgYXQgdmly dGRvbmUrMHg3MA0KPj4NCj4+IExvb2tzIGxpa2Ugdm1idXNfc3lzaW5pdCgpIGlzIGdldHRpbmcg Y2FsbGVkIGV2ZW4gYmVmb3JlIFNNUCBoYXMgc3RhcnRlZCBhZnRlcg0KPmluc3RhbGxhdGlvbi4N Cj4+IFdvbmRlcmluZyB3aGF0IGNvdWxkIGJlIHRoZSByZWFzb24uDQo+PiBBbHNvIGFueSBpZGVh IGhvdyB0byBkaXNhYmxlIHRoaXMgcmVib290IGN5Y2xlIHBvc3QgcG9zdCBwYW5pYywgc28gdGhh dCBJIGNhbiBnZXQgdGhlDQo+YmFja3RyYWNlIHByb3Blcmx5Lg0KPg0KPk9kZC0gSSB0aG91Z2h0 IHdlIGhhZCBwdXNoZWQgdGhhdCBiYWNrIHRvIFNJX1NVQl9TTVAgbGF0ZSBvciBTSV9TVUJfU01Q ICsgMS4NCj5UaGUgYmFja3RyYWNlIGhlcmUgbG9va3MgcmVhc29uYWJseSBjb21wbGV0ZS0NCj5t aV9zdGFydHVwKCkgZG9lcyBpbnZva2Ugc3lzaW5pdHMgZGlyZWN0bHkuIENhbiB5b3UgcG9zdCB5 b3VyIGZ1bGwgZGlmZiBhZ2FpbnN0DQo+c3lzL2Rldi9oeXBlcnYgc29tZXdoZXJlLCBwbGVhc2U/ DQpbU291cmFkZWVwXSAgSSBoYXZlIHBhc3RlZCBteSBkaWZmIGhlcmUgaHR0cHM6Ly9wYXN0ZWJp bi5jb20vQk5VUUxHRDEgLg0KQWxzbyB0aGlzIHRpbWUgcGFuaWMgd2FzIGRpZmZlcmVudCBhZnRl ciBpbnN0YWxsIDogaHR0cHM6Ly9wYXN0ZWJpbi5jb20vQ0VtVE0xc2EgLg0KPg0KPlRoYW5rcywN Cj4NCj5LeWxlIEV2YW5zDQo= From nobody Mon May 15 19:14:06 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QKpvg6dPLz49ysD for ; Mon, 15 May 2023 19:14:23 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-21.consmr.mail.gq1.yahoo.com (sonic310-21.consmr.mail.gq1.yahoo.com [98.137.69.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QKpvd3y5vz411Z for ; Mon, 15 May 2023 19:14:21 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=ELKOKY6Z; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1684178059; bh=0aJTkavWviQNntnKJNEnyMq8uAy9GuJit0F6/GZ1rw0=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=ELKOKY6ZMB3PrhGVEFTHQtDaQwlla8X12UF4D9IrNlKWk4xLHH/Z1yWaymO8Cfu2lV/Sfq5uGABwk9jEtL+nJi9ldMFCKpoiAmrDBBCO+4BkkKHpTDS/Ucownd6YVdtMtlBttlwvtchqS9n5Xu+zjFm4pwF5R/WNmuZ843V8o9c2XYIX13ygMZhe8nDPTe6fCHCG+liIChBd2Egb5zuoc5k1RJJEm1KwTh1O/ev4cWnw3a0OgZIXwspEg/XmSgPiNvQy98Tcj3HN2T4mf4I1AfXJBz94PY0Tn8NaYz+6G7tT+XURwH1LqpeFXUr5+fM9vpqGyHpP7RPFAaUvzsMHTg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1684178059; bh=fFN7KHaZtwkyU+rcIocszw2wslTHjST9/TaOx2T2oOD=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=DOfZgwsGdWirhHfEIRRV1vAvnZRNjIqKmzbBk0Cve1WBGAdaTwj4EfgDIKcOPvKpquSZFyI0soupnuwXTNsYQm/iyb+Z6ZsstPkLc4pMsmA5vfUkfN7IvCbAgAwS1/LbfZoTuEgruj+QSSsdUP1YhM9ZkFZD5V9uq1/zZHRxAy0hr1pCYLRSe+gm8j+cnMLN7Scr53p23Mj6SidAAEwLgAj6tk7s/HyoJ3wGnjl5qoWt7HhRXvFZzodlhHr9IjzdHJzNcZT12yZxbSc1d6+DM2+J5AOk6lFZDGBPdBezYlaH9WUN2J+lrpPBoIfw40XZYPDXjWjnInfqnh9oqep/qw== X-YMail-OSG: rZzDOwQVM1m.aPAJnWHh18C4eZdTGOdu7UDMFM5IV69h1p_X.qLz1dk1JqP.jvT ry1ESGcEJmWf3curJJ.WEcEaQocay34X6NYTeydokwBHLzOZQIG1VjypzpAwvt63Z58p38NBd_2B sUc0.2A5RfUuy4wXW0McJYRdiC7njkuEPWeTyD_MWjJqerbviN.jZREgbxVtrCFWFMzvQ4cwju4Z dsJal0AMF1cGVxVZ.Y.ZSkNcOEXR4g5pnPE452uuc..Qdf7ImXRmUQ5KvxfK_.WNF2HHeQZqid8A rNErh13Fmxww9DuMtvb74sN3zCulx94gUTbX1FIw7jjxOs.oPwWVPXV6R.Xro_XtQDJAiqbvbc80 Y3_9Zp1RO6JDZx5qHcXZ4jX9jNUmryLdH1i2M7dXDleW7jLeCI0Z6tdNzeKzcfqEuQOGJ_gEwHO2 6lDfbfZLfwEgUl5Zyr5U3x8RADSscESlZ3pcu0fUZl4EVlq3S2yG93FOqwiI3l.oEAlMe1uhMRdv 5WUe.WpQeYMBtfrxRPJNG4sfWVPAsq1Y1e36DcEeKPh3WA9X4VqROz68R196E3g2RN2Bh.BAsaC8 BWwair99x80hM7WRGSP4MB0e9oryU.kA1jGCMI_xX4P7VjAhlNndfKAoTxXbMnMhviNGgzg8yH2m 1FxYCvwvg3LtJXdCx9mm91DMVztJmU6uh5DKKfOUXntlbCh.RzeSVAafDu63kmw41W1n5BayHEOW 1SRlDhtEbJc5xa_VD88_KsHQYCQ3c94xso1sBaSwvYOAB2yS6faxaJNKanO7YUSWbSnssa7.1dp4 t4J6ktJKPFLiWkT3R._zqZY.o85vSUEX2_2mM99Z8OnXgeEsdhAjY0QqMyIkvegDO7GKm8pwT8vA uCwPOVTI3cQsHbmT.KTMDVi4CG7D7rkIkM0iQqQKRWwacR66VUFs.UfXsGhUMB5GD3JiJzwHmoJw 0O2kspLaDByfwcJYd9JpGYzRdIYdxJKj9NvJC7ScZFJpk4vJbnMXr4hNr5tPCRDzAi3HB1smJPDK YDEA3JlOV7TJEe8YYXyMhDpwySC.m2ukTR9UcTNSzOzviu6LZBGj4d93aIn9lPLONzdPQZQ7p8JC aO6UQJYfmYumrM4zQqROd.a45N_OucovPpC47vXVoDDsNw9P9VVT.0M5f50QCjb2u_zeZP2B3TH0 PKbFTFidrqJtUIHWWhTpx9OrgcDhyPHQT61WZxwnxU8XhOAepKtb1i1iPxzqryzW6ZwMjHf.pvoK 1doo0FllQl8SKJu7QV01y6ylSMyqduTnWyy8NavBx5mZhIAbeY6.zunAjQtCLGa84C5x0n.s2USu v6zgASaKK.pYI9mfnz8obgU.WDnWXwmMizBUeUT3p10KNoooyu0edp9qxjjv0mi7yolYp2iSY8c0 UAxHa1xxRCdRA1B0i_pcwYAazGhKawWj.cAluAyMDLL5Co1utwugWlbelqN.U6cPyQSK3QaHX5M. _6vdHt.w8EtviQnOVP6oUWYznQaY4YOzcShZDQg2wNYF2UIm7iz8HgzvPbtUI6aBuLmvJIXfBkl7 PX5aRkFJndYOWs.RUx1tBb1cwX7YqAMO1WhvCE1GE9cLyE_ixE.Zix7AFnRi2gQ_tKHET5kzRw6w WCNy87Jlg3y9Awod3BoHa2FQvULVzZf9YSfuYJC5y6VA30tAB1mpH4OW3CtrgxXg3et.S0nRtRvp 9_aRWrKKKExQIsWIp8Z3qEqUy7YYdhGlLHDC.v0jyqY_UZXF7_YLQD0muAeIyflYT3OpDo8SxJ85 BnBtW8tKyW__zdDG4WEDBKrb_r2.NpmEUgkUbh220AqMZEY.1SnEWrK9T4AQgN_hrj4VL.VEMQpr udo_7K4CZry7t763FWn.wdHwihPf_RSdu_mv7tEOZxN6phe4pMHL35qsGtC4wyKi5q8Vzw34c5QW AVvPiSw80aPdlCSz8KbZW_mibmoZ8xdTwlIqJA0EWSB.qUvRdgYv5rGbetWROgSpNfzNQ1jFcwDM wfCihDnvny345x5.xptmLrjmRKn3k.zqjLdPsNRz3rHO0iqJIWFNMfJa_7aXiSXNoIroVTjm3oOH z7G3Q5tRdg_n0sY.WX1.CqKibnpsMTRnUthqj.lwSgtdD9j_eH16ZK20CBPq4JWiJz3Ubpc5cRoA WgUsmOK.i3vafH0JOcBo5XahqNjNXkgiAWmesGgByOYC2sRQdbFDc6znPaDzdhaqf7tY.p4d0Gio waa1Q8l9UAbn.RCacIQWcDqbFZviKg0jZkifV_Brgv7g3Th79A.1dYpK2960jfM7eUMKNd1c2pMg - X-Sonic-MF: X-Sonic-ID: 5332199f-a968-42e0-9abd-8aaee810364f Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Mon, 15 May 2023 19:14:19 +0000 Received: by hermes--production-gq1-6db989bfb-hz24p (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID b58914027e833400d5ed5ce1870fd9e6; Mon, 15 May 2023 19:14:17 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Cores of different performance vs. time spent creating threads: Windows Dev Kit 2023 example From: Mark Millard In-Reply-To: <11EBAA22-6E0F-4B27-9799-7786E149D9B1@yahoo.com> Date: Mon, 15 May 2023 12:14:06 -0700 Cc: freebsd-arm Content-Transfer-Encoding: 7bit Message-Id: <47DE0BF6-3A16-4F87-AEEF-6D320BBC90E5@yahoo.com> References: <11EBAA22-6E0F-4B27-9799-7786E149D9B1@yahoo.com> To: FreeBSD Hackers X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.38 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.88)[-0.884]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.147:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4QKpvd3y5vz411Z X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On May 9, 2023, at 19:19, Mark Millard wrote: > First some context that reaches an oddity that seems to > be involved in the time to create threads . . . > > The Windows Dev Kit 2023 (WDK23 abbrevation here) boot reports: > > CPUs (cores) 0..3: cortex-a78c (the slower cores) > CPUs (cores) 4..7: cortex-x1c (the faster cores) > > Building a kernel explicitly via involving -mcpu= use > gets the following oddity relative to cpu numbering > when the kernel is used: > > -mcpu=cortex-x1c or -mcpu=cortex-a78c: > Benchmarking tracks that number/performance pairing. > > -mcpu=cortex-a72: > The slower vs. faster gets swapped number blocks. > > So, for -mcpu=cortex-a72 , 0..3 are the faster cores. > > This sets up for the following . . . > > But I also observe (a relative comparison of contexts > via some benchmark-like activity): > > -mcpu=cortex-x1c or -mcpu=cortex-a78c based kernel: > threads take more time to create > > -mcpu=cortex-a72 based kernel: > threads take less time to create > > The difference is not trivial for the activity involved > for this WDK23 context. > > If there is a bias as to which core(s) are involved in part > of thread creation generally, it would appear to be important > that the bias to be to the more performant cores (for what the > activity involves). The above suggests that such is possibly > not necessarily the case for FreeBSD as is. BIG/little (and > analogous?) cause this to become more relevant. > > Does this hypothesis about what type of thing is going on > fit with how FreeBSD actually works? > > As stands, I'm going to experiment with the WDK23 using > a cortex-a72 targeted kernel but a cortex-x1c/cortex-a78c > targeted world for my general operation of the WDK23. > > > Note: While the benchmark results allow seeing in plots > what traces back to thread creation time contributions, > the benchmark itself does not directly measure that time. > It is more like, the average work rate for a time changes > based on the fraction of the time involved in the thread > creations for each given problem size. The actual definition > of work here involves a mathematical quantity for a > mathematical problem (that need not be limited to computers > doing the work). > > The benchmark results are more useful for discovering that > there is something to potentially investigate than to > actually do an investigation with. > Never mind: Starting over did not reproduce the oddity. So: operator oddity/error, though I've no clue of how to reproduce the odd swap of which cpu number ranges took more vs. less time for each given size problem. (Or any other aspect that might be considered also odd, such as specific performance figures.) Retry details: I booted the WDK23 via UFS media set up for cortex-a72, media that I use for UFS activities on the HoneyComb (for example). I built the benchmark and ran it. As stands, I've only done the "cpu lock down" case. It produces less messy data by avoiding cpu migration once the lockdown completes (singleton cpuset for the thread). I'll also run the variant that does not have the cpu lock downs (standard C++ code without FreeBSD specifics added). === Mark Millard marklmi at yahoo.com From nobody Mon May 15 19:30:28 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QKqGW3w2Vz4B0kG for ; Mon, 15 May 2023 19:30:43 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-yw1-x1136.google.com (mail-yw1-x1136.google.com [IPv6:2607:f8b0:4864:20::1136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QKqGV5GXMz4475 for ; Mon, 15 May 2023 19:30:42 +0000 (UTC) (envelope-from tomek@cedro.info) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cedro.info header.s=google header.b=YBu1pfV7; spf=none (mx1.freebsd.org: domain of tomek@cedro.info has no SPF policy when checking 2607:f8b0:4864:20::1136) smtp.mailfrom=tomek@cedro.info; dmarc=none Received: by mail-yw1-x1136.google.com with SMTP id 00721157ae682-55dc3431c64so105086127b3.2 for ; Mon, 15 May 2023 12:30:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; t=1684179041; x=1686771041; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=yzqknJtRIWEFUHDm55oTUz51oYeY/w2swQuuczYjIuA=; b=YBu1pfV7anL93mgF3ULP4SM9rcnxU62pWIe/2pO3ax/mrSIivpy7ItgqQ4ratw+kjC E2LUt9c0FcUwprm7xAPii06Hd9ENWf+6szGAcHzvoJZmu0tJg64uWEAitbCGv3VvQjuw FMK67AlNGtlxUR3Gt8ZO53eZMHZx6NfSWwH3wQHrkIR4RlBUE8bJPPJKmMsGkVcdTZ+o MISiHnm2biiEdoxYHdH3hbtpcp/0Yotlb8HsC00DzwRrSQU1GQLP/4MjRkWRC9UFU8hg LfhCRsmloNShFiRkY/gwsCxjog81QmTNijJNqSm4WtwvXNIWt/vxgQWurkwI+6vDqEQu /YNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684179041; x=1686771041; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=yzqknJtRIWEFUHDm55oTUz51oYeY/w2swQuuczYjIuA=; b=czwirn1coKyDrQTVgIR+Bt8ulGdYhL0rcY70jHqxgypIAN5uQMDhIKj1+Tzmz4P4Kk bjzayR72FNrGR8sk++zEQG5CdujmlcBCQHtIL1tddgrxTArkUsryzkMhxUHTKLktFoL1 X719+ctCNIMe3amB5/rWJQMhH3EYLBJqonVmt4MBlpFHbX+Mhwr4AqVFMehW4h5takAK FylKa10PBR/77I0io8eS1PR9IksFg7Xe/ezpSB1I0jcsqO3xikUY5xQQySJ9u4aCI5HF 9KPkPRdyGlg2CK2Di0xzoJIlxYgAb4wXSqXanX0OvMZhNZcX6alnoKW+80TLXS9pHOaZ eBBA== X-Gm-Message-State: AC+VfDxVeYa++52zQRIo5qlojS0lWl9hgMMZwifV4qvTIME5qnKqLIIM ToVmNGDfrntAzXViSBt3mMfVQQ== X-Google-Smtp-Source: ACHHUZ7d1ePmXCK1QJz6OjKV3cwrYq2J1ZbvSJiycSXFjV4T97wa2GOa4R3Qku0eTt14LYEO1HeBbQ== X-Received: by 2002:a81:6604:0:b0:55a:6100:c0e6 with SMTP id a4-20020a816604000000b0055a6100c0e6mr30469668ywc.47.1684179041605; Mon, 15 May 2023 12:30:41 -0700 (PDT) Received: from mail-yb1-f176.google.com (mail-yb1-f176.google.com. [209.85.219.176]) by smtp.gmail.com with ESMTPSA id m15-20020a0dca0f000000b0055aafcef659sm39731ywd.5.2023.05.15.12.30.41 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 15 May 2023 12:30:41 -0700 (PDT) Received: by mail-yb1-f176.google.com with SMTP id 3f1490d57ef6-b9daef8681fso11363400276.1; Mon, 15 May 2023 12:30:41 -0700 (PDT) X-Received: by 2002:a25:d8c6:0:b0:ba8:18c6:a14e with SMTP id p189-20020a25d8c6000000b00ba818c6a14emr3955ybg.51.1684179040858; Mon, 15 May 2023 12:30:40 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <20230501181449.GJ1219@FreeBSD.org> In-Reply-To: <20230501181449.GJ1219@FreeBSD.org> From: Tomek CEDRO Date: Mon, 15 May 2023 21:30:28 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Delay in 14.0-RELEASE cycle and blocking items To: Glen Barber Cc: freebsd-arch , FreeBSD Hackers , FreeBSD Current , FreeBSD Release Engineering Team , FreeBSD Security Team Content-Type: multipart/alternative; boundary="0000000000007803f905fbc07d63" X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[cedro.info:s=google]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_NONE(0.00)[209.85.219.176:received,2607:f8b0:4864:20::1136:from]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; R_SPF_NA(0.00)[no SPF record]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; BLOCKLISTDE_FAIL(0.00)[209.85.219.176:server fail,2607:f8b0:4864:20::1136:server fail]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DKIM_TRACE(0.00)[cedro.info:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_FIVE(0.00)[6]; DMARC_NA(0.00)[cedro.info]; RCVD_COUNT_THREE(0.00)[4]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4QKqGV5GXMz4475 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --0000000000007803f905fbc07d63 Content-Type: text/plain; charset="UTF-8" Thanks Glen, no rush, quality first, it takes time :-) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info --0000000000007803f905fbc07d63 Content-Type: text/html; charset="UTF-8"
Thanks Glen, no rush, quality first, it takes time :-)

--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
--0000000000007803f905fbc07d63-- From nobody Mon May 15 21:16:21 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QKscc401Dz4B5wD for ; Mon, 15 May 2023 21:16:32 +0000 (UTC) (envelope-from jo@bruelltuete.com) Received: from email.jo-t.de (seppel.jo-t.de [45.132.244.126]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QKscZ4XXZz4PcS for ; Mon, 15 May 2023 21:16:30 +0000 (UTC) (envelope-from jo@bruelltuete.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bruelltuete.com header.s=bruelltuete18a header.b=TxYZIb5h; spf=pass (mx1.freebsd.org: domain of jo@bruelltuete.com designates 45.132.244.126 as permitted sender) smtp.mailfrom=jo@bruelltuete.com; dmarc=pass (policy=none) header.from=bruelltuete.com DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=bruelltuete.com; s=bruelltuete18a; t=1684185384; bh=1NUu+6Mxuvfc8E01Jro4y7FNV4Lv/IMWhTkJBX3umdc=; h=Message-ID:Date:MIME-Version:To:From:Subject:From; b=TxYZIb5hIxB/rnXVtrxgrBWYwav1OD7J+mdqsx0V0/lTCJGEFaBUAkbrweJ7vhDg0 QK52GwnIVmusQpWVoJq0K7FhJVVooq76Nsvw0EAumrCEzq1jPgZjdGLN0Kcn6z35pR t2yXwkVf8p0xVZ6c+JTIGdk6KLkRKds4tDT2HPVHjl99uZW0detJfDCXNBM71oL+Zd Wph+TXQgP7YUwPN0CDx5opTcZbXEvFu8lgrQSlHt3Te13fu9US4USptJYGQbgbt6Ez p+YZnnhV0QynP0pzRIRoCsGbrBb44usqlnJDKamuOxObaEVonJBT2GsjWHSGxMPBZf qMJwU69ZMtN5w== Message-ID: <576641a3-8b9c-fedb-67a6-a5c61a52f654@bruelltuete.com> Date: Mon, 15 May 2023 22:16:21 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-Mozilla-News-Host: news://news.gmane.io:119 Content-Language: en-GB To: FreeBSD Hackers From: Johannes Totz Subject: cpufreq & hwpstate_amd & Zen 2 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[bruelltuete.com,none]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[bruelltuete.com:s=bruelltuete18a]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@FreeBSD.org]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; ASN(0.00)[asn:197540, ipnet:45.132.244.0/22, country:DE]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[bruelltuete.com:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4QKscZ4XXZz4PcS X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Hi all, I'm poking cpufreq's hwpstate_amd to see what I can tune re performance vs power vs heat trade-off. I'm struggling with the P-state behaviour though. The code looks really straight-forward: https://github.com/freebsd/freebsd-src/blob/main/sys/x86/cpufreq/hwpstate_amd.c#L172 But enabling hwpstate_verify, it looks like P-state transitions never go as requested. For this, I'm not running powerd. In addition to the existing verify code, I've sprinkled in a few more printfs. PStateCurLim (aka MSR_AMD_10H_11H_LIMIT = 0x20) and PStateDef (aka MSR_AMD_10H_11H_CONFIG = eg 0x8000000049120890) look all reasonable. $ sysctl dev.cpu.0 dev.cpu.0.freq_levels: 3600/3960 2800/2800 2200/1980 dev.cpu.0.freq: 2800 $ sysctl dev.cpu.0.freq=3600 dev.cpu.0.freq: 2800 -> 3600 $ cat /var/log/messages [...extra printf debugging...] kernel: hwpstate0: setting P0-state on cpu0 kernel: hwpstate0: setting P1(2) -> P0 on cpu1 [...same for all the other cpus...] kernel: hwpstate0: setting P1(2) -> P0 on cpu15 This shows that cpufreq thought we were at P1 and wanted to transition to P0. But actually, the CPU was in P2 (the 2 in brackets). We want to go from P0 to P2... $ sysctl dev.cpu.0.freq=2200 dev.cpu.0.freq: 3600 -> 2200 $ cat /var/log/messages kernel: hwpstate0: setting P2-state on cpu0 kernel: hwpstate0: setting P0(1) -> P2 on cpu1 ...but CPU was in P1 at that time. Wanting to go from P2 back to P1... $ sysctl dev.cpu.0.freq=2800 dev.cpu.0.freq: 2200 -> 2800 $ cat /var/log/messages kernel: hwpstate0: setting P1-state on cpu0 kernel: hwpstate0: setting P2(2) -> P1 on cpu1 ...shows that this time the CPU really was in P2 (yeay). But it did not transition to P1, it stayed in P2 (not shown in the log). So question is: what else could be interfering with P-state? thanks, Johannes From nobody Tue May 16 09:03:50 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QL9K32QKnz4BmjW for ; Tue, 16 May 2023 09:04:07 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-54.consmr.mail.gq1.yahoo.com (sonic307-54.consmr.mail.gq1.yahoo.com [98.137.64.30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QL9K16GrJz4L9b for ; Tue, 16 May 2023 09:04:05 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=B8aTa6Xe; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.30 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1684227843; bh=uHAm/Id2CiAHOzt0dBbycNC8tIqrvrpr5tlPoU65UJs=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=B8aTa6Xe3dJjd6tPEETZ+foI8Wbguxbbj6kvl5ySN/+TM+IxRpHhzB8bYlgAoPO3RZ+XgQ+SgzX3nXh5ho5HtYJZlxPN2Jl0EqFmQ0pEKVqRaz2ONnghaQsSorTH4tNCflgTysFGxa1mic9pvRLnD4Nid2ef3W5xLBLPqU9e27cmUrcWbqFqoQznEJt+7kW+ZygVTzlJAlJaJ81AtWe3AOVjjlEApbgi/lwtb6wtDb2e5Hcr2NsiHfyiYwZXLejd8ut7gk8CWozXr9DFwwLcdMRK9dZDZFe3FcEVjiaYbnsbxwj5y55TGxjDwHDPKXZLF1pFFY/ZgPSOypbiDf2taA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1684227843; bh=sq6I+q2uLd+pRzGDtTqWbtNVf67Cc7NvM/IolWPAYeD=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=REMYnE0XLb8MQIR6S5Q+OKqNAUTY0FJTV1pYGPnCeUafoWuuB3kdL6sHAnhRz1HuErjKj3nmJrI+oEc8TkagVhW2KWqH9FB3ltLfvGYbWStIZnmSMDFLyPJAOQqmN8kYrwUdgZSyPOVhz2ZMnuybzFxzyAWenGLMxlTTuNvI5FhTtpb5qLA+kcSsrSyqYggcyaK6E+Y0zqnSyJxVjZryq0pHXbWq38YUO3PbU8RlatzKfL3z/IYKD4bC7lWJrNeYnMOw6BIsjInY02R53gP5sxvlUm5IX4P9Xh+Qcy5t1Dr4pXPLIyFOrRbq1o1fCTH62T/r8KS/wfcuNcAeYA6wEQ== X-YMail-OSG: Wsgz6dQVM1nFzJdoAKEmSNG1znuEg8HWQ5sf8UOMC6NIC2uJiXL.JrP9y.5RXF6 2D02jSBCEVUVw3f7fTz2nYYkxp6NDstXW54JOMVny9BT0laLYS48krHKRVFSVgE32Y96akiNE432 hFXwoyBJIO.x86Xh4_mA9CVwSMa.cdKBhs.ZwesXKfR4LEaurBPsw36phPd7rAgYqFylZZ4Qfern g6sVptiBBYuNlRH26mC5qbqWgSIretL6FLIpZ3gl4sCmzlkkTC1DQX1eIzQVJSKM5F23TvZA2DYu RPXxeUSgQdsgoFH35ZBnmZXUP7v3Q3GUTLYhiKqX.6d8OH5insjmuqW4.S7S0J5AAowI1M17KTDP ec9S44GbE9RMopBzrmW93LVF6d6ta3O9O4rgqG7.hI9IH1aI0ljJL73EpQvkYXJ6weYL7XPRE4M5 8Ury2kFYAZxewHGfN5epq.Yp3XY4c2Vtr_BtlC5XyVd1HmnQrraDrsewfU3C2bQN4nT4EHRhlZ8m fdDph60ai4TzKrYVzWWmA5tnYRqMtS8vXRwqusGQtuJgQXm9fVyUkWKSwGqk3_dv406nlZRoFZRs Sh2KVdpUm7LY2EfT6gR7NBvnKHTUaga7Xr6whp4eMkYQnMacys1SwmmiyZur5.5hraPOpxnjYEyx 1zEMVN6WCgMhVjmkirDKMyoADyQHodFUIzwijW9kLOFgazWsBQIs.9Bo.3.eCnmWp0RcqpCmQtBA _JBnByEM4YKbaWVUtEK1slOsnz0TxH.u26OjWW14H5mD51a1m9C4reuKO3qiQHn8fy9mswOKgS.K CdEBpXyPkv49ehvPcrK5hTKA8zeiC4s23h2OsJTFC1rSHVMiteoA25td5ZuGp_owv_foNsLUsQy6 lzmclo5LTKgZJGlGs.94Jw5DhzraO7kX61cEPyKp_Y2kuarFzOLfvc7BCklUGNNT8RcbQWixhQgz a7T_4_PSrdAzuo7sFXa_XFh2a2IwU.U_2dilaht_4W6vfrCIHrMlWUQrQeq244Dej60fUx88I4r1 MRjNOihuo1qUnGCSDXj9eOUL1yU6LUEsT.Eavt.wrfAQcN5v6m_r5SeSX0_C9nv3__QOkIh3A9XS 1tcmluZ9DsR6T7uZvxgfhMV5R3muhNB700M2PlZkcW.g.piRjiXrN4Mlh0nnEQnasl5NJMTHMu2S KulcjvBzOatBvUhJJkd753zfBQZyiTBdN6ky9jZrO0znNd1dOKIEeMeqCFconSz7xU.D82SyKAJz cyhql9rfLF8Whtv2dRNTMpIwvscZEXYzIrZCFEOQr3VGY8TrfOB_952TrBv5OjsxZo8nNnPvNHs1 VIsHbp1wwI2.fcCysRS.ZuagOvkxwi15MT6bGfYMQ_hu8kXBXMRb0QH756LyJIEQACVargqAgrXb G.45JLlhC8tAk1FHXoQivsvFIgJ2R_UBsl0WMgO4yJP_87mL3O5STJmyoA8RdvVokpqEmW.VHzoM XjuNxWa699NFGBCrqc47fQLrmNyCA7uvqq_FjCtdN8rY8BA5FT0s_E_y2EezNHiKblMlvEPWRhhB VrBkSBcI.ZKZGIo3D6T0tPJ_1myQjT7AL6axvcENs2AlO8LwqH8buv4Lm9W4pMUgBvspllpIeJC1 scaluikVf.0qz5ng8TuYWpbyE5x9gD_GwKmlxyTPGRnpRRJcJLFFwXoi.VfJZNloeQ9V51QXSt_8 G_Oe.3W54oWzNcEuB_BoDMknf8Urqn5D1UjWGqsll4Ucyayrkp7EmrpnDRDR3fopV6L8rhq3Zxrj LgXEPxZn_fLmo4NuJsKhv557rFz0b_4MXvtNi0UnWtPhTsOtZUJag2aPZDxy8g9sz28pntbvQ0BJ b7B4qsrkvZiUBZemfUKR6lA8rI9Wu.uJ_1Nj3Cpi6Qv_.FqTixE5ly7E0JgCCPN6uK.k8x0DbqRz bwsd4NO13FeLQ.VzWkJQ2oEqouWKuG_3R.jVjFuztwW5ukDintoaFyQP.UebPJ9CrQS34h5xv0YD 7eZuGxGchRH0kjO7D.PhI_1eOj_PoWI8clYVmsqGKDiGWdqCqmdDt7xHZfPElWlFh1661t1KFRan mfPJiCNUaXqDGE5liyfZ9myqaDdsQIo3l6yVRyY13IuX_h.R7uZ8XgyYpIZcs0vSU.bpA9vh2fo. GNMBswukjWGx3bVbhHKM_jwJzxG_PrZyB7zFw03.E92rsfTvhKYLBZ1U9qqiYVW83LIuozijU.eL G8hJuWjeBJTpi.6qqvvI_ouzgyCsHFKHrYlXAo95VLOYHF0QAy2xkcGZVI4USRN0.CA2HOyMOtL3 ZEw-- X-Sonic-MF: X-Sonic-ID: 7c9b72b1-7fb1-49e3-bf91-8a98be359ae6 Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Tue, 16 May 2023 09:04:03 +0000 Received: by hermes--production-gq1-6db989bfb-hz24p (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 3ef801103db473dfff557383acd5d8e8; Tue, 16 May 2023 09:04:01 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Cores of different performance vs. time spent creating threads: Windows Dev Kit 2023 example [Oddity is back!] From: Mark Millard In-Reply-To: <47DE0BF6-3A16-4F87-AEEF-6D320BBC90E5@yahoo.com> Date: Tue, 16 May 2023 02:03:50 -0700 Cc: freebsd-arm Content-Transfer-Encoding: 7bit Message-Id: <91F3816A-EFF5-462A-8580-EE5C73A0FBEB@yahoo.com> References: <11EBAA22-6E0F-4B27-9799-7786E149D9B1@yahoo.com> <47DE0BF6-3A16-4F87-AEEF-6D320BBC90E5@yahoo.com> To: FreeBSD Hackers X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.30:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; SUBJECT_HAS_EXCLAIM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[yahoo.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4QL9K16GrJz4L9b X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On May 15, 2023, at 12:14, Mark Millard wrote: > On May 9, 2023, at 19:19, Mark Millard wrote: > >> First some context that reaches an oddity that seems to >> be involved in the time to create threads . . . >> >> The Windows Dev Kit 2023 (WDK23 abbrevation here) boot reports: >> >> CPUs (cores) 0..3: cortex-a78c (the slower cores) >> CPUs (cores) 4..7: cortex-x1c (the faster cores) >> >> Building a kernel explicitly via involving -mcpu= use >> gets the following oddity relative to cpu numbering >> when the kernel is used: >> >> -mcpu=cortex-x1c or -mcpu=cortex-a78c: >> Benchmarking tracks that number/performance pairing. >> >> -mcpu=cortex-a72: >> The slower vs. faster gets swapped number blocks. >> >> So, for -mcpu=cortex-a72 , 0..3 are the faster cores. >> >> This sets up for the following . . . >> >> But I also observe (a relative comparison of contexts >> via some benchmark-like activity): >> >> -mcpu=cortex-x1c or -mcpu=cortex-a78c based kernel: >> threads take more time to create >> >> -mcpu=cortex-a72 based kernel: >> threads take less time to create >> >> The difference is not trivial for the activity involved >> for this WDK23 context. >> >> If there is a bias as to which core(s) are involved in part >> of thread creation generally, it would appear to be important >> that the bias to be to the more performant cores (for what the >> activity involves). The above suggests that such is possibly >> not necessarily the case for FreeBSD as is. BIG/little (and >> analogous?) cause this to become more relevant. >> >> Does this hypothesis about what type of thing is going on >> fit with how FreeBSD actually works? >> >> As stands, I'm going to experiment with the WDK23 using >> a cortex-a72 targeted kernel but a cortex-x1c/cortex-a78c >> targeted world for my general operation of the WDK23. >> >> >> Note: While the benchmark results allow seeing in plots >> what traces back to thread creation time contributions, >> the benchmark itself does not directly measure that time. >> It is more like, the average work rate for a time changes >> based on the fraction of the time involved in the thread >> creations for each given problem size. The actual definition >> of work here involves a mathematical quantity for a >> mathematical problem (that need not be limited to computers >> doing the work). >> >> The benchmark results are more useful for discovering that >> there is something to potentially investigate than to >> actually do an investigation with. >> > > Never mind: I was wrong about that . . . its back. (See later below.) > Starting over did not reproduce the oddity. So: > operator oddity/error, though I've no clue of how > to reproduce the odd swap of which cpu number ranges > took more vs. less time for each given size problem. > (Or any other aspect that might be considered also > odd, such as specific performance figures.) > > Retry details: > > I booted the WDK23 via UFS media set up for > cortex-a72, media that I use for UFS activities on > the HoneyComb (for example). I built the benchmark > and ran it. > > As stands, I've only done the "cpu lock down" case. > It produces less messy data by avoiding cpu > migration once the lockdown completes (singleton > cpuset for the thread). I'll also run the variant > that does not have the cpu lock downs (standard > C++ code without FreeBSD specifics added). I got the swapped number blocks vs. performance again, but not for cortext-a72 tailored FreeBSD, but for cortex-x1c/cortex-a78c +nolse tailored FreeBSD. Not rebooting for now, the oddity exists for the benchmark built with each of: clang 16 plus libc++ g++ 13 plus libc++ g++ 13 plus libstdc++ As before, top shows the name CPU's for STATE that the benchmark does for the cpuset based cpu id (bit numbering). As before, the measured performance for "faster" is also higher than normal. As a cross check: Avoiding use of my benchmark program . . . # cpuset -l0-3 openssl speed Doing mdc2 for 3s on 16 size blocks: 1705580 mdc2's in 3.10s . . . vs. # cpuset -l4-7 openssl speed Doing mdc2 for 3s on 16 size blocks: 1079870 mdc2's in 3.03s . . . So, openssl speed also shows the oddity: 0-3 usage being faster than 4-7 usage. The 1705580 is also somewhat large compared to a normal "4-7 is faster" context: 1705580/3.1 approx= 550187/sec . Compare to the similar calculation results in the below. For example: Shutting down, powering off, powering on, booting, and doing the openssl speed type of examples: # cpuset -l0-3 openssl speed Doing mdc2 for 3s on 16 size blocks: 997679 mdc2's in 3.09s . . . # cpuset -l4-7 openssl speed Doing mdc2 for 3s on 16 size blocks: 1360400 mdc2's in 3.02s . . . # cpuset -l0-3 openssl speed Doing mdc2 for 3s on 16 size blocks: 967253 mdc2's in 3.00s . . . # cpuset -l4-7 openssl speed Doing mdc2 for 3s on 16 size blocks: 1406978 mdc2's in 3.08s . . . So (2 similar calculations to earlier above): About 550187/sec vs. about 450463/sec and 456811/sec That is about 1.2 times faster. I've no clue about the cause or what stage(s) lead to the odd context happening. === Mark Millard marklmi at yahoo.com From nobody Fri May 19 00:29:29 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QMnm650YQz4CBx1 for ; Fri, 19 May 2023 00:29:42 +0000 (UTC) (envelope-from jo@bruelltuete.com) Received: from email.jo-t.de (seppel.jo-t.de [45.132.244.126]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QMnm45H2Dz4G29 for ; Fri, 19 May 2023 00:29:40 +0000 (UTC) (envelope-from jo@bruelltuete.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bruelltuete.com header.s=bruelltuete18a header.b="m3/ajP1Z"; spf=pass (mx1.freebsd.org: domain of jo@bruelltuete.com designates 45.132.244.126 as permitted sender) smtp.mailfrom=jo@bruelltuete.com; dmarc=pass (policy=none) header.from=bruelltuete.com DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=bruelltuete.com; s=bruelltuete18a; t=1684456174; bh=2F0/hTZUB3VmTSjENeOKyGLe5ge89eZweYlK9TA1YUM=; h=Message-ID:Date:MIME-Version:Subject:From:To:References:From; b=m3/ajP1Z6d9xWHXYY+EzA9YykNdauHXpc4G0AtqomBEHppMxooXzcg814up/wDx3x hiwCZCDpDk19Oycs6654L51pclRKJ/GG8NdLCDLgmbWJS6OcpQNUHoq5g5GJBRmuTP 5ETlNIAhI6n0W39kVW0nj45xutglI7LH6KGPiQjgainJfdro4T8SJpK3vazuZAFScJ +xnigorL/e8UI2NYcu5BESk2Ff1/9nSEMc4uD7RIh2S1lmqRKXrdxJUmG/RxlmDTUC NiflGZpk2bQeFxWeO73H1EVNFThAIE95TfOyOgNhdmDc0vPCqNTzBMv/TwFKRoBn+c 7oKANFIk5iGTg== Message-ID: Date: Fri, 19 May 2023 01:29:29 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Subject: Re: cpufreq & hwpstate_amd & Zen 2 From: Johannes Totz To: FreeBSD Hackers References: <576641a3-8b9c-fedb-67a6-a5c61a52f654@bruelltuete.com> Content-Language: en-GB In-Reply-To: <576641a3-8b9c-fedb-67a6-a5c61a52f654@bruelltuete.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.986]; DMARC_POLICY_ALLOW(-0.50)[bruelltuete.com,none]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[bruelltuete.com:s=bruelltuete18a]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@FreeBSD.org]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; ASN(0.00)[asn:197540, ipnet:45.132.244.0/22, country:DE]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[bruelltuete.com:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4QMnm45H2Dz4G29 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 15/05/2023 22:16, Johannes Totz wrote: > Hi all, > > I'm poking cpufreq's hwpstate_amd to see what I can tune re performance > vs power vs heat trade-off. Here are some patches, if anyone is interested: https://reviews.freebsd.org/D40139 Adds a tunable for cpufreq/hwpstate to get the P-state info from the CPU's MSR instead of acpi_perf. https://reviews.freebsd.org/D40158 Adds another tunable that allows overriding the default (or BIOS-configured?) P-state configuration. Stuff like over- or underclocking and -volting. https://reviews.freebsd.org/D40140 Adds power calculation if P-state info comes from MSR. This was missing until now but is really just cosmetic. These do not solve the mystery below though :( And fwiw, C-state power saving is really effective. Messing with the P-states does not do much while idle, it's measurable only when the CPU is busy. > I'm struggling with the P-state behaviour though. > The code looks really straight-forward: > https://github.com/freebsd/freebsd-src/blob/main/sys/x86/cpufreq/hwpstate_amd.c#L172 > > But enabling hwpstate_verify, it looks like P-state transitions never go > as requested. > For this, I'm not running powerd. > In addition to the existing verify code, I've sprinkled in a few more > printfs. > > PStateCurLim (aka MSR_AMD_10H_11H_LIMIT = 0x20) and PStateDef (aka > MSR_AMD_10H_11H_CONFIG = eg 0x8000000049120890) look all reasonable. > > > $ sysctl dev.cpu.0 > dev.cpu.0.freq_levels: 3600/3960 2800/2800 2200/1980 > dev.cpu.0.freq: 2800 > > $ sysctl dev.cpu.0.freq=3600 > dev.cpu.0.freq: 2800 -> 3600 > > $ cat /var/log/messages > [...extra printf debugging...] > kernel: hwpstate0: setting P0-state on cpu0 > kernel: hwpstate0: setting P1(2) -> P0 on cpu1 > [...same for all the other cpus...] > kernel: hwpstate0: setting P1(2) -> P0 on cpu15 > > > This shows that cpufreq thought we were at P1 and wanted to transition > to P0. But actually, the CPU was in P2 (the 2 in brackets). > > We want to go from P0 to P2... > > > $ sysctl dev.cpu.0.freq=2200 > dev.cpu.0.freq: 3600 -> 2200 > > $ cat /var/log/messages > kernel: hwpstate0: setting P2-state on cpu0 > kernel: hwpstate0: setting P0(1) -> P2 on cpu1 > > > ...but CPU was in P1 at that time. > > Wanting to go from P2 back to P1... > > > $ sysctl dev.cpu.0.freq=2800 > dev.cpu.0.freq: 2200 -> 2800 > > $ cat /var/log/messages > kernel: hwpstate0: setting P1-state on cpu0 > kernel: hwpstate0: setting P2(2) -> P1 on cpu1 > > > ...shows that this time the CPU really was in P2 (yeay). But it did not > transition to P1, it stayed in P2 (not shown in the log). > > > So question is: what else could be interfering with P-state? > > > thanks, > > Johannes