Date: Tue, 23 May 2023 19:57:59 +0000 From: "Abboud, Osama" <osamaabb@amazon.com> To: "freebsd-questions@FreeBSD.org" <freebsd-questions@FreeBSD.org> Subject: Interrupts Affinity on ARM systems Message-ID: <8D487376-CB7F-45DE-9A44-28F8052E1CAA@amazon.com>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] I'm using the same freebsd kernel and same network driver on ARM based instance and intel based instance. ARM based instance uname: uname -a FreeBSD freebsd 13.2-RELEASE FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC arm64 Intel based instance uname: uname -a FreeBSD freebsd 13.2-RELEASE FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC amd64 For the intel based instance, I want to see the affinity of then hardware interrupts so i run the following: $ vmstat -i interrupt total rate . . irq41: ena0 137 4 irq42: ena0 97 3 irq43: ena0 18 1 irq44: ena0 5254 159 Now after I run cpuset I get: $ cpuset -g -x 44 irq 44 mask: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 However on ARM instance I do the same and get the following: $ vmstat -i interrupt total rate gic0,p11:-ic_timer0 44021813 503 .. its0,75: ena0 580 0 its0,76: ena0 2943 0 its0,77: ena0 586 0 After running cpuset: $ cpuset -g -x 75 cpuset: getaffinity: No such process The device interrupts are allocated in the driver using bus_alloc_resource_any() & bus_setup_intr(). Why are the interrupts names different in ARM/Intel and how can I get the interrupts affinity in ARM systems? Followup question: Upon printing the cpu on which the HW interrupts are received (curcpu parameter within the msix interrupt handler) on ARM systems I can see that all interrupts are received on CPU 0. Why is that? Is it an acceptable practice in FreeBSD to spread the interrupt affinity upon different cores by default using bus_bind_intr() calls? Within the intel based instance I can see that the interrupts are spread across all CPU’s by default. [-- Attachment #2 --] <html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">; <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; font-size:11.0pt; font-family:"Calibri",sans-serif; mso-ligatures:standardcontextual;} span.EmailStyle17 {mso-style-type:personal-compose; font-family:"Calibri",sans-serif; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-family:"Calibri",sans-serif;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 72.0pt 72.0pt 72.0pt;} div.WordSection1 {page:WordSection1;} --></style> </head> <body lang="en-IL" link="#0563C1" vlink="#954F72" style="word-wrap:break-word"> <div class="WordSection1"> <p class="MsoNormal"><span style="mso-ligatures:none">I'm using the same freebsd kernel</span><span lang="EN-US" style="mso-ligatures:none"> and same network driver</span><span style="mso-ligatures:none"> on ARM based instance and intel based instance.<br> ARM based instance uname:<br> </span><span style="font-size:10.0pt;font-family:"Courier New";mso-ligatures:none">uname -a FreeBSD freebsd 13.2-RELEASE FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC arm64</span><span style="mso-ligatures:none"><br> Intel based instance uname:<br> </span><span style="font-size:10.0pt;font-family:"Courier New";mso-ligatures:none">uname -a FreeBSD freebsd 13.2-RELEASE FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC amd64</span><span style="mso-ligatures:none"><br> <br> For the intel based instance, I want to see the affinity of then hardware interrupts so i run the following:<br> </span><span style="font-size:10.0pt;font-family:"Courier New";mso-ligatures:none">$ vmstat -i <br> interrupt total rate .<br> . <br> irq41: ena0 137 4 <br> irq42: ena0 97 3 <br> irq43: ena0 18 1 <br> irq44: ena0 5254 159<br> </span><span style="mso-ligatures:none"><br> Now after I run cpuset I get:<br> </span><span style="font-size:10.0pt;font-family:"Courier New";mso-ligatures:none">$ cpuset -g -x 44 <br> irq 44 mask: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15</span><span style="mso-ligatures:none"><br> <br> However on ARM instance I do the same and get the following:<br> </span><span style="font-size:10.0pt;font-family:"Courier New";mso-ligatures:none">$ vmstat -i <br> interrupt total rate <br> gic0,p11:-ic_timer0 44021813 503 <br> .. <br> its0,75: ena0 580 0 <br> its0,76: ena0 2943 0 <br> its0,77: ena0 586 0 <o:p></o:p></span></p> <p class="MsoNormal"><span style="mso-ligatures:none"><br> After running cpuset:<br> </span><span style="font-size:10.0pt;font-family:"Courier New";mso-ligatures:none">$ cpuset -g -x 75 <br> cpuset: getaffinity: No such process</span><span style="mso-ligatures:none"><br> <br> </span><span lang="EN-US" style="mso-ligatures:none">The device interrupts are allocated in the driver using bus_alloc_resource_any() & bus_setup_intr().</span><span style="mso-ligatures:none"><br> Why are the interrupts name</span><span lang="EN-US" style="mso-ligatures:none">s</span><span lang="EN-US" style="mso-ligatures:none"> </span><span style="mso-ligatures:none">different in ARM/Intel and how can I get the interrupts affinity in ARM systems?<br> <br> Followup question:<br> Upon printing the cpu on which the HW interrupts are received (curcpu parameter within the msix interrupt handler) on ARM systems I can see that all interrupts are received on CPU 0. Why is that?<br> Is it an acceptable practice in FreeBSD to spread the interrupt affinity upon different cores by default using bus_bind_intr() calls?<br> </span><span lang="EN-US" style="mso-ligatures:none">Within the intel based instance I can see that the interrupts are spread across all CPU’s by default.<o:p></o:p></span></p> <p class="MsoNormal"><o:p> </o:p></p> </div> </body> </html>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8D487376-CB7F-45DE-9A44-28F8052E1CAA>