From owner-dev-commits-src-all@freebsd.org Tue Jan 19 20:46:44 2021 Return-Path: Delivered-To: dev-commits-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B71684DEDA6; Tue, 19 Jan 2021 20:46:44 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DL10g4rC3z4gdd; Tue, 19 Jan 2021 20:46:43 +0000 (UTC) (envelope-from avg@FreeBSD.org) X-Originating-IP: 195.64.148.76 Received: from [192.168.0.24] (unknown [195.64.148.76]) (Authenticated sender: andriy.gapon@uabsd.com) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id A1DE760002; Tue, 19 Jan 2021 20:46:40 +0000 (UTC) Subject: Re: git: 248f0cabca75 - main - make maximum interrupt number tunable on ARM, ARM64, MIPS, and RISC-V To: meloun.michal@gmail.com, John Baldwin , Jessica Clarke , Oleksandr Tymoshenko Cc: "src-committers@freebsd.org" , "dev-commits-src-all@freebsd.org" , "dev-commits-src-main@freebsd.org" References: <202101190036.10J0aj3O033256@gitrepo.freebsd.org> <8FE79D0F-602A-4C98-893A-806E72ED991B@freebsd.org> <80a3722c-16d1-0e8e-b6e1-4bdcfb554aa4@gmail.com> From: Andriy Gapon Message-ID: Date: Tue, 19 Jan 2021 22:46:37 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.6.1 MIME-Version: 1.0 In-Reply-To: <80a3722c-16d1-0e8e-b6e1-4bdcfb554aa4@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4DL10g4rC3z4gdd X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; TAGGED_RCPT(0.00)[]; ASN(0.00)[asn:29169, ipnet:217.70.176.0/20, country:FR] X-BeenThere: dev-commits-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jan 2021 20:46:44 -0000 On 2021-01-19 21:42, Michal Meloun wrote: > > > On 19.01.2021 19:41, John Baldwin wrote: >> On 1/19/21 8:43 AM, Jessica Clarke wrote: >>> On 19 Jan 2021, at 00:36, Oleksandr Tymoshenko >>> wrote: >>>> @@ -142,21 +143,15 @@ static bool irq_assign_cpu = false; >>>> #endif >>>> #endif >>>> >>>> -/* >>>> - * - 2 counters for each I/O interrupt. >>>> - * - MAXCPU counters for each IPI counters for SMP. >>>> - */ >>>> -#ifdef SMP >>>> -#define INTRCNT_COUNT   (NIRQ * 2 + INTR_IPI_COUNT * MAXCPU) >>>> -#else >>>> -#define INTRCNT_COUNT   (NIRQ * 2) >>>> -#endif >>>> +int intr_nirq = NIRQ; >>>> +SYSCTL_UINT(_machdep, OID_AUTO, nirq, CTLFLAG_RDTUN, &intr_nirq, 0, >>>> +    "Number of IRQs"); >>> >>> Unsigned integer, given the SYSCTL_UINT? >>> >>> What's stopping us from dynamically resizing rather than forcing the >>> user to use a tunable (which also means that, in practice, the limit >>> will remain unchanged, because you want GENERIC kernels to work out of >>> the box)? >> >> For x86 (and probably the same here), the biggest headache is that there >> is a dynamically allocated array indexed by IRQ value at interrupt >> invocation time.  Currently it uses no locks since the value of the >> pointer is set during boot time and then never changes.  Making it >> dynamic would require allowing that pointer to change and dealing with >> what that entails in a very hot path.  Generally speaking, the majority >> of interrupts used in a given system are allocated during boot and not >> by hotplug devices added post-boot, so there is also not a lot of utility >> in having this run-time tunable vs a boot-time tunable. >> > To be exact, the main headaches are  intrcnt and intrnames arrays and > their relationship to the sysctl interface. These fields are used > without locking, they do not have the ability to mark interrupt at given > index as unused. This is the real reason why we can't unload a driver > that acts as an interrupt controller (e.g. USB parallel port with the > possibility of interrupts on each pin). > Everything else in subr_intr.c should be prepared for dynamic > allocation/delete of interrupt sources. x86 does not use subr_intr / intrng though. I've been playing with an idea of using two-level interrupt "arrays" for things like interrupt counts and descriptiopns. That is, having fixed size chunks and a dynamically growing array of pointers to the chunks with a restriction that once a chunk is allocated it is never deallocated. So a pointer to an entry stored in such a chunk would remain stable. I also tried to use vmem(9) to keep track of allocated interrupts (interrupt sources) so that they could be dynamically created and removed. I think that it is a feasible idea, but I got lost in implementation details. -- Andriy Gapon