From owner-freebsd-arm@freebsd.org Sun Apr 21 18:05:29 2019 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6BC3215801C7 for ; Sun, 21 Apr 2019 18:05:29 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1a.eu.mailhop.org (outbound1a.eu.mailhop.org [52.58.109.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9774782692 for ; Sun, 21 Apr 2019 18:05:28 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1555869920; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=NXI68mORWXem7iIkOyzTKlUKnfdv6B0LadE/k3TLNH5E45OSlTrCIT8s5W9xhHqoRGqFKs1HYsgZ1 DuuGT+8JmjxSnZIXpb3gbMsBR1UidLW0vDSXwokk00I7+NP8uUZsw7NeO5sN9QTDZT01VZXfnV6VK7 S3hYQpH7KPnWyLkB8DmKGzuhHALSVSTNBaP6AnVO36nxO/ys/nuJLlcdOD+e3V7byeMcHaaHjqijcs vAvrVVYkhPY4uwdx/w5N1ujB9tpmGlkDcUEj3yJwUhrKY1DPi7pRxeBmMQKw9N6gwXSzKujNb6l+xH 6o3KZAvaISu6PgZYwLSzg+jcFH5XbtQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:dkim-signature:from; bh=rA4Eq1tLPF0K70gt0Aw2zjvsRJMrnARA0h5c6Tb2ezs=; b=NYxXhtzsfqQNTWrDfMFwh4KaPVIY+cpFuJNVVjNDioK7oXj4XErP5f6y+UQgFREufLju4rLtmRaG4 rmMj8hK+Okdh9O6kt4tcrnKv53nWhii7Jf0HkIgEShNkm9vrY/6ABGixkYRP4QpKKC/WupzkiocHe8 X/WnZGH25D8KvFmouI+Y64/T30qgYFZ0a6+c2Y9SVS3VCbetEDRXhK6RF1UnDx+xgeOyySWjwlA2Wl WzrQNyjE4axJVcIXqu8P0AdzNpRJCogk29BQYMDCehDeswbUInPZoJVflozptUtiCiDXseqoLL74sQ +LQrEwcz7zmDYFB7J4J+tfIjVG7HpXw== ARC-Authentication-Results: i=1; outbound2.eu.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:from; bh=rA4Eq1tLPF0K70gt0Aw2zjvsRJMrnARA0h5c6Tb2ezs=; b=M3t2kVXxLC0Z0JEZUyGMkiYm4G/EFqJy2c7OsI2e2+4XB6uwYbREFSq5958rtKQAv4hYL+T1mEeTu IDY6hNV21EDrD+5NrrQdWxFlhjY37DwLQUFYvGMmFY8kPq5o0IXpFuG5y3Gm3TMI68oz8kfJyzwOWk RLvOKaO3fZcdCoy1JPBqsGeF7zocmGWOC4gqrFpJLRvcKD1L2V6Dd+hFGxeVsDqV/Cafggc6Nkwkls Cz5SFKMQ9sGjEwm6lsY0rioi3Q+AgNsOyI70OE2CZk2dIK1zDgslCP5SLa/5fFQqtfCri9xu6zOW9F KMJYNxXSyr+S+NZwCfkB/lMhejigZ5w== X-MHO-RoutePath: aGlwcGll X-MHO-User: 02387791-6460-11e9-803b-31925da7267c X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound2.eu.mailhop.org (Halon) with ESMTPSA id 02387791-6460-11e9-803b-31925da7267c; Sun, 21 Apr 2019 18:05:15 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id x3LI5Caw061231; Sun, 21 Apr 2019 12:05:12 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <26962ee10cf8c61416dde40d5e8c0c24400316f9.camel@freebsd.org> Subject: Re: insanely-high interrupt rates -- PARTIAL resolution (Pi2) From: Ian Lepore To: Warner Losh Cc: Karl Denninger , Andrew Gierth , "freebsd-arm@freebsd.org" , ticso@cicely.de Date: Sun, 21 Apr 2019 12:05:12 -0600 In-Reply-To: References: <004ddba628b94b80845d8e509ddcb648d21fd6c9.camel@freebsd.org> <20190319161423.GH57400@cicely7.cicely.de> <52df098fdc0caf5de1879c93239534fffbd49b56.camel@freebsd.org> <40f57de2-2b25-3981-a416-b9958cc97636@denninger.net> <669892ac3fc37b0843a156c0ab102316829103fd.camel@freebsd.org> <663f2566-b035-7011-70eb-4163b41e6e55@denninger.net> <20190325164827.GL57400@cicely7.cicely.de> <3db9cf8a-68ee-e339-67bf-760ee51464fd@denninger.net> <874l7fyrpr.fsf@news-spur.riddles.org.uk> <701e011f-3088-8ed4-4fbb-6fa93ac698f5@denninger.net> <67133e19-2be5-ccd1-2ded-008b36a866ec@denninger.net> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5 FreeBSD GNOME Team Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 9774782692 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.984,0]; ASN(0.00)[asn:16509, ipnet:52.58.0.0/15, country:US] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Apr 2019 18:05:29 -0000 On Sun, 2019-04-21 at 12:00 -0600, Warner Losh wrote: > On Sun, Apr 21, 2019 at 11:58 AM Ian Lepore wrote: > > > On Wed, 2019-04-17 at 14:56 -0500, Karl Denninger wrote: > > > On 4/9/2019 19:25, Ian Lepore wrote: > > > > On Tue, 2019-04-09 at 09:55 -0500, Karl Denninger wrote: > > > > > On 4/3/2019 11:48, Andrew Gierth wrote: > > > > > > [...] > > > > > > > > I've just posted https://reviews.freebsd.org/D19871 for this. > > > > Hopefully I'll get it committed in a day or so and merged to > > > > 12- > > > > stable > > > > a few days after that. > > > > > > > > -- Ian > > > > > > I am running that now on a Pi2 and so far the load problem is > > > gone > > > but > > > the spurious interrupt warnings are not.... > > > > > > > > > > [...] > > > > > > On my bench without the I2c inputs connected (which do analog > > > reads) I > > > do NOT get the spurious interrupt prints. With it connected I > > > do. The > > > process that reads them is code that is running in both cases, > > > but if it > > > cannot find the I2c devices it logs the error but continues, so > > > all it > > > gets to is trying to open the unit, doesn't see it when probed, > > > and > > > gives up. > > > > > > It appears that I2c is an inherent part of the spurious interrupt > > > thing > > > still and while the timer issue appears to be fixed that doesn't > > > resolve > > > the other problem. > > > > > > Any ideas on how to track down exactly what is generating those > > > warnings? > > > > > > > > > > After spending the whole day yesterday trying all the usual driver > > techniques for eliminating spurious interrupts, I was unable to > > make > > them go away completely, but I also convinced myself they're > > harmless. > > > > I was a little surprised that the "read after write" technique > > didn't > > work. That is, after writing to the i2c control register to clear > > all > > the interrupt-enable bits, read back that register. In theory, at > > least on normal arm chips, that ensures that the prior write has > > reached the hardware before the read can procede, so it's a way to > > guarantee that the write has taken effect and the interrupt can no > > longer be asserted, before returning from the interrupt > > handler. But, > > on the rpi chips even that doesn't work... you can read back the > > register and verify the interrupt-enable bits are cleared, and > > still > > after returning from the handler, it re-interrupts immediately. > > > > If you stick in a nice long DELAY() after clearing the control > > register, the spurious interrupts go away, but that's a horrible > > fix. > > It would be especially horrible for i2c devices that do a lot of > > transfers, you'd end up with the delay time overwhelming the time > > to do > > the actual transfers themselves. > > > > So, in r346489, I moved the reporting of the spurious transfers > > under > > the bootverbose flag, so that normally you just won't see them > > anymore, > > but we can still enable the reporting if we suspect some device > > driver > > is behaving badly. I'll mfc that change to 12-stable after a few > > days. > > > > vmstat -i will also show if you're system has an unusually high interrupt > rate in general as well, and is preferable to spamming the console with > printfs :) > > Warner vmstat doesn't report spurious interrupts in any way, though. I considered making it do so as one of the possible fixes here, but it turns out to be complicated... we need to do a bit of reworking of the INTRNG code as it related to interrupt counts. For example, on x86 you get this from vmstat -i: cpu0:timer 42006521 80 cpu1:timer 32510560 62 But on arm, all timer interrupts are counted as belonging to the generic_timer0 device. When I tried to figure out how to split that into per-cpu reporting like x86 does, I discovered what a mess the intrstats stuff in INTRNG is right now. So, a project for another day, I guess. -- Ian