Date: Sat, 25 Apr 2015 11:45:10 -0700 From: Adrian Chadd <adrian@freebsd.org> To: Davide Italiano <davide@freebsd.org> Cc: "freebsd-arch@freebsd.org" <freebsd-arch@freebsd.org> Subject: Re: RFC: setting performance_cx_lowest=C2 in -HEAD to avoid lock contention on many-CPU boxes Message-ID: <CAJ-Vmo=RrcS2dYX38BW4ezMTBrY3-t6=rCnc79AwSYD2gJy55w@mail.gmail.com> In-Reply-To: <CACYV=-F_p_Pe=y%2Bs4COk%2BJf1Y8EEfxFcCKPmOXX9RE0k-KqGAA@mail.gmail.com> References: <CAJ-VmonG%2By5gzoYmer70KAswUorvezcZxRSDsQWj47=jsAZ71w@mail.gmail.com> <CACYV=-F_p_Pe=y%2Bs4COk%2BJf1Y8EEfxFcCKPmOXX9RE0k-KqGAA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 25 April 2015 at 11:18, Davide Italiano <davide@freebsd.org> wrote: > On Sat, Apr 25, 2015 at 9:31 AM, Adrian Chadd <adrian@freebsd.org> wrote: >> Hi! >> >> I've been doing some NUMA testing on large boxes and I've found that >> there's lock contention in the ACPI path. It's due to my change a >> while ago to start using sleep states above ACPI C1 by default. The >> ACPI C3 state involves a bunch of register fiddling in the ACPI sleep >> path that grabs a serialiser lock, and on an 80 thread box this is >> costly. >> >> I'd like to drop performance_cx_lowest to C2 in -HEAD. ACPI C2 state >> doesn't require the same register fiddling (to disable bus mastering, >> if I'm reading it right) and so it doesn't enter that particular >> serialised path. I've verified on Westmere-EX, Sandybridge, Ivybridge >> and Haswell boxes that ACPI C2 does let one drop down into a deeper >> CPU sleep state (C6 on each of these). I think is still a good default >> for both servers and desktops. >> >> If no-one has a problem with this then I'll do it after the weekend. >> > > This sounds to me just a way to hide a problem. > Very few people nowaday run on NUMA and they can tune the machine as > they like when they do testing. > If there's a lock contention problem, it needs to be fixed and not > hidden under another default. The lock contention problem is inside ACPI and how it's designed/implemented. We're not going to easily be able to make ACPI lock "better" as we're constrained by how ACPI implements things in the shared ACPICA code. > Also, as already noted this is a problem on 80-core machines but > probably not on a 2-core Atom. I think you need to understand factors > better and come up with a more sensible relation. In other words, your > bet needs to be proven before changing a default useful for frew that > can impact many. I've just described the differences in behaviour. I've checked the C states on all the intel servers too - with power plugged in, ACPI C2 and ACPI C3 still result in entering CPU C6 state, not CPU C7 state - so it's not going to result in worse behaviour. For reference, "all" being the following list: * westmere-EX * nehalem * sandybridge * sandybridge mobile * sandybridge xeon * ivybridge mobile * ivybridge xeon * haswell mobile * haswell * haswell xeon * haswell xeon v3 -adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmo=RrcS2dYX38BW4ezMTBrY3-t6=rCnc79AwSYD2gJy55w>