From owner-freebsd-stable@freebsd.org Sun Sep 4 16:29:33 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 787D6A9DFE3 for ; Sun, 4 Sep 2016 16:29:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 5F68F82B for ; Sun, 4 Sep 2016 16:29:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 5EC1CA9DFE2; Sun, 4 Sep 2016 16:29:33 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E6A1A9DFE1 for ; Sun, 4 Sep 2016 16:29:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E1B2582A; Sun, 4 Sep 2016 16:29:32 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u84GTR0c089020 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sun, 4 Sep 2016 19:29:27 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u84GTR0c089020 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u84GTRHv089019; Sun, 4 Sep 2016 19:29:27 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 4 Sep 2016 19:29:27 +0300 From: Konstantin Belousov To: Andriy Gapon Cc: Slawa Olhovchenkov , stable@FreeBSD.org Subject: Re: X2APIC support Message-ID: <20160904162926.GF83214@kib.kiev.ua> References: <20151212130615.GE70867@zxy.spb.ru> <20151212133513.GL82577@kib.kiev.ua> <20160901112724.GX88122@zxy.spb.ru> <20160901114500.GJ83214@kib.kiev.ua> <20160901121300.GZ88122@zxy.spb.ru> <4ba05c00-f737-f562-553d-a7fa59145768@FreeBSD.org> <20160904151415.GE83214@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.6.1 (2016-04-27) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Sep 2016 16:29:33 -0000 On Sun, Sep 04, 2016 at 06:49:43PM +0300, Andriy Gapon wrote: > On 04/09/2016 18:14, Konstantin Belousov wrote: > > On Sun, Sep 04, 2016 at 11:19:16AM +0300, Andriy Gapon wrote: > >> Kostik, I see one strange thing which is common to both successful and > >> unsuccessful configurations. All "SMP: Added CPU..." lines have "AP" in them. > >> It seems like the platform does not tell explicitly tell which CPU is the BSP, > >> see cpu_add() function. This can break quite a few assumption. And I am not > >> even sure how the successful scenario works. > >> Ah... I see that there is a backup code in cpu_mp_start() where boot_cpu_id is > >> set based on the current CPU's Local APIC ID. I suspect then that this > >> information is incorrect in the failing case. > >> > > Well, there is no easy way to read the LAPIC Id of BSP before LAPICs > > are initialized. BIOS might reprogram LAPIC Ids, so reading from > > CPUID[1].EBX[31:24] might return incorrect data. Even more incorrect > > it might be in the x2APIC state, since 8 bits are not enough for 32bit > > x2APIC Id. > > Hmm, I am not sure how what you are saying here is relevant to the problem. > I believe that cpu_mp_start() is executed (on the BSP) after the BSP's LAPIC is > initialized. So, the code should just work. The order is madt_probe()->madt_probe_cpus()->madt_setup_local(). madt_probe() and madt_probe_cpus() are called from apic_init9) at SI_SUB_TUNABLEs, and madt_probe_cpus()->madt_add_cpu()->lapic_create()-> cpu_add() is how the SMP: ... lines are printed. The madt_setup_local() code is called from apic_setup_local() at SI_SUB_CPU, this is where APIC window is mapped or x2APIC mode is enabled by the call to native_lapic_init(). You cannot get BSP LAPIC Id earlier than native_lapic_init() was executed. mp_start()->cpu_mp_start() is called right after the madt_setup_local(). > > My theory was that the BSP's LAPIC ID was incorrectly programmed by BIOS (firmware). This is possible, of course. But it would not affect "SMP: Added CPU ..." lines.