From owner-freebsd-amd64@FreeBSD.ORG Fri Jul 18 16:07:14 2014 Return-Path: Delivered-To: amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EEBEFF7A; Fri, 18 Jul 2014 16:07:14 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 753AF23AD; Fri, 18 Jul 2014 16:07:14 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s6IG786b093036 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 18 Jul 2014 19:07:08 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s6IG786b093036 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s6IG78YB093035; Fri, 18 Jul 2014 19:07:08 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 18 Jul 2014 19:07:08 +0300 From: Konstantin Belousov To: arch@freebsd.org Subject: KDB entry on NMI Message-ID: <20140718160708.GO93733@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="IOwL3FhNvW0Xz3At" Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: amd64@freebsd.org X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jul 2014 16:07:15 -0000 --IOwL3FhNvW0Xz3At Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable It was mentioned somewhere recently, that typical BIOS today configures NMI delivery on the hardware events as broadcast. When I developerd the dmar(4) busdma backend, I indeed met the problem, and wrote a prototype which avoided startup of ddb on all cores. Instead, the patch implements custom spinlock, which allows only one core to win, other cores ignore the NMI, by spinning on lock. The issue which I see on at least two different machines with different Intel chipsets, is that NMI is somehow sticky, i.e. it is re-delivered after the handler executes iret. I am not sure what the problem is, whether it is due to hardware needing some ACK, or a bug in code. Anyway, even on two-cores machine, having both cores simultaneously enter NMI makes the use of ddb impossible, so I believe the patch is improvement. I make measures to ensure that reboot from ddb prompt works. Thought ? diff --git a/sys/amd64/amd64/mp_machdep.c b/sys/amd64/amd64/mp_machdep.c index 9b12449..76b992a 100644 --- a/sys/amd64/amd64/mp_machdep.c +++ b/sys/amd64/amd64/mp_machdep.c @@ -32,14 +32,18 @@ __FBSDID("$FreeBSD$"); #include "opt_kstack_pages.h" #include "opt_sched.h" #include "opt_smp.h" +#include "opt_isa.h" +#include "opt_kdb.h" =20 #include #include #include +#include #include #ifdef GPROF=20 #include #endif +#include #include #include #include @@ -60,6 +64,7 @@ __FBSDID("$FreeBSD$"); =20 #include #include +#include #include #include #include @@ -164,6 +169,7 @@ static int cpu_logical; /* logical cpus per core */ static int cpu_cores; /* cores per package */ =20 static void assign_cpu_ids(void); +static void cpustop_handler_post(u_int cpu); static void set_interrupt_apic_ids(void); static int start_ap(int apic_id); static void release_aps(void *dummy); @@ -1415,26 +1421,44 @@ ipi_nmi_handler() cpustop_handler(); return (0); } - =20 -/* - * Handle an IPI_STOP by saving our current context and spinning until we - * are resumed. - */ -void -cpustop_handler(void) -{ - u_int cpu; =20 - cpu =3D PCPU_GET(cpuid); +#ifdef DEV_ISA +static int nmi_kdb_lock; +#endif =20 - savectx(&stoppcbs[cpu]); +#ifdef DEV_ISA +bool +nmi_call_kdb_smp(u_int type, int code, struct trapframe *frame, bool do_pa= nic) +{ + int cpu; + bool call_post, ret; =20 - /* Indicate that we are stopped */ - CPU_SET_ATOMIC(cpu, &stopped_cpus); + call_post =3D false; + cpu =3D PCPU_GET(cpuid); + if (atomic_cmpset_acq_int(&nmi_kdb_lock, 0, 1)) { + ret =3D nmi_call_kdb(cpu, type, code, frame, do_panic); + } else { + ret =3D true; + savectx(&stoppcbs[cpu]); + while (!atomic_cmpset_acq_int(&nmi_kdb_lock, 0, 1)) { + if (CPU_ISSET(cpu, &ipi_nmi_pending)) { + CPU_CLR_ATOMIC(cpu, &ipi_nmi_pending); + call_post =3D true; + } + cpustop_handler_post(cpu); + cpu_spinwait(); + } + } + atomic_store_rel_int(&nmi_kdb_lock, 0); + if (call_post) + cpustop_handler_post(cpu); + return (ret); +} +#endif =20 - /* Wait for restart */ - while (!CPU_ISSET(cpu, &started_cpus)) - ia32_pause(); +static void +cpustop_handler_post(u_int cpu) +{ =20 CPU_CLR_ATOMIC(cpu, &started_cpus); CPU_CLR_ATOMIC(cpu, &stopped_cpus); @@ -1450,6 +1474,25 @@ cpustop_handler(void) } =20 /* + * Handle an IPI_STOP by saving our current context and spinning until we + * are resumed. + */ +void +cpustop_handler(void) +{ + u_int cpu; + + cpu =3D PCPU_GET(cpuid); + savectx(&stoppcbs[cpu]); + /* Indicate that we are stopped */ + CPU_SET_ATOMIC(cpu, &stopped_cpus); + /* Wait for restart */ + while (!CPU_ISSET(cpu, &started_cpus)) + ia32_pause(); + cpustop_handler_post(cpu); +} + +/* * Handle an IPI_SUSPEND by saving our current context and spinning until = we * are resumed. */ diff --git a/sys/amd64/amd64/trap.c b/sys/amd64/amd64/trap.c index d9203bc..6fa576e 100644 --- a/sys/amd64/amd64/trap.c +++ b/sys/amd64/amd64/trap.c @@ -74,6 +74,7 @@ PMC_SOFT_DEFINE( , , page_fault, all); PMC_SOFT_DEFINE( , , page_fault, read); PMC_SOFT_DEFINE( , , page_fault, write); #endif +#include =20 #include #include @@ -158,6 +159,44 @@ SYSCTL_INT(_machdep, OID_AUTO, uprintf_signal, CTLFLAG= _RWTUN, &uprintf_signal, 0, "Print debugging information on trap signal to ctty"); =20 +#ifdef DEV_ISA +bool +nmi_call_kdb(u_int cpu, u_int type, int code, struct trapframe *frame, + bool do_panic) +{ + + /* machine/parity/power fail/"kitchen sink" faults */ + if (isa_nmi(code) =3D=3D 0) { +#ifdef KDB + /* + * NMI can be hooked up to a pushbutton for debugging. + */ + if (kdb_on_nmi) { + printf ("NMI/cpu%d ... going to debugger\n", cpu); + kdb_trap(type, 0, frame); + return (true); + } + } else +#endif /* KDB */ + if (do_panic) + panic("NMI indicates hardware failure"); + return (false); +} +#endif + +static int +handle_nmi_intr(u_int type, int code, struct trapframe *frame, bool panic) +{ + +#ifdef DEV_ISA +#ifdef SMP + return (nmi_call_kdb_smp(type, code, frame, panic)); +#else + return (nmi_call_kdb(0, type, code, frame, panic)); +#endif +#endif +} + /* * Exception, fault, and trap interface to the FreeBSD kernel. * This common code is called from assembly language IDT gate entry @@ -357,25 +396,9 @@ trap(struct trapframe *frame) i =3D SIGFPE; break; =20 -#ifdef DEV_ISA case T_NMI: - /* machine/parity/power fail/"kitchen sink" faults */ - if (isa_nmi(code) =3D=3D 0) { -#ifdef KDB - /* - * NMI can be hooked up to a pushbutton - * for debugging. - */ - if (kdb_on_nmi) { - printf ("NMI ... going to debugger\n"); - kdb_trap(type, 0, frame); - } -#endif /* KDB */ - goto userout; - } else if (panic_on_nmi) - panic("NMI indicates hardware failure"); + handle_nmi_intr(type, code, frame, true); break; -#endif /* DEV_ISA */ =20 case T_OFLOW: /* integer overflow fault */ ucode =3D FPE_INTOVF; @@ -543,25 +566,11 @@ trap(struct trapframe *frame) #endif break; =20 -#ifdef DEV_ISA case T_NMI: - /* machine/parity/power fail/"kitchen sink" faults */ - if (isa_nmi(code) =3D=3D 0) { -#ifdef KDB - /* - * NMI can be hooked up to a pushbutton - * for debugging. - */ - if (kdb_on_nmi) { - printf ("NMI ... going to debugger\n"); - kdb_trap(type, 0, frame); - } -#endif /* KDB */ - goto out; - } else if (panic_on_nmi =3D=3D 0) + if (handle_nmi_intr(type, code, frame, false) || + !panic_on_nmi) goto out; /* FALLTHROUGH */ -#endif /* DEV_ISA */ } =20 trap_fatal(frame, 0); diff --git a/sys/amd64/include/md_var.h b/sys/amd64/include/md_var.h index 5ddfbbd..da170f2 100644 --- a/sys/amd64/include/md_var.h +++ b/sys/amd64/include/md_var.h @@ -120,5 +120,9 @@ struct savefpu *get_pcb_user_save_td(struct thread *td); struct savefpu *get_pcb_user_save_pcb(struct pcb *pcb); struct pcb *get_pcb_td(struct thread *td); void amd64_db_resume_dbreg(void); +bool nmi_call_kdb(u_int cpu, u_int type, int code, struct trapframe *frame, + bool panic); +bool nmi_call_kdb_smp(u_int type, int code, struct trapframe *frame, + bool panic); =20 #endif /* !_MACHINE_MD_VAR_H_ */ --IOwL3FhNvW0Xz3At Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJTyUYsAAoJEJDCuSvBvK1Bl5YP/2IEO296ZUqjVYUqvcGoZm/x Ov+1xzBFsWd+M81IHKHtyQ91B5dzS4vDnTBrlQR7/PjEKpMuuscLPkXDMJ2Q3KG7 3RC/DHZO2gtNKuppUW72Wtr5bZwRc7WChFE/q48/N0f3Pan8o25ZqHtdt+j3pMlM xj3GjtH5xlnlOzHUgBUeGgZEnzA9bl1kvBYDz53j+JSqbjZCsoJpSfhq5zP0fuYM O6L+gyQhqPN7NjkVcJaG4wfsA+spVfHt72+67HWj8AbXvj6NJsNifUCtyNlTnRwf czp+j6m9NfZyC3aSgvnC6uI+txLm7ambdRuuxtzvLEsQb4f2Lst1HKPgq3yvexcI yNAwmlkCFWl7GBpUFknk9I6+MTW+bgicrnU0F49JUj247V1jTpLImQPVORD7wWC3 vhcudmCDCiB4Qj/meRMfKjaIBtGcM6OYFvbkLUKse2zHwc8vl41B/tGdECwgt3Bd ZWhj6jHm0ck6mKvNCxlqDrEFL4cXgeeaV823AEBWJi9M4zczLe+W4NIk1sihePtJ OSHBUtsK3ExytjUoV1Q7cf45G0+gJuHZkrk7V53Kty1qAd1JrHi5dC57hDOwUqyE EvXRUi4z9NYvrSxyyGOzbcNaaQMqwlTo39hsuTNUUGX5dmQjYIasby07M+9OE51b oA1RaVbv/lGOcnqaCEY0 =CQqR -----END PGP SIGNATURE----- --IOwL3FhNvW0Xz3At-- From owner-freebsd-amd64@FreeBSD.ORG Sat Jul 19 17:58:35 2014 Return-Path: Delivered-To: amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 09942120; Sat, 19 Jul 2014 17:58:35 +0000 (UTC) Received: from mail.xcllnt.net (mail.xcllnt.net [50.0.150.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9A7392FA3; Sat, 19 Jul 2014 17:58:33 +0000 (UTC) Received: from [172.29.5.30] ([66.129.239.11]) (authenticated bits=0) by mail.xcllnt.net (8.14.9/8.14.9) with ESMTP id s6JHwOUU023109 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Sat, 19 Jul 2014 10:58:26 -0700 (PDT) (envelope-from marcel@xcllnt.net) Content-Type: multipart/signed; boundary="Apple-Mail=_332A8ACD-E11F-490C-B854-38BF11F64932"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: KDB entry on NMI From: Marcel Moolenaar In-Reply-To: <20140718160708.GO93733@kib.kiev.ua> Date: Sat, 19 Jul 2014 10:58:18 -0700 Message-Id: References: <20140718160708.GO93733@kib.kiev.ua> To: Konstantin Belousov X-Mailer: Apple Mail (2.1878.6) Cc: amd64@freebsd.org, arch@freebsd.org X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Jul 2014 17:58:35 -0000 --Apple-Mail=_332A8ACD-E11F-490C-B854-38BF11F64932 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii On Jul 18, 2014, at 9:07 AM, Konstantin Belousov wrote: > It was mentioned somewhere recently, that typical BIOS today configures > NMI delivery on the hardware events as broadcast. When I developerd > the dmar(4) busdma backend, I indeed met the problem, and wrote a > prototype which avoided startup of ddb on all cores. Instead, the patch > implements custom spinlock, which allows only one core to win, other > cores ignore the NMI, by spinning on lock. > > The issue which I see on at least two different machines with different > Intel chipsets, is that NMI is somehow sticky, i.e. it is re-delivered > after the handler executes iret. I am not sure what the problem is, > whether it is due to hardware needing some ACK, or a bug in code. > > Anyway, even on two-cores machine, having both cores simultaneously > enter NMI makes the use of ddb impossible, so I believe the patch is > improvement. I make measures to ensure that reboot from ddb prompt > works. > > Thought ? One may call kdb_enter on different CPUs at the same time and it's also possible to call panic on multiple CPUs at the same time (but we serialize panic() right now). What if we let kdb_enter at al deal with concurrency, instead of doing it specifically for NMIs? Also: we may want to do something else than going to the debugger when we see an NMI. More complexity in the NMI handler and specific to entering the debugger seems to move us away from doing other things more easily. Aside: I've always wanted to have the ability to have the kernel debugger switch to a different CPU so that you can create DDB commands that dump hardware resources like TLBs, etc. To support this, you want the KDB layer to have good CPU handling, which possibly makes it also a good place to handle concurrent entry into the debugger from different CPUs. FYI, -- Marcel Moolenaar marcel@xcllnt.net --Apple-Mail=_332A8ACD-E11F-490C-B854-38BF11F64932 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlPKsboACgkQpgWlLWHuifbE1ACeLhNWhD1eu/5acOCnK+zTedWY uq4AniYoVSg/fF9DIDEWJiDjMIkTAhbS =SCRD -----END PGP SIGNATURE----- --Apple-Mail=_332A8ACD-E11F-490C-B854-38BF11F64932-- From owner-freebsd-amd64@FreeBSD.ORG Sat Jul 19 18:29:20 2014 Return-Path: Delivered-To: amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 063B0677; Sat, 19 Jul 2014 18:29:20 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8158D2202; Sat, 19 Jul 2014 18:29:19 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s6JIT930088464 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 19 Jul 2014 21:29:09 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s6JIT930088464 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s6JIT9Ix088463; Sat, 19 Jul 2014 21:29:09 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 19 Jul 2014 21:29:09 +0300 From: Konstantin Belousov To: Marcel Moolenaar Subject: Re: KDB entry on NMI Message-ID: <20140719182909.GU93733@kib.kiev.ua> References: <20140718160708.GO93733@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Tx24CLeOShJjHgAY" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: amd64@freebsd.org, arch@freebsd.org X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Jul 2014 18:29:20 -0000 --Tx24CLeOShJjHgAY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jul 19, 2014 at 10:58:18AM -0700, Marcel Moolenaar wrote: >=20 > On Jul 18, 2014, at 9:07 AM, Konstantin Belousov wr= ote: >=20 > > It was mentioned somewhere recently, that typical BIOS today configures > > NMI delivery on the hardware events as broadcast. When I developerd > > the dmar(4) busdma backend, I indeed met the problem, and wrote a > > prototype which avoided startup of ddb on all cores. Instead, the patch > > implements custom spinlock, which allows only one core to win, other > > cores ignore the NMI, by spinning on lock. > >=20 > > The issue which I see on at least two different machines with different > > Intel chipsets, is that NMI is somehow sticky, i.e. it is re-delivered > > after the handler executes iret. I am not sure what the problem is, > > whether it is due to hardware needing some ACK, or a bug in code. > >=20 > > Anyway, even on two-cores machine, having both cores simultaneously > > enter NMI makes the use of ddb impossible, so I believe the patch is > > improvement. I make measures to ensure that reboot from ddb prompt > > works. > >=20 > > Thought ? >=20 > One may call kdb_enter on different CPUs at the same time and it's > also possible to call panic on multiple CPUs at the same time (but > we serialize panic() right now). What if we let kdb_enter at al deal > with concurrency, instead of doing it specifically for NMIs? Then, on 80-threads machine I get the 80 ddb sessions on NMI broadcast, like now. With your proposal, it will be somewhat better, since sessions are serialized, so I can do the reboot from the first one. Still, I hope to understand what I am missing to stop NMI from delivering in loop. Then, having only one ddb entry would mean that I should return only once. >=20 > Also: we may want to do something else than going to the debugger > when we see an NMI. More complexity in the NMI handler and specific > to entering the debugger seems to move us away from doing other > things more easily. I agree there. >=20 > Aside: I've always wanted to have the ability to have the kernel > debugger switch to a different CPU so that you can create DDB > commands that dump hardware resources like TLBs, etc. To support > this, you want the KDB layer to have good CPU handling, which > possibly makes it also a good place to handle concurrent entry > into the debugger from different CPUs. Me too. I have another half-finished patch which does this, it allows to migrate the ddb from one cpu to another. It worked by signalling a destination cpu that it should activate, while source cpu starts spinning. I do not remember exact problems which were unresolved. I needed this because some state is CPU-local, cannot be accessed =66rom other cores, and is not saved in pcb. I definitely looked at EFER and MISC_FEATURES MSRs, and local apic state. --Tx24CLeOShJjHgAY Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJTyrj1AAoJEJDCuSvBvK1BodgQAIk2aC6e9so+6O0rLJAizzzz 1SZ3bYLSpNN/MbdPxQM/JudjI4qRuqMXl4vSOdOpN8oUlrKKpZBKBj7YfDyjwA3L aEeyG1iAhBjEMLhF2Rty2wDVXfOxBM/bZiTmPdl69VGbmvsqIchhJFNbUh3tTY1i a4JKQDg32LyUNvkbxvw++wco+Ts2ptIASHVkayI1zM428uaxA2r2DnFajCFFAbot dh/jvWJsXHEiQhHFOCoP1UfrQaLWbl+mTsvBjXW3lSldL4SkAQPSuW8NtUE1+kfY DBVEjMHXFvR2cT9TVJgkOhzosFKQ+Z/hrjd2tyIknaptk2kQPZJUX/qO9KH1Yt1+ UoeTVBKXUySYcfDZX/CzThLmsZwURnfq/7ZjV64P0x44FfWIBe+os5sUA0dKBmS2 62kkOFFkOwi4oSFM6ymm0JJhYLxHfGULIgqTeVa+XEO4RHb8qmU199559R/YSSDK f6nlV8hLaHkbhBSZ2EZAhRmXyuBB+P6l6rOoIEbujTAQGIy3xHvCEUFIHDl3BtN6 6aiFEKXPQJFoEvItYwDW3VSjbvCO9jkyA0+suCaL36isEM8w4Z4OfAQszgBRouqk /SXHoI6KIe/14z0JZohYYavPSC3q51/WeHYO1k7q50IIEiwhKUNwkyRAv0DJPIf2 WwSwITcNPpypeeamVfaB =qos5 -----END PGP SIGNATURE----- --Tx24CLeOShJjHgAY-- From owner-freebsd-amd64@FreeBSD.ORG Sat Jul 19 19:57:27 2014 Return-Path: Delivered-To: amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4EDA5BCB; Sat, 19 Jul 2014 19:57:27 +0000 (UTC) Received: from mail.xcllnt.net (mail.xcllnt.net [50.0.150.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0D57F293B; Sat, 19 Jul 2014 19:57:26 +0000 (UTC) Received: from [192.168.2.31] (atc.xcllnt.net [50.0.150.213]) (authenticated bits=0) by mail.xcllnt.net (8.14.9/8.14.9) with ESMTP id s6JJvOxX024820 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Sat, 19 Jul 2014 12:57:25 -0700 (PDT) (envelope-from marcel@xcllnt.net) Content-Type: multipart/signed; boundary="Apple-Mail=_9EC7C332-AEF2-47D7-BE1B-8C3AA7C81F99"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: KDB entry on NMI From: Marcel Moolenaar In-Reply-To: <20140719182909.GU93733@kib.kiev.ua> Date: Sat, 19 Jul 2014 12:57:24 -0700 Message-Id: <18C85F15-FC9E-480C-BFB9-4CD0894FD93A@xcllnt.net> References: <20140718160708.GO93733@kib.kiev.ua> <20140719182909.GU93733@kib.kiev.ua> To: Konstantin Belousov X-Mailer: Apple Mail (2.1878.6) Cc: amd64@freebsd.org, arch@freebsd.org X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Jul 2014 19:57:27 -0000 --Apple-Mail=_9EC7C332-AEF2-47D7-BE1B-8C3AA7C81F99 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On Jul 19, 2014, at 11:29 AM, Konstantin Belousov = wrote: >>=20 >> One may call kdb_enter on different CPUs at the same time and it's >> also possible to call panic on multiple CPUs at the same time (but >> we serialize panic() right now). What if we let kdb_enter at al deal >> with concurrency, instead of doing it specifically for NMIs? > Then, on 80-threads machine I get the 80 ddb sessions on NMI = broadcast, > like now. With your proposal, it will be somewhat better, since > sessions are serialized, so I can do the reboot from the first one. There's value to send the NMI to all CPUs: you'll be pretty sure that if there's a CPU that can handle it, it will get the NMI. Sending it to a single CPU has the downside that if that CPU is unable to handle the NMI (corrupted page tables, locked on some chipset access, held in reset, powering down, whatever one can think of) you're out of luck. Are we acking the NMI on all CPUs right now? --=20 Marcel Moolenaar marcel@xcllnt.net --Apple-Mail=_9EC7C332-AEF2-47D7-BE1B-8C3AA7C81F99 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlPKzaQACgkQpgWlLWHuifbMMwCeMnNysi89BXze/Aatu3VkRZk8 G9cAnj7IsKTxaQh5eZ3xtNcwSryWBfHl =m2cx -----END PGP SIGNATURE----- --Apple-Mail=_9EC7C332-AEF2-47D7-BE1B-8C3AA7C81F99-- From owner-freebsd-amd64@FreeBSD.ORG Sat Jul 19 20:09:55 2014 Return-Path: Delivered-To: amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 158B61B5; Sat, 19 Jul 2014 20:09:55 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AA82C2A40; Sat, 19 Jul 2014 20:09:54 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s6JK9nVH011576 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 19 Jul 2014 23:09:49 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s6JK9nVH011576 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s6JK9mMn011575; Sat, 19 Jul 2014 23:09:48 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 19 Jul 2014 23:09:48 +0300 From: Konstantin Belousov To: Marcel Moolenaar Subject: Re: KDB entry on NMI Message-ID: <20140719200948.GW93733@kib.kiev.ua> References: <20140718160708.GO93733@kib.kiev.ua> <20140719182909.GU93733@kib.kiev.ua> <18C85F15-FC9E-480C-BFB9-4CD0894FD93A@xcllnt.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="/kNZZTualZuJSMMX" Content-Disposition: inline In-Reply-To: <18C85F15-FC9E-480C-BFB9-4CD0894FD93A@xcllnt.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: amd64@freebsd.org, arch@freebsd.org X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Jul 2014 20:09:55 -0000 --/kNZZTualZuJSMMX Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jul 19, 2014 at 12:57:24PM -0700, Marcel Moolenaar wrote: >=20 > On Jul 19, 2014, at 11:29 AM, Konstantin Belousov w= rote: > >>=20 > >> One may call kdb_enter on different CPUs at the same time and it's > >> also possible to call panic on multiple CPUs at the same time (but > >> we serialize panic() right now). What if we let kdb_enter at al deal > >> with concurrency, instead of doing it specifically for NMIs? > > Then, on 80-threads machine I get the 80 ddb sessions on NMI broadcast, > > like now. With your proposal, it will be somewhat better, since > > sessions are serialized, so I can do the reboot from the first one. >=20 > There's value to send the NMI to all CPUs: you'll be pretty sure > that if there's a CPU that can handle it, it will get the NMI. > Sending it to a single CPU has the downside that if that CPU is > unable to handle the NMI (corrupted page tables, locked on some > chipset access, held in reset, powering down, whatever one can > think of) you're out of luck. Right, and this is what my patch aim to make useable. The first CPU which gets the NMI wins the permission to enter kdb, other CPUs ignore interrupt, if they are interrupted while winner is still in NMI handler. >=20 > Are we acking the NMI on all CPUs right now? I am not sure what you are asking about. There is no any code which ACKs NMI, and possibly there should be, but I do not know what the ACK could consist of. It might be that the external signal which initiates the NMI is programmed to level-triggered. and we need to ack it in chipset ? But I do not know this for sure, and the action would be definitely chipset-depended. The NMI is blocked on cpu by internal NMI disable bit upon NMI delivery, until iret is executed. The bit is not available in the CPU architectural state. --/kNZZTualZuJSMMX Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJTytCMAAoJEJDCuSvBvK1BPBAQAJhX5OO90h5g62cs0VxX3l4c lnyG6vK/oZQBSIJHWPf8evPIGuFhOQH8pULC2FA4z5fwRJmgIoI+s/QVKxotD1nw AjGuBM2MVfwtOzafvfkXq+x/jlL7iJgHBHEGde4HHWxgbCzvkENVQUADDDXftex/ oDLkFzE0vfnVlm8/UAC6si08AmawSeEI5gQ+oss1onrrC/RCKconMqJZ4nlQ/EWJ VMm6QLJb7+HWuWNKWTFznD9n/+FNM6S5n8UTthvbKz7EIZHtfuESpXdIhx9ctt81 USbz7r+GM27MYwoupwkd4c33qetobsTBVZW9L4vuhj2QCmKX+wWU+lozxtG18gF3 T2ZH+AGvr+qLn6ROQq5J51PT7Et/JFkMn4o5fyOvJaEm+9FeGdEa70pp0hRDYWBQ pmVVqANKtqvo4PnspN4bhPY3W3XBCx1vQkI32g4bdgifaZsjW1YLqQVkLqDz50Mh uwXmoB1wFPZ+cwHo+tBsmMynjJtXmatKdnhRXbN56FDz9PEjyOVLhON0OummAftC BsS2txJNy2vyOVYRypU2PqLWTY+wED3Ks092Yjw/Qlu61Xo7/N+U6IQ4jC2Xzm5l S34U+0twBm1vAcEZhO1hV1up3b0DqfKPbvdFMQOA1ojRY+qX7Lte5dAXwhM59KOG qbqOkKDPlTM1Ri9EOTGd =qT// -----END PGP SIGNATURE----- --/kNZZTualZuJSMMX--