From owner-freebsd-current@freebsd.org Wed Jan 6 14:46:12 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 97B1EA64279 for ; Wed, 6 Jan 2016 14:46:12 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-yk0-x22f.google.com (mail-yk0-x22f.google.com [IPv6:2607:f8b0:4002:c07::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5293213D6 for ; Wed, 6 Jan 2016 14:46:12 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-yk0-x22f.google.com with SMTP id x67so317398252ykd.2 for ; Wed, 06 Jan 2016 06:46:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=Q9Y89c1l2/x1pbuuQVrU7eqkObDS1SK4C3U9SBwK3I0=; b=PehgsHkqrXN7RisyVuA3XDh4SoXqcprmiu7d8XAfaRq24oYA+OIo71MkkEjYRRotCn cglbjD7qm8mGdgcBep46iFBdXp3JkFwNV1nEb8Cv6ab8Y5NWTmivCGW5PVwz//aOO32N uhRkdmfz3UDfE0rqIrUkmbCqQUbl66vdHfX/qcImRqBT4qirKXfnpD41nsEWUzgflPiI M4eARExvBoO2JqoEaT6lxYdueQry/VJCyOnyTZNg0FoODczwR7Zy3zbc2N5RyFWR9pZD 7wrb0B2Ebbmzs4e4FeAuHHBkYu5ZnSkWjCCmq6keeXvBJLglNHbaG9F4UzSpuvG42SfN 1V/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=Q9Y89c1l2/x1pbuuQVrU7eqkObDS1SK4C3U9SBwK3I0=; b=gWjIuVrQzGY8FWjVz1Jy6xxrCHDq1ALQZMYopaIJ4PNiDOYhYrlEqct3ZgHfzHR7jf UysSorfVd2AMLeF+0nql0yn7wNeTqWXqmcd4vXYihOTAx7HBQgiqlHaSl3wFWZh9tgWM OmxbhSKVuFXgIaT08Xsg+vuXOjTqYMyOFfsr53EsadfuL6vtki/at4ws3Jgrw/OumwsJ tmEkpRu0E2Q9UPiBFS8O+CkfAMzUjU0OTCUUAwauwY1L1jQvx/eJtPojBT/D3ZoMPStD ziMbo5DGXsytPIFM5m5RCKMI14UVBHeUvM5VoRdJzujsCYynTjf/+r8Y68l6NTzo5tPC esOQ== X-Gm-Message-State: ALoCoQlMcnm1iSoUMQVXGNSlHCb5vHEBssbT0dz8RKXTdRVOYOkU4gOXZ1ZAM8djTQQgvDr2Q/69jhm4qoLRhVIY0yl0EgkTvQ== X-Received: by 10.129.17.66 with SMTP id 63mr66363003ywr.122.1452091571454; Wed, 06 Jan 2016 06:46:11 -0800 (PST) Received: from mutt-hardenedbsd ([63.88.83.104]) by smtp.gmail.com with ESMTPSA id v70sm51883081ywa.30.2016.01.06.06.46.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Jan 2016 06:46:10 -0800 (PST) Date: Wed, 6 Jan 2016 09:46:08 -0500 From: Shawn Webb To: Adrian Chadd Cc: freebsd-current Subject: Re: kernel panic by enabling net.inet.ip.random_id Message-ID: <20160106144608.GA71037@mutt-hardenedbsd> References: <20160106015742.GA8405@mutt-hardenedbsd> <20160106021316.GB8405@mutt-hardenedbsd> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="yrj/dFKFPuw6o+aM" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD mutt-hardenedbsd 11.0-CURRENT-HBSD FreeBSD 11.0-CURRENT-HBSD X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jan 2016 14:46:12 -0000 --yrj/dFKFPuw6o+aM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable (kgdb) list *(0xffffffff80b5de9e) 0xffffffff80b5de9e is in ip_fillid (/usr/src/sys/netinet/ip_id.c:237). warning: Source file is more recent than executable. 232 new_id =3D 0; 233 do { 234 if (new_id !=3D 0) 235 V_random_id_collisions++; 236 arc4rand(&new_id, sizeof(new_id), 0); 237 } while (bit_test(V_id_bits, new_id) || new_id =3D=3D 0); 238 bit_clear(V_id_bits, V_id_array[V_array_ptr]); 239 bit_set(V_id_bits, new_id); 240 V_id_array[V_array_ptr] =3D new_id; 241 V_array_ptr++; This is the change I made to ip_id.c that caused the underlying kernel panic: https://github.com/HardenedBSD/hardenedBSD/commit/52d5a93b92097e7a79be8d2e0= eb9c1a58b8337d1 Ideally, we should be able to just toggle that variable and all would be well. But it seems with the VIMAGE work, something is preventing that. Thanks, Shawn On Tue, Jan 05, 2016 at 06:22:34PM -0800, Adrian Chadd wrote: > try list *(0x[address]) . >=20 > That line is mtx_unlock(), which makes no sense (as mtx_lock succeeded fi= ne.) >=20 >=20 > -a >=20 >=20 > On 5 January 2016 at 18:13, Shawn Webb wrote: > > Thanks for the quick reply! Here's some more debugging output: > > > > =3D=3D=3D Begin Log =3D=3D=3D > > (kgdb) bt > > #0 doadump (textdump=3D0) at pcpu.h:221 > > #1 0xffffffff8037c78b in db_dump (dummy=3D, dummy= 2=3Dfalse, dummy3=3D0, dummy4=3D0x0) at /usr/src/sys/ddb/db_command.c:533 > > #2 0xffffffff8037c57e in db_command (cmd_table=3D0x0) at /usr/src/sys/= ddb/db_command.c:440 > > #3 0xffffffff8037c314 in db_command_loop () at /usr/src/sys/ddb/db_com= mand.c:493 > > #4 0xffffffff8037edab in db_trap (type=3D, code= =3D0) at /usr/src/sys/ddb/db_main.c:251 > > #5 0xffffffff80a5c563 in kdb_trap (type=3D12, code=3D0, tf=3D) at /usr/src/sys/kern/subr_kdb.c:654 > > #6 0xffffffff80e6b7e1 in trap_fatal (frame=3D0xfffffe02c33894d0, eva= =3D) at /usr/src/sys/amd64/amd64/trap.c:829 > > #7 0xffffffff80e6ba2d in trap_pfault (frame=3D0xfffffe02c33894d0, user= mode=3D) at /usr/src/sys/amd64/amd64/trap.c:684 > > #8 0xffffffff80e6b15f in trap (frame=3D0xfffffe02c33894d0) at /usr/src= /sys/amd64/amd64/trap.c:435 > > #9 0xffffffff80e4af97 in calltrap () at /usr/src/sys/amd64/amd64/excep= tion.S:234 > > #10 0xffffffff80b5de9e in ip_fillid (ip=3D0xfffff8000ef8cb88) at /usr/s= rc/sys/netinet/ip_id.c:237 > > #11 0xffffffff80b6c41b in ip_output (m=3D, opt=3D<= value optimized out>, ro=3D, flags=3D0, imo=3D0x0, inp= =3D0xfffff8000e66e960) at /usr/src/sys/netinet/ip_output.c:268 > > #12 0xffffffff80bf0612 in udp_send (so=3D, flags= =3D, m=3D, addr=3D0x0, control=3D= , td=3D0xfffff8000ef8cb88) at /usr/src/sys/netinet/udp= _usrreq.c:1517 > > #13 0xffffffff80aa3872 in sosend_dgram (so=3D0xfffff8000e6422e8, addr= =3D0x0, uio=3D, top=3D0xfffff8000ef8cb00, control=3D0x= 0, flags=3D, td=3D0xffffffff81bef2ec) at /usr/src/sys/= kern/uipc_socket.c:1164 > > #13 0xffffffff80aa3872 in sosend_dgram (so=3D0xfffff8000e6422e8, addr= =3D0x0, uio=3D, top=3D0xfffff8000ef8cb00, control=3D0x= 0, flags=3D, td=3D0xffffffff81bef2ec) at /usr/src/sys/= kern/uipc_socket.c:1164 > > #14 0xffffffff80aaa03b in kern_sendit (td=3D0xfffff8000e4cd9c0, s=3D6, = mp=3D, flags=3D0, control=3D0x0, segflg=3DUIO_USERSPAC= E) at /usr/src/sys/kern/uipc_syscalls.c:906 > > #15 0xffffffff80aaa336 in sendit (td=3D0xfffff8000e4cd9c0, s=3D, mp=3D0xfffffe02c3389970, flags=3D3980) at /usr/src/sys/kern/= uipc_syscalls.c:833 > > #16 0xffffffff80aaa1fd in sys_sendto (td=3D0x0, uap=3D) at /usr/src/sys/kern/uipc_syscalls.c:957 > > #17 0xffffffff80e6bfdb in amd64_syscall (td=3D0xfffff8000e4cd9c0, trace= d=3D0) at subr_syscall.c:135 > > #18 0xffffffff80e4b27b in Xfast_syscall () at /usr/src/sys/amd64/amd64/= exception.S:394 > > #19 0x000003e339782e8a in ?? () > > (kgdb) x/i 0xffffffff80b5de9e > > 0xffffffff80b5de9e : movzbl (%rax,%rcx,1),%esi > > (kgdb) info reg > > rax 0x0 0 > > rbx 0x0 0 > > rcx 0x0 0 > > rdx 0x0 0 > > rsi 0x0 0 > > rdi 0x0 0 > > rbp 0xfffffe02c3388fe0 0xfffffe02c3388fe0 > > rsp 0xfffffe02c3388fc8 0xfffffe02c3388fc8 > > r8 0x0 0 > > r9 0x0 0 > > r10 0x0 0 > > r11 0x0 0 > > r12 0xffffffff817c0b80 -2122577024 > > r13 0xffffffff817c1470 -2122574736 > > r14 0x1 1 > > r15 0x4 4 > > rip 0xffffffff80a1fae3 0xffffffff80a1fae3 > > eflags 0x0 0 > > cs 0x0 0 > > ss 0x0 0 > > ds 0x0 0 > > es 0x0 0 > > fs 0x0 0 > > gs 0x0 0 > > =3D=3D=3D End Log =3D=3D=3D > > > > Thanks, > > > > Shawn > > > > On Tue, Jan 05, 2016 at 06:06:41PM -0800, Adrian Chadd wrote: > >> looks like a null pointer deference. What's kgdb show at that IP? > >> > >> > >> -a > >> > >> > >> On 5 January 2016 at 17:57, Shawn Webb wr= ote: > >> > Hey All, > >> > > >> > Here's a kernel panic I'm experiencing by enabling net.inet.ip.rando= m_id > >> > at boot. > >> > > >> > I'm on latest HEAD on amd64 in bhyve. I'll soon-ish be testing on na= tive > >> > hardware with VIMAGE enabled. > >> > > >> > =3D=3D=3D Begin Log =3D=3D=3D > >> > Kernel page fault with the following non-sleepable locks held: > >> > exclusive sleep mutex ip_id_mtx (ip_id_mtx) r =3D 0 (0xffffffff81c54= 830) locked @ /usr/src/sys/netinet/ip_id.c:227 > >> > stack backtrace: > >> > #0 0xffffffff80a79620 at witness_debugger+0x70 > >> > #1 0xffffffff80a7a937 at witness_warn+0x3d7 > >> > #2 0xffffffff80e6b887 at trap_pfault+0x57 > >> > #3 0xffffffff80e6b15f at trap+0x4bf > >> > #4 0xffffffff80e4af97 at calltrap+0x8 > >> > #5 0xffffffff80b6c41b at ip_output+0x16b > >> > #6 0xffffffff80b68e82 at icmp_reflect+0x5b2 > >> > #7 0xffffffff80b6883f at icmp_error+0x46f > >> > #8 0xffffffff80beeb12 at udp_input+0x982 > >> > #9 0xffffffff80b69d1d at ip_input+0x17d > >> > #10 0xffffffff80b08ba1 at netisr_dispatch_src+0x81 > >> > #11 0xffffffff80afecce at ether_demux+0x15e > >> > #12 0xffffffff80affa14 at ether_nh_input+0x344 > >> > #13 0xffffffff80b08ba1 at netisr_dispatch_src+0x81 > >> > #14 0xffffffff80afefcf at ether_input+0x4f > >> > #15 0xffffffff8089a5c3 at vtnet_rxq_eof+0x823 > >> > #16 0xffffffff8089b2ce at vtnet_rx_vq_intr+0x4e > >> > #17 0xffffffff809e9ba6 at intr_event_execute_handlers+0x96 > >> > > >> > > >> > Fatal trap 12: page fault while in kernel mode > >> > cpuid =3D 6; apic id =3D 06 > >> > fault virtual address =3D 0x5bd > >> > fault code =3D supervisor read data, page not present > >> > instruction pointer =3D 0x20:0xffffffff80b5de9e > >> > stack pointer =3D 0x28:0xfffffe02b8d483e0 > >> > frame pointer =3D 0x28:0xfffffe02b8d48410 > >> > code segment =3D base 0x0, limit 0xfffff, type 0x1b > >> > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > >> > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > >> > current process =3D 12 (irq265: virtio_pci0) > >> > [ thread pid 12 tid 100040 ] > >> > Stopped at ip_fillid+0x8e: movzbl (%rax,%rcx,1),%esi > >> > =3D=3D=3D End Log =3D=3D=3D > >> > > >> > Thanks, > >> > > >> > -- > >> > Shawn Webb > >> > HardenedBSD > >> > > >> > GPG Key ID: 0x6A84658F52456EEE > >> > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6= EEE > > > > -- > > Shawn Webb > > HardenedBSD > > > > GPG Key ID: 0x6A84658F52456EEE > > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --=20 Shawn Webb HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --yrj/dFKFPuw6o+aM Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJWjSitAAoJEGqEZY9SRW7udgUQALOysjXSlzrANZkBVvAgraNk hGXfRTZFWacRY9IdB2cskjpDg3xKrKzBaUUKb38QGd0UHOXwJCLKe/5MOwJObagX cx6eS/NYROcRhgBC3/u/fuFzUwmeICB33CIIzBrTjiTCQFoEV3RkF3zhea+fV6LN AF4l8tb1f2C5N4xOmUicwmYLzGqzShPg1YYdHFsKpr9iAfivk/zIVidblblQ1ZwE 0ytNWGzJ1l2Jk37UnyeI5n/8ovbUHjmLSa9R60iJoY6rkCNSDBjQacEy6c32QXun vckPVRGAqS765xooczBU7zJ29CMAr8T7xSA0htzwmI+uj8O4IjfuiCSucfkKoADh win05LCZfzp9z3JuHLmrL8Cc2TJVbtWBeCLegyz7ZmDQka/q+z1z18VbYvjqvQjC gf9+WJotEVvN27g9u4Jhqw8+piVFu/dGA6RUeh+n2122NPodDYLQiVnWMeJ8GSa+ j5uaUOKR+ULgFz5qbtPi0EnD1U2wpBc2AIsIWcaerhgk1raLXYEXCUJ08vPiEtd2 b7i/i6if/FWPyk1vPnLwisCjy4Uc52nfd6ep3r/C6EGeQLmHsXsHn79X6G3LlR4J 35rne7yX5yT30Gin+tgjBF/zfZ2J7O1JY6uIVtgJRRGAnJxTdmcZVsqzGXBBEDiS W29Jo/ezCFHaVJEiiC4W =4Hot -----END PGP SIGNATURE----- --yrj/dFKFPuw6o+aM--