From owner-freebsd-current@freebsd.org Wed Jan 6 18:15:09 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 12E2DA65372 for ; Wed, 6 Jan 2016 18:15:09 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x236.google.com (mail-ig0-x236.google.com [IPv6:2607:f8b0:4001:c05::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D7AA819DC for ; Wed, 6 Jan 2016 18:15:08 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-ig0-x236.google.com with SMTP id t15so10756221igr.0 for ; Wed, 06 Jan 2016 10:15:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=15CAIE0oZU78+RvFUmsf5DgFcGrDE6AmaG4QKa7wwdQ=; b=bZjjH8sm9vT/HTNytbD6Kej70XCqx7G7MzxtmQ8TYHK3XfdGgj5Qvk/cZA6t45bNex BywKBsiQjYljs1gGiZ1wn5Yv/129X24xImcL+W3XBpK6ZUTXbwgfQarvK0o5FtZe+6of +jA0Ron4jGKUn+Es4NliajNhSBGKld+ncVOmcgnTTkcv76NRbtGeQ2SvmrTEswBW/Nbd VerAMZLPwPOkyy+bN7LMu7WmrI0tpajj0rKqF2kRlAcRZidj95ZquGht78jckXC2v7L8 XR44Xf9W2407tc44oI5z/s2oE6LgUNvt98vM3hTxw76Xe3sCacQjlZhOQPW0nPbvFt1N yUlg== MIME-Version: 1.0 X-Received: by 10.50.171.225 with SMTP id ax1mr9161361igc.61.1452104108214; Wed, 06 Jan 2016 10:15:08 -0800 (PST) Received: by 10.36.121.202 with HTTP; Wed, 6 Jan 2016 10:15:08 -0800 (PST) In-Reply-To: <20160106144608.GA71037@mutt-hardenedbsd> References: <20160106015742.GA8405@mutt-hardenedbsd> <20160106021316.GB8405@mutt-hardenedbsd> <20160106144608.GA71037@mutt-hardenedbsd> Date: Wed, 6 Jan 2016 10:15:08 -0800 Message-ID: Subject: Re: kernel panic by enabling net.inet.ip.random_id From: Adrian Chadd To: Shawn Webb Cc: freebsd-current Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jan 2016 18:15:09 -0000 Why'd you condition the vimage definition? :) -a On 6 January 2016 at 06:46, Shawn Webb wrote: > (kgdb) list *(0xffffffff80b5de9e) > 0xffffffff80b5de9e is in ip_fillid (/usr/src/sys/netinet/ip_id.c:237). > warning: Source file is more recent than executable. > > 232 new_id = 0; > 233 do { > 234 if (new_id != 0) > 235 V_random_id_collisions++; > 236 arc4rand(&new_id, sizeof(new_id), 0); > 237 } while (bit_test(V_id_bits, new_id) || new_id == 0); > 238 bit_clear(V_id_bits, V_id_array[V_array_ptr]); > 239 bit_set(V_id_bits, new_id); > 240 V_id_array[V_array_ptr] = new_id; > 241 V_array_ptr++; > > This is the change I made to ip_id.c that caused the underlying kernel > panic: > https://github.com/HardenedBSD/hardenedBSD/commit/52d5a93b92097e7a79be8d2e0eb9c1a58b8337d1 > > Ideally, we should be able to just toggle that variable and all would be > well. But it seems with the VIMAGE work, something is preventing that. > > Thanks, > > Shawn > > On Tue, Jan 05, 2016 at 06:22:34PM -0800, Adrian Chadd wrote: >> try list *(0x[address]) . >> >> That line is mtx_unlock(), which makes no sense (as mtx_lock succeeded fine.) >> >> >> -a >> >> >> On 5 January 2016 at 18:13, Shawn Webb wrote: >> > Thanks for the quick reply! Here's some more debugging output: >> > >> > === Begin Log === >> > (kgdb) bt >> > #0 doadump (textdump=0) at pcpu.h:221 >> > #1 0xffffffff8037c78b in db_dump (dummy=, dummy2=false, dummy3=0, dummy4=0x0) at /usr/src/sys/ddb/db_command.c:533 >> > #2 0xffffffff8037c57e in db_command (cmd_table=0x0) at /usr/src/sys/ddb/db_command.c:440 >> > #3 0xffffffff8037c314 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493 >> > #4 0xffffffff8037edab in db_trap (type=, code=0) at /usr/src/sys/ddb/db_main.c:251 >> > #5 0xffffffff80a5c563 in kdb_trap (type=12, code=0, tf=) at /usr/src/sys/kern/subr_kdb.c:654 >> > #6 0xffffffff80e6b7e1 in trap_fatal (frame=0xfffffe02c33894d0, eva=) at /usr/src/sys/amd64/amd64/trap.c:829 >> > #7 0xffffffff80e6ba2d in trap_pfault (frame=0xfffffe02c33894d0, usermode=) at /usr/src/sys/amd64/amd64/trap.c:684 >> > #8 0xffffffff80e6b15f in trap (frame=0xfffffe02c33894d0) at /usr/src/sys/amd64/amd64/trap.c:435 >> > #9 0xffffffff80e4af97 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:234 >> > #10 0xffffffff80b5de9e in ip_fillid (ip=0xfffff8000ef8cb88) at /usr/src/sys/netinet/ip_id.c:237 >> > #11 0xffffffff80b6c41b in ip_output (m=, opt=, ro=, flags=0, imo=0x0, inp=0xfffff8000e66e960) at /usr/src/sys/netinet/ip_output.c:268 >> > #12 0xffffffff80bf0612 in udp_send (so=, flags=, m=, addr=0x0, control=, td=0xfffff8000ef8cb88) at /usr/src/sys/netinet/udp_usrreq.c:1517 >> > #13 0xffffffff80aa3872 in sosend_dgram (so=0xfffff8000e6422e8, addr=0x0, uio=, top=0xfffff8000ef8cb00, control=0x0, flags=, td=0xffffffff81bef2ec) at /usr/src/sys/kern/uipc_socket.c:1164 >> > #13 0xffffffff80aa3872 in sosend_dgram (so=0xfffff8000e6422e8, addr=0x0, uio=, top=0xfffff8000ef8cb00, control=0x0, flags=, td=0xffffffff81bef2ec) at /usr/src/sys/kern/uipc_socket.c:1164 >> > #14 0xffffffff80aaa03b in kern_sendit (td=0xfffff8000e4cd9c0, s=6, mp=, flags=0, control=0x0, segflg=UIO_USERSPACE) at /usr/src/sys/kern/uipc_syscalls.c:906 >> > #15 0xffffffff80aaa336 in sendit (td=0xfffff8000e4cd9c0, s=, mp=0xfffffe02c3389970, flags=3980) at /usr/src/sys/kern/uipc_syscalls.c:833 >> > #16 0xffffffff80aaa1fd in sys_sendto (td=0x0, uap=) at /usr/src/sys/kern/uipc_syscalls.c:957 >> > #17 0xffffffff80e6bfdb in amd64_syscall (td=0xfffff8000e4cd9c0, traced=0) at subr_syscall.c:135 >> > #18 0xffffffff80e4b27b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:394 >> > #19 0x000003e339782e8a in ?? () >> > (kgdb) x/i 0xffffffff80b5de9e >> > 0xffffffff80b5de9e : movzbl (%rax,%rcx,1),%esi >> > (kgdb) info reg >> > rax 0x0 0 >> > rbx 0x0 0 >> > rcx 0x0 0 >> > rdx 0x0 0 >> > rsi 0x0 0 >> > rdi 0x0 0 >> > rbp 0xfffffe02c3388fe0 0xfffffe02c3388fe0 >> > rsp 0xfffffe02c3388fc8 0xfffffe02c3388fc8 >> > r8 0x0 0 >> > r9 0x0 0 >> > r10 0x0 0 >> > r11 0x0 0 >> > r12 0xffffffff817c0b80 -2122577024 >> > r13 0xffffffff817c1470 -2122574736 >> > r14 0x1 1 >> > r15 0x4 4 >> > rip 0xffffffff80a1fae3 0xffffffff80a1fae3 >> > eflags 0x0 0 >> > cs 0x0 0 >> > ss 0x0 0 >> > ds 0x0 0 >> > es 0x0 0 >> > fs 0x0 0 >> > gs 0x0 0 >> > === End Log === >> > >> > Thanks, >> > >> > Shawn >> > >> > On Tue, Jan 05, 2016 at 06:06:41PM -0800, Adrian Chadd wrote: >> >> looks like a null pointer deference. What's kgdb show at that IP? >> >> >> >> >> >> -a >> >> >> >> >> >> On 5 January 2016 at 17:57, Shawn Webb wrote: >> >> > Hey All, >> >> > >> >> > Here's a kernel panic I'm experiencing by enabling net.inet.ip.random_id >> >> > at boot. >> >> > >> >> > I'm on latest HEAD on amd64 in bhyve. I'll soon-ish be testing on native >> >> > hardware with VIMAGE enabled. >> >> > >> >> > === Begin Log === >> >> > Kernel page fault with the following non-sleepable locks held: >> >> > exclusive sleep mutex ip_id_mtx (ip_id_mtx) r = 0 (0xffffffff81c54830) locked @ /usr/src/sys/netinet/ip_id.c:227 >> >> > stack backtrace: >> >> > #0 0xffffffff80a79620 at witness_debugger+0x70 >> >> > #1 0xffffffff80a7a937 at witness_warn+0x3d7 >> >> > #2 0xffffffff80e6b887 at trap_pfault+0x57 >> >> > #3 0xffffffff80e6b15f at trap+0x4bf >> >> > #4 0xffffffff80e4af97 at calltrap+0x8 >> >> > #5 0xffffffff80b6c41b at ip_output+0x16b >> >> > #6 0xffffffff80b68e82 at icmp_reflect+0x5b2 >> >> > #7 0xffffffff80b6883f at icmp_error+0x46f >> >> > #8 0xffffffff80beeb12 at udp_input+0x982 >> >> > #9 0xffffffff80b69d1d at ip_input+0x17d >> >> > #10 0xffffffff80b08ba1 at netisr_dispatch_src+0x81 >> >> > #11 0xffffffff80afecce at ether_demux+0x15e >> >> > #12 0xffffffff80affa14 at ether_nh_input+0x344 >> >> > #13 0xffffffff80b08ba1 at netisr_dispatch_src+0x81 >> >> > #14 0xffffffff80afefcf at ether_input+0x4f >> >> > #15 0xffffffff8089a5c3 at vtnet_rxq_eof+0x823 >> >> > #16 0xffffffff8089b2ce at vtnet_rx_vq_intr+0x4e >> >> > #17 0xffffffff809e9ba6 at intr_event_execute_handlers+0x96 >> >> > >> >> > >> >> > Fatal trap 12: page fault while in kernel mode >> >> > cpuid = 6; apic id = 06 >> >> > fault virtual address = 0x5bd >> >> > fault code = supervisor read data, page not present >> >> > instruction pointer = 0x20:0xffffffff80b5de9e >> >> > stack pointer = 0x28:0xfffffe02b8d483e0 >> >> > frame pointer = 0x28:0xfffffe02b8d48410 >> >> > code segment = base 0x0, limit 0xfffff, type 0x1b >> >> > = DPL 0, pres 1, long 1, def32 0, gran 1 >> >> > processor eflags = interrupt enabled, resume, IOPL = 0 >> >> > current process = 12 (irq265: virtio_pci0) >> >> > [ thread pid 12 tid 100040 ] >> >> > Stopped at ip_fillid+0x8e: movzbl (%rax,%rcx,1),%esi >> >> > === End Log === >> >> > >> >> > Thanks, >> >> > >> >> > -- >> >> > Shawn Webb >> >> > HardenedBSD >> >> > >> >> > GPG Key ID: 0x6A84658F52456EEE >> >> > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE >> > >> > -- >> > Shawn Webb >> > HardenedBSD >> > >> > GPG Key ID: 0x6A84658F52456EEE >> > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE > > -- > Shawn Webb > HardenedBSD > > GPG Key ID: 0x6A84658F52456EEE > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE