From owner-freebsd-hackers@freebsd.org Sun Dec 4 18:32:34 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8DD69C66646 for ; Sun, 4 Dec 2016 18:32:34 +0000 (UTC) (envelope-from embaudarm@gmail.com) Received: from mail-qk0-x230.google.com (mail-qk0-x230.google.com [IPv6:2607:f8b0:400d:c09::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 508E110D1 for ; Sun, 4 Dec 2016 18:32:34 +0000 (UTC) (envelope-from embaudarm@gmail.com) Received: by mail-qk0-x230.google.com with SMTP id n204so326659743qke.2 for ; Sun, 04 Dec 2016 10:32:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=XL/Rce53lYqLgNsuEiZSzP3a4UQ6QK/nZCHtlmJkWRY=; b=t5ho+T9Cp9LSiq21Eu5+70PH/ICjh1pgaEIe4aKq2/xiddh9NUKHZjhDBKXZrmfN2p sdsHC4XYFuU5I5IP850RHf6nu5vqR6rukZhCH0x4kSdqOr74orn6oNbKUwGAH12LecHO EMuxOZJNoee9OUzQvywwgFB6dqnBy789dU20g2EPUoGfX8KCporFha/vf3CR4AD0IXWT 2kHu1ErQMT9H5SlFnAL0u/eIjAmiA4d0ORxyuhlXBmG+DgIE9vt7dyDxvZ3Vzk3mhqrx s+Cn2G12Vau+kUR5KWzOU7PrdOZ/X+cGP0v3ed7VH8K2n/B4YJfecf58yPOXmF4+McqA q3Gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=XL/Rce53lYqLgNsuEiZSzP3a4UQ6QK/nZCHtlmJkWRY=; b=cz3t3OxZ5RosYCrZGlcM5rZd3ZuDdKCyCg3YHbtqrGQDPDw/ajSmPIaQm88X78l0uw e41T8Fp3aOrrwnIJdL0yxLArhKXlkg7E6cv5aDq0Z9+2lilFX08kFt7A0PFnaEdFeyO8 gFnk0faOc7TuRU9SYiw7mYaLO4ny3ueb3mlSJwJuLInj3euYzGQ3A1Yz4BVagy6lE1yo 5TChOJTjd5Hn3Rx7CspOWhmwrVMxUtE0jNvFsQNbWKsrNfbyJb3JtwuXWJLT7L78KmP+ 1aO15565GidSMraa2uTjgzhQipt7YNslvuffjFsIWgyDN8d00qa9rrGwZzUwpyekl4sj zmJg== X-Gm-Message-State: AKaTC01mUleoJUwcQDa3hZoLjbBVrnQEBbGcIEgshaeJ8daE9B/91ggxFsLyNSC6HfRQbhHg1RX6p6MGlx6fMw== X-Received: by 10.55.154.205 with SMTP id c196mr44264469qke.25.1480876353163; Sun, 04 Dec 2016 10:32:33 -0800 (PST) MIME-Version: 1.0 Received: by 10.237.54.225 with HTTP; Sun, 4 Dec 2016 10:32:32 -0800 (PST) From: Lee D Date: Sun, 4 Dec 2016 13:32:32 -0500 Message-ID: Subject: Please help me understand "Translation Fault" in custom device drivers, and how to debug To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Dec 2016 18:32:34 -0000 Hello, I need help understanding what a translation fault is, and how to debug it. I have googled like crazy but can't seem to find any detailed information. I am working on an embedded system using an ARM processor, and consequently am writing a bunch of device device drivers for my custom hardware. I am having a problem with occasional crashes when kldload'ing my modules in a boot script. I get various errors, including "Translation Fault" (L1 or L2), "Alignment Fault", "vm_fault", and "undefined instruction in kernel". My code works 95% of the time though. I never see any crashes while running, so I don't think this is a flaky hardware problem. Any suggestions on what kernel debugger commands to enter to gather information would also be helpful. Here are the commands I am currently recording the output of when I get a crash: db> bt db> ps db> show intr db> show proc 618 db> show allpcpu db> show allrman db> show intrcnt db> show proc db> show procvm For a single concrete example, here is a backtrace of a device driver that failed with a translation fault on kldload. This BT is unique in that it actually seems to contain useful information. Most of the backtraces just show some abort/exeception related calls and then say "Unable to unwind into user space" (paraphrased), leaving me no info about where my crash happened. FreeBSD 10.3 Thanks, Lee db> bt Tracing pid 622 tid 100079 td 0xc2d68000 db_trace_self() at db_trace_self pc = 0xc057a1e4 lr = 0xc0137c68 (db_stack_trace+0x108) sp = 0xde966670 fp = 0xde966688 r10 = 0xc074b240 db_stack_trace() at db_stack_trace+0x108 pc = 0xc0137c68 lr = 0xc013760c (db_command+0x294) sp = 0xde966690 fp = 0xde966730 r4 = 0x00000000 r5 = 0x00000000 r6 = 0x00000000 db_command() at db_command+0x294 pc = 0xc013760c lr = 0xc0137364 (db_command_loop+0x78) sp = 0xde966738 fp = 0xde966748 r4 = 0xc05c7ed4 r5 = 0xc05dd87c r6 = 0xc074b22c r7 = 0xde966978 r8 = 0x00000001 r9 = 0xc0673520 r10 = 0xc0740f44 db_command_loop() at db_command_loop+0x78 pc = 0xc0137364 lr = 0xc0139e6c (db_trap+0x108) sp = 0xde966750 fp = 0xde966870 r4 = 0x00000000 r5 = 0xc074b238 r6 = 0xc0740f70 db_trap() at db_trap+0x108 pc = 0xc0139e6c lr = 0xc02ec8f8 (kdb_trap+0x188) sp = 0xde966878 fp = 0xde966898 r4 = 0x00000000 r5 = 0x00000017 r6 = 0xc0740f70 r7 = 0xde966978 kdb_trap() at kdb_trap+0x188 pc = 0xc02ec8f8 lr = 0xc05919ec (abort_fatal+0x1d4) sp = 0xde9668a0 fp = 0xde9668b8 r4 = 0xde966978 r5 = 0x00000013 r6 = 0x00000004 r7 = 0x00000007 r8 = 0x00000017 r9 = 0x00000004 r10 = 0x00000000 abort_fatal() at abort_fatal+0x1d4 pc = 0xc05919ec lr = 0xc0591818 (abort_fatal) sp = 0xde9668c0 fp = 0xde966970 r4 = 0xde966978 r5 = 0x00000007 r6 = 0x00000013 r7 = 0x00000017 r8 = 0x00000000 abort_fatal() at abort_fatal pc = 0xc0591818 lr = 0xc057bf20 (exception_exit) sp = 0xde966978 fp = 0xde966a00 r4 = 0x00000000 r5 = 0x00000000 r6 = 0x00000000 r7 = 0xc2643440 r8 = 0xffffffec exception_exit() at exception_exit pc = 0xc057bf20 lr = 0xc02866c0 (free+0xc0) sp = 0xde9669c8 fp = 0xde966a00 r0 = 0x00000000 r1 = 0x00000001 r2 = 0xffffffec r3 = 0x00000000 r4 = 0xc26b2900 r5 = 0xc0740d50 r6 = 0x00000000 r7 = 0x00000000 r8 = 0x00000000 r9 = 0xc2643440 r10 = 0xffffffec r12 = 0x00000002 device_probe_child() at device_probe_child+0x298 pc = 0xc02e1110 lr = 0xc02e1d00 (device_probe+0x40) sp = 0xde966a08 fp = 0xde966a18 r4 = 0xc26b2900 r5 = 0xffffffff r6 = 0x00000000 r7 = 0xc26b2d00 r8 = 0xc06869f8 r9 = 0xc0692ec0 r10 = 0x00000000 device_probe() at device_probe+0x40 pc = 0xc02e1d00 lr = 0xc02e389c (bus_generic_driver_added+0x88) sp = 0xde966a20 fp = 0xde966a28 r4 = 0xc26b2900 r5 = 0xc2e2ff14 r6 = 0x00000000 bus_generic_driver_added() at bus_generic_driver_added+0x88 pc = 0xc02e389c lr = 0xc02e02a0 (devclass_driver_added+0x80) sp = 0xde966a30 fp = 0xde966a48 r4 = 0xc2e2ff14 r5 = 0xc2643440 devclass_driver_added() at devclass_driver_added+0x80 pc = 0xc02e02a0 lr = 0xc02e0208 (devclass_add_driver+0x12c) sp = 0xde966a50 fp = 0xde966a70 r4 = 0xc2e2ff14 r5 = 0xc2e2ff90 r6 = 0x7fffffff r7 = 0xc274d520 r8 = 0xc2643440 devclass_add_driver() at devclass_add_driver+0x12c pc = 0xc02e0208 lr = 0xc02e5224 (driver_module_handler+0x1ec) sp = 0xde966a78 fp = 0xde966a98 r4 = 0xc2e2fefc r5 = 0xc0692340 r6 = 0xc2c7fd00 r7 = 0x00000000 r8 = 0xc074cbac r9 = 0xc2c7fd00 r10 = 0xc2643440 driver_module_handler() at driver_module_handler+0x1ec pc = 0xc02e5224 lr = 0xc0289a8c (module_register_init+0x1fc) sp = 0xde966aa0 fp = 0xde966ad0 r4 = 0xc074cb80 r5 = 0xc0692340 r6 = 0xc2c7fd00 r7 = 0xc2e27970 r8 = 0xc074cbac r9 = 0xc0730ea8 r10 = 0xc2e2fec0 module_register_init() at module_register_init+0x1fc pc = 0xc0289a8c lr = 0xc027b430 (linker_load_module+0xc78) sp = 0xde966ad8 fp = 0xde966d38 r4 = 0xc074cbac r5 = 0xc0692340 r6 = 0xc072f9e0 r7 = 0xc2e27d7c r8 = 0xc2c7fd00 r9 = 0xc274d8c0 r10 = 0xc072f9b0 linker_load_module() at linker_load_module+0xc78 pc = 0xc027b430 lr = 0xc027d398 (kern_kldload+0x128) sp = 0xde966d40 fp = 0xde966d70 r4 = 0xde966d78 r5 = 0x00000000 r6 = 0xc26d5800 r7 = 0x00000001 r8 = 0xc072f9b0 r9 = 0xc072f9e0 r10 = 0x00000000 kern_kldload() at kern_kldload+0x128 pc = 0xc027d398 lr = 0xc027d508 (sys_kldload+0x64) sp = 0xde966d78 fp = 0xde966d88 r4 = 0xc2d68000 r5 = 0xc26d5800 r6 = 0x00000000 r7 = 0x00000000 r8 = 0xde966df0 r9 = 0xc2daa670 sys_kldload() at sys_kldload+0x64 pc = 0xc027d508 lr = 0xc05908fc (swi_handler+0x5e8) sp = 0xde966d90 fp = 0xde966e48 r4 = 0xc2d68000 r5 = 0xde966e50 r6 = 0xbffffe58 swi_handler() at swi_handler+0x5e8 pc = 0xc05908fc lr = 0xc057beb0 (swi_exit) sp = 0xde966e50 fp = 0xbffffe18 r4 = 0xbfffff42 r5 = 0x00000000 r6 = 0xbffffe58 r7 = 0x00000130 r8 = 0x00000000 r9 = 0xbffff9dc r10 = 0x00000000 swi_exit() at swi_exit pc = 0xc057beb0 lr = 0xc057beb0 (swi_exit) sp = 0xde966e50 fp = 0xbffffe18 Unable to unwind further