From owner-freebsd-arm@freebsd.org Wed Dec 23 16:53:40 2015 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F3D82A4F081 for ; Wed, 23 Dec 2015 16:53:39 +0000 (UTC) (envelope-from zbb@semihalf.com) Received: from mail-lb0-x22f.google.com (mail-lb0-x22f.google.com [IPv6:2a00:1450:4010:c04::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9139F1E31 for ; Wed, 23 Dec 2015 16:53:39 +0000 (UTC) (envelope-from zbb@semihalf.com) Received: by mail-lb0-x22f.google.com with SMTP id sv6so42546146lbb.0 for ; Wed, 23 Dec 2015 08:53:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=ZMR3TfwaKlwamLjRVO9JUJ8JQOXYlRm8tMNQR8N2FpU=; b=xT8Vnrgtzg58FpDmfoDTu6gOV3FNx2axxHvQYVV2K2MYok7W0KLLv+0RWjeoAFrUgl cM/U2mNZYYNpZxK1YHJX4mKaUuvqedPDKissbUCnzYpDzQIDAk4etlkI+VBGvpeQa/yx VFO7g7k0I9NkY/3OASj/Mb7E0lTxRSbd2k07hRP+TLN3Y3ckO8xKGrd299ED72LKuhS8 TjoJmSmOQMwZyiFxQqhNI+hsnW0OmX/xEtnxLpIBAaNzh6eVAmhvdrCCc4ayNWxGn48F wY5sb2ICzVQkiZ8DbJSRG2j5f0lOCMl3WoVbnhDKXFWWJhikFH4m85p/JsumnDYh8Kwj bDXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=ZMR3TfwaKlwamLjRVO9JUJ8JQOXYlRm8tMNQR8N2FpU=; b=SZPysuSK8GoCBDuXOPhoWbRH5fx7tI+RbgpP3ZzFxolRJ9PrXK096NS5fuzBQj/8YB wZeHk4kpVu9XQuw8rUDqq03dU9hDhY/M8ILd7XppFFvnbKiseOA4LZr/ra2DExJGWXHG IyGbT9hSXPajpL+nLnqes6bvGNePWX8Y/bRb8Isd+zzXEgJOPhUuq+/jpXHrHEhsLf3A xeJFsnih/lBLp24wmSrcgsycoE53VOl5RHijvl0F/fvN7dijd+8nhGiyNsG0MVvCEUWb i79fVo93Y0/3hMpEz92AtevOOCoID4EMZseEAfmcGqdpQ9U1CooUUtrQKnuSYdUqGf0D f3og== X-Gm-Message-State: ALoCoQnD1iQ2RHwR5ZXULUq39bhmtb2M0NmkTfwRDXkp5QX3bVqAA2ng+F5U4j7ALMqT764bgqPl9KsWSbre6dMUR7jrD1/lkQ== X-Received: by 10.112.160.202 with SMTP id xm10mr8965463lbb.22.1450889616951; Wed, 23 Dec 2015 08:53:36 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.139.194 with HTTP; Wed, 23 Dec 2015 08:53:17 -0800 (PST) In-Reply-To: References: From: Zbigniew Bodek Date: Wed, 23 Dec 2015 17:53:17 +0100 Message-ID: Subject: Re: Various panics while using HWPMC on ARM64 To: Ruslan Bukin Cc: Andrew Turner , "freebsd-arm@freebsd.org" , Ed Maste Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Dec 2015 16:53:40 -0000 Hi, I go into new panics although quite similar to the previously mentioned in both SMP and UP but this time I tried to use DTrace. Can anyone check this on another ARM64 platform? I used ThunderX and FreeBSD-CURRENT r292654. Best regards zbb Please check out below for panic outputs: SMP: -------- root@thunder_crb4:~ # procsystime Tracing... Hit Ctrl-C to end... ^C Elapsed Times for all processes, x0: 1 x1: 0 x2: ffffff800052c911 x3: deadc0d8 x4: 1a2 x5: ffffff854a976860 x6: 0 x7: 100 x8: deadc0de x9: 0 x10: c4d4ae68 x11: 6c4d4ae68 x12: 80000000 x13: 0 x14: b x15: 274f x16: ffffff80481a97f8 x17: ffffff8048196814 x18: ffffff854a9764b0 x19: 989680 x20: ffffff8000708000 x21: ffffff8000702038 x22: ffffffc004219000 x23: ffffff8000702050 x24: 3938700 x25: 3938700 x26: ffffff80006e2000 x27: ffffffc0041129a0 x28: 39386ff x29: ffffff854a976530 x30: ffffff854a976530 sp: ffffff854a9764b0 lr: ffffff8000244760 elr: ffffff80002448e8 spsr: 600003c5 far: deadc174 esr: 96000007 panic: data abort in critical section or under mutex cpuid = 0 KDB: enter: panic [ thread pid 11 tid 100004 ] Stopped at kdb_enter+0x40: db> UP: -------- root@thunder_crb4:~ # kldload dtraceall IMPLEMENT ME: dtrace_toxic_ranges x0: ffffff8000299740 x1: ffffffc0040cd100 x2: ffffff8000299740 x3: ffffff8043655a70 x4: ffffff8000299740 x5: 0 x6: ffffff87cba9c0a8 x7: 0 x8: ffffffc0129484d0 x9: 1 x10: 1 x11: 0 x12: ffffffc003e7ed88 x13: ffffffc007ab0280 x14: 400000 x15: ffffff800068c000 x16: ffffff80436687b8 x17: ffffff8000299690 x18: ffffff87cba9c010 x19: ffffff8043655a70 x20: ffffff8000299740 x21: ffffff8000299740 x22: ffffffc0040cd100 x23: ffffffc003e7ed40 x24: 8 x25: 1 x26: ffffff8043669000 x27: ffffff8043669000 x28: ffffff8043669000 x29: ffffff87cba9c030 x30: ffffff87cba9c030 sp: ffffff87cba9c010 lr: ffffff80002996c0 elr: ffffffc0040cd100 spsr: 800003c5 panic: Unknown kernel exception 0 esr_el1 2000000 KDB: enter: panic [ thread pid 719 tid 100062 ] Stopped at kdb_enter+0x40: db> 2015-12-14 16:24 GMT+01:00 Zbigniew Bodek : > Hello, > > Did you have time to look into that? Do you have any clues what could > be wrong here? > We would like to use hwpmc for profiling so your help will be very > much appreciated. > > Best regards > zbb > > 2015-12-09 13:06 GMT+01:00 Zbigniew Bodek : >> Hello Ed, >> >> Done. I also check what happens when SMP is disabled and the kassert >> is triggered: >> >> root@thunderx_crb4:~ # pmcstat -S CPU_CYCLES -O cpu_cycles.pmc >> ^Cpanic: [pmc,4256] cpu 0 didn't find a sample to collect >> KDB: stack backtrace: >> db_trace_self() at db_trace_self_wrapper+0x28 >> pc = 0xffffff80004e9aac lr = 0xffffff800006d8b4 >> sp = 0xffffff87cba976e0 fp = 0xffffff87cba97800 >> >> db_trace_self_wrapper() at vpanic+0x9c >> pc = 0xffffff800006d8b4 lr = 0xffffff800027136c >> sp = 0xffffff87cba97810 fp = 0xffffff87cba97880 >> >> vpanic() at kassert_panic+0x160 >> pc = 0xffffff800027136c lr = 0xffffff80002712cc >> sp = 0xffffff87cba97890 fp = 0xffffff87cba97950 >> >> kassert_panic() at pmc_capture_user_callchain+0x1a4 >> pc = 0xffffff80002712cc lr = 0xffffff80000e1444 >> sp = 0xffffff87cba97960 fp = 0xffffff87cba979c0 >> >> pmc_capture_user_callchain() at pmc_hook_handler+0x7c0 >> pc = 0xffffff80000e1444 lr = 0xffffff80000dfb78 >> sp = 0xffffff87cba979d0 fp = 0xffffff87cba97a50 >> >> pmc_hook_handler() at ast+0x14c >> pc = 0xffffff80000dfb78 lr = 0xffffff80002b976c >> sp = 0xffffff87cba97a60 fp = 0xffffff87cba97a90 >> >> ast() at handle_el0_sync+0x90 >> pc = 0xffffff80002b976c lr = 0xffffff80004eb224 >> sp = 0xffffff87cba97aa0 fp = 0xffffff87cba97bb0 >> >> handle_el0_sync() at 0x406d60 >> pc = 0xffffff80004eb224 lr = 0x0000000000406d60 >> sp = 0xffffff87cba97bc0 fp = 0x0000007ffffff540 >> >> KDB: enter: panic >> [ thread pid 679 tid 100061 ] >> Stopped at kdb_enter+0x40: >> db> >> >> >> when invariants options is disabled I only get: >> >> root@thunderx_crb4:~ # pmcstat -S CPU_CYCLES -O cpu_cycles.pmc >> ^Cpmcstat: WARNING: sampling was paused at least 1 time. >> Please consider tuning the "kern.hwpmc.nsamples" tunable. >> >> >> Best regards >> zbb >> >> 2015-12-08 20:59 GMT+01:00 Ed Maste : >>> On 8 December 2015 at 14:34, Zbigniew Bodek wrote: >>>> Hello, >>>> >>>> I encountered some problems with FreeBSD on ARM64 while using hwpmc. >>>> Some of the errors that I found are listed below: >>>> >>>> * panic: Unknown kernel exception 0 esr_el1 2000000 >>>> * panic: data abort in critical section or under mutex >>>> * panic: VFP exception in the kernel >>>> * panic: Unknown kernel exception 21 esr_el1 86000006 >>> >>> Can you add these notes to PR 204686? I think there are SMP issues in >>> arm64 hwpmc that need to be resolved.