From owner-freebsd-current@freebsd.org Tue May 29 11:20:48 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B557F739F9 for ; Tue, 29 May 2018 11:20:48 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-wm0-f53.google.com (mail-wm0-f53.google.com [74.125.82.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0B2D5845FF for ; Tue, 29 May 2018 11:20:47 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-wm0-f53.google.com with SMTP id o78-v6so39611397wmg.0 for ; Tue, 29 May 2018 04:20:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:openpgp:autocrypt:message-id :date:user-agent:mime-version:content-language :content-transfer-encoding; bh=GlTXdlKsojUqf/84rpoYYQffsUd7ulRXD+7xzLYC0ew=; b=U28czQNSq31I5poKrpmr2XbCJmksddnbACZ51TgOXMal0ZgtFGvJC88UGVB1ZZw4Wp +QO7ATsfjWU4x6ccC/3HWIto1zPLWqQZHd/jUDi/ssHh359bL0dmtrchJaQxIMGt+L4t A8caaMB6VO1TFS7j1UJKUYGB6Xuck2BPV6WrgHniWhff0P3rSuDO+yd0hP5/jkuCqwAS Sg7RwTHrShIsRn9KYYgZM1OLMStCWuGyjbgLvB/U56larKMe4SMHTYHv9DOjqOfYtIEM LYl5BJMi2PayzQXiPCDeKhCsfhK6jTHvaD0G5Bqnv+AW8Ddc85XtTdmzmq3xhfRd7dLz gUvQ== X-Gm-Message-State: ALKqPwfVcFNQ427BrOzBha+NsKEDQx3ZI1SRBWn/Mk63mFY47wGb0eHW pjMFYdkTc2Cd1GApxN1i0phWyjWw X-Google-Smtp-Source: ADUXVKIDp6fXsk+BO4VkOx4CYg12ksTcNQ12Bqwl0Cv+F9eeXVhFpmTg2HbbS0IN73sLLk6LkxxbYA== X-Received: by 2002:a2e:4082:: with SMTP id r2-v6mr10706006lje.48.1527592840433; Tue, 29 May 2018 04:20:40 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id t14-v6sm7293789lfi.92.2018.05.29.04.20.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 29 May 2018 04:20:39 -0700 (PDT) To: FreeBSD Current From: Andriy Gapon Subject: head@r334204, Bad link elm in callout_process() Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: Date: Tue, 29 May 2018 14:20:38 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2018 11:20:48 -0000 Has anyone else seen this panic? Any suspect commits? Any suggestion on analyzing the problem? Thanks! panic: Bad link elm 0xfffffe004bfed798 next->prev != elm cpuid = 0 time = 1527257839 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00413be770 vpanic() at vpanic+0x1a3/frame 0xfffffe00413be7d0 panic() at panic+0x43/frame 0xfffffe00413be830 callout_process() at callout_process+0x30d/frame 0xfffffe00413be8b0 handleevents() at handleevents+0x1a5/frame 0xfffffe00413be8f0 timercb() at timercb+0x254/frame 0xfffffe00413be940 lapic_handle_timer() at lapic_handle_timer+0xa7/frame 0xfffffe00413be980 Xtimerint() at Xtimerint+0xa5/frame 0xfffffe00413be980 --- interrupt, rip = 0xffffffff811cf8d6, rsp = 0xfffffe00413bea50, rbp = 0xfffffe00413bea50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00413bea50 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe00413beaa0 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe00413beac0 cpu_idle() at cpu_idle+0x95/frame 0xfffffe00413beae0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe00413bebb0 fork_exit() at fork_exit+0x84/frame 0xfffffe00413bebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00413bebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- KDB: enter: panic (kgdb) bt #0 kdb_enter (why=0xffffffff812f802e "panic", msg=) at /usr/devel/svn/head/sys/kern/subr_kdb.c:479 #1 0xffffffff80b93390 in vpanic (fmt=, ap=0xfffffe00413be810) at /usr/devel/svn/head/sys/kern/kern_shutdown.c:852 #2 0xffffffff80b93423 in panic (fmt=0xffffffff81df35b8 "\362\315+\201\377\377\377\377") at /usr/devel/svn/head/sys/kern/kern_shutdown.c:790 #3 0xffffffff80bab86d in callout_process (now=8718488703935) at /usr/devel/svn/head/sys/kern/kern_timeout.c:562 #4 0xffffffff8119a205 in handleevents (now=8718488703935, fake=0) at /usr/devel/svn/head/sys/kern/kern_clocksource.c:217 #5 0xffffffff8119a984 in timercb (et=0xffffffff81feb1d8 , arg=) at /usr/devel/svn/head/sys/kern/kern_clocksource.c:361 #6 0xffffffff811d9097 in lapic_handle_timer (frame=0xfffffe00413be990) at /usr/devel/svn/head/sys/x86/x86/local_apic.c:1308 #7 #8 acpi_cpu_c1 () at /usr/devel/svn/head/sys/x86/x86/cpu_machdep.c:137 #9 0xffffffff804602ee in acpi_cpu_idle (sbt=) at /usr/devel/svn/head/sys/dev/acpica/acpi_cpu.c:1187 #10 0xffffffff811cfeff in cpu_idle_acpi (sbt=178798794) at /usr/devel/svn/head/sys/x86/x86/cpu_machdep.c:430 #11 0xffffffff811cffa5 in cpu_idle (busy=0) at /usr/devel/svn/head/sys/x86/x86/cpu_machdep.c:578 #12 0xffffffff80bc28d7 in sched_idletd (dummy=) at /usr/devel/svn/head/sys/kern/sched_ule.c:2832 #13 0xffffffff80b524c4 in fork_exit (callout=0xffffffff80bc23c0 , arg=0x0, frame=0xfffffe00413bec00) at /usr/devel/svn/head/sys/kern/kern_fork.c:1039 #14 (kgdb) fr 3 #3 0xffffffff80bab86d in callout_process (now=8718488703935) at /usr/devel/svn/head/sys/kern/kern_timeout.c:562 562 } (kgdb) i loc first = 8719562445759 firstb = 519662 nowb = 519662 lookahead = last = 8720629104640 lastb = 519789 cc = 0xffffffff82011980 max = 8720629104640 tmp = tmpn = tmp_max = sc = (kgdb) p *cc $2 = { cc_lock = { lock_object = { lo_name = 0xffffffff8131227d "callout", lo_flags = 720896, lo_data = 0, lo_witness = 0x0 }, mtx_lock = 18446735277670449152 }, cc_exec_entity = {{ cc_curr = 0x0, cc_drain = 0x0, ce_migration_func = 0x0, ce_migration_arg = 0x0, ce_migration_cpu = 256, ce_migration_time = 0, ce_migration_prec = 0, cc_cancel = true, cc_waiting = false }, { cc_curr = 0x0, cc_drain = 0x0, ce_migration_func = 0x0, ce_migration_arg = 0x0, ce_migration_cpu = 256, ce_migration_time = 0, ce_migration_prec = 0, cc_cancel = false, cc_waiting = false }}, cc_next = 0x0, cc_callout = 0xfffffe0003841000, cc_callwheel = 0xfffffe0003963000, cc_expireq = { tqh_first = 0xffffffff81af68c8 , tqh_last = 0xffffffff81af68c8 }, cc_callfree = { slh_first = 0xfffffe00039622c0 }, cc_firstevent = 8718488678823, cc_lastscan = 8718488703935, cc_cookie = 0xfffff8000338fd00, cc_bucket = 28118, cc_inited = 1, cc_ktr_event_name = "callwheel cpu 0\000\000\000\000" } (kgdb) p cc->cc_callwheel[firstb & callwheelmask] $4 = { lh_first = 0xfffffe004bfed798 } (kgdb) p *$4.lh_first $5 = { c_links = { le = { le_next = 0xffffffff81bb08c8 , le_prev = 0xfffffe0003999f70 }, sle = { sle_next = 0xffffffff81bb08c8 }, tqe = { tqe_next = 0xffffffff81bb08c8 , tqe_prev = 0xfffffe0003999f70 } }, c_time = 8718482236373, c_precision = 26843543, c_arg = 0xfffffe004bfe9000, c_func = 0xffffffff8060cef0 , c_lock = 0x0, c_flags = 2, c_iflags = 20, c_cpu = 0 } (kgdb) p *$4.lh_first->c_links.le.le_next $6 = { c_links = { le = { le_next = 0x0, le_prev = 0xfffffe0003999f98 }, sle = { sle_next = 0x0 }, tqe = { tqe_next = 0x0, tqe_prev = 0xfffffe0003999f98 } }, c_time = 8718576725647, c_precision = 26843543, c_arg = 0x0, c_func = 0xffffffff8045bfa0 , c_lock = 0x0, c_flags = 2, c_iflags = 20, c_cpu = 0 } -- Andriy Gapon