From owner-freebsd-net@freebsd.org Tue May 14 11:02:53 2019 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A2C7415ABDAD for ; Tue, 14 May 2019 11:02:53 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 324958F9A9 for ; Tue, 14 May 2019 11:02:52 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f193.google.com with SMTP id r76so13863159lja.12 for ; Tue, 14 May 2019 04:02:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=BIe+F5EdIjCM2sPUAtmbCmG22aw/k43eqoabCZ1eLpw=; b=a5Xl0tp7hfldsza/9kVZfM/OU2XR8R+KdwXoMEI/Qn4/4y98hqDe4ohvaXAzUMAb6D S4VlrsotiqNQMEIwvTWbdU/lm5PlPCW/PVvF1ZueBGl5UMQBL4O9kNerehKgz4tAi6qD ZTPBwZlaVKXglCTxVikD6VWPz2pcnJjdh8o816MGLoIjpcrt2thE4qO1dAVWZllCojrl da8yu5KqR3QD1q2lNpuMRLAwwY0nJ/Btv5UowBebdu/Lioq6DVvsCv0dFv2gUmRm5bW8 df2S+NlWSBuvuZ/qbWPtcu7H3b4XrsryPhvFHsgF89wCGsWP/MuBjZU7CHy9fuqsoGbG ybEA== X-Gm-Message-State: APjAAAUr41/UBDXWY0aeCVf3leDBoAtBVxpX62LVCWX4cOP9tV6ajpH0 z6y0Jy4jCCqInc05N11+jUCuSpzN X-Google-Smtp-Source: APXvYqwbZpZrN9yGq66om70RkY2wKOADd/vBMEzzperBb0s2AuKl0tWI2B9A3Z6GiQycGhTSNKx3vw== X-Received: by 2002:a2e:98c6:: with SMTP id s6mr16553018ljj.161.1557831379988; Tue, 14 May 2019 03:56:19 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id c10sm2108614lfh.79.2019.05.14.03.56.19 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 May 2019 03:56:19 -0700 (PDT) Subject: Re: crash in dummynet, drain_scheduler_cb, si->sched == NULL From: Andriy Gapon To: freebsd-net@FreeBSD.org References: <60237dd7-e507-ecf3-c9b0-ef3c4c2116ad@FreeBSD.org> Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: Date: Tue, 14 May 2019 13:56:18 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <60237dd7-e507-ecf3-c9b0-ef3c4c2116ad@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 324958F9A9 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.193 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-4.17 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.98)[-0.977,0]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-1.18)[ipnet: 209.85.128.0/17(-3.59), asn: 15169(-2.27), country: US(-0.06)]; RCVD_IN_DNSWL_NONE(0.00)[193.208.85.209.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[193.208.85.209.rep.mailspike.net : 127.0.0.17] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2019 11:02:54 -0000 Just fixing the subject. On 14/05/2019 12:17, Andriy Gapon wrote: > > Unfortunately, all we have is some information from a ddb text dump. We do not > have a vmcore and we do not have a way to re-create the crash. It happened just > once on a production system. > > So, the information follows. > > dn_enqueue fs 0 si 0, dropping > > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x60 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff8077bdff > stack pointer = 0x28:0xfffffe1096343910 > frame pointer = 0x28:0xfffffe1096343920 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 0 (dummynet) > > db:3:psinfo> bt > Tracing pid 0 tid 100248 td 0xfffff8002829d4d0 > stack1 drain_scheduler_cb+0x1f drain_scheduler_sch_cb+0x25 > dn_ht_scan_bucket+0x7a dn_drain_scheduler+0x20 dummynet_task+0x219 > taskqueue_run_locked+0x71 taskqueue_thread_loop+0x56 fork_exit+0x121 > fork_trampoline+0xe > > drain_scheduler_cb+0x1f: movq 0x60(%rdx),%rax > > Here is disassembly of the function with some notes of mine: > 0xffffffff8077bde0 <+0>: push %rbp > 0xffffffff8077bde1 <+1>: testb $0x20,0x90(%rdi) // test DN_ACTIVE > 0xffffffff8077bde8 <+8>: mov %rsp,%rbp > 0xffffffff8077bdeb <+11>: jne 0xffffffff8077bdf4 > 0xffffffff8077bded <+13>: cmpq $0x0,0x78(%rdi) > 0xffffffff8077bdf2 <+18>: je 0xffffffff8077bdf8 > 0xffffffff8077bdf4 <+20>: leaveq > 0xffffffff8077bdf5 <+21>: xor %eax,%eax > 0xffffffff8077bdf7 <+23>: retq > 0xffffffff8077bdf8 <+24>: mov 0x88(%rdi),%rdx // rdx = si->sched > 0xffffffff8077bdff <+31>: mov 0x60(%rdx),%rax // rax = si->sched->fp > 0xffffffff8077be03 <+35>: testb $0x1,0x10(%rax) > > So, it seems that dummynet ran into dn_sch_inst with sched field being NULL. > > I am not sure how that could be possible. > Also, I am not sure if that "dn_enqueue ..." message is related to the crash. > > Does anyone have any ideas? > Thank you. > > P.S. > I found a somewhat similar but different and very old report: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=166937 > It seems that it was not really root-caused and fixed, but marked as fixed > because of a chance that it could have been caused by flaky hardware. > -- Andriy Gapon