From owner-freebsd-current@freebsd.org Fri Feb 15 07:51:05 2019 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3BE1B14F3207 for ; Fri, 15 Feb 2019 07:51:05 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf1-f41.google.com (mail-lf1-f41.google.com [209.85.167.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 34C8A836C3; Fri, 15 Feb 2019 07:51:04 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf1-f41.google.com with SMTP id t14so6493725lfk.7; Thu, 14 Feb 2019 23:51:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=JWBamcafWogNOCIklhB1QSw+yZJB51hBQbxJYxXqtWA=; b=t6aIO2DPx4VbIIdRbj068i0l+WUzPuJbiIem5PnxwLuZN+MKMXf5w0kveu2VoaAmTu gnWJkn4E6R6T0dkwJccslfH54LSDnsy0m1Rsge+/sHJf4Qd9FyG0Wqfo1ytcYIU5+3Ch +4j41zSuRgxLTCLvLksvSR+u5CAzb5vJyqcBeQzPNre1aokd7DF9d8tE6YU1vjl03C+q 1yBJxDXGRrEyimzyXjP8gFdeoglwYE4MpI6zggRRKBoYs8BxNbpCuB63ZbDfS8P6mIhA j5fHeNiKUvLRgOKobFz8zWxtIyqxsn429DIB1nsw0sDz2UHneFRtAOAxVy3CrI9gUUxA saiw== X-Gm-Message-State: AHQUAuYon4U1JSNXX6WDBDslYGzTUs+FtFx7vySZILlm5s3rHnlWaZ6k nbffaH2BnwBaIA+VwPRLDVfp2K0X X-Google-Smtp-Source: AHgI3IZ70IzecEEJXd6kNe9k+DdguQySBdXbrfYU7hsizLFFO8dnMmrICGJBjtEE/mUNLHIBQJjaOg== X-Received: by 2002:a19:f508:: with SMTP id j8mr4672872lfb.87.1550217057005; Thu, 14 Feb 2019 23:50:57 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id d24-v6sm995212ljg.2.2019.02.14.23.50.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Feb 2019 23:50:56 -0800 (PST) Subject: Re: Panic in sys_fstatat() To: John Baldwin , sgk@troutmask.apl.washington.edu, freebsd-current@freebsd.org References: <20190214024703.GA51003@troutmask.apl.washington.edu> <786f8034-b3ef-54cb-043b-e189e752b18b@FreeBSD.org> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <03a9628d-9101-c9c4-fb4a-96bc57308c38@FreeBSD.org> Date: Fri, 15 Feb 2019 09:50:55 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <786f8034-b3ef-54cb-043b-e189e752b18b@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 34C8A836C3 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.167.41 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-4.05 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[FreeBSD.org]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.88)[-0.876,0]; RCVD_IN_DNSWL_NONE(0.00)[41.167.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; IP_SCORE(-1.17)[ipnet: 209.85.128.0/17(-3.78), asn: 15169(-1.99), country: US(-0.07)]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Feb 2019 07:51:05 -0000 On 14/02/2019 22:26, John Baldwin wrote: > On 2/13/19 6:47 PM, Steve Kargl wrote: ... >> panic: vm_fault_hold: fault on nofault entry, addr: 0x202000 What's very suspicious here is that the fault address looks a lot like LK_SHARED | LK_NODDLKTREAT, which would be 'flags' passed to vn_lock and which should never be used as an address. In a later email Steve reported that cn_lkflags = 2097152 and that's 0x200000, LK_SHARED. compute_cn_lkflags() adds LK_NODDLKTREAT. However, LK_RETRY is missing. >> cpuid = 1 >> time = 1550111772 >> KDB: stack backtrace: >> db_trace_self_wrapper(10b42f3,8c96000,1,9341bd0,2e7b6590,...) at db_trace_self_wrapper+0x2a/frame 0x2e7b6560 >> kdb_backtrace(109973a,5c64d41c,0,2e7b661c,1,...) at kdb_backtrace+0x2d/frame 0x2e7b65c8 >> vpanic(108d309,2e7b661c,2e7b661c,2e7b6700,f734a9,...) at vpanic+0x141/frame 0x2e7b65fc >> panic(108d309,103dfa3,202000,2e7b6664,2e7b6654,...) at panic+0x1b/frame 0x2e7b6610 >> vm_fault_hold(1ea5000,202000,1,0,0,...) at vm_fault_hold+0x29e9/frame 0x2e7b6700 >> vm_fault(1ea5000,202000,1,0,0,...) at vm_fault+0x5e/frame 0x2e7b6728 >> trap_pfault(202462,40,109e2f2,316d3480,2e7b67c0,...) at trap_pfault+0xb2/frame 0x2e7b6770 >> trap(2e7b6880,8,28,28,1836a120,...) at trap+0x3cb/frame 0x2e7b6874 >> calltrap() at PTDpde+0x4165/frame 0x2e7b6874 >> --- trap 0xc, eip = 0x1027fb8, esp = 0x2e7b68c0, ebp = 0x2e7b68f8 --- >> VOP_LOCK1_APV(1836a120,202400,1099cc5,2c8,2e7b6ab0,...) at VOP_LOCK1_APV+0x8/frame 0x2e7b68f8 And [0x]202400 here confirms the above observations. [0x]2c8 is 712, the line number in vfs_lookup.c. >> lookup(2e7b6a50,0,400,2e7b6aa0,2e7b6a18,...) at lookup+0xc4/frame 0x2e7b6960 >> namei(2e7b6a50,0,4000144,0,2cced08e,...) at namei+0x4f3/frame 0x2e7b6a20 >> kern_statat(3c5dc700,0,ffffff9c,2cced08e,0,...) at kern_statat+0x85/frame 0x2e7b6af0 >> sys_fstatat(3c5dc700,3c5dc988,1384bb0,3c5dc700,0,...) at sys_fstatat+0x49/frame 0x2e7b6c00 >> syscall(2e7b6ce8,3b,3b,3b,fbafbbc8,...) at syscall+0x3ea/frame 0x2e7b6cdc >> Xint0x80_syscall() at PTDpde+0x43af/frame 0x2e7b6cdc > > Frame 18 is probably the root problem, though it doesn't look like kgdb is > able to unwind it correctly. Looking at frame 19 might help though. It > seems like a NULL pointer dereference when invoking VOP_LOCK. > So, I suspect something exotic like some sort of a stack alignment issue, or a CPU bug, or a mismatch between object files, or some local experiment, etc. -- Andriy Gapon