From owner-freebsd-current@freebsd.org Sat May 9 20:33:45 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7C0702F5068 for ; Sat, 9 May 2020 20:33:45 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49KJnN4rBbz4JL7 for ; Sat, 9 May 2020 20:33:44 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f196.google.com with SMTP id f18so5296998lja.13 for ; Sat, 09 May 2020 13:33:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=IccQXs/1thqS8XvKaAJzbjfCvK1RfHGfh4P96C0wYyE=; b=dztVHnPwnM5jRk9/D/v/z1urtljlSnnAHTALN4BjTtbZR/N1v1sCkWUPM6z1cY7xl0 GUInq0Q3gf6N/ZOsfkBQ86OPBSqq6rq5yYk/iOrww7P6Fhq0e+f8opB9j/rmmoD/yyGd dUiFfsyH2SDF2DfMgwGC2wPxzcLdEsSAaB1UPP8+U/RryJVH7iBpfRcPOVjL4JME7i00 rbuDRyv4N8Jp8xupUlCF4n1w1h4IllUENXeHZL83y9z645Bge/VVr/Hs6CBJueFLfV5V CLCYd1Ga8OxIGUwnIKRx0gWFZ3t86TrIFm/yom09nUciPpkl4+/ew2i61AbICjIbRCGE lk/w== X-Gm-Message-State: AOAM532WYzHGrrMmlfmUMHI4larbuZLQOXFvRtBTcCf9o3RrRw/tg6ye DQUZoQErrZYXHESoOm7Fx2LnGfnoVF0= X-Google-Smtp-Source: ABdhPJwCf5DuOnvslQU4NRMNn/SciaIyBEVnpPI5RnMF5giBkqDEqUiJCen5I7yCnqedJZEcDLNfcg== X-Received: by 2002:a2e:9712:: with SMTP id r18mr5593686lji.225.1589056422564; Sat, 09 May 2020 13:33:42 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id a10sm4660922ljp.16.2020.05.09.13.33.41 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 09 May 2020 13:33:41 -0700 (PDT) Subject: Re: CHANGE_PV_LIST_LOCK_TO_PHYS is not correct when !NUMA ? To: Konstantin Belousov Cc: FreeBSD Current References: <0d7db402-621e-cc6b-2918-2078f63e2a9b@FreeBSD.org> <20200508161500.GC44519@kib.kiev.ua> <6485ab77-a3d0-8916-9431-74e4da1e3ea7@FreeBSD.org> <20200509161325.GH44519@kib.kiev.ua> <20200509165010.GI44519@kib.kiev.ua> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <88ebc8f9-b10d-366d-a12a-fb74417904bc@FreeBSD.org> Date: Sat, 9 May 2020 23:33:40 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Firefox/60.0 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <20200509165010.GI44519@kib.kiev.ua> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 49KJnN4rBbz4JL7 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.196 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-0.96 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; SUBJECT_HAS_EXCLAIM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.95)[-0.955,0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.99)[-0.988,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[196.208.85.209.list.dnswl.org : 127.0.5.0]; SUBJECT_ENDS_QUESTION(1.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[196.208.85.209.rep.mailspike.net : 127.0.0.17]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(-0.02)[ip: (0.76), ipnet: 209.85.128.0/17(-0.39), asn: 15169(-0.43), country: US(-0.05)] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.32 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 May 2020 20:33:45 -0000 On 09/05/2020 19:50, Konstantin Belousov wrote: > On Sat, May 09, 2020 at 07:16:27PM +0300, Andriy Gapon wrote: >> On 09/05/2020 19:13, Konstantin Belousov wrote: >>> On Sat, May 09, 2020 at 06:52:24PM +0300, Andriy Gapon wrote: >>>> On 08/05/2020 19:15, Konstantin Belousov wrote: >>>>> On Fri, May 08, 2020 at 06:53:24PM +0300, Andriy Gapon wrote: >>>>>> >>>>>> I have a reproducible panic with a custom kernel without option NUMA while using >>>>>> amdgpu driver from linuxkpi-based drm: >>>>>> >>>>>> panic: address 41ec00000 beyond the last segment >>>>>> >>>>>> I did some quick debugging and the panic happens when Xorg server tries to >>>>>> access a frame buffer (or something like that). There is a page fault that gets >>>>>> satisfied by ttm with a fictitious page. >>>>>> >>>>>> The stack trace is: >>>>>> #11 0xffffffff808031a3 in panic (fmt=0xffffffff8119a998 >>>>>> "5\003ʀ\377\377\377\377") at /usr/devel/git/motil/sys/kern/kern_shutdown.c:839 >>>>>> #12 0xffffffff80bbc552 in pmap_enter (pmap=, va=34504441856, >>>>>> m=, prot=, flags=, psind=>>>>> out>) at /usr/devel/git/motil/sys/amd64/amd64/pmap.c:6035 >>>>>> #13 0xffffffff80b288be in vm_fault_populate (fs=) at >>>>>> /usr/devel/git/motil/sys/vm/vm_fault.c:519 >>>>>> #14 vm_fault_allocate (fs=) at >>>>>> /usr/devel/git/motil/sys/vm/vm_fault.c:1032 >>>>>> #15 vm_fault (map=, vaddr=, fault_type=>>>>> out>, fault_flags=, m_hold=) at >>>>>> /usr/devel/git/motil/sys/vm/vm_fault.c:1342 >>>>>> #16 0xffffffff80b26e7e in vm_fault_trap (map=0xfffffe0017cd39e8, >>>>>> vaddr=, fault_type=, fault_flags=0, >>>>>> signo=0xfffffe00a810dbc4, ucode=0xfffffe00a810dbc0) at >>>>>> /usr/devel/git/motil/sys/vm/vm_fault.c:589 >>>>>> #17 0xffffffff80bcf89c in trap_pfault (frame=0xfffffe00a810dc00, >>>>>> usermode=, signo=, ucode=0xffffffff80853250 >>>>>> ) at /usr/devel/git/motil/sys/amd64/amd64/trap.c:821 >>>>>> #18 0xffffffff80bceeec in trap (frame=0xfffffe00a810dc00) at >>>>>> /usr/devel/git/motil/sys/amd64/amd64/trap.c:34 >>>>>> >>>>>> >>>>>> The line number in pmap_enter() is incorrect, I guess because of optimizations. >>>>>> The assert seems to be reached via pmap_enter -> CHANGE_PV_LIST_LOCK_TO_PHYS -> >>>>>> PHYS_TO_PV_LIST_LOCK -> pa_index(). >>>>>> >>>>>> The panic in correct in that the page is fictitious and its physical address is >>>>>> beyond the end of real physical memory. >>>>>> It seems that NUMA PHYS_TO_PV_LIST_LOCK() is aware of such pages, but !NUMA one >>>>>> is not. >>>>> >>>>> I think you can remove this assert. pa_index() is always taken by >>>>> % NVP_LIST_LOCKS, because fictitious mappings are not promoted. >>>>> >>>>> Try that and commit if it works for you. >>>> >>>> I tried this change: >>>> diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c >>>> index 4deed86a76d1a..b834b7f0388b7 100644 >>>> --- a/sys/amd64/amd64/pmap.c >>>> +++ b/sys/amd64/amd64/pmap.c >>>> @@ -345,7 +345,7 @@ pmap_pku_mask_bit(pmap_t pmap) >>>> #define NPV_LIST_LOCKS MAXCPU >>>> >>>> #define PHYS_TO_PV_LIST_LOCK(pa) \ >>>> - (&pv_list_locks[pa_index(pa) % NPV_LIST_LOCKS]) >>>> + (&pv_list_locks[((pa) >> PDRSHIFT) % NPV_LIST_LOCKS]) >>>> #endif >>>> >>>> #define CHANGE_PV_LIST_LOCK_TO_PHYS(lockp, pa) do { \ >>>> >>>> It fixed the original problem, but I got a new panic. >>>> "DI already started" in pmap_remove() -> pmap_delayed_invl_start_u(). >>>> I guess that !NUMA variant does not get much testing, so I'll probably just >>>> stick with the default. >>> Why didn't you just removed the KASSERT from pa_index ? >> >> Well, I thought it might be useful in the NUMA case. >> pa_index() definition is shared between both cases. > Might be define the macro two times, for NUMA/non-NUMA. non-NUMA case > does not need the assert, because users take it mod NPV_LIST_LOCKS. > I still don't see how that could help with "DI already started" panic. -- Andriy Gapon