From owner-freebsd-net@freebsd.org Thu Oct 17 16:31:23 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0FABD156C6E for ; Thu, 17 Oct 2019 16:31:23 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46vF6L6hqlz4PMP; Thu, 17 Oct 2019 16:31:22 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from John-Baldwins-MacBook-Pro-4.local (ralph.baldwin.cx [66.234.199.215]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id 3B45799EB; Thu, 17 Oct 2019 16:31:22 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Subject: Re: panic on invalid ifp pointer in iflib drivers To: "Keller, Jacob E" , "freebsd-net@freebsd.org" Cc: "shurd@llnw.com" , "Joyner, Eric" References: <02874ECE860811409154E81DA85FBB589692E0D4@ORSMSX121.amr.corp.intel.com> From: John Baldwin Openpgp: preference=signencrypt Autocrypt: addr=jhb@FreeBSD.org; keydata= mQGiBETQ+XcRBADMFybiq69u+fJRy/0wzqTNS8jFfWaBTs5/OfcV7wWezVmf9sgwn8TW0Dk0 c9MBl0pz+H01dA2ZSGZ5fXlmFIsee1WEzqeJzpiwd/pejPgSzXB9ijbLHZ2/E0jhGBcVy5Yo /Tw5+U/+laeYKu2xb0XPvM0zMNls1ah5OnP9a6Ql6wCgupaoMySb7DXm2LHD1Z9jTsHcAQMD /1jzh2BoHriy/Q2s4KzzjVp/mQO5DSm2z14BvbQRcXU48oAosHA1u3Wrov6LfPY+0U1tG47X 1BGfnQH+rNAaH0livoSBQ0IPI/8WfIW7ub4qV6HYwWKVqkDkqwcpmGNDbz3gfaDht6nsie5Z pcuCcul4M9CW7Md6zzyvktjnbz61BADGDCopfZC4of0Z3Ka0u8Wik6UJOuqShBt1WcFS8ya1 oB4rc4tXfSHyMF63aPUBMxHR5DXeH+EO2edoSwViDMqWk1jTnYza51rbGY+pebLQOVOxAY7k do5Ordl3wklBPMVEPWoZ61SdbcjhHVwaC5zfiskcxj5wwXd2E9qYlBqRg7QeSm9obiBCYWxk d2luIDxqaGJARnJlZUJTRC5vcmc+iGAEExECACAFAkTQ+awCGwMGCwkIBwMCBBUCCAMEFgID AQIeAQIXgAAKCRBy3lIGd+N/BI6RAJ9S97fvbME+3hxzE3JUyUZ6vTewDACdE1stFuSfqMvM jomvZdYxIYyTUpC5Ag0ERND5ghAIAPwsO0B7BL+bz8sLlLoQktGxXwXQfS5cInvL17Dsgnr3 1AKa94j9EnXQyPEj7u0d+LmEe6CGEGDh1OcGFTMVrof2ZzkSy4+FkZwMKJpTiqeaShMh+Goj XlwIMDxyADYvBIg3eN5YdFKaPQpfgSqhT+7El7w+wSZZD8pPQuLAnie5iz9C8iKy4/cMSOrH YUK/tO+Nhw8Jjlw94Ik0T80iEhI2t+XBVjwdfjbq3HrJ0ehqdBwukyeJRYKmbn298KOFQVHO EVbHA4rF/37jzaMadK43FgJ0SAhPPF5l4l89z5oPu0b/+5e2inA3b8J3iGZxywjM+Csq1tqz hltEc7Q+E08AAwUIAL+15XH8bPbjNJdVyg2CMl10JNW2wWg2Q6qdljeaRqeR6zFus7EZTwtX sNzs5bP8y51PSUDJbeiy2RNCNKWFMndM22TZnk3GNG45nQd4OwYK0RZVrikalmJY5Q6m7Z16 4yrZgIXFdKj2t8F+x613/SJW1lIr9/bDp4U9tw0V1g3l2dFtD3p3ZrQ3hpoDtoK70ioIAjjH aIXIAcm3FGZFXy503DOA0KaTWwvOVdYCFLm3zWuSOmrX/GsEc7ovasOWwjPn878qVjbUKWwx Q4QkF4OhUV9zPtf9tDSAZ3x7QSwoKbCoRCZ/xbyTUPyQ1VvNy/mYrBcYlzHodsaqUDjHuW+I SQQYEQIACQUCRND5ggIbDAAKCRBy3lIGd+N/BCO8AJ9j1dWVQWxw/YdTbEyrRKOY8YZNwwCf afMAg8QvmOWnHx3wl8WslCaXaE8= Message-ID: <23f1e835-5dbb-055b-3768-f361311a9387@FreeBSD.org> Date: Thu, 17 Oct 2019 09:31:17 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <02874ECE860811409154E81DA85FBB589692E0D4@ORSMSX121.amr.corp.intel.com> Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Oct 2019 16:31:23 -0000 On 10/16/19 2:16 PM, Keller, Jacob E wrote: > Hi, > > I'm investigating an issue on the iflib ixl driver in 11.3-RELEASE as well as 12-RELEASE. We found a panic in that occurs if SCTP/IPv6 traffic is being transmitted while the device is detached: > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0xfffffe0000411e38 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff80c84700 > stack pointer = 0x28:0xfffffe2f4351b600 > frame pointer = 0x28:0xfffffe2f4351b650 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 12 (swi4: clock (0)) > trap number = 12 > panic: page fault > cpuid = 0 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe2f4351b2c0 > vpanic() at vpanic+0x17e/frame 0xfffffe2f4351b320 > panic() at panic+0x43/frame 0xfffffe2f4351b380 > trap_fatal() at trap_fatal+0x369/frame 0xfffffe2f4351b3d0 > trap_pfault() at trap_pfault+0x62/frame 0xfffffe2f4351b420 > trap() at trap+0x2b3/frame 0xfffffe2f4351b530 > calltrap() at calltrap+0x8/frame 0xfffffe2f4351b530 > --- trap 0xc, rip = 0xffffffff80c84700, rsp = 0xfffffe2f4351b600, rbp = 0xfffffe2f4351b650 --- > in6_selecthlim() at in6_selecthlim+0x20/frame 0xfffffe2f4351b650 > sctp_lowlevel_chunk_output() at sctp_lowlevel_chunk_output+0xeb2/frame 0xfffffe2f4351b790 > sctp_chunk_output() at sctp_chunk_output+0x68c/frame 0xfffffe2f4351c110 > sctp_timeout_handler() at sctp_timeout_handler+0x2d8/frame 0xfffffe2f4351c180 > softclock_call_cc() at softclock_call_cc+0x15b/frame 0xfffffe2f4351c230 > softclock() at softclock+0x7c/frame 0xfffffe2f4351c260 > intr_event_execute_handlers() at intr_event_execute_handlers+0x9a/frame 0xfffffe2f4351c2a0 > ithread_loop() at ithread_loop+0xb7/frame 0xfffffe2f4351c2f0 > fork_exit() at fork_exit+0x84/frame 0xfffffe2f4351c330 > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe2f4351c330 > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > KDB: enter: panic > > > From what I've gathered so far, it appears that the issue is a use-after-free where the SCTP stack gets an ifp pointer that's no longer valid. We've reproduced this issue on multiple iflib-based drivers, including ixl and the recently published ice driver code (available on phabricator). > > Additionally, we cannot reproduce it on legacy-stack drivers for ixl, or a mellanox 100G board we have. This leads me to believe that it's an issue in iflib rather than in the specific device drivers. > > I am not sure exactly what's going wrong here... anyone have suggestions? I thought it might be an issue of when ether_ifdetach is called. That function is supposed to clear all of the pre-existing routes from the route entry list. I'm thinking maybe somehow a route gets added after ether_ifdetach is called. > > In the iflib_device_deregister function, ether_ifdetach is called just after iflib_stop, (which would call a device's if_stop routine), and then the task queues are shutdown, a driver's ifdi_detach handler is called, and the ifp is free'd at the end. In the ixl legacy driver, ether_ifdetach is called prior to the stop routine. However, in the mlx5 driver, it's called after a call to close_locked()... > > So I'm really not sure exactly what could cause a stale ifp pointer to get into the route entry list. Nominally, ifnet drivers should call ether_ifdetach first to remove public references to the ifnet and only call their stop routine after that has returned. This ensures any open if_ioctl invocations have completed, etc. before the stop routine is invoked. Otherwise you are open to a race where the inteface can be upped via an ioctl after you have stopped the hardware. Any other references to the ifnet via eventhandlers, etc. should also be deregistered before calling the stop routine. After the hardware is stopped, interrupt handlers should be torn down and callouts and tasks drained to ensure there are no other references to the ifp outside of the thread running detach. After that you can release device resources, destroy mutexes, free the ifp, etc. Note that drivers have to be prepared for ether_ifdetach to invoke if_ioctl (e.g. when detaching bpf), but of the drivers I've looked at this has generally been a non-issue. It sounds like iflib should be doing the detach before calling iflib_stop. -- John Baldwin