From owner-freebsd-drivers@freebsd.org Mon Jan 6 14:40:16 2020 Return-Path: Delivered-To: freebsd-drivers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7213B1E6B08; Mon, 6 Jan 2020 14:40:16 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x841.google.com (mail-qt1-x841.google.com [IPv6:2607:f8b0:4864:20::841]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47rypl10Bcz3Jbv; Mon, 6 Jan 2020 14:40:14 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qt1-x841.google.com with SMTP id d5so42626608qto.0; Mon, 06 Jan 2020 06:40:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=xIOySfCvlml6aHUEwcHEjaRQSFhQ8mxxsx0kVoYakmg=; b=cug2jdKc2TRCURa1T95YZdLVnMrXsD3LARGkkZGXziZNdIzFjAB8MH8O3h5pWwEIJ+ 1ItvQ/kuwyieFnB4/ByD/jCl9bSUIp+DTNqjOz3PxpKVtE0OeC9mZ2UX/tOHoS8tbrf7 xIVnb3rLAZRo7NYP4jFR6vbfSG5KAZAH627OGdwF2ZcPqlCDHueXO9LT06qpNziO4SjH uTgyLx+DYGKehOnswkYgBfZwM3u8IGCKgwt/fSsiA6nGdz1fhF1UOK1MpBoz5JXrnZ1m Z2gP+j1P4a5XDFbQIegNP72E6k8lV88aL29IGUcUs1OPINy7r8DEvXszdIrzquZStfQa Wkrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to; bh=xIOySfCvlml6aHUEwcHEjaRQSFhQ8mxxsx0kVoYakmg=; b=fqLUIxpyU9CgRhChmCADrhoGGFuJNMqg/3PlOBxD2p9W8SKrcqyYvppvNKMCkOI9te ACwKaaqPMzJkhR+TKkLTx1E09ZPkIBzzl6Rdm9o27YHXzFLEWHEzen6v6BykGUl5gzf7 3jlC4tr7t7pc/2Ugj8ccq4ezaF5G4AE0cm9FJGeDowAFn7UhrRB7fiB3wu8MSvz/gKb0 a1Dz+gQ5/kZzahpLccjiAdd2mf5XJC70p7BNOLZResXzoLp0EmZK0KIq8cNRA7hH8L48 bIzV15GY+t3HjquiPxNholJ/N2BS3u5WL6cYxvDIA4gF4p/V7Hw6hDyD2VVJdxCWiWRZ A4WQ== X-Gm-Message-State: APjAAAW8UW8fteAl/c4JH39TY2yvaSJLc99I4q9vXNgfi0Y2kE6gzWne PuvI3Z5g+VVhw8QVDB6roDE= X-Google-Smtp-Source: APXvYqy3xpR8zwXJLJeC1m3Un1WT+wiqMFY9o4waZQLnnoml476qVEoOJ1/vYZRKftj9X2mPIhaWvA== X-Received: by 2002:ac8:2a06:: with SMTP id k6mr75800114qtk.145.1578321614228; Mon, 06 Jan 2020 06:40:14 -0800 (PST) Received: from spy ([24.114.48.177]) by smtp.gmail.com with ESMTPSA id p126sm20698013qke.108.2020.01.06.06.40.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jan 2020 06:40:13 -0800 (PST) Sender: Mark Johnston Date: Mon, 6 Jan 2020 09:40:08 -0500 From: Mark Johnston To: Rajesh Kumar Cc: freebsd-drivers@freebsd.org, freebsd-amd64@freebsd.org Subject: Re: panic: smp_targeted_tlb_shootdown: interrupts disabled - FreeBSD 12.0 Message-ID: <20200106144008.GA23672@spy> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 47rypl10Bcz3Jbv X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=cug2jdKc; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::841 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-2.01 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-0.99)[-0.993,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; NEURAL_HAM_LONG(-1.00)[-0.997,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_IN_DNSWL_NONE(0.00)[1.4.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.32)[ip: (2.42), ipnet: 2607:f8b0::/32(-2.13), asn: 15169(-1.85), country: US(-0.05)]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[177.48.114.24.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jan 2020 14:40:16 -0000 On Thu, Jan 02, 2020 at 11:17:10PM +0530, Rajesh Kumar wrote: > Hi, > > I am hitting the below panic when I am testing couple of my drivers. When i > looked at the source, the reason is PSL_I bit (Interrupt enabled) bit is > not set when read in "smp_targeted_tlb_shootdown" routine. But I am > wondering what leads to this. > > *Driver test scenario:* > 1. Driver has a kernel thread, which allocates memory and dma maps it and > send the dma addr to Driver 2. Then wait (msleep) till it gets a > notification from Driver 2. > 2. Driver 2 has a kernel thread which will submit the received address from > Driver1 to hardware and wait (msleep) for an interrupt > 3. Driver 2 receives a interrupt and wakes up the thread (wakeup_one) in > Driver2. Thread in Driver 2 then notifies Driver1 through a callback. > 4. Callback in Driver 1 wakeup (wakeup_one) it's sleeping kernel thread. > The kernel thread then dma unmap the buffer the frees the memory > > Kernel Panics while the memory is freed at last. I am sure this is a > driver, but not sure what am I doing wrong here. Please help. Does your driver hold a spin mutex when it calls kmem_free()? If so, that is causing your problem: interrupts are disabled while a spin mutex is held. Most kernel interfaces do not expect to be called with a spin mutex held and will misbehave in this case. > panic: smp_targeted_tlb_shootdown: interrupts disabled > cpuid = 44 > time = 1577463701 > KDB: stack backtrace: > #0 0xffffffff80be7977 at kdb_backtrace+0x67 > #1 0xffffffff80b9b563 at vpanic+0x1a3 > #2 0xffffffff80b9b3b3 at panic+0x43 > #3 0xffffffff811eea8f at smp_targeted_tlb_shootdown+0x40f > #4 0xffffffff811eeb22 at smp_masked_invlpg_range+0x42 > #5 0xffffffff8105e67c at pmap_invalidate_range+0x25c > #6 0xffffffff81061cdd at pmap_remove_ptes+0x6d > #7 0xffffffff8106079b at pmap_remove+0x35b > #8 0xffffffff80ee0366 at _kmem_unback+0x36 > #9 0xffffffff80ee04cd at kmem_free+0x2d > #10 0xffffffff829fc5f8 at driver_thread_work+0x798 > #11 0xffffffff80bf9cb4 at taskqueue_run_locked+0x154 > #12 0xffffffff80bfae18 at taskqueue_thread_loop+0x98 > #13 0xffffffff80b5bf33 at fork_exit+0x83 > #14 0xffffffff810501be at fork_trampoline+0xe > > Please let me know if any details needed. > > Thanks, > Rajesh. > _______________________________________________ > freebsd-amd64@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-amd64 > To unsubscribe, send any mail to "freebsd-amd64-unsubscribe@freebsd.org"