From owner-freebsd-net@freebsd.org Wed Jul 8 10:30:03 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E835E35E920 for ; Wed, 8 Jul 2020 10:30:03 +0000 (UTC) (envelope-from vegeta@tuxpowered.net) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4B1wY74qvXz3cfY for ; Wed, 8 Jul 2020 10:30:03 +0000 (UTC) (envelope-from vegeta@tuxpowered.net) Received: by mailman.nyi.freebsd.org (Postfix) id A3DDD35E79F; Wed, 8 Jul 2020 10:30:03 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A398335E6C4 for ; Wed, 8 Jul 2020 10:30:03 +0000 (UTC) (envelope-from vegeta@tuxpowered.net) Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B1wY64B0Cz3cLM for ; Wed, 8 Jul 2020 10:30:02 +0000 (UTC) (envelope-from vegeta@tuxpowered.net) Received: by mail-ed1-x534.google.com with SMTP id e15so41224216edr.2 for ; Wed, 08 Jul 2020 03:30:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuxpowered-net.20150623.gappssmtp.com; s=20150623; h=to:from:subject:autocrypt:message-id:date:user-agent:mime-version; bh=VLhMkV7A37FRbrH1mabzCb46A9ldXiCH4t3Y5C9jYIA=; b=bHQBHqodZb2qqTISqEHTCb/IRhejNhK8W1I/oqdqTTnrT32C48RFioC9GkuXsH6n5B PZoQypkcOpQy8OeMWsf9CIfAVcCi8tL1GrZ0J6JWkd+g9Fn5ZLDII9x2VvvUlFMcrR1F v/VYWPp+MZ/vAUXXc7VTj6Nqf8DI/CuqsnGM2WBp+BrLioQGOebjB25c8723HQtSLwOf pKakpo9q8J02y5O00U9a7wkdHYFG9y49F1drD5aPlub6snipmAHILwpwg/3mRv5LzeNZ kA0pMkplOUbxl1n62zCL6EX46mX4Gij8n60VlE8A35rg2dW97lIkfLRWtvD3EonAu+ET PWrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:autocrypt:message-id:date :user-agent:mime-version; bh=VLhMkV7A37FRbrH1mabzCb46A9ldXiCH4t3Y5C9jYIA=; b=rziu5CBCBfaB6AMRvxhzJZiwYAuVuP9BWD+sn+gv77dGTw/WW3RAe9lwk7q6wmj2Ol agxELhK1gsKXb7GnAkEJuWJ+FuUn5z25dD0WsOb8nsZ4T1/yaDE7q1et1dsKYP01L6T6 /c+WZbsaPMvyQidf6wCujF1NOGOPWoq6tXdsLuFY7FUjkw0HYO4KFVefjf1IHRYv4iEZ 6Rkey6/Evh4ceIKQJxpkiaHcp9j4y+A8y6EHVXcRyoZDPIgzWcJloKTSK836W54kzT6n WihBPspvLIMdaiNWC+RlzvwzaqaXDqvZ3QpsvLLrWLZrIEfSKRMEwf0V8ujQDvVS9tJG cHSg== X-Gm-Message-State: AOAM5301ueBbM+9FLHuJf7zehSehfZaQLcEJHasQkxQwYY76EAASm+22 Trb7RbxnGvnyT4okJU3pKDaDYO9CqEo= X-Google-Smtp-Source: ABdhPJxwZ+q774aVGKyy6dsm1VOrVoUsVpNs8taScyANVm2fPSwEu2STPrJQyuvccBIafbChvTsyfQ== X-Received: by 2002:a05:6402:cb3:: with SMTP id cn19mr64176256edb.368.1594204198892; Wed, 08 Jul 2020 03:29:58 -0700 (PDT) Received: from Proton.local ([2a02:8108:50bf:eb94:c0e3:6edb:170:1bcd]) by smtp.gmail.com with ESMTPSA id y22sm1732149ejf.108.2020.07.08.03.29.57 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 08 Jul 2020 03:29:57 -0700 (PDT) To: net@FreeBSD.org From: Kajetan Staszkiewicz Subject: Deadlocks when using pf tags or socket owner matching Autocrypt: addr=vegeta@tuxpowered.net; keydata= mQGiBELvVycRBADVGZM8mHAsH+R87EBg4O+QTOkL0TjroqamohMlCdBEZgFGcGVoKA9c9Az6 e7xpk90DuaWYrzBKJ+I5drx2ddqdqejLhgNm3QZubE8Cf9cCxBAxnxBZHzmmgVJMOg93lJUQ e9L1BstntodE2xz4jSBB++Zh9eZgRqbn/EICcQmmKwCg9pQfnXRAMr4tFxhsFenxa/JCvFME AK/03irNfB8DezORCfpt7lZuwL5oRJ/TvpoCfwgVkNd6gTLMgSQpKbFytLzAAmRsE+EwVpBo sUzKt4vzmW4bllgPao14TyuVcViah27/da3fHm1HIMkjvro/ONtUivInn+5L33S0meT3KyuK ofwc1A6KucNxhv4rG7RsXuhwZZmQA/0QVni2wq7yc6t15dfCxuDCxG7yXp4pE5Dghp/MMwts leIxJ3JdHaTZ9aIrYT2Rxw8mTXUs89pDi7PCqXA2N4C+RvkoZI0Q6cWs6jHNZGiZRVzkw38r 8ctqtAlcfzlAynX5+Ym9oiNMJ/c/4fAiFrWerMR1rFWDSD56ltQHk0X0oLQsS2FqZXRhbiBT dGFzemtpZXdpY3ogPHZlZ2V0YUB0dXhwb3dlcmVkLm5ldD6IewQTEQgAOwYLCQgHAwIDFQID AxYCAQIeAQIXgAIZARYhBI4RBk5u/YHyZ/QlueO0UK9tezoUBQJd5qIlBQkeucAOAAoJEOO0 UK9tezoUR7cAoIk0VDEW+znh8hbw3zDgnLhKMOj3AJ92fUPkB7huUEtUHtnjJWXNlHAnqLkB DQRC71cpEAQAjXEOKfj9O4eYTWcifEApMYzel9+aWmhNRqqUhJuNO40UDF73biRJ0cjd8miV hZGxcqIdjnZUmxn8Okr+ta7ZU4Q2KNw7B23VKd1jzDKalaUGtCbv8pnvFdBCJwwzdhHJ2vxr e7zkGMrU4x5Od/92YZRCgX229Ic8y7muveQty4sAAwYD/A/FKDQkIu16GVOu9g8ZBLLBi1HS h2eiem/efmfZS1APR7Q5Ouf6KJMeEgBCKY9yqEp9wg97Bt93oi3zP0H1I8rLmrj5hoEE/VEj Cc4XSQ3qrthmQ9bE8fPDZIgodPG1h+dlOzDQoUxKM/YZdbKmV8VkegbAmEng9rJk90gJ+7Qt iGMEGBEIACMWIQSOEQZObv2B8mf0JbnjtFCvbXs6FAUCXhcbkAUJHuo5dwAKCRDjtFCvbXs6 FK0WAJ96lSx5vRW6HaUbplfnVnARDYNo5gCgvYSyV/mBrnEGfUgA5CAx9Jdudog= Message-ID: <48de3ee4-ba52-f6a7-b2d7-c7d77a635eb9@tuxpowered.net> Date: Wed, 8 Jul 2020 12:29:55 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6Of8gmt13SoPjymQr9LyK3pjeHp5qDQwP" X-Rspamd-Queue-Id: 4B1wY64B0Cz3cLM X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tuxpowered-net.20150623.gappssmtp.com header.s=20150623 header.b=bHQBHqod; dmarc=none; spf=pass (mx1.freebsd.org: domain of vegeta@tuxpowered.net designates 2a00:1450:4864:20::534 as permitted sender) smtp.mailfrom=vegeta@tuxpowered.net X-Spamd-Result: default: False [-4.64 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[tuxpowered-net.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-0.96)[-0.959]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[net@freebsd.org]; TO_DN_NONE(0.00)[]; HAS_ATTACHMENT(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.04)[-1.040]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[tuxpowered.net]; DKIM_TRACE(0.00)[tuxpowered-net.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-0.04)[-0.039]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::534:from]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jul 2020 10:30:04 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --6Of8gmt13SoPjymQr9LyK3pjeHp5qDQwP Content-Type: multipart/mixed; boundary="UkvR4H5J5VCKr7C31oruLwzrE00EhNufW" --UkvR4H5J5VCKr7C31oruLwzrE00EhNufW Content-Type: text/plain; charset=utf-8 Content-Language: en-GB Content-Transfer-Encoding: quoted-printable Hello group, I've decided to give a try to two functions of pf: tags, so that I can match packets on outgoing side of my loadbalancer to the ones which came in and socket ownership, so that I can match packets originating on the loadbalancer itself (dns, puppet and such). My kernel has some additional patches so I'm starting here before reporting this in Bugzilla= =2E After a few hours runnig with the new ruleset some of my systems freeze. I've built a kernel without KDB_UNATTENDED, and NMI from IPMI card got me the debugger console where I dumped the memory and rebooted the system. So far so good. Now I'm having problems understading who holds the lock which causes trouble. Looking at what NIC queue threads do, I find that all of threads apart from one are locked on backtraces similar (the exact path to reach pf_test() might be a bit different, or it's pf_test6 instead) to this: info threads: 85 Thread 100096 (PID=3D12: intr/irq301: ixl0:q7) sched_switch (td=3D0xfffff800118c6620, newtd=3D0xfffff8001513b620, flags=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/sched_ule.c:2103 (kgdb) thr 85 [Switching to thread 85 (Thread 100096)]#0 sched_switch (td=3D0xfffff800118c6620, newtd=3D0xfffff8001513b620, flags=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/sched_ule.c:2103 2103 /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/sched_ule.c: No such file or directory. in /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/sched_ule.c (kgdb) bt #0 sched_switch (td=3D0xfffff800118c6620, newtd=3D0xfffff8001513b620, flags=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/sched_ule.c:2103 #1 0xffffffff806719d8 in mi_switch (flags=3D259, newtd=3D0x0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/kern_synch.c:439 #2 0xffffffff806c3741 in turnstile_wait (ts=3D0xfffff8001503f180, owner=3D, queue=3D-500) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/subr_turnstile.c:751 #3 0xffffffff80647d81 in __mtx_lock_sleep (c=3D0xffffffff8184f278, v=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/kern_mutex.c:637 #4 0xffffffff80661e53 in _rm_rlock (rm=3D0xffffffff8184f220, tracker=3D0xfffffe0c04808a38, trylock=3D0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/kern_rmlock.c:408 #5 0xffffffff81825604 in pf_test (dir=3D2, ifp=3D0xfffff80012f9a000, m0=3D0xfffffe0c04808cc8, inp=3D0x0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/netpfil/pf/pf.c:6006 #6 0xffffffff81826759 in pf_test (dir=3D1, ifp=3D, m0=3D, inp=3D0x0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/netpfil/pf/pf.c:5594 #7 0xffffffff818370ed in pf_check_in (arg=3D, m=3D0xfffffe0c04808dc0, ifp=3D, dir=3D, flags=3D, inp=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/netpfil/pf/pf_ioctl.c:3916= #8 0xffffffff8077b647 in pfil_run_hooks (ph=3D0xffffffff81335108, mp=3D, ifp=3D0xfffff80012f2c000, dir=3D1, flags=3D0,= inp=3D0x0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/net/pfil.c:1= 09 =2E.. My pf.c, function pf_test(): 6003 6004 pd.pf_mtag =3D pf_find_mtag(m); 6005 6006 PF_RULES_RLOCK(); 6007 6008 if (ip_divert_ptr !=3D NULL && 6009 ((ipfwtag =3D m_tag_locate(m, MTAG_IPFW_RULE, 0, NULL)) !=3D NULL)) { 6010 struct ipfw_rule_ref *rr =3D (struct ipfw_rule_ref *)(ipfwtag+1); All those NIC queues threads and some pfctl processes (the loadbalancer collects stats from pf labels or updates loadbalancing tables) are waiting for one more (I hope that's the correct way to get this information): (kgdb) print rm->rm_activeReaders->lh_first->rmp_thread->td_name $27 =3D 0xfffff800118c3294 "irq296: ixl0:q2" [9:17:44] hwlb-foe-aw-02 ~/ # grep ixl0:q2 ps.core2 90 Thread 100091 (PID=3D12: intr/irq296: ixl0:q2) sched_switch (td=3D0xfffff800118c3000, newtd=3D0xfffff8000b0d7000, flags=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/sched_ule.c:2103 And that one is waiting on: (kgdb) thr 90 [Switching to thread 90 (Thread 100091)]#0 sched_switch (td=3D0xfffff800118c3000, newtd=3D0xfffff8000b0d7000, flags=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/sched_ule.c:2103 2103 in /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/sched_ule.c (kgdb) bt #0 sched_switch (td=3D0xfffff800118c3000, newtd=3D0xfffff8000b0d7000, flags=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/sched_ule.c:2103 #1 0xffffffff806719d8 in mi_switch (flags=3D259, newtd=3D0x0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/kern_synch.c:439 #2 0xffffffff806c3741 in turnstile_wait (ts=3D0xfffff8000b9ecb40, owner=3D, queue=3D1) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/subr_turnstile.c:751 #3 0xffffffff80663511 in __rw_rlock_hard (rw=3D0xffffffff81336188, td=3D0xfffff800118c3000, v=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/kern_rwlock.c:601 #4 0xffffffff80795f61 in in_pcblookup_hash (pcbinfo=3D0xffffffff81336130= , faddr=3D{s_addr =3D 2208380372}, fport=3D13698, laddr=3D{s_addr =3D 27357= 72508}, lport=3D17822, lookupflags=3D2) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/netinet/in_pcb.c:19= 75 #5 0xffffffff818251c2 in pf_socket_lookup (direction=3D, pd=3D0xfffffe0c047efa98, m=3D0xfffff80015509200) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/netpfil/pf/pf.c:3052 #6 0xffffffff8182aa84 in pf_test_rule (rm=3D0xfffffe0c047efb40, sm=3D0xfffffe0c047efb48, direction=3D2, kif=3D0xfffff80012e4bb00, m=3D, off=3D20, pd=3D0xfffffe0c047efa98, am=3D0xfffffe0c047efb2= 8, rsm=3D0xfffffe0c047efb18, inp=3D0x0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/netpfil/pf/pf.c:348= 9 #7 0xffffffff818269e2 in pf_test (dir=3D2, ifp=3D, m0=3D, inp=3D0x0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/netpfil/pf/pf.c:6115 #8 0xffffffff81826759 in pf_test (dir=3D1, ifp=3D, m0=3D, inp=3D0x0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/netpfil/pf/pf.c:5594 #9 0xffffffff818370ed in pf_check_in (arg=3D, m=3D0xfffffe0c047efdc0, ifp=3D, dir=3D, flags=3D, inp=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/netpfil/pf/pf_ioctl.c:3916= #10 0xffffffff8077b647 in pfil_run_hooks (ph=3D0xffffffff81335108, mp=3D, ifp=3D0xfffff80012f2c000, dir=3D1, flags=3D0,= inp=3D0x0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/net/pfil.c:1= 09 #11 0xffffffff8079a8dd in ip_input (m=3D0xfffff80015509200) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/netinet/ip_input.c:604 =2E.. pf.c line 3052 is pf_socket_lookup(): 3049 switch (pd->af) { 3050 #ifdef INET 3051 case AF_INET: 3052 inp =3D in_pcblookup_mbuf(pi, saddr->v4, sport, daddr->v4, 3053 dport, INPLOOKUP_RLOCKPCB, NULL, m); (kgdb) f 3 #3 0xffffffff80663511 in __rw_rlock_hard (rw=3D0xffffffff81336188, td=3D0xfffff800118c3000, v=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/kern_rwlock.c:601 601 /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/kern_rwlock.c: No such file or directory. in /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/kern_rwlock.c (kgdb) print *rw $37 =3D {lock_object =3D {lo_name =3D 0xffffffff80b8490f "pcbinfohash", lo_flags =3D 86179840, lo_data =3D 0, lo_witness =3D 0x0}, rw_lock =3D 18446735277970142758} (kgdb) f 2 #2 0xffffffff806c3741 in turnstile_wait (ts=3D0xfffff8000b9ecb40, owner=3D, queue=3D1) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/subr_turnstile.c:751 751 /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/subr_turnstile.c: No such file or directory. in /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/subr_turnstile.c (kgdb) print *ts $38 =3D {ts_lock =3D {lock_object =3D {lo_name =3D 0xffffffff80b73a17 "tu= rnstile lock", lo_flags =3D 720896, lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock = =3D 0}, ts_blocked =3D 0xfffff8000b9ecb60, ts_pending =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff8000b9ecb80}, ts_hash =3D {le_next =3D 0x0, le_pr= ev =3D 0xffffffff8127cba8}, ts_link =3D {le_next =3D 0x0, le_prev =3D 0xfffff8001513b748}, ts_free =3D {lh_first =3D 0xfffff80015273000}, ts_lockobj =3D 0xffffffff81336188, ts_owner =3D 0xfffff8001513b620} (kgdb) print *ts->ts_owner =2E.. td_name =3D 0xfffff8001513b8b4 "unbound" =2E.. [9:32:27] hwlb-foe-aw-02 ~/ # grep '(td=3D0xfffff8001513b620' ps.core2 175 Thread 100215 (PID=3D1308: unbound) sched_switch (td=3D0xfffff8001513b620, newtd=3D0xfffff8000b0d9620, flags=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/sched_ule.c:2103 (kgdb) thr 175 [Switching to thread 175 (Thread 100215)]#0 sched_switch (td=3D0xfffff8001513b620, newtd=3D0xfffff8000b0d9620, flags=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/sched_ule.c:2103 2103 /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/sched_ule.c: No such file or directory. in /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/sched_ule.c (kgdb) bt #0 sched_switch (td=3D0xfffff8001513b620, newtd=3D0xfffff8000b0d9620, flags=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/sched_ule.c:2103 #1 0xffffffff806719d8 in mi_switch (flags=3D259, newtd=3D0x0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/kern_synch.c:439 #2 0xffffffff806c3741 in turnstile_wait (ts=3D0xfffff8001503f180, owner=3D, queue=3D0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/subr_turnstile.c:751 #3 0xffffffff80647d81 in __mtx_lock_sleep (c=3D0xffffffff8184f278, v=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/kern_mutex.c:637 #4 0xffffffff80661e53 in _rm_rlock (rm=3D0xffffffff8184f220, tracker=3D0xfffffe0c04aeea48, trylock=3D0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/kern_rmlock.c:408 #5 0xffffffff8182e935 in pf_test6 (dir=3D2, pflags=3D0, ifp=3D0xfffff80012f2c000, m0=3D0xfffffe0c04aeec40, inp=3D0xfffff800154911= d0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/netpfil/pf/pf.c:6395 #6 0xffffffff818371ed in pf_check6_out (arg=3D, m=3D0xfffffe0c04aeec40, ifp=3D, dir=3D, flags=3D, inp=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/netpfil/pf/pf_ioctl.c:3976= #7 0xffffffff8077b647 in pfil_run_hooks (ph=3D0xffffffff81338060, mp=3D, ifp=3D0xfffff80012f2c000, dir=3D2, flags=3D0,= inp=3D0xfffff800154911d0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/net/pfil.c:109 #8 0xffffffff8084fdbf in ip6_output (m0=3D, opt=3D0xfffff8010cf99400, ro=3D0xfffffe0c04aeee70, flags=3D0, im6o=3D0x0,= ifpp=3D0x0, inp=3D0xfffff800154911d0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/netinet6/ip6_output.c:835 #9 0xffffffff80868b86 in udp6_send (so=3D, flags=3D, m=3D, addr=3D, control=3D, td=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/netinet6/udp6_usrreq.c:897= #10 0xffffffff806f6ef4 in sosend_dgram (so=3D0xfffff80302b2b6d0, addr=3D, uio=3D, top=3D, control=3D0x0, flags=3D, td=3D0xfffff8001513b620) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/uipc_socket.c:1174 #11 0xffffffff806fdf85 in kern_sendit (td=3D, s=3D20= , mp=3D, flags=3D0, control=3D0x0, segflg=3DUIO_USERSP= ACE) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/uipc_syscalls.c:884 #12 0xffffffff806fe2de in sendit (td=3D0xfffff8001513b620, s=3D20, mp=3D0xfffffe0c04aef1c0, flags=3D0) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/uipc_syscalls.c:804 #13 0xffffffff806fe12d in sys_sendto (td=3D, uap=3D) at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/kern/uipc_syscalls.c:935 #14 0xffffffff8095bb46 in amd64_syscall (td=3D0xfffff8001513b620, traced=3D0) at freebsd.git/sys/amd64/amd64/../../kern/subr_syscall.c:132 #15 0xffffffff80939cdd in fast_syscall_common () at /usr/home/kajetan.staszkiewicz/freebsd.git/sys/amd64/amd64/exception.S:49= 4 #16 0x00000008017de0aa in ?? () Previous frame inner to this frame (corrupt stack?) pf.c 6395 is pf_test6(): 6392 if (m->m_flags & M_SKIP_FIREWALL) 6393 return (PF_PASS); 6394 6395 PF_RULES_RLOCK(); 6396 6397 /* We do IP header normalization and packet reassembly here = */ 6398 if (pf_normalize_ip6(m0, dir, kif, &reason, &pd) !=3D PF_PAS= S) { 6399 action =3D PF_DROP; So as far as I understand: - most NIC queues want to access "pf rulesets" lock - which is locked held by one more NIC queue which tries to lock "pcbinfohash" - which is held by unbound process - which tries to lock "pf rulesets" --=20 | pozdrawiam / greetings | Powered by macOS, Debian and FreeBSD | | Kajetan Staszkiewicz | www: http://vegeta.tuxpowered.net | `------------------------^--------------------------------------' --UkvR4H5J5VCKr7C31oruLwzrE00EhNufW-- --6Of8gmt13SoPjymQr9LyK3pjeHp5qDQwP Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQSOEQZObv2B8mf0JbnjtFCvbXs6FAUCXwWgJAAKCRDjtFCvbXs6 FIneAJ9ZUQR3DNic3s+Ztfh2Extfv+FUSQCeP6cCCPx/9HailKuFN8ywSb3Bzsw= =nVvL -----END PGP SIGNATURE----- --6Of8gmt13SoPjymQr9LyK3pjeHp5qDQwP--