Date: Thu, 22 Jun 2017 23:17:49 +0000 From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 220217] deadlock on enc and pf Message-ID: <bug-220217-8@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220217 Bug ID: 220217 Summary: deadlock on enc and pf Product: Base System Version: 11.0-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: vegeta@tuxpowered.net I have a bunch of routers running FreeBSD 10 and 11. They have gif tunnels encrypted with IPsec transport mode between them and this worked fine for years. On each of them there is pf protecting many VLANs and the aforementi= oned tunnels. Recently I had to connect them to some external cloud service provider and because of that I needed to use IPsec tunnel mode. In order to be able to r= un pf correctly and without redesigning my ruleset to work on interface facing= the Internet and covering internal IP addresses (I understand this is how (de)encapsulation of IPsec tunnel mode would work) I decided to use enc(4) device. Unfortunately as soon as the device is up the system locks up. Sometimes it will work for a minute or two with 100% cpu usage done in bird (which I use= to establish BGP sesion to that cloud) or in "intr/swi4: clock". Afterwards it just becomes unresponsive. I built a custom kernel with lock debugging options (https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-deadl= ocks.html) enabled and managed to get the following crash: #0 doadump (textdump=3D0) at pcpu.h:221=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20 #1 0xffffffff80332a2b in db_dump (dummy=3D<value optimized out>, dummy2=3D= false, dummy3=3D0, dummy4=3D0x0) at /mnt/builder/freebsd.git/sys/ddb/db_command.c:= 533=20=20=20=20=20=20 #2 0xffffffff80332829 in db_command (cmd_table=3D<value optimized out>) at /mnt/builder/freebsd.git/sys/ddb/db_command.c:440=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 #3 0xffffffff80332584 in db_command_loop () at /mnt/builder/freebsd.git/sys/ddb/db_command.c:493=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 #4 0xffffffff803354bb in db_trap (type=3D<value optimized out>, code=3D<va= lue optimized out>) at /mnt/builder/freebsd.git/sys/ddb/db_main.c:251=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20 #5 0xffffffff80646913 in kdb_trap (type=3D<value optimized out>, code=3D<v= alue optimized out>, tf=3D<value optimized out>) at /mnt/builder/freebsd.git/sys/kern/subr_kdb.c:654=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 #6 0xffffffff808d6a1d in trap (frame=3D0xfffffe01e47de850) at /mnt/builder/freebsd.git/sys/amd64/amd64/trap.c:556=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 #7 0xffffffff808b7631 in calltrap () at /mnt/builder/freebsd.git/sys/amd64/amd64/exception.S:236=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 #8 0xffffffff80645fbb in kdb_enter (why=3D0xffffffff80abf8f9 "panic", msg= =3D0x80 <Address 0x80 out of bounds>) at cpufunc.h:63=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 #9 0xffffffff80605ccf in vpanic (fmt=3D<value optimized out>, ap=3D0xfffffe01e47de9e0) at /mnt/builder/freebsd.git/sys/kern/kern_shutdown= .c:752=20 #10 0xffffffff80605b26 in kassert_panic (fmt=3D<value optimized out>) at /mnt/builder/freebsd.git/sys/kern/kern_shutdown.c:649=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 #11 0xffffffff80601e73 in __rw_rlock (c=3D0xfffff802449e8e30, file=3D0xffffffff80ade5f6 "/mnt/builder/freebsd.git/sys/netinet/in_pcb.c", line=3D1968) at /mnt/builder/freebsd.git/sys/kern/kern_rwlock.c:411=20=20= =20=20=20=20=20=20=20=20=20=20 #12 0xffffffff807252d1 in in_pcblookup_hash () at /mnt/builder/freebsd.git/sys/netinet/in_pcb.c:1968=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 #13 0xffffffff81825495 in pf_socket_lookup (direction=3D<value optimized ou= t>, pd=3D0xfffffe01e47df0a8, m=3D0xfffff80059759600) at /mnt/builder/freebsd.git/sys/modules/pf/../../netpfil/pf/pf.c:2932=20=20=20= =20=20=20=20=20=20=20=20=20=20=20 #14 0xffffffff8182ad04 in pf_test_rule (rm=3D0xfffffe01e47df128, sm=3D0xfffffe01e47df120, direction=3D<value optimized out>, kif=3D<value op= timized out>, m=3D0xfffff80059759600, off=3D<value optimized out>,=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20 pd=3D<value optimized out>, am=3D0xfffffe01e47df130, rsm=3D0xfffffe01e4= 7df118, inp=3D<value optimized out>) at /mnt/builder/freebsd.git/sys/modules/pf/../../netpfil/pf/pf.c:3369=20=20=20= =20=20=20=20=20=20=20=20=20=20=20 #15 0xffffffff8182705d in pf_test (dir=3D<value optimized out>, ifp=3D<value optimized out>, m0=3D0xfffffe01e47df208, inp=3D0x0) at /mnt/builder/freebsd.git/sys/modules/pf/../../netpfil/pf/pf.c:5973=20=20=20= =20=20=20=20=20=20=20=20=20=20=20 #16 0xffffffff81837f0d in pf_check_out (arg=3D<value optimized out>, m=3D0xfffffe01e47df208, ifp=3D0x80, dir=3D<value optimized out>, inp=3D0x0)= =20=20=20=20=20=20=20=20=20=20=20=20=20 at /mnt/builder/freebsd.git/sys/modules/pf/../../netpfil/pf/pf_ioctl.c:= 3742=20 #17 0xffffffff8070a46b in pfil_run_hooks (ph=3D<value optimized out>, mp=3D= <value optimized out>, ifp=3D<value optimized out>, dir=3D<value optimized out>, inp=3D<value optimized out>)=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 at /mnt/builder/freebsd.git/sys/net/pfil.c:83=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 #18 0xffffffff8185a4e5 in enc_hhook (hhook_type=3D4, hhook_id=3D2, udata=3D= <value optimized out>, ctx_data=3D0xfffffe01e47df3b0, hdata=3D<value optimized out= >, hosd=3D0x1d0)=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20 at /mnt/builder/freebsd.git/sys/net/if_enc.c:287=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 #19 0xffffffff805c8f33 in hhook_run_hooks (hhh=3D<value optimized out>, ctx_data=3D<value optimized out>, hosd=3D<value optimized out>) at /mnt/builder/freebsd.git/sys/kern/kern_hhook.c:121 #20 0xffffffff807ebe1f in ipsec_run_hhooks (ctx=3D0xfffffe01e47df3b0, type= =3D<value optimized out>) at /mnt/builder/freebsd.git/sys/netipsec/ipsec.c:838 #21 0xffffffff807eff05 in ipsec4_process_packet (m=3D<value optimized out>, isr=3D0xfffff80008d46080) at /mnt/builder/freebsd.git/sys/netipsec/ipsec_output.c:556 #22 0xffffffff8072b374 in ip_ipsec_output (m=3D0xfffffe01e47df588, inp=3D0xfffff802449e8cb0, error=3D0xfffffe01e47df54c) at /mnt/builder/freebsd.git/sys/netinet/ip_ipsec.c:205 #23 0xffffffff8072d076 in ip_output (m=3D<value optimized out>, opt=3D<value optimized out>, ro=3D<value optimized out>, flags=3D0, imo=3D0x0, inp=3D<va= lue optimized out>) at /mnt/builder/freebsd.git/sys/netinet/ip_output.c:556 #24 0xffffffff807a3ff0 in tcp_output (tp=3D<value optimized out>) at /mnt/builder/freebsd.git/sys/netinet/tcp_output.c:1422 #25 0xffffffff807af419 in tcp_timer_rexmt (xtp=3D<value optimized out>) at /mnt/builder/freebsd.git/sys/netinet/tcp_timer.c:812 #26 0xffffffff8061c1cc in softclock_call_cc (c=3D<value optimized out>, cc= =3D<value optimized out>, direct=3D<value optimized out>) at /mnt/builder/freebsd.git/sys/kern/kern_timeout.c:729 #27 0xffffffff8061c5c7 in softclock (arg=3D<value optimized out>) at /mnt/builder/freebsd.git/sys/kern/kern_timeout.c:867 #28 0xffffffff805cb3c6 in intr_event_execute_handlers (p=3D<value optimized= out>, ie=3D<value optimized out>) at /mnt/builder/freebsd.git/sys/kern/kern_intr.= c:1262 #29 0xffffffff805cba46 in ithread_loop (arg=3D<value optimized out>) at /mnt/builder/freebsd.git/sys/kern/kern_intr.c:1275 #30 0xffffffff805c8ac4 in fork_exit (callout=3D0xffffffff805cb9a0 <ithread_= loop>, arg=3D0xfffff800034ea360, frame=3D0xfffffe01e47df9c0) at /mnt/builder/freebsd.git/sys/kern/kern_fork.c:1038 #31 0xffffffff808b7b6e in fork_trampoline () at /mnt/builder/freebsd.git/sys/amd64/amd64/exception.S:611 #32 0x0000000000000000 in ?? () I do have in firewall a rule which allows all connections having a UID, so = the ones created by programs on the router. Apart from that most other rules are for forwarding. I'm afraid line numbers in pf.c might be altered, I build this kernel with debug options from the branch I use on my LoadBalancers which contains a few patches for pf, but outside of pf it should be standard kernel. The problem= of course exists also on standard FreeBSD 11.0 kernel. I see ticket PR 188063 deals with a very similar issue. --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-220217-8>