From owner-freebsd-net@freebsd.org Sun Sep 8 07:21:06 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B78B1EAB71 for ; Sun, 8 Sep 2019 07:21:06 +0000 (UTC) (envelope-from slava.shwartsman@gmail.com) Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46R2lM5KlRz4b3m for ; Sun, 8 Sep 2019 07:21:03 +0000 (UTC) (envelope-from slava.shwartsman@gmail.com) Received: by mail-wm1-x32f.google.com with SMTP id r195so11214815wme.2 for ; Sun, 08 Sep 2019 00:21:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=R6m9668BOViRVjRr75M5oUWLCKOVZswM7Ek4lS/XqE0=; b=dzko4R5ADEaulekPlQcGNlomHO6homUD+nr4LTgg/fNVdBroicpnDDcAzSXvCDq0TR CDRMS6JqVZVQCfuOqPvK1iEB9yUHZlc8zi7W2f2SZqF/FRDxVcVNCY+bRWd3sR4g9C2q Tc5fza5shF/4tSYNVKmpJvEMk0u2vE2KOUpjZyyJBlM7TIa0fvLuWWbMIg61pE25wOX5 6mQg3UIr9YHKRyDczON3qzPo31v0KufYXTNqbj75DyvrqK6VbOukAeB/WGyHO4iyHQ2d qn8C7fbFtgl986CMAQwT8qxsAzUVzLX2E7tfWBPbh7lA8Pz3q/VI2IWHZVOuQmdEC0we Cz7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=R6m9668BOViRVjRr75M5oUWLCKOVZswM7Ek4lS/XqE0=; b=fB0iSlkP+MsoXiItajIggKnM+4jp1vLWK5IQfGYLuzBGDQRKy3CIPdsscXhvPEi8/Q r3y6e5AC99kVZcYUKE4h3tncH3yYiD6utZyTEniUM8XIIItJWD0/e1GuIfiuFCUvaQq5 9JddvEv11B3ntRoKlErRjCGxlK/emxulxHBFHhzmC8ot2bjpPmk8TniqwpRSXC6Jmev2 U5T/Z1sJakcqnhDZu0qu4CGLiFk15U+QVQu3uHfwNdp0K/jggSREy3b+lB47PuI9RvhT cVxSb2y2R+Y+iAhckgOwMeZRuwoIltzELK0ZcltIKQD/HcppLaRa/n5fqkoWoVjtdd30 xhqg== X-Gm-Message-State: APjAAAXmCOUclVNap1gETGgqL2tNYpA8hklwaX7mvT3Vfr9O9BdYI1tt h+soGbjzEovTvcGEm85OGk+3pvE8 X-Google-Smtp-Source: APXvYqxDXTq3QxJB9uXhpMOVgr91BRjBR3d4wgUMCCfgRPUVTlDcNDdA4rQsYWJ05gwc4sdv+GEf/g== X-Received: by 2002:a1c:4485:: with SMTP id r127mr13249213wma.59.1567927259255; Sun, 08 Sep 2019 00:20:59 -0700 (PDT) Received: from [10.223.3.125] ([193.47.165.251]) by smtp.gmail.com with ESMTPSA id u68sm16651509wmu.12.2019.09.08.00.20.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 08 Sep 2019 00:20:58 -0700 (PDT) Subject: Re: panic: rcv_start < rcv_end To: Piotr Kubaj , freebsd-net@freebsd.org References: <20190905194017.GA29827@KGPE-D16> From: Slava Shwartsman Message-ID: <344c571a-d7ca-6027-5901-808927923860@gmail.com> Date: Sun, 8 Sep 2019 10:20:56 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <20190905194017.GA29827@KGPE-D16> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 46R2lM5KlRz4b3m X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=dzko4R5A; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of slavashwartsman@gmail.com designates 2a00:1450:4864:20::32f as permitted sender) smtp.mailfrom=slavashwartsman@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-0.996,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (-9.02), ipnet: 2a00:1450::/32(-2.97), asn: 15169(-2.27), country: US(-0.05)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[f.2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Sep 2019 07:21:06 -0000 +1 On 05-Sep-19 22:40, Piotr Kubaj via freebsd-net wrote: > Hello, > > today I had a panic on CURRENT/powerpc64. I guess running make fetch for one of ports had something to do with it: > db> show panic > panic: rcv_start < rcv_end > db> bt > Tracing pid 0 tid 100176 td 0xc000000006fa4560 > 0xe000000099ad2b30: at vpanic+0x1fc > 0xe000000099ad2be0: at panic+0x44 > 0xe000000099ad2c10: at tcp_update_dsack_list+0x3e8 > 0xe000000099ad2ce0: at tcp_do_segment+0x2eb8 > 0xe000000099ad2e80: at tcp_input+0xec0 > 0xe000000099ad3060: at tcp6_input+0x1a8 > 0xe000000099ad30b0: at ip6_input+0x11d4 > 0xe000000099ad3210: at netisr_dispatch_src+0x108 > 0xe000000099ad32c0: at netisr_dispatch+0x24 > 0xe000000099ad32f0: at ether_demux+0x1b4 > 0xe000000099ad3370: at ether_gen_addr+0x660 > 0xe000000099ad3430: at netisr_dispatch_src+0x108 > 0xe000000099ad34e0: at netisr_dispatch+0x24 > 0xe000000099ad3510: at ether_ifattach+0x414 > 0xe000000099ad35b0: at if_input+0x28 > 0xe000000099ad35e0: at ata_sii_chipinit+0xe92c > 0xe000000099ad3690: at ata_sii_chipinit+0xb8bc > 0xe000000099ad3700: at taskqueue_run+0x200 > 0xe000000099ad37d0: at taskqueue_thread_loop+0xd4 > 0xe000000099ad3820: at fork_exit+0xc4 > 0xe000000099ad38c0: at fork_trampoline+0x18 > 0xe000000099ad38f0: at -0x4 > > pkubaj@talos:$/usr/home/pkubaj$ uname -a > FreeBSD talos 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r351820M: Wed Sep 4 23:37:46 CEST 2019 root@talos:/usr/obj/usr/src/powerpc.powerpc64/sys/ZFS powerpc > > My kernel: > pkubaj@talos:$/usr/home/pkubaj$ cat /usr/src/sys/powerpc/conf/ZFS > include GENERIC64 > options GEOM_ELI > options ZFS (kgdb) bt #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1  doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:392 #2  0xffffffff804a1f3b in db_dump (dummy=, dummy2=, dummy3=, dummy4=) at /usr/src/sys/ddb/db_command.c:575 #3  0xffffffff804a1d09 in db_command (last_cmdp=, cmd_table=, dopager=1) at /usr/src/sys/ddb/db_command.c:482 #4  0xffffffff804a1a84 in db_command_loop () at /usr/src/sys/ddb/db_command.c:535 #5  0xffffffff804a4c2f in db_trap (type=, code=) at /usr/src/sys/ddb/db_main.c:252 #6  0xffffffff80c1e67c in kdb_trap (type=3, code=0, tf=) at /usr/src/sys/kern/subr_kdb.c:692 #7  0xffffffff8119387f in trap (frame=0xfffffe0011dfd9c0) at /usr/src/sys/amd64/amd64/trap.c:621 #8  #9  kdb_enter (why=0xffffffff8133d85a "panic", msg=) at /usr/src/sys/kern/subr_kdb.c:479 #10 0xffffffff80bd2f4a in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:897 #11 0xffffffff80bd2cc3 in panic (fmt=0xffffffff81e8a3b8 "\211'0\201\377\377\377\377") at /usr/src/sys/kern/kern_shutdown.c:835 #12 0xffffffff80dfae89 in tcp_update_dsack_list (tp=0xfffff800bb670000, rcv_start=828121842, rcv_end=828121842) at /usr/src/sys/netinet/tcp_sack.c:166 #13 0xffffffff80deef63 in tcp_do_segment (m=0xfffff800bb4dab00, th=0xfffff800bb4dab7c, so=0xfffff8013035ea98, tp=0xfffff800bb670000, drop_hdrlen=52, tlen=,     iptos=0 '\000') at /usr/src/sys/netinet/tcp_input.c:3076 #14 0xffffffff80deb993 in tcp_input (mp=, offp=, proto=) at /usr/src/sys/netinet/tcp_input.c:1373 #15 0xffffffff80d6f39f in ip_input (m=0x0) at /usr/src/sys/netinet/ip_input.c:829 #16 0xffffffff80cffd02 in netisr_dispatch_src (proto=1, source=, m=0x80) at /usr/src/sys/net/netisr.c:1123 #17 0xffffffff80ce3c7b in ether_demux (ifp=0xfffff800036b2800, m=0x80) at /usr/src/sys/net/if_ethersubr.c:913 #18 0xffffffff80ce51fd in ether_input_internal (ifp=0xfffff800036b2800, m=0x80) at /usr/src/sys/net/if_ethersubr.c:705 #19 ether_nh_input (m=) at /usr/src/sys/net/if_ethersubr.c:735 #20 0xffffffff80cffd02 in netisr_dispatch_src (proto=5, source=, m=0x80) at /usr/src/sys/net/netisr.c:1123 #21 0xffffffff80ce410d in ether_input (ifp=0xfffff800036b2800, m=0x0) at /usr/src/sys/net/if_ethersubr.c:823 #22 0xffffffff80df5051 in tcp_flush_out_le (tp=0x0, lc=0xfffffe0056544fc0, le=0xfffffe005ea6c498, locked=0) at /usr/src/sys/netinet/tcp_lro.c:569 #23 0xffffffff80df4d93 in tcp_lro_flush (lc=0xfffffe0056544fc0, le=0xfffffe005ea6c498) at /usr/src/sys/netinet/tcp_lro.c:971 #24 0xffffffff80df53f2 in tcp_lro_rx_done (lc=0xfffffe0056544fc0) at /usr/src/sys/netinet/tcp_lro.c:356 #25 0xffffffff80df50fc in tcp_lro_flush_all (lc=0xfffffe0056544fc0) at /usr/src/sys/netinet/tcp_lro.c:1116 #26 0xffffffff82546056 in mlx5e_rx_cq_comp (mcq=) at /usr/src/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c:581 #27 0xffffffff8256f4a1 in mlx5_cq_completion (dev=0xfffffe0056318000, cqn=1035) at /usr/src/sys/dev/mlx5/mlx5_core/mlx5_cq.c:71 #28 0xffffffff8257080a in mlx5_eq_int (dev=0xfffffe0056318000, eq=0xfffff800036ae800) at /usr/src/sys/dev/mlx5/mlx5_core/mlx5_eq.c:248 #29 mlx5_msix_handler (irq=, eq_ptr=0xfffff800036ae800) at /usr/src/sys/dev/mlx5/mlx5_core/mlx5_eq.c:400 #30 0xffffffff80b94307 in intr_event_execute_handlers (p=, ie=) at /usr/src/sys/kern/kern_intr.c:1148 #31 ithread_execute_handlers (p=, ie=) at /usr/src/sys/kern/kern_intr.c:1161 #32 ithread_loop (arg=) at /usr/src/sys/kern/kern_intr.c:1241 #33 0xffffffff80b90e84 in fork_exit (callout=0xffffffff80b94180 , arg=0xfffff800036ebe20, frame=0xfffffe0011dfe340) at /usr/src/sys/kern/kern_fork.c:1046 #34 (kgdb) f 12 #12 0xffffffff80dfae89 in tcp_update_dsack_list (tp=0xfffff800bb670000, rcv_start=828121842, rcv_end=828121842) at /usr/src/sys/netinet/tcp_sack.c:166 166             KASSERT(SEQ_LT(rcv_start, rcv_end), ("rcv_start < rcv_end")); (kgdb) l 161             int i, j, n, identical; 162             tcp_seq start, end; 163 164             INP_WLOCK_ASSERT(tp->t_inpcb); 165 166             KASSERT(SEQ_LT(rcv_start, rcv_end), ("rcv_start < rcv_end")); 167 168             if (tp->t_inpcb->inp_socket->so_options & SO_DEBUG) { 169                     log(LOG_DEBUG, "\nDSACK update: %d..%d, rcv_nxt: %u\n", 170                     rcv_start, rcv_end, tp->rcv_nxt); Slava