From owner-freebsd-testing@freebsd.org Sun Apr 12 17:45:53 2020 Return-Path: Delivered-To: freebsd-testing@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 425272C44F4 for ; Sun, 12 Apr 2020 17:45:53 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: from mail-yb1-f179.google.com (mail-yb1-f179.google.com [209.85.219.179]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 490fL818PCz4Qqs; Sun, 12 Apr 2020 17:45:51 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: by mail-yb1-f179.google.com with SMTP id f14so4088709ybr.13; Sun, 12 Apr 2020 10:45:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=U6Htn2SzGY6c5CNspqjoy9dTDQaa2jc2/10eY9toI+4=; b=uj+6atLCwRIfy/zGGdrKV/M8uWfgDyoMWLCAr3XkkdgYox3F2nPRpir7Oikmx7YBPc EAIj/K1iPQ1dq309PQ+PytmSdD+uMgoA1yDCbLCj1ZEcwkBITz88sAebmc0NWhoYOpyD VIywzj1mXHn6aveW4abKDz2i9iM/Mh45HHFvuIqQQyK5oUUhHkPfsvdOSy6BYzpiIcq8 K3p79vFU8CnPZAByiqqgYzW7Rk+IcU8hu8GISPF6cM5QQhndOILbE0gXNWK+G42pQ+7T CbbTEMgcCXgOk+kL4DjMOh/xoJQ3QDLPfXroH00NaIONWsRGNEv6O8hQlyIN0K0eDgLm bXFA== X-Gm-Message-State: AGi0PuZQ/T6deIs1i2O1VQNaQ9OSg0/cAEA4N4WPMnDWyHqyVls1XLx3 PVIg+s5rqgxdTnB9oX8+fW6t6R9OJuAQP695u/SAESri X-Google-Smtp-Source: APiQypKyyopiHpKeNKpgIC20/74Y2RoGs2DDuXTf8CMLBN5engN19TkFKrLyTTeCjCX8uAce5qmDPUlphOwhvcRRoT0= X-Received: by 2002:a25:cd43:: with SMTP id d64mr17501881ybf.451.1586713550736; Sun, 12 Apr 2020 10:45:50 -0700 (PDT) MIME-Version: 1.0 From: Li-Wen Hsu Date: Mon, 13 Apr 2020 01:45:39 +0800 Message-ID: Subject: ipdivert tests and net.inet.ip.fw.default_to_accept To: Alexander Chernikov Cc: freebsd-testing@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 490fL818PCz4Qqs X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of lwhsufreebsd@gmail.com designates 209.85.219.179 as permitted sender) smtp.mailfrom=lwhsufreebsd@gmail.com X-Spamd-Result: default: False [-3.88 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-1.88)[ip: (-8.55), ipnet: 209.85.128.0/17(-0.40), asn: 15169(-0.43), country: US(-0.05)]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[179.219.85.209.list.dnswl.org : 127.0.5.0]; FORGED_SENDER(0.30)[lwhsu@freebsd.org,lwhsufreebsd@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[179.219.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; TAGGED_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[lwhsu@freebsd.org,lwhsufreebsd@gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 17:45:53 -0000 (CC freebsd-testing@ for more people can join the discussion) Hi Alexander, I'm checking test failures and panics, and found that the two ipdivert tests we added previously have never been tested correctly. I've loaded ipdivert.ko to unskip them in test VM, and found that these two tests will fail or panic the kernel when net.inet.ip.fw.default_to_accept is not 1. I'm thinking: 1) Also checks for net.inet.ip.fw.default_to_accept=1 at the beginning and skip the test if 0 2) Is this lock order reversal normal? 3) Does this panic indicate a bug in the relative code? How do you think? Details of running the two tests are as following: # kyua test divert:ipdivert_ip_input_local_success divert:ipdivert_ip_input_local_success -> lock order reversal: 1st 0xffffffff81c8e558 allprison (allprison) @ /usr/src/sys/kern/kern_jail.c:953 2nd 0xffffffff81d9a750 vnet_sysinit_sxlock (vnet_sysinit_sxlock) @ /usr/src/sys/net/vnet.c:577 stack backtrace: #0 0xffffffff80c28671 at witness_debugger+0x71 #1 0xffffffff80bc7747 at _sx_slock_int+0x67 #2 0xffffffff80cfd2ce at vnet_alloc+0x11e #3 0xffffffff80b7fd93 at kern_jail_set+0x1a23 #4 0xffffffff80b81670 at sys_jail_set+0x40 #5 0xffffffff8105d09d at amd64_syscall+0x73d #6 0xffffffff810328d0 at fast_syscall_common+0x101 failed: 1 != 0 (1 != 0) [0.814s] # kyua test divert:ipdivert_ip_output_remote_success divert:ipdivert_ip_output_remote_success -> panic: Duplicate free of 0xfffff80003c3fa00 from zone 0xfffffe000a1d0000(mbuf) slab 0xfffff80003c3ffd8(10) cpuid = 3 time = 1586710625 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0038476f70 vpanic() at vpanic+0x182/frame 0xfffffe0038476fc0 panic() at panic+0x43/frame 0xfffffe0038477020 uma_dbg_free() at uma_dbg_free+0x1f2/frame 0xfffffe0038477060 uma_zfree_arg() at uma_zfree_arg+0x130/frame 0xfffffe00384770b0 m_free() at m_free+0xb9/frame 0xfffffe00384770e0 m_freem() at m_freem+0x28/frame 0xfffffe0038477100 div_send() at div_send+0x43c/frame 0xfffffe0038477170 sosend_generic() at sosend_generic+0x44c/frame 0xfffffe0038477220 sosend() at sosend+0x66/frame 0xfffffe0038477250 kern_sendit() at kern_sendit+0x246/frame 0xfffffe00384772f0 sendit() at sendit+0x1cc/frame 0xfffffe0038477340 sys_sendto() at sys_sendto+0x4d/frame 0xfffffe0038477390 amd64_syscall() at amd64_syscall+0x73d/frame 0xfffffe00384774b0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00384774b0 --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x8007beeba, rsp = 0x7fffffffdc68, rbp = 0x7fffffffdcb0 --- KDB: enter: panic Best, Li-Wen From owner-freebsd-testing@freebsd.org Sun Apr 12 18:09:35 2020 Return-Path: Delivered-To: freebsd-testing@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B25452C4B34 for ; Sun, 12 Apr 2020 18:09:35 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from forward500p.mail.yandex.net (forward500p.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 490fsT6cQNz4Rtr; Sun, 12 Apr 2020 18:09:33 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from mxback5q.mail.yandex.net (mxback5q.mail.yandex.net [IPv6:2a02:6b8:c0e:1ba:0:640:b716:ad89]) by forward500p.mail.yandex.net (Yandex) with ESMTP id 7547B940579; Sun, 12 Apr 2020 21:09:29 +0300 (MSK) Received: from localhost (localhost [::1]) by mxback5q.mail.yandex.net (mxback/Yandex) with ESMTP id GI6H6DKzNu-9SlSUBOK; Sun, 12 Apr 2020 21:09:28 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfw.ru; s=mail; t=1586714968; bh=/Gds0gssrjC/rda92OqXdHDw7NpaVtOFBNXXEF2ubRA=; h=Message-Id:Cc:Subject:In-Reply-To:Date:References:To:From; b=dwkvV1iYvDOLPZo3NpALK6Ebgw+T825oh5QfYY6OiE5Ol00L7uUgwtq5FY8vyggTw zZ8qokRtT+0y4TYjp+1ugyroPU7QWZJVTjFa6zrGADk1wIXPbuZ+5kIp1evvGuGcYP JTioWBHCAbN1/qVC99SdbxhdpssKa9aUxiD1jWqg= Received: by vla1-c477e3898c96.qloud-c.yandex.net with HTTP; Sun, 12 Apr 2020 21:09:28 +0300 From: Alexander V. Chernikov Envelope-From: melifaro@ipfw.ru To: Li-Wen Hsu Cc: "freebsd-testing@freebsd.org" In-Reply-To: References: Subject: Re: ipdivert tests and net.inet.ip.fw.default_to_accept MIME-Version: 1.0 X-Mailer: Yamail [ http://yandex.ru ] 5.0 Date: Sun, 12 Apr 2020 19:09:28 +0100 Message-Id: <6963991586714482@vla4-4046ec513d04.qloud-c.yandex.net> Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=utf-8 X-Rspamd-Queue-Id: 490fsT6cQNz4Rtr X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ipfw.ru header.s=mail header.b=dwkvV1iY; dmarc=none; spf=pass (mx1.freebsd.org: domain of melifaro@ipfw.ru designates 2a02:6b8:0:1472:2741:0:8b7:110 as permitted sender) smtp.mailfrom=melifaro@ipfw.ru X-Spamd-Result: default: False [-5.77 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[ipfw.ru:s=mail]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a02:6b8:0:1000::/52]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; IP_SCORE(-3.57)[ip: (-9.23), ipnet: 2a02:6b8::/32(-4.77), asn: 13238(-3.85), country: RU(0.01)]; DKIM_TRACE(0.00)[ipfw.ru:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[0.1.1.0.7.b.8.0.0.0.0.0.1.4.7.2.2.7.4.1.0.0.0.0.8.b.6.0.2.0.a.2.list.dnswl.org : 127.0.5.0]; FORGED_SENDER(0.30)[melifaro@freebsd.org,melifaro@ipfw.ru]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:13238, ipnet:2a02:6b8::/32, country:RU]; FROM_NEQ_ENVFROM(0.00)[melifaro@freebsd.org,melifaro@ipfw.ru] X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 18:09:35 -0000 12.04.2020, 18:45, "Li-Wen Hsu" : > (CC freebsd-testing@ for more people can join the discussion) > > Hi Alexander, Ehlo! > > I'm checking test failures and panics, and found that the two ipdivert > tests we added previously have never been tested correctly. I've Yep, I've been running tests manually but haven't managed to raise a pull request to auto-load ipdivert prior to the tests yet. > loaded ipdivert.ko to unskip them in test VM, and found that these > two tests will fail or panic the kernel when > net.inet.ip.fw.default_to_accept is not 1. Wow. That's an interesting one! I've been always testing them with default_to_accept set to 1, as it eases test configuration. > > I'm thinking: > > 1) Also checks for net.inet.ip.fw.default_to_accept=1 at the beginning > and skip the test if 0 Well, ipfw defaults are not that well isolated in vnet. Maybe a better way would be to just install allow-all rule in VNET jail and proceed normally. > 2) Is this lock order reversal normal? No. > 3) Does this panic indicate a bug in the relative code? Yes. Let me investigate it. > > How do you think? > > Details of running the two tests are as following: > > # kyua test divert:ipdivert_ip_input_local_success > divert:ipdivert_ip_input_local_success -> lock order reversal: >  1st 0xffffffff81c8e558 allprison (allprison) @ > /usr/src/sys/kern/kern_jail.c:953 >  2nd 0xffffffff81d9a750 vnet_sysinit_sxlock (vnet_sysinit_sxlock) @ > /usr/src/sys/net/vnet.c:577 > stack backtrace: > #0 0xffffffff80c28671 at witness_debugger+0x71 > #1 0xffffffff80bc7747 at _sx_slock_int+0x67 > #2 0xffffffff80cfd2ce at vnet_alloc+0x11e > #3 0xffffffff80b7fd93 at kern_jail_set+0x1a23 > #4 0xffffffff80b81670 at sys_jail_set+0x40 > #5 0xffffffff8105d09d at amd64_syscall+0x73d > #6 0xffffffff810328d0 at fast_syscall_common+0x101 > failed: 1 != 0 (1 != 0) [0.814s] > > # kyua test divert:ipdivert_ip_output_remote_success > divert:ipdivert_ip_output_remote_success -> panic: Duplicate free of > 0xfffff80003c3fa00 from zone 0xfffffe000a1d0000(mbuf) slab > 0xfffff80003c3ffd8(10) > cpuid = 3 > time = 1586710625 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0038476f70 > vpanic() at vpanic+0x182/frame 0xfffffe0038476fc0 > panic() at panic+0x43/frame 0xfffffe0038477020 > uma_dbg_free() at uma_dbg_free+0x1f2/frame 0xfffffe0038477060 > uma_zfree_arg() at uma_zfree_arg+0x130/frame 0xfffffe00384770b0 > m_free() at m_free+0xb9/frame 0xfffffe00384770e0 > m_freem() at m_freem+0x28/frame 0xfffffe0038477100 > div_send() at div_send+0x43c/frame 0xfffffe0038477170 > sosend_generic() at sosend_generic+0x44c/frame 0xfffffe0038477220 > sosend() at sosend+0x66/frame 0xfffffe0038477250 > kern_sendit() at kern_sendit+0x246/frame 0xfffffe00384772f0 > sendit() at sendit+0x1cc/frame 0xfffffe0038477340 > sys_sendto() at sys_sendto+0x4d/frame 0xfffffe0038477390 > amd64_syscall() at amd64_syscall+0x73d/frame 0xfffffe00384774b0 > fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00384774b0 > --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x8007beeba, rsp = > 0x7fffffffdc68, rbp = 0x7fffffffdcb0 --- > KDB: enter: panic > > Best, > Li-Wen From owner-freebsd-testing@freebsd.org Sun Apr 12 18:46:30 2020 Return-Path: Delivered-To: freebsd-testing@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3A0E72C58DB for ; Sun, 12 Apr 2020 18:46:30 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: from mail-yb1-f174.google.com (mail-yb1-f174.google.com [209.85.219.174]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 490gh60lGrz4TvK; Sun, 12 Apr 2020 18:46:29 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: by mail-yb1-f174.google.com with SMTP id t10so4161494ybk.8; Sun, 12 Apr 2020 11:46:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=f5Jsp/mvEhVFIqDI2A1V+g3bkZpfBqwDytxluEVG74M=; b=dSGvpJG7uL/d8J33JCWjpIunRmOmScgZ+y1O2mpo9kJ3QTbkprbyxr0S2OuygUJGRO qJZxeYjhp7sFNXt0EBaPUf0DoEhAemG9IGFCIOBxyjiHngNH3/6LxCXV9GY8ii5KlIoC bGMJNyY8l6x+74nejwmuEu3pBLNsZQv65v5y7hrR8MnTCtpVquOrvnxtpooFj7zbD5MZ X1qkzsJBJFQ0a5Kk0S4tR9QpJ4PP+yj6HWUedincbsXdcIpBFT0MdgWlclyNAbgXWrc6 1ozdhE/1TTI33Iztmbj2LJEAoROV8+ODjlcfAKGFUFJmpG6bhWR0wqJikabr1HYkDGpn zm6w== X-Gm-Message-State: AGi0PuZhPeElvdsLXcq/6K3+HJ+0vIzyJXJYBdBDrp6ZgCtGY9FQkjAh RvSbwnbi9YqvFwCy1aABqbovPJxuep+7q3qrZN/NwZNK X-Google-Smtp-Source: APiQypJMgDo2lyUL0GB1DbDu0+d2nBPDvX2oQhjvWj0gBR5nSfV8eoy84IdMluKM7Gb8FrKt0zdgNXksUCs/CLnfjkA= X-Received: by 2002:a25:80c4:: with SMTP id c4mr22974962ybm.110.1586717188720; Sun, 12 Apr 2020 11:46:28 -0700 (PDT) MIME-Version: 1.0 References: <6963991586714482@vla4-4046ec513d04.qloud-c.yandex.net> In-Reply-To: <6963991586714482@vla4-4046ec513d04.qloud-c.yandex.net> From: Li-Wen Hsu Date: Mon, 13 Apr 2020 02:46:17 +0800 Message-ID: Subject: Re: ipdivert tests and net.inet.ip.fw.default_to_accept To: "Alexander V. Chernikov" Cc: "freebsd-testing@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 490gh60lGrz4TvK X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[] X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Apr 2020 18:46:30 -0000 On Mon, Apr 13, 2020 at 2:09 AM Alexander V. Chernikov wrote: > 12.04.2020, 18:45, "Li-Wen Hsu" : > > I'm checking test failures and panics, and found that the two ipdivert > > tests we added previously have never been tested correctly. I've > Yep, I've been running tests manually but haven't managed to raise a pull request to auto-load ipdivert prior to the tests yet. It think it's okay for now. I've merged this PR https://github.com/freebsd/freebsd-ci/pull/34 and now we have loaded all the modules needed. I somehow feel it's crazy and default_to_accept might have side effect like what we already saw. Anyway. I'm leaving them there for now and move to checking other failures and panics first. Also thanks for checking relative codes. That's the reason we need tests! From owner-freebsd-testing@freebsd.org Tue Apr 14 22:35:19 2020 Return-Path: Delivered-To: freebsd-testing@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DD5492CB305; Tue, 14 Apr 2020 22:35:19 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4920gC5Vj1z3NMG; Tue, 14 Apr 2020 22:35:19 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 1129) id 96ADF16895; Tue, 14 Apr 2020 22:35:19 +0000 (UTC) Date: Tue, 14 Apr 2020 22:35:19 +0000 From: Li-Wen Hsu To: freebsd-testing@freebsd.org Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: FreeBSD CI Weekly Report 2020-04-05 Message-ID: <20200414223519.GA33328@freefall.freebsd.org> Reply-To: freebsd-testing@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Apr 2020 22:35:19 -0000 (Please send the followup to freebsd-testing@ and note Reply-To is set.) FreeBSD CI Weekly Report 2020-04-05 =================================== Here is a summary of the FreeBSD Continuous Integration results for the period from 2020-03-30 to 2020-04-05. During this period, we have: * 1715 builds (93.6% (+0.1) passed, 6.4% (-0.1) failed) of buildworld and buildkernel (GENERIC and LINT) were executed on aarch64, amd64, armv6, armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, riscv64, sparc64 architectures for head, stable/12, stable/11 branches. * 274 test runs (49.7% (-4.6) passed, 19.3% (-24.4) unstable, 31.0% (+29.0) exception) were executed on amd64, i386, riscv64 architectures for head, stable/12, stable/11 branches. * 26 doc and www builds (84.6 (-10.6) passed, 15.4% (+10.6) failed) Test case status (on 2020-04-05 23:59): | Branch/Architecture | Total | Pass | Fail | Skipped | | ------------------- | --------- | --------- | ------- | -------- | | head/amd64 | 7740 (+4) | 7619 (-6) | 9 (+4) | 112 (+6) | | head/i386 | 7738 (+4) | 7613 (-4) | 11 (+6) | 114 (+2) | | 12-STABLE/amd64 | 7508 (+2) | 7452 (+2) | 0 (0) | 56 (0) | | 12-STABLE/i386 | 7506 (+2) | 7442 (-1) | 0 (0) | 64 (+3) | | 11-STABLE/amd64 | 6882 (+2) | 6835 (+5) | 0 (0) | 47 (-3) | | 11-STABLE/i386 | 6880 (+2) | 6831 (+5) | 0 (0) | 49 (-3) | (The statistics from experimental jobs are omitted) If any of the issues found by CI are in your area of interest or expertise please investigate the PRs listed below. The latest web version of this report is available at https://hackmd.io/@FreeBSD-CI/report-20200405 and archive is available at https://hackmd.io/@FreeBSD-CI/ , any help is welcome. ## News * New armv6 job is added: * FreeBSD-head-armv6-testvm The images are available at https://artifact.ci.freebsd.org ## Failing jobs * https://ci.freebsd.org/job/FreeBSD-head-amd64-gcc6_build/ * See console log for the error details. ## Failing tests * https://ci.freebsd.org/job/FreeBSD-head-amd64-test/ Some test cases not correctly skipped because test env loads the wrong kyua.conf after kyua imported to the base. * lib.libproc.proc_test.symbol_lookup * lib.msun.ctrig_test.test_inf_inputs * local.kyua.integration.cmd_about_test.topic__authors__installed * sys.kern.ptrace_test.ptrace__parent_wait_after_attach * sys.netipsec.tunnel.empty.v4 * sys.netipsec.tunnel.empty.v6 * sys.opencrypto.blake2_test.blake2b_vectors_x86 * sys.opencrypto.blake2_test.blake2s_vectors_x86 * usr.bin.calendar.legacy_test.main * https://ci.freebsd.org/job/FreeBSD-head-i386-test/ All amd64 failures and: * sys.kern.ptrace_test.ptrace__parent_exits_before_child * sys.net.if_lagg_test.witness ## Regressions * 3 tests start failing after llvm10 import * lib.libproc.proc_test.symbol_lookup * lib.msun.ctrig_test.test_inf_inputs https://bugs.freebsd.org/244732 * (DTrace) common.pid.t_dtrace_contrib.err_D_PROC_OFF_toobig_d https://bugs.freebsd.org/244823 * fusefs tests fail when mac_bsdextended.ko is loaded https://bugs.freebsd.org/244229 * `dtrace -c` causes program dumps core after somewhere between (r357694, r357701] https://bugs.freebsd.org/244053 * Lock-order reversals triggered by tests under sys.net.if_lagg_test.* on i386 https://bugs.freebsd.org/244163 Discovered by newly endabled sys.net.* tests. ([r357857](https://svnweb.freebsd.org/changeset/base/357857)) * sys.net.if_lagg_test.lacp_linkstate_destroy_stress panics i386 kernel https://bugs.freebsd.org/244168 Discovered by newly endabled sys.net.* tests. ([r357857](https://svnweb.freebsd.org/changeset/base/357857)) * (test case) sys.geom.class.multipath.misc.fail_on_error (on 12-STABLE) https://bugs.freebsd.org/244158 ## Failing and Flaky tests (from experimental jobs) * https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/ * cddl.usr.sbin.dtrace.common.misc.t_dtrace_contrib.tst_dynopt_d * https://bugs.freebsd.org/237641 * cddl.usr.sbin.dtrace.common.pid.t_dtrace_contrib.err_D_PROC_OFF_toobig_d * https://bugs.freebsd.org/244823 * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/ * There are ~13 failing and ~109 skipped cases, including flakey ones, see https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/lastCompletedBuild/testReport/ for more details * Work for cleaning these failing cass are in progress * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_ltp/ * Total 3670 tests (0), 2285 success (-5), 579 failures (+5), 806 skipped (0) ## Disabled Tests * sys.fs.tmpfs.mount_test.large https://bugs.freebsd.org/212862 * sys.fs.tmpfs.link_test.kqueue https://bugs.freebsd.org/213662 * sys.kqueue.libkqueue.kqueue_test.main https://bugs.freebsd.org/233586 * sys.kern.ptrace_test.ptrace__PT_KILL_competing_stop https://bugs.freebsd.org/220841 * lib.libc.regex.exhaust_test.regcomp_too_big (i386 only) https://bugs.freebsd.org/237450 * sys.netinet.socket_afinet.socket_afinet_bind_zero https://bugs.freebsd.org/238781 * sys.netpfil.pf.names.names * sys.netpfil.pf.synproxy.synproxy https://bugs.freebsd.org/238870 * sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger https://bugs.freebsd.org/239292 * sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger https://bugs.freebsd.org/239397 * sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger https://bugs.freebsd.org/239399 * sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger https://bugs.freebsd.org/239425 * lib.libc.gen.getmntinfo_test.getmntinfo_test https://bugs.freebsd.org/240049 * sys.sys.qmath_test.qdivq_s64q https://bugs.freebsd.org/240219 * sys.kern.ptrace_test.ptrace__getppid https://bugs.freebsd.org/240510 * lib.libc.sys.stat_test.stat_socket https://bugs.freebsd.org/240621 * lib.libarchive.functional_test.test_write_filter_zstd https://bugs.freebsd.org/240683 * lib.libcasper.services.cap_dns.dns_test.main https://bugs.freebsd.org/241435 * local.kyua.* (31 cases) & local.lutok.* (3 cases) on 11-i386 https://ci.freebsd.org/job/FreeBSD-stable-11-i386-test/2278/testReport/ * sys.geom.class.multipath.failloop.failloop https://bugs.freebsd.org/242689 * sys.kern.ptrace_test.ptrace__procdesc_reparent_wait_child https://bugs.freebsd.org/243605 * skip sys.geom.class.multipath.failloop.failloop https://bugs.freebsd.org/244053 * sys.kern.ptrace_test.ptrace__parent_wait_after_attach https://bugs.freebsd.org/244055 * sys.kern.ptrace_test.ptrace__parent_exits_before_child https://bugs.freebsd.org/244056 * sys.geom.class.multipath.misc.fail_on_error (12-STABLE) https://bugs.freebsd.org/244158 * sys.net.if_lagg_test.witness (i386) https://bugs.freebsd.org/244163 * PipePdfork.WildcardWait in sys.capsicum.capsicum-test.main https://bugs.freebsd.org/244165 * sys.net.if_lagg_test.lacp_linkstate_destroy_stress (i386) https://bugs.freebsd.org/244168 * sys.netinet6.frag6.frag6_07.frag6_07 https://bugs.freebsd.org/244170 * sys.netinet.fibs_test.udp_dontroute6 https://bugs.freebsd.org/244172 * sys.netpfil.pf.nat.exhaust https://bugs.freebsd.org/244703 * sys.geom.class.gate.ggate_test.ggated https://bugs.freebsd.org/244737 ## Issues ### Cause build fails * https://bugs.freebsd.org/233735 Possible build race: genoffset.o /usr/src/sys/sys/types.h: error: machine/endian.h: No such file or directory * https://bugs.freebsd.org/233769 Possible build race: ld: error: unable to find library -lgcc_s ### Cause kernel panics * https://bugs.freebsd.org/238870 sys.netpfil.pf.names.names and sys.netpfil.pf.synproxy.synproxy cause panic Patch exists: * https://reviews.freebsd.org/D20868 * https://reviews.freebsd.org/D20869 ### Open * https://bugs.freebsd.org/237403 Tests in sys/opencrypto should be converted to Python3 * https://bugs.freebsd.org/237641 Flakey test case: common.misc.t_dtrace_contrib.tst_dynopt_d * https://bugs.freebsd.org/237656 "Freed UMA keg (rtentry) was not empty (18 items). Lost 1 pages of memory." seen when running sys/netipsec tests * https://bugs.freebsd.org/238781 sys.netinet.socket_afinet.socket_afinet_bind_zero does not work when mac_portacl(4) loaded * https://bugs.freebsd.org/239292 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger * https://bugs.freebsd.org/239397 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger * https://bugs.freebsd.org/239399 Flakey test case: sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger * https://bugs.freebsd.org/239425 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger * https://bugs.freebsd.org/241662 Flakey test case: lib.libarchive.functional_test.test_fuzz_iso9660 ### Others * [Tickets related to testing@](https://preview.tinyurl.com/y9maauwg) From owner-freebsd-testing@freebsd.org Tue Apr 14 22:37:10 2020 Return-Path: Delivered-To: freebsd-testing@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D1D772CB631; Tue, 14 Apr 2020 22:37:10 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4920jL5CmXz3Nhv; Tue, 14 Apr 2020 22:37:10 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 1129) id A440116A26; Tue, 14 Apr 2020 22:37:10 +0000 (UTC) Date: Tue, 14 Apr 2020 22:37:10 +0000 From: Li-Wen Hsu To: freebsd-testing@freebsd.org Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: FreeBSD CI Weekly Report 2020-04-12 Message-ID: <20200414223710.GB33328@freefall.freebsd.org> Reply-To: freebsd-testing@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Apr 2020 22:37:10 -0000 (Please send the followup to freebsd-testing@ and note Reply-To is set.) FreeBSD CI Weekly Report 2020-04-12 =================================== Here is a summary of the FreeBSD Continuous Integration results for the period from 2020-04-06 to 2020-04-12. During this period, we have: * 1801 builds (94.0% (+0.4) passed, 6.0% (-0.4) failed) of buildworld and buildkernel (GENERIC and LINT) were executed on aarch64, amd64, armv6, armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, riscv64, sparc64 architectures for head, stable/12, stable/11 branches. * 288 test runs (25.1% (-24.6) passed, 29.9% (+10.6) unstable, 45.1% (+14.1) exception) were executed on amd64, i386, riscv64 architectures for head, stable/12, stable/11 branches. * 30 doc and www builds (83.3% (-1.3) passed, 16.7% (+1.3) failed) Test case status (on 2020-04-12 23:59): | Branch/Architecture | Total | Pass | Fail | Skipped | | ------------------- | --------- | ---------- | -------- | -------- | | head/amd64 | 7744 (+4) | 7638 (+19) | 14 (+5) | 92 (-20) | | head/i386 | 7742 (+4) | 7628 (+15) | 16 (+5) | 98 (-16) | | 12-STABLE/amd64 | 7508 (0) | 7449 (-3) | 1 (+1) | 58 (+2) | | 12-STABLE/i386 | 7506 (0) | 7425 (-17) | 2 (+2) | 79 (+15) | | 11-STABLE/amd64 | 6882 (0) | 6829 (-6) | 1 (+1) | 52 (+5) | | 11-STABLE/i386 | 6880 (0) | 6749 (-82) | 80 (+80) | 51 (+2) | (The statistics from experimental jobs are omitted) If any of the issues found by CI are in your area of interest or expertise please investigate the PRs listed below. The latest web version of this report is available at https://hackmd.io/@FreeBSD-CI/report-20200412 and archive is available at https://hackmd.io/@FreeBSD-CI/ , any help is welcome. ## News * The test env now loads the required module for firewall tests. * New armv7 job is added (to replace armv6 one): * FreeBSD-head-armv7-testvm The images are available at https://artifact.ci.freebsd.org FreeBSD-head-armv7-test is ready but needs test env update. ## Failing jobs * https://ci.freebsd.org/job/FreeBSD-head-amd64-gcc6_build/ * See console log for the error details. ## Failing tests * https://ci.freebsd.org/job/FreeBSD-head-amd64-test/ * local.kyua.integration.cmd_about_test.topic__authors__installed * sys.netipsec.tunnel.empty.v4 * sys.netipsec.tunnel.empty.v6 * sys.netpfil.common.forward.ipf_v4 * sys.netpfil.common.forward.ipfw_v4 * sys.netpfil.common.forward.pf_v4 * sys.netpfil.common.tos.ipfw_tos * sys.netpfil.common.tos.pf_tos * sys.netpfil.pf.forward.v4 * sys.netpfil.pf.set_tos.v4 * sys.opencrypto.blake2_test.blake2b_vectors_x86 * sys.opencrypto.blake2_test.blake2s_vectors_x86 * sbin.nvmecontrol.basic.__test_cases_list__ Fixed in r359903 * usr.bin.calendar.legacy_test.main Fixed in r359862 * https://ci.freebsd.org/job/FreeBSD-head-i386-test/ All amd64 failures and: * sys.netinet.divert.ipdivert_ip_input_local_success * sys.netinet.divert.ipdivert_ip_output_remote_success * https://ci.freebsd.org/job/FreeBSD-stable-12-amd64-test/ * usr.bin.calendar.legacy_test.main Fixed in r359863 * sbin.nvmecontrol.basic.__test_cases_list__ Fixed in r359904 * https://ci.freebsd.org/job/FreeBSD-stable-12-i386-test/ * usr.bin.calendar.legacy_test.main Fixed in r359863 * https://ci.freebsd.org/job/FreeBSD-stable-11-amd64-test/ * usr.bin.calendar.legacy_test.main Fixed in r359864 * https://ci.freebsd.org/job/FreeBSD-stable-11-i386-test/ Many failure all due to kyua.conf issue. Already fixed on 2020-04-13. ## Regressions * 3 tests start failing after llvm10 import * lib.libproc.proc_test.symbol_lookup * lib.msun.ctrig_test.test_inf_inputs https://bugs.freebsd.org/244732 * (DTrace) common.pid.t_dtrace_contrib.err_D_PROC_OFF_toobig_d https://bugs.freebsd.org/244823 * fusefs tests fail when mac_bsdextended.ko is loaded https://bugs.freebsd.org/244229 * `dtrace -c` causes program dumps core after somewhere between (r357694, r357701] https://bugs.freebsd.org/244053 * Lock-order reversals triggered by tests under sys.net.if_lagg_test.* on i386 https://bugs.freebsd.org/244163 Discovered by newly endabled sys.net.* tests. ([r357857](https://svnweb.freebsd.org/changeset/base/357857)) * sys.net.if_lagg_test.lacp_linkstate_destroy_stress panics i386 kernel https://bugs.freebsd.org/244168 Discovered by newly endabled sys.net.* tests. ([r357857](https://svnweb.freebsd.org/changeset/base/357857)) * (test case) sys.geom.class.multipath.misc.fail_on_error (on 12-STABLE) https://bugs.freebsd.org/244158 ## Failing and Flaky tests (from experimental jobs) * https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/ * cddl.usr.sbin.dtrace.common.misc.t_dtrace_contrib.tst_dynopt_d * https://bugs.freebsd.org/237641 * cddl.usr.sbin.dtrace.common.pid.t_dtrace_contrib.err_D_PROC_OFF_toobig_d * https://bugs.freebsd.org/244823 * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/ * There are ~13 failing and ~109 skipped cases, including flakey ones, see https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/lastCompletedBuild/testReport/ for more details * Work for cleaning these failing cass are in progress * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_ltp/ * Total 3670 tests (0), 2285 success (-5), 579 failures (+5), 806 skipped (0) ## Disabled Tests * sys.fs.tmpfs.mount_test.large https://bugs.freebsd.org/212862 * sys.fs.tmpfs.link_test.kqueue https://bugs.freebsd.org/213662 * sys.kqueue.libkqueue.kqueue_test.main https://bugs.freebsd.org/233586 * sys.kern.ptrace_test.ptrace__PT_KILL_competing_stop https://bugs.freebsd.org/220841 * lib.libc.regex.exhaust_test.regcomp_too_big (i386 only) https://bugs.freebsd.org/237450 * sys.netinet.socket_afinet.socket_afinet_bind_zero https://bugs.freebsd.org/238781 * sys.netpfil.pf.names.names * sys.netpfil.pf.synproxy.synproxy https://bugs.freebsd.org/238870 * sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger https://bugs.freebsd.org/239292 * sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger https://bugs.freebsd.org/239397 * sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger https://bugs.freebsd.org/239399 * sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger https://bugs.freebsd.org/239425 * lib.libc.gen.getmntinfo_test.getmntinfo_test https://bugs.freebsd.org/240049 * sys.sys.qmath_test.qdivq_s64q https://bugs.freebsd.org/240219 * sys.kern.ptrace_test.ptrace__getppid https://bugs.freebsd.org/240510 * lib.libc.sys.stat_test.stat_socket https://bugs.freebsd.org/240621 * lib.libarchive.functional_test.test_write_filter_zstd https://bugs.freebsd.org/240683 * lib.libcasper.services.cap_dns.dns_test.main https://bugs.freebsd.org/241435 * local.kyua.* (31 cases) & local.lutok.* (3 cases) on 11-i386 https://ci.freebsd.org/job/FreeBSD-stable-11-i386-test/2278/testReport/ * sys.geom.class.multipath.failloop.failloop https://bugs.freebsd.org/242689 * sys.kern.ptrace_test.ptrace__procdesc_reparent_wait_child https://bugs.freebsd.org/243605 * skip sys.geom.class.multipath.failloop.failloop https://bugs.freebsd.org/244053 * sys.kern.ptrace_test.ptrace__parent_wait_after_attach https://bugs.freebsd.org/244055 * sys.kern.ptrace_test.ptrace__parent_exits_before_child https://bugs.freebsd.org/244056 * sys.geom.class.multipath.misc.fail_on_error (12-STABLE) https://bugs.freebsd.org/244158 * sys.net.if_lagg_test.witness (i386) https://bugs.freebsd.org/244163 * PipePdfork.WildcardWait in sys.capsicum.capsicum-test.main https://bugs.freebsd.org/244165 * sys.net.if_lagg_test.lacp_linkstate_destroy_stress (i386) https://bugs.freebsd.org/244168 * sys.netinet6.frag6.frag6_07.frag6_07 https://bugs.freebsd.org/244170 * sys.netinet.fibs_test.udp_dontroute6 https://bugs.freebsd.org/244172 * sys.netpfil.pf.nat.exhaust https://bugs.freebsd.org/244703 * sys.geom.class.gate.ggate_test.ggated https://bugs.freebsd.org/244737 ## Issues ### Cause build fails * https://bugs.freebsd.org/233735 Possible build race: genoffset.o /usr/src/sys/sys/types.h: error: machine/endian.h: No such file or directory * https://bugs.freebsd.org/233769 Possible build race: ld: error: unable to find library -lgcc_s ### Cause kernel panics * https://bugs.freebsd.org/238870 sys.netpfil.pf.names.names and sys.netpfil.pf.synproxy.synproxy cause panic Patch exists: * https://reviews.freebsd.org/D20868 * https://reviews.freebsd.org/D20869 ### Open * https://bugs.freebsd.org/237403 Tests in sys/opencrypto should be converted to Python3 * https://bugs.freebsd.org/237641 Flakey test case: common.misc.t_dtrace_contrib.tst_dynopt_d * https://bugs.freebsd.org/237656 "Freed UMA keg (rtentry) was not empty (18 items). Lost 1 pages of memory." seen when running sys/netipsec tests * https://bugs.freebsd.org/238781 sys.netinet.socket_afinet.socket_afinet_bind_zero does not work when mac_portacl(4) loaded * https://bugs.freebsd.org/239292 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger * https://bugs.freebsd.org/239397 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger * https://bugs.freebsd.org/239399 Flakey test case: sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger * https://bugs.freebsd.org/239425 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger * https://bugs.freebsd.org/241662 Flakey test case: lib.libarchive.functional_test.test_fuzz_iso9660 ### Others * [Tickets related to testing@](https://preview.tinyurl.com/y9maauwg) From owner-freebsd-testing@freebsd.org Wed Apr 15 13:34:17 2020 Return-Path: Delivered-To: freebsd-testing@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D46692B65FB; Wed, 15 Apr 2020 13:34:17 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 492NcT4tFSz3M1b; Wed, 15 Apr 2020 13:34:17 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 74F9514BAF; Wed, 15 Apr 2020 13:34:17 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id BF7DBEA1B; Wed, 15 Apr 2020 15:34:15 +0200 (CEST) From: "Kristof Provost" To: freebsd-testing@freebsd.org Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: FreeBSD CI Weekly Report 2020-04-12 Date: Wed, 15 Apr 2020 15:34:12 +0200 X-Mailer: MailMate (1.13.1r5671) Message-ID: In-Reply-To: <20200414223710.GB33328@freefall.freebsd.org> References: <20200414223710.GB33328@freefall.freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Apr 2020 13:34:17 -0000 On 15 Apr 2020, at 0:37, Li-Wen Hsu wrote: > (Please send the followup to freebsd-testing@ and note Reply-To is > set.) > > FreeBSD CI Weekly Report 2020-04-12 > =================================== > > Here is a summary of the FreeBSD Continuous Integration results for > the period > from 2020-04-06 to 2020-04-12. > > During this period, we have: > > * 1801 builds (94.0% (+0.4) passed, 6.0% (-0.4) failed) of buildworld > and > buildkernel (GENERIC and LINT) were executed on aarch64, amd64, > armv6, > armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, riscv64, > sparc64 architectures for head, stable/12, stable/11 branches. > * 288 test runs (25.1% (-24.6) passed, 29.9% (+10.6) unstable, 45.1% > (+14.1) > exception) were executed on amd64, i386, riscv64 architectures for > head, > stable/12, stable/11 branches. > * 30 doc and www builds (83.3% (-1.3) passed, 16.7% (+1.3) failed) > > Test case status (on 2020-04-12 23:59): > | Branch/Architecture | Total | Pass | Fail | Skipped | > | ------------------- | --------- | ---------- | -------- | -------- | > | head/amd64 | 7744 (+4) | 7638 (+19) | 14 (+5) | 92 (-20) | > | head/i386 | 7742 (+4) | 7628 (+15) | 16 (+5) | 98 (-16) | > | 12-STABLE/amd64 | 7508 (0) | 7449 (-3) | 1 (+1) | 58 (+2) | > | 12-STABLE/i386 | 7506 (0) | 7425 (-17) | 2 (+2) | 79 (+15) | > | 11-STABLE/amd64 | 6882 (0) | 6829 (-6) | 1 (+1) | 52 (+5) | > | 11-STABLE/i386 | 6880 (0) | 6749 (-82) | 80 (+80) | 51 (+2) | > > (The statistics from experimental jobs are omitted) > > If any of the issues found by CI are in your area of interest or > expertise > please investigate the PRs listed below. > > The latest web version of this report is available at > https://hackmd.io/@FreeBSD-CI/report-20200412 and archive is available > at > https://hackmd.io/@FreeBSD-CI/ , any help is welcome. > > ## News > > * The test env now loads the required module for firewall tests. > > * New armv7 job is added (to replace armv6 one): > * FreeBSD-head-armv7-testvm > The images are available at https://artifact.ci.freebsd.org > FreeBSD-head-armv7-test is ready but needs test env update. > > ## Failing jobs > > * https://ci.freebsd.org/job/FreeBSD-head-amd64-gcc6_build/ > * See console log for the error details. > > ## Failing tests > > * https://ci.freebsd.org/job/FreeBSD-head-amd64-test/ > * local.kyua.integration.cmd_about_test.topic__authors__installed > * sys.netipsec.tunnel.empty.v4 > * sys.netipsec.tunnel.empty.v6 > * sys.netpfil.common.forward.ipf_v4 > * sys.netpfil.common.forward.ipfw_v4 > * sys.netpfil.common.forward.pf_v4 > * sys.netpfil.common.tos.ipfw_tos > * sys.netpfil.common.tos.pf_tos > * sys.netpfil.pf.forward.v4 I can’t actually reproduce this failure in my test VM, but with the ci test VM I can reproduce the problem. It’s not related to pf, because the sanity check ping we do before we set up pf already fails. Or rather pft_ping.py sends an incorrect packet, because `ping` does get the packet to go where it’s supposed to go. Scapy seems to fail to find the source IP address, so we get this: 12:12:22.152652 IP 0.0.0.0 > 198.51.100.3: ICMP echo request, id 0, seq 0, length 12 I have a vague recollection that we’ve seem this problem before, but I can’t remember what we did about it. In all likelihood most of the other netpfil tests fail for exactly the same reason. Best regards, Kristof From owner-freebsd-testing@freebsd.org Wed Apr 15 14:09:57 2020 Return-Path: Delivered-To: freebsd-testing@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BFF8B2B7FD3; Wed, 15 Apr 2020 14:09:57 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 492PPd53FVz3PW8; Wed, 15 Apr 2020 14:09:57 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 7A9EE14F66; Wed, 15 Apr 2020 14:09:57 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id 20857EAAA; Wed, 15 Apr 2020 16:09:56 +0200 (CEST) From: "Kristof Provost" To: freebsd-testing@freebsd.org Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org, bofh@freebsd.org, "Alexander V. Chernikov" Subject: Re: FreeBSD CI Weekly Report 2020-04-12 Date: Wed, 15 Apr 2020 16:09:55 +0200 X-Mailer: MailMate (1.13.1r5671) Message-ID: In-Reply-To: References: <20200414223710.GB33328@freefall.freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Apr 2020 14:09:57 -0000 On 15 Apr 2020, at 15:34, Kristof Provost wrote: > On 15 Apr 2020, at 0:37, Li-Wen Hsu wrote: >> (Please send the followup to freebsd-testing@ and note Reply-To is >> set.) >> >> FreeBSD CI Weekly Report 2020-04-12 >> =================================== >> >> Here is a summary of the FreeBSD Continuous Integration results for >> the period >> from 2020-04-06 to 2020-04-12. >> >> During this period, we have: >> >> * 1801 builds (94.0% (+0.4) passed, 6.0% (-0.4) failed) of buildworld >> and >> buildkernel (GENERIC and LINT) were executed on aarch64, amd64, >> armv6, >> armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, riscv64, >> sparc64 architectures for head, stable/12, stable/11 branches. >> * 288 test runs (25.1% (-24.6) passed, 29.9% (+10.6) unstable, 45.1% >> (+14.1) >> exception) were executed on amd64, i386, riscv64 architectures for >> head, >> stable/12, stable/11 branches. >> * 30 doc and www builds (83.3% (-1.3) passed, 16.7% (+1.3) failed) >> >> Test case status (on 2020-04-12 23:59): >> | Branch/Architecture | Total | Pass | Fail | Skipped >> | >> | ------------------- | --------- | ---------- | -------- | -------- >> | >> | head/amd64 | 7744 (+4) | 7638 (+19) | 14 (+5) | 92 (-20) >> | >> | head/i386 | 7742 (+4) | 7628 (+15) | 16 (+5) | 98 (-16) >> | >> | 12-STABLE/amd64 | 7508 (0) | 7449 (-3) | 1 (+1) | 58 (+2) >> | >> | 12-STABLE/i386 | 7506 (0) | 7425 (-17) | 2 (+2) | 79 (+15) >> | >> | 11-STABLE/amd64 | 6882 (0) | 6829 (-6) | 1 (+1) | 52 (+5) >> | >> | 11-STABLE/i386 | 6880 (0) | 6749 (-82) | 80 (+80) | 51 (+2) >> | >> >> (The statistics from experimental jobs are omitted) >> >> If any of the issues found by CI are in your area of interest or >> expertise >> please investigate the PRs listed below. >> >> The latest web version of this report is available at >> https://hackmd.io/@FreeBSD-CI/report-20200412 and archive is >> available at >> https://hackmd.io/@FreeBSD-CI/ , any help is welcome. >> >> ## News >> >> * The test env now loads the required module for firewall tests. >> >> * New armv7 job is added (to replace armv6 one): >> * FreeBSD-head-armv7-testvm >> The images are available at https://artifact.ci.freebsd.org >> FreeBSD-head-armv7-test is ready but needs test env update. >> >> ## Failing jobs >> >> * https://ci.freebsd.org/job/FreeBSD-head-amd64-gcc6_build/ >> * See console log for the error details. >> >> ## Failing tests >> >> * https://ci.freebsd.org/job/FreeBSD-head-amd64-test/ >> * local.kyua.integration.cmd_about_test.topic__authors__installed >> * sys.netipsec.tunnel.empty.v4 >> * sys.netipsec.tunnel.empty.v6 >> * sys.netpfil.common.forward.ipf_v4 >> * sys.netpfil.common.forward.ipfw_v4 >> * sys.netpfil.common.forward.pf_v4 >> * sys.netpfil.common.tos.ipfw_tos >> * sys.netpfil.common.tos.pf_tos >> * sys.netpfil.pf.forward.v4 > I can’t actually reproduce this failure in my test VM, but with the > ci test VM I can reproduce the problem. > It’s not related to pf, because the sanity check ping we do before > we set up pf already fails. > Or rather pft_ping.py sends an incorrect packet, because `ping` does > get the packet to go where it’s supposed to go. > > Scapy seems to fail to find the source IP address, so we get this: > > 12:12:22.152652 IP 0.0.0.0 > 198.51.100.3: ICMP echo request, id 0, > seq 0, length 12 > > I have a vague recollection that we’ve seem this problem before, but > I can’t remember what we did about it. > > In all likelihood most of the other netpfil tests fail for exactly the > same reason. The problem appears to be that /usr/local/lib/python3.7/site-packages/scapy/arch/unix.py is misparsing the `netstat -rnW` output. For reference, this is the output in the test VM: Routing tables Internet: Destination Gateway Flags Nhop# Mtu Netif Expire 127.0.0.1 link#2 UH 1 16384 lo0 192.0.2.0/24 link#4 U 2 1500 epair0a 192.0.2.1 link#4 UHS 1 16384 lo0 198.51.100.0/24 192.0.2.2 UGS 3 1500 epair0a Internet6: Destination Gateway Flags Nhop# Mtu Netif Expire ::/96 ::1 UGRS 4 16384 lo0 ::1 link#2 UH 1 16384 lo0 ::ffff:0.0.0.0/96 ::1 UGRS 4 16384 lo0 fe80::/10 ::1 UGRS 4 16384 lo0 fe80::%lo0/64 link#2 U 3 16384 lo0 fe80::1%lo0 link#2 UHS 2 16384 lo0 fe80::%epair0a/64 link#4 U 5 1500 epair0a fe80::3d:9dff:fe7c:d70a%epair0a link#4 UHS 1 16384 lo0 fe80::%epair1a/64 link#6 U 6 1500 epair1a fe80::37:9eff:fe03:250a%epair1a link#6 UHS 1 16384 lo0 ff02::/16 ::1 UGRS 4 16384 lo0 The parsing code seems to think that the netif for the 198.51.100.0/24 route is 1500 rather than epair0a. This may be related to the difference in netstat output, because on my VM it looks like this: Routing tables Internet: Destination Gateway Flags Use Mtu Netif Expire default 172.16.2.1 UGS 319 1500 vtnet0 127.0.0.1 link#2 UH 0 16384 lo0 172.16.2.0/24 link#1 U 14 1500 vtnet0 172.16.2.2 link#1 UHS 0 16384 lo0 Internet6: Destination Gateway Flags Use Mtu Netif Expire ::/96 ::1 UGRS 0 16384 lo0 ::1 link#2 UH 0 16384 lo0 ::ffff:0.0.0.0/96 ::1 UGRS 0 16384 lo0 fe80::/10 ::1 UGRS 0 16384 lo0 fe80::%vtnet0/64 link#1 U 0 1500 vtnet0 fe80::5a9c:fcff:fe02:a95e%vtnet0 link#1 UHS 0 16384 lo0 fe80::%lo0/64 link#2 U 0 16384 lo0 fe80::1%lo0 link#2 UHS 0 16384 lo0 ff02::/16 ::1 UGRS 0 16384 lo0 I suspect that this change was introduced in r359823 (Introduce nexthop objects and new routing KPI). Best regards, Kristof From owner-freebsd-testing@freebsd.org Wed Apr 15 14:49:44 2020 Return-Path: Delivered-To: freebsd-testing@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4048E2B8F3B; Wed, 15 Apr 2020 14:49:44 +0000 (UTC) (envelope-from olivier@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 492QHX0xqyz3wt2; Wed, 15 Apr 2020 14:49:44 +0000 (UTC) (envelope-from olivier@freebsd.org) Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: olivier/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id E6B86170E7; Wed, 15 Apr 2020 14:49:43 +0000 (UTC) (envelope-from olivier@freebsd.org) Received: by mail-pg1-f179.google.com with SMTP id r4so17093pgg.4; Wed, 15 Apr 2020 07:49:43 -0700 (PDT) X-Gm-Message-State: AGi0PuYCsN/5awg9udJTfwL7GvOf1DjTBmnISu7CpUvEVvY6WRFj07ag imRgxkdYAlQg+6ygtL28GIdD2tOMvK/So9jB1CM= X-Google-Smtp-Source: APiQypKecHW77MF5DIXi8FXRktWsuQds0xTOB4APspMaWMrkHF4euHjE8RZvfkDfLkoNh/Z3pJhQk8izB38QA7UpQLQ= X-Received: by 2002:a62:4e0c:: with SMTP id c12mr27056707pfb.87.1586962182720; Wed, 15 Apr 2020 07:49:42 -0700 (PDT) MIME-Version: 1.0 References: <20200414223710.GB33328@freefall.freebsd.org> In-Reply-To: From: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= Date: Wed, 15 Apr 2020 16:49:31 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: FreeBSD CI Weekly Report 2020-04-12 To: Kristof Provost Cc: freebsd-testing@freebsd.org, freebsd-current , "freebsd-stable@freebsd.org Stable" , bofh@freebsd.org, "Alexander V. Chernikov" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Apr 2020 14:49:44 -0000 On Wed, Apr 15, 2020 at 4:10 PM Kristof Provost wrote: > > The problem appears to be that > /usr/local/lib/python3.7/site-packages/scapy/arch/unix.py is misparsing > the `netstat -rnW` output. > Shouldn't scapy use the libxo output of netstat to mitigate this regression ? From owner-freebsd-testing@freebsd.org Wed Apr 15 15:28:18 2020 Return-Path: Delivered-To: freebsd-testing@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D5E5D2BA023; Wed, 15 Apr 2020 15:28:18 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 492R825Nhvz40DN; Wed, 15 Apr 2020 15:28:18 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 9A0EA175E8; Wed, 15 Apr 2020 15:28:18 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id 35C53ED27; Wed, 15 Apr 2020 17:28:17 +0200 (CEST) From: "Kristof Provost" To: "Olivier =?utf-8?q?Cochard-Labb=C3=A9?=" Cc: freebsd-testing@freebsd.org, freebsd-current , Stable , bofh@freebsd.org, "Alexander V. Chernikov" Subject: Re: FreeBSD CI Weekly Report 2020-04-12 Date: Wed, 15 Apr 2020 17:28:13 +0200 X-Mailer: MailMate (1.13.1r5671) Message-ID: In-Reply-To: References: <20200414223710.GB33328@freefall.freebsd.org> MIME-Version: 1.0 Embedded-HTML: [{"HTML":[362, 854], "plain":[58, 303], "uuid":"C83DB664-8363-477B-9B11-992FD4A5E8E1"}] Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Apr 2020 15:28:18 -0000 On 15 Apr 2020, at 16:49, Olivier Cochard-Labbé wrote: > On Wed, Apr 15, 2020 at 4:10 PM Kristof Provost > wrote: > >> >> The problem appears to be that >> /usr/local/lib/python3.7/site-packages/scapy/arch/unix.py is >> misparsing >> the `netstat -rnW` output. >> > > Shouldn't scapy use the libxo output of netstat to mitigate this > regression > ? That would likely help, yes. I’m going to leave that decision up to the maintainer, because I’m not going to do the work :) I’m also not sure how “stable” we want the netstat output to be. Best regards, Kristof From owner-freebsd-testing@freebsd.org Wed Apr 15 20:21:32 2020 Return-Path: Delivered-To: freebsd-testing@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4467A2C1022; Wed, 15 Apr 2020 20:21:32 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from forward103o.mail.yandex.net (forward103o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::606]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 492YfK5H1Wz4Lyl; Wed, 15 Apr 2020 20:21:29 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from forward501o.mail.yandex.net (forward501o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::611]) by forward103o.mail.yandex.net (Yandex) with ESMTP id 4208D5F8320A; Wed, 15 Apr 2020 23:21:11 +0300 (MSK) Received: from mxback24o.mail.yandex.net (mxback24o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::75]) by forward501o.mail.yandex.net (Yandex) with ESMTP id 3EACF1E80002; Wed, 15 Apr 2020 23:21:11 +0300 (MSK) Received: from localhost (localhost [::1]) by mxback24o.mail.yandex.net (mxback/Yandex) with ESMTP id DIwk9NYhfE-LA2S1Nra; Wed, 15 Apr 2020 23:21:10 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfw.ru; s=mail; t=1586982070; bh=pcnmyfi5q3HAMA53d7VJPHEH6EjFCQiuew2bY/lFFC8=; h=Message-Id:Cc:Subject:In-Reply-To:Date:References:To:From; b=cKmyl/jtOeyU7W0titJHlwzJpZoXtxvNdNC+PcrsWiw84mXhxbY4PhcnXFas3sUM0 hdfN1pj4ZFMHWU/2I4LZ9B/pWwFiwMbeRj0CIeN9vYSOumPm5e8S5X86FnqzQ2aWYc GA0fNVnuk6de2MIPE3itCUSd7V3zGnPVvUDvANEc= Received: by myt2-c3952fd46804.qloud-c.yandex.net with HTTP; Wed, 15 Apr 2020 23:21:10 +0300 From: Alexander V. Chernikov To: Kristof Provost , "freebsd-testing@freebsd.org" Cc: "freebsd-current@freebsd.org" , "freebsd-stable@freebsd.org" , "bofh@freebsd.org" , =?utf-8?B?T2xpdmllciBDb2NoYXJkLUxhYmLDqQ==?= In-Reply-To: References: <20200414223710.GB33328@freefall.freebsd.org> Subject: Re: FreeBSD CI Weekly Report 2020-04-12 MIME-Version: 1.0 X-Mailer: Yamail [ http://yandex.ru ] 5.0 Date: Wed, 15 Apr 2020 21:21:10 +0100 Message-Id: <885331586982061@myt5-bc0f9d8e5f27.qloud-c.yandex.net> Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=utf-8 X-Rspamd-Queue-Id: 492YfK5H1Wz4Lyl X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ipfw.ru header.s=mail header.b=cKmyl/jt; dmarc=none; spf=pass (mx1.freebsd.org: domain of melifaro@ipfw.ru designates 2a02:6b8:0:1a2d::606 as permitted sender) smtp.mailfrom=melifaro@ipfw.ru X-Spamd-Result: default: False [-6.16 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[ipfw.ru:s=mail]; RCVD_COUNT_FIVE(0.00)[5]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a02:6b8:0:1000::/52]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[ipfw.ru]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_FIVE(0.00)[6]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; DKIM_TRACE(0.00)[ipfw.ru:+]; RCVD_IN_DNSWL_NONE(0.00)[6.0.6.0.0.0.0.0.0.0.0.0.0.0.0.0.d.2.a.1.0.0.0.0.8.b.6.0.2.0.a.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:13238, ipnet:2a02:6b8::/32, country:RU]; IP_SCORE(-3.66)[ip: (-9.72), ipnet: 2a02:6b8::/32(-4.76), asn: 13238(-3.85), country: RU(0.01)] X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Apr 2020 20:21:32 -0000 15.04.2020, 15:10, "Kristof Provost" : > On 15 Apr 2020, at 15:34, Kristof Provost wrote: >>  On 15 Apr 2020, at 0:37, Li-Wen Hsu wrote: >>>  (Please send the followup to freebsd-testing@ and note Reply-To is >>>  set.) >>> >>>  FreeBSD CI Weekly Report 2020-04-12 >>>  =================================== >>> >>>  Here is a summary of the FreeBSD Continuous Integration results for >>>  the period >>>  from 2020-04-06 to 2020-04-12. >>> >>>  During this period, we have: >>> >>>  * 1801 builds (94.0% (+0.4) passed, 6.0% (-0.4) failed) of buildworld >>>  and >>>    buildkernel (GENERIC and LINT) were executed on aarch64, amd64, >>>  armv6, >>>    armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, riscv64, >>>    sparc64 architectures for head, stable/12, stable/11 branches. >>>  * 288 test runs (25.1% (-24.6) passed, 29.9% (+10.6) unstable, 45.1% >>>  (+14.1) >>>    exception) were executed on amd64, i386, riscv64 architectures for >>>  head, >>>    stable/12, stable/11 branches. >>>  * 30 doc and www builds (83.3% (-1.3) passed, 16.7% (+1.3) failed) >>> >>>  Test case status (on 2020-04-12 23:59): >>>  | Branch/Architecture | Total | Pass | Fail | Skipped >>>  | >>>  | ------------------- | --------- | ---------- | -------- | -------- >>>  | >>>  | head/amd64 | 7744 (+4) | 7638 (+19) | 14 (+5) | 92 (-20) >>>  | >>>  | head/i386 | 7742 (+4) | 7628 (+15) | 16 (+5) | 98 (-16) >>>  | >>>  | 12-STABLE/amd64 | 7508 (0) | 7449 (-3) | 1 (+1) | 58 (+2) >>>  | >>>  | 12-STABLE/i386 | 7506 (0) | 7425 (-17) | 2 (+2) | 79 (+15) >>>  | >>>  | 11-STABLE/amd64 | 6882 (0) | 6829 (-6) | 1 (+1) | 52 (+5) >>>  | >>>  | 11-STABLE/i386 | 6880 (0) | 6749 (-82) | 80 (+80) | 51 (+2) >>>  | >>> >>>  (The statistics from experimental jobs are omitted) >>> >>>  If any of the issues found by CI are in your area of interest or >>>  expertise >>>  please investigate the PRs listed below. >>> >>>  The latest web version of this report is available at >>>  https://hackmd.io/@FreeBSD-CI/report-20200412 and archive is >>>  available at >>>  https://hackmd.io/@FreeBSD-CI/ , any help is welcome. >>> >>>  ## News >>> >>>  * The test env now loads the required module for firewall tests. >>> >>>  * New armv7 job is added (to replace armv6 one): >>>    * FreeBSD-head-armv7-testvm >>>    The images are available at https://artifact.ci.freebsd.org >>>    FreeBSD-head-armv7-test is ready but needs test env update. >>> >>>  ## Failing jobs >>> >>>  * https://ci.freebsd.org/job/FreeBSD-head-amd64-gcc6_build/ >>>    * See console log for the error details. >>> >>>  ## Failing tests >>> >>>  * https://ci.freebsd.org/job/FreeBSD-head-amd64-test/ >>>    * local.kyua.integration.cmd_about_test.topic__authors__installed >>>    * sys.netipsec.tunnel.empty.v4 >>>    * sys.netipsec.tunnel.empty.v6 >>>    * sys.netpfil.common.forward.ipf_v4 >>>    * sys.netpfil.common.forward.ipfw_v4 >>>    * sys.netpfil.common.forward.pf_v4 >>>    * sys.netpfil.common.tos.ipfw_tos >>>    * sys.netpfil.common.tos.pf_tos >>>    * sys.netpfil.pf.forward.v4 >>  I can’t actually reproduce this failure in my test VM, but with the >>  ci test VM I can reproduce the problem. >>  It’s not related to pf, because the sanity check ping we do before >>  we set up pf already fails. >>  Or rather pft_ping.py sends an incorrect packet, because `ping` does >>  get the packet to go where it’s supposed to go. >> >>  Scapy seems to fail to find the source IP address, so we get this: >> >>          12:12:22.152652 IP 0.0.0.0 > 198.51.100.3: ICMP echo request, id 0, >>  seq 0, length 12 >> >>  I have a vague recollection that we’ve seem this problem before, but >>  I can’t remember what we did about it. >> >>  In all likelihood most of the other netpfil tests fail for exactly the >>  same reason. > > The problem appears to be that > /usr/local/lib/python3.7/site-packages/scapy/arch/unix.py is misparsing > the `netstat -rnW` output. Thanks for the analysis! Sorry for breaking the tests. I should have run tests with userland changes installed. I'll revert the netstat output changes shortly to unbreak the tests. Re longer-term: parsing textual output for the routes does not seem to be a good habit, especially in these days. Structural (libxo) approach looks better, however I guess this will make scapy unusable on the routers with full-view. So far light-weight sysctl-route reader looks like the best option. What do you folks think? > > For reference, this is the output in the test VM: > >         Routing tables > >         Internet: >         Destination Gateway Flags Nhop# Mtu Netif > Expire >         127.0.0.1 link#2 UH 1 16384 lo0 >         192.0.2.0/24 link#4 U 2 1500 epair0a >         192.0.2.1 link#4 UHS 1 16384 lo0 >         198.51.100.0/24 192.0.2.2 UGS 3 1500 epair0a > >         Internet6: >         Destination Gateway Flags > Nhop# Mtu Netif Expire >         ::/96 ::1 UGRS >      4 16384 lo0 >         ::1 link#2 UH >      1 16384 lo0 >         ::ffff:0.0.0.0/96 ::1 UGRS >      4 16384 lo0 >         fe80::/10 ::1 UGRS >      4 16384 lo0 >         fe80::%lo0/64 link#2 U >      3 16384 lo0 >         fe80::1%lo0 link#2 UHS >      2 16384 lo0 >         fe80::%epair0a/64 link#4 U >      5 1500 epair0a >         fe80::3d:9dff:fe7c:d70a%epair0a link#4 UHS >      1 16384 lo0 >         fe80::%epair1a/64 link#6 U >      6 1500 epair1a >         fe80::37:9eff:fe03:250a%epair1a link#6 UHS >      1 16384 lo0 >         ff02::/16 ::1 UGRS >      4 16384 lo0 > > The parsing code seems to think that the netif for the 198.51.100.0/24 > route is 1500 rather than epair0a. > This may be related to the difference in netstat output, because on my > VM it looks like this: > >         Routing tables > >         Internet: >         Destination Gateway Flags Use Mtu Netif > Expire >         default 172.16.2.1 UGS 319 1500 vtnet0 >         127.0.0.1 link#2 UH 0 16384 lo0 >         172.16.2.0/24 link#1 U 14 1500 vtnet0 >         172.16.2.2 link#1 UHS 0 16384 lo0 > >         Internet6: >         Destination Gateway Flags >      Use Mtu Netif Expire >         ::/96 ::1 UGRS >        0 16384 lo0 >         ::1 link#2 UH >        0 16384 lo0 >         ::ffff:0.0.0.0/96 ::1 UGRS >        0 16384 lo0 >         fe80::/10 ::1 UGRS >        0 16384 lo0 >         fe80::%vtnet0/64 link#1 U >        0 1500 vtnet0 >         fe80::5a9c:fcff:fe02:a95e%vtnet0 link#1 UHS >        0 16384 lo0 >         fe80::%lo0/64 link#2 U >        0 16384 lo0 >         fe80::1%lo0 link#2 UHS >        0 16384 lo0 >         ff02::/16 ::1 UGRS >        0 16384 lo0 > > I suspect that this change was introduced in r359823 (Introduce nexthop > objects and new routing KPI). > > Best regards, > Kristof > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" From owner-freebsd-testing@freebsd.org Wed Apr 15 20:27:45 2020 Return-Path: Delivered-To: freebsd-testing@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 02D362C15DF; Wed, 15 Apr 2020 20:27:45 +0000 (UTC) (envelope-from bofh@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 492YnX6F3Cz4N2q; Wed, 15 Apr 2020 20:27:44 +0000 (UTC) (envelope-from bofh@freebsd.org) Received: from mx.bofh.network (mx.bofh.network [IPv6:2001:19f0:5001:2b77:5400:2ff:fe7b:aa2c]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx.bofh.network", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: bofh/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 3461619AC0; Wed, 15 Apr 2020 20:27:44 +0000 (UTC) (envelope-from bofh@freebsd.org) Received: from [IPv6:2402:54c0:ffff:ffff:9188:3af7:1bd8:3b5d] ( [2402:54c0:ffff:ffff:9188:3af7:1bd8:3b5d]) by mx.bofh.network (OpenSMTPD) with ESMTPSA id 74b240b3 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Wed, 15 Apr 2020 20:27:41 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: FreeBSD CI Weekly Report 2020-04-12 From: Muhammad Moinur Rahman In-Reply-To: <885331586982061@myt5-bc0f9d8e5f27.qloud-c.yandex.net> Date: Thu, 16 Apr 2020 02:27:36 +0600 Cc: Kristof Provost , "freebsd-testing@freebsd.org" , "freebsd-current@freebsd.org" , "freebsd-stable@freebsd.org" , =?utf-8?Q?Olivier_Cochard-Labb=C3=A9?= Content-Transfer-Encoding: quoted-printable Message-Id: References: <20200414223710.GB33328@freefall.freebsd.org> <885331586982061@myt5-bc0f9d8e5f27.qloud-c.yandex.net> To: "Alexander V. Chernikov" X-Mailer: Apple Mail (2.3608.60.0.2.5) X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Apr 2020 20:27:45 -0000 Hi Alexandar, I have changed net/scapy to apply a simple patch as a workaround for the = time being. ports@r531789 should do the trick for now. So no need to = revert. Kind Regards, Moin > On 16 Apr, 2020, at 02:21, Alexander V. Chernikov = wrote: >=20 >=20 >=20 > 15.04.2020, 15:10, "Kristof Provost" : >> On 15 Apr 2020, at 15:34, Kristof Provost wrote: >>> On 15 Apr 2020, at 0:37, Li-Wen Hsu wrote: >>>> (Please send the followup to freebsd-testing@ and note Reply-To is >>>> set.) >>>>=20 >>>> FreeBSD CI Weekly Report 2020-04-12 >>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>>=20 >>>> Here is a summary of the FreeBSD Continuous Integration results = for >>>> the period >>>> from 2020-04-06 to 2020-04-12. >>>>=20 >>>> During this period, we have: >>>>=20 >>>> * 1801 builds (94.0% (+0.4) passed, 6.0% (-0.4) failed) of = buildworld >>>> and >>>> buildkernel (GENERIC and LINT) were executed on aarch64, amd64, >>>> armv6, >>>> armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, = riscv64, >>>> sparc64 architectures for head, stable/12, stable/11 branches. >>>> * 288 test runs (25.1% (-24.6) passed, 29.9% (+10.6) unstable, = 45.1% >>>> (+14.1) >>>> exception) were executed on amd64, i386, riscv64 architectures = for >>>> head, >>>> stable/12, stable/11 branches. >>>> * 30 doc and www builds (83.3% (-1.3) passed, 16.7% (+1.3) failed) >>>>=20 >>>> Test case status (on 2020-04-12 23:59): >>>> | Branch/Architecture | Total | Pass | Fail | Skipped >>>> | >>>> | ------------------- | --------- | ---------- | -------- | = -------- >>>> | >>>> | head/amd64 | 7744 (+4) | 7638 (+19) | 14 (+5) | 92 (-20) >>>> | >>>> | head/i386 | 7742 (+4) | 7628 (+15) | 16 (+5) | 98 (-16) >>>> | >>>> | 12-STABLE/amd64 | 7508 (0) | 7449 (-3) | 1 (+1) | 58 (+2) >>>> | >>>> | 12-STABLE/i386 | 7506 (0) | 7425 (-17) | 2 (+2) | 79 (+15) >>>> | >>>> | 11-STABLE/amd64 | 6882 (0) | 6829 (-6) | 1 (+1) | 52 (+5) >>>> | >>>> | 11-STABLE/i386 | 6880 (0) | 6749 (-82) | 80 (+80) | 51 (+2) >>>> | >>>>=20 >>>> (The statistics from experimental jobs are omitted) >>>>=20 >>>> If any of the issues found by CI are in your area of interest or >>>> expertise >>>> please investigate the PRs listed below. >>>>=20 >>>> The latest web version of this report is available at >>>> https://hackmd.io/@FreeBSD-CI/report-20200412 and archive is >>>> available at >>>> https://hackmd.io/@FreeBSD-CI/ , any help is welcome. >>>>=20 >>>> ## News >>>>=20 >>>> * The test env now loads the required module for firewall tests. >>>>=20 >>>> * New armv7 job is added (to replace armv6 one): >>>> * FreeBSD-head-armv7-testvm >>>> The images are available at https://artifact.ci.freebsd.org >>>> FreeBSD-head-armv7-test is ready but needs test env update. >>>>=20 >>>> ## Failing jobs >>>>=20 >>>> * https://ci.freebsd.org/job/FreeBSD-head-amd64-gcc6_build/ >>>> * See console log for the error details. >>>>=20 >>>> ## Failing tests >>>>=20 >>>> * https://ci.freebsd.org/job/FreeBSD-head-amd64-test/ >>>> * = local.kyua.integration.cmd_about_test.topic__authors__installed >>>> * sys.netipsec.tunnel.empty.v4 >>>> * sys.netipsec.tunnel.empty.v6 >>>> * sys.netpfil.common.forward.ipf_v4 >>>> * sys.netpfil.common.forward.ipfw_v4 >>>> * sys.netpfil.common.forward.pf_v4 >>>> * sys.netpfil.common.tos.ipfw_tos >>>> * sys.netpfil.common.tos.pf_tos >>>> * sys.netpfil.pf.forward.v4 >>> I can=E2=80=99t actually reproduce this failure in my test VM, but = with the >>> ci test VM I can reproduce the problem. >>> It=E2=80=99s not related to pf, because the sanity check ping we do = before >>> we set up pf already fails. >>> Or rather pft_ping.py sends an incorrect packet, because `ping` = does >>> get the packet to go where it=E2=80=99s supposed to go. >>>=20 >>> Scapy seems to fail to find the source IP address, so we get this: >>>=20 >>> 12:12:22.152652 IP 0.0.0.0 > 198.51.100.3: ICMP echo = request, id 0, >>> seq 0, length 12 >>>=20 >>> I have a vague recollection that we=E2=80=99ve seem this problem = before, but >>> I can=E2=80=99t remember what we did about it. >>>=20 >>> In all likelihood most of the other netpfil tests fail for exactly = the >>> same reason. >>=20 >> The problem appears to be that >> /usr/local/lib/python3.7/site-packages/scapy/arch/unix.py is = misparsing >> the `netstat -rnW` output. > Thanks for the analysis! >=20 > Sorry for breaking the tests. > I should have run tests with userland changes installed. >=20 > I'll revert the netstat output changes shortly to unbreak the tests. > Re longer-term: parsing textual output for the routes does not seem to = be a good habit, especially in these days. > Structural (libxo) approach looks better, however I guess this will = make scapy unusable on the routers with full-view. >=20 > So far light-weight sysctl-route reader looks like the best option. > What do you folks think? >=20 >>=20 >> For reference, this is the output in the test VM: >>=20 >> Routing tables >>=20 >> Internet: >> Destination Gateway Flags Nhop# Mtu Netif >> Expire >> 127.0.0.1 link#2 UH 1 16384 lo0 >> 192.0.2.0/24 link#4 U 2 1500 epair0a >> 192.0.2.1 link#4 UHS 1 16384 lo0 >> 198.51.100.0/24 192.0.2.2 UGS 3 1500 epair0a >>=20 >> Internet6: >> Destination Gateway Flags >> Nhop# Mtu Netif Expire >> ::/96 ::1 UGRS >> 4 16384 lo0 >> ::1 link#2 UH >> 1 16384 lo0 >> ::ffff:0.0.0.0/96 ::1 UGRS >> 4 16384 lo0 >> fe80::/10 ::1 UGRS >> 4 16384 lo0 >> fe80::%lo0/64 link#2 U >> 3 16384 lo0 >> fe80::1%lo0 link#2 UHS >> 2 16384 lo0 >> fe80::%epair0a/64 link#4 U >> 5 1500 epair0a >> fe80::3d:9dff:fe7c:d70a%epair0a link#4 UHS >> 1 16384 lo0 >> fe80::%epair1a/64 link#6 U >> 6 1500 epair1a >> fe80::37:9eff:fe03:250a%epair1a link#6 UHS >> 1 16384 lo0 >> ff02::/16 ::1 UGRS >> 4 16384 lo0 >>=20 >> The parsing code seems to think that the netif for the = 198.51.100.0/24 >> route is 1500 rather than epair0a. >> This may be related to the difference in netstat output, because on = my >> VM it looks like this: >>=20 >> Routing tables >>=20 >> Internet: >> Destination Gateway Flags Use Mtu Netif >> Expire >> default 172.16.2.1 UGS 319 1500 vtnet0 >> 127.0.0.1 link#2 UH 0 16384 lo0 >> 172.16.2.0/24 link#1 U 14 1500 vtnet0 >> 172.16.2.2 link#1 UHS 0 16384 lo0 >>=20 >> Internet6: >> Destination Gateway Flags >> Use Mtu Netif Expire >> ::/96 ::1 UGRS >> 0 16384 lo0 >> ::1 link#2 UH >> 0 16384 lo0 >> ::ffff:0.0.0.0/96 ::1 UGRS >> 0 16384 lo0 >> fe80::/10 ::1 UGRS >> 0 16384 lo0 >> fe80::%vtnet0/64 link#1 U >> 0 1500 vtnet0 >> fe80::5a9c:fcff:fe02:a95e%vtnet0 link#1 UHS >> 0 16384 lo0 >> fe80::%lo0/64 link#2 U >> 0 16384 lo0 >> fe80::1%lo0 link#2 UHS >> 0 16384 lo0 >> ff02::/16 ::1 UGRS >> 0 16384 lo0 >>=20 >> I suspect that this change was introduced in r359823 (Introduce = nexthop >> objects and new routing KPI). >>=20 >> Best regards, >> Kristof >> _______________________________________________ >> freebsd-current@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to = "freebsd-current-unsubscribe@freebsd.org"