From owner-freebsd-testing@freebsd.org Wed Jun 12 15:13:30 2019 Return-Path: Delivered-To: freebsd-testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9C5015B9302; Wed, 12 Jun 2019 15:13:30 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: from mail-yw1-f65.google.com (mail-yw1-f65.google.com [209.85.161.65]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7772680F47; Wed, 12 Jun 2019 15:13:29 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: by mail-yw1-f65.google.com with SMTP id t2so6939809ywe.10; Wed, 12 Jun 2019 08:13:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=HgXElayOnBH1Ca1XjFBa3vAyFilXIHncrM7KvWSdyQY=; b=UlTpsOTw25FjKiIjxX6nRyl8GeQcNKa5P+E0KkT8+k1VKYRz016asj8MJRlm8tglNm Cr0De/2Xi9dxxsNh/OiPxAr4t7M/0KMEXvIPDfVRO7O2AT6KsJFh+iSSEjYCLivT/mfc lBILT3AcB049iE6O09ijOUJzYJkQM3uolSzYPYJUmH4dgXWDTnF7q+elaelsNGp3jUbB ZHfMnXnytIOh8rHnBGRGp4lG+eXiI4jj1TXMrSZcm0LgWWrzjaFO8sy7kzykNzXMzfaD msevxdZ2AAiI5Sm0CJMAhm0TvwNdIv+VcN5pjMZ3cbnjkOc1Dkk3yZ5frwO/C+yoi24L iJnA== X-Gm-Message-State: APjAAAUIW+KleuQIdUjvw2CoHdFE9HARlsAcGyhgNvC1ZCmn6Hw66phc c57h6cJH5Lxemt98N6YA41T2gDxGwJ3W1YqmJZyxYOK/ X-Google-Smtp-Source: APXvYqzudl0/0Ba/w9Xuh7IuwR+mnH4IorVgVSiwTAY7dZ1S8pIOH6aDlVAxJisPMVPNf1OWJGFvAnqWMy7sn4vMLr4= X-Received: by 2002:a0d:ce86:: with SMTP id q128mr13485802ywd.360.1560350963656; Wed, 12 Jun 2019 07:49:23 -0700 (PDT) MIME-Version: 1.0 From: Li-Wen Hsu Date: Wed, 12 Jun 2019 10:49:13 -0400 Message-ID: Subject: FreeBSD CI Weekly Report 2019-06-09 To: freebsd-testing@freebsd.org Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 7772680F47 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of lwhsufreebsd@gmail.com designates 209.85.161.65 as permitted sender) smtp.mailfrom=lwhsufreebsd@gmail.com X-Spamd-Result: default: False [-4.20 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_NEQ_ENVFROM(0.00)[lwhsu@freebsd.org,lwhsufreebsd@gmail.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_TLS_LAST(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.89)[-0.890,0]; RCVD_IN_DNSWL_NONE(0.00)[65.161.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.30)[ip: (-0.74), ipnet: 209.85.128.0/17(-3.41), asn: 15169(-2.30), country: US(-0.06)]; FORGED_SENDER(0.30)[lwhsu@freebsd.org,lwhsufreebsd@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; TAGGED_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jun 2019 15:13:31 -0000 (Please send the followup discussions to freebsd-testing@ list.) FreeBSD CI Weekly Report 2019-06-09 =================================== Here is a summary of the FreeBSD Continuous Integration results for the period from 2019-06-03 to 2019-06-09. During this period, we have: * 2444 builds (92.9% passed, 7.1% failed) were executed on aarch64, amd64, armv6, armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, riscv64, sparc64 architectures for head, stable/12, stable/11 branches. * 247 test runs (51% passed, 49% unstable) were executed on amd64, i386, riscv64 architectures for head, stable/12, stable/11 branches. * 76 doc builds (100% passed) (The statistics from experimental jobs are omitted) If any of the issues found by CI are in your area of interest or expertise please investigate the PRs listed below. The latest web version of this report is available at https://hackmd.io/s/BJI5k8RRV and archive is available at http://hackfoldr.org/freebsd-ci-report/, any help is welcome. ## Fixed Tests * https://ci.freebsd.org/job/FreeBSD-head-i386-test/ * [r348808](https://reviews.freebsd.org/rS348808) fixed i386 kernel panics on loading ipsec(4) kernel module, which is needed after [347410](https://svnweb.freebsd.org/changeset/base/347410). For more information, see: * https://bugs.freebsd.org/238012 * https://bugs.freebsd.org/230857 * https://reviews.freebsd.org/D17512 ## Failing builds * https://ci.freebsd.org/job/FreeBSD-head-amd64-gcc/ * See https://reviews.freebsd.org/D20622 * pkg(8) now requires `/dev/null` which doesn't exist on jail builds test VM image. Fix is under development now. ## Failing Tests * https://ci.freebsd.org/job/FreeBSD-head-amd64-test/ * sys.netinet.socket_afinet.socket_afinet_bind_zero Affected by mac_portacl(4), which is loaded by MAC tests. Need to specify AF_INET to workaround and fix is being discussed. * https://ci.freebsd.org/job/FreeBSD-head-i386-test/ * Same as amd64: * sys.netinet.socket_afinet.socket_afinet_bind_zero * Others: * sys.netpfil.pf.forward.v6 * sys.netpfil.pf.forward.v4 * sys.netpfil.pf.set_tos.v4 * sys.opencrypto.runtests.main * https://ci.freebsd.org/job/FreeBSD-stable-12-i386-test/ * sys.netpfil.pf.forward.v6 * sys.netpfil.pf.forward.v4 * sys.netpfil.pf.set_tos.v4 * lib.libc.regex.exhaust_test.regcomp_too_big * lib.libregex.exhaust_test.regcomp_too_big * sys.opencrypto.runtests.main Failed with: ``` File "/usr/tests/sys/opencrypto/cryptodev.py", line 179, in __init__ ioctl(_cryptodev, CIOCGSESSION2, s, 1) IOError: [Errno 22] Invalid argument ``` * https://ci.freebsd.org/job/FreeBSD-stable-11-i386-test/ * local.kyua.* (31 cases) * local.lutok.* (3 cases) ## Failing and Flaky Tests (from experimental jobs) * https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/ * Flakey test case: common.misc.t_dtrace_contrib.tst_dynopt_d https://bugs.freebsd.org/237641 * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/ * This job is currently suffering from timeout because of https://bugs.freebsd.org/237652 * There are ~60 failing cases, including flakey ones, see https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/lastCompletedBuild/testReport/ for more details ## Disabled Tests * lib.libc.sys.mmap_test.mmap_truncate_signal https://bugs.freebsd.org/211924 * sys.fs.tmpfs.mount_test.large https://bugs.freebsd.org/212862 * sys.fs.tmpfs.link_test.kqueue https://bugs.freebsd.org/213662 * sys.kqueue.libkqueue.kqueue_test.main https://bugs.freebsd.org/233586 * usr.bin.procstat.procstat_test.command_loogle.com/ine_arguments https://bugs.freebsd.org/233587 * usr.bin.procstat.procstat_test.environment https://bugs.freebsd.org/233588 ## Open Issues * https://bugs.freebsd.org/237077 possible race in build: /usr/src/sys/amd64/linux/linux_support.s:38:2: error: expected relocatable expression * https://bugs.freebsd.org/237403 Tests in sys/opencrypto should be converted to Python3 * https://bugs.freebsd.org/237641 Flakey test case: common.misc.t_dtrace_contrib.tst_dynopt_d * https://bugs.freebsd.org/237652 tests.hotspare.hotspare_test.hotspare_snapshot_001_pos timeout since somewhere in (r346814, r 346845] * https://bugs.freebsd.org/237655 Non-deterministic panic when running pf tests in interface ioctl code (NULL passed to strncmp) * https://bugs.freebsd.org/237656 "Freed UMA keg (rtentry) was not empty (18 items). Lost 1 pages of memory." seen when running sys/netipsec tests * https://bugs.freebsd.org/237657 sys.kern.pdeathsig.signal_delivered_ptrace timing out periodically on i386 ### Cause build fails * [233735: Possible build race: genoffset.o /usr/src/sys/sys/types.h: error: machine/endian.h: No such file or directory](https://bugs.freebsd.org/233735) * [233769: Possible build race: ld: error: unable to find library -lgcc_s](https://bugs.freebsd.org/233769) ### Others * [Tickets related to testing@](https://preview.tinyurl.com/y9maauwg) * https://issues.tmatesoft.com/issue/SVNKIT-740 The patch is asked to be updated and help wanted. * https://bugs.freebsd.org/235356 Help on how to reproduce and analyze is wanted. From owner-freebsd-testing@freebsd.org Sat Jun 15 09:36:02 2019 Return-Path: Delivered-To: freebsd-testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9122015C2082; Sat, 15 Jun 2019 09:36:02 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 33BBC6EE94; Sat, 15 Jun 2019 09:36:02 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id E6C14BD11; Sat, 15 Jun 2019 09:36:01 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [10.10.132.5] (vpn.bally-wulff.de [212.144.118.13]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id 840861F35C; Sat, 15 Jun 2019 11:36:00 +0200 (CEST) From: "Kristof Provost" To: "Li-Wen Hsu" Cc: freebsd-testing@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: FreeBSD CI Weekly Report 2019-06-09 Date: Sat, 15 Jun 2019 11:35:59 +0200 X-Mailer: MailMate (2.0BETAr6137) Message-ID: In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 33BBC6EE94 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.99 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.988,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Jun 2019 09:36:02 -0000 On 12 Jun 2019, at 16:49, Li-Wen Hsu wrote: > * https://ci.freebsd.org/job/FreeBSD-head-i386-test/ > * Same as amd64: > * sys.netinet.socket_afinet.socket_afinet_bind_zero > * Others: > * sys.netpfil.pf.forward.v6 > * sys.netpfil.pf.forward.v4 > * sys.netpfil.pf.set_tos.v4 I’ve finally gotten around to taking a look at this, and it appears to not be a pf problem. forward:v4 already fails at its sanity check, before it configures pf. It creates a vnet jail, telling it to route traffic through, and then we run a sanity check with pft_ping.py. Scapy tries to resolve the MAC address of the gateway (jail, 192.0.2.1). The jail replies, but scapy never picks up the reply, so the traffic looks like this: 13:19:29.953468 02:be:b4:57:9f:0a > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.0.2.2 tell 192.0.2.1, length 28 13:19:29.953572 02:be:b4:57:9f:0b > 00:a0:98:b2:48:59, ethertype ARP (0x0806), length 42: Reply 192.0.2.2 is-at 02:be:b4:57:9f:0b, length 28 13:19:32.082843 02:be:b4:57:9f:0a > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 52: 192.0.2.1 > 198.51.100.3: ICMP echo request, id 0, seq 0, length 18 The jail doesn’t forward the broadcast ICMP echo request and the test fails. My current guess is that it’s related to bpf. It’s interesting to note that it fails on i386, but succeeds on amd64. -- Kristof From owner-freebsd-testing@freebsd.org Sat Jun 15 13:34:58 2019 Return-Path: Delivered-To: freebsd-testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B33515C79AB; Sat, 15 Jun 2019 13:34:58 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 84178759F9; Sat, 15 Jun 2019 13:34:57 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 29656D936; Sat, 15 Jun 2019 13:34:57 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [172.26.30.5] (vpn.bally-wulff.de [212.144.118.13]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id BD6E31F626; Sat, 15 Jun 2019 15:34:53 +0200 (CEST) From: "Kristof Provost" To: "Li-Wen Hsu" Cc: freebsd-testing@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: FreeBSD CI Weekly Report 2019-06-09 Date: Sat, 15 Jun 2019 15:34:53 +0200 X-Mailer: MailMate (2.0BETAr6137) Message-ID: <356AE1B0-B3D7-4BE2-A052-7A1F996E3F37@FreeBSD.org> In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 84178759F9 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.978,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Jun 2019 13:34:58 -0000 On 15 Jun 2019, at 11:35, Kristof Provost wrote: > On 12 Jun 2019, at 16:49, Li-Wen Hsu wrote: >> * https://ci.freebsd.org/job/FreeBSD-head-i386-test/ >> * Same as amd64: >> * sys.netinet.socket_afinet.socket_afinet_bind_zero >> * Others: >> * sys.netpfil.pf.forward.v6 >> * sys.netpfil.pf.forward.v4 >> * sys.netpfil.pf.set_tos.v4 > > I’ve finally gotten around to taking a look at this, and it appears > to not be a pf problem. forward:v4 already fails at its sanity check, > before it configures pf. > > It creates a vnet jail, telling it to route traffic through, and then > we run a sanity check with pft_ping.py. > Scapy tries to resolve the MAC address of the gateway (jail, > 192.0.2.1). The jail replies, but scapy never picks up the reply, so > the traffic looks like this: > > 13:19:29.953468 02:be:b4:57:9f:0a > ff:ff:ff:ff:ff:ff, ethertype ARP > (0x0806), length 42: Request who-has 192.0.2.2 tell 192.0.2.1, length > 28 > 13:19:29.953572 02:be:b4:57:9f:0b > 00:a0:98:b2:48:59, ethertype ARP > (0x0806), length 42: Reply 192.0.2.2 is-at 02:be:b4:57:9f:0b, length > 28 > 13:19:32.082843 02:be:b4:57:9f:0a > ff:ff:ff:ff:ff:ff, ethertype IPv4 > (0x0800), length 52: 192.0.2.1 > 198.51.100.3: ICMP echo request, id > 0, seq 0, length 18 > > The jail doesn’t forward the broadcast ICMP echo request and the > test fails. > > My current guess is that it’s related to bpf. It’s interesting to > note that it fails on i386, but succeeds on amd64. > I’ve done a little dtracing, and I think that points at bpf too: #!/usr/sbin/dtrace -s fbt:kernel:bpf_buffer_uiomove:entry { tracemem(arg1, 1500, arg2); stack(); } Results in: 1 49539 bpf_buffer_uiomove:entry 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef 0: ce 0e 05 5d 17 ea 00 00 2a 00 00 00 2a 00 00 00 ...]....*...*... 10: 12 00 ff ff ff ff ff ff 02 fd 10 30 e6 0a 08 06 ...........0.... 20: 00 01 08 00 06 04 00 01 00 a0 98 b2 48 59 c0 00 ............HY.. 30: 02 01 00 00 00 00 00 00 c0 00 02 02 ce 0e 05 5d ...............] 40: 60 ea 00 00 2a 00 00 00 2a 00 00 00 12 00 00 a0 `...*...*....... 50: 98 b2 48 59 02 fd 10 30 e6 0b 08 06 00 01 08 00 ..HY...0........ 60: 06 04 00 02 02 fd 10 30 e6 0b c0 00 02 02 00 a0 .......0........ 70: 98 b2 48 59 c0 00 02 01 ..HY.... kernel`bpfread+0x137 kernel`dofileread+0x6d kernel`kern_readv+0x3b kernel`sys_read+0x48 kernel`syscall+0x2b4 0xffc033b7 So, we see the ARP request through bpf, but we don’t see the reply, despite tcpdump capturing it. I have no idea how that’d happen, so I’d very much like someone more familiar with bpf to take a look at this problem. Regards, Kristof