From owner-freebsd-hackers@freebsd.org Sun Nov 24 06:01:16 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4E4D41CCDE7 for ; Sun, 24 Nov 2019 06:01:16 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47LKKm1JY5z4tPf for ; Sun, 24 Nov 2019 06:01:16 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 048FE2BCCD for ; Sun, 24 Nov 2019 06:01:16 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qt1-f182.google.com with SMTP id y39so13253605qty.0 for ; Sat, 23 Nov 2019 22:01:15 -0800 (PST) X-Gm-Message-State: APjAAAULeHuTVNOU++iZ70PUrrgNAWZPtem85WyBu6MoJ2mnAujTLY/B h2so5yx2cNKQ/fE8aGaLRTnwDCj5HY3ZTGkfRTk= X-Google-Smtp-Source: APXvYqz5xlJCgA79Pln6YcGn6PYJvM/V0zet/8wftAU713tNsF2dRNVYJ4FJMUjsJvzjASnTYO5Z/0ZbDVjV4pzMMqE= X-Received: by 2002:ac8:66d7:: with SMTP id m23mr13424108qtp.53.1574575275481; Sat, 23 Nov 2019 22:01:15 -0800 (PST) MIME-Version: 1.0 From: Kyle Evans Date: Sun, 24 Nov 2019 00:01:04 -0600 X-Gmail-Original-Message-ID: Message-ID: Subject: ptrace(2) debugging To: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2019 06:01:16 -0000 Hi, I'm working on implementing `reptyr -T` on FreeBSD because I'm pretty bad about starting long-running jobs outside of tmux and often desire to reparent these jobs into tmux. I've gotten to a point where it's getting stuck in waitpid(2) when attempting to work over the session leader to ignore SIGHUP. The chain of operations looks roughly like this: PT_ATTACH -> waitpid -> kill(SIGCONT) -> PT_TO_SCE -> waitpid -> PT_TO_SCE -> waitpid Each of the waitpids are paired with a PT_LWPINFO. The first waitpid observes SIGSTOP. The second waitpid observes SIGCONT. I would expect the third to observe PL_FLAG_SCE on ptrace_lwpinfo->pl_flags, but instead it actually hangs as the target process is now sleep-inhibited and stuck in "pause" wchan. I've uploaded a truss excerpt at [0] in case it's helpful -- pid=10204 is the process I'm reparenting, initially just attached/detached to make sure reptyr *can* do this. pid=10187 is the sshd that it's running under, and pid=10188 is the shell running under that. Anyone have good advice on debugging this? It seems like it might be some kind of kernel bug, as it's already done this same dance once before when grabbing sshd and my attempts to distill it down to a simple test case failed. The FreeBSD part of reptyr needed some love, though, so that can't be discounted either. Thanks, Kyle Evans [0] https://people.freebsd.org/~kevans/truss.log From owner-freebsd-hackers@freebsd.org Sun Nov 24 11:40:11 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3AC7A1AC0A6 for ; Sun, 24 Nov 2019 11:40:11 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47LSrq00pDz3R1g; Sun, 24 Nov 2019 11:40:10 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id xAOBdudj013776 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 24 Nov 2019 13:39:59 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua xAOBdudj013776 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id xAOBdub9013775; Sun, 24 Nov 2019 13:39:56 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 24 Nov 2019 13:39:56 +0200 From: Konstantin Belousov To: Kyle Evans Cc: FreeBSD Hackers Subject: Re: ptrace(2) debugging Message-ID: <20191124113956.GY2707@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.2 (2019-09-21) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 47LSrq00pDz3R1g X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-5.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.996,0]; NEURAL_HAM_LONG(-1.00)[-0.997,0]; REPLY(-4.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2019 11:40:11 -0000 On Sun, Nov 24, 2019 at 12:01:04AM -0600, Kyle Evans wrote: > Hi, > > I'm working on implementing `reptyr -T` on FreeBSD because I'm pretty > bad about starting long-running jobs outside of tmux and often desire > to reparent these jobs into tmux. I've gotten to a point where it's > getting stuck in waitpid(2) when attempting to work over the session > leader to ignore SIGHUP. The chain of operations looks roughly like > this: > > PT_ATTACH -> waitpid -> kill(SIGCONT) -> PT_TO_SCE -> waitpid -> > PT_TO_SCE -> waitpid > > Each of the waitpids are paired with a PT_LWPINFO. The first waitpid > observes SIGSTOP. The second waitpid observes SIGCONT. I would expect > the third to observe PL_FLAG_SCE on ptrace_lwpinfo->pl_flags, but > instead it actually hangs as the target process is now sleep-inhibited > and stuck in "pause" wchan. > > I've uploaded a truss excerpt at [0] in case it's helpful -- pid=10204 > is the process I'm reparenting, initially just attached/detached to > make sure reptyr *can* do this. pid=10187 is the sshd that it's > running under, and pid=10188 is the shell running under that. > > Anyone have good advice on debugging this? It seems like it might be > some kind of kernel bug, as it's already done this same dance once > before when grabbing sshd and my attempts to distill it down to a > simple test case failed. The FreeBSD part of reptyr needed some love, > though, so that can't be discounted either. > > Thanks, > > Kyle Evans > > [0] https://people.freebsd.org/~kevans/truss.log How much work would be to provide a self-contained standalone test ? From owner-freebsd-hackers@freebsd.org Sun Nov 24 15:17:17 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9809D1B1B4F for ; Sun, 24 Nov 2019 15:17:17 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47LYgK3YjMz46wR for ; Sun, 24 Nov 2019 15:17:17 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from mail-qk1-f182.google.com (mail-qk1-f182.google.com [209.85.222.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 4FD3A2FC66 for ; Sun, 24 Nov 2019 15:17:17 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qk1-f182.google.com with SMTP id e187so10550189qkf.4 for ; Sun, 24 Nov 2019 07:17:17 -0800 (PST) X-Gm-Message-State: APjAAAVU2peYmblh87EYRNPHWKA/fFH9GTTg73xGX39npHQILaowaPZO j/4/gCX2WFw0cBG0tiuOBUxnNt7ruTxMUuu6rUI= X-Google-Smtp-Source: APXvYqynsS1GtuxhczZQ3INyrCn3YiSUVzgsugSwbvQynznXzhvrv4pNXWqyw6VDuiPWZvQy8GnTOz8nMKnk1m+v42s= X-Received: by 2002:a37:7305:: with SMTP id o5mr14857556qkc.120.1574608636757; Sun, 24 Nov 2019 07:17:16 -0800 (PST) MIME-Version: 1.0 References: <20191124113956.GY2707@kib.kiev.ua> In-Reply-To: <20191124113956.GY2707@kib.kiev.ua> From: Kyle Evans Date: Sun, 24 Nov 2019 09:17:05 -0600 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: ptrace(2) debugging To: Konstantin Belousov Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2019 15:17:17 -0000 On Sun, Nov 24, 2019 at 5:40 AM Konstantin Belousov wrote: > > On Sun, Nov 24, 2019 at 12:01:04AM -0600, Kyle Evans wrote: > > Hi, > > > > I'm working on implementing `reptyr -T` on FreeBSD because I'm pretty > > bad about starting long-running jobs outside of tmux and often desire > > to reparent these jobs into tmux. I've gotten to a point where it's > > getting stuck in waitpid(2) when attempting to work over the session > > leader to ignore SIGHUP. The chain of operations looks roughly like > > this: > > > > PT_ATTACH -> waitpid -> kill(SIGCONT) -> PT_TO_SCE -> waitpid -> > > PT_TO_SCE -> waitpid > > > > Each of the waitpids are paired with a PT_LWPINFO. The first waitpid > > observes SIGSTOP. The second waitpid observes SIGCONT. I would expect > > the third to observe PL_FLAG_SCE on ptrace_lwpinfo->pl_flags, but > > instead it actually hangs as the target process is now sleep-inhibited > > and stuck in "pause" wchan. > > > > I've uploaded a truss excerpt at [0] in case it's helpful -- pid=10204 > > is the process I'm reparenting, initially just attached/detached to > > make sure reptyr *can* do this. pid=10187 is the sshd that it's > > running under, and pid=10188 is the shell running under that. > > > > Anyone have good advice on debugging this? It seems like it might be > > some kind of kernel bug, as it's already done this same dance once > > before when grabbing sshd and my attempts to distill it down to a > > simple test case failed. The FreeBSD part of reptyr needed some love, > > though, so that can't be discounted either. > > > > Thanks, > > > > Kyle Evans > > > > [0] https://people.freebsd.org/~kevans/truss.log > How much work would be to provide a self-contained standalone test ? I'm still struggling to write a self-contained example... unfortunately a basic attach and trace them all to syscall entry isn't sufficient. I'm slowly removing surface area from reptyr to try and narrow it down- its operations between attaching to sshd and the misbehavior are quite extensive, as it mmaps a page into the target, opens a socket established by reptyr and passes an fd back over it. From owner-freebsd-hackers@freebsd.org Mon Nov 25 02:26:09 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 33BCC1C4A2D for ; Mon, 25 Nov 2019 02:26:09 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47LrW50fGxz3KD6 for ; Mon, 25 Nov 2019 02:26:09 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id DC6044DB3 for ; Mon, 25 Nov 2019 02:26:08 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qt1-f181.google.com with SMTP id y10so15503373qto.3 for ; Sun, 24 Nov 2019 18:26:08 -0800 (PST) X-Gm-Message-State: APjAAAVaFkvTSlErqLsK3ZwjdUq0lGNNa/72qlZUjb6S5u403EjdurSd MVdAQkWWDJcel0hgQepcbDxyHxBerrLvPnaVXxI= X-Google-Smtp-Source: APXvYqw92oZ2hzaM0Yd+Nz00j/Wn7H/LyLRhxzRf6lqn0xRaKmE5vFSrXn3j7zcjy5FMoDIM6VhDbQl/EugyejXvRlY= X-Received: by 2002:ac8:41cc:: with SMTP id o12mr26198457qtm.310.1574648768219; Sun, 24 Nov 2019 18:26:08 -0800 (PST) MIME-Version: 1.0 References: <20191124113956.GY2707@kib.kiev.ua> In-Reply-To: From: Kyle Evans Date: Sun, 24 Nov 2019 20:25:56 -0600 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: ptrace(2) debugging To: Konstantin Belousov Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Nov 2019 02:26:09 -0000 On Sun, Nov 24, 2019 at 9:17 AM Kyle Evans wrote: > > On Sun, Nov 24, 2019 at 5:40 AM Konstantin Belousov wrote: > > > > On Sun, Nov 24, 2019 at 12:01:04AM -0600, Kyle Evans wrote: > > > Hi, > > > > > > I'm working on implementing `reptyr -T` on FreeBSD because I'm pretty > > > bad about starting long-running jobs outside of tmux and often desire > > > to reparent these jobs into tmux. I've gotten to a point where it's > > > getting stuck in waitpid(2) when attempting to work over the session > > > leader to ignore SIGHUP. The chain of operations looks roughly like > > > this: > > > > > > PT_ATTACH -> waitpid -> kill(SIGCONT) -> PT_TO_SCE -> waitpid -> > > > PT_TO_SCE -> waitpid > > > > > > Each of the waitpids are paired with a PT_LWPINFO. The first waitpid > > > observes SIGSTOP. The second waitpid observes SIGCONT. I would expect > > > the third to observe PL_FLAG_SCE on ptrace_lwpinfo->pl_flags, but > > > instead it actually hangs as the target process is now sleep-inhibited > > > and stuck in "pause" wchan. > > > > > > I've uploaded a truss excerpt at [0] in case it's helpful -- pid=10204 > > > is the process I'm reparenting, initially just attached/detached to > > > make sure reptyr *can* do this. pid=10187 is the sshd that it's > > > running under, and pid=10188 is the shell running under that. > > > > > > Anyone have good advice on debugging this? It seems like it might be > > > some kind of kernel bug, as it's already done this same dance once > > > before when grabbing sshd and my attempts to distill it down to a > > > simple test case failed. The FreeBSD part of reptyr needed some love, > > > though, so that can't be discounted either. > > > > > > Thanks, > > > > > > Kyle Evans > > > > > > [0] https://people.freebsd.org/~kevans/truss.log > > How much work would be to provide a self-contained standalone test ? > > I'm still struggling to write a self-contained example... > unfortunately a basic attach and trace them all to syscall entry isn't > sufficient. I'm slowly removing surface area from reptyr to try and > narrow it down- its operations between attaching to sshd and the > misbehavior are quite extensive, as it mmaps a page into the target, > opens a socket established by reptyr and passes an fd back over it. I managed to narrow it down, kind of. The problem is specifically with trying to trace zsh as a session leader. Easiest reproducer is to change shell to zsh and run this: https://people.freebsd.org/~kevans/ptrace_test.c -> you'll hang and have to ^C that sucker. My experiments showed that running this on zsh spawned any other way is fine, and changing shell to /bin/sh is also fine. Thanks, Kyle Evans From owner-freebsd-hackers@freebsd.org Mon Nov 25 03:26:52 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D32E61C64C2 for ; Mon, 25 Nov 2019 03:26:52 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Lss85H7Pz3N19 for ; Mon, 25 Nov 2019 03:26:52 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 8CDA6549B for ; Mon, 25 Nov 2019 03:26:52 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qk1-f169.google.com with SMTP id z65so6701941qka.6 for ; Sun, 24 Nov 2019 19:26:52 -0800 (PST) X-Gm-Message-State: APjAAAVqelOQlJJil7j2sIOUDKxO/SKCu8DE7C7Jy+jfwSnIquYBef0T EWL+wiirDvfnSR6RN52d/StbtroH8C1Qplrtvn4= X-Received: by 2002:a37:a00f:: with SMTP id j15mt21978937qke.103.1574652411853; Sun, 24 Nov 2019 19:26:51 -0800 (PST) MIME-Version: 1.0 References: <20191124113956.GY2707@kib.kiev.ua> In-Reply-To: From: Kyle Evans Date: Sun, 24 Nov 2019 21:26:40 -0600 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: ptrace(2) debugging Cc: Konstantin Belousov , FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Nov 2019 03:26:52 -0000 On Sun, Nov 24, 2019 at 8:25 PM Kyle Evans wrote: > > On Sun, Nov 24, 2019 at 9:17 AM Kyle Evans wrote: > > > > On Sun, Nov 24, 2019 at 5:40 AM Konstantin Belousov wrote: > > > > > > On Sun, Nov 24, 2019 at 12:01:04AM -0600, Kyle Evans wrote: > > > > Hi, > > > > > > > > I'm working on implementing `reptyr -T` on FreeBSD because I'm pretty > > > > bad about starting long-running jobs outside of tmux and often desire > > > > to reparent these jobs into tmux. I've gotten to a point where it's > > > > getting stuck in waitpid(2) when attempting to work over the session > > > > leader to ignore SIGHUP. The chain of operations looks roughly like > > > > this: > > > > > > > > PT_ATTACH -> waitpid -> kill(SIGCONT) -> PT_TO_SCE -> waitpid -> > > > > PT_TO_SCE -> waitpid > > > > > > > > Each of the waitpids are paired with a PT_LWPINFO. The first waitpid > > > > observes SIGSTOP. The second waitpid observes SIGCONT. I would expect > > > > the third to observe PL_FLAG_SCE on ptrace_lwpinfo->pl_flags, but > > > > instead it actually hangs as the target process is now sleep-inhibited > > > > and stuck in "pause" wchan. > > > > > > > > I've uploaded a truss excerpt at [0] in case it's helpful -- pid=10204 > > > > is the process I'm reparenting, initially just attached/detached to > > > > make sure reptyr *can* do this. pid=10187 is the sshd that it's > > > > running under, and pid=10188 is the shell running under that. > > > > > > > > Anyone have good advice on debugging this? It seems like it might be > > > > some kind of kernel bug, as it's already done this same dance once > > > > before when grabbing sshd and my attempts to distill it down to a > > > > simple test case failed. The FreeBSD part of reptyr needed some love, > > > > though, so that can't be discounted either. > > > > > > > > Thanks, > > > > > > > > Kyle Evans > > > > > > > > [0] https://people.freebsd.org/~kevans/truss.log > > > How much work would be to provide a self-contained standalone test ? > > > > I'm still struggling to write a self-contained example... > > unfortunately a basic attach and trace them all to syscall entry isn't > > sufficient. I'm slowly removing surface area from reptyr to try and > > narrow it down- its operations between attaching to sshd and the > > misbehavior are quite extensive, as it mmaps a page into the target, > > opens a socket established by reptyr and passes an fd back over it. > > I managed to narrow it down, kind of. The problem is specifically with > trying to trace zsh as a session leader. Easiest reproducer is to > change shell to zsh and run this: > https://people.freebsd.org/~kevans/ptrace_test.c -> you'll hang and > have to ^C that sucker. My experiments showed that running this on zsh > spawned any other way is fine, and changing shell to /bin/sh is also > fine. > Follow up part three, zsh is in sigsuspend() while a child is executing and this is the cause. More effective reproducer: https://people.freebsd.org/~kevans/ptrace_test2.c -> the behavior makes a little more sense to me, but that seems less than ideal. From owner-freebsd-hackers@freebsd.org Mon Nov 25 14:45:55 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 90DBC1AF024 for ; Mon, 25 Nov 2019 14:45:55 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47M8wg1kgSz4VkS; Mon, 25 Nov 2019 14:45:54 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id xAPEjkdk002853 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 25 Nov 2019 16:45:50 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua xAPEjkdk002853 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id xAPEjkY8002852; Mon, 25 Nov 2019 16:45:46 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 25 Nov 2019 16:45:46 +0200 From: Konstantin Belousov To: Kyle Evans Cc: FreeBSD Hackers Subject: Re: ptrace(2) debugging Message-ID: <20191125144546.GD10580@kib.kiev.ua> References: <20191124113956.GY2707@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.2 (2019-09-21) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 47M8wg1kgSz4VkS X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-5.99 / 15.00]; NEURAL_HAM_MEDIUM(-0.99)[-0.995,0]; NEURAL_HAM_LONG(-1.00)[-0.997,0]; REPLY(-4.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Nov 2019 14:45:55 -0000 On Sun, Nov 24, 2019 at 09:26:40PM -0600, Kyle Evans wrote: > On Sun, Nov 24, 2019 at 8:25 PM Kyle Evans wrote: > > > > On Sun, Nov 24, 2019 at 9:17 AM Kyle Evans wrote: > > > > > > On Sun, Nov 24, 2019 at 5:40 AM Konstantin Belousov wrote: > > > > > > > > On Sun, Nov 24, 2019 at 12:01:04AM -0600, Kyle Evans wrote: > > > > > Hi, > > > > > > > > > > I'm working on implementing `reptyr -T` on FreeBSD because I'm pretty > > > > > bad about starting long-running jobs outside of tmux and often desire > > > > > to reparent these jobs into tmux. I've gotten to a point where it's > > > > > getting stuck in waitpid(2) when attempting to work over the session > > > > > leader to ignore SIGHUP. The chain of operations looks roughly like > > > > > this: > > > > > > > > > > PT_ATTACH -> waitpid -> kill(SIGCONT) -> PT_TO_SCE -> waitpid -> > > > > > PT_TO_SCE -> waitpid > > > > > > > > > > Each of the waitpids are paired with a PT_LWPINFO. The first waitpid > > > > > observes SIGSTOP. The second waitpid observes SIGCONT. I would expect > > > > > the third to observe PL_FLAG_SCE on ptrace_lwpinfo->pl_flags, but > > > > > instead it actually hangs as the target process is now sleep-inhibited > > > > > and stuck in "pause" wchan. > > > > > > > > > > I've uploaded a truss excerpt at [0] in case it's helpful -- pid=10204 > > > > > is the process I'm reparenting, initially just attached/detached to > > > > > make sure reptyr *can* do this. pid=10187 is the sshd that it's > > > > > running under, and pid=10188 is the shell running under that. > > > > > > > > > > Anyone have good advice on debugging this? It seems like it might be > > > > > some kind of kernel bug, as it's already done this same dance once > > > > > before when grabbing sshd and my attempts to distill it down to a > > > > > simple test case failed. The FreeBSD part of reptyr needed some love, > > > > > though, so that can't be discounted either. > > > > > > > > > > Thanks, > > > > > > > > > > Kyle Evans > > > > > > > > > > [0] https://people.freebsd.org/~kevans/truss.log > > > > How much work would be to provide a self-contained standalone test ? > > > > > > I'm still struggling to write a self-contained example... > > > unfortunately a basic attach and trace them all to syscall entry isn't > > > sufficient. I'm slowly removing surface area from reptyr to try and > > > narrow it down- its operations between attaching to sshd and the > > > misbehavior are quite extensive, as it mmaps a page into the target, > > > opens a socket established by reptyr and passes an fd back over it. > > > > I managed to narrow it down, kind of. The problem is specifically with > > trying to trace zsh as a session leader. Easiest reproducer is to > > change shell to zsh and run this: > > https://people.freebsd.org/~kevans/ptrace_test.c -> you'll hang and > > have to ^C that sucker. My experiments showed that running this on zsh > > spawned any other way is fine, and changing shell to /bin/sh is also > > fine. > > > > Follow up part three, zsh is in sigsuspend() while a child is > executing and this is the cause. More effective reproducer: > https://people.freebsd.org/~kevans/ptrace_test2.c -> the behavior > makes a little more sense to me, but that seems less than ideal. It is still not quite a reproducer. Can you modify the test to clearly indicate what you want vs. what you get ? I see the legitimate loop of the parent (debugger) doing 59326 ptrace_test2 CALL ptrace(PT_ATTACH,0xe7bf,0x1,0) 59326 ptrace_test2 RET ptrace 0 59326 ptrace_test2 CALL wait4(0xe7bf,0x7fffffffe3bc,0,0) 59326 ptrace_test2 RET wait4 59327/0xe7bf 59326 ptrace_test2 CALL ptrace(PT_LWPINFO,0xe7bf,0x7fffffffe3c0,0xa0) 59326 ptrace_test2 RET ptrace 0 59326 ptrace_test2 CALL ptrace(PT_FOLLOW_FORK,0xe7bf,0,0x1) 59326 ptrace_test2 RET ptrace 0 59326 ptrace_test2 CALL ptrace(PT_GETREGS,0xe7bf,0x7fffffffe300,0) 59326 ptrace_test2 RET ptrace 0 59326 ptrace_test2 CALL kill(0xe7bf,SIGCONT) 59326 ptrace_test2 RET kill 0 59326 ptrace_test2 CALL ptrace(PT_TO_SCE,0xe7bf,0x1,0) 59326 ptrace_test2 RET ptrace 0 59326 ptrace_test2 CALL wait4(0xe7bf,0x7fffffffe25c,0,0) 59326 ptrace_test2 RET wait4 59327/0xe7bf 59326 ptrace_test2 CALL ptrace(PT_LWPINFO,0xe7bf,0x7fffffffe260,0xa0) 59326 ptrace_test2 RET ptrace 0 59326 ptrace_test2 CALL ptrace(PT_DETACH,0xe7bf,0x1,0) 59326 ptrace_test2 RET ptrace 0 in a loop, and I do not see anything wrong with it. From owner-freebsd-hackers@freebsd.org Mon Nov 25 18:00:38 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 099861B487B for ; Mon, 25 Nov 2019 18:00:38 +0000 (UTC) (envelope-from joel.bertrand@systella.fr) Received: from rayleigh.systella.fr (newton-ipv6.systella.fr [IPv6:2001:7a8:a8ed:253::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "*.systella.fr", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47MFFJ3D3kz3Dfd for ; Mon, 25 Nov 2019 18:00:35 +0000 (UTC) (envelope-from joel.bertrand@systella.fr) Received: from [192.168.10.103] (hilbert.systella.fr [192.168.10.103]) (authenticated bits=0) by rayleigh.systella.fr (8.15.2/8.15.2/Debian-15) with ESMTPSA id xAPI0CZ3012929 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 25 Nov 2019 19:00:13 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=systella.fr; s=mail; t=1574704827; bh=He1t5kUId077NMUsRECZU2NJkzMbMywm946JvvTh5ZM=; h=To:From:Subject:Date:From; b=mfH/jLrMIMRTxpzLlqe31YmTVg9eP+vQZnDsDWC4r545IZl8+iyn6ilVi7g7kkCLj QBjkbMGpLxf7YnrTIKqmmYNlYkBwmE6Tk1ZNKVVgk7CXSouFsIutPf+YbaDM7q4B16 ztGCX7+0I/okDt2FPOCtEVxK4VHo2fR6GWYGxyDmHo5LXmnL2rQ6TGkzn/NPNONhH5 HxADcWYB/85yLjub3dHT3WXGQWvOr3GYYJsY748BNmNC92dpw2Btcowm3PXL6Ejyhe s/xk1By/qlK521Ul2b5XVdMbJAFNmjJzBxokYgJ+1zcWP3PIXzVB8yAo8wmkg/pbwv /mgwTDVHn2/BQ== To: freebsd-hackers@freebsd.org From: =?UTF-8?Q?BERTRAND_Jo=c3=abl?= Subject: Segfault on graphical console Openpgp: preference=signencrypt Autocrypt: addr=joel.bertrand@systella.fr; prefer-encrypt=mutual; keydata= mQINBFphqV8BEAC+ye6YQBdlEn9p/mbZhiQLkZQjIbGL84M0XOd66AgWqJ3T2TiwEyiExQyM Of0VswmB1q6SaIyh0x4bzaRyKwJLWDJy+AMGLGT2cpmsrDgllUxiBv3xAoLpwR8yDuLs5+bT vPpu81Pm/nzO2NDl85+3WAQbW+bUDAUdBhkg7X07bjJePypRxoRh4LF/syalOjzPEFARFNBY VrXFCTS/F7ZwmUHLv2xWJpEyKHfsC4BSo4ZPjrKmPJBxBPxuR+KiSYG/TkjU6CzoFvdwRY33 GNrK+dUmt9/VnPC/l1rGWS3durgah4OEkxciesKiTvQBUzVfMY0dvzBQKJeNNMFLstnjq3NP qvo3g5DnhFYFSAjI7wzqLkcYO8qg01wrWYyY/NLfAY6CvK0VjlenlKu84ePQDu7zh9/DUzBs 75ZAP2vZv4o00B2A3ksbkXSIs9eSDDx19OS1EUkSqw1VtFRfupoMkK7I7zrGR+DBENl5oK09 SOYJw594XVAoPpZ5zVUlEBDpatBNICTTT17EHrVpEB222TVfChvoE0cwYGkS40nVRIhQ1Yo3 A8qeKb2EeeCtslgiNcb1ajeZOSGUSHnczWHTaX5jMB/evNxZpLJH1WX8PqZ/+7wVRYuZGZ/b r8V3wXjZVNzPSTONwq3w/VjizPcQqdwicoNtxvuB6hM1J1tLGQARAQABtCpCRVJUUkFORCBK b8OrbCA8am9lbC5iZXJ0cmFuZEBzeXN0ZWxsYS5mcj6JAlQEEwEKAD4WIQSrhgKgAkzAsSVl Vhc4B+jSUpDz5wUCWmGpXwIbIwUJCWYBgAULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRA4 B+jSUpDz5xwED/462ki+I97keYMSDPVjzx9MmcE/pznOqv8b4IbFbWzaSCx5J3BByBVU1IyP LdDCdZgvc7vM5l8Gc6mqxeABbgdyxBGMwoXBBADXt9dcAKW3xl6URMLoor8DwgnuluCdQT+K 7VW6ron23x7wI9iscuXV2M6xG2G84o5kDgW2NrkKBIwfWqS/XATNrC8e9j71h29cv1RvKN2a 3XQMI6kvBRirb9zM8jWaP1K/QCLZpvhuiXCJwJvl+MGTuOUTxvp44MjVaM++Wfljc9NAVyD8 s3UxBTjhei1zIHiLUMANzeFLnW21UnoCLLoqzD558VC2Gyqk9I9kaZ/jgQqEu6drbJG+6LbY zbiYt1OhURCrMh7zXjPbBCF1rjtjhEZx2EmT2U7KWQvgcW75JBEGCRveTXQga8ytBTcNoEC8 e7ZjM0ob769ZQ1HmeWAOJy6xKEnv1A2P3erQ3xTZEHFVesoruhMAzrf2fdWJ9vGvndMt7sV/ G0NabDwlI+eIZ77vEUCdFiCZuE8vk0BglwXHVH5zjgJNVLNgNFSs3uTSf5zOIqLXyQTOd0px 5jNe2mePxdMuI9MuQWXE8Z7InUaU1WZ+95eTMonTs+DRUJzQ+XWYbqllGudx230T/Y6cYxSW stw5bAQl1aNhNCZjHutNUigb8row2cH2kCVJexv4LYUx8vdc2bkCDQRaYalfARAA000pOiAt CMcbNPczj/sFU5UZ6zaEOPj08nNv48UZAfo8ZiWejSp7YbaU4HW0VxcT2ZQvhsHor2wjqI5K Ii7gmGyjMA6NJPhHVoJw8+j64m38mRcOzlSaQEZV+Pp+TAX8PyucZvNPXHy40UQfDqCoYxAw A0bSWwcSwH+N/eEijrEOf7k1QRjEodjGTxaE50XOWXz/NMVx7l9ORcd5r2oq2DLjqnnQzl6k XxmxSV5cm+HDIojLmQz1qxq7r2GhC5hGuR5KXrO/p4bNiqCTw+rwm4bO6YjTfaH+eNRwzg6L OpW7ZNw0pfWf4wO/h+ozZTY5q9EbZSmZyVoSyPu2J1+mX3gF2ZLSzZ7+XgV4z6EIAcF1sjGE hsqA9yF4NVHGlrP9dkhZFoBVtkJgjSfdSWGXp40X+pUROeVuexD6zDadB5rCv05o9/EPDFu0 W7mJ+l8h3rGI0E5ObmR6+HyFGqalByGFxYbia+x51Kj6WbHNMuddkPk0YbHs53zS9VGnRcnh xTGdga2rMzO9ZycKo7DPrdZVi7bZWKM/WM0IL5m6FTPSJteJ886NP9oc8U6o2GxZ1cMeZwnu AprT77mISL0Z1CCcoSDZEnD+EmOcKxYnkxJuhMY2kdMd2x47I2HFxTa1ix6UQ7OY6i0VQ4gG WZ3tgiHYIHbeAyZXn0P4nM5ofgEAEQEAAYkCPAQYAQoAJhYhBKuGAqACTMCxJWVWFzgH6NJS kPPnBQJaYalfAhsMBQkJZgGAAAoJEDgH6NJSkPPn9mEP/2mEFZGT0AaecRRXfpfrVnxxIwK8 YK3mmaa8jqSLXzDgubTO6PWojVt/SCrvhCtEf5vxATPbeFz5Ho0foI/iGys9SQkYmsIn13m0 z9oY8LBIyrPFud56RrIqYBno3qR6N7SWBWtZeUw+gc6HYbMyk2L7//wz4DkneJYLLcTfMxb/ +Ok0tNmWp6YfuyRBt5yHU6VfW4tZxF5qzg+9niW3VbB6BEZbM+ya7qBZD8su4e1EfUjaKb2z Egyw09laSgBW4NzGBwlhX3zeDsNTiReqa78e1pv52Ur3dI5GH4psLw4rH7ghh/aA/eVDMcKn LPNeTNl+Sz/1PK+oVNxz6tGBVsTVbZpwbanv6+qQP3yPvX0MS1x5QQPp/SAsjJv6lpFkoUGK n2clMYH8pSefR7jp0UPCrMBeFv5bom8tNMTHrIQrpnWo7vXUmeJ7OP/lHUtbBB26882pYbpa 05D79xUkBMYX2BGvtM687+CZaWwA4u/Tl7cu3PpIavPWgmmvIBJOwsDKxNgopLkix8AGFbfs wPcE7d03t9tdauheHA8pssNQmy3scoThC3DQc4eIEBUU5M9uIW2HARl3OgJP9h9OePsgaT8g zTlN3q6QmDWQwRmxF6J7Z9dtIDmXr+014XtK7UCzxkBIFS5jPGzL8dSKDu5jIx8cboy9QUeH Tr6FXnLg Message-ID: <49d29b45-953b-a33a-66ce-e6da60a4e415@systella.fr> Date: Mon, 25 Nov 2019 19:00:12 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.5 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.101.4 at rayleigh X-Virus-Status: Clean X-Rspamd-Queue-Id: 47MFFJ3D3kz3Dfd X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=systella.fr header.s=mail header.b=mfH/jLrM; dmarc=none; spf=pass (mx1.freebsd.org: domain of joel.bertrand@systella.fr designates 2001:7a8:a8ed:253::1 as permitted sender) smtp.mailfrom=joel.bertrand@systella.fr X-Spamd-Result: default: False [-0.37 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[systella.fr:s=mail]; NEURAL_HAM_MEDIUM(-0.36)[-0.364,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2001:7a8:a8ed:253::1/64]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.28)[-0.282,0]; DMARC_NA(0.00)[systella.fr]; DKIM_TRACE(0.00)[systella.fr:+]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(0.77)[asn: 13193(3.87), country: FR(-0.00)]; ASN(0.00)[asn:13193, ipnet:2001:7a8::/32, country:FR]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Nov 2019 18:00:38 -0000 Hello, For a long time, I have seen that some programs abort (for example vlc or okular) with segfault when I try to launch them in a regular xterm. But if I do 'ssh -Y localhost', the same program run as expected ! I don't find any explanation. I suppose this bug is X related. My Xorg.0.log doesn't contain error and I use Intel graphic adapter (i5-4th generaion). In dmesg, I have : root@pythagore:/var/log # dmesg vm_fault: pager read error, pid 6909 (vlc) pid 6909 (vlc), uid 1001: exited on signal 11 Help will be welcome. Best regards, JB From owner-freebsd-hackers@freebsd.org Tue Nov 26 07:00:56 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 612591AB9A5 for ; Tue, 26 Nov 2019 07:00:56 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47MZYg07lKz4St9 for ; Tue, 26 Nov 2019 07:00:54 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id xAQ70oNx005403 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 26 Nov 2019 08:00:51 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1574751651; bh=BWiAh8j3si8VwipSSraJfkAGzh5QDW2oBZIrCTZF0RQ=; h=Date:From:To:Subject; b=UUpkbZg5m0LWtINVcyNNMcgbpk9ZRoNbNbFwNEgs2Hup60eBWN6TmBzMEh58/SIjJ 9yaFP1/zv7S94/YaBX1f9nnzNSohZa31rl9GxHRPlFREE7y+/ApoBL273fj7RF/A4L ZFwS6w4RJZ3aJbA9s1CCoeiTxJpwFwLytz4bBrZs= Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id xAQ70o0r005400 for ; Tue, 26 Nov 2019 08:00:50 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Date: Tue, 26 Nov 2019 08:00:50 +0100 (CET) From: Wojciech Puchar To: freebsd-hackers@freebsd.org Subject: what's got broken in bhyve after upgrade Message-ID: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-Rspamd-Queue-Id: 47MZYg07lKz4St9 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=fail (rsa verify failed) header.d=puchar.net header.s=default header.b=UUpkbZg5; dmarc=none; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-4.71 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_REJECT(1.00)[puchar.net:s=default]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; DMARC_NA(0.00)[puchar.net]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[puchar.net:-]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-3.41)[ip: (-9.01), ipnet: 194.1.144.0/24(-4.50), asn: 43476(-3.60), country: PL(0.07)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2019 07:00:56 -0000 upgraded to FreeBSD puchar.net 11.3-STABLE FreeBSD 11.3-STABLE #0 r354807: Fri Nov 22 22:45:43 CET 2019 root@puchar.net:/usr/src/sys/amd64/compile/puchar amd64 i have setting hw.vmm.topology.cores_per_package=16 in loader.conf before upgrade - bhyve virtual machines (windows 7) see one CPU with 2 or 4 cores (depending how much i use with -c option in bhyve). after upgrade - bhyve sees single core CPUs. Which in case of windows 7 pro means no more than 2 cores will be used. What's wrong and how to fix it? From owner-freebsd-hackers@freebsd.org Tue Nov 26 09:44:47 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D83081B02CF for ; Tue, 26 Nov 2019 09:44:47 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47MfBk4x1bz4cFG for ; Tue, 26 Nov 2019 09:44:46 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id xAQ9ieqs072518 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 26 Nov 2019 10:44:40 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1574761481; bh=0jZTHgEav3OEEHfmZpV9E4YySw2UdfVBlxW0YyEypgo=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=E+wckRdIsrIIWJ7qDgVXcP0g700Znk9tCdhY+OIpDPfD+86Mch2knwZj+c9TvSryP vOPBp6U3SN2BVuJnrIglO1T60GqAo7jD2doUci0r2TQr7G9US05WcfrD75ZFQEfT6c 5X9t7aVjsU/KwrwZ9Mkir22msU81e4l3lnWOos+Q= Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id xAQ9ieWp072515; Tue, 26 Nov 2019 10:44:40 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Date: Tue, 26 Nov 2019 10:44:40 +0100 (CET) From: Wojciech Puchar To: Wojciech Puchar cc: freebsd-hackers@freebsd.org Subject: Nothing is broken, but changed WAS: what's got broken in bhyve after upgrade In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 47MfBk4x1bz4cFG X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=fail (rsa verify failed) header.d=puchar.net header.s=default header.b=E+wckRdI; dmarc=none; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-4.70 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx:c]; R_DKIM_REJECT(1.00)[puchar.net:s=default]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[puchar.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[puchar.net:-]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-3.40)[ip: (-8.98), ipnet: 194.1.144.0/24(-4.49), asn: 43476(-3.59), country: PL(0.07)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2019 09:44:47 -0000 New bhyve gets sockets, cores and threads parameters from command line. fixed now. Question - is 16 core limit in bhyve a hard one because of something, or it can be changed? On Tue, 26 Nov 2019, Wojciech Puchar wrote: > upgraded to > > FreeBSD puchar.net 11.3-STABLE FreeBSD 11.3-STABLE #0 r354807: Fri Nov 22 > 22:45:43 CET 2019 root@puchar.net:/usr/src/sys/amd64/compile/puchar amd64 > > > i have setting hw.vmm.topology.cores_per_package=16 in loader.conf > > > before upgrade - bhyve virtual machines (windows 7) see one CPU with 2 or 4 > cores (depending how much i use with -c option in bhyve). > > after upgrade - bhyve sees single core CPUs. Which in case of windows 7 pro > means no more than 2 cores will be used. > > What's wrong and how to fix it? > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > From owner-freebsd-hackers@freebsd.org Tue Nov 26 10:29:22 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4324C1B15A1 for ; Tue, 26 Nov 2019 10:29:22 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47MgB86SpBz4fFH for ; Tue, 26 Nov 2019 10:29:20 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id xAQAPF2t047107 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 26 Nov 2019 10:25:16 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: wojtek@puchar.net Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id xAQAPBHq009596 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 26 Nov 2019 17:25:11 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Nothing is broken, but changed WAS: what's got broken in bhyve after upgrade To: Wojciech Puchar References: Cc: freebsd-hackers@freebsd.org From: Eugene Grosbein Message-ID: <0d4379a6-d71d-a2b1-35e3-2dec23df5df4@grosbein.net> Date: Tue, 26 Nov 2019 17:25:03 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Spam-Flag: YES X-Spam-Status: Yes, score=6.0 required=5.0 tests=BAYES_00, DATE_IN_FUTURE_96_Q, HELO_MISC_IP,LOCAL_FROM,RDNS_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.7 DATE_IN_FUTURE_96_Q Date: is 4 days to 4 months after Received: * date * -0.0 SPF_PASS SPF: sender matches SPF record * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains * 1.1 HELO_MISC_IP Looking for more Dynamic IP Relays X-Spam-Level: ***** X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on eg.sd.rdtc.ru X-Rspamd-Queue-Id: 47MgB86SpBz4fFH X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [1.40 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.86)[-0.857,0]; SPAM_FLAG(5.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.98)[-0.983,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-1.66)[ip: (-4.37), ipnet: 2a01:4f8::/29(-2.33), asn: 24940(-1.57), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2019 10:29:22 -0000 26.11.2019 16:44, Wojciech Puchar wrote: > New bhyve gets sockets, cores and threads parameters from command line. fixed now. I observer significant memory leaks in bhyve processes after upgrade from 11.2 to 11.3-STABLE, do you? Way over limits for guests. From owner-freebsd-hackers@freebsd.org Tue Nov 26 10:59:24 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B17DF1B1F0C for ; Tue, 26 Nov 2019 10:59:24 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47Mgrq1MXyz3Bqx for ; Tue, 26 Nov 2019 10:59:22 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id xAQAxJcY002278; Tue, 26 Nov 2019 02:59:19 -0800 (PST) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id xAQAxJTE002277; Tue, 26 Nov 2019 02:59:19 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201911261059.xAQAxJTE002277@gndrsh.dnsmgr.net> Subject: Re: Nothing is broken, but changed WAS: what's got broken in bhyve after upgrade In-Reply-To: To: Wojciech Puchar Date: Tue, 26 Nov 2019 02:59:19 -0800 (PST) CC: freebsd-hackers@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 47Mgrq1MXyz3Bqx X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd-rwg@gndrsh.dnsmgr.net has no SPF policy when checking 69.59.192.140) smtp.mailfrom=freebsd-rwg@gndrsh.dnsmgr.net X-Spamd-Result: default: False [1.96 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.50)[0.500,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.52)[0.522,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.04)[ip: (0.14), ipnet: 69.59.192.0/19(0.07), asn: 13868(0.03), country: US(-0.05)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2019 10:59:24 -0000 > New bhyve gets sockets, cores and threads parameters from command line. > fixed now. > > Question - is 16 core limit in bhyve a hard one because of something, or > it can be changed? Yes, modify this line: #define VM_MAXCPU 16 /* maximum virtual cpus */ in ./amd64/include/vmm.h You'll need to rebuild vmm.,ko, and anything else that includes vmm.h > On Tue, 26 Nov 2019, Wojciech Puchar wrote: > > > upgraded to > > > > FreeBSD puchar.net 11.3-STABLE FreeBSD 11.3-STABLE #0 r354807: Fri Nov 22 > > 22:45:43 CET 2019 root@puchar.net:/usr/src/sys/amd64/compile/puchar amd64 > > > > > > i have setting hw.vmm.topology.cores_per_package=16 in loader.conf > > > > > > before upgrade - bhyve virtual machines (windows 7) see one CPU with 2 or 4 > > cores (depending how much i use with -c option in bhyve). > > > > after upgrade - bhyve sees single core CPUs. Which in case of windows 7 pro > > means no more than 2 cores will be used. > > > > What's wrong and how to fix it? > > > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-hackers@freebsd.org Tue Nov 26 13:44:12 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 069401B6794 for ; Tue, 26 Nov 2019 13:44:12 +0000 (UTC) (envelope-from salvadore@FreeBSD.org) Received: from MailHost (unknown [80.211.33.142]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47MlVz1CfJz3LDM for ; Tue, 26 Nov 2019 13:44:11 +0000 (UTC) (envelope-from salvadore@FreeBSD.org) Received: from root (uid 0) (envelope-from salvadore@FreeBSD.org) id 2276 by MailHost (DragonFly Mail Agent v0.11+); Tue, 26 Nov 2019 14:43:32 +0100 To: freebsd-hackers@FreeBSD.org Subject: FreeBSD Quarterly Status Report - Third Quarter 2019 Cc: freebsd-current@FreeBSD.org, freebsd-stable@FreeBSD.org Date: Tue, 26 Nov 2019 14:43:32 +0100 Message-Id: <5ddd2c04.2276.7c14d497@MailHost> From: X-Rspamd-Queue-Id: 47MlVz1CfJz3LDM X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-1.80 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-0.91)[-0.912,0]; NEURAL_HAM_LONG(-0.89)[-0.890,0]; ASN(0.00)[asn:31034, ipnet:80.211.0.0/17, country:IT] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2019 13:44:12 -0000 FreeBSD Project Quarterly Status Report - Third Quarter 2019 Here is the third quarterly status report for 2019. This quarter the reports team has been more active than usual thanks to a better organization: calls for reports and reminders have been sent regularly, reports have been reviewed and merged quickly (I would like to thank debdrup@ in particular for his reviewing work). Efficiency could still be improved with the help of our community. In particular, the quarterly team has found that many reports have arrived in the last days before the deadline or even after. I would like to invite the community to follow the guidelines below that can help us sending out the reports sooner. Starting from next quarter, all quarterly status reports will be prepared the last month of the quarter itself, instead of the first month after the quarter's end. This means that deadlines for submitting reports will be the 1st of January, April, July and October. Next quarter will then be a short one, covering the months of November and December only and the report will probably be out in mid January. -- Lorenzo Salvadore __________________________________________________________________ FreeBSD Team Reports * Cluster Administration Team * Continuous Integration * FreeBSD Core Team * FreeBSD Foundation * FreeBSD Graphics Team status report * FreeBSD Release Engineering Team * FreeBSD Security Team Projects * FAT / msdosfs support for makefs(8) * FUSE * Google Summer of Code 2019 * GSoC'19 Project - MAC policy on IP addresses in Jail: mac_ipacl * Improving laptop support * NFS Version 4.2 implementation * Rockchip RK3399 SoC's eMMC support * syzkaller on FreeBSD * TPM2 Software Stack (TSS2) Kernel * Casueword(9) livelock * Kernel Mapping Protections * Kernel ZLIB Update * PROT_MAX mmap/mprotect maximum protections API * Randomized Top of Stack pointer * Signals delivered on unhandled Page Faults Architectures * Broadcom ARM64 SoC support * FreeBSD support for the forthcoming Arm Morello CPU, SoC, and board * FreeBSD/powerpc Project * NXP ARM64 SoC support Userland Programs * gets(3) retirement Ports * FreshPorts * Java on FreeBSD * KDE on FreeBSD * Ports Collection * XFCE 4.14 update Third-Party Projects * ClonOS: virtualization platform on top of FreeBSD Operating System * ENA FreeBSD Driver Update * Nomad pot driver - Orchestrating jails via nomad * sysctlinfo __________________________________________________________________ FreeBSD Team Reports Entries from the various official and semi-official teams, as found in the Administration Page. Cluster Administration Team Contact: Cluster Administration Team The FreeBSD Cluster Administration Team consists of the people responsible for administering the machines that the Project relies on for its distributed work and communications to be synchronised. In this quarter, the team has worked on the following: * Change IPv6 address in TWN site. * Solved hardware issues in KWC site (with hrs@). * Moved remaining infrastructure from the YSV (Yahoo!) site to NYI (New York Internet) (peter@). * YSV hosted most of FreeBSD.org between 2000 and 2019. Installed new machines for portmgr@ courtesy of the FreeBSD Foundation. Resolved outtages (thanks uqs@) with GitHub exporter, Bugzilla and hg-beta (thanks bapt@). PowerPC64 servers are online (power8) building pkgs and reference hosts. Ongoing systems administration work: * Creating accounts for new committers. * Backups of critical infrastructure. * Keeping up with security updates in 3rd party software. Work in progress: * Review the service jails and service administrators operation. * South Africa Mirror (JINX) in progress. * NVME issues on PowerPC64 Power9 blocking dual socket machine from being used as pkg builder. * Drive upgrade test for pkg builders (SSDs) courtesy of the FreeBSD Foundation. * Boot issues with Aarch64 reference machines. * New NYI.net sponsored colocation space in Chicago-land area. * Setup new host for CI staging environment. __________________________________________________________________ Continuous Integration Links FreeBSD Jenkins Instance URL: https://ci.FreeBSD.org FreeBSD CI artifact archive URL: https://artifact.ci.FreeBSD.org/ FreeBSD Jenkins wiki URL: https://wiki.freebsd.org/Jenkins freebsd-testing Mailing List URL: https://lists.FreeBSD.org/mailman/listinfo/freebsd-testing FreeBSD CI Repository URL: https://github.com/freebsd/freebsd-ci Tickets related to freebsd-testing@ URL: https://preview.tinyurl.com/y9maauwg Hosted CI wiki URL: https://wiki.freebsd.org/HostedCI FreeBSD CI weekly report URL: https://hackmd.io/@FreeBSD-CI Contact: Jenkins Admin Contact: Li-Wen Hsu The FreeBSD CI team maintains continuous integration system and related tasks for the FreeBSD project. The CI system regularly checks the committed changes can be successfully built, then performs various tests and analysis of the results. The results from build jobs are archived in an artifact server, for the further testing and debugging needs. The CI team members examine the failing builds and unstable tests, and work with the experts in that area to fix the code or adjust test infrastructure. The details are of these efforts are available in the weekly CI reports. We had a testing working group at the 201909 DevSummit lwhsu@ has presented the Testing/CI project status and "how to work with the FreeBSD CI system", slides are available at the DevSummit page. Some contents have been migrated to https://wiki.freebsd.org/Jenkins/Debug , extending is welcomed. We continue publishing CI Weekly Report and moved the archive to https://hackmd.io/@FreeBSD-CI Work in progress: * Collecting and sorting CI tasks and ideas at https://hackmd.io/bWCGgdDFTTK_FG0X7J1Vmg * Setup the CI stage environment and put the experimental jobs on it * Extending and publishing the embedded boards testbed * Implementing automatic tests on bare metal hardware * Adding drm ports building test against -CURRENT * Testing and merging pull requests at https://github.com/freebsd/freebsd-ci/pulls * Planning for running ztest and network stack tests * Help more 3rd software get CI on FreeBSD through a hosted CI solution Please see freebsd-testing@ related tickets for more WIP information. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ FreeBSD Core Team Contact: FreeBSD Core Team The FreeBSD Core Team is the governing body of FreeBSD. * Core has provisionally accepted the BSD+patent license for use in some cases. The Core Team must approve the import of new BSD+Patent licensed components or the change of license of existing components to the BSD+Patent License. https://opensource.org/licenses/BSDplusPatent * Kernel Pseudo Random Number Generator (PRNG) maintainership was updated to reduce the contribution barrier for committers who have demonstrated competence in this part of the tree. * Core approved a source commit bit for Pawel/ Biernacki. Konstantin Belousov will mentor Pawel/ and Mateusz Guzik will be co-mentor. * The Core-initiated Git Transition Working Group met over the last quarter, however a report is still forthcoming. Discussions will continue in the fourth quarter of 2019. There are many issues to resolve including how to deal with contrib/, whether to re-generate hashes in the current Git repository, and how to best implement commit testing. __________________________________________________________________ FreeBSD Foundation Contact: Deb Goodkin The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the FreeBSD Project and community worldwide. Funding comes from individual and corporate donations and is used to fund and manage software development projects, conferences and developer summits, and provide travel grants to FreeBSD contributors. The Foundation purchases and supports hardware to improve and maintain FreeBSD infrastructure and provides resources to improve security and quality assurance efforts; publishes marketing material to promote, educate, and advocate for the FreeBSD Project; facilitates collaboration between commercial vendors and FreeBSD developers; and finally, represents the FreeBSD Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity. Here are some highlights of what we did to help FreeBSD last quarter: Partnerships and Commercial User Support We help facilitate collaboration between commercial users and FreeBSD developers. We also meet with companies to discuss their needs and bring that information back to the Project. In Q3, Ed Maste and Deb Goodkin met with a few commercial users in the US. It is not only beneficial for the above, but it also helps us understand some of the applications where FreeBSD is used. We were also able to meet with a good number of commercial users at vBSDCon and EuroBSDCon. These venues provide an excellent opportunity to meet with commercial and individual users and contributors to FreeBSD. Fundraising Efforts Our work is 100% funded by your donations. We are continuing to work hard to get more commercial users to give back to help us continue our work supporting FreeBSD. More importantly, we'd like to thank our individual donors for making $10-$1,000 donations last quarter, for more than $16,000! Please consider making a donation to help us continue and increase our support for FreeBSD! We also have the Partnership Program, to provide more benefits for our larger commercial donors. Find out more information at www.FreeBSDfoundation.org/FreeBSD-foundation-partnership-program/ and share with your companies. OS Improvements The Foundation supports software development projects to improve the FreeBSD operating system through our full time technical staff, contractors, and project grant recipients. They maintain and improve critical kernel subsystems, add new features and functionality, and fix problems. Over the last quarter there were 345 commits to the FreeBSD base system repository sponsored by the FreeBSD Foundation - this represents about one fifth of all commits during this period. Many of these projects have their own entries in this quarterly report (and are not repeated here). Foundation staff member Konstantin Belousov committed many improvements to multiple kernel subsystems, as well as low-level 32-bit and 64-bit x86 infrastructure. These included fixes for robust mutexes, unionfs, the out of memory (OOM) handler, and per-cpu allocators. Additional work included fixes for security issues and introduction and maintenance of vulnerability mitigations, and improving POSIX conformance. Ed Maste committed a number of minor security bug fixes and improvements, as well as the first iteration of a tool for editing the mitigation control ELF note. Additional work included effort on build infrastructure and the tool chain. Clang's integrated assembler (IAS) is now used more widely, as part of the path to retiring the assembler from GNU binutils 2.17.50. The readelf tool now decodes some additional ELF note information. Ed also enabled the Linuxulator (Linux binary support layer) on arm64, and added a trivial implementation of the renameat2 system call (handling common options). Mark Johnston added Capsicum support to a number of ELF Tool Chain utilities, and committed a number of other Capsicum kernel and userland fixes. Mark worked on a number of changes related to security improvements, including integration and support of the Syzkaller automated system call fuzzer, and fixing issues identified by Syzkaller. Other changes included addressing failures caused by refcount wraparound, improvements to the prot_max memory protection. Other work included NUMA, locking, kernel debugging, RISC-V and arm64 kernel improvements. Edward Napierala continued working on Linuxulator improvements over the quarter. The primary focus continued to be tool improvements - strace is now more usable for diagnosing issues with Linux binaries running under the Linuxulator. That said, as with previous work a number of issues have been fixed along the way. These are generally minor issues with a large impact - for example, every binary linked against up-to-date glibc previously segfaulted on startup. This is now fixed. Continuous Integration and Quality Assurance The Foundation provides a full-time staff member who is working on improving our automated testing, continuous integration, and overall quality assurance efforts. During the third quarter of 2019, Foundation staff continued to improve the project's CI infrastructure, worked with contributors to fix the failing build and test cases, and worked with other teams in the Project for their testing needs. We added several new CI jobs and worked on getting the hardware regression testing lab ready. Li-Wen Hsu gave presentations "Testing/CI status update" and "How to work with the FreeBSD CI system" at the 201909 DevSummit. Slides are available at the DevSummit page. We continue publishing the CI weekly report on the freebsd-testing@. mailing list, and an archive is available. See the FreeBSD CI section of this report for completed work items and detailed information. Supporting FreeBSD Infrastructure The Foundation provides hardware and support to improve the FreeBSD infrastructure. Last quarter, we continued supporting FreeBSD hardware located around the world. FreeBSD Advocacy and Education A large part of our efforts are dedicated to advocating for the Project. This includes promoting work being done by others with FreeBSD; producing advocacy literature to teach people about FreeBSD and help make the path to starting using FreeBSD or contributing to the Project easier; and attending and getting other FreeBSD contributors to volunteer to run FreeBSD events, staff FreeBSD tables, and give FreeBSD presentations. The FreeBSD Foundation sponsors many conferences, events, and summits around the globe. These events can be BSD-related, open source, or technology events geared towards underrepresented groups. We support the FreeBSD-focused events to help provide a venue for sharing knowledge, to work together on projects, and to facilitate collaboration between developers and commercial users. This all helps provide a healthy ecosystem. We support the non-FreeBSD events to promote and raise awareness of FreeBSD, to increase the use of FreeBSD in different applications, and to recruit more contributors to the Project. Check out some of the advocacy and education work we did last quarter: * Sponsored USENIX 2019 Annual Technical Conference as an Industry Partner * Represented FreeBSD at OSCON 2019 in Portland, OR * Represented FreeBSD at COSCUP 2019 in Taiwan * Presented at the Open Source Summit, North American in San Diego, CA * Executive Director Deb Goodkin was interviewed by TFiR https://www.freebsdfoundation.org/news-and-events/latest-news/tfir-interview-freebsd-meets-linux-at-the-open-source-summit/ * Sponsored FreeBSD Hackathon at vBSDcon 2019 in Reston, VA * Sponsored the attendee bags and attended vBSDcon 2019 in Reston VA * Represented FreeBSD at APNIC-48 in Chiang Mai, Thailand * Represented FreeBSD at MNNOG-1 in Ulaanbaatar, Mongolia * Served as an administrator for the Project's Google Summer of Code Session. See the Google Summer of Code section of this report for more information. * Sponsored FreeBSD Developers Summit at EuroBSDCon in Lillehammer, Norway * Sponsored and attended EuroBSDcon 2019 in Lillehammer, Norway * Applied and was accepted for a FreeBSD Miniconf at linux.conf.au, in Gold Coast, Australia, Jan 14, 2020 * Our FreeBSD talk was accepted at seaGL, Seattle, WA, November 15 and 16. We continued producing FreeBSD advocacy material to help people promote FreeBSD. Learn more about our recent efforts to advocate for FreeBSD around the world: https://www.freebsdfoundation.org/blog/freebsd-around-the-world/ Our Faces of FreeBSD series is back. Check out the latest post: Roller Angel. Read more about our conference adventures in the conference recaps and trip reports in our monthly newsletters: https://www.freebsdfoundation.org/news-and-events/newsletter/ We help educate the world about FreeBSD by publishing the professionally produced FreeBSD Journal. As we mentioned previously, the FreeBSD Journal is now a free publication. Find out more and access the latest issues at https://www.FreeBSDfoundation.org/journal/. You can find out more about events we attended and upcoming events. We opened our official FreeBSD Swag Store. Get stickers, shirts, mugs and more at ShopFreeBSD. We have continued our work with a new website developer to help us improve our website. Work has begun to make it easier for community members to find information and to make the site more efficient. Legal/FreeBSD IP The Foundation owns the FreeBSD trademarks, and it is our responsibility to protect them. We also provide legal support for the core team to investigate questions that arise. Go to http://www.FreeBSDfoundation.org to find out how we support FreeBSD and how we can help you! __________________________________________________________________ FreeBSD Graphics Team status report Links Project GitHub page URL: https://github.com/FreeBSDDesktop Contact: FreeBSD Graphics Team Contact: Niclas Zeising The FreeBSD X11/Graphics team maintains the lower levels of the FreeBSD graphics stack. This includes graphics drivers, graphics libraries such as the MESA OpenGL implementation, the X.org xserver with related libraries and applications, and Wayland with related libraries and applications. During the last period, several changes have been made, but most of them has been behind the scene. We have also worked on general clean up of old xorg ports that have been deprecated upstream. The ports infrastructure for xorg ports and ports that depend on xorg ports have been updated. We have switched USE_XORG and XORG_CAT to use the USES framework, instead of the old way of including bsd.xorg.mk from bsd.port.mk. This infrastructure work has been fairly substantial, and new ports depending on xorg ports should add USES=xorg to their makefiles. As part of this bsd.xorg.mk was split up, and the XORG_CAT part was split out to USES=xorg-cat. This is used for the xorg ports themselves, and sets up a common environment for building all xorg ports. In addition, framework for pulling xorg ports directly from freedesktop.org gitlab was added, which will make improve development and testing, since it makes it possible to create ports of unreleased versions. Further improvements in this area includes framework for using meson instead of autotools for building xorg ports. This is still a work in progress. We have also worked to clean up and deprecate several old xorg ports and libraries. Some of these ports have already been removed, and some are still waiting on removal after a sufficient deprecation period. Most notably amongst the deprecations are x11/libXp, which required to fix several dependencies. Several other old libraries have also been deprecated, such as x11/Xxf86misc, x11-fonts/libXfontcache and graphics/libGLw. Some applications and drivers have also been deprecated during the period. With the remaining removals in this area, we should be up to speed with deprecations upstream. We are currently investigating if there are new software added upstream that we need to port to FreeBSD. We have also continued our regularly scheduled bi-weekly meetings. People who are interested in helping out can find us on the x11@FreeBSD.org mailing list, or on our gitter chat: https://gitter.im/FreeBSDDesktop/Lobby. We are also available in #freebsd-xorg on EFNet. We also have a team area on GitHub where our work repositories can be found: https://github.com/FreeBSDDesktop __________________________________________________________________ FreeBSD Release Engineering Team Links FreeBSD 11.3-RELEASE announcement URL: https://www.freebsd.org/releases/11.3R/announce.html FreeBSD 12.1-RELEASE schedule URL: https://www.freebsd.org/releases/12.1R/schedule.html FreeBSD 12.1-RELEASE BETA/RC builds URL: https://download.freebsd.org/ftp/releases/ISO-IMAGES/12.1/ FreeBSD development snapshots URL: https://download.freebsd.org/ftp/snapshots/ISO-IMAGES/ Contact: FreeBSD Release Engineering Team The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code freezes and maintaining the respective branches, among other things. During the third quarter of 2019, the FreeBSD Release Engineering team finished the 11.3-RELEASE cycle, with the final release build started on July 5th and the official announcement sent on July 9th. FreeBSD 11.3-RELEASE is the fourth release from the stable/11 branch, building on the stability and reliability of 11.2-RELEASE. The FreeBSD Release Engineering Team also started work on the upcoming 12.1-RELEASE, which started September 6th. This release cycle is the first "freeze-less" release from the Subversion repository, and the test bed for eliminating the requirement of a hard code freeze on development branches. Commits to the releng/12.1 branch still require explicit approval from the Release Engineering Team, however. At present, there have been three BETA builds, and so far, two RC builds, with the final 12.1-RELEASE build scheduled for November 4th. Additionally throughout the quarter, several development snapshots builds were released for the head and stable/11 branches; snapshots for stable/12 were released as well although not during the 12.1-RELEASE cycle. Much of this work was sponsored by Rubicon Communications, LLC (Netgate) and the FreeBSD Foundation. __________________________________________________________________ FreeBSD Security Team Links FreeBSD security information URL: https://www.freebsd.org/security/ Contact: Security Team Several members of the security team met at the Vendor Summit in October to formalize team structure dedicated for architecture and crypto engineering in addition to the existing product security incident response function. Since June we have started having fortnightly conference calls to discuss important issues and to collaborate closely on advisories and errata notices in the pipeline. * Security advisories sent out in 2019-Q3: 7 * Errata Notices sent out in 2019-Q3: 5 __________________________________________________________________ Projects Projects that span multiple categories, from the kernel and userspace to the Ports Collection or external projects. FAT / msdosfs support for makefs(8) Contact: Ed Maste In order to streamline the process of creating install or virtual machine system images we needed FAT filesystem support in makefs(8). Makefs was originally developed in NetBSD, and FAT support was added there not much later, but after the tool was ported to FreeBSD. Siva Mahadevan, one of the FreeBSD Foundation's interns from the University of Waterloo, worked on porting FAT support from NetBSD. I rebased and updated Siva's work, and committed it during this quarter. After a few follow-up fixes we are able to build FAT filesystem images without using md(4) and without requiring root. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ FUSE Contact: Alan Somers FUSE (File system in USErspace) allows a userspace program to implement a file system. It is widely used to support out-of-tree file systems like NTFS, as well as for exotic pseudo file systems like sshfs. FreeBSD's fuse driver was added as a GSoC project in 2012. Since that time, it has been largely neglected. The FUSE software is buggy and out-of-date. Our implementation is about 11 years behind. I completed this work during Q3. I fixed a few newly-introduced bugs, fixed a long-standing sendfile bug that affects FUSE ([236466](https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236466)) and merged everything to head and stable/12. Then I fixed the resulting Coverity CIDs. There have been no new FUSE-related bug reports, so I can only assume that everything is working great. Report any problems to asomers@FreeBSD.org. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ Google Summer of Code 2019 Links 2019 Summer of Code Project Wikis URL: https://wiki.freebsd.org/SummerOfCode2019Projects 2019 Summer of Code Projects URL: https://summerofcode.withgoogle.com/archive/2019/organizations/6504969929228288/ Contact: Summer of Code Admins The FreeBSD Project is pleased to have participated in Google Summer of Code 2019 marking our 14th year of participation. This year we had six successful projects: * Dual-stack ping command by Ján Sucan * Firewall test suite by Ahsan Barkati * Kernel sanitizers by Costin Carabas * MAC policy on IP addresses for FreeBSD Jail by Shivank Garg * Separation of ports build process from local installation by Theron Tarigo * Virtual memory compression by Paavo-Einari Kaipila We thank Google for the opportunity to work with these students and hope they continue to work with FreeBSD in the future. This project was sponsored by Google Summer of Code. __________________________________________________________________ GSoC'19 Project - MAC policy on IP addresses in Jail: mac_ipacl Links FreeBSD's Phabricator Differential Link URL: https://reviews.freebsd.org/D20967 Github Diff Link URL: https://github.com/freebsd/freebsd/compare/master...shivankgarg98:shivank_MACPolicyIPAddressJail Project Wiki Page URL: https://wiki.freebsd.org/SummerOfCode2019Projects/MACPolicyIPAddressJail Contact: Shivank Garg About - With the introduction of VNET(9) in FreeBSD, Jails are free to set their IP addresses. However, this privilege may need to be limited by the host as per its need for multiple security reasons. This project uses mac(9) for an access control framework to impose restrictions on FreeBSD jails according to rules defined by the root of the host using sysctl(8). It involves the development of a dynamically loadable kernel module (mac_ipacl) based on The TrustedBSD MAC Framework to implement a security policy for configuring the network stack. This project allows the root of the host to define the policy rules to limit the root of a jail to a set of IP (v4 or v6) addresses and/or subnets for a set of interfaces. Features this new MAC policy module are: * The host can define one or more lists of IP addresses/subnets for the jail to choose from. * The host can restrict the jail from setting certain IP addresses or prefixes (subnets). * The host can restrict this privilege to a few network interfaces. Implementation - The mac_ipacl module is a loadable kernel module. It implements mac checks in netinet/in.c and netinet6/in6.c to check the IP addresses requested by jail. The idea to implement these checks at these places comes from the fact that SIOCAIFADDR (for IPv4) and SIOCAIFADDR_IN6 (for IPv6) ioctl handlers are defined for adding the IP addresses to an interface. This is used by ifconfig (in userspace) for setting the IP address. The MAC Framework acts as multiplexer between the netinet and the module. The requested IP and the credentials are checked with the rules in mac_ipacl and output is returned accordingly to netinet. The module can be tuned with various sysctl and similarly, policy rules are also be defined with sysctl. TestSuite - Test scripts integrated with kyua and ATF are included with the module. Using the module - I have written a man page for the module. Please refer to the mac_ipacl(4) for using the new MAC module and various examples. Final Deliverables - * A loadable kernel module - mac_ipacl in sys/security/mac_ipacl * ATF tests for the module in tests/sys/mac/ipacl * A man page for this new mac module - mac_ipacl.4 in share/man/man4/mac_ipacl.4 This is a new project, developed as part of Google Summer of Code'19 under the guidance of Bjoern A. Zeeb . The module is reviewed and Revision for this project is accepted and ready to land. It is yet to be merged with FreeBSD HEAD, and waiting to be tested by few more hands in the industry. I'll be very thankful if you can give this module a try and share your valuable experience about it. Please be free to share your ideas and feedback on this module and please do not hesitate to send me an email. __________________________________________________________________ Improving laptop support Contact: Ed Maste The FreeBSD Foundation would like to ensure that running FreeBSD on contemporary hardware, including laptops, remains viable. To that end we plan to purchase the latest generation of one or more of a family of laptops preferred by members of the FreeBSD community, evaluate the existing state of hardware support, and implement missing hardware support where possible. As the first laptop for this project we have selected a 7th Generation Lenovo X1 Carbon. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ NFS Version 4.2 implementation Contact: Rick Macklem RFC-7862 describes a new minor revision to the NFS Version 4 protocol. This project implements this new minor revision. The NFS Version 4 Minor Version 2 protocol adds several optional features to NFS, such as support for SEEK_DATA/SEEK_HOLE, file copying done on the server that avoids data transfer over the wire and support for posix_fallocate(), posix_fadvise(). Hopefully these features can improve performance for certain applications. The implementation is now nearing completion and recent work has been mostly testing. A cycle of interoperability testing with Linux has just been completed. The main area that still needs testing is use of the pNFS server with NFSv4.2 and that should start soon. Once testing of pNFS is completed, I believe the code is ready to be incorporated into FreeBSD head/current. Testing by others would be appreciated. The modified kernel can be found at https://svn.freebsd.org/base/projects/nfsv42/sys and should run with a recent FreeBSD head/current system. Client mounts need the "minorversion=2" mount option to enable this protocol. __________________________________________________________________ Rockchip RK3399 SoC's eMMC support Contact: Ganbold Tsagaankhuu The followings features have been added to support RK3399 SoC eMMC on FreeBSD: * Extended simple_mfd driver to expose a syscon interface if that node is also compatible with syscon. For instance, Rockchip RK3399's GRF (General Register Files) is compatible with simple-mfd as well as syscon and has devices like usb2-phy, emmc-phy and pcie-phy etc. under it. * Made Rockchip's General Register Files driver the subclass of Simple MFD driver * Added driver for Rockchip RK3399 eMMC PHY. * Added eMMC support codes for Rockchip RK3399 SoC. * All above was tested on NanoPC-T4 board. __________________________________________________________________ syzkaller on FreeBSD Contact: Andrew Turner Contact: Mark Johnston Contact: Michael Tuexen Contact: Samy Al Bahra See the syzkaller entry in the 2019q1 quarterly report for an introduction to syzkaller. Work has continued on fixing bugs uncovered by syzkaller. Over a dozen kernel bugs fixed in the past three months have been directly attributed to syzkaller, and a number of syzkaller reproducers have been incorporated into our test suite. backtrace.io, via Samy, has graciously provided a large server for running a dedicated syzkaller instance to fuzz FreeBSD under bhyve. Though syzbot, the public syzkaller instance run by Google, already fuzzes FreeBSD, it has proven fruitful to run multiple syzkaller instances: different instances find different bugs, and syzkaller.backtrace.io allows us to experiment with FreeBSD-specific extensions. In particular, this instance currently uploads data about each crash to backtrace.io, making it much easier to triage and analyze crashes. We plan to make this service generally available to FreeBSD developers in the near future. Going forward we will continue to extend syzkaller coverage and make it simpler for users and developers to run private instances, and optionally collect kernel crash information for debugging or for submission. This project was sponsored by backtrace.io, and The FreeBSD Foundation. __________________________________________________________________ TPM2 Software Stack (TSS2) Links tpm2-tss Documentation URL: https://tpm2-tss.readthedocs.io/en/latest/index.html tpm2 Source Repository URL: https://github.com/tpm2-software/ tpm2 mailing list URL: https://lists.01.org/postorius/lists/tpm2.lists.01.org/ tpm2 irc channel URL: ircs://chat.freenode.net:6697/tpm2.0-tss Contact: D Scott Phillips Intel has contributed ports of the TPM2 Software Stack to the ports tree, with the new ports securtity/tpm2-tss, security/tpm2-tools, security/tpm2-abrmd. tpm2-tss contains a set of libraries which expose various TPM2 APIs for using a TPM conforming to the TCG TPM 2.0 specification. tpm2-tools provides a set of command line tools which use the tpm2-tss libraries to perform tpm operations. Finally, tpm2-abrmd is a userspace daemon which acts as a TPM Access Broker and Resource Manager, multiplexing many TPM users onto a single TPM device. Sponsored by: Intel Corporation __________________________________________________________________ Kernel Updates to kernel subsystems/features, driver support, filesystems, and more. Casueword(9) livelock Contact: Konstantin Belousov The Compare-And-Swap (CAS) is one of the fundamental building blocks for hardware-assisted atomic read/modify/write operations. Some architectures support it directly, for instance x86 and sparc. Others provide different building blocks, usually the pair of Load Linked/Store Conditional instructions (ll/sc) which can be used to construct CAS or other atomic operations like Fetch-And-Add or any atomic arithmetic ops using plain arithmetic instructions. An example is the LDXR/STXR pair on AArch64. The ll/sc operation is performed by first using the load linked instruction to load a value from memory and simultaneously mark the cache line for exclusive access. The value is then updated by the store conditional instruction, but only if there were not any writes to the marked cache line. Any write by another CPU makes the store instruction fail. So typically atomic primitives on ll/sc architectures retry the whole operation if only store failed, because it just means that this CPU either lost a race, or even the failure was spurious. This is so called strong version of CAS and atomics. If primitive returns failure instead, the CAS variant is called weak by C standard. For the FreeBSD threading library lock implementation, when the fast path mode in userspace was not possible, the kernel often needs to do a CAS operation on user memory location. In addition to all guarantees of normal CAS, it also must ensure the safety and tolerance to paging. In FreeBSD, the casueword32(9) primitive provides CAS on usermode 32bit words for kernel. Casueword32(9) was implemented as strong CAS, similarly to the mode of operation of hardware CAS instructions on x86. Using the strong implementation for casueword may be dangerous, since the same address is potentially accessible to other, potentially malicious, threads in the same or other processes. If such a thread constantly dirties the cache line used by a ll/sc loop, it could practically force the kernel-mode thread to remain stuck in the loop forever. Since the loop is tight, and it does not check for signals, the thread cannot be stopped or killed. The solution is to make the casueword implementation weak, which means that the interface of the primitive must be changed to allow notifying the caller about spurious failures. Also, it is now the caller's responsibility to retry as appropriate. The change to casueword was made for all architectures. Even on x86, the PSL.ZF value after the CMPXCHG instruction was returned directly for the new casueword. All two dozens uses of the primitive, all located in kern_umtx.c, were inspected and retry added as needed. As a somewhat related note, in-kernel atomic_cmpset(9) operations are strong, while atomic_fcmpset(9) should be weak, unless broken by a specific architecture. The general attitude seems to be that retry is the duty of the primitive's caller. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ Kernel Mapping Protections Contact: Mark Johnston Modern CPU architectures have the ability to flag memory mappings as "no-execute" (NX), which prevents that memory from being executed by a processor. NX mappings are an important security mitigation since they help segregate code and data, blocking unintentional execution of memory whose contents is controlled by an attacker. W^X (write XOR execute) is a security policy which disallows the creation of mappings that are simultaneously writeable and executable. Under this policy, memory whose contents can be modified must be mapped with the NX flag. This policy makes it harder to exploit bugs that permit an attacker to arbitrarily overwrite data. FreeBSD has long made use of the NX flag for userspace mappings whenever possible, but only in the past several years has it been applied to kernel mappings. A recent project has sought to implement a W^X-by-default policy for the amd64 kernel by completing an audit of the remaining executable mappings in the kernel, and making modifications to either apply the NX bit to those mappings or to make them read-only. This work has landed in HEAD and will be available in FreeBSD 13.0 and 12.2. Similar work for other CPU architectures is also planned. To help audit kernel mapping protections, the vm.pmap.kernel_maps sysctl was added; it dumps a snapshot of the kernel's page table entries and their attributes. This way, mappings violating the W^X policy can easily be discovered and investigated, and the sysctl provides information useful to anyone interested in kernel memory layout. With a few rare exceptions, the only kernel mappings which require execute permission are those of the kernel executable itself, and loadable kernel modules. A number of other regions of memory were unnecessarily being mapped without NX, and these were identified and corrected first. To address the kernel code mappings, the amd64 kernel linker script was modified to pad the .text segment to a 2MB boundary. To improve performance, the kernel creates superpage mappings of its .text segment, but this means that any data cohabiting the final 2MB .text mapping is mapped with execute permissions. The padding allows executable code to be segregated from data which follows in the kernel image, avoiding this problem and maintaining the superpage optimization at the expense of some wasted RAM. Enforcing W^X turned out to be somewhat trickier. Unlike other CPU architectures supported by FreeBSD, amd64 kernel modules are linked as relocatable object files, i.e., .o files. (On other architectures, they are dynamically shared objects (DSOs, or .so files), as one might naively expect.) The use of .o files means that amd64 kernel modules contain more efficient code than they would if linked as DSOs, since DSOs inherently make use of certain types of indirection which allow shared libraries to be loaded at arbitrary addresses, and this indirection is useless in the kernel. As part of this project an attempt was made to switch amd64 to using DSOs as well, since the cost of this indirection can largely be mitigated with modern toolchains, but it was found that the use of DSOs would also force a change to the code model used when compiling amd64 kernel code, resulting in a further performance penalty. The main obstacle with the use of .o files is that sections are not page-aligned by default; the segregation of sections with differing mapping protection requirements is done by the static linker when linking a DSO or executable file. Since mapping protections are applied at the granularity of the page size (4KB on amd64), the overlap of sections within a page causes problems since, for example, the end of the read-only .text section may overlap with the beginning of the read-write .data section. One possible solution is to perform any required section reordering and padding at kernel module load time, but this approach breaks debugging tools such as dtrace(1) and kgdb which assume that the kernel linker does not modify the layout of loadable modules. As a result, an amd64 kernel module linker script is now used to insert padding for specific sections. Finally, the kernel linker was modified to restrict mapping protections based on section flags. As a result of all of this, amd64 kernels now boot without any writeable, executable mappings. Though some of the work was architecture-specific, much of it can and will be leveraged to provide the same policy for our other supported architectures. This project was sponsored by Netflix. __________________________________________________________________ Kernel ZLIB Update Contact: Xin Li Contact: Yoshihiro Ota The ZLIB is a compression library is widely used in various software. The FreeBSD system had used an ancient (over 20 year-old) version of zlib (version 1.0.4) and total of 3 versions, one in user-land, one in ZFS, and one in kernel. Xin and Yoshihiro upgraded zlib to the latest and eliminated 2 extra copies. Along the efforts, zlib version was upgraded to 1.2.11, netgraph's ppp is re-implemented to use the standard zlib, and removed unmaintained code. We now only have one and the latest version of zlib in FreeBSD code base, new compress, compress2, and uncompress APIs exposed in the kernel, and importing changes from zlib will be simple. Kernel zlib changes * sys/crc32.h is split to avoid object file name conflict with ZLIB's * contrib/zlib is moved to sys/contrib/zlib * Kernel started switching to sys/contrib/zlib and ZFS copy dropped * Kernel zcalloc is introduced and compress, compress2, and uncompress APIs are exposed to kernel * Removed zlib 1.0.4 from kernel Kernel zlib user updates * kern_ctf.c updated * opencryptodeflate updated * geom_uzip updated * subr_compressor updated * if_mxge updated * bxe updated * ng_deflate updated Legacy code removals * Removed MIPS zlib elf trampoline * Removed kgzip and kgzldr * Removed gzip'ed a.out support * Removed ArmvX elf_trampoline.c __________________________________________________________________ PROT_MAX mmap/mprotect maximum protections API Links PROT_MAX commit URL: https://reviews.freebsd.org/rS349240 Contact: Brooks Davis Unix-like systems provide the mmap(2) system call to allocate memory or map files or devices into memory with specified access protection, and the mprotect(2) system call to change protections on mapped memory. Protection flags specify whether the memory may be read, written, and/or executed (PROT_READ, PROT_WRITE, PROT_EXEC respectively). Traditionally, mprotect(2) can be used to set any desired protections (except that a shared mapping of a file opened read-only cannot have PROT_WRITE set). A new macro PROT_MAX() adds a facility for specifying the maximum possible protection flags for mmap(2) and mprotect(2) calls. The program can then specify whether a mapping may be changed in the future to allow a given access protection. For example, a memory mapping can be set such that it can have read and write protections set, but may never be made executable. Maximum protection values are provided to the PROT_MAX() macro, and are OR'd with regular protection values. This change allows (e.g.) a region that must be writable during run-time linking or JIT code generation to be made permanently read+execute after writes are complete. This complements Write-XOR-Execute (W^X) protections available on other operating systems, allowing more precise control by the programmer. For example, to request memory that cannot be made executable: mmap(NULL, size, PROT_READ | PROT_WRITE | PROT_MAX(PROT_READ | PROT_WRITE), MAP_ANON, -1, 0); and to request memory that may have execute permission enabled later on, but is not currently executable: mmap(NULL, size, PROT_READ | PROT_WRITE | PROT_MAX(PROT_READ | PROT_WRITE | PROT_EXECUTE), MAP_ANON, -1, 0); This change alters mprotect argument checking and returns an error when unhandled protection flags are set. This differs from POSIX (in that POSIX only specifies an error if a valid permission can not be set), but is the documented behavior on Linux and more closely matches historical mmap behavior. In addition to explicit setting of the maximum permissions, an experimental sysctl vm.imply_prot_max causes mmap to assume that the initial permissions requested should be the maximum when the sysctl is set to 1. This behavior is known to break code that uses PROT_NONE reservations before mapping contents into part of the reservation. A later version of this work is expected to provide per-binary and per-process opt-in/out options and this sysctl may be removed in its current form. As such it is undocumented. While these flags are non-portable, they can be used in portable code with simple ifdefs to expand PROT_MAX() to 0. Sponsors: DARPA, AFRL __________________________________________________________________ Randomized Top of Stack pointer Contact: Konstantin Belousov After the ASLR so useful addition, next in the series of the buzzword-compliant checkboxes is the stack addresses randomization. The main userspace thread stack on FreeBSD is traditionally allocated from the top of the user address space and grows down. Besides the initial pointer for the stack on userspace entry, this area also contains structures for program arguments and environment (so called strings), and aux vector data for ELF images. Considering the goal of randomizing the addresses of strings and main thread initial frame, moving the main stack area in the address space is not feasible. It would cause significant ABI breakage, invalidates the knowledge already coded into many introspection tools, and causes unneeded additional fragmentation of the user address space. Instead a typical approach of adding a gap of randomized size between top of stack and a place for strings was used. It is done in a way which preserves the stack alignment requirement. Stack gap is only enabled if ASLR is enabled (not default) and stack gap itself is enabled (default if ASLR is enabled). Stack gap is specified in percentage of the total stack size that can be used for maximum gap. The main drawback of the gap approach was shortly identified. Since gap is cut from the normal stack area, attempts of the programs to reduce stack size using rlimit(RLIMIT_STACK) could cut the active stack region if new limit happens to be smaller than the gap. E.g. on amd64 with its default 512M main thread stack, even one percent of the max gap gives approximately 5M of unused stack, that can blow up the limit of several KBs. Typical reason for using rlimit(2) this way is for programs that wire all of its address space with mlockall(2), trying to reduce potential wired stack size to avoid exceeding RLIMIT_MEMLOCK. First victim of that issue was ntpd, which resets the stack limit after start for a really small value. Currently the wiring was removed from ntpd, because apparently it does not make the timekeeping better by any means, contrary to popular belief. My opinion is that the problem is more in the user interface area than in the gap approach itself. We should make it easy to specify small gap sizes, which cannot be done with integral percentage interface. So far I did not formulated a way to do this which I would like, and since nobody looked for a good solution because after ntpd was fixed, the severity of the issue seems very low. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ Signals delivered on unhandled Page Faults Contact: Konstantin Belousov By necessity, handling of page faults is separated into two pieces. The first is the architecture-dependent low-level machine exception handler, and the second is the architecture-independent vm_fault() function in sys/vm/vm_fault.c. Machine-dependent code for page faults typically consists of three components: a trampoline written in assembly which creates the C execution environment and gathers hardware-supplied data about page fault reason, a trap() function which is common C-level entry point to dispatch all exceptions processing, and the trap_pfault() C function to specifically handle page faults. trap_pfault() calls vm_fault() when help from VM is needed to resolve the situation. If the fault was handled either by trap()/trap_pfault() or vm_fault(), the faulting instruction is restarted. If fault cannot be handled for any reason, the next action depends on the mode in which the fault occured. If it was in kernel, and the kernel installed a helper, then the helper is called instead of returning to the faulting instruction. Otherwise, a kernel-mode page fault causes a panic. If the unhandled fault occured in usermode, then all Unixes send a signal to the thread whose execution caused the exception. POSIX (or Single Unix Specification) lists several cases where a signal should be sent, and specifies the signal number and si_code from siginfo that must be supplied with the signal. Unfortunately, FreeBSD was rather non-compliant in this regard. A long time ago, to improve compliance, we changed the signal sent on access to a page with permissions incompatible with the access mode. That caused multiple problems with garbage collection (GC)-based runtimes which incorporated knowledge of the FreeBSD quirks, so the machdep.prot_fault_translation sysctl knob was added. More cases of incompatibility were identified recently. Part of the problem is that code to calculate the signal and si_code from the Mach error returned by vm_fault() was located in trap_pfault(). In other words, each architecture did that on its own, with a specific set of bugs and non-compliance. Sometimes code even mis-interpreted returned Mach errors as errno. A new helper function vm_fault_trap() was added, that does several things for trap handlers: it creates ktrace points for faults, calls vm_fault(), and then interprets the result in terms of the user-visible error condition, returning precalculated signal number and si_code to the caller. Now trap_pfault() only needs to provide signal numbers for truly machine-dependent errors. For amd64, an example of such a case is a protection key violation. Besides compliance and bug fixes, we also provided some refinements to userspace about the reason of the delivered signal. For instance, on SIGBUS caused by copy-on-write failure due to exceeding swap reservation limit, we provide specific si_code BUS_OOMERR. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ Architectures Updating platform-specific features and bringing in support for new hardware platforms. Broadcom ARM64 SoC support Contact: Michal Stanek Contact: Kornel Duleba Contact: Marcin Wojtas The Semihalf team continued working on FreeBSD support for the Broadcom BCM5871X SoC series BCM5871X are quad-core 64-bit ARMv8 Cortex-A57 communication processors targeted for networking applications such as 10G routers, gateways, control plane processing and NAS. Completed since the last update: * iProc PCIe root complex (internal and external buses): fixes and improvements, * Crypto engine acceleration for IPsec offloading. Todo: * Upstreaming of work. This work is expected to be submitted/merged to HEAD in the Q4 of 2019. This project was sponsored by Juniper Networks, Inc. __________________________________________________________________ FreeBSD support for the forthcoming Arm Morello CPU, SoC, and board Contact: Robert Watson Contact: Andrew Turner Contact: Brooks Davis In September 2019, Arm announced Morello, an experimental multicore superscalar ARMv8-A CPU, SoC, and prototype board extended to implement the CHERI protection model. Morello will become available in 2021. More details can be found in Arm's blog, a Light Blue Touchpaper blog, and the main CHERI website. We have updated CheriBSD, a CHERI-adapted version of FreeBSD originally targeted at the MIPS-based SRI/Cambridge CHERI processor prototype, to support the current draft architecture. This includes full userspace spatial and referential memory safety CheriABI, as well as backwards compatibility to support running existing ARMv8-A userspace binaries. We will continue to update CheriBSD/Morello as the ISA is finalised. Will also closely track the public CheriBSD/MIPS trunk, picking up new software features utilizing CHERI as they mature, as well as to pick up changes from the baseline FreeBSD development trunk. We currently anticipate releasing CheriBSD/Morello as open source once the ISA and toolchain are published in 2020. Sponsors: DARPA, AFRL __________________________________________________________________ FreeBSD/powerpc Project Links Status of FreeBSD ports on PowerPC URL: https://wiki.freebsd.org/powerpc/ports Status of FreeBSD ports on PowerPC built using gcc URL: https://wiki.freebsd.org/powerpc/ports/PortsOnGcc Status of FreeBSD ports on PowerPC built using clang URL: https://wiki.freebsd.org/powerpc/ports/PortsOnClang Bringing LLVM to PowerPC64 target, using OpenPower ELF v2 ABI URL: https://wiki.freebsd.org/powerpc/llvm-elfv2 Contact: Mark Linimon (ports) Contact: Justin Hibbits (src) Contact: Piotr Kubaj (ports) The FreeBSD/powerpc project continues to make great strides. However, as we discovered at BSDCan 2019, we have not done a great job of letting people know. Key points: * powerpc64 src on recent hardware has been production-quality for over a year now. * powerpc64 ports has been developer-quality for over 18 months now. The main targeted platforms: * powerpc64 on IBM POWER8 and POWER9 chips (the most recent available). This is the primary focus going forward. FreeBSD 12 is required; FreeBSD 13 is recommended. * powerpc64 on older Apple Power Macs, on a continuing but secondary basis. Any FreeBSD version should work. * powerpc64 on x5000. However, this is still developer-only quality. FreeBSD 13 is recommended. * powerpcspe on Amiga A1222. However, this is still developer-only quality. FreeBSD 13 is recommended. The software status: * powerpc*/12 and earlier still remain on the antiquated gcc4.2.1 in base. * powerpc*/13 will soon be switched to llvm90 in base. A great deal of work has been undertaken to ensure as few regressions as possible. Once that switch has occurred (see llvm-elfv2 link above), powerpc*/13 support on gcc4.2.1 will no longer be a priority. * FreeBSD.org package sets are available for powerpc64/12 (quarterly) and powerpc64/13 (default) through the usual method. * Firefox works on powerpc64 using experimental (not-yet committed) patches, * As of the most recent package build on powerpc64/13 (still gcc4.2.1), the following statistics apply: Queued Built Failed Skipped Ignored 33306 30514 245 1686 861 * More ports fixes are being committed every day. Also, Piotr would like to thank the FreeBSD Foundation for funding his personal Talos, and Raptor (via its IntegriCloud subsidiary) for loaning a server on which talos.anongoth.pl runs. The team would like to thank IBM for the loan of two POWER8 machines, and Oregon State University (OSU) for providing the hosting. As well, we would like to thank the clusteradm team for keeping the Tyan POWER8 machines online that are hosted at NYI. __________________________________________________________________ NXP ARM64 SoC support Contact: Marcin Wojtas Contact: Artur Rojek The Semihalf team initiated working on FreeBSD support for the NXP LS1046A SoC LS1046A are quad-core 64-bit ARMv8 Cortex-A72 processors with integrated packet processing acceleration and high speed peripherals including 10 Gb Ethernet, PCIe 3.0, SATA 3.0 and USB 3.0 for a wide range of networking, storage, security and industrial applications. Completed since the last update: * DPAA Network interface support * SD/MMC * USB3.0 * I2C * GPIO In progress: * QSPI * Network performance improvements Todo: * Upstreaming of developed features. This work is expected to be submitted/merged to HEAD in the Q4 of 2019. This project was sponsored by Alstom Group. __________________________________________________________________ Userland Programs Changes affecting the base system and programs in it. gets(3) retirement Contact: Ed Maste gets is an obsolete C library routine for reading a string from standard input. It was removed from the C standard as of C11 because there was no way to use it safely. Prompted by a comment during Paul Vixie's talk at vBSDCon 2017 I started investigating what it would take to remove gets from libc. The patch was posted to Phabricator and refined several times, and the portmgr team performed several exp-runs to identify ports broken by the removal. Symbol versioning is used to preserve binary compatibility for existing software that uses gets. The change was committed in September, and will be in FreeBSD 13.0. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ Ports Changes affecting the Ports Collection, whether sweeping changes that touch most of the tree, or individual ports themselves. FreshPorts Links FreshPorts URL: https://www.FreshPorts.org/ git_proc_commit code URL: https://github.com/FreshPorts/git_proc_commit Things you didn't know FreshPorts can do URL: https://news.freshports.org/2019/09/03/things-you-didnt-know-freshports-can-do/ Contact: Dan Langille FreshPorts consolidates commits into an easy-to-follow format so you can track changes to your favorite ports. It also processes src, doc, and www commit. FreshPorts parses incoming emails and refreshes the database with what it finds. In early September I started looking at how FreshPorts could use a git repository for processing commits. The result was an approach for identifying new commits and for iterating through them. The next idea was commit hooks but the most interesting bit of that exercise was commit iteration. At the EuroBSDCon 2019 FreeBSD Developer summit, I wrote up a small requirements section and then received great help from two sources: * Jan-Piet MENS recommended a Python library and it turned out to be great * Sergey Kozlov wrote Python code to create xml using that Python library On the trip home, I was able to get the code to parse a git commit into XML and loaded into a FreshPorts database. Returning home from the conference, I created a new FreshPorts instance for processing git based on the above. The website has needed no changes related to git. The remaining tasks: * automate the script (git pull, etc) * detect new commits (for later iteration) * design a light-weight git hook Event: EuroBSDCon 2019 Hackathon Sponsored by: Intel Corporation (work done by Sergey Kozlov) __________________________________________________________________ Java on FreeBSD Links OpenJDK 11 repository at FreeBSD GitHub URL: https://github.com/freebsd/openjdk-jdk11u Contact: Greg Lewis Over the last few quarters there has been considerable work in improving support for Java 11 and higher, with some work being backported to Java 8. Starting with the initial release in March on amd64, over the intervening months FreeBSD support was added for features such as: * Java Flight Recorder * HotSpot Serviceability Agent * HotSpot Debugger * DTrace * Javac Server * Java Sound * SCTP In addition, support for the aarch64, i386 and powerpc64 architectures was also added. With most features supported, attention turned to compliance, using the internal JDK test suite as a method of measuring this. Most work during the third quarter has focused on this, with test failures dropping from 50+ failures to only 2 tier1 test failures (which don't appear to impact functionality at all). Some significant fixes include: * A stack overflow crash * Errors in external process handling * The unpack200 utility (on little endian systems) * HotSpot Debugger functionality such as thread enumeration * Networking functionality Finally, this work has been forward ported to Java 12 and 13, with FreeBSD gaining support for these versions on or just after the day of release. Note that this work has been a collaboration with many others. While there are too many contributors to list here (please take a look at the commit history of the GitHub repository), I'd like to recognise Kurt Miller of OpenBSD for his tireless work as a co-collaborator on Java for BSD through many years. I'm also grateful to the sponsorship of the FreeBSD Foundation, which has allowed me to focus on this work for Q3. This project was sponsored by FreeBSD Foundation. __________________________________________________________________ KDE on FreeBSD Links KDE FreeBSD URL: https://freebsd.kde.org/ KDE Community FreeBSD URL: https://community.kde.org/FreeBSD Contact: Adriaan de Groot The KDE on FreeBSD project packages the software produced by the KDE Community for FreeBSD. The software includes a full desktop environment, the art application https://kdenlive.org and hundreds of other applications that can be used on any FreeBSD desktop machine. Along with KDE itself, the team maintains the Qt libraries, the CMake build system, and a handful of other C++ libraries used in the KDE stack. The upstream releases KDE Frameworks (libraries) on a monthly cycle, KDE Plasma (the desktop environment) quarterly and a collection of KDE Applications (usable everywhere) twice a year. The KDE on FreeBSD team chased a dozen updates to these components so that FreeBSD remains one of the most up-to-date systems with KDE software (and Qt). A large (and possibly breaking, still needs more investigation) change came with the release to KDE's Digikam 6.3.0, which stopped using its previous plugins system (the "old" plugins are still used by other KDE software). A new entry in the net-im category was added for Ruqola, a Rocket.chat client for rich instant-messaging. CMake was updated twice. This forces the rebuild of several thousand C++-based ports. The KDE on FreeBSD team then fixes those ports, regardless of whether the error is in the CMake update, or in the upstream code. These updates tend to take a large amount of effort, since they touch unfamiliar (and often very-very-legacy) code. The open bugs list grew to 24 this quarter. It tends to hover around 20 items as new things come in and old ones are resolved. We welcome detailed bug reports and patches. KDE packaging updates are prepared in a copy of the ports repository on GitHub and then merged in SVN. We welcome pull requests there as well. __________________________________________________________________ Ports Collection Links About FreeBSD Ports URL: https://www.FreeBSD.org/ports/ Contributing to Ports URL: https://www.freebsd.org/doc/en_US.ISO8859-1/articles/contributing/ports-contributing.html FreeBSD Ports Monitoring URL: http://portsmon.freebsd.org/index.html Ports Management Team URL: https://www.freebsd.org/portmgr/index.html Contact: René Ladan Contact: FreeBSD Ports Management Team The FreeBSD Ports Management Team, tasked with overseeing the Ports Tree and its committers, reports that the following events happened during 2019Q3: The number of ports grew to just under 38,000 during the last quarter. We have just over 2,000 open ports PRs, of which 400 are unassigned. In this period, 169 committers made 7,340 commits to HEAD and 561 commits to the quarterly branch. This shows that the trend of last quarter of increased activity is continuing. During the last quarter, we welcomed Santhosh Raju (fox@) and Dmitri Goutnik (dmgk@) to the team, and said goodbye to gabor@. During the last quarter, feld@ decided to step down from the portmgr@ and ports-secteam@ teams. We would like to thank them for their past services. In the last three months, bapt@ put on his engineering hat and released a new version of pkg (1.12), added support for overlays to the Ports Tree, fixed two Make targets (deinstall-depends and reinstall), and cleaned up bsd.sites.mk. On the infrastructure side, USES=pure became obsolete and has therefore been removed, and two new USES, xorg and xorg-cat have been added and replace the old bsd.xorg.mk. Two new keywords, ldconfig and ldconfig-linux, were added to simplify formatting of package lists. A number of default versions changed: Lazarus to 2.0.4, Linux to CentOS 7, LLVM to 9.0, Perl to 5.30, PostgreSQL to 11, and Ruby to 2.6. Of the big user-visible ports, Firefox got updated to 69.0.1, Firefox-esr to 68.1.0, Chromium to 76.0.3809.132, and Xfce to 4.14. During the last quarter, antoine@ ran 48 exp-runs to test package updates, test updating bsd.java.mk, test the new ldconfig and ldconfig-linux keywords, test default version updates, test the new xorg and xorg-cat USES, test base updates, and test various improvements to Go and Ruby. __________________________________________________________________ XFCE 4.14 update Links XFCE 4.14 announce URL: https://xfce.org/about/news/?post=1565568000 Contact: Guido Falsi On September 19th the XFCE desktop environment ports have been updated to the recently released XFCE 4.14 version. These updates include upgrades of all the main XFCE components to the latest versions which have been migrated to GTK3, with few exceptions. Base components still require and link to GTK2 in addition to GTK3 to allow older GTK2 XFCE applications, for example panel plugins, to keep working. Due to this change the gtk-xfce-engine theme is deprecated since it only supports GTK2. The XFCE metaport by default installs the greybird theme, but it is not enabled by default. This new version also includes now it's own xfce4-screensaver program which is installed by default. Finally the default Display Manager on which XFCE depends has been changed to LightDM. Some regressions were reported in bugzilla. In particular the one affecting most users is a regression in the window manager: on specific graphic display hardware the window manager fails to properly draw window decorations, which appear black and blocky on affected systems. This problem has also been reported in the upstream bug tracker and a solution is being sought. If anyone is experiencing this display glitch and is able to test, please contact xfce@freebsd.org to help trying to figure out a solution. __________________________________________________________________ Third-Party Projects Many projects build upon FreeBSD or incorporate components of FreeBSD into their project. As these projects may be of interest to the broader FreeBSD community, we sometimes include brief updates submitted by these projects in our quarterly report. The FreeBSD project makes no representation as to the accuracy or veracity of any claims in these submissions. ClonOS: virtualization platform on top of FreeBSD Operating System Links ClonOS 19.09 URL: https://clonos.tekroutine.com/download.html CBSD URL: https://www.bsdstore.ru/ Contact: Oleg Ginzburg What is ClonOS? ClonOS is a turnkey open-source platform based on FreeBSD and the CBSD framework. ClonOS offers a complete web UI for an easy control, deployment and management of FreeBSD jails containers and bhyve/Xen hypervisor virtual environments. ClonOS is currently the only available platforms which allow both Xen and bhyve hypervisors to coexist on the same host. Since ClonOS is a FreeBSD-based platform, it has the ability to create and manage jails natively, allowing you to run FreeBSD applications without losing performance. ClonOS/CBSD 2019Q3 Status Report * Added support for cloud-init (Linux/BSD VMs) and cloudbase-init (Windows VMs). It gives the ability to use FreeBSD as IaaS platform for instant deployment and usage of virtual machines based on bhyve hypervisor. * Project started to use own cloud images for better stability and resiliency. * New mirrors in France, US and Canada were added for distributing ISO/cloud-init images in addition to Russia, Latvia and Ukraine. * Now we're using Jenkins CI for testing regular ClonOS builds: Update clonos packages (Thanks to Daniel Shafer) * New pkg repository was deployed to support ClonOS installation from packages (at this moment only FreeBSD-12 packages are available) ClonOS package repo (Thanks to Daniel Shafer) Open issues and tasks: * Volunteers, contributors and testers are urgently needed to distribute ClonOS on FreeBSD environments. * We'd like to expand our mirrors number geographically, your help and contribution will be much appriciated. * We're urgently looking for hosting sponsorship for various developing and testing activities. __________________________________________________________________ ENA FreeBSD Driver Update Links ENA README URL: https://github.com/amzn/amzn-drivers/blob/master/kernel/fbsd/ena/README Contact: Michal Krawczyk Contact: Maciej Bielski Contact: Marcin Wojtas ENA (Elastic Network Adapter) is the smart NIC available in the virtualized environment of Amazon Web Services (AWS). The ENA driver supports multiple transmit and receive queues and can handle up to 100 Gb/s of network traffic, depending on the instance type on which it is used. ENAv2 has been under development for FreeBSD, similar to Linux and DPDK. Since the last update internal review and improvements of the patches were done, followed by validation on various AWS instances. Completed since the last update: * Verification and review of the NETMAP support * Mapping of the memory as WC on A1 instances in order to enable LLQ mode Todo: * Upstream of NETMAP support * Upstream of the fix for LLQ mode on A1 instances Amazon.com Inc __________________________________________________________________ Nomad pot driver - Orchestrating jails via nomad Links Nomad pot driver URL: https://github.com/trivago/nomad-pot-driver Pot project URL: https://github.com/pizzamig/pot Contact: Luca Pizzamiglio Contact: Esteban Barrios An experimental project has started to provide jail orchestration based on nomad and the jail framework pot, similarly to how orchestration works with docker. This model allows us to split the jail creation and the jail deployment. Jail images can be created and exported using the pot framework. The images can be deployed and orchestrated using nomad. A driver has been developed to allow nomad to interact with pot. One of the goals of this project is to use non-persistent jails as containers, allowing us to: * define containers similar to Docker (but not identical, because the underlaying OS is different) * identify potential missing features in FreeBSD to support such a computational model In the next quarter, we will launch the first service based on this project. Next steps are: * provide more guides and howtos * improve stability, extending the tests suite * improving tooling to create/manage images This project was sponsored by trivago N.V.. __________________________________________________________________ sysctlinfo Links gitlab.com/alfix/sysctlinfo URL: https://gitlab.com/alfix/sysctlinfo Contact: Alfonso Sabato Siciliano The FreeBSD kernel maintains a Management Information Base (MIB) where a component (object) represents a parameter of the system. The sysctl system call explores the MIB to find an object by its Object Identifier (OID) and calls its handler to get or set the value of the parameter. It is often necessary to find an object not to call its handler but to get its info (description, type, name, next object, etc.), so the kernel provides an undocumented interface implemented in kern_sysctl.c. sysctlinfo is a new interface to explore the sysctl MIB and to pass the info of an object to the userland. The project provides: a README, a manual, helper macros, examples, tests and converted tools. Primarily sysctlinfo provides a new set of sysctl nodes (corresponding to the kernel interface) to handle an object up to CTL_MAXNAME levels: sysctl.entryfakename, sysctl.entrydesc, sysctl.entrylabel, sysctl.entrykind, sysctl.entryfmt and sysctl.entrynextleaf. Moreover new features have been implemented: the support for the capability mode, sysctl.entryname, sysctl.entryidbyname and sysctl.entrynextnode. To get all the info about an object the kernel needs to find it many times, then the new sysctl.entryallinfo* nodes were written, they are 30% more efficient. Finally, *byname nodes were added: they search the object by its name avoiding to call sysctl.name2oid (or similar) to explore the MIB just to find the corresponding OID. sysctlinfo can be installed via sysutils/sysctlinfo-kmod or by applying the sysctlinfo.diff patch (more efficient than the module because uses a shared lock). The interface is used by deskutils/sysctlview 1.5, sysutils/nsysctl 1.2 and the converted version of sysctl(8), sysctlbyname(3), sysctlnametomib(3), they should be used to get the value of an object with 23/24 levels or if some level-name has only the '\0' character. In the future a new byname node will be added to allow sysctlbyname() to manage a CTLTYPE_NODE with a no-NULL handler, example sysctlbyname("kern.proc.pid.\"). __________________________________________________________________ From owner-freebsd-hackers@freebsd.org Tue Nov 26 13:53:05 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7B0341B6E97 for ; Tue, 26 Nov 2019 13:53:05 +0000 (UTC) (envelope-from salvadore@FreeBSD.org) Received: from MailHost (unknown [80.211.33.142]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47MljD5wwNz3M1R for ; Tue, 26 Nov 2019 13:53:04 +0000 (UTC) (envelope-from salvadore@FreeBSD.org) Received: from root (uid 0) (envelope-from salvadore@FreeBSD.org) id 229c by MailHost (DragonFly Mail Agent v0.11+); Tue, 26 Nov 2019 14:52:33 +0100 To: freebsd-hackers@FreeBSD.org Subject: FreeBSD Quarterly Status Report - Third Quarter 2019 Cc: freebsd-stable@FreeBSD.org Date: Tue, 26 Nov 2019 14:52:33 +0100 Message-Id: <5ddd2e21.229c.5acaaa0a@MailHost> From: X-Rspamd-Queue-Id: 47MljD5wwNz3M1R X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-1.80 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-0.91)[-0.912,0]; ASN(0.00)[asn:31034, ipnet:80.211.0.0/17, country:IT]; NEURAL_HAM_LONG(-0.89)[-0.885,0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2019 13:53:05 -0000 FreeBSD Project Quarterly Status Report - Third Quarter 2019 Here is the third quarterly status report for 2019. This quarter the reports team has been more active than usual thanks to a better organization: calls for reports and reminders have been sent regularly, reports have been reviewed and merged quickly (I would like to thank debdrup@ in particular for his reviewing work). Efficiency could still be improved with the help of our community. In particular, the quarterly team has found that many reports have arrived in the last days before the deadline or even after. I would like to invite the community to follow the guidelines below that can help us sending out the reports sooner. Starting from next quarter, all quarterly status reports will be prepared the last month of the quarter itself, instead of the first month after the quarter's end. This means that deadlines for submitting reports will be the 1st of January, April, July and October. Next quarter will then be a short one, covering the months of November and December only and the report will probably be out in mid January. -- Lorenzo Salvadore __________________________________________________________________ FreeBSD Team Reports * Cluster Administration Team * Continuous Integration * FreeBSD Core Team * FreeBSD Foundation * FreeBSD Graphics Team status report * FreeBSD Release Engineering Team * FreeBSD Security Team Projects * FAT / msdosfs support for makefs(8) * FUSE * Google Summer of Code 2019 * GSoC'19 Project - MAC policy on IP addresses in Jail: mac_ipacl * Improving laptop support * NFS Version 4.2 implementation * Rockchip RK3399 SoC's eMMC support * syzkaller on FreeBSD * TPM2 Software Stack (TSS2) Kernel * Casueword(9) livelock * Kernel Mapping Protections * Kernel ZLIB Update * PROT_MAX mmap/mprotect maximum protections API * Randomized Top of Stack pointer * Signals delivered on unhandled Page Faults Architectures * Broadcom ARM64 SoC support * FreeBSD support for the forthcoming Arm Morello CPU, SoC, and board * FreeBSD/powerpc Project * NXP ARM64 SoC support Userland Programs * gets(3) retirement Ports * FreshPorts * Java on FreeBSD * KDE on FreeBSD * Ports Collection * XFCE 4.14 update Third-Party Projects * ClonOS: virtualization platform on top of FreeBSD Operating System * ENA FreeBSD Driver Update * Nomad pot driver - Orchestrating jails via nomad * sysctlinfo __________________________________________________________________ FreeBSD Team Reports Entries from the various official and semi-official teams, as found in the Administration Page. Cluster Administration Team Contact: Cluster Administration Team The FreeBSD Cluster Administration Team consists of the people responsible for administering the machines that the Project relies on for its distributed work and communications to be synchronised. In this quarter, the team has worked on the following: * Change IPv6 address in TWN site. * Solved hardware issues in KWC site (with hrs@). * Moved remaining infrastructure from the YSV (Yahoo!) site to NYI (New York Internet) (peter@). * YSV hosted most of FreeBSD.org between 2000 and 2019. Installed new machines for portmgr@ courtesy of the FreeBSD Foundation. Resolved outtages (thanks uqs@) with GitHub exporter, Bugzilla and hg-beta (thanks bapt@). PowerPC64 servers are online (power8) building pkgs and reference hosts. Ongoing systems administration work: * Creating accounts for new committers. * Backups of critical infrastructure. * Keeping up with security updates in 3rd party software. Work in progress: * Review the service jails and service administrators operation. * South Africa Mirror (JINX) in progress. * NVME issues on PowerPC64 Power9 blocking dual socket machine from being used as pkg builder. * Drive upgrade test for pkg builders (SSDs) courtesy of the FreeBSD Foundation. * Boot issues with Aarch64 reference machines. * New NYI.net sponsored colocation space in Chicago-land area. * Setup new host for CI staging environment. __________________________________________________________________ Continuous Integration Links FreeBSD Jenkins Instance URL: https://ci.FreeBSD.org FreeBSD CI artifact archive URL: https://artifact.ci.FreeBSD.org/ FreeBSD Jenkins wiki URL: https://wiki.freebsd.org/Jenkins freebsd-testing Mailing List URL: https://lists.FreeBSD.org/mailman/listinfo/freebsd-testing FreeBSD CI Repository URL: https://github.com/freebsd/freebsd-ci Tickets related to freebsd-testing@ URL: https://preview.tinyurl.com/y9maauwg Hosted CI wiki URL: https://wiki.freebsd.org/HostedCI FreeBSD CI weekly report URL: https://hackmd.io/@FreeBSD-CI Contact: Jenkins Admin Contact: Li-Wen Hsu The FreeBSD CI team maintains continuous integration system and related tasks for the FreeBSD project. The CI system regularly checks the committed changes can be successfully built, then performs various tests and analysis of the results. The results from build jobs are archived in an artifact server, for the further testing and debugging needs. The CI team members examine the failing builds and unstable tests, and work with the experts in that area to fix the code or adjust test infrastructure. The details are of these efforts are available in the weekly CI reports. We had a testing working group at the 201909 DevSummit lwhsu@ has presented the Testing/CI project status and "how to work with the FreeBSD CI system", slides are available at the DevSummit page. Some contents have been migrated to https://wiki.freebsd.org/Jenkins/Debug , extending is welcomed. We continue publishing CI Weekly Report and moved the archive to https://hackmd.io/@FreeBSD-CI Work in progress: * Collecting and sorting CI tasks and ideas at https://hackmd.io/bWCGgdDFTTK_FG0X7J1Vmg * Setup the CI stage environment and put the experimental jobs on it * Extending and publishing the embedded boards testbed * Implementing automatic tests on bare metal hardware * Adding drm ports building test against -CURRENT * Testing and merging pull requests at https://github.com/freebsd/freebsd-ci/pulls * Planning for running ztest and network stack tests * Help more 3rd software get CI on FreeBSD through a hosted CI solution Please see freebsd-testing@ related tickets for more WIP information. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ FreeBSD Core Team Contact: FreeBSD Core Team The FreeBSD Core Team is the governing body of FreeBSD. * Core has provisionally accepted the BSD+patent license for use in some cases. The Core Team must approve the import of new BSD+Patent licensed components or the change of license of existing components to the BSD+Patent License. https://opensource.org/licenses/BSDplusPatent * Kernel Pseudo Random Number Generator (PRNG) maintainership was updated to reduce the contribution barrier for committers who have demonstrated competence in this part of the tree. * Core approved a source commit bit for Pawel/ Biernacki. Konstantin Belousov will mentor Pawel/ and Mateusz Guzik will be co-mentor. * The Core-initiated Git Transition Working Group met over the last quarter, however a report is still forthcoming. Discussions will continue in the fourth quarter of 2019. There are many issues to resolve including how to deal with contrib/, whether to re-generate hashes in the current Git repository, and how to best implement commit testing. __________________________________________________________________ FreeBSD Foundation Contact: Deb Goodkin The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the FreeBSD Project and community worldwide. Funding comes from individual and corporate donations and is used to fund and manage software development projects, conferences and developer summits, and provide travel grants to FreeBSD contributors. The Foundation purchases and supports hardware to improve and maintain FreeBSD infrastructure and provides resources to improve security and quality assurance efforts; publishes marketing material to promote, educate, and advocate for the FreeBSD Project; facilitates collaboration between commercial vendors and FreeBSD developers; and finally, represents the FreeBSD Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity. Here are some highlights of what we did to help FreeBSD last quarter: Partnerships and Commercial User Support We help facilitate collaboration between commercial users and FreeBSD developers. We also meet with companies to discuss their needs and bring that information back to the Project. In Q3, Ed Maste and Deb Goodkin met with a few commercial users in the US. It is not only beneficial for the above, but it also helps us understand some of the applications where FreeBSD is used. We were also able to meet with a good number of commercial users at vBSDCon and EuroBSDCon. These venues provide an excellent opportunity to meet with commercial and individual users and contributors to FreeBSD. Fundraising Efforts Our work is 100% funded by your donations. We are continuing to work hard to get more commercial users to give back to help us continue our work supporting FreeBSD. More importantly, we'd like to thank our individual donors for making $10-$1,000 donations last quarter, for more than $16,000! Please consider making a donation to help us continue and increase our support for FreeBSD! We also have the Partnership Program, to provide more benefits for our larger commercial donors. Find out more information at www.FreeBSDfoundation.org/FreeBSD-foundation-partnership-program/ and share with your companies. OS Improvements The Foundation supports software development projects to improve the FreeBSD operating system through our full time technical staff, contractors, and project grant recipients. They maintain and improve critical kernel subsystems, add new features and functionality, and fix problems. Over the last quarter there were 345 commits to the FreeBSD base system repository sponsored by the FreeBSD Foundation - this represents about one fifth of all commits during this period. Many of these projects have their own entries in this quarterly report (and are not repeated here). Foundation staff member Konstantin Belousov committed many improvements to multiple kernel subsystems, as well as low-level 32-bit and 64-bit x86 infrastructure. These included fixes for robust mutexes, unionfs, the out of memory (OOM) handler, and per-cpu allocators. Additional work included fixes for security issues and introduction and maintenance of vulnerability mitigations, and improving POSIX conformance. Ed Maste committed a number of minor security bug fixes and improvements, as well as the first iteration of a tool for editing the mitigation control ELF note. Additional work included effort on build infrastructure and the tool chain. Clang's integrated assembler (IAS) is now used more widely, as part of the path to retiring the assembler from GNU binutils 2.17.50. The readelf tool now decodes some additional ELF note information. Ed also enabled the Linuxulator (Linux binary support layer) on arm64, and added a trivial implementation of the renameat2 system call (handling common options). Mark Johnston added Capsicum support to a number of ELF Tool Chain utilities, and committed a number of other Capsicum kernel and userland fixes. Mark worked on a number of changes related to security improvements, including integration and support of the Syzkaller automated system call fuzzer, and fixing issues identified by Syzkaller. Other changes included addressing failures caused by refcount wraparound, improvements to the prot_max memory protection. Other work included NUMA, locking, kernel debugging, RISC-V and arm64 kernel improvements. Edward Napierala continued working on Linuxulator improvements over the quarter. The primary focus continued to be tool improvements - strace is now more usable for diagnosing issues with Linux binaries running under the Linuxulator. That said, as with previous work a number of issues have been fixed along the way. These are generally minor issues with a large impact - for example, every binary linked against up-to-date glibc previously segfaulted on startup. This is now fixed. Continuous Integration and Quality Assurance The Foundation provides a full-time staff member who is working on improving our automated testing, continuous integration, and overall quality assurance efforts. During the third quarter of 2019, Foundation staff continued to improve the project's CI infrastructure, worked with contributors to fix the failing build and test cases, and worked with other teams in the Project for their testing needs. We added several new CI jobs and worked on getting the hardware regression testing lab ready. Li-Wen Hsu gave presentations "Testing/CI status update" and "How to work with the FreeBSD CI system" at the 201909 DevSummit. Slides are available at the DevSummit page. We continue publishing the CI weekly report on the freebsd-testing@. mailing list, and an archive is available. See the FreeBSD CI section of this report for completed work items and detailed information. Supporting FreeBSD Infrastructure The Foundation provides hardware and support to improve the FreeBSD infrastructure. Last quarter, we continued supporting FreeBSD hardware located around the world. FreeBSD Advocacy and Education A large part of our efforts are dedicated to advocating for the Project. This includes promoting work being done by others with FreeBSD; producing advocacy literature to teach people about FreeBSD and help make the path to starting using FreeBSD or contributing to the Project easier; and attending and getting other FreeBSD contributors to volunteer to run FreeBSD events, staff FreeBSD tables, and give FreeBSD presentations. The FreeBSD Foundation sponsors many conferences, events, and summits around the globe. These events can be BSD-related, open source, or technology events geared towards underrepresented groups. We support the FreeBSD-focused events to help provide a venue for sharing knowledge, to work together on projects, and to facilitate collaboration between developers and commercial users. This all helps provide a healthy ecosystem. We support the non-FreeBSD events to promote and raise awareness of FreeBSD, to increase the use of FreeBSD in different applications, and to recruit more contributors to the Project. Check out some of the advocacy and education work we did last quarter: * Sponsored USENIX 2019 Annual Technical Conference as an Industry Partner * Represented FreeBSD at OSCON 2019 in Portland, OR * Represented FreeBSD at COSCUP 2019 in Taiwan * Presented at the Open Source Summit, North American in San Diego, CA * Executive Director Deb Goodkin was interviewed by TFiR https://www.freebsdfoundation.org/news-and-events/latest-news/tfir-interview-freebsd-meets-linux-at-the-open-source-summit/ * Sponsored FreeBSD Hackathon at vBSDcon 2019 in Reston, VA * Sponsored the attendee bags and attended vBSDcon 2019 in Reston VA * Represented FreeBSD at APNIC-48 in Chiang Mai, Thailand * Represented FreeBSD at MNNOG-1 in Ulaanbaatar, Mongolia * Served as an administrator for the Project's Google Summer of Code Session. See the Google Summer of Code section of this report for more information. * Sponsored FreeBSD Developers Summit at EuroBSDCon in Lillehammer, Norway * Sponsored and attended EuroBSDcon 2019 in Lillehammer, Norway * Applied and was accepted for a FreeBSD Miniconf at linux.conf.au, in Gold Coast, Australia, Jan 14, 2020 * Our FreeBSD talk was accepted at seaGL, Seattle, WA, November 15 and 16. We continued producing FreeBSD advocacy material to help people promote FreeBSD. Learn more about our recent efforts to advocate for FreeBSD around the world: https://www.freebsdfoundation.org/blog/freebsd-around-the-world/ Our Faces of FreeBSD series is back. Check out the latest post: Roller Angel. Read more about our conference adventures in the conference recaps and trip reports in our monthly newsletters: https://www.freebsdfoundation.org/news-and-events/newsletter/ We help educate the world about FreeBSD by publishing the professionally produced FreeBSD Journal. As we mentioned previously, the FreeBSD Journal is now a free publication. Find out more and access the latest issues at https://www.FreeBSDfoundation.org/journal/. You can find out more about events we attended and upcoming events. We opened our official FreeBSD Swag Store. Get stickers, shirts, mugs and more at ShopFreeBSD. We have continued our work with a new website developer to help us improve our website. Work has begun to make it easier for community members to find information and to make the site more efficient. Legal/FreeBSD IP The Foundation owns the FreeBSD trademarks, and it is our responsibility to protect them. We also provide legal support for the core team to investigate questions that arise. Go to http://www.FreeBSDfoundation.org to find out how we support FreeBSD and how we can help you! __________________________________________________________________ FreeBSD Graphics Team status report Links Project GitHub page URL: https://github.com/FreeBSDDesktop Contact: FreeBSD Graphics Team Contact: Niclas Zeising The FreeBSD X11/Graphics team maintains the lower levels of the FreeBSD graphics stack. This includes graphics drivers, graphics libraries such as the MESA OpenGL implementation, the X.org xserver with related libraries and applications, and Wayland with related libraries and applications. During the last period, several changes have been made, but most of them has been behind the scene. We have also worked on general clean up of old xorg ports that have been deprecated upstream. The ports infrastructure for xorg ports and ports that depend on xorg ports have been updated. We have switched USE_XORG and XORG_CAT to use the USES framework, instead of the old way of including bsd.xorg.mk from bsd.port.mk. This infrastructure work has been fairly substantial, and new ports depending on xorg ports should add USES=xorg to their makefiles. As part of this bsd.xorg.mk was split up, and the XORG_CAT part was split out to USES=xorg-cat. This is used for the xorg ports themselves, and sets up a common environment for building all xorg ports. In addition, framework for pulling xorg ports directly from freedesktop.org gitlab was added, which will make improve development and testing, since it makes it possible to create ports of unreleased versions. Further improvements in this area includes framework for using meson instead of autotools for building xorg ports. This is still a work in progress. We have also worked to clean up and deprecate several old xorg ports and libraries. Some of these ports have already been removed, and some are still waiting on removal after a sufficient deprecation period. Most notably amongst the deprecations are x11/libXp, which required to fix several dependencies. Several other old libraries have also been deprecated, such as x11/Xxf86misc, x11-fonts/libXfontcache and graphics/libGLw. Some applications and drivers have also been deprecated during the period. With the remaining removals in this area, we should be up to speed with deprecations upstream. We are currently investigating if there are new software added upstream that we need to port to FreeBSD. We have also continued our regularly scheduled bi-weekly meetings. People who are interested in helping out can find us on the x11@FreeBSD.org mailing list, or on our gitter chat: https://gitter.im/FreeBSDDesktop/Lobby. We are also available in #freebsd-xorg on EFNet. We also have a team area on GitHub where our work repositories can be found: https://github.com/FreeBSDDesktop __________________________________________________________________ FreeBSD Release Engineering Team Links FreeBSD 11.3-RELEASE announcement URL: https://www.freebsd.org/releases/11.3R/announce.html FreeBSD 12.1-RELEASE schedule URL: https://www.freebsd.org/releases/12.1R/schedule.html FreeBSD 12.1-RELEASE BETA/RC builds URL: https://download.freebsd.org/ftp/releases/ISO-IMAGES/12.1/ FreeBSD development snapshots URL: https://download.freebsd.org/ftp/snapshots/ISO-IMAGES/ Contact: FreeBSD Release Engineering Team The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code freezes and maintaining the respective branches, among other things. During the third quarter of 2019, the FreeBSD Release Engineering team finished the 11.3-RELEASE cycle, with the final release build started on July 5th and the official announcement sent on July 9th. FreeBSD 11.3-RELEASE is the fourth release from the stable/11 branch, building on the stability and reliability of 11.2-RELEASE. The FreeBSD Release Engineering Team also started work on the upcoming 12.1-RELEASE, which started September 6th. This release cycle is the first "freeze-less" release from the Subversion repository, and the test bed for eliminating the requirement of a hard code freeze on development branches. Commits to the releng/12.1 branch still require explicit approval from the Release Engineering Team, however. At present, there have been three BETA builds, and so far, two RC builds, with the final 12.1-RELEASE build scheduled for November 4th. Additionally throughout the quarter, several development snapshots builds were released for the head and stable/11 branches; snapshots for stable/12 were released as well although not during the 12.1-RELEASE cycle. Much of this work was sponsored by Rubicon Communications, LLC (Netgate) and the FreeBSD Foundation. __________________________________________________________________ FreeBSD Security Team Links FreeBSD security information URL: https://www.freebsd.org/security/ Contact: Security Team Several members of the security team met at the Vendor Summit in October to formalize team structure dedicated for architecture and crypto engineering in addition to the existing product security incident response function. Since June we have started having fortnightly conference calls to discuss important issues and to collaborate closely on advisories and errata notices in the pipeline. * Security advisories sent out in 2019-Q3: 7 * Errata Notices sent out in 2019-Q3: 5 __________________________________________________________________ Projects Projects that span multiple categories, from the kernel and userspace to the Ports Collection or external projects. FAT / msdosfs support for makefs(8) Contact: Ed Maste In order to streamline the process of creating install or virtual machine system images we needed FAT filesystem support in makefs(8). Makefs was originally developed in NetBSD, and FAT support was added there not much later, but after the tool was ported to FreeBSD. Siva Mahadevan, one of the FreeBSD Foundation's interns from the University of Waterloo, worked on porting FAT support from NetBSD. I rebased and updated Siva's work, and committed it during this quarter. After a few follow-up fixes we are able to build FAT filesystem images without using md(4) and without requiring root. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ FUSE Contact: Alan Somers FUSE (File system in USErspace) allows a userspace program to implement a file system. It is widely used to support out-of-tree file systems like NTFS, as well as for exotic pseudo file systems like sshfs. FreeBSD's fuse driver was added as a GSoC project in 2012. Since that time, it has been largely neglected. The FUSE software is buggy and out-of-date. Our implementation is about 11 years behind. I completed this work during Q3. I fixed a few newly-introduced bugs, fixed a long-standing sendfile bug that affects FUSE ([236466](https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236466)) and merged everything to head and stable/12. Then I fixed the resulting Coverity CIDs. There have been no new FUSE-related bug reports, so I can only assume that everything is working great. Report any problems to asomers@FreeBSD.org. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ Google Summer of Code 2019 Links 2019 Summer of Code Project Wikis URL: https://wiki.freebsd.org/SummerOfCode2019Projects 2019 Summer of Code Projects URL: https://summerofcode.withgoogle.com/archive/2019/organizations/6504969929228288/ Contact: Summer of Code Admins The FreeBSD Project is pleased to have participated in Google Summer of Code 2019 marking our 14th year of participation. This year we had six successful projects: * Dual-stack ping command by Ján Sucan * Firewall test suite by Ahsan Barkati * Kernel sanitizers by Costin Carabas * MAC policy on IP addresses for FreeBSD Jail by Shivank Garg * Separation of ports build process from local installation by Theron Tarigo * Virtual memory compression by Paavo-Einari Kaipila We thank Google for the opportunity to work with these students and hope they continue to work with FreeBSD in the future. This project was sponsored by Google Summer of Code. __________________________________________________________________ GSoC'19 Project - MAC policy on IP addresses in Jail: mac_ipacl Links FreeBSD's Phabricator Differential Link URL: https://reviews.freebsd.org/D20967 Github Diff Link URL: https://github.com/freebsd/freebsd/compare/master...shivankgarg98:shivank_MACPolicyIPAddressJail Project Wiki Page URL: https://wiki.freebsd.org/SummerOfCode2019Projects/MACPolicyIPAddressJail Contact: Shivank Garg About - With the introduction of VNET(9) in FreeBSD, Jails are free to set their IP addresses. However, this privilege may need to be limited by the host as per its need for multiple security reasons. This project uses mac(9) for an access control framework to impose restrictions on FreeBSD jails according to rules defined by the root of the host using sysctl(8). It involves the development of a dynamically loadable kernel module (mac_ipacl) based on The TrustedBSD MAC Framework to implement a security policy for configuring the network stack. This project allows the root of the host to define the policy rules to limit the root of a jail to a set of IP (v4 or v6) addresses and/or subnets for a set of interfaces. Features this new MAC policy module are: * The host can define one or more lists of IP addresses/subnets for the jail to choose from. * The host can restrict the jail from setting certain IP addresses or prefixes (subnets). * The host can restrict this privilege to a few network interfaces. Implementation - The mac_ipacl module is a loadable kernel module. It implements mac checks in netinet/in.c and netinet6/in6.c to check the IP addresses requested by jail. The idea to implement these checks at these places comes from the fact that SIOCAIFADDR (for IPv4) and SIOCAIFADDR_IN6 (for IPv6) ioctl handlers are defined for adding the IP addresses to an interface. This is used by ifconfig (in userspace) for setting the IP address. The MAC Framework acts as multiplexer between the netinet and the module. The requested IP and the credentials are checked with the rules in mac_ipacl and output is returned accordingly to netinet. The module can be tuned with various sysctl and similarly, policy rules are also be defined with sysctl. TestSuite - Test scripts integrated with kyua and ATF are included with the module. Using the module - I have written a man page for the module. Please refer to the mac_ipacl(4) for using the new MAC module and various examples. Final Deliverables - * A loadable kernel module - mac_ipacl in sys/security/mac_ipacl * ATF tests for the module in tests/sys/mac/ipacl * A man page for this new mac module - mac_ipacl.4 in share/man/man4/mac_ipacl.4 This is a new project, developed as part of Google Summer of Code'19 under the guidance of Bjoern A. Zeeb . The module is reviewed and Revision for this project is accepted and ready to land. It is yet to be merged with FreeBSD HEAD, and waiting to be tested by few more hands in the industry. I'll be very thankful if you can give this module a try and share your valuable experience about it. Please be free to share your ideas and feedback on this module and please do not hesitate to send me an email. __________________________________________________________________ Improving laptop support Contact: Ed Maste The FreeBSD Foundation would like to ensure that running FreeBSD on contemporary hardware, including laptops, remains viable. To that end we plan to purchase the latest generation of one or more of a family of laptops preferred by members of the FreeBSD community, evaluate the existing state of hardware support, and implement missing hardware support where possible. As the first laptop for this project we have selected a 7th Generation Lenovo X1 Carbon. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ NFS Version 4.2 implementation Contact: Rick Macklem RFC-7862 describes a new minor revision to the NFS Version 4 protocol. This project implements this new minor revision. The NFS Version 4 Minor Version 2 protocol adds several optional features to NFS, such as support for SEEK_DATA/SEEK_HOLE, file copying done on the server that avoids data transfer over the wire and support for posix_fallocate(), posix_fadvise(). Hopefully these features can improve performance for certain applications. The implementation is now nearing completion and recent work has been mostly testing. A cycle of interoperability testing with Linux has just been completed. The main area that still needs testing is use of the pNFS server with NFSv4.2 and that should start soon. Once testing of pNFS is completed, I believe the code is ready to be incorporated into FreeBSD head/current. Testing by others would be appreciated. The modified kernel can be found at https://svn.freebsd.org/base/projects/nfsv42/sys and should run with a recent FreeBSD head/current system. Client mounts need the "minorversion=2" mount option to enable this protocol. __________________________________________________________________ Rockchip RK3399 SoC's eMMC support Contact: Ganbold Tsagaankhuu The followings features have been added to support RK3399 SoC eMMC on FreeBSD: * Extended simple_mfd driver to expose a syscon interface if that node is also compatible with syscon. For instance, Rockchip RK3399's GRF (General Register Files) is compatible with simple-mfd as well as syscon and has devices like usb2-phy, emmc-phy and pcie-phy etc. under it. * Made Rockchip's General Register Files driver the subclass of Simple MFD driver * Added driver for Rockchip RK3399 eMMC PHY. * Added eMMC support codes for Rockchip RK3399 SoC. * All above was tested on NanoPC-T4 board. __________________________________________________________________ syzkaller on FreeBSD Contact: Andrew Turner Contact: Mark Johnston Contact: Michael Tuexen Contact: Samy Al Bahra See the syzkaller entry in the 2019q1 quarterly report for an introduction to syzkaller. Work has continued on fixing bugs uncovered by syzkaller. Over a dozen kernel bugs fixed in the past three months have been directly attributed to syzkaller, and a number of syzkaller reproducers have been incorporated into our test suite. backtrace.io, via Samy, has graciously provided a large server for running a dedicated syzkaller instance to fuzz FreeBSD under bhyve. Though syzbot, the public syzkaller instance run by Google, already fuzzes FreeBSD, it has proven fruitful to run multiple syzkaller instances: different instances find different bugs, and syzkaller.backtrace.io allows us to experiment with FreeBSD-specific extensions. In particular, this instance currently uploads data about each crash to backtrace.io, making it much easier to triage and analyze crashes. We plan to make this service generally available to FreeBSD developers in the near future. Going forward we will continue to extend syzkaller coverage and make it simpler for users and developers to run private instances, and optionally collect kernel crash information for debugging or for submission. This project was sponsored by backtrace.io, and The FreeBSD Foundation. __________________________________________________________________ TPM2 Software Stack (TSS2) Links tpm2-tss Documentation URL: https://tpm2-tss.readthedocs.io/en/latest/index.html tpm2 Source Repository URL: https://github.com/tpm2-software/ tpm2 mailing list URL: https://lists.01.org/postorius/lists/tpm2.lists.01.org/ tpm2 irc channel URL: ircs://chat.freenode.net:6697/tpm2.0-tss Contact: D Scott Phillips Intel has contributed ports of the TPM2 Software Stack to the ports tree, with the new ports securtity/tpm2-tss, security/tpm2-tools, security/tpm2-abrmd. tpm2-tss contains a set of libraries which expose various TPM2 APIs for using a TPM conforming to the TCG TPM 2.0 specification. tpm2-tools provides a set of command line tools which use the tpm2-tss libraries to perform tpm operations. Finally, tpm2-abrmd is a userspace daemon which acts as a TPM Access Broker and Resource Manager, multiplexing many TPM users onto a single TPM device. Sponsored by: Intel Corporation __________________________________________________________________ Kernel Updates to kernel subsystems/features, driver support, filesystems, and more. Casueword(9) livelock Contact: Konstantin Belousov The Compare-And-Swap (CAS) is one of the fundamental building blocks for hardware-assisted atomic read/modify/write operations. Some architectures support it directly, for instance x86 and sparc. Others provide different building blocks, usually the pair of Load Linked/Store Conditional instructions (ll/sc) which can be used to construct CAS or other atomic operations like Fetch-And-Add or any atomic arithmetic ops using plain arithmetic instructions. An example is the LDXR/STXR pair on AArch64. The ll/sc operation is performed by first using the load linked instruction to load a value from memory and simultaneously mark the cache line for exclusive access. The value is then updated by the store conditional instruction, but only if there were not any writes to the marked cache line. Any write by another CPU makes the store instruction fail. So typically atomic primitives on ll/sc architectures retry the whole operation if only store failed, because it just means that this CPU either lost a race, or even the failure was spurious. This is so called strong version of CAS and atomics. If primitive returns failure instead, the CAS variant is called weak by C standard. For the FreeBSD threading library lock implementation, when the fast path mode in userspace was not possible, the kernel often needs to do a CAS operation on user memory location. In addition to all guarantees of normal CAS, it also must ensure the safety and tolerance to paging. In FreeBSD, the casueword32(9) primitive provides CAS on usermode 32bit words for kernel. Casueword32(9) was implemented as strong CAS, similarly to the mode of operation of hardware CAS instructions on x86. Using the strong implementation for casueword may be dangerous, since the same address is potentially accessible to other, potentially malicious, threads in the same or other processes. If such a thread constantly dirties the cache line used by a ll/sc loop, it could practically force the kernel-mode thread to remain stuck in the loop forever. Since the loop is tight, and it does not check for signals, the thread cannot be stopped or killed. The solution is to make the casueword implementation weak, which means that the interface of the primitive must be changed to allow notifying the caller about spurious failures. Also, it is now the caller's responsibility to retry as appropriate. The change to casueword was made for all architectures. Even on x86, the PSL.ZF value after the CMPXCHG instruction was returned directly for the new casueword. All two dozens uses of the primitive, all located in kern_umtx.c, were inspected and retry added as needed. As a somewhat related note, in-kernel atomic_cmpset(9) operations are strong, while atomic_fcmpset(9) should be weak, unless broken by a specific architecture. The general attitude seems to be that retry is the duty of the primitive's caller. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ Kernel Mapping Protections Contact: Mark Johnston Modern CPU architectures have the ability to flag memory mappings as "no-execute" (NX), which prevents that memory from being executed by a processor. NX mappings are an important security mitigation since they help segregate code and data, blocking unintentional execution of memory whose contents is controlled by an attacker. W^X (write XOR execute) is a security policy which disallows the creation of mappings that are simultaneously writeable and executable. Under this policy, memory whose contents can be modified must be mapped with the NX flag. This policy makes it harder to exploit bugs that permit an attacker to arbitrarily overwrite data. FreeBSD has long made use of the NX flag for userspace mappings whenever possible, but only in the past several years has it been applied to kernel mappings. A recent project has sought to implement a W^X-by-default policy for the amd64 kernel by completing an audit of the remaining executable mappings in the kernel, and making modifications to either apply the NX bit to those mappings or to make them read-only. This work has landed in HEAD and will be available in FreeBSD 13.0 and 12.2. Similar work for other CPU architectures is also planned. To help audit kernel mapping protections, the vm.pmap.kernel_maps sysctl was added; it dumps a snapshot of the kernel's page table entries and their attributes. This way, mappings violating the W^X policy can easily be discovered and investigated, and the sysctl provides information useful to anyone interested in kernel memory layout. With a few rare exceptions, the only kernel mappings which require execute permission are those of the kernel executable itself, and loadable kernel modules. A number of other regions of memory were unnecessarily being mapped without NX, and these were identified and corrected first. To address the kernel code mappings, the amd64 kernel linker script was modified to pad the .text segment to a 2MB boundary. To improve performance, the kernel creates superpage mappings of its .text segment, but this means that any data cohabiting the final 2MB .text mapping is mapped with execute permissions. The padding allows executable code to be segregated from data which follows in the kernel image, avoiding this problem and maintaining the superpage optimization at the expense of some wasted RAM. Enforcing W^X turned out to be somewhat trickier. Unlike other CPU architectures supported by FreeBSD, amd64 kernel modules are linked as relocatable object files, i.e., .o files. (On other architectures, they are dynamically shared objects (DSOs, or .so files), as one might naively expect.) The use of .o files means that amd64 kernel modules contain more efficient code than they would if linked as DSOs, since DSOs inherently make use of certain types of indirection which allow shared libraries to be loaded at arbitrary addresses, and this indirection is useless in the kernel. As part of this project an attempt was made to switch amd64 to using DSOs as well, since the cost of this indirection can largely be mitigated with modern toolchains, but it was found that the use of DSOs would also force a change to the code model used when compiling amd64 kernel code, resulting in a further performance penalty. The main obstacle with the use of .o files is that sections are not page-aligned by default; the segregation of sections with differing mapping protection requirements is done by the static linker when linking a DSO or executable file. Since mapping protections are applied at the granularity of the page size (4KB on amd64), the overlap of sections within a page causes problems since, for example, the end of the read-only .text section may overlap with the beginning of the read-write .data section. One possible solution is to perform any required section reordering and padding at kernel module load time, but this approach breaks debugging tools such as dtrace(1) and kgdb which assume that the kernel linker does not modify the layout of loadable modules. As a result, an amd64 kernel module linker script is now used to insert padding for specific sections. Finally, the kernel linker was modified to restrict mapping protections based on section flags. As a result of all of this, amd64 kernels now boot without any writeable, executable mappings. Though some of the work was architecture-specific, much of it can and will be leveraged to provide the same policy for our other supported architectures. This project was sponsored by Netflix. __________________________________________________________________ Kernel ZLIB Update Contact: Xin Li Contact: Yoshihiro Ota The ZLIB is a compression library is widely used in various software. The FreeBSD system had used an ancient (over 20 year-old) version of zlib (version 1.0.4) and total of 3 versions, one in user-land, one in ZFS, and one in kernel. Xin and Yoshihiro upgraded zlib to the latest and eliminated 2 extra copies. Along the efforts, zlib version was upgraded to 1.2.11, netgraph's ppp is re-implemented to use the standard zlib, and removed unmaintained code. We now only have one and the latest version of zlib in FreeBSD code base, new compress, compress2, and uncompress APIs exposed in the kernel, and importing changes from zlib will be simple. Kernel zlib changes * sys/crc32.h is split to avoid object file name conflict with ZLIB's * contrib/zlib is moved to sys/contrib/zlib * Kernel started switching to sys/contrib/zlib and ZFS copy dropped * Kernel zcalloc is introduced and compress, compress2, and uncompress APIs are exposed to kernel * Removed zlib 1.0.4 from kernel Kernel zlib user updates * kern_ctf.c updated * opencryptodeflate updated * geom_uzip updated * subr_compressor updated * if_mxge updated * bxe updated * ng_deflate updated Legacy code removals * Removed MIPS zlib elf trampoline * Removed kgzip and kgzldr * Removed gzip'ed a.out support * Removed ArmvX elf_trampoline.c __________________________________________________________________ PROT_MAX mmap/mprotect maximum protections API Links PROT_MAX commit URL: https://reviews.freebsd.org/rS349240 Contact: Brooks Davis Unix-like systems provide the mmap(2) system call to allocate memory or map files or devices into memory with specified access protection, and the mprotect(2) system call to change protections on mapped memory. Protection flags specify whether the memory may be read, written, and/or executed (PROT_READ, PROT_WRITE, PROT_EXEC respectively). Traditionally, mprotect(2) can be used to set any desired protections (except that a shared mapping of a file opened read-only cannot have PROT_WRITE set). A new macro PROT_MAX() adds a facility for specifying the maximum possible protection flags for mmap(2) and mprotect(2) calls. The program can then specify whether a mapping may be changed in the future to allow a given access protection. For example, a memory mapping can be set such that it can have read and write protections set, but may never be made executable. Maximum protection values are provided to the PROT_MAX() macro, and are OR'd with regular protection values. This change allows (e.g.) a region that must be writable during run-time linking or JIT code generation to be made permanently read+execute after writes are complete. This complements Write-XOR-Execute (W^X) protections available on other operating systems, allowing more precise control by the programmer. For example, to request memory that cannot be made executable: mmap(NULL, size, PROT_READ | PROT_WRITE | PROT_MAX(PROT_READ | PROT_WRITE), MAP_ANON, -1, 0); and to request memory that may have execute permission enabled later on, but is not currently executable: mmap(NULL, size, PROT_READ | PROT_WRITE | PROT_MAX(PROT_READ | PROT_WRITE | PROT_EXECUTE), MAP_ANON, -1, 0); This change alters mprotect argument checking and returns an error when unhandled protection flags are set. This differs from POSIX (in that POSIX only specifies an error if a valid permission can not be set), but is the documented behavior on Linux and more closely matches historical mmap behavior. In addition to explicit setting of the maximum permissions, an experimental sysctl vm.imply_prot_max causes mmap to assume that the initial permissions requested should be the maximum when the sysctl is set to 1. This behavior is known to break code that uses PROT_NONE reservations before mapping contents into part of the reservation. A later version of this work is expected to provide per-binary and per-process opt-in/out options and this sysctl may be removed in its current form. As such it is undocumented. While these flags are non-portable, they can be used in portable code with simple ifdefs to expand PROT_MAX() to 0. Sponsors: DARPA, AFRL __________________________________________________________________ Randomized Top of Stack pointer Contact: Konstantin Belousov After the ASLR so useful addition, next in the series of the buzzword-compliant checkboxes is the stack addresses randomization. The main userspace thread stack on FreeBSD is traditionally allocated from the top of the user address space and grows down. Besides the initial pointer for the stack on userspace entry, this area also contains structures for program arguments and environment (so called strings), and aux vector data for ELF images. Considering the goal of randomizing the addresses of strings and main thread initial frame, moving the main stack area in the address space is not feasible. It would cause significant ABI breakage, invalidates the knowledge already coded into many introspection tools, and causes unneeded additional fragmentation of the user address space. Instead a typical approach of adding a gap of randomized size between top of stack and a place for strings was used. It is done in a way which preserves the stack alignment requirement. Stack gap is only enabled if ASLR is enabled (not default) and stack gap itself is enabled (default if ASLR is enabled). Stack gap is specified in percentage of the total stack size that can be used for maximum gap. The main drawback of the gap approach was shortly identified. Since gap is cut from the normal stack area, attempts of the programs to reduce stack size using rlimit(RLIMIT_STACK) could cut the active stack region if new limit happens to be smaller than the gap. E.g. on amd64 with its default 512M main thread stack, even one percent of the max gap gives approximately 5M of unused stack, that can blow up the limit of several KBs. Typical reason for using rlimit(2) this way is for programs that wire all of its address space with mlockall(2), trying to reduce potential wired stack size to avoid exceeding RLIMIT_MEMLOCK. First victim of that issue was ntpd, which resets the stack limit after start for a really small value. Currently the wiring was removed from ntpd, because apparently it does not make the timekeeping better by any means, contrary to popular belief. My opinion is that the problem is more in the user interface area than in the gap approach itself. We should make it easy to specify small gap sizes, which cannot be done with integral percentage interface. So far I did not formulated a way to do this which I would like, and since nobody looked for a good solution because after ntpd was fixed, the severity of the issue seems very low. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ Signals delivered on unhandled Page Faults Contact: Konstantin Belousov By necessity, handling of page faults is separated into two pieces. The first is the architecture-dependent low-level machine exception handler, and the second is the architecture-independent vm_fault() function in sys/vm/vm_fault.c. Machine-dependent code for page faults typically consists of three components: a trampoline written in assembly which creates the C execution environment and gathers hardware-supplied data about page fault reason, a trap() function which is common C-level entry point to dispatch all exceptions processing, and the trap_pfault() C function to specifically handle page faults. trap_pfault() calls vm_fault() when help from VM is needed to resolve the situation. If the fault was handled either by trap()/trap_pfault() or vm_fault(), the faulting instruction is restarted. If fault cannot be handled for any reason, the next action depends on the mode in which the fault occured. If it was in kernel, and the kernel installed a helper, then the helper is called instead of returning to the faulting instruction. Otherwise, a kernel-mode page fault causes a panic. If the unhandled fault occured in usermode, then all Unixes send a signal to the thread whose execution caused the exception. POSIX (or Single Unix Specification) lists several cases where a signal should be sent, and specifies the signal number and si_code from siginfo that must be supplied with the signal. Unfortunately, FreeBSD was rather non-compliant in this regard. A long time ago, to improve compliance, we changed the signal sent on access to a page with permissions incompatible with the access mode. That caused multiple problems with garbage collection (GC)-based runtimes which incorporated knowledge of the FreeBSD quirks, so the machdep.prot_fault_translation sysctl knob was added. More cases of incompatibility were identified recently. Part of the problem is that code to calculate the signal and si_code from the Mach error returned by vm_fault() was located in trap_pfault(). In other words, each architecture did that on its own, with a specific set of bugs and non-compliance. Sometimes code even mis-interpreted returned Mach errors as errno. A new helper function vm_fault_trap() was added, that does several things for trap handlers: it creates ktrace points for faults, calls vm_fault(), and then interprets the result in terms of the user-visible error condition, returning precalculated signal number and si_code to the caller. Now trap_pfault() only needs to provide signal numbers for truly machine-dependent errors. For amd64, an example of such a case is a protection key violation. Besides compliance and bug fixes, we also provided some refinements to userspace about the reason of the delivered signal. For instance, on SIGBUS caused by copy-on-write failure due to exceeding swap reservation limit, we provide specific si_code BUS_OOMERR. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ Architectures Updating platform-specific features and bringing in support for new hardware platforms. Broadcom ARM64 SoC support Contact: Michal Stanek Contact: Kornel Duleba Contact: Marcin Wojtas The Semihalf team continued working on FreeBSD support for the Broadcom BCM5871X SoC series BCM5871X are quad-core 64-bit ARMv8 Cortex-A57 communication processors targeted for networking applications such as 10G routers, gateways, control plane processing and NAS. Completed since the last update: * iProc PCIe root complex (internal and external buses): fixes and improvements, * Crypto engine acceleration for IPsec offloading. Todo: * Upstreaming of work. This work is expected to be submitted/merged to HEAD in the Q4 of 2019. This project was sponsored by Juniper Networks, Inc. __________________________________________________________________ FreeBSD support for the forthcoming Arm Morello CPU, SoC, and board Contact: Robert Watson Contact: Andrew Turner Contact: Brooks Davis In September 2019, Arm announced Morello, an experimental multicore superscalar ARMv8-A CPU, SoC, and prototype board extended to implement the CHERI protection model. Morello will become available in 2021. More details can be found in Arm's blog, a Light Blue Touchpaper blog, and the main CHERI website. We have updated CheriBSD, a CHERI-adapted version of FreeBSD originally targeted at the MIPS-based SRI/Cambridge CHERI processor prototype, to support the current draft architecture. This includes full userspace spatial and referential memory safety CheriABI, as well as backwards compatibility to support running existing ARMv8-A userspace binaries. We will continue to update CheriBSD/Morello as the ISA is finalised. Will also closely track the public CheriBSD/MIPS trunk, picking up new software features utilizing CHERI as they mature, as well as to pick up changes from the baseline FreeBSD development trunk. We currently anticipate releasing CheriBSD/Morello as open source once the ISA and toolchain are published in 2020. Sponsors: DARPA, AFRL __________________________________________________________________ FreeBSD/powerpc Project Links Status of FreeBSD ports on PowerPC URL: https://wiki.freebsd.org/powerpc/ports Status of FreeBSD ports on PowerPC built using gcc URL: https://wiki.freebsd.org/powerpc/ports/PortsOnGcc Status of FreeBSD ports on PowerPC built using clang URL: https://wiki.freebsd.org/powerpc/ports/PortsOnClang Bringing LLVM to PowerPC64 target, using OpenPower ELF v2 ABI URL: https://wiki.freebsd.org/powerpc/llvm-elfv2 Contact: Mark Linimon (ports) Contact: Justin Hibbits (src) Contact: Piotr Kubaj (ports) The FreeBSD/powerpc project continues to make great strides. However, as we discovered at BSDCan 2019, we have not done a great job of letting people know. Key points: * powerpc64 src on recent hardware has been production-quality for over a year now. * powerpc64 ports has been developer-quality for over 18 months now. The main targeted platforms: * powerpc64 on IBM POWER8 and POWER9 chips (the most recent available). This is the primary focus going forward. FreeBSD 12 is required; FreeBSD 13 is recommended. * powerpc64 on older Apple Power Macs, on a continuing but secondary basis. Any FreeBSD version should work. * powerpc64 on x5000. However, this is still developer-only quality. FreeBSD 13 is recommended. * powerpcspe on Amiga A1222. However, this is still developer-only quality. FreeBSD 13 is recommended. The software status: * powerpc*/12 and earlier still remain on the antiquated gcc4.2.1 in base. * powerpc*/13 will soon be switched to llvm90 in base. A great deal of work has been undertaken to ensure as few regressions as possible. Once that switch has occurred (see llvm-elfv2 link above), powerpc*/13 support on gcc4.2.1 will no longer be a priority. * FreeBSD.org package sets are available for powerpc64/12 (quarterly) and powerpc64/13 (default) through the usual method. * Firefox works on powerpc64 using experimental (not-yet committed) patches, * As of the most recent package build on powerpc64/13 (still gcc4.2.1), the following statistics apply: Queued Built Failed Skipped Ignored 33306 30514 245 1686 861 * More ports fixes are being committed every day. Also, Piotr would like to thank the FreeBSD Foundation for funding his personal Talos, and Raptor (via its IntegriCloud subsidiary) for loaning a server on which talos.anongoth.pl runs. The team would like to thank IBM for the loan of two POWER8 machines, and Oregon State University (OSU) for providing the hosting. As well, we would like to thank the clusteradm team for keeping the Tyan POWER8 machines online that are hosted at NYI. __________________________________________________________________ NXP ARM64 SoC support Contact: Marcin Wojtas Contact: Artur Rojek The Semihalf team initiated working on FreeBSD support for the NXP LS1046A SoC LS1046A are quad-core 64-bit ARMv8 Cortex-A72 processors with integrated packet processing acceleration and high speed peripherals including 10 Gb Ethernet, PCIe 3.0, SATA 3.0 and USB 3.0 for a wide range of networking, storage, security and industrial applications. Completed since the last update: * DPAA Network interface support * SD/MMC * USB3.0 * I2C * GPIO In progress: * QSPI * Network performance improvements Todo: * Upstreaming of developed features. This work is expected to be submitted/merged to HEAD in the Q4 of 2019. This project was sponsored by Alstom Group. __________________________________________________________________ Userland Programs Changes affecting the base system and programs in it. gets(3) retirement Contact: Ed Maste gets is an obsolete C library routine for reading a string from standard input. It was removed from the C standard as of C11 because there was no way to use it safely. Prompted by a comment during Paul Vixie's talk at vBSDCon 2017 I started investigating what it would take to remove gets from libc. The patch was posted to Phabricator and refined several times, and the portmgr team performed several exp-runs to identify ports broken by the removal. Symbol versioning is used to preserve binary compatibility for existing software that uses gets. The change was committed in September, and will be in FreeBSD 13.0. This project was sponsored by The FreeBSD Foundation. __________________________________________________________________ Ports Changes affecting the Ports Collection, whether sweeping changes that touch most of the tree, or individual ports themselves. FreshPorts Links FreshPorts URL: https://www.FreshPorts.org/ git_proc_commit code URL: https://github.com/FreshPorts/git_proc_commit Things you didn't know FreshPorts can do URL: https://news.freshports.org/2019/09/03/things-you-didnt-know-freshports-can-do/ Contact: Dan Langille FreshPorts consolidates commits into an easy-to-follow format so you can track changes to your favorite ports. It also processes src, doc, and www commit. FreshPorts parses incoming emails and refreshes the database with what it finds. In early September I started looking at how FreshPorts could use a git repository for processing commits. The result was an approach for identifying new commits and for iterating through them. The next idea was commit hooks but the most interesting bit of that exercise was commit iteration. At the EuroBSDCon 2019 FreeBSD Developer summit, I wrote up a small requirements section and then received great help from two sources: * Jan-Piet MENS recommended a Python library and it turned out to be great * Sergey Kozlov wrote Python code to create xml using that Python library On the trip home, I was able to get the code to parse a git commit into XML and loaded into a FreshPorts database. Returning home from the conference, I created a new FreshPorts instance for processing git based on the above. The website has needed no changes related to git. The remaining tasks: * automate the script (git pull, etc) * detect new commits (for later iteration) * design a light-weight git hook Event: EuroBSDCon 2019 Hackathon Sponsored by: Intel Corporation (work done by Sergey Kozlov) __________________________________________________________________ Java on FreeBSD Links OpenJDK 11 repository at FreeBSD GitHub URL: https://github.com/freebsd/openjdk-jdk11u Contact: Greg Lewis Over the last few quarters there has been considerable work in improving support for Java 11 and higher, with some work being backported to Java 8. Starting with the initial release in March on amd64, over the intervening months FreeBSD support was added for features such as: * Java Flight Recorder * HotSpot Serviceability Agent * HotSpot Debugger * DTrace * Javac Server * Java Sound * SCTP In addition, support for the aarch64, i386 and powerpc64 architectures was also added. With most features supported, attention turned to compliance, using the internal JDK test suite as a method of measuring this. Most work during the third quarter has focused on this, with test failures dropping from 50+ failures to only 2 tier1 test failures (which don't appear to impact functionality at all). Some significant fixes include: * A stack overflow crash * Errors in external process handling * The unpack200 utility (on little endian systems) * HotSpot Debugger functionality such as thread enumeration * Networking functionality Finally, this work has been forward ported to Java 12 and 13, with FreeBSD gaining support for these versions on or just after the day of release. Note that this work has been a collaboration with many others. While there are too many contributors to list here (please take a look at the commit history of the GitHub repository), I'd like to recognise Kurt Miller of OpenBSD for his tireless work as a co-collaborator on Java for BSD through many years. I'm also grateful to the sponsorship of the FreeBSD Foundation, which has allowed me to focus on this work for Q3. This project was sponsored by FreeBSD Foundation. __________________________________________________________________ KDE on FreeBSD Links KDE FreeBSD URL: https://freebsd.kde.org/ KDE Community FreeBSD URL: https://community.kde.org/FreeBSD Contact: Adriaan de Groot The KDE on FreeBSD project packages the software produced by the KDE Community for FreeBSD. The software includes a full desktop environment, the art application https://kdenlive.org and hundreds of other applications that can be used on any FreeBSD desktop machine. Along with KDE itself, the team maintains the Qt libraries, the CMake build system, and a handful of other C++ libraries used in the KDE stack. The upstream releases KDE Frameworks (libraries) on a monthly cycle, KDE Plasma (the desktop environment) quarterly and a collection of KDE Applications (usable everywhere) twice a year. The KDE on FreeBSD team chased a dozen updates to these components so that FreeBSD remains one of the most up-to-date systems with KDE software (and Qt). A large (and possibly breaking, still needs more investigation) change came with the release to KDE's Digikam 6.3.0, which stopped using its previous plugins system (the "old" plugins are still used by other KDE software). A new entry in the net-im category was added for Ruqola, a Rocket.chat client for rich instant-messaging. CMake was updated twice. This forces the rebuild of several thousand C++-based ports. The KDE on FreeBSD team then fixes those ports, regardless of whether the error is in the CMake update, or in the upstream code. These updates tend to take a large amount of effort, since they touch unfamiliar (and often very-very-legacy) code. The open bugs list grew to 24 this quarter. It tends to hover around 20 items as new things come in and old ones are resolved. We welcome detailed bug reports and patches. KDE packaging updates are prepared in a copy of the ports repository on GitHub and then merged in SVN. We welcome pull requests there as well. __________________________________________________________________ Ports Collection Links About FreeBSD Ports URL: https://www.FreeBSD.org/ports/ Contributing to Ports URL: https://www.freebsd.org/doc/en_US.ISO8859-1/articles/contributing/ports-contributing.html FreeBSD Ports Monitoring URL: http://portsmon.freebsd.org/index.html Ports Management Team URL: https://www.freebsd.org/portmgr/index.html Contact: René Ladan Contact: FreeBSD Ports Management Team The FreeBSD Ports Management Team, tasked with overseeing the Ports Tree and its committers, reports that the following events happened during 2019Q3: The number of ports grew to just under 38,000 during the last quarter. We have just over 2,000 open ports PRs, of which 400 are unassigned. In this period, 169 committers made 7,340 commits to HEAD and 561 commits to the quarterly branch. This shows that the trend of last quarter of increased activity is continuing. During the last quarter, we welcomed Santhosh Raju (fox@) and Dmitri Goutnik (dmgk@) to the team, and said goodbye to gabor@. During the last quarter, feld@ decided to step down from the portmgr@ and ports-secteam@ teams. We would like to thank them for their past services. In the last three months, bapt@ put on his engineering hat and released a new version of pkg (1.12), added support for overlays to the Ports Tree, fixed two Make targets (deinstall-depends and reinstall), and cleaned up bsd.sites.mk. On the infrastructure side, USES=pure became obsolete and has therefore been removed, and two new USES, xorg and xorg-cat have been added and replace the old bsd.xorg.mk. Two new keywords, ldconfig and ldconfig-linux, were added to simplify formatting of package lists. A number of default versions changed: Lazarus to 2.0.4, Linux to CentOS 7, LLVM to 9.0, Perl to 5.30, PostgreSQL to 11, and Ruby to 2.6. Of the big user-visible ports, Firefox got updated to 69.0.1, Firefox-esr to 68.1.0, Chromium to 76.0.3809.132, and Xfce to 4.14. During the last quarter, antoine@ ran 48 exp-runs to test package updates, test updating bsd.java.mk, test the new ldconfig and ldconfig-linux keywords, test default version updates, test the new xorg and xorg-cat USES, test base updates, and test various improvements to Go and Ruby. __________________________________________________________________ XFCE 4.14 update Links XFCE 4.14 announce URL: https://xfce.org/about/news/?post=1565568000 Contact: Guido Falsi On September 19th the XFCE desktop environment ports have been updated to the recently released XFCE 4.14 version. These updates include upgrades of all the main XFCE components to the latest versions which have been migrated to GTK3, with few exceptions. Base components still require and link to GTK2 in addition to GTK3 to allow older GTK2 XFCE applications, for example panel plugins, to keep working. Due to this change the gtk-xfce-engine theme is deprecated since it only supports GTK2. The XFCE metaport by default installs the greybird theme, but it is not enabled by default. This new version also includes now it's own xfce4-screensaver program which is installed by default. Finally the default Display Manager on which XFCE depends has been changed to LightDM. Some regressions were reported in bugzilla. In particular the one affecting most users is a regression in the window manager: on specific graphic display hardware the window manager fails to properly draw window decorations, which appear black and blocky on affected systems. This problem has also been reported in the upstream bug tracker and a solution is being sought. If anyone is experiencing this display glitch and is able to test, please contact xfce@freebsd.org to help trying to figure out a solution. __________________________________________________________________ Third-Party Projects Many projects build upon FreeBSD or incorporate components of FreeBSD into their project. As these projects may be of interest to the broader FreeBSD community, we sometimes include brief updates submitted by these projects in our quarterly report. The FreeBSD project makes no representation as to the accuracy or veracity of any claims in these submissions. ClonOS: virtualization platform on top of FreeBSD Operating System Links ClonOS 19.09 URL: https://clonos.tekroutine.com/download.html CBSD URL: https://www.bsdstore.ru/ Contact: Oleg Ginzburg What is ClonOS? ClonOS is a turnkey open-source platform based on FreeBSD and the CBSD framework. ClonOS offers a complete web UI for an easy control, deployment and management of FreeBSD jails containers and bhyve/Xen hypervisor virtual environments. ClonOS is currently the only available platforms which allow both Xen and bhyve hypervisors to coexist on the same host. Since ClonOS is a FreeBSD-based platform, it has the ability to create and manage jails natively, allowing you to run FreeBSD applications without losing performance. ClonOS/CBSD 2019Q3 Status Report * Added support for cloud-init (Linux/BSD VMs) and cloudbase-init (Windows VMs). It gives the ability to use FreeBSD as IaaS platform for instant deployment and usage of virtual machines based on bhyve hypervisor. * Project started to use own cloud images for better stability and resiliency. * New mirrors in France, US and Canada were added for distributing ISO/cloud-init images in addition to Russia, Latvia and Ukraine. * Now we're using Jenkins CI for testing regular ClonOS builds: Update clonos packages (Thanks to Daniel Shafer) * New pkg repository was deployed to support ClonOS installation from packages (at this moment only FreeBSD-12 packages are available) ClonOS package repo (Thanks to Daniel Shafer) Open issues and tasks: * Volunteers, contributors and testers are urgently needed to distribute ClonOS on FreeBSD environments. * We'd like to expand our mirrors number geographically, your help and contribution will be much appriciated. * We're urgently looking for hosting sponsorship for various developing and testing activities. __________________________________________________________________ ENA FreeBSD Driver Update Links ENA README URL: https://github.com/amzn/amzn-drivers/blob/master/kernel/fbsd/ena/README Contact: Michal Krawczyk Contact: Maciej Bielski Contact: Marcin Wojtas ENA (Elastic Network Adapter) is the smart NIC available in the virtualized environment of Amazon Web Services (AWS). The ENA driver supports multiple transmit and receive queues and can handle up to 100 Gb/s of network traffic, depending on the instance type on which it is used. ENAv2 has been under development for FreeBSD, similar to Linux and DPDK. Since the last update internal review and improvements of the patches were done, followed by validation on various AWS instances. Completed since the last update: * Verification and review of the NETMAP support * Mapping of the memory as WC on A1 instances in order to enable LLQ mode Todo: * Upstream of NETMAP support * Upstream of the fix for LLQ mode on A1 instances Amazon.com Inc __________________________________________________________________ Nomad pot driver - Orchestrating jails via nomad Links Nomad pot driver URL: https://github.com/trivago/nomad-pot-driver Pot project URL: https://github.com/pizzamig/pot Contact: Luca Pizzamiglio Contact: Esteban Barrios An experimental project has started to provide jail orchestration based on nomad and the jail framework pot, similarly to how orchestration works with docker. This model allows us to split the jail creation and the jail deployment. Jail images can be created and exported using the pot framework. The images can be deployed and orchestrated using nomad. A driver has been developed to allow nomad to interact with pot. One of the goals of this project is to use non-persistent jails as containers, allowing us to: * define containers similar to Docker (but not identical, because the underlaying OS is different) * identify potential missing features in FreeBSD to support such a computational model In the next quarter, we will launch the first service based on this project. Next steps are: * provide more guides and howtos * improve stability, extending the tests suite * improving tooling to create/manage images This project was sponsored by trivago N.V.. __________________________________________________________________ sysctlinfo Links gitlab.com/alfix/sysctlinfo URL: https://gitlab.com/alfix/sysctlinfo Contact: Alfonso Sabato Siciliano The FreeBSD kernel maintains a Management Information Base (MIB) where a component (object) represents a parameter of the system. The sysctl system call explores the MIB to find an object by its Object Identifier (OID) and calls its handler to get or set the value of the parameter. It is often necessary to find an object not to call its handler but to get its info (description, type, name, next object, etc.), so the kernel provides an undocumented interface implemented in kern_sysctl.c. sysctlinfo is a new interface to explore the sysctl MIB and to pass the info of an object to the userland. The project provides: a README, a manual, helper macros, examples, tests and converted tools. Primarily sysctlinfo provides a new set of sysctl nodes (corresponding to the kernel interface) to handle an object up to CTL_MAXNAME levels: sysctl.entryfakename, sysctl.entrydesc, sysctl.entrylabel, sysctl.entrykind, sysctl.entryfmt and sysctl.entrynextleaf. Moreover new features have been implemented: the support for the capability mode, sysctl.entryname, sysctl.entryidbyname and sysctl.entrynextnode. To get all the info about an object the kernel needs to find it many times, then the new sysctl.entryallinfo* nodes were written, they are 30% more efficient. Finally, *byname nodes were added: they search the object by its name avoiding to call sysctl.name2oid (or similar) to explore the MIB just to find the corresponding OID. sysctlinfo can be installed via sysutils/sysctlinfo-kmod or by applying the sysctlinfo.diff patch (more efficient than the module because uses a shared lock). The interface is used by deskutils/sysctlview 1.5, sysutils/nsysctl 1.2 and the converted version of sysctl(8), sysctlbyname(3), sysctlnametomib(3), they should be used to get the value of an object with 23/24 levels or if some level-name has only the '\0' character. In the future a new byname node will be added to allow sysctlbyname() to manage a CTLTYPE_NODE with a no-NULL handler, example sysctlbyname("kern.proc.pid.\"). __________________________________________________________________ From owner-freebsd-hackers@freebsd.org Tue Nov 26 20:42:25 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 436B61C0DBB for ; Tue, 26 Nov 2019 20:42:25 +0000 (UTC) (envelope-from mashtizadeh@gmail.com) Received: from mail-io1-xd30.google.com (mail-io1-xd30.google.com [IPv6:2607:f8b0:4864:20::d30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47MwnW3DWbz4FFR for ; Tue, 26 Nov 2019 20:42:23 +0000 (UTC) (envelope-from mashtizadeh@gmail.com) Received: by mail-io1-xd30.google.com with SMTP id s3so22231821ioe.3 for ; Tue, 26 Nov 2019 12:42:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=Gxp6JtYSCjfIQk2JBdcLicHwUTtoRf4BnC81Mm7e0SQ=; b=uWAKKjFSbVhEca/kHuPA8cJoVnjfP6yOTN1BeDho1IEatEnjha9JojeIYZ5T6NHvNd lmoNNHw53SPGujeuf65J/jebnpS5QO3zUmejMB8DIKaEbxiQgvVYACh+yzCn+2bQDURE R+sa0KJVjuBERkSijMaSG5A7xk3y98eeHJB96vDJD8UeLJwrfASoln3jLRABLtMr6YNa MmBQcu1nNLC5evetpE6NtEyv5yaUbjl3YICMV9Vzk9+6iusr8TZSiZloP70shy/OG1cF GocKAMIChShEIv7VZLtyDagrsmY8pM9x+b9G6LWRs5DOcP8kBPinyXKZ5XACLYBZ91Qf lyAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=Gxp6JtYSCjfIQk2JBdcLicHwUTtoRf4BnC81Mm7e0SQ=; b=P2N4zE3JA2ytoCCyHoDz0k0BlaPYpL+7qrIGjgEDlD89mGxRG3CESUiPzVWpNG0D4Q GoXi1G/AltZPHxnAbUx67Gfn6sHvnl5x7+KJmtXWoeCuYVGaL2t7EiDWKonPDP6htYBv KkMffVf0fRqA2qq42vtdvnlI7k0L3lqVjmcI1iZJyEka6pHI6GnFa6NzqTHDDK97K1oI OBCYGms7x/iaz+H7JZKgLo42mZ6VptlZkJvhXVqIQi3cLlQRViPZIRjKfnG7IgNANzkv DpPojJa6aef+hWaG6ffDnfy8R0eC5RDhA6yH8oXjfS7EZxhakVpHbXWG25rkl+zEdNZN WLSQ== X-Gm-Message-State: APjAAAUm1gZQw/AXx4NQNFKj+M2sZ2Amcv4SqcW+Vwubt+rIvv/1FL9o c1VPuhDdhoXlPFnnug5H6PDT3os6atc+oN4RHc0mm3Bc X-Google-Smtp-Source: APXvYqyODbbsnOoYO2HuAwBwcX+Qbs41nTdvXM2LlDo0MJNTCWKv36d6BtkaAopwkM2ln5TdA+KxnhSnXpiqqTz5dho= X-Received: by 2002:a6b:c982:: with SMTP id z124mr34542229iof.291.1574800941127; Tue, 26 Nov 2019 12:42:21 -0800 (PST) MIME-Version: 1.0 From: Ali Mashtizadeh Date: Tue, 26 Nov 2019 15:42:10 -0500 Message-ID: Subject: Mellanox NIC + Bridge + VM traffic issues on 12.1-RELEASE To: FreeBSD Hackers X-Rspamd-Queue-Id: 47MwnW3DWbz4FFR X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=uWAKKjFS; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mashtizadeh@gmail.com designates 2607:f8b0:4864:20::d30 as permitted sender) smtp.mailfrom=mashtizadeh@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(0.00)[ip: (-5.94), ipnet: 2607:f8b0::/32(-2.27), asn: 15169(-1.95), country: US(-0.05)]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[0.3.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2019 20:42:25 -0000 Hello Everyone, I've noticed something strange happening after I upgraded to FreeBSD 12.1 on one machine. My VM is bridged to a Mellanox ConnectX4 that connects to our cluster of machines. Traffic appears to go between the VM and external machines except for ARP replies. I noticed ARP replies from the VM do not make it to the external network and appear to be dropped by the NIC. All network connectivity is fine when using an Intel ixl NIC. Has anyone seen this? Does this have something to do with the NIC offloads? I experimented with them but did not succeed in fixing ARP traffic. Best, Ali From owner-freebsd-hackers@freebsd.org Tue Nov 26 20:50:51 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 43FC21C1191 for ; Tue, 26 Nov 2019 20:50:51 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47MwzG06lCz4FjF for ; Tue, 26 Nov 2019 20:50:49 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id xAQKoilx055888 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 26 Nov 2019 21:50:44 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1574801444; bh=iHcLbDMCXV2AO2zRYjGbui3IXla9rvsgLlkmONzED/s=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=cOaHqCF8AzhSxbtfsNZZFTFaX46IUSRBgUyVeXPOG50hxSo/u1saI969QV3YjDjY7 twnRnk9cb73bhGCQ5y+Dj7fMmdtcT0nGfLUqkKHjiFkWu4FDYA7ioKiisT2mHUuIya IzOUcchqkB511tm5iWNB7T5g9dT2+whkty558vWk= Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id xAQKoi3f055885; Tue, 26 Nov 2019 21:50:44 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Date: Tue, 26 Nov 2019 21:50:44 +0100 (CET) From: Wojciech Puchar To: Eugene Grosbein cc: Wojciech Puchar , freebsd-hackers@freebsd.org Subject: Re: Nothing is broken, but changed WAS: what's got broken in bhyve after upgrade In-Reply-To: <0d4379a6-d71d-a2b1-35e3-2dec23df5df4@grosbein.net> Message-ID: References: <0d4379a6-d71d-a2b1-35e3-2dec23df5df4@grosbein.net> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 47MwzG06lCz4FjF X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=fail (rsa verify failed) header.d=puchar.net header.s=default header.b=cOaHqCF8; dmarc=none; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-4.69 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_REJECT(1.00)[puchar.net:s=default]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[puchar.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[puchar.net:-]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-3.39)[ip: (-8.95), ipnet: 194.1.144.0/24(-4.47), asn: 43476(-3.58), country: PL(0.07)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2019 20:50:51 -0000 doesn't seem like for a day. ask me after few days. On Tue, 26 Nov 2019, Eugene Grosbein wrote: > 26.11.2019 16:44, Wojciech Puchar wrote: > >> New bhyve gets sockets, cores and threads parameters from command line. fixed now. > > I observer significant memory leaks in bhyve processes after upgrade from 11.2 to 11.3-STABLE, do you? > Way over limits for guests. > > > > From owner-freebsd-hackers@freebsd.org Tue Nov 26 20:51:04 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 03DBD1C132D for ; Tue, 26 Nov 2019 20:51:04 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47MwzW0hqfz4Fp1 for ; Tue, 26 Nov 2019 20:51:02 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id xAQKowIA055999 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 26 Nov 2019 21:50:58 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1574801459; bh=DF+xykDEbueR03bfyd18C/KGNWTtir8JeMNklyz3QeM=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=XwJ1W94vXBQEKlf6gPW1GpWRGPJyDXIiJWp9km8L+xLX9p1kWOUC/3qpGND+b2XbX m0/I9NqSPzU2bLJ0wCbu9PImTHKByJDFioRJBNkGlmYeRrNfEc51TDfZNwETFZ53/M J0Dzc+pcNi5n0CpmY2FrAavIQudRQ2EIVF5uGMG4= Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id xAQKowO5055996; Tue, 26 Nov 2019 21:50:58 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Date: Tue, 26 Nov 2019 21:50:58 +0100 (CET) From: Wojciech Puchar To: "Rodney W. Grimes" cc: Wojciech Puchar , freebsd-hackers@freebsd.org Subject: Re: Nothing is broken, but changed WAS: what's got broken in bhyve after upgrade In-Reply-To: <201911261059.xAQAxJTE002277@gndrsh.dnsmgr.net> Message-ID: References: <201911261059.xAQAxJTE002277@gndrsh.dnsmgr.net> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 47MwzW0hqfz4Fp1 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=fail (rsa verify failed) header.d=puchar.net header.s=default header.b=XwJ1W94v; dmarc=none; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-4.67 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx:c]; R_DKIM_REJECT(1.00)[puchar.net:s=default]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[puchar.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[puchar.net:-]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-3.37)[ip: (-8.92), ipnet: 194.1.144.0/24(-4.46), asn: 43476(-3.57), country: PL(0.07)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2019 20:51:04 -0000 thank you On Tue, 26 Nov 2019, Rodney W. Grimes wrote: >> New bhyve gets sockets, cores and threads parameters from command line. >> fixed now. >> >> Question - is 16 core limit in bhyve a hard one because of something, or >> it can be changed? > > Yes, modify this line: > #define VM_MAXCPU 16 /* maximum virtual cpus */ > in ./amd64/include/vmm.h > > You'll need to rebuild vmm.,ko, and anything else that includes vmm.h > > >> On Tue, 26 Nov 2019, Wojciech Puchar wrote: >> >>> upgraded to >>> >>> FreeBSD puchar.net 11.3-STABLE FreeBSD 11.3-STABLE #0 r354807: Fri Nov 22 >>> 22:45:43 CET 2019 root@puchar.net:/usr/src/sys/amd64/compile/puchar amd64 >>> >>> >>> i have setting hw.vmm.topology.cores_per_package=16 in loader.conf >>> >>> >>> before upgrade - bhyve virtual machines (windows 7) see one CPU with 2 or 4 >>> cores (depending how much i use with -c option in bhyve). >>> >>> after upgrade - bhyve sees single core CPUs. Which in case of windows 7 pro >>> means no more than 2 cores will be used. >>> >>> What's wrong and how to fix it? >>> >>> _______________________________________________ >>> freebsd-hackers@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >>> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >>> >>> >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >> > > -- > Rod Grimes rgrimes@freebsd.org > > From owner-freebsd-hackers@freebsd.org Tue Nov 26 21:25:54 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6EBBD1C1D6B for ; Tue, 26 Nov 2019 21:25:54 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47Mxlj2V7sz4HRf for ; Tue, 26 Nov 2019 21:25:52 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id xAQLPnm4004542; Tue, 26 Nov 2019 13:25:49 -0800 (PST) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id xAQLPnwU004541; Tue, 26 Nov 2019 13:25:49 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201911262125.xAQLPnwU004541@gndrsh.dnsmgr.net> Subject: Re: Nothing is broken, but changed WAS: what's got broken in bhyve after upgrade In-Reply-To: To: Wojciech Puchar Date: Tue, 26 Nov 2019 13:25:49 -0800 (PST) CC: "Rodney W. Grimes" , freebsd-hackers@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 47Mxlj2V7sz4HRf X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd-rwg@gndrsh.dnsmgr.net has no SPF policy when checking 69.59.192.140) smtp.mailfrom=freebsd-rwg@gndrsh.dnsmgr.net X-Spamd-Result: default: False [2.15 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.70)[0.697,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.52)[0.519,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.04)[ip: (0.14), ipnet: 69.59.192.0/19(0.07), asn: 13868(0.03), country: US(-0.05)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2019 21:25:54 -0000 Almost forgot, be sure you install it to /usr/include before trying to compile stuff, you may end up picking up old values unless you use buildworld. cp /usr/src/sys/amd64/include/vmm.h /usr/include/machine/vmm.h > thank you > > On Tue, 26 Nov 2019, Rodney W. Grimes wrote: > > >> New bhyve gets sockets, cores and threads parameters from command line. > >> fixed now. > >> > >> Question - is 16 core limit in bhyve a hard one because of something, or > >> it can be changed? > > > > Yes, modify this line: > > #define VM_MAXCPU 16 /* maximum virtual cpus */ > > in ./amd64/include/vmm.h > > > > You'll need to rebuild vmm.,ko, and anything else that includes vmm.h > > > > > >> On Tue, 26 Nov 2019, Wojciech Puchar wrote: > >> > >>> upgraded to > >>> > >>> FreeBSD puchar.net 11.3-STABLE FreeBSD 11.3-STABLE #0 r354807: Fri Nov 22 > >>> 22:45:43 CET 2019 root@puchar.net:/usr/src/sys/amd64/compile/puchar amd64 > >>> > >>> > >>> i have setting hw.vmm.topology.cores_per_package=16 in loader.conf > >>> > >>> > >>> before upgrade - bhyve virtual machines (windows 7) see one CPU with 2 or 4 > >>> cores (depending how much i use with -c option in bhyve). > >>> > >>> after upgrade - bhyve sees single core CPUs. Which in case of windows 7 pro > >>> means no more than 2 cores will be used. > >>> > >>> What's wrong and how to fix it? > >>> > >>> _______________________________________________ > >>> freebsd-hackers@freebsd.org mailing list > >>> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > >>> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > >>> > >>> > >> _______________________________________________ > >> freebsd-hackers@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > >> > > > > -- > > Rod Grimes rgrimes@freebsd.org > > > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-hackers@freebsd.org Tue Nov 26 21:55:05 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5E1B51C2944 for ; Tue, 26 Nov 2019 21:55:05 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [IPv6:2a01:4f8:c17:6c4b::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47MyPN4zmTz4Jxk for ; Tue, 26 Nov 2019 21:55:04 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2016.home.selasky.org (unknown [62.141.129.235]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id F2745260182; Tue, 26 Nov 2019 22:55:02 +0100 (CET) Subject: Re: Mellanox NIC + Bridge + VM traffic issues on 12.1-RELEASE To: Ali Mashtizadeh , FreeBSD Hackers References: From: Hans Petter Selasky Message-ID: <16933f4d-c9cc-9a7a-3937-197ed131b9a4@selasky.org> Date: Tue, 26 Nov 2019 22:53:55 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.1.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 47MyPN4zmTz4Jxk X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 2a01:4f8:c17:6c4b::2 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-4.92 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[selasky.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-2.62)[ip: (-9.17), ipnet: 2a01:4f8::/29(-2.33), asn: 24940(-1.57), country: DE(-0.01)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2019 21:55:05 -0000 On 2019-11-26 21:42, Ali Mashtizadeh wrote: > I noticed ARP replies from the VM do not make it to the external network > and appear to be dropped by the NIC. All network connectivity is fine when > using an Intel ixl NIC. Hi, The output from "ifmcstat" on working and non-working machine might give some clues. The CX-4 has some steering rules, which if are not present, traffic will be dropped. Can also try to set promiscious mode for a test "ifconfig mceX promisc". --HPS From owner-freebsd-hackers@freebsd.org Wed Nov 27 02:59:52 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5DD721C8346 for ; Wed, 27 Nov 2019 02:59:52 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47N593119sz4YlF for ; Wed, 27 Nov 2019 02:59:50 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id xAR2xeAI005662; Tue, 26 Nov 2019 18:59:40 -0800 (PST) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id xAR2xdS3005661; Tue, 26 Nov 2019 18:59:39 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201911270259.xAR2xdS3005661@gndrsh.dnsmgr.net> Subject: Re: what's got broken in bhyve after upgrade In-Reply-To: To: Wojciech Puchar Date: Tue, 26 Nov 2019 18:59:39 -0800 (PST) CC: freebsd-hackers@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 47N593119sz4YlF X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd-rwg@gndrsh.dnsmgr.net has no SPF policy when checking 69.59.192.140) smtp.mailfrom=freebsd-rwg@gndrsh.dnsmgr.net X-Spamd-Result: default: False [2.35 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.74)[0.741,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.67)[0.666,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.04)[ip: (0.14), ipnet: 69.59.192.0/19(0.07), asn: 13868(0.03), country: US(-0.05)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2019 02:59:52 -0000 > upgraded to > > FreeBSD puchar.net 11.3-STABLE FreeBSD 11.3-STABLE #0 r354807: Fri Nov 22 > 22:45:43 CET 2019 root@puchar.net:/usr/src/sys/amd64/compile/puchar > amd64 > > > i have setting hw.vmm.topology.cores_per_package=16 in loader.conf I do not have a 11.3-STABLE reference box, but could you check man bhyve and see if it has the new -c options: SYNOPSIS bhyve [-abehuwxACHPSWY] [-c [[cpus=]numcpus][,sockets=n][,cores=n][,threads=n]] If it has that form of -c you SHOULD stop using hw.vmm.topology.cores_per_package and start to use the new -c options. I do not believe this feature is in 11.3, but it may of been merged. > > > before upgrade - bhyve virtual machines (windows 7) see one CPU with 2 or > 4 cores (depending how much i use with -c option in bhyve). > > after upgrade - bhyve sees single core CPUs. Which in case of windows 7 > pro means no more than 2 cores will be used. > > What's wrong and how to fix it? -c cores=4 or -c cores=2 Note unspecified values default to 1. See the man page for additional details. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-hackers@freebsd.org Wed Nov 27 08:06:21 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 129FC1CD4E5 for ; Wed, 27 Nov 2019 08:06:21 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47NCyg5zQtz3Jbd for ; Wed, 27 Nov 2019 08:06:19 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id xAR86DLU090506 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 27 Nov 2019 09:06:14 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1574841974; bh=PVjzQJ3UL5zl8Sgm3BpOgbtkX5EI0+fw2Ks7BGpgjZ8=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=krw7Ene8aBkZSt+9OxRis/Hj1Bm0SSztoc335svhrKwiwFWS0/WMdlrC2hz4OX2xl ErtFo25uqPvErRLlBJ7KqtHYSiu8FSru9hEpPzfaXnunXnqrjuzU059gzYU5NRlNCQ TatD2n/Y0MuaOFgvWmvp8C98oziHZxvpzoWiXtU4= Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id xAR86DQ9090502; Wed, 27 Nov 2019 09:06:13 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Date: Wed, 27 Nov 2019 09:06:13 +0100 (CET) From: Wojciech Puchar To: "Rodney W. Grimes" cc: Wojciech Puchar , freebsd-hackers@freebsd.org Subject: Re: what's got broken in bhyve after upgrade In-Reply-To: <201911270259.xAR2xdS3005661@gndrsh.dnsmgr.net> Message-ID: References: <201911270259.xAR2xdS3005661@gndrsh.dnsmgr.net> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 47NCyg5zQtz3Jbd X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=fail (rsa verify failed) header.d=puchar.net header.s=default header.b=krw7Ene8; dmarc=none; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-4.66 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_REJECT(1.00)[puchar.net:s=default]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[puchar.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[puchar.net:-]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-3.36)[ip: (-8.89), ipnet: 194.1.144.0/24(-4.44), asn: 43476(-3.55), country: PL(0.07)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2019 08:06:21 -0000 > > If it has that form of -c you SHOULD stop using hw.vmm.topology.cores_per_package > and start to use the new -c options. > > I do not believe this feature is in 11.3, but it may of been merged. yes this was that. all works fine From owner-freebsd-hackers@freebsd.org Wed Nov 27 08:07:18 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 289381CD5E8 for ; Wed, 27 Nov 2019 08:07:18 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47NCzn3kSRz3JlC for ; Wed, 27 Nov 2019 08:07:17 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id xAR87Cj7092146 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 27 Nov 2019 09:07:13 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1574842033; bh=WGdM/+BE1NV+J3HXJUr9UOpVXWZn9Rc6o5bLBTvFt3U=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=hK97bsTnJJm8s44ldUd5Ix4tJsEW5vTN5fT5Rp3L6sBsZUaSisn+6bt5I20G0buUz clVd9Ei8ius+g5MiqrAH2pba1gd4k7SZrHC2jgkP7OU1k9DKiW5eFGiDUYcN9ILiZ9 VJbscLHbxk0Ij3pNLeWeC9kuVxulnBZQbMTPYqhw= Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id xAR87CK0092143; Wed, 27 Nov 2019 09:07:12 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Date: Wed, 27 Nov 2019 09:07:12 +0100 (CET) From: Wojciech Puchar To: "Rodney W. Grimes" cc: Wojciech Puchar , freebsd-hackers@freebsd.org Subject: Re: Nothing is broken, but changed WAS: what's got broken in bhyve after upgrade In-Reply-To: <201911262125.xAQLPnwU004541@gndrsh.dnsmgr.net> Message-ID: References: <201911262125.xAQLPnwU004541@gndrsh.dnsmgr.net> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 47NCzn3kSRz3JlC X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=fail (rsa verify failed) header.d=puchar.net header.s=default header.b=hK97bsTn; dmarc=none; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-4.65 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx:c]; R_DKIM_REJECT(1.00)[puchar.net:s=default]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[puchar.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[puchar.net:-]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-3.35)[ip: (-8.85), ipnet: 194.1.144.0/24(-4.43), asn: 43476(-3.54), country: PL(0.07)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2019 08:07:18 -0000 thanks On Tue, 26 Nov 2019, Rodney W. Grimes wrote: > Almost forgot, be sure you install it to /usr/include before > trying to compile stuff, you may end up picking up old values > unless you use buildworld. > > cp /usr/src/sys/amd64/include/vmm.h /usr/include/machine/vmm.h > > >> thank you >> >> On Tue, 26 Nov 2019, Rodney W. Grimes wrote: >> >>>> New bhyve gets sockets, cores and threads parameters from command line. >>>> fixed now. >>>> >>>> Question - is 16 core limit in bhyve a hard one because of something, or >>>> it can be changed? >>> >>> Yes, modify this line: >>> #define VM_MAXCPU 16 /* maximum virtual cpus */ >>> in ./amd64/include/vmm.h >>> >>> You'll need to rebuild vmm.,ko, and anything else that includes vmm.h >>> >>> >>>> On Tue, 26 Nov 2019, Wojciech Puchar wrote: >>>> >>>>> upgraded to >>>>> >>>>> FreeBSD puchar.net 11.3-STABLE FreeBSD 11.3-STABLE #0 r354807: Fri Nov 22 >>>>> 22:45:43 CET 2019 root@puchar.net:/usr/src/sys/amd64/compile/puchar amd64 >>>>> >>>>> >>>>> i have setting hw.vmm.topology.cores_per_package=16 in loader.conf >>>>> >>>>> >>>>> before upgrade - bhyve virtual machines (windows 7) see one CPU with 2 or 4 >>>>> cores (depending how much i use with -c option in bhyve). >>>>> >>>>> after upgrade - bhyve sees single core CPUs. Which in case of windows 7 pro >>>>> means no more than 2 cores will be used. >>>>> >>>>> What's wrong and how to fix it? >>>>> >>>>> _______________________________________________ >>>>> freebsd-hackers@freebsd.org mailing list >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >>>>> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >>>>> >>>>> >>>> _______________________________________________ >>>> freebsd-hackers@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >>>> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >>>> >>> >>> -- >>> Rod Grimes rgrimes@freebsd.org >>> >>> >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >> > > -- > Rod Grimes rgrimes@freebsd.org > > From owner-freebsd-hackers@freebsd.org Wed Nov 27 08:08:14 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6B9BD1CD719 for ; Wed, 27 Nov 2019 08:08:14 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47ND0s3vrJz3JtG for ; Wed, 27 Nov 2019 08:08:13 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id xAR888hL093042 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 27 Nov 2019 09:08:08 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1574842088; bh=4tSgbCACcCfsiD9UR1uaqs2WG02xTjawiITpPBLMbkk=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=BTY/GFcH1LeB12kjf20am2ZDyyFVuzik9ErES/ThVMv1RVlOkYrtLGETLTETXva1W qFd4R9M2RGPLETAdnkqzS3zD5UcWzF0Xy0XFrEoqXhywrGWcAVW0idobO6Om4dNNG1 6nzz3H91/btIHS4Cp22SIlEy38EbWRNPPL2ACdj4= Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id xAR888PD093036; Wed, 27 Nov 2019 09:08:08 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Date: Wed, 27 Nov 2019 09:08:08 +0100 (CET) From: Wojciech Puchar To: Eugene Grosbein cc: Wojciech Puchar , freebsd-hackers@freebsd.org Subject: Re: Nothing is broken, but changed WAS: what's got broken in bhyve after upgrade In-Reply-To: <0d4379a6-d71d-a2b1-35e3-2dec23df5df4@grosbein.net> Message-ID: References: <0d4379a6-d71d-a2b1-35e3-2dec23df5df4@grosbein.net> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Rspamd-Queue-Id: 47ND0s3vrJz3JtG X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=fail (rsa verify failed) header.d=puchar.net header.s=default header.b=BTY/GFcH; dmarc=none; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-4.64 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx:c]; R_DKIM_REJECT(1.00)[puchar.net:s=default]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[puchar.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[puchar.net:-]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-3.34)[ip: (-8.82), ipnet: 194.1.144.0/24(-4.41), asn: 43476(-3.53), country: PL(0.07)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2019 08:08:14 -0000 on first server - bhyve running 3 days after upgrade, 10 VMs total. No leaks on other machine - one VM with 11 virtual cores and 24GB RAM and -S option - no leaks. On Tue, 26 Nov 2019, Eugene Grosbein wrote: > 26.11.2019 16:44, Wojciech Puchar wrote: > >> New bhyve gets sockets, cores and threads parameters from command line. fixed now. > > I observer significant memory leaks in bhyve processes after upgrade from 11.2 to 11.3-STABLE, do you? > Way over limits for guests. > > > > From owner-freebsd-hackers@freebsd.org Wed Nov 27 08:19:21 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8DA851CDA15 for ; Wed, 27 Nov 2019 08:19:21 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47NDFh3Jq9z3KKR for ; Wed, 27 Nov 2019 08:19:20 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id xAR8JHE7007877 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 27 Nov 2019 09:19:17 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1574842757; bh=/KSgjXHzfLH7EMr1rgjjfJktl2AjXRvkqO4J7MQ5Vvk=; h=Date:From:To:Subject; b=TCr2hQ8RsyyGtsde4jeOtMtbrhMSRuw7JvNpZMbrZGWg/4y2YGpo6zzjeM1a55GcG JZ5aPjPTLSrMoNvqkU+LjwHyma2v2ZM4wfGgLdIpKfKgO2XaYwQBW8HAfIIde4yxOM pJO6qobsACt05PJHmPQBH/cASMaQKjd2xP9TJvrE= Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id xAR8JHD9007874 for ; Wed, 27 Nov 2019 09:19:17 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Date: Wed, 27 Nov 2019 09:19:17 +0100 (CET) From: Wojciech Puchar To: freebsd-hackers@freebsd.org Subject: bhyve - lots of cores Message-ID: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-Rspamd-Queue-Id: 47NDFh3Jq9z3KKR X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=fail (rsa verify failed) header.d=puchar.net header.s=default header.b=TCr2hQ8R; dmarc=none; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-4.63 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx:c]; R_DKIM_REJECT(1.00)[puchar.net:s=default]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; DMARC_NA(0.00)[puchar.net]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[puchar.net:-]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-3.33)[ip: (-8.79), ipnet: 194.1.144.0/24(-4.40), asn: 43476(-3.52), country: PL(0.07)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2019 08:19:21 -0000 does anyone have FreeBSD running host with LOTS of cores (>=32) and allow me to run short test of bhyve? I would like to know a) does bhyve have scaling limits. b) can does windows 7 x64 (which is for what i use bhyve :) run efficiently on this. At least up to 12 cores i see no issues both with bhyve and windows 7. From owner-freebsd-hackers@freebsd.org Wed Nov 27 14:12:58 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DF4701AD046 for ; Wed, 27 Nov 2019 14:12:58 +0000 (UTC) (envelope-from bsdlisten@gmail.com) Received: from mail-il1-x129.google.com (mail-il1-x129.google.com [IPv6:2607:f8b0:4864:20::129]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47NN5j6dWBz4740 for ; Wed, 27 Nov 2019 14:12:57 +0000 (UTC) (envelope-from bsdlisten@gmail.com) Received: by mail-il1-x129.google.com with SMTP id o18so21110790ils.12 for ; Wed, 27 Nov 2019 06:12:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=2fjBaFBSJ78pcBkRC1vFaErHnGKXtFZX8exuyyoqJA4=; b=eSFT3HmLQaObae4LNY+ZW/q6WAhG8v2RBwbmXWS+zHnEFgZxVWrwjVxbaEGllBL5WJ 3hcxbspcutn13zZ9YO9HvEr5FFiYYjEUPKFBHUsXk9xHqBqikVlQYN0yp//NHvw5rCnJ VbBrU2Z9yLlYtM1B75VmpUUODJSL3YmcRr5zOdOMKTfXqFcpF0TppZqtnkxy0Mu7hZoI 5laSCluPlcSELWl0lKEmZ/lsxyQ8C2EBVUrHrn3RcFAhM4wYjLiQxcGmO4EZ3utkQFiF njaKBaERRIQg7v6xTu1mEpKXzU37T8Eddwj7y90HVkixkWMAhCI0yNym88LrNY2d2YsC mYFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=2fjBaFBSJ78pcBkRC1vFaErHnGKXtFZX8exuyyoqJA4=; b=e82MJEv8obPMdSFjkPmyTBgOXNKlJDL6rMr7QTVueGBXDz2h/ThD2+oct0JXDXSK2N nY1JS14R3S/QYmhWDnH6CUqj9TkiZShlpvaXGWC8d63ZDcS687dl3SFZcTqM8ZQ6FWlQ H2xhoLBRzDQH66aw5M6dBEimTnBjZnuXcKZGu67MXEvX9WqUz0JH2/8KWk4Ch6CVGbD7 We6vnc114jNZsJDUI12dC8+lQ4YZLQbxdCHKwMPzEhW2O8PNW+QBxBx/WKnifdaGtUdu huu8Y5yqAxhVF1pVB95ZtNjB0sYUGWWz/zMQj0au/RFDlVR+1lnM+l7hi8ZUDKrdFhpw 3a9A== X-Gm-Message-State: APjAAAWBbuc49yG2X5QVM2q/KoZwbDvHNStS/fskV+jBaTK2ESTUJKq7 t4nTRwIC4CMjyTP79ZWEiTzQKCW3rRdB016iji4XuA== X-Google-Smtp-Source: APXvYqz0KKBfkbRwMDuBjzmNfjBgWGCzo48gytKz7HyffahMPZonAxODVPBTcXxz1KUCZVJtws7uYm1TvAxPIhf7pIo= X-Received: by 2002:a05:6e02:d0c:: with SMTP id g12mr46760678ilj.114.1574863976094; Wed, 27 Nov 2019 06:12:56 -0800 (PST) MIME-Version: 1.0 From: Rares Aioanei Date: Wed, 27 Nov 2019 16:12:45 +0200 Message-ID: Subject: LLVM flags when building CURRENT To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 47NN5j6dWBz4740 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=eSFT3HmL; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of bsdlisten@gmail.com designates 2607:f8b0:4864:20::129 as permitted sender) smtp.mailfrom=bsdlisten@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(0.00)[ip: (-8.98), ipnet: 2607:f8b0::/32(-2.27), asn: 15169(-1.95), country: US(-0.05)]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[9.2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2019 14:12:58 -0000 Hey, Using 13.0-CURRENT so I'm using source to update the machine, which is a bit constrained when it comes to resources. My question is : Where is the best place to add a flag to cmake for LLVM (-DLLVM_TARGETS_TO_BUILD) so that it only builds for X86? From owner-freebsd-hackers@freebsd.org Wed Nov 27 14:32:22 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EF7EC1AD74A for ; Wed, 27 Nov 2019 14:32:22 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47NNX56s2Kz47x9 for ; Wed, 27 Nov 2019 14:32:21 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id xAREWIPN008131; Wed, 27 Nov 2019 06:32:18 -0800 (PST) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id xAREWIxM008130; Wed, 27 Nov 2019 06:32:18 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201911271432.xAREWIxM008130@gndrsh.dnsmgr.net> Subject: Re: what's got broken in bhyve after upgrade In-Reply-To: To: Wojciech Puchar Date: Wed, 27 Nov 2019 06:32:18 -0800 (PST) CC: "Rodney W. Grimes" , freebsd-hackers@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 47NNX56s2Kz47x9 X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd-rwg@gndrsh.dnsmgr.net has no SPF policy when checking 69.59.192.140) smtp.mailfrom=freebsd-rwg@gndrsh.dnsmgr.net X-Spamd-Result: default: False [1.16 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.02)[-0.022,0]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.24)[0.240,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.04)[ip: (0.14), ipnet: 69.59.192.0/19(0.07), asn: 13868(0.03), country: US(-0.05)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2019 14:32:23 -0000 > > > > If it has that form of -c you SHOULD stop using hw.vmm.topology.cores_per_package > > and start to use the new -c options. > > > > I do not believe this feature is in 11.3, but it may of been merged. > > yes this was that. all works fine Thank you, I do need to investigate the issue you saw because this should of been backwards compatible. I may of over looked a case that needs to be handled. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-hackers@freebsd.org Wed Nov 27 14:38:43 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 490D71ADA87 for ; Wed, 27 Nov 2019 14:38:43 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47NNgQ3nj0z48DQ for ; Wed, 27 Nov 2019 14:38:42 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id xAREcemR008163; Wed, 27 Nov 2019 06:38:40 -0800 (PST) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id xAREce2g008162; Wed, 27 Nov 2019 06:38:40 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201911271438.xAREce2g008162@gndrsh.dnsmgr.net> Subject: Re: bhyve - lots of cores In-Reply-To: To: Wojciech Puchar Date: Wed, 27 Nov 2019 06:38:40 -0800 (PST) CC: freebsd-hackers@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 47NNgQ3nj0z48DQ X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd-rwg@gndrsh.dnsmgr.net has no SPF policy when checking 69.59.192.140) smtp.mailfrom=freebsd-rwg@gndrsh.dnsmgr.net X-Spamd-Result: default: False [1.24 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.01)[-0.012,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.31)[0.313,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.04)[ip: (0.14), ipnet: 69.59.192.0/19(0.07), asn: 13868(0.03), country: US(-0.05)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2019 14:38:43 -0000 > does anyone have FreeBSD running host with LOTS of cores (>=32) and allow > me to run short test of bhyve? > > I would like to know > > a) does bhyve have scaling limits. > b) can does windows 7 x64 (which is for what i use bhyve :) run > efficiently on this. > > At least up to 12 cores i see no issues both with bhyve and windows 7. Since you are running on 11.3 I would need to verify some patches that are needed to get above 24 cores are infact in 11.3. The VM_MAXCPU cleanup I did was verified with a 32 core box, and has run a over commit vCPU count of 254, though that is not usable in any shape. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-hackers@freebsd.org Wed Nov 27 15:11:47 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1FA0A1AEB1E for ; Wed, 27 Nov 2019 15:11:47 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47NPPY3Qnjz4B6m for ; Wed, 27 Nov 2019 15:11:45 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 565A2C3169; Wed, 27 Nov 2019 16:11:36 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Y4FUXNDFDT-F; Wed, 27 Nov 2019 16:11:35 +0100 (CET) Received: from [192.168.10.67] (opteron [192.168.10.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id B1BA7C3168 for ; Wed, 27 Nov 2019 16:11:35 +0100 (CET) From: Willem Jan Withagen Subject: Process in T state does not want to die..... To: FreeBSD Hackers Message-ID: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> Date: Wed, 27 Nov 2019 16:11:33 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 47NPPY3Qnjz4B6m X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 2001:4cb8:90:ffff::3 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.26 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[digiware.nl]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; IP_SCORE(-2.96)[ip: (-9.52), ipnet: 2001:4cb8::/29(-4.61), asn: 28878(-0.71), country: NL(0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:2001:4cb8::/29, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2019 15:11:47 -0000 Hi, Probably a "dumb" question, but still I wondering what is going on... I have this ceph server running several OSDs (ceph-osd), now when they do not get certain responses within a time limit, they commit suicide. That is a rather convoluted process where they - call abort() - which is then trapped the ABORT signal handler Try to dump the logging state Try to dump stacktrace - either call _exit() or call reraise_fatal - reraise_fatal does some logging and calls exit(1) And then the process ends up as: root 3433 0.0 4.2 699944 353716 - TsJ 11Nov19 38:10.17 ceph-osd -i 2 Where the I state make it Terminated and no more processing is consumed. But the process one way or another is not going away and keeps resources locked that prevents starting a new daemon. It stays in that state for a 1) few minutes, and then it is gone from the processtable. 2) forever (>24h) But why doesn't the process die (right away)? Killing it -9 does not help. Trying to attach gdb brings nothing. If it disappears from the processtable, somethings there is a core. Do how do I debug this? --WjW From owner-freebsd-hackers@freebsd.org Wed Nov 27 15:36:56 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A0BF81AF58F for ; Wed, 27 Nov 2019 15:36:56 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47NPyb28b5z4CQh for ; Wed, 27 Nov 2019 15:36:54 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id xARFaHB9055857 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 27 Nov 2019 15:36:24 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: wjw@digiware.nl Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id xARFaEZG024709 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 27 Nov 2019 22:36:14 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Process in T state does not want to die..... To: Willem Jan Withagen , FreeBSD Hackers References: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> From: Eugene Grosbein Message-ID: <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> Date: Wed, 27 Nov 2019 22:36:05 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 47NPyb28b5z4CQh X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.77 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-1.67)[ip: (-4.41), ipnet: 2a01:4f8::/29(-2.34), asn: 24940(-1.57), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2019 15:36:56 -0000 27.11.2019 22:11, Willem Jan Withagen wrote: [skip] Process in state T is STOPPED and prohibited for execution. It cannot even process signal like SIGTERM or SIGKILL because they are queied waiting for continuation. You need to resume it with kill -CONT first. From owner-freebsd-hackers@freebsd.org Wed Nov 27 15:51:15 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A13291AFB51 for ; Wed, 27 Nov 2019 15:51:15 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-io1-f44.google.com (mail-io1-f44.google.com [209.85.166.44]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47NQH64lKZz4CyF for ; Wed, 27 Nov 2019 15:51:14 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-io1-f44.google.com with SMTP id j13so25439069ioe.0 for ; Wed, 27 Nov 2019 07:51:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=+xySph0V6fnBXNZpME8ToWi774iBEq3EfuWaE13OInc=; b=B+Pnb/E60hHb47lXfdAQJvCaLjwy9r63yeEOyeIesLGZGhdXuUHujv0hPqOOztuE6G KsHE+0PBdKN+BroxJawldVhYsxvbzNgD0Lk3ar/OB0avjkN9/a/QOxUWxss9mdXPkptd ZO/r7ooXxdkZdoOfXD8/QdOsg6XlTQ4nu/04W4Jh6/x5swJL32TGc6N3Zs3QWYdkZggl c91cygqzXzGTyTo/fsOurbvLKBPtt4xHi37cnkKJAUpABVz2hBwZ5/qHcq0i/YNXmK24 Zagc5KhLZDRWzQpiTE2kfuaulucg0c1u8Aci++Js4ik8C0kkDk0jSJRKAYjP3DoBsfA1 1FAA== X-Gm-Message-State: APjAAAW+SNapmkp6RXezwetUVjemg41732V5YqTnp5x4tQHPOg8f4e3W C6QvMGPIGIdFElPy3UR9LMLC83BVG7OWO3Kmenk= X-Google-Smtp-Source: APXvYqz80AFmTWozBSsW7vlcjm6oGLTSgnEYc6YM5Xs6jcXm9FOWikLgPw8gVbOtqDf0/5dyABOnSJpVMA63oN0zCeU= X-Received: by 2002:a02:9144:: with SMTP id b4mr5165349jag.118.1574869872615; Wed, 27 Nov 2019 07:51:12 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Ed Maste Date: Wed, 27 Nov 2019 07:04:38 -0500 Message-ID: Subject: Re: LLVM flags when building CURRENT To: Rares Aioanei Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 47NQH64lKZz4CyF X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.166.44 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [-3.71 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; MIME_TRACE(0.00)[0:+]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[44.166.85.209.list.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-1.71)[ip: (-3.39), ipnet: 209.85.128.0/17(-3.16), asn: 15169(-1.95), country: US(-0.05)]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[44.166.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2019 15:51:15 -0000 On Wed, 27 Nov 2019 at 09:13, Rares Aioanei wrote: > > Hey, > Using 13.0-CURRENT so I'm using source to update the machine, which is > a bit constrained when it comes to resources. My question is : Where > is the best place to add a flag to cmake for LLVM > (-DLLVM_TARGETS_TO_BUILD) so that it only builds for X86? Have a look at the src.conf man page and the WITHOUT_LLVM_TARGET_ALL option. From owner-freebsd-hackers@freebsd.org Thu Nov 28 02:55:55 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 74C411C1B8E for ; Thu, 28 Nov 2019 02:55:55 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Nj2224R9z3xk4 for ; Thu, 28 Nov 2019 02:55:53 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id xAS2pjk4060984 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 28 Nov 2019 02:51:49 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: bsdlisten@gmail.com Received: from [10.58.0.4] (dadv@[10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id xAS2pbA8031083 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 28 Nov 2019 09:51:37 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: LLVM flags when building CURRENT To: Rares Aioanei , freebsd-hackers@freebsd.org References: From: Eugene Grosbein Message-ID: <53796231-2772-da45-2829-11a0e6e57956@grosbein.net> Date: Thu, 28 Nov 2019 09:51:34 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=4.1 required=5.0 tests=BAYES_00, DATE_IN_FUTURE_96_Q, HELO_MISC_IP, LOCAL_FROM, SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.7 DATE_IN_FUTURE_96_Q Date: is 4 days to 4 months after Received: * date * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains * 1.1 HELO_MISC_IP Looking for more Dynamic IP Relays X-Spam-Level: **** X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on eg.sd.rdtc.ru X-Rspamd-Queue-Id: 47Nj2224R9z3xk4 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.77 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-1.67)[ip: (-4.43), ipnet: 2a01:4f8::/29(-2.34), asn: 24940(-1.57), country: DE(-0.01)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 02:55:55 -0000 27.11.2019 21:12, Rares Aioanei wrote: > Hey, > Using 13.0-CURRENT so I'm using source to update the machine, which is > a bit constrained when it comes to resources. My question is : Where > is the best place to add a flag to cmake for LLVM > (-DLLVM_TARGETS_TO_BUILD) so that it only builds for X86? 1. kldload filemon # and perhaps auto-load it with /boot/loader.conf or /etc/rc.conf 2. echo WITH_META_MODE=yes >> /etc/src-env.conf 3. for /etc/src.conf: WITHOUT_LLVM_TARGET_ALL= WITH_LLVM_TARGET_X86= WITHOUT_CLANG_FULL= 4. If its resources are really constrained, also add this to /etc/src.conf: WITHOUT_KERNEL_SYMBOLS= WITHOUT_DEBUG_FILES= WITHOUT_TESTS= From owner-freebsd-hackers@freebsd.org Thu Nov 28 09:10:59 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4186B1CA1A9 for ; Thu, 28 Nov 2019 09:10:59 +0000 (UTC) (envelope-from bsdlisten@gmail.com) Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47NsLq11lsz4HnT; Thu, 28 Nov 2019 09:10:58 +0000 (UTC) (envelope-from bsdlisten@gmail.com) Received: by mail-io1-xd36.google.com with SMTP id x21so28183960ior.2; Thu, 28 Nov 2019 01:10:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=g0x+KiMFWiZVea0w1ZOk8/KxrkdCHxiM3TL2Qx6S9qw=; b=PYp7CegLIIkPSfBrX+WdKr3Wb7rOZyf9Z3ZQNSb49MyJI8y/EJFZpgI8tDwcgA3ahd O6JEwhj9OJqZNn/CRNrBlT6M+6s9uBF0MftZoul65/EMcV/nqDGPxg08zFeCKE0xCwna HRfyWytBjTqy+FAnzYeOT6kMYDgAhBFY3P1BDy8SAWD3T7Ql9546HapE/UDaHOupR9bL xL6MM/8WMW1UAvhChyyccXcKl1s+EBVBl96mdoqkS1Yic6/7SyI20rIwE7wxuD14Uc7W F/kk39bqdVKADEnWPdkErPpEsY5sr9dNRj+S8UPvewhs2Q9gjraqk2nRJQtG47sxf8eS JBFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=g0x+KiMFWiZVea0w1ZOk8/KxrkdCHxiM3TL2Qx6S9qw=; b=swFnKkZ+7rSz4eouDN0ZzUHAZb9ERxKOo/O+uYfRSxf1MHoGo7nD87COX6i443+LHs aV4MIE+MDpBRRUbBzOCqhg49oYZm9+3z+0Kpf+G3d79HkFPlMZSVUFeDwp+A5bVgBQir VWmpeiepIsQruPktf397kGZYDUo5481i3tbxkx82zsgwb0PPfh8QWlSbIUFc23M3I1Et 3lUHezRk2fWNnFaZW855HoYGEzGO2mR99oHLDdWloJ3su4yNOLQk/7KIWrH31hqg0PFx gYNz6yA0r0icFgb/qDKBHuhfXUpS+03xHizqD9dPqO+SwTNFzJQK/8o6jIIMn7RlMlvq GgnQ== X-Gm-Message-State: APjAAAUosqgK25WWTVlIMFLfBn9+gKcSoJHHs6ldjiUsNg+MqTRy14bS SyY6+NJ2/WwPZLOlyENIfE+3c0vxL0bihugRbVAXvw== X-Google-Smtp-Source: APXvYqyuxbkTOIgDfwa7sKRyXjIgEM0cgzT6QTJsquIyDqKBfg2xrsyz8JtNrwpFddUg6rSUteGvGNBsnuczHTcFLwA= X-Received: by 2002:a02:a148:: with SMTP id m8mr6243008jah.69.1574932257768; Thu, 28 Nov 2019 01:10:57 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Rares Aioanei Date: Thu, 28 Nov 2019 11:10:46 +0200 Message-ID: Subject: Re: LLVM flags when building CURRENT To: Ed Maste Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 47NsLq11lsz4HnT X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 09:10:59 -0000 Thank you, it's what I needed. On Wed, Nov 27, 2019 at 5:51 PM Ed Maste wrote: > > On Wed, 27 Nov 2019 at 09:13, Rares Aioanei wrote: > > > > Hey, > > Using 13.0-CURRENT so I'm using source to update the machine, which is > > a bit constrained when it comes to resources. My question is : Where > > is the best place to add a flag to cmake for LLVM > > (-DLLVM_TARGETS_TO_BUILD) so that it only builds for X86? > > Have a look at the src.conf man page and the WITHOUT_LLVM_TARGET_ALL option. From owner-freebsd-hackers@freebsd.org Thu Nov 28 09:43:26 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B791C1CAEF0; Thu, 28 Nov 2019 09:43:26 +0000 (UTC) (envelope-from f0andrey@gmail.com) Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Nt4F60KBz4Kwc; Thu, 28 Nov 2019 09:43:25 +0000 (UTC) (envelope-from f0andrey@gmail.com) Received: by mail-wm1-x32f.google.com with SMTP id z19so10960669wmk.3; Thu, 28 Nov 2019 01:43:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=xI7O7tW7yg5xJPC3tIY4r5a3pCvrxjHlIYP1qOysTvg=; b=EOr2NuXdoRC9qjsObYGDPRK6FkJDgm5nanIF+QXIc2FK78PNp/W3YKuRq9Xq7AAJba dBCHdSs8+8s6OxNu9wk0BzAg/RKYXTWuTl8dj9DPKoAspr1RkwItAXDyhT0rR7MSchf+ 0xeJFDMjmIeZLLN0v8yqn+b47WLw/R8lKyrAP0IzCgwBnx/QGFsEHG7lptiFPZKKlo4Y HVySWZyyG4l/8F9+K+fHQ/6j29jx7tyLdGT0XbhsRljvLNYcDLJmkQ/NWM5E2XHLeRa2 ezV+KtxV8mIuuq1zv078Ur7nrZkISqMqr5KYWGTVs7Wus8bcTwFvCAAkpPUnCK26Bl/a to7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=xI7O7tW7yg5xJPC3tIY4r5a3pCvrxjHlIYP1qOysTvg=; b=AG3jOvXtHoGRYa+/dWrBcuTupKjEMEfXq3jxG92yWvpdKI9SUueQNb6Q4M0EqMdlnF dO+5Vvsy8c5tRFnZRMAUm6C09WkwPcgr4nbFwgr8YwUHFdfOlT4lKYy/DS0L0cJfcDXG g7p64W4lkzyDDjO7xZGd2iEsFMyie8w7uxzX3AtKwr5wTo73GEMF7nIK6dB4owGvg8VX 1Ta8g3hNR69kj+Y4/aaNwGOp+1L/aPuBfPQb/nBNebamgDUJLdg/k5+HWpVufKgvdYih 1wO6WtKHMXVZUdbB1xq6VZt+jive8G5t4ShdkZWQXoeDuqudwKHWXr6rzkQO+DJI5RDy Kztw== X-Gm-Message-State: APjAAAUh7CbL9cPFc/Gzp3HBJakeuL6hO0+umVtPc2QKI0jnybntU1xf 5JeH/AzbkGqoqJXPNBe/+aN+/R9gY4ovsZsE3Wuzr8ZGr5M= X-Google-Smtp-Source: APXvYqwM5cT2ZLi3FAgdSC8KZS7CLmKc4f8vqqoEt3cqLZHeQpmAjbtjqmT9sQTslMciLLsc1thUcJFGd0MCBvYKyRk= X-Received: by 2002:a1c:2c43:: with SMTP id s64mr5239576wms.135.1574934203877; Thu, 28 Nov 2019 01:43:23 -0800 (PST) MIME-Version: 1.0 From: Andrey Fesenko Date: Thu, 28 Nov 2019 12:43:09 +0300 Message-ID: Subject: EFI loader failure, after 20191114-r354699 Z87MX-D3H To: freebsd-current , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 47Nt4F60KBz4Kwc X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=EOr2NuXd; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of f0andrey@gmail.com designates 2a00:1450:4864:20::32f as permitted sender) smtp.mailfrom=f0andrey@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; IP_SCORE(0.00)[ip: (-8.90), ipnet: 2a00:1450::/32(-2.69), asn: 15169(-1.95), country: US(-0.05)]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[f.2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 09:43:26 -0000 Hello, Around starting 20191114 r354699 (memstick tested), my desktop not boot normally. Boot only loader menu (black and white mode) after start i'm see load modules, after this monitor gray, and 15-20s disabled, system block not disable but run silently. system not boot. If i'm change efi (EFI/BOOT/bootx64.efi), 20191031-r354207 or 12.1 release, system boot normally My notebook x220 work fine any loaders. desktop BIOS Information Vendor: American Megatrends Inc. Version: F2 Release Date: 05/03/2013 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 16 MB Characteristics: PCI is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported BIOS ROM is socketed EDD is supported 5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 kB floppy services are supported (int 13h) 3.5"/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) ACPI is supported USB legacy is supported BIOS boot specification is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 4.6 Handle 0x0001, DMI type 1, 27 bytes System Information Manufacturer: Gigabyte Technology Co., Ltd. Product Name: Z87MX-D3H Version: To be filled by O.E.M. Serial Number: To be filled by O.E.M. UUID: 03de0294-0480-057d-0c06-7a0700080009 Wake-up Type: Power Switch SKU Number: To be filled by O.E.M. Family: To be filled by O.E.M. From owner-freebsd-hackers@freebsd.org Thu Nov 28 10:24:48 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9D4FF1CC4BC; Thu, 28 Nov 2019 10:24:48 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Ntzz4WYrz4NYd; Thu, 28 Nov 2019 10:24:47 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id xASAKhi3064730 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 28 Nov 2019 10:20:44 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: f0andrey@gmail.com Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id xASAKffe036226 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 28 Nov 2019 17:20:41 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: EFI loader failure, after 20191114-r354699 Z87MX-D3H To: Andrey Fesenko , freebsd-current , freebsd-hackers@freebsd.org References: From: Eugene Grosbein Message-ID: Date: Thu, 28 Nov 2019 17:20:35 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Spam-Flag: YES X-Spam-Status: Yes, score=6.0 required=5.0 tests=BAYES_00, DATE_IN_FUTURE_96_Q, HELO_MISC_IP,LOCAL_FROM,RDNS_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.7 DATE_IN_FUTURE_96_Q Date: is 4 days to 4 months after Received: * date * -0.0 SPF_PASS SPF: sender matches SPF record * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains * 1.1 HELO_MISC_IP Looking for more Dynamic IP Relays X-Spam-Level: ***** X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on eg.sd.rdtc.ru X-Rspamd-Queue-Id: 47Ntzz4WYrz4NYd X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [1.35 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.92)[-0.920,0]; SPAM_FLAG(5.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.96)[-0.961,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; IP_SCORE(-1.67)[ip: (-4.42), ipnet: 2a01:4f8::/29(-2.33), asn: 24940(-1.57), country: DE(-0.01)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 10:24:48 -0000 28.11.2019 16:43, Andrey Fesenko wrote: > Hello, > > Around starting 20191114 r354699 (memstick tested), my desktop not > boot normally. Boot only loader menu (black and white mode) after > start i'm see load modules, after this monitor gray, and 15-20s > disabled, system block not disable but run silently. system not boot. So loader works. Try breaking to loader prompt and do: set hw.vga.acpi_ignore_no_vga=1 boot See if it helps. From owner-freebsd-hackers@freebsd.org Thu Nov 28 10:37:57 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 15C0E1CCAA8 for ; Thu, 28 Nov 2019 10:37:57 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47NvH80kC5z4PKZ for ; Thu, 28 Nov 2019 10:37:55 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 5D01EC2A0C; Thu, 28 Nov 2019 11:37:51 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 24rYapctTzJc; Thu, 28 Nov 2019 11:37:50 +0100 (CET) Received: from [192.168.10.9] (vaio [192.168.10.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 9E1B2C2A0B; Thu, 28 Nov 2019 11:37:50 +0100 (CET) Subject: Re: Process in T state does not want to die..... To: Eugene Grosbein , FreeBSD Hackers References: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> From: Willem Jan Withagen Message-ID: <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> Date: Thu, 28 Nov 2019 11:37:50 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1 MIME-Version: 1.0 In-Reply-To: <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: nl X-Rspamd-Queue-Id: 47NvH80kC5z4PKZ X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 2001:4cb8:90:ffff::3 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.27 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[digiware.nl]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-2.97)[ip: (-9.52), ipnet: 2001:4cb8::/29(-4.62), asn: 28878(-0.73), country: NL(0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:2001:4cb8::/29, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 10:37:57 -0000 On 27-11-2019 16:36, Eugene Grosbein wrote: > 27.11.2019 22:11, Willem Jan Withagen wrote: > > [skip] > > Process in state T is STOPPED and prohibited for execution. > It cannot even process signal like SIGTERM or SIGKILL because they are queied waiting for continuation. > You need to resume it with kill -CONT first. Tried that several times, but does not really have any effect. I could check and see if the signals (TERM, KILL) were waiting somewhere? With procstat?? But the original question was more for a way on preventing this state of affairs. Because uptill now the only resolution was to reboot the server, which is not a nice thing for a storage sollution. Hence the: how to debug? question. --WjW From owner-freebsd-hackers@freebsd.org Thu Nov 28 11:08:56 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 40A451CDA28 for ; Thu, 28 Nov 2019 11:08:56 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Nvyv2kRpz4RHt for ; Thu, 28 Nov 2019 11:08:54 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id xASB4lT1065042 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 28 Nov 2019 11:04:50 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: wjw@digiware.nl Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id xASB4iFY036634 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 28 Nov 2019 18:04:44 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Process in T state does not want to die..... To: Willem Jan Withagen , FreeBSD Hackers References: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> From: Eugene Grosbein Message-ID: <0d698f4e-decf-8d4b-7a05-d5985558fb07@grosbein.net> Date: Thu, 28 Nov 2019 18:04:39 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Flag: YES X-Spam-Status: Yes, score=6.0 required=5.0 tests=BAYES_00, DATE_IN_FUTURE_96_Q, HELO_MISC_IP,LOCAL_FROM,RDNS_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.7 DATE_IN_FUTURE_96_Q Date: is 4 days to 4 months after Received: * date * -0.0 SPF_PASS SPF: sender matches SPF record * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains * 1.1 HELO_MISC_IP Looking for more Dynamic IP Relays X-Spam-Level: ***** X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on eg.sd.rdtc.ru X-Rspamd-Queue-Id: 47Nvyv2kRpz4RHt X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [1.30 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.96)[-0.960,0]; SPAM_FLAG(5.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.98)[-0.981,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-1.66)[ip: (-4.38), ipnet: 2a01:4f8::/29(-2.33), asn: 24940(-1.57), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 11:08:56 -0000 28.11.2019 17:37, Willem Jan Withagen wrote: >> Process in state T is STOPPED and prohibited for execution. >> It cannot even process signal like SIGTERM or SIGKILL because they are queied waiting for continuation. >> You need to resume it with kill -CONT first. > > Tried that several times, but does not really have any effect. > I could check and see if the signals (TERM, KILL) were waiting somewhere? > With procstat?? > > But the original question was more for a way on preventing this state of affairs. > Because uptill now the only resolution was to reboot the server, which is not a nice > thing for a storage sollution. > > Hence the: how to debug? question. Sometimes background process is (mistakenly) started without closing/redirecting output and if it tries to print something, it might get stopped depending on used shell. Use ktrace/kdump to verify what's going on with the process if you can reproduce the problem. You did not specify which version you use. Sometimes flags shown by ps are wrong due to kernel bugs and corrupted process state flag and process is not really Stopped but Exiting for example. ktrace/kdump should show if process got SIGSTOP/SIGTSTP really. And if SIGCONT was delivered. From owner-freebsd-hackers@freebsd.org Thu Nov 28 11:51:33 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 677311CE744 for ; Thu, 28 Nov 2019 11:51:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47Nww40qcHz4Svp for ; Thu, 28 Nov 2019 11:51:31 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id xASBpND4092771 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 28 Nov 2019 13:51:26 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua xASBpND4092771 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id xASBpMKg092770; Thu, 28 Nov 2019 13:51:22 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 28 Nov 2019 13:51:22 +0200 From: Konstantin Belousov To: Willem Jan Withagen Cc: Eugene Grosbein , FreeBSD Hackers Subject: Re: Process in T state does not want to die..... Message-ID: <20191128115122.GN10580@kib.kiev.ua> References: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> User-Agent: Mutt/1.12.2 (2019-09-21) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 47Nww40qcHz4Svp X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; IP_SCORE(0.00)[ip: (-2.79), ipnet: 2001:470::/32(-4.64), asn: 6939(-3.52), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 11:51:33 -0000 On Thu, Nov 28, 2019 at 11:37:50AM +0100, Willem Jan Withagen wrote: > On 27-11-2019 16:36, Eugene Grosbein wrote: > > 27.11.2019 22:11, Willem Jan Withagen wrote: > > > > [skip] > > > > Process in state T is STOPPED and prohibited for execution. > > It cannot even process signal like SIGTERM or SIGKILL because they are queied waiting for continuation. > > You need to resume it with kill -CONT first. > > Tried that several times, but does not really have any effect. > I could check and see if the signals (TERM, KILL) were waiting somewhere? > With procstat?? > > But the original question was more for a way on preventing this state of > affairs. > Because uptill now the only resolution was to reboot the server, which > is not a nice > thing for a storage sollution. > > Hence the: how to debug? question. Start with the output from $ procstat -kk -L Also useful is the output from $ ps -o pid,lwp,flags,flags2,state,tracer,command -p After that is shown, it could be determined where to look next. What is your system version ? From owner-freebsd-hackers@freebsd.org Thu Nov 28 12:50:21 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6F21A1A861A for ; Thu, 28 Nov 2019 12:50:21 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from smtpq2.tb.mail.iss.as9143.net (smtpq2.tb.mail.iss.as9143.net [212.54.42.165]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47NyCw4SHlz4X7V for ; Thu, 28 Nov 2019 12:50:20 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from [212.54.42.136] (helo=smtp12.tb.mail.iss.as9143.net) by smtpq2.tb.mail.iss.as9143.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1iaJFR-0006s4-Tf for freebsd-hackers@freebsd.org; Thu, 28 Nov 2019 13:50:17 +0100 Received: from 94-209-85-88.cable.dynamic.v4.ziggo.nl ([94.209.85.88] helo=wan0.bsd4all.org) by smtp12.tb.mail.iss.as9143.net with esmtp (Exim 4.90_1) (envelope-from ) id 1iaJFR-00041C-Lf for freebsd-hackers@freebsd.org; Thu, 28 Nov 2019 13:50:17 +0100 Received: from newnas.bsd4all.local (localhost [127.0.0.1]) by wan0.bsd4all.org (Postfix) with ESMTP id 25C4A8F for ; Thu, 28 Nov 2019 13:50:17 +0100 (CET) X-Virus-Scanned: amavisd-new at bsd4all.org Received: from wan0.bsd4all.org ([127.0.0.1]) by newnas.bsd4all.local (newnas.bsd4all.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dNhmH3nhjBTu for ; Thu, 28 Nov 2019 13:50:16 +0100 (CET) Received: from [192.168.1.65] (unknown [192.168.1.65]) by wan0.bsd4all.org (Postfix) with ESMTPSA id 258C040 for ; Thu, 28 Nov 2019 13:50:16 +0100 (CET) From: Peter Blok Content-Type: multipart/signed; boundary="Apple-Mail=_A7A4EEBC-984D-4FC9-AAF9-79BF83D504E0"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: dynamic loadable library multiple degined symbols Message-Id: Date: Thu, 28 Nov 2019 13:50:15 +0100 To: freebsd-hackers@freebsd.org X-Mailer: Apple Mail (2.3445.104.11) X-SourceIP: 94.209.85.88 X-Ziggo-spambar: / X-Ziggo-spamscore: 0.0 X-Ziggo-spamreport: CMAE Analysis: v=2.3 cv=Rvq70xuK c=1 sm=1 tr=0 a=LYXyOGYQqFYBMgK+Y6iqTg==:17 a=MeAgGD-zjQ4A:10 a=MW9R8kzCsCf5i6-9gGMA:9 a=CjuIK1q_8ugA:10 a=g2jUX3k5JNHoXd8hjKIA:9 a=ZVk8-NSrHBgA:10 X-Ziggo-Spam-Status: No X-Spam-Status: No X-Spam-Flag: No X-Rspamd-Queue-Id: 47NyCw4SHlz4X7V X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pblok@bsd4all.org designates 212.54.42.165 as permitted sender) smtp.mailfrom=pblok@bsd4all.org X-Spamd-Result: default: False [-5.26 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; RECEIVED_SPAMHAUS_PBL(0.00)[88.85.209.94.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; R_SPF_ALLOW(-0.20)[+a:smtp.ziggo.nl/16]; TO_MATCH_ENVRCPT_ALL(0.00)[]; HAS_ATTACHMENT(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_TLS_LAST(0.00)[]; MV_CASE(0.50)[]; DMARC_NA(0.00)[bsd4all.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; IP_SCORE(-1.26)[ipnet: 212.54.32.0/20(-3.97), asn: 33915(-2.33), country: NL(0.02)]; RCVD_IN_DNSWL_LOW(-0.10)[165.42.54.212.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:33915, ipnet:212.54.32.0/20, country:NL]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 12:50:21 -0000 --Apple-Mail=_A7A4EEBC-984D-4FC9-AAF9-79BF83D504E0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, named (bind9.14) has a function called dns_name_equal. (0000000000443ac0 = T dns_name_equal) named dynamically loads dlz_bind9_14.so is build from dlz_bind9.c and = calls this function, but dns_name_equal was undefined so it got resolved = to the version of named. The function is defined in dns_utils.c, so I changed the building to = include that file. Now dlz_bind9_14.so is using dlz_bind9.c and dns_utils.c also has the = right dns_name_equal (000000000000bee0 T dns_name_equal) defined =20 Unfortunately the code inside dlz_bind9_14.so still calls the function = out of named. Is this something that should have been resolved at compile/link time of = dlz_bind9_14.so? If so, how? Note that the samba build uses waf and wscript files. Peter --Apple-Mail=_A7A4EEBC-984D-4FC9-AAF9-79BF83D504E0 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCBSAw ggUcMIIEBKADAgECAhEAq2wFIs+rCK6H6/2jbblXhDANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgwNDE0MDAwMDAwWhcNMjEwNDEzMjM1 OTU5WjBEMQswCQYDVQQGEwJOTDETMBEGA1UEAxMKUGV0ZXIgQmxvazEgMB4GCSqGSIb3DQEJARYR cGJsb2tAYnNkNGFsbC5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPT/3evs2a zLSIVepGa9qFVcSISd5HzoJt9xAyQ4od7NM6Qzwm446OyhzWsIN/a6+nDNB4AxzSg00QXKx4afEa FrdLzmREEfv24f88j2UZYqHAls0j26jyED5FZ068xs4gWZBG2U7EVTUNNJuUrrmqBNZkGxTIrFrD Cgr1EpRULpN+HrEelHHh7uR0twAjvwcyXkG9DbDJXnw8HzKGR80ik4+13HDxx4mDxOY4NOvWSSiM kEFS2Z2AKtxXSMBQZHazAUvbka27c1m93/QsjnDF+P6Aef9NEvUDL9mU9Jbf/+5V+anT2KdPGP4p rQ9gA/Nup61qxDkwc+RupiXD5NSbAgMBAAGjggGzMIIBrzAfBgNVHSMEGDAWgBSCr2yM+MX+lmF8 6B89K3FIXsSLwDAdBgNVHQ4EFgQUjwe7n1zvxFkTeCUYWrsaJpOGP14wDgYDVR0PAQH/BAQDAgWg MAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUFBwMCMEYGA1UdIAQ/MD0w OwYMKwYBBAGyMQECAQMFMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQv Q1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNs aWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmwwgYsGCCsGAQUFBwEBBH8wfTBV BggrBgEFBQcwAoZJaHR0cDovL2NydC5jb21vZG9jYS5jb20vQ09NT0RPUlNBQ2xpZW50QXV0aGVu dGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29t b2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQC85hVlqTVwt218IJR/WjMiMnDtZ7hY860XKjzO uB3sUUQwHxHj+ZYuMbAfVLZGGqh1EekbwDMVgkK9cezIHM+ZzxrNGX2SJyl1YW+3FLn52P0uIlmA VPFjUowf5qBhOHl2NJo+WXYZhQY7rT/xSygE81o3oLE/A4zO6WtO3PeZpFpZNrBvizAsjTDfPeXW iQzXz6NLrgwert0Wml95ov2rG5oCzHYPijabubSNm2NdUjPRtcVylcqAThXOvp6X4UvW8/L0uhkp 9WsKP2JEJ3Zukv7Ib+vMBsdE4tf4rmv89pQC+lLpD08ze/QDCIeFBCRIihcC2PycDQrnNIp1RAIh MYIDyjCCA8YCAQEwga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0 ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQD EzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEA q2wFIs+rCK6H6/2jbblXhDANBglghkgBZQMEAgEFAKCCAe0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkxMTI4MTI1MDE1WjAvBgkqhkiG9w0BCQQxIgQgSZIINZoV RHNmO2X6/05jrJVsva6meHbuqSsmH0tlz3gwgb4GCSsGAQQBgjcQBDGBsDCBrTCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCrbAUiz6sIrofr/aNtuVeEMIHABgsqhkiG 9w0BCRACCzGBsKCBrTCBlzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3Rl cjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMT NENPTU9ETyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCr bAUiz6sIrofr/aNtuVeEMA0GCSqGSIb3DQEBAQUABIIBACZs+LDawAMdcozsniergc46PPPHZVre 5+DIVoAP/V9+O0na7SvvbPbz18/NoQ3ERlcPmCxIWj4/hksB/TGFqdHmx+21nZ/KSGd1LplO7RGG 2t6bMYoaa7WWFi/PH5O5l2WTm7s8HWfSU6tB1z101U9d6bgmWo4egmJpFsTe6YV61hTS1w94tKip c/ccfEc9gAMhvYn+uT4kn/oM8zW+dwWTdXrNAI0QheTq4qaljVi9Q7/rfG9UODlRwyCvNYYxalm2 Eaw8bDCdx8/JdNUG8g8JeMiEX84jtEVR7Rd0rDCY+27iazR66F1br96fetYI811TWgMpgvXzipUL F4oBuY4AAAAAAAA= --Apple-Mail=_A7A4EEBC-984D-4FC9-AAF9-79BF83D504E0-- From owner-freebsd-hackers@freebsd.org Thu Nov 28 13:12:21 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EF15D1A9327 for ; Thu, 28 Nov 2019 13:12:21 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47NyjK1m5Nz4YjS for ; Thu, 28 Nov 2019 13:12:21 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id xASDCBFY012713 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 28 Nov 2019 15:12:15 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua xASDCBFY012713 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id xASDCBP4012712; Thu, 28 Nov 2019 15:12:11 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 28 Nov 2019 15:12:11 +0200 From: Konstantin Belousov To: Peter Blok Cc: freebsd-hackers@freebsd.org Subject: Re: dynamic loadable library multiple degined symbols Message-ID: <20191128131211.GQ10580@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.2 (2019-09-21) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 47NyjK1m5Nz4YjS X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(0.00)[ip: (-2.77), ipnet: 2001:470::/32(-4.64), asn: 6939(-3.52), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 13:12:22 -0000 On Thu, Nov 28, 2019 at 01:50:15PM +0100, Peter Blok wrote: > Hi, > > named (bind9.14) has a function called dns_name_equal. (0000000000443ac0 T dns_name_equal) > > named dynamically loads dlz_bind9_14.so is build from dlz_bind9.c and calls this function, but dns_name_equal was undefined so it got resolved to the version of named. > > The function is defined in dns_utils.c, so I changed the building to include that file. > > Now dlz_bind9_14.so is using dlz_bind9.c and dns_utils.c also has the right dns_name_equal (000000000000bee0 T dns_name_equal) defined > > Unfortunately the code inside dlz_bind9_14.so still calls the function out of named. > > Is this something that should have been resolved at compile/link time of dlz_bind9_14.so? If so, how? No, default ELF name resolution rules would give the behaviour you described, assuming the main binary was linked with -Wl,-E (and it must be to export symbols to loadable modules). The shared libraries and loadable modules are interposable by default, unless linked with -B symbolic, and the symbol resolution order starts from the main binary object. Why do you try to change this ? > > Note that the samba build uses waf and wscript files. > > Peter > > From owner-freebsd-hackers@freebsd.org Thu Nov 28 13:48:31 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A08DB1AA391 for ; Thu, 28 Nov 2019 13:48:31 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from smtpq6.tb.mail.iss.as9143.net (smtpq6.tb.mail.iss.as9143.net [212.54.42.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47NzW264mZz4bRK for ; Thu, 28 Nov 2019 13:48:30 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from [212.54.42.134] (helo=smtp10.tb.mail.iss.as9143.net) by smtpq6.tb.mail.iss.as9143.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1iaK9l-0006Nk-3h; Thu, 28 Nov 2019 14:48:29 +0100 Received: from 94-209-85-88.cable.dynamic.v4.ziggo.nl ([94.209.85.88] helo=wan0.bsd4all.org) by smtp10.tb.mail.iss.as9143.net with esmtp (Exim 4.90_1) (envelope-from ) id 1iaK9k-0000cP-Vg; Thu, 28 Nov 2019 14:48:29 +0100 Received: from newnas.bsd4all.local (localhost [127.0.0.1]) by wan0.bsd4all.org (Postfix) with ESMTP id 8A4FF92; Thu, 28 Nov 2019 14:48:28 +0100 (CET) X-Virus-Scanned: amavisd-new at bsd4all.org Received: from wan0.bsd4all.org ([127.0.0.1]) by newnas.bsd4all.local (newnas.bsd4all.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GF3t9egZbyi2; Thu, 28 Nov 2019 14:48:27 +0100 (CET) Received: from [192.168.1.65] (unknown [192.168.1.65]) by wan0.bsd4all.org (Postfix) with ESMTPSA id A436F1B7; Thu, 28 Nov 2019 14:48:27 +0100 (CET) From: Peter Blok Message-Id: <58EC7517-CFE7-442F-9A9B-00849E32BCA4@bsd4all.org> Content-Type: multipart/signed; boundary="Apple-Mail=_373C4CF8-B65C-41D2-AF20-6B293E9FE8BA"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: dynamic loadable library multiple degined symbols Date: Thu, 28 Nov 2019 14:48:27 +0100 In-Reply-To: <20191128131211.GQ10580@kib.kiev.ua> Cc: freebsd-hackers@freebsd.org To: Konstantin Belousov References: <20191128131211.GQ10580@kib.kiev.ua> X-Mailer: Apple Mail (2.3445.104.11) X-SourceIP: 94.209.85.88 X-Ziggo-spambar: / X-Ziggo-spamscore: 0.0 X-Ziggo-spamreport: CMAE Analysis: v=2.3 cv=R5595uZX c=1 sm=1 tr=0 a=LYXyOGYQqFYBMgK+Y6iqTg==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=MeAgGD-zjQ4A:10 a=pGLkceISAAAA:8 a=Q-j6Hl_hjpYZ2wg1R-kA:9 a=QEXdDO2ut3YA:10 a=vu0-PkTyB2cw7_YyQL0A:9 a=ZVk8-NSrHBgA:10 X-Ziggo-Spam-Status: No X-Spam-Status: No X-Spam-Flag: No X-Rspamd-Queue-Id: 47NzW264mZz4bRK X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pblok@bsd4all.org designates 212.54.42.169 as permitted sender) smtp.mailfrom=pblok@bsd4all.org X-Spamd-Result: default: False [-5.29 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; RECEIVED_SPAMHAUS_PBL(0.00)[88.85.209.94.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; R_SPF_ALLOW(-0.20)[+a:smtp.ziggo.nl/16]; MV_CASE(0.50)[]; HAS_ATTACHMENT(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[bsd4all.org]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_IN_DNSWL_LOW(-0.10)[169.42.54.212.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:33915, ipnet:212.54.32.0/20, country:NL]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-1.29)[ipnet: 212.54.32.0/20(-4.03), asn: 33915(-2.43), country: NL(0.03)]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 13:48:31 -0000 --Apple-Mail=_373C4CF8-B65C-41D2-AF20-6B293E9FE8BA Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I=E2=80=99m trying to change this because named dies with an assert. = named checks the arguments of dns_name_equal which is completely = different from the one intended out of the shared module. > On 28 Nov 2019, at 14:12, Konstantin Belousov = wrote: >=20 > On Thu, Nov 28, 2019 at 01:50:15PM +0100, Peter Blok wrote: >> Hi, >>=20 >> named (bind9.14) has a function called dns_name_equal. = (0000000000443ac0 T dns_name_equal) >>=20 >> named dynamically loads dlz_bind9_14.so is build from dlz_bind9.c = and calls this function, but dns_name_equal was undefined so it got = resolved to the version of named. >>=20 >> The function is defined in dns_utils.c, so I changed the building to = include that file. >>=20 >> Now dlz_bind9_14.so is using dlz_bind9.c and dns_utils.c also has the = right dns_name_equal (000000000000bee0 T dns_name_equal) defined >>=20 >> Unfortunately the code inside dlz_bind9_14.so still calls the = function out of named. >>=20 >> Is this something that should have been resolved at compile/link time = of dlz_bind9_14.so? If so, how? > No, default ELF name resolution rules would give the behaviour you = described, > assuming the main binary was linked with -Wl,-E (and it must be to = export > symbols to loadable modules). The shared libraries and loadable = modules > are interposable by default, unless linked with -B symbolic, and the = symbol > resolution order starts from the main binary object. >=20 > Why do you try to change this ? >=20 >>=20 >> Note that the samba build uses waf and wscript files. >>=20 >> Peter >>=20 >>=20 >=20 >=20 --Apple-Mail=_373C4CF8-B65C-41D2-AF20-6B293E9FE8BA Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCBSAw ggUcMIIEBKADAgECAhEAq2wFIs+rCK6H6/2jbblXhDANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgwNDE0MDAwMDAwWhcNMjEwNDEzMjM1 OTU5WjBEMQswCQYDVQQGEwJOTDETMBEGA1UEAxMKUGV0ZXIgQmxvazEgMB4GCSqGSIb3DQEJARYR cGJsb2tAYnNkNGFsbC5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPT/3evs2a zLSIVepGa9qFVcSISd5HzoJt9xAyQ4od7NM6Qzwm446OyhzWsIN/a6+nDNB4AxzSg00QXKx4afEa FrdLzmREEfv24f88j2UZYqHAls0j26jyED5FZ068xs4gWZBG2U7EVTUNNJuUrrmqBNZkGxTIrFrD Cgr1EpRULpN+HrEelHHh7uR0twAjvwcyXkG9DbDJXnw8HzKGR80ik4+13HDxx4mDxOY4NOvWSSiM kEFS2Z2AKtxXSMBQZHazAUvbka27c1m93/QsjnDF+P6Aef9NEvUDL9mU9Jbf/+5V+anT2KdPGP4p rQ9gA/Nup61qxDkwc+RupiXD5NSbAgMBAAGjggGzMIIBrzAfBgNVHSMEGDAWgBSCr2yM+MX+lmF8 6B89K3FIXsSLwDAdBgNVHQ4EFgQUjwe7n1zvxFkTeCUYWrsaJpOGP14wDgYDVR0PAQH/BAQDAgWg MAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUFBwMCMEYGA1UdIAQ/MD0w OwYMKwYBBAGyMQECAQMFMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQv Q1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNs aWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmwwgYsGCCsGAQUFBwEBBH8wfTBV BggrBgEFBQcwAoZJaHR0cDovL2NydC5jb21vZG9jYS5jb20vQ09NT0RPUlNBQ2xpZW50QXV0aGVu dGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29t b2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQC85hVlqTVwt218IJR/WjMiMnDtZ7hY860XKjzO uB3sUUQwHxHj+ZYuMbAfVLZGGqh1EekbwDMVgkK9cezIHM+ZzxrNGX2SJyl1YW+3FLn52P0uIlmA VPFjUowf5qBhOHl2NJo+WXYZhQY7rT/xSygE81o3oLE/A4zO6WtO3PeZpFpZNrBvizAsjTDfPeXW iQzXz6NLrgwert0Wml95ov2rG5oCzHYPijabubSNm2NdUjPRtcVylcqAThXOvp6X4UvW8/L0uhkp 9WsKP2JEJ3Zukv7Ib+vMBsdE4tf4rmv89pQC+lLpD08ze/QDCIeFBCRIihcC2PycDQrnNIp1RAIh MYIDyjCCA8YCAQEwga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0 ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQD EzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEA q2wFIs+rCK6H6/2jbblXhDANBglghkgBZQMEAgEFAKCCAe0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkxMTI4MTM0ODI3WjAvBgkqhkiG9w0BCQQxIgQgZtk+4otn BRsOhvnPl4CGM22TsoWKAIZ6+QKru4B8i+owgb4GCSsGAQQBgjcQBDGBsDCBrTCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCrbAUiz6sIrofr/aNtuVeEMIHABgsqhkiG 9w0BCRACCzGBsKCBrTCBlzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3Rl cjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMT NENPTU9ETyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCr bAUiz6sIrofr/aNtuVeEMA0GCSqGSIb3DQEBAQUABIIBAHs9RWySFRuBuz59l0egPswBp60NP6Jt W4EB3ocfJqxmHed+LBoR4eXQIdTFm9izgT41XfbsMUB0wFqznXdnIJwT+1HGsolMYLVod9WU6bm/ YvztC1i3LSVJ1H4uv0w8Thm6VT9Px56+2sGg0LJXPpNHxZl4mMMzu6LiX9b5gE+3Jhf0hRitLywb da4R1X9ZorA6LZRnDcaDAcgH9UwY5kUbK0INiqwlS8ONJwlXm8epv8uqfdvIYt741/8G9ruSW8aa UtMGX/noPQ1TFnooJkqRz/hwpGhopfxr8r8/Dl8M2ZISykjYvcFI0TiPCcRDu4mlSzcLpHZwa1v1 qdz9bvMAAAAAAAA= --Apple-Mail=_373C4CF8-B65C-41D2-AF20-6B293E9FE8BA-- From owner-freebsd-hackers@freebsd.org Thu Nov 28 14:27:24 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1E06D1AB575 for ; Thu, 28 Nov 2019 14:27:24 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from smtpq2.tb.mail.iss.as9143.net (smtpq2.tb.mail.iss.as9143.net [212.54.42.165]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47P0Mv2rKDz4dKt for ; Thu, 28 Nov 2019 14:27:23 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from [212.54.42.136] (helo=smtp12.tb.mail.iss.as9143.net) by smtpq2.tb.mail.iss.as9143.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1iaKlN-0002TU-C7; Thu, 28 Nov 2019 15:27:21 +0100 Received: from 94-209-85-88.cable.dynamic.v4.ziggo.nl ([94.209.85.88] helo=wan0.bsd4all.org) by smtp12.tb.mail.iss.as9143.net with esmtp (Exim 4.90_1) (envelope-from ) id 1iaKlN-0003T9-3q; Thu, 28 Nov 2019 15:27:21 +0100 Received: from newnas.bsd4all.local (localhost [127.0.0.1]) by wan0.bsd4all.org (Postfix) with ESMTP id 797A8319; Thu, 28 Nov 2019 15:27:20 +0100 (CET) X-Virus-Scanned: amavisd-new at bsd4all.org Received: from wan0.bsd4all.org ([127.0.0.1]) by newnas.bsd4all.local (newnas.bsd4all.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ppcyapnq3jr5; Thu, 28 Nov 2019 15:27:18 +0100 (CET) Received: from [192.168.1.65] (unknown [192.168.1.65]) by wan0.bsd4all.org (Postfix) with ESMTPSA id 7D3832AD; Thu, 28 Nov 2019 15:27:18 +0100 (CET) From: Peter Blok Content-Type: multipart/signed; boundary="Apple-Mail=_C171FF62-931F-469B-9146-B31C901F3B65"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: dynamic loadable library multiple degined symbols Date: Thu, 28 Nov 2019 15:27:17 +0100 References: <20191128131211.GQ10580@kib.kiev.ua> <58EC7517-CFE7-442F-9A9B-00849E32BCA4@bsd4all.org> To: Konstantin Belousov , freebsd-hackers@freebsd.org In-Reply-To: <58EC7517-CFE7-442F-9A9B-00849E32BCA4@bsd4all.org> Message-Id: X-Mailer: Apple Mail (2.3445.104.11) X-SourceIP: 94.209.85.88 X-Ziggo-spambar: / X-Ziggo-spamscore: 0.0 X-Ziggo-spamreport: CMAE Analysis: v=2.3 cv=Rvq70xuK c=1 sm=1 tr=0 a=LYXyOGYQqFYBMgK+Y6iqTg==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=MeAgGD-zjQ4A:10 a=6Q3WNqvRAAAA:8 a=pGLkceISAAAA:8 a=cepaolNN1VHjxSdKrm0A:9 a=QEXdDO2ut3YA:10 a=g2jUX3k5JNHoXd8hjKIA:9 a=ZVk8-NSrHBgA:10 a=I8PBwKCn76L9oNdl0isp:22 X-Ziggo-Spam-Status: No X-Spam-Status: No X-Spam-Flag: No X-Rspamd-Queue-Id: 47P0Mv2rKDz4dKt X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pblok@bsd4all.org designates 212.54.42.165 as permitted sender) smtp.mailfrom=pblok@bsd4all.org X-Spamd-Result: default: False [-5.32 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; RECEIVED_SPAMHAUS_PBL(0.00)[88.85.209.94.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; R_SPF_ALLOW(-0.20)[+a:smtp.ziggo.nl/16]; MV_CASE(0.50)[]; HAS_ATTACHMENT(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[bsd4all.org]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_IN_DNSWL_LOW(-0.10)[165.42.54.212.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:33915, ipnet:212.54.32.0/20, country:NL]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-1.32)[ipnet: 212.54.32.0/20(-4.09), asn: 33915(-2.51), country: NL(0.03)]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 14:27:24 -0000 --Apple-Mail=_C171FF62-931F-469B-9146-B31C901F3B65 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Your pointers helped me find a solution. The samba build environment generates a runner script to build the = module. I have added this symbol in the script amongst other = =E2=80=9Chidden=E2=80=9D ones and it now works. Now I have to find out where and how the script is generated and have a = patch ready for upstream. > On 28 Nov 2019, at 14:48, Peter Blok wrote: >=20 > I=E2=80=99m trying to change this because named dies with an assert. = named checks the arguments of dns_name_equal which is completely = different from the one intended out of the shared module. >=20 >=20 >> On 28 Nov 2019, at 14:12, Konstantin Belousov = wrote: >>=20 >> On Thu, Nov 28, 2019 at 01:50:15PM +0100, Peter Blok wrote: >>> Hi, >>>=20 >>> named (bind9.14) has a function called dns_name_equal. = (0000000000443ac0 T dns_name_equal) >>>=20 >>> named dynamically loads dlz_bind9_14.so is build from dlz_bind9.c = and calls this function, but dns_name_equal was undefined so it got = resolved to the version of named. >>>=20 >>> The function is defined in dns_utils.c, so I changed the building to = include that file. >>>=20 >>> Now dlz_bind9_14.so is using dlz_bind9.c and dns_utils.c also has = the right dns_name_equal (000000000000bee0 T dns_name_equal) defined >>>=20 >>> Unfortunately the code inside dlz_bind9_14.so still calls the = function out of named. >>>=20 >>> Is this something that should have been resolved at compile/link = time of dlz_bind9_14.so? If so, how? >> No, default ELF name resolution rules would give the behaviour you = described, >> assuming the main binary was linked with -Wl,-E (and it must be to = export >> symbols to loadable modules). The shared libraries and loadable = modules >> are interposable by default, unless linked with -B symbolic, and the = symbol >> resolution order starts from the main binary object. >>=20 >> Why do you try to change this ? >>=20 >>>=20 >>> Note that the samba build uses waf and wscript files. >>>=20 >>> Peter >>>=20 >>>=20 >>=20 >>=20 >=20 --Apple-Mail=_C171FF62-931F-469B-9146-B31C901F3B65 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCBSAw ggUcMIIEBKADAgECAhEAq2wFIs+rCK6H6/2jbblXhDANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgwNDE0MDAwMDAwWhcNMjEwNDEzMjM1 OTU5WjBEMQswCQYDVQQGEwJOTDETMBEGA1UEAxMKUGV0ZXIgQmxvazEgMB4GCSqGSIb3DQEJARYR cGJsb2tAYnNkNGFsbC5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPT/3evs2a zLSIVepGa9qFVcSISd5HzoJt9xAyQ4od7NM6Qzwm446OyhzWsIN/a6+nDNB4AxzSg00QXKx4afEa FrdLzmREEfv24f88j2UZYqHAls0j26jyED5FZ068xs4gWZBG2U7EVTUNNJuUrrmqBNZkGxTIrFrD Cgr1EpRULpN+HrEelHHh7uR0twAjvwcyXkG9DbDJXnw8HzKGR80ik4+13HDxx4mDxOY4NOvWSSiM kEFS2Z2AKtxXSMBQZHazAUvbka27c1m93/QsjnDF+P6Aef9NEvUDL9mU9Jbf/+5V+anT2KdPGP4p rQ9gA/Nup61qxDkwc+RupiXD5NSbAgMBAAGjggGzMIIBrzAfBgNVHSMEGDAWgBSCr2yM+MX+lmF8 6B89K3FIXsSLwDAdBgNVHQ4EFgQUjwe7n1zvxFkTeCUYWrsaJpOGP14wDgYDVR0PAQH/BAQDAgWg MAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUFBwMCMEYGA1UdIAQ/MD0w OwYMKwYBBAGyMQECAQMFMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQv Q1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNs aWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmwwgYsGCCsGAQUFBwEBBH8wfTBV BggrBgEFBQcwAoZJaHR0cDovL2NydC5jb21vZG9jYS5jb20vQ09NT0RPUlNBQ2xpZW50QXV0aGVu dGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29t b2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQC85hVlqTVwt218IJR/WjMiMnDtZ7hY860XKjzO uB3sUUQwHxHj+ZYuMbAfVLZGGqh1EekbwDMVgkK9cezIHM+ZzxrNGX2SJyl1YW+3FLn52P0uIlmA VPFjUowf5qBhOHl2NJo+WXYZhQY7rT/xSygE81o3oLE/A4zO6WtO3PeZpFpZNrBvizAsjTDfPeXW iQzXz6NLrgwert0Wml95ov2rG5oCzHYPijabubSNm2NdUjPRtcVylcqAThXOvp6X4UvW8/L0uhkp 9WsKP2JEJ3Zukv7Ib+vMBsdE4tf4rmv89pQC+lLpD08ze/QDCIeFBCRIihcC2PycDQrnNIp1RAIh MYIDyjCCA8YCAQEwga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0 ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQD EzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEA q2wFIs+rCK6H6/2jbblXhDANBglghkgBZQMEAgEFAKCCAe0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkxMTI4MTQyNzE4WjAvBgkqhkiG9w0BCQQxIgQgBw5Q8d9r CVwl4qLJN7Esem/xZ5c+N/cIY9kbNeV7/w8wgb4GCSsGAQQBgjcQBDGBsDCBrTCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCrbAUiz6sIrofr/aNtuVeEMIHABgsqhkiG 9w0BCRACCzGBsKCBrTCBlzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3Rl cjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMT NENPTU9ETyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCr bAUiz6sIrofr/aNtuVeEMA0GCSqGSIb3DQEBAQUABIIBAAY945hgvIrb6d+lhct8wspXye5PBgQW IjzDB9JpX2ra9GB4/I8faYnRKt9XuRpcXwQcR7wPu+57GgyDsA0/5/R5uy9IauSLdz4C+3iQZ35Q qB9YP8XjP32eu+aUyQiEzZ9DO9RHm1SgXON8BNbCQkZ6qPD2V4dsspKDw5Zo/FOBTntxHCvLxKJF P0Thx0qWiyJMGAUThPHMaD+sMTOM6XG+1uhHNeZhUSf3SvNxjYBkdRKM4sbSI7pNQh6N6iyma0bz +WPPvP/W97QlQXBqfQ4so2czpd6SCKILlaEqp12E7+yegKWhdEojZ943HXJWBCKvNHfPeTEkI46H xBrXtoEAAAAAAAA= --Apple-Mail=_C171FF62-931F-469B-9146-B31C901F3B65-- From owner-freebsd-hackers@freebsd.org Thu Nov 28 18:20:36 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5F3831B1F0B; Thu, 28 Nov 2019 18:20:36 +0000 (UTC) (envelope-from f0andrey@gmail.com) Received: from mail-wm1-x341.google.com (mail-wm1-x341.google.com [IPv6:2a00:1450:4864:20::341]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47P5Xz0ByPz3P5s; Thu, 28 Nov 2019 18:20:34 +0000 (UTC) (envelope-from f0andrey@gmail.com) Received: by mail-wm1-x341.google.com with SMTP id a131so7525512wme.5; Thu, 28 Nov 2019 10:20:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=+vwxV5TZzszkuN1ruxhUQAKmp7LOFGivhbvieE8ymes=; b=C2PS1Q+xkEwrwOECzp9KIpT2a7/rhMbY4Q+2xCUyUqOfq+WcRPxJF2FTES6IlUUYMq fkfgq4gl2xvbPchXYVUqfjcKYEeegys3iHzGVrvMFcfUbnwbHDm/k2Vx+sdTDdRfzTSs I3rhEToqAp3WqkL+9y5+T4yAOnAe2xL9lkIYRCPXBI4nQayGicpY3Mky3cNbDTQDIuBT PgQn5OlySGB55uqrYaJ++qhrx2v3ZX3Y3ov+nzPcwZc9npCHmL+MhrrBfvckNMIVP72O WqqwuSxUh0ZCjC9ZgdYX7e4NPiPWpdZPplwIYu9w/KwcyK9hpch5Dqvnb69l4igyaN1X KuLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=+vwxV5TZzszkuN1ruxhUQAKmp7LOFGivhbvieE8ymes=; b=gMImXnFM2j42QWHLLwpInoSIWQEXi5vUpkHFuTUJT7ScIOFvKu2rP7d9wu+O4eTjaJ F/7+RZenLR59lUJw3keN9kzZwddfxGzbc3h8yyuMTn8LbHfqbZyokrdI6rGRkQHs5Qqs eLlEuNWJAmEpyRW4ykbVk2bV5wN2WlhBCr9raJjzR892NpE1IrxE37tWMWFuf29q33P3 TkYD0+vS2p+eOCBWu5mUuNS6Nmn5cAgWENCjPow08lO7wS6IO2YJNVuTnI+IKbEJRTV4 lOn0NGJ0n2oZAZwKRudxp93kMexVkvWHBtFgywqXGpTYqHZyeghOpXS/WlH3rjg3HNqb UKyg== X-Gm-Message-State: APjAAAXLwqo2RqBiKdufgLdkaWSr4n7fQSgPOxIKCpBH3HNN1T20TjIl BMlgoCDrQQC7x7YwxjeVhGLljY54HMIpbBpe9Mc5xI5J3O4= X-Google-Smtp-Source: APXvYqxrglBfhTNRoq6i1ZSnFv75dGLIwFvvjDLtzt8vXBuge9UPdq9sNsS9D0VtSmucsPHHQrpUDv2FiAa/8E7jL44= X-Received: by 2002:a1c:f415:: with SMTP id z21mr10919292wma.140.1574965233481; Thu, 28 Nov 2019 10:20:33 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Andrey Fesenko Date: Thu, 28 Nov 2019 21:20:22 +0300 Message-ID: Subject: Re: EFI loader failure, after 20191114-r354699 Z87MX-D3H To: Eugene Grosbein Cc: freebsd-current , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 47P5Xz0ByPz3P5s X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=C2PS1Q+x; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of f0andrey@gmail.com designates 2a00:1450:4864:20::341 as permitted sender) smtp.mailfrom=f0andrey@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (2.86), ipnet: 2a00:1450::/32(-2.69), asn: 15169(-1.95), country: US(-0.05)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[1.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 18:20:36 -0000 On Thu, Nov 28, 2019 at 1:24 PM Eugene Grosbein wrote: > > 28.11.2019 16:43, Andrey Fesenko wrote: > > > Hello, > > > > Around starting 20191114 r354699 (memstick tested), my desktop not > > boot normally. Boot only loader menu (black and white mode) after > > start i'm see load modules, after this monitor gray, and 15-20s > > disabled, system block not disable but run silently. system not boot. > > So loader works. Try breaking to loader prompt and do: > > set hw.vga.acpi_ignore_no_vga=1 > boot > Hm.... new test, new problems. Old and new efi, if breaking to loader prompt work only alphabet keys, nums not work :) (on notebook loader work fine) If make echo 'hw.vga.acpi_ignore_no_vga=1' > /boot/loader.conf on flash, and reboot, not fix boot. From owner-freebsd-hackers@freebsd.org Thu Nov 28 19:13:41 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CA0F61B44A7 for ; Thu, 28 Nov 2019 19:13:41 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47P6kD6Qz8z3y4Z for ; Thu, 28 Nov 2019 19:13:40 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id BD2F2A917B; Thu, 28 Nov 2019 20:13:38 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Mn9WGJS_ncg1; Thu, 28 Nov 2019 20:13:37 +0100 (CET) Received: from [192.168.10.9] (vaio [192.168.10.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id DC885A9176; Thu, 28 Nov 2019 20:13:37 +0100 (CET) Subject: Re: Process in T state does not want to die..... To: Eugene Grosbein , FreeBSD Hackers References: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> <0d698f4e-decf-8d4b-7a05-d5985558fb07@grosbein.net> From: Willem Jan Withagen Message-ID: Date: Thu, 28 Nov 2019 20:13:38 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1 MIME-Version: 1.0 In-Reply-To: <0d698f4e-decf-8d4b-7a05-d5985558fb07@grosbein.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: nl X-Rspamd-Queue-Id: 47P6kD6Qz8z3y4Z X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 176.74.240.9 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.59 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[digiware.nl]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[9.240.74.176.list.dnswl.org : 127.0.9.2]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-3.09)[ip: (-9.79), ipnet: 176.74.224.0/19(-4.89), asn: 28878(-0.77), country: NL(0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:176.74.224.0/19, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 19:13:41 -0000 On 28-11-2019 12:04, Eugene Grosbein wrote: > 28.11.2019 17:37, Willem Jan Withagen wrote: > >>> Process in state T is STOPPED and prohibited for execution. >>> It cannot even process signal like SIGTERM or SIGKILL because they are queied waiting for continuation. >>> You need to resume it with kill -CONT first. >> Tried that several times, but does not really have any effect. >> I could check and see if the signals (TERM, KILL) were waiting somewhere? >> With procstat?? >> >> But the original question was more for a way on preventing this state of affairs. >> Because uptill now the only resolution was to reboot the server, which is not a nice >> thing for a storage sollution. >> >> Hence the: how to debug? question. > Sometimes background process is (mistakenly) started without closing/redirecting output > and if it tries to print something, it might get stopped depending on used shell. Oke, that could very wel be the case.... When starting it prints to console, but I also know it forks. And I have seen some dev discusion on closing so they could be closing before the fork. > Use ktrace/kdump to verify what's going on with the process if you can reproduce the problem. > You did not specify which version you use. Sometimes flags shown by ps are wrong due to kernel bugs > and corrupted process state flag and process is not really Stopped but Exiting for example. Running 12.1-RELEASE: FreeBSD cephtest 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERIC amd64 > ktrace/kdump should show if process got SIGSTOP/SIGTSTP really. And if SIGCONT was delivered. Thanx for the suggestion, I'll try and read up. It is a bit of work, but the problem is "easily" reproduced. --WjW > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@freebsd.org Thu Nov 28 19:24:46 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0A6321B492E for ; Thu, 28 Nov 2019 19:24:46 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47P6z133bgz3yZG for ; Thu, 28 Nov 2019 19:24:45 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id BEB01A97BE; Thu, 28 Nov 2019 20:24:43 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HRAfXtXtKt3c; Thu, 28 Nov 2019 20:24:43 +0100 (CET) Received: from [192.168.10.9] (vaio [192.168.10.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id EEA98A97BD; Thu, 28 Nov 2019 20:24:42 +0100 (CET) Subject: Re: Process in T state does not want to die..... To: Konstantin Belousov Cc: FreeBSD Hackers , Eugene Grosbein References: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> <20191128115122.GN10580@kib.kiev.ua> From: Willem Jan Withagen Message-ID: <296874db-40f0-c7c9-a573-410e4c86049a@digiware.nl> Date: Thu, 28 Nov 2019 20:24:43 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1 MIME-Version: 1.0 In-Reply-To: <20191128115122.GN10580@kib.kiev.ua> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: nl X-Rspamd-Queue-Id: 47P6z133bgz3yZG X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 176.74.240.9 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.59 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[digiware.nl]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[9.240.74.176.list.dnswl.org : 127.0.9.2]; IP_SCORE(-3.09)[ip: (-9.79), ipnet: 176.74.224.0/19(-4.89), asn: 28878(-0.81), country: NL(0.02)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:176.74.224.0/19, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 19:24:46 -0000 On 28-11-2019 12:51, Konstantin Belousov wrote: > On Thu, Nov 28, 2019 at 11:37:50AM +0100, Willem Jan Withagen wrote: >> On 27-11-2019 16:36, Eugene Grosbein wrote: >>> 27.11.2019 22:11, Willem Jan Withagen wrote: >>> >>> [skip] >>> >>> Process in state T is STOPPED and prohibited for execution. >>> It cannot even process signal like SIGTERM or SIGKILL because they are queied waiting for continuation. >>> You need to resume it with kill -CONT first. >> Tried that several times, but does not really have any effect. >> I could check and see if the signals (TERM, KILL) were waiting somewhere? >> With procstat?? >> >> But the original question was more for a way on preventing this state of >> affairs. >> Because uptill now the only resolution was to reboot the server, which >> is not a nice >> thing for a storage sollution. >> >> Hence the: how to debug? question. > Start with the output from > $ procstat -kk -L > Also useful is the output from > $ ps -o pid,lwp,flags,flags2,state,tracer,command -p > After that is shown, it could be determined where to look next. > What is your system version ? > This is 12.1 RELEASE... But I get this: # procstat -L 3448 procstat: sysctl method is not supported And # procstat -kk -L Gives me the help info --WjW From owner-freebsd-hackers@freebsd.org Thu Nov 28 19:50:23 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 091E61B53CC for ; Thu, 28 Nov 2019 19:50:23 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47P7XY58VTz40tp for ; Thu, 28 Nov 2019 19:50:21 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id xASJoDHP004779 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 28 Nov 2019 21:50:16 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua xASJoDHP004779 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id xASJoDlY004776; Thu, 28 Nov 2019 21:50:13 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 28 Nov 2019 21:50:13 +0200 From: Konstantin Belousov To: Willem Jan Withagen Cc: FreeBSD Hackers , Eugene Grosbein Subject: Re: Process in T state does not want to die..... Message-ID: <20191128195013.GU10580@kib.kiev.ua> References: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> <20191128115122.GN10580@kib.kiev.ua> <296874db-40f0-c7c9-a573-410e4c86049a@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <296874db-40f0-c7c9-a573-410e4c86049a@digiware.nl> User-Agent: Mutt/1.12.2 (2019-09-21) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 47P7XY58VTz40tp X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; IP_SCORE(0.00)[ip: (-2.75), ipnet: 2001:470::/32(-4.64), asn: 6939(-3.52), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 19:50:23 -0000 On Thu, Nov 28, 2019 at 08:24:43PM +0100, Willem Jan Withagen wrote: > On 28-11-2019 12:51, Konstantin Belousov wrote: > > On Thu, Nov 28, 2019 at 11:37:50AM +0100, Willem Jan Withagen wrote: > >> On 27-11-2019 16:36, Eugene Grosbein wrote: > >>> 27.11.2019 22:11, Willem Jan Withagen wrote: > >>> > >>> [skip] > >>> > >>> Process in state T is STOPPED and prohibited for execution. > >>> It cannot even process signal like SIGTERM or SIGKILL because they are queied waiting for continuation. > >>> You need to resume it with kill -CONT first. > >> Tried that several times, but does not really have any effect. > >> I could check and see if the signals (TERM, KILL) were waiting somewhere? > >> With procstat?? > >> > >> But the original question was more for a way on preventing this state of > >> affairs. > >> Because uptill now the only resolution was to reboot the server, which > >> is not a nice > >> thing for a storage sollution. > >> > >> Hence the: how to debug? question. > > Start with the output from > > $ procstat -kk -L > > Also useful is the output from > > $ ps -o pid,lwp,flags,flags2,state,tracer,command -p > > After that is shown, it could be determined where to look next. > > What is your system version ? > > > This is 12.1 RELEASE... > But I get this: > # procstat -L 3448 > procstat: sysctl method is not supported > > And > > # procstat -kk -L > > Gives me the help info Remove -L. From owner-freebsd-hackers@freebsd.org Thu Nov 28 20:17:13 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4C3181B5FFE; Thu, 28 Nov 2019 20:17:13 +0000 (UTC) (envelope-from f0andrey@gmail.com) Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47P87W6bqPz42G8; Thu, 28 Nov 2019 20:17:10 +0000 (UTC) (envelope-from f0andrey@gmail.com) Received: by mail-wm1-x342.google.com with SMTP id p17so6938732wma.1; Thu, 28 Nov 2019 12:17:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=18W98a8dfQ6ztFrfx5TxmFHXzLjV4AeiR2oUuc+n9Tw=; b=qDxFiWsd9W9qVEp/L6wswZyBEkPWbr6qek+xZEV2rfSAEdjZF1yUyEpWnjg19GyEEN nRMJd95LifZlWVQWvLk/JQYeGubThIQ402diD82RrnCchNok6jdcFT1F924cTI/gyYid XQOd497nFmTLiu0L2bVnyts31eJ58WFhVdngUDDJdRMAaWn+Q0je1/SQvdI86k+oSqny eKvhxb5j/mWZgrAkJnLm8j4Zdi83OhzPXAjK0TH+jmaUaIwEzm+TEX8w1bHeeKTNDSVb HOyTdsQIVMhgaDtlCm8JwYkwSrczcPDQarqnFBNS32VW2S5IG0IUlHDHbnnshQvIEg+W /a/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=18W98a8dfQ6ztFrfx5TxmFHXzLjV4AeiR2oUuc+n9Tw=; b=aP9RwcuyzK0+nXpdKmQbSMPCve34m2lCygPom96eGZ/QZq6kFld2a3Q6qJzCLnmyB1 g74hNGXkv1orBRV42Ea13nN4lplZbNe4Za4+2tvexC53+jY5+DMZtKA+0ZVLPKVMRJPl ZVe7+sGz6O/+DjR0UyXR7OXd9AznarQTD0cFacEsZ6arMM2+ViRDq58NeaMMKxdf0KnM 1D4OFU4xk1W9NqYWIHGoUw3Si5ej/xsrAGjn5JxQ+4I3Q02GXHbNFpKQp0rGcwfQ9K9M +j1CzTccXplHZNz8oNco4FQa8B6+jRk/n444WUBiYVP52+M/3NIMUJs2o+F14hYMXLjz Mrhw== X-Gm-Message-State: APjAAAXToxflz77Z1dEM/cIfL9faWNl9Qs8v0y3YDTykKEd42j4wO1Pp ptnQgDka4X/nfY9evRg4yid8h9++HnnfRLptvS3YYlqR0p4= X-Google-Smtp-Source: APXvYqyPtTzq6zeR9xsP72HIRTTX18fvVeB6qIzzpK7qO+4/NVh9TKBASnWFZDc35PdZKk5GSn3kvVN+WR6ZBaaR8AA= X-Received: by 2002:a1c:2c43:: with SMTP id s64mr7746286wms.135.1574972227378; Thu, 28 Nov 2019 12:17:07 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Andrey Fesenko Date: Thu, 28 Nov 2019 23:16:56 +0300 Message-ID: Subject: Re: EFI loader failure, after 20191114-r354699 Z87MX-D3H To: Toomas Soome Cc: freebsd-current , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 47P87W6bqPz42G8 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=qDxFiWsd; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of f0andrey@gmail.com designates 2a00:1450:4864:20::342 as permitted sender) smtp.mailfrom=f0andrey@gmail.com X-Spamd-Result: default: False [-1.99 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(0.00)[+ip6:2a00:1450:4000::/36:c]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(0.00)[gmail.com,none]; FREEMAIL_TO(0.00)[me.com]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (2.64), ipnet: 2a00:1450::/32(-2.69), asn: 15169(-1.95), country: US(-0.05)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(0.00)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE_FREEMAIL(0.00)[]; BAD_REP_POLICIES(0.10)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; URIBL_PBL(0.01)[bsdnir.info]; RCVD_IN_DNSWL_NONE(0.00)[2.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 20:17:13 -0000 On Thu, Nov 28, 2019 at 3:03 PM Toomas Soome wrote: > > hi! > > I did try to reach you, but mail did bounce back=E2=80=A6 > > unicast ping me:) > > rgds, > toomas > > > On 28. Nov 2019, at 11:43, Andrey Fesenko wrote: > > > > Hello, > > > > Around starting 20191114 r354699 (memstick tested), my desktop not > > boot normally. Boot only loader menu (black and white mode) after > > start i'm see load modules, after this monitor gray, and 15-20s > > disabled, system block not disable but run silently. system not boot. > > > > If i'm change efi (EFI/BOOT/bootx64.efi), 20191031-r354207 or 12.1 > > release, system boot normally > > > This video boot second version (Name: "loader.efi") 545 KB https://bsdnir.info/files/efi_fail.mp4 From owner-freebsd-hackers@freebsd.org Thu Nov 28 20:19:05 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6CE1C1B6391; Thu, 28 Nov 2019 20:19:05 +0000 (UTC) (envelope-from f0andrey@gmail.com) Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47P89g6kMhz42mN; Thu, 28 Nov 2019 20:19:03 +0000 (UTC) (envelope-from f0andrey@gmail.com) Received: by mail-wr1-x441.google.com with SMTP id c14so7908678wrn.7; Thu, 28 Nov 2019 12:19:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=3WEPQvh+urOXzuhx7smg8ad5bbFkaiT4T9Azlh/witM=; b=qRd3BmloExBKkFN8vPPrwyD8zcaKXaBlU4csD0QnXoqKmD77qaINDEXXksHgTS1OVk /zFCRWG5jU185vScMjW1Ewt4BU17XnCidUIHbJRESmsgwCkMFSyortf2/m38HHJv6Qio lQNA3qjTKuagF38Onovs48Lf/JXgxsCGCSUihueb4xWdSBHzdIb2J7cVndBZ1Mk7rSkD 00UyFc2Dclt1wXtpODMr73BFb7+SmxC4Nn4q1mqD4yp67CIY9+CXJEpYv+B2CvIgeqPi 4zbQT4rrhBwZWaNT/OY9/qqRGUl/Mp501nWyeLdNFos84ksK/7WsKS00m09nDupj3fH8 OS3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=3WEPQvh+urOXzuhx7smg8ad5bbFkaiT4T9Azlh/witM=; b=rmkjT1irKTlG4wB+bUXFXU/wLkn+2jJLKBBnSLAK+tHHthlY/EmkVO3974e8g8wY5Y rYQDqoCQR52n218bJbg7tWceoV/NDAlb0vFEKmYsd3qXJVNdc7kwaqzHlCO3lRd9iC5F dJRXuxvWltSA0s8RgFqVVVBSJy9DPmy/iezSgglJ7LeNMnDxPnzEAusiQg0YXRxrEIDZ 71qXdOuxParq4pg4eTKvHHc7PRHny+7bGhPHWg0zEhWrlXgG2TRXF7sTU8VR85aR1sj6 1zv2WUqxTMyBTpvezDWVPk9uzzacbkTwU3HGh0fmgBIajeiMS/Psqa8wIRFpXkNbeGSA ySAQ== X-Gm-Message-State: APjAAAX6i8z/xfXnN1puRJAA0dYY4B4fHXXLUkRh6vmHa6V0V59v/TmM LIYHx8wJMewnjIFu/IsPDHEUFDe+ygM5o8mZp5qiT/UlAY8= X-Google-Smtp-Source: APXvYqxqZ89/5tXASBofsq2fptWl+iIY9KVr/qCJf8xUGXsyL22k15hL+Y3dEyXKuMHJ8t/6PNf5FGJlnkuBoXBjO9I= X-Received: by 2002:adf:ff84:: with SMTP id j4mr9355766wrr.27.1574972337491; Thu, 28 Nov 2019 12:18:57 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Andrey Fesenko Date: Thu, 28 Nov 2019 23:18:46 +0300 Message-ID: Subject: Re: EFI loader failure, after 20191114-r354699 Z87MX-D3H To: Eugene Grosbein Cc: freebsd-current , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 47P89g6kMhz42mN X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=qRd3Bmlo; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of f0andrey@gmail.com designates 2a00:1450:4864:20::441 as permitted sender) smtp.mailfrom=f0andrey@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (2.53), ipnet: 2a00:1450::/32(-2.69), asn: 15169(-1.95), country: US(-0.05)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[1.4.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 20:19:05 -0000 On Thu, Nov 28, 2019 at 9:20 PM Andrey Fesenko wrote: > > On Thu, Nov 28, 2019 at 1:24 PM Eugene Grosbein wrote: > > > > 28.11.2019 16:43, Andrey Fesenko wrote: > > > > > Hello, > > > > > > Around starting 20191114 r354699 (memstick tested), my desktop not > > > boot normally. Boot only loader menu (black and white mode) after > > > start i'm see load modules, after this monitor gray, and 15-20s > > > disabled, system block not disable but run silently. system not boot. > > > > So loader works. Try breaking to loader prompt and do: > > > > set hw.vga.acpi_ignore_no_vga=1 > > boot > > > > Hm.... new test, new problems. Old and new efi, if breaking to loader > prompt work only alphabet keys, nums not work :) (on notebook loader > work fine) > If make > echo 'hw.vga.acpi_ignore_no_vga=1' > /boot/loader.conf > > on flash, and reboot, not fix boot. Correction number and underline work, but set hw.vga.acpi_ignore_no_vga not fix boot From owner-freebsd-hackers@freebsd.org Thu Nov 28 20:20:30 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 96BEE1B673F; Thu, 28 Nov 2019 20:20:30 +0000 (UTC) (envelope-from f0andrey@gmail.com) Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47P8CK6Q51z436r; Thu, 28 Nov 2019 20:20:29 +0000 (UTC) (envelope-from f0andrey@gmail.com) Received: by mail-wr1-x441.google.com with SMTP id w15so1565578wru.4; Thu, 28 Nov 2019 12:20:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=D51YHK1w32jujqOU/ZtQn6G9ZBYhpHhxYNjlVP4L8yc=; b=V1ifRpj5Pr16dqjTd4IVIobwzcPvfIX7huwoE9p1k/F8q+HyF71UurGpow23E6BMzW cDMxcDATfckAGRSFAbwtAcSRqREhZ8kYdpuY5nz1rvoOj6nlKlTajR4JxeW2eYdgShwZ f/ajq5+QIWDOpBhgd7esIhaYsIROGkAsjdwrTi/dYYSvTE+OqtYrNqCoR9m7OKsW7gPh 5BElPpMumarvcAU4wucFSsUQ2NhBtx7GCGYw5dq2ATll52GWMzQH63P3B83vbm3MS7P/ 6GH2XbS877x9vaxlMZOhjogpy97C1CN+r3NEhto0aT5TiyYbLeT1N+c0k+4IdrjsYKPf GXYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=D51YHK1w32jujqOU/ZtQn6G9ZBYhpHhxYNjlVP4L8yc=; b=TL3ZRtDpu6TsYg1mjFteFOpde3gUgQinDzQ3Eh/FMBxfwDBwFa7cLOx01zLyw86ILA IikOnm6YJSSkkXiEKQ3AK0SP8Lplkszrec85wbw/NaKD5AKDxKfLzpykq7nKtKpL2GVG piGZpPXn4WdjLpJiQIGEjzJ9h3IOgpziEqaId8ws+47s1/lmEe7v/fuMTWehtY9wQoOf L9DxUuH8+Qb5djM9tmpmBHKFDPxShSBX/yawEoc+mgGBAHrTeKDqHoKHLKbp1WlqXOzV 3DAITr4IfRUmkMGegWTvgCD4Mve4pcx9ugnD4Vmi6WZoZ3ZHvMX42C0pfML4FviEjaul AaWg== X-Gm-Message-State: APjAAAWB6q/3/DEa1lbn8TOjSLgj3GWKZhh5mlqOotbsAkW6odmqqL8o tB7MNAkNhccglO3lsO86MM0xHXqpegT/fsdjv7ZT4Oht X-Google-Smtp-Source: APXvYqwxvjEuZn+VLxeMmW/K53OyPIqYbiTE0w1VKJ0fRzqIzHmfKnyfeJTRrQ4jwkpZvpjr+/6dovJxmvDmZqjXs2c= X-Received: by 2002:adf:de86:: with SMTP id w6mr29486131wrl.115.1574972428256; Thu, 28 Nov 2019 12:20:28 -0800 (PST) MIME-Version: 1.0 References: <49B58A7F-0DE2-4BEB-8A8C-D996962FDDE1@me.com> In-Reply-To: <49B58A7F-0DE2-4BEB-8A8C-D996962FDDE1@me.com> From: Andrey Fesenko Date: Thu, 28 Nov 2019 23:20:17 +0300 Message-ID: Subject: Re: EFI loader failure, after 20191114-r354699 Z87MX-D3H To: Toomas Soome Cc: freebsd-current , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 47P8CK6Q51z436r X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=V1ifRpj5; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of f0andrey@gmail.com designates 2a00:1450:4864:20::441 as permitted sender) smtp.mailfrom=f0andrey@gmail.com X-Spamd-Result: default: False [-1.99 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(0.00)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(0.00)[gmail.com,none]; FREEMAIL_TO(0.00)[me.com]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (2.53), ipnet: 2a00:1450::/32(-2.69), asn: 15169(-1.95), country: US(-0.05)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(0.00)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE_FREEMAIL(0.00)[]; BAD_REP_POLICIES(0.10)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; URIBL_PBL(0.01)[bsdnir.info]; RCVD_IN_DNSWL_NONE(0.00)[1.4.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 20:20:30 -0000 Fixed On Thu, Nov 28, 2019 at 11:18 PM Toomas Soome wrote: > > > > On 28. Nov 2019, at 22:16, Andrey Fesenko wrote: > > On Thu, Nov 28, 2019 at 3:03 PM Toomas Soome wrote: > > > hi! > > I did try to reach you, but mail did bounce back=E2=80=A6 > > unicast ping me:) > > rgds, > toomas > > On 28. Nov 2019, at 11:43, Andrey Fesenko wrote: > > Hello, > > Around starting 20191114 r354699 (memstick tested), my desktop not > boot normally. Boot only loader menu (black and white mode) after > start i'm see load modules, after this monitor gray, and 15-20s > disabled, system block not disable but run silently. system not boot. > > If i'm change efi (EFI/BOOT/bootx64.efi), 20191031-r354207 or 12.1 > release, system boot normally > > > > This video boot second version (Name: "loader.efi") 545 KB > https://bsdnir.info/files/efi_fail.mp4 > > > 403 Forbidden > > :=3D) > > rgds, > toomas From owner-freebsd-hackers@freebsd.org Thu Nov 28 20:53:03 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D6A601B7AF5 for ; Thu, 28 Nov 2019 20:53:03 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47P8wt3Ppdz453X for ; Thu, 28 Nov 2019 20:53:02 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 40262AD361; Thu, 28 Nov 2019 21:52:52 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qiGLNNNhYSwT; Thu, 28 Nov 2019 21:52:50 +0100 (CET) Received: from [192.168.10.9] (vaio [192.168.10.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 8C3CEAD360; Thu, 28 Nov 2019 21:52:50 +0100 (CET) Subject: Re: Process in T state does not want to die..... To: Konstantin Belousov Cc: FreeBSD Hackers , Eugene Grosbein References: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> <20191128115122.GN10580@kib.kiev.ua> <296874db-40f0-c7c9-a573-410e4c86049a@digiware.nl> <20191128195013.GU10580@kib.kiev.ua> From: Willem Jan Withagen Message-ID: <1ae7ad65-902c-8e5f-bcf1-1e98448c64bb@digiware.nl> Date: Thu, 28 Nov 2019 21:52:50 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1 MIME-Version: 1.0 In-Reply-To: <20191128195013.GU10580@kib.kiev.ua> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: nl X-Rspamd-Queue-Id: 47P8wt3Ppdz453X X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 2001:4cb8:90:ffff::3 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.29 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[digiware.nl]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; IP_SCORE(-2.99)[ip: (-9.52), ipnet: 2001:4cb8::/29(-4.63), asn: 28878(-0.83), country: NL(0.02)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:2001:4cb8::/29, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 20:53:03 -0000  # ps -o pid,lwp,flags,flags2,state,tracer,command -p 3532  PID    LWP        F       F2 STAT TRACER COMMAND 3532 103955 11080081 00000000 TsJ       0 ceph-osd -i 5 # procstat -kk 3532   PID    TID COMM                TDNAME              KSTACK  3532 103166 ceph-osd            log                 mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103167 ceph-osd            service             mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103168 ceph-osd            admin_socket        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103169 ceph-osd            msgr-worker-0       mi_switch+0xe2 thread_suspend_switch+0x140 thread_single+0x47b sigexit+0x53 postsig+0x304 ast+0x327 fast_syscall_common+0x198  3532 103170 ceph-osd            msgr-worker-1       mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103171 ceph-osd            msgr-worker-2       mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103172 ceph-osd            signal_handler      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103406 ceph-osd            OpHistorySvc        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103407 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103418 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103419 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103421 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103427 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103663 ceph-osd            fn_anonymous        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103675 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103677 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103678 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103679 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103680 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103681 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103682 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103683 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103684 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103685 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103955 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104621 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104826 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104827 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104828 ceph-osd            wb_throttle         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104829 ceph-osd            filestore_sync      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104830 ceph-osd            journal_write       mi_switch+0xe2 sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 dofilewrite+0x93 sys_writev+0x6e amd64_syscall+0x364 fast_syscall_common+0x101  3532 104831 ceph-osd            fn_jrn_objstore     mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104832 ceph-osd            tp_fstore_op        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104833 ceph-osd            tp_fstore_op        mi_switch+0xe2 sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 vn_open_cred+0xc8 zfs_setextattr+0x216 VOP_SETEXTATTR_APV+0x7c extattr_set_vp+0x11d sys_extattr_set_fd+0xee amd64_syscall+0x364 fast_syscall_common+0x101  3532 104834 ceph-osd            fn_odsk_fstore      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104835 ceph-osd            fn_appl_fstore      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104836 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104838 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104839 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104841 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104843 ceph-osd            fn_anonymous        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105398 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105399 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105400 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105401 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105402 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105403 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105404 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105405 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105406 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105407 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105408 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105409 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105410 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105411 ceph-osd            fn_anonymous        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105412 ceph-osd            finisher            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105413 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105414 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105415 ceph-osd            osd_srv_agent       mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105416 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105417 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105418 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105419 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105420 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105421 ceph-osd            tp_osd_cmd          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105422 ceph-osd            osd_srv_heartbt     mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f root@cephtest:~ # procstat -kk 3532 | less   PID    TID COMM                TDNAME              KSTACK  3532 103166 ceph-osd            log                 mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103167 ceph-osd            service             mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103168 ceph-osd            admin_socket        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103169 ceph-osd            msgr-worker-0       mi_switch+0xe2 thread_suspend_switch+0x140 thread_single+0x47b sigexit+0x53 postsig+0x304 ast+0x327 fast_syscall_common+0x198  3532 103170 ceph-osd            msgr-worker-1       mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103171 ceph-osd            msgr-worker-2       mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103172 ceph-osd            signal_handler      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103406 ceph-osd            OpHistorySvc        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103407 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103418 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103419 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103421 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103427 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103663 ceph-osd            fn_anonymous        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103675 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103677 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103678 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103679 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103680 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103681 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103682 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103683 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103684 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103685 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 103955 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104621 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104826 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104827 ceph-osd            -                   mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104828 ceph-osd            wb_throttle         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104829 ceph-osd            filestore_sync      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104830 ceph-osd            journal_write       mi_switch+0xe2 sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 dofilewrite+0x93 sys_writev+0x6e amd64_syscall+0x364 fast_syscall_common+0x101  3532 104831 ceph-osd            fn_jrn_objstore     mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104832 ceph-osd            tp_fstore_op        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104833 ceph-osd            tp_fstore_op        mi_switch+0xe2 sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 vn_open_cred+0xc8 zfs_setextattr+0x216 VOP_SETEXTATTR_APV+0x7c extattr_set_vp+0x11d sys_extattr_set_fd+0xee amd64_syscall+0x364 fast_syscall_common+0x101  3532 104834 ceph-osd            fn_odsk_fstore      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104835 ceph-osd            fn_appl_fstore      mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104836 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104838 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104839 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104841 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 104843 ceph-osd            fn_anonymous        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105398 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105399 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105400 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105401 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105402 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105403 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105404 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105405 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105406 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105407 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105408 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105409 ceph-osd            ms_dispatch         mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105410 ceph-osd            ms_local            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105411 ceph-osd            fn_anonymous        mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105412 ceph-osd            finisher            mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105413 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105414 ceph-osd            safe_timer          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105415 ceph-osd            osd_srv_agent       mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105416 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105417 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105418 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105419 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105420 ceph-osd            tp_osd_tp           mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105421 ceph-osd            tp_osd_cmd          mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f  3532 105422 ceph-osd            osd_srv_heartbt     mi_switch+0xe2 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f root@cephtest:~ # From owner-freebsd-hackers@freebsd.org Thu Nov 28 21:01:19 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B01491B7EE0; Thu, 28 Nov 2019 21:01:19 +0000 (UTC) (envelope-from f0andrey@gmail.com) Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47P96Q48FGz45Kq; Thu, 28 Nov 2019 21:01:18 +0000 (UTC) (envelope-from f0andrey@gmail.com) Received: by mail-wr1-x442.google.com with SMTP id b6so2987971wrq.0; Thu, 28 Nov 2019 13:01:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=fW6J4PxiMto8xq31NanXnNJZ+AFybLygTbISDpFw8R8=; b=dfWkVps3y7lg5OUJhnTeCOjOnayj4iwpGMAOGlZOaUVJrKEi8zflTlo0W0Uwu+q6IN vSWJs/pr32b718RlWzgUNB0jEHWS81UPW5VCtMycDskJvEHqwzZ9BqgRnvXrMdAFct08 MAg6imo/ta6KHhd8JyiG4iDKac/Gk5cZuWEeVLaMMlxcNHSXgeAQiWtkMp+SSW9Mz2hu vlWuZXFBWyp41vWHd2ZjbAAWmVGgC8K3Y0zJTuRVqkCjVJLsBrQd5KOBJjKGdnKCqQu3 ouXFLwctSlh7Wow+VxxfLmEMhFhFC7XmQM6zpVS4jPd+7T9X513zn4ojcIs4/R+e1OtA yn8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=fW6J4PxiMto8xq31NanXnNJZ+AFybLygTbISDpFw8R8=; b=BsQ7ZhfFV2oSMjgR+WUFnW6xBUyNy2KRZXdlFKw2q547hSRdHcGNuLx6DQ7a95RMV/ /aplVVI06X0/siJKRH3GPeGHyPd6gArwH0LBRWeFaImP0y0Qr0DGuonWwXoXN+Lz24AI zYnpQKPp3zIfA3LwQUSEdhedSh8RZH2jl0+uLMmAwF58bDGLNePWZ/W8N+w8tW7BhjAY JZJG1acP4Qi1KaGms1W3pVdc/rlP7E5eBbmRUHdoS4iJLycBTiDOmCV6eKrxCLOaf3id 8cKAfl0H4fm/PWK9cC3UkgyR51PG8YmQRreiF0LVc9R5j8y2EhMnNKSH6+Jii8oc7+UB igrw== X-Gm-Message-State: APjAAAUXk1/iHCWMzh2xetoK2ob3GhJv9HM8BzV24cMTAs/i/yQEAxuy mFgQGWxGnplSzLXKjT/pG5Hec6CWGjNWZaA5HC0= X-Google-Smtp-Source: APXvYqysshI1oTvQW0XGx2t2FikrDUn/wgdiIkTAidQ3R9i9qKLikmE8kJBRABzz4pYjYXPbQuY4XU9KWJKjQkG/caA= X-Received: by 2002:adf:ff84:: with SMTP id j4mr9501629wrr.27.1574974876789; Thu, 28 Nov 2019 13:01:16 -0800 (PST) MIME-Version: 1.0 References: <49B58A7F-0DE2-4BEB-8A8C-D996962FDDE1@me.com> <8C721EA7-09EA-4154-AC29-273B3A881FA4@me.com> In-Reply-To: <8C721EA7-09EA-4154-AC29-273B3A881FA4@me.com> From: Andrey Fesenko Date: Fri, 29 Nov 2019 00:01:05 +0300 Message-ID: Subject: Re: EFI loader failure, after 20191114-r354699 Z87MX-D3H To: Toomas Soome Cc: freebsd-current , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 47P96Q48FGz45Kq X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=dfWkVps3; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of f0andrey@gmail.com designates 2a00:1450:4864:20::442 as permitted sender) smtp.mailfrom=f0andrey@gmail.com X-Spamd-Result: default: False [-1.99 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(0.00)[+ip6:2a00:1450:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(0.00)[gmail.com,none]; FREEMAIL_TO(0.00)[me.com]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (2.49), ipnet: 2a00:1450::/32(-2.69), asn: 15169(-1.95), country: US(-0.05)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(0.00)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE_FREEMAIL(0.00)[]; BAD_REP_POLICIES(0.10)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; URIBL_PBL(0.01)[bsdnir.info]; RCVD_IN_DNSWL_NONE(0.00)[2.4.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 21:01:19 -0000 On Thu, Nov 28, 2019 at 11:31 PM Toomas Soome wrote: > > > > > On 28. Nov 2019, at 22:20, Andrey Fesenko wrote: > > > > Fixed > > > > Thanks. And yea, that one is nasty. I have some guess but nothing too sol= id=E2=80=A6 it may take time to get figured out. > > Have you tested BIOS boot? > > rgds, > toomas > > > On Thu, Nov 28, 2019 at 11:18 PM Toomas Soome wrote: > >> > >> > >> > >> On 28. Nov 2019, at 22:16, Andrey Fesenko wrote: > >> > >> On Thu, Nov 28, 2019 at 3:03 PM Toomas Soome wrote: > >> > >> > >> hi! > >> > >> I did try to reach you, but mail did bounce back=E2=80=A6 > >> > >> unicast ping me:) > >> > >> rgds, > >> toomas > >> > >> On 28. Nov 2019, at 11:43, Andrey Fesenko wrote: > >> > >> Hello, > >> > >> Around starting 20191114 r354699 (memstick tested), my desktop not > >> boot normally. Boot only loader menu (black and white mode) after > >> start i'm see load modules, after this monitor gray, and 15-20s > >> disabled, system block not disable but run silently. system not boot. > >> > >> If i'm change efi (EFI/BOOT/bootx64.efi), 20191031-r354207 or 12.1 > >> release, system boot normally > >> > >> > >> > >> This video boot second version (Name: "loader.efi") 545 KB > >> https://bsdnir.info/files/efi_fail.mp4 > >> > >> > >> 403 Forbidden > >> > >> :=3D) > >> > >> rgds, > >> toomas > BIOS boot fine, loader colored, installer start normal. From owner-freebsd-hackers@freebsd.org Thu Nov 28 21:46:50 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 527901B9983 for ; Thu, 28 Nov 2019 21:46:50 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47PB6w5vYNz483y for ; Thu, 28 Nov 2019 21:46:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id xASLkY4W033250 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 28 Nov 2019 23:46:37 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua xASLkY4W033250 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id xASLkXjS033249; Thu, 28 Nov 2019 23:46:33 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 28 Nov 2019 23:46:33 +0200 From: Konstantin Belousov To: Willem Jan Withagen Cc: FreeBSD Hackers , Eugene Grosbein Subject: Re: Process in T state does not want to die..... Message-ID: <20191128214633.GV10580@kib.kiev.ua> References: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> <20191128115122.GN10580@kib.kiev.ua> <296874db-40f0-c7c9-a573-410e4c86049a@digiware.nl> <20191128195013.GU10580@kib.kiev.ua> <1ae7ad65-902c-8e5f-bcf1-1e98448c64bb@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1ae7ad65-902c-8e5f-bcf1-1e98448c64bb@digiware.nl> User-Agent: Mutt/1.12.2 (2019-09-21) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 47PB6w5vYNz483y X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; IP_SCORE(0.00)[ip: (-2.74), ipnet: 2001:470::/32(-4.64), asn: 6939(-3.51), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Nov 2019 21:46:50 -0000 On Thu, Nov 28, 2019 at 09:52:50PM +0100, Willem Jan Withagen wrote: >  # ps -o pid,lwp,flags,flags2,state,tracer,command -p 3532 >  PID    LWP        F       F2 STAT TRACER COMMAND > 3532 103955 11080081 00000000 TsJ       0 ceph-osd -i 5 > > # procstat -kk 3532 >   PID    TID COMM                TDNAME              KSTACK >  3532 103166 ceph-osd            log                 mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103167 ceph-osd            service             mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103168 ceph-osd            admin_socket        mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103169 ceph-osd            msgr-worker-0       mi_switch+0xe2 > thread_suspend_switch+0x140 thread_single+0x47b sigexit+0x53 > postsig+0x304 ast+0x327 fast_syscall_common+0x198 >  3532 103170 ceph-osd            msgr-worker-1       mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103171 ceph-osd            msgr-worker-2       mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103172 ceph-osd            signal_handler      mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103406 ceph-osd            OpHistorySvc        mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103407 ceph-osd            -                   mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103418 ceph-osd            safe_timer          mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103419 ceph-osd            safe_timer          mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103421 ceph-osd            safe_timer          mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103427 ceph-osd            safe_timer          mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103663 ceph-osd            fn_anonymous        mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103675 ceph-osd            -                   mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103677 ceph-osd            -                   mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103678 ceph-osd            -                   mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103679 ceph-osd            -                   mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103680 ceph-osd            -                   mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103681 ceph-osd            -                   mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103682 ceph-osd            -                   mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103683 ceph-osd            -                   mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103684 ceph-osd            -                   mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103685 ceph-osd            -                   mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 103955 ceph-osd            -                   mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 104621 ceph-osd            -                   mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 104826 ceph-osd            -                   mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 104827 ceph-osd            -                   mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 104828 ceph-osd            wb_throttle         mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 104829 ceph-osd            filestore_sync      mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 104830 ceph-osd            journal_write       mi_switch+0xe2 > sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 dofilewrite+0x93 > sys_writev+0x6e amd64_syscall+0x364 fast_syscall_common+0x101 >  3532 104831 ceph-osd            fn_jrn_objstore     mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 104832 ceph-osd            tp_fstore_op        mi_switch+0xe2 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >  3532 104833 ceph-osd            tp_fstore_op        mi_switch+0xe2 > sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 vn_open_cred+0xc8 > zfs_setextattr+0x216 VOP_SETEXTATTR_APV+0x7c extattr_set_vp+0x11d > sys_extattr_set_fd+0xee amd64_syscall+0x364 fast_syscall_common+0x101 This is an example of the cause for your problem. The thread is executing some ZFS code, zfs_setextattr() VOP probably to do something with the ext attrs. There, it recurses into VFS to open a file, and vn_open_cred() waits for buffer space pressure because it is assumed the vn_open_cred() is called from top level, not from inside VFS/fs code. Until this thread finished its operation and safely returned back to kernel/user boundary, the process cannot exit. There are two problems. One is this call to bwillwrite(), and it is easy to get rid of it, see the patch at the end of the message. But I wonder why do you have so many dirty buffers and why it does not resolve itself. Note that ZFS does not use buffer cache, you must have some other very active fs, using buffer cache, that is somehow blocked on writes. diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c index ebcc0ad92e0..ae37dd1fba1 100644 --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c @@ -5490,7 +5490,7 @@ vop_getextattr { flags = FREAD; NDINIT_ATVP(&nd, LOOKUP, NOFOLLOW, UIO_SYSSPACE, attrname, xvp, td); - error = vn_open_cred(&nd, &flags, 0, 0, ap->a_cred, NULL); + error = vn_open_cred(&nd, &flags, VN_OPEN_INVFS, 0, ap->a_cred, NULL); vp = nd.ni_vp; NDFREE(&nd, NDF_ONLY_PNBUF); if (error != 0) { @@ -5627,7 +5627,8 @@ vop_setextattr { flags = FFLAGS(O_WRONLY | O_CREAT); NDINIT_ATVP(&nd, LOOKUP, NOFOLLOW, UIO_SYSSPACE, attrname, xvp, td); - error = vn_open_cred(&nd, &flags, 0600, 0, ap->a_cred, NULL); + error = vn_open_cred(&nd, &flags, 0600, VN_OPEN_INVFS, ap->a_cred, + NULL); vp = nd.ni_vp; NDFREE(&nd, NDF_ONLY_PNBUF); if (error != 0) { diff --git a/sys/kern/vfs_vnops.c b/sys/kern/vfs_vnops.c index a0c018deb32..c69010dd999 100644 --- a/sys/kern/vfs_vnops.c +++ b/sys/kern/vfs_vnops.c @@ -219,7 +219,8 @@ vn_open_cred(struct nameidata *ndp, int *flagp, int cmode, u_int vn_open_flags, ndp->ni_cnd.cn_flags |= AUDITVNODE1; if (vn_open_flags & VN_OPEN_NOCAPCHECK) ndp->ni_cnd.cn_flags |= NOCAPCHECK; - bwillwrite(); + if ((vn_open_flags & VN_OPEN_INVFS) == 0) + bwillwrite(); if ((error = namei(ndp)) != 0) return (error); if (ndp->ni_vp == NULL) { diff --git a/sys/sys/vnode.h b/sys/sys/vnode.h index 8472bc0fb7b..27dbcbc58b1 100644 --- a/sys/sys/vnode.h +++ b/sys/sys/vnode.h @@ -579,6 +579,7 @@ typedef void vop_getpages_iodone_t(void *, vm_page_t *, int, int); #define VN_OPEN_NOAUDIT 0x00000001 #define VN_OPEN_NOCAPCHECK 0x00000002 #define VN_OPEN_NAMECACHE 0x00000004 +#define VN_OPEN_INVFS 0x00000008 /* * Public vnode manipulation functions. From owner-freebsd-hackers@freebsd.org Fri Nov 29 09:08:45 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0A89F1A999C for ; Fri, 29 Nov 2019 09:08:45 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47PTFl2kpwz3Bnp for ; Fri, 29 Nov 2019 09:08:43 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 6F69C17A73; Fri, 29 Nov 2019 10:08:40 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wn73f945ZW9b; Fri, 29 Nov 2019 10:08:39 +0100 (CET) Received: from [192.168.10.9] (vaio [192.168.10.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 9324617A72; Fri, 29 Nov 2019 10:08:39 +0100 (CET) Subject: Re: Process in T state does not want to die..... To: Konstantin Belousov Cc: FreeBSD Hackers , Eugene Grosbein References: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> <20191128115122.GN10580@kib.kiev.ua> <296874db-40f0-c7c9-a573-410e4c86049a@digiware.nl> <20191128195013.GU10580@kib.kiev.ua> <1ae7ad65-902c-8e5f-bcf1-1e98448c64bb@digiware.nl> <20191128214633.GV10580@kib.kiev.ua> From: Willem Jan Withagen Message-ID: <24afad3c-331c-099a-e5f2-32e1de74c985@digiware.nl> Date: Fri, 29 Nov 2019 10:08:39 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1 MIME-Version: 1.0 In-Reply-To: <20191128214633.GV10580@kib.kiev.ua> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-Rspamd-Queue-Id: 47PTFl2kpwz3Bnp X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 176.74.240.9 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.61 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[digiware.nl]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[9.240.74.176.list.dnswl.org : 127.0.9.2]; IP_SCORE(-3.11)[ip: (-9.79), ipnet: 176.74.224.0/19(-4.90), asn: 28878(-0.87), country: NL(0.02)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:176.74.224.0/19, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Nov 2019 09:08:45 -0000 On 28-11-2019 22:46, Konstantin Belousov wrote: > On Thu, Nov 28, 2019 at 09:52:50PM +0100, Willem Jan Withagen wrote: >>  # ps -o pid,lwp,flags,flags2,state,tracer,command -p 3532 >>  PID    LWP        F       F2 STAT TRACER COMMAND >> 3532 103955 11080081 00000000 TsJ       0 ceph-osd -i 5 >> >> # procstat -kk 3532 >>   PID    TID COMM                TDNAME              KSTACK ....... >>  3532 104829 ceph-osd            filestore_sync      mi_switch+0xe2 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >>  3532 104830 ceph-osd            journal_write       mi_switch+0xe2 >> sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 dofilewrite+0x93 >> sys_writev+0x6e amd64_syscall+0x364 fast_syscall_common+0x101 >>  3532 104831 ceph-osd            fn_jrn_objstore     mi_switch+0xe2 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >>  3532 104832 ceph-osd            tp_fstore_op        mi_switch+0xe2 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >>  3532 104833 ceph-osd            tp_fstore_op        mi_switch+0xe2 >> sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 vn_open_cred+0xc8 >> zfs_setextattr+0x216 VOP_SETEXTATTR_APV+0x7c extattr_set_vp+0x11d >> sys_extattr_set_fd+0xee amd64_syscall+0x364 fast_syscall_common+0x101 > This is an example of the cause for your problem. > > The thread is executing some ZFS code, zfs_setextattr() VOP probably to > do something with the ext attrs. There, it recurses into VFS to open a > file, and vn_open_cred() waits for buffer space pressure because it is > assumed the vn_open_cred() is called from top level, not from inside > VFS/fs code. > > Until this thread finished its operation and safely returned back to > kernel/user boundary, the process cannot exit. > There are two problems. One is this call to bwillwrite(), and it is easy > to get rid of it, see the patch at the end of the message. But I wonder > why do you have so many dirty buffers and why it does not resolve itself. > Note that ZFS does not use buffer cache, you must have some other very > active fs, using buffer cache, that is somehow blocked on writes. Oke, Thanx for the analysis. I'll try the patch.. I think the use of the buffer cache comes from bonnie++ test that is hammering the UFS filesystem that is mounted on a ceph rbd-ggate device. rbd-ggate uses geom-gate to offer a disk device that is backed by an rbd-image in the ceph cluster. And some of the nodes in the cluster run on the same node as the test, so there is a lot of ZFS activity as well. Likely this server's memory is a bit small for the load thrown at it, but atm. I do not have more beefy hardware. Bonnie is actually the only way thus far to get this type of problems... This would probably also explain why this problem does not occur when using small testsizes in bonnie: the memory pressure does not get critical. --WjW From owner-freebsd-hackers@freebsd.org Fri Nov 29 09:28:19 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 11A971AA3B4 for ; Fri, 29 Nov 2019 09:28:19 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47PThL1CFyz3ClJ; Fri, 29 Nov 2019 09:28:17 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id xAT9OBmQ077824 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Nov 2019 09:24:12 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: kostikbel@gmail.com Received: from [10.58.0.4] (dadv@[10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id xAT9O8JS050223 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 29 Nov 2019 16:24:08 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Process in T state does not want to die..... To: Konstantin Belousov , Willem Jan Withagen References: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> <20191128115122.GN10580@kib.kiev.ua> <296874db-40f0-c7c9-a573-410e4c86049a@digiware.nl> <20191128195013.GU10580@kib.kiev.ua> <1ae7ad65-902c-8e5f-bcf1-1e98448c64bb@digiware.nl> <20191128214633.GV10580@kib.kiev.ua> Cc: FreeBSD Hackers , Alexander Motin , Andriy Gapon From: Eugene Grosbein Message-ID: Date: Fri, 29 Nov 2019 16:24:09 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20191128214633.GV10580@kib.kiev.ua> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=4.0 required=5.0 tests=BAYES_00, DATE_IN_FUTURE_96_Q, HELO_MISC_IP, LOCAL_FROM, SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.7 DATE_IN_FUTURE_96_Q Date: is 4 days to 4 months after Received: * date * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains * 1.1 HELO_MISC_IP Looking for more Dynamic IP Relays X-Spam-Level: **** X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on eg.sd.rdtc.ru X-Rspamd-Queue-Id: 47PThL1CFyz3ClJ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.77 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[]; IP_SCORE(-1.67)[ip: (-4.42), ipnet: 2a01:4f8::/29(-2.34), asn: 24940(-1.57), country: DE(-0.01)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Nov 2019 09:28:19 -0000 29.11.2019 4:46, Konstantin Belousov write: >> sys_extattr_set_fd+0xee amd64_syscall+0x364 fast_syscall_common+0x101 > This is an example of the cause for your problem. I observe this problem too, but my use case is different. I have several bhyve instances running Windows guests over ZVOLs over SSD-only RAIDZ1 pool. "zfs destroy" for snapshots with large "used" numbers takes long time (several minutes) due to slow TRIM. Sometimes this makes virtual guest unresponsible and attempt to restart the bhyve instance may bring it to Exiting (E) state for several minutes and it finishes successfully after that. But sometimes bhyve process hangs in T state indefinitely. This is 11.3-STABLE/amd64 r354667. Should I try your patch too? From owner-freebsd-hackers@freebsd.org Fri Nov 29 10:43:55 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 176551ABBEE for ; Fri, 29 Nov 2019 10:43:55 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47PWMZ19ypz3GlK; Fri, 29 Nov 2019 10:43:53 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id xATAhRHE078354 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Nov 2019 10:43:30 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: kostikbel@gmail.com Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id xATAhNRj050969 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 29 Nov 2019 17:43:23 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Process in T state does not want to die..... To: Konstantin Belousov , Willem Jan Withagen References: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> <20191128115122.GN10580@kib.kiev.ua> <296874db-40f0-c7c9-a573-410e4c86049a@digiware.nl> <20191128195013.GU10580@kib.kiev.ua> <1ae7ad65-902c-8e5f-bcf1-1e98448c64bb@digiware.nl> <20191128214633.GV10580@kib.kiev.ua> Cc: FreeBSD Hackers , Alexander Motin , Andriy Gapon From: Eugene Grosbein Message-ID: Date: Fri, 29 Nov 2019 17:43:18 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 47PWMZ19ypz3GlK X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.77 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[]; IP_SCORE(-1.67)[ip: (-4.44), ipnet: 2a01:4f8::/29(-2.35), asn: 24940(-1.58), country: DE(-0.01)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Nov 2019 10:43:55 -0000 29.11.2019 16:24, Eugene Grosbein wrote: > 29.11.2019 4:46, Konstantin Belousov write: > >>> sys_extattr_set_fd+0xee amd64_syscall+0x364 fast_syscall_common+0x101 >> This is an example of the cause for your problem. > > I observe this problem too, but my use case is different. > > I have several bhyve instances running Windows guests over ZVOLs over SSD-only RAIDZ1 pool. > "zfs destroy" for snapshots with large "used" numbers takes long time (several minutes) due to slow TRIM. > Sometimes this makes virtual guest unresponsible and attempt to restart the bhyve instance may bring it to Exiting (E) > state for several minutes and it finishes successfully after that. But sometimes bhyve process hangs in T state indefinitely. > > This is 11.3-STABLE/amd64 r354667. Should I try your patch too? OTOH, same system has several FreeBSD jails over mounted ZFS (file systems) over same pool. These file systems have snapshots created/removed too and snapshot are large (upto 10G). From owner-freebsd-hackers@freebsd.org Fri Nov 29 11:05:39 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C3DA91AC3B5 for ; Fri, 29 Nov 2019 11:05:39 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47PWrf4xBPz3Hbs; Fri, 29 Nov 2019 11:05:37 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 7F3A82C660; Fri, 29 Nov 2019 12:05:35 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id s3ngmaYEjX8k; Fri, 29 Nov 2019 12:05:34 +0100 (CET) Received: from [192.168.101.70] (unknown [192.168.101.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 6C7AC2C65C; Fri, 29 Nov 2019 12:05:34 +0100 (CET) Subject: Re: Process in T state does not want to die..... To: Eugene Grosbein , Konstantin Belousov Cc: FreeBSD Hackers , Alexander Motin , Andriy Gapon References: <966f830c-bf09-3683-90da-e70aa343cc16@digiware.nl> <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> <20191128115122.GN10580@kib.kiev.ua> <296874db-40f0-c7c9-a573-410e4c86049a@digiware.nl> <20191128195013.GU10580@kib.kiev.ua> <1ae7ad65-902c-8e5f-bcf1-1e98448c64bb@digiware.nl> <20191128214633.GV10580@kib.kiev.ua> From: Willem Jan Withagen Message-ID: Date: Fri, 29 Nov 2019 12:05:34 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 47PWrf4xBPz3Hbs X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 176.74.240.9 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.61 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[digiware.nl]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[9.240.74.176.list.dnswl.org : 127.0.9.2]; IP_SCORE(-3.11)[ip: (-9.80), ipnet: 176.74.224.0/19(-4.90), asn: 28878(-0.90), country: NL(0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:176.74.224.0/19, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Nov 2019 11:05:39 -0000 On 29-11-2019 11:43, Eugene Grosbein wrote: > 29.11.2019 16:24, Eugene Grosbein wrote: > >> 29.11.2019 4:46, Konstantin Belousov write: >> >>>> sys_extattr_set_fd+0xee amd64_syscall+0x364 fast_syscall_common+0x101 >>> This is an example of the cause for your problem. >> >> I observe this problem too, but my use case is different. >> >> I have several bhyve instances running Windows guests over ZVOLs over SSD-only RAIDZ1 pool. >> "zfs destroy" for snapshots with large "used" numbers takes long time (several minutes) due to slow TRIM. >> Sometimes this makes virtual guest unresponsible and attempt to restart the bhyve instance may bring it to Exiting (E) >> state for several minutes and it finishes successfully after that. But sometimes bhyve process hangs in T state indefinitely. >> >> This is 11.3-STABLE/amd64 r354667. Should I try your patch too? > > OTOH, same system has several FreeBSD jails over mounted ZFS (file systems) over same pool. > These file systems have snapshots created/removed too and snapshot are large (upto 10G). > From what I get from Konstantin is that this problem is due to memory pressure build by both ZFS and the buffercache used by UFS. And the buffercache is waiting for some buffer memory to be able to do its work. If wanted I can try and put a ZFS fs on /dev/ggate0 so that any buffering would be in ZFS and not in UFS. But even with the patch I still now have: root 3471 0.0 5.8 646768 480276 - TsJ 11:16 0:10.74 ceph-osd -i 0 root 3530 0.0 11.8 1153860 985020 - TsJ 11:17 0:11.51 ceph-osd -i 1 root 3532 0.0 5.3 608760 438676 - TsJ 11:17 0:07.31 ceph-osd -i 2 root 3534 0.0 3.2 435564 266328 - IsJ 11:17 0:07.35 ceph-osd -i 3 root 3536 0.0 4.8 565792 398392 - IsJ 11:17 0:08.73 ceph-osd -i 5 root 3553 0.0 2.3 362892 192348 - TsJ 11:17 0:04.21 ceph-osd -i 6 root 3556 0.0 3.0 421516 246956 - TsJ 11:17 0:04.81 ceph-osd -i 4 And from procstat -kk below it looks like things are still stuck in bwillwrite, but now with another set of functions. I guess not writing an extattrib() but writing a file. # ps -o pid,lwp,flags,flags2,state,tracer,command -p 3471 PID LWP F F2 STAT TRACER COMMAND 3471 104097 11080081 00000000 TsJ 0 ceph-osd -i 0 # procstat -kk 3471: 3471 104310 ceph-osd journal_write mi_switch+0xe0 sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 dofilewrite+0x93 sys_writev+0x6e amd64_syscall+0x362 fast_syscall_common+0x101 3471 104311 ceph-osd fn_jrn_objstore mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104312 ceph-osd tp_fstore_op mi_switch+0xe0 sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 dofilewrite+0x93 sys_write+0xc1 amd64_syscall+0x362 fast_syscall_common+0x101 3471 104313 ceph-osd tp_fstore_op mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104314 ceph-osd fn_odsk_fstore mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104315 ceph-osd fn_appl_fstore mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104316 ceph-osd safe_timer mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104355 ceph-osd ms_dispatch mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104356 ceph-osd ms_local mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104357 ceph-osd safe_timer mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104358 ceph-osd fn_anonymous mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104359 ceph-osd safe_timer mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104360 ceph-osd ms_dispatch mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104361 ceph-osd ms_local mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104362 ceph-osd ms_dispatch mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104363 ceph-osd ms_local mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104364 ceph-osd ms_dispatch mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104365 ceph-osd ms_local mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104366 ceph-osd ms_dispatch mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104367 ceph-osd ms_local mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104368 ceph-osd ms_dispatch mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104369 ceph-osd ms_local mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104370 ceph-osd ms_dispatch mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104371 ceph-osd ms_local mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104372 ceph-osd fn_anonymous mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104373 ceph-osd finisher mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104374 ceph-osd safe_timer mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104375 ceph-osd safe_timer mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104376 ceph-osd osd_srv_agent mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104377 ceph-osd tp_osd_tp mi_switch+0xe0 thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f 3471 104378 ceph-osd tp_osd_tp mi_switch+0xe0 thread_suspend_switch+0x140 thread_single+0x47b sigexit+0x53 postsig+0x304 ast+0x327 fast_syscall_common+0x198 From owner-freebsd-hackers@freebsd.org Fri Nov 29 11:40:40 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F30A11AD0F7 for ; Fri, 29 Nov 2019 11:40:40 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47PXd36mp3z3K7h; Fri, 29 Nov 2019 11:40:39 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id xATBeUvb028642 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 29 Nov 2019 13:40:33 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua xATBeUvb028642 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id xATBeTmN028640; Fri, 29 Nov 2019 13:40:29 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 29 Nov 2019 13:40:29 +0200 From: Konstantin Belousov To: Willem Jan Withagen Cc: Eugene Grosbein , FreeBSD Hackers , Alexander Motin , Andriy Gapon Subject: Re: Process in T state does not want to die..... Message-ID: <20191129114029.GX10580@kib.kiev.ua> References: <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> <20191128115122.GN10580@kib.kiev.ua> <296874db-40f0-c7c9-a573-410e4c86049a@digiware.nl> <20191128195013.GU10580@kib.kiev.ua> <1ae7ad65-902c-8e5f-bcf1-1e98448c64bb@digiware.nl> <20191128214633.GV10580@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.2 (2019-09-21) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 47PXd36mp3z3K7h X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; RCPT_COUNT_FIVE(0.00)[5]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; IP_SCORE(0.00)[ip: (-2.73), ipnet: 2001:470::/32(-4.64), asn: 6939(-3.51), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Nov 2019 11:40:41 -0000 On Fri, Nov 29, 2019 at 12:05:34PM +0100, Willem Jan Withagen wrote: > On 29-11-2019 11:43, Eugene Grosbein wrote: > > 29.11.2019 16:24, Eugene Grosbein wrote: > > > >> 29.11.2019 4:46, Konstantin Belousov write: > >> > >>>> sys_extattr_set_fd+0xee amd64_syscall+0x364 fast_syscall_common+0x101 > >>> This is an example of the cause for your problem. > >> > >> I observe this problem too, but my use case is different. > >> > >> I have several bhyve instances running Windows guests over ZVOLs over SSD-only RAIDZ1 pool. > >> "zfs destroy" for snapshots with large "used" numbers takes long time (several minutes) due to slow TRIM. > >> Sometimes this makes virtual guest unresponsible and attempt to restart the bhyve instance may bring it to Exiting (E) > >> state for several minutes and it finishes successfully after that. But sometimes bhyve process hangs in T state indefinitely. > >> > >> This is 11.3-STABLE/amd64 r354667. Should I try your patch too? > > > > OTOH, same system has several FreeBSD jails over mounted ZFS (file systems) over same pool. > > These file systems have snapshots created/removed too and snapshot are large (upto 10G). > > > > From what I get from Konstantin is that this problem is due to memory > pressure build by both ZFS and the buffercache used by UFS. > And the buffercache is waiting for some buffer memory to be able to do > its work. > > If wanted I can try and put a ZFS fs on /dev/ggate0 so that any > buffering would be in ZFS and not in UFS. > > But even with the patch I still now have: > root 3471 0.0 5.8 646768 480276 - TsJ 11:16 0:10.74 ceph-osd -i 0 > root 3530 0.0 11.8 1153860 985020 - TsJ 11:17 0:11.51 ceph-osd -i 1 > root 3532 0.0 5.3 608760 438676 - TsJ 11:17 0:07.31 ceph-osd -i 2 > root 3534 0.0 3.2 435564 266328 - IsJ 11:17 0:07.35 ceph-osd -i 3 > root 3536 0.0 4.8 565792 398392 - IsJ 11:17 0:08.73 ceph-osd -i 5 > root 3553 0.0 2.3 362892 192348 - TsJ 11:17 0:04.21 ceph-osd -i 6 > root 3556 0.0 3.0 421516 246956 - TsJ 11:17 0:04.81 ceph-osd -i 4 > > And from procstat -kk below it looks like things are still stuck in > bwillwrite, but now with another set of functions. I guess not writing > an extattrib() but writing a file. Yes, it should resolve after you end the load that starves the buffer cache' dirty space. Or wait some time until the thread gets its portion of share, which is unfair and could take a long time. I will commit the VN_OPEN_INVFS patch shortly. > > # ps -o pid,lwp,flags,flags2,state,tracer,command -p 3471 > PID LWP F F2 STAT TRACER COMMAND > 3471 104097 11080081 00000000 TsJ 0 ceph-osd -i 0 > > # procstat -kk 3471: > 3471 104310 ceph-osd journal_write mi_switch+0xe0 > sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 dofilewrite+0x93 > sys_writev+0x6e amd64_syscall+0x362 fast_syscall_common+0x101 > 3471 104311 ceph-osd fn_jrn_objstore mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104312 ceph-osd tp_fstore_op mi_switch+0xe0 > sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 dofilewrite+0x93 > sys_write+0xc1 amd64_syscall+0x362 fast_syscall_common+0x101 > 3471 104313 ceph-osd tp_fstore_op mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104314 ceph-osd fn_odsk_fstore mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104315 ceph-osd fn_appl_fstore mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104316 ceph-osd safe_timer mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104355 ceph-osd ms_dispatch mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104356 ceph-osd ms_local mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104357 ceph-osd safe_timer mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104358 ceph-osd fn_anonymous mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104359 ceph-osd safe_timer mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104360 ceph-osd ms_dispatch mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104361 ceph-osd ms_local mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104362 ceph-osd ms_dispatch mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104363 ceph-osd ms_local mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104364 ceph-osd ms_dispatch mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104365 ceph-osd ms_local mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104366 ceph-osd ms_dispatch mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104367 ceph-osd ms_local mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104368 ceph-osd ms_dispatch mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104369 ceph-osd ms_local mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104370 ceph-osd ms_dispatch mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104371 ceph-osd ms_local mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104372 ceph-osd fn_anonymous mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104373 ceph-osd finisher mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104374 ceph-osd safe_timer mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104375 ceph-osd safe_timer mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104376 ceph-osd osd_srv_agent mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104377 ceph-osd tp_osd_tp mi_switch+0xe0 > thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f > 3471 104378 ceph-osd tp_osd_tp mi_switch+0xe0 > thread_suspend_switch+0x140 thread_single+0x47b sigexit+0x53 > postsig+0x304 ast+0x327 fast_syscall_common+0x198 > > From owner-freebsd-hackers@freebsd.org Fri Nov 29 14:24:50 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ACAB81B1E78 for ; Fri, 29 Nov 2019 14:24:50 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47PcGT19TWz3xsZ; Fri, 29 Nov 2019 14:24:48 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id xATEOMNg079260 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Nov 2019 14:24:24 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: kostikbel@gmail.com Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id xATEOI2V056231 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 29 Nov 2019 21:24:18 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Process in T state does not want to die..... To: Konstantin Belousov , Willem Jan Withagen References: <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> <20191128115122.GN10580@kib.kiev.ua> <296874db-40f0-c7c9-a573-410e4c86049a@digiware.nl> <20191128195013.GU10580@kib.kiev.ua> <1ae7ad65-902c-8e5f-bcf1-1e98448c64bb@digiware.nl> <20191128214633.GV10580@kib.kiev.ua> <20191129114029.GX10580@kib.kiev.ua> Cc: FreeBSD Hackers , Alexander Motin , Andriy Gapon From: Eugene Grosbein Message-ID: <61899086-0272-6d72-b250-b697efe8b4f5@grosbein.net> Date: Fri, 29 Nov 2019 21:24:13 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20191129114029.GX10580@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 47PcGT19TWz3xsZ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.78 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[]; IP_SCORE(-1.68)[ip: (-4.46), ipnet: 2a01:4f8::/29(-2.35), asn: 24940(-1.58), country: DE(-0.01)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Nov 2019 14:24:50 -0000 29.11.2019 18:40, Konstantin Belousov wrote: >> From what I get from Konstantin is that this problem is due to memory >> pressure build by both ZFS and the buffercache used by UFS. >> And the buffercache is waiting for some buffer memory to be able to do >> its work. >> >> If wanted I can try and put a ZFS fs on /dev/ggate0 so that any >> buffering would be in ZFS and not in UFS. >> >> But even with the patch I still now have: >> root 3471 0.0 5.8 646768 480276 - TsJ 11:16 0:10.74 ceph-osd -i 0 >> root 3530 0.0 11.8 1153860 985020 - TsJ 11:17 0:11.51 ceph-osd -i 1 >> root 3532 0.0 5.3 608760 438676 - TsJ 11:17 0:07.31 ceph-osd -i 2 >> root 3534 0.0 3.2 435564 266328 - IsJ 11:17 0:07.35 ceph-osd -i 3 >> root 3536 0.0 4.8 565792 398392 - IsJ 11:17 0:08.73 ceph-osd -i 5 >> root 3553 0.0 2.3 362892 192348 - TsJ 11:17 0:04.21 ceph-osd -i 6 >> root 3556 0.0 3.0 421516 246956 - TsJ 11:17 0:04.81 ceph-osd -i 4 >> >> And from procstat -kk below it looks like things are still stuck in >> bwillwrite, but now with another set of functions. I guess not writing >> an extattrib() but writing a file. > Yes, it should resolve after you end the load that starves the buffer > cache' dirty space. Or wait some time until the thread gets its portion > of share, which is unfair and could take a long time. > > I will commit the VN_OPEN_INVFS patch shortly. My system is ZFS-only. OTOH, it has vfs.zfs.arc_max=160g out of 360G RAM plus it runs several bhyve instances that leak host's memory significantly upto overflowing swap space (64G): Nov 25 06:55:01 col02 kernel: swap_pager_getswapspace(32): failed Is patch relevant for me? Before update from 11.2-STABLE, there were no leaks. From owner-freebsd-hackers@freebsd.org Fri Nov 29 15:06:19 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8262F1B2F22 for ; Fri, 29 Nov 2019 15:06:19 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47PdBK3MLsz414r; Fri, 29 Nov 2019 15:06:16 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id E3555A9D08; Fri, 29 Nov 2019 16:06:10 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bshASlAQhIJH; Fri, 29 Nov 2019 16:06:09 +0100 (CET) Received: from [192.168.101.70] (unknown [192.168.101.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 5D407A9CE9; Fri, 29 Nov 2019 16:06:09 +0100 (CET) Subject: Re: Process in T state does not want to die..... To: Konstantin Belousov Cc: FreeBSD Hackers , Alexander Motin , Andriy Gapon , Eugene Grosbein References: <3c57e51d-fa36-39a3-9691-49698e8d2124@grosbein.net> <91490c30-45e9-3c38-c55b-12534fd09e28@digiware.nl> <20191128115122.GN10580@kib.kiev.ua> <296874db-40f0-c7c9-a573-410e4c86049a@digiware.nl> <20191128195013.GU10580@kib.kiev.ua> <1ae7ad65-902c-8e5f-bcf1-1e98448c64bb@digiware.nl> <20191128214633.GV10580@kib.kiev.ua> <20191129114029.GX10580@kib.kiev.ua> From: Willem Jan Withagen Message-ID: <21b8b806-614b-cc9c-7b5f-496e0e8c541c@digiware.nl> Date: Fri, 29 Nov 2019 16:06:10 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1 MIME-Version: 1.0 In-Reply-To: <20191129114029.GX10580@kib.kiev.ua> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 47PdBK3MLsz414r X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 2001:4cb8:90:ffff::3 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.31 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[digiware.nl]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; IP_SCORE(-3.01)[ip: (-9.52), ipnet: 2001:4cb8::/29(-4.64), asn: 28878(-0.93), country: NL(0.02)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:2001:4cb8::/29, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Nov 2019 15:06:19 -0000 On 29-11-2019 12:40, Konstantin Belousov wrote: > On Fri, Nov 29, 2019 at 12:05:34PM +0100, Willem Jan Withagen wrote: >> On 29-11-2019 11:43, Eugene Grosbein wrote: >>> 29.11.2019 16:24, Eugene Grosbein wrote: >>> >>>> 29.11.2019 4:46, Konstantin Belousov write: >>>> >>>>>> sys_extattr_set_fd+0xee amd64_syscall+0x364 fast_syscall_common+0x101 >>>>> This is an example of the cause for your problem. >>>> >>>> I observe this problem too, but my use case is different. >>>> >>>> I have several bhyve instances running Windows guests over ZVOLs over SSD-only RAIDZ1 pool. >>>> "zfs destroy" for snapshots with large "used" numbers takes long time (several minutes) due to slow TRIM. >>>> Sometimes this makes virtual guest unresponsible and attempt to restart the bhyve instance may bring it to Exiting (E) >>>> state for several minutes and it finishes successfully after that. But sometimes bhyve process hangs in T state indefinitely. >>>> >>>> This is 11.3-STABLE/amd64 r354667. Should I try your patch too? >>> >>> OTOH, same system has several FreeBSD jails over mounted ZFS (file systems) over same pool. >>> These file systems have snapshots created/removed too and snapshot are large (upto 10G). >>> >> >> From what I get from Konstantin is that this problem is due to memory >> pressure build by both ZFS and the buffercache used by UFS. >> And the buffercache is waiting for some buffer memory to be able to do >> its work. >> >> If wanted I can try and put a ZFS fs on /dev/ggate0 so that any >> buffering would be in ZFS and not in UFS. >> >> But even with the patch I still now have: >> root 3471 0.0 5.8 646768 480276 - TsJ 11:16 0:10.74 ceph-osd -i 0 >> root 3530 0.0 11.8 1153860 985020 - TsJ 11:17 0:11.51 ceph-osd -i 1 >> root 3532 0.0 5.3 608760 438676 - TsJ 11:17 0:07.31 ceph-osd -i 2 >> root 3534 0.0 3.2 435564 266328 - IsJ 11:17 0:07.35 ceph-osd -i 3 >> root 3536 0.0 4.8 565792 398392 - IsJ 11:17 0:08.73 ceph-osd -i 5 >> root 3553 0.0 2.3 362892 192348 - TsJ 11:17 0:04.21 ceph-osd -i 6 >> root 3556 0.0 3.0 421516 246956 - TsJ 11:17 0:04.81 ceph-osd -i 4 >> >> And from procstat -kk below it looks like things are still stuck in >> bwillwrite, but now with another set of functions. I guess not writing >> an extattrib() but writing a file. > Yes, it should resolve after you end the load that starves the buffer > cache' dirty space. Or wait some time until the thread gets its portion > of share, which is unfair and could take a long time. Eh, right.... This pointed me in a direction to offer some stress relief. The other process is: root 3581 0.0 0.0 10724 2372 v1 D+ 11:20 0:00.91 bonnie -s 256 Which is also stuck in disk i/o, in the kernel I guess. So killing it only works once any of the writes succeed. Luckely the geom-gateway (rbd-ggate) is also in userspace, and can be killed. Which I guess is where the buffers collected, because shooting that down, immediately allows the ceph-osd to continue crashing. So is there any controls that I can apply as to make all these components behave better? - One thing would be more memory but this board only allows 8G. (It s an oldy) - Don't run heavy UFS buffer consumers. Are there any sysctl values I can monitor to check used buffersize? I guess the system has these values, since top can find 'm # sysctl -a | grep buffer vfs.hifreebuffers: 768 vfs.lofreebuffers: 512 vfs.numfreebuffers: 52820 vfs.hidirtybuffers: 13225 vfs.lodirtybuffers: 6612 vfs.numdirtybuffers: 0 vfs.altbufferflushes: 0 vfs.dirtybufferflushes: 0 --WjW > > I will commit the VN_OPEN_INVFS patch shortly. >> >> # ps -o pid,lwp,flags,flags2,state,tracer,command -p 3471 >> PID LWP F F2 STAT TRACER COMMAND >> 3471 104097 11080081 00000000 TsJ 0 ceph-osd -i 0 >> >> # procstat -kk 3471: >> 3471 104310 ceph-osd journal_write mi_switch+0xe0 >> sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 dofilewrite+0x93 >> sys_writev+0x6e amd64_syscall+0x362 fast_syscall_common+0x101 >> 3471 104311 ceph-osd fn_jrn_objstore mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104312 ceph-osd tp_fstore_op mi_switch+0xe0 >> sleepq_wait+0x2c _sleep+0x247 bwillwrite+0x97 dofilewrite+0x93 >> sys_write+0xc1 amd64_syscall+0x362 fast_syscall_common+0x101 >> 3471 104313 ceph-osd tp_fstore_op mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104314 ceph-osd fn_odsk_fstore mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104315 ceph-osd fn_appl_fstore mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104316 ceph-osd safe_timer mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104355 ceph-osd ms_dispatch mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104356 ceph-osd ms_local mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104357 ceph-osd safe_timer mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104358 ceph-osd fn_anonymous mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104359 ceph-osd safe_timer mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104360 ceph-osd ms_dispatch mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104361 ceph-osd ms_local mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104362 ceph-osd ms_dispatch mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104363 ceph-osd ms_local mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104364 ceph-osd ms_dispatch mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104365 ceph-osd ms_local mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104366 ceph-osd ms_dispatch mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104367 ceph-osd ms_local mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104368 ceph-osd ms_dispatch mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104369 ceph-osd ms_local mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104370 ceph-osd ms_dispatch mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104371 ceph-osd ms_local mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104372 ceph-osd fn_anonymous mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104373 ceph-osd finisher mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104374 ceph-osd safe_timer mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104375 ceph-osd safe_timer mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104376 ceph-osd osd_srv_agent mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104377 ceph-osd tp_osd_tp mi_switch+0xe0 >> thread_suspend_check+0x297 ast+0x3b9 doreti_ast+0x1f >> 3471 104378 ceph-osd tp_osd_tp mi_switch+0xe0 >> thread_suspend_switch+0x140 thread_single+0x47b sigexit+0x53 >> postsig+0x304 ast+0x327 fast_syscall_common+0x198 >> >> > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@freebsd.org Sat Nov 30 20:04:42 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6B5561B5BCB for ; Sat, 30 Nov 2019 20:04:42 +0000 (UTC) (envelope-from salvadore@FreeBSD.org) Received: from MailHost (unknown [80.211.33.142]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47QMmB05gGz4QLc for ; Sat, 30 Nov 2019 20:04:41 +0000 (UTC) (envelope-from salvadore@FreeBSD.org) Received: from root (uid 0) (envelope-from salvadore@FreeBSD.org) id 2775 by MailHost (DragonFly Mail Agent v0.11+); Sat, 30 Nov 2019 21:04:10 +0100 To: freebsd-hackers@FreeBSD.org Subject: Call for 2019Q4 quarterly status reports Date: Sat, 30 Nov 2019 21:04:10 +0100 Message-Id: <5de2cb3a.2775.6221377a@MailHost> From: X-Rspamd-Queue-Id: 47QMmB05gGz4QLc X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-1.89 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-0.92)[-0.917,0]; ASN(0.00)[asn:31034, ipnet:80.211.0.0/17, country:IT]; NEURAL_HAM_LONG(-0.98)[-0.977,0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Nov 2019 20:04:42 -0000 Dear FreeBSD Community, The deadline for the next FreeBSD Quarterly Status update is January 1, 2020, for work done since the last round of Quarterly Reports: October, 2019 - December, 2019. Starting with this quarter, reports will be collected during the last month of every quarter (instead of the first month of the next quarter, as was previously the case). Status report submissions do not need to be very long. They may be about anything happening in the FreeBSD project and community, and provide a great way to inform FreeBSD users and developers about work that is underway or has been completed. Report submissions are not limited to committers; anyone doing anything interesting and FreeBSD related can -- and should -- write one! The preferred method is to follow the guidelines at the Quarterly GitHub repository: https://github.com/freebsd/freebsd-quarterly Alternatively you can fetch the Markdown template, fill it in, and email it to quarterly@FreeBSD.org. The template can be found at: https://raw.githubusercontent.com/freebsd/freebsd-quarterly/master/report-sample.md We look forward to seeing your 2019Q4 reports! Thanks, Lorenzo Salvadore (on behalf of quarterly@)