From nobody Fri Sep 6 13:15:45 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0cDM2xqDz5V2XL for ; Fri, 06 Sep 2024 13:15:47 +0000 (UTC) (envelope-from jan@digitaldaemon.com) Received: from digitaldaemon.com (digitaldaemon.com [162.217.114.50]) by mx1.freebsd.org (Postfix) with SMTP id 4X0cDL6YL3z4Fjn for ; Fri, 6 Sep 2024 13:15:46 +0000 (UTC) (envelope-from jan@digitaldaemon.com) Authentication-Results: mx1.freebsd.org; none Received: (qmail 35542 invoked by uid 89); 6 Sep 2024 13:15:45 -0000 Received: from c-69-142-153-99.hsd1.nj.comcast.net (HELO ?10.0.0.22?) (jan@digitaldaemon.com@69.142.153.99) by digitaldaemon.com with SMTP; 6 Sep 2024 13:15:45 -0000 Content-Type: multipart/alternative; boundary="------------SAXOS04t0oRybAieYmCfmxlX" Message-ID: <1dcfee7b-11e7-4bde-aa5f-6d90101a6022@digitaldaemon.com> Date: Fri, 6 Sep 2024 09:15:45 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: The Case for Rust (in any system) To: David Chisnall , Alan Somers Cc: Dmitry Salychev , freebsd-hackers@freebsd.org References: <6FEF9D06-01DC-48DC-93D2-178F9726C1D3@freebsd.org> Content-Language: en-US From: Jan Knepper In-Reply-To: <6FEF9D06-01DC-48DC-93D2-178F9726C1D3@freebsd.org> X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36236, ipnet:162.217.112.0/22, country:US] X-Rspamd-Queue-Id: 4X0cDL6YL3z4Fjn This is a multi-part message in MIME format. --------------SAXOS04t0oRybAieYmCfmxlX Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 9/6/24 03:02, David Chisnall wrote: > On 5 Sep 2024, at 22:13, Alan Somers wrote: >> I used to check it, years ago. But I gave up. The UI is too hard to >> use and false alarms are both too frequent and too hard to suppress. >> Plus, it's a real drag that I can't run the tool myself. Instead, I >> need to wait for the next scheduled run. > In general, it’s very hard to add static analysis to existing projects. Not the experience I have. It was as simple as wrapping the build with 'cov-'. (Also, Coverity would come and visit us, and assist with setting up and getting analyzes started). > The property that you want is that there are no *new* static analyser errors in a new commit, but that’s requires tracking all of the existing ones. Coverity, when we used it did keep track of all the issues: A false positive, when marked as false, would not be reported again. A true positive, thus an actual problem, most (all!) of the time would disappear after the code was fixed and Coverity was run again. Every analyzer I have ever used gives false positives. VeraCode was one of the worst. Coverity was one of the best. (But times have changed) > In CHERIoT RTOS, we run the clang analyser in CI as one of the checks that must pass before a PR can be merged. This is possible because we started doing it very early on. Good! :-) > It may be possible for some individual parts of FreeBSD, but when we started with Coverity I looked at the reports and the first ten I looked at were all false positives. That can happen. I think when we first started with Coverity the false positives where just less then 50%. Which is truly extremely good for an analyzer. Then Coverity was tuned/reconfigured a bit and it became much better. > There are probably some serious issues in there but the effort to find them is high. For a new project, that cost is a small incremental cost in each commit and code review (if the analyser finds something, reviewer has to agree that it’s a false positive). > Well, the cost is high on an existing project alike FreeBSD, because this has not been done... My guess is... It could be considered 'Technical Debt'. What helps quite a bit is tuning Coverity to the right configuration, so the false positives are not overwhelming. Then, when the true positives are resolved, dial the tuning back a little. Jan --------------SAXOS04t0oRybAieYmCfmxlX Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit On 9/6/24 03:02, David Chisnall wrote:
On 5 Sep 2024, at 22:13, Alan Somers <asomers@freebsd.org> wrote:

I used to check it, years ago.  But I gave up.  The UI is too hard to
use and false alarms are both too frequent and too hard to suppress.
Plus, it's a real drag that I can't run the tool myself.  Instead, I
need to wait for the next scheduled run.

In general, it’s very hard to add static analysis to existing projects.
Not the experience I have.
It was as simple as wrapping the build with 'cov-<something>'.
(Also, Coverity would come and visit us, and assist with setting up and getting analyzes started).
 The property that you want is that there are no *new* static analyser errors in a new commit, but that’s requires tracking all of the existing ones.
Coverity, when we used it did keep track of all the issues:
A false positive, when marked as false, would not be reported again.
A true positive, thus an actual problem, most (all!) of the time would disappear after the code was fixed and Coverity was run again.

Every analyzer I have ever used gives false positives.
VeraCode was one of the worst.
Coverity was one of the best.
(But times have changed)
 In CHERIoT RTOS, we run the clang analyser in CI as one of the checks that must pass before a PR can be merged. This is possible because we started doing it very early on.
Good! :-)
 It may be possible for some individual parts of FreeBSD, but when we started with Coverity I looked at the reports and the first ten I looked at were all false positives.
That can happen. I think when we first started with Coverity the false positives where just less then 50%. Which is truly extremely good for an analyzer. Then Coverity was tuned/reconfigured a bit and it became much better.
 There are probably some serious issues in there but the effort to find them is high. For a new project, that cost is a small incremental cost in each commit and code review (if the analyser finds something, reviewer has to agree that it’s a false positive).
Well, the cost is high on an existing project alike FreeBSD, because this has not been done... My guess is... It could be considered 'Technical Debt'.
What helps quite a bit is tuning Coverity to the right configuration, so the false positives are not overwhelming. Then, when the true positives are resolved, dial the tuning back a little.

Jan


--------------SAXOS04t0oRybAieYmCfmxlX--