From owner-freebsd-arch@freebsd.org Tue May 1 17:38:22 2018 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 75108FB1964 for ; Tue, 1 May 2018 17:38:22 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-io0-x229.google.com (mail-io0-x229.google.com [IPv6:2607:f8b0:4001:c06::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 056F37CB22 for ; Tue, 1 May 2018 17:38:22 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-io0-x229.google.com with SMTP id z4-v6so14464486iof.5 for ; Tue, 01 May 2018 10:38:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=3f+VC7234ELXxi15r/cbzaPwTJ3rHTaJvcMmx3j/dps=; b=Zb54VfrmQWzUgPtZEtsYRliJtFjaI7jN74yyZ7HSg+lIu25K1BaWwccYWYYLGTEk8/ GMiW9p5jUZ7rz5CEcgWHwwffYoGfyUeHXZ6ZAwSDr7Nne1KIFeEK8QMHP58sW1SgF162 B2v8+qycd2qADv4XydPcM2U4Amyi/9VbqepIu7qvMlGHDNxg+zEufpEqSDMtXmJSSy9p Yi57HtsQIuo1ZdfK+ttf8Bdhk8fJH2epWR5y7C/HeCvLwrEVy8Ze/G6g/LTkWh56Xcki tZDgyFvAqLUopqaZhSgdiA4MggjHe9+BpZ6NO//8EdMhm+VmquVcXdL2oN9hFEvX3n7m oXqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=3f+VC7234ELXxi15r/cbzaPwTJ3rHTaJvcMmx3j/dps=; b=pD/DQ4aWxtcXa8QwSK2ONVo9oIJHgQsZ27dRmEuaxwYbrbxGskBkQfQyUqD75qC5IX b7JyPyJ+XX2azicvjSGXRxp9JKxsJTj4IpQWNQCsOE2faBjVa7Y1/CaaeBk2DEDxWyvv AOh/GAFjHEInoEUtVP2k/CJ9Ow+XM9ws+ajBUesl5tlzoMxIIpmUl8vU8VNQ/dAVSni0 vwhuQTqzPia6h3ZURXu1NzGedoT38dt0Hfsh1OshDBNqjBOzB8I4jOeb7XNHIy7gyJmW /aEVkfJ4cEVbwemYAb3FFR+uAqKnzUR+7FyUwDbGsWh8dzyqQYm4ffDss/6tD0QmdVNK MMNQ== X-Gm-Message-State: ALQs6tBnILCNrFP5yw2ajdJBXMDeKBUcNyA69NryZccoWWcj6vTtMMPs ce3X8IrdhaROQ2/spM0nhAD4qQ== X-Google-Smtp-Source: AB8JxZrNt9nwxMf0Kug0xHCECia0Cxn9tYyjqSMaebEMOCQT3FxE3RNaoxDDkUMCz5Kt7d0uiVvabg== X-Received: by 2002:a6b:fd0:: with SMTP id 77-v6mr18423402iop.108.1525196300995; Tue, 01 May 2018 10:38:20 -0700 (PDT) Received: from raichu (toroon0560w-lp130-04-184-145-252-74.dsl.bell.ca. [184.145.252.74]) by smtp.gmail.com with ESMTPSA id e18-v6sm4884237itc.3.2018.05.01.10.38.19 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 May 2018 10:38:20 -0700 (PDT) Sender: Mark Johnston Date: Tue, 1 May 2018 13:38:17 -0400 From: Mark Johnston To: freebsd-arch@FreeBSD.org Subject: Re: CFT: netdump Message-ID: <20180501173817.GB47315@raichu> References: <20180411210445.GC43015@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180411210445.GC43015@raichu> User-Agent: Mutt/1.9.5 (2018-04-13) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 May 2018 17:38:22 -0000 On Wed, Apr 11, 2018 at 05:04:45PM -0400, Mark Johnston wrote: > Hi, > > In user/markj/netdump I've been working on a feature which allows the > kernel to transmit kernel dumps over a network rather than writing to a > local storage device. This is convenient for VMs, diskless or embedded > devices, and devices with limited storage space. The server component of > netdump is a UDP daemon which is in ports as ftp/netdumpd. > > The original code has been around for a long time and wasn't authored by > me. However, I've fixed a number of bugs, made it more reliable, and > integrated it with dumpon(8). I also largely rewrote the userland > daemon. At this point I'd like to propose committing it to HEAD. > > The netdump(4) man page included in the patch lists the drivers > currently supported by netdump[*]. To be supported, a driver must be > augmented with a set of routines for use by the generic code. These > allow the generic code to transmit packets and poll for completions. I > added a mechanism to overwrite the mbuf/cluster UMA zone pointers at > panic time so that they instead fetch mbufs from a pre-allocated pool. > This way, existing driver code can be used without modification, and we > don't impose any runtime overhead in the regular mbuf allocation path. > > Of course, the netdump data path is not fully reliable since the panic > may have interrupted a thread as it was processing tx/rx ring entries > and thus left them in an inconsistent state. In my experience this > hasn't proved to be an issue, and when testing I will typically have > several iperf streams running over the interface at panic time. That > said, the same issue exists in principle with local storage devices > as well. > > A kernel dump will generally contain sensitive data. Thus, netdump > should only be used on trusted networks. It is possible to use the > kernel dump encryption feature (dumpon -k) with netdump, however I > am not certain that the encryption scheme used is well-suited for > this situation. Any feedback on this would be appreciated. Please note > that you need netdumpd-20180411 or later when using this feature, as the > daemon needs to accept and save the symmetric key in addition to the > dump itself and support for this isn't present in earlier versions. > > A single large patch for the src tree is available here: > https://people.freebsd.org/~markj/patches/netdump/netdump-20180411.diff > Of course, the user/markj/netdump branch in SVN or its clone in the > GitHub mirror may be used as well. Documentation for configuration is in > the dumpon(8) man page. The gist of it is that one specifies some > additional network parameters to dumpon(8), along with an interface > name. For instance, > > # dumpon -c 192.168.2.11 -s 192.168.2.1 vtnet0 > > tells the kernel to transmit to netdumpd at 192.168.2.1 using a source > address of 192.168.2.11. If there is a router between the two, the > gateway IP for the client must be specified as well, with -g. > > I'm interested in any testing reports that folks may have, especially > with ixgbe or bnxt, or with NICs under load. If you have a favourite > NIC driver which does not yet support netdump, I'm happy to help add > support or review patches. Depending on how testing goes, I plan to > create reviews for various components of the patch over the next several > days. I've created reviews for various components of the change: https://reviews.freebsd.org/D15251 (post-panic mbuf allocator) https://reviews.freebsd.org/D15252 (kernel dump refactoring for netdump) https://reviews.freebsd.org/D15253 (core netdump client) https://reviews.freebsd.org/D15254 (dumpon(8) extension for netdump) https://reviews.freebsd.org/D15255 (alc(4) hooks) https://reviews.freebsd.org/D15256 (bge(4) hooks) https://reviews.freebsd.org/D15257 (bxe(4) hooks) https://reviews.freebsd.org/D15258 (cxgb(4) hooks) https://reviews.freebsd.org/D15259 (mlxen(4) hooks) https://reviews.freebsd.org/D15260 (re(4) hooks) https://reviews.freebsd.org/D15261 (vtnet(4) hooks) https://reviews.freebsd.org/D15262 (iflib hooks) If anyone is interested in reviewing some or all of these diffs, please feel free to add yourself to the corresponding review. I plan to add driver maintainers as reviewers to the corresponding driver changes.