From owner-freebsd-net@FreeBSD.ORG Tue Sep 28 18:08:27 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 54111106564A for ; Tue, 28 Sep 2010 18:08:27 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0DE7A8FC1D for ; Tue, 28 Sep 2010 18:08:26 +0000 (UTC) Received: by gxk8 with SMTP id 8so2492696gxk.13 for ; Tue, 28 Sep 2010 11:08:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=qmCwzUz5VAUR1sJJQ97Ri01gFuQw2DPjHlEMtM5xj70=; b=fT/COwpUYT42UsH+aQ+1QQTSNbz/62TVRPsWmOFIP2kkw64Z5TFDMWp453ptXntxhb V6hrZdHJtZ5UTDI+9I/IRloN3w9BEk60RZuMJB+vq0hmhwnWzoWCqJ+UsvLsN6OCvLzm vYOcA1GAs2H+IHjeD0OhmD4ydBCZJAmgw6EbE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type; b=mQGkv7qlsnkKUHgGKa/FN0szwZYcwo0KbQt+2/vvWQ6SYGkEWHMwHuhQJYDzmfoQIM Bmkam+H2pw1rg7G/OkLyrXxbIpef43b8jMu2oX9NzddfulZBz5InqOVkVU58//tvuYpm Ed0YZuhTvTplRvwfIG9ukni3snGLWn2+6fIdw= MIME-Version: 1.0 Received: by 10.150.52.11 with SMTP id z11mr508444ybz.149.1285695583833; Tue, 28 Sep 2010 10:39:43 -0700 (PDT) Sender: asmrookie@gmail.com Received: by 10.150.158.20 with HTTP; Tue, 28 Sep 2010 10:39:43 -0700 (PDT) Date: Tue, 28 Sep 2010 19:39:43 +0200 X-Google-Sender-Auth: 50houPhnTXGyhYeLaLPt8apOQf4 Message-ID: From: Attilio Rao To: freebsd-net@freebsd.org, FreeBSD Current Content-Type: text/plain; charset=UTF-8 Cc: Ryan Stone , Ed Maste Subject: [PATCH] Netdump for review and testing -- preliminary version X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Sep 2010 18:08:27 -0000 In the last weeks I worked for porting the netdump infrastructure to FreeBSD-CURRENT on the behalf of Sandvine Incorporated. Netdump is a framework that aims for handling kernel coredumps over the TCP/IP suite in order to dump to a separate machine than the running one. That may be used on an interesting number of cases involving disk-less workstations, disk driver debugging or embedded devices. GENERAL FRAMEWORK ARCHITECTURE Netdump is composed, right now, by an userland "server" and a kernel "client". The former is run on the target machine (where the dump will phisically happen) and it is responsible for receiving the packets containing coredumps frame and for correctly writing them on-disk. The latter is part of the kernel installed on the source machine (where the dump is initiated) and is responsible for building correctly UDP packets containing the coredump frames, pushing through the network interface and routing them appropriately. While the server may appear as, pretty much, a simple userland deamon dealing with UDP packets, the client imposes interesting problems as long as its activity is linked to handling kernel core dumping. More precisely, as long as the client is part of the dumping mechanism and the kernel may be in general panic conditions, netdump must ensure "robustness" of operations. That is partially achieved by reworking (and someway replicating) locally some UDP, ARP and IP operations, hardsetting some values (like the default gateway, the destination and the client address) and reducing further interactions with the kernel just to the network interface acitivities. More specifically, it implements a very basic UDP / IPv4 / ARP stack separate from the standard stack (since that may not be in a consistent state). It can dump to a server on the same LAN or via a router (correctly specifying the connection gateway). In order to receive packet on critical conditions, netdump polls the interface. Every network driver can implement hooks to be used by netdump independently by DEVICE_POLLING option, even if it is probabilly a good idea to share some code among them. The reference set of hooks is contained into "struct netdump_methods". And if_lem/if_em driver modifies may be set as reference for netdump hooks implementation. In order to work into an "up and running" system (meant as with all the devices in place) the netdump handler hooks as a pre-sync handler (differently from other dumping routines). It however suffers some problems typical of other dumping mechanism. For example, on DDB entering unlocked version of polling handler is used, in order to reduce the risk of deadlocks during inspections*. That reflects, among the netdump methods, the existence of 2 versions of polling hooks, where the "unlocked" is meant as reducing locking as much as possible. PATCH AND FURTHER WORK The patch is not totally complete and it is not intended to be committed in SVN yet. What I'm looking for now is more testing and review (in particular in terms of architecture) coverage by community. The server should be in realtively "committable" state, though, but I encourage its stress-testing. A manpage is provided along that should be very easy to understand how to use it. Things that can be further improved, as it is now, in the client, are: - Deciding if hardcoding of the kernel parameter is done properly. I personally don't like the sysctl usage and I would prefer an userland small utility used to testing and maybe add some tunables for enabling netdump early in the boot. You may have several opinions on this though. - VIMAGE and IPv6 support. - More drivers support. Right now only if_em (and if_lem) are converted to use netdump and can be used as a draft for other device drivers. if_ixgb should came along in the final, committing, version too. In general I think that all drivers supporting device polling could easilly support also netdump - Ideally dumpsys() in FreeBSD is too much disk-activity oriented. It should be made, instead, more neutral and more flexible to cope better with different interfaces. It is a quite a bit of work, however, and beyond the scope of netdump introduction (even if it could be beneficial for it) Netdump has been developed on a FreeBSD project branch located here: svn://svn.freebsd.org/base/projects/sv/ which could also forsee further informations about every single change. However, for your convenience, also a patch has been made public which is located here (against FreeBSD-CURRENT@213246): http://www.freebsd.org/~attilio/Sandvine/STABLE_8/netdump/netdump_alpha_0.diff In order to enable the client it is enough to add at your kernel config: options NETDUMP_CLIENT NETDUMP_CLIENT_DEBUG can be specified too in order to have further debuggin traces but I'd encourage to use them just for developing Netdump itself. TYPICAL USAGE This is a set of the available sysctls for netdump: # sysctl -d net.dump net.dump: netdump net.dump.enable: enable network dump net.dump.retries: times to retransmit lost packets net.dump.polls: times to poll NIC per retry net.dump.nic: NIC to dump on net.dump.gateway: dump default gateway net.dump.client: dump client And when compiled with the NETDUMP_CLIENT_DEBUG option: net.dump.serve# sysctl -d debug.netdump debug.netdump: Netdump debugging debug.netdump.crash: force crashingr: dump server A tipycal setup for netdump can be the following: Run the server on the receiving interface: # ifconfig em2 em2: flags=8843 metric 0 mtu 1500 options=209b ether 00:30:48:28:b2:7a inet 10.135.14.118 netmask 0xfffffffc broadcast 10.135.14.119 media: Ethernet autoselect (1000baseT ) status: active # netdumpsrv -a 10.135.14.118 Listening on IP 10.135.14.118 Default: dumping on /var/crash/ While on the client: # netstat -nr Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire 10.128.0.0/10 10.135.12.113 UGS 18 149 em2 10.135.12.112/30 link#3 U 0 0 em2 127.0.0.1 link#5 UH 0 118 lo0 ... # ifconfig em2 em2: flags=8843 metric 0 mtu 1500 options=209b ether 00:30:48:28:9f:b2 inet6 fe80::230:48ff:fe28:9fb2%em2 prefixlen 64 scopeid 0x3 inet 10.135.12.114 netmask 0xfffffffc broadcast 10.135.12.115 nd6 options=21 media: Ethernet autoselect (1000baseT ) status: active # sysctl net.dump.server=10.135.14.118 net.dump.server: 0.0.0.0 -> 10.135.14.118 # sysctl net.dump.client=10.135.12.114 net.dump.client: 0.0.0.0 -> 10.135.12.114 # sysctl net.dump.gateway=10.135.12.113 net.dump.gateway: 0.0.0.0 -> 10.135.12.113 # sysctl net.dump.nic=em2 net.dump.nic: none -> em2 # sysctl net.dump.enable=1 net.dump.enable: 0 -> 1 and at next dumping operation on client, it must netdump. Thanks, Attilio [* The dumping infrastructure in FreeBSD has several weaknesses that needs to be discussed and treacted separately] -- Peace can only be achieved by understanding - A. Einstein