From owner-freebsd-questions@FreeBSD.ORG Wed Mar 2 14:27:19 2011 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5A8D71065672 for ; Wed, 2 Mar 2011 14:27:19 +0000 (UTC) (envelope-from cyber366@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0EAB78FC16 for ; Wed, 2 Mar 2011 14:27:18 +0000 (UTC) Received: by vxc34 with SMTP id 34so6030351vxc.13 for ; Wed, 02 Mar 2011 06:27:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer; bh=neylw04T3TvMqdesrtJFgjfGMkSyY14CYh9HyaC9rPU=; b=S7evG1QjWSSqM0Ohk3kKfbj9BWOkAIL/rVUN0ifWP/aXRs0T3bdj5+ob8+Xtc7Qe7K g42yPodNCdORa7jQWh7HIrKqJQmx4E1lNIjKmivjOEwe0Y9r2aL3aQSe1EqAtjggEmQs 3KnpoPJJvfQrhmWWZdC7e315RGnWFVxGFOfhs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=hg1sz/udq5mpRGwUUsuaiKpMqQ7ZVIvm6g0Ylqy3N+aZ+kt+nsVJiN1Ybe8TWObPFt DFTsFYPQcgeSH1nrYTCeEH8hVVkTz7hiUZJeh1D2ZZNe9nXqmUQP1yYkCBFlGWCNyRrJ 0S3nCN/yNdUlumOe4sus1WYH7xLrhDFsy84WU= Received: by 10.52.157.161 with SMTP id wn1mr4448193vdb.160.1299075987442; Wed, 02 Mar 2011 06:26:27 -0800 (PST) Received: from lib1-lt.haystack.mit.edu (lib1-lt.haystack.mit.edu [192.52.61.207]) by mx.google.com with ESMTPS id g2sm154vbz.0.2011.03.02.06.26.26 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 02 Mar 2011 06:26:26 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: David In-Reply-To: <201102272143.p1RLhr0J027801@mail.r-bonomi.com> Date: Wed, 2 Mar 2011 09:26:26 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: References: <201102272143.p1RLhr0J027801@mail.r-bonomi.com> To: Robert Bonomi X-Mailer: Apple Mail (2.1082) Cc: freebsd-questions@freebsd.org, David Subject: Re: FreeBSD Performance X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2011 14:27:19 -0000 On Feb 27, 2011, at 4:43 PM, Robert Bonomi wrote: >> =46rom owner-freebsd-questions@freebsd.org Sun Feb 27 14:54:09 2011 >> From: David >> Date: Sun, 27 Feb 2011 15:46:03 -0500 >> To: freebsd-questions@freebsd.org >> Subject: FreeBSD Performance >>=20 >> Hello All: >>=20 >> I am curious... does anyone know of a reasonably priced commodity = server=20 >> capable of sourcing/sinking 10 Gbps of data from/to disk via 2 x 10 = GE=20 >> network interfaces? Any ideas on how hard this would be to do with=20 >> FreeBSD? >>=20 >> I know of a proprietary linux-based system, but looking for = open-source=20 >> FreeBSD based system. Thanks for the comments Robert... > A lot depends on what you need to do with the data. At the moment, I'm just looking to see if anyone has tried anything = similar. I have a detailed set of requirements/results, but wanted to keep things = simple initially. For now, let's just say there are two use cases: 1. Record 10 Gbps of data received from 2 10xGE cards onto hard disk = array 2. Playback 10 Gbps of data out over 2 10xGE cards onto network. > Do you need just the 'contents' of the network packets -- i.e. are = you > trying to send/recieve a single stream of data -- or do you need=20 > complete headers, augmented with timestamps, such that you can re- > construct/replay what was 'seen on the wire'? Just contents is fine. > Is the box 'dedicated' to receiving (or sending), and does = -nothing-else- > while that operation is in process? or do you need to sample the data = in > real-time as well? Dedicated. > Another question is _how_long_ you need to handle the 2x10gbit/sec of=20= > data. a few seconds? a few tens of seconds? minutes? hours? One hour (for now). > If you need to 'go to disk' in real-time, you're looking at needing > at least 3-4 gigabyte/sec of bandwith to disk. No commodity drives=20= > provide that kind of capacity, so you're looking at multiple drives=20= > 'in parallel' -- the logical equivalent of a 'striped' RAID array. =20= > Probably 12-16 spindles paralleled. Best handled with _hardware_=20 > raid, directly in the disk controller, but I don't know of a = commodity=20 Yes. > controller that supports enough spindles to give that bandwidth. > This means one is best off doing it in the application softwre = itself, > rather than trusting the O/S to get it right. Yes. > You're also looking at a _big_ disk array. Around 200 gigs for ONE=20= > MINUTE of data. Need 'only' an hour? That's merely 12 terabytes. Yes :) > The O/S is -relatively- unimportant. OK. As a recent convert to FreeBSD, I was hoping you would tell me=20 that the clean architecture and efficient implementation of FreeBSD = would solve all of my problems :) > You need _good_ network cards, with good drivers -- preferably ones = where > most of the network stack can be off-loaded onto the card itself. Yes. Something like TOE, batched interrupts, etc. > You also need good disk controllers, ideally semi-autonomous (like = SCSI), > with fairly large data buffers. Yes. OK. Thanks for the comments, that is helpful. I would be very interested to hear if anyone has had experience implementing a system like this (or close to it). I'm trying to decide whether I should try this = myself or proceed with the current linux-based system.