From owner-freebsd-questions@FreeBSD.ORG Sun Sep 28 12:30:10 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 274B0106568F for ; Sun, 28 Sep 2008 12:30:10 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from smtp.infracaninophile.co.uk (gate6.infracaninophile.co.uk [IPv6:2001:8b0:151:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 8F6A08FC15 for ; Sun, 28 Sep 2008 12:30:09 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from happy-idiot-talk.infracaninophile.co.uk (localhost [IPv6:::1]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.14.3/8.14.3) with ESMTP id m8SCU2d6076541; Sun, 28 Sep 2008 13:30:03 +0100 (BST) (envelope-from m.seaman@infracaninophile.co.uk) X-DKIM: Sendmail DKIM Filter v2.7.2 smtp.infracaninophile.co.uk m8SCU2d6076541 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=infracaninophile.co.uk; s=200708; t=1222605003; bh=qXWIFIlaPKyC6+ IoUdcajS2yUH80EZlYvRR5u/tf8a8=; h=Message-ID:Date:From:MIME-Version: To:CC:Subject:References:In-Reply-To:Content-Type:Cc:Content-Type: Date:From:In-Reply-To:Message-ID:Mime-Version:References:To; z=Mes sage-ID:=20<48DF78C3.9080401@infracaninophile.co.uk>|Date:=20Sun,=2 028=20Sep=202008=2013:29:55=20+0100|From:=20Matthew=20Seaman=20|Organization:=20Infracaninophile|User -Agent:=20Thunderbird=202.0.0.16=20(X11/20080726)|MIME-Version:=201 .0|To:=20Danny=20Do=20|CC:=20freebsd-questions@fr eebsd.org|Subject:=20Re:=20Hard=20disk=20bottle=20neck.|References: =20<00d101c92154$691f53c0$3b5dfb40$@com>|In-Reply-To:=20<00d101c921 54$691f53c0$3b5dfb40$@com>|X-Enigmail-Version:=200.95.6|Content-Typ e:=20multipart/signed=3B=20micalg=3Dpgp-sha256=3B=0D=0A=20protocol= 3D"application/pgp-signature"=3B=0D=0A=20boundary=3D"------------en igECF6016D7D5E6D5C89AE0FDE"; b=qvvHYiB+x9Wh1qHpN3SYQPa+y59k3qa9PPoL ZPsmWYA4JT3ZwpVhEX8+imkehIerwNhmVjP2mC05bjrxyFcOA94CP6SygX0cL3pUUbd E/vcGGdtpiCiSeFieDawp3n4H3xGbqxU9/EKjOyBObcdWXMzePp7MmV8lWSIVVP2aeQ Q= Message-ID: <48DF78C3.9080401@infracaninophile.co.uk> Date: Sun, 28 Sep 2008 13:29:55 +0100 From: Matthew Seaman Organization: Infracaninophile User-Agent: Thunderbird 2.0.0.16 (X11/20080726) MIME-Version: 1.0 To: Danny Do References: <00d101c92154$691f53c0$3b5dfb40$@com> In-Reply-To: <00d101c92154$691f53c0$3b5dfb40$@com> X-Enigmail-Version: 0.95.6 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enigECF6016D7D5E6D5C89AE0FDE" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0 (smtp.infracaninophile.co.uk [IPv6:::1]); Sun, 28 Sep 2008 13:30:03 +0100 (BST) X-Virus-Scanned: ClamAV 0.94/8347/Sun Sep 28 00:35:46 2008 on happy-idiot-talk.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VERIFIED,NO_RELAYS autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on happy-idiot-talk.infracaninophile.co.uk Cc: freebsd-questions@freebsd.org Subject: Re: Hard disk bottle neck. X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Sep 2008 12:30:10 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigECF6016D7D5E6D5C89AE0FDE Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Danny Do wrote: > Hi guys, >=20 > =20 >=20 > I have this problem for years but couldn't find a way to solve it. >=20 > I have a file server handling large files from 1MByte to 1GByte.=20 >=20 > Server Info: > FreeBSD 6.2=20 > Apache 2.2.9 >=20 > DELL PowerEdge 1850 > 2GB RAM (only 184MB is active) > 6x300MB SCSI 10K RPM RAID5 > Gigabit Ethernet Connection >=20 > My server can output NO MORE than 60Mbps (read only).=20 >=20 > The bottle neck is the hard disk. If I use ONE connection to download f= ile > from my server, the speed can go up to about 400Mbps.=20 >=20 > If I let visitors download using multiple connections, the server canno= t > output more than 60Mbps.=20 >=20 > My service is similar to rapidshare/megaupload, I am wondering how they= > configure their servers? >=20 > If I recall correctly, it doesn't cost much time to read the data from = the > disk but it does cost a lot of time to seek for the data. Correct me if= I am > wrong, if I increase the read buffer size, there would be less disk see= k > (disk access). Let's say the read buffer is 64K, if I increase it to 64= 0K, > the disk seek would reduce by 90%. Thus, more data can be read from the= hard > drive. >=20 > What should I do now? Try some different webservers. Apache is great, but it is designed to be maximally flexible and capable of doing anything you can imagine rather than to be absolutely as fast as possible. There are some light-weight servers which have put work into optimizing delivery of static content -- usually spoken of in the context of serving= =20 images but any static files will be suitable material. Personally, I=20 really like nginx for this. Lots of people go for lighttpd and there are= a number of other alternatives in ports. Also, depending on exactly how much content you have to serve and whether= certain items are very much more popular than others, a reverse proxy / m= emory cache (a.k.a http accelerator) may help. varnish is the obvious candidate here, but you'll have to experiment a bit to see what the optim= al settings are and if it actually helps at all. If your website runs using a scripting language such as PHP, then another= possibility is memcached -- although described as a cache for dynamically= generated pages, it can cache just about anything, but you will need some= sort of scripting language to interface to it from your web server. Ther= e are memcached APIs for all popular languages and probably a few you've=20 never heard of... The various caching strategies basically work because they keep recently accessed files in RAM, avoiding an expensive round-trip to the HDD to retrieve the data (memory access takes nano- or micro- seconds: disk=20 accesses take milliseconds). Of course, the OS itself also does exactly = the same thing in a general way, and FreeBSD is already very good in this= =20 respect. Caching software however gives you more control over what gets= =20 cached and for how long, enabling you to tune this specific application = for maximum performance. Cheers, Matthew --=20 Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW --------------enigECF6016D7D5E6D5C89AE0FDE Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEAREIAAYFAkjfeMoACgkQ8Mjk52CukIyrswCfWjuor/UMlEofLjPCMnEkzxqQ N1IAoIIw0Y4Ay48yuXTnwrmvpnXcZ+IE =Ba6b -----END PGP SIGNATURE----- --------------enigECF6016D7D5E6D5C89AE0FDE--