Date: Sat, 28 Oct 2000 19:21:10 -0600 (CST) From: Ryan Thompson <ryan@sasknow.com> To: Mike <mike@fdhosting.com> Cc: freebsd-isp@FreeBSD.ORG Subject: Re: OT: Program Speed Message-ID: <Pine.BSF.4.21.0010281901210.44584-100000@ren.sasknow.com> In-Reply-To: <4.3.2.7.2.20001028164421.03021200@mail.futuredesigns.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Mike wrote to freebsd-isp@FreeBSD.ORG: > Sorry for the off topic question, but I'm not quite sure where else to find > unbiased opinions on this. We currently have a cgi program, written in C, > that is ran about 500,000+ times a day. Each time it is run, it reads from > reads and writes to 2-5 flat text files. What does it do? That's quite a lot of accesses for anything... You might do well to make it into an apache module for fast access. If your accesses were absolutely homogenous, your script will be run about 6 times per sec... But, as with most things, it probably runs in spurts. > I am in the process of rewriting this program so to add more features and > what not. I am wondering if it would be faster / more efficient if it were > to use MySQL instead of flat text files (over 500,000 accesses daily) in > the C version of the program, or would PHP or a combination of PHP/MySQL be > faster? Making and breaking SQL connections is VERY expensive in comparison to process creation and most (local) file IO. If you want to go this route, you MUST use something that maintains persistent connections. Depending on the size of the data, SQL is very efficient, and, providing you can optimize your requests, is faster than flat-file implementations. PHP will assist with persistent connections. SQL isn't necessarily faster than everything else... You might be able to write an optimized hash key algorithm tailored to the data, with a speedy file format and memory cache, that will be faster than a very good SQL implementation. SQL is standardized, highly portable, and easy, though. mySQL also allows you to write custom functions, which will gain you most of the advantages of your own implementation. Another thing to consider is: is the result of this script important? I.e., must it be run "live"? Writing a backend daemon to cache the requests and write them out asynchronously every few seconds, or per hundred entries, for example, would probably increase overall efficiency greatly. Even if the user needs to see the results, maybe you can: a) return the result immediately, without querying the files, and still cache the update query. This depends on your application--you may be able to do this. b) return asynchronously... i.e., build in some lag to the responses of a second or so, so that your system can run much more efficiently, at the cost of a slight delay to the user. You could use a pause() or sleep() in the client script, and have the backend daemon send a signal via kill(2) when the data is available. Taking that further, maybe you could use sockets to pass data back and fourth for maximum efficiency. A great deal depends on the purpose of the application. Another thought: How large is the data? Can it be stored in memory? Is persistent storage necessary? If you have a stable server, and the data is small enough to easily fit in RAM, you can usually improve performance by an order of magnitude or so (even _with_ decent caching) by storing everything in memory and flushing once every 30 seconds or so... Or longer, if you can risk losing that much data. Don't fill your main memory needlessly, though, and force other critical applications to swap out... That could surely bring your system to its knees ;-) > Any thoughts/ideas, or a reference as to where I could find such > information would be greatly appreciated. > > Thanks in advance! > > Mike Hope this helps, - Ryan -- Ryan Thompson <ryan@sasknow.com> Network Administrator, Accounts Phone: +1 (306) 664-1161 SaskNow Technologies http://www.sasknow.com #106-380 3120 8th St E Saskatoon, SK S7H 0W2 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.21.0010281901210.44584-100000>