Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 11 Jan 2000 11:25:35 +1100
From:      Patryk Zadarnowski <patrykz@ilion.eu.org>
To:        "Scott Hess" <scott@avantgo.com>
Cc:        freebsd-hackers@FreeBSD.ORG, developer@lists.mysql.com
Subject:   Re: Concept check: iothreads addition to pthreads for MYSQL+FreeBSD. 
Message-ID:  <200001110025.LAA31411@mycenae.ilion.eu.org>
In-Reply-To: Your message of "Mon, 10 Jan 2000 15:20:38 -0800." <1a6101bf5bc1$4e364b20$1e80000a@avantgo.com> 

next in thread | previous in thread | raw e-mail | index | archive | help
> Recently I was tasked to find a way to scale up our MYSQL server, running
> MYSQL3.22.15 on FreeBSD3.3.  I've been testing a hardware RAID solution,
> and found that with 6 disks in a RAID5 configuration, the system was only
> perhaps 30% faster than when running on a single disk.  [The 6 disks in the
> RAID5 are the same model as the single-disk test I was comparing against.]
> 
> Experimentation determined that pthreads was the problem.  FreeBSD's
> implementation of pthreads using a select() loop, and select() always says
> that disk I/O is ready to proceed, and disk I/O never return EWOULDBLOCK.
> Essentially, pthreads was serializing the MYSQL read() requests, and if the
> dataset exceeds memory size, performance becomes entirely seek bound.
> 
> I've implemented a rough fix, which is to rfork() processes which I label
> "iothreads" to handle the disk I/O.  The parent process running pthreads
> has a socketpair() to each of the iothreads.  The iothreads wait for
> requests on the socketpair, and since socketpairs can block, pthreads can
> handle them efficiently.  This essentially allows me to turn blocking disk
> I/O calls into non-blocking calls.  Having multiple pending seeks turns out
> to be a huge win for MYSQL, allowing it to scale much better as disks are
> added to the RAID5 array.
> 
> Unfortunately, I'm concerned about using this code in production, because
> it needs a fair amount of cleanup to handle signals and administrative
> functions correctly.  For this reason and others, I'm starting a project to
> move it into the pthreads library itself.  Before I embark on that effort,
> I have a couple questions:
> 
> 1) Does this seem like a reasonable approach?  [It _works_, and well.  But
> it tastes strongly of hack.]
> 
> 2) Does anyone have suggestions for a solution that will be cleaner and
> won't take man-months to implement?  [Which is the redeeming quality of
> what I've got - it took me two days to zero in on a very workable
> solution.]
> 
> 3) Is anyone working on something I might leverage off of in this area?
> [For instance, a select()-based interface to async I/O?  Or support for
> blocking disk I/O in select() and read()?]
> 
> 4) Is there anyone willing to commit to testing my modified pthreads
> library against MYSQL?  [I'll be stress testing it quite heavily, of
> course.  It would probably also be testable against Squid with async I/O
> and multithreaded Apache 2.0.]

I'm no expert on pthreads, but, if you decide to proceed with
implementing a mixed user-land/rfork pthread implementation, may I
suggest that you implement is through POSIX pthread_attr_setscope()
interfaces instead of some local extension. pthread_attr_setscope()
sounds like a portable interface to precisely what you're trying to
achieve.

Pat.


To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200001110025.LAA31411>