Date: Thu, 13 Oct 2005 12:26:32 +0930 From: Greg 'groggy' Lehey <grog@MySQL.com> To: NMH <drumslayer2@yahoo.com>, Kris Kennaway <kris@FreeBSD.org>, justin.bastedo@gmail.com Cc: FreeBSD-DataBase <freebsd-database@freebsd.org>, questions <freebsd-questions@freebsd.org> Subject: Re: Mysql server not able to stay running on anything but Linux? Message-ID: <20051013025632.GN49168@wantadilla.lemis.com> In-Reply-To: <20051012225544.83673.qmail@web32901.mail.mud.yahoo.com> <20051012214154.17245.qmail@web32913.mail.mud.yahoo.com> <20051012210757.GA97184@xor.obsecurity.org> <20051012201622.49704.qmail@web32905.mail.mud.yahoo.com> References: <20051012225544.83673.qmail@web32901.mail.mud.yahoo.com> <20051012210757.GA97184@xor.obsecurity.org> <20051012214154.17245.qmail@web32913.mail.mud.yahoo.com> <20051012201622.49704.qmail@web32905.mail.mud.yahoo.com> <20051012210757.GA97184@xor.obsecurity.org> <20051012201622.49704.qmail@web32905.mail.mud.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--GTjM2GI4zMe3xHZ5
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Note the From: address.
On Wednesday, 12 October 2005 at 13:16:22 -0700, NMH wrote:
>
> I am stuck with a delema and I feel like a damn troll. But.. I have
> a Mysql Database that I posted about earlier. It seems that it is
> only able to not die by running on BSD 4.11 with Linux Threads. My
> boss is convinced this means that Linux is better for MySQL and
> wants that installed now.
>
> We even got a support contact from Mysql that so far has gotten us
> nothing for almost a month while our production database server died
> up to 3 times a day. (and lots of we're looking into it's)
One of the reasons why you haven't got much more than "we're looking
into it"s is because we haven't been able to reproduce the problem;
you acknowledge this in follow-up mail quoted below.
As you know from various threads on the FreeBSD lists, including this
one, the typical answer is "works fine for me". That doesn't mean
that we're not taking your problems seriously, but we do have a
significant issue just reproducing the problem. We have a number of
choices:
1. Try different hardware or a different version of FreeBSD. It's
conceivable that there's something about your specific hardware,
or about the combination of i386 kernel on amd64 in general, that
triggers the problem.
2. Do debugging on your production servers. This isn't really a
choice at all: it would involve even more down time.
3. Get you to run a more stable version of FreeBSD while we
investigate the problem. This is the method we chose. I haven't
heard from you since the weekend, so I hope I'm correct in
understanding that you currently don't have stability problems.
On our side, we have installed FreeBSD 5.4 on one of our internal
machines, and we're trying to reproduce the problem there.
> We were running fine but a little slow on FreeBSD 5.1-P11 and MySQL
> 4.0.18.(apperantly before a big Lib change) We had to move quicker
> than we wanted to a new server running FreeBSD 5.4 and MySQL 4.11
> (becouse of a dual HD death) Under production load the new 5.4
> server fell over regulary. It has only now become stable by wiping
> it and running it on FreeBSD 4.11 with Linux Threads. (it regularly
> has over 400+ threads)
Kris obviously understood that by this statement you meant a kernel
crash. My understanding is that only the mysqld server is crashing.
Is this still correct?
> I want to try FreeBSD 5.4 AMD64 (the machines are Opteron) or 6.0
> but my boss feels that would be a waste given that MYSQL doesn't
> support Mysql on AMD64 well enough.
I think it would be a good idea to try this. It's one of the things
that we intend to do in-house as soon as we can reproduce the problem
at all.
> Can anyone help or offer assistance to help track this down? Perhaps
> also any annecdotes or examples I can show my boss that other people
> have as busy MYSQL databases on BSD 5.X. We paid 3K to Mysql for
> help and so far they have been unable to offer any clues as to why
> ours will not stay stable on anything but Linux threads.
Have you had any kind of crash under 4.x? I don't think that the
issue is so much linuxthreads as 5.x.
> As I say I only manage the server, I don't program the databases. Is
> there anything I should/could look for database wise that could
> trigger such things?
So far we've had the machine up in-house and have not reproduced the
problem. If you have a spare machine that we could run under more
typical conditions on your premises, this might help.
On Wednesday, 12 October 2005 at 17:07:57 -0400, Kris Kennaway wrote:
>
> Unfortunately you'll need to provide details of how it "fell over"
> (e.g. panic messages + backtraces).
As I mention above, I think this is only a server crash. I mentioned
this on the list a couple of weeks ago: all the backtraces I have seen
have been a SIGSEGV out of mutex_unlock_common.
>> Can anyone help or offer assistance to help track this down?
>> Perhaps also any annecdotes or examples I can show my boss that
>> other people have as busy MYSQL databases on BSD 5.X. We paid 3K to
>> Mysql for help and so far they have been unable to offer any clues
>> as to why ours will not stay stable on anything but Linux
>> threads. I feel really sad that so far no one has responded to my
>> posts and it feels like a victory for linux.
>
> If I was your boss I'd be asking why mysql hasn't delivered on their
> support contract.
Indeed. I think we have, though. There's a certain class of bugs
which are almost impossible to fix because they're so hard to chase
down. This is one of them.
On Wednesday, 12 October 2005 at 14:41:54 -0700, NMH wrote:
> --- Kris Kennaway <kris@obsecurity.org> wrote:
>
>> If I was your boss I'd be asking why mysql hasn't delivered on
>> their support contract.
>
> Well I think support has many meanings. I decided to look at what he
> paid for and it says:
> We get "access" to the mysql devlopers...
> We get "access" to certified binaries. (none of which
> are FreeBSD)
>
> So, sad as it may seem, if your running on FreeBSD, $3000.00 buys
> you someone to talk to. It doesn't mean they have to say anything
> meaningful back or within any reasonable time. :(
We try to handle all problems within a reasonable time. The fact that
you're running FreeBSD does mean that you don't get certified
binaries, but that's the only drawback. And the fact that the time
for this problem has been unreasonable has nothing to do with the fact
you're running FreeBSD: it's because it's a bugger to track down.
> Also it doesn't cover tuning or code review etc. So if our problem
> is caused by anything like that.. they don't even have to tell us I
> would imagine.
FWIW, my personal opinion (I don't speak for MySQL, but I don't think
the official line would be much different) is that a database server
shouldn't crash. If it does, it's a bug. The question we need to
clarify is not whether your code is a problem, but whether the crash
is due to FreeBSD or MySQL. Currently the evidence is pointing to
FreeBSD, but we're keeping an open mind on that.
> So.. that is why as always, these lists and fellow FreeBSD people
> are their own best support.
Apparently we've given you the impression that we're not doing
anything. I'm sorry about that, and it's certainly not the case. So
far all these lists have done is to provide the valuable information
that this is not a general problem.
On Wednesday, 12 October 2005 at 15:55:44 -0700, NMH wrote:
> --- Justin Bastedo <justin.bastedo@gmail.com> wrote:
>
>
> Also no one from Mysql seemed to think going to STABLE would do
> anything. It's a very very tough thing to play lets see what
> happens if.. On production servers. The time it takes us just to
> bounce from one machine to another is pretty decent. Let alone
> downtime etc.
This is puzzling. The bug report is visible to the world at
http://bugs.mysql.com/bug.php?id=12251 : it was entered against
5.4-STABLE. I did express the opinion, based on code analysis, that
upgrading wouldn't make much difference.
>> I'm definately interested to see what the final solution is on this
>> matter. I can easily forsee our db load getting close to that level
>> within the next couple of months depending on how these contracts
>> play out.
Whether or not you have a contract, I'd be very interested in hearing
of any problems you have with high load. Obviously you'll get better
support if you do have a contract, Nicole's impressions
notwithstanding.
> Well, I will post whatever happens. As I said the most annoying
> part to all this is that so far we have been unable to crash the
> 5.4 servers via any testing means we (and Mysql) could come up
> with. Yet in production they fell over as soon as the load started
> to climb and points in between. But never on the slope of
> decreasing load for the day. (always felt that was odd) IE it would
> die on the bell curve increases but never at the coresponding
> downward cycle of the daily busy curve.
I noticed that too; I suspect it's not statistically relevant.
> If anyone knows of any particulary evil testing programs I would
> love to know. It's tough to simulate 30+ servers asking lots
> questions and forcing tons of writes.
If you (anyone) do have useful input, please keep me in the loop too.
Greg
--
Greg Lehey, Senior Software Engineer
MySQL AB, http://www.mysql.com/
Echunga, South Australia
See complete headers for address and phone numbers.
Are you MySQL certified? http://www.mysql.com/certification/
--GTjM2GI4zMe3xHZ5
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (FreeBSD)
iD8DBQFDTczgIubykFB6QiMRAs+yAJ9YsDJE1iooFZzplYwcmm9KFJC7JwCcCtMy
FfcmSvl0PU3T9UD0EWgm5mY=
=PmgN
-----END PGP SIGNATURE-----
--GTjM2GI4zMe3xHZ5--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20051013025632.GN49168>