Date: Tue, 13 Mar 2012 00:22:23 +0100 From: Phil Regnauld <regnauld@x0.dk> To: Mikolaj Golub <to.my.trociny@gmail.com> Cc: freebsd-stable@freebsd.org Subject: Re: Issue with hast replication Message-ID: <20120312232223.GG12975@macbook.bluepipe.net> In-Reply-To: <86k42pu0tb.fsf@kopusha.home.net> References: <20120311185457.GB1684@macbook.bluepipe.net> <861uoyvpzh.fsf@kopusha.home.net> <20120311220911.GD1684@macbook.bluepipe.net> <20120312143127.GM12975@macbook.bluepipe.net> <86k42pu0tb.fsf@kopusha.home.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Mikolaj Golub (to.my.trociny) writes:
>
> It looks like in the case of hastd this was send(2) who returned ENOMEM, but
> it would be good to check. Could you please start synchronization again,
> ktrace primary worker process when ENOMEM errors are observed and show output
> here?
Ok, took a little while, as running ktrace on the hastd does slow it down
significantly, and the error normally occurs at 30-90 sec intervals.
0x0f90 b2f3 3ad5 e657 7f0f 3e50 698f 5deb 12af |..:..W..>Pi.]...|
0x0fa0 740d c343 6e80 75f3 e1a7 bfdf a4c1 f6a6 |t..Cn.u.........|
0x0fb0 ea85 655d e423 bd5e 42f7 7e9a 05d2 363a |..e].#.^B.~...6:|
0x0fc0 025e a7b5 0956 417c f31c a6eb 2cd9 d073 |.^...VA|....,..s|
0x0fd0 2589 e8c0 d76a 889f 8345 eeaf f2a0 c2d6 |%....j...E......|
0x0fe0 b89e aaef fee2 6593 e515 7271 88aa cf66 |......e...rq...f|
0x0ff0 d272 411a 7289 d6c9 6643 bdbe 3c8c 8ae8 |.rA.r...fC..<...|
50959 hastd RET sendto 32768/0x8000
50959 hastd CALL sendto(0x6,0x8024bf000,0x8000,0x20000<MSG_NOSIGNAL>,0,0)
50959 hastd RET sendto -1 errno 12 Cannot allocate memory
50959 hastd CALL clock_gettime(0xd,0x7fffff3f86f0)
50959 hastd RET clock_gettime 0
50959 hastd CALL getpid
50959 hastd RET getpid 50959/0xc70f
50959 hastd CALL sendto(0x3,0x7fffff3f8780,0x84,0,0,0)
50959 hastd GIO fd 3 wrote 132 bytes
"<27>Mar 12 23:42:43 hastd[50959]: [hvol] (primary) Unable to sen\
d request (Cannot allocate memory): WRITE(8626634752, 131072)."
50959 hastd RET sendto 132/0x84
50959 hastd CALL close(0x7)
50959 hastd RET close 0
> If it is send(2) who fails then monitoring netstat and network driver
> statistics might be helpful. Something like
>
> netstat -nax
> netstat -naT
> netstat -m
> netstat -nid
I could run this in a loop, but that would be a lot of data, and might
not be appropriate to paste here.
I didn't see any obvious errors, but I'm not sure what I'm looking for.
netstat -m didn't show anything close to running out of buffers or
clusters...
> sysctl -a dev.<nic>
>
> And may be
>
> vmstat -m
> vmstat -z
No obvious errors there either, but again what should I look out for ?
In the meantime, I've also experimented with a few different scenarios, and
I'm quite puzzled.
For instance, I configured one of the other gigabit cards on each host to
provide a dedicated replication network. The main difference is that up
until now this has been running using tagged vlans. To be on the safe side,
I decided to use an untagged interface (the second gigabit adapter in each
machine).
Here's where I observed, and it is very odd:
- doing a dd ... | ssh dd fails in the same fashion as before
- I created a second zvol + hast resource of just 1 GB, and it replicated
without any problems, peaking at 75 MB / sec (!) - maybe 1GB is too small
?
(side note: hastd doesn't pick up configuration changes even with SIGHUP,
which makes it hard to provision new resources on the fly)
- I restarted replication on the 100 G hast resource, and it's currently
replicating without any problems over the second ethernet, but it's
dragging along at 9-10 MB/sec, peaking at 29 MB/sec occasionally.
Earlier, I was observing peaks at 65-70 MB sec in between failures...
So I don't really know what to conclude :-|
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120312232223.GG12975>