Date: Fri, 29 Jun 2007 10:48:24 -0700 From: Steve Kargl <sgk@troutmask.apl.washington.edu> To: David Malone <dwmalone@maths.tcd.ie> Cc: freebsd-current@freebsd.org Subject: Re: SYNCOOKIE authentication problems Message-ID: <20070629174824.GA6989@troutmask.apl.washington.edu> In-Reply-To: <20070629105140.GA51586@walton.maths.tcd.ie> References: <20070628014311.GA50012@troutmask.apl.washington.edu> <20070629105140.GA51586@walton.maths.tcd.ie>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Jun 29, 2007 at 11:51:40AM +0100, David Malone wrote:
> On Wed, Jun 27, 2007 at 06:43:11PM -0700, Steve Kargl wrote:
> > Any advice on how to isolate or avoid?
> >
> > Jun 27 18:31:19 node11 kernel: TCP: [192.168.0.11]:59661 to
> > [192.168.0.11]:63266 tcpflags 0x10<ACK>; syncache_expand: Segment failed
> > SYNCOOKIE authentication, segment rejected (probably spoofed)
>
> It looks like you tried to open a TCP connection to yourself, but
> the connection failed. You could try leaving a tcpdump running:
>
> tcpdump -i whatever_interface -w /tmp/synfinrstdata -s 1500 'tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) != 0'
>
> while your MPI app runs and then we can have a look at the packets
> that caused the problem. The above should collect all TCP SYN, FIN
> and RST packets, which would probably be enough to diagnose the
> problem.
>
Another tidbit, once the MPI app started to trash, I ran truss
on rank=0 process. I have a very file containing a
sigprocmask(SIG_UNBLOCK,SIGCHLD,0x0) = 0 (0x0)
poll({4/POLLIN 5/POLLIN 6/POLLIN 7/POLLIN 9/POLLIN 10/POLLIN 11/POLLIN 13/POLLIN 8/POLLIN 12/POLLIN 14/POLLIN 15/POLLIN 16/POLLIN 17/POLLIN 18/POLLIN 19/POLLIN 20/POLLIN 21/POLLIN 22/POLLIN 23/POLLIN 24/POLLIN 25/POLLIN 26/POLLIN 27/POLLIN 28/POLLIN 29/POLLIN 30/POLLIN},27,0) = 0 (0x0)
sigprocmask(SIG_BLOCK,SIGCHLD,0x0) = 0 (0x0)
sigaction(SIGCHLD,{ 0x3c0d2c850 SA_RESTART ss_t },0x0) = 0 (0x0)
gettimeofday({1183138884.532826},0x0) = 0 (0x0)
sched_yield(0x3c1d44180,0x3c0b39ec0,0x0,0x3c0b39ec0,0x3c1d44280) = 0 (0x0)
sigprocmask(SIG_BLOCK,SIGCHLD,0x0) = 0 (0x0)
sigaction(SIGCHLD,{ 0x3c0d2c850 SA_RESTART ss_t },0x0) = 0 (0x0)
gettimeofday({1183138884.535137},0x0) = 0 (0x0)
sigprocmask(SIG_UNBLOCK,SIGCHLD,0x0) = 0 (0x0)
poll({4/POLLIN 5/POLLIN 6/POLLIN 7/POLLIN 9/POLLIN 10/POLLIN 11/POLLIN 13/POLLIN 8/POLLIN 12/POLLIN 14/POLLIN 15/POLLIN 16/POLLIN 17/POLLIN 18/POLLIN 19/POLLIN 20/POLLIN 21/POLLIN 22/POLLIN 23/POLLIN 24/POLLIN 25/POLLIN 26/POLLIN 27/POLLIN 28/POLLIN 29/POLLIN 30/POLLIN},27,0) = 0 (0x0)
sigprocmask(SIG_BLOCK,SIGCHLD,0x0) = 0 (0x0)
sigaction(SIGCHLD,{ 0x3c0d2c850 SA_RESTART ss_t },0x0) = 0 (0x0)
gettimeofday({1183138884.538484},0x0) = 0 (0x0)
ad nausem
I'm using the 4BSD scheduler.
--
Steve
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070629174824.GA6989>