From owner-freebsd-hackers  Thu May 18 21:15:39 1995
Return-Path: hackers-owner
Received: (from majordom@localhost)
          by freefall.cdrom.com (8.6.10/8.6.6) id VAA11544
          for hackers-outgoing; Thu, 18 May 1995 21:15:39 -0700
Received: from haywire.DIALix.COM (peter@haywire.DIALix.COM [192.203.228.65])
          by freefall.cdrom.com (8.6.10/8.6.6) with ESMTP id VAA11535
          for <hackers@freebsd.org>; Thu, 18 May 1995 21:15:32 -0700
Received: (from peter@localhost) by haywire.DIALix.COM (8.6.12/8.6.12/DIALix) id MAA09745; Fri, 19 May 1995 12:15:21 +0800
Date: Fri, 19 May 1995 12:15:18 +0800 (WST)
From: Peter Wemm <peter@haywire.dialix.com>
To: hackers@FreeBSD.org
Subject: More on "Hmm.. Strange..."
Message-ID: <Pine.SV4.3.91.950519105053.23362C-100000@haywire.DIALix.COM>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: hackers-owner@FreeBSD.org
Precedence: bulk

Well. Something really wierd is going on.  I think it's a bug, and 
potentially serious, as it makes gated practically useless..

There is some interaction with SO_REUSEADDR.. I have not figured out why 
yet, but take the following program (chopped down rwhod.c) which 
deterministically demonstrates the problem.

-----------
#include <sys/param.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/signal.h>
#include <sys/ioctl.h>
#include <sys/sysctl.h>

#include <net/if.h>
#include <net/if_dl.h>
#include <net/route.h>
#include <netinet/in.h>
#include <protocols/rwhod.h>

#include <ctype.h>
#include <errno.h>
#include <fcntl.h>
#include <netdb.h>
#include <paths.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <syslog.h>
#include <unistd.h>
#include <utmp.h>

int
main(argc, argv)
	int argc;
	char *argv[];
{
	int on = 1;
	char *cp;
	struct sockaddr_in sin;
	struct servent *sp;
	struct hostent *hp;
	int s;

	sp = getservbyname("router", "udp");
	if (sp == NULL) {
		fprintf(stderr, "rwhod: man/who: unknown service\n");
		exit(1);
	}

	if ((s = socket(AF_INET, SOCK_DGRAM, 0)) < 0) {
		syslog(LOG_ERR, "socket: %m");
		exit(1);
	}

	if (setsockopt(s, SOL_SOCKET, SO_BROADCAST, &on, sizeof(on)) < 0) {
		syslog(LOG_ERR, "setsockopt SO_BROADCAST: %m");
		exit(1);
	}

	memset(&sin, 0, sizeof(sin));

	sin.sin_family = AF_INET;
	sin.sin_port = sp->s_port;
#if 0
	if (bind(s, (struct sockaddr *)&sin, sizeof(sin)) < 0) {
		syslog(LOG_ERR, "bind: %m");
		exit(1);
	}
#endif

	hp = gethostbyname(argv[1]);
	if (hp == NULL) {
		fprintf(stderr, "rwhod: %s unknown host\n", argv[1]);
		exit(1);
	}
	sin.sin_addr.s_addr = *((long *)hp->h_addr);

	(void)sendto(s, "hello", 5, 0, (struct sockaddr *)&sin, sizeof(sin));
	(void)sendto(s, "hellothere", 10, SO_REUSEADDR, (struct sockaddr *)&sin, 
sizeof(sin));

}

The first sendto() always works, and the second sendto() sends the 
datagram to the *wrong interface*!

Just a (yet another) reminder of the config:

ed0: flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	inet 192.203.228.69 netmask 0xfffffff0 broadcast 192.203.228.79
ppp0: flags=151<UP,POINTOPOINT,RUNNING,PROMISC> mtu 1500
	inet 192.203.228.69 --> 192.203.228.3 netmask 0xfffffff0 

In this case, in the above program, the first sendto() sends a 5 byte 
broadcast to 192.203.228.79 on the ethernet (correct!). The second 
sendto(), the 10 byte datagram gets sent to 192.203.228.79 on the PPP 
interface!!!!!  (of which the remote sends it straight back!  after a 
game of ping-pong, an icmp timer exceeded message is sent).

I recompiled gated to not turn on SO_REUSEADDR at all, but there's some 
kernel glue that appears to turn it on inside the kernel if the socket is 
implicated in any multicasting (as gated does with OSPF and RIP-II)..  

I'm not awake enough to interpret the code to understand what's going 
on.. But I thought I'd mention it in case somebody else might recognised 
it, or knew what was going wrong.

Any suggestions what to try next?

-Peter