From owner-freebsd-net  Mon Apr 30 16: 2:46 2001
Delivered-To: freebsd-net@freebsd.org
Received: from rgmail.regenstrief.org (rgmail.regenstrief.org [134.68.31.197])
	by hub.freebsd.org (Postfix) with ESMTP id 2100C37B422
	for <freebsd-net@freebsd.org>; Mon, 30 Apr 2001 16:02:35 -0700 (PDT)
	(envelope-from gunther@aurora.regenstrief.org)
Received: from aurora.regenstrief.org (rgnout.regenstrief.org [134.68.31.38])
	by rgmail.regenstrief.org (8.11.0/8.8.7) with ESMTP id f3UN7OX18068;
	Mon, 30 Apr 2001 18:07:24 -0500
Message-ID: <3AEDEEFA.60DD4AC4@aurora.regenstrief.org>
Date: Mon, 30 Apr 2001 23:02:18 +0000
From: Gunther Schadow <gunther@aurora.regenstrief.org>
Organization: Regenstrief Institute for Health Care
X-Mailer: Mozilla 4.75 [en] (Win98; U)
X-Accept-Language: en
MIME-Version: 1.0
To: snap-users@kame.net
Cc: Shoichi Sakane <sakane@ydc.co.jp>, freebsd-net@freebsd.org,
	itojun@iijlab.net
Subject: Re: (KAME-snap 4523) Re: KAME SPD bug, please try and confirm ...
References: <20010424040539N.sakane@ydc.co.jp> <20010424041925C.sakane@ydc.co.jp> <3AE4BB51.FC1400BD@aurora.regenstrief.org>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Sender: owner-freebsd-net@FreeBSD.ORG
Precedence: bulk
X-Loop: FreeBSD.org

Earlier last week I wrote:
> I just built and tested the latest KAME-SNAP, and it appears as if
> the two ipsec tunnels work together now. I will have a final word
> on this later tomorrow, but for now it looks as if this problem
> requires no further action on your part.

Unfortunately I found out that the problem still exists deep down,
it's just harder to reproduce. It comes when I try to use multiple
SPD rules to route packets into the same ESP tunnel. 

The following setkey here script shows an exaple. This is for a 
VPN with a central headquarter (home) and multiple remote sites 
(sattelites 1, 2, 3, ...). We use shell variables to make this
both easily reproduceable and more readable:

${homeip} - the outside IP address of the central VPN gateway, e.g.,
homeip=204.152.64.103

${sat1ip} - the outside IP address of satellite-1's VPN gateway, e.g,
sat1ip=24.16.273.28

${sat2ip} - the outside IP address of sattelite-2's VPN gateway, e.g.,
sat2ip=65.34.283.122

etc.

The VPN establishes one common class B network (typically using local IP 
numbers, e.g., 192.168.0.0/26). We use the shell variable ${vpn} to make
this more readable. For example, we could set 

vpn=192.168

The problem with the KAME prolicy processing becomes manifest when
we add another realistic twist, i.e., in adittion to the VPN network
we want other routes using the VPN tunnel. For example, some folks
in our company use the 10.0.0.0/8 network at the headquarter, and
we have an affiliated organization with network 123.4.5.0/24 which
we want to contact only via the headquarter using NAT such that 
the affiliated network would not need to know about the structure of
our VPN.

We configure the central home VPN gateway using the following setkey
here script:


setkey -c <<EOF

# VPN TUNNEL FROM HOME TO SATELLITE 2

# SAD ENTRIES
  add ${homeip} ${sat2ip} esp ${sat2spi}            -E simple;
  add ${sat2ip} ${homeip} esp $(( ${sat2spi} + 1 )) -E simple;

# PRIMARY TUNNEL INGRESS AND EGRESS FOR VPN
  spdadd ${vpn}.0.0/16 ${vpn}.2.0/24 -P in  ipsec
    esp/tunnel/${homeip}-${sat1ip}/require;
  spdadd ${vpn}.2.0/24 ${vpn}.0.0/16 -P out ipsec
    esp/tunnel/${sat2ip}-${homeip}/require;

# ADDITIONAL INGRESS AND EGRESS RULES ...

# ... FOR THE OTHER LOCAL NETWORK 10.0.0.0/8
  spdadd 10.0.0.0/8    ${vpn}.2.0/24 -P in  ipsec
    esp/tunnel/${homeip}-${sat2ip}/require;
  spdadd ${vpn}.2.0/24 10.0.0.0/8    -P out ipsec
    esp/tunnel/${sat2ip}-${homeip}/require;

# ... FOR SOME GLOBAL IP NETWORK THAT SHOULD BE ROUTED THROUGH TUNNEL
  spdadd 123.4.5.0/24  ${vpn}.2.0/24 -P in  ipsec
    esp/tunnel/${homeip}-${sat2ip}/require;
  spdadd ${vpn}.2.0/24 123.4.5.0/24  -P out ipsec
    esp/tunnel/${sat2ip}-${homeip}/require;

# END OF SATTELITE 2

# VPN TUNNEL FROM HOME TO SATELLITE 3

# SAD ENTRIES
  add ${homeip} ${sat3ip} esp ${sat3spi}            -E simple;
  add ${sat3ip} ${homeip} esp $(( ${sat3spi} + 1 )) -E simple;

# PRIMARY TUNNEL INGRESS AND EGRESS FOR VPN
  spdadd ${vpn}.0.0/16 ${vpn}.3.0/24 -P in  ipsec
    esp/tunnel/${homeip}-${sat3ip}/require;
  spdadd ${vpn}.3.0/24 ${vpn}.0.0/16 -P out ipsec
    esp/tunnel/${sat3ip}-${homeip}/require;

# ADDITIONAL INGRESS AND EGRESS RULES ...

# ... FOR THE OTHER LOCAL NETWORK 10.0.0.0/8
  spdadd 10.0.0.0/8    ${vpn}.3.0/24 -P in  ipsec
    esp/tunnel/${homeip}-${sat3ip}/require;
  spdadd ${vpn}.3.0/24 10.0.0.0/8    -P out ipsec
    esp/tunnel/${sat3ip}-${homeip}/require;

# ... FOR SOME GLOBAL IP NETWORK THAT SHOULD BE ROUTED THROUGH TUNNEL
  spdadd 123.4.5.0/24  ${vpn}.3.0/24 -P in  ipsec
    esp/tunnel/${homeip}-${sat3ip}/require;
  spdadd ${vpn}.3.0/24 123.4.5.0/24  -P out ipsec
    esp/tunnel/${sat3ip}-${homeip}/require;

# END OF SATTELITE 3

EOF
 
Each sattelite is configured using the following script, where we
have shell variables defined similarly to the above. With the
following differences: instead of sat1.., sat2.., sat3.. we simply
say sat.., i.e.:

satip  - the outside ip address of this sattelite
satspi - the spi (base) number for this tunnel

satno  - the ordinal number of the sattelite, e.g., 1 for sat1, 2
         for sat2, 3 for sat3, etc.

Here goes the script:

setkey -c <<EOF

# VPN TUNNEL FROM HOME TO SATELLITE

# SAD ENTRIES
  add ${homeip} ${satip}  esp ${satspi}            -E simple;
  add ${satip}  ${homeip} esp $(( ${satspi} + 1 )) -E simple;

# PRIMARY TUNNEL INGRESS AND EGRESS FOR VPN
  spdadd ${vpn}.0.0/16        ${vpn}.${satno}.0/24 -P in  ipsec
    esp/tunnel/${homeip}-${satip}/require;
  spdadd ${vpn}.${satno}.0/24 ${vpn}.0.0/16        -P out ipsec
    esp/tunnel/${satip}-${homeip}/require;

# ADDITIONAL INGRESS AND EGRESS RULES ...

# ... FOR THE OTHER LOCAL NETWORK 10.0.0.0/8
  spdadd 10.0.0.0/8           ${vpn}.${satno}.0/24 -P in  ipsec
    esp/tunnel/${homeip}-${satip}/require;
  spdadd ${vpn}.${satno}.0/24 10.0.0.0/8           -P out ipsec
    esp/tunnel/${satip}-${homeip}/require;

# ... FOR SOME GLOBAL IP NETWORK THAT SHOULD BE ROUTED THROUGH TUNNEL
  spdadd 123.4.5.0/24  ${vpn}.${satno}.0/24 -P in  ipsec
    esp/tunnel/${homeip}-${satip}/require;
  spdadd ${vpn}.${satno}.0/24 123.4.5.0/24  -P out ipsec
    esp/tunnel/${satip}-${homeip}/require;

EOF

Now, doing this with last week's KAME SNAP works only partly. That
is, all those first degree VPN tunnels do work now. But the 
tunnels rules to the 10.0.0.0/8 and 123.4.5.0/24 networks work
only half way, that is, the central VPN home gateway will not
act upon those rules. When I delete the SPD rules for the first
degree VPN, one of the others suddenly work. Same problem as 
reported last time, just a little more involved to make it 
manifest.

Please let me know if you can or cannot reproduce this. I will
also try different things, but I do suspect that there is one
systematic error in the implementation of the SPD rules engine,
unless this systematic problem is found, the bug will only be
put off to become manifest in only weird situations that are 
even harder to reproduce.

Since my project is in jeopardy because of this bug, I have 
now engaged plan B, which is to use IPsec in transport mode 
and one gif tunnel connecting each sattelite with the home
gateway. I can then use as many static routes as I like to
add ingress rules for the tunnel. I am convinced this works
because the SPD entries are much simpler for this. However, on 
the way discovered another weirdness similar to the above.
Take the above setkey scripts and since the additional
tunnel ingress and egress rules do not work, let's delete
them and use a gif-tunnel hack instead. On the central home
gateway I would say:

for satno in 1 2 3 ... ; do
  gifconfig gif${satno} inet ${vpn}.0.1 ${vpn}.${satno}.1
  central=${pseudo}.223
  remote=${pseudo}.${satno}
  ifconfig  gif${satno} inet ${central} ${remote} netmask 0xffffff00
done

Here, ${pseudo} is another local network that is meaningful
only for the gif-link, for example, 

pseudo=172.16.1

would work. 

Now, on each sattelite ${satno} I would have defined this:

gifconfig gif0 inet ${vpn}.${satno}.1 ${vpn}.0.1
central=${pseudo}.223
remote=${pseudo}.${satno}
ifconfig  gif0 inet ${remote} ${central} netmask 0xffffff00

and then I'd have added all the additional ingress rules as 
static routes:

route add -net 10.0.0.0/8   ${central}
route add -net 123.4.5.0/24 ${central}


However, this, again does not work and it shows the same symptoms,
i.e., the "netstat -s -p ip" will show how the counter for 
"unknown / unsupported protocols" goes up. If I use the same 
gif tunnel with gifconfig not using the ${vpn}.0.x addresses 
but the respective global addresses, it all works -- but of
course the point is to make this gif tunnel hidden in the VPN
tunnel. May be this will help debugging the problem in KAME's
SPD rules engine.

Epilogue: I am open to the suggestion that I'm just doing a
stupid thing here and that x-thousand installations are out
there who establish similarly complex VPNs using KAME IPsec
tunnels with no problems at all. If so, please let me know
and if possible, tell me what I should do differently. 

regards
-Gunther

-- 
Gunther Schadow, M.D., Ph.D.                    gschadow@regenstrief.org
Medical Information Scientist      Regenstrief Institute for Health Care
Adjunct Assistent Professor        Indiana University School of Medicine
tel:1(317)630-7960                         http://aurora.regenstrief.org

To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-net" in the body of the message