Date: Wed, 6 Oct 2010 16:06:56 -0300 From: Eduardo Meyer <dudu.meyer@gmail.com> To: Julian Elischer <julian@freebsd.org>, Adrian Chadd <adrian@ucc.gu.uwa.edu.au>, ipfw@freebsd.org Subject: Re: layer2 ipfw 'fwd' support Message-ID: <AANLkTi=hoe%2BCaV6%2BbyagXYwzDRAHqCseh-M_44OxEeJO@mail.gmail.com> In-Reply-To: <4CAB8B35.7020703@freebsd.org> References: <AANLkTi=wHkmfDmoPrKN1SRcE9m=1_5iieAd85hQNWHs1@mail.gmail.com> <AANLkTinj8wd9AbROwRzUAUK=XraYmTDkoB3MGddqq-Tn@mail.gmail.com> <AANLkTin1vXOMPT6m8ybhNQk9G7WjDrCcSArP3Zwf65cR@mail.gmail.com> <4CAA1E7B.1020107@freebsd.org> <AANLkTikExTKMWvvDwn=rVUSqwz6UeVXi8WOSsHROQYq%2B@mail.gmail.com> <4CAA45CC.8020304@freebsd.org> <AANLkTikAd_fke1HfMgRy3h4fXpo7_DcX3E4%2BTu__3my8@mail.gmail.com> <4CAB8B35.7020703@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Oct 5, 2010 at 5:31 PM, Julian Elischer <julian@freebsd.org> wrote: > =A0On 10/5/10 12:56 PM, Eduardo Meyer wrote: >> >> On Mon, Oct 4, 2010 at 6:23 PM, Julian Elischer<julian@freebsd.org> >> =A0wrote: >>> >>> =A0On 10/4/10 12:18 PM, Eduardo Meyer wrote: >>>> >>>> On Mon, Oct 4, 2010 at 3:35 PM, Julian Elischer<julian@freebsd.org> >>>> =A0wrote: >>>>> >>>>> =A0On 10/4/10 10:16 AM, Eduardo Meyer wrote: >>>>>> >>>>>> On Mon, Oct 4, 2010 at 2:02 PM, Brandon Gooch >>>>>> <jamesbrandongooch@gmail.com> =A0 =A0 =A0wrote: >>>>>>> >>>>>>> On Mon, Oct 4, 2010 at 9:44 AM, Eduardo Meyer<dudu.meyer@gmail.com> >>>>>>> =A0wrote: >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> In the past I have used this patch by Luigi Rizzo, which helped me >>>>>>>> well. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> http://lists.freebsd.org/pipermail/freebsd-ipfw/2003-September/000= 526.html >>>>>>>> >>>>>>>> I tried with a friend to port it to -STABLE, but we were not able = to >>>>>>>> find out what has replaced mt_tag. Also on ip_input.c we dirty >>>>>>>> hacked >>>>>>>> to following piece of code: >>>>>>>> >>>>>>>> #ifdef IPFIREWALL_FORWARD >>>>>>>> =A0 =A0 =A0 =A0if (m->m_flags& =A0 =A0 =A0M_FASTFWD_OURS) { >>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0m->m_flags&=3D ~M_FASTFWD_OURS; >>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0goto pass; /* XXX was 'ours' - SHOU= LD WE MODIFY IT >>>>>>>> HERE >>>>>>>> */ >>>>>>>> =A0 =A0 =A0 =A0} >>>>>>>> =A0 =A0 =A0 =A0if ((dchg =3D (m_tag_find(m, PACKET_TAG_IPFORWARD, = NULL) !=3D >>>>>>>> NULL)) >>>>>>>> !=3D 0) { >>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0/* >>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * Directly ship the packet on. =A0= This allows >>>>>>>> forwarding >>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * packets originally destined to u= s to some other >>>>>>>> directly >>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * connected host. >>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 */ >>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0ip_forward(m, dchg); >>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return; >>>>>>>> =A0 =A0 =A0 =A0} >>>>>>>> #endif /* IPFIREWALL_FORWARD */ >>>>>>>> >>>>>>>> And this is something we are not sure if its correct. >>>>>>>> >>>>>>>> So my very obvious question is: >>>>>>>> >>>>>>>> Does anyone has a recent version of this patch to share? >>>>>>>> >>>>>>>> Can anyone familiar with ipfw source code help me with that? >>>>>>>> >>>>>>> I'm certainly not an expert, but I wonder if the patch your referri= ng >>>>>>> to is still required? Can you provide more detail about your >>>>>>> particular application? >>>>>>> >>>>>>> -Brandon >>>>>> >>>>>> Yes, its still required since ipfw fwd ignores layer2 frames. >>>>>> >>>>>> The application is the very same: squid. I mean, Lusca in fact (squi= d >>>>>> fork). >>>>>> >>>>>> Thank you for your interest. >>>>> >>>>> Cisco/Ironport have a patch that does this.. >>>>> I had permission to bring it back when I worked there but never got i= t >>>>> committed. >>>>> >>>>> Adrian, was it part of the set I gave you? >>>> >>>> Hello Elischer, >>>> >>>> Was this made public? >>>> >>>> I hope Chadd has some good news. In fact I tent to use with Lusca in >>>> tproxy mode. I bet this is the only missing piece of software. >>>> >>> I just dug up my old changes. >>> do you want to fwd from a bridge? or what? >>> (it makes a difference what patches are needed) >>> >>> If you want to fwd from a bridge to make a transparent layer 2 proxy, >>> this >>> may help.. >>> >>> >>> Here are parts of it that may be relevent: >>> these are old (2007 I think) but may be of use still. >>> >>> adrian had the full set at >>> >>> =3D=3Dquote adrian=3D=3D=3D=3D=3D >>> =A0The stuff is in p4 now, but I haven't tested it out at all. >>> >>> =A0 =A0//depo/projects/adrian_spoof_clientip/ =A0 I -think-. >>> =3D=3D end quote=3D=3D=3D >>> >>> >>> >>> >>> Index: net/if_bridge.c >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>> RCS file: /usr/local/cvsroot/freebsd/src/sys/net/if_bridge.c,v >>> retrieving revision 1.107 >>> diff -u -r1.107 if_bridge.c >>> --- net/if_bridge.c =A0 =A0 6 Nov 2007 23:01:42 -0000 =A0 =A0 =A0 1.107 >>> +++ net/if_bridge.c =A0 =A0 28 Nov 2007 06:59:10 -0000 >>> @@ -2908,6 +2908,11 @@ >>> =A0 =A0 =A0 =A0struct ip *ip; >>> =A0 =A0 =A0 =A0struct llc llc1; >>> =A0 =A0 =A0 =A0u_int16_t ether_type; >>> + =A0 =A0 =A0 int =A0 =A0 is_ip =3D 0; >>> +#ifdef IPFIREWALL_FORWARD >>> + =A0 =A0 =A0 struct m_tag *fwd_tag; >>> +#endif >>> + >>> >>> =A0 =A0 =A0 =A0snap =3D 0; >>> =A0 =A0 =A0 =A0error =3D -1; =A0 =A0 /* Default error if not error =3D= =3D 0 */ >>> @@ -2967,6 +2972,7 @@ >>> =A0#ifdef INET6 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0case ETHERTYPE_IPV6: >>> =A0#endif /* INET6 */ >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 is_ip =3D 1; >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0break; >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0default: >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0/* >>> @@ -3024,6 +3030,30 @@ >>> >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (*mp =3D=3D NULL) >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return (error); >>> + >>> +#ifdef IPFIREWALL_FORWARD >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0/* >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 * Did the firewall want to forward it som= ewhere? >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 * If so, let the ip stack handle it. >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 */ >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0if (i =3D=3D 0&& =A0 =A0args.next_hop !=3D= NULL&& >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 is_ip /*&& =A0 =A0src != =3D NULL */) { >>> + >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0fwd_tag =3D m_tag_get(PACK= ET_TAG_IPFORWARD, >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0sizeof(struct sockaddr_in), >>> M_NOWAIT); >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (fwd_tag =3D=3D NULL) >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0goto drop; >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0bcopy(args.next_hop, (fwd_= tag+1), >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 sizeof(st= ruct sockaddr_in)); >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0m_tag_prepend(*mp, fwd_tag= ); >>> + >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (in_localip(args.next_h= op->sin_addr)) >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0(*mp)->m_f= lags |=3D M_FASTFWD_OURS; >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0ether_demux(src, *mp); >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return (NULL); >>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0} >>> +#endif >>> + >>> >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (DUMMYNET_LOADED&& =A0 =A0(i =3D=3D I= P_FW_DUMMYNET)) { >>> >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>> Index: netinet/ip_fw2.c >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>> RCS file: /usr/local/cvsroot/freebsd/src/sys/netinet/ip_fw2.c,v >>> retrieving revision 1.178 >>> diff -u -r1.178 ip_fw2.c >>> --- netinet/ip_fw2.c =A0 =A028 Oct 2007 17:12:47 -0000 =A0 =A0 =A01.178 >>> +++ netinet/ip_fw2.c =A0 =A028 Nov 2007 06:59:10 -0000 >>> >>> @@ -3446,8 +3507,10 @@ >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0case O_FORWARD_IP: { >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0struct s= ockaddr_in *sa; >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sa =3D&(= ((ipfw_insn_sa *)cmd)->sa); >>> +#if 0 >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (args= ->eh) =A0 /* not valid on layer2 >>> pkts >>> */ >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0break; >>> +#endif >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (!q |= | dyn_dir =3D=3D MATCH_FORWARD) { >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0if (sa->sin_addr.s_addr =3D=3D >>> INADDR_ANY) { >>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0bcopy(sa,&args->hopstore, >>> >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>> Index: netinet/ip_output.c >> >> Dear Julian, >> >> Is anything missing from the above code? Say, like ip_output stuff? >> >> I have tried what you sent me, compiled fine but did not work. >> >> Here is my only rule (I have tried both with and without layer2 on the >> rule): >> >> 00001 =A0 =A0 =A0 =A036 =A0 =A0 =A0 =A04338 fwd 127.0.0.1,80 tcp from an= y to not me >> dst-port 80 layer2 >> 65535 32842101 2107060460 allow ip from any to any >> >> Here are the sysctl tunables: >> >> net.link.bridge.ipfw: 1 >> net.link.bridge.inherit_mac: 0 >> net.link.bridge.log_stp: 0 >> net.link.bridge.pfil_local_phys: 0 >> net.link.bridge.pfil_member: 0 >> net.link.bridge.pfil_bridge: 1 >> net.link.bridge.ipfw_arp: 0 >> net.link.bridge.pfil_onlyip: 0 >> net.link.ether.inet.log_arp_permanent_modify: 1 >> net.link.ether.inet.log_arp_movements: 1 >> net.link.ether.inet.log_arp_wrong_iface: 1 >> net.link.ether.inet.proxyall: 0 >> net.link.ether.inet.useloopback: 1 >> net.link.ether.inet.maxtries: 5 >> net.link.ether.inet.max_age: 1200 >> net.link.ether.ipfw: 1 >> >> And my bridge: >> >> bridge0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> =A0metric = 0 mtu >> 1500 >> =A0 =A0 =A0 =A0 ether 16:52:8e:91:2f:45 >> =A0 =A0 =A0 =A0 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay= 15 >> =A0 =A0 =A0 =A0 maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200 >> =A0 =A0 =A0 =A0 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 >> =A0 =A0 =A0 =A0 member: vr0 flags=3D143<LEARNING,DISCOVER,AUTOEDGE,AUTOP= TP> >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ifmaxaddr 0 port 5 priority 128 path cos= t 200000 >> =A0 =A0 =A0 =A0 member: sis0 flags=3D143<LEARNING,DISCOVER,AUTOEDGE,AUTO= PTP> >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ifmaxaddr 0 port 1 priority 128 path cos= t 200000 >> >> The ipfw counter gets increased by nothing hits by Apache. Instead I >> go to Internet directly. >> >> sis0 is on internet, vr0 is cross-over to the laptop (customer). >> >> How should I debug it? >> >> > basically I woud suggest code inspection for a start.. > > look at where ipfw is called (just before where the patch went in) and > follow the packet up into ipfw > and back, =A0and read what it would do.. > > It's actually not a very hard path to follow. > > I'll try look at it after work.. Hello Julian / Adrian. Thank you for your attention. A friend added some log entries so we could try to find out what gets run and what doesnt. Here is my current patch against RELENG_8: --- if_bridge.c.orig 2010-09-11 22:02:36.000000000 +0000 +++ if_bridge.c 2010-10-05 17:59:13.000000000 +0000 @@ -2957,6 +2957,13 @@ struct ip *ip; struct llc llc1; u_int16_t ether_type; + int is_ip =3D 0; +#ifdef IPFIREWALL_FORWARD + struct m_tag *fwd_tag; +#endif + + + snap =3D 0; error =3D -1; /* Default error if not error =3D=3D 0 */ @@ -3016,6 +3023,8 @@ #ifdef INET6 case ETHERTYPE_IPV6: #endif /* INET6 */ + is_ip=3D1; + log(LOG_NOTICE, "Entered 0: is_ip=3D%i\n",is_ip); break; default: /* @@ -3091,6 +3100,32 @@ if (*mp =3D=3D NULL) return (error); +#ifdef IPFIREWALL_FORWARD + /* + * Did the firewall want to forward it somewhere? + * If so, let the ip stack handle it. + */ + log(LOG_NOTICE, "Entered 1"); + if (i =3D=3D 0&& args.next_hop !=3D NULL&& + is_ip /*&& src !=3D NULL */) { + log(LOG_NOTICE, "Entered 2"); + + fwd_tag =3D m_tag_get(PACKET_TAG_IPFORWARD, + sizeof(struct sockaddr_in), M_NOWAIT= ); + if (fwd_tag =3D=3D NULL) + goto drop; + bcopy(args.next_hop, (fwd_tag+1), + sizeof(struct sockaddr_in)); + m_tag_prepend(*mp, fwd_tag); + + if (in_localip(args.next_hop->sin_addr)) + (*mp)->m_flags |=3D M_FASTFWD_OURS; + ether_demux(src, *mp); + return (NULL); + } +#endif + + if (ip_dn_io_ptr && (i =3D=3D IP_FW_DUMMYNET)) { /* put the Ethernet header back on */ --- ../netinet/ipfw/ip_fw2.c.orig 2010-09-16 15:11:17.000000000 +0000 +++ ../netinet/ipfw/ip_fw2.c 2010-10-06 12:17:12.000000000 +0000 @@ -2059,8 +2059,14 @@ break; case O_FORWARD_IP: - if (args->eh) /* not valid on layer2 pkts= */ +#if 0 + /* not valid on layer2 pkts */ + if (args->eh) { + log(LOG_NOTICE, "ip_fw2.c Entered 1= "); break; + } + log(LOG_NOTICE, "ip_fw2.c Entered 2"); +#endif if (!q || dyn_dir =3D=3D MATCH_FORWARD) { struct sockaddr_in *sa; sa =3D &(((ipfw_insn_sa *)cmd)->sa); Please notice the log entries for debugging. When I try with the very same rule: fwd 127.0.0.1,80 tcp from any to not me dst-port 80 layer2 This is what I get on /var/log/messages: Oct 6 15:58:44 phoenix kernel: Entered 0: is_ip=3D1 Oct 6 15:59:16 phoenix last message repeated 93 times Oct 6 16:01:20 phoenix last message repeated 189 times So we never "Entered 1", never "Entered 2" nor ever entered "ip_fw2.c Entered 2"; Seem like FWD is never triggered... I don't know where to move forth. Any help is appreciated. > > --=20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Eduardo Meyer pessoal: dudu.meyer@gmail.com profissional: ddm.farmaciap@saude.gov.br
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTi=hoe%2BCaV6%2BbyagXYwzDRAHqCseh-M_44OxEeJO>