Date: Fri, 21 Aug 2015 16:24:18 GMT From: dpl@FreeBSD.org To: svn-soc-all@FreeBSD.org Subject: socsvn commit: r290032 - in soc2015/dpl: . extra extra/sys extra/sys/contrib extra/sys/contrib/pf extra/sys/contrib/pf/net extra/sys/sys ipfw presentation sys sys/net sys/netgraph sys/netinet sys/n... Message-ID: <201508211624.t7LGOI0X055661@socsvn.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: dpl Date: Fri Aug 21 16:24:17 2015 New Revision: 290032 URL: http://svnweb.FreeBSD.org/socsvn/?view=rev&rev=290032 Log: Import all the code from github. This is just a copy of github.com/dplbsd/netmap-ipfwjit. Added: soc2015/dpl/BSDmakefile soc2015/dpl/BUGS soc2015/dpl/LICENSE soc2015/dpl/Makefile soc2015/dpl/Makefile.inc soc2015/dpl/Makefile.kipfw soc2015/dpl/README.md soc2015/dpl/extra/ soc2015/dpl/extra/expand_number.c soc2015/dpl/extra/glue.c soc2015/dpl/extra/glue.h soc2015/dpl/extra/humanize_number.c soc2015/dpl/extra/ipfw2_mod.c soc2015/dpl/extra/linux_defs.h soc2015/dpl/extra/missing.c soc2015/dpl/extra/missing.h soc2015/dpl/extra/netmap_io.c soc2015/dpl/extra/session.c soc2015/dpl/extra/sys/ soc2015/dpl/extra/sys/contrib/ soc2015/dpl/extra/sys/contrib/pf/ soc2015/dpl/extra/sys/contrib/pf/net/ soc2015/dpl/extra/sys/contrib/pf/net/pfvar.h soc2015/dpl/extra/sys/sys/ soc2015/dpl/extra/sys/sys/kernel.h soc2015/dpl/extra/sys/sys/malloc.h soc2015/dpl/extra/sys/sys/mbuf.h soc2015/dpl/extra/sys/sys/module.h soc2015/dpl/extra/sys/sys/systm.h soc2015/dpl/extra/sys/sys/taskqueue.h soc2015/dpl/ipfw/ soc2015/dpl/ipfw/Makefile soc2015/dpl/ipfw/altq.c soc2015/dpl/ipfw/dummynet.c soc2015/dpl/ipfw/ipfw2.c soc2015/dpl/ipfw/ipfw2.h soc2015/dpl/ipfw/ipv6.c soc2015/dpl/ipfw/main.c soc2015/dpl/ipfw/nat.c soc2015/dpl/presentation/ soc2015/dpl/presentation/Figure1.png (contents, props changed) soc2015/dpl/presentation/jit.pdf (contents, props changed) soc2015/dpl/presentation/jit.tex (contents, props changed) soc2015/dpl/sys/ soc2015/dpl/sys/net/ soc2015/dpl/sys/net/pfil.h soc2015/dpl/sys/net/radix.c soc2015/dpl/sys/net/radix.h soc2015/dpl/sys/netgraph/ soc2015/dpl/sys/netgraph/ng_ipfw.h soc2015/dpl/sys/netinet/ soc2015/dpl/sys/netinet/in_cksum.c soc2015/dpl/sys/netinet/ip_dummynet.h soc2015/dpl/sys/netinet/ip_fw.h soc2015/dpl/sys/netinet/tcp.h soc2015/dpl/sys/netinet/udp.h soc2015/dpl/sys/netpfil/ soc2015/dpl/sys/netpfil/ipfw/ soc2015/dpl/sys/netpfil/ipfw/dn_heap.c soc2015/dpl/sys/netpfil/ipfw/dn_heap.h soc2015/dpl/sys/netpfil/ipfw/dn_sched.h soc2015/dpl/sys/netpfil/ipfw/dn_sched_fifo.c soc2015/dpl/sys/netpfil/ipfw/dn_sched_prio.c soc2015/dpl/sys/netpfil/ipfw/dn_sched_qfq.c soc2015/dpl/sys/netpfil/ipfw/dn_sched_rr.c soc2015/dpl/sys/netpfil/ipfw/dn_sched_wf2q.c soc2015/dpl/sys/netpfil/ipfw/ip_dn_glue.c soc2015/dpl/sys/netpfil/ipfw/ip_dn_io.c soc2015/dpl/sys/netpfil/ipfw/ip_dn_private.h soc2015/dpl/sys/netpfil/ipfw/ip_dummynet.c soc2015/dpl/sys/netpfil/ipfw/ip_fw2.c soc2015/dpl/sys/netpfil/ipfw/ip_fw_dynamic.c soc2015/dpl/sys/netpfil/ipfw/ip_fw_log.c soc2015/dpl/sys/netpfil/ipfw/ip_fw_pfil.c soc2015/dpl/sys/netpfil/ipfw/ip_fw_private.h soc2015/dpl/sys/netpfil/ipfw/ip_fw_rules.c soc2015/dpl/sys/netpfil/ipfw/ip_fw_rules.h soc2015/dpl/sys/netpfil/ipfw/ip_fw_sockopt.c soc2015/dpl/sys/netpfil/ipfw/ip_fw_table.c soc2015/dpl/sys/netpfil/ipfw/jit.cc soc2015/dpl/sys/netpfil/ipfw/jit.h soc2015/dpl/tests/ soc2015/dpl/tests/README.md soc2015/dpl/tests/configs/ soc2015/dpl/tests/configs/kipfwclear.sh soc2015/dpl/tests/configs/kipfwpassDNS.sh soc2015/dpl/tests/configs/kipfwpassUDP.sh soc2015/dpl/tests/tests/ soc2015/dpl/tests/tests/pktgen-passall/ soc2015/dpl/tests/tests/pktgen-passall/conductor.cfg soc2015/dpl/tests/tests/pktgen-passall/dut.cfg soc2015/dpl/tests/tests/pktgen-passall/sink.cfg soc2015/dpl/tests/tests/pktgen-passall/source.cfg soc2015/dpl/tools/ soc2015/dpl/tools/GNUmakefile soc2015/dpl/tools/Makefile soc2015/dpl/tools/README soc2015/dpl/tools/bridge.c soc2015/dpl/tools/click-test.cfg soc2015/dpl/tools/pkt-gen.c soc2015/dpl/tools/test_nm.c soc2015/dpl/tools/test_select.c soc2015/dpl/tools/testcsum.c soc2015/dpl/tools/testlock.c soc2015/dpl/tools/testmmap.c soc2015/dpl/tools/testmod/ soc2015/dpl/tools/testmod/Makefile soc2015/dpl/tools/testmod/kern_test.c soc2015/dpl/tools/vale-ctl.c Added: soc2015/dpl/BSDmakefile ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ soc2015/dpl/BSDmakefile Fri Aug 21 16:24:17 2015 (r290032) @@ -0,0 +1,8 @@ +# forward to use gmake +.PHONY: ipfw kipfw + +all: + gmake + +$(.TARGETS) : + gmake MAKE=gmake $(.TARGETS) Added: soc2015/dpl/BUGS ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ soc2015/dpl/BUGS Fri Aug 21 16:24:17 2015 (r290032) @@ -0,0 +1,73 @@ +# This file contains the information for the bugs we've encountered during the +# development process. It sort of follows the guideline of: +# http://courses.cse.tamu.edu/cscd606fall13/walker/Papers/BugScientificMethod.pdf + +1 BUG: Firewall can't compile code. -- Solved. + Comments: We think this happens due to the change in the LLVM version we're + using now. During the last summer, I used devel/llvm-devel, and we're now + settling with LLVM 3.5, and I'll probably end up porting to the C API. + Error message(s): + * (LLVM ERROR: Target does not support MC emission!) + * () (empty errsrt after creating ExecutionEngine). + * (LLVM ERROR: Inline asm not supported by this streamer because we + don't have an asm parser for this target) When using + InitializeNativeTargetAsmParser(). + * KERNEL PANIC: Happens when at initialization we call: + InitializeNativeTarget(); + InitializeNativeTargetAsmPrinter(); + InitializeNativeTargetAsmParser(); + LLVMLinkInMCJIT(); + This is not expected at all. We're supposedly playing at userland. + + + TEST: LLVM build is broken. + PROOF: Use same LLVM build with another compiler using MCJIT as well. + DISPROVEN: Another LLVM project works perfectly + (CompileTeaching/MysoreScript). + + TEST: Final IR is wrong. + PROOF: Dump IR into a file and compile with clang's assembler. + DISPROVEN: Not needed having a module verfier. + PROOF: Use the runtime code checker. (.setVerifyModule(true)) + DISPROVEN: Even when checking the module, it doesn't work. + + TEST: Function and module optimization are the culprit. + PROOF: Just do not perform any optimization. I also didn't deleted + "void_function" a function used so that we can use an initial bitcode + with the needed structs and type information. + DISPROVEN: Not working despite change. + + TEST: LLVM is initialized inadequately. + PROOF: Perform the minimal initialization, see if it works. + DISPROVEN: It doesn't work this way. + PROOF: Myscript is correct, therefore, we can debug its initialization + and perform it ourselves in the same way. + Comment: Instead of run a debugger, I'm trying to use something to + get a function call graph. That should do the work too. + I ended up copying the code from MysoreScript after searching for + too long. + DISPROVEN: Still getting the "Target does not support MC emission!" + error. + + TEST: Project is not build adequatelly. + Comment: This has been double checked now. + PROOF: Compile a different compiler and compare results. + DISPROVEN: Changed the Makefile to reflect the compiling and linking + process of another working compiler (CompileTeaching/MysoreScript). + It works perfectly. + +2 BUG: Kernel panic when filtering an unbounded quantity of packets. + TEST: Is there any hardcoded limit at the number of packets? + OBS: 500 packets - Kernel panic. + 625 packets - Kernel panic. + 750 packets - Kernel panic. + 1000 packets - Kernel panic. + +3 BUG: Code emission didn't work properly for some functions. -- Solved. +4 BUG: Code emission for some rules isn't yet completed. -- Solved. + - Still working on call/return. + +5 BUG: Compilation error when compiling code emitted by emit_jail(). -- Solved. + Comment: There's something wrong going on with the BasicBlocks used by this + function. It's emission is correct, though. + Solved by instead creating two branchs, OR'ed the result of two comparisons. Added: soc2015/dpl/LICENSE ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ soc2015/dpl/LICENSE Fri Aug 21 16:24:17 2015 (r290032) @@ -0,0 +1,24 @@ +Copyright (c) 2014, dpl +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + Added: soc2015/dpl/Makefile ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ soc2015/dpl/Makefile Fri Aug 21 16:24:17 2015 (r290032) @@ -0,0 +1,38 @@ +# +# This is a gnu makefile to build ipfw in userspace. +# Usage: +# +# make NETMAP_INC=/some/place/with/netmap-release/sys +# +# build with make NETMAP_INC=/place/with/netmap/sys + +SUBDIRS= ipfw dummynet +.PHONY: ipfw kipfw + +include Makefile.inc +all: ipfw kipfw + +ipfw: $(OBJDIR) + $(MSG) Building userspace ... + @(cd ipfw && $(MAKE) $(MAKECMDGOALS) ) + +$(OBJDIR): + -@mkdir $(OBJDIR) + +kipfw: $(OBJDIR) + $(MSG) Building datapath ... + @(cd $(OBJDIR) && $(MAKE) -f ../Makefile.kipfw && cp kipfw ..) + +clean: + -@rm -rf $(OBJDIR) kipfw + @(cd ipfw && $(MAKE) clean ) + -@rm -f *.bc + +tgz: + @$(MAKE) clean + (cd ..; tar cvzf /tmp/ipfw-user.tgz --exclude .svn ipfw-user) + +# compute diffs wrt FreeBSD head tree in BSD_HEAD +diffs: + -@diff -urp --exclude Makefile $(BSD_HEAD)/sbin/ipfw ipfw + -@diff -urp --exclude Makefile $(BSD_HEAD)/sys sys Added: soc2015/dpl/Makefile.inc ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ soc2015/dpl/Makefile.inc Fri Aug 21 16:24:17 2015 (r290032) @@ -0,0 +1,40 @@ +# +# this is a gnu makefile + +# Use LLVM 3.5. +# CXXFLAGS includes -g +LLVM_CONFIG=/home/dpl/wrk/llvm_build/bin/llvm-config +CC=clang35 +CCXX=clang++35 +LLVM_VERSION=-DLLVM_MAJOR=3 -DLLVM_MINOR=5 +LLVM_CXXFLAGS=`${LLVM_CONFIG} --cxxflags` +LLVM_LDFLAGS=`${LLVM_CONFIG} --ldflags` +LLVM_SYSLIBS=`${LLVM_CONFIG} --system-libs` +LLVM_LIBS=`${LLVM_CONFIG} --libs all` + + +BSD_HEAD ?= /home/luigi/FreeBSD/head +NETMAP_INC ?= ../netmap-release/sys + +OBJDIR=objs +OSARCH := $(shell uname) +OSARCH := $(findstring $(OSARCH),FreeBSD Linux Darwin) +ifeq ($(OSARCH),) + OSARCH := Windows +endif + +ifeq ($V,) # no echo + MSG=@echo + HIDE=@ +else + MSG=@\# + HIDE= +endif + +# ipfw and kipfw are built in subdirs so the paths for +# headers refer to one directory up +INCDIRS += -I ../$(OBJDIR)/include_e -DEMULATE_SYSCTL +INCDIRS += -I ../sys -I ../extra/sys -I ../extra/sys/contrib/pf +.c.o: + $(MSG) " CC $<" + $(HIDE) $(CC) $(CFLAGS) -O0 -g -c $< -o $@ Added: soc2015/dpl/Makefile.kipfw ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ soc2015/dpl/Makefile.kipfw Fri Aug 21 16:24:17 2015 (r290032) @@ -0,0 +1,183 @@ +# gnu Makefile to build a userland version of the +# kernel code for ipfw+dummynet +# +# The kernel code is compiled with appropriate flags to make +# it see a kernel-like environment. +# The userland emulation code is compiler with regular flags. + +# M is the current directory, used in recursive builds +# so we allow it to be overridden +include ../Makefile.inc +VPATH = ../extra:../sys/netpfil/ipfw:../sys/netinet:../sys/net +M ?= $(shell pwd) +OBJPATH = $(M)/../$(OBJDIR) + +ifeq ($(OSARCH),Darwin) + CFLAGS2 += -D__BSD_VISIBLE + EFILES_. += libutil.h + EFILES_sys += condvar.h priv.h _lock.h rmlock.h + EFILES_machine += in_cksum.h + EFILES_netinet += ip_carp.h pim.h sctp.h + EFILES_net += netisr.h vnet.h +endif + +ifeq ($(OSARCH),Linux) + CFLAGS2 += -D__BSD_VISIBLE + CFLAGS2 += -include ../extra/linux_defs.h + CFLAGS2 += -Wno-unused-but-set-variable + EFILES_. += libutil.h + EFILES_sys += condvar.h priv.h _lock.h rmlock.h + EFILES_sys += lock.h ucred.h # taskqueue.h + EFILES_sys += sockio.h + EFILES_sys += cpuset.h + EFILES_machine += in_cksum.h + EFILES_netinet += in_pcb.h ip_carp.h pim.h sctp.h tcp_var.h + EFILES_net += if_types.h bpf.h netisr.h vnet.h + EFILES_linux += module.h +endif + +ifeq ($(OSARCH),Windows) + CFLAGS2 += -D__BSD_VISIBLE +# CFLAGS2 += -include ../extra/linux_defs.h + CFLAGS2 += -Wno-unused-but-set-variable +# EFILES_. += libutil.h +# EFILES_sys += condvar.h priv.h _lock.h rmlock.h +# EFILES_sys += lock.h ucred.h # taskqueue.h +# EFILES_sys += sockio.h +# EFILES_machine += in_cksum.h +# EFILES_netinet += in_pcb.h ip_carp.h pim.h sctp.h tcp_var.h +# EFILES_net += if_types.h bpf.h netisr.h vnet.h +# EFILES_linux += module.h + EFILES_sys += sockio.h + EFILES_net += ethernet.h + EFILES_sys += condvar.h priv.h socketvar.h ucred.h + EFILES_net += vnet.h + EFILES_netinet += in_pcb.h ip_carp.h pim.h sctp.h tcp_var.h +endif + +NETMAP_FLAGS = -DWITH_NETMAP -I$(NETMAP_INC) + +E_CFLAGS += $(INCDIRS) +E_CFLAGS += -include $(M)/../extra/glue.h # headers +E_CFLAGS += -include $(M)/../extra/missing.h # headers +E_CFLAGS += -O2 -Wall -Werror -fno-strict-aliasing +E_CFLAGS += -g +E_CFLAGS += -DKERNEL_SIDE # build the kernel side of the firewall +E_CFLAGS += -DUSERSPACE # communicate through userspace +E_CFLAGS += $(EFLAGS) $(NETMAP_FLAGS) +E_CFLAGS += -DINET +E_CFLAGS += -DIPFIREWALL_DEFAULT_TO_ACCEPT +E_CFLAGS += -D_BSD_SOURCE +# many of the kernel headers need _KERNEL +E_CFLAGS += -D_KERNEL +E_CFLAGS += $(CFLAGS2) + +#ipfw + dummynet section, other parts are not compiled in +SRCS_IPFW = ip_fw2.c ip_fw_pfil.c ip_fw_sockopt.c +SRCS_IPFW += ip_fw_dynamic.c ip_fw_table.c +SRCS_IPFW += ip_fw_log.c +SRCS_IPFW += ip_dummynet.c ip_dn_io.c ip_dn_glue.c +SRCS_IPFW += dn_heap.c +SRCS_IPFW += dn_sched_fifo.c dn_sched_wf2q.c +SRCS_IPFW += dn_sched_rr.c dn_sched_qfq.c +SRCS_IPFW += dn_sched_prio.c +SRCS_NET = radix.c +SRCS_NETINET = in_cksum.c +# Module glue and functions missing in linux +IPFW_SRCS = $(SRCS_IPFW) $(SRCS_NET) $(SRCS_NETINET) +IPFW_SRCS += ipfw2_mod.c # bsd_compat.c + +IPFW_SRCS += missing.c session.c netmap_io.c +IPFW_CFLAGS= -DINET + +E_CFLAGS += -Dradix +MOD := kipfw + +CFLAGS = $(E_CFLAGS) + +IPFW_OBJS= $(IPFW_SRCS:%.c=%.o) +IPFW_OBJS+= jit.o + +all: include_e $(MOD) + +# entries to create empty files +EFILES_. += opt_inet.h opt_ipsec.h opt_ipdivert.h +EFILES_. += opt_inet6.h opt_ipfw.h opt_mpath.h +EFILES_. += opt_mbuf_stress_test.h opt_param.h +EFILES_. += timeconv.h + +EFILES_altq += if_altq.h + +EFILES_net += if_var.h route.h if_clone.h +EFILES_netpfil/pf += pf_mtag.h +EFILES_netinet += in_var.h ip_var.h udp_var.h +EFILES_netinet6 += ip6_var.h +EFILES_sys += proc.h sockopt.h sysctl.h +# new +EFILES_sys += mutex.h _mutex.h _rwlock.h rwlock.h +EFILES_sys += eventhandler.h +EFILES_sys += jail.h ktr.h + +#EFILES += sys/_lock.h sys/_rwlock.h sys/rwlock.h sys/rmlock.h sys/_mutex.h sys/mutex.h +#EFILES += sys/condvar.h sys/eventhandler.h # sys/domain.h +#EFILES += sys/limits.h sys/lock.h sys/mutex.h sys/priv.h +#EFILES += sys/proc.h sys/rwlock.h sys/socket.h sys/socketvar.h +#EFILES += sys/sysctl.h sys/time.h sys/ucred.h + +#EFILES += vm/uma_int.h vm/vm_int.h vm/uma_dbg.h +#EFILES += vm/vm_dbg.h vm/vm_page.h vm/vm.h +#EFILES += sys/rwlock.h sys/sysctl.h + +# first make a list of directories from variable names +EDIRS= $(subst EFILES_,,$(filter EFILES_%,$(.VARIABLES))) +# then prepend the directory name to individual files. +# $(empty) serves to interpret the following space literally, +# and the ": = " substitution packs spaces into one. +EFILES = $(foreach i,$(EDIRS),$(subst $(empty) , $(i)/, $(EFILES_$(i): = ))) + +include_e: + -@echo "Building $(OBJPATH)/include_e ..." + -$(HIDE)rm -rf $(OBJPATH)/include_e opt_* + -$(HIDE)mkdir -p $(OBJPATH)/include_e + -$(HIDE)(cd $(OBJPATH)/include_e; mkdir -p $(EDIRS); touch $(EFILES) ) + +$(IPFW_OBJS) : ../extra/glue.h + +ip_fw2.o ip_dummynet.o: # EFLAGS= -include missing.h +ip_fw2.o: jit.o + +# Generate the actual bytecode to be used, +# since we don't want to get our symbols removed, we're compiling with -O0. +../rules.bc: + $(CC) $(CFLAGS) -O0 -emit-llvm -c -o ../rules.bc ../sys/netpfil/ipfw/ip_fw_rules.c + +radix.o:# CFLAGS += -U_KERNEL + +# session.o: CFLAGS = -O2 +nm_util.o: CFLAGS = -O2 -Wall -Werror $(NETMAP_FLAGS) + +# kipfw, CXXFLAGS includes -g +LINKTIME_FLAGS += -std=c++11 -Wno-zero-length-array +LINKTIME_FLAGS += ${LLVM_CXXFLAGS} $(LLVM_VERSION) ${LLVM_LDFLAGS} +$(MOD): $(IPFW_OBJS) + $(MSG) " LD $@" + $(HIDE)${CCXX} $(LINKTIME_FLAGS) -Wl,--export-dynamic $^ -o $@ ${LLVM_LIBS} ${LLVM_SYSLIBS} + + +#Flags needed for jit.cc +JIT_CFLAGS += $(INCDIRS) $(NETMAP_FLAGS) +JIT_CFLAGS += -DINET -D_KERNEL -D_BSD_SOURCE -DKERNEL_SIDE -DUSERSPACE +JIT_CFLAGS += -I ../extra/ -Wno-extern-c-compat -Wno-comment +JIT_CFLAGS += ${LLVM_CXXFLAGS} $(LLVM_VERSION) +jit.o: jit.cc ../rules.bc + $(MSG) " CC $<" + $(HIDE)${CCXX} $(JIT_CFLAGS) -O0 -c -o ./jit.o ../sys/netpfil/ipfw/jit.cc + +clean: + -rm -f *.o $(DN) $(MOD) + -rm -rf include_e + +diff: + @-(for i in $(SRCS_IPFW) ; do diff -ubw $(BSD_HEAD)/sys/netpfil/ipfw/$$i .; done) + @-(for i in $(SRCS_NET) ; do diff -ubw $(BSD_HEAD)/sys/net/$$i . ; done) + @-(for i in $(SRCS_NETINET) ; do diff -ubw $(BSD_HEAD)/sys/netinet/$$i .; done) Added: soc2015/dpl/README.md ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ soc2015/dpl/README.md Fri Aug 21 16:24:17 2015 (r290032) @@ -0,0 +1,99 @@ +Note: This README is taken from the original netmap-ipfw, and adopted as needed +here. + +netmap-ipfwjit +============== +This directory contains a version of ipfw and dummynet that can run in +userland, using NETMAP as the backend for packet I/O. This permits a +throughput about 10 times higher than the corresponding in-kernel version. +Luigi has measured about 6.5 Mpps for plain filtering, and 2.2 Mpps going +through a pipe. Some optimizations are possible when running on netmap pipes, +or other netmap ports that support zero copy. + +To build the code simply run + make NETMAP_INC=/some/where/with/netmap-release/sys + +pointing to the netmap 'sys' directory (make will use gmake underneath). + +The base version comes from FreeBSD-HEAD -r '{2012-08-03}' +(and subsequently updated in late 2013) with small modifications listed below + + netinet/ipfw + ip_dn_io.c + support for on-stack mbufs + ip_fw2.c + some conditional compilation for functions not + available in userspace + ip_fw_log.c + revise snprintf, SNPARGS (MAC) + + +sbin/ipfw and the kernel counterpart communicate through a TCP socket +(localhost:5555) carrying the raw data that would normally be carried on +seg/getsockopt. + +Testing +======= +For testing purposes, opening a telnet session to port 5556 and typing some +bytes will start a fake 'infinite source' so you can check how fast your +ruleset works. + + gmake + dummynet/ipfw & # preferably in another window + telnet localhost 5556 # type some bytes to start 'traffic' + + sh -c "while true; do ipfw/ipfw show; ipfw/ipfw zero; sleep 1; done" + +(on an i7-3400 Luigi gets about 15 Mpps) + +Real packet I/O is possible using netmap info.iet.unipi.it/~luigi/netmap/ You +can use a couple of VALE switches (part of netmap, included in ./tools/) to +connect a source and sink to the userspace firewall, as follows: + + [pkt-gen]-->--[valeA:s]-->--[kipfw]-->--[valeA:r]-->--[pkt-gen] + +The commands to run (in separate windows) are: + + # preliminarly, load the netmap module if needed: + sudo kldload netmap.ko + + # connect the firewall to two vale switches + ./kipfw valeA:f valeB:f & + + # configure ipfw/dummynet + ipfw/ipfw show # or other + + # start the sink + pkt-gen -i valeB:d -f rx + + # start an infinite source + pkt-gen -i valeA:s -f tx + + # plain again with the firewall and enjoy + ipfw/ipfw show # or other + +Luigi reports that on his i7-3400 he got about 6.5 Mpps with a single rule, and +about 2.2 Mpps when going through a dummynet pipe using netmap-ipfw. This is +for a single process handling the traffic. + +Unfortunately, this is not enough to test it properly, as we'll have to test +different protocols besides IP, which is what ./tools/pkt-gen does. At some +point, we'll test the firewall properly with a set of rulesets, that will be +added to ./rulesets and commented adequately. + +Simple benchmarch +================= +We executed and tested it with just one rule (accept all), and 1k packets and +this is what we found (This was done on a computer with a 3 year-old i7): +- Compilation time: 130ms (Amortized when 41440 packets are filtered). +- Filtering time (JIT): 523us +- Filtering time (Interpreter): 3664us + +This basically means we'll have a x7 speedup compared to the interpreter, and +the more rules we have, the better the speedup will be. + +Current state +============= +- The JIT compiler is not working. +- All the commands except the flow-modifying ones should work well. + Added: soc2015/dpl/extra/expand_number.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ soc2015/dpl/extra/expand_number.c Fri Aug 21 16:24:17 2015 (r290032) @@ -0,0 +1,101 @@ +/*- + * Copyright (c) 2007 Eric Anderson <anderson@FreeBSD.org> + * Copyright (c) 2007 Pawel Jakub Dawidek <pjd@FreeBSD.org> + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <sys/cdefs.h> +__FBSDID("$FreeBSD: head/lib/libutil/expand_number.c 211343 2010-08-15 18:32:06Z des $"); + +#include <sys/types.h> +#include <ctype.h> +#include <errno.h> +#include <inttypes.h> +//#include <libutil.h> +#include <stdint.h> + +/* + * Convert an expression of the following forms to a uint64_t. + * 1) A positive decimal number. + * 2) A positive decimal number followed by a 'b' or 'B' (mult by 1). + * 3) A positive decimal number followed by a 'k' or 'K' (mult by 1 << 10). + * 4) A positive decimal number followed by a 'm' or 'M' (mult by 1 << 20). + * 5) A positive decimal number followed by a 'g' or 'G' (mult by 1 << 30). + * 6) A positive decimal number followed by a 't' or 'T' (mult by 1 << 40). + * 7) A positive decimal number followed by a 'p' or 'P' (mult by 1 << 50). + * 8) A positive decimal number followed by a 'e' or 'E' (mult by 1 << 60). + */ +int +expand_number(const char *buf, uint64_t *num) +{ + uint64_t number; + unsigned shift; + char *endptr; + + number = strtoumax(buf, &endptr, 0); + + if (endptr == buf) { + /* No valid digits. */ + errno = EINVAL; + return (-1); + } + + switch (tolower((unsigned char)*endptr)) { + case 'e': + shift = 60; + break; + case 'p': + shift = 50; + break; + case 't': + shift = 40; + break; + case 'g': + shift = 30; + break; + case 'm': + shift = 20; + break; + case 'k': + shift = 10; + break; + case 'b': + case '\0': /* No unit. */ + *num = number; + return (0); + default: + /* Unrecognized unit. */ + errno = EINVAL; + return (-1); + } + + if ((number << shift) >> shift != number) { + /* Overflow */ + errno = ERANGE; + return (-1); + } + + *num = number << shift; + return (0); +} Added: soc2015/dpl/extra/glue.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ soc2015/dpl/extra/glue.c Fri Aug 21 16:24:17 2015 (r290032) @@ -0,0 +1,540 @@ +/* + * Userland functions missing in linux + * taken from /usr/src/lib/libc/stdtime/time32.c + */ + +#include <stdlib.h> +#include <stdio.h> +#include <sys/types.h> +#include <sys/socket.h> +#include <netinet/in.h> /* sockaddr_in */ +#include <sys/uio.h> +#include <unistd.h> /* uint* types */ +#include <errno.h> +#include <string.h> /* bzero */ +#include <arpa/inet.h> /* htonl */ + +#ifndef HAVE_NAT +/* dummy nat functions */ +void +ipfw_show_nat(int ac, char **av) +{ + D("unsupported"); +} + +void +ipfw_config_nat(int ac, char **av) +{ + D("unsupported"); +} +#endif /* HAVE_NAT */ + +#ifdef NEED_STRTONUM +/* missing in linux and windows */ +long long int +strtonum(const char *nptr, long long minval, long long maxval, + const char **errstr) +{ + long long ret; + int errno_c = errno; /* save actual errno */ + + errno = 0; +#ifdef TCC + ret = strtol(nptr, (char **)errstr, 0); +#else + ret = strtoll(nptr, (char **)errstr, 0); +#endif + /* We accept only a string that represent exactly a number (ie. start + * and end with a digit). + * FreeBSD version wants errstr==NULL if no error occurs, otherwise + * errstr should point to an error string. + * For our purspose, we implement only the invalid error, ranges + * error aren't checked + */ + if (errno != 0 || nptr == *errstr || **errstr != '\0') + *errstr = "invalid"; + else { + *errstr = NULL; + errno = errno_c; + } + return ret; +} + +int +ishexnumber(int c) +{ + return ((c >= '0' && c <= '9') || + (c >= 'a' && c <= 'f') || + (c >= 'A' && c <= 'F') ); +} + +#endif /* NEED_STRTONUM */ + +#ifdef __linux__ + + +int optreset; /* missing in linux */ + +/* + * not implemented in linux. + * taken from /usr/src/lib/libc/string/strlcpy.c + */ +size_t +strlcpy(dst, src, siz) + char *dst; + const char *src; + size_t siz; +{ + char *d = dst; + const char *s = src; + size_t n = siz; + + /* Copy as many bytes as will fit */ + if (n != 0 && --n != 0) { + do { + if ((*d++ = *s++) == 0) + break; + } while (--n != 0); + } + + /* Not enough room in dst, add NUL and traverse rest of src */ + if (n == 0) { + if (siz != 0) + *d = '\0'; /* NUL-terminate dst */ + while (*s++) + ; + } + + return(s - src - 1); /* count does not include NUL */ +} + + +#endif /* __linux__ */ + + +#if defined (EMULATE_SYSCTL) +//XXX missing prerequisites +#include <net/if.h> //openwrt +#include <netinet/ip.h> //openwrt +#include <netinet/ip_fw.h> +#include <netinet/ip_dummynet.h> +int do_cmd(int optname, void *optval, uintptr_t optlen); +#endif /* EMULATE_SYSCTL */ + +/* + * set or get system information + * XXX lock acquisition/serialize calls + * + * we export this as sys/module/ipfw_mod/parameters/___ + * This function get or/and set the value of the sysctl passed by + * the name parameter. If the old value is not desired, + * oldp and oldlenp should be set to NULL. + * + * XXX + * I do not know how this works in FreeBSD in the case + * where there are no write permission on the sysctl var. + * We read the value and set return variables in any way + * but returns -1 on write failures, regardless the + * read success. + * + * Since there is no information on types, in the following + * code we assume a length of 4 is a int. + * + * Returns 0 on success, -1 on errors. + */ +int +sysctlbyname(const char *name, void *oldp, size_t *oldlenp, void *newp, + size_t newlen) +{ +#if defined (EMULATE_SYSCTL) + /* + * we embed the sysctl request in the usual sockopt mechanics. + * the sockopt buffer il filled with a dn_id with IP_DUMMYNET3 + * command, and the special DN_SYSCTL_GET and DN_SYSCTL_SET + * subcommands. + * the syntax of this function is fully compatible with + * POSIX sysctlby name: + * if newp and newlen are != 0 => this is a set + * else if oldp and oldlen are != 0 => this is a get + * to avoid too much overhead in the module, the whole + * sysctltable is returned, and the parsing is done in userland, + * a probe request is done to retrieve the size needed to + * transfer the table, before the real request + * if both old and new params = 0 => this is a print + * this is a special request, done only by main() + * to implement the extension './ipfw sysctl', + * a command that bypasses the normal getopt, and that + * is available on those platforms that use this + * sysctl emulation. + * in this case, a negative oldlen signals that *oldp + * is actually a FILE* to print somewhere else than stdout + */ + + int l; + int ret; + struct dn_id* oid; + struct sysctlhead* entry; + char* pstring; + char* pdata; + FILE* fp; + + if((oldlenp != NULL) && ((int)*oldlenp < 0)) + fp = (FILE*)oldp; + else + fp = stdout; + if(newp != NULL && newlen != 0) + { + //this is a set + l = sizeof(struct dn_id) + sizeof(struct sysctlhead) + strlen(name)+1 + newlen; + oid = malloc(l); + if (oid == NULL) + return -1; + oid->len = l; + oid->type = DN_SYSCTL_SET; + oid->id = DN_API_VERSION; + + entry = (struct sysctlhead*)(oid+1); + pdata = (char*)(entry+1); + pstring = pdata + newlen; + + entry->blocklen = ((sizeof(struct sysctlhead) + strlen(name)+1 + newlen) + 3) & ~3; + entry->namelen = strlen(name)+1; + entry->flags = 0; + entry->datalen = newlen; + + bcopy(newp, pdata, newlen); + bcopy(name, pstring, strlen(name)+1); + + ret = do_cmd(IP_DUMMYNET3, oid, (uintptr_t)l); + if (ret != 0) + return -1; + } + else + { + //this is a get or a print + l = sizeof(struct dn_id); + oid = malloc(l); + if (oid == NULL) + return -1; + oid->len = l; + oid->type = DN_SYSCTL_GET; + oid->id = DN_API_VERSION; + + ret = do_cmd(-IP_DUMMYNET3, oid, (uintptr_t)&l); + if (ret != 0) + return -1; + + l=oid->id; + free(oid); + oid = malloc(l); + if (oid == NULL) + return -1; + oid->len = l; + oid->type = DN_SYSCTL_GET; + oid->id = DN_API_VERSION; + + ret = do_cmd(-IP_DUMMYNET3, oid, (uintptr_t)&l); + if (ret != 0) + return -1; + + entry = (struct sysctlhead*)(oid+1); + while(entry->blocklen != 0) + { + pdata = (char*)(entry+1); + pstring = pdata+entry->datalen; + + //time to check if this is a get or a print + if(name != NULL && oldp != NULL && *oldlenp > 0) + { + //this is a get + if(strcmp(name,pstring) == 0) + { + //match found, sanity chech on len + if(*oldlenp < entry->datalen) + { + printf("%s error: buffer too small\n",__FUNCTION__); + return -1; + } + *oldlenp = entry->datalen; + bcopy(pdata, oldp, *oldlenp); + return 0; + } + } + else + { + //this is a print + if( name == NULL ) + goto print; + if ( (strncmp(pstring,name,strlen(name)) == 0) && ( pstring[strlen(name)]=='\0' || pstring[strlen(name)]=='.' ) ) + goto print; + else + goto skip; +print: + fprintf(fp, "%s: ",pstring); + switch( entry->flags >> 2 ) + { + case SYSCTLTYPE_LONG: + fprintf(fp, "%li ", *(long*)(pdata)); + break; + case SYSCTLTYPE_UINT: + fprintf(fp, "%u ", *(unsigned int*)(pdata)); + break; + case SYSCTLTYPE_ULONG: + fprintf(fp, "%lu ", *(unsigned long*)(pdata)); + break; + case SYSCTLTYPE_INT: + default: + fprintf(fp, "%i ", *(int*)(pdata)); + } + if( (entry->flags & 0x00000003) == CTLFLAG_RD ) + fprintf(fp, "\t(read only)\n"); + else + fprintf(fp, "\n"); +skip: ; + } + entry = (struct sysctlhead*)((unsigned char*)entry + entry->blocklen); + } + free(oid); + return 0; + } + //fallback for invalid options + return -1; + +#else /* __linux__ */ + FILE *fp; + char *basename = "/sys/module/ipfw_mod/parameters/"; + char filename[256]; /* full filename */ + char *varp; + int ret = 0; /* return value */ + int d; + + if (name == NULL) /* XXX set errno */ + return -1; + + /* locate the filename */ + varp = strrchr(name, '.'); + if (varp == NULL) /* XXX set errno */ + return -1; + + snprintf(filename, sizeof(filename), "%s%s", basename, varp+1); + + /* + * XXX we could open the file here, in rw mode + * but need to check if a file have write + * permissions. + */ + + /* check parameters */ + if (oldp && oldlenp) { /* read mode */ + fp = fopen(filename, "r"); + if (fp == NULL) { + fprintf(stderr, "%s fopen error reading filename %s\n", __FUNCTION__, filename); + return -1; + } + if (*oldlenp == 4) { + if (fscanf(fp, "%d", &d) == 1) + memcpy(oldp, &d, *oldlenp); + else + ret = -1; + } + fclose(fp); + } + + if (newp && newlen) { /* write */ + fp = fopen(filename, "w"); + if (fp == NULL) { + fprintf(stderr, "%s fopen error writing filename %s\n", __FUNCTION__, filename); + return -1; + } + if (newlen == 4) { + if (fprintf(fp, "%d", *(int*)newp) < 1) + ret = -1; + } + + fclose(fp); + } + + return ret; +#endif /* __linux__ */ +} + +/* + * The following two functions implement getsockopt/setsockopt + * replacements to talk over a TCP socket. + * Because the calls are synchronous, we can run blocking code + * and do not need to play special tricks to be selectable. + * The wire protocol for the emulation is the following: + * REQUEST: n32 req_size, level, optname; u8 data[req_size] + * RESPONSE: n32 resp_size, ret_code; u8 data[resp_size] + * data is only present if ret_code == 0 + * + * Return 0 if the message wan sent to the remote + * endpoint, -1 on error. + * + * If the required lenght is greater then the + * available buffer size, -1 is returned and + * optlen is the required lenght. + */ +enum sock_type {GET_SOCKOPT, SET_SOCKOPT}; + +struct wire_hdr { + uint32_t optlen; /* actual data len */ + uint32_t level; /* or error */ + uint32_t optname; /* or act len */ + uint32_t dir; /* in or out */ +}; + +/* do a complete write of the buffer */ +static int +writen(int fd, const char *buf, int len) +{ + int i; + + for (; len > 0; buf += i, len -= i) { + i = write(fd, buf, len); + ND("have %d wrote %d", len, i); + if (i < 0) { + if (errno == EAGAIN) + continue; + return -1; + } + } + return 0; +} + +/* do a complete read */ +static int +readn(int fd, char *buf, int len) *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201508211624.t7LGOI0X055661>