Date: Mon, 26 Jul 1999 16:36:28 -0500 (CDT) From: Joe Greco <jgreco@ns.sol.net> To: nate@mt.sri.com (Nate Williams) Cc: jgreco@ns.sol.net, nate@mt.sri.com, dillon@apollo.backplane.com, hackers@FreeBSD.ORG, freebsd-ipfw@FreeBSD.ORG Subject: Re: securelevel and ipfw zero Message-ID: <199907262136.QAA19479@aurora.sol.net> In-Reply-To: <199907262045.OAA21485@mt.sri.com> from Nate Williams at "Jul 26, 1999 2:45:13 pm"
next in thread | previous in thread | raw e-mail | index | archive | help
> > > > Having a per-rule limit means that you > > > > actually have a "IPFW_VERBOSE_LIMIT * number_of_rules_specifying_log" limit > > > > (assuming an attacker exploits multiple rules) rather than a limit of > > > > "IPFW_VERBOSE_LIMIT". > > > > > > I disagree. I have *tons* of firewall rules, and I don't want to have > > > to recompile my kernel if I modify my script at a later point and make > > > the rules 'more verbose'. > > > > Huh? > > > > I think you completely missed what I said - or maybe the opposite. > > > > Right now, if you have ten "log" rules, I can make your system log up > > to 10 * IPFW_VERBOSE_LIMIT rules (assuming I can generate violations for > > each rule). If you have 200 rules, it becomes 200 * IPFW_VERBOSE_LIMIT - > > again assuming I can generate appropriate violations. > > > > This eats up a grossly variable amount of disk space, which seems contrary > > to the whole concept of IPFW_VERBOSE_LIMIT. > > If I'm getting attacked such that I'm logging data, I *want* as much > information on the attack as possible. I realize this when I setup my > IPFW_VERBOSE_LIMIT 'limits' as well as the logging rules. > > If an attack takes is 1 rule 100 times, and the other 10 rules 1 time, > I'd like to see all of this. With a 'global' limit set to 100, I > wouldn't see these kinds of hits. > > Also, from analyzing attacks for years, most attacks have a pattern that > is *best* seen from seeing it hit a number of rules. By settin a > per-rule limit, you 'generally' can determine that an attack is going to > have a signature the same as the previous 'n' hits on that rule, but you > want to see the rest of the attack. > > I don't care to see a scan of the IMAP 1000 times when right after they > finish scanning it they also try to attack my POP3 port 1000 times. > Rather, I'd rather see the first 100 attempts on the IMAP port, then the > first 100 attempts on POP3, then the first 100 attempts on port 7, > etc... > > You get *better* information on per-rule limits than on a global limit. No, you simply get a finer-grained ability to select. > > If I'm an admin, I'm going to think "Well lets see, I want to store a > > month of bad packets in it. > > If you're an admin on today's internet, you'd realize that there is *NO* > way to get save a month worth of bad packets. Heck, at least once/week > I can't even store a day's worth of bad packets (I assume when we say > bad packets, we're talking about the If I'm an admin on today's internet (and I am), of course I realize that I'm not going to save a month worth of all bad packets, but I _can_ choose to hold onto a reasonable amount of logging information, which is what we are discussing. That is part of the _point_ of the IPFW_VERBOSE_LIMIT option... it allows you to sample the beginning of each 5 minute period, and you ought to be able to calculate the space required in a manner that allows you to guarantee that you have sufficient space. Now, if you're on an OC3c and you want to set IPFW_VERBOSE_LIMIT to 100000, and you reset the counter every 5 minutes, you still _ought_ to be able to determine that you require 2GB of disk space per day to keep those records. (And if you're getting 100K rejectable packets every 5 minutes, something is seriously wrong) > > I zero > > my counters every five minutes and there is a limit of 100 per five mins, > > and the average line length is 80 (or whatever) so therefore I need 80 * 100 > > * 12 * 24 * 30 or 70MB for my worst case scenario." > > Not. If the attack is the same, then the syslog error reporting will > same something like (this is a real message, with IP addresses change to > protect the guilty..): > > Jun 29 16:43:09 ns /kernel: ipfw: 64000 Deny UDP 1.2.3.4:53 4.3.2.1:54927 out via ed0 > Jun 29 16:43:44 ns last message repeated 3 times > > If you've got seperate attacks, then you've got too much stuff you're > logging. > > > Now given 200 rules, I can fill his disk in substantially less than a day. > > If you're logging that much information, then you're logging too much > information. In any case, you've got information overload, so adding a > global limit on the rules means you're losing as much (or more) > information than you're logging, which is just as bad (or worse). No. You're not necessarily logging too much information. I do log a whole bunch of shouldn't-happen scenarios, because if they do happen, it means something is either horribly broken or somebody is doing something horribly wrong. I don't want an event such as an outbound source 10-net address to happen... in theory it can't because I filter at my borders and use no 10-net stuff internally. But I damn well want to know if it happens ANYWAYS. > If you're worried about logging too much information, then don't reset > your counters every 5 minutes. Besides, you're losing information if > you're max'd out every 5 minutes anyway, so it really doesn't matter > when you zero out your counters. You just argued my point for me, thank you. I _want_ to see the things that I choose to log, and by definition once I pass IPFW_VERBOSE_LIMIT, then the information is beyond the point of being useful. > > I don't know what you mean by "make the rules 'more verbose'" but what I > > am advocating is not any change in what currently exists... I am talking > > about limiting the number of entries possible in a manner that would seem > > to be more consistent with the intent behind IPFW_VERBOSE_LIMIT. > > IPFW_VERBOSE_LIMIT is a per/rule limit. This is what is intended. Most > attacks don't hit the entire system on every rule in order to do a DoS > attack. Even so, each rule with eventually 'limit-out' and the system > will no longer log packets. This is a 'bad thing' from a security > standpoint, since at that point we are losing information. Okay, so what YOU really want is to not set up an IPFW_VERBOSE_LIMIT at all, and that's perfectly valid as well. > The idea behind IPFW_VERBOSE_LIMIT is to avoid DoS attacks, while still > allowing *MAXIMUM* data collection when a DoS attack is not happening. > > It's not there to make your logfiles small. If you want small logfiles, > turn of IPFW logging altogether. > > (I'm actually serious here.) Oh, that's a f'ing crock. What _I_ really want is to be able to GET the god damn information when the system is in securemode 3 and the damn IPFW has stopped logging due to the VERBOSE_LIMIT. You think I'm interested in turning OFF IPFW logging? You're nuts if you think so. I need to be able to log a reasonable number of things without logging so many that it turns into a DoS attack. I'm trying to figure out how to do that while retaining (or increasing) intended functionality - because I've clearly spelled out a broken (but quite legitimate) situation. The stated rationale behind adding VERBOSE_LIMIT, when it was added, was to reduce or eliminate the chance of a DoS attack. Now, the best way to eliminate the chance of a DoS attack is _algorithmically_. Giving someone a mechanism that allows them to determine the maximum resource commitment for a given scenario eliminates the chance for that DoS attack. Giving someone a mechanism that may use a variable quantity of resources is the typical programming practice that leads to a resource starvation DoS. In my own scenario, what I'm trying to do is to log a reasonable amount of stuff when something "odd" happens. I don't want someone to be able to take down my syslogd, but logging a hundred events every five minutes is not going to result in that. I can increase that number to whatever I feel is needed to make me "comfortable" that I will catch sufficient information about an incident. I also want to be able to catch multiple instances... so if someone attacks my server a half hour later, I need that counter to have been zero'ed (and I can presumably determine whether or not any new data is part of the previous attack or a new attack). I also want it to be completely automatic, so that if someone attacks my server while I'm asleep on the console, the machine does a reasonable job of trying to catch the information. What I absolutely need is a way to clear out the VERBOSE_LIMIT counter. If it is one counter, that is trivial. If it is a per-rule counter, I can at least write a C program to DTRT quickly via the ipfirewall ioctl interface. Just turning off logging is NOT a cool thing to suggest, unless you would not mind me suggesting that you don't need a firewall. > > > Case in point, I may at one time want to 'log' all connections to a > > > particular port, and then later ignore (no longer log) all the > > > connections to that port *except* to a particular host. Or, the reverse > > > may be true. In these cases, I don't want to have to recompile my > > > kernel to allow 'more logging' of the information. > > > > Huh? Why would you have to recompile your kernel? Assuming securelevel < > > 3, which is required regardless... you just delete the less restrictive > > rule and replace it with a more restrictive rule. > > I want *both* rules, and I want *full* information disclosure since I > want to know how I'm being attacked. See, knowing how I'm attacked is > useful since it helps me avoid future attacks, as well as allow me > potentially stock future attacks from happening by informing an ISP of > the behavior of one of their users. Maybe that person will get on > another ISP, or maybe he'll cleanup his act. Who knows, but sitting > idle means nothing happens. Okay, so you put in both rules. It doesn't matter. What stops you from doing # sysctl -w net.inet.ip.fw.verbose_limit=0 net.inet.ip.fw.verbose_limit: 100 -> 0 to temporarily remove the limit entirely, thereby allowing both entries to log? > > > I think a 'per-rule' limit works best, instead of a global limit. This > > > also works well in the case of rootkit attempt breakins, since they tend > > > to hit one rule multiple times, and then another multiple times, etc... > > > > > > With a 'global' limit, I may end up 'limiting out' on the first rule, > > > and then miss all the rules hit later on with the attack. > > > > Yes, you might. But you might miss them anyways, and I can run you out of > > disk space quicker. > > Sure, but you have to know alot about my box to do that, and if you have > that kind of information, I can run your box out of disk space just as > easily, it takes a bit longer. > > (And, it would take you months to run my box out of disk space in it's > current setup, which is a engineering tradeoff for maximum information > retrieval balanced with the history of attacks on my system, along with > engineering paranoia about how a DoS attack could occur.) > > Again, IPFW_VERBOSE_LIMIT is *NOT* there to make for smaller logfiles, > it's there to avoid a single kind of attack made on the system. These > kinds of attacks 'tend' to focus on a single port and/or rule. Also, > having a global count also leaves you 'wide-open' for doing a DoS > attack, following by a real attack. You would no longer have any idea > how the 'real' attack is occuring, so with the current implementation, > the attacker must know your exact FW configuration in order to attack > you well. (And, I would argue that *IF* they had my FW configuration > and/or rulesets, my firewall would be almost useless since in any > system, there are tradeoffs made for 'ease-of-use' vs. security.) I'm not arguing for smaller logfiles. You don't tell a person with a several gigabyte /var about how they are arguing for smaller logfiles. I'm looking strictly at the intended purpose of IPFW_VERBOSE_LIMIT. > > > > It also makes it more difficult to code in a bunch > > > > of "log" rules, since your periodic "zero" script has to know the number of > > > > each one, and if you just do an "ipfw zero rule1 rule2 rule3...." then you > > > > get a bunch of > > > > > > > > /kernel: ipfw: Entry rule1 cleared. > > > > /kernel: ipfw: Entry rule2 cleared. > > > > /kernel: ipfw: Entry rule3 cleared. > > > > > > > > each time you do this. > > > > > > Or, you do like I do, and have a cronjob 'zero' out the entire log > > > ruleset everynight right after it sends the results to me to analyze. > > > > Interferes with your accounting counters. You need to be able to do > > them individually to avoid that. > > I'm not disagreeing. ipfw zero rule1 also interfers with your counters > as well, but only on those rules. I count on separate rules right now. :-/ > > > However, at times (during breakins I happen to catch, or during ruleset > > > debugging sessions) I still want to have control over individual rules. > > > > > > > I would rather see something like > > > > > > > > /kernel: ipfw: logging limit reached, suspending. > > > > # /sbin/ipfw zerolog > > > > /kernel: ipfw: logging limit reset, resuming. > > > > > > This is essentially what I do. But, you can do 'ipfw zero' which > > > accomplishes the same thing. > > > > No, it nukes the accounting counters. > > I've not argued that point. > > > > However, this is really a side-issue. The last thing I'll say is that I > > > think a 'global' counter is a bad idea, and this is from using IPFW for > > > years in a real/deployed FreeBSD firewall situation. It would cause me > > > more trouble than it's worth, and provide 'less' flexibility than > > > currently exists (which I use). > > > > Then your argument devolves down to one of wanting a per-rule counter and > > you don't care if it is the accounting one or a new one... > > Correct. The existing infrastructure works for me. I don't use my FW > for accounting purposes (that's what the wireless bridge is for), so I > don't care if the counters are zero'd. But, I *don't* want to see the > system modified to replace the per-rule counters with a global counter > since it has some exteremely negative side-effects, the most obnoxious > is to lose 'valid' breakin information, which is the point of having > logging in the first place. Well, you certainly lose it now, anyways, with securelevel >= 3 unless you happen to be made aware of the attack in some other manner, because once you hit IPFW_VERBOSE_LIMIT, the system just stops logging. You haven't provided any helpful suggestions about how to deal with this. You can argue that you want per-rule limits, and you've done so, but I think it is safe to say that both ways have merits. > > And I've been using FreeBSD and IPFW in a real/deployed environment as well, > > and I keep hitting up against these goofy sorts of problems. It is much, > > much more usable than something like ipfw in 2.1R or earlier, but these > > little things still matter. > > The changes in 2.1 affected quite a bit, but all it required me to do > was to rewrite my FW rules. I saw very little 'functionality' > additions, just changes made that required me to re-write how I was > doing things. Since the firewall is a 'side' job of mine, it was a pain > in the butt to do it, but now that's (long) past. :) > > > > The real issue from your first email is '*should* ipfw rules be > > > 'zero-able' at securelevel 3'. I'm of two minds. The paranoid > > > administrator can't think of any bad things that could occur, but I can > > > imagine something bad happening if someone were allowed to do this, > > > although it's not completely concrete. I don't have the brain-cells to > > > dedicate to thinking about the full implications of a 'bad-guy' zeroing > > > out my rules. > > > > He interferes with whatever other features you've built into the system > > which rely on those accounting numbers. I agree - you probably don't > > want to do that. > > One could argue that accounting numbers in a firewall shouldn't be > trusted, but I won't argue that point since the firewall is often the > most 'natural' place to stick network accounting software. If you can't trust something in the kernel, then you just can't trust anything at all. > > Well, the 'right' thing is clear: pull the limit count out of the > > accounting count. This solves both the problem of zeroing accounting > > counters and potentially messing with other things, and also of letting > > people do anything 'negative' in securelevel 3. > > Is it the 'right' thing to do? I think that someone messing with the > 'logging' rule counts could be just as dangerous as someone messing with > the accounting counts. It is no more dangerous, and somebody zeroing (the only thing they should be able to do!) the logging rule counts would result in additional logging going on... this might mess up any sort of deterministic disk space calculation that you had going on, but if it was really an issue, you could also set up a sysctl variable or kernel #define that would forbid even this - which lands you back right where we are, if you choose that, but gives everyone else who needs it the option to do automated logging. > Again, if you're paranoid enough to have securelevel 3, you've got to be > paranoid enough to assume that someone *HAS* root access on your box, > and is going to do anything nasty they could. Yes, but causing additional logging to happen is going to be something that is likely to _attract_ my attention. > > By pulling the limit count out into a separate entity, nothing > > 'negative' can happen (because stopping the logging process is > > definitely negative, and being able to restart it without messing with > > other stuff is positive). > > I'm not sure this is an acceptable change either, but I'll leave that > discussion to the networking folks. > > > Now that we are agreed that the limit count needs to be divorced from the > > accounting count, we can debate whether it should be a per-rule or global > > limit. > > Hold on to your horses, Hoss. I haven't agreed to that. Don't be > putting words in my mouth. :) :) Well, then, how do you "fix" this problem? ... Joe ------------------------------------------------------------------------------- Joe Greco - Systems Administrator jgreco@ns.sol.net Solaria Public Access UNIX - Milwaukee, WI 414/342-4847 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199907262136.QAA19479>