Saturday, March 23, 2019

Too Many Rules

 on  with No comments 
In ,  
There is definitely a number of local rules in Firepower that makes the system unhappy, and that number is somewhere in the neighborhood of 50,000. At some point when you pass by 50,000, your scheduled SRU fails each day, with the following text to be found at System > Updates > Rule Updates.  Cisco Bug CSCv81754 talks about this, though it only mentions version 6.2.3.

This failure, which will happen every time your system attempts a SRU, causes other issues as well, such as you'll be unable to upgrade the system.  I discovered this because we had a change window last weekend to upgrade the system, and the Readiness Check fails because of this.

So I opened a case with TAC to see what our options are. Naturally, they want us to delete all local rules and be done with it. However, we're currently paying for the Emerging Threats rules, so that option is a non-starter.

But the email from TAC (which told me to use to do it) got me thinking. I could delete the oldest rules (2500 open and 2500 pro for example). But there's also a file in the bundle, deleted.rules, which are all of the obsolete rules which are safe to delete. We appear to have been disabling these, but not deleting them. There's also well over 5000 of them, so that will get me back on the right side of 50,000.

I extracted the SIDs out of the file, generated a list titled sidlist.txt on the FMC, and then run this command

cat sidlist.txt | while read line ; do -n $line --prune <<< PRUNE ; done

and restart the FMC.  The documentation and everything I read online said that using pmtool to restart services would do the trick, but that hasn't ever worked for me.

So what's going on here?  cat sidlist.txt outputs the list of deleted sids, which is then piped into a while loop that feeds one line at a time into The output of this command's help is as follows:

So with each sid, it runs -n ##sidnumber##  --pruneThe n switch specifies an individual SID, and --prune is required to actually remove it from the database. Finally the end, <<< PRUNE enters the text PRUNE where asks for it to confirm.

Not much documentation for exists on the web, and probably for good reason, direct manipulation of the database is always a dangerous task. The best I could find is this bug report which will give you an idea of how sparse it is.

Other ideas

This is obviously a stop gap meant to lower the rule count one time. However, there's a few other things ET subscribers can do. First, there is a rule set called et-compromised (or compromised.rules) that block traffic from known compromised hosts on the internet. These IPs are also available as an IP list, so you could install these as a feed under Object Managment > Security Intelligence Network Lists and Feeds in Firepower, or in the appropriate spot of your firewall, which would make the rules redundant and therefore able to be removed.

Next, there is a rule base called et-botcc (or botcc.rules), that has an equivalent feed.  This feed collects various feeds from, Spamhaus and Dshield, so if you have this feed or it's source feeds, the same applies.

Finally, when all else fails, you're going to have to just delete old rules. Until Firepower stops choking at 50k, here's a way to delete a number of old Open rules:

for i in {2000000..2005000}; do -n $i  --prune <<< PRUNE; done

This will delete the first 5000 rules from the Open Set, change the leading 20 to 28 to do the same for Pro rules.  You can find the SID allocation here.

Saturday, February 16, 2019

Default User Role in Firepower

 on  with No comments 
When I took over the Firepower system at work, we had a long standing issue relating to a couple of users. Everyone in Security Operations is an admin in the FMC, however, one of our SecOps guy could not see all the events in the FMC. He can see everything in the SIEM, just not in the FMC, verifying that this is indeed a FMC issue.  2 separate TAC cases had been raised before, neither of which got to the bottom of the matter.

One of the first tasks I had in Firepower was adding in the remaining IR guys who did not have accounts yet. One if the IR guy immediately noticed the same behavior as the previous mentioned SecOps guys. He could see some events, just not all of them.

Here's what happened. In the FMC, we had a default user role defined for RADIUS users under External Authentications. This user role had Restrictive Search enabled everywhere possible in the user role. User role permissions are cumulative, however it appears that Restrictive Search overrides everything.

Because this is the default role, it gets applied to all RADIUS authenticated users upon login. Even if you uncheck the role from their user profile later, it will get rechecked the next time they login. You can see that this is the case when you look at the user profile and see that the role is marked Externally Set.

We were hesitant to change the default user role to Security Analyst (Read Only) which is what the IR guys were getting because the more restrictive role was set up for users outside of the SOC to view events related to their devices and it was feared that a new non-SOC user in the future would accidentally get more access than they required.

So the answer was to create a new user role with zero access, set that as the default, and then add another role on top of that be it Administrator, Security Analyst (Read Only), or the previous mentioned custom role with the Restricted Search filters.

Saturday, February 9, 2019

Database Integrity Check Failed

 on  with No comments 
A quick tip for Firepower when you have a database error.  This seems to come up a lot especially if you're manually importing snort rules from an outside source such as Emerging Threats.

Every once in a while you'll get a database error that will cause an operation to fail. In my case, it's the scheduled backup. The web page doesn't give you a lot go to on at first glance, but under your schedule, it should tell you more at the bottom of the calendar.

If you ssh in and run, you'll see something like this:

[Tue Feb  5 13:15:50 2019][FATAL]  [missing eo reference]    database [sybase],table [rule_header] column [uuid], uuid [fcd5fc7a-2889-11e9-ab40-b1c5df20061d],rev_uuid [NULL]

indicating a bad database entry for a snort rule.

This is easily fixed by typing -db sdb -e "delete from rule_header where uuid ='fcd5fc7a-2889-11e9-ab40-b1c5df20061d';"

Pay close attention to the table (rule_header in this case). I've also seen EOPermissions, EORevisionStore, and EOContainerStore (or all 4). 

Again, this seems to only be a problem here for the Emerging Threats rules that I import, I've never seen it happen for the default rule set downloaded and installed automatically by Firepower.

TAC also likes to suggest the "slow repair,"

for i in `ls -1|egrep -v "\.BAK$"|sed 's/\....$//'|sort|uniq`; do -frames $i;done 

but I've never had that actually fix anything. Note here that you have to actually be in /var/lib/mysql/sfsnort for this to actually do anything despite TAC's insistence that it can be run from anywhere in the file system.

Saturday, January 19, 2019

Saturday, January 12, 2019

My New Favorite ISE Setting

 on  with No comments 
In ,  
And by favorite, of course I mean least favorite ever.

This one has been a thorn in my side for a while now. In the User Password Policy (Administration > Identity Management > Settings > User Password Policy), under Password Lifetime, there's a setting called "Disable user account after ____ days if password not changed." This setting is enabled by default, with a value of 60 days.

What happened is that I set up a pretty weak password policy for convenience including passwords that do not expire (it's a lab environment, and I only have the one local user account in ISE) but seemingly out of the blue, the account would become disabled. At one point, I even deleted the account and created a new one. Who notices it's been exactly 60 days? I'd reenable the account, and the next day it would be disabled again. The only thing I use this ISE local account for is the automated testing feature on my 802.1x enabled switches. This account becoming disabled leads to a long list of nasty red failed authentications in the Live Log.

For reference, here is the setting: