Saturday, March 23, 2019

Too Many Rules

 on  with No comments 
In ,  
There is definitely a number of local rules in Firepower that makes the system unhappy, and that number is somewhere in the neighborhood of 50,000. At some point when you pass by 50,000, your scheduled SRU fails each day, with the following text to be found at System > Updates > Rule Updates.  Cisco Bug CSCv81754 talks about this, though it only mentions version 6.2.3.

This failure, which will happen every time your system attempts a SRU, causes other issues as well, such as you'll be unable to upgrade the system.  I discovered this because we had a change window last weekend to upgrade the system, and the Readiness Check fails because of this.

So I opened a case with TAC to see what our options are. Naturally, they want us to delete all local rules and be done with it. However, we're currently paying for the Emerging Threats rules, so that option is a non-starter.

But the email from TAC (which told me to use to do it) got me thinking. I could delete the oldest rules (2500 open and 2500 pro for example). But there's also a file in the bundle, deleted.rules, which are all of the obsolete rules which are safe to delete. We appear to have been disabling these, but not deleting them. There's also well over 5000 of them, so that will get me back on the right side of 50,000.

I extracted the SIDs out of the file, generated a list titled sidlist.txt on the FMC, and then run this command

cat sidlist.txt | while read line ; do -n $line --prune <<< PRUNE ; done

and restart the FMC.  The documentation and everything I read online said that using pmtool to restart services would do the trick, but that hasn't ever worked for me.

So what's going on here?  cat sidlist.txt outputs the list of deleted sids, which is then piped into a while loop that feeds one line at a time into The output of this command's help is as follows:

So with each sid, it runs -n ##sidnumber##  --pruneThe n switch specifies an individual SID, and --prune is required to actually remove it from the database. Finally the end, <<< PRUNE enters the text PRUNE where asks for it to confirm.

Not much documentation for exists on the web, and probably for good reason, direct manipulation of the database is always a dangerous task. The best I could find is this bug report which will give you an idea of how sparse it is.

Other ideas

This is obviously a stop gap meant to lower the rule count one time. However, there's a few other things ET subscribers can do. First, there is a rule set called et-compromised (or compromised.rules) that block traffic from known compromised hosts on the internet. These IPs are also available as an IP list, so you could install these as a feed under Object Managment > Security Intelligence Network Lists and Feeds in Firepower, or in the appropriate spot of your firewall, which would make the rules redundant and therefore able to be removed.

Next, there is a rule base called et-botcc (or botcc.rules), that has an equivalent feed.  This feed collects various feeds from, Spamhaus and Dshield, so if you have this feed or it's source feeds, the same applies.

Finally, when all else fails, you're going to have to just delete old rules. Until Firepower stops choking at 50k, here's a way to delete a number of old Open rules:

for i in {2000000..2005000}; do -n $i  --prune <<< PRUNE; done

This will delete the first 5000 rules from the Open Set, change the leading 20 to 28 to do the same for Pro rules.  You can find the SID allocation here.

Saturday, February 16, 2019

Default User Role in Firepower

 on  with No comments 
When I took over the Firepower system at work, we had a long standing issue relating to a couple of users. Everyone in Security Operations is an admin in the FMC, however, one of our SecOps guy could not see all the events in the FMC. He can see everything in the SIEM, just not in the FMC, verifying that this is indeed a FMC issue.  2 separate TAC cases had been raised before, neither of which got to the bottom of the matter.

One of the first tasks I had in Firepower was adding in the remaining IR guys who did not have accounts yet. One if the IR guy immediately noticed the same behavior as the previous mentioned SecOps guys. He could see some events, just not all of them.

Here's what happened. In the FMC, we had a default user role defined for RADIUS users under External Authentications. This user role had Restrictive Search enabled everywhere possible in the user role. User role permissions are cumulative, however it appears that Restrictive Search overrides everything.

Because this is the default role, it gets applied to all RADIUS authenticated users upon login. Even if you uncheck the role from their user profile later, it will get rechecked the next time they login. You can see that this is the case when you look at the user profile and see that the role is marked Externally Set.

We were hesitant to change the default user role to Security Analyst (Read Only) which is what the IR guys were getting because the more restrictive role was set up for users outside of the SOC to view events related to their devices and it was feared that a new non-SOC user in the future would accidentally get more access than they required.

So the answer was to create a new user role with zero access, set that as the default, and then add another role on top of that be it Administrator, Security Analyst (Read Only), or the previous mentioned custom role with the Restricted Search filters.

Saturday, February 9, 2019

Database Integrity Check Failed

 on  with No comments 
A quick tip for Firepower when you have a database error.  This seems to come up a lot especially if you're manually importing snort rules from an outside source such as Emerging Threats.

Every once in a while you'll get a database error that will cause an operation to fail. In my case, it's the scheduled backup. The web page doesn't give you a lot go to on at first glance, but under your schedule, it should tell you more at the bottom of the calendar.

If you ssh in and run, you'll see something like this:

[Tue Feb  5 13:15:50 2019][FATAL]  [missing eo reference]    database [sybase],table [rule_header] column [uuid], uuid [fcd5fc7a-2889-11e9-ab40-b1c5df20061d],rev_uuid [NULL]

indicating a bad database entry for a snort rule.

This is easily fixed by typing -db sdb -e "delete from rule_header where uuid ='fcd5fc7a-2889-11e9-ab40-b1c5df20061d';"

Pay close attention to the table (rule_header in this case). I've also seen EOPermissions, EORevisionStore, and EOContainerStore (or all 4). 

Again, this seems to only be a problem here for the Emerging Threats rules that I import, I've never seen it happen for the default rule set downloaded and installed automatically by Firepower.

TAC also likes to suggest the "slow repair,"

for i in `ls -1|egrep -v "\.BAK$"|sed 's/\....$//'|sort|uniq`; do -frames $i;done 

but I've never had that actually fix anything. Note here that you have to actually be in /var/lib/mysql/sfsnort for this to actually do anything despite TAC's insistence that it can be run from anywhere in the file system.

Saturday, January 19, 2019

Saturday, January 12, 2019

My New Favorite ISE Setting

 on  with No comments 
In ,  
And by favorite, of course I mean least favorite ever.

This one has been a thorn in my side for a while now. In the User Password Policy (Administration > Identity Management > Settings > User Password Policy), under Password Lifetime, there's a setting called "Disable user account after ____ days if password not changed." This setting is enabled by default, with a value of 60 days.

What happened is that I set up a pretty weak password policy for convenience including passwords that do not expire (it's a lab environment, and I only have the one local user account in ISE) but seemingly out of the blue, the account would become disabled. At one point, I even deleted the account and created a new one. Who notices it's been exactly 60 days? I'd reenable the account, and the next day it would be disabled again. The only thing I use this ISE local account for is the automated testing feature on my 802.1x enabled switches. This account becoming disabled leads to a long list of nasty red failed authentications in the Live Log.

For reference, here is the setting:


Saturday, December 29, 2018

802.1x in GNS3

 on  with No comments 
In , ,  
After tiring of the trips up and down the stairs dealing with a physical 3750 switch and Windows 7 workstation testing 802.1x, I decided to give it a shot virtually. And I'm pleased to report that it works like a champ.  This post is not a how to guide on setting up 802.1x, it's to show another complex configuration topic that can be done exclusively in the virtual domain.  And while I've never done it personally, I've seen elsewhere that it's also possible to do in EVE-ng.

I setup a basic topology in GNS3. In the light blue oval is the connection to the physical network.  The cloud connects to VLAN10 on my physical 3750 switch, which is the home network. There's a generic GNS3 switch there just in case I want to tie anything in at that point in the future, but at this point it's unnecessary and not really being utilized. Next we have BB2, which is a IOU router (i86bi-linux-l3-adventerprisek9-15.4.1T.bin if you care) that provides a degree of separation between the "real network" and the test area. SW11 is a L2 IOU image (i86bi-linux-l2-adventerprisek9-15.2d.bin) playing the role of the supplicant in the chain.  And finally we have fn-ws70008, which is a domain connected Windows 7 workstation which runs in qemu on the GNS3 VM.  The Yellow section branches out into another area of the topology that isn't relevant to this discussion.

Be advised that using a nested VM like this (qemu VM running inside of the GNS3 VM) is horrible from a performance standpoint. However the performance is not bad enough that delays lead to timeouts, so I can live with it for now.

At this time, I'm using the Windows 7 VM to confirm both 802.1x and MAB operation. I had a basic VPCS host that I tried to use for MAB, but for whatever reason, the switch didn't want to play nice with that host so I just set it to try both but prefer 802.1x on the Windows 7 workstation so I could see both play out in the ISE logs. If you can get a MAB authentication to trigger with a VPCS host or a router, I'd love to hear about it.

Turning our attention over to ISE, the next 2 images show the basic NAD configuration for SW11. Nothing out of the ordinary here, straight out of the documentation.


The switch configuration comes straight from the SISAS certification guide in Chapter 12.  You can also find a great stand alone PDF right here from Cisco.  As far as I recall, everything in the cert guide and the PDF work in this version of the L2 IOU image except for LLDP device tracking (Edit: its working now with no changes on my part.  No idea why it didn't before). CDP and DHCP tracking work, so that's definitely not the end of the world.  One caveat to keep in mind is that ip routing is enable by default on these switches which led to a short session of head scratching when I couldn't reach anything off of the local network from the switch the first time I started seriously labbing with it.

From what I can tell, the L2 IOUv works great as well, however I haven't labbed to seriously with it yet since the L2 IOU image does everything I've asked of it and I've been using it since before IOUv was a thing. As another note, there is a significant syntax difference with IOU 15.6 so I'm holding off on that one as well.

Configuring ISE for 802.1x is also straight from the SISAS certification guide, so no need to beat a dead horse rehashing it here.  Chapters 10 and 11 cover AuthZ and AuthC policy creation.  So with all that completed, let's check the Live Log in ISE. For the record, I'm running ISE 2.0 with no patches applied.

Here we'll see the workstation performing MAB and 802.1x authentication successfully. You'll also note the switch doing the automate-tester authentication with the local-bob account. For ease of use, I have an AD account called domain-bob, a local account on ISE called local-bob, and each router has a username called router-bob. Thanks to Keith Barker inspiring me to name all the users Bob, though I at least saw the potential for confusion with both the domain user and ISE user being just "bob."

And as one last show of compatibility, I've configured sw11 to send syslog to my local Splunk server. I do use the ISE interface for debugging as much as possible while labbing, as you should when you're trying to learn everything about ISE, having one location for long term log retention for troubleshooting purposes is very valuable when the problem involves multiple devices, and/or you want to look back at the messages for a similar situation that did work.

BB2 is also configured with ip-helper to forward DNS requests to one of my Windows domain controllers, both of which exist outside of this GNS3 topology. That also works flawlessly. 

So bottom line is that I'm really liking these IOU images (both L2 and L3) and they're almost fully meeting my labbing needs.  Since more recent ISE versions appear to work in GNS3 as well, you could theoretically work through the SISAS almost completely in GNS3 at this point. Connecting the dots in the exam topics tell me I still want to know ACS, though I cannot say for sure. It's not specifically listed on the topics and I haven't taken the exam yet. But the exam is written for ISE 1.2 (which doesn't have TACACS+ support) and the topics do say implement device administration with both TACACS+ and RADIUS.

Next up, 802.1s on NX-OS using the Cisco Nexus 1000V switch image, though I'm not sure when I'll get around to that. Nexus switches do not appear anywhere in the security track (yet) but I do want some exposure to them. Maybe I can integrate a 1000V image into ESXi and ditch the nested Windows 7 VM.

Saturday, December 22, 2018

Troubleshooting With Near Zero Access

 on  with No comments 
In , ,  
Early one morning last week I attempted to RDP into my lab to test something out I was looking into at work. Access to my terminal server was fine, but from there, I was unable to access any other system on my network. Every system that I attempted to RDP into came back stating that my user account was unauthorized for RDP access on that system. The user is a Domain Admin so there should be no reason for that. Not too long after, I noticed that the terminal server was asking for a username and password for everything with is out of character for my user account. And after accounting, I get access denied errors for anything requiring elevated privileges. My first thought was that my network was compromised.


Sunday, December 16, 2018

Saturday, August 12, 2017


 on  with No comments 
In ,  

1. What kind of Questions should I expect to see on the CCNA exam?

All the information that is known about the exam is available on the Cisco Certifications pagerelating to the exam in question. The Cisco NDA forbids us from telling you anything more. This includes how many of each type of question we saw, whether or not a specific topic is covered heavily, and what specific questions we were asked.

2. How much does the CCNA exam cost?

You can find the cost of the exam in your country at the above mentioned site where you register. Again, don't take anyone else's word for it. You're going to pay Pearson Vue to take the exam, not me. So it only makes sense to see what they say it costs, right?

3. How do I register to take the exam?

Cisco exams are proctored Pearson VUE. You will find everything you need to know about registration and exam locations on their site. Don't just take the word of some random person on Facebook, get the information directly from the authoritative source.

4. What book should I read for the CCNA?

I've blogged about this before very recently, and you can find that post elsewhere on my blog. But to summarize, I'm always going to recommend the Official Certification Guide from Cisco Press. I don't care if someone said Wendell Odom is too dry, or the books are too long. There's a lot to know, and you may as well understand now that there are no shortcuts.

Many like Todd Lammle's book because it's shorter and it reads more like a stand up comedy routine.  First, Lammle assumes you have a certain level of knowledge already before you pick up his book.  Odom doesn't.  Lammle gives you the bare minimum needed for the exam.  Odom doesn't.  If passing the exam is your only priority, you're in for a rude awakening on your first day on the job. The router that's down isn't going to go easy on you because the root cause wasn't a CCNA exam topic.

You can find a large list of free resources around the Internet to assist your studies.

5. I have a question about my final results.

There's a great post over at the Cisco Learning Network titled Demystifying the Cisco Score Report. It says it's posted by the Certs & Lab Admin, so I'm going to assume this is legit. And while this particular breakdown is for a CCIE Written exam, pretty much everything that is said in the post applies to any written exam.

6. What degree should I get for networking?

Nobody can speak for every hiring manager out there. Every one is different, so there is no answer for this. In my experience, a college degree was a yes/no question without too much focus on what the degree was in. Some hiring managers don't care about a degree while others only care that you have one, and others still want it to be something specific. I chose a degree that closely matched the work I wanted to do in the future. You can't go wrong with that way of going about it. One thing I can guarantee is that nobody is looking specifically for somebody with a masters in Botany to work in their NOC.

7. Why did I get banned from the CCNA group?

The group has rules, and the rules are enforced. Its nobody else's fault that you either posted without knowing the rules, or you knew the rules but chose to disregard them. Every Facebook group, Website, Online Forum, etc. has rules and it's up to you, the user, to seek them out and follow them. This is true with anything you do in life.

8. Will you do my homework for me?

No, we absolutely will not do your homework. Do your own homework. I already know the material and your homework is one of the steps involved in your education. You want to pas the exam and earn the certification? Then you need to learn this stuff, not just write down what I told you. Obvious homework questions will be deleted, and the poster will be mocked. I don't care if you say it's not homework, we know homework when we see it.

Besides that, half of the "answers" I see given to homework questions are wrong anyway.

9. Which certification pays more, Cert A or Cert B?

Follow your heart, not your greed. If you want to strike it rich, maybe come up with the next Google or Facebook.  Every certification path has great opportunities, and you'll be paid well in any IT specialty if you do your job well. And you will do your job a lot better if you're doing something that you actually enjoy doing. If you don't know what you enjoy doing yet, you should be spending more time getting a job and gaining experience and less time worrying about 3 jobs ahead.

Bottom line, if it wasn't a marketable job skill, Cisco wouldn't bother maintaining a certification program for the skill.

10. What is router?

This question, and any basic question like it, doesn't have any place in a Facebook group or an online forum. It's something that you can easily Google, and not waste everyone's time. In case you're still unsure how, I wrote a post on how to Google things. Try it sometime, you may like it.
"What is the CCNA" is especially offensive. Why did you join our group if you don't even know what it is? If you don't know, then Google it.

When in doubt, try this post on how to ask better questions. Better questions get better answers. Take note when you will not show any router configs but still want us to figure out why can't ping across your 5 router topology.

11. Any question that conntains "Class A," "Class B," and/or "Class C".

Let me stop you right there. Classful networking is a relic of ancient days, and we simply don't use it any more. CIDR and VLSM aren't something that's off in the distant future (insert IPv6 or Duke Nuke 'em Forever joke here), they are the here and now. If your "practice test" or study guide that you obtained from some shady website talks about classful networking, you're going to fail.

12. I don't like the rules, who can I complain to?

Basically nobody. The admins set and enforce the rules. And as long as we do not violate any part of Facebook's TOS, that will not change. If you dislike the group that much, there's certainly other groups elsewhere that you would like better. Try one of them.

Of course, if you do want to scream at somebody, then by all means send me a private message on Facebook, even if it was a different admin that annoyed you. I've actually grown to enjoy the abuse.

13. I was treated unfairly by an admin, who can I complain to?

See question #12. I have nothing but respect for the other admins of this group, otherwise they wouldn't be admins. It's not very likely I'll take your side over them.

But again, feel free to scream at me all you want on Facebook.  I love the abuse.

14. Will the CCNA help me find a job?

The same goes for this as it does for the degree question. While experience trumps all, certifications and a degree will help you stand out from other similar candidates who you are otherwise similar to. But at the end of the day, it's up to the individual company to decide what they want in a candidate. Some want a degree, some want certifications, some want something else still. The job posting will tell you exactly what they're looking for in a candidate.

The bottom line is that having a degree and a CCNA helped me get my first job in I.T. Everyone had zero experience at one point, and yet the senior members of the group are all working in the field, right?

15. I has cert, job me now!

Sorry, but this isn't a jobs board, and these types of posts can and will drown out everything else if we allow them.. Since there's really no such thing as a "CCNA Job" despite what many posters say, there's no connection here. If you wish to explore career opportunities in your area, seek out a group dedicated to that topic.

16. I'm the network administrator or engineer and everything is broke. Help me!

Let me guess, you passed the CCNA by using Testking, and somehow lucked your way through the job interview. The hiring manager must have figured they could pay you less since you really didn't interview well. Now you see why you shouldn't have cheated. You can't afford my hourly rate as a consultant, so pick up a book and start learning. You're in for a long night.

Saturday, July 22, 2017

That'll Never Work!

 on  with No comments 
In , ,  
While reading through a discussion on Slashdot today, someone linked a thread from 1999 as proof that the Slashdot community has always been pretty dumb collectively.  The thread was on the upcoming release of VMWare, a class of products that clearly hasn't been heard of on PCs at the time.

The thread can be found here.  It's amazing how a concept so ingrained today appears to be so foreign just 18 years ago.  My personal favorite is the very bottom comment "a great idea ROFL."