Security Ripcord

Once again I find myself pointing to tweets by Richard Bejtlich. This time it was actually a retweet of Hogfly who runs the Forensic Incident Response blog. Hogfly recently pointed out an article in Aviation Week titled “China’s Role In JSF’s Spiraling Costs“. This article demonstrates the actual cost for a specific project associated with industrial espionage, nation state infiltration of critical infrastructures, and general criminal activity. A blog post by Richard Stiennon, titled “The first thing we do, is hack all the lawyers“, also showed how these same threat agents are leveraging third-party relationships to impact specific projects. I like these articles because they provide specific numbers relating to cost of impact. “…$40 billion…” “…costs at tens of billions of dollars…”

These two examples show both ends of the information technology (IT) spectrum. The defense contractors responsible for the Joint Strike Fighter (JSF) should have had decent security in place to include security testing and monitoring. The majority of law offices will not have security practices that met these standards. In most cases the IT security considerations employed by law offices will actually fall at the other end of the spectrum: operating without significant information security considerations. But, in the end the result of both cases was the same.

I believe that it is pretty safe to conclude that the threat agents responsible for these and similar breaches did not just attach to the networks and start exfiltrating data. Rather, these successful attacks very likely required the use of a combination of known and discovered exploits to gain access, persistent, and propagate within these networks. It is also logical to conclude that their activities generated system and network-based artifacts that outlined their activity, even if that activity mimicked normal and authorized operational activity. Understanding these system and network-based artifacts is an important step to preventing and detecting attempts to infiltrate a network. Another key component is detecting exploitable vulnerabilities before they can be leveraged against the resources in a network.

Early detection is why security professionals encourage network mapping, vulnerability scanning, penetration testing, research and development, and monitoring. Security teams that provide penetration testing services should have an understanding of the techniques applied by current and past threat agents. They should also have a keen eye for leveraging the resources and services to their advantage to demonstrate the ingenuity attackers will implement after the initial compromise of a network. Tom Liston of InGuardians has a plethora of stories that demonstrate how he circumvented good security implementations using common sense and experience with a wide variety of technologies. The result of such penetration testing will generate system and network-based artifacts that can be leveraged to train a organization’s security and information technology administrators to detect and identify similar activity.

Research into technologies deployed in test networks and off-line implementations provide valuable information without impacting an organization’s business assets. It also reduces the cost of effort by providing beneficial information to all businesses employing a technology rather than a single instance where the organization hordes the information to prevent distribution to threat agents. When the results of research are presented to security professionals, IT administrators, IT management, and corporate executives all of these parties benefit from the knowledge and are able to leverage the information to assess and use it to improve their security and effectiveness of their business assets.

While writing this Richard tweeted (if you are not following him then you should stop reading and take care of it right now) to additional insights that are relative to my point. “Exploits aren’t as important as some think. I worked cases where not even active intruders on a corp network inspired appropriate concern!” and “Tech people should consider that IT and sec are one of many factors that mgt weighs. I fear it’s underweighted, but it’s not tech’s call.” Both of these are very true and have played a significant role in the persistence of many breaches including those pointed out at the beginning of this post. I too have experienced active intrusions with manual (rather than automated) system interactions, supported by specific system and network-based artifacts, that were downplayed and eventually treated as a malware infestation.  Even the experiences of multiple incident response professionals was not enough to change the opinions of the administrations, IT managers, and executives of these organizations. This attitude was born out of inexperience in the types of activities associated with a network’s initial compromise, persistence of the compromise, propagation to additional resources, and exfiltration of data.  Experience that might have been achieved by monitoring the data produced by penetration testing. Of course, it also requires a mind that is not specifically limited by business restrictions and is open to new possibilities that are born out of sublime and criminal research into information technologies.

Go forth and do good things,

Don C. Weber

It is more than obvious now that my ShmooCon talk, Looking into the Eye of the Meter, was canceled.  Kelly Jackson Higgins in her  Dark Reading article Researchers Postpone Release Of Free Smart Meter Security Testing Tool did a good job describing what InGuardians and I can say about the topic. But even one week later there is not much news or discussion about this. Robert Former wrote an interesting blog post, Security researchers: Spawn of Satan, Necessary Evil, or Security Salvation?, about his opinion of the event. I liked it, but I am biased (Free SMACK!! Indeed.). Other than that the Smart Grid lists and media feeds have been very quiet about the whole issue. It seems to be turning into the little meter talk that faded into the night. I cannot say if this is the goal of the vendor that asked us to pull the talk, but I can say that it was my fear when they asked us to pull it from the ShmooCon venue.

I cannot provide any more details about the vendor’s issues until we speak with the vendor and get some things cleared up. I want to talk a bit about my opinions on why we decided to pull the talk at the request of a vendor. Of course, these opinions are my own and do not reflect the official opinions of InGuardians, Inc.

I wrote the Smart Meter presentation to educate. I wanted to educate security professionals that will be dealing with the fast proliferation of embedded devices. I wanted to show that following a good testing methodology and implementing the hardware analysis basics well could lead to findings that will strengthen future development and implementation of a product and associated services. ShmooCon might be considered a “hacker” conference, but the people attending this event are security professionals and researchers that will be providing guidance to information technology deployments throughout the world. By educating these people I am not teaching the public how to hack the Smart Grid, I am informing them on the basics for security assessments and providing them with the skills to pass on to their teams and organizations.

In addition to security professionals I was hoping to educate the Smart Grid-related vendors and implementors. The Smart Grid industry has come a long way in the last three years. Most of the vendors and utilities have been listening to security professionals (internal and external) and addressing issues as quickly as they can. As with all industries there are individuals and even teams of people who are afraid of information disclosure and security tools. Basically, the Smart Grid industry, although moving as quickly as they can, are still new to the positive impacts that research, education, and assessment tools can provide to their efforts. They are not familiar with Wright’s Law: “Security will not get better until tools for practical exploration of the attack surface are made available.” I believe that Richard Bejtlich put it very well in a tweet posted as I was arriving in Washington, D.C.: “Proponents argue that releasing weaponized exploits will have long term positive impact; critics worry about short/med term negative impact.” Although I was not intending on releasing a “weaponized exploit” the statement hits very close to home for all security-related research and tools.

InGuardians has experienced the effects of security-related misconceptions within the Smart Grid arena. In fact, I almost didn’t get hired at InGuardians because of the fear, uncertainty, and doubt raised by the Associated Press article: ‘Smart’ meters have security holes. The title alone was enough to raise a stir within the Smart Grid industry not to mention the fact that the reporter left out many of the positive statements made by Joshua Wright. I now have a good understanding of how Josh might have been feeling at this time. I respect him immensely for his continued courage and desire to educate.

Because of Josh’s, InGuardians’, and other experiences, I approached presenting a Smart Meter talk at ShmooCon very carefully. I first had to sell it to InGuardians. I provided our team with an outline of my presentation idea with specific emphasis on the mitigations already employed by many meter vendors and third-party solutions. Mike Poor (yes, Tom Liston, it is Mike Poor’s fault) especially liked the idea as our approach would replace Fear, Uncertainty, and Doubt with Understanding, Certainty, and Knowledge (TM, I think. ). With InGuardians’ approval I had to sell the optical toolkit (the actual basis for my presentation) and the presentation content to the Smart Grid industry. I did this by providing access to the toolkit to our contacts at as many Smart Grid vendors as I could. These people were provided access to the toolkit on January 8, 2012 (approximately, I would have to double check for accuracy). I received immediate positive feedback from the list. Robert Former of Itron provided me with an understanding of the impacts of the tool to Advanced Metering Infrastructure (AMI) solutions and helped me understand implementable mitigations. Ed Beroset of Elster provided me with code modifications to make the tool more stable and usable by meter research and development teams. Several other individual, who have asked not to be named, provided us with their thoughts on the tool as well. As we did not receive any complaints about the release of the tool we continued with developing the presentation. After intense internal and legal review, I provided a draft presentation to the same group on the Monday before the presentation. This gave the group at least five days to provide us with their input about the content. The same individuals that provided me input about the tool also provided me input about the content of the presentation. All of the input was positive and really helped tighten up the concepts and content of the presentation.

We did not hear any objections to the toolkit or presentation until the first day of ShmooCon. One of the vendors on our list contacted us with generic objections to the content and venue of the presentation. As we had almost two full days to work through their objections InGuardians began a concerted effort to identify and address their issues so that the information could be presented and the toolkit released. I am very proud of the InGuardians team and the professionalism we displayed during these discussions. There were times when I wanted to call it and move forward as planned, but cooler heads prevailed and we adhered to the request of the vendor. And looking back, keeping a cool head at these times is the key to responsible disclosure, education, and maintaining positive relationships with our vendors and clients. InGuardians strives to be an organization of thought leaders and information security professionals that can be turned to for guidance and expertise. Of course we make mistakes, but we are quick to own them and address them as swiftly and professionally as possible. With these ideals in mind we could not move forward with the tool release and presentation until all parties were comfortable. It could be said that trying to please everybody is a recipe for disaster, but we cannot ignore direct requests for additional information. Because in the long run it provides each party with a better understanding of the positive and negative impacts of publicly releasing information and toolkits relating to specific products or solutions.

The result of all of this is that my Smart Meter presentation and the release of the Smart Meter Assessment Communication Kit (SMACK) has been delayed until further notice. I have already received an offer to give the presentation at the next Smart Grid Security Summit in Atlanta. I am hoping we can work through the vendor issues before they have to fill the slot with another presentation. I am also hoping to present this at another information security-related conference such as Black Hat USA and/or DefCon so that this information can educate the future security professionals that will be moving into Smart Grid-related positions or conducting authorized research on Smart Grid solutions. Of course this requires that the presentation is accepted for these venues.

I am also hoping the the vendor will come forward with a statement that they requested us to hold the presentation and release of the toolkit and InGuardians worked with them in good faith. This would show the public that they have a concern about the information being provided and that all parties are working to address the issues and concerns before moving forward. This would have a positive impact on the Smart Grid industry and security researchers.It would demonstrate that we are working as a team and not playing a fancy political game to suppress information or teach criminals how to hack the Smart Grid.

Go forth and do good things,

Don C. Weber

All of the opinions expressed in this blog post are mine alone. They do not represent the opinions of my employer, InGuardians, Inc.

When I started working for IBM’s Emergency Response Team I was a little intimidated about walking into a client’s environment and quickly providing incident response leadership. Luckily I was trained by Chris Pogue and Harlan Carvey to consider three things when I got on-site:

• What are you trying to answer?
• What data do you need to answer it?
• How do you get that data?

Obviously the first question is the hardest to answer (although, perhaps, not the most challenging). I often find myself having Car Talk-style conversations where you talk about the situation for an hour or two and then somebody says something to the effect of: “Oh, yeah. And we have this service running over here.”

It always amazes me how similar many of the security issues are across companies with decent security programs and teams. The security teams work hard to identify situations before they happen but incidents still occur and the teams struggle with pulling all of the information together. Disciplined incident response comes with experience. Not just experience within the security team, but experience throughout the organization. Most companies have information technology (IT) divisions with silo-ed experience (I am not saying it is bad, just natural). The web application developers and administrators know the web applications and web logs. Database administrators know the database logs, how verbose they are, how to retrieve them, and what can be turned on and off. Network administrators know what is logging where, what has been disabled, what can be turned up. Etc, Etc. Of course there is some overlap, but this example demonstrates that a lot of internal interaction may be required to initiate an incident response.

This silo-ing can complicate things but it is often manageable. The situations that prove to be more challenging are those that involve external services and equipment. Security teams are not usually consulted when service level agreements with external entities are formulated. Any third-party organization worth their salt will be prepared to provide information and action during critical times. But experience is the only way to determine the proper channels and methods of request to initiate the actions necessary to facilitate an efficient incident response. Acquiring data, removing services, increasing logging, and physically accessing a facility are just a few of the things that could increase the gap between incident identification and response.

Another thing that comes into play during an active incident response is the review of unique data types. I have assisted in several instances where an incident involved a web application running on an Apache server. The Apache web logs did not include any POST information which may have provided artifacts relating to the incident. As these web logs were the primary method for detecting anomalies the security team quickly realized that with the default configuration they may have been missing a few things. Digging a little deeper by including web application administrators and developers additional logs were identified. In a recent case the development team used Apache log4j to produce excellent activity information for debugging. As it did not introduce a performance hit to the server the developers just left the debugging code alone. The end result was a detailed log of the web server activity. The only real hick-up was that each entry was undocumented and multi-lined making it difficult to review by common automated processes. But, in the end, very helpful.

So, again, we come back around to the same incident response-related conclusion: Preparation is KEY. Understanding how things interact and who is involved is critical to reducing the time involved during each step of an incident response. But where to start with preparation? The results of an actual incident or a penetration test provide excellent examples and supporting data. Incident response scenarios are also very helpful but can be difficult to design and get everybody involved. To be honest, I have been a big advocate of “incident response scenarios” and I have told many people to develop them. But, looking back, I realize how hard it is for an organization to do this. Heck, it was hard for me when I was specifically thinking about and devoting time to developing a good incident response scenario. Therefore I am going to change my future recommendations from  developing incident response scenarios to performing system walk-throughs.

I made this decision when I found myself leaning back in a chair, looking at a white-boarded diagram of a system (by this I mean a group of resources combined for a specific purpose), and trying to determine what we needed to understand what might have happened during an incident. It was excellent because I also found myself and the security team asking questions to which we did not have immediate answers. I also found myself thinking about how I would have accomplished a specific set of actions relating to the incident. Coming up with these possibilities lead me to asking additional questions and identifying other resources-of-concern and data that could be used to identify useful network and system artifacts.

Therefore, my recommendation is that security teams periodically perform a system walk-through for various IT implementations in their organization. See how systems walk-throughs work and let us know your experiences. Pick one thing at a time when you start and see where it takes you. Focus on the things that will provide you information about a specific activity. Brain-storm situations and talk about the system and network-based artifacts that would be helpful. Folllow up on these ideas and pull in personnel to provide additional information when necessary. This will increase their experience and begin the collaboration process that may prove vitally important in future incidents. I also believe that these interactions will help identify risks, weaknesses, and vulnerabilities that can be easily addressed and increase the overall security of the organization.

Go forth and do good things,

Don C. Weber

Mentoring can be a powerful learning tool for learning specific topics. I have been thinking about mentoring a little bit because I have often found myself thinking that a mentor would be beneficial to my technological and managerial growth.  From my experiences I have determined there are a few requirements to setting up a good mentoring situation. Now, I am not a professional mentor, so, your mileage may vary.

• Focus: a broad range of mentoring subjects do not really work. A mentoring experience should be focused on a specific area.
• Time: both mentor and mentee must have time and inclination for the mentoring experience. Mentoring requires consistent meetings and reviews.
• Experience: A mentor must have knowledge of the subject and mentoring. A mentee must be able to learn the subject and be willing to branch out without encouragement.
• Desire: Both parties need to have a strong desire to participate from the beginning to the end of the mentoring experience.
• End State: both mentor and mentee must have an understanding of when the mentoring experience will end.  This may be after a specific set of goals is reached or a after a specific time period.

Personally, I don’t think that a blossoming Information Technology (IT) Security professional needs mentoring outside of the educational practices implemented by their place of employment. But that is the key: place of employment. Without a job it is difficult to implement the requirements that I listed above. Obviously this means that my list should include the additional requirement of: a job. Of course, a job is generally why a blossoming IT Security professional wants to enter into a mentoring experience.

How do people attempting to enter the IT Security profession get past this gap? My thoughts: effort, tenacity, and the will to teach. A well rounded IT Security professional starts by knowing a lot of things. They slowly, via experience, move to knowing a lot about a lot of things and eventually realize that they do not know and have not done everything. Once they realize they do not know everything they accept the fact that they will be learning during their whole career. I also believe that continuous effort and vocalism are requirements for becoming a well-rounded IT Security professional. Not only does this get you noticed by your peers, it expresses confidence to other administrators and managers.

As IT Security professionals mature they tend to filter into specific areas of focus. This usually means that the influential people in their lives become more narrow and they start to enter into good peer-mentoring relationships. By “peer-mentoring” I mean that each person grows from the other’s experiences. They drive each other to dig deeper and accomplish more. Eventually, people start developing other interests and the “peer-mentoring” relationships shift. But the excellent thing is that, generally, the previous relationships (business or friend) remain strong and continue throughout the years. At some point, interests will diverge again and collaboration will continue.

So, if you are just getting into IT Security or attempting to transition to a new focus area, do not get stifled or discouraged by the lack of mentoring opportunities. Rather, put yourself out there. Start working on personal projects. Tell people about your experiences, failures and success (because BOTH are important to the learning process). Do not be discouraged by critical feed back and have fun even if the subject is hard from start to finish. Remember, experience comes with time and effort. Learning the hard way helps people understand the easy and efficient way. In the end, people will notice and you will have achieved your goal.

Go forth and do good things,

Don C. Weber

There are several reasons that I am drawn to IT security and incident response.

• The discovery of what occurred.
• Protecting a business and its employees from people doing them harm.
• The need for a breadth and depth of knowledge in many areas.

When I was but a young security professional I always wanted to actively go head to head with a malicious hacker.  Last month, I finally got my chance.

Tom Liston (OMG, he has a Wikipedia page!?!) and myself, of InGuardians, came to our client’s site and integrated ourselves with the other security team members already on-site.  Our addition rounded out the team with a good cross-section of skill sets and experiences.  Perfect for what we were going to experience in the next few days.

By the time Tom and I arrived on-site the incident response team understood that the client’s website was being modified to serve malware to its visitors but they did know know how the modifications were being accomplished.  They had already start getting resources in place to provide data that would get us answers our questions.

• Snapshots of the dynamic websites were being saved to a strategic location.
• Web logs were configured to log centrally instead of locally.
• Full network packet captures were configured and stored in another location the incident responders could easily access.
• A method for remotely conducting data analysis and data acquisition was implemented.

Although I won’t say that I always ask for these things during an incident response, I will say that these are usually some of the recommendations that are made at the end of an engagement.  The reason these technologies are valuable is because they provide responders with the system and network-based artifacts that can be used to make quick decisions.

The response effort started normally.  We used our time to become familiar with the environment, located compromised webpages, and tried to piece together the initial compromise vector, persistence mechanisms, and propagation techniques being utilized by the attacker.  During this time we were able to devise methods for quickly locating anomalous activity within the information provided by the weblogs, packet captures, and file differences.  Analysis of systems did not detect anything significant so the team believed that the compromise was limited to the modification of dynamic web pages.

We quickly came to realize that the attackers had been modifying files for quite a while.  The specific code that provided them their initial access was eluding us and we made various recommendations to reduce the number of blatantly vulnerable web applications.  Our analysis identified various obfuscated PHP web administration tools (such as the web shell WSO and Web-based file manager webadmin.php) that the attackers were using to modify files and stomp the file’s metadata.  The combination of all of these things made timeline analysis very difficult and less fruitful than I am use to during active incidents.

This type of activity is fairly normal during incident response and we were starting to get to the point where our team was just going through the motions of detecting modifications and addressing them as quickly as possible.  While this was happening the systems and website administrators worked on individual sites and servers to close vulnerabilities (no small task, indeed).  These websites are the primary business of this client and each of the affected web applications and their servers had to be left online during the attack.  In other words, it was cost effective for them to keep us chasing the bad guys and to keep the sites alive despite the attack.

And then, we had contact with the enemy.

Tom and I had just sat down for our last day on-site.  All of our team had their systems up and running and we were going through our usual motions.  I had just sat down with my coffee when the client’s security manager walked in and told us that one of their primary sites was being blocked due to malicious code.  The only thing I can relate this moment to is somebody yelling “Contact Left!!” and the Marine squad turning in unison and entering the fray.  Each one of our team turned to our computers and start gathering information with a flurry of keystrokes.  One man starting rolling through the web logs for the last hour.  Another man starting pulling down the last hour’s worth of packet captures.  Tom located the malicious code in the infected webpage and I, using that information, tracked down the source of the inserted code to a PHP include file (obviously used since it would get tacked onto every page served by the web application).  I quickly handed the PHP code off to Tom for de-obfuscation (damn, he is seriously fast at it, too) and I turned to investigating the last “known good” snapshot of the website to try and get a good timeline for the team members conducting log analysis. At the same time, two of us remotely acquired images of the web servers in an attempt to detect any malicious interactions with the operating system and provide more detail to our timeline information.

All of our quick action payed off.  About fifteen minutes into this activity the team member analyzing the packet captures said “Holy sh*t!!! I just grabbed the latest packet capture and I think he is on the system right now.”  He paused and then said, “Holy sh*t!!  He’s got root.”  It turns out that all of our work during the week had hampered the attacker enough that s/he wanted to know what as going on with the system.  So, instead of merely managing the malicious activities via the web shell s/he dropped to interacting with the operating system.  The packet captures pointed us to several system-based artifacts which we grabbed for analysis.  This was a boon because until this point, as I mentioned, we had not detected this type of interaction (those web shells provide some nasty functionality – “It’s a feature!!”).  Now we knew the types of tools s/he was using, how s/he got them onto the system, how s/he was hiding them, and the tool’s capabilities.

About thirty minutes in the fray the excitement wore off.  We could see where the attacker logged off of the system (possibly moving onto their next compromised site).  We spent the rest of the day working through the information we found and firming up our findings and recommendations.  I actually flew out of there later that day; high on adrenaline and wishing I could stay.  After all, I would much rather stay and face the enemy than to return to the rear and write a report about it.

So, the incident response-related lessons learned from this engagement:

• Centralized logging of application logs and full network captures are critical during an incident response;
• Being prepared with remote acquisition and data analysis techniques is also extremely important;
• Team organization, tool familiarization, and overall experience will produce positive results from stressful situations;
• Management support and confidence breeds success.

These are the tools and concepts that will help your incident response team fight-the-fight while the mission critical assets stay online and ensures that everybody (client and consultant) gets paid.  When an incident response team asks for these resources or recommends they be put in place, they are not doing it because the technologies are cool.  They are making the recommendations because they are effective and will reduce the time and effort associated with the incident.  Pre-deploying these technologies or generating procedures for doing so will make them even more effective.

Go forth and do good things,

Don C. Weber

Drop The Refrain

The refrain “make it too expensive for the attackers” needs to be retired from the security professional’s vocabulary.  It is not going to happen.  Making it “too expensive” is not S.M.A.R.T. It also means absolutely nothing to the attackers.  The guidance security professionals need to be pushing is that managed business processes and security controls will reduce the overall cost associated with responding to each attack as they are experienced.  It is not a matter of who will do the attacking, when they will do it, or how many resources they have behind them.  It is the understanding that attacks are going to occur, most attacks will involve techniques to which an organization can identify and respond, and some attacks will occur using new methodologies and technologies to which the organization can identify and respond.

Attacker’s Viewpoint

Let’s take a look at the malicious hacker list Roger Grimes put together (which I like very much, BTW) and see how each group is affected by the overall “expense” of their efforts..

Cyber criminals: as these people are after big financial gain they understand that there will be cost involved with their efforts.  The bigger the pay-out the more time and effort they are willing to spend.  However, they just don’t turn away from targets.  Hard targets are softened by time.  Over time there will be new vulnerabilities, new attack methodologies, new personnel, etc.  If something is currently difficult they may roll over to other prey, but they will come back around.

Spammers and adware spreaders: these attackers are generally not concerned with specific targets.  However, the way spammers and adware spreaders conduct their business is interesting to other malicious hackers.  The research and development of this group can easily be leveraged for other purposes.  They try everything in the book and then write new pages when those do not work any more.  They are not concerned about how much it costs to get around your controls because they are best at presenting a moving target.  Basically making it more expensive for customers to keep up with them.

Advanced persistent threat (APT) agents: these guys are good at getting in.  They are also good at being patient.  Biding their time for specific opportunities.  They try a few things to get in and, if that doesn’t work, they try something else.  As they have multiple targets (or so it appears) they move onto the next target on their list.  Then, after a time, they roll back to a targeted organization where their tactics have been unsuccessful and they try something else.  Security programs and controls are not making it more expensive for them.  They know it is a part of the game and they just continue until they are successful.

Corporate spies: well, this attacker is most likely (yes, I’m guessing) expensive to begin with.  I have not run into any of these people or cases, nor have I heard much about them.   But, I understand the mentality.  They could be heavily trained or simply people exploiting an opportunity.  These people are usually in a position where they can manipulate the security controls, or feel they can get away with their efforts despite the controls.  In other words, the reward already out-weighs the risk.

Hacktivists: this group is probably the most affected by the cost of their efforts.  However, they will most likely be associated with the Rogue hacker category and thereby reap the rewards of their efforts.  Which, in turn, means that cost does not affect them much as they rely on targets of opportunity that will produce the actions they desire.

Cyber warriors: although military commanders do take into consideration cost of resources, their limits are beyond those of most corporations.  Additionally, once the cost reaches a certain point then the tactics for this group changes to kinetic solutions.

Rogue hackers: one word for these guys: “challenge.”  If it is a challenge is it really too expensive?  To this group time and effort is nothing.  Once they set their sights they either work it till they are bored with it or they accomplish their objectives.  If a group of these individuals gets their collective heads together, then cost matters even less.  Certainly a good security effort will prevent many unmotivated rogue hackers, but those that are motivated have a purpose where cost is of little consequence.

In Need of a New Catchphrase

Therefore I say out with the old refrain and in with something new.  Preferably something that is a little more Specific, Measurable, Attainable, Relevant, and Time-bound.  I prefer more direct and implementable guidance that helps build cross-functional incident response efforts and teams.  However, for those who require those “elevator statements” (don’t we all at some point) maybe try one of my favorites: “reducing the gap between compromise and identification.”  Because the old “make it too expensive for the attackers” statement is placing the wrong emphasis on the overall effort and makes people, particularly executives, think that 100% secure is an obtainable goal.  When we all know that information security is a sustained effort that will continue as long as the organization exists.

Go forth and do good things,

Don C. Weber

Shmoo Con

First of all, I have to say that my talk at ShmooCon 2011 was a great experience.  Here is a view from my stand point.

Q and Atlas did a great job.  You can experience our talk yourself by downloading and watching the presentation generously provided by ShmooCon.

Purpose

Although this started as one of my first full-fledged hardware projects, the intent was always to evaluate ways to cheaply assess deployments frequency hopping spread spectrum (FHSS) technologies (please review and I’ll assume you did).  Why is this important to you?  Well, because there is a rift in the radio and information techology (IT) community about how much protection FHSS provides.  The most concerning of these view points is that FHSS acts as a pseudo-encryption technique.  People that think FHSS is, by itself, secure believe that the channel hops happen too fast to predict and simulate without knowing  the actual algorithm used to generate the hopping pattern.  This manner of thinking leads to implementations that send data “protected by FHSS” in clear text or obfuscated by techniques such as XOR.  In any system the transmission of clear text information could lead to information disclosure and, in the worst case, data injection.

Some deployments of FHSS are going to be more secure than others.  Military units that use  FHSS have the best capabilities for modifying algorithms such that hopping patterns are reduced or even  eliminated.    Commercial implementations are always going to be based on risk assessment.  This means that the resources necessary to consistently modify algorithms to limit hopping patterns is going to suffer unless it is constantly maintained.  Other limitations will also play into repeated hopping patterns.  Embedded devices with limited memory and processing capabilities naturally limit the effective rotation of generated hopping patterns.  Poor implementations of FHSS will reduce this even more and could result in one pattern being repeated over and over.

The intent of security researchers is to identify weak implementations in any technology.  This lead our team to focus on producing tools that would provide engineers, IT administrators, and security professionals with the capabilities to assess FHSS deployments.  As weak implementations are identified they can be improved with newer technologies or augmented with additional security controls.

FHSS itself is not a vulnerable or “broken” technology.  Just like most technologies it must be deployed and configured correctly to provide its intended functionality.  It is the deployment that may be vulnerable.   FHSS itself increases resiliency (availability) and does make it harder to intercept and interact with the transmitted data.  It, like other technologies, is improved and becomes even more robust when augmented by other technologies such as encryption.

Hedy Attack

The way our team approached identifying hopping patterns is simple in theory.  Certain FHSS deployments will, when given enough time, repeat the channels to which they hop.  This might happen over a year, month, week, day, hour, or even minute.  By monitoring FHSS transmissions it should be possible to determine the point where an FHSS deployment repeats or, in a term that might help some understand, loop.  Loops are the weakness.  Once loops are know the actual algorithm that generated the pattern is not necessary.  A simple ordered list is all that is necessary to intercept and interact with the complete transmission.

HedyAttack approaches this problem by analysing the minimum noise on each channel.  The first step is to record the average minimum received signal strength indicator (RSSI) value of each channel.  Later, any RSSI value that is greater than the minimum plus a boundary value should indicate a transmission on that channel.  Thus, by quickly looping through the spectrum, transmissions on any channel should be detectable.  As hopping patterns are pseudo-random scans must be very fast.  To increase scan speed the actions taken by the radio and controlling microcontroller must be limited to tuning the radio and recording any detected channels that jump above the minimum value.

For FHSS sources that have known configurations, this is enough to begin the analysis.  But, as with every technology, different vendors have their own methods of implementations.  For instance, let’s take the frequency range we mentioned before.  a simple breakdown of this range would be to cut it into 500 KHs pieces which are also referred to as channels.  This would give us 53 channels and make it easy to gather data.  However, the channel spacing does not have to be 500 KHs.  Vendors can split this up any way they choose making their implementations have more or fewer channels. This means that for systems where the channel spacing is unknown tools must analyse and determine these values so that hopping patterns are as accurate as possible.

Similar to channel spacing, the amount of data that each vendor will send during each hop is going to vary.  Data analysis to determine the hopping time increments will also be helpful when analysing the hopping patterns.  As time increments decrease these values will be very valuable to determine the type of equipment necessary to conduct the research.

Hedy Attack provides all of these tools that attempt to provide all of the above with the exception of determining hopping time increments.  Time increments will be worked into the toolset as the other tools become more stable.

Hardware

The hardware selected for this project was the CC1111EMK868-915 Evaluation Module Kit.  This was picked because of the cost, Chipcon radio, TI support, USB capabilities, and breakout of the Data Debug pins.  The fact that other pins were broken out was an added bonus.  Extra added bonus is that work for the CC1111EMK will could also be leveraged for work with the CC2511EMK Evaluation Module Kit and other implementations of Chipcon chips.

Of course interaction with the CC1111EMK would not be possible without the Goodfet.  Travis and his neighbours have developed tools such as goodfet.cc that allow for easy debugging and interaction with the Chipcon radio.   All of the capabilities provided by the Goodfet are also provided by the CC Debugger.  However, I recommend that you get a Goodfet and use it because of its flexibility.  As you develop you will move to other platforms and it is very likely that the Goodfet does, or will, provide you the support you need.

Source Code

Since the talk we have firmed up the code base and uploaded it to HedyAttack at Google Code.  The primary tools (at the time of this post) are:

• specan – Spectrum analysis code ported from the IM-ME project.  This code was the starting point for all of the other tools.  It has had the display functionality removed.  The scan data is written to XDATA and can be pulled using the Data Debug functionality provided by the Goodfet.  The specantap_gnuplot.py client can be used to display the results.  As the
• maxscan – Spectrum analysis code with a focus on the 902 thru 928 MHz range.  This tool generates data in the same manner as specan.  The best way to display this data is to use the maxscan_gnuploy.py client.
• hoptrans – Transmission code that mimics frequency hopping.  This code can be adjusted to hop at different speeds and across as many channels as configured.  Default is set to a channel spacing of 500 KHz which equates to 53 channels.  The intent of this tool is to provide a test environment.  Using this in conjunction with the other tools will require a second CC1111EMK.
• minscan – This tool starts out by recording the average minimum RSSI value for each channel using a series of initialisation passes.  Once the minimum values have been recorded the tool changes mode to monitor ups in the RSSI value that should indicate transmission on that channel.  To track when jumps occurred a loop counter is also maintained.  The loop counter lets us create a tracked array which will contain an ordered list of channel hops.  This list can be pulled off using the minscan.py client.
• hedyattack – The culmination of all the tools.  This tool initialises in much the same manner as minscan.  Several iterations are run to detect the minimum and maximum values of each channel.  Initially this information is used to attempt to determine the channel spacing of the FHSS device.  Although channel spacing can be determine programmatically reviewing the data manually is also helpful.  After the initial iterations the tool moves onto detecting and recording hops in the same manner as minscan.  The added strength of the hedyattack tool is the USB functionality.  This means that interactions with the radio and XDATA no longer need to be accomplished using the Data Debug feature which actually pauses the microprocessor to function.
• Makefile – each tools has a corresponding Makefile in their individual tool folders.  The Makefiles build  the corresponding firmware and place it in the firmware directory.  The Makefiles can also be used to have the Goodfet install the firmware onto the CC1111EMK.

Get To Hacking…err…Assessing

That is it.  HedyAttack in a nutshell.  A methodology and toolset to be used for assessing the FHSS implementations and deployments.  Hopefully it is helpful. Should you wish to join this effort just drop the administrators at HedyAttack a message.  We’ll get back to you as soon as possible.

Go forth and do good things,

Don C. Weber

When I left the United States Marine Corps and started college I knew two things.  1. I wanted my career to be in Computer Security and 2. I wanted to work for a group of professionals who operate at the same level of the Force Reconnaissance unit I had the pleasure of serving with for six and a half years.  I chose computer security because I figured it would be a growth industry with a strong job market for years to come (I was correct).  My vision of a Force Recon like security organization was born out of the need to surround myself with individuals who are self-driven, educators, drawn from various circumstances, volunteers, and flat-out just ready to do whatever it takes to do a good and professional job.

I achieved my first goal right out of college and basically due to sheer luck.  The company I was hired with decided to expand a security team and I was in the right place at the right time.  My second goal was a little harder to fulfill.  It has taken me eight years, five different jobs, collaboration with many outstanding security and IT professionals, countless hours of extra work, and the will to try and turn bad situations into mutually beneficial experiences for all involved.  It has been a long row to hoe but I have finally achieve the second goal.  At the beginning of this month I became a new member of InGuardians, Inc.

For the past two weeks I have been transitioning from the world of Incident Response back into the world of penetration testing, research and development, and security architecture.  I have been exposed to the world of Smart Grids, hardware hacking, and software evaluation.  To say the very least I have been thrust from the frying pan and into the fire.  The world does not stop spinning because of personal transition and I am experiencing this first hand right now.  But, it is exciting and educational.  My eyes have already opened to a new realm of possibilities and I intend to use that knowledge to ensure our team is a pivotal part of security developments and implementation in the years to come.

I would like to thank all of the member of InGuardians, Inc for their confidence in me and bringing me on-board.  I would also like to thank everybody out there who has helped me get to this point.  There are too many of you to do this individually, but I think you can all agree that I can show my thanks my continuing with my open education stance and providing you with an insight as to what I am experiencing and how I think it affects individuals as well as the industry.

Go forth and do good things,

Don C. Weber

I recently attended SANS Security 508 at SANS 2010-Orlando.  When I told Harlan Carvey that I was going to attend this training he was concerned that I would not be exposed to anything I had not already exposed myself to through work and personal effort.  When I arrived on-site I got the same feeling from Rob Lee although his concerned seemed to be more centered around the value added by the course to more experienced incident response professionals.  Well, although their concerns were valid, I have to say that attending this class was a very valuable experience from the networking I accomplished, to the new (to me) concepts about how file systems work, to the concerns about how some applications leverage that information to produce system artifacts.

I am not going to delve into too much about the topics covered in the class.  It is outlined for you on SANS’ website and, well, Rob and his crew worked very hard on pulling all of the concepts together.  For that you should attended the course or purchase the course material if you would like a deeper understanding.  However, there are a bunch of priceless illustrations that help the students understand some of the complex topics that can be confusing when first exposed to the information.  The ones that popped out to me the most were the images covering the Forensic Investigation Methodology, Date/Time correlations, and the Filesystem/Sleuthkit Review.  Each of them, at a glance provides excellent clarification.

So, to alleviate the concerns of Harlan and Rob, I learned a lot by attending the course.  I guess I am just one of those people.  I will admit that I was not as challenged as when I attended some of the other SANS trainings I have attended.  Certainly it was a fire hose for most of the attendees.    I just figure that this means that for the past two years I have been approaching incident response properly.  Rob’s class most validated the processes I have formed surrounding my acquisition and initial analysis techniques.    A lot of this came to me via Harlan’s training both professionally, individually, and via reading his blog and books.  Some of these concepts were also developed via the experiences of Chris Pogue.  His Sniper Forensics talk is a direct representation of many of the concepts I employ.  (It was good to finally meet him after two years.)

Since I am not going to go too deep into the concepts covered by the class (although they should shape some of the future content here) I will provide you with some of the notable quotes that came from Rob.

“…training the new breed of incident responder.”

Absolutely, SEC508 provides a sound foundation.  It exposes incident responders to the basics of the field.  Starting with a sound foundation is what is necessary.   (Tangent Alert!)  It also takes incident response and digital forensics out of the court room and back into the data center.  Which is important because the data center changes much faster than the court room.  By letting the court room lead our incident response processes we are limiting our capabilities to adapt to new threats and attack methodologies.  Let the court room keep up with us.

“…EMTs do not worry about adjusting evidence …”

Another statement enforcing the point I just made.  Of course, what should be noted is that EMTs approach an incident with a specific methodology.  They have a plan and they execute it.  When necessary, they deviate from that plan.  But familiarization and continuous training around the basics of that plan make it second nature to them.  This means that their actions can be accounted for and justified when evidence is necessary.

“Evidence integrity goes to the weight of what the evidence can be used for….”

Basically, be more concerned about the actions you have taken to gather information.  Once again, following your plan, knowing the basics, and documenting deviations.  Just because there is or is not a hash does not mean that, if necessary, the information will not be admissible during a court case.  But court cases should not be your major concern.  Consistent and repeatable process should be your concern.  This is necessary in case there is a need to repeat the data analysis in a court room, for a Board of Directors, or for a team of auditors.

“Tools do not have to be validated.  The output, what was found, is more important than the tool that was used to interpret the data.”

This is one of the first concepts that Harlan explained to me when I started working with him.  Different tools display information better than other tools (which is why we have a wide variety of them).  But just because a tools presents the data in a certain way, or has been doing so for X number of years, does not mean it is doing so correctly.  Other methods may be necessary to validate tool output.  This concept holds true for a perl/python script that was written last night by a kid in Poughkeepsie, NY or a long standing data analysis tool such as EnCase or FTK.

The forensic industry “is not a fad.  Organizations are spinning up internal teams to handle incident response and investigations.”

This is nothing new but it is a great validation.  Rob is exposed to a wide range of people from many different operational backgrounds.  This statement is also supported by the explosion of process and tool development in the digital forensic and incident response field.

I will end with a personal favorite of mine.  The following quote validates a realization I recently came to while cleaning up after an incident response.  If you have a weak heart, and hold onto old concepts  dearly, you may want to skip the following quote.  (I am paraphrasing because I just realized I didn’t write it down.)

“How many passes does it take to destroy data so that  forensic analysis tools cannot recover it?  One, yes you are correct.”

Yes, you read that right.  Only one pass is necessary.  Wow, that will save a lot of time not to mention a lot of energy related to processor intensive multiple writes using random data.  I am not going to track down all of the links that support this statement.  Basically, once information has been overwritten it cannot be accessed by the tools we typcially deploy.  Even advanced tools can only guess at the former state of a bit.  The cool thing is that since there are multiple layers to the file systems, there is a chance that a tool or process did not correctly overwrite the information.  This is a key concept covered by SEC508.  And as incident responders we also realize that just because data was destroyed in one location that it is not stored in some other location.  Which is why our processes include involving an organization’s network, workstation, server, and application administrators as well as management.  These people will understand where residual data resides within the organization.

So, to wrap this up, I highly recommend SEC508 to new and experienced incident response and digital forensic professionals.  You are going to learn something you did not know.  You are going to make contacts that will be invaluable in the future.  And, if you obtain the GIAC certification, you are going to have a valuable certification in a growing and increasingly important field that is having global impact.

Go forth and do good things,

Don C. Weber

The success of Into The Boxes Issue 0×0 was only possible because of the collaboration provided by members of the Digital Forensics and Incident Response community.  In order for this publication to continue we need more people to step up and provide their input.  As you can see from the first issue we are looking for input that can be implemented by people in the DF/IR fields.  This input can be in the form of detailed articles or quick tips.  All input will be given serious consideration.  The ITB editors will provide authors with recommendations to strengthen their write-ups to ensure the best value to the community and help the authors develop as DF/IR professionals and writers.

Please help ITB by providing your submissions or letting us know about your intent to submit via the ITB Call Box.  We are also looking for article recommendations which we will place in the ITB Research Box so that others have good ideas as to what will help the DF/IR Community.  Obviously, if you would like to contribute but do not know what to write about, check out the ITB Research Box for recommendations.

Go forth and do good things,

Don C. Weber