Last month, I introduced my good friend Ninja G, an accomplished professional penetration tester for a large financial institution (who also happens to be a practitioner of ancient martial arts developed by ninjas). We talked about his daily activities, his unusual career path, his perspectives on WLAN and Linux security, and the uncanny preoccupation among Ninja G and his coworkers with ninjutsu.

This month, I wrap up the interview. I think this installment is every bit as wide-ranging, thought-provoking and entertaining as part I. I hope you agree!

MB: I've got to ask a somewhat selfish question, because so much of my own career has involved firewalls. In the age of Web applications, where the browser is the platform and so much of the world's business is conducted over TCP ports 80 and 443, do traditional layer 3 firewalls still have a useful role?

NG: I believe so; however, I think most people would agree that a traditional (state-aware) firewall by itself isn't a complete solution. It is only the first of many layers of defense. Intrusion detection (and prevention) systems, system hardening, secure centralized logging and application-aware firewalls should supplement a state-aware firewall to help round out areas where it would otherwise be lacking.

MB: What's your opinion of the new generation of (Web) application firewalls that can be trained to block exceptions to “expected” Web application behavior? Are we finally reaching a point in this class of technology where it's possible to create a complete enough notion of “expected” application or network behavior to reduce the very high rate of false positives we've traditionally associated with this approach?

NG: I don't think I ever would recommend a Web application firewall in lieu of Web application hardening. A lot of the problems that are being caught by Web application firewalls usually can be solved with proper tainted variable checks on all user-supplied input. This includes the Web browser's user-agent string (which may find its way into log files), all cookies and, of course, GET/POST data, AJAX requests and so on.

In an ideal world, the Web application should provide real-time logging of things like cross-site script, SQL injection and brute-force attempts, but that is more often the exception than the rule. Often organizations rely on (SSL breaking) Web application firewalls to obtain the same types of data; however, you have to be very confident in low false positive rates when deploying in any inline device that actually prevents malicious traffic from reaching the target host, or else you could be creating your own denial-of-service condition for what could be business-critical applications. One should keep in mind that even the top-of-the-line Web application firewalls occasionally will generate false positives and plan one's use of them accordingly.

MB: Some people are very skeptical not only about the worth of pen testing, but also especially of the trustworthiness of anybody who has amassed experience in system cracking, which they (the skeptics) seem to think is inherently corrupting. But my own experience has been that overall, security researchers and penetration testers, precisely because they understand so well how easy it is to get caught, tend to be very responsible.

And, I'd unhesitatingly put you in that category. I think of you as a highly ethical and trustworthy person. So, what's your take on this? Are security ninjas generally like you, or is criminal recidivism (à la Alberto Gonzales) rampant?

NG: Of course, I can't speak for anyone else but myself. People have their own unique motivations, passions or “demons” that drive them along this path. I can say it has been my experience that most people who perform penetration testing for a living generally wouldn't risk their livelihood by hacking illegally. For a person like myself, I view my work almost like a flip-side version of the Buddhist concept of Right Livelihood, which states that people should try to find an occupation where they won't sacrifice their moral code. Except for me, I found a job where I get to express myself creatively in a criminal-like way, yet with the most ethical of outcomes.

At the end of testing, I create detailed write-ups of all of my security findings, write proof-of-concept code, help vendors understand and re-create these issues, and assign a risk rating to help others understand their severity—like a lot of the other “unsung heroes” of this field, who never release their exploit code and provide full details directly to vendors and code writers. I would like to think I am helping improve the overall security of not only the company I work for, but also indirectly helping all the other companies who use the same systems, services and products.

MB: It's easy for someone in my position (and that of my readers, I hope!) to see the value of your methods and your reports. I don't think “unsung hero” is any exaggeration, having seen firsthand how tangibly things tend to be improved after an unfavorable penetration test. But how do vendors and developers react to your work? How do you get them past what I'm sure tends to be an initially defensive reaction?

NG: Yes, you're correct. The typical initial response is to view the request for a penetration test as simply a check-box item on a list somewhere, which may or may not prove any sort of real due diligence. My assumption is that this is their usual experience with dealing with other companies with limited information security personnel and budgets.

Once they actually start hearing about high- or critical-severity security vulnerabilities, they almost always shift to a defensive position. At this point, I give them my (somewhat canned) speech about how they are in essence getting weeks of free security consulting. I assure them that I will share the details of all security findings completely and recommend steps for remediation where appropriate.

It is usually around this point when they realize that although their experience may be painful or humbling, they really are getting something for nothing, and far better to receive the information in this way, rather than having the same issues discovered by others who may not give the vendor any sort of warning before releasing the information to the general public.

MB: In my own consulting work years ago, now and then I'd be called on to do port scanning or security scanning (though not actual penetration testing) of live, production systems, and I always found that nerve-wracking. Have you ever inadvertently (or, come to think of it, intentionally) disrupted an important production system?

NG: I once took down a large group of firewalls simply by port scanning them. This was completely inadvertent, as I didn't know that they had been configured to effectively “turn themselves off” when they detected heavy port scanning. (?!) This caused a large production outage (and several people were howling for my blood), yet the configuration was beyond foolish, so by helping to “correct” the issue, I managed to escape unscathed.

I have never intentionally disrupted an important production system. Even in my lawless years, the goal was never to “take down” a system. I viewed that action as providing a very clear sign that an intrusion had occurred. The end result would be one less system to explore, which I viewed as sub-optimal.

MB: There still seems to be tension between two camps in information security: those of us who self-identify as hackers and those who don't. Yet attendance at Black Hat Briefings, DEFCON and other hacker conventions continues to balloon. Do you think the “hackers vs. suits” situation is getting better, or are we simply gaining numeric superiority?

NG: I would say that there was much more tension a decade or two ago, back when the corporate world wouldn't dare ever send someone off to a “hacker conference”, let alone pay for it. Now conferences such as Black Hat and DEFCON are considered valued sources of information for those defending rather than attacking corporate resources.

Personally, I think there is a rather new element at hacker cons that may soon upset the delicate “hacker/suit” ecosystem: the US military—and I'm not talking about their old guys, I'm talking a new wave of young military. So don't be surprised in the coming years if that guy sitting next to you at DEFCON happens to be a SIGINT analyst for the US Navy.

MB: I've already had that type of experience, and agree with your observation about the military (which, I supposed, tends to be a youthful demographic, so maybe this isn't too surprising). DEFCON's “Spot the Fed” contest stopped being easy ages ago!

I think I'd like to wrap up with some more or less random questions—feel free to answer as tersely or completely as you like and, of course, omit names to protect whomever you like!

MB: What's the stupidest design decision you've ever seen?

NG: Okay, this one goes back way over a decade, so I feel comfortable discussing it. I once saw a large financial organization solve their backup issues by installing FIDDI interfaces in all of their UNIX servers. They dual-homed all of the machines into one large FIDDI ring and then kicked off all of the backups via rhost. I rooted one backup server in the FIDDI ring and ended up owning the entire back office settlement environment across the globe in minutes, due to rhost trusting the backup server. Keep in mind these hosts' primary interfaces were segregated into protected zones by firewalls, and access was limited into each zone based on your job function.

Epic fail.

MB: What's the coolest security control you've encountered lately?

NG: People are probably sick of hearing about it, but DNSSEC is actually pretty cool, especially when you factor in the potential impact associated with traditional DNS. (Yes, I have drunk Dan Kaminsky's Kool-Aid!)

MB: What's your favorite (non-secret) ninja weapon or fighting technique?

NG: Hmmm...for this question, I believe a little bit of ninja lore is necessary. First, forget everything you know about ninjas being the bad guys dressed head to toe in black uniforms; rather, they were often highly skilled martial artists acting covertly in enemy territory much like a modern-day spy. The worst possible fate would be to be caught, so contrary to all of those bad ninja movies of the 1980s, escape was always way more desirable than fighting.

So in the study of ninjutsu, you will encounter things like the Santo Tonko kata (forms of the escaping rat), which includes techniques for escaping grabs, the use of eye-blinding powders and using small thrown objects (such as shuriken) to dissuade continued attack or pursuit. Nowadays, a small tactical flashlight can be used instead of the irritating blinding power, and as shuriken have been outlawed in most states, a pocket full of loose change does the same thing when unexpectedly tossed into the face of a would-be attacker. “Oh! These are a few of my favorite things...!”

In one particular situation, I was greatly outnumbered, so I decided I needed to even the odds a bit. Using a Surefire E2E flashlight, I blinded the advancing “front row” only to have my “fighting with light” completely misidentified as a police tactic. Next thing I heard was, “This guy is a cop!” and everyone scattered. That alone was worth the price of the flashlight.

So rather than doing something silly or dangerous (like arming your loved ones), instead get them an inexpensive high lumen output flashlight. If their attacker disarms them, it isn't anything that could really be used against them. Plus, you would be surprised how often a flashlight comes in handy, even when you aren't being jumped by a large group of people.

MB: Which is harder, being stealthful in meat-space or being stealthful in cyberspace?

NG: If you understand both environments, then one really isn't more difficult to manage than the other. These are not set in stone, but I generally think of four basic principles that apply fairly equally to both physical and cyberspace stealth considerations.

The first principle would be disguise. In physical space, this includes concepts, such as wearing appropriate clothing to blend in with the rest of the population in the area, and the use of multiuse, hidden or improvised tools/weapons. In cyberspace, this would include avoiding the generation of network traffic that would be typically associated with either reconnaissance or attack. Instead, the stealthy attacker would choose to generate completely “normal” network traffic that could also be used to glean useful information.

A good modern example of this would be the tool named FOCA, which uses search engines to identify the location of documents that commonly contain metadata. The tool downloads these files a few at a time and then rips out the metadata and presents the extracted data. All of the generated Web traffic is 100% normal. All of the documents downloaded were intended to be downloaded. Unless a company is militant about scrubbing metadata, it isn't unusual to find about one file out of 100 that actually contains something useful enough to be leveraged in further attacks, such as an employee name, e-mail address or user ID. With larger companies, it is not uncommon to discover thousands of metadata-laden files available for download.

The second principle would be distraction (or attention control). In physical space, there are lots of variations here, as the human mind is very limited. We may claim to multitask, but the reality is that we can think of only one thing at a time. All you have to do is fill the mind with something interesting, and it will miss everything else. A good example of this can be found on-line by Googling Daniel J. Simons' video titled “selective attention test”. In cyberspace, the same effect can be achieved by intentionally generating known malicious traffic to create overt “noise” in which to hide. Tools like snot and stick were designed to do exactly this sort of thing.

The third principle would be exhaustion (or frustration). In physical space, a modern analog would be to repeatedly set off a building's burglar alarm. The first time it goes off it usually will get all sorts of attention. The police will show up first, then eventually, the building manager will arrive on-site. After about the fifth or sixth time (in the same night) of the alarm going off with no visible signs of attempted forced entry, that alarm is going to be turned off until morning and a repair call will be made to the burglar alarm vendor.

The exact same thing happens in cyberspace with intrusion detection systems that constantly generate alarms that seem to be false positives. A security-conscious person will put up with an interruption or two in the middle of the night, but nobody can tolerate unwarranted nonstop interruptions. Discover this “breaking point” limit, and you can help “tune” remote intrusion detection systems to be more “friendly” toward your future stealthy endeavors.

The fourth principle I would call fu-sui (wind and water), but most people know it as feng shui. This is intentionally selecting the most advantageous positioning, terrain, weather or timing. In physical space, this could be something as simple as choosing to attack at dawn or dusk with the sun low in the sky behind you. Your enemy will be staring directly into the sun allowing you to hide in bright glare. Another example would be to choose a foggy night to hide in the darkness and mist. In cyberspace, this could include performing your activities during holidays celebrated by your target, choosing late hours when monitoring personnel may be more scarce or choosing peak traffic hours if you are trying to “blend in” with normal activities.

These principles are not mutually exclusive; in fact, it is often better to blend them together when creating your plan of attack.

MB: When does the fun of Brute Force trump the righteousness of the Elegant Solution?

NG: Wow, that definitely applies to both penetration testing and ninjutsu: “When it is the quickest and most direct solution to the problem.”

MB: Who's more elite, pirates or ninjas, and why?

NG: Wow, that is the eternal debate. Here is my take.

If you research the origin of each of the nine martial arts that I study, you will find that one originally was used as a naval military art, and as a result, the body positions and movements were designed to be used on ships that were constantly rocking and slippery. Even to this day, techniques from this school are sometimes practiced using a boat oar as a weapon. This means that historically, some ninjas were pirates; however, I seriously doubt that the contrary was ever true. I'm sure this won't surprise you, but I vote ninjas!

MB: And on that lighthearted note, I'll sadly but gratefully wrap up what I hope, dear readers, you agree was a fun and informative conversation. Thanks so much for playing, Ninja G!