When it comes to data protection, Black Hat puts its money where its mouth is

From embarrassing dating profiles to unprotected corporate earning reports, Cybernews discovers what really happens to all that sensitive information flowing – ahem, leaking – through the Black Hat Network Operations Center (NOC) once summer camp for hackers finally ends.

There are new reports of tech giants collecting troves of personal data on unsuspecting individuals almost every day.

From social media apps, like Meta’s Threads, to companies like Microsoft's OpenAI who perform mass data scraping to train their generative AI models, keeping track of where one’s sensitive data ends up can turn into a full-time job.

So, when rumors of the “Black Hat guarantee” that any tech equipment brought to the six-day cybersec event would most likely be hacked by threat actors rolling their dice in Sin City – or at least kindly compromised by IT folks playing around with their newfound pen testing skills – I had to know more.

After making sure my VPN, firewalls, and anti-virus software were all in place before arrival, I decided to search out the nexus of where it all goes down at Black Hat – the NOC.

At first glance, the dimly lit space filled with dozens of computers, monitors, and faces highlighted by the glow of blue light, looks like a makeshift cybersecurity version of a James Bond-ish control room.

Protected from unauthorized outsiders behind panels of see-through plexiglass, the state-of-the-art NOC “provides a high security, high availability network in one of the most demanding environments in the world," according to its Black Hat description.

The setup

For nearly two decades, the NOC has been in the hands of security researchers Neil Wyler (aka Grifter), Global Lead of Threat Assessment at IBM Security X-Force, and NetWitness Senior Systems Engineer Bart Stump.

This year, the duo led a team of 15 volunteers to create the NOC from scratch right inside the Mandalay Convention center in a matter of days (although planning takes place for months).

Cybernews caught up with one of those volunteers, who, lucky for us, stepped outside the see-through partition to try and explain how it all works from the perspective of an observer looking in.

“We built the entire conference. From the fiber, switches to VP’s (virtual path) to the firewalls, to NetWitness, to everything, we set it up and take down,” said Sandy Wenzel, NOC’s lead incident responder and threat hunter – and customer engineer at Mandiant for her day job.

Wenzel says that the Black Hat network is probably the safest network to connect to at any conference. Stability is the first goal of the operations center, which is made up of a plethora of equipment provided by the biggest names in security, including Cisco, Lumen, and Arista.

“We are actively threat hunting, we build detection specifically for the show, both behavioral and signature detections as well. We bring in Palo Alto, we have a Corelight. We have a couple other open source tooling as well, so we’re monitoring everything, as much as we can,” she said.

Needle in a haystack

Most of the time, Wenzel said, the NOC team says “it's like looking for a needle within a haystack of needles, because most of the traffic we expect.”

“If we start to see things kind of get out of control, if someone is trying to doxx our agency, or be really malicious if they’re in class and try to go after other students. We’ll enforce that or isolate them,” she said.

In the past, the network was set up where classes could see each other, and the offensive classes would immediately attack the defensive classes, according to Wyler. This led the team to implement segmentation as part of their strategy.

In one instance, during classroom sessions, which take place on the first four days of Black Hat and involve hands-on live simulation training, Wenzel said that the NOC team happened to see some questionable things light up.

“We were like…Why does that say PlayStation 5 or why does it say Blizzard? Who's playing video games in class?” she said.

Turns out it was just some “folks from EVO leaning over and connecting to our stuff and clients. We're able to identify that very quickly and rapidly, not only by location but from the exact IP coordinates, locations, and even labeling the classes with geotags,” Wenzel explained.

"We can say, oh, this is a cloud class, or they're doing red team offense for Azure. That's normal stuff. But if, obviously, if we see a transfer to another area or location, we're like, no, that stays in the classroom."

Another big thing the classes provide the team is the ability to track what the students are working on, which ultimately reveals future trends in the wild.

“Typically what happens is you come to a conference like this, you see what the classes are doing, then you'll see those active things happening during the holidays, such as over Thanksgiving. You do notice that,” Wenzel said.

Another way to monitor trends is to follow what’s known as the Black Hat Arsenal, another interactive, hands-on lab environment for security professionals.

Wenzel says that’s where threat researchers and different vendors will bring their zero-day vulnerabilities to test out and show others how to defend against them, giving the NOC yet another reason to make sure what happens at Black Hat stays in Black Hat.

Clear text more often than not

The NOC boasts full package capture, allowing the team to grab whatever files are coming across the network and then dump them into a sandbox to see what they do.

"It's a lot of submissions; 29,000 different files were submitted to Threat Grid, just checking them to see if there's anything in there. Sometimes they're not malicious files, but they are sensitive in nature," Wyler said.

Documents are no different, the duo says. "Things come through, and we're like, ooh, that's actually bad, and we have to tell people, 'you're leaking your earnings reports before you've made them public, or, hey, you just dumped your entire financial history onto the Black Hat Network.'"

Surprisingly, the NOC team also tends to see a lot of security vendors and the attendees paying them for VPNs or security features that aren’t very secure.

“We see a lot of clear text credentials go through… your user name… your password, and we just find those people warn them ‘if you're paying for this tech service they’re not doing you a great service because we're seeing everything.’”

“We can intercept their traffic, we’ll say ‘here’s your machine name, here’s your IP, we’ve noticed this behavior, whether it's odd, or clear text credentials, or you're doing something bad. If you think you're infected, we’ll help you out.’ And we have real life analysts, people that will,” Wenzel said.

Post NOC

Wenzel says the most interesting of all those sensitive files and exposed clear text credentials are often collected by the team and revealed as part of the annual Black Hat NOC report.

For the past nine years, besides the setup and operations, leads Wyler and Stump have been presenting the NOC report as one of the last briefings to take place on the final day of the conference.

This year’s report was also chock full of images of each dashboard monitor, depicting everything from DNS traffic flow and the number of files analyzed for malware, to the breakdowns of VPN usage, network connections, and identifying the top threats by name.

The briefing is comparable to a post-mortem snapshot filled with lots of goodies about the tools and techniques used to set up, stabilize, and secure the network, including of course what notable data was exposed during the event, as well as other anomalies and changes made since previous years.

In fact, the threat hunting itself came about because of all the hacking attempts that take place during Black Hat (and at DefCon held the same week, which the two are also a part of).

“Black Hat is one of the most hostile networks in the world and that is true if you're looking at the data at face value," Wyler said.

“It’s intensely malicious. It's the type of data that if you saw that in your corporate environment you would be horrified by it, your hair would turn white or if you have gray hair already, it would fall out,” he said.

Back in the day, Wyler said they wanted to know if that was true, and threat hunting at Black Hat was born.

Here’s some of the sensitive data that was seen by volunteers this year, according to the duo.

Many of the images from the 2023 event – as in years past when Tinder first came out and user swipes were all shown to be in the clear – were related to social media and dating apps.

This user, whose profile (and lack of matches) leaking from a site for arranged marriages, was embarrassingly used as an example by Wyler in the presentation.

Another example of a Black Hat attendee whose personal information and photos were leaked from his social media app, one of the most common exposures at the event.

Another app designed for parents to keep track of their children on an Android phone using GPS, was found exposing the child’s exact location in the clear. Wyler also warned attendees to be wary of webcam apps, especially those designed to watch your pets.

It’s all about the data, dummy

Those who were compromised during Black Hat can take comfort in the fact that every stitch of data, and the equipment it is stored on, is thankfully destroyed after the conference ends, and that goes for all BlackHat events.

“Even though we have access to all this data, it gets destroyed right after the show,” Wenzel said.

“We take it all down… all the data gets reset to factory, all the physical hard drives get destroyed, and equipment goes back to whereever it needs to go,” she explained.

She described the entire process as “rinse, repeat, and reiterate” for all Black Hat shows, including those in Europe and Asia.

“We do get a lot of people that come in from different companies saying “I’d love to get my hands on the data, just to understand trends,’” she explained, noting that NOC’s answer has always been a “Sorry, but no.”

Data privacy is a really, really big thing for us, and that's the reason why,” Winzel said, “We keep a lot of stuff, hardware in-house in the American data center, not doing everything in the cloud, which a lot of people are moving to, because we can't really.”

“We don't understand data sovereignty, especially when we go over to Europe and Asia for those shows, so we keep everything in-house, on hard drives, that we can handle, destroy, do what we need to do,” she said.

“Anything you see here is where you'll see it, and then after the show, it’s gone,” Wenzel added.