I can't help but notice how little of our industry actually reads what its own tools are saying. We scan for patterns we care about, we alert on the ones that match, and we move on. The message itself (what was actually said, why the source transmitted, who in the organisation should care) rarely gets decoded. That's a gap I want to close, and earlier this week we shipped the dataset that makes closing it cheap.
We pushed the largest open-source, structured cybersecurity dataset in existence to HuggingFace. It carries more than a 100 million labelled records, built from real adversary traffic (not lab traffic), co-published with the University of Canterbury (UC) | Te Whare Wānanga o Waitaha Computer Science and Software Engineering department, and released under Apache 2.0. The dataset card is here. The four-stage sanitization pipeline is here, also open-sourced, so anyone can verify exactly how we cleaned the data before publishing it.
My hope is that academic and industry researchers will use it to advance innovations that reduce risk and increase deterrence against cybercrime. No doubt about that. But there's a nearer-term use worth walking through: you can use the dataset (and its smaller cousin, witfoo/syslog-to-artifact) as a reference corpus to apply WitFoo's Empathetic Processing in a stack you already own. Three prompts. Three phases. Grafana, Sentinel, Splunk, or Elastic; the approach is the same.
The step we skip
The first phase in Empathetic Processing (EP) is fully comprehending every signal. I know that sounds like a no-brainer, but as an industry we skip it. We scan or skim messages for patterns we're interested in but rarely spend the compute to understand exactly what was said, why the data source transmitted, or who in the organisation or community may be interested.
The reason we skip it is that the number of "languages" cybersecurity tools speak runs into the tens of thousands, and the dialects of those signals run into the millions. Over the last decade, WitFoo has identified over 200 language families and their etymologies from everything ranging from firewalls to application logs. That study has produced a baseline of listening to each data source with an understanding of the linguistics (and, yes, the philosophies) that created it. That understanding is what the dataset represents.
The first task in EP is taking the thousands of dialects in an organisation and translating them into a normalised schema. Over the last three decades vendors have tried: ArcSight's Common Event Format (CEF), Splunk's Open Cybersecurity Schema, a handful of others. The practical challenge is that the weight of the translation falls on the contributing vendor. First, many vendors fail to implement a schema at all. Second, when a vendor does implement one, they often translate incorrectly.
That second failure has causes worth naming. A common one: vendors can't understand their own log formats. The core contributors to this are inheritance of logging packages from open source (nobody on the current team wrote the original code) and internal contributors who have left the organisation. As confusing as log formats are, they are often under-documented or the actual implementation has drifted quite far from initial scope.
The second issue in accurate translation to a format is lack of understanding of the output format. Cybersecurity formats require a deep understanding of the craft. Most software engineers working at cybersecurity companies (including WitFoo) are not domain experts on the craft of cybersecurity or on these formats. It creates a myriad of problems. A common example is lack of understanding of RFC 5424, which defines severity levels in syslog. The RFC defines 0 as Emergency and 7 as Debug. A software engineer unfamiliar with the RFC would not necessarily know that, and might quite reasonably assume severity is 1 (low) to 10 (critical). We see this error in a wide array of logs (most famously in McAfee ePO). I point it out because the step of translation is critical, and it is extremely complicated. One good use of the Precinct 6 dataset is to detect exactly these kinds of translation errors in existing datasets.
I wrote about this class of problem in 2014 in Parsing Vendor Claims of APT Detection, and again in 2023 in 4 Types of Data Analytics. It has not gotten better. It might actually be getting worse as vendors layer AI on top of the same broken parsers.
The reason translation matters so much is this: if you get it right, cybersecurity analysts can elevate their work away from being tool-specific and focus on the business level instead. Rule-sets can be universal. If signals are mapped to frameworks like MITRE ATT&CK®, detection rules become as simple as alert on any MITRE ATT&CK technique detected across any tool. That simplifies rule management, and (importantly) it is itself an example of EP applied to the analyst's workflow. The message has already been tagged and shaped into a form that is ready for human handoff. The broader argument for EP as a human-centric model is in my November 2025 post, Empathetic Processing and Temporal Link Analysis: Research Pathways for AI in Cyber Defense.
OK. Enough theory. Let me put it to work.
Phase 1: Standardise format
Assume you're feeding data into Grafana (it could be Sentinel, Splunk, Elastic, or any other data lake; the approach is the same). You can craft the following prompt to accomplish phase-1 Empathetic Processing using your own tooling:
Analyse the WitFoo Empathetic Processing whitepaper at https://www.witfoo.com/documents/Empathetic-Processing-Whitepaper.pdf. Study the dataset at https://huggingface.co/datasets/witfoo/precinct6-cybersecurity-100m to understand how signals are translated into WitFoo artifacts. Once you understand that, analyse my data in Grafana. If formats in my data are not in the 100m dataset, examine https://huggingface.co/datasets/witfoo/syslog-to-artifact for matches. For any missing translations, search the internet using the same WitFoo EP approaches. I would like to create Grafana parsers that translate my data into WitFoo Artifacts before storing in Grafana.
That prompt will create parsers for your existing data and get it into a standardised format. If you would prefer a different output schema (OCSF, ECS, your own), provide the LLM with documentation for your target format and have the parsers do one additional transformation. If nothing else, this creates a pipeline that fully parses your data, which is already more than most of the industry is doing.
Phase 2: Alert rules
Next is flagging alerts. That is easily accomplished with this prompt:
Analyse the lead rules included in the Precinct 6 dataset here: https://huggingface.co/datasets/witfoo/precinct6-cybersecurity-100m/blob/main/reference/lead_rules_catalog.json. Build detection logic utilising those rules.
The lead_rules_catalog.json file is the rule corpus WitFoo's own platform uses to triage incidents. Using it as a reference for your own detection logic means your rules start life aligned with something that has been sharpened on real adversary behaviour, not on whatever last week's marketing blog told a vendor to chase.
Phase 3: Visualisation
With the data and the rules in place, we can now generate dashboards.
Analyse the new WitFoo-structured index. Generate Grafana dashboards that show MITRE ATT&CK® hits. Also add dashboards I can use to hunt for anomalies.
Three prompts. You now have an MITRE ATT&CK-compatible dashboard on top of your own data sources, and you can query and visualise across a normalised schema. The heavy lift (building a deterministic, security-aware parsing layer that understands 200+ language families) has already been done and is sitting on Hugging Face waiting for you.
Where this goes next
The phases above are the signal-comprehension half of EP. The next phases pivot these signals to graph theory and apply WitFoo's research in Temporal Link Analysis (TLA). The Precinct 6 dataset has additional subsets for graph nodes, edges, and incidents, and a similar prompt-driven approach works on a graph-friendly backend like Neo4j or Cassandra 5.0 with SAI.
The theory behind the graph work sits in the November 2025 EP/TLA post.
Wrap Up
I wanted this post to do two things: show a practical use of the Precinct 6 dataset in stacks people already own, and draw attention to the gaps in current detection pipelines that the dataset can help you find. Translation errors are invisible until you put a well-labelled reference corpus next to your current pipeline; then they jump off the screen. The same is true for missing rules, missed techniques, and dashboards that visualise the wrong thing.
The dataset and the sanitisation pipeline are both Apache 2.0. The only thing I ask in return is that if you find a translation error (ours or someone else's) you say so publicly. That is how the craft gets better. I hope this, and the datasets we have published, help you keep your organisations and communities safer from cybercrime.
