Today’s cybersecurity teams need comprehensive access to threat intelligence, detailed reports, and the latest news updates, but must navigate multiple tools, databases, and external resources in their daily work. This complexity often leads to information overload and increases the risk of missing critical information for correlations that could help prevent or mitigate cyberattacks. While cloud-based language models may reduce operational costs, they introduce significant data privacy concerns, especially for critical infrastructure, and create dependencies on external providers. In contrast, deploying LLMs on-premise provides a way to maintain full control over sensitive data and local infrastructure, although it demands substantial computational resources. Smaller models can operate with fewer GPUs but can only manage simpler queries, whereas larger models can handle more complex tasks if given more robust hardware, particularly powerful GPUs.
Our initial minimum viable product (MVP) used a quantized Llama 7B model to query a threat database and process various threat reports. Early evaluations suggested that increasing the model’s size would lead to more accurate and insightful responses. To explore this, we utilized two locally hosted NVIDIA L40 GPUs, providing a total of 80GB of VRAM, to run a 70B model using INT4 quantization. While this scaling was intended to improve performance for both structured queries and document search, generating accurate SQL queries from natural language posed a major challenge. The threat database schemas often reflect complex business logic and relationships, conditions that hindered the model’s ability to produce reliable SQL queries directly from natural language prompts without having the context of database table's business context and not having enough subject matter experts to evaluate the queries and their outputs. Although document retrieval and analysis worked effectively, the text-to-SQL issue became the principal technical hurdle, which made us reconsider our overall approach.
In response, we shifted our strategy. Instead of starting with hardware limitations, we tested a full 70B model via API-based services—representing a model roughly 70% larger than its quantized counterpart. This allowed us to confirm model capabilities before finalizing on-premise deployments. By doing so, we established clearer requirements for clients who plan to implement local LLM solutions, spanning from initial API evaluations to full on-premise setups with appropriate infrastructure.
After confirming the model’s potential, our focus turned to data architecture. We integrated data from multiple security sources, including Tenable, Qualys, Microsoft Defender, and the National Vulnerability Database (NVD), into a unified schema. This consolidation allows security analysts to use natural language queries to access both structured security data—such as vulnerability scan results, endpoint alerts, and CVE data—and unstructured data like threat intelligence reports and advisories. By prioritizing data integration and simplifying the underlying database structures before final deployment, we created a system that preserves data sovereignty, enables efficient retrieval of security-relevant information from multiple sources, and operates entirely without reliance on external cloud providers.
Project Lead
Sonja Judith Fink
IKARUS Security Software GmbH
office(at)ikarus.at
Partners
CyberACI GmbH
+43 1 58995-0
Contact
Sonja Judith Fink
Blechturmgasse 11, 1050 Wien
01 58995 0
office(at)ikarus.at
https://www.IKARUSsecurity.com
https://www.cyberaci.com/minerva/
