Engineering Real-Time Vulnerability Intelligence: Inside Farion's Architecture

Christoph Ebeling, Founder of Farion

On paper, software vulnerability data looks like a static catalog. A bug is discovered, a CVE identifier is assigned, and a severity score is attached. If you design a security system around that assumption, the architecture is simple: package a snapshot of public feeds into a scanner container, compare it against an application's dependencies during a build, and output a report.

In production environments, vulnerability intelligence behaves less like a static catalog and more like an eventually consistent stream of conflicting state updates.

A language ecosystem feed like RustSec or a vendor security tracker often publishes an advisory first. The corresponding CVE record arrives later, sometimes the following day. Threat scoring such as EPSS follows in a subsequent update, while Linux distributions independently patch older release branches and revise their affected version ranges. Throughout this sequence, the deployed application has not changed, yet the inputs required to evaluate its dependencies have shifted multiple times.

If an assessment pipeline only evaluates dependencies when a scan is triggered, the accuracy of its findings is coupled to the timestamp of the last scan.

Decoupling vulnerability assessment from scan timestamps changes the systems engineering problem. Instead of running a batch lookup job during CI/CD, we have to maintain a continuous matching pipeline against known dependency states while simultaneously serving high-concurrency lookups for new scans. Building and operating that system forced us to solve four connected problems: collecting and versioning primary source data without losing provenance, serving package-specific context in memory for large Software Bills of Materials (SBOMs), protecting our PostgreSQL cluster from replication lag as our historical archive approached 500 GB, and using that accumulated revision history to study how advisories mature over time.

Direct Collection and the Provenance Problem

Relying on secondary aggregators or a single upstream feed introduces two failure modes for automated analysis. First, you have no control over when records are updated or whether historical revisions are overwritten. Second, a generic CVE summary rarely contains the package-level context required to determine whether a specific library build is actually affected.

Consider how upstream maintainers and Linux distributions record fixes for the same vulnerability. In curl's upstream advisory for CVE-2024-7264 (an ASN.1 date parser out-of-bounds read in libcurl), the project lists an overall affected range of 7.32.0 through 8.9.0, with reachability depending on the TLS backend, and identifies 8.9.1 as the fixed release. An analysis engine using only upstream version ranges could therefore classify a patched Debian 12 (Bookworm) package as vulnerable, without considering the distribution's backport or the build's TLS backend. However, Debian's security tracker for CVE-2024-7264 records that the Bookworm package was patched via backport in version 7.88.1-10+deb12u7. Both records are accurate within their respective boundaries, and merging them into a single flattened version range breaks assessments for one of the two environments.

To capture these distinctions directly from primary sources, we run dozens of independent workers and scrapers covering vendor security advisories, programming language ecosystems such as RustSec, Linux distribution trackers, public package registries, and GitHub and GitLab repositories (where we also collect original software licence texts), alongside threat scoring feeds like EPSS. Main advisory feeds are scraped continuously, while secondary sources run on staggered schedules.

Owning this collection layer is also a deliberate sovereignty decision. Because our downstream analysis services depend on this database, we need full control over which records we retain, how their version histories are stored, and how internal consumers retrieve them, rather than relying on data bundled into an external scanner image.

Ingestion, Explicit Consolidation Rules, and Versioning

Scrapers publish raw records into Kafka, which acts as the ingestion backbone for our consolidation, vulnerability, and packaging services. Our operational target inside the vulnerability service is to process incoming updates within a maximum latency window of ten minutes.

As overlapping records arrive through Kafka, the consolidation pipeline has to reconcile them without destroying the underlying source evidence. Rather than using an opaque model to guess which feed is right, our consolidation logic follows explicit, deterministic rules informed by ongoing operational reviews of source quality. We track how advisories from different providers develop over time and which sources consistently supply complete, accurate metadata for specific technologies. When existing feeds lack sufficient detail for an ecosystem, we add direct scrapers for primary sources in that space.

Every raw record is preserved and versioned. From those raw inputs, the pipeline generates two distinct consolidated views:

Consolidated View	Engineering Purpose
General view	Provides a clear, human-readable explanation of the vulnerability across affected products.
Technology-specific view	Preserves the exact package, ecosystem, and distribution parameters needed by downstream analysis agents.

Retaining the raw source history alongside the consolidated outputs makes merge decisions reversible. When a vendor revises an advisory days later, or when our source-quality reviews lead us to adjust our consolidation rules, we can re-evaluate the record from the original inputs rather than trying to reconstruct state from a destructive overwrite. It also allows engineers investigating an incident to inspect the exact intelligence state that was recorded at an earlier point in time, before subsequent CVE updates or severity changes arrived.

Continuous Matching vs. Exploitability Reassessment

When a newly consolidated update changes our knowledge of a vulnerability, the pipeline matches that update against existing dependency records already known to the system. Customers do not need to trigger a new scan to see that an advisory affecting their software has changed.

In designing this workflow, we separated three operations that are often conflated in vulnerability management tools: updating the underlying finding state, notifying users, and re-evaluating code exploitability.

1. State updates are continuous: As soon as new intelligence is reconciled, the affected findings and their historical timelines are updated.

2. Notifications are decoupled from ingestion: Flooding engineering teams with an alert every time an upstream tracker tweaks a version range creates alert fatigue. By default, changes are batched into a daily morning update, though teams that want immediate notifications can opt into real-time delivery.

3. Exploitability reassessment is triggered selectively: Our exploitability evaluations rely on static data-flow and taint analysis of the application. Because the data-flow graph reflects the structure of the code itself, routine metadata updates to an advisory usually do not require re-running the heavy data-flow analysis. However, if new advisory details alter the relevant trigger conditions, an exploitability reassessment can be run optionally without requiring a full rescan of the repository.

Serving Concurrent SBOM Lookups in Memory with Rust

While Kafka handles the asynchronous ingestion and update pipeline, internal services and exploitability agents need request-driven access to current intelligence. An enterprise SBOM routinely contains several thousand libraries. When multiple scans run concurrently, each resolving thousands of package URLs (PURLs) while background services also query advisory context, lookup latency directly dictates the throughput of the entire analysis pipeline.

To keep read latency low, we hold the active serving dataset of vulnerabilities and advisories in memory and expose it to internal consumers through a GraphQL API. A typical query passes a PURL to the GraphQL layer, which combines data across our backing stores and returns the technology-specific advisory context required by the calling analysis service or exploitability agent. To support availability and limit cross-zone network traffic, we run our serving infrastructure and read replicas across three AWS availability zones.

Our choice of programming language for these services grew out of an earlier workload. Roughly eighteen months ago, we adopted Rust for our taint-analysis engine, which performs fixed-point computations and breadth-first graph traversals where predictable memory layout and the absence of garbage-collection pauses were critical.

When we built out the scraper fleet and the vulnerability serving layer, we standardized on Rust for workload-specific reasons:

• Fleet density and idle footprint: Running dozens of independent scraper services with high availability means per-service overhead adds up quickly. Compiling scrapers to small native binaries inside minimal containers keeps idle memory consumption low when workers are waiting between scheduled runs, compared to our prior experience with dynamic languages and Go runtimes.

• Predictable memory behavior in the GraphQL layer: The vulnerability service has to hold active lookup structures in memory while executing complex GraphQL resolvers that stitch together records from PostgreSQL and object storage. Rust's ownership model and explicit allocation lifetimes let us reason about memory usage under concurrent scan spikes without provisioning for runtime collector overhead.

Controlling memory and compute inside the application layer solved our lookup latency, but it exposed a much harder bottleneck in our storage tier.

The Hardest Operational Problem: PostgreSQL Under Ingestion Pressure

As our scraper coverage expanded and our archive of versioned advisories and software licences grew toward 500 GB, keeping our PostgreSQL cluster healthy became our primary engineering challenge.

Initially, PostgreSQL served as the single persistent store for current operational intelligence, full historical advisory revisions, scraped licence texts, and time-series metrics. Because we distribute readers across three AWS availability zones to keep local reads fast, write bursts on the primary instance must be replicated and replayed across zones. As documented in PostgreSQL's streaming replication architecture, sustaining high write volume while serving complex reads puts pressure on write-ahead log (WAL) shipping and replay, which surfaced in our environment as significant replication lag on the read replicas.

We were effectively allowing bulk archival writes and analytical queries to degrade the replicas serving live security assessments. We addressed this through two architectural changes: urgency-based ingestion throttling and workload-specific storage separation.

1. Priority-Based Backpressure on Writers

Not every record entering Kafka has the same operational urgency. A newly published security advisory or an updated affected-version range needs to reach the serving layer quickly. A scraped software licence text or a routine documentation update does not.

We modified our writer pipeline to regulate Kafka ingestion based on database pressure. When write pressure or replica lag climbs, the writers throttle lower-priority ingestion streams, holding licence texts and documentation updates in the queue for a minute or two. Accepting a brief delay on archival material preserves write capacity and replication headroom for urgent advisory updates.

2. Splitting Storage by Access Pattern

Throttling smoothed out ingestion bursts, but the underlying dataset was still too large and varied for a single relational database profile. We decomposed the persistence layer according to how each data type is written and queried:

Storage Component	Workload Responsibility
In-memory dataset	Hot vulnerability and advisory lookups by PURL for concurrent scans and internal analysis agents.
PostgreSQL	Operational state required for current threat intelligence, active dashboards, and finding timelines.
RustFS	Distributed, S3-compatible object storage for offloaded historical licence texts and lengthy or older advisory revisions.
ClickHouse	Indexing and serving time-series metric queries directly, removing analytical scans from PostgreSQL.

Offloading bulky historical licence texts and older advisory bodies into RustFS shrank the operational footprint inside PostgreSQL, while routing time-series queries to ClickHouse eliminated heavy analytical reads from the transactional replicas. When an engineer or an internal workflow requests the full text of a historical advisory, the vulnerability service retrieves the detailed payload on demand without keeping every historical document inside the primary relational tables.

Using the Revision Archive for Model Research

Going through the operational effort to preserve almost 500 GB of historical advisory states, rather than simply overwriting rows with the latest feed data, was driven by our research roadmap as much as by auditability.

Over time, this archive has become one of our most valuable assets for building agent knowledge graphs and training internal security evaluation models. It allows us to study not just what vulnerabilities exist today, but how vulnerability information evolves across different ecosystems and providers.

One active research problem we are working on addresses how early uncertainty and writing style skew automated risk prioritization. When a vulnerability is first disclosed, the initial advisory is frequently incomplete. Depending on the author and source, an uncertain Day 0 report may be written in alarming language to encourage caution. Conversely, an advisory revised two weeks later might sound far more dangerous simply because researchers have filled in concrete technical mechanics and exploit conditions, or it might narrow the actual risk by restricting the affected configurations.

If an evaluation model relies naively on surface wording, its assessment will drift based on rhetorical style and reporting maturity rather than the underlying technical evidence.

Because our database retains the sequence of revisions as we observed them, we can construct training pairs that compare an advisory on the day of initial disclosure with its version roughly two weeks later. Studying the delta between Day 0 and Day 14 across different advisory sources gives us the training material to help models account for incomplete early reporting and normalize across different writing styles.

Summary of Architectural Trade-Offs

Every layer of this pipeline reflects a trade-off dictated by the workload:

• Direct collection over feed aggregation: Maintaining dozens of primary-source scrapers requires continuous data hygiene and source-quality review, but it gives us package-specific context and sovereignty over our historical dataset.

• In-memory GraphQL reads over direct database queries: Holding active advisory mappings in memory in Rust services adds operational state to manage, but keeps multi-thousand-package SBOM lookups from bottlenecking concurrent scans.

• Selective ingestion throttling over uniform FIFO processing: Delaying licence and documentation writes by one to two minutes under database load introduces a minor lag for archival data, in exchange for keeping PostgreSQL read replicas responsive across availability zones.

• Polyglot persistence over a single database: Splitting data across PostgreSQL, RustFS, and ClickHouse increases architectural surface area, but matches each storage engine to its actual access pattern.

Vulnerability intelligence rarely arrives complete on day one. Designing our infrastructure around the assumption that security data will mutate allows us to keep assessments current as new evidence emerges, while preserving the historical record needed to build better analysis tools.