At Red Canary, storing large amounts of small data is a big deal.
Their Linux Endpoint Detection and Response (EDR) sensors run on hundreds of thousands of endpoints (VMs, laptops, EC2 instances, etc.) They continuously generate small payloads with telemetry details (file changes, process launches, and network connections), as well as self-reported health checks. Uploading all these small payloads to S3, which charges per PUT operation, became quite costly at scale.
The logical approach – merging data into larger chunks before uploading them to S3 — did cut costs. However, it also introduced file-size limits, delayed data freshness, and complicated debugging of specific endpoint records. After some experimentation, they cut latency and costs with an unexpected approach: storing large amounts of small data in a write-optimized database, ScyllaDB.
“After some experimentation, they cut latency and costs with an unexpected approach: storing large amounts of small data in a write-optimized database, ScyllaDB.”
Andres Medina, a staff software engineer on the LinuxEDR team, took the Monster Scale Summit stage to share their journey from S3 and Postgres to a Firehose buffer to ScyllaDB – and the lessons learned along the way. You can watch his complete talk or read the highlights below.
Checking the health check data flow
Medina’s team decided to start pipeline optimization with the health check data. As a lower-risk part of the pipeline than the actual telemetry data, it seemed like the safest spot for experimentation.

Problem 1: cost
The cost problem was the most pressing pain that they wanted to address. “S3 charges a flat fee per S3 PUT, which occurs for every payload upload,” Medina explained. “So if we have hundreds of thousands of endpoints, we’re being charged for hundreds of thousands of S3 PUTs.”

They hoped that creating a buffer would provide a short-term fix, at least. Using AWS Firehose and Kinesis, they combined small data streams into larger files and wrote them to S3 once they reached a defined threshold. It did reduce costs – but at the cost of introducing new issues:
- Firehose and Kinesis impose a strict one-megabyte-per-shard limit, and the larger files just didn’t fit.
- Visibility into the health data was delayed. Before, it was passed over to S3 immediately. With the change, they had to wait until the designated threshold was met.
- Debugging specific endpoints became difficult. When lots of endpoint IDs are mashed together, it’s much harder to find a particular endpoint in a large customer fleet.

Problem 2: database scalability
Another issue: Postgres was used to store endpoint metadata, and it was just overwhelmed at their scale. “Sometimes a large customer might restart a chunk of their fleet,” Medina shared. “That means that many endpoints will try to check in at very similar times, causing a spike of writes in the database.” They also had to throttle their incoming database requests, as sudden bursts could take down the database.
Yes, they could have tweaked and tuned their existing database for yet more short-term solutions. But the team wanted a longer-term fix.
Looking for a longer-term solution
The team set the following criteria for a new approach:
- Ability to sustain their large and growing throughput
- Improve time-to-visibility in their portal
- Minimal admin burden
They ultimately chose ScyllaDB Cloud, with the Bring Your Own Account (BYOA) deployment option. Medina explained, “We installed it in our own AWS account, in a separate account from our remaining data to reduce any risk there. That way, ScyllaDB can still manage that cluster without having access to our data.”
Their shift to ScyllaDB involved two phases: replacing Postgres, then replacing S3.
Replacing Postgres
Replacing Postgres was simpler than replacing S3. Each sensor event appends a single row, and the web portal always queries it in a time series manner – asking for a customer’s active endpoints over a given period of time. Still, Postgres and ScyllaDB are very different databases, which pushed the team to learn a few lessons regarding tombstones and uneven data volumes.
When you delete data in ScyllaDB, it isn’t just removed. ScyllaDB leaves a timestamped marker called a tombstone. This prevents data resurrection, which occurs if a node misses a deletion (e.g., due to a network error) and later restores the old data during a cluster repair.
Tombstones were particularly troubling for Red Canary’s ephemeral endpoints, which are online only for a short time. Medina said, “Some clusters might scale up for an hour and then scale back down, which means an endpoint is alive for a very small amount of time. That meant some customers had partitions that were mostly just tombstones. When you query it, ScyllaDB has to iterate through all these tombstones, making our latency worse than it should be.”
The fix here was simple. Since they don’t need to delete any data, they didn’t need tombstones. They cleared them from the read path by setting a short TTL on the table and tuning tombstone garbage collection to evict tombstones faster. The latency improved as a result.
“Some clusters might scale up for an hour and then scale back down. That meant some customers had partitions that were mostly just tombstones. When you query it, ScyllaDB has to iterate through all these tombstones, making our latency worse than it should be.”
The second issue was that their data volume varied wildly across customers. Some customers have a few dozen endpoints, while others have hundreds of thousands. To avoid creating hot partitions in ScyllaDB, Medina and the team developed a bucketing strategy.
They split each customer’s data into smaller numbered “buckets,” which ScyllaDB distributes across shards. When fetching a customer’s data, the system checks how many buckets they have and queries them all at once (in parallel) for faster responses.
The following diagrams compare the short-term solution to the long-term one. In the After diagram, the Postgres endpoint database is gone. Instead, the service forwards sensor health data to ScyllaDB, and the portal queries the service for the metadata needed for recently active endpoints.


Replacing S3
The next challenge was the unconventional move to replace S3 usage with ScyllaDB. Throughput was higher, and access patterns were less predictable. Medina explained, “It’s not a service trying to query in one way. Instead, it’s developers debugging.” They could predict one thing at least: developers would debug one specific sensor at a time. So, they defined “sensor ID” as the partition for their queries.
There was an ulterior motive here. Medina shared, “One of our biggest reasons for wanting to do this was to see if ScyllaDB could also be used for our detection data in the future.” Since the health data had similar throughput and was less critical to their operations, it was a safe way to test the waters for a broader shift to ScyllaDB.
The first complicating factor in the S3 replacement was that their events contain very large arrays used to track a given process’s memory consumption at a given endpoint. Some endpoints have thousands of processes running at the same time, which could create rather large rows. The solution was to unnest the array and store it in an auxiliary table, tracking each process’s memory usage separately from the primary table. “We can still put every row in the same partition so that our querying is still easy enough, as long as the partition doesn’t get too big (too many rows),” Medina added.

Next up: batching. They wanted to batch rather than write each health event individually to avoid multiple network round trips to the database. However, batching is more efficient if every statement in a batch belongs to the same partition. Ideally, they split each health payload’s statements by partition when they receive it. The ScyllaDB Rust driver wasn’t designed for this, though.
Wrangling the Rust driver
More specifically, the Rust driver doesn’t expose a prepared statement with values already serialized (a bound statement). To get a partition for a statement with values, you would have to serialize it once to get the corresponding hashed token, then serialize it again to execute the statement.
If you try to keep these values unserialized together with the statement, it’s difficult to put them into collections in Rust (a Rust collection like a vector or hash map requires everything to be the same type). If different statements have different types, it’s complicated. However, once you serialize them, those values are just a stream of bytes. You can keep all that data in the same collection.
Medina detailed his workaround like this: “To get around this, I forked the Rust driver to add the idea of a bound statement and then a bound batch, which is just a collection of statements. With this now in place, we can ask the bound statement what its token is without having to reserialize it, and then execute it again without reserializing it. A second benefit is that we can separate CPU work from I/O work. In Rust, you mostly use Tokio for I/O operations, and doing CPU work there can block the runtime and reduce efficiency.
“When we made this change, our average time to process a file dropped from about 150 milliseconds to 50 milliseconds — a drop of about 60%.”
So now we can do all of the CPU work in a CPU-bound worker using Rayon, and then we send it back to Tokio when we’re done. Then in Tokio, you can just do the I/O for the batch execution. When we made this change, our average time to process a file dropped from about 150 milliseconds to 50 milliseconds — a drop of about 60%. Of course, this is for our use case, so your mileage may vary.”
Did it work?
Here’s the final system architecture.

The LinuxEDR services now own the health data for the LinuxEDR product, and that data no longer goes to the engine. ScyllaDB stores all the check-ins and health metrics. There’s a separate metrics database for now, but that might be replaced soon. The portal can query the services to get the list of active endpoints, and any developer can query ScyllaDB for debugging.
So far, the team has significantly reduced their per-endpoint cost. Batching has boosted their I/O per second, with a ~20X reduction in writes versus rows inserted (~670K rows/second, but only ~36K writes/second due to batching). And the portal now retrieves endpoints in seconds instead of minutes for its largest customers. “Each query to ScyllaDB is in the double-digit millisecond range,” Medina noted.
Next steps
Medina wrapped up with this:
“Since the transition has gone fairly smoothly, we’re now exploring using it for data that is more critical. One of my team’s main goals is to store our detection data in ScyllaDB to avoid sending data that doesn’t need to go to the engine. Also, we’re looking at ways to use it outside my team to replace some uses of OpenSearch, which has been giving us issues with cost, performance, and sometimes paging…no one likes being paged. Ideally, I would also like to merge the fork at some point so I don’t need to rebase the driver on every new release.”
The post S3 vs. ScyllaDB for tiny telemetry: What Red Canary learned the hard way appeared first on The New Stack.



