Originally published on the HostingPodium Enterprise Engineering Blog.

Free SRE & Developer Diagnostics: Test your stack live with our Global DNS Propagation Checker & SSL Health Auditor, Free Website Speed & Core Web Vitals Checker, or explore all 9 Free Webmaster & Cloud Engineering Tools.

Architectural guide detailing how to replace resource-heavy ELK stacks with ClickHouse columnar storage for high-cardinality log analytics and trace observability at 10x lower cost.

Architectural Imperatives for High-Volume Observability

Traditional log management stacks built on Elasticsearch and Logstash suffer from exorbitant memory overhead, catastrophic garbage collection pauses, and non-linear storage expansion costs when ingesting high-cardinality telemetry. At scale exceeding 50,000 events per second, the inverted index architecture of Elasticsearch becomes an operational liability. Indexes require massive RAM allocations—frequently exceeding 30% of total raw data volume—just to maintain segment metadata and term dictionaries. This operational reality creates severe financial friction for engineering organizations processing petabytes of telemetry daily.

At HostingPodium, our infrastructure engineering group migrated tier-1 production clusters from the ELK stack to a column-oriented architecture utilizing ClickHouse. By leveraging vectorized execution engines, aggressive columnar dictionary encoding, and pluggable storage engines, this migration reduced infrastructure total cost of ownership by an average of 11.4x while accelerating analytical query performance by two orders of magnitude. This guide details the exact architectural patterns, kernel tunings, ingestion topologies, and data lifecycle policies required to implement high-cardinality observability at scale.

Columnar Storage vs Inverted Indexes

The foundational architectural divergence between Elasticsearch and ClickHouse lies in how data is stored and retrieved. Elasticsearch relies on Lucene inverted indexes, mapping terms to document IDs. To accelerate range scans and aggregations, field values must be duplicated in doc_values structures, doubling the storage footprint. Conversely, ClickHouse utilizes a pure columnar storage engine. Values for a single attribute are stored sequentially on disk, enabling ultra-fast vector math and SIMD (Single Instruction, Multiple Data) optimizations.

Consider the memory footprint comparison for a 10TB uncompressed log corpus containing rich JSON attributes, Kubernetes metadata, and distributed tracing identifiers:

Metric Elasticsearch (Lucene) ClickHouse (MergeTree)
RAM Footprint per TB Raw ~300 GB (Index + Heap) ~15 GB (Marks & Indexes)
Storage Compression Ratio 2.5:1 (Zlib default) 8:1 to 12:1 (ZSTD level 3)
Ingestion CPU Cost High (JSON parsing, inverted index building) Low (Direct columnar streaming)
Query Latency (P99 Scan) Seconds to Timeout on Wildcards Milliseconds (Parallelized CPU vectorization)

By eliminating the inverted index entirely, ClickHouse reduces the working set memory requirements from hundreds of gigabytes of heap to a lean collection of primary index files (.idx) and column mark files (.mrk). This allows the Linux page cache to effectively cache the entire index metadata set without thrashing.

Vector and Fluent Bit Ingestion Pipelines

High-performance log ingestion requires decoupling collection agents from analytical storage engines using an intermediate buffering and transformation layer. While Logstash introduces heavy JVM memory overhead and high processing latency, Vector and Fluent Bit provide low-latency, memory-safe data routing written in Rust and C respectively.

For large-scale deployments, Vector acts as the primary daemonset within Kubernetes clusters, scraping container standard output, parsing JSON payloads, enriching records with cluster metadata, and streaming them directly to ClickHouse via the native binary protocol over TCP port 9000 or HTTP payload batching.

sources: kubernetes_logs: type: kubernetes_logs extra_namespace_annotations: ["hostingpodium.com/scrape"] transforms: parsed_logs: type: remap inputs: ["kubernetes_logs"] source: | .parsed, err = parse_json(.message) if err == null { . = merge(., .parsed) del(.message) } .timestamp = to_unix_timestamp(._timestamp, unit: "nanoseconds") sinks: clickhouse_sink: type: clickhouse inputs: ["parsed_logs"] endpoint: "http://clickhouse-cluster.monitoring.svc:8123" database: "observability" table: "app_logs" auth: strategy: "basic" user: "vector_writer" password: "${CLICKHOUSE_PASSWORD}" batch: max_events: 50000 timeout_ms: 1000 encoding: codec: "json" 

To optimize ingestion throughput, Vector must utilize batching parameters aligned with ClickHouse part generation limits. Writing tiny batches (under 10,000 rows) causes excessive small-part proliferation, forcing the ClickHouse background MergeTree cleaner to consume excessive CPU cycles combining parts.

Designing the MergeTree Schema for High-Cardinality Telemetry

Defining the table schema determines the query latency profile and storage efficiency of the cluster. The primary key in ClickHouse does not enforce uniqueness; instead, it determines the sparse index sorting order on disk. Choosing an optimal ORDER BY clause is critical for high-cardinality trace analytics and log filtering.

CREATE DATABASE IF NOT EXISTS observability; CREATE TABLE observability.app_logs ( timestamp DateTime64(3, 'UTC') CODEC(DoubleDelta, ZSTD(1)), level LowCardinality(String) CODEC(ZSTD(1)), namespace LowCardinality(String) CODEC(ZSTD(1)), pod_name String CODEC(ZSTD(3)), container_name LowCardinality(String) CODEC(ZSTD(1)), trace_id String CODEC(ZSTD(1)), span_id String CODEC(ZSTD(1)), message String CODEC(ZSTD(3)), attributes Map(String, String) CODEC(ZSTD(3)), INDEX idx_message message TOKENBF_V1(30720, 2, 0) ) ENGINE = MergeTree() PARTITION BY toYYYYMM(timestamp) ORDER BY (namespace, level, timestamp) SETTINGS index_granularity = 8192, storage_policy = 'tiered_storage'; 

Key design considerations for this schema include:

  • LowCardinality Data Type: Strings with fewer than 10,000 distinct values per data part (such as log levels, namespaces, and container names) must be wrapped in LowCardinality(String). This instructs ClickHouse to maintain an integer-to-string dictionary dictionary lookup, drastically accelerating grouping and filtering operations via bit-vector operations.

  • Map Data Types: High-cardinality metadata and arbitrary user tags should be stored in Map(String, String) rather than flattened columns. This avoids schema explosion while allowing efficient retrieval via functions like mapContainsKey and element(attributes, 'key').

  • Secondary Indexes: The TOKENBF_V1 bloom filter index on the message column provides sub-linear substring searching without the storage penalty of an inverted index, skipping data blocks that do not contain specific tokens.

Memory Footprint Optimization and Kernel Tuning

ClickHouse relies heavily on the Linux virtual memory subsystem. To prevent out-of-memory (OOM) kills during heavy concurrent aggregations, production nodes require explicit configuration adjustments at both the application and kernel layers.

First, configure the memory limits inside /etc/clickhouse-server/users.d/limits.xml:

 32000000000 128000000000 0.5 

At the operating system level, configure transparent huge pages (THP) and virtual memory memory overcommit ratios in /etc/sysctl.d/99-clickhouse.conf to maintain predictable latency profiles under saturated I/O conditions:

vm.swappiness = 1 vm.max_map_count = 262144 vm.overcommit_memory = 1 

Disable Transparent Huge Pages via systemd or grub, as THP frequently introduces kernel locking contention during memory allocations in large hash tables utilized by ClickHouse GROUP BY aggregations.

Data Lifecycle Management and Tiered Storage

Retaining petabytes of logs on high-performance NVMe storage arrays is economically unsustainable. ClickHouse provides native tiered storage mechanics via storage policies, allowing seamless migration of cold log partitions from NVMe to object storage (AWS S3, MinIO, or Ceph RGW) without altering table schemas or application query logic.

Configure tiered storage within /etc/clickhouse-server/config.d/storage.xml:

 /var/lib/clickhouse/disks/nvme/ s3 https://s3.us-west-2.amazonaws.com/bucket/clickhouse/ ${S3_KEY} ${S3_SECRET} nvme>/disk> s3_cold 0.1 

Combine this policy with automated TTL (Time-To-Live) expressions directly on the table definition to execute background data movement:

ALTER TABLE observability.app_logs MODIFY TTL timestamp + INTERVAL 7 DAY TO VOLUME 'cold', timestamp + INTERVAL 90 DAY DELETE; 

This automated data migration ensures that frequent recent queries execute against local NVMe page caches, while historical compliance auditing queries transparently pull blocks from cost-effective object storage over high-bandwidth internal networks.

Operational Runbook: Handling Part Merges and Disk Pressure

Monitoring a production ClickHouse cluster requires tracking specific metrics exposed via the system.metrics and system.asynchronous_metrics tables. The most critical operational failure mode is "Too many parts" (ErrorCode: 252), which occurs when ingestion rate exceeds the background MergeTree processing capacity.

Run the following diagnostics query to identify tables experiencing part explosion:

SELECT database, table, count() as parts_count, sum(bytes_on_disk) / 1024 / 1024 / 1024 as size_gb FROM system.parts WHERE active GROUP BY database, table ORDER BY parts_count DESC LIMIT 10; 

If part counts exceed 3,000 per partition, immediately scale background merge threads by executing a dynamic settings adjustment:

SET max_background_merges_mutations_threads = 16; 

To prevent ingestion stalls, ensure that producers implement exponential backoff when receiving HTTP 429 or ClickHouse native TCP throttling errors. By coupling Vector's durable disk buffering with ClickHouse's high-throughput columnar storage, engineering teams achieve resilient, petabyte-scale observability at a fraction of legacy operational costs.

Production Verification Checklist & Operational Runbook: ClickHouse Log Analytics & High-Cardinality Observability: Replacing ELK at 10x Lower Cost

Before promoting any ClickHouse Log Analytics & High-Cardinality Observability: Replacing ELK at 10x Lower Cost deployment into production traffic, senior platform and SRE teams validate deterministic failure handling, kernel resource ceilings, and observability telemetry across clickhouse mergetree log compression zstd, vector fluent-bit log ingestion pipeline, high-cardinality trace analytics, elasticsearch vs clickhouse memory footprint. Unlike lab prototypes, production clusters must survive upstream network partitions, sudden concurrency bursts, and rolling node maintenance without dropping in-flight client transactions.

Pre-Cutover Acceptance Gates

  • Synthetic Load & Tail-Latency Benchmarking: Execute sustained concurrency tests against staging endpoints for ClickHouse Log Analytics & High-Cardinality Observability: Replacing ELK at 10x Lower Cost, verifying that p95 and p99 response latencies remain within contractual SLO boundaries under 3x peak projected load.

  • Fault Injection & Chaos Verification: Simulate primary worker termination, AZ link degradation, and storage IOPS saturation to confirm automated failover and circuit-breaker recovery complete within sub-second thresholds.

  • Telemetry & Alert Routing Validation: Confirm Prometheus scrape targets, structured JSON log pipelines, and OpenTelemetry trace propagation emit actionable metrics for clickhouse-log-analytics-high-cardinality-observability-vs-elk before live traffic cutover.

  • Immutable Rollback Readiness: Verify that declarative GitOps manifests, database migration snapshots, and previous container image digests can restore the prior production state in under 120 seconds.

# Production Readiness Verification Probe (clickhouse-log-analytics-high-cardinality-observability-vs-elk) systemctl is-active --quiet prometheus-node-exporter && echo "TELEMETRY_OK" sysctl net.core.somaxconn net.ipv4.tcp_tw_reuse vm.swappiness curl -fsSL -o /dev/null -w "http_code=%{http_code} ttfb=%{time_starttransfer}s " http://127.0.0.1:8080/healthz 

By enforcing these pre-cutover verification gates alongside 15-minute guaranteed human engineer response for critical P1 incidents with 99.99% platform availability, engineering teams ensure ClickHouse Log Analytics & High-Cardinality Observability: Replacing ELK at 10x Lower Cost operates with predictable throughput, cryptographic isolation, and continuous audit compliance across SOC2 audit reporting, continuous configuration drift detection, and daily immutable offsite snapshots.

Free Webmaster, SRE & Cloud Diagnostic Tools (No Signup Required)

Verify and benchmark your production infrastructure directly from your browser using the HostingPodium Free Webmaster & Developer Suite:

Free Engineering Utility Primary Diagnostic Use Case
Global DNS Propagation & SSL Checker Verify live A, AAAA, CNAME, MX, TXT, NS & CAA records across 21+ global resolvers + TLS certificate audits
Website Speed Test & Core Web Vitals Auditor Measure server TTFB, LCP, INP, CLS, and HTTP/2 compression latency in real time
Rich Snippets & Schema.org JSON-LD Generator Generate & validate structured JSON-LD markup for TechArticles, FAQs, Products & Organizations
HTTP Redirect Chain & Status Code Tracer Trace 301/302 redirect hops, inspect response headers, and verify load balancer health checks
DMARC, SPF & DKIM Email Security Validator Audit domain email authentication policies and generate compliant DMARC records
Google SERP Simulator & Meta Tag Previewer Preview desktop/mobile Google snippets, OpenGraph cards, and pixel-width truncation
Free WebP Image Converter & Optimizer Client-side PNG/JPG to WebP compression for faster Core Web Vitals LCP

Related Enterprise Architecture & Benchmarks