Backend
Asymptotic Analysis of Data Deduplication Under a Constant Edit Model
arXiv math.IT0 views
arXiv:2611.09842v1 Announce Type: new Abstract: Data deduplication has become increasingly important in large-scale storage systems, driven by the rapid growth of digital data. By identifying and removing repeats in the data stream, deduplication saves storage space more efficiently than traditional compression methods. In this paper, we study the performance of variable-length deduplication and multi-chunk deduplication algorithms from an information-theoretic perspective, focusing on data streams where repeats are approximate rather than exact. We consider two probabilistic source models in which source strings consist of repeated blocks that are modified by either a constant number of substitution edits or burst-insertion edits per block, the latter capturing the localized edit patterns commonly observed in real-world data. For both models, we show that variable-length deduplication achieves asymptotically arbitrarily large compression ratios, and that multi-chunk deduplication achieves compression within a constant factor of the source entropy under mild conditions.
Source:https://arxiv.org/abs/2611.09842PreviousAn Efficient Synchronization Scheme for Distributed BanditsNextHow far does a hash-only sender go? A pre-registered map of sender memory versus bits for short text messages
Related posts

What a good resume actually looks like
0dev.to Career

System Design Interviews From Production Scars, Not Blog Diagrams
0dev.to Career
Pennsylvania Layoffs November 2026: WARN Numbers Explained
0dev.to Career

PHP Dies After Every Request. How the Zend Engine Actually Runs Your Code.
0dev.to PHP
No Sign-Up, No Accounts: The Privacy Posture of a Simple Downloader
0dev.to PHP
Compute crypto perpetual funding cost from a public API (Python)
0dev.to Beginners