Backend
Asymptotic Analysis of Data Deduplication Under a Constant Edit Model
arXiv math.IT0 views
arXiv:2611.09842v1 Announce Type: new Abstract: Data deduplication has become increasingly important in large-scale storage systems, driven by the rapid growth of digital data. By identifying and removing repeats in the data stream, deduplication saves storage space more efficiently than traditional compression methods. In this paper, we study the performance of variable-length deduplication and multi-chunk deduplication algorithms from an information-theoretic perspective, focusing on data streams where repeats are approximate rather than exact. We consider two probabilistic source models in which source strings consist of repeated blocks that are modified by either a constant number of substitution edits or burst-insertion edits per block, the latter capturing the localized edit patterns commonly observed in real-world data. For both models, we show that variable-length deduplication achieves asymptotically arbitrarily large compression ratios, and that multi-chunk deduplication achieves compression within a constant factor of the source entropy under mild conditions.
Source:https://arxiv.org/abs/2611.09842PreviousAn Efficient Synchronization Scheme for Distributed BanditsNextHow far does a hash-only sender go? A pre-registered map of sender memory versus bits for short text messages
Related posts
AI가 단일세포 데이터를 읽는 시대, 우리는 분자 3D 구조를 '읽고' 있는가? 10x Genomics가 Anthropic의 Claude를 Chromium...
0dev.to Data
Wenn ein Drittel der Diagnosen falsch war: Was molekulare Präzision für Allergologie und Arzneimittelforschung bedeutet
0dev.to Data
Planning a Tattoo Idea Before the First Appointment
0dev.to Design

Comment la vectorisation d'image transforme-t-elle un logo PNG en SVG ?
0dev.to Design
I Had the Studio Before I Had the Business Idea
0dev.to Design

Best AI Humanizer for Essays and Academic Writing in 2026: Improve Clarity and Natural Writing
0dev.to Discuss