Technical deep-dive into the architecture decisions behind CaptureOCR

CaptureOCR is a Windows desktop application that transforms screenshots into structured data (Markdown tables, CSV) using local OCR. Built with Python, PySide6, and ONNX Runtime, it processes everything on-device with zero cloud dependencies.

Product link: CaptureOCR on Microsoft Store — $2.99 one-time, no subscription.

High-Level Architecture

┌─────────────────────────────────────────────────────────────┐ │ UI Layer (PySide6) │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ │ │ Full-screen │ │ Region │ │ Canvas/Annotation │ │ │ │ Selection │ │ Drawer │ │ Toolbar │ │ │ └─────────────┘ └─────────────┘ └─────────────────────┘ │ └────────────────────────────┬────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Application Services │ │ ┌─────────────────┐ ┌─────────────────────────────────┐ │ │ │ Capture Manager │ │ OCR Workflow Orchestrator │ │ │ │ - mss grab │ │ - Queue management │ │ │ │ - DPI mapping │ │ - Model tier switching │ │ │ │ - Multi-monitor │ │ - Progress reporting │ │ │ └─────────────────┘ └─────────────────────────────────┘ │ │ │ │ │ ┌────────▼────────┐ │ │ │ Export Service │ │ │ │ - Markdown/CSV │ │ │ │ - Session (.snipx)│ │ │ └─────────────────┘ │ └────────────────────────────┬────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Infrastructure Layer │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │ │ │ OCR Backend │ │ Tiling │ │ Scroll Capture │ │ │ │ (RapidOCR + │ │ Engine │ │ (Row signatures) │ │ │ │ ONNX Runtime)│ │ │ │ │ │ │ └──────────────┘ └──────────────┘ └──────────────────┘ │ └─────────────────────────────────────────────────────────────┘ 

Key Design Decisions

1. Single OCR Worker Thread

Decision: All model inference runs on a dedicated background thread.

Rationale:

• ONNX Runtime's InferenceSession is not guaranteed thread-safe
• Avoids complex locking around model loads/unloads
• For a desktop tool processing ~10 regions, serial throughput is sufficient
• Predictable memory profile (no concurrent model instances)

# Simplified worker pattern class OcrWorker(QThread): result_ready = pyqtSignal(OcrResult) def __init__(self): super().__init__() self.queue = Queue() self.session = None self.current_model = None def run(self): while True: task = self.queue.get() if task is None: break result = self._process(task) self.result_ready.emit(result) 

2. Two-Tier Model Strategy

Implementation:

class ModelManager: def __init__(self): self.tiny_session = self._load_tiny() # Always loaded  self.small_session = None self.last_small_use = 0 def get_session(self, tier: str): if tier == "fast": return self.tiny_session if self.small_session is None: self.small_session = self._download_and_load_small() self.last_small_use = time.time() return self.small_session def maybe_unload_small(self): if self.small_session and time.time() - self.last_small_use > 5: self.small_session = None gc.collect() 

3. DPI-Aware Multi-Monitor Capture

Windows captures in physical pixels; Qt works in logical coordinates. On 125-150% scaling (and mixed-DPI setups), this mismatch causes blurry captures or offset selections.

Solution: Per-monitor ScreenInfo with both logical and physical rects + device pixel ratio.

@dataclass class ScreenInfo: index: int logical_rect: QRect # Qt coordinates  physical_rect: QRect # Screen pixels  dpr: float # Device pixel ratio  def logical_to_physical(screen: ScreenInfo, logical_rect: QRect) -> QRect: """Convert Qt logical rect to physical pixels for mss capture.""" x = int((logical_rect.x() - screen.logical_rect.x()) * screen.dpr) y = int((logical_rect.y() - screen.logical_rect.y()) * screen.dpr) w = int(logical_rect.width() * screen.dpr) h = int(logical_rect.height() * screen.dpr) return QRect(x, y, w, h) 

4. Tiled OCR for Long Images

Feeding a 20,000px tall screenshot into a 1600px OCR cap scales text to ~2px — unreadable. Instead, tile vertically:

Tile parameters: - Target height: 1600px - Overlap: 64px (≈2 lines at 1080p) - Core bands: [0, 1536), [1536, 3072), ... - Each tile extends 32px above/below core for context - Line assigned to tile where box center Y falls in core band 

This ensures every line is processed at native resolution, claimed exactly once.

5. Row-Signature Scroll Stitching

Template matching fails for scrolling (neither frame contains the other). Instead:

1. Convert frame to grayscale, ignore right 8% (scrollbars)
2. Compress each row to 32 column-bucket means → (H, 32) signature
3. For each candidate shift s, compute MAE between prev[s:] and cur[:H-s]
4. Accept shift only if best score < threshold AND clearly better than median
5. Sticky headers: retry skipping 8%/16%/25% of top rows
6. Adaptive wheel step: measure first scroll, target ~45% of selection height

Performance Characteristics

Test set: 23 mixed Chinese/English screenshots, 1600px cap, local CPU. Not a public benchmark — treat as order of magnitude.

Distribution

• Format: MSIX (runFullTrust), distributed via Microsoft Store
• Size: ~126 MB (includes tiny model, ONNX Runtime, Python runtime)
• Dependencies: Zero — fully self-contained
• Auto-updates: Handled by Store

Lessons Learned

1. Detection config matters more than model choice: Switching from limit_type="min" (upscales short side) to limit_type="max" (only downscale long side) was a bigger win than model tuning.

2. Disable classification for screenshots: use_cls=False — the 0°/180° classifier adds latency with no signal for upright captures.

3. DPI bugs are subtle: Logical/physical coordinate mismatch only appears on scaled displays. Test on 125%, 150%, and mixed-DPI multi-monitor.

4. Serialization format for reproducibility: The .snipx session package (ZIP with session.json + original PNGs + Markdown/CSV) enables lossless round-tripping and collaboration.

Resources

• GitHub: github.com/lijiannan1241/CaptureOCR — product page & issue tracker
• Microsoft Store: CaptureOCR
• Source availability: Application is closed-source; repo hosts product page and issue tracker only

Disclosure: I'm the developer of CaptureOCR. This is a paid product ($6.99 one-time). The application source code is not public — the repository serves as the product page and issue tracker.