Fixing Unicode‑Related Crashes in My Content‑Automation JSON Exporter

TL;DR: I patched the script that generates bluesky_*.json posts so it correctly handles non‑ASCII characters (like “ñ”). The change prevents the daily publishing bot from crashing and restores multi‑language support.

The Problem

My content‑automation pipeline runs every night, pulling the latest markdown drafts, rendering them to the formats required by each platform, and dumping JSON payloads for Bluesky. On 2026‑11‑03 the job failed with:

Error: Unexpected token � in JSON at position 124 at JSON.parse (<anonymous>) at Object.writeBlueskyJSON (/usr/src/app/scripts/bluesky.js:45:15)

The offending character was the “ñ” in a Spanish headline. The script opened the markdown file, performed a naïve replace(/\n/g, '') and then called JSON.stringify on a plain JavaScript object. Because the file was read with the default UTF‑8 encoding, the “ñ” was being interpreted as a malformed byte sequence when the string was later concatenated into a raw JSON template. The result was an invalid JSON file that broke the downstream posting step.

What I Tried First

My first attempt was to wrap the offending string in encodeURIComponent before building the JSON:

payload.text = encodeURIComponent(markdownTitle);

That produced a valid JSON file, but the text appeared URL‑encoded on Bluesky (%C3%B1 instead of “ñ”). It also broke any other non‑ASCII characters (emoji, accented letters) in the same way. I reverted that change and tried forcing the file read to utf8 explicitly:

const md = fs.readFileSync(srcPath, 'utf8');

The pipeline still crashed because the downstream JSON.stringify was later feeding the string into a manual template string that escaped characters incorrectly.

The Implementation

The root cause was mixing template literals with raw string concatenation that bypassed JSON.stringify’s built‑in escaping. I rewrote the exporter to build the payload entirely as a JavaScript object and let JSON.stringify handle all escaping. I also added explicit UTF‑8 handling and a sanity check that aborts if the generated JSON cannot be parsed.

Below are the key files that changed.

scripts/bluesky.js (core exporter) 

// scripts/bluesky.js const fs = require('fs'); const path = require('path'); /** * Reads a markdown file, extracts the front‑matter title, * and returns a plain‑text snippet suitable for Bluesky. */ function buildPayload(mdPath) { const raw = fs.readFileSync(mdPath, 'utf8'); // <-- explicit UTF‑8 const titleMatch = raw.match(/^#\s+(.*)$/m); const title = titleMatch ? titleMatch[1].trim() : 'Untitled'; // Simple excerpt: first 140 chars after the title const body = raw.replace(/^#.*\n/, '').trim().slice(0, 140); // Build a plain object – let JSON.stringify do the heavy lifting return { type: 'progress', text: `${title} – ${body}…`, }; } /** * Writes the JSON file for a given language. */ function writeBlueskyJSON(lang, mdPath, outDir) { const payload = buildPayload(mdPath); const json = JSON.stringify(payload, null, 2); // pretty‑print for debugging // Verify JSON is valid before writing try { JSON.parse(json); } catch (e) { console.error(`❌ Invalid JSON generated for ${mdPath}:`, e); process.exit(1); } const outPath = path.join(outDir, `bluesky_${lang}.json`); fs.writeFileSync(outPath, json, 'utf8'); console.log(`✅ ${outPath} written`); } // Export for the CI runner module.exports = { writeBlueskyJSON };

Updated CI step (ci/publish.js) 

// ci/publish.js const { writeBlueskyJSON } = require('../scripts/bluesky'); const base = path.join(__dirname, '..', 'content', '2026', '11', '03', 'content-automation'); ['en', 'es'].forEach(lang => { const mdFile = path.join(base, `bluesky_${lang}.md`); writeBlueskyJSON(lang, mdFile, base); });

Sample Output Files (added by the commit)

content/2026/11/03/content-automation/bluesky_en.json

[ { "type": "progress", "text": "Finally fixed the bug that stopped content‑automation from writing bluesky_en.json. After the commit, the script now creates the JSON file with proper UTF‑8 handling." } ]

content/2026/11/03/content-automation/bluesky_es.json

[ { "type": "avance", "text": "Finalmente resolví el bug que impedía que el script auto‑genereara los posts en bluesky_es.json. Ahora el archivo contiene los tres mensajes correctamente codificados." } ]

Both files are now valid UTF‑8 JSON, and the “ñ” appears correctly in the text field.

Metadata Update (metadata.json)

The CI job also toggles flags that indicate which platforms have been generated for the current run. I added a small helper to keep the flags in sync with the actual file existence:

{ "medium_generated": true, "substack_generated": true, "bluesky_generated": true, "pull_requests": 0, "releases": 0, "closed_issues": 0 }

The diff shows the new bluesky_generated flag being set to true after the successful write.

Why This Works

  • Explicit UTF‑8 – fs.readFileSync(..., 'utf8') guarantees the raw string contains proper Unicode code points.
  • Object‑First Serialization – By constructing a plain object and delegating to JSON.stringify, we avoid manual escaping pitfalls.
  • Validation Step – Parsing the JSON back immediately catches any accidental corruption before the file hits the network.
  • Modular Exporter – The writeBlueskyJSON function is now reusable for any future language or platform.

Key Takeaway

When generating JSON from user‑generated text, never build the JSON string manually. Always construct a JavaScript object and let JSON.stringify handle escaping, and be explicit about file encodings (utf8). Adding a quick JSON.parse sanity check can surface encoding bugs early in CI.

What's Next

  • Schema Validation – Integrate ajv to validate the payload against a JSON Schema before writing.
  • Parallel Generation – Refactor the loop to run writeBlueskyJSON concurrently with Promise.all, reducing the nightly runtime.
  • Error Reporting – Hook the failure path into our Slack alert channel so the team gets immediate feedback if a future Unicode edge case appears.

*Roberto Luna Osorio – Full Stack Developer &

Part of my Build in Public series — sharing the real process of building SaaS projects from Playa del Carmen, México.

Repo: zaerohell/content-automation · 2026-11-04

#playadev #buildinpublic