Traditional Publishing:
Local Editor (Markdown) ──> Copy/Paste ──> Web CMS Dashboard ──> Manual Formatting ──> Publish ──> Repeat for 4 Platforms

GitOps Publishing:
Local Editor (Markdown) ──> git commit & push ──> GitHub Actions (CI/CD) ──> ZyVop API ──> Automated Multi-Platform Fanout

---
title: "The Anatomy of a Resilient Distributed Task Queue"
subtitle: "Deep dive into NestJS, Fastify, BullMQ, and Redis worker processes"
excerpt: "Learn how to architect high-throughput asynchronous job pipelines that survive network partitions and node crashes."
category: backend
tags:
  - typescript
  - architecture
  - redis
  - devops
canonical_url: https://myblog.com/posts/distributed-queues
cover_image: https://assets.myblog.com/covers/task-queue.webp
status: PUBLISHED
generate_toc: true
cross_post:
  devto: true
  hashnode: true
  medium: true
  bluesky: true
---

# Introduction

When designing scalable web architectures, separating synchronous request-response cycles from background task execution is essential...

import fs from "node:fs";
import path from "node:path";
import matter from "gray-matter";
import { marked } from "marked";
import { publishArticleRestApi } from "../api.js";

export async function publishCommand(filePath, options) {
  const resolvedPath = path.resolve(process.cwd(), filePath);
  const rawFile = fs.readFileSync(resolvedPath, "utf-8");

  // 1. Extract frontmatter and raw Markdown AST
  const parsed = matter(rawFile);
  const frontmatter = parsed.data || {};
  const content = parsed.content || "";

  // 2. Resolve authentication credentials
  const token = process.env.ZYVOP_TOKEN || options.token;
  const endpoint = options.endpoint || "https://api.zyvop.com";

  // 3. Dispatch to the Headless REST Endpoint
  if (token.startsWith("zv_")) {
    const post = await publishArticleRestApi(rawFile, token, endpoint);
    console.log(`✅ Live URL: ${post.url}`);
    return;
  }
}

// Dynamically increase fence length if the content contains triple backticks
let fence = '```';
while (text.includes(fence)) {
  fence += '`';
}
return `\n\n${fence}${lang}\n${text}\n${fence}\n\n`;

This ensures that your code snippets—no matter how complex or nested—remain syntactically intact across all platforms.

* * *

## 4\. Setting Up the Production GitHub Actions Workflow

To achieve true GitOps publishing, we don't want to re-publish every single post on every commit. We only want to publish **new or modified Markdown files** in the current push.

Here is the production-ready GitHub Actions workflow.

### File: `.github/workflows/publish.yml`

```

yaml
name: Continuous Publishing (GitOps)

on:
  push:
    branches:
      - main
    paths:
      - 'posts/**.md'
  workflow_dispatch: # Allows manual trigger from the GitHub Actions UI

concurrency:
  group: publishing-${{ github.ref }}
  cancel-in-progress: false

jobs:
  publish-articles:
    name: Validate & Publish to ZyVop
    runs-on: ubuntu-latest

    steps:
      - name: 📥 Checkout Repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 2 # Fetch the previous commit for accurate git diffing

      - name: ⚙️ Setup Node.js Runtime
        uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'npm'

      - name: 🔍 Detect Changed Markdown Posts
        id: changed-files
        run: |
          # If initial commit or forced push, fallback to all posts
          if [ "${{ github.event.before }}" = "0000000000000000000000000000000000000000" ]; then
            FILES=$(git ls-files 'posts/*.md')
          else
            FILES=$(git diff --name-only --diff-filter=ACMR ${{ github.event.before }} ${{ github.sha }} | grep '^posts/.*\.md$' || true)
          fi

          if [ -z "$FILES" ]; then
            echo "No Markdown files modified."
            echo "has_changes=false" >> $GITHUB_OUTPUT
          else
            echo "Files to publish:"
            echo "$FILES"
            # Format files into a space-separated list
            FILES_CLEAN=$(echo "$FILES" | tr '\n' ' ')
            echo "files=$FILES_CLEAN" >> $GITHUB_OUTPUT
            echo "has_changes=true" >> $GITHUB_OUTPUT
          fi

      - name: 🚀 Run ZyVop CLI Publisher
        if: steps.changed-files.outputs.has_changes == 'true'
        env:
          ZYVOP_TOKEN: ${{ secrets.ZYVOP_TOKEN }}
        run: |
          for file in ${{ steps.changed-files.outputs.files }}; do
            if [ -f "$file" ]; then
              echo "──────────────────────────────────────────────"
              echo "📦 Deploying: $file"
              npx zyvop publish "$file"
            fi
          done

      - name: 📊 Summary Report
        if: steps.changed-files.outputs.has_changes == 'true'
        run: |
          echo "### 🚀 GitOps Publishing Complete" >> $GITHUB_STEP_SUMMARY
          echo "The following articles were verified and deployed:" >> $GITHUB_STEP_SUMMARY
          for file in ${{ steps.changed-files.outputs.files }}; do
            echo "- \`$file\`" >> $GITHUB_STEP_SUMMARY
          done



```

* * *

## 5\. Securing the Pipeline with Personal Access Tokens

Authentication in CI/CD pipelines requires zero interactive prompts. ZyVop uses cryptographically hashed **Developer Personal Access Tokens** (`zv_live_...`).

![Mermaid Diagram](https://mermaid.ink/img/pako:eNqFklFv2jAUhf_K3X3YU0jVrQ8bQkhpRwGNrhFhSGsyIce5EK-OHdlOWij898kBsW5StSdfXx37O_fYL8h1QdjHtdRPvGTGwWyeKQDb5BvD6hJmmjMJaYaXIXyhlqSuycAd46VQlOFPLwaIo0Wa4ZgUGeYIFvqRFAgFCTkn1MYOcnMxHHjWcNeupGhp9Xn9iYdhOLjouqebSBWZ-os_nnj4hxDGwk2aHOZUayucNtszPCFuyKUZjhQ329pRcWq9oj78WN7Hq8X919G3_xHnjVJkPPXjmRpxJ7SyZ-RItWmGU_WLuMeNVCuMVhWp18zaaE7WhqTa8E0-wM1smmb4sF3q2tdv-krItEdfVyEc5VE8PV8TNa5MM1wyKQr_BgnXNVl4D3O_m4lKuD_-4yaXwnr9qYJYW_cvOo4W0OsN91FRgNPnx4ThKV-7PxVefKw6_TEXYM5n6URFe5-QF41U6xXdnKob_QhoXKmN2DGfch-uiRkysGtXYRjuu8m82q_d6ZNnDLAiUzFRYP8FXUmV_8kFrVkjHQbHzpIZwXJJ1mvWWrlbVgm5xT72WF1L6tmtdVQFcC2FerxjPOn2t1q5ADJMaKMJvk8zDGCuc-10ABOSLTnBWQCREUwGYJmyPUtGrDHoIInYeS-XV_UzHg4B5psbLbXBPr57KoUjPPwGR8Yolw?type=png)

### Steps to Configure:

1.  Log in to your **Dashboard** and navigate to **Settings > Developer API**.

2.  Click **Generate New Token**, assign a descriptive name (e.g., `github-actions-blog`), and copy the generated key.

3.  In your GitHub repository, navigate to **Settings > Secrets and variables > Actions**.

4.  Click **New repository secret**, set the name to `ZYVOP_TOKEN`, and paste the token string.


* * *

## 6\. Engineering Best Practices for Repository Structure

When managing your publication as code, organizing your directory structure helps maintain readability and simplifies pre-commit validations.

### Recommended Repository Layout:

```

yaml
my-tech-blog/
├── .github/
│   └── workflows/
│       ├── publish.yml          # Automated deployment pipeline
│       └── lint.yml             # Pre-merge validation (Markdownlint, CSpell)
├── .markdownlint.json           # Style consistency rules
├── posts/
│   ├── 2026-08-20-distributed-queues.md
│   ├── 2026-08-22-at-protocol-internals.md
│   └── drafts/
│       └── upcoming-raft-consensus.md
├── static/
│   └── diagrams/
│       └── queue-architecture.png
└── README.md



```

### Pre-Merge Quality Gates (Pull Request Workflow)

Before an article is merged into `main`, you can enforce the same quality checks you use on software projects:

-   **Markdown Linting (**`markdownlint-cli2`**):** Ensures heading hierarchies are semantically correct (e.g., single `h1`, no skipped header levels).

-   **Spell Checking (**`cspell`**):** Catches typographical errors and unknown terminology before publication.

-   **Link Validation (**`lychee`**):** Verifies that all outbound references, documentation links, and image URLs are reachable and return HTTP 200.


```

yaml
# .github/workflows/lint.yml
name: Content Verification

on:
  pull_request:
    paths:
      - 'posts/**.md'

jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Check Markdown formatting
        uses: DavidAnson/markdownlint-cli2-action@v16
        with:
          globs: 'posts/**/*.md'
      - name: Check spelling
        uses: streetsidesoftware/cspell-action@v6
        with:
          files: 'posts/**/*.md'



```

* * *

## 7\. Summary & Getting Started

By shifting technical blogging to a GitOps workflow:

-   **Your content stays in your hands:** You own the raw Markdown files, version history, and branch reviews in your repository.

-   **You write where you are productive:** No more pasting between browser tabs. Stay in your terminal, IDE, and Git workflow.

-   **Continuous syndication happens automatically:** One `git push` simultaneously publishes your post across Dev.to, Hashnode, Medium, and Bluesky, while maintaining your canonical SEO ranking.


To test publishing an article directly from your terminal:

```

bash
# 1. Login to your account
npx zyvop login

# 2. Test publishing any local Markdown file
npx zyvop publish ./posts/my-article.md



```



---

*Originally published on [ZyVOP](https://zyvop.com/gitops-for-technical-writers-continuous-publishing-with-the-zyvop-cli-and-github-actions-fb690)*

💡 For more articles like this, [subscribe to the ZyVOP newsletter](https://zyvop.com/newsletter)!