Skip to content

Workflow · Content

Build an AI YouTube Shorts Generator

An end-to-end pipeline that turns a single topic prompt into a finished 30-second vertical video with voiceover, captions, and B-roll.

Advanced3–5 hoursStack: Claude / GPT-5, ElevenLabs, Replicate or Runway, ffmpeg, Remotion
YouTubeShortsVideoTTS
Edited by The AIKnowHub team · Editorial team

Problem

Faceless YouTube Shorts channels need volume to grow — 1–3 videos a day — and editing each one manually takes hours. Most automation tools produce visibly bad output that hurts the channel.

Final output

1080×1920 MP4 with: a strong 3-second hook, animated word-level captions in the lower middle, B-roll synced to the script, a takeaway final beat, and clean audio levels. Upload-ready.

Architecture

Topic prompt
  → LLM (script: hook + 4-6 beats + takeaway)
  → ElevenLabs (voiceover, Turbo for speed)
  → B-roll: Replicate AI clips OR Pexels stock
  → Whisper (word-level captions)
  → Remotion (compose: 1080x1920, animated text + B-roll)
  → Render MP4
  → Review

Step-by-step

  1. 01

    Generate the script

    LLM produces ~75 words: hook, 4–6 beats, screenshot-worthy takeaway. Output structured JSON.

  2. 02

    Synthesize voiceover

    ElevenLabs Turbo with high-energy voice settings. Save as voice.mp3 with timestamps.

  3. 03

    Generate or pull B-roll

    For each beat, generate a 3-5s clip on Replicate (Veo, Pika, Kling) OR fetch a Pexels stock clip. Hybrid is often best.

  4. 04

    Time the captions

    Whisper-large-v3 on the voice.mp3 returns word-level timestamps. Save as JSON.

  5. 05

    Compose with Remotion

    Build a Remotion template: top 1/3 B-roll, middle word-by-word captions, bottom handle/tag. Render at 1080x1920.

What you'll build

A pipeline: "explain the difference between RAG and fine-tuning" → 30-second vertical MP4 ready to upload.

Pipeline

  1. Generate script (LLM)
  2. Generate voiceover (ElevenLabs)
  3. Generate or pull B-roll clips (Replicate / Pexels / stock)
  4. Caption the voiceover (Whisper)
  5. Compose with Remotion or ffmpeg
  6. Render vertical 1080×1920 MP4

Step 5 — Compose

Remotion is the cleanest path: programmatic video in React. Build a template with:

  • Top 1/3: visual (B-roll, with subtle zoom)
  • Middle: animated caption text
  • Bottom 1/6: channel handle + topic tag

Render with @remotion/cli.

Or use ffmpeg for a simpler (less polished) result:

ffmpeg -i bg.mp4 -i voice.mp3 -vf "scale=1080:1920,subtitles=captions.ass" out.mp4

What makes Shorts go viral (briefly)

  • First-second hook
  • High caption legibility (large text, high contrast)
  • One single idea per video — not a montage
  • Loop-friendly ending (so it auto-replays)

This pipeline gives you the production. The content judgment is still on you.

Prompt examples

Copy any of these, replace the placeholders, run.

Script generation

Write a 30-second YouTube Short script about: {{topic}}.

Rules:
- ~75 words total
- First 3 seconds must hook: question, bold claim, or surprising stat
- 4–6 beats, one sentence each, conversational
- Final beat = a one-line takeaway viewers can screenshot
- Energetic, no jargon without explanation, no "let's dive into"

Output:
{
  "hook": "...",
  "beats": ["...", "..."],
  "takeaway": "..."
}

B-roll prompt per beat

Generate a 5-second cinematic video clip.

Subject: {{beat_text}}
Mood: {{mood}}
Camera: slow dolly in, shallow depth of field, warm color grading
Avoid: text on screen, talking heads, fast cuts

Cost estimate

Line itemApprox. cost
Script (LLM)$0.01
Voiceover (ElevenLabs Turbo)$0.30
B-roll (AI, 5–6 clips on Replicate)$1.50
Whisper transcription$0.01
Remotion render compute$0.05
Total per run (approx.)~$1.87

Costs depend on model choice, content length, and how aggressively you cache.

Optimizations

  • Pre-generate a library of B-roll for common topics — saves the most expensive step on repeat themes.
  • Use Pexels for 50% of clips, AI for the hero shots — drops cost by half with minimal quality loss.
  • Render captions client-side; only the final video composite needs heavy compute.
  • Batch process 10 Shorts per night — script generation is the only stage that benefits from running individually.
  • Cache voiceover by script hash — re-renders cost nothing if the script didn't change.

Common mistakes

  • Hook is too long — if the first 2 seconds don't grab, the viewer is gone.
  • Captions too small — viewers watch Shorts on mute most of the time.
  • AI B-roll with uncanny artifacts — always watch the output before uploading.
  • Mismatched audio levels — voice and bed need to be balanced or it sounds amateur.
  • Cramming too many ideas — one Short = one idea. Multi-idea Shorts underperform.

Frequently asked questions

YouTube allows AI-generated content but requires disclosure when content is realistic or could mislead. Faceless explainer Shorts with disclosed AI narration are well within policy.

Related on AIKnowHub