OpenShorts Deployment Guide: Turning Long Videos into AI Shorts
Anyone who makes podcasts, live streams, or interview shows knows this pain point: in a one-hour video, the moments truly worth posting as short clips might only be a few minutes. Manually reviewing, editing, adding subtitles, and reframing can easily eat up half a day. OpenShorts is an open-source alternative to OpusClip that automates this pipeline—throw in a long video, and it automatically transcribes, uses AI to pick highlight moments, reframes them to 9:16 vertical format, burns in subtitles, and finally outputs a ready-to-publish product.
Let's look at how it works: it first uses faster-whisper for word-level timestamp transcription, uses PySceneDetect to identify scene boundaries, then uses Gemini or any OpenAI-compatible local model to score the transcribed text, selecting 3-15 highlight moments of 15-60 seconds each. Each clip is also intelligently reframed—stacking vertically for two-person conversations, face-tracking crops for solo shots, with subtitles burned into the seams of the frame. The entire pipeline is orchestrated with Docker Compose, and self-hosting is completely free with no watermarks.
Core Dependencies and Environment
Before you start, make sure your machine meets these requirements:
- Docker and Docker Compose: The only dependency for deployment; no need to install Python or Node on the host machine
- Memory: 8GB or more, as model inference and video encoding are memory-intensive
- Storage: Video files plus transcription and rendering intermediate artifacts; we recommend leaving at least 20GB free
- Network: Ability to pull Docker images and download models (HuggingFace, Ultralytics)
- CPU is the default path, with optional NVIDIA GPU acceleration, which will be covered later
Let's clarify the division of AI services in advance:
| Service | Purpose | When Required |
|---|---|---|
| Gemini or OpenAI-compatible LLM | Text highlight selection, script generation, titles | At least one |
| Gemini (vision) | Automatic layout selection, on-screen content recognition, silent video | Optional |
| fal.ai | AI actor generation, digital human video, lip sync | Only for AI Shorts feature |
| ElevenLabs | Voiceover, multilingual cloning | Optional |
| Upload-Post | Multi-platform auto-publishing | Optional |
The main feature only needs a text model that can call v1/chat/completions. Gemini vision, fal.ai, and ElevenLabs do not affect the core clipping pipeline.
Project Structure Tree
openshorts/
├── app.py # FastAPI backend entry
├── main.py # Core pipeline: transcription, highlight detection, clipping
├── llm_backend.py # OpenAI-compatible LLM adapter layer (core file)
├── gemini_worker.py # Gemini calls and JSON parsing
├── layout_picker.py # Automatic layout selection
├── s3_uploader.py # S3 storage (optional)
├── render-service/ # Remotion rendering service
│ └── Dockerfile
├── dashboard/ # React frontend (Vite + Tailwind)
│ ├── src/
│ └── package.json
├── docker-compose.yml # Three-service orchestration
├── .env.example # Environment variable template
└── Dockerfile # CPU/GPU multi-stage build
Three core services: backend (FastAPI, port 8000), frontend (Vite dev server, mapped to 5175), and renderer (Remotion rendering, port 3100).
Step-by-Step Deployment Guide
1. Clone the Repository
git clone https://github.com/mutonby/openshorts.git
cd openshorts
2. Configure AI Services
First, copy the environment variable template:
cp .env.example .env
Open .env and find the LLM_BASE_URL configuration. This is where you connect your text highlight selection service. OpenShorts' llm_backend.py appends /chat/completions to LLM_BASE_URL, fully compatible with OpenAI's protocol format. If you don't want to use Google's Gemini, you can point this text selection part to any compatible LLM service.
Here's a cost-saving tip: the Defapi platform charges only half of the official price, official website is Defapi. Its models are largely compatible with v1/chat/completions, v1/messages, and v1beta/models/ protocols, so OpenShorts can connect directly. Here's how to configure it:
LLM_BASE_URL=https://api.defapi.org/v1
LLM_MODEL=your-model-id # Replace with the actual model ID you use on Defapi
LLM_API_KEY=your_defapi_key
It's worth noting that this entry point only handles highlighting scores and detailed selection from text transcription; it does not replace Gemini's vision features (automatic layout, silent video recognition), nor does it involve fal.ai's video generation or ElevenLabs' voice. The main clipping pipeline does not depend on AI actors, social publishing, or cloud Autopilot—those are extra features. Defapi's models are basically compatible with v1/chat/completions, v1/messages, and v1beta/models/ protocols, so switching integration won't incur extra adaptation costs.
After configuration, the /api/config endpoint will return localLlm info, and the frontend's top-right corner won't force a Gemini Key. One note: if you also set LLM_PROVIDER=gemini, it will force the Gemini route, so comment out or delete that line.
3. Start the Services
docker compose up --build
The first build needs to pull base images like Python 3.11, Node 22, Deno, plus install Python dependencies and frontend npm packages; time depends on network speed. After the build completes, the backend, frontend, and rendering services will start in sequence.
4. Upload a Video and Generate Highlights
Open your browser and go to http://localhost:5175, entering the Dashboard. First, fill in your API Key in Settings (if you've configured a local LLM, the Gemini Key can be left empty). Then:
- Click Clip Generator
- Upload a video with audio, or paste a YouTube link
- Wait for transcription and selection to complete (an 8-minute video takes about 5-8 minutes on CPU)
Once done, you'll see each highlight clip in the interface, with virality scores, platform title suggestions, and editable clip tracks.
5. Edit Subtitles and Composition
Each clip comes with a full editing editor: source monitor on the left, program monitor on the right, and full transcription below. You can click any word to set a cut point, add extra clips, and switch composition modes (single-person tracking, two-person vertical split, vertical blur background, screen recording). After changes, just click to re-render, and you'll get the output in seconds.
6. Export the Final Product
Once you're satisfied, click Export. The rendering service will call FFmpeg to burn in subtitles, hook text, and frame transformations into the final file, outputting to the ./output directory. The exported files are ready-to-post 9:16 vertical videos for TikTok, YouTube Shorts, or Instagram Reels.
Common Troubleshooting
Image Build Fails or Model Download Times Out
In domestic network environments, pulling HuggingFace models (whisper, YOLO) or Docker images often gets interrupted. Configure a mirror accelerator and set a proxy for the Docker daemon. If model downloads fail, delete container volumes and retry:
docker compose down -v
docker compose up --build
Configured LLM_BASE_URL but Still Being Asked for Gemini Key
Check if .env contains LLM_PROVIDER=gemini; if so, delete it. Also ensure LLM_BASE_URL doesn't have a trailing slash—llm_backend.py will append the path itself.
Local Model Does Not Return Valid JSON
OpenShorts tries three response_format strategies in order: json_schema → json_object → no format restriction. Models above 7B usually return reliably; 3B models tend to fail. If you frequently get JSON parsing errors, switch to a larger model or lower LLM_SCORE_BATCH (default 3) to reduce context size per request.
Dashboard Opens but Backend Returns 502
The three services have dependencies, and the frontend starts faster than the backend. Wait until the backend logs show Uvicorn running before refreshing the page. If the 502 persists, check the backend container:
docker logs openshorts-backend
Missing Features Without a Gemini Key
After configuring a local LLM, text selection works, but the following features degrade: automatic layout selection falls back to pure face-tracking cropping, silent videos raise errors directly, and on-screen content recognition is disabled. To get these features, you need to fill in a Gemini Key in Settings. Both routes can coexist—text via local model, vision via Gemini.
Insufficient Memory or Disk
Both transcription and rendering use significant resources. 8GB memory is the minimum; 16GB is safer. For disk, the whisper model is about 3GB, YOLO is about 6MB, plus video files and intermediate artifacts—an 8-minute video takes about 10GB of temporary space. Remember to clean up the output directory after processing.
Further Reading and Advanced Directions
GPU Acceleration: The default CPU image processes an 8-minute video in 5-8 minutes; with an NVIDIA GPU (any NVENC-capable model), you can reduce that to about 1 minute. Refer to the docker-compose.override.yml configuration in the README, enable the GPU: "1" build argument, and set WHISPER_DEVICE=cuda and FFMPEG_ENCODER=auto.
API and MCP Integration: OpenShorts provides a complete REST API and MCP server, supporting natural language operations via Claude or ChatGPT: "Cut this podcast into 3 TikToks." When self-hosting, no Key is required; the service is ready to use once started.
Complete Local Run Without Google Key: The README provides a detailed guide for configuring local LLMs. Besides text selection, you can also configure tools like Ollama to keep the entire pipeline offline. Note that Ollama needs OLLAMA_CONTEXT_LENGTH=16384; otherwise, the default 4096 context can silently truncate and cause incomplete JSON.
License Note: The core application is MIT-licensed and completely free. However, the code in the cloud/ directory (billing, hosted infrastructure) uses the OpenShorts Commercial License, which can be used personally or internally, but cannot be provided to third parties as a paid hosting service. Self-hosting the core features doesn't need this directory. The differences between the cloud version (openshorts.app) and self-hosting mainly lie in Autopilot (automatic clipping of new videos) and auto-publishing (one-click platform connections), which in self-hosting need to be implemented via your own Upload-Post setup.