OpenShorts Deployment Practice: Turning Long Videos into AI Shorts
People who make podcasts, livestreams, or interview shows all share this pain point: in an hour-long video, the moments truly suitable for short-form clips might only be a few minutes. Manually reviewing, editing, adding subtitles, and reframing takes half a day. OpenShorts is an open-source OpusClip alternative that automates this pipeline—feed it a long video, and it automatically transcribes, uses AI to select highlight clips, reframes them into 9:16 vertical, burns in subtitles, and finally outputs a ready-to-publish product.
Let's look at how it works: it uses faster-whisper for word-level timestamp transcription, PySceneDetect to identify scene boundaries, then Gemini or any OpenAI-compatible local model scores the transcript text to select 3-15 highlight moments of 15-60 seconds. Each clip is also intelligently reframed—stacked vertically for two-person conversations, face-tracking crop for solo shots, with subtitles burned at the scene seams. The entire process is orchestrated via Docker Compose, and self-hosting is completely free with no watermarks.
Core Dependencies and Environment
Before you start, confirm your machine meets these requirements:
- Docker and Docker Compose: The only deployment dependency—no need to install Python or Node on the host
- Memory: 8GB or more; model inference and video encoding both consume memory
- Storage: Video files plus intermediate transcription and rendering artifacts—recommend reserving at least 20GB
- Network: Ability to pull Docker images and download models (HuggingFace, Ultralytics)
- CPU is the default route, NVIDIA GPU optional for acceleration, covered later
Let's clarify the division of AI services upfront:
| Service | Purpose | When Required |
|---|---|---|
| Gemini or OpenAI-compatible LLM | Text highlight selection, script generation, titles | At least one |
| Gemini (vision) | Auto-layout selection, on-screen content recognition, silent video | Optional |
| fal.ai | AI actor generation, digital human video, lip sync | Only for AI Shorts feature |
| ElevenLabs | Voiceover, multilingual cloning | Optional |
| Upload-Post | Multi-platform auto-publishing | Optional |
The main pipeline only requires a text model that can call v1/chat/completions; Gemini vision, fal.ai, and ElevenLabs do not affect the core clipping workflow.
Project Structure Tree
openshorts/
├── app.py # FastAPI backend entry point
├── main.py # Core pipeline: transcription, highlight detection, clipping
├── llm_backend.py # OpenAI-compatible LLM adapter layer (core file)
├── gemini_worker.py # Gemini calls and JSON parsing
├── layout_picker.py # Auto-layout selection
├── s3_uploader.py # S3 storage (optional)
├── render-service/ # Remotion rendering service
│ └── Dockerfile
├── dashboard/ # React frontend (Vite + Tailwind)
│ ├── src/
│ └── package.json
├── docker-compose.yml # Three-service orchestration
├── .env.example # Environment variable template
└── Dockerfile # CPU/GPU multi-stage build
Three core services: backend (FastAPI, port 8000), frontend (Vite dev server, mapped to 5175), renderer (Remotion rendering, port 3100).
Step-by-Step Deployment Guide
1. Clone the Repository
git clone https://github.com/mutonby/openshorts.git
cd openshorts
2. Configure AI Services
First, copy the environment variable template:
cp .env.example .env
Open .env and find the LLM_BASE_URL configuration. This is where you connect the text highlight selection service. OpenShorts' llm_backend.py appends /chat/completions to LLM_BASE_URL, fully compatible with OpenAI's protocol format. If you don't want to use Google's Gemini, you can point the text selection part to any compatible LLM service.
Here's a cost-saving idea: Defapi platform's prices are half of official channels; the website is Defapi. Its models are largely compatible with v1/chat/completions, v1/messages, and v1beta/models/ protocols, so OpenShorts can connect directly. Configuration is as follows:
LLM_BASE_URL=https://api.defapi.org/v1
LLM_MODEL=your-model-id # Replace with the actual model ID you use on Defapi
LLM_API_KEY=your_defapi_key
To clarify, this entry point only handles text transcription highlight scoring and detailed selection; it does not replace Gemini's vision features (auto-layout, silent video recognition), nor does it involve fal.ai's video generation or ElevenLabs' voice. The main clipping pipeline does not depend on AI actors, social publishing, or cloud Autopilot; these are extra features. Defapi's models are broadly compatible with protocols like v1/chat/completions, v1/messages, and v1beta/models/, so switching integration incurs no additional adaptation cost.
After configuration, the /api/config endpoint returns localLlm info, so the frontend top-right no longer forces a Gemini Key. Note: if you also configure LLM_PROVIDER=gemini, it forces the Gemini route—just comment out or delete that line.
3. Start the Services
docker compose up --build
The first build pulls base images like Python 3.11, Node 22, Deno, plus installing Python dependencies and frontend npm packages; time depends on network. After build completes, backend, frontend, and rendering services start in sequence.
4. Upload Video and Generate Highlights
Open your browser to http://localhost:5175 and enter the Dashboard. First, fill in the API Key in Settings (if you configured a local LLM, Gemini Key can be left blank). Then:
- Click Clip Generator
- Upload a video with audio, or paste a YouTube link
- Wait for transcription and selection to complete (an 8-minute video takes about 5-8 minutes on CPU)
On completion, you'll see each highlight clip in the interface with a virality score, suggested platform titles, and an editable clip track.
5. Edit Subtitles and Composition
Each clip has a full built-in clip editor: source monitor on the left, program monitor on the right, and full transcript below. You can click any word to set a cut point, add additional clips, switch composition modes (solo tracking, two-person vertical split, vertical blurred background, screen recording). After changes, just re-render and get the video out in seconds.
6. Export the Final Product
Once satisfied, click Export. The rendering service calls FFmpeg to burn subtitles, hook text, and aspect ratio changes into the final file, outputting to the ./output directory. The exported videos are 9:16 vertical videos ready to post to TikTok, YouTube Shorts, or Instagram Reels.
Common Troubleshooting
Image build failures or model download timeouts
In domestic network environments, pulling HuggingFace models (whisper, YOLO) or Docker images often fails. Configure a mirror accelerator and set a proxy for the Docker daemon. If model downloads fail, delete the container volumes and retry:
docker compose down -v
docker compose up --build
Configured LLM_BASE_URL but still getting Gemini Key requests
Check if .env has LLM_PROVIDER=gemini; if so, delete it. Also confirm that LLM_BASE_URL does not end with a trailing slash—llm_backend.py concatenates the path itself.
Local model returns invalid JSON
OpenShorts tries three response_format strategies in order: json_schema → json_object → no format restriction. Models above 7B typically return stable results; 3B models often fail. If JSON parsing errors occur frequently, switch to a larger model or reduce LLM_SCORE_BATCH (default 3) to lower the context size per request.
Dashboard opens but backend API returns 502
The three services have dependencies; the frontend starts faster than the backend. Wait until the backend log shows Uvicorn running before refreshing the page. If 502 persists, check the backend container:
docker logs openshorts-backend
Feature degradation without Gemini Key
With a local LLM configured, text selection works fine, but the following features degrade: auto-layout selection falls back to pure face-tracking cropping, silent video raises an error, and on-screen content recognition is disabled. To use these features, fill in a Gemini Key in Settings; both routes can coexist—text via local model, vision via Gemini.
Insufficient memory or disk
Transcription and rendering both consume significant resources. 8GB memory is the bare minimum; 16GB is more robust. For disk, the whisper model is about 3GB, YOLO model about 6MB, plus video files and intermediate artifacts; an 8-minute video needs roughly 10GB of temporary space throughout. After processing, remember to clean the output directory.
Extended Reading and Advanced Directions
GPU Acceleration: The default CPU image processes an 8-minute video in 5-8 minutes; with an NVIDIA GPU (any model supporting NVENC), this can drop to 1 minute. Refer to the docker-compose.override.yml configuration in the README, enable the GPU: "1" build argument, and set WHISPER_DEVICE=cuda and FFMPEG_ENCODER=auto.
API and MCP Integration: OpenShorts provides a full REST API and MCP server, supporting natural-language control via Claude or ChatGPT: "Cut this podcast into 3 TikToks." When self-hosting, no key is needed; the service is ready to use upon startup.
Complete local run without Google Key: The README provides detailed guidance for local LLM configuration. Besides text selection, you can configure tools like Ollama to keep the entire pipeline offline. Note that Ollama requires setting OLLAMA_CONTEXT_LENGTH=16384; otherwise, the default 4096 context will silently truncate, leading to incomplete JSON.
Notes on Licensing: The core application is MIT-licensed, fully free. However, code in the cloud/ directory (billing, hosted infrastructure) uses the OpenShorts Commercial License, which can be used personally or internally, but cannot be offered to third parties as a paid hosting service. Self-hosting the core features does not require this directory. The functional differences between the cloud version (openshorts.app) and self-hosting are mainly Autopilot (auto-clipping new videos) and auto-publishing (one-click platform connections), which need to be configured with Upload-Post in self-hosting.