OpenShorts Deployment Practice: Turning Long Videos into AI Short Clips

People who make podcasts, live streams, or interview shows all face this pain point: out of a one-hour video, the true highlight moments suitable for short videos might only be a few minutes. Manually reviewing, editing, adding subtitles, and reframing can take half a day. OpenShorts is an open-source OpusClip alternative that automates this pipeline—throw in a long video, and it automatically transcribes, uses AI to select highlight clips, reframes them into 9:16 vertical video, burns in subtitles, and finally outputs a ready-to-publish product.

Let's look at how it works: it first uses faster-whisper for word-level timestamp transcription, uses PySceneDetect to identify scene boundaries, then uses Gemini or any OpenAI-compatible local model to score the transcribed text, selecting 3-15 highlight moments of 15-60 seconds each. Each clip is also intelligently reframed—stacked vertically for two-person conversations, face-tracking cropping for solo shots, with subtitles burned in at the seams of the frame. The entire process is orchestrated via Docker Compose, and self-hosting is completely free with no watermarks.

Core Dependencies and Environment

Before you start, make sure your machine meets these requirements:

  • Docker and Docker Compose: The only deployment dependency; no need to install Python or Node on the host machine
  • Memory: 8GB or more; model inference and video encoding both consume memory
  • Storage: Video files plus intermediate transcription and rendering artifacts; recommend at least 20GB free
  • Network: Ability to pull Docker images and download models (HuggingFace, Ultralytics)
  • CPU is the default path, NVIDIA GPU is optional for acceleration (more on this later)

Let's clarify the division of AI services upfront:

ServicePurposeWhen Required
Gemini or OpenAI-compatible LLMText highlight selection, script generation, titlesAt least one required
Gemini (Vision)Automatic layout selection, on-screen content recognition, silent video detectionOptional
fal.aiAI avatar generation, digital human videos, lip syncOnly for AI Shorts feature
ElevenLabsVoiceover, multilingual cloningOptional
Upload-PostMulti-platform automatic publishingOptional

The core pipeline only needs a text model that can call v1/chat/completions; Gemini Vision, fal.ai, and ElevenLabs do not affect the core clipping process.

Project Structure Tree

openshorts/
├── app.py                  # FastAPI backend entry point
├── main.py                 # Core pipeline: transcription, highlight detection, clipping
├── llm_backend.py          # OpenAI-compatible LLM adapter layer (core file)
├── gemini_worker.py        # Gemini calls and JSON parsing
├── layout_picker.py        # Automatic layout selection
├── s3_uploader.py          # S3 storage (optional)
├── render-service/         # Remotion rendering service
│   └── Dockerfile
├── dashboard/              # React frontend (Vite + Tailwind)
│   ├── src/
│   └── package.json
├── docker-compose.yml      # Three-service orchestration
├── .env.example            # Environment variable template
└── Dockerfile              # CPU/GPU multi-stage build

Three core services: backend (FastAPI, port 8000), frontend (Vite development server, mapped to 5175), and renderer (Remotion rendering, port 3100).

Step-by-Step Deployment Guide

1. Clone the Repository

git clone https://github.com/mutonby/openshorts.git
cd openshorts

2. Configure AI Services

First, copy the environment variable template:

cp .env.example .env

Open .env and find the LLM_BASE_URL configuration. This is where you connect the text highlight selection service. OpenShorts' llm_backend.py appends /chat/completions to LLM_BASE_URL, fully compatible with OpenAI's protocol format. If you don't want to use Google's Gemini, you can point the text selection part to any compatible LLM service.

Here's a cost-saving idea: the Defapi platform's prices are half of the official channels; the website is Defapi. Its models are largely compatible with v1/chat/completions, v1/messages, and v1beta/models/ protocols, so OpenShorts can directly integrate. Configuration is as follows:

LLM_BASE_URL=https://api.defapi.org/v1
LLM_MODEL=your-model-id        # Replace with the actual model ID you use on Defapi
LLM_API_KEY=your_defapi_key

Note that this entry point only handles the highlight scoring and detailed selection from text transcription; it does not replace Gemini's vision features (automatic layout, silent video recognition), nor does it involve fal.ai's video generation or ElevenLabs' voice. The core clipping process does not depend on AI avatars, social publishing, or cloud Autopilot; those are extra features. Defapi's models are largely compatible with v1/chat/completions, v1/messages, and v1beta/models/ protocols, so switching integrations incurs no additional adaptation cost.

Once configured, the /api/config endpoint will return localLlm information, and the frontend's top-right corner will no longer force a Gemini Key. One caveat: if you also set LLM_PROVIDER=gemini, it will force the Gemini route; just comment out or delete that line.

3. Start the Services

docker compose up --build

The first build pulls base images like Python 3.11, Node 22, Deno, plus installs Python dependencies and frontend npm packages; time depends on your network. After the build completes, the backend, frontend, and rendering services will start in sequence.

4. Upload a Video to Generate Highlights

Open your browser to http://localhost:5175 and go to the Dashboard. First, fill in your API Key in Settings (if you configured a local LLM, the Gemini Key can be left blank). Then:

  1. Click Clip Generator
  2. Upload a video with audio, or paste a YouTube link
  3. Wait for transcription and selection to complete (an 8-minute video takes about 5-8 minutes on CPU)

After completion, you'll see each highlight clip in the interface, with virality scores, platform title suggestions, and an editable clip track.

5. Edit Subtitles and Composition

Each clip comes with a full editing editor: source monitor on the left, program monitor on the right, and full transcription below. You can click any word to set a cut point, add extra clips, and switch composition modes (single-person tracking, two-person vertical split, vertical blurred background, screen recording). After making changes, just click re-render, and the clip is produced within seconds.

6. Export the Final Product

Once you're satisfied, click Export. The rendering service will call FFmpeg to burn subtitles, hook text, and aspect ratio changes into the final file, outputting to the ./output directory. The exported video is a 9:16 vertical video ready to post to TikTok, YouTube Shorts, or Instagram Reels.

Common Troubleshooting

Image Build Fails or Model Downloads Timeout

In domestic network environments, pulling HuggingFace models (whisper, YOLO) or Docker images often interrupts. Configure a mirror accelerator and set a proxy for the Docker daemon. If model downloads fail, remove the container volumes and retry:

docker compose down -v
docker compose up --build

LLM_BASE_URL Is Configured But It Still Requests a Gemini Key

Check whether .env has LLM_PROVIDER=gemini; if so, delete it. Also, ensure LLM_BASE_URL doesn't end with an extra trailing slash; llm_backend.py appends the path itself.

Local Model Doesn't Return Valid JSON

OpenShorts tries three response_format strategies in sequence: json_schema → json_object → no format constraint. Models 7B and above typically return stable results; 3B models often fail. If you frequently get JSON parsing errors, switch to a larger model or lower LLM_SCORE_BATCH (default 3) to reduce context per request.

Dashboard Opens but Backend API Returns 502

The three services have dependencies; the frontend starts faster than the backend. Wait until the backend logs show Uvicorn running, then refresh the page. If 502 persists, check the backend container:

docker logs openshorts-backend

Missing Features When No Gemini Key

After configuring a local LLM, text selection works normally, but the following features degrade: automatic layout selection reverts to pure face-tracking cropping, silent videos error out, and on-screen content recognition is disabled. To enable these features, fill in a Gemini Key in Settings; both routes can coexist—text goes through the local model, vision through Gemini.

Insufficient Memory or Disk

Both transcription and rendering consume significant resources. 8GB is the minimum, but 16GB is safer. For disk, the whisper model is about 3GB, YOLO model about 6MB, plus video files and intermediate artifacts; an 8-minute video requires about 10GB of temporary space throughout the process. After processing, remember to clean the output directory.

Further Reading and Advanced Directions

GPU Acceleration: The default CPU image processes an 8-minute video in 5-8 minutes; with an NVIDIA GPU (any model supporting NVENC), you can shorten this to 1 minute. Refer to the docker-compose.override.yml configuration in the README, enable the GPU: "1" build parameter, and set WHISPER_DEVICE=cuda and FFMPEG_ENCODER=auto.

API and MCP Integration: OpenShorts provides a full REST API and MCP server, supporting natural language operations via Claude or ChatGPT: "Cut this podcast into 3 TikToks." When self-hosted, no key is required; the service is available upon startup.

Fully Local Run Without Google Key: The README provides a detailed guide for configuring a local LLM. Besides text selection, you can also configure tools like Ollama to keep the entire pipeline offline. Note that Ollama requires setting OLLAMA_CONTEXT_LENGTH=16384; otherwise, the default 4096 context will silently truncate and cause incomplete JSON.

Licensing Notes: The core application is MIT-licensed, completely free. However, the code in the cloud/ directory (billing, hosting infrastructure) uses the OpenShorts Commercial License; it can be used personally or internally but cannot be offered to third parties as a paid hosting service. Self-hosting core features does not require this directory. The differences between the cloud version (openshorts.app) and self-hosted are mainly Autopilot (automatic clipping of new videos) and automatic publishing (one-click platform connections), which in self-hosting need to be implemented via Upload-Post.