Operation

This document describes django-video-transcoding operation details.

Proper shutdown

Processing video files is a very long operation, so waiting for celery task complition while shutting down is inacceptable. On the other hand, if celery worker processes are killed, lost tasks an zombie ffmpeg processes may appear.

  • if you are running transcoder in a container, make sure that celery master process has pid 1 (docker will send SIGTERM to it by default)

  • for separate celery app make sure that it sets process group for all workers and propagates SIGUSR1 to trigger graceful worker shutdown

Resumable workers

For modern cloud-based installations an optimal solution is to use Preemptible VM instances. On the one hand they are inexpensive, but on the another these VMs can be shut down at any time.

Transcoding VOD files is a very time and CPU consuming operation, so the implemetation should support resumable transcoding.

For large container-based deployments (docker, K8s) resumable transcoding simplifies hardware failure handling and release management.

Django-video-transcoding implements resumable strategy for transcoding:

  • It splits source file into 1-minute chunks while downloading

  • Each chunk is transcoded independently

  • At the end all chunks are concatenated and segmented to HLS streams

  • After restart each step can be skipped if it’s result already exists

So, temporary storage should be persistent and host-independent. We recommend mounting S3 bucket as a file system.

Getting sources

ffmpeg supports a large number of ingest protocols, such as http and ftp. This allows downloading source video and splitting it to chunks in a single pass. The main drawback is inability to tune some protocol options, especially for ftp. We recommend to use http for source videos.

Storing temporary files

For some reasons the segment muxer is used to split source video into chunks, and this muxer does not support setting http method. If temporary storage is accessed via HTTP, it must support storing files via POST requests.

Serving HLS streams

Serving video requires high network bandwidth and fast drives, especially if number of users is large. It’s recommended to use CDN to protect video storage from high repetitive load.

  • Use CloudFront with S3 or there analogues from other cloud providers

  • Or use CDN provider in front of HTTP server

  • For self-hosted solutions distribute network load across multiple edge servers (round robin is supported by multiple hosts in VIDEO_EDGES env variable).