Deploy Whisper Speech Recognition on Serverless GPUs

Deploy Whisper by validating transcription locally, packaging the selected runtime for a serverless GPU worker, and checking the returned transcript with audio you control. A complete deployment also needs the provider’s exact handler, request schema, authentication, and endpoint URL; where the linked documentation does not settle those details, this guide leaves them unresolved. For ai model deployment, treat model choice, worker lifecycle, and measurement as separate decisions.

Prepare the runtime and choose a model

The Whisper README defines the setup, model list, and task behavior used here. It says the codebase is expected to work with Python 3.8-3.11 and recent PyTorch versions, installs from PyPI as openai-whisper, and requires the system ffmpeg command-line tool. Rust may be needed if tiktoken has no pre-built wheel for your platform.

Install and run locally

On Ubuntu or Debian, use the documented installation commands:

sudo apt update && sudo apt install ffmpeg
pip install -U openai-whisper

Validate the CLI with files already present:

whisper audio.flac audio.mp3 audio.wav --model turbo

The documented Python transcription calls are:

model = whisper.load_model("turbo")
result = model.transcribe("audio.mp3")
print(result["text"])

Whisper’s transcribe() method reads the entire file and processes it with sliding 30-second windows. Treat this as offline file transcription rather than a ready-made real-time streaming endpoint. A successful local result validates the model and audio path, but it does not prove that a remote request contract is correct.

Match model memory to the worker

The documented model list contains six sizes, four of which have English-only variants:

Model	Parameters	English-only variant	README VRAM figure
tiny	39 M	tiny.en	~1 GB
base	74 M	base.en	~1 GB
small	244 M	small.en	~2 GB
medium	769 M	medium.en	~5 GB
large	1550 M	N/A	~10 GB
turbo	809 M	N/A	~6 GB

Compare the selected model with the GPU memory in the provider’s current catalog. The VRAM figures are approximate, so they do not guarantee that the complete worker will fit a particular GPU.

The README describes turbo as an optimized version of large-v3 with faster transcription and minimal accuracy degradation. It is not trained for translation: even when --task translate is supplied, it returns the original language. Its relative-speed figures were measured on English speech with an A100, and the README warns that real-world speed can vary with language, speaking speed, and available hardware. Do not convert those figures into a latency promise for your endpoint.

Choose a GPU runtime deliberately

The original openai-whisper package and faster-whisper are different runtimes. The faster-whisper README describes faster-whisper as a reimplementation of Whisper using the CTranslate2 inference engine. It requires Python 3.9+ and does not require a system FFmpeg installation because PyAV bundles the FFmpeg libraries.

The latest CTranslate2 path supports CUDA 12 and cuDNN 9. The README names this NVIDIA CUDA base image:

nvidia/cuda:12.3.2-cudnn9-runtime-ubuntu22.04

Its installation and GPU model-initialization examples are:

pip install faster-whisper

model = WhisperModel(model_size, device="cuda", compute_type="float16")

This is a model-initialization fragment, not a complete serverless handler. Obtain the surrounding code, model identifier, and request integration from the current package documentation rather than inventing them. The same documentation maps float16 to FP16 on GPU, int8_float16 to INT8 on GPU, and int8 to INT8 on CPU.

Pay particular attention to the segments result: it is a generator, and transcription starts only when it is iterated. A handler that returns the generator without consuming it may appear to accept a request without performing the work.

Plan ai model deployment around the worker lifecycle

RunPod’s Serverless pricing documentation says billing starts when a worker starts and continues until it fully stops, rounded up to the next second. Its billable worker time includes container initialization and model loading, request execution, and the idle-timeout period. Container startup therefore belongs in both the latency record and the cost calculation.

RunPod’s Serverless workers overview says Flex workers scale to zero when unused and cost nothing while unused, but incur cold starts when scaling up. The endpoint settings documentation says Active workers remain warm and incur charges continuously, including while idle. Its documented endpoint defaults are 0 Active workers, 3 maximum workers, a 5 s idle timeout, a 600 s execution timeout, a 24 h job TTL, and FlashBoot enabled.

These values are configuration defaults, not workload guarantees. Select a worker mode and its settings against your request pattern, audio duration, concurrency, and tolerance for startup delay.

The cited material does not settle the complete handler entry point, request body keys, response schema, authentication syntax, endpoint URL format, or endpoint-creation command. Before deploying, obtain these exact values from RunPod’s current Serverless deployment documentation:

• the supported handler entry point;
• the image or build settings;
• the request field carrying audio and its accepted encoding;
• the response returned to the caller;
• the authentication header;
• the endpoint URL and creation method.

Do not guess a file path, JSON key, header, or URL.

Build and verify the endpoint in a fixed order

After obtaining the provider contract, map its audio field to the selected Whisper runtime, return the documented response shape, and place model loading where the current handler template says it belongs. Package the runtime in the provider-supported environment, create the endpoint, and record the selected worker mode and timeout settings.

Use an audio file with an expected transcript. Confirm the local result first, then submit the same file through the exact remote request format. Verify the provider’s documented success signal, returned text, detected language, and error behavior. A worker startup log is not sufficient: the endpoint must return the expected transcript through its supported response path.

Keep the initial request separate from later requests so cold-start effects do not disappear inside an average. Record request submission, worker readiness, model availability, transcription start and completion, and response return.

Measure latency and calculate the bill

For benchmark terminology, the MLPerf Inference Rules require every submission to use LoadGen, which generates queries, tracks latency, validates accuracy, and computes final metrics. The rules define latency from the scheduled handoff of a query to the system under test until a reply is received.

In the MLPerf Server scenario, queries follow a Poisson distribution and the reported metric is maximum Poisson throughput under a benchmark-specific latency constraint. In the Offline scenario, LoadGen sends all samples at the start, applies no latency constraint, and reports measured throughput. These definitions can structure your test plan, but a custom request script is not by itself an MLPerf submission.

For RunPod worker time, fill this calculation from your own timers, current quote, or bill:

BILLABLE_SECONDS_PER_WORKER = ceil(
  YOUR_START_SECONDS + YOUR_EXECUTION_SECONDS + YOUR_BILLED_IDLE_SECONDS
)
WORKER_COST = BILLABLE_SECONDS_PER_WORKER * YOUR_GPU_RATE
TOTAL_COST = WORKER_COST + YOUR_OTHER_LINE_ITEMS

Read the current GPU rate from the provider’s pricing page, the worker mode and timeout settings from the endpoint configuration, and the billed worker period and any additional categories from your own invoice. Do not insert a price from this article. Include an additional cost category only if it appears in your quote or bill.

For an Active worker, carry the full continuously billed period into the calculation rather than summing only request execution times. For a Flex worker, retain the startup period in the latency record even though unused capacity can scale to zero.

Where the setup usually goes wrong

• Skipping runtime prerequisites: Check the selected package’s Python, media-tool, CUDA, and platform requirements before changing model code.
• Treating the local command as an API: Derive the remote method, body, encoding, authentication, and response from current provider documentation.
• Measuring only model execution: Preserve submission, startup, execution, response, and billed idle phases as separate fields.
• Leaving worker settings implicit: Record the worker mode, maximum concurrency, idle timeout, execution timeout, and TTL used for the run.
• Using published speed as an SLA: Measure your own audio, language, concurrency, worker mode, and hardware before setting an expectation.
• Calling a sample response a benchmark: Define the input set, arrival pattern, accuracy check, latency definition, and reporting period before comparing configurations.

Production gate

Archive the chosen model, runtime, image, worker settings, provider documentation, sample input, expected transcript, actual response, error behavior, latency breakdown, and billed-worker calculation. Recheck the current documentation whenever any deployment input changes. These steps are compiled from the official documentation linked in the text; nothing was run or measured for the article, and the current version should be checked before use.

Sources

• OpenAI Whisper README
• SYSTRAN faster-whisper README
• RunPod Serverless pricing
• RunPod Serverless workers overview
• RunPod endpoint settings
• MLPerf Inference Rules