Replicate — Run Open-Source AI Models Guide

In this tutorial, you'll learn about Replicate. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

The Replicate API provides cloud-hosted access to thousands of open-source AI models for image generation, language processing, audio transcription, and video creation without managing GPU infrastructure.

What You'll Learn

• Setting up Replicate API tokens and client libraries
• Running popular open-source models like Llama 3 and Stable Diffusion
• Passing inputs, handling outputs, and polling for predictions
• Building Webhook-based notification workflows
• Fine-tuning models with custom datasets
• Managing costs, scaling, and production deployment

Why Replicate Matters

Replicate eliminates GPU infrastructure complexity. Instead of provisioning and maintaining expensive GPU servers, you call a REST API to run open-source models. This is critical for startups and solo developers who need State-of-the-art AI without DevOps overhead. DodaTech's DodaZIP uses Replicate's Whisper model for batch audio transcription, and Durga Antivirus Pro uses image generation models to create synthetic training data for malware screenshot classification. This guide walks through every feature of the Replicate API.

flowchart LR
    A["Replicate API Token\n& Client Setup"] --> B["Run Models\nLlama 3, SDXL"]
    A --> C["Webhooks\nAsync Workflows"]
    A --> D["Fine-Tuning\nCustom Training"]
    B --> E["Prediction Polling\nStatus & Output"]
    D --> F["Trained Model\nDeployment"]
    B --> G["Output Handling\nImages, Text, Audio"]
    style B fill:#dbeafe,stroke:#2563eb

Getting Started with Replicate

Replicate uses API tokens for authentication. Sign up at replicate.com, Go to Account Settings, and copy your API token.

import replicate
import os

client = replicate.Client(API_token=os.environ["REPLICATE_API_TOKEN"])

Expected output: No output — the client initializes silently. If REPLICATE_API_TOKEN is missing, you'll see replicate.exceptions.ReplicateError: authentication required.

Replicate hosts thousands of models. Here are the most popular categories:

Running a Language Model

Running an open-source LLM on Replicate is as simple as calling a function. Let's run Meta's Llama 3 model.

import replicate
import os

output = replicate.run(
    "meta/meta-llama-3-70b-instruct",
    input={
        "prompt": "Explain how AES-256 encryption works in simple terms.",
        "temperature": 0.7,
        "max_tokens": 300,
        "top_p": 0.95
    }
)

full_text = ""
for chunk in output:
    full_text += chunk
print(full_text)

Expected output:

AES-256 (Advanced Encryption Standard with 256-bit keys) is a symmetric encryption algorithm that protects data using a single secret key. Think of it like a high-security lockbox:

1. **Key Generation** — A 256-bit key (32 characters) is created. This key is the only way to lock and unlock the data.
2. **Encryption** — The algorithm divides data into 128-bit blocks and applies multiple rounds of substitution and permutation (14 rounds for 256-bit keys).
3. **Decryption** — The same key reverses the process, restoring the original data.

AES-256 is the standard used by governments and security tools like Durga Antivirus Pro to protect threat intelligence databases. Without the correct key, brute-forcing would take billions of years with current hardware.

The output is a generator that yields text chunks. You can also set stream=False to receive the complete output as a single string. Replicate supports both synchronous (streaming) and asynchronous (polling) modes.

Image Generation with Stable Diffusion

Let's generate an image using Stable Diffusion XL. The output is a URL to the generated image.

output = replicate.run(
    "stability-ai/stable-diffusion-3.5",
    input={
        "prompt": "A futuristic data center with blue neon lighting, server racks, holographic displays, cyberpunk aesthetic",
        "negative_prompt": "blurry, low quality, distorted, ugly",
        "width": 1024,
        "height": 1024,
        "num_outputs": 1,
        "scheduler": "DPMSolverMultistep",
        "num_inference_steps": 28,
        "guidance_scale": 7.5
    }
)

print(output[0])

Expected output:

https://replicate.delivery/pbxt/ABCDEF123456789_output.png

The output is a list of image URLs that are valid for approximately one hour. Download them immediately or store them in your own cloud storage. DodaZIP uses this pattern to generate custom archive cover images for users.

Webhooks for Asynchronous Workflows

For long-running predictions (fine-tuning, video generation), use Webhooks to receive notifications instead of polling.

from replicate import Client
import os

client = replicate.Client(API_token=os.environ["REPLICATE_API_TOKEN"])

prediction = client.predictions.create(
    model="meta/meta-llama-3-70b-instruct",
    input={
        "prompt": "Write a 500-word blog post about cybersecurity trends in 2026"
    },
    Webhook="HTTPS://API.mysite.com/replicate-Webhook",
    Webhook_events_filter=["completed", "failed"]
)

print(f"Prediction ID: {prediction.id}")
print(f"Status: {prediction.status}")

Expected output:

Prediction ID: abc123def456
Status: starting

Replicate POSTs to your Webhook URL when the prediction completes or fails. The Webhook payload includes the full prediction object with outputs. This is how Doda Browser handles long-running AI tasks without blocking the user interface:

# Flask Webhook handler example
from Flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/replicate-Webhook", methods=["POST"])
def handle_Webhook():
    data = request.JSON
    if data["status"] == "completed":
        outputs = data["output"]
        print(f"Prediction {data['id']} completed with output: {outputs}")
    elif data["status"] == "failed":
        print(f"Prediction {data['id']} failed: {data.get('error')}")
    return jsonify({"status": "ok"}), 200

Expected output: Console logs showing prediction completion or failure as Webhooks arrive.

Fine-Tuning Models

Replicate supports fine-tuning certain models with your own data. This teaches a BASE model your specific patterns or style.

training = replicate.trainings.create(
    version="stability-ai/sdxl:39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b",
    input={
        "input_images": "HTTPS://storage.mysite.com/training-images.zip",
        "caption_prefix": "A photo of DODA_STYLE",
        "max_train_steps": 1000,
        "learning_rate": 1e-4
    },
    destination="mydodatech/my-sdxl-model"
)

print(f"Training ID: {training.id}")
print(f"Status: {training.status}")

Expected output:

Training ID: train_abcdef123456
Status: starting

The destination parameter specifies where the trained model will be saved in the format owner/model-name. Training takes 30-90 minutes depending on dataset size. Once complete, you run your fine-tuned model just like any other:

output = replicate.run(
    "mydodatech/my-sdxl-model",
    input={
        "prompt": "A photo of DODA_STYLE server room with blue lighting",
        "width": 1024,
        "height": 1024
    }
)
print(output[0])

Expected output: A URL to an image generated in your fine-tuned style — consistent server room images with the DODA_STYLE aesthetic applied.

Durga Antivirus Pro uses fine-tuned models to generate synthetic screenshots of malware infection scenarios for training its detection systems.

Audio Transcription with Whisper

Replicate hosts Whisper for high-accuracy speech-to-text. This is useful for transcribing meetings, podcasts, or security audio logs.

output = replicate.run(
    "OpenAI/whisper",
    input={
        "audio": "HTTPS://storage.mysite.com/meeting_recording.mp3",
        "model": "large-v3",
        "language": "en",
        "response_format": "JSON",
        "temperature": 0.0
    }
)

print(output["text"][:500])

Expected output:

All right, let's start the security review meeting. Today we're going to discuss the findings from last week's penetration test. First item on the agenda is the SQL injection vulnerability found in the user login endpoint. John, can you walk us through that?

Yeah, so we found that the login form doesn't sanitize input properly. An attacker could inject SQL commands through the username field. We're recommending parameterized queries to fix this...

Whisper large-v3 provides near-human accuracy. DodaZIP uses this to transcribe user voice notes that describe what files they want to compress.

Batch Processing and Queue Management

For production workloads, you can manage prediction queues and check status programmatically.

predictions = client.predictions.list()

for pred in predictions.results[:5]:
    print(f"ID: {pred.id} | Model: {pred.model} | Status: {pred.status} | "
          f"Created: {pred.created_at}")

# Cancel a running prediction
running = [p for p in predictions.results if p.status == "starting"]
if running:
    running[0].cancel()
    print(f"Cancelled prediction {running[0].id}")

Expected output:

ID: pred_abc123 | Model: meta/meta-llama-3-70b-instruct | Status: succeeded | Created: 2026-06-21T10:00:00Z
ID: pred_def456 | Model: stability-ai/stable-diffusion-3.5 | Status: processing | Created: 2026-06-21T10:05:00Z
Cancelled prediction pred_def456

You can list, inspect, and cancel predictions programmatically. This is essential for building Queue-based systems where multiple users submit AI tasks concurrently.

Common Errors

1. ReplicateError: authentication required

The API token is missing or invalid. Set REPLICATE_API_TOKEN environment variable or pass it directly. Verify the token at replicate.com/account.

2. ModelError: Model not found

The model identifier is incorrect. Format is always owner/model-name (e.g., meta/meta-llama-3-70b-instruct). Check the exact identifier on Replicate's model page.

3. InputValidationError: Invalid input parameter

A required input is missing or has the wrong type. Each model documents its inputs. Use replicate.models.get("owner/model-name") to inspect the schema.

4. InsufficientCreditsError: Out of credits

Your account has insufficient credits. Replicate is pay-as-you-Go. Check usage at replicate.com/usage and add credits.

5. RateLimitError: Too many requests

Free tier: 10 concurrent predictions. Paid tiers: 50-200 depending on plan. Queue or throttle requests to stay within limits.

6. PredictionFailed: Model execution error

The model itself encountered an error. This can happen with GPU memory issues on very large inputs. Try reducing input size or using a smaller model variant.

7. TimeoutError: Prediction took too long

Predictions time out after 60 seconds by default for sync calls. Use async mode with Webhooks for longer-running models like video generation.

Practice Questions

1. What is the difference between replicate.run() and client predictions for async workflows?
2. How do Webhooks improve prediction handling for long-running models?
3. What does the destination parameter do in fine-tuning?
4. How does Replicate handle GPU infrastructure for users?
5. What format must model identifiers follow?

Answers:

1. replicate.run() blocks until the prediction completes (synchronous). Client predictions return immediately with a prediction ID for status polling or Webhook delivery.
2. Webhooks POST to your URL when prediction status changes (completed/failed), eliminating the need for constant polling.
3. The destination parameter specifies where the fine-tuned model is saved (owner/model-name) on Replicate's platform.
4. Replicate manages all GPU infrastructure — users only call REST APIs and never provision or manage GPU servers themselves.
5. Model identifiers follow the format owner/model-name (e.g., meta/meta-llama-3-70b-instruct).

Challenge: Doda Browser needs a bulk image processing feature that: takes a ZIP of images, generates captions using BLIP (another Replicate model), creates new images with Stable Diffusion based on those captions, packages the results, and notifies the user via Webhook. Build this pipeline.

Mini Project: AI Content Generation Pipeline

Build a complete pipeline that generates a blog post and a hero image for it:

import replicate
import os
import requests
from datetime import datetime

client = replicate.Client(API_token=os.environ["REPLICATE_API_TOKEN"])

def generate_blog_post(topic: str) -> tuple[str, str]:
    text_output = replicate.run(
        "meta/meta-llama-3-70b-instruct",
        input={
            "prompt": f"Write a 300-word blog post about {topic}. "
                      f"Use a professional but engaging tone.",
            "max_tokens": 600,
            "temperature": 0.8
        }
    )
    post = "".join(text_output)

    image_output = replicate.run(
        "stability-ai/stable-diffusion-3.5",
        input={
            "prompt": f"A professional blog header image about {topic}, "
                      f"clean design, blue and white color scheme, modern style",
            "width": 1024,
            "height": 512,
            "num_outputs": 1
        }
    )
    img_URL = image_output[0]

    img_data = requests.get(img_URL).content
    filename = f"blog_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"
    with open(filename, "wb") as f:
        f.write(img_data)

    return post, filename

post, image = generate_blog_post("cybersecurity best practices for small businesses")
print(f"Generated blog post ({len(post)} chars)")
print(f"Hero image saved as: {image}")
print(f"\nPost preview:\n{post[:200]}...")

Expected output:

Generated blog post (1850 chars)
Hero image saved as: blog_20260621_103000.png

Post preview:
In today's interconnected world, small businesses face the same cybersecurity threats as large enterprises — but often with fewer resources to defend themselves. The good news is that effective security doesn't require a massive budget...

Try it: Extend the pipeline to publish the post to a WordPress site via the REST API. Add SEO metadata generation and social media image variants.

FAQ

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.