🛟

OpenAI Lifeboat

Switch your language model provider to Meta Llama 3 70B with 3 lines of code.

Sometimes, open is just better.

Board the lifeboatLearn more about Llama 3 
1import OpenAI from "openai";
2
3const openai = new OpenAI({
4  apiKey: process.env["REPLICATE_API_TOKEN"],
5  baseURL: "https://openai-proxy.replicate.com/v1",
6});
7
8const completions = await openai.chat.completions.create({
9  model: "meta/meta-llama-3-70b-instruct",
10  messages: [
11    {
12      role: "user",
13      content: "Write a haiku about camelids",
14    },
15  ],
16  maxTokens: 64,
17  stream: true,
18});
19for await (const part of completions) {
20  process.stdout.write(part.choices[0]?.delta || "");
21}

Quick Start

To get started, create a Replicate account and copy your API token. Set your token as an environment variable. Install the OpenAI client if you haven't already.

Next:
  1. Change your api_key from your OpenAI key to your REPLICATE_API_TOKEN environment variable wherever you initialize the OpenAI client.
  2. Point your base_url to https://openai-proxy.replicate.com/v1
  3. Set your model to meta/meta-llama-3-70b-instruct

Your code should now look like this:

import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env["REPLICATE_API_TOKEN"],
  baseURL: "https://openai-proxy.replicate.com/v1",
});

const completions = await openai.chat.completions.create({
  model: "meta/meta-llama-3-70b-instruct",
  messages: [
    {
      role: "user",
      content: "Write a haiku about camelids",
    },
  ],
  maxTokens: 64,
  stream: true,
});
for await (const part of completions) {
  process.stdout.write(part.choices[0]?.delta || "");
}

That's it! You're now using an open source model. Enjoy.

Pricing

Lifeboat is the same price as language models on Replicate.

Function Calling

Function calling is also available via lifeboat. Function calling allows you to define structured data for the model to return.

You can enforce JSON responses by specifying response_format: { type: "json_object" }. Or, you can define a specific schema by adding a tool with type function. See below for an example.

Note: function calling is currently only supported with Llama 2. Llama 3 support is on the way!

import OpenAI from "openai";

// initialize the OpenAI client
const openai = new OpenAI({
  apiKey: process.env["REPLICATE_API_TOKEN"],
  baseURL: "https://openai-proxy.replicate.com/v1",
});

// You can enforce a JSON object response format
const completions = await openai.chat.completions.create({
    model: "meta/llama-2-70b-chat",
    messages: [
        {
            role: "system",
            content: "Respond in JSON format",
        },
        {
            role: "user",
            content: "Sweden's top three tourist attractions",
        }
    ],
    response_format: { "type": "json_object" },
    max_tokens: 128,
    stream: true
});
for await (const part of completions) {
    process.stdout.write(part.choices[0]?.delta || '');
}

// Or, you can also define your own tools for function calling
const completions = await openai.chat.completions.create({
    model: "meta/llama-2-70b-chat",
    messages: [
        {
            role: "user",
            content: "What's the weather in San Francisco? (Celsius, please)",
        }
    ],
    tools: [
        {
            type: "function",
            function: {
                "name": "get_current_weather",
                "description": "Get the current weather in a given location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g. San Francisco, CA"
                        },
                        "unit": {
                            "type": "string",
                            "enum": ["celsius", "fahrenheit"]
                        }
                    },
                    "required": ["location"]
                }
            }
        }
    ],
    max_tokens: 500,
    stream: true
});
for await (const part of completions) {
    process.stdout.write(part.choices[0]?.delta || '');
}

"Until recently, I would have recommended myself. But with the latest developments, there's one obvious choice: go open source."

🤖
- GPT-4

Go open source

It's stormy out there.
Jump on a lifeboat.

Three line change

Keep your OpenAI client code. Just change the base URL, API key, and model name, and you're good to go.

Control your weights

With open source models, you control your weights. The API isn't going to change from under you.

Fast

Llama 3 70B is competitive with GPT-4, Claude 3, and Mistral-Large. Use it out of the box, or fine-tune Llama 2 to do things that aren't possible with proprietary models.

Board the lifeboatRun Llama 3 on Replicate