Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,8 +118,8 @@ bun test # Run tests
- Release flow follows `CONTRIBUTING.md` — bump version, update changelog, commit, push, then `gh release create` triggers the NPM publish workflow; changelog only tracks changes that affect the npm package (not examples, tests, or docs); the publish workflow auto-detects prerelease versions (e.g. `0.10.0-beta.0`) and publishes to npm with the prerelease identifier as the dist-tag (`--tag beta`)
- `consumeSession` converts `sessionId + sessionKey` → WebRTC credentials (`serverUrl`, `token`, `roomName`) on the client in the SDK; `@runwayml/avatars-react/api` is the server-safe entry point (no React, no `'use client'`) for Next.js API routes or server components so imports do not pull in the client bundle
- `AvatarSession` connects `LiveKitRoom` with local `audio` and `video` off, then enables the mic and camera after the room reaches `Connected` so exclusive camera/mic use by another app (e.g. Zoom) does not block the session from becoming active; media acquisition failures surface via `MediaDeviceError` context / `useLocalMedia` with retry instead of leaving the room stuck connecting
- Primary quickstart reference is `examples/nextjs/` (API routes, more universally understood than server actions); documentation lives on an external docs website — `docs/` and `skills/` folders were intentionally removed from the repo; screen share is enabled via `<ControlBar showScreenShare />` (defaults `false` for backwards compatibility) — the quickstart does not render `<ScreenShareVideo />` by default; the avatar stays full-size and the active button state is the sharing indicator
- Dev scripts auto-detect portless (`command -v portless`) and use it when available; VS Code launch configs (`.vscode/launch.json`) bake SDK watch mode into every example via a background `Watch SDK` task — each example's `preLaunchTask` runs `bun run build && bun link`, then a background tsup watch starts automatically; no separate terminal needed for continuous SDK rebuilds
- Primary quickstart reference is `examples/nextjs/` (API routes, more universally understood than server actions); `examples/nextjs/app/api/avatar/connect/route.ts` may demonstrate optional `baseUrl`, client-event `tools`, and `personality` alongside defaults; documentation lives on an external docs website — `docs/` and `skills/` folders were intentionally removed from the repo; screen share is enabled via `<ControlBar showScreenShare />` (defaults `false` for backwards compatibility) — the quickstart does not render `<ScreenShareVideo />` by default; the avatar stays full-size and the active button state is the sharing indicator
- Dev scripts auto-detect portless (`command -v portless`) and use it when available; VS Code launch configs (`.vscode/launch.json`) bake SDK watch mode into every example via a background `Watch SDK` task — each example's `preLaunchTask` runs `bun run build && bun link`, then a background tsup watch starts automatically; no separate terminal needed for continuous SDK rebuilds; for examples that list `@runwayml/avatars-react` from the registry, prefer that build-and-link flow over changing `package.json` to `file:../..`
- tsup CSS-only entries silently fail to emit in watch mode; the `dev` script pre-copies `src/styles.css` to `dist/` before starting tsup watch, and `rm -rf dist` cleanup only runs during `bun run build` (not during watch)
- Graphite `gt submit` only works after the GitHub repo is added under **Synced repos** in Graphite ([settings](https://app.graphite.dev/settings/synced-repos)); otherwise it errors with “You can only submit to repos synced with Graphite” (org admins may need to enable the Graphite GitHub app for `runwayml/avatars-sdk-react`). The CLI resolves the repo from `git remote origin` — it must match that GitHub slug exactly (do not confuse the NPM package name `@runwayml/avatars-react` with the repo path `runwayml/avatars-sdk-react`, or typo `avatar-sdk-react` vs `avatars-sdk-react`). Until then, use `git push -u origin <branch>` + `gh pr create`. Local commands (`gt ls`, `gt sync`, `gt checkout`, `gt modify`, `gt create`) still work.
- Client events are fire-and-forget messages from the avatar model delivered via LiveKit data channel (`RoomEvent.DataReceived`); exposed through `onClientEvent` prop, `useClientEvents<T>` (catch-all), and `useClientEvent<E, T>` (filtered by tool name; latest args as state + optional callback); server also sends ack messages with `args: { status: "event_sent" }` that `parseClientEvent` filters out; examples with rich UI should include a `/dev` page for testing states (question cards, score, confetti, error) without a live avatar session
Expand All @@ -128,4 +128,4 @@ bun test # Run tests
- The trivia examples (`examples/nextjs-client-events/`, `examples/nextjs-rpc/`) use a single `next_step` client event per turn with personality/startScript as repo constants in `lib/trivia-personality.ts`; `personality`/`startScript` are set only in server `realtimeSessions.create` (not passed from the client)—keep `startScript` short so the model does not treat the greeting as already having asked the first question; intro → tool → spoken `question` order is instruction-only (no SDK hook to wait for playback before the client event); keep personality within the API character limit (~2000) and each tool's `timeoutSeconds` ≤8; exceeding the char limit returns a length-specific 400; a *different* 400 ("This text cannot be used for an avatar") is content moderation — avoid pop culture character names and suggestive phrasing in `personality`/`startScript`; realtime create fields may still be cast with `as any` until `@runwayml/sdk` types include them; the RPC trivia example adds `@runwayml/avatars-node-rpc` (GitHub dep) — `next.config.ts` must include `serverExternalPackages: ['@runwayml/avatars-node-rpc', '@livekit/rtc-node']`; `examples/nextjs-rpc-weather/`
is a standalone RPC-only example (no client events); all tool-calling examples use preset avatars (`runway-preset`) with `personality`/`startScript`/`tools` overrides targeting the production API
- Cross-session audio routing (two avatars hearing each other) is not supported by the SDK; achieving avatar-to-avatar conversation requires Web Audio API bridging in the browser or server-side LiveKit audio forwarding — the SDK intentionally does not expose the underlying LiveKit room object to consumer code
- Public-facing examples target production API only (`new Runway()` with no `baseURL` override); don't build multi-environment infrastructure — keep `.env.example` minimal (just `RUNWAYML_API_SECRET`); hardcode preset IDs and other constants directly in code rather than env var indirection; for internal staging/dev testing, pass `baseUrl` to `AvatarCall`/`AvatarSession` and set `NEXT_PUBLIC_RUNWAYML_BASE_URL` in `.env.local`
- Public-facing examples target production API only (`new Runway()` with no `baseURL` override); don't build multi-environment infrastructure — keep `.env.example` minimal (just `RUNWAYML_API_SECRET`); hardcode preset IDs and other constants directly in code rather than env var indirection; keep example code straightforward — don't add defensive error-handling utilities, env-var validation guards, or staging base URL plumbing to committed examples; for internal staging/dev testing, pass `baseUrl` to `AvatarCall`/`AvatarSession` and set `NEXT_PUBLIC_RUNWAYML_BASE_URL` in `.env.local`; also pass `baseUrl` to `createRpcHandler` since it defaults independently to production and does not auto-read `RUNWAYML_BASE_URL`
7 changes: 7 additions & 0 deletions examples/nextjs-rpc-external-api/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Runway API secret (required)
# Get yours at https://dev.runwayml.com/
RUNWAYML_API_SECRET=

# Avatar ID (required)
# Create one at https://dev.runwayml.com/
NEXT_PUBLIC_AVATAR_ID=
8 changes: 8 additions & 0 deletions examples/nextjs-rpc-external-api/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
.next/
node_modules/
.env
.env.local
package-lock.json
bun.lock
tsconfig.tsbuildinfo
next-env.d.ts
41 changes: 41 additions & 0 deletions examples/nextjs-rpc-external-api/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# External API Calling — Backend RPC Example

An AI avatar that calls **real external APIs** using **backend RPC tool calls**. Uses the ESPN public API as an example — ask about scores, standings, news, or league leaders and the avatar fetches live data and speaks the results.

## What it demonstrates

- **Backend RPC tools** — the avatar triggers server-side functions via `@runwayml/avatars-node-rpc`
- **Real external API calls** — each tool makes an HTTP request to the ESPN public API (not mock data)
- **Multiple tools** — four distinct tools the avatar can invoke depending on the question
- **Network latency** — demonstrates how external API round-trips affect avatar responsiveness

## Tools

| Tool | What it does |
|------|-------------|
| `get_scores` | Fetches current/recent game scores and matchups |
| `get_standings` | Fetches league standings by division/conference |
| `get_news` | Fetches the 3 latest news headlines |
| `get_leaders` | Fetches statistical leaders for a season |

## Quick start

```bash
npx degit runwayml/avatars-sdk-react/examples/nextjs-rpc-external-api my-api-app
cd my-api-app
cp .env.example .env.local
# Add your RUNWAYML_API_SECRET and NEXT_PUBLIC_AVATAR_ID

npm install
npm run dev
```

## Architecture

```
lib/personality.ts (sports assistant persona)
lib/sports-api.ts (ESPN API calls — real HTTP requests)
└── app/api/.../route.ts (server: creates session + RPC handler)
└── app/page.tsx (client: AvatarCall + video + controls)
```
144 changes: 144 additions & 0 deletions examples/nextjs-rpc-external-api/app/api/avatar/connect/route.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
import Runway from '@runwayml/sdk';
import type { RealtimeSessionCreateParams } from '@runwayml/sdk/resources/realtime-sessions';
import { createRpcHandler, type RpcHandler } from '@runwayml/avatars-node-rpc';
import { getScores, getStandings, getNews, getLeaders } from '@/lib/sports-api';
import { SPORTS_PERSONALITY, SPORTS_START_SCRIPT } from '@/lib/personality';

export const runtime = 'nodejs';

const client = new Runway();

const activeHandlers = new Map<string, RpcHandler>();

let callSequence = 0;

function timed<T extends Record<string, unknown>>(
name: string,
fn: (args: T) => Promise<unknown>,
) {
return async (args: T) => {
const seq = ++callSequence;
const start = performance.now();
console.log(`[rpc #${seq}] ▶ ${name} called — args: ${JSON.stringify(args)}`);

const result = await fn(args);

const elapsed = (performance.now() - start).toFixed(0);
const resultSize = JSON.stringify(result).length;
console.log(
`[rpc #${seq}] ◀ ${name} done — ${elapsed}ms (${resultSize} bytes returned to model)`,
);
return result;
};
}

const rpcTools: RealtimeSessionCreateParams['tools'] = [
{
type: 'backend_rpc',
name: 'get_scores',
description:
'Get current/recent game scores for a league. Returns game matchups, scores, and status.',
parameters: [
{ name: 'league', type: 'string', description: 'League abbreviation: NFL, NBA, MLB, NHL, or MLS' },
],
timeoutSeconds: 8,
},
{
type: 'backend_rpc',
name: 'get_standings',
description: 'Get league standings with wins and losses by division/conference.',
parameters: [
{ name: 'league', type: 'string', description: 'League abbreviation: NFL, NBA, MLB, NHL, or MLS' },
],
timeoutSeconds: 8,
},
{
type: 'backend_rpc',
name: 'get_news',
description: 'Get the latest 3 news headlines for a league.',
parameters: [
{ name: 'league', type: 'string', description: 'League abbreviation: NFL, NBA, MLB, NHL, or MLS' },
],
timeoutSeconds: 8,
},
{
type: 'backend_rpc',
name: 'get_leaders',
description: 'Get statistical leaders for a league (top scorers, passers, etc.).',
parameters: [
{ name: 'league', type: 'string', description: 'League abbreviation: NFL, NBA, MLB, NHL, or MLS' },
{ name: 'season', type: 'string', description: 'Season year (e.g. "2025"). Defaults to current season.' },
],
timeoutSeconds: 8,
},
];

export async function POST(req: Request) {
try {
const { avatarId } = (await req.json()) as { avatarId: string };

const { id: sessionId } = await client.realtimeSessions.create({
model: 'gwm1_avatars',
avatar: { type: 'custom', avatarId },
tools: rpcTools,
personality: SPORTS_PERSONALITY,
startScript: SPORTS_START_SCRIPT,
});

const session = await pollSessionUntilReady(sessionId);

const handler = await createRpcHandler({
apiKey: process.env.RUNWAYML_API_SECRET!,
sessionId,
tools: {
get_scores: timed('get_scores', async (args) => {
const league = typeof args.league === 'string' ? args.league : 'nba';
return getScores(league);
}),
get_standings: timed('get_standings', async (args) => {
const league = typeof args.league === 'string' ? args.league : 'nba';
return getStandings(league);
}),
get_news: timed('get_news', async (args) => {
const league = typeof args.league === 'string' ? args.league : 'nba';
return getNews(league);
}),
get_leaders: timed('get_leaders', async (args) => {
const league = typeof args.league === 'string' ? args.league : 'nba';
const season = typeof args.season === 'string' ? args.season : undefined;
return getLeaders(league, season);
}),
},
onDisconnected: () => activeHandlers.delete(sessionId),
onError: (error: Error) => console.error('[rpc] Handler error:', error.message),
});

activeHandlers.set(sessionId, handler);

return Response.json({ sessionId, sessionKey: session.sessionKey });
} catch (error) {
console.error('[connect] Failed:', error);
const message = error instanceof Error ? error.message : 'Unknown error';
return Response.json({ error: message }, { status: 500 });
}
}

async function pollSessionUntilReady(sessionId: string) {
const TIMEOUT_MS = 30_000;
const POLL_INTERVAL_MS = 1_000;
const deadline = Date.now() + TIMEOUT_MS;

while (Date.now() < deadline) {
const session = await client.realtimeSessions.retrieve(sessionId);

if (session.status === 'READY') return session;

if (session.status === 'COMPLETED' || session.status === 'FAILED' || session.status === 'CANCELLED') {
throw new Error(`Session ${session.status.toLowerCase()} before becoming ready`);
}

await new Promise((resolve) => setTimeout(resolve, POLL_INTERVAL_MS));
}

throw new Error('Session creation timed out');
}
92 changes: 92 additions & 0 deletions examples/nextjs-rpc-external-api/app/globals.css
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
* {
box-sizing: border-box;
margin: 0;
padding: 0;
}

:root {
--foreground: #ededed;
--background: #0a0a0a;
--muted: #a1a1aa;
--border: rgba(255, 255, 255, 0.1);
--accent: #38bdf8;
--radius: 12px;
}

html,
body {
height: 100%;
}

body {
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
background: var(--background);
color: var(--foreground);
line-height: 1.5;
-webkit-font-smoothing: antialiased;
}

.page {
min-height: 100vh;
min-height: 100dvh;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
padding: 24px;
}

.page-call {
padding: 0;
}

.hero {
display: flex;
flex-direction: column;
align-items: center;
gap: 12px;
text-align: center;
}

.hero-emoji {
font-size: 56px;
line-height: 1;
}

.title {
font-size: 36px;
font-weight: 700;
letter-spacing: -0.03em;
}

.subtitle {
color: var(--muted);
font-size: 16px;
max-width: 420px;
}

.connect-button {
margin-top: 8px;
padding: 14px 36px;
font-size: 16px;
font-weight: 600;
border: none;
border-radius: var(--radius);
background: var(--accent);
color: #fff;
cursor: pointer;
transition: opacity 0.15s;
}

.connect-button:hover:not(:disabled) {
opacity: 0.85;
}

.connect-button:disabled {
opacity: 0.4;
cursor: not-allowed;
}

.loading {
color: var(--muted);
}
19 changes: 19 additions & 0 deletions examples/nextjs-rpc-external-api/app/layout.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import type { Metadata } from 'next';
import './globals.css';

export const metadata: Metadata = {
title: 'External API Calling — Backend RPC',
description: 'AI avatar that calls external APIs using backend RPC tool calls',
};

export default function RootLayout({
children,
}: {
children: React.ReactNode;
}) {
return (
<html lang="en">
<body>{children}</body>
</html>
);
}
Loading
Loading