AI Town 🏠💻💌

Live Demo

Join our community Discord: AI Stack Devs

AI Town is a virtual town where AI characters live, chat and socialize.

This project is a deployable starter kit for easily building and customizing your own version of AI town. Inspired by the research paper Generative Agents: Interactive Simulacra of Human Behavior.

The primary goal of this project, beyond just being a lot of fun to work on, is to provide a platform with a strong foundation that is meant to be extended. The back-end natively supports shared global state, transactions, and a simulation engine and should be suitable from everything from a simple project to play around with to a scalable, multi-player game. A secondary goal is to make a JS/TS framework available as most simulators in this space (including the original paper above) are written in Python.

Overview

• 💻 Stack
• 🧠 Installation (cloud, local, Docker, self-host, Fly.io, ...)
• 💻️ Windows Pre-requisites
• 🤖 Configure your LLM of choice (Ollama, OpenAI, Together.ai, ...)
• 👤 Customize - YOUR OWN simulated world
• 👩‍💻 Deploying to production
• 🐛 Troubleshooting

Stack

• Game engine, database, and vector search: Convex
• Auth (Optional): Clerk
• Default chat model is llama3 and embeddings with mxbai-embed-large.
• Local inference: Ollama
• Configurable for other cloud LLMs: Together.ai or anything that speaks the OpenAI API. PRs welcome to add more cloud provider support.
• Background Music Generation: Replicate using MusicGen

Other credits:

• Pixel Art Generation: Replicate, Fal.ai
• All interactions, background music and rendering on the component in the project are powered by PixiJS.
• Tilesheet:
  • https://opengameart.org/content/16x16-game-assets by George Bailey
  • https://opengameart.org/content/16x16-rpg-tileset by hilau
• We used https://github.com/pierpo/phaser3-simple-rpg for the original POC of this project. We have since re-wrote the whole app, but appreciated the easy starting point
• Original assets by ansimuz
• The UI is based on original assets by Mounir Tohami

Installation

The overall steps are:

1. Build and deploy
2. Connect it to an LLM

Build and Deploy

There are a few ways to run the app on top of Convex (the backend).

1. The standard Convex setup, where you develop locally or in the cloud. This requires a Convex account(free). This is the easiest way to depoy it to the cloud and seriously develop.
2. If you want to try it out without an account and you're okay with Docker, the Docker Compose setup is nice and self-contained.
3. There's a community fork of this project offering a one-click install on Pinokio for anyone interested in running but not modifying it 😎.
4. You can also deploy it to Fly.io. See ./fly for instructions.

Standard Setup

Note, if you're on Windows, see below.

git clone https://github.com/a16z-infra/ai-town.git
cd ai-town
npm install

This will require logging into your Convex account, if you haven't already.

To run it:

npm run dev

You can now visit http://localhost:5173.

If you'd rather run the frontend and backend separately (which syncs your backend functions as they're saved), you can run these in two terminals:

npm run dev:frontend
npm run dev:backend

See package.json for details.

Using Docker Compose with self-hosted Convex

You can also run the Convex backend with the self-hosted Docker container. Here we'll set it up to run the frontend, backend, and dashboard all via docker compose.

docker compose up --build -d

The container will keep running in the background if you pass -d. After you've done it once, you can stop and start services.

• The frontend will be running on http://localhost:5173.
• The backend will be running on http://localhost:3210 (3211 for the http api).
• The dashboard will be running on http://localhost:6791.

To log into the dashboard and deploy from the convex CLI, you will need to generate an admin key.

docker compose exec backend ./generate_admin_key.sh

Add it to your .env.local file. Note: If you run down and up, you'll have to generate the key again and update the .env.local file.

# in .env.local
CONVEX_SELF_HOST_ADMIN_KEY="<admin-key>" # Ensure there are quotes around it
CONVEX_SELF_HOST_URL="http://127.0.0.1:3210"

Then set up the Convex backend (one time):

npx convex self-host dev --run init --until-success

To continuously deploy new code to the backend and print logs:

npx convex self-host dev --tail-logs

To see the dashboard, visit http://localhost:6791 and provide the admin key you generated earlier.

Configuring Docker for Ollama

If you'll be using Ollama for local inference, you'll need to configure Docker to connect to it.

npx convex self-host env set OLLAMA_HOST http://host.docker.internal:11434

To test the connection (after you have it running):

docker compose exec backend /bin/bash curl http://host.docker.internal:11434

If it says "Ollama is running", it's good! Otherwise, check out the Troubleshooting section.

Connect an LLM

Note: If you want to run the backend in the cloud, you can either use a cloud-based LLM API, like OpenAI or Together.ai or you can proxy the traffic from the cloud to your local Ollama. See below for instructions.

Ollama (default)

By default, the app tries to use Ollama to run it entirely locally.

1. Download and install Ollama.
2. Open the app or run ollama serve in a terminal. ollama serve will warn you if the app is already running.
3. Run ollama pull llama3 to have it download llama3.
4. Test it out with ollama run llama3.

Ollama model options can be found here.

If you want to customize which model to use, adjust convex/util/llm.ts or set npx convex env set OLLAMA_MODEL # model. If you want to edit the embedding model:

1. Change the OLLAMA_EMBEDDING_DIMENSION in convex/util/llm.ts and ensure: export const EMBEDDING_DIMENSION = OLLAMA_EMBEDDING_DIMENSION;
2. Set npx convex env set OLLAMA_EMBEDDING_MODEL # model.

Note: You might want to set NUM_MEMORIES_TO_SEARCH to 1 in constants.ts, to reduce the size of conversation prompts, if you see slowness.

OpenAI

To use OpenAI, you need to:

// In convex/util/llm.ts change the following line:
export const EMBEDDING_DIMENSION = OPENAI_EMBEDDING_DIMENSION;

Set the OPENAI_API_KEY environment variable. Visit https://platform.openai.com/account/api-keys if you don't have one.

npx convex env set OPENAI_API_KEY 'your-key'

Optional: choose models with OPENAI_CHAT_MODEL and OPENAI_EMBEDDING_MODEL.

Together.ai

To use Together.ai, you need to:

// In convex/util/llm.ts change the following line:
export const EMBEDDING_DIMENSION = TOGETHER_EMBEDDING_DIMENSION;

Set the TOGETHER_API_KEY environment variable. Visit https://api.together.xyz/settings/api-keys if you don't have one.

npx convex env set TOGETHER_API_KEY 'your-key'

Optional: choose models via TOGETHER_CHAT_MODEL, TOGETHER_EMBEDDING_MODEL. The embedding model's dimension must match EMBEDDING_DIMENSION.

Other OpenAI-compatible API

You can use any OpenAI-compatible API, such as Anthropic, Groq, or Azure.

• Change the EMBEDDING_DIMENSION in convex/util/llm.ts to match the dimension of your embedding model.
• Edit getLLMConfig in llm.ts or set environment variables:

npx convex env set LLM_API_URL 'your-url'
npx convex env set LLM_API_KEY 'your-key'
npx convex env set LLM_MODEL 'your-chat-model'
npx convex env set LLM_EMBEDDING_MODEL 'your-embedding-model'

Note: if LLM_API_KEY is not required, don't set it.

Note on changing the LLM provider or embedding model:

If you change the LLM provider or embedding model, you should delete your data and start over. The embeddings used for memory are based on the embedding model you choose, and the dimension of the vector database must match the embedding model's dimension. See below for how to do that.

Customize your own simulation

NOTE: every time you change character data, you should re-run npx convex run testing:wipeAllTables and then npm run dev to re-upload everything to Convex. This is because character data is sent to Convex on the initial load. However, beware that npx convex run testing:wipeAllTables WILL wipe all of your data.

1. Create your own characters and stories: All characters and stories, as well as their spritesheet references are stored in characters.ts. You can start by changing character descriptions.

2. Updating spritesheets: in data/characters.ts, you will see this code:

   export const characters = [
     {
       name: 'f1',
       textureUrl: '/assets/32x32folk.png',
       spritesheetData: f1SpritesheetData,
       speed: 0.1,
     },
     ...
   ];

   You should find a sprite sheet for your character, and define sprite motion / assets in the corresponding file (in the above example, f1SpritesheetData was defined in f1.ts)

3. Update the Background (Environment): The map gets loaded in convex/init.ts from data/gentle.js. To update the map, follow these steps:

   • Use Tiled to export tilemaps as a JSON file (2 layers named bgtiles and objmap)
   • Use the convertMap.js script to convert the JSON to a format that the engine can use.

   node data/convertMap.js <mapDataPath> <assetPath> <tilesetpxw> <tilesetpxh>

   • <mapDataPath>: Path to the Tiled JSON file.
   • <assetPath>: Path to tileset images.
   • <tilesetpxw>: Tileset width in pixels.
   • <tilesetpxh>: Tileset height in pixels. Generates converted-map.js that you can use like gentle.js

4. Adding background music with Replicate (Optional)

   For Daily background music generation, create a Replicate account and create a token in your Profile's API Token page. npx convex env set REPLICATE_API_TOKEN # token

   This only works if you can receive the webhook from Replicate. If it's running in the normal Convex cloud, it will work by default. If you're self-hosting, you'll need to configure it to hit your app's url on /http. If you're using Docker Compose, it will be http://localhost:3211, but you'll need to proxy the traffic to your local machine.

   Note: The simulation will pause after 5 minutes if the window is idle. Loading the page will unpause it. You can also manually freeze & unfreeze the world with a button in the UI. If you want to run the world without the browser, you can comment-out the "stop inactive worlds" cron in convex/crons.ts.

   • Change the background music by modifying the prompt in convex/music.ts
   • Change how often to generate new music at convex/crons.ts by modifying the generate new background music job

Commands to run / test / debug

To stop the back end, in case of too much activity

This will stop running the engine and agents. You can still run queries and run functions to debug.

npx convex run testing:stop

To restart the back end after stopping it

npx convex run testing:resume

To kick the engine in case the game engine or agents aren't running

npx convex run testing:kick

To archive the world

If you'd like to reset the world and start from scratch, you can archive the current world:

npx convex run testing:archive

Then, you can still look at the world's data in the dashboard, but the engine and agents will no longer run.

You can then create a fresh world with init.

npx convex run init

To pause your backend deployment

You can go to the dashboard to your deployment settings to pause and un-pause your deployment. This will stop all functions, whether invoked from the client, scheduled, or as a cron job. See this as a last resort, as there are gentler ways of stopping above.

Windows Installation

Prerequisites

1. Windows 10/11 with WSL2 installed
2. Internet connection

Steps:

1. Install WSL2

   First, you need to install WSL2. Follow this guide to set up WSL2 on your Windows machine. We recommend using Ubuntu as your Linux distribution.

2. Update Packages

   Open your WSL terminal (Ubuntu) and update your packages:

   sudo apt update

3. Install NVM and Node.js

   NVM (Node Version Manager) helps manage multiple versions of Node.js. Install NVM and Node.js 18 (the stable version):

   curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.2/install.sh | bash
   export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")"
   [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
   source ~/.bashrc
   nvm install 18
   nvm use 18

4. Install Python and Pip

   Python is required for some dependencies. Install Python and Pip:

   sudo apt-get install python3 python3-pip sudo ln -s /usr/bin/python3 /usr/bin/python

At this point, you can follow the instructions above.

Deploy the app to production

Deploy Convex functions to prod environment

Before you can run the app, you will need to make sure the Convex functions are deployed to its production environment. Note: this is assuming you're using the default Convex cloud product.

1. Run npx convex deploy to deploy the convex functions to production
2. Run npx convex run init --prod

To transfer your local data to the cloud, you can run npx convex export and then import it with npx convex import --prod.

If you have existing data you want to clear, you can run npx convex run testing:wipeAllTables --prod

Adding Auth (Optional)

You can add clerk auth back in with git revert b44a436. Or just look at that diff for what changed to remove it.

Make a Clerk account

• Go to https://dashboard.clerk.com/ and click on "Add Application"
• Name your application and select the sign-in providers you would like to offer users
• Create Application
• Add VITE_CLERK_PUBLISHABLE_KEY and CLERK_SECRET_KEY to .env.local

VITE_CLERK_PUBLISHABLE_KEY=pk_***
CLERK_SECRET_KEY=sk_***

• Go to JWT Templates and create a new Convex Template.
• Copy the JWKS endpoint URL for use below.

npx convex env set CLERK_ISSUER_URL # e.g. https://your-issuer-url.clerk.accounts.dev/

Deploy the frontend to Vercel

• Register an account on Vercel and then install the Vercel CLI.
• If you are using Github Codespaces: You will need to install the Vercel CLI and authenticate from your codespaces cli by running vercel login.
• Deploy the app to Vercel with vercel --prod.

Using local inference from a cloud deployment

We support using Ollama for conversation generations. To have it accessible from the web, you can use Tunnelmole or Ngrok or similar so the cloud backend can send requests to Ollama running on your local machine.

Steps:

1. Set up either Tunnelmole or Ngrok.
2. Add Ollama endpoint to Convex

   npx convex env set OLLAMA_HOST # your tunnelmole/ngrok unique url from the previous step

3. Update Ollama domains Ollama has a list of accepted domains. Add the ngrok domain so it won't reject traffic. see ollama.ai for more details.

Using Tunnelmole

Tunnelmole is an open source tunneling tool.

You can install Tunnelmole using one of the following options:

• NPM: npm install -g tunnelmole
• Linux: curl -s https://tunnelmole.com/sh/install-linux.sh | sudo bash
• Mac: curl -s https://tunnelmole.com/sh/install-mac.sh --output install-mac.sh && sudo bash install-mac.sh
• Windows: Install with NPM, or if you don't have NodeJS installed, download the exe file for Windows here and put it somewhere in your PATH.

Once Tunnelmole is installed, run the following command:

tmole 11434

Tunnelmole should output a unique url once you run this command.

Using Ngrok

Ngrok is a popular closed source tunneling tool.

• Install Ngrok

Once ngrok is installed and authenticated, run the following command:

ngrok http http://localhost:11434

Ngrok should output a unique url once you run this command.

Troubleshooting

Wiping the database and starting over

You can wipe the database by running:

npx convex run testing:wipeAllTables

Then reset with:

npx convex run init

Incompatible Node.js versions

If you encounter a node version error on the convex server upon application startup, please use node version 18, which is the most stable. One way to do this is by installing nvm and running nvm install 18 and nvm use 18.

Reaching Ollama

If you're having trouble with the backend communicating with Ollama, it depends on your setup how to debug:

1. If you're running directly on Windows, see Windows Ollama connection issues.
2. If you're using Docker, see Docker to Ollama connection issues.
3. If you're running locally, you can try the following:

npx convex env set OLLAMA_HOST http://localhost:11434

By default, the host is set to http://127.0.0.1:11434. Some systems prefer localhost ¯_(ツ)_/¯.

Windows Ollama connection issues

If the above didn't work after following the windows and regular installation instructions, you can try the following, assuming you're not using Docker.

If you're using Docker, see the next section for Docker troubleshooting.

For running directly on Windows, you can try the following:

1. Install unzip and socat:

   sudo apt install unzip socat

2. Configure socat to Bridge Ports for Ollama

   Run the following command to bridge ports:

   socat TCP-LISTEN:11434,fork TCP:$(cat /etc/resolv.conf | grep nameserver | awk '{print $2}'):11434 &

3. Test if it's working:

   curl http://127.0.0.1:11434

   If it responds OK, the Ollama API should be accessible.

Docker to Ollama connection issues

If you're having trouble with the backend communicating with Ollama, there's a couple things to check:

1. Is Docker at least verion 18.03 ? That allows you to use the host.docker.internal hostname to connect to the host from inside the container.

2. Is Ollama running? You can check this by running curl http://localhost:11434 from outside the container.

3. Is Ollama accessible from inside the container? You can check this by running docker compose exec backend curl http://host.docker.internal:11434.

If 1 & 2 work, but 3 does not, you can use socat to bridge the traffic from inside the container to Ollama running on the host.

1. Configure socat with the host's IP address (not the Docker IP).

   docker compose exec backend /bin/bash
   HOST_IP=YOUR-HOST-IP
   socat TCP-LISTEN:11434,fork TCP:$HOST_IP:11434

   Keep this running.

2. Then from outside of the container:

   npx convex env set OLLAMA_HOST http://localhost:11434

3. Test if it's working:

   docker compose exec backend curl http://localhost:11434

   If it responds OK, the Ollama API is accessible. Otherwise, try changing the previous two to http://127.0.0.1:11434.

Launching an Interactive Docker Terminal

If you wan to investigate inside the container, you can launch an interactive Docker terminal, for the frontend, backend or dashboard service:

docker compose exec frontend /bin/bash

To exit the container, run exit.

Updating the browser list

docker compose exec frontend npx update-browserslist-db@latest

🧑‍🏫 What is Convex?

Convex is a hosted backend platform with a built-in database that lets you write your database schema and server functions in TypeScript. Server-side database queries automatically cache and subscribe to data, powering a realtime useQuery hook in our React client. There are also clients for Python, Rust, ReactNative, and Node, as well as a straightforward HTTP API.

The database supports NoSQL-style documents with opt-in schema validation, relationships and custom indexes (including on fields in nested objects).

The query and mutation server functions have transactional, low latency access to the database and leverage our v8 runtime with determinism guardrails to provide the strongest ACID guarantees on the market: immediate consistency, serializable isolation, and automatic conflict resolution via optimistic multi-version concurrency control (OCC / MVCC).

The action server functions have access to external APIs and enable other side-effects and non-determinism in either our optimized v8 runtime or a more flexible node runtime.

Functions can run in the background via scheduling and cron jobs.

Development is cloud-first, with hot reloads for server function editing via the CLI, preview deployments, logging and exception reporting integrations, There is a dashboard UI to browse and edit data, edit environment variables, view logs, run server functions, and more.

There are built-in features for reactive pagination, file storage, reactive text search, vector search, https endpoints (for webhooks), snapshot import/export, streaming import/export, and runtime validation for function arguments and database data.

Everything scales automatically, and it’s free to start.