Create an AI Assistant With Its Own Phone Number (Eleven Labs Walkthrough)

In this guide, you’ll build an AI personal assistant that you can call from your phone. It will know your calendar, your business, and any additional context you add to its knowledge base. The setup takes about 10 minutes and costs $0 to test.

  • Intermediate
  • 4 steps
  • By The Rundown University

Rundown Pro

Every course and recording, the certification program and member perks, for $29 a month. Guides stay free.

Go Pro for $29

Step by step 4 steps

  1. Step 1 of 4

    Create your agent in ElevenLabs

    Sign in to ElevenLabs and go to the Agents section.

    • Click New Agent.
    • Select Personal Assistant as the template.
    • Name it whatever you want.

    Pick a voice. Browse the voice library, filter by language, gender, and age, preview a few options, and choose one that fits.

    Pro tip: If the default voice speed feels slow, bump it up slightly in the voice settings.

    Review the system prompt. The personal assistant template comes with a solid default. Update your time zone and customize the first message to your liking.

    Add to the knowledge base. This is what makes your assistant actually useful:

    • Paste a URL to your website or product page so the agent can scrape it.
    • Upload a text or markdown file of your calendar. You can ask Claude or ChatGPT to format your Google Calendar as markdown.
    • Add any other static information, such as client lists, project notes, or your services.

    Hit Publish when you’re done.

    image-2079
    Pro tip: The knowledge base is static, so it is best for things that do not change often, such as your offerings, brand info, and current projects. For dynamic data like live calendar events, use the webhook approach in the Going Further section.

  2. Step 2 of 4

    Set up Twilio and get a phone number

    Go to twilio.com and create a new account. Twilio gives you $15 in free credits. A phone number costs about $1 to buy and $1/month to keep.

    image-2080
    • Click Phone Numbers → Buy a Number.
    • Pick one with voice capability. SMS requires extra paperwork and will not be same-day.
    • Copy down three things from your Twilio dashboard: the phone number you just bought, the Account SID from the main dashboard, and the Auth Token below the SID.
    image-2081
    Pro tip: During the free trial, callers will hear a short Twilio message before connecting. It goes away once you upgrade to pay-as-you-go, which is fractions of a cent per minute.

  3. Step 3 of 4

    Connect Twilio to ElevenLabs

    Go back to ElevenLabs. In the sidebar, find Phone Numbers and click Import Number from Twilio.

    • Paste your Account SID.
    • Paste your Auth Token.
    • Paste your Twilio phone number.
    • Name the number and assign your agent to it.
    image-2082

    That’s it. Your agent now has a real phone number. Call it and start talking. On a free Twilio account, it will make you hit the keypad to connect to your agent. If you upgrade, this will go away.

  4. Step 4 of 4

    Extend your assistant with tools and integrations

    Once the basic assistant is working, you can make it much more powerful by adding tools.

    Here are the system tools you can install:

    image-2083

    You can also add integrations from your agent by going to Add Tool → Integration. Your options include:

    • Cal.com for live calendar access.
    • HubSpot for CRM lookups.
    • Custom MCP server for advanced integrations.
    • Webhooks that connect to Make.com, n8n, or Zapier.

    With webhooks, your assistant can pull live calendar events, schedule meetings, check your email, or look up client info, all triggered by a phone call. If you’ve taken any of the Rundown automation workshops, this is a natural next step.

That's the whole workflow

Mark it complete to keep your progress. The background follows.

Go further with ElevenLabs

This Rundown Pro recording covers the same tool or topic as this guide.

$29/mo, cancel anytime

Background and context

Who This Is Useful For

  • Busy founders or operators who want a hands-free way to check their schedule.
  • Consultants juggling multiple clients who need quick access to project information on the go.
  • Anyone who wants to experiment with voice AI agents without writing code.

What You Will Build

You’ll create a personal AI phone assistant powered by ElevenLabs that answers calls, uses your calendar and business context, and can later be extended with tools like webhooks and automations.

  • An ElevenLabs personal assistant agent.
  • A Twilio phone number connected to the agent.
  • A knowledge base with your calendar, business page, or other static context.
  • An optional path for adding live integrations later.

What You Need

  • ElevenLabs account. The free plan works.
  • Twilio account. It is free to sign up and includes $15 in credits.
  • A calendar export or text file with your schedule.
  • Optional: a URL to your business or product page.

Going Further

  • After the basic assistant is working, you can add tools and integrations for live calendar access, CRM lookups, advanced integrations, or webhooks connected to Make.com, n8n, or Zapier. With webhooks, your assistant can pull live calendar events, schedule meetings, check email, or look up client information from a phone call.