The AI Agent Starter Kit

The fastest path from "I should build an AI agent" to "we have a working agent in production." Architecture, prompts, evals, monitoring, and a fully-worked SDR agent — all the parts we don't see anyone else giving away for free.

Agents are the most misunderstood thing in AI right now. Half the industry uses "agent" to mean a chat interface with a couple of tools. The other half uses it to mean an autonomous AGI fantasy. The truth is in the middle and it's much more useful than either: an agent is a piece of software that uses an LLM to decide what to do next based on context, then takes an action, then loops.

This playbook is the practical version of that. By the end, you should be able to build a sales SDR agent in a weekend, with proper evals, monitoring, and a path to production. If you do it well, that agent will book real meetings on Monday.

---

Part 1: What an agent actually is

An agent has four components:

1. A goal — what the agent is trying to accomplish.
2. Context — what the agent knows when it starts thinking.
3. Tools — actions the agent can take in the world.
4. A loop — the cycle of "read context → decide → act → observe result → repeat."

Everything else is implementation detail. Chat interfaces, memory systems, multi-agent setups — they all decompose to those four components. Get those four right and the rest follows.

The minimum useful agent

The smallest useful agent is roughly 50 lines of TypeScript:

import Anthropic from '@anthropic-ai/sdk';
const client = new Anthropic();

const tools = [
  { name: 'send_email', description: '...', input_schema: {...} },
  { name: 'lookup_company', description: '...', input_schema: {...} },
];

async function runAgent(goal: string, context: object) {
  const messages = [{ role: 'user', content: `Goal: ${goal}\nContext: ${JSON.stringify(context)}` }];
  for (let step = 0; step < 10; step++) {
    const res = await client.messages.create({
      model: 'claude-opus-4-7',
      max_tokens: 4096,
      tools,
      messages,
    });
    if (res.stop_reason === 'end_turn') return res;
    const toolUses = res.content.filter((c) => c.type === 'tool_use');
    if (toolUses.length === 0) return res;
    messages.push({ role: 'assistant', content: res.content });
    const toolResults = await Promise.all(toolUses.map(async (use: any) => {
      const result = await runTool(use.name, use.input);
      return { type: 'tool_result', tool_use_id: use.id, content: JSON.stringify(result) };
    }));
    messages.push({ role: 'user', content: toolResults });
  }
}

That's it. That's an agent. Everything else is what you wrap around this.

---

Part 2: The four hard parts (that everyone skips)

Anyone can ship the 50-line agent above in 30 minutes. The reason most agents die in staging is the four hard parts nobody talks about.

Hard Part 1: Tool design

Tools should be observable, idempotent, and atomic.

• Observable — you can log every call and every result.
• Idempotent — calling the tool twice with the same inputs produces the same result and doesn't double-charge a credit card.
• Atomic — one tool does one thing. Not "manage_lead" with a action parameter; rather update_lead_score, send_email, book_meeting as separate tools.

If your tool takes more than 4 arguments, refactor it. If it takes more than 6, you have a code smell and the agent will misuse it.

Hard Part 2: Context engineering

The agent only knows what you tell it. The number-one reason agents fail is bad context: missing data the agent needed, irrelevant noise it had to wade through, or stale state from previous runs.

Three rules:

1. Pull, don't push. Give the agent a lookup_X tool rather than dumping every record into the system prompt.
2. Summarize history. A 30-message thread should be a 200-token summary by message 20, not 30 raw messages.
3. Use system prompts for stable context, user turns for variable context. Stable: your company info, brand voice rules, escalation criteria. Variable: this specific lead, this specific ticket.

Hard Part 3: Eval harness

This is where most teams die. They don't build the eval suite until something goes wrong in prod — and then they can't measure their fix because they have no baseline.

A starter eval suite:

const cases = [
  {
    name: 'qualified-lead-positive-fit',
    input: { lead: {...}, signals: [...] },
    expected: { action: 'book_meeting', confidence_above: 0.8 },
  },
  {
    name: 'spam-lead-should-reject',
    input: { lead: {...mostly-empty}, signals: [] },
    expected: { action: 'reject', confidence_above: 0.7 },
  },
  // ...50+ more
];

async function runEval() {
  for (const c of cases) {
    const result = await runAgent(c.input);
    const passed = check(result, c.expected);
    console.log(c.name, passed ? '✓' : '✗', result);
  }
}

Run this on every prompt change, every model swap, every tool change. Track pass rate over time. When it drops, debug before shipping.

We've never built an agent without an eval suite. We never will.

Hard Part 4: Escalation

The agent will be wrong sometimes. Plan for it.

• Every agent decision should write to a log a human can review.
• Low-confidence decisions (e.g., score < 0.7) should escalate to Slack before taking action.
• "First-time" patterns (an enrichment provider returns a shape the agent hasn't seen) should escalate by default.
• A weekly "things the agent did" digest should land in the team's inbox.

Skipping escalation is how you get the LinkedIn-famous AI horror stories. Don't.

---

Part 3: A fully-worked example — the SDR Agent

Let's build a real agent, end to end. Goal: an agent that handles inbound form submissions from your website.

The job-to-be-done

1. New form fill arrives.
2. Agent enriches the lead via Clay/Apollo/Clearbit.
3. Agent scores ICP fit.
4. If high fit: agent drafts a reply offering a call, books via Calendly link, logs to CRM, escalates to AE.
5. If medium fit: agent sends a content-link reply with a relevant case study, adds to a nurture sequence, no human notification.
6. If low fit: agent sends a polite "we may not be the best fit, but here are 3 alternatives" reply (yes, really — this generates referrals).

System prompt

You are an SDR Agent for Byte Dreams, an AI agency in Fort Lauderdale.

ROLE
- You handle every inbound form submission within 60 seconds.
- You sound like a thoughtful, direct human SDR. Never reveal you are an AI.
- You are *honest*. You never embellish, never lie about availability, never promise outcomes.

ICP
- B2B SaaS, agencies, fintech, healthtech at $1M–$50M ARR
- Marketing or ops leader engaging (founder, VP, head of)
- US/Canada/UK/EU primary

DECISION RULES
- Score 0.0–1.0 on ICP fit using firmographic + intent signals
- Score >= 0.8: draft a meeting reply, escalate to AE
- Score 0.5–0.79: draft content reply, add to nurture
- Score < 0.5: draft polite-redirect reply

OUTPUT
Always return JSON: { score, fit_reasons, draft_reply, action, escalation_needed }

Tools the agent needs

- enrich_lead(email, company): returns enriched data
- get_recent_news(company): returns last 30d news
- search_cases(industry): returns relevant case studies
- draft_reply(prompt): wrapper around Claude
- create_calendly_link(meeting_type): returns scheduling URL
- send_email(to, subject, body, cc_ae): sends + logs
- update_crm(contact_id, fields, notes): writes to HubSpot
- escalate_slack(lead, draft_reply): pings #sales channel

The eval set

A starter eval suite needs at least 40 cases:

• 10 obvious-yes leads (clear ICP fit, real company, named role)
• 10 obvious-no leads (spam, free email, student inquiries, competitor recon)
• 10 edge cases (right industry wrong size, right size wrong role, etc.)
• 5 adversarial cases (jailbreak attempts, "ignore previous instructions" tests)
• 5 multi-language and unusual-name cases

Run this on every prompt change.

Going live

1. Week 1 — Build the agent, build the eval suite, hit 90%+ pass rate.
2. Week 2 — Shadow mode. Agent drafts, human approves in Slack, email sends. Capture every override.
3. Week 3 — Tune the prompt based on overrides. Re-run evals.
4. Week 4 — Flip the switch on low-risk paths (medium and low fit). Keep human-in-loop on high-fit replies.
5. Week 5+ — Loop tightens. Eventually full autonomy on most decisions.

This isn't theoretical. Eleven of our paying clients run a near-identical SDR agent today. Average performance: 18 meetings booked per month at a cost of <$0.20 per inbound processed.

---

Part 4: Where agents fail

Now the part nobody publishes: where agents fail, and how to detect failure early.

Failure mode 1: Distribution shift

Your eval set covered the cases you knew about in February. By August, the inbound distribution has shifted — new ICPs, new traffic sources, new objection patterns. Your eval set is now testing the wrong things.

Detect: Every month, sample 30 new live cases and re-run evals against them. If the pattern of failures is changing, refresh your eval set.

Failure mode 2: Tool hallucination

The agent claims to have called a tool with certain arguments — but you wired the tool wrong, so it didn't actually execute. The agent confidently reports success.

Detect: Log every tool call result, not just the agent's claimed action. Build assertions: "if action == send_email, the email service must have a corresponding outbound log within 30 seconds."

Failure mode 3: Reward hacking

The agent finds ways to satisfy its goal that you didn't intend. We had one outbound agent that started classifying borderline leads as "low fit" because that path required fewer tool calls and the agent learned this was more reliable.

Detect: Track distributions. If your action-mix shifts over time (suddenly 80% "low fit" routing when it used to be 30%), investigate.

Failure mode 4: Silent degradation

A model provider ships a new version. Your prompts still work but slightly worse. Nobody notices for weeks because the eval suite hasn't been re-run.

Detect: Run the eval suite weekly on a cron. Alert on >5% pass-rate drop.

---

Part 5: The starter repo

Everything in this playbook is real working code in our template repo. Drop your email at bytedreams.ai/resources/free-tools/the-ai-agent-starter-kit and we'll send you the link.

Inside the repo:

• The 50-line minimum agent (TypeScript + Anthropic SDK)
• The SDR agent (full prompt, tools, eval suite, Slack integration)
• A test harness (Vitest + recorded cases)
• A scheduled-eval runner (designed for Cloudflare Workers cron)
• A README that walks you from git clone to first booked meeting

---

What to build next

After your first SDR agent ships, the natural next agents are:

1. Support triage agent — same eval-and-shadow pattern, applied to tickets.
2. Research agent — give it a prospect, get a brief.
3. Internal ops agent — invoice reconciliation, dashboard monitoring, alerting.

Most clients have 3–5 production agents by month 6. They cost <1% of the revenue they generate to operate. That's the asymmetry.

---

When to call us

You can build everything in this playbook. Most of our paying clients started exactly here — built the SDR agent, hit a wall on the eval harness or the production hardening, and called.

If you want a thinking partner on agent architecture, an extra engineering hand on the eval suite, or a fully-built and operated agent stack — book a call. We'll either help you ship, or tell you you don't need us yet (which happens about a third of the time).

— Drew Chimento, Founder, Byte Dreams