🔍 What 4 Billion Tokens Actually Bought Me

I've spent over ten billion tokens building with AI this summer.

One of the most useful lessons came from something as specific as a podcast's volume.

Before we do a deep dive, I want to invite you to a free session this upcoming Thursday…

Let me ask you this:

What if editing your podcast took minutes instead of days? What would you do with that free time?

This Thursday, October 15, my friend Chris Harder and I are going to show you what's possible when AI does the heavy lifting. 

We'll take a raw audio recording and turn it into a polished, ready-to-publish podcast episode, right in front of you.

Join us live on Zoom, October 15, from 2:00–3:00 PM EDT. 

It's free, and if you have a podcast (or have been thinking about starting one), you'll want to see this.

→ Save your free seat here

Now, on to my 10 billion token lesson…

📐Your AI Defaults When It Should Measure

While building this summer, AI taught me it has a habit of applying the standard answer, even when the right answer is sitting right in front of it.

Let me show you what I mean.

🔎 What I Found

When I started building Casthouse, I wanted the system to take raw podcast audio and produce a finished episode that sounded consistent with everything that show had already published.

Sounds straightforward (and awesome), right?

Except I hit a block on something as basic as audio levels.

There's a measurement called LUFS that tells us how loud audio sounds to the human ear. 

And most podcast production tools work from recommended targets:

• Apple recommends -16 LUFS as a target for podcast loudness.
• The show I was setting up is mastered closer to -20 LUFS, which means the established sound is quieter than the industry average.

Following the recommendation would have broken continuity for the listener.

That means the system could have delivered a technically acceptable episode that didn't match the rest of his show.

That's when I realized the system needed to do something different.

Instead of starting with what a podcast should sound like, it needed to analyze what each podcast already sounded like.

The difference was giving AI a measurable target instead of a generic instruction.

And once I saw that distinction in audio production, I started recognizing it in nearly every AI workflow I was building.

🧠 What It Means for You

Building Casthouse changed how I think about the instructions we give AI.

And more importantly, the way we evaluate what comes back.

I've spent much of this summer figuring out how to build systems that can make hundreds and thousands of micro-decisions without someone supervising every step.

And I've learned that the quality of an AI system depends heavily on how well you've defined what success actually looks like at each step of the process.

Here are four things I'd encourage you to put into practice:

1. Stop treating defaults like requirements.

Every AI model comes with assumptions about how work should be done. 

Some come from its training. Others come from the tools, templates, and instructions wrapped around it.

And honestly, most are perfectly reasonable starting points.

But reasonable and appropriate aren't always the same thing.

Ask AI to write a sales email, and it might produce something with:

• a punchy subject line
• a short opening
• three benefits
• and a call to action

That's a recognizable sales email structure.

But what if your highest-converting emails are long, personal stories?

What if your audience responds better to a straightforward subject line?

What if the format you've spent years developing outperforms conventional advice?

Without that information, AI has no reason to prioritize your approach.

In other words, before you accept a recommendation, find out where it came from. 

Is it based on your preferences and wins, or is the system applying a standard convention?

2. Give AI something real to measure and assess.

This has become one of the most valuable changes I've made to my workflows.

I used to spend a lot of time describing what I wanted.

Now, whenever possible, I give AI the work I’ve done and ask it to investigate.

When I built Casthouse, I didn't want to manually specify every detail that made Chris's podcast sound like Chris's podcast.

I wanted the system to analyze his published episodes and identify those details for itself.

The same principle applies well beyond audio production.

If you're building an AI workflow for sales proposals, give it proposals that you've already closed. 

Let it examine:

• how they're structured
• what information they include
• how pricing is presented
• and what changes across different types of clients

If you're building a content system, give it published work with actual performance data. 

Ask what patterns appear in the pieces that consistently resonate with your audience.

If you're trying to reproduce your writing style, give it enough examples to identify the patterns you may not even realize you follow.

Your existing work is one of your best sources of evidence. Treat it that way.

And here's an important distinction: ask AI to show you what it found.

I want measurements, examples, and references to the original material. 

I want to know whether a pattern appears in 9/10 examples or only 2.

That gives me something to evaluate instead of another AI-generated opinion on what good work should look like.

That's the whole point.

The best use of AI isn't to generate cookie-cutter work that feeds the Dead Internet. 

The best use of AI is to make your unique contribution to the world easier by handling the tedious steps, so you can do more of what you do best.

3. Build verification into the process.

Here's something I've become increasingly convinced of: the longer an AI workflow runs without checking its own work, the more expensive its mistakes can become.

Think about what happens when you have a system performing five consecutive tasks.

The first stage gathers information.
The second analyzes it.
The third makes recommendations.
The fourth produces something based on those recommendations.
The fifth prepares the final deliverable.

What happens if the first stage pulls incorrect information?

Every stage that follows can build on that error. 

And because the final result may look polished and complete, you might not catch the mistake until much later.

I've been designing Casthouse to check the output of each production stage before moving to the next.

Did the audio meet the measured specifications? 

Were the correct assets included? 

Does the resulting episode match the requirements established for that particular show?

If something fails, I want the system to catch it while the problem is still isolated.

The principle is simple: verify the output before you make it someone else's input.

You can apply this even if you're building relatively simple automations.

4. Optimize for work you can trust, not just work you can generate.

Billions and billions of tokens later, and I still haven't written a line of code myself.

That's an extraordinary amount of iteration, testing, analyzing, and rebuilding.

Could I have generated a working prototype with far fewer tokens? 

Absolutely.

But getting something to work once is a very different challenge from building a system that produces reliable results repeatedly.

The first successful output tells you something is possible.

The next hundred tell you whether you've actually built something that works without constant supervision. (Who wants to babysit their AI? That defeats the purpose, doesn’t it?)

Yes, all of this takes more time and more tokens.

I've learned to be willing to spend more on the process when it means I can spend less time fixing the results.

The goal is to build something I'd confidently put my name on, even when I'm not standing there watching it work.

✌️ Your Turn

Let's put this into practice.

Think about something you regularly ask AI to produce. Maybe it's a newsletter, a sales proposal, a client report, or a LinkedIn post.

Find more than five real examples of that work. 

Ideally, choose examples that performed well or that you would happily produce again.

Then upload them to your AI tool and give it this prompt:

I'm giving you ten examples of work I've actually produced.

I want you to reverse-engineer the patterns in these examples and create a measurable specification that I can use to reproduce this type of work consistently.

Analyze the examples themselves rather than relying on conventional best practices.

First, identify the patterns:

• Typical length, including averages and ranges
• Sentence structure, rhythm, and complexity
• How each piece opens, develops, and closes
• Recurring words, phrases, and stylistic choices
• How information is organized and presented
• Differences between examples, including when and how the structure changes
• Characteristics that distinguish this work from conventional examples of the same type

Second, show me the evidence.

For each significant pattern, tell me how frequently it occurs and provide examples from the original material. Separate consistent patterns from occasional choices.

Third, challenge the assumptions.

Identify where my work differs from common recommendations. Explain the trade-offs of those differences, and flag anything that might warrant further investigation.

Don't assume a difference is a mistake. Don't assume it's intentional, either. Tell me what the available evidence supports.

Finally, turn your findings into a reusable specification.

Give me clear instructions, measurable guidelines, and a quality-control checklist that another AI system could use to produce similar work.

Include a list of questions you would need me to answer before treating this specification as final.

When you get the results, pay particular attention to the patterns that surprise you.

P.S. If you missed it above, this Thursday, October 15, Chris Harder and I are taking raw podcast audio and turning it into a finished, ready-to-publish episode live on Zoom. You'll see what the production process looks like when AI handles the heavy lifting. It's free, and we're starting at 11 AM PT / 2 PM ET. Know someone who podcasts? Forward this their way.

→ Save your free seat here.