Building Your AI Team in 2025

Throughout 2025, I had one-on-one conversations with teams building AI products, from three-person startups to engineering organizations shipping generative features inside large companies. One pattern showed up almost everywhere. The teams that shipped fast matched the shape of the team to the shape of the product. Prompt engineering and evaluation belonged to the people with enough context to judge the output.

The market moved quickly around these teams. The 2025 Stanford AI Index found that organizational AI use rose from 55% in 2023 to 78% in 2024. The cost of inference at GPT-3.5-level performance fell more than 280-fold between November 2022 and October 2024. Models became easier and cheaper to use. Team organization became a larger constraint.

What effective teams have in common

A few traits showed up across almost every team I talked to, regardless of size or industry.

Comfortable with ambiguity. LLM output rarely resolves into a clean pass or fail. Teams that struggled kept waiting for a complete specification. Teams that moved fast treated judgment calls about tone, correctness, and tradeoffs as part of the job.

Familiar with models and prompt engineering. Everyone touching the product needs enough hands-on experience to understand where the model is reliable, where it fails, and how a prompt change affects the output.

Collaborative across disciplines. Prompt engineering, evaluation, and product decisions all touch the same output. Teams that kept these three siloed produced worse results than teams that let engineers, product managers, and domain experts read each other's work.

Systematic and iterative. The teams that improved fastest treated every prompt change like a shippable unit of work with a before-and-after comparison attached.

The five team shapes

LLM products vary widely, and their teams should reflect those differences. Across the conversations I had this year, five recurring shapes came up again and again. Most teams map cleanly onto one of these rows.

A few things are true across all five shapes. The founder, product manager, or domain expert plays a similar role no matter the label on their title. Product-minded engineers, ones who care about the output as much as the implementation, consistently drive better results than engineers who treat the model as a black box behind an API call. And a dedicated data engineer only earns their seat when the product's quality depends on retrieval over the team's own data. If it doesn't, that role is better spent elsewhere.

Team size follows a simple pattern. Most teams start with one or two people and grow to three or four once they have traction. Adding headcount before you have a working product mostly adds coordination overhead.

The capability stack

Regardless of team shape, the same set of capabilities has to exist somewhere on the team.

• Customer and domain understanding. Someone has to know what a good answer looks like to the end user, beyond what the model is technically capable of producing.
• Full-stack product engineering. Most of an AI application is still ordinary software: authentication, data models, UI state, deployment. Someone has to move across the whole stack.
• Data engineering. Collecting, cleaning, and structuring the data that retrieval and fine-tuning depend on.
• Interaction design for streamed and probabilistic output. Output arrives token by token and is sometimes wrong. Designing for that reality changes how you build loading states, corrections, and confidence signals.
• Prompt engineering. Writing and iterating on the natural-language instructions that shape model behavior.
• Retrieval. Getting the right context in front of the model at inference time.
• Evaluation. Measuring whether a change made the product better or worse before it ships.
• Selective fine-tuning. Training a smaller, specialized model or adapter when prompting and retrieval hit a ceiling.

McKinsey's 2025 State of AI survey found that software and data engineers were the most in-demand AI hires. Larger companies also reported more hiring across specialized AI roles. That gap matches what I heard directly. Small teams get further with generalists who can move across this stack. Narrow specialists become useful when the product exposes a specific constraint.

The same survey found that AI high performers were three times more likely to report strong senior-leadership ownership and commitment. Team shape matters. Leadership attention matters too.

The practices LLM products add

Instruction-following models turned prompt engineering into a shared product practice. They also made side-by-side evaluation of prompts and model versions part of the normal release process.

The rest of the operating loop builds on familiar software practices:

• Gathering user feedback on generations, the same way you'd gather feedback on any feature
• Tracking usage and second-order activity, the same analytics discipline applied to a new surface
• Running A/B experiments, unchanged in method, applied to prompt and model versions
• Running evaluation checks in CI as part of the normal pull request process, the same instinct that produces unit tests

Recognizing which practices are new and which are adapted matters because it changes who should own them. The new practices need new muscle. The adapted ones can lean on skills your team already has.

A concrete evaluation loop

Vague statements that evaluation matters do not help anyone ship. What worked, informed by Hamel Husain's field guide to improving AI products, looks like this:

1. Start with real traces. Pull actual production or test conversations you've already observed.
2. Read the failures manually. Before writing an automated check, review a representative sample and record where each output goes wrong.
3. Write explicit criteria. Turn what you noticed into specific, checkable statements.
4. Build a small initial test set. A handful of representative cases is enough to start comparing prompt versions against each other.
5. Grow a durable golden dataset. As you find new failure modes, add them permanently so they get checked on every future comparison.
6. Separate deterministic checks from subjective judgment. Code checks handle things like format, length, and presence of required fields. Model-based judges, validated against human judgment first, handle subjective quality.
7. Compare versions before release. Run the new prompt or model against the same dataset as the old one and compare the two outputs directly.
8. Capture production feedback and feed it back in. When something fails in production, add it to the golden dataset so it never silently regresses again.

One thing worth flagging directly: if your evaluation set shows a perfect pass rate, that usually means the eval itself is too weak.

Domain experts belong in the loop

Teams working in regulated or accredited industries need a domain expert in the loop, and we saw experts contributing directly to strategy, marketing, legal review, and prompt language itself. In most of these cases, engineering alone cannot keep the measure-learn-build loop moving without that expert's judgment. Code generation is the one clear exception, where engineers are usually the domain experts themselves.

Two public case studies illustrate this well. At Twain, linguists directly owned prompt engineering, and that work was kept separate from the core codebase so it could move at its own pace. Humanloop's case study estimated annual savings of around $70,000 per team from that setup. At Duolingo, curriculum and content experts define the course structure and raw learning material, while AI supports content production, exercise creation, and personalization. Teaching experts retain authority over what a correct lesson looks like.

The lesson from both is the same. When the person who understands quality writes the prompt directly, the feedback loop gets shorter and the output gets better.

The team after product-market fit

The most productive teams I talked to, the ones shipping quickly and finding product-market fit, used hosted foundation models. Custom training makes sense for the small minority of teams with a differentiated data advantage and a specific product gap that training will close.

Chip Huyen's framing in AI Engineering is useful here. Prompt engineering, retrieval, fine-tuning, and dataset engineering form the practical stack for building on foundation models. Fine-tuning becomes useful after prompting and retrieval expose a persistent, measurable gap. In our experience, teams that invest heavily in custom model training often move back toward a best-in-class hosted model because the quality gain rarely justifies the ongoing cost of data, infrastructure, evaluation, and maintenance. Use that as the working assumption until you have hard evidence for custom training.

The practical sequence looks like this. Start on a hosted model. Add retrieval when the model needs context it doesn't already have. Reach for fine-tuning only after prompting and retrieval have been pushed as far as they'll go and you can point to a specific gap. Consider custom training only when you have a data advantage large enough to justify the ongoing infrastructure cost.

How the best teams collaborate

Working on generative AI features pushes engineers and product people closer to domain experts than most feature work does, because everyone is looking at the same output and arguing about the same tradeoffs. Concretely, that looked like a few repeatable practices:

• Prompt versions and forks shared across the whole team in a common place
• Cross-functional review at release time, where engineering, product, and domain expertise all look at the same output before it ships
• Written notes tracking what changed between prompt versions and why, so the reasoning survives past the person who made the change

The operating rule is simple. Treat prompts, datasets, and evaluations as shared, reviewable artifacts.

Where Klu fits

Klu exists to move prompt and evaluation work out of scattered documents, Notion pages, and spreadsheets and into one place teams can actually collaborate in. Product managers, engineers, and domain experts share and version prompts together, run the evaluation workflow described above against a shared dataset, and track feedback from production in the same system. Engineers connect that workflow to the actual product using the Klu SDK, so what gets tested in evaluation is the same thing that runs live.

That's the extent of it here. The team practices above matter regardless of which tool you use to run them.