Mixture-of-Agents (MoA)

What is Mixture-of-Agents (MoA)?

Mixture-of-Agents (MoA) is a layered multi-agent orchestration architecture in which multiple large language models generate candidate responses that other models synthesize across successive layers, rather than relying on a single model to answer directly.

The architecture comes from a June 2024 Together AI paper (Wang et al., arXiv:2406.04692), which reported that a MoA stack built entirely from open-source models reached a 65.1 percent length-controlled win rate on AlpacaEval 2.0, against 57.5 percent for GPT-4o, then the benchmark leader, with further gains on MT-Bench and FLASK. Several mid-tier models layered together beat every individual model in the stack, including models far larger than any single proposer.

The core premise of MoA rests on collaborative synthesis. It relies on a characteristic of LLMs sometimes called "collaborativeness": a model tends to produce better output when given other models' outputs as additional context, even when those other models are individually less capable.

In a MoA setup, models are organized into sequential layers. In the first layer, multiple distinct LLMs independently generate responses to a prompt. In later layers, a new set of models (or the same models reapplied) take the prompt plus the previous layer's responses as input, synthesizing and refining the information. The process ends in a final layer where one capable model acts as the aggregator, producing the output returned to the user.

MoA vs. Mixture-of-Experts (MoE)

While they share similar names, Mixture-of-Agents (MoA) and Mixture-of-Experts (MoE) operate at different levels of the AI stack. MoE is a model-level architecture; MoA is an application-level orchestration strategy.

MoE makes a single model more efficient. MoA combines multiple different models to increase collective output quality.

Layered architecture and information flow

The architecture of a MoA system is defined by its layered structure and the specialized roles models play within those layers: proposers and aggregators.

Proposer models

In any given layer except the final one, models act as proposers. Their job is to generate candidate responses, explore different reasoning paths, and provide a range of perspectives.

• Layer 1 proposers receive only the original user prompt. Each generates an independent response.
• Layer N proposers (intermediate layers) receive the original prompt plus the concatenated responses from the proposers in layer N-1. They synthesize these prior outputs while adding their own reasoning to produce a refined candidate response.

Aggregator models

The aggregator is the role taken by the model in the final layer. It receives the prompt and all candidate responses from the preceding layer. Its job is synthesis, fact-checking, and final polish, similar to the evaluative role described in LLM-as-a-judge.

• The aggregator is typically the most capable model in the cluster (for example, GPT-5, Claude Sonnet 4.5, or Llama 4 Maverick).
• It reviews the candidate responses from the proposers, selects the most accurate reasoning chains, and combines them into a single, coherent final output.

Information flow

1. Input: The user submits a prompt.
2. Layer 1 (generation): Models A, B, C, and D independently process the prompt and output responses 1A, 1B, 1C, 1D.
3. Layer 2 (refinement): Models E, F, G, and H (which can be the same as A-D) receive the prompt plus [1A, 1B, 1C, 1D]. They synthesize this to output responses 2E, 2F, 2G, 2H.
4. Final layer (aggregation): Model Z (the aggregator) receives the prompt plus [2E, 2F, 2G, 2H] and produces the final answer.

Benchmark performance, token multiplication, and latency trade-offs

The primary advantage of MoA is a leap in benchmark performance, particularly on complex reasoning tasks, coding, and open-ended generation. The original paper's headline result, 65.1 percent on AlpacaEval 2.0 with open-source models versus 57.5 percent for GPT-4o, held across MT-Bench and FLASK as well. Multiple proposers cover more of the answer space, while iterative synthesis corrects errors that any single model might miss.

These gains come with substantial trade-offs:

Token multiplication (cost)

MoA is token-intensive. Because intermediate models must ingest the outputs of all previous models in the layer, the context window grows rapidly.

• If layer 1 has 4 models generating 500 tokens each, layer 2 models must process the user prompt plus 2,000 tokens of context.
• This growth in input tokens leads to higher inference costs than a single API call.

Latency (time to first token)

Latency is the most significant bottleneck for MoA in production.

• A single LLM call might take 2-5 seconds.
• A 3-layer MoA requires waiting for all layer 1 models to finish, then all layer 2 models to finish, and finally the aggregator to finish. This can push response times to 15-30 seconds, making it unsuitable for real-time conversational interfaces.

Production implementation strategies and model diversity

To manage the cost and latency issues while keeping the performance benefits of MoA, production implementations need to be optimized.

Does model diversity actually help? The Self-MoA challenge

The original paper argued that MoA's gains scale with the diversity of the proposer models. A February 2025 Princeton study (Li et al., arXiv:2502.00674) tested that premise directly and found it mostly does not hold. Their Self-MoA variant, which samples multiple outputs from the single best model and aggregates those, beat the mixed-model MoA by 6.6 percentage points on AlpacaEval 2.0 and by an average of 3.8 points across MMLU, CRUX, and MATH. Adding weaker models for the sake of diversity dragged the aggregate down more than the extra perspectives helped. Even on a task constructed so that each model excelled at a distinct subtask, mixed MoA barely edged out Self-MoA.

The practical reading: proposer quality dominates proposer diversity. If you have access to one clearly strongest model, sampling it several times at nonzero temperature and aggregating is the better default. Reserve mixed-model MoA for cases where models have genuinely complementary strengths you have measured, not assumed, or where cost forces you onto several cheaper models (for example, mixing Qwen3, Llama 4, and DeepSeek-V3.2 because the strongest single model is out of budget).

Python pseudo-workflow

Here is a simplified Python example demonstrating a basic 2-layer MoA orchestration using an asynchronous approach to minimize latency:

import asyncio

async def generate_response(model_name, prompt, context=""):
  '''Mock API call to an LLM'''
  full_prompt = f"{prompt}\n\nContext:\n{context}" if context else prompt
  # return await llm_api.call(model_name, full_prompt)
  return f"[Output from {model_name}]"

async def mixture_of_agents_inference(user_prompt):
  # Layer 1: Proposers (Run concurrently)
  proposer_models = ['llama-4-scout', 'qwen3-8b', 'deepseek-v3.2']

  tasks = [generate_response(model, user_prompt) for model in proposer_models]
  layer_1_outputs = await asyncio.gather(*tasks)

  # Concatenate outputs for the aggregator
  synthesized_context = '\n\n---\n\n'.join(layer_1_outputs)

  # Layer 2: Final Aggregator
  aggregator_model = 'llama-4-maverick'

  final_aggregator_prompt = (
    "You are an expert synthesizer. Review the user prompt and the candidate "
    "responses below. Combine the best insights, correct any errors, and produce "
    "a final, definitive answer.\n\n"
  )

  final_result = await generate_response(
    aggregator_model,
    final_aggregator_prompt + user_prompt,
    context=synthesized_context
  )

  return final_result

Production optimizations

• Streaming aggregation: While intermediate layers cannot be fully streamed to the user, the final aggregator layer can stream its output, providing faster perceived latency.
• Dynamic MoA (routing): Instead of always using a MoA pipeline, use a fast LLM as a router to determine if a prompt is complex enough to warrant MoA. Simple queries ("What is the capital of France?") go to a single model, while complex reasoning queries trigger the MoA pipeline.
• Smaller proposers, large aggregator: Use fast, cheap models for the proposer layers, relying on a more capable, more expensive model for the final aggregation layer to balance cost and quality.

Frequently Asked Questions

Is MoA suitable for real-time chat applications? Generally, no. The multi-layered, sequential nature of MoA introduces significant latency, often 10-30 seconds, making it unsuitable for standard chat interfaces. MoA is best used for asynchronous tasks, batch processing, complex coding generation, or research tasks where quality matters more than speed.

Do I need to fine-tune models to use them in a MoA setup? No. One of the primary benefits of MoA is that it uses off-the-shelf, pre-trained, instruction-tuned models. The collaborative capability comes from the prompting and orchestration pipeline, not from specialized weight updates.

How do I choose which models to use as proposers? Start with quality, not variety. The 2025 Self-MoA results showed that repeated samples from the single best available model usually beat a mix of different model families, because weak proposers pollute the aggregator's context. Mix families (Llama, Qwen, DeepSeek) only when you have evaluation data showing they contribute complementary strengths on your task, or when budget rules out multiple calls to the strongest model.

Is Mixture-of-Agents the same as an LLM ensemble? MoA is a specific ensemble design. Classic ensembling picks or votes among candidate outputs, as in self-consistency decoding. MoA instead feeds all candidates back into a model as context and asks it to synthesize a new answer, so the final output can combine partial insights that no single candidate contained.