AI system design and integration

How to select an AI model for a task

Pick an AI model by defining the task's inputs, outputs and success metric first, then filter candidates against hard constraints like privacy, latency and budget, before testing two to four shortlisted models against a small eval set built from your own real data. No model wins every task. Benchmarks only narrow the field, they don't make the final call for you.

This matters because most teams do the opposite. They pick the model first, usually whatever's trending, then try to bend the task to fit it. That's backwards, and it's expensive to fix once it's live.

Why there's no single best AI model

Benchmarks measure averages, not your task

Leaderboards tell you how a model performs across thousands of generic prompts. Your task is not generic. A model that scores well on MMLU or a coding benchmark can still fumble a specific extraction job on messy WhatsApp text full of typos, voice-note transcriptions and mixed English-Afrikaans slang. Benchmarks are a starting filter, not a verdict.

Model tiers in 2026: flagship, balanced, budget

Most providers now ship three rough tiers: a flagship model built for reasoning-heavy, high-stakes work, a balanced mid-tier model for everyday generation tasks, and a budget or "fast" model built for classification, routing and short replies at low cost. Pricing between the top and bottom tier can differ by 10 to 30x per call. Picking the wrong tier for a simple task is where most AI budgets get wasted.

Why a 92% accurate model can beat a 95% accurate one

Accuracy alone doesn't tell you if the model fails safely. A 92% accurate model that clearly flags its 8% of uncertain cases for human review beats a 95% accurate model that answers everything with false confidence. For customer-facing work, predictable failure mode matters more than raw score.

Step 1: define the task before you touch a model list

Write down inputs, outputs and the success metric

Before comparing a single model, write down exactly what goes in, what comes out, and how you'll know it worked. "Classify this WhatsApp message into one of six categories" is a task. "Help customers" is not. If you can't write the success metric in one sentence, you're not ready to pick a model yet. You're still scoping the project. This is the same discipline we use when scoping the task before you scope the model.

Classification and extraction vs generation and reasoning

These are different jobs and they need different models. Classification and extraction (intent detection, pulling an order number out of a message, sentiment tagging) are narrow, deterministic-ish tasks that small, fast models handle well. Generation and reasoning (drafting a reply, resolving a complaint, deciding on a refund) need more context and nuance, and usually justify a bigger model.

Decide if this is a single call or part of an agent workflow

A one-shot classification call is a different decision than a multi-step agent that calls tools, checks a database and decides whether to escalate. Agent workflows often need different models at different steps, which is why you shouldn't lock in one model for the whole pipeline before mapping the steps.

Step 2: set hard constraints first

Data residency and POPIA for South African businesses

If customer data can't leave South Africa, or can't sit on a third-party US-hosted API under your current data processing agreements, that rules out models before you even open a benchmark page. This is a legal and compliance decision, not a technical preference, and it should happen before cost or accuracy conversations. Check what POPIA actually requires before you pick a model if you're handling WhatsApp conversations, ID numbers, or payment details.

Self-hosted vs API models

Self-hosting (open-weight models like Llama or Mistral variants, run on your own infrastructure or a local cloud provider) gives you full data control but adds operational overhead: GPU costs, maintenance, and someone on your team who can actually keep it running. API models are faster to ship but mean data crosses a third party's infrastructure, sometimes outside South Africa. Most SA businesses land on API models for speed to market, with self-hosting reserved for cases where POPIA or a client contract demands it.

Language support and context window limits

If your customers write in Afrikaans, Zulu, Xhosa or code-switch between English and a local language mid-sentence, test that specifically. Flagship models handle this better than budget ones, but not perfectly. Also check context window: a model that can't hold a full WhatsApp conversation history will start forgetting earlier context mid-chat, which shows up as the bot asking the same question twice.

Step 3: set your cost and latency budget before shopping

Cost per message in ZAR, not just per token in USD

Providers price per token in USD. Your client cares about cost per message in ZAR, because that's what lands on their monthly PayFast or Yoco-linked invoice. Convert early. A model that looks cheap at $0.15 per million tokens can still blow a budget if your average WhatsApp exchange runs 15 back-and-forth messages with growing context each time. Run the real math on a sample conversation, not a single prompt, before you commit. For a longer view, see what model costs actually look like over two years.

Latency targets for WhatsApp and real-time chat

WhatsApp users expect a reply within a few seconds, not the 10 to 20 seconds a large reasoning model can take on a complex call. If your use case is real-time chat, latency is a hard constraint, not a nice-to-have. This alone rules out some flagship models for anything beyond backend processing or batch work.

Why finance should approve the ceiling before the technical team picks a model

Set the maximum acceptable cost per message and the maximum acceptable latency before anyone touches a model comparison. Otherwise the technical team optimises for accuracy, picks the best-scoring model, and finance finds out three weeks later that it's 8x over budget at scale.

Step 4: shortlist 2-4 models using benchmarks as a filter

How to read benchmark leaderboards without being misled

Use benchmarks to eliminate obviously wrong candidates, not to pick a winner. If a model scores poorly on reasoning benchmarks and your task is pure classification, that's irrelevant, discard the benchmark, not the model. Look for benchmarks closest to your actual task type (tool-use, multilingual, instruction-following) rather than general knowledge scores.

When to test a cheap model first vs starting with a flagship

Start cheap if the task is narrow: classification, routing, FAQ replies, simple extraction. Start with a flagship only if the task genuinely needs multi-step reasoning or handles unreviewed replies to paying customers. Testing cheap first and upgrading only where it fails is cheaper than starting flagship and trying to downgrade later, because teams rarely bother to downgrade once something works.

Step 5: build a small eval set from your own data

Why 10-20 real examples beat any public benchmark

Fifteen real conversations from your own support history will tell you more than any public leaderboard. Pull actual WhatsApp threads, actual support tickets, actual documents your extraction task will face. A model that aces a generic benchmark can still fail on your specific product names, local slang, or the particular way your customers phrase refund requests.

Scoring models consistently across the same test set

Run every shortlisted model against the exact same eval set, score them on the same rubric, and keep a human reviewing the outputs blind to which model produced which answer. Inconsistent scoring is the fastest way to end up picking a model based on vibes instead of evidence.

What to measure beyond accuracy: tool-call reliability, escalation behaviour, cost at volume

For agent workflows, check whether the model reliably calls the right tool with the right parameters, not just whether its text sounds right. Check how it behaves when it's unsure, does it escalate cleanly or guess confidently. And project the cost at your real expected volume, not at test-run volume, because cost differences that look trivial at 50 messages can be significant at 50,000.

Step 6: pilot before full rollout

Canary testing in production

Roll the new model out to 5-10% of real traffic before switching everyone over. Watch error rates, escalation rates, and customer complaints for a week or two before expanding. This catches failure modes that never showed up in your eval set because real customers are more creative than test data.

Building fallback and model-switching into your architecture

Model calls fail, get rate-limited, or occasionally return garbage. Build retry logic and a fallback model into the workflow from day one rather than bolting it on after an outage. This is the same approach we use when building fallback logic when a model call fails inside n8n flows.

When to route between models inside one workflow

Don't force one model to handle everything. Route simple intents to a cheap, fast model and only invoke the flagship model when the conversation needs real reasoning. This routing decision, done at the workflow level, is often where the real cost savings come from, more than any single model swap.

A practical example: choosing a model for a WhatsApp support agent

Routing simple queries to a cheap model and complex ones to a flagship

At Sagentics we never pick a model before we pick a task boundary. For WhatsApp automation builds we default to a two-tier setup: a cheap, fast model (Haiku or Flash-class) handles classification, routing, and FAQ-style replies, and a flagship model only gets called when the conversation needs real reasoning or when the reply goes straight to a customer unreviewed. We size this against ZAR cost per message, not USD per token, because that's what actually shows up on the client's PayFast or Yoco-linked invoice at month end. If you're wiring this into n8n, here's how to wire a specific model into an n8n workflow natively.

We also treat POPIA as a hard constraint before cost. If a client's data can't leave South Africa or needs to stay off a third-party API, that rules out models before we even open a benchmark page. And the eval set we build is never generic. It's 15-20 real WhatsApp threads pulled from the client's own history, because a model that's great at customer support in English often falls apart on Afrikaans slang or local payment terminology that a US-trained benchmark never tests. See how model choice plays out in a WhatsApp support agent for the full build pattern.

Where human handoff fits into the model decision

No model choice removes the need for a human escalation path. Define upfront which situations always route to a person, regardless of how confident the model is, refunds above a certain amount, complaints, anything legal. This is less a model decision and more an architecture decision, but it changes which model you need, because a model that only has to handle the easy 80% doesn't need flagship-level reasoning. Read more on how human handoff fits into model selection and where human review belongs in a model-based workflow.

Common questions

What's the most common mistake when choosing an AI model? Picking the model before defining the task. Teams default to the most talked-about flagship model, then try to force every use case through it, including simple classification jobs a cheap model would handle at a fraction of the cost. Define inputs, outputs and success metric first, then shop for a model that fits.

Is there a single best AI model overall, or does it depend on the task? It depends on the task. Benchmarks rank models on broad averages, but your specific job, language, format, and risk tolerance will favour different models for different steps. A model that wins on general reasoning benchmarks can still lose to a smaller model on your narrow classification task.

Should I use a small/cheap model or a large flagship model? Use the cheap model for narrow, low-risk tasks like routing, classification and FAQ replies. Reserve the flagship model for steps that need real reasoning or produce unreviewed customer-facing output. Most production workflows need both, routed by task, not one model handling everything by default.

How does model choice differ between a simple chatbot and an AI agent workflow? A simple chatbot usually needs one model tuned for conversational quality and speed. An agent workflow often needs different models at different steps, a fast model for intent detection, a reasoning model for complex decisions, and reliable tool-calling behaviour throughout. Test each step separately rather than picking one model for the whole pipeline.

How much does model choice affect cost at scale versus during testing? Massively. A cost difference that looks negligible at 100 test messages can multiply into thousands of rand a month at production volume. Always project cost at real expected monthly volume in ZAR before committing, not at the volume you tested with during development.

Do I need to fine-tune a model or is prompting/RAG enough? For most business automation tasks, well-structured prompting plus retrieval (RAG) over your own documents is enough and ships faster. Fine-tuning makes sense when you have a very specific, repetitive output format and thousands of labelled examples. Start with prompting and RAG, fine-tune only if evaluation shows a persistent gap that prompting can't close.

How do I test AI models before committing to one in production? Build a 15-20 example eval set from your own real data, run every shortlisted model against it using the same rubric, and have a human score the outputs blind. Then canary test the winner on 5-10% of real traffic before a full rollout, watching error and escalation rates closely.

What's the hallucination risk for my specific task, and how do I check it? Risk depends on task type: extraction and classification have lower hallucination risk than open-ended generation or answering questions outside the provided context. Check it by feeding your eval set questions it shouldn't be able to answer from the given data and seeing if the model invents an answer instead of saying it doesn't know.

Should I self-host a model or use an API, and how does POPIA factor into that decision? Self-host only if your data cannot legally or contractually leave South Africa or sit on third-party infrastructure, since POPIA governs where personal data flows and who processes it. For most businesses, API models from compliant providers are faster to ship and cheaper to run. Confirm your provider's data processing terms before deciding either way.

How do I choose an AI model for business work specifically in South Africa? Start with POPIA and data residency as hard constraints, then convert all pricing to ZAR cost per message at your real volume, not USD per token. Test local language handling, Afrikaans, Zulu, Xhosa, and code-switching specifically, since most benchmarks don't cover this. Build your eval set from your own customer conversations, not generic examples.

If you're trying to work out which model fits your workflow, or whether to build custom at all, you can also read about deciding whether to build around a model or buy a packaged tool and how a development partner should approach model selection.

If you want a second pair of eyes on this before you commit budget to it, message us on WhatsApp and we'll walk through your task with you.