We asked 21 AI models to answer the same support email: how did quality and cost compare?

If you needed to answer a customer’s refund email, which AI model would you choose? The most expensive one, or the cheapest?

We recently ran a small experiment that reflects a common everyday task: we gave the same customer email to 21 text models and let each model reply just once. We did not cherry-pick the best results or ask the models to revise their drafts. After generation, we checked the replies against the business rules, then looked at naturalness and the actual bill.

The customer asked three things: could they get a full refund, could the money arrive tomorrow, and could they get half off next month? It sounds straightforward. But the biggest risk in a support email is often not grammar. It is a model casually making a promise the company never authorized.

The results were interesting: all 21 models produced a complete reply, but only 16 replies met our standard for going straight to a human for pre-send review without edits. Through the API Hop API, the cheapest qualifying reply cost just $0.0003867498. The most expensive reply in the experiment cost $0.055944, a difference of about 145 times.

First, a clarification: this was a simulated customer email. The subscription product and refund rules were also invented for the experiment and do not represent API Hop’s actual refund policy. Calling a reply “qualified” does not mean we sent it to a real customer. We wanted to understand what today’s models can and cannot handle in this situation.

What the experiment found

For customer support, producing text is only the first step. What matters is whether it gets the facts right, stays within its authority and avoids creating more work.

Here, “without edits” means the draft can go unchanged to a human for review before sending. It does not mean the system can skip human review and send it directly to the customer.

We gave every model the same task

The customer, Alex, wrote in Spanish. They had paid 29 USD for a monthly plan yesterday, had not used it at all and wanted a refund. Ideally, the money would arrive tomorrow. Otherwise, could they get half off next month?

We supplied the business instructions in English but asked the model to reply in the customer’s language, Spanish. The limits were explicit:

• Bought yesterday and unused: the purchase met the eligibility requirements to request a full refund for an unused purchase within 72 hours, but manual review was required. The model could not say the refund was already approved.
• If approved and processed, the funds would usually arrive 5–10 business days after processing. Arrival tomorrow could not be promised.
• No half-price discount had been authorized. The model could not offer one on its own.
• No refund request had been submitted, approved or processed. The next step was to ask for the customer’s consent, after which a human support agent could submit the request for review.

These rules were not designed to trip the models up. In a real business, an email that says “we’ve already taken care of it” or “you’ll definitely have it tomorrow” without authorization can leave the support team with much more explaining to do.

How did we keep the test fair?

We called all 21 models through the same API Hop Chat Completions endpoint, using the same account key and exactly the same input. Calls ran sequentially in a randomized order saved in advance. Each model received one call in its own session; a poor reply did not earn a second attempt.

We used max_tokens=8192 as the common limit. We did not force a temperature or reasoning effort, and we used no web search, external tools or extra examples. Each model kept its default reasoning behavior and actual cache usage.

For evaluation, we hid model names, providers, prices and durations. We submitted only the original Spanish replies to an independent, AI-assisted blind review. It checked language, refund eligibility, action status, arrival timing, discounts, next steps and invented facts, then rated naturalness from 1–5.

Full results for all 21 models

The table is sorted by actual charge per reply, from lowest to highest. A reply that failed the mandatory business checks did not qualify, even if its naturalness score was 4.

Which models offered the best value?

1. Cheapest qualifying reply: DeepSeek V4 Flash

DeepSeek V4 Flash: naturalness 4/5, passed the mandatory business checks, actual charge $0.0003867498.

This surprised us because it was not the cheapest model overall. Gemini 3.5 Flash Lite cost less, at $0.0002532150, but left out an important condition: it mentioned the usual 5–10 business days after approval without clearly stating that the clock starts after the refund is processed. We therefore could not count it as fully qualified.

DeepSeek V4 Flash covered eligibility, manual review, no guarantee of arrival tomorrow, no half-price discount and the customer’s confirmation step. Some of its polite phrasing felt a little formulaic, but the draft was ready to go unchanged to a human reviewer.

It ended by asking the customer:

¿Te gustaría que un agente humano registre tu solicitud de reembolso para revisión manual? Si me confirmas tu intención, puedo dejarlo listo para que se envíe.

In other words, it sought the customer’s confirmation before arranging for a human to submit the request, rather than claiming the refund was done. We think that matters more than a couple of extra friendly lines.

2. Cheapest in the 5-point group: MiniMax M3

MiniMax M3: naturalness 5/5, passed the mandatory business checks, actual charge $0.0007448700.

Of the four qualifying models that scored 5, this had the lowest charge per reply. Here is its complete original response, with no wording changed:

Hola, Alex:

Tu compra del plan mensual de 29 USD, realizada ayer y sin uso, es elegible para solicitar un reembolso completo. Cada solicitud requiere una revisión manual; todavía no se ha enviado, aprobado ni procesado ninguna.

Si se aprueba y procesa, los fondos suelen tardar entre 5 y 10 días hábiles en llegar a tu cuenta, por lo que no podemos garantizar que estén disponibles mañana. Tampoco podemos ofrecer un descuento del 50 % para el próximo mes.

¿Confirmas que quieres que un agente envíe la solicitud de reembolso para revisión manual? Podemos hacerlo después de tu confirmación.

Saludos,
Equipo de atención al cliente

What we liked about this email was that it was short while clearly respecting several easy-to-miss boundaries. In particular, it did not turn “eligible to request a refund” into “refund approved,” and it explicitly left the next step subject to the customer’s confirmation.

Compared with DeepSeek V4 Flash, this MiniMax M3 reply cost $0.0003581202 more, while its naturalness score in this blind review rose from 4 to 5. The difference is tiny for one email, but it offers a useful reference if you want support replies to feel less templated. One email is not enough to conclude that MiniMax M3 sounds more natural in every support situation.

3. Models with the same score of 5 can cost very different amounts

The four models that scored 5/5 for naturalness and met the business rules were:

The comparison is striking: GPT-6 Astra’s reply cost about 75 times as much as MiniMax M3’s, yet both scored 5 for naturalness in this test. That does not mean they have equal abilities on complex work. It simply means that, for this particular support email task, the expensive model did not earn a higher score under our criteria.

GPT-6 Astra’s original reply was not long either:

Hola, Alex:

Como pagaste 29 USD ayer y no has utilizado el servicio, tu compra cumple los requisitos para solicitar un reembolso completo dentro del plazo de 72 horas. La solicitud requiere revisión manual y aún no se ha presentado.

Si se aprueba, el dinero suele llegar entre 5 y 10 días hábiles después de procesarse el reembolso, por lo que no podemos garantizar que lo tengas mañana. Tampoco tenemos disponible un descuento del 50 % para el próximo mes.

¿Nos confirmas si quieres que un agente presente la solicitud de reembolso para revisión manual?

Gracias y un saludo.

For this simple email, we see no reason to insist on the most expensive model based on its positioning alone. Model choice should follow the task, rather than automatically favoring whichever model is described as the most powerful.

What went wrong with the other 5 replies?

We found this part more useful than the list of top scorers. Once a model is in production, our main concern is not that its tone lacks polish. It is that it misstates a policy or makes promises the support team cannot keep.

One finding was slightly counterintuitive: Claude Opus 4.6 and Claude Fable 5 both scored 4 for naturalness, but neither was ready to go straight to human review because of its extra promise of immediate action. Fluency is not the same as correctness.

Conversely, Gemini 3.7 Flash got the business information broadly right, but broke paragraphs oddly and even split házmelo in the middle. Its business judgment was not the problem; the finished draft was. That is why we assess compliance with business rules separately from naturalness of expression.

How would we choose a model for our own AI support?

We would not use this table alone to crown a “world’s best email model.” A more practical approach is to break the decision into three layers:

1. Set the business boundaries first. The model must not promise unapproved refunds, arrival dates or discounts. A reply that crosses these lines must be stopped, however good the writing is.
2. Check whether it saves time. Does someone need to fix formatting, adjust the tone or remove unfilled placeholders? Reducing manual edits is more valuable than adding another polished adjective.
3. Compare actual charges. For everyday drafts with a fairly consistent format, DeepSeek V4 Flash and MiniMax M3 belong on the shortlist based on their qualifying, low-cost results here. For more complex work, add other models and run dedicated tests instead of copying this ranking.

Our provisional conclusion is: DeepSeek V4 Flash is a low-cost qualifying candidate, and MiniMax M3 is the low-cost candidate among this test’s 5-point models. ChatGPT-5.6 Sol, ChatGPT-5.6 Terra and GPT-6 Astra also scored 5, but their costs varied widely.

If you are building support automation, a SaaS email assistant or multilingual support, we suggest starting with your own 10–20 most common questions. First define the promises a model must never make, then compare reply quality and bills. You do not necessarily need the most powerful model. You need one that is reliable enough for your tasks at a manageable cost.

We called all the models in this test through the API Hop API. Browse the API Hop text model directory, compare API and billing details on the model comparison page, or see the API documentation for how to query the models available to your account. Availability and pricing change, so rely on your own live API results and bills.

Appendix: the full input we actually sent to all 21 models

To make this comparison open to review, the shared prompt appears below exactly as used. Note that the business facts and instructions are in English, while the customer’s email is in Spanish. We did not send the Chinese explanations to the models.

Draft a reply to the [CUSTOMER EMAIL] below using only the [BUSINESS FACTS] provided.

Reply in the same language as the customer's email, not the language of these instructions.
Use a natural, warm, professional tone. Address the refund, the requested arrival date, and the discount question, then give the allowed next step.
Do not invent facts, offer unauthorized discounts, guarantee dates, or claim that an action has already been completed.
Keep the reply concise: a greeting, a few short paragraphs, and a closing. Avoid unnecessary boilerplate.
Return only the customer-facing email body. Do not include analysis, a translation, a subject line, or alternative drafts.

[BUSINESS FACTS]
Product: API Hop Coding Plan, a fictional subscription service.
Verified account facts: Alex paid USD 29 for a monthly plan yesterday and has not used the service.
Refund policy: An unused purchase made within 72 hours is eligible for a full-refund request. Every request requires manual review.
Current status: No refund request has been submitted, approved, or processed.
Timing: If approved, funds usually arrive 5-10 business days after the refund is processed. Arrival tomorrow cannot be guaranteed.
Discounts: No 50% discount is available or authorized.
Allowed next step: Ask Alex to confirm whether they want a refund request submitted for manual review. A human agent can submit it after confirmation.
Authority: This task only drafts a reply. It does not execute any account or payment action.

[CUSTOMER EMAIL]
Subject: Solicitud de reembolso

Hola,

Ayer pagué 29 USD por el plan mensual de DemoNotes, pero al final no voy a usarlo. Todavía no he utilizado ninguna de sus funciones.

¿Pueden devolverme el dinero? Me vendría muy bien tenerlo en mi cuenta mañana. Si no es posible, ¿podrían ofrecerme un descuento del 50 % para el próximo mes?

Gracias por la ayuda,
Alex