Engineering · Language models

"Boleh ke?" How well AI reads the way Malaysians actually write

AI models score well on textbook Bahasa Malaysia. On colloquial particles and mixed-language chat the published evidence is much weaker, for local and frontier models alike. What the benchmarks say, and how to test a model on your own customers' messages.

Pikabyte Engineering 6 min read

28%

average accuracy of ten leading AI models at saying what a Malay particle such as "kan" or "ke" is doing in a sentence, in a May 2026 study. Guessing scores 14%

A customer writes to a distributor in Klang: “Boss, esok delivery boleh ke? Kalau x sempat, Friday pun ok lah. 那个蓝色的还有吗?”

That is one request in three languages, with a short form (“x” for “tak”), two particles, and a question about “the blue one” in Mandarin. No school teaches this style and almost every Malaysian business receives it daily.

So the useful question is not whether an AI model “supports Malay”. It is whether it understands your customers. The published evidence gives three different answers, depending on which kind of Malay you mean.

Formal Bahasa Malaysia: strong scores, with caveats

MalayMMLU is the main public test. It has 24,213 multiple-choice questions drawn from the Malaysian primary and secondary school syllabus, across 22 subjects.

Its public results table reads like this:

Model Score
ILMU 0.1 (YTL AI Labs) 86.98
GPT-4o (OpenAI) 84.98
Llama 3.1, 70B (Meta) 78.44
Qwen 2.5, 72B (Alibaba) 77.79
MaLLaM v2.5 Small (Mesolitica) 71.53
Random guessing 38.02

Textbook Malay is not the hard part any more. But read the table with three things in mind.

It is dated. Most entries are models from 2024. Claude and Gemini are not in the table at all, and neither is any model released in 2026.

The top score has not been independently reproduced, as far as we could find. The benchmark’s repository belongs to a Universiti Malaya and YTL AI Labs group, and ILMU is YTL AI Labs’ model. That does not make the number wrong. It means it is the maker’s own.

Exam questions are not customer messages. A model that knows Form 4 Sejarah has shown nothing about “x sempat”.

Colloquial Malay: the scores fall

In May 2026, researchers led from Nanyang Technological University published MalayPrag, a set of 1,137 colloquial Malay sentences, drawn from Reddit and X and supplemented with sentences written for the study. It looks at two small words, “kan” and “ke”, which change what a sentence is doing: checking, doubting, asking, nudging.

They tested ten models, including GPT-5, Claude Sonnet 4.6, Gemini 3.1 Pro, DeepSeek and two SEA-LION models built for Southeast Asian languages.

  • Asked what the particle was doing in the sentence, the models averaged 28.0%. Guessing would score 14.3%.
  • Asked to choose the right particle for a sentence, they averaged 43.0%. Guessing would score 33.3%.
  • One of the regional models scored 33.2% on that second task, which is chance.

The authors’ summary: “Without the attributes, models struggle significantly to predict pragmatic functions, averaging only 27.96% accuracy.” Given structured hints prepared by annotators, the average rose to 52.5%.

This is one study of two particles, and a model can misread a particle and still get the order right. But in customer service the tone is often the message. “Boleh kan?” and “boleh ke?” are not the same question, and a model that cannot tell them apart will sometimes answer the wrong one.

Mixed-language chat: fluent summaries, wrong facts

CS-Sum, also from Nanyang Technological University, tested how well models understand conversations that switch language mid-sentence. It includes 918 Malay-English dialogues, 1,320 Mandarin-English and 1,000 Tamil-English. Ten models summarised them in English.

GPT-4o did best overall. The finding that matters is about how the others failed: “LLMs make subtle mistakes that alter the complete meaning of the dialogue.” Summaries scored well on automatic similarity measures while getting facts wrong, such as who had confirmed an arrangement.

Two limits. The paper is from May 2025, and nine of its ten models were small open ones. And we could not find a published test of three languages in one message, which is the Klang example above. On real Malaysian chat, the evidence is thin.

Local, regional or frontier

ILMU. Launched on 12 August 2025 by YTL AI Labs with Universiti Malaya. The company describes it as “Fluent in Bahasa Melayu, Manglish and local dialects like Kelate”, and as hosted on the YTL AI Cloud with “local data residency”. By October 2026 its platform also listed models built with partners, such as ILMU-GLM-5.3, priced in ringgit at RM5.15 per million input tokens and RM16.05 per million output tokens. The Manglish and dialect claims are the vendor’s. We found no independent evaluation of them.

MaLLaM. Small open models from Mesolitica, with 1.1 to 5 billion parameters, trained from scratch on 349GB of Malaysian text. Small enough to run on your own hardware.

SEA-LION. Open models from AI Singapore covering Malay and Tamil among other regional languages. Its SEA-HELM leaderboard now includes Malay and, as of November 2026, lists 101 models, 21 of them closed commercial ones. A recent model card is candid: “The model has not been aligned for safety.”

Frontier models from OpenAI, Anthropic and Google. Where the studies above compared them with regional models, the large general models came out ahead or close: GPT-4o was second on MalayMMLU and best on CS-Sum, and the two regional models were generally weaker at choosing the right particle. The frontier models still stumbled on particles like everyone else.

A reasonable reading:

  • Choose a local or self-hosted model when the data must stay in Malaysia, when you want ringgit billing, or when the job is narrow and in formal Malay. See where your AI runs.
  • Choose a frontier model when the job needs several steps, tools, documents or images as well as language.
  • Trust neither on your customers’ messages until you have tested it on them.

How to test a model on your own messages

No benchmark contains your customers. Build a small one.

  1. Collect 200 real messages. Remove names and numbers. Keep the ugly ones: short forms, voice-note transcripts, three languages in a line.
  2. Write the right answer for each. A senior staff member records what the customer wants, the items and quantities, the urgency, and the language to reply in.
  3. Run every candidate model on all 200, with the same instructions.
  4. Have staff mark the results blind, without knowing which model produced which. Count wrong meaning separately from clumsy wording. Only the first is dangerous.
  5. Sort the failures by type: short forms, tone and particles, language mixing, dialect, product and place names. This tells you what to fix in the instructions and what to hand to a person.
  6. Keep the set. Run it again whenever the model or the prompt changes.

Two design habits reduce the damage from the errors that remain. Have the system repeat the request back in plain words before acting on anything that costs money: “Confirm: 2 cartons, deliver Friday?” And give it a clear way to say it is unsure and pass the chat to staff.

That is how we treat language in our own work. The content platform we built for a Malaysian tuition provider produces material in English and Bahasa Melayu, and nothing is published until it has been through AI review and an editor’s approval. The model is not trusted because of a leaderboard.

What to do

  • Ask any vendor which model they use and what they tested it on. “It supports Malay” is not an answer.
  • Ask to see it run on your own messages before you sign.
  • Decide in advance which messages always go to a person.
  • If the agent will work on WhatsApp, read what an AI agent may do there.

Sources

Written by

Pikabyte Engineering

Pikabyte builds custom business systems, applied AI and secure cloud infrastructure from Malaysia. Talk to us.

Start a conversation

Want this applied to your business?

Tell us what is slowing the business down. You will talk to an engineer, not a salesperson, and get a straight answer on whether software is the right fix.