locally led development

Locally Led Evidence: How Kabakoo Tracks What Happens After the AI Responds

2026-08-25 · EN

Kabakoo's WhatsApp-enabled upskilling program pairs an AI Mentor with a four-level evaluation framework that examines model behavior, product use, learner change, and longitudinal income outcomes.

In March 2026, a learner in Bamako recorded the introduction video required for her first challenge in Kabakoo's upskilling program. The AI Mentor assessed it, found that it did not meet the criteria, rejected it, and the learner never tried again. When Kabakoo later called her, she explained plainly: "I dropped out after the first challenge, when the AI Mentor did not accept my introduction video." The system had done what it was built to do. And still, a learner walked away.

That moment sits behind how Kabakoo now evaluates artificial intelligence inside its upskilling programs. In July 2026, Kabakoo engineers were in Nairobi for the sprint of the Agency Fund's AI for Global Development Accelerator, where progress is assessed against a framework the Agency Fund built with the Center for Global Development. There, Kabakoo's engineers presented a four-card framework covering Model, Product, User, and Impact.

Same product, different encounters

Focus groups and hundreds of individual phone calls with dropouts from Kabakoo programs revealed that some learners became trapped in loops, with many reporting remaking a video up to 5 times and some speaking of 10 attempts on a single submission without validation. Yet Maimouna described the same AI Mentor as encouraging: it gave her feedback, showed her what to add or change, and helped her find the courage to submit her videos. Same product, a strikingly different encounter.

Language as it is actually lived

Kabakoo's Mentor works in French text and Bamanankan voice, and across 11,171 bilingual conversations, nearly 40% of voice interactions happened in Bamanankan. That figure is a reminder that a locally led design has to be built around the languages people actually use, not around a single fixed interface imported from elsewhere.

Where people actually show up

Kabakoo's June 2026 analysis of one WhatsApp Upskilling System cohort included 826 learners, defining an active learner as one who sent a message, completed a module assignment, worked on a portfolio, or attended an in-person activity, excluding those who merely received an AI Mentor message. That distinction keeps the measurement honest about what genuine engagement looks like on the ground.

What changes, and what still needs proof

Kabakoo followed learners over time and compared them with randomized counterfactuals, finding baseline-adjusted growth-mindset gains of about 0.14 to 0.25 standard deviations and an increase of 0.33 in the Kabakoo Behavioral Index. For the broader hybrid program, learners were surveyed at baseline and again 12 to 15 months later, with some followed for as long as 36 months, using a randomized recruit-and-delay design comparing participants with eligible learners who had not yet entered the program, adjusting for baseline differences and cohort.

In Kabakoo's July 2026 round of analyses, 46% of the clean comparison group remained in the lowest monthly income bracket, at or below 40,000 FCFA, compared with 38% of program participants. At the upper end, 7% of comparison learners earned above 250,000 FCFA per month, compared with 11% of program learners, and learners also showed more continued investment in their professional lives, including connectivity and tools, and were less likely to report no professional investment at all.

Kabakoo is precise about the limits of this evidence. It does not yet have causal Level 4 evidence for the AI Mentor in isolation; the presented evidence is longitudinal evidence for Kabakoo's broader hybrid program, within which AI is increasingly important, and separating the AI's contribution from peers, facilitators, and in-person experiences remains part of the work ahead.

An honest boundary problem

Applying the four-level framework exposed a boundary problem Kabakoo names rather than smooths over. The current state of Level 3 defines its subject as an engaged user who has received adequate dosage, so the learner from the opening story never qualifies. Her discouragement appears at Level 2 as drop-off, then disappears from the level meant to ask what changed for her. If Level 3 measures only those who stayed long enough to be measured, it risks becoming an evaluation of survivors, a gap that matters most for the learners any locally led program exists to reach.