Why health AI needs to read between the lines

Sahar Abid is a Science Associate at Ema EQ, where she works on cultural sensitivity and bias in AI.

A woman asks an AI health assistant about postpartum depression.

She mentions that her in-laws are telling her to “push through” and skip medical help, even as her symptoms get harder to manage. She never says where she is from or names her background.

The assistant describes the condition and gives her a hotline number. It sounds correct, but it misses what she needs.

That gap is more common than the industry admits, and it points to a blind spot in how we test health AI for bias.

Most bias testing looks at what people explicitly say.

The typical way to check an AI for bias is to label a prompt with someone’s demographic details and see if the answer changes. That catches some problems but misses a bigger one.

Most people do not lead with their identity. They lead with their situation. The woman above told the assistant everything it needed to help her, just not in the form of a label.

Her real question was not only “what is postpartum depression?” It was “how do I get care when the people around me don’t want me to?

When family members hold sway over health decisions, and in many communities they do, advice that asks someone to overrule their family is not something they can act on.

The AI didn’t say anything factually wrong. It answered a different question than the one she was living.

We call this culturally implicit bias, meaning the AI misses the cultural context a situation implies rather than the context a person spells out.

When systems are trained to notice only the explicit cues, they fall back on a default answer built for the majority. For everyone else, the response can feel generic, off-target, or discouraging enough that they stop looking for help.

In health, that is not small. The people most likely to be missed are often the ones the system already underserves.

What we set out to test.

At Ema, we wanted to know how well AI picks up on cultural context that is implied but never stated. So we built our own way to test for it, across a range of communities and real situations like postpartum depression and fertility, using questions that carried cultural meaning without announcing it.

The patterns were consistent. Models often missed the meaning underneath the question. They dropped the specific details a person did share and smoothed them into something generic.

And even when they pointed toward real care, they tended to offer one option instead of choices that might actually fit a person’s life. Any one of those can be the difference between someone following the advice and walking away from care.

Why this matters for anyone building health AI.

Getting this right is the right thing to do, and it also works better.

When an answer reflects a person’s real context, people trust and act on the recommendations more, so they get the help and support they need.

Testing for it is harder than the shortcut most teams use. Swapping a name or a demographic label in and out is easy. Checking whether a model actually understands the human context around a question takes more care.

The shortcut teaches models to perform cultural competence instead of practicing it. No matter how much or how little someone chooses to share, they deserve an answer that is warm, complete, and usable.

A better question.

The bar for equitable health AI should be “does it serve someone who never told you who they are?” It is the harder test, but it determines whether real people get help.

The work of getting there is far from finished, and it is exactly what we are building toward at Ema.

Sources: Naidoo, V., & Chadha, K. K. (2025), Culturally responsive AI chatbots: from framework to field evidence, Computers in Human Behavior: Artificial Humans. Souligne, N., & Subbian, V. (2026), FairLogue: A toolkit for intersectional fairness analysis in clinical machine learning models.

Learn more about Ema EQ