Kakao publishes safety results for open source Kanana-2, outscoring Google Gemma and Alibaba Qwen.
Kakao has published safety evaluation results for its open-source Kanana-2 small language models, showing both models outperforming comparable models from Google and Alibaba. Kanana-2-1.3B-Instruct and Kanana-2-3B-Instruct were released as open source on Hugging Face on July 28.
According to Kakao, both models ranked first overall among comparable models. Kanana-2-1.3B-Instruct scored 0.70, ahead of Google Gemma at 0.68 and Alibaba Qwen at 0.58. Kanana-2-3B-Instruct also scored 0.70, surpassing Gemma at 0.66 and Qwen at 0.62.
The evaluation used Assur AI, a Korean-language safety benchmark developed by the Korea Information and Communication Technology Association (TTA), KAIST and Kakao under a Ministry of Science and ICT project. It covers 9,560 evaluation items across 35 risk categories, including social risk, sexual content and child protection, crime and illegal activity, violence, and rights violations.
Kakao used an LLM-as-a-Judge method to score responses against predefined criteria. Kanana-2-1.3B-Instruct led in crime and illegal activity, while the 3B model outperformed rivals in sexual content and child protection and rights violations.
“This evaluation was an effort to proactively verify the safety of our open-source models through our own verification framework and to make those results public,” said Kim Gyeong-hun, Kakao’s AI safety leader.
Kakao plans to make pre-release safety evaluations standard for future in-house AI models and extend testing to multimodal and agentic AI.
