Rednote Launches Its First Open-Source Language Model with Innovative Mixture-of-Experts Architecture

Rednote, a social networking company, has launched its first major open-source language model. The Mixture-of-Experts (MoE) system, called dots.llm1, aims to match the performance of competing models while significantly reducing costs.

According to Rednote’s [technical report](https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf), dots.llm1 utilizes 14 billion active parameters out of a total of 142 billion. The MoE architecture divides the model into 128 specialized expert modules, of which only the top six modules and two always-active modules are engaged for each token. This selective approach helps conserve computational resources without compromising quality.

Rednote claims a substantial increase in efficiency. Training dots.llm1 on one trillion tokens required only 130,000 GPU hours, compared to 340,000 hours for Qwen2.5-72B. Overall, the entire pre-training process for dots.llm1 took 1.46 million GPU hours, while Qwen2.5-72B took 6.12 million hours—about four times more. Despite this, Rednote asserts that both models yield similar results.

Tests reveal that dots.llm1 excels in Chinese language tasks. In assessments like C-Eval (which gauges Chinese language proficiency) and CMMLU (the Chinese variant of MMLU), the model outperforms Qwen2.5-72B and Deepseek-V3.

In English language tests, dots.llm1 slightly trails the leaders. According to MMLU and the more challenging MMLU-Pro, which assess general knowledge and reasoning ability, the model falls short compared to Qwen2.5-72B.

When it comes to mathematics, dots.llm1 performs well but typically lags behind the largest models. However, its code generation capabilities are impressive, as seen in HumanEval, a standard programming test where dots.llm1 outperforms Qwen2.5-72B and is competitive in other coding challenges.

Rednote trained the model on 11.2 trillion high-quality tokens, exclusively using real internet text and avoiding synthetic data. The data processing involved three stages: document preparation, rule-based filtering, and model-based processing. Two innovations were highlighted: a system that removes distracting elements from websites, such as ads and navigation panels, and an automatic content categorization system.

The company developed a classifier featuring 200 categories to optimize its training data set. This enhanced the proportion of factual and knowledge-based content (like encyclopedia articles and academic papers) while reducing the amount of fictional and highly structured web pages, such as product listings.

Rednote publishes interim checkpoints for every trillion tokens trained, giving the research community insight into the learning dynamic of large models. These models can be accessed on [Hugging Face](https://huggingface.co/rednote-hilab) under an Apache 2.0 license, with the source code available on [GitHub](https://github.com/rednote-hilab/dots.llm1).

With 300 million monthly users, Rednote is entering the crowded Chinese AI market, dominated by companies like Alibaba, Baidu, Tencent, Bytedance, and [Deepseek](https://the-decoder.com/deepseek-r1-triggers-boom-in-reasoning-enabled-language-models/). The new model was developed in Rednote’s lab focused on human intelligence, which split from the company’s AI team and is now hiring more researchers with humanities backgrounds.

Rednote is already testing its research assistant platform, [Diandian](https://www.reuters.com/business/media-telecom/rednote-joins-wave-chinese-firms-releasing-open-source-ai-models-2025-06-09/), based on its own AI model.

This spring, the social networking app briefly made [international headlines](https://edition.cnn.com/2025/01/14/tech/rednote-china-popularity-us-tiktok-ban-intl-hnk) as a potential refuge for US users amid looming TikTok bans. After the ban was lifted, [interest outside China](https://www.start.io/blog/new-data-as-expected-rednote-installs-plunge-after-u-s-tiktok-ban-reversed/) waned.

Nevertheless, on June 7, Rednote opened its first office outside mainland China in Hong Kong and plans to expand internationally. According to [Bloomberg](https://www.bloomberg.com/news/articles/2025-06-04/chinese-social-media-app-xiaohongshu-s-26-billion-valuation-bolsters-gsr-fund), its valuation reached $26 billion this year, surpassing its peak during the pandemic, with an IPO expected in 2025.

I would like to take this opportunity to recommend [BotHub](https://bothub.chat/?utm_source=contentmarketing&utm_medium=habr&utm_campaign=news&utm_content=REDNOTE_LAUNCHES_ITS_FIRST_OPEN-SOURCE_LANGUAGE_MODEL_WITH_MIXTURE-OF-EXPERTS_ARCHITECTURE)—a platform to test all popular models without restrictions. No VPN is necessary for access, and it accepts Russian cards. [Follow this link](https://bothub.chat/?invitedBy=m_aGCkuyTgqllHCK0dUc7) to get 100,000 free tokens for your initial tasks and start working right away!

[Source](https://the-decoder.com/rednote-releases-its-first-open-source-llm-with-a-mixture-of-experts-architecture/)