Getting Access to the Beta

I recently received beta access to ERNIE Bot, which finally gave me a chance to see what this much-discussed “Chinese ChatGPT” actually looks like in practice.

Rather than judging it by marketing language alone, I tested it across several areas that matter most for a large language model: language understanding, coding, practical everyday problem-solving, document reading, and image generation.

Language Understanding

The first test was aimed at a very basic but crucial question: how well does Baidu’s so-called “new-generation knowledge-enhanced large language model” actually understand language?

If an AI assistant still struggles with ordinary language comprehension, then talking about catching up with ChatGPT is basically wishful thinking. Some domestic models have given exactly that kind of disappointing impression before, so this part was worth checking carefully.

Because ERNIE Bot has some limitations when handling long screenshots, the conversation results had to be shown in separate images.

Judging from this round of testing, ERNIE Bot is already quite close to the level of ChatGPT 3.x in general language interaction, though it is clearly not flawless. Overall, its grasp of language is fairly solid. Near the end, when I asked it to answer in the style of “Pangu’s White^1,” it seemed to stumble a little at first, but it corrected itself quickly enough.

So on this front, the result was better than expected. Compared with earlier impressions of similar products, the progress is obvious.

Coding Ability

For programmers, code generation is one of the features that can genuinely improve productivity. If ERNIE Bot can understand a developer’s requirements and produce correct, reasonable code, then that counts as meaningful progress for the model.

For this test, I used a basic quicksort algorithm and added a few simple constraints, then compiled and ran the generated code.

The test environment used TDM-GCC 4.9.2 64bit with the C++11 standard.

Test code:

<table> <thead> <tr> <th>1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48</th> <th>#include <iostream> #include <vector> template<typename T> class SortAlgorithm { public: virtual void sort(std::vector<T>& array) = 0; }; template<typename T> class QuickSort : public SortAlgorithm<T> { public: void sort(std::vector<T>& array) override { quickSort(array, 0, array.size() - 1); } private: void quickSort(std::vector<T>& array, int left, int right) { if (left < right) { int pivotIndex = partition(array, left, right); quickSort(array, left, pivotIndex - 1); quickSort(array, pivotIndex + 1, right); } } int partition(std::vector<T>& array, int left, int right) { T pivot = array[right]; int i = left - 1; for (int j = left; j < right; j++) { if (array[j] <= pivot) { i++; std::swap(array[i], array[j]); } } std::swap(array[i + 1], array[right]); return i + 1; } }; int main() { std::vector<int> data = {64, 25, 12, 22, 11}; QuickSort<int> sorter; sorter.sort(data); for (int i : data) { std::cout << i << " "; } return 0; }</th> </tr> </thead> <tbody> <tr> <td></td> <td></td> </tr> </tbody> </table>

For a simple sorting algorithm like this, ERNIE Bot handled the task easily and basically followed the requested design. More importantly, the code ran successfully on the first try, with almost no need for modification.

That is a pretty noticeable improvement. The contrast is especially clear when compared with some other domestic models that have performed much worse in similar coding scenarios.

Handling Everyday Problems

For most users, the ability to solve real-life problems may matter more than programming or technical conversation. The key question is whether ERNIE Bot can take a person’s actual needs and provide answers that are genuinely usable.

For this part, I used weight loss as the example and conducted a practical Q&A test around an everyday life scenario.

Based only on these three answers, the overall performance was quite good. The responses were organized into fairly clear categories, and the advice was generally useful.

Of course, the final quality still depends heavily on how the question is asked. If the prompt is vague or does not provide enough information, the answer can be affected significantly. But with a clear and reasonable question, ERNIE Bot is basically capable of helping with some everyday problems.

Document Reading

Document reading is a relatively newer feature. At the moment, it supports only doc, docx, and pdf formats, and each uploaded file must be no larger than 10MB. Even with those limits, it should still be enough for simple reading and summarization tasks.

To test it, I used Scott Meyers’ Effective Modern C++: 42 Specific Ways to Improve Your Use of C++11 and C++14 and asked ERNIE Bot to interpret and summarize the content.

From these answers, ERNIE Bot’s understanding of the book appeared relatively comprehensive and accurate. It was not merely extracting surface-level text; it was able to identify the key points and condense the content into a fairly precise summary.

This is probably one of the more impressive parts of the test. It has to be admitted that among domestic large language models, ERNIE Bot is one of the few that feels reasonably capable in this area.

Image Generation

AI image generation has been extremely popular recently, so I also wanted to give ERNIE Bot’s drawing ability a quick test. Unfortunately, this was the most disappointing part.

Compared with its text-related abilities, ERNIE Bot’s image generation did not feel particularly strong. The gap here is quite obvious.

Overall Impression

Across these tests, ERNIE Bot was evaluated in five areas: language understanding, code generation, everyday problem-solving, document reading, and image generation.

Its language understanding was generally in line with expectations. It can handle basic intelligent conversation and shows real progress, though there are still rough edges. Its coding performance was surprisingly smooth in the quicksort test: it understood the requirements, produced reasonable C++ code, and the code compiled and ran successfully without much adjustment.

For practical life questions, the answers were also solid as long as the prompt was framed clearly. The document-reading feature was the strongest surprise, especially in its ability to understand and summarize content from a technical book. Image generation, however, was clearly the weak point and left a poor impression.

Taken as a whole, ERNIE Bot is no longer just a product that exists in announcements. It performs well in several important areas, especially language comprehension, programming assistance, and document summarization. But it still needs more improvement, particularly on the image generation side.