DeepSeek-R1 and Ollama3.2 - Judge

To assess the quality of response generated by LLMs, we designed and implemented a test suite called LLM-as-Judge. This framework evaluates the performance of an LLM as if it were a judge in a competition, where the model's responses are scored on several aspect by a supposed more intelligent model. In this study, we utilized Mixtral 8x7b as our judge model, which is designed to provide accurate and unbiased scores.

Despite the potential benefits of the LLM-as-Judge test suite, it remains an experimental approach that requires further refinement. As expected, the initial results were not satisfactory, with the rates provided by Mixtral not yet meeting our stringent requirements for accuracy and reliability. This limitation emphasizes the need for continued research and development to improve the quality and consistency of human judge models like Mixtral, as well as the evaluation frameworks that rely on them. As a result, while the LLM-as-Judge test suite holds promise, it is still in its early stages of development and requires additional work before it can be considered a reliable tool for evaluating the performance of LLMs.

We still provide the results and let humans evaluate and compare the quality of answers given by the different models. You can find questions/answers categorized realms in the following tabs.