Kihagyás

HuGME – Hungarian Generative Model Evaluation Benchmark

How good is your Hungarian?

HuGME is an advanced evaluation framework designed to assess the linguistic and cultural capabilities of Large Language Models (LLMs) in Hungarian. Partially employing the DeepEval methodology, HuGME provides a comprehensive, multi-dimensional assessment that covers everything from bias and toxicity to readability and factual accuracy.

What is HuGME?

HuGME is the first benchmark system dedicated to testing Hungarian LLMs. It not only evaluates general performance but also focuses on key linguistic skills unique to Hungarian, including:

  • Linguistic proficiency: Assessment of spelling, grammatical correctness, and readability.
  • Cultural & contextual understanding: Evaluation of the model’s ability to process Hungarian cultural references, and domain-specific knowledge.
  • Ethical considerations: Analysis of bias and toxicity to ensure safe, respectful language outputs.

How it works

HuGME combines several evaluation strategies to deliver a robust assessment:

  • LLM-as-a-Judge: Utilizes GPT-4 within the DeepEval framework for automated, consistent evaluation of outputs.
  • Specialized modules: Custom tests, such as the Needle-in-the-Haystack for context retention, alongside domain-specific assessments for factual accuracy.
  • Comprehensive metrics: Each module is designed to measure specific dimensions including bias mitigation, prompt alignment, and factual consistency.

For detailed documentation, refer to our module breakdown:

  • LLM-as-a-judge
  • Language proficiency
  • World knowledge
  • Needle in the haystack
  • Modules: Access detailed documentation on each evaluation module.
  • Results: View comprehensive evaluation reports and leaderboards.
  • Downloads: Get the HuGME paper and evaluation scripts.
  • News: Keep up-to-date with HuGME.
  • About: Discover our team, partners, and the inspiration behind HuGME.
  • Contact: Reach out for collaborations or inquiries.

Get started

Dive into our documentation to understand how HuGME can help you evaluate and enhance your Hungarian language models. Whether you’re a researcher or a developer, our resources and detailed guidelines are designed to support you every step of the way.

HuGME is continuously evolving. For the latest updates, check our [News] section and follow us on social media.

© 2025 HuGME. All rights reserved. Licensed under Apache 2.0.