IGEL is an LLM model family developed for German. The first version of IGEL is built on top BigScience BLOOM, adapted to German from Malte Ostendorff. IGEL is designed to provide accurate and reliable language understanding capabilities for a wide range of natural language understanding tasks, including sentiment analysis, language translation, and question answering.
You can try out the model at igel-playground.
The IGEL family currently includes instruct-igel-001 and chat-igel-001
(coming soon).
The 001
version of IGEL is designed as a naive proof of concept to determine whether it is possible to create a German instruction-tuned model using a set of available open-source models and German translated instruction dataset. The goal is to explore the potential of the LLMs for German language modeling tasks that require instruction-based responses.
To achieve this goal, we used a pre-trained adapted BLOOM model (6B) and fine-tuned it using the translated instruction-based dataset. The dataset was created by taking instructions in English and translating them into German using an automated translation tool. While this approach may introduce errors in the translated content, we wanted to test whether the model could still learn to generate instruction-based responses.
We are pleased to announce that we had success. 🥳 Instruct-igel-001 is LoRA-tuned BLOOM-CLP German (6.4B parameters) with merged weights to make it is to load and use with Hugging Face Transformers.
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
# load model
# load pipeline
generator = pipeline("text-generation",model=model,tokenizer=tokenizer)
# run generation
generator("### Anweisung:\n{{input}}\n\n### Antwort:")
instruct-igel-001
is trained on naive translated instruction datasets without much any data-cleaning, filtering, or post-processing.
instruct-igel-001
also exhibits several common deficiencies of language models, including hallucination, toxicity, and stereotypes.
For example, in the following figure, instruct-igel-001
wrongly says that the chancellor of Germany is Angela Merkel.