README.md

metadata

title: README
emoji: 🚀
colorFrom: blue
colorTo: gray
sdk: static
pinned: true

We are a small but ambitious AI research group, focused on developing performant and highly capable Large Language Models that aim to excel in their domains. Our specialty lies in the exploration of cutting-edge model finetuning methods and innovative data preparation techniques.

Our Mission

At Zeus Labs, we strive to push the boundaries of AI capabilities, particularly in the realm of language models. Our goal is to create models that not only perform well but also demonstrate exceptional abilities in specific domains, contributing to the advancement of AI technology and its applications.

Our Approach

• Cutting-edge finetuning methods for Large Language Models
• Innovative data preparation and curation techniques
• Focus on domain-specific excellence and versatility
• Open collaboration and knowledge sharing within the AI community
• Advancing LLM research via novel techniques, which has been applied by all of our members.

Team:

Chief ML Engineer, M.S.

@elinas - HuggingFace Profile

Senior Data Scientist, PhD

@ToastyPigeon - HuggingFace Profile

Operations Engineer

@fizz - HuggingFace Profile

ML / DS Engineer

@SteelSkull - HuggingFace Profile

Notable Achievements

• Revival of Llama 1 33B by training on over 500M tokens
• We did this based on the original pretraining token count of 1.4T and decided to add another 500M tokens to it, to which our surprise ended up surpassing expectations in both quality and length
• It was trained at 16384 context legth with an *effective* context legnth around 12k due to the nature of the samples, but exceeds in RP.
• Our next goal is to apply GQA to it, but in the meantime, we will appreciate quanters who will help with running this model on less VRAM!

• Development of L3-Aethora-15B series, The first heavily fintuned 15b model that focuses in creative writing and general intelligence using a novel technique known as "zeroing layers."
• Creation of the Aether-Lite-V1.8.1 dataset, a carefully curated dataset for AI training

Join Us

We are currently growing and looking for passionate individuals interested in machine learning and AI research. Whether you're a seasoned researcher or an enthusiastic beginner, there's a place for you in our community.

Join our Discord to connect with like-minded individuals, share ideas, and potentially collaborate on exciting AI projects!

Join The Zeus Labs Discord

Our Work

Explore our independently developed work and collaborations on our HuggingFace profiles. We're always pushing the boundaries of what's possible with AI!

Model Quanters!

If you create quants for our models and we miss them, please post a discussion to that model and we will add it to the Model card!