👨🏻‍💻 LLM Engineer Toolkit

This repository contains a curated list of 120+ LLM libraries category wise.

Quick links

🚀 LLM Training	🧱 LLM Application Development	🩸LLM RAG
🟩 LLM Inference	🚧 LLM Serving	📤 LLM Data Extraction
🌠 LLM Data Generation	💎 LLM Agents	⚖️ LLM Evaluation
🔍 LLM Monitoring	📅 LLM Prompts	📝 LLM Structured Outputs
🛑 LLM Safety and Security	💠 LLM Embedding Models	❇️ Others

LLM Training and Fine-Tuning

Library	Description	Link
unsloth	Fine-tune LLMs faster with less memory.	Link
PEFT	State-of-the-art Parameter-Efficient Fine-Tuning library.	Link
TRL	Train transformer language models with reinforcement learning.	Link
Transformers	Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio.	Link
Axolotl	Tool designed to streamline post-training for various AI models.	Link
LLMBox	A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation.	Link
LitGPT	Train and fine-tune LLM lightning fast.	Link
Mergoo	A library for easily merging multiple LLM experts, and efficiently train the merged LLM.	Link
Llama-Factory	Easy and efficient LLM fine-tuning.	Link
Ludwig	Low-code framework for building custom LLMs, neural networks, and other AI models.	Link
Txtinstruct	A framework for training instruction-tuned models.	Link
Lamini	An integrated LLM inference and tuning platform.	Link
XTuring	xTuring provides fast, efficient and simple fine-tuning of open-source LLMs, such as Mistral, LLaMA, GPT-J, and more.	Link
RL4LMs	A modular RL library to fine-tune language models to human preferences.	Link
DeepSpeed	DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.	Link
torchtune	A PyTorch-native library specifically designed for fine-tuning LLMs.	Link
PyTorch Lightning	A library that offers a high-level interface for pretraining and fine-tuning LLMs.	Link

LLM Application Development

Frameworks

Library	Description	Link
LangChain	LangChain is a framework for developing applications powered by large language models (LLMs).	Link
Llama Index	LlamaIndex is a data framework for your LLM applications.	Link
HayStack	Haystack is an end-to-end LLM framework that allows you to build applications powered by LLMs, Transformer models, vector search and more.	Link
Prompt flow	A suite of development tools designed to streamline the end-to-end development cycle of LLM-based AI applications.	Link
Griptape	A modular Python framework for building AI-powered applications.	Link
Weave	Weave is a toolkit for developing Generative AI applications.	Link
Llama Stack	Build Llama Apps.	Link

Multi API Access

Library	Description	Link
LiteLLM	Library to call 100+ LLM APIs in OpenAI format.	Link
AI Gateway	A Blazing Fast AI Gateway with integrated Guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API.	Link

Routers

Library	Description	Link
RouteLLM	Framework for serving and evaluating LLM routers - save LLM costs without compromising quality. Drop-in replacement for OpenAI's client to route simpler queries to cheaper models.	Link

Memory

Library	Description	Link
mem0	The Memory layer for your AI apps.	Link
Memoripy	An AI memory layer with short- and long-term storage, semantic clustering, and optional memory decay for context-aware applications.	Link

Interface

Library	Description	Link
Streamlit	A faster way to build and share data apps. Streamlit lets you transform Python scripts into interactive web apps in minutes	Link
Gradio	Build and share delightful machine learning apps, all in Python.	Link
AI SDK UI	Build chat and generative user interfaces.	Link
AI-Gradio	Create AI apps powered by various AI providers.	Link
Simpleaichat	Python package for easily interfacing with chat apps, with robust features and minimal code complexity.	Link
Chainlit	Build production-ready Conversational AI applications in minutes.	Link

Low Code

Library	Description	Link
LangFlow	LangFlow is a low-code app builder for RAG and multi-agent AI applications. It’s Python-based and agnostic to any model, API, or database.	Link

Cache

Library	Description	Link
GPTCache	A Library for Creating Semantic Cache for LLM Queries. Slash Your LLM API Costs by 10x 💰, Boost Speed by 100x. Fully integrated with LangChain and LlamaIndex.	Link

LLM RAG

Library	Description	Link
FastGraph RAG	Streamlined and promptable Fast GraphRAG framework designed for interpretable, high-precision, agent-driven retrieval workflows.	Link
Chonkie	RAG chunking library that is lightweight, lightning-fast, and easy to use.	Link
RAGChecker	A Fine-grained Framework For Diagnosing RAG.	Link
RAG to Riches	Build, scale, and deploy state-of-the-art Retrieval-Augmented Generation applications.	Link
BeyondLLM	Beyond LLM offers an all-in-one toolkit for experimentation, evaluation, and deployment of Retrieval-Augmented Generation (RAG) systems.	Link
SQLite-Vec	A vector search SQLite extension that runs anywhere!	Link
fastRAG	fastRAG is a research framework for efficient and optimized retrieval-augmented generative pipelines, incorporating state-of-the-art LLMs and Information Retrieval.	Link
FlashRAG	A Python Toolkit for Efficient RAG Research.	Link
Llmware	Unified framework for building enterprise RAG pipelines with small, specialized models.	Link
Rerankers	A lightweight unified API for various reranking models.	Link
Vectara	Build Agentic RAG applications.	Link

LLM Inference

Library	Description	Link
LLM Compressor	Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment.	Link
LightLLM	Python-based LLM inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.	Link
vLLM	High-throughput and memory-efficient inference and serving engine for LLMs.	Link
torchchat	Run PyTorch LLMs locally on servers, desktop, and mobile.	Link
TensorRT-LLM	TensorRT-LLM is a library for optimizing Large Language Model (LLM) inference.	Link
WebLLM	High-performance In-browser LLM Inference Engine.	Link

LLM Serving

Library	Description	Link
Langcorn	Serving LangChain LLM apps and agents automagically with FastAPI.	Link
LitServe	Lightning-fast serving engine for any AI model of any size. It augments FastAPI with features like batching, streaming, and GPU autoscaling.	Link

LLM Data Extraction

Library	Description	Link
Crawl4AI	Open-source LLM Friendly Web Crawler & Scraper.	Link
ScrapeGraphAI	A web scraping Python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.).	Link
Docling	Docling parses documents and exports them to the desired format with ease and speed.	Link
Llama Parse	GenAI-native document parser that can parse complex document data for any downstream LLM use case (RAG, agents).	Link
PyMuPDF4LLM	PyMuPDF4LLM library makes it easier to extract PDF content in the format you need for LLM & RAG environments.	Link
Crawlee	A web scraping and browser automation library.	Link
MegaParse	Parser for every type of document.	Link
ExtractThinker	Document Intelligence library for LLMs.	Link

LLM Data Generation

Library	Description	Link
DataDreamer	DataDreamer is a powerful open-source Python library for prompting, synthetic data generation, and training workflows.	Link
fabricator	A flexible open-source framework to generate datasets with large language models.	Link
Promptwright	Synthetic Dataset Generation Library.	Link
EasyInstruct	An Easy-to-use Instruction Processing Framework for Large Language Models.	Link

LLM Agents

Library	Description	Link
CrewAI	Framework for orchestrating role-playing, autonomous AI agents.	Link
LangGraph	Build resilient language agents as graphs.	Link
Agno	Build AI Agents with memory, knowledge, tools, and reasoning. Chat with them using a beautiful Agent UI.	Link
AutoGen	An open-source framework for building AI agent systems.	Link
Smolagents	Library to build powerful agents in a few lines of code.	Link
Pydantic AI	Python agent framework to build production grade applications with Generative AI.	Link
gradio-tools	A Python library for converting Gradio apps into tools that can be leveraged by an LLM-based agent to complete its task.	Link
Composio	Production Ready Toolset for AI Agents.	Link
Atomic Agents	Building AI agents, atomically.	Link
Memary	Open Source Memory Layer For Autonomous Agents.	Link
Browser Use	Make websites accessible for AI agents.	Link
OpenWebAgent	An Open Toolkit to Enable Web Agents on Large Language Models.	Link
Lagent	A lightweight framework for building LLM-based agents.	Link
LazyLLM	A Low-code Development Tool For Building Multi-agent LLMs Applications.	Link
Swarms	The Enterprise-Grade Production-Ready Multi-Agent Orchestration Framework.	Link
ChatArena	ChatArena is a library that provides multi-agent language game environments and facilitates research about autonomous LLM agents and their social interactions.	Link
Swarm	Educational framework exploring ergonomic, lightweight multi-agent orchestration.	Link
AgentStack	The fastest way to build robust AI agents.	Link
Archgw	Intelligent gateway for Agents.	Link
Flow	A lightweight task engine for building AI agents.	Link
AgentOps	Python SDK for AI agent monitoring.	Link
Langroid	Multi-Agent framework.	Link
Agentarium	Framework for creating and managing simulations populated with AI-powered agents.	Link

LLM Evaluation

Library	Description	Link
Ragas	Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications.	Link
Giskard	Open-Source Evaluation & Testing for ML & LLM systems.	Link
DeepEval	LLM Evaluation Framework	Link
Lighteval	All-in-one toolkit for evaluating LLMs.	Link
Trulens	Evaluation and Tracking for LLM Experiments	Link
PromptBench	A unified evaluation framework for large language models.	Link
LangTest	Deliver Safe & Effective Language Models. 60+ Test Types for Comparing LLM & NLP Models on Accuracy, Bias, Fairness, Robustness & More.	Link
EvalPlus	A rigorous evaluation framework for LLM4Code.	Link
FastChat	An open platform for training, serving, and evaluating large language model-based chatbots.	Link
judges	A small library of LLM judges.	Link
Evals	Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.	Link
AgentEvals	Evaluators and utilities for evaluating the performance of your agents.	Link
LLMBox	A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation.	Link
Opik	An open-source end-to-end LLM Development Platform which also includes LLM evaluation.	Link

LLM Monitoring

Library	Description	Link
Opik	An open-source end-to-end LLM Development Platform which also includes LLM monitoring.	Link
LangSmith	Provides tools for logging, monitoring, and improving your LLM applications.	Link
Weights & Biases (W&B)	W&B provides features for tracking LLM performance.	Link
Helicone	Open source LLM-Observability Platform for Developers. One-line integration for monitoring, metrics, evals, agent tracing, prompt management, playground, etc.	Link
Evidently	An open-source ML and LLM observability framework.	Link
Phoenix	An open-source AI observability platform designed for experimentation, evaluation, and troubleshooting.	Link
Observers	A Lightweight Library for AI Observability.	Link

LLM Prompts

Library	Description	Link
PCToolkit	A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models.	Link
Selective Context	Selective Context compresses your prompt and context to allow LLMs (such as ChatGPT) to process 2x more content.	Link
LLMLingua	Library for compressing prompts to accelerate LLM inference.	Link
betterprompt	Test suite for LLM prompts before pushing them to production.	Link
Promptify	Solve NLP Problems with LLMs & easily generate different NLP Task prompts for popular generative models like GPT, PaLM, and more with Promptify.	Link
PromptSource	PromptSource is a toolkit for creating, sharing, and using natural language prompts.	Link
DSPy	DSPy is the open-source framework for programming—rather than prompting—language models.	Link
Py-priompt	Prompt design library.	Link
Promptimizer	Prompt optimization library.	Link

LLM Structured Outputs

Library	Description	Link
Instructor	Python library for working with structured outputs from large language models (LLMs). Built on top of Pydantic, it provides a simple, transparent, and user-friendly API.	Link
XGrammar	An open-source library for efficient, flexible, and portable structured generation.	Link
Outlines	Robust (structured) text generation	Link
Guidance	Guidance is an efficient programming paradigm for steering language models.	Link
LMQL	A language for constraint-guided and efficient LLM programming.	Link
Jsonformer	A Bulletproof Way to Generate Structured JSON from Language Models.	Link

LLM Safety and Security

Library	Description	Link
JailbreakEval	A collection of automated evaluators for assessing jailbreak attempts.	Link
EasyJailbreak	An easy-to-use Python framework to generate adversarial jailbreak prompts.	Link
Guardrails	Adding guardrails to large language models.	Link
LLM Guard	The Security Toolkit for LLM Interactions.	Link
AuditNLG	AuditNLG is an open-source library that can help reduce the risks associated with using generative AI systems for language.	Link
NeMo Guardrails	NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.	Link
Garak	LLM vulnerability scanner	Link

LLM Embedding Models

Library	Description	Link
Sentence-Transformers	State-of-the-Art Text Embeddings	Link
Model2Vec	Fast State-of-the-Art Static Embeddings	Link
Text Embedding Inference	A blazing fast inference solution for text embeddings models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding, Ember, GTE and E5.	Link

Others

Library	Description	Link
Text Machina	A modular and extensible Python framework, designed to aid in the creation of high-quality, unbiased datasets to build robust models for MGT-related tasks such as detection, attribution, and boundary detection.	Link
LLM Reasoners	A library for advanced large language model reasoning.	Link
EasyEdit	An Easy-to-use Knowledge Editing Framework for Large Language Models.	Link
CodeTF	CodeTF: One-stop Transformer Library for State-of-the-art Code LLM.	Link
spacy-llm	This package integrates Large Language Models (LLMs) into spaCy, featuring a modular system for fast prototyping and prompting, and turning unstructured responses into robust outputs for various NLP tasks.	Link
pandas-ai	Chat with your database (SQL, CSV, pandas, polars, MongoDB, NoSQL, etc.).	Link
LLM Transparency Tool	An open-source interactive toolkit for analyzing internal workings of Transformer-based language models.	Link
Vanna	Chat with your SQL database. Accurate Text-to-SQL Generation via LLMs using RAG.	Link
mergekit	Tools for merging pretrained large language models.	Link
MarkLLM	An Open-Source Toolkit for LLM Watermarking.	Link
LLMSanitize	An open-source library for contamination detection in NLP datasets and Large Language Models (LLMs).	Link
Annotateai	Automatically annotate papers using LLMs.	Link
LLM Reasoner	Make any LLM think like OpenAI o1 and DeepSeek R1.	Link

⭐️ Star History

Please consider giving a star, if you find this repository useful.