Four open-source projects dominate large language model fine-tuning today: Unsloth, Axolotl, TRL, and LLaMA-Factory. All four wrap the same underlying PyTorch and Hugging Face stack, but they diverge in where they concentrate engineering effort. Unsloth rewrites kernels for raw speed. Axolotl focuses on composable parallelism strategies. TRL defines the trainer APIs that the others build upon. LLaMA-Factory prioritizes breadth of model coverage and zero-code operation.

As open-weight models from Meta, Alibaba, Google, and others close the quality gap with proprietary APIs, enterprises increasingly fine-tune locally rather than pay per-token premiums — making the framework layer a direct cost lever, since GPU hours remain the dominant expense in any customization pipeline.

This comparison evaluates three axes that practitioners routinely encounter: training throughput, peak VRAM usage, and multi-GPU scaling.

Framework Overview

TRL serves as the reference implementation layer. It ships SFTTrainer, DPOTrainer, GRPOTrainer, KTOTrainer, RewardTrainer, and RLOOTrainer. Both Axolotl and LLaMA-Factory call into it internally. The current stable release line is v1.8.0.

Unsloth replaces portions of the modeling code with hand-written Triton kernels. Backpropagation steps are manually derived rather than autograd-generated. Hugging Face's own writeup notes that accuracy degradation is 0% versus standard QLoRA because no approximations are introduced.

Axolotl is a YAML-driven wrapper over Transformers, PEFT, TRL, Accelerate, and DeepSpeed. Its key differentiator is the composability of parallelism strategies, not kernel-level optimization.

LLaMA-Factory is documented in an ACL 2024 system demonstration paper and includes a Gradio web UI called LlamaBoard. The repository covers over 100 LLMs and VLMs.

Speed

Unsloth: Kernel-Level Gains on a Single GPU

Unsloth's published benchmarks show 2x training speed for Llama 3.1 8B and Llama 3.3 70B. The test setup used the Alpaca dataset, batch size 2, and gradient accumulation 4. QLoRA ran at rank 32 across all linear layers.

The results for Mixture-of-Experts (MoE) models are more pronounced. Unsloth fine-tuned unsloth/gpt-oss-20b-BF16 on an NVIDIA B200, reporting 712.33 ms per step at 8K context versus 5,226.86 ms for Transformers v5 — a 7.3x gap. At 4K context the gap narrows to 4.82x, and at 1K it is only 1.37x.

The trend direction is model-dependent. Unsloth's MoE documentation scopes this particular claim to gpt-oss, where speedup grows with sequence length, attributed to Flex Attention and the MoE kernels.

Qwen3-30B-A3B on B200 exhibits the opposite pattern. Its reported speedup falls from 1.7x at 1K context to 1.1x at 16K. Memory savings move in the reverse direction, rising from approximately 2% to 15%.

Qwen3-30B-A3B on H100 reaches up to 1.77x. GLM-4.7-Flash on RTX PRO 6000 reaches 2.1x. A collaboration with AMD measured Llama-3.1-8B LoRA SFT at 2.07 s/step, while TRL plus FlashAttention-2 took 2.87 s/step — a 1.39x gap with matching loss curves.

Axolotl: Borrowed Kernels, Native Parallelism

Axolotl added custom Triton kernels and autograd functions for LoRA in February 2025, explicitly citing Unsloth as inspiration. These are opt-in through lora_mlp_kernel, lora_qkv_kernel, and lora_o_kernel.

Recent release notes add SonicMoE LoRA support, delivering up to 1.45x speedup and 30% memory reduction over a grouped_mm baseline for Qwen3.5-35B-A3B 8-bit LoRA on a single H100 SXM.

Axolotl also ships FlashAttention 2/3/4, xFormers, Flex Attention, SageAttention, Liger Kernel, Cut Cross Entropy, and ScatterMoE.

TRL: The Baseline Everyone Measures Against

TRL typically serves as the reference point rather than the winner on raw single-GPU throughput. It compensates with a breadth of memory and speed levers documented in Reducing Memory Usage and Speeding Up Training. These levers include packing, padding-free batching, truncation, Liger Kernel, and vLLM sleep mode for GRPO. TRL also has a first-party Unsloth integration, so the two are not mutually exclusive.

LLaMA-Factory: Speed by Delegation

LLaMA-Factory does not write its own kernels. Instead, it exposes other frameworks' optimizations through configuration flags. Setting use_unsloth: true activates the Unsloth patch, with the project's changelog reporting 170% relative speed from that path. Unsloth's long-sequence training is listed at 117% speed and 50% memory. LLaMA-Factory also supports enable_liger_kernel: true and FlashAttention-2 via flash_attn: fa2.

VRAM

Reported Memory Floors

Unsloth publishes a VRAM requirements table sorted by parameter count. It lists 6 GB for an 8B model in 4-bit QLoRA and 41 GB for 70B. LoRA at 16-bit costs 22 GB and 164 GB for the same models respectively.

LLaMA-Factory's README hardware table covers the same 4-bit QLoRA regime, listing 6 GB at 7B, 24 GB at 30B, and 48 GB at 70B. Full bf16 fine-tuning of 70B is listed at 600 GB.

Both tables describe minimums. Batch size, sequence length, and optimizer choice all affect actual consumption.

Context Length as the Sharper Differentiator

Peak VRAM at a fixed context length matters less than the maximum context a given VRAM budget allows. Unsloth's context length benchmarks for Llama 3.1 8B QLoRA at rank 32 and batch size 1 are stark. Unsloth attributes this to its gradient checkpointing algorithm combined with Apple's Cut Cross Entropy. For Llama 3.3 70B on an 80 GB A100, it reports 89,389 tokens — compared to 6,916 for the FA2 baseline.

The MoE Memory Story

MoE training is where memory behavior has shifted most in 2026. Unsloth reports gpt-oss-20b fine-tuning within 12.8 GB, while Qwen3-30B-A3B at 16-bit LoRA requires 63 GB.

On its B200 gpt-oss run, Unsloth used 47.43 GB at 8K context where Transformers v5 used 73.80 GB. At 16K, Transformers v5 went out of memory while Unsloth used 55.13 GB.

The mechanism is a split-LoRA formulation. PEFT materializes the LoRA delta across all experts before the MoE matmul. Unsloth reorders the operations instead, which is mathematically identical but avoids the materialization step.

Axolotl addresses the same problem through MoE expert quantization, quantizing expert weights during model loading and freeing the original bf16 tensor immediately. This became necessary after a Transformers v5 change moved expert layers from nn.Linear to fused nn.Parameter 3D tensors, preventing bitsandbytes from quantizing them on load. Axolotl's docs report GLM-4.7-Flash QLoRA dropping from roughly 127 GiB to roughly 23 GiB reserved memory with quantize_moe_experts: true.

Multi-GPU

Multi-GPU is where the single-GPU ranking inverts. Unsloth's lead on a single GPU does not carry over.

Axolotl: The Deepest Parallelism Matrix

Axolotl's multi-GPU guide offers three mutually exclusive sharding strategies: DeepSpeed ZeRO stages 1 through 3, FSDP, and DDP. FSDP2 is the recommended path, with FSDP1 deprecated.

On top of those, its N-D Parallelism guide composes data, tensor, context, and expert parallelism through PyTorch's DeviceMesh. The documented support matrix confirms FSDP+TP, HSDP+TP, FSDP+CP, FSDP+TP+CP, and FSDP+EP. Two combinations are explicitly unsupported: expert parallelism cannot compose with TP or CP in v1, and pure DDP cannot compose with them either.

Axolotl's sequence parallelism uses the ring-flash-attention library. Its published H100 benchmark for Llama 3.1 8B QLoRA illustrates the tradeoff: context scales close to linearly, but throughput efficiency collapses. On 4090s at SP degree 8, the same benchmark records 0.88x speedup — training actually got slower while context reached 32,768 tokens.

Axolotl also supports multi-node training through torchrun and Ray.

TRL: Two Sequence-Splitting Backends

TRL's distributed training guide draws a clean distinction between two approaches. Context Parallelism uses Ring Attention on FSDP2, requiring Accelerate 1.11.0+, using cp_size with cp_backend="torch", and currently supporting SDPA only — FlashAttention is not supported on this path. Sequences must divide evenly by cp_size * 2.

Sequence Parallelism uses ALST/Ulysses on DeepSpeed, requiring DeepSpeed 0.18.1+ and Accelerate 1.12.0+. It uses sp_size with sp_backend="deepspeed" and works with FlashAttention-2, but is bounded by attention head count, requiring num_heads >= sp_size.

TRL's guidance is specific: Ring Attention suits 1M+ token sequences and limited network topology, while Ulysses suits NVLink or InfiniBand interconnects and sequences up to roughly 500k tokens. TRL's own Ring Attention benchmark fine-tuned Qwen3-8B across 1, 2, 4, and 8 H100 GPUs, with context lengths above 300k tokens becoming trainable at 8 GPUs.

LLaMA-Factory: Standard Engines with Megatron

LLaMA-Factory's distributed training docs cover DDP, DeepSpeed, and FSDP, including FSDP2 and Ray for single-node and multi-node runs. The docs also describe DeepSpeed AutoTP, which combines tensor parallelism with ZeRO.

Its most consequential 2025 addition was a Megatron-core training backend through mcore_adapter, opening a genuine large-scale pretraining path. The FSDP+QLoRA path fine-tunes a 70B model on two 24 GB GPUs — the cheapest documented route to 70B in this comparison.

The friction point is the interface itself. Distributed configuration lives in YAML and CLI, not in LlamaBoard. Teams that adopted LLaMA-Factory for its zero-code UI lose that property when scaling past one GPU.

Unsloth: The Open Gap

Unsloth's multi-GPU documentation states that multi-GPU works through Accelerate and DeepSpeed, giving access to FSDP and DDP. However, it also states the process is complex and requires manual setup, with official support still being announced. The practical route is accelerate launch train.py or torchrun --nproc_per_node N_GPUS train.py. For models too large for one GPU, device_map = "balanced" splits the model across devices.

Unsloth's PyPI listing marks multi-GPU as available with major improvements pending. The Studio changelog describes preliminary automatic multi-GPU allocation for inference and training as of March 2026.

Taken together, the position is clear: Unsloth supports multi-GPU but does not yet offer the composable parallelism matrix that Axolotl and TRL document.

Limitations of Each Framework

Unsloth breaks when tensor, context, or expert parallelism is needed as first-class configuration. It also breaks when a model is not in its supported list, since gains come from architecture-specific kernels.

Axolotl breaks on the learning curve. Users must configure FSDP2 versus DeepSpeed, SP degree, and divisibility constraints spanning GPU count, sequence length, and attention heads.

TRL breaks on defaults. It provides correct primitives rather than tuned ones — users must supply the Accelerate config, memory optimizations, and parallelism plan.

LLaMA-Factory breaks at the UI boundary. Its abstraction is effective up to one node but thin above it.

Practical Guidance

For a single consumer GPU with a supported architecture running LoRA or QLoRA, Unsloth is the strongest choice — the context-length headroom alone justifies it.

For two to eight GPUs with long context and full fine-tuning or RLHF pipelines, Axolotl is recommended. FSDP2 plus sequence parallelism is the best-documented path.

For custom training loops, novel post-training algorithms, or tight Hugging Face coupling, TRL is the foundation to build on, as it is the layer the others wrap.

For broadest model coverage, non-engineer operators, and fastest initial runs, LLaMA-Factory is optimal — with a transition to CLI when scaling.

These choices are not mutually exclusive. LLaMA-Factory can run Unsloth as a backend. TRL ships an Unsloth integration. Axolotl calls TRL trainers internally. The frameworks are visibly converging on each other's strengths — Unsloth adding multi-GPU, LLaMA-Factory adding Megatron, Axolotl adopting kernel-level tricks — which suggests that over the next cycle, differentiation may shift from raw performance toward developer ergonomics and integration breadth.

Source: MarkTechPost