Running large language models on a single GPU for throughput-oriented scenarios.
-
Updated
Oct 28, 2024 - Python
Running large language models on a single GPU for throughput-oriented scenarios.
Run Mixtral-8x7B models in Colab or consumer desktops
Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s agent, ~41 fast, 512K context. Plus Inkling-2-Mini (own engine: 512K on 24 GB, vision, MTP drafter), GLM-5.3-Flash, Qwen3.8, LFM3-Omni. MoE experts tiered across VRAM, RAM and NVMe (Xolotl Direct). OpenAI + Anthropic API.
A QoE-Oriented Computation Offloading Algorithm based on Deep Reinforcement Learning (DRL) for Mobile Edge Computing (MEC) | This algorithm captures the dynamics of the MEC environment by integrating the Dueling Double Deep Q-Network (D3QN) model with Long Short-Term Memory (LSTM) networks.
ZO2 (Zeroth-Order Offloading): Full Parameter Fine-Tuning 175B LLMs with 18GB GPU Memory [COLM2025]
LLM Inference on consumer devices
dpdk infrastructure for software acceleration. Currently working on RX and ACL pre-filter
Run 50-280 GB MoE LLMs (DeepSeek V4/V4.1 Flash, GLM-5.3-Flash, Qwen3.8-Flash-Next, Ornith) on one consumer GPU: experts spread over VRAM, RAM and NVMe SSD. Browser chat + OpenAI/Anthropic APIs. CUDA, ROCm, DGX Spark.
A collection of tests for the Open vSwitch HW offload.
A Dynamic Programming Offloading Algorithm for Mobile Cloud Computing
LeapIO: Efficient and Portable Virtual NVMe Storage on ARM SoCs (ASPLOS'20)
Monero hardware wallet protocol implementation for Trezor, agent
A framework for IoT devices to offload tasks to the cloud, resulting in efficient computation and decreased cloud costs.
A lightweight framework that enables serverless users to reduce their bills by harvesting non-serverless compute resources such as their VMs, on-premise servers, or personal computers.
30 tok/s for 20B MoE on 8 GB VRAM. Flat throughput to 32K context. Native MXFP4 + GGUF Q4_K/Q5_K/Q6_K via ggml CUDA kernels — zero dequant. Expert offloading for models that don't fit in GPU memory.
The container-based cloud platform for mobile code offloading
Code for paper "Real-time Neural Network Inference on Extremely Weak Devices: Agile Offloading with Explainable AI" (MobiCom'22)
Oscar: Elastic LLM Serving for Large LLMs on Apple Silicon
Monero wallet Trezor integration documentation
To associate your repository with the offloading topic, visit your repo's landing page and select "manage topics."