MobileTransformers - An On-Device LLM PEFT Framework for Fine-Tuning and Inference

Project information

MobileTransformers is a lightweight, modular framework based on ONNX Runtime for running and adapting large language models (LLMs) directly on mobile and edge devices. It supports on-device fine-tuning (PEFT), efficient inference, quantization, weight merging, and direct inference from merged models. It includes advanced generation techniques like Retrieval-Augmented Generation (RAG) with vector databases and KV-cache with embedding reuse. The framework also provides export scripts for converting custom Huggingface SLM/LLM for on-device deployment with custom PEFT methods.

16 Commits
2 Branches
0 Tags

README
Creative Commons Attribution Non Commercial 4.0 International

Created on

January 10, 2025

Copy SSH clone URLgit@gitlab.fri.uni-lj.si:lrk/mobiletransformers.git
Copy HTTPS clone URLhttps://gitlab.fri.uni-lj.si/lrk/mobiletransformers.git

Loading