English
Content on This Page

Model Fine-Tuning ​

Fine-tuning is to train a pre-trained model on a dataset of a specific task, so that the trained model can enhance its capability on the training dataset while maintaining the original common capability. This technology allows you to use the most advanced models without training the them from scratch, reducing the cost of using computing resources. This document explains how to use openMind Library to fine-tune a model.

PyTorch Model Fine-Tuning ​

Transformers is a mainstream suite for PyTorch model development. Based on the mainstream ecosystem libraries on PyTorch, openMind Library provides you with one-click training startup commands, helping you quickly implement the entire process of data processing, multi-site weight loading, low-parameter fine-tuning, quantization adaptation, fine-tuning, and training tracing.

The following features are supported:

  • Fine-tuning algorithm: SFT
  • Efficient parameter fine-tuning algorithms: Full, LoRA, and QLoRA
  • Acceleration: npu_fusion_attention and npu_rms_norm
  • Training monitoring: SwanLab
  • Distributed training: single-node single-card, single-node multi-cardd, DeepSpeed
  • Export: LoRA weight merge

MindSpore Model Fine-Tuning ​

Mindformers is a mainstream suite for MindSpore model development. To facilitate model fine-tuning, openMind Library encapsulates the Auto Classes, Trainer, and TrainingArguments APIs in a unified manner.

  • The following table lists the Auto Classes APIs encapsulated by openMind Library. openMind library keeps the other Auto Classes APIs unchanged. To use the APIs, you can import them from the Mindformers.

    APIMindSpore
    AutoConfig✅
    AutoTokenizer✅
    AutoProcessor✅
    AutoImageProcessor✅
    AutoFeatureExtractor❌
    AutoModel✅
    AutoModelForCausalLM✅
    AutoModelForSequenceClassification✅

    For details about the parameters of the Auto Class API, see the Auto Class APIs provided by openMind Library.

  • For the MindSpore framework, MindSpore framework users can use the Trainer and TrainingArguments provided by openMind Library for fine-tuning while retaining the original usage habits of Mindformers. For details about the supported parameters, see the API Reference of Trainer and TrainingArguments provided by openMind Library.

The sub-sections explain how to use openMind to fine-tune PyTorch and MindSpore models.