How to convert fine-tuned .pt to huggingface .safetensors

[yuzhenmao]

Hi, I have successfully fine-tuned the Llama 3.1 model and obtained the fine-tuned checkpoint in the format hf_model_000*_2.pt. However, I haven’t been able to find a solution for converting the .pt file into the .safetensors format used in Huggingface. I have noticed some similar issues: #878 and #832. I would greatly appreciate any guidance on how to achieve this.

[felipemello1]

hey @yuzhenmao , I am sorry that you are having issues with it. We are about to land this PR that will solve this problem for our users: #2074

Is it ok if you wait until the EoD and retrain the model? If you need it right away or retraining is a big lift, then you can check this similar issue and follow the steps: #2048

[yuzhenmao]

Hi @felipemello1, thank you for the reply! I can wait until the EoD. Please keep me posted, thanks!

[felipemello1]

hey @yuzhenmao , PR is merged: #2074

Now it should be much easier to use vllm/huggingface. Instructions are in the readme.

We will update the docs soon. Let us know if you find any issues and thanks for your patience :).

[yuzhenmao]

Hi @felipemello1, thank you so much for your help! I have a quick question regarding the PR: In section 3.1 (Using Hugging Face), if an alternative approach involves something similar to 3.2—specifically, deleting adapter_model.safetensors from the folder—will Hugging Face then automatically use ft-model-00001-of-00001.safetensors?

Thanks again!

[felipemello1]

Np! Please feel free to ask more or reopen the issue. If you want to use the model (not adapter) with hf, there’s no need to delete the adapter. Hugginface will use the file index.safetensor.json to know which ckpt to load @yuzhenmao

[felipemello1]

@yuzhenmao , I updated the PR description. There was an issue with the file name that was fixed today. You can use it with huggingface like this:

from transformers import AutoModelForCausalLM, AutoTokenizer

# Define the model and adapter paths
trained_model_path = "/tmp/torchtune/llama3_2_1B/full_single_device/base_model"

model = AutoModelForCausalLM.from_pretrained(
    pretrained_model_name_or_path=trained_model_path,
)

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(trained_model_path, safetensors=True)


# Function to generate text
def generate_text(model, tokenizer, prompt, max_length=50):
    inputs = tokenizer(prompt, return_tensors="pt")
    outputs = model.generate(**inputs, max_length=max_length)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)


prompt = "Complete the sentence: 'Once upon a time...'"
print("Base model output:", generate_text(model, tokenizer, prompt))

[yuzhenmao]

Hi @felipemello1, thank you so much for letting me know the update!