Fine-tuning didn't work after ONNX export
I fine-tuned the model, but it didn’t work after exporting to ONNX format. However, the agentic UI is working well with the original FP16 version. Any help with the export would be greatly appreciated.
~> Demo: https://huggingface.co/spaces/harlley/basic-agentic-ui
Hi
@harlley
When you run the ONNX model, does it output specific error messages (e.g., Shape mismatch), or does it output "gibberish" text?
If possible , can you also please share the exact code used to export the model to ONNX , Library versions and any other steps which will help us to understand the issue better and repro from our end once .
Thanks
Hi @harlley
When you run the ONNX model, does it output specific error messages (e.g., Shape mismatch), or does it output "gibberish" text?
If possible , can you also please share the exact code used to export the model to ONNX , Library versions and any other steps which will help us to understand the issue better and repro from our end once .
Thanks
Yes, the raw output is showing invalid tokens:
[Worker] Model loaded!
worker.ts:52 [Worker] initModel completed
worker.ts:59 [Worker] ========== GENERATE ==========
worker.ts:60 [Worker] Messages: [
{
"role": "developer",
"content": "You are a model that can do function calling with the following functions"
},
{
"role": "user",
"content": "what is the square color?"
}
]
worker.ts:72 [Worker] Input token count: 153
worker.ts:85 [Worker] Raw output: <unused32><start_function_call><unused23><pad><unused18><unused30><unused45><eos>
It seems the quantization corrupts the model
This was the notebook to export to ONNX
https://colab.research.google.com/drive/1QBZE22t3y-x1981XbNKEqOKH4EK0ff_N?usp=sharing
I believe this is due to FunctionGemma model which uses custom control tokens that needs to be properly defined in the tokenizer. When you exported to ONNX, the mapping between these semantic tokens and their underlying IDs got lost or wasn't properly configured . As the model was trained with these special tokens, so it knows when to use them, but during ONNX export, the tokenizer could have lost the mapping.
Can you check if the reference model (onnx-community/functiongemma-270m-it-ONNX) has these tokens properly defined and make sure 6 function tokens are added with their correct IDs?
Reference to control tokens : https://ai.google.dev/gemma/docs/functiongemma/formatting-and-best-practices#control-tokens
Thanks