Applying GRPO to Qwen-0.5B-Instruct using GSM8K ends up outputting a low-performing model.

For context: I had just read and learned about GRPO last week. This week, I decided to apply this method by training Qwen-0.5B-Instruct on the GSM8K dataset. Using GRPOTrainer from TRL, I set 2 training epochs and reference model synch every 25 steps. I only used two reward functions: strict formatting (i.e., must follow <reasoning>…</reasoning><answer>…</answer> format) and accuracy (i.e., must output the correct answer).

However when I tried to ask it a simple question after training phase was done, it wasn't able to answer it. It just instead answers \n (newline) character. I checked the graphs of the reward function and they were "stable" at 1.0 towards the end of training.

Did I miss something? Would like to hear your thoughts. Thank you.