During the reproduction of AgentArk‘’s PAD method,I encountered several discrepancies between the paper and the open-source code, and I was hoping you could clarify a few points:
1.The requirements.txt file specifies transformers==4.46.3, but this version does not actually support Qwen3, even though your paper mentions using models related to it
2.The openrlhf/datasets module is missing, but there are multiple import statements referencing it in the code.
3.In your paper, the PRM (Process Reward Model) requires a two-stage training process, but the README only provides a script for a single stage.
4.PRM contrastive_projector wiring (prm/finetune2.py): The projection head is attached to the Trainer (self.contrastive_projector) rather than self.model. Since saving uses model.state_dict() and the HF Trainer optimizer typically only includes model.named_parameters(), the projector appears neither optimized nor checkpointed (and would be randomly re-initialized if training continues from the saved checkpoint). Is this intentional, or should the projector be registered on the model?
If possible, could you kindly share the complete/updated code and detailed execution scripts to help with the reproduction? Any guidance would be immensely appreciated!
During the reproduction of AgentArk‘’s PAD method,I encountered several discrepancies between the paper and the open-source code, and I was hoping you could clarify a few points:
1.The requirements.txt file specifies transformers==4.46.3, but this version does not actually support Qwen3, even though your paper mentions using models related to it
2.The openrlhf/datasets module is missing, but there are multiple import statements referencing it in the code.
3.In your paper, the PRM (Process Reward Model) requires a two-stage training process, but the README only provides a script for a single stage.
4.PRM contrastive_projector wiring (prm/finetune2.py): The projection head is attached to the Trainer (self.contrastive_projector) rather than self.model. Since saving uses model.state_dict() and the HF Trainer optimizer typically only includes model.named_parameters(), the projector appears neither optimized nor checkpointed (and would be randomly re-initialized if training continues from the saved checkpoint). Is this intentional, or should the projector be registered on the model?
If possible, could you kindly share the complete/updated code and detailed execution scripts to help with the reproduction? Any guidance would be immensely appreciated!