Dataloader always drop last batch in PPO

[allanj]

trl/trl/trainer/ppo_trainer.py

Lines 387 to 393 in e23a541

	dataloader = torch.utils.data.DataLoader(
	dataset,
	batch_size=self.config.batch_size,
	collate_fn=data_collator,
	shuffle=True,
	drop_last=True,
	)

Is this expected?

I believe we should have an option for that?

[younesbelkada]

Hi @allanj Yes this is expected to avoid some issues we had in the past, please check out this PR: #154 for more details

[allanj]

Thanks. I just checked. Is it better to let the developer pass in the data loader in that case? XD. In my experiments, I prefer to go through all the batches.

Not sure if there are more elegant ways for me to do that

[younesbelkada]

Thanks!
I think in all cases you need to drop the last batch to avoid issues such as: #148 , this is because of the way PPOTrainer has been designed :/
To make sure you pass all your batches perhaps you can just have a dataset that has a multiple of batch_size examples and you add a single dummy instance to it at the end to that it gets dropped.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.