What is the purpose of reload_dataloaders_every_epoch?

[isabellahuang]

I'm looking to train on chunks of my entire dataset at a time per epoch (preferably over every n epochs, but this is not yet implemented officially), since the size of my dataset exceeds my total RAM. I'd therefore like to update the data in the DataLoaders every epoch. From all the examples I've seen, the actual data content is saved in memory a class attribute (in the following code snippet, it's saved in self.mnist_train and self.mnist_val). It seems that train_dataloader() and val_dataloader() only read self.mnist_train and self.mnist_val into DataLoaders.

From my understanding, reload_dataloaders_every_epoch=True calls train_dataloader() and val_dataloader() at every epoch, but I don't see the point of doing this if self.mnist_test and self.mnist_train aren't actually being changed. How do I make train_dataloader() and test_dataloader() return new data every epoch?

class MyDataModule(pl.LightningDataModule):

    def __init__(
        self,
        batch_size: int = 32,
    ):
        super().__init__()
        dataset = MNIST(_DATASETS_PATH, train=True, download=True, transform=transforms.ToTensor())
        self.mnist_test = MNIST(_DATASETS_PATH, train=False, download=True, transform=transforms.ToTensor())
        self.mnist_train, self.mnist_val = random_split(dataset, [55000, 5000])
        self.batch_size = batch_size

    def train_dataloader(self):
        return DataLoader(self.mnist_train, batch_size=self.batch_size)

    def val_dataloader(self):
        return DataLoader(self.mnist_val, batch_size=self.batch_size)

    def test_dataloader(self):
        return DataLoader(self.mnist_test, batch_size=self.batch_size)

[awaelchli]

since the size of my dataset exceeds my total RAM

That's not unusual. Often datasets don't fit into the RAM and that's fine. DataLoaders are designed to asynchronously load the data from your hard disk into ram and then onto the GPU.

From my understanding, reload_dataloaders_every_epoch=True calls train_dataloader() and val_dataloader() at every epoch, but I don't see the point of doing this if self.mnist_test and self.mnist_train aren't actually being changed.

Yes but first of all this is a toy example and doesn't really do anything interesting. Second even though the dataset does not change, the dataloader will be constructed newly every epoch. You could return dataloader with a new or growing dataset every epoch or do something crazy like increase the batch size every epoch or turn shuffle on and off haha.

[awaelchli]

Example: cycle through N datasets:

class MyDataModule(pl.LightningDataModule):

    def __init__(
        self,
        datasets,
    ):
        super().__init__()
        self.datasets = datasets. # a list of datasets
        self.batch_size = batch_size

    def train_dataloader(self):
        dataset =self.datasets[self.current_epoch % len(self.datasets)]
        return DataLoader(dataset, batch_size=self.batch_size)


# important:
trainer = Trainer(reload_dataloaders_every_epoch=True)

[isabellahuang]

Thanks so much Adrian!