Introduction

Continual Learning is a field of machine learning where the data distribution changes through time. For instance, instead of learning to classify all animals in the world at once, the model learns to classify them 10 by 10. To experiment in this setting, the data management is quite different from usual machine learning.

Continuum is a Python based library that proposes various data loaders for Continual Learning scenarios. It is designed to prevent researchers from spending time on designing classical data loaders and eliminate time-consuming errors. With Continuum, it is possible to focus on the model design using multiple scenarios directly. It also includes tools fro logging algorithm results and compute continual learning metrics.

Our philosophy is inspired by UNIX's: we aim to do few things but do it well without too many assumptions about your research code. Continuum is designed to be pluggable seamlessly in any codebase.

Installation

The library can be installed easily thanks to pip.

%%capture
!pip install continuum

Create your First Scenario

One of the simplest scenarios that continuum proposes is the split MNIST.

This scenario consists of learning MNIST by part. We split MNIST into several parts that contain different classes, i.e. we learn incrementally new classes. In our case, we see at first only images of "0" and "1", then images of "2" and "3", etc. Continuum proposes to use the class ClassIncremental to split the MNIST dataset into tasks.

First, we create an dataset object that contains the MNIST dataset.

%%capture
from six.moves import urllib    
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
urllib.request.install_opener(opener)
from continuum.datasets import MNIST   # Import the dataset
dataset = MNIST(".", download=True, train=True)

Then, we create a scenario of 5 tasks containing each the data of two classes.

continuum_scenario.png

from continuum import ClassIncremental # Import the scenarios template

# Create the scenario object. 
scenario = ClassIncremental(
    dataset,
    increment=2  # each new task contains, 2 classes
)

print("******************************************")
print(f"Number of classes: {scenario.nb_classes}.")
print(f"Number of tasks: {scenario.nb_tasks}.")
print("******************************************")
******************************************
Number of classes: 10.
Number of tasks: 5.
******************************************

The scenario object can be iterate into the 5 TaskSets that can be used with a Pytorch dataloader.

Each batch contains three kinds of tensors:

  • x: the pixels of the images
  • y: the labels of the images (i.e. 0, 1, 2, 3, 4, etc.)
  • t: the task identifiers of the images.

Screen Shot 2021-03-02 at 10.22.52.png

from torch.utils.data import DataLoader
import matplotlib.pyplot as plt
fig, axs = plt.subplots(1, len(scenario), figsize=(15,3))

for task_id, taskset in enumerate(scenario):

    # plot samples for each tasks
    ind_plot = 100 + len(scenario)*10 + task_id+1
    plt.subplot(ind_plot)
    taskset.plot(nb_samples=25)
    plt.title(f"Samples \n task {task_id}")

    # convert taskset into a the data loader for one task
    loader = DataLoader(taskset, batch_size=10)
    for x, y, t in loader:
        # x: data, y: label, t:task label
        # Train your model here
        print("**********************")
        print(f"Task       : {task_id}")
        print(f"Data Shape : {x.shape}")
        print(f"Labels     : {y}")
        print(f"Task label : {t}")

        break # we do not want to iterate through all the data in this example
**********************
Task       : 0
Data Shape : torch.Size([10, 1, 28, 28])
Labels     : tensor([0, 1, 1, 1, 1, 0, 1, 1, 0, 0])
Task label : tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
**********************
Task       : 1
Data Shape : torch.Size([10, 1, 28, 28])
Labels     : tensor([2, 3, 3, 3, 2, 2, 3, 2, 3, 3])
Task label : tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
**********************
Task       : 2
Data Shape : torch.Size([10, 1, 28, 28])
Labels     : tensor([5, 4, 4, 5, 4, 4, 5, 5, 4, 4])
Task label : tensor([2, 2, 2, 2, 2, 2, 2, 2, 2, 2])
**********************
Task       : 3
Data Shape : torch.Size([10, 1, 28, 28])
Labels     : tensor([6, 7, 6, 7, 6, 6, 7, 6, 7, 7])
Task label : tensor([3, 3, 3, 3, 3, 3, 3, 3, 3, 3])
**********************
Task       : 4
Data Shape : torch.Size([10, 1, 28, 28])
Labels     : tensor([9, 8, 9, 9, 8, 9, 8, 9, 9, 8])
Task label : tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4])

Note that you can index the scenario object to select a specific taskset or set of tasksets:

task_set = scenario[1]  # Second task
task_set = scenario[:3]  # First three tasks
task_set = scenario[::2]  # All even tasks
task_set = scenario[:]  # All tasks

The ClassIncremental scenario, as most proposed scenarios, can accept different datasets than MNIST. Here is a list of the existing datasets:

Name Nb classes Image Size Automatic Download Type
MNIST 10 28x28x1 YES Images
Fashion MNIST 10 28x28x1 YES Images
KMNIST 10 28x28x1 YES. Images
EMNIST 10 28x28x1 YES Images
QMNIST 10 28x28x1 YES Images
MNIST Fellowship 30 28x28x1 YES Images
CIFAR10 10 32x32x3 YES Images
CIFAR100 100 32x32x3 YES Images
CIFAR Fellowship 110 32x32x3 YES Images
ImageNet100 100 224x224x3 NO Images
ImageNet1000 1000 224x224x3 NO Images
CORe50 50 224x224x3 YES Images
CORe50-v2-79 50 224x224x3 YES Images
CORe50-v2-196 50 224x224x3 YES Images
CORe50-v2-391 50 224x224x3 YES Images
Synbols 50 32x32x3 YES Images
MultiNLI 5 YES Text
Stream51 51 ?x?x3 YES Images
PascalVOC-2012 20 + 1 ?x?x3 YES Segmentation

This list may not be up-to-date, as new datasets are released regularly. Furthermore, you can easily add your own dataset with only a few lines of code and benefit from the Continuum ecosystem: Adding Your Own Datasets ).

The most recent addition (at the time of writing) to Continuuum is PascalVOC-2012: a segmentation dataset where images are partially labelized, a challenging setting:

image.png

ClassIncremental is not the only scenario we propose. For example, the InstanceIncremental scenario consists of learning several tasks with the same classes, but the task data are always different.

Famous examples of instance incremental scenarios are permuted MNIST and rotated MNIST.

In permuted MNIST, each task contains all MNIST data with a fixed permutation of pixels, but each task has a different permutation. In the following code snippet, we create a 5 tasks permuted MNIST scenario. The scenario created with class Permutation can be used like previously with the ClassIncremental scenario.

NB: The Permutation scenario is a scpecial case of InstanceIncremental scenario.

from continuum import Permutations
from continuum.datasets import MNIST

train_dataset = MNIST(".", download=True, train=True)
train_scenario = Permutations(cl_dataset=dataset, nb_tasks=5)

# create test scenario with all test data in a single tasks
test_dataset = MNIST(".", download=True, train=True)
test_scenario = Permutations(cl_dataset=dataset, nb_tasks=5)

Logging with Continuum

Continuum also proposes tools to log algorithms' performance and compute easily various continual learning metrics.

import torch
from continuum.metrics import Logger 
from torch.utils.data import DataLoader

logger = Logger()

# training
for taskset in train_scenario:
    loader = DataLoader(taskset, batch_size=64)
    for (x,y,t) in loader:
        #predictions = model(x,y,t) # if we had a model, we could get a prediction
        # we will assume here the model is perfect, therefore,
        # we replace the prediction by the expected labels
        preds = y
        logger.add([preds, y, t])
    print(f"Online accuracy on train: {logger.online_accuracy*100} %")

# evaluation
for taskset in test_scenario:
    loader = DataLoader(taskset, batch_size=64)
    for (x,y,t) in loader:
        #once again we have perfect prediction
        preds = y
        logger.add([preds, y, t], subset='test')
    print(f"Test Accuracy: {logger.accuracy*100} %")
Online accuracy on train: 100.0 %
Online accuracy on train: 100.0 %
Online accuracy on train: 100.0 %
Online accuracy on train: 100.0 %
Online accuracy on train: 100.0 %
Test Accuracy: 100.0 %
Test Accuracy: 100.0 %
Test Accuracy: 100.0 %
Test Accuracy: 100.0 %

The continuum logger can be used to calculate several other continual learning metrics:

Name Code ↑ / ↓
Accuracy accuracy

Accuracy A accuracy_A

Backward Transfer backward_transfer

Positive Backward Transfer positive_backward_transfer

Remembering remembering

Forward Transfer forward_transfer

Forgetting forgetting

Model Size Growth model_size_growth

Online Accuracy online_accuracy

(Details here)

Conclusion

Continuum is an open-source project which aims at simplifying data management for continual learning algorithms. It aims at covering the various types of scenarios in the continual learning research field, and it is developed such as being easily adaptable to specific needs.

Continuum is made to save development time, reduce code size in continual project, and avoid classical data loader bugs.

Moreover, we believe that using a common plateform for continual learning experiments will benefit the reproducibility problems that researchers often experiment with. We believe by providing the right tools, Continuum can enable better research in Continual Learning: several research papers already used this framework, and the Continual Learning workshop of CVPR 2021 will hold a competition where the participants will also have to use Continuum.