quapy.classification package

Submodules

quapy.classification.calibration module

class quapy.classification.calibration.BCTSCalibration(classifier, val_split=5, n_jobs=None, verbose=False, random_state=None)

Bases: RecalibratedProbabilisticClassifierBase

Applies the Bias-Corrected Temperature Scaling (BCTS) calibration method from abstention.calibration, as defined in Alexandari et al. paper:

Parameters:

• classifier – a scikit-learn probabilistic classifier

• val_split – indicate an integer k for performing kFCV to obtain the posterior prevalences, or a float p in (0,1) to indicate that the posteriors are obtained in a stratified validation split containing p% of the training instances (the rest is used for training). In any case, the classifier is retrained in the whole training set afterwards. Default value is 5.

• n_jobs – indicate the number of parallel workers (only when val_split is an integer)

• verbose – whether or not to display information in the standard output

• random_state – controls the shuffling of the train/validation split when val_split is a float (has no effect when val_split is an integer); pass an int for reproducible splits

class quapy.classification.calibration.NBVSCalibration(classifier, val_split=5, n_jobs=None, verbose=False, random_state=None)

Bases: RecalibratedProbabilisticClassifierBase

Applies the No-Bias Vector Scaling (NBVS) calibration method from abstention.calibration, as defined in Alexandari et al. paper:

Parameters:

• classifier – a scikit-learn probabilistic classifier

• val_split – indicate an integer k for performing kFCV to obtain the posterior prevalences, or a float p in (0,1) to indicate that the posteriors are obtained in a stratified validation split containing p% of the training instances (the rest is used for training). In any case, the classifier is retrained in the whole training set afterwards. Default value is 5.

• n_jobs – indicate the number of parallel workers (only when val_split is an integer)

• verbose – whether or not to display information in the standard output

• random_state – controls the shuffling of the train/validation split when val_split is a float (has no effect when val_split is an integer); pass an int for reproducible splits

class quapy.classification.calibration.RecalibratedProbabilisticClassifier

Bases: object

Abstract class for (re)calibration method from abstention.calibration, as defined in Alexandari, A., Kundaje, A., & Shrikumar, A. (2020, November). Maximum likelihood with bias-corrected calibration is hard-to-beat at label shift adaptation. In International Conference on Machine Learning (pp. 222-232). PMLR.:

class quapy.classification.calibration.RecalibratedProbabilisticClassifierBase(classifier, calibrator, val_split=5, n_jobs=None, verbose=False, random_state=None)

Bases: BaseEstimator, RecalibratedProbabilisticClassifier

Applies a (re)calibration method from abstention.calibration, as defined in Alexandari et al. paper.

Parameters:

• classifier – a scikit-learn probabilistic classifier

• calibrator – the calibration object (an instance of abstention.calibration.CalibratorFactory)

• val_split – indicate an integer k for performing kFCV to obtain the posterior probabilities, or a float p in (0,1) to indicate that the posteriors are obtained in a stratified validation split containing p% of the training instances (the rest is used for training). In any case, the classifier is retrained in the whole training set afterwards. Default value is 5.

• n_jobs – indicate the number of parallel workers (only when val_split is an integer); default=None

• verbose – whether or not to display information in the standard output

• random_state – controls the shuffling of the train/validation split when val_split is a float (has no effect when val_split is an integer); pass an int for reproducible splits

property classes_

Returns the classes on which the classifier has been trained on

Returns:

array-like of shape (n_classes)

fit(X, y)

Fits the calibration for the probabilistic classifier.

Parameters:

• X – array-like of shape (n_samples, n_features) with the data instances

• y – array-like of shape (n_samples,) with the class labels

Returns:

self

fit_cv(X, y)

Fits the calibration in a cross-validation manner, i.e., it generates posterior probabilities for all training instances via cross-validation, and then retrains the classifier on all training instances. The posterior probabilities thus generated are used for calibrating the outputs of the classifier.

Parameters:

• X – array-like of shape (n_samples, n_features) with the data instances

• y – array-like of shape (n_samples,) with the class labels

Returns:

self

fit_tr_val(X, y)

Fits the calibration in a train/val-split manner, i.e.t, it partitions the training instances into a training and a validation set, and then uses the training samples to learn classifier which is then used to generate posterior probabilities for the held-out validation data. These posteriors are used to calibrate the classifier. The classifier is not retrained on the whole dataset.

Parameters:

• X – array-like of shape (n_samples, n_features) with the data instances

• y – array-like of shape (n_samples,) with the class labels

Returns:

self

predict(X)

Predicts class labels for the data instances in X

Parameters:

X – array-like of shape (n_samples, n_features) with the data instances

Returns:

array-like of shape (n_samples,) with the class label predictions

predict_proba(X)

Generates posterior probabilities for the data instances in X

Parameters:

X – array-like of shape (n_samples, n_features) with the data instances

Returns:

array-like of shape (n_samples, n_classes) with posterior probabilities

class quapy.classification.calibration.TSCalibration(classifier, val_split=5, n_jobs=None, verbose=False, random_state=None)

Bases: RecalibratedProbabilisticClassifierBase

Applies the Temperature Scaling (TS) calibration method from abstention.calibration, as defined in Alexandari et al. paper:

Parameters:

• classifier – a scikit-learn probabilistic classifier

• val_split – indicate an integer k for performing kFCV to obtain the posterior prevalences, or a float p in (0,1) to indicate that the posteriors are obtained in a stratified validation split containing p% of the training instances (the rest is used for training). In any case, the classifier is retrained in the whole training set afterwards. Default value is 5.

• n_jobs – indicate the number of parallel workers (only when val_split is an integer)

• verbose – whether or not to display information in the standard output

• random_state – controls the shuffling of the train/validation split when val_split is a float (has no effect when val_split is an integer); pass an int for reproducible splits

class quapy.classification.calibration.TemperatureScalingFromLogits(bias_corrected=False, verbose=False)

Bases: BaseEstimator

Calibrates a matrix of logits by learning a temperature-scaling mapping with the calibration methods from abstention.calibration.

This estimator is useful when the inputs are already logits produced by a pretrained classifier, and the goal is to transform them directly into calibrated posterior probabilities without retraining the underlying model.

Parameters:

• bias_corrected – if True, uses Bias-Corrected Temperature Scaling (BCTS); otherwise, uses standard Temperature Scaling (TS)

• verbose – whether the underlying calibrator should display progress information

fit(X, y)

Fits the logits calibrator.

Parameters:

• X – array-like of shape (n_samples, n_classes) containing logits

• y – array-like of shape (n_samples,) containing class labels

Returns:

self

predict(X)

Predicts class labels after calibration.

Parameters:

X – array-like of shape (n_samples, n_classes) containing logits

Returns:

array-like of shape (n_samples,) with class label predictions

predict_proba(X)

Converts logits into calibrated posterior probabilities.

Parameters:

X – array-like of shape (n_samples, n_classes) containing logits

Returns:

array-like of shape (n_samples, n_classes) with calibrated posterior probabilities

class quapy.classification.calibration.VSCalibration(classifier, val_split=5, n_jobs=None, verbose=False, random_state=None)

Bases: RecalibratedProbabilisticClassifierBase

Applies the Vector Scaling (VS) calibration method from abstention.calibration, as defined in Alexandari et al. paper:

Parameters:

• classifier – a scikit-learn probabilistic classifier

• val_split – indicate an integer k for performing kFCV to obtain the posterior prevalences, or a float p in (0,1) to indicate that the posteriors are obtained in a stratified validation split containing p% of the training instances (the rest is used for training). In any case, the classifier is retrained in the whole training set afterwards. Default value is 5.

• n_jobs – indicate the number of parallel workers (only when val_split is an integer)

• verbose – whether or not to display information in the standard output

• random_state – controls the shuffling of the train/validation split when val_split is a float (has no effect when val_split is an integer); pass an int for reproducible splits

quapy.classification.labelshift module

class quapy.classification.labelshift.LabelShiftedClassifier(classifier, quantifier, weight_epsilon=0.0001)

Bases: BaseEstimator

Adapts a classifier to a shifted (unlabelled) target distribution by means of a quapy quantifier.

Given a base classifier h and a quantifier q, this wrapper: (i) at fit time, fits q on the labelled training data and keeps a copy of this data; (ii) at prediction time, for a given batch of (unlabelled) target instances, estimates the importance weights \(w_y=Q(y)/P(y)\) that would explain the shift between the training distribution and the target one (directly, if q is an quapy.method.aggregative.ImportanceWeightQuantifier, or otherwise by dividing q’s prevalence estimate by the training prevalence); and (iii) retrains a fresh copy of h, reweighted according to w via class_weight, on the original training data, before delegating the classification of the target instances to this adapted classifier.

Note that a fresh copy of h is retrained for every batch of target instances passed to predict, predict_proba, or get_classifier, since the adaptation is specific to that batch’s estimated shift; q and its own internal classifier (if any) are otherwise left untouched. h itself is required to accept a class_weight parameter in its constructor (e.g., sklearn.linear_model.LogisticRegression).

Parameters:

• classifier – a scikit-learn classifier that accepts a class_weight parameter in its constructor

• quantifier – a quapy quantifier, used to estimate the target prevalence (or, if it is an quapy.method.aggregative.ImportanceWeightQuantifier, the importance weights directly)

• weight_epsilon – float, a small positive floor applied to the estimated importance weights (default 1e-4), so that a class with training support is never assigned a class_weight of exactly 0 (which would silently drop it from the retraining loss)

fit(X, y)

Fits the internal quantifier on the training data, and stores the training data for the subsequent, per-target-batch, classifier adaptation.

Parameters:

• X – array-like of shape (n_samples, n_features) with the training instances

• y – array-like of shape (n_samples,) with the class labels

Returns:

self

get_classifier(X)

Returns a fresh classifier instance, sharing classifier’s hyperparameters, retrained on the original training data with class_weight set according to the importance weights estimated for the given target instances. The classifier passed at construction time, and the one used internally by the quantifier, are left untouched.

Parameters:

X – array-like of shape (n_samples, n_features) with the (unlabelled) target instances

Returns:

a fitted scikit-learn classifier

get_importance_weights(X)

Estimates the vector of importance weights \(w_y=Q(y)/P(y)\) that would explain the shift between the training distribution and the distribution of the given target instances. This method does not mutate any internal state, so it is safe to call concurrently for different batches of target instances.

Parameters:

X – array-like of shape (n_samples, n_features) with the (unlabelled) target instances

Returns:

np.ndarray of shape (n_classes,)

predict(X)

Adapts the classifier to the target instances in X, and returns its label predictions for X. Adaptation implies retraining. If you intend to generate inferences over many samples from the same suspected distribution, then call the get_classifier method once, and use the returned classifier over such samples.

Parameters:

X – array-like of shape (n_samples, n_features) with the (unlabelled) target instances

Returns:

array-like of shape (n_samples,) with the class label predictions

predict_proba(X)

Adapts the classifier to the target instances in X, and returns its posterior probabilities for X. Adaptation implies retraining. If you intend to generate inferences over many samples from the same suspected distribution, then call the get_classifier method once, and use the returned classifier over such samples.

Parameters:

X – array-like of shape (n_samples, n_features) with the (unlabelled) target instances

Returns:

array-like of shape (n_samples, n_classes) with posterior probabilities

quapy.classification.methods module

class quapy.classification.methods.LowRankLogisticRegression(n_components=100, **kwargs)

Bases: BaseEstimator

An example of a classification method (i.e., an object that implements fit, predict, and predict_proba) that also generates embedded inputs (i.e., that implements transform), as those required for quapy.method.neural.QuaNet. This is a mock method to allow for easily instantiating quapy.method.neural.QuaNet on array-like real-valued instances. The transformation consists of applying sklearn.decomposition.TruncatedSVD while classification is performed using sklearn.linear_model.LogisticRegression on the low-rank space.

Parameters:

• n_components – the number of principal components to retain

• kwargs – parameters for the Logistic Regression classifier

fit(X, y)

Fit the model according to the given training data. The fit consists of fitting TruncatedSVD and then LogisticRegression on the low-rank representation.

Parameters:

• X – array-like of shape (n_samples, n_features) with the instances

• y – array-like of shape (n_samples, n_classes) with the class labels

Returns:

self

get_params()

Get hyper-parameters for this estimator.

Returns:

a dictionary with parameter names mapped to their values

predict(X)

Predicts labels for the instances X embedded into the low-rank space.

Parameters:

X – array-like of shape (n_samples, n_features) instances to classify

Returns:

a numpy array of length n containing the label predictions, where n is the number of instances in X

predict_proba(X)

Predicts posterior probabilities for the instances X embedded into the low-rank space.

Parameters:

X – array-like of shape (n_samples, n_features) instances to classify

Returns:

array-like of shape (n_samples, n_classes) with the posterior probabilities

set_params(**params)

Set the parameters of this estimator.

Parameters:

parameters – a **kwargs dictionary with the estimator parameters for Logistic Regression and eventually also n_components for TruncatedSVD

transform(X)

Returns the low-rank approximation of X with n_components dimensions, or X unaltered if n_components >= X.shape[1].

Parameters:

X – array-like of shape (n_samples, n_features) instances to embed

Returns:

array-like of shape (n_samples, n_components) with the embedded instances

class quapy.classification.methods.MockClassifierFromPosteriors

Bases: BaseEstimator

Mock classifier that bypasses classifier training when the input instances are already posterior probabilities produced by a pretrained probabilistic classifier.

Parameters:

X – arrays of shape (n_samples, n_classes) are interpreted as posterior probabilities

fit(X, y)

predict(X)

predict_proba(X)

quapy.classification.neural module

class quapy.classification.neural.CNNnet(vocabulary_size, n_classes, embedding_size=100, hidden_size=256, repr_size=100, kernel_heights=[3, 5, 7], stride=1, padding=0, drop_p=0.5)

Bases: TextClassifierNet

An implementation of quapy.classification.neural.TextClassifierNet based on Convolutional Neural Networks.

Parameters:

• vocabulary_size – the size of the vocabulary

• n_classes – number of target classes

• embedding_size – the dimensionality of the word embeddings space (default 100)

• hidden_size – the dimensionality of the hidden space (default 256)

• repr_size – the dimensionality of the document embeddings space (default 100)

• kernel_heights – list of kernel lengths (default [3,5,7]), i.e., the number of consecutive tokens that each kernel covers

• stride – convolutional stride (default 1)

• stride – convolutional pad (default 0)

• drop_p – drop probability for dropout (default 0.5)

document_embedding(input)

Embeds documents (i.e., performs the forward pass up to the next-to-last layer).

Parameters:

input – a batch of instances, typically generated by a torch’s DataLoader instance (see quapy.classification.neural.TorchDataset)

Returns:

a torch tensor of shape (n_samples, n_dimensions), where n_samples is the number of documents, and n_dimensions is the dimensionality of the embedding

get_params()

Get hyper-parameters for this estimator

Returns:

a dictionary with parameter names mapped to their values

property vocabulary_size

Return the size of the vocabulary

Returns:

integer

class quapy.classification.neural.LSTMnet(vocabulary_size, n_classes, embedding_size=100, hidden_size=256, repr_size=100, lstm_class_nlayers=1, drop_p=0.5)

Bases: TextClassifierNet

An implementation of quapy.classification.neural.TextClassifierNet based on Long Short Term Memory networks.

Parameters:

• vocabulary_size – the size of the vocabulary

• n_classes – number of target classes

• embedding_size – the dimensionality of the word embeddings space (default 100)

• hidden_size – the dimensionality of the hidden space (default 256)

• repr_size – the dimensionality of the document embeddings space (default 100)

• lstm_class_nlayers – number of LSTM layers (default 1)

• drop_p – drop probability for dropout (default 0.5)

document_embedding(x)

Embeds documents (i.e., performs the forward pass up to the next-to-last layer).

Parameters:

x – a batch of instances, typically generated by a torch’s DataLoader instance (see quapy.classification.neural.TorchDataset)

Returns:

a torch tensor of shape (n_samples, n_dimensions), where n_samples is the number of documents, and n_dimensions is the dimensionality of the embedding

get_params()

Get hyper-parameters for this estimator

Returns:

a dictionary with parameter names mapped to their values

property vocabulary_size

Return the size of the vocabulary

Returns:

integer

class quapy.classification.neural.NeuralClassifierTrainer(net: TextClassifierNet, lr=0.001, weight_decay=0, patience=10, epochs=200, batch_size=64, batch_size_test=512, padding_length=300, device='cpu', checkpointpath='../checkpoint/classifier_net.dat')

Bases: object

Trains a neural network for text classification.

Parameters:

• net – an instance of TextClassifierNet implementing the forward pass

• lr – learning rate (default 1e-3)

• weight_decay – weight decay (default 0)

• patience – number of epochs that do not show any improvement in validation to wait before applying early stop (default 10)

• epochs – maximum number of training epochs (default 200)

• batch_size – batch size for training (default 64)

• batch_size_test – batch size for test (default 512)

• padding_length – maximum number of tokens to consider in a document (default 300)

• device – specify ‘cpu’ (default) or ‘cuda’ for enabling gpu

• checkpointpath – where to store the parameters of the best model found so far according to the evaluation in the held-out validation split (default ‘../checkpoint/classifier_net.dat’)

property device

Gets the device in which the network is allocated

Returns:

device

fit(instances, labels, val_split=0.3)

Fits the model according to the given training data.

Parameters:

• instances – list of lists of indexed tokens

• labels – array-like of shape (n_samples, n_classes) with the class labels

• val_split – proportion of training documents to be taken as the validation set (default 0.3)

Returns:

get_params()

Get hyper-parameters for this estimator

Returns:

a dictionary with parameter names mapped to their values

predict(instances)

Predicts labels for the instances

Parameters:

instances – list of lists of indexed tokens

Returns:

a numpy array of length n containing the label predictions, where n is the number of instances in X

predict_proba(instances)

Predicts posterior probabilities for the instances

Parameters:

X – array-like of shape (n_samples, n_features) instances to classify

Returns:

array-like of shape (n_samples, n_classes) with the posterior probabilities

reset_net_params(vocab_size, n_classes)

Reinitialize the network parameters

Parameters:

• vocab_size – the size of the vocabulary

• n_classes – the number of target classes

set_params(**params)

Set the parameters of this trainer and the learner it is training. In this current version, parameter names for the trainer and learner should be disjoint.

Parameters:

params – a **kwargs dictionary with the parameters

transform(instances)

Returns the embeddings of the instances

Parameters:

instances – list of lists of indexed tokens

Returns:

array-like of shape (n_samples, embed_size) with the embedded instances, where embed_size is defined by the classification network

class quapy.classification.neural.TextClassifierNet(*args: Any, **kwargs: Any)

Bases: Module

Abstract Text classifier (torch.nn.Module)

dimensions()

Gets the number of dimensions of the embedding space

Returns:

integer

abstractmethod document_embedding(x)

Embeds documents (i.e., performs the forward pass up to the next-to-last layer).

Parameters:

x – a batch of instances, typically generated by a torch’s DataLoader instance (see quapy.classification.neural.TorchDataset)

Returns:

a torch tensor of shape (n_samples, n_dimensions), where n_samples is the number of documents, and n_dimensions is the dimensionality of the embedding

forward(x)

Performs the forward pass.

Parameters:

x – a batch of instances, typically generated by a torch’s DataLoader instance (see quapy.classification.neural.TorchDataset)

Returns:

a tensor of shape (n_instances, n_classes) with the decision scores for each of the instances and classes

abstractmethod get_params()

Get hyper-parameters for this estimator

Returns:

a dictionary with parameter names mapped to their values

predict_proba(x)

Predicts posterior probabilities for the instances in x

Parameters:

x – a torch tensor of indexed tokens with shape (n_instances, pad_length) where n_instances is the number of instances in the batch, and pad_length is length of the pad in the batch

Returns:

array-like of shape (n_samples, n_classes) with the posterior probabilities

property vocabulary_size

Return the size of the vocabulary

Returns:

integer

xavier_uniform()

Performs Xavier initialization of the network parameters

class quapy.classification.neural.TorchDataset(instances, labels=None)

Bases: Dataset

Transforms labelled instances into a Torch’s torch.utils.data.DataLoader object

Parameters:

• instances – list of lists of indexed tokens

• labels – array-like of shape (n_samples, n_classes) with the class labels

asDataloader(batch_size, shuffle, pad_length, device)

Converts the labelled collection into a Torch DataLoader with dynamic padding for the batch

Parameters:

• batch_size – batch size

• shuffle – whether or not to shuffle instances

• pad_length – the maximum length for the list of tokens (dynamic padding is applied, meaning that if the longest document in the batch is shorter than pad_length, then the batch is padded up to its length, and not to pad_length.

• device – whether to allocate tensors in cpu or in cuda

Returns:

a torch.utils.data.DataLoader object

quapy.classification.svmperf module

class quapy.classification.svmperf.SVMperf(svmperf_base, C=0.01, verbose=False, loss='01', host_folder=None)

Bases: BaseEstimator, ClassifierMixin

A wrapper for the SVM-perf package by Thorsten Joachims. When using losses for quantification, the source code has to be patched. See the installation documentation for further details.

References

• Esuli et al.2015

• Barranquero et al.2015

Parameters:

• svmperf_base – path to directory containing the binary files svm_perf_learn and svm_perf_classify

• C – trade-off between training error and margin (default 0.01)

• verbose – set to True to print svm-perf std outputs

• loss – the loss to optimize for. Available losses are “01”, “f1”, “kld”, “nkld”, “q”, “qacc”, “qf1”, “qgm”, “mae”, “mrae”.

• host_folder – directory where to store the trained model; set to None (default) for using a tmp directory (temporal directories are automatically deleted)

decision_function(X, y=None)

Evaluate the decision function for the samples in X.

Parameters:

• X – array-like of shape (n_samples, n_features) containing the instances to classify

• y – unused

Returns:

array-like of shape (n_samples,) containing the decision scores of the instances

fit(X, y)

Trains the SVM for the multivariate performance loss

Parameters:

• X – training instances

• y – a binary vector of labels

Returns:

self

predict(X)

Predicts labels for the instances X

Parameters:

X – array-like of shape (n_samples, n_features) instances to classify

Returns:

a numpy array of length n containing the label predictions, where n is the number of instances in X

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → SVMperf

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

• True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

• False: metadata is not requested and the meta-estimator will not pass it to score.

• None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

• str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

sample_weight (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for sample_weight parameter in score.

Returns:

self – The updated object.

Return type:

object

valid_losses = {'01': 0, 'f1': 1, 'kld': 12, 'mae': 26, 'mrae': 27, 'nkld': 13, 'q': 22, 'qacc': 23, 'qf1': 24, 'qgm': 25}

Module contents