Neural Networks

Evaluate

Accuracy

Helper function for computing the (top-k) accuracy of a model on a dataset.

rfml.nn.eval.accuracy.compute_accuracy(model: rfml.nn.model.base.Model, data: rfml.data.dataset.Dataset, le: rfml.data.encoder.Encoder, batch_size: int = 512, mask: pandas.core.generic.NDFrame.mask = None) → float[source]

Compute the Top-1 accuracy of this model on the dataset.

Parameters

  • model (Model) – (Trained) model to evaluate.

  • data (Dataset) – (Testing) data to use for evaluation.

  • le (Encoder) – Mapping from human readable to machine readable.

  • batch_size (int, optional) – Defaults to 512.

  • mask (pd.DataFrame.mask, optional) – Mask to apply to the data before computing accuracy. Defaults to None.

Returns

Top-1 Accuracy

Return type

float

rfml.nn.eval.accuracy.compute_accuracy_on_cross_sections(model: rfml.nn.model.base.Model, data: rfml.data.dataset.Dataset, le: rfml.data.encoder.Encoder, column: str, batch_size: int = 512) → Tuple[List[float], List][source]

Compute an accuracy on each unique value in the column (such as SNR or CFO)

Parameters

  • model (Model) – (Trained) model to evaluate.

  • data (Dataset) – (Testing) data to use for evaluation.

  • le (Encoder) – Mapping from human readable to machine readable.

  • column (str) – Name of the column to use for computing cross sections (e.g. SNR)

  • batch_size (int, optional) – Defaults to 512.

Returns

Accuracy vs Column, Column Values

Return type

List[float], List[object]

rfml.nn.eval.accuracy.compute_topk_accuracy(model: rfml.nn.model.base.Model, data: rfml.data.dataset.Dataset, le: rfml.data.encoder.Encoder, k: int, batch_size: int = 512, mask: pandas.core.generic.NDFrame.mask = None) → float[source]

Computes the probability that the true class is in the top k outputs of the network.

Warning

If you only want Top-1 Accuracy (if you don’t know what this is, then that is what you want). Then you should just use compute_accuracy instead.

Parameters

  • model (Model) – (Trained) model to evaluate.

  • data (Dataset) – (Testing) data to use for evaluation.

  • le (Encoder) – Mapping from human readable to machine readable.

  • k (int) – Value to use when determining the “top k”.

  • batch_size (int, optional) – Defaults to 512.

  • mask (pd.DataFrame.mask, optional) – Mask to apply to the data before computing top-k accuracy. Defaults to None.

Returns

Top-K Accuracy

Return type

float

Confusion

Helper function for computing the confusion matrix of a model on a dataset.

rfml.nn.eval.confusion.compute_confusion(model: rfml.nn.model.base.Model, data: rfml.data.dataset.Dataset, le: rfml.data.encoder.Encoder, batch_size: int = 512, mask: pandas.core.generic.NDFrame.mask = None) → numpy.ndarray[source]

Compute and normalize a confusion matrix of this model on the dataset.

Parameters

  • model (Model) – (Trained) model to evaluate.

  • data (Dataset) – (Testing) data to use for evaluation.

  • le (Encoder) – Mapping from human readable to machine readable.

  • batch_size (int, optional) – Defaults to 512.

  • mask (pd.DataFrame.mask, optional) – Mask to apply to the data before computing accuracy. Defaults to None.

Returns

Normalized Confusion Matrix

Return type

np.ndarray

Functional

Energy

Calculate the average energy (per symbol if provided) for each example.

rfml.nn.F.energy.energy(x: torch.Tensor, sps: float = 1.0)[source]

Calculate the average energy (per symbol if provided) for each example.

This function assumes that the signal is structured as:

\[Batch x Channel x IQ x Time.\]
Parameters

  • x (torch.Tensor) – Input Tensor (BxCxIQxT)

  • sps (int, optional) – Samples per symbol, essentially the power is multiplied by this value in order to calculate average energy per symbol. Defaults to 1.0.

\[ \begin{align}\begin{aligned}\mathbb{E}[E_{s}] = \frac{\text{sps}}{N} \sum_{i=0}^{N} |s_i|^2\\|s_i| = \sqrt{\mathbb{R}^2 + \mathbb{C}^2}\end{aligned}\end{align} \]
Returns

Average energy per example per channel (BxC)

Return type

[torch.Tensor]

Error Vector Magnitude

PyTorch implementation of Error Vector Magnitude

rfml.nn.F.evm.evm(x: torch.Tensor, y: torch.Tensor) → torch.Tensor[source]

Compute the Error Vector Magnitude (EVM) between two signals.

\[\operatorname{EVM}(s_{rx}, s_{tx}) = \sqrt{\mathbb{R}(s_{rx} - s_{tx})^2 + \mathbb{C}(s_{rx} - s_{tx})^2}\]
Parameters

  • x (torch.Tensor) – First signal (BxCxIQxN)

  • y (torch.Tensor) – Second signal (BxCxIQxN)

Returns

Error Vector Magnitude per sample (BxCx1xN). Note that the

returned signal is no longer complex as it is only a magnitude and therefore only has a dimension size of 1 where IQ used to be.

Return type

[torch.Tensor]

Power Spectral Density

Calculate the power spectral density (PSD) of an input signal.

Layers

Flatten

Flatten signals/images into features.

class rfml.nn.layers.flatten.Flatten(preserve_time: bool = False)[source]

Flatten the channel, IQ, and time dims into a single feature dim.

This module assumes that the input signal is structured as:

\[Batch x Channel x IQ x Time\]
Parameters

preserve_time (bool, optional) – If provided as True then the time dimension is preserved in the outputs and only the IQ and Channel dimensions are concatenated together. Otherwise, the time dimension is also collapsed to form a single feature dimension. Generally, you will set this to False if the layer after Flatten will be a Linear layer and set this to True if the layer after Flatten will be a Recurrent layer that utilizes the time dimension. Defaults to False.

The outputs of this layer, if preserve_time is not set to True, are:

\[Batch x Features\]

Where features is the product of the flattened dimensions:

\[(Channel x IQ x Time)\]

The outputs of this layer, if preserve_time is set to True, are:

\[Batch x Time x Features\]

Where features is the product of the flattened dimensions:

\[(Channel x IQ)\]
forward(x: torch.Tensor)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Power Normalization

Normalize the power across each example/channel to 1.

class rfml.nn.layers.power_normalization.PowerNormalization[source]

Perform average energy per sample (power) normalization.

Power Normalization would be performed as follows for each batch/channel:

\[x = \frac{x}{\sqrt{\mathbb{E}[x]}}\]

This module assumes that the signal is structured as:

\[Batch x Channel x IQ x Time.\]

Where the power normalization is performed along the T axis using the power measured in the complex valued I/Q dimension.

The outputs of this layer match the inputs:

\[Batch x Channel x IQ x Time\]
forward(x: torch.Tensor)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Models

Base

Base class for all neural network models

class rfml.nn.model.base.Model(input_samples: int, n_classes: int)[source]

Base class that all neural network models inherit from.

Parameters

  • input_samples (int) – The number of samples that will be given to this Model for each inference.

  • n_classes (int) – The number of classes that this Model will predict.

This model supports standard switching between training/evaluation through PyTorch (e.g. Model.train() and Model.eval()) but also supports a custom command to allow transfer learning by freezing only portions of the network (e.g. Model.freeze() and Model.unfreeze()). Note that some subclasses of this may not necessarily support this feature.

This class also provides all of the common functionality to the child classes such as save() and load().

property device

Retrieve the most probable device that this model is currently on.

Warning

This method is not guaranteed to work if the model is split onto multiple devices (e.g. part on CPU, part on GPU 1, and part on GPU 2).

Returns

Device that this model is likely located on

Return type

torch.device

freeze()[source]

Freeze part of the model so that only parts of the model are updated.

property input_samples

The expected number of complex samples on the input to this model.

load(path: str = None, map_location: str = 'cpu')[source]

Load pretrained weights from disk.

Parameters

  • path (str, optional) – If provided, then load immortal weights from this path. If not set, then the temporary weights path is used (for reloading the “best weights” in an early stopping procedure). Defaults to None.

  • map_location (str, optional) – String representing the device to load the model/weights into. If this is set to None, then the weights will be loaded onto the same device they were saved from. This can cause failures if the devices do not exist on the machine calling this function. This can occur if the model is trained on one device (with GPUs) and then used on another device where GPUs do not exist. It can also occur on the same device if the GPU configurations are changed (by setting CUDA_VISIBLE_DEVICES) or if the desired device is out of memory. See torch.load() documentation for further details and options as this parameter is simply a passthrough for that. Defaults to “cpu” if path is provided, else it is set to None and the input provided by the user is ignored.

Warning

This doesn’t provide safety against weights paths existing; therefore, it will throw the arguments back up the stack instead of silencing them.

property n_classes

The number of outputs of this model per inference.

property n_parameters

The total number of parameters in the model.

property n_trainable_parameters

The number of parameters that would be ‘learned’ during training.

outputs(x: torch.Tensor) → torch.Tensor[source]

Convenience method for receiving the full outputs of the neural network.

Note

This method should only be used during testing – training should operate directly on the forward/backward calls provided by PyTorch.

This method is opinionated in order to reduce complexity of receiving model outputs for the caller. To that end, it does four things for the caller:

  • Puts the model in eval mode so that Batch Normalization/Dropout aren’t induced

  • Pushes the data to whatever device this model is currently on (such as cuda:0/cuda:1/cpu/etc.) so the caller doesn’t have to know where the model resides

  • Obtains the outputs of the network (using whichever device the model is currently on)

  • Pulls the outputs back to CPU for further analysis by the caller

Parameters

x (torch.Tensor) – Inputs to the network.

Returns

Full outputs of this network for each input.

Return type

torch.Tensor

predict(x: torch.Tensor) → torch.Tensor[source]

Return a categorical prediction using an argmax strategy.

Parameters

x (torch.Tensor) – Inputs to the network.

Returns

Label of the highest class for each input.

Return type

torch.Tensor

See also

outputs

save(path: str = None)[source]

Save the currently loaded/trained weights to disk.

Parameters

path (str, optional) – If provided, the weights will be saved at this path, which is useful for immortalizing the weights once training is completed. If not provided, then the model will create a temporary file with a unique ID to store the current weights at, and delete that file when this object is deleted. This can be useful for storing intermediate weights that will be used to reload “the best weights” for an early stopping procedure, without requiring the caller to care where they are stored at. Defaults to None.

Warning

This will overwrite the weights saved at this path (or the temporary weights).

unfreeze()[source]

Re-enable learning of all parts of the model.

CLDNN

Convolutional long deep neural network (CNN + GRU + MLP)

class rfml.nn.model.cldnn.CLDNN(input_samples: int, n_classes: int)[source]

Convolutional Long Deep Neural Network (CNN + GRU + MLP)

This network is based off of a network for modulation classification first introduced in West/O’Shea.

The following modifications/interpretations were made by Bryse Flowers:

  • Batch Normalization was added otherwise the model was not stable enough to train in many cases (its unclear whether this was included in West’s model)

  • The filter sizes were changed to 7 and the padding was set to 3 (where as West’s paper said they used size 8 filters and did not mention padding)

    • An odd sized filter is necessary to ensure that the intermediate signal/feature map lengths are the same size and thus can be concatenated back together

  • A Gated Recurrent Unit (GRU) was used in place of a Long-Short Term Memory (LSTM).

    • These two submodules should behave nearly identically but GRU has one fewer equation

  • Bias was not used in the first convolution in order to more closely mimic the implementation of the CNN.

  • The hidden size of the GRU was set to be the number of classes it is trying to predict – it makes the most sense instead of trying to find an arbritrary best hidden size.

References

N. E. West and T. O’Shea, “Deep architectures for modulation recognition,” in IEEE International Symposium on Dynamic Spectrum Access Networks (DySPAN), pp. 1–6, IEEE, 2017.

forward(x)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

CNN

Simplistic convolutional neural network.

class rfml.nn.model.cnn.CNN(input_samples: int, n_classes: int)[source]

Convolutional Neural Network based on the “VT_CNN2” Architecture

This network is based off of a network for modulation classification first introduced in O’Shea et al and later updated by West/Oshea and Hauser et al to have larger filter sizes.

Modifying the first convolutional layer to not use a bias term is a modification made by Bryse Flowers due to the observation of vanishing gradients during training when ported to PyTorch (other authors used Keras).

Including the PowerNormalization inside this network is a simplification made by Bryse Flowers so that utilization of DSP blocks in real time for data generation does not require knowledge of the normalization used during training as that is encapsulated in the network and not in a pre-processing stage that must be matched up.

References

T. J. O’Shea, J. Corgan, and T. C. Clancy, “Convolutional radio modulation recognition networks,” in International Conference on Engineering Applications of Neural Networks, pp. 213–226, Springer,2016.

N. E. West and T. O’Shea, “Deep architectures for modulation recognition,” in IEEE International Symposium on Dynamic Spectrum Access Networks (DySPAN), pp. 1–6, IEEE, 2017.

S. C. Hauser, W. C. Headley, and A. J. Michaels, “Signal detection effects on deep neural networks utilizing raw iq for modulation classification,” in Military Communications Conference, pp. 121–127, IEEE, 2017.

forward(x)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Factory

Simplistic factory pattern for config defined swapping of architecture used.

rfml.nn.model.factory.build_model(model_name: str, input_samples: int, n_classes: int)[source]

Factory method for dynamic creation of multiple neural architectures.

Parameters

  • model_name (str) –

    The name of the model to build. Currently supported models are:

    • ”CNN”

    • ”CLDNN”

  • input_samples (int) – Number of complex input samples to the model.

  • n_classes (int) – Number of output classes the model should predict.

Returns

The built model described by the provided parameters

Return type

Model

Train

Base

Base class that all trainers inherit from (strategy pattern).

Factory

Simplistic factory pattern for swapping of training strategies.

rfml.nn.train.factory.build_trainer(strategy: str, lr: float = 0.001, max_epochs: int = 50, patience: int = 5, batch_size: int = 512, gpu: bool = True, **kwargs) → rfml.nn.train.base.TrainingStrategy[source]

Construct a training strategy from the given parameters.

Parameters

  • strategy (str) –

    Strategy to use when training the network, current options are:

    • ”standard”

  • lr (float, optional) – Learning rate to be used by the optimizer. Defaults to 10e-4.

  • max_epochs (int, optional) – Maximum number of epochs to train before stopping training to preserve computing resources (even if the network is still improving). Defaults to 50.

  • patience (int, optional) – Maximum number of epochs to continue to train for even if the network is not still improving before deciding that overfitting is occurring and stopping. Defaults to 5.

  • batch_size (int, optional) – Number of examples to give to the model at one time. If this value is set too high, then an out of memory error could occur. If the value is set too low then training will take a longer time (and be more variable). Defaults to 512.

  • gpu (bool, optional) – Flag describing whether the GPU is used or the training is performed wholly on the CPU. Defaults to True.

  • **kwargs – The remainder of the keyword arguments are directly passed through to the constructor of the class being instantied.

Example

>>> trainer = build_trainer("standard")
>>> trainer(model, training, validation, encoder)
>>> model.save("/path/to/weights.pt")

Printing Training Listener

TrainingListener for quick prototypes that only log to stdout.

class rfml.nn.train.printing_training_listener.PrintingTrainingListener[source]

TrainingListener implementation for quick prototypes that only logs to stdout.

on_epoch_completed(mean_loss: float, epoch: int)[source]

Called after a full training epoch has been completed.

Parameters

  • mean_loss (float) – The mean training loss during this epoch.

  • epoch (int) – Epoch that was just trained.

on_training_completed(best_loss: float, best_epoch: int, total_epochs: int)[source]

Called when a stopping condition has been reached.

Parameters

  • best_loss (float) – The best loss that was achieved by the model.

  • best_epoch (int) – The epoch where that loss was achieved. The weights for the model is reloaded from this epoch.

  • total_epochs (int) – The total number of epochs that this model trained for.

on_validation_completed(mean_loss: float, epoch: int)[source]

Called after the validation loss has been computed.

Parameters

  • mean_loss (float) – Mean training loss for this epoch.

  • epoch (int) – Epoch that was just validated.

Standard

An implementation of a typical multi-class classification training loop.

class rfml.nn.train.standard.StandardTrainingStrategy(lr: float = 0.001, max_epochs: int = 50, patience: int = 5, batch_size: int = 512, gpu: bool = True)[source]

A typical strategy that would be used to train a multi-class classifier.

Parameters

  • lr (float, optional) – Learning rate to be used by the optimizer. Defaults to 10e-4.

  • max_epochs (int, optional) – Maximum number of epochs to train before stopping training to preserve computing resources (even if the network is still improving). Defaults to 50.

  • patience (int, optional) – Maximum number of epochs to continue to train for even if the network is not still improving before deciding that overfitting is occurring and stopping. Defaults to 5.

  • batch_size (int, optional) – Number of examples to give to the model at one time. If this value is set too high, then an out of memory error could occur. If the value is set too low then training will take a longer time (and be more variable). Defaults to 512.

  • gpu (bool, optional) – Flag describing whether the GPU is used or the training is performed wholly on the CPU. Defaults to True.

Training Listener

Interface to receive callbacks about training progress (observer patten).

class rfml.nn.train.training_listener.TrainingListener[source]

Interface for receiving callbacks during training with current progress.

A user should inherit from this base class, override the corresponding callbacks that it intends to receive, and register this listener with a trainer in order to receive status updates during training.

on_epoch_completed(mean_loss: float, epoch: int)[source]

Called after a full training epoch has been completed.

Parameters

  • mean_loss (float) – The mean training loss during this epoch.

  • epoch (int) – Epoch that was just trained.

on_training_completed(best_loss: float, best_epoch: int, total_epochs: int)[source]

Called when a stopping condition has been reached.

Parameters

  • best_loss (float) – The best loss that was achieved by the model.

  • best_epoch (int) – The epoch where that loss was achieved. The weights for the model is reloaded from this epoch.

  • total_epochs (int) – The total number of epochs that this model trained for.

on_validation_completed(mean_loss: float, epoch: int)[source]

Called after the validation loss has been computed.

Parameters

  • mean_loss (float) – Mean training loss for this epoch.

  • epoch (int) – Epoch that was just validated.