Testing Coverage in Bessy Python

This spike is a review of the testing currently implemented in Bessy Python with Unit Test Cases. The table below provides a brief summary of each test case. For a more detailed description of testing coverage, including descriptions of the specific testing methods defined in each test case, please refer to the section Review of Testing Files.

Testing Coverage Summary

Unit test case File it is located in Description
TestChannelSelection test_channel_selection.py Tests the functionality of the BessyPy’s method channel_selection_by_method(...). For the channel selection methods (SFS, SBS, etc.), it performs relevant tests regarding time limit, min/max channels, and performance delta. See more
TestClassifierSave test_classifier_save.py Tests the save and load functionality of MiClassifier and ErpRgClassifier. See more
TestBciController test_eeg_data.py Tests the functionality of BciController with NullClassifier. It tests the behavior of BciController for various scenarios, including changing channel types to “stim,” running a single step offline or online, handling invalid marker or EEG data when running online, and more. Note that each method initializes BciController with the paradigm set to MiParadigm. See more
TestLslSourceTimeouts test_lsl_sources.py Tests that LslMarkerSource or LslEegSource raise an exception when initialized with a timeout of 0. See more
TestLslMarkerSource test_lsl_sources.py Tests functionality of LslMarkerSource, including verification of the marker name and checking time correction. See more
TestLslEegSource test_lsl_sources.py Tests functionality of LslEegSource, including verification of the marker name, verification of the value of several attributes, and checking time correction. See more
TestRemoveFlats test_remove_flats.py Runs MI and P300 classification, and checks that a model was trained, and that offline classification accuracy, precision, recall, and the confusion matrix exist. Also checks if the list of class predictions exists. See more
TestLoadData test_signal_processing.py Tests that bandpass, lowpass, highpass, and notch signal processing correctly processes generated testing data. See more
TestSmoke test_smoke.py Tests offline classification for MI, P300, and SSVEP using smoke test data. See more
TestRawTrialSave test_trial_saving.py Tests the save and load functionality of EEG trial data using NullClassifier with the MiParadigm. See more
TestXdfSourceNonexistentFiles test_xdf_sources Checks if XdfMarkerSource or XdfEegSource raise exceptions for non-existent files. See more
TestXdfMarkerSource test_xdf_sources Tests functionality of XdfMarkerSource, including verification of the marker name, verification that all marker samples are provided at once, and checking time correction. See more
TestXdfEegSource test_xdf_sources Tests functionality of XdfEegSource, including verification of the EEG source name, verification of the value of several attributes, etc. See more

Review of Testing Files

This is a more detailed review and description of the unit test cases as defined in the following files (located in bci-essentials-python/tests). Descriptions are provided for the methods of each test case, to review which tests are specifically covered by each method.

test_channel_selection.py

TestChannelSelection class

Tests channel_selection_by_method(...) functionality using the following methods:

Method Description
test_time_limit() Tests if channel selection by SFS returns a channel subset within the correct range, and that it completes within a time limit.
test_max_channels() Tests if channel selection by SFS returns a channel subset within the correct range, and checks that the algorithm didn’t check combinations outside the range.
test_min_channels() Tests if channel selection by SBS and SBFS returns a channel subset within the correct range, and checks that the algorithm didn’t check combinations outside the range.
test_performance_delta() Tests if channel selection by SFS didn’t check combinations outside the range, and checks that all but the last difference in scores are greater than the performance delta.

test_classifier_save.py

TestClassifierSave class

This Test Case contains two methods to test the save and load functionality of MiClassifier and ErpRgClassifier.

After setting up the respective classifiers, each of these methods performs similar tests: run classification, then save the classifier to a file. Load the classifier from that file, and test that the numpy arrays of the original classifier are the same as the newly loaded classifier. Then, the methods run classification with the loaded classifier and verifies that the predictions are the same.

test_eeg_data.py

TestBciController class

This test case sets up NullClassifier as the classifier. It tests the functionality of the BciController with the following methods:

Method Description
test_trg_channel_types_changed_to_stim() Makes sure that BciController properly deals with "trg" channel types that need to be changed to "stim”
test_when_offline_loop_stops_when_no_more_data() Tests that BciController stops the loop when there is no more data when running offline.
test_offline_single_step_runs_single_loop() Tests that running a single step offline only runs a single loop.
test_when_online_loop_continues_even_when_no_data() Tests that the loop continues even without data when running online.
test_online_single_step_runs_single_loop() Tests that running a single step online only runs a single loop.
test_step_does_not_wait_for_more_data() Makes sure that the step() method of BciController does not wait for more data when running online.
test_when_online_and_invalid_markers_then_loop_continues() Tests that the loop continues even when invalid marker data is provided when running online.
test_when_online_and_invalid_eeg_then_loop_continues() Tests that the loop continues on even when invalid EEG data is provided when running online.

test_lsl_sources.py

TestLslSourceTimeouts class

This test case checks that LslMarkerSource or LslEegSource raise an exception when initialized with a timeout of 0.

TestLslMarkerSource class

Unit test case for LslMarkerSource. It is initialized in setUp() with instances of LslSender and LslMarkerSource, and has the following methods:

Methods Description
test_marker_name() Tests that the name attribute of the LslMarkerSource instance equals “LSL Test”
test_marker_time_corretion_is_probably_not_zero() Tests that the time_correction method of LslMarkerSource returns 0.0

TestLslEegSource class

Unit test case for LslEegSource. It is initialized in setUp() with instances of LslSender and LslEegSource, and has the following methods:

Methods Description
test_marker_name() Tests that the name attribute of the LslEegSource instance equals “LSL Test”
test_eeg_fsample() Checks that the fsample attribute is set to 128.0
test_eeg_nchannel() Checks that the n_channels attribute is set to 8
test_eeg_channel() Checks that the channel_types attribute has 8 elements.
test_eeg_channel_units() Checks that the channel_units attribute has 8 elements.
test_eeg_channel_labels() Checks that the channel_labels attribute has 8 elements.
test_marker_time_corretion_is_probably_not_zero() Tests that the time_correction method of LslEegSource returns 0.0

test_remove_flats.py

TestRemoveFlats class

Tests the removal of flat channels from EEG data for the MI and P300 paradigms with the following methods:

Method Description
test_mi_remove_flats() Tests removing flats for MI Classification (MiClassifier and MiParadigm), with the data from the “mi_smoke_dsi24.xdf” file.
test_p300_remove_flats() Tests removing flats for P300 Classification (ErpRgClassifier and P300Paradigm), with the data from the “p300_smoke_dsi24.xdf” file.

Each of the above methods runs classification. Then, it checks that a model was trained, and that the following metrics exist: accuracy, precision, recall, confusion matrix, and the list of class predictions.

test_signal_processing.py

TestLoadData class

Tests the BessyPy signal processing methods: bandpass, lowpass, highpass, and notch. It makes sure the signal processing outputs have the correct shape, and that the MSE between the filtered and original signal was reduced.

test_smoke.py

TestSmoke class

Includes the following methods to test classification paradigms offline:

Method Description
test_mi_offline() Conducts offline test of MI Classification using MiClassifier, with data from mi_smoke.xdf.
test_p300_offline() Conducts offline test of P300 Classification using the ErpRgClassifier, with data from p300_smoke.xdf.
test_ssvep_offline() Conducts offline test of SSVEP Classification using SsvepRiemannianMdmClassifier, with data from ssvep_smoke.xdf.

test_trial_saving.py

TestRawTrialSave class

Tests the saving and loading of trial EEG data with its method test_xy_trial_save(). Currently, it just implements the MiParadigm with the NullClassifier(). It creates temporary .npy files to save the EEG trials and labels. Then it loads that data again, and checks that the classifier’s numpy arrays are the same.

test_xdf_sources.py

TestXdfSourceNonexistentFiles class

This test case checks that XdfMarkerSource or XdfEegSource raise exceptions when attempting to use a file that does not actually exist.

TestXdfMarkerSource class

This test case creates an instance of XdfMarkerSource with xdf_test_data.xdf. It has the following methods:

Methods Description
test_marker_name() Tests that the name attribute of the XdfMarkerSource instance equals “UnityMarkerStream”
test_marker_get_samples_provides_all_samples_in_one_go() Checks that the get_markers method returns all samples at once and checks array shapes. Then it checks that calling get_markers a second time returns empty lists.
test_marker_time_correction_is_zero_for_xdf() Tests that the time_correction method returns 0.0

TestXdfEegSource class

This test case creates an instance of XdfEegSource with xdf_test_data.xdf. It has the following methods:

Methods Description
test_eeg_name() Tests that the name attribute of the XdfEegSource instance is equal to “g.USBamp-1”
test_eeg_fsample() Checks that the fsample attribute is set to 256.0
test_eeg_nchannel() Checks that the n_channels attribute is set to 16
test_eeg_channel() Checks that the channel_types attribute has 16 elements and that the first element is “eeg”
test_eeg_channel_units() Checks that the channel_units attribute has 16 elements and that the first element is “microvolts”
test_eeg_channel_labels() Checks that the channel_labels attribute has 16 elements and that the first element is “FP1”
test_eeg_get_samples_provides_all_samples_in_one_go() Checks that the get_samples method returns all samples at once and checks array shapes. Then it checks that calling get_samples a second time returns empty lists.
test_eeg_time_correction_is_zero_for_xdf() Tests that the time_correction method returns 0.0