FlagRFIAnalyzer

Identifies data contaminated by RFI.

Wrapper for ch_util.rfi.number_deviations. Flags data as RFI if the stacked autocorrelations are greater than some number of local median absolute deviations (MAD) from the local median.

DocLib 777 describes this analyzer and the associated theremin graph.

Metrics

dias_task_<task_name>_run_time_seconds

Time to process single run.

dias_task_<task_name>_files_total

Number of files processed.

dias_task_<task_name>_file_time_seconds

Time to process single file.

dias_data_<task_name>_masked_missing_ratio

Fraction of data that is missing (e.g., dropped packets or down GPU nodes.)

Labels
stack : string of format <POL>-<CYL>
The polarisation and cylinder of the feeds used to construct the stacked autocorrelation. The special value ALL indicates all feeds.

dias_data_<task_name>_masked_before_ratio

Fraction of data considered bad before applying MAD threshold. Includes missing data and static frequency mask from ch_util.rfi.frequency_mask.

Labels
stack : string of format <POL>-<CYL>
The polarisation and cylinder of the feeds used to construct the stacked autocorrelation. The special value ALL indicates all feeds.

dias_data_<task_name>_masked_after_ratio

Fraction of data considered bad after applying MAD threshold. Includes missing data, static frequency mask from ch_util.rfi.frequency_mask, and MAD threshold mask.

Labels
stack : string of format <POL>-<CYL>
The polarisation and cylinder of the feeds used to construct the stacked autocorrelation. The special value ALL indicates all feeds.

Output Data

File naming

<YYYYMMDD>T<HHMMSS>Z_chimestack_rfimask/<SSSSSSSS>.h5
YYYYMMDD and HHMMSS are the date and time (in UTC) of the start of the underlying chimestack data acquisition from which the RFI mask was derived. SSSSSSSS is the number of seconds elapsed between the start of the file and the start of the acquisition.

Indexes

freq
1D structured array containing the centre and width of the frequency channels in MHz.
stack
1D array containing strings of format <POL>-<CYL> that indicate the polarisation and cylinder of the feeds used to construct the stacked autocorrelation. The special value ALL indicates all feeds.
time
1D array contaning the unix timestamps of the centre of the integrations.

Datasets

auto
3D array of type float with axes [freq, stack, time] that contains the calibrated autocorrelations stacked over inputs. Units are Jansky.
ndev
3D array of type float with axes [freq, stack, time] that contains the number of local median absolute deviations of the autocorrelations from the local median.
mask
3D array of type bool with axes [freq, stack, time] that indicates data that is likely contamined by RFI.

Config Variables

dias.analyzers.flag_rfi_analyzer.FlagRFIAnalyzer.offset

Process data this timedelta before current time.

Type:str
dias.analyzers.flag_rfi_analyzer.FlagRFIAnalyzer.instrument

Search archive for corr acquisitions from this instrument.

Type:str
dias.analyzers.flag_rfi_analyzer.FlagRFIAnalyzer.max_num_file

Maximum number of files to load into memory at once.

Type:int
dias.analyzers.flag_rfi_analyzer.FlagRFIAnalyzer.output_suffix

Name for the output acquisition type.

Type:str
dias.analyzers.flag_rfi_analyzer.FlagRFIAnalyzer.freq_low

Generate RFI flags for all frequencies above this threshold.

Type:float
dias.analyzers.flag_rfi_analyzer.FlagRFIAnalyzer.freq_high

Generate RFI flags for all frequencies below this threshold.

Type:float
dias.analyzers.flag_rfi_analyzer.FlagRFIAnalyzer.separate_cyl_pol

Construct a mask for each cylinder and polarisation in addition to the mask for the entire array.

Type:bool
dias.analyzers.flag_rfi_analyzer.FlagRFIAnalyzer.apply_static_mask

Apply static mask obtained from ch_util.rfi.frequency_mask before computing statistics.

Type:bool
dias.analyzers.flag_rfi_analyzer.FlagRFIAnalyzer.freq_width

Frequency interval in MHz for computing local statistics.

Type:float
dias.analyzers.flag_rfi_analyzer.FlagRFIAnalyzer.time_width

Time interval in seconds for computing local statistics.

Type:float
dias.analyzers.flag_rfi_analyzer.FlagRFIAnalyzer.rolling

Use a rolling window instead of distinct blocks.

Type:bool
dias.analyzers.flag_rfi_analyzer.FlagRFIAnalyzer.threshold_mad

Flag data as RFI if greater than this number of median absolute deviations.

Type:float

Additionally, the config parameters of the base class are expected in the task file

dias.analyzer.Analyzer(name, write_dir, …) Base class for all dias analyzers.