<jupyter_start><jupyter_text>Convert MVF dataset(s) to MeasurementSetConverting the MeerKAT MVF observation data to a CASA measurement set is done using the `mvftoms.py` script provided with the `katdal` package installation.`Usage: mvftoms.py [options]  []*`For detail help: `mvftoms.py -h`  OptionsOptions when converting MVF to MeasurementSet                        Option            Value            Description                                    -o            OUTPUT_MS            Output Measurement Set                            -r            REF_ANT            Reference antenna. Default is first antenna used by script                            -t                        Tar-ball the MS                            -f                        Produce a full polarisation MS in CASA canonical order (HH, HV, VH, VV). Default is to produce HH,VV only                            -v                        More verbose progress information                            -w                        Use W term to stop fringes for each baseline                            -p                        Select polarisation products to include in MS as comma separated list (from: HH, HV, VH, VV). Default is all available from HH, VV                            -a                        MeasurementSet will exclude autocorrelation data                            -s                        Keep spaces in source names, default removes spaces                            -C            CHANNEL_RANGE            Range of frequency channels to keep (zero-based inclusive 'first_chan,last_chan', default is all channels)                            -e            ELEVATION_RANGE            Flag elevations outside the range 'lowest_elevation,highest_elevation'                            -m                        Add MODEL_DATA and CORRECTED_DATA columns to the MS. MODEL_DATA initialised to unity amplitude zero phase, CORRECTED_DATA initialised to DATA                            --flags            FLAGS            List of online flags to apply (from 'static,cam,data_lost,ingest_rfi,cal_rfi,predicted_rfi', default is all flags, '' will apply no flags)                            --dumptime            DUMPTIME            Output time averaging interval in seconds, default is no averaging                            --chanbin            CHANBIN            Bin width for channel averaging in channels, default is no averaging                            --flagav                        Create calibration tables from gain solutions in the dataset, if present                            --caltables                        If a single element in an averaging bin is flagged, flag the averaged bin                            --quack            N            Discard the first N dumps (which are frequently incomplete)                            --applycal            APPLYCAL            List of calibration solutions to apply to data as a string of comma-separated names, e.g. 'l1' or 'K,B,G'. Use 'default' for L1 + L2 and 'all' for all available products.              FilenameThe `` provided to `katdal` has to include the complete web path the data in the MeerKAT archive, as well as a valid token, in the format:captureblockID```mvftoms.py [options] https://archive-gw-1.kat.ac.za//_sdp_l0.full.rdb?token=```  Example 1Convert the full observation to an output MS, and use the `-o` option to specify an appropriate name for the output Measurement Set```mvftoms.py https://archive-gw-1.kat.ac.za/1548417874/1548417874_sdp_l0.full.rdb?token=<KEY> -o exms_full.ms```Default behaviour that should be noted:* If a reference antenna is not specified, the first antenna used will be selected by default* Since full polarisation was not specified, only HH, VV products will be produced* All flags to remove suspected bad data are applied by default. Flagging during conversion is by no means exhaustive and some flagging will still be required in CASA during data processing.   **Note: pre-flagging is generally good for continuum imaging, care should be taken for spectral lines, since it may flag out science lines.**    For older files you may encounder the following error:```katsdptelstate.errors.DecodeError: The telescope state contains pickled values. This is a security risk, so is disabled by default. If you trust the source of the data, you can allow the pickles to be loaded by setting KATSDPTELSTATE_ALLOW_PICKLE=1 in the environment. This is needed for MeerKAT data up to March 2019.```To fix this:```export KATSDPTELSTATE_ALLOW_PICKLE=1```Refer to https://pypi.org/project/katsdptelstate/ for more information<jupyter_code>msfile = 'exms_full.ms'
target = 'PKS1934-63'
listobs(vis=msfile)
plotms(vis=msfile, xaxis='freq', yaxis='amp', field=target, coloraxis='baseline', antenna='*&')
plotms(vis=msfile, xaxis='freq', yaxis='amp', field=target, coloraxis='baseline', antenna='*&&&')
plotms(vis=msfile, xaxis='freq', yaxis='phase', field=target, coloraxis='baseline')<jupyter_output><empty_output><jupyter_text>Example 2Selection of options based on information shown in observation [calibration report](https://github.com/ska-sa/MeerKAT-Cookbook/blob/master/archive/Calibration%20pipeline%20report.pdf)Convert only the channels defining the passband and exclude the transition band channels from the output MS```mvftoms.py https://archive-gw-1.kat.ac.za/1557528200/1557528200_sdp_l0.full.rdb?token=<KEY>o exms_passband.ms -f -a -C '163,3885' -r 'm053' --flags 'static,cam,data_lost,ingest_rfi' --quack=3 ```Options specified:* `-f` produce a full polarisation MS in CASA canonical order (HH, HV, VH, VV)* `-a` exclude autocorrelation data* `-C` range of frequency channels to keep. Selection is `'163,3885'` which will discard the edges of the band and only use the passband channels during conversion. Refer to the *RFI and Flagging summary* section in the observation calibration report for usable section channels/frequencies* `-r` specify a reference antenna selected to be the same reference antenna used in the calibration report* `--flags` remove selection of non-usable data and RFI, good default selection `'static,cam,data_lost,ingest_rfi'`* `--quack` discard the first N dumps. Refer to the observation summary in the *Description* provided by the MeerKAT archive interface Smaller example showing similar results```mvftoms.py https://archive-gw-1.kat.ac.za/1548417874/1548417874_sdp_l0.full.rdb?token=<KEY> -o exms_passband.ms -f -a -C '163,3885' -r 'm053' --flags 'static,cam,data_lost,ingest_rfi' --quack=3```<jupyter_code>msfile = 'exms_passband.ms'
target = 'PKS1934-63'
listobs(vis=msfile)
plotants(vis=msfile)
plotms(vis=msfile, xaxis='freq', yaxis='amp', field=target, coloraxis='baseline')
plotms(vis=msfile, xaxis='freq', yaxis='phase', field=target, coloraxis='baseline')<jupyter_output><empty_output><jupyter_text>Example 3More selective example    Only select the frequency range between 1.3 GHz and 1.5 GHz for convertion into MS```mvftoms.py https://archive-gw-1.kat.ac.za/1548417874/1548417874_sdp_l0.full.rdb?token=<KEY> -o 1548417874.ms -f -a -C '2150,3150' -r 'm053' --flags 'static,cam,data_lost,ingest_rfi' --quack=1 ```<jupyter_code>plotms(vis='1548417874.ms', xaxis='chan', yaxis='amp', coloraxis='corr')
plotms(vis='1548417874.ms', xaxis='freq', yaxis='amp', coloraxis='corr')<jupyter_output><empty_output><jupyter_text>Example 4Good default conversion:    Convert only the channels defining the passband and apply antenna based pipeline calibration results to the output MS```mvftoms.py https://archive-gw-1.kat.ac.za/1548417874/1548417874_sdp_l0.full.rdb?token=<KEY> -o 1548417874.ms -f -C '163,3885' --flags 'static,cam,data_lost,ingest_rfi' --quack=1 --applycal all```Options specified:* `-f` produce a full polarisation MS in CASA canonical order (HH, HV, VH, VV)* `-C` range of frequency channels to keep. Selection is `'163,3885'` which will discard the edges of the band and only use the passband channels during conversion. Refer to the *RFI and Flagging summary* section in the observation calibration report for usable section channels/frequencies* `--flags` remove selection of non-usable data and RFI, good default selection `'static,cam,data_lost,ingest_rfi'`* `--quack` discard the first N dumps. Refer to the observation summary in the *Description* provided by the MeerKAT archive interface* `--applycal` apply `all` available calibration products `mvftoms.py https://archive-gw-1.kat.ac.za/1557528200/1557528200_sdp_l0.full.rdb?token=<KEY> -o 1557528200.ms -f -a -C '2150,3150' --quack=3 --applycal all`<jupyter_code>msfile = '1557528200.ms'
target = '3C286'
listobs(vis=msfile)
plotms(vis=msfile, xaxis='channel', yaxis='amp', antenna='*&', field=target, averagedata=True, avgtime='30000')
plotms(vis=msfile, xaxis='channel', yaxis='phase', antenna='*&', field=target, averagedata=True, avgtime='30000')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09130826390452601, 'fraction_numerical': 0.034451700067552354, 'mean_word_length': 3.099215505306876, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '164349', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 2768, 'n_tokens_neox': 2471, 'n_words': 897, 'path': 'archive/Convert MVF dataset(s) to MeasurementSet.ipynb', 'repo_name': 'BongaNjamela001/MeerKAT-Cookbook', 'size': '10798'}