IO Operations

IO Operations

JAXTAM._config_edit — Function.
_config_edit(key_name::String, key_value::String;
        config_path=string(__sourcedir__, "user_configs.jld2"))

Edits (or adds) key_name to key_value in the config file

source
JAXTAM._config_gen — Function.
_config_gen(string(__sourcedir__, "user_configs.jld2"))

Generates a user configuration file at config_path, by default, file is placed in the JAXTAM module dir

Config file is /user_configs.jld2, excluded from git

Also used to overwrite the old config file

source
JAXTAM._config_key_value — Function.
_config_key_value(key_name::Symbol,
    config_path=string(__sourcedir__, "user_configs.jld2"))

Loads and returns the value for key_name

source
JAXTAM._config_load — Function.
_config_load(config_path=string(__sourcedir__, "user_configs.jld2"))

Loads data from the configuration file

source
JAXTAM._config_rm — Function.
_config_rm(key_name::String;
        config_path=string(__sourcedir__, "user_configs.jld2"))

Removes a the key key_name from the configuration file

source
JAXTAM.config — Method.
config(key_name::Union{String,Symbol}, key_value::Union{String,Symbol})
    _config_edit(String(key_name), String(key_value))

Adds key_name with key_value to the configuration file and saves

source
JAXTAM.config — Method.
config(key_name::Union{String,Symbol})

Returns the value of key_name

source
JAXTAM.config — Method.
config()

Returns the full configuration file data

source
JAXTAM.config_rm — Method.
config_rm(key_name::Union{String,Symbol})

Removes key_name from the configuration file and saves changes.

source
JAXTAM._master_download — Method.
_master_download(master_path, master_url)

Downloads (and unzips) a master table from HEASARC given its url and a destination path

source
JAXTAM._master_read_tdat — Method.
_master_read_tdat(master_path::String)

Reads a raw .tdat table from HEASARC mastertable archives, parses the ASCII data, finds and stores column names, cleans punctuation, converts to DataFrame, strongly types the columsn, and finally returns cleaned table as DataFrame

source
JAXTAM._master_save — Method.
_master_save(master_path_feather, master_data)

Saves the DataFrame master table to a .feather file

source
JAXTAM._public_date_int — Method.
_public_date_int(public_date)

Converts the public date to an integer, if that fails just returns the arbitrary sort-of-far-away 2e10 date

TODO: Don't return 2e10 on Float64 parse error

source
JAXTAM._type_master_df! — Method.
_type_master_df!(master_df)

Slightly janky way to strongly type columns in the master table, this needs to be done to ensure the .feather file is saved/read correctly

TODO: Make this less... stupid

source
JAXTAM.master — Method.
master(mission_name::Union{String,Symbol})

Reads in a previously created .feather master table for a specific mission_name using a path provided by _config_key_value(mission_name)

source
JAXTAM.master — Method.
master()

Loads a default mission, if one is set, otherwise throws error

source
JAXTAM.master_query — Method.
master_query(master_df::DataFrame, key_type::Symbol, key_value::Any)

Wrapper for a query, takes in an already loaded DataFrame master_df, a key_type to search over (e.g. obsid), and a key_value to find (e.g. 0123456789)

Returns the full row for any observations matching the search criteria

TODO: Fix the DataValue bug properly

source
JAXTAM.master_query — Method.
master_query(mission_name::Symbol, key_type::Symbol, key_value::Any)

Calls master_query(master_df::DataFrame, key_type::Symbol, key_value::Any) by loading the master and append tables for mission_name

source
JAXTAM.master_query_public — Method.
master_query_public(master_df::DataFrame, key_type::Symbol, key_value::Any)

Calls master_query for given query, but restricted to currently public observations

source
JAXTAM.master_query_public — Method.
master_query_public(master_df::DataFrame)

Returns all the currently public observations in master_df

source
JAXTAM.master_query_public — Method.
master_query_public(mission_name::Symbol, key_type::Symbol, key_value::Any)

Loads mission master table, then calls master_query_public(master_df::DataFrame, key_type::Symbol, key_value::Any)

source
JAXTAM.master_query_public — Method.
master_query_public(mission_name::Symbol)

Loads master table for mission_name, calls master_query_public(master_df::DataFrame) returning all currently public observations

source
JAXTAM.master_update — Method.
master_update(mission_name::Union{String,Symbol})

Downloads mastertable from HEASARC and overwrites old conerted tables

source
JAXTAM.master_update — Method.
master_update()

Calls master_update(mission_name::Union{String,Symbol}) using the :default mission

source
JAXTAM._add_append_analysed! — Method.
_add_append_analysed!(append_df, mission_name)

Appends column of Union{Bool,Missing}, true if the JAXTAM directory exists

TODO: Improve this function, currently an empty JAXTAM folder means it has been analysed

source
JAXTAM._add_append_cl! — Method.
_add_append_cl!(append_df, master_df, mission_name)

Appends column of Union{Tuple{String},Missing}, tuple of local paths to the cl files

source
JAXTAM._add_append_downloaded! — Method.
_add_append_downloaded!(append_df, mission_name)

Appends column of Union{Bool,Missing}, true if all cl files exist

source
JAXTAM._add_append_obspath! — Method.
_add_append_obspath!(append_df, master_df, mission_name)

Appends column of Union{String,Missing}, with the local path to the observation

source
JAXTAM._add_append_publicity! — Method.
_add_append_publicity!(append_df, master_df)

Appends column of Union{Bool,Missing}, true if public_date <=now()`

source
JAXTAM._add_append_results! — Method.
_add_append_results!(append_df, mission_name)

Appends column of String, if the results.html file exists for an observation the path to the file is returned, otherwise "NA" is returned

source
JAXTAM._add_append_uf! — Method.
_add_append_uf!(append_df, master_df, mission_name)

Appends column of Union{Tuple{String},Missing}, tuple of local paths to the uf files

source
JAXTAM._append_gen — Method.
_append_gen(mission_name, master_df)

Runs all the _add_append functions, returns the full append_df

source
JAXTAM._append_gen — Method.
_append_gen(mission_name)

Generates the append file for a mission

source
JAXTAM._append_load — Method.
_append_load(append_path_feather)

Loads a saved append_df, runs _feather2tuple() and returns the DataFrame

source
JAXTAM._append_save — Method.
_append_save(append_path_feather, append_df)

Runs _tuple2feather() on append_df then saves to the save path

source
JAXTAM._build_append — Method.
_build_append(master_df)

First step in creating append table, just returns the obsid column from a missions master table

source
JAXTAM._feather2tuple — Method.
_feather2tuple(append_df::DataFrames.DataFrame)

Function that takes in a DataFrame which has been run through _tuple2feather() and joins the split tuples together

source
JAXTAM._tuple2feather — Method.
_tuple2feather(append_df::DataFrames.DataFrame)

Feather.jl, and probably Feather files in general, can't save Tuples, this function selects and columns in the DataFrame of type Tuple, then it splits the tuples up into a DataFrame, with column names of the original column name with __tuple__$col$i appended to the end

Only works if all the tuples in a column are of the same length

TODO: Make edge cases of tuples with over 9 elements work, test methods to allow tuples of different lengths to be split and saved as well

source
JAXTAM.append — Method.
append(mission_name)

If no append file exists, crates one using the _append_gen() function, then saves the file with _append_save()

If the append file exists, loads via _append_load()

source
JAXTAM.append — Method.
append()

Calld append(mission_name) with the default mission config_dict[:default] if one exists

source
JAXTAM.append_update — Method.
append_update(mission_name)

Re-generates the append file

source
JAXTAM.master_a — Method.
master_a(mission_name)

Joins the master_df (raw, unedited HEASARC master table) and the append_df DataFrames together on :obsid, returns the joined tables

source
JAXTAM._clean_path_dots — Method.
_clean_path_dots(dir)

FTP directories use hidden dot folders frequentyl, function removes from a path for local use

source
JAXTAM._ftp_dir — Method.
_ftp_dir(mission_name::Symbol, master::DataFrames.DataFrame, obsid::String)

Same as _ftp_dir(mission_name::Symbol, obsid::String), takes in master as argument to avoid running master() to load the master table each time

source
JAXTAM._ftp_dir — Method.
_ftp_dir(mission_name::Symbol, obs_row::DataFrames.DataFrame)

Returns the HEASARC FTP server path to an observation using the mission defined path_obs function

source
JAXTAM._ftp_dir — Method.
_ftp_dir(mission_name::Symbol, obsid::String)

Uses master_query to get obs_row for obsid, calls _ftp_dir(mission_name::Symbol, obs_row::DataFrames.DataFrame)

Calls master(mission_name) each time

source
JAXTAM.download — Method.
download(mission_name::Symbol, obsids::Array; overwrite=false)

Calls master(mission_name), then download(mission_name::Symbol, master::DataFrames.DataFrame, obsids::Array; overwrite=false)

source
JAXTAM.download — Method.
download(mission_name::Symbol, master::DataFrames.DataFrame, obsids::Array; overwrite=false)

Calls download(mission_name::Symbol, master::DataFrames.DataFrame, obsid::String; overwrite=false) with an array of multiple obsids

source
JAXTAM.download — Method.
download(mission_name::Symbol, master::DataFrames.DataFrame, obsid::String; overwrite=false)

Finds the FTP server-side path via _ftp_dir, downloads folder using lftp, currently excludes the uf files assuming calibrations are up to date. Saves download folder to local, dot-free, path

source
JAXTAM.download — Method.
download(mission_name::Symbol, obsid::String; overwrite=false)

Calls master(mission_name), then calls download(mission_name::Symbol, master::DataFrames.DataFrame, obsid::String; overwrite=false)

source
JAXTAM._datetime2mjd — Method.
_datetime2mjd(human_time::Dates.DateTime)

Converts (human-readable) DataTime to MJD (Julian - 2400000.5) time

source
JAXTAM._mjd2datetime — Method.
_mjd2datetime(mjd_time::Nothing)

Returns missing if the mjd_time is Nothing

source
JAXTAM._mjd2datetime — Method.
_mjd2datetime(mjd_time::Number)

Converts MJD (Julian - 2400000.5) time to (human-readable) DataTime

source
JAXTAM.unzip! — Method.
unzip!(path)

Unzip function, using the bundled 7z.exe for Windows, and p7zip-full for is_linux

Used to unzip the mastertables after download from HEASARC

source

Mission Definitions

A mutable struct is used for the mission definition:

mutable struct MissionDefinition
    name            ::String
    url             ::String
    path            ::String
    path_obs        ::Function
    path_cl         ::Function
    path_uf         ::Function
    path_rmf        ::String
    path_web        ::String
    good_energy_max ::Number
    good_energy_min ::Number
    instruments     ::Array
end

name simply defines the mission name.

url contains the url to the HEASARC mastertable.

path contains the local path to the main mission folder.

path_obs, path_cl, and path_uf are three functions, which take in a row from a master table, and return the HEASARC server-sice path to the pbservation, cleaned files, and unfiltered files.

path_rmf is a local path to the mission RMF files in caldb.

path_web is the local path to the folder where the report pages will be saved.

good_energy_max and good_energy_min are the energies the instrument has been rated for.

instruments is an array of symbols of instrument names.