Processing AQG raw data

This part shows basic functionalities of reading and processing AQG data. This workflow is related to the AQGRawData-class. After processing is finished, the raw data is converted to a AQGDataset. Furher usage of this can be found in the documentation on "Handling processed AQG data" (next section).

Reading AQG raw data

You can load data from a folder, containing the raw measurement data:

from gravitools.aqg import read_aqg_raw_dataset
from gravitools.corrections import read_eop_data

path_to_raw_dataset = "../data/20240620_163341/"
# please note: reading EOP data explicitly is only necessary for this documentation (due to missing local file system)
read_eop_data('../data/finals2000A.all.csv')
raw = read_aqg_raw_dataset(path_to_raw_dataset)
raw

<AQGRawData: 20240620_163341>

Otherwise data can also be loaded directly from a compressed file (of the raw measurement data):

from gravitools.aqg import read_aqg_raw_dataset

path_to_raw_dataset = "../data/20240620_163341.zip"
raw = read_aqg_raw_dataset(path_to_raw_dataset)
raw

<AQGRawData: 20240620_163341>

AQGRawData has an attribute name, which is a unique string that identifies the dataset. By default, its value is infered from the source filename.

raw.name

'20240620_163341'

AQGRawData wraps a pandas.DataFrame, which can also be accessed directly.

raw.df.iloc[0]

gps_timestamp                                 0.0
gps_state                                       0
ref_time_mode                                   S
g_raw                                  9808372019
dg_tilt                                        -1
dg_pressure                                   -21
dg_earth_tide                                 637
dg_polar                                        0
atmospheric_pressure                       949.56
external_temperature                          NaN
sensor_head_temperature                      37.2
vacuum_chamber_temperature                   48.4
tiltmeter_temperature                        38.1
x_tilt                                    0.00001
y_tilt                                  -0.000001
sat_abs_offset_correction                 -890000
sat_abs_offset_correction_applied               1
raw_quartz_frequency                 99999995.981
cooler_phi                                 -0.076
cooler_eta                                  0.098
rep_phi                                     1.544
rep_eta                                       NaN
atoms_number                                68634
laser_temperature                             NaN
is_outlier                                  False
Name: 2024-06-20 16:33:41.790999889+00:00, dtype: object

Metadata from the .info-file of the AQG raw data is mapped to the AQGRawData.metadata attribute, which is a simple dictionary.

raw.metadata["software"]

'1.7.4'

Perform standard data processing

raw.process()

<AQGRawData: 20240620_163341>

You can also configure this procedure as needed

raw.process(pressure_admittance=-2.9)

<AQGRawData: 20240620_163341>

Since the configuration parameters are passed as a dictionary, they can of course also be read from a file format that maps to a dictionary, such as JSON or YAML. You can thus load a configuration file and pass it to the processing command. For how config files are setup and used, please see section Configuration files.

from gravitools.config import read_config

# load config
config = read_config("../data/config_file.yml")

# processes with loaded configuration parameters
raw.process(**config)

<AQGRawData: 20240620_163341>

Finalize the dataset

The class AQGRawData is a wrapper to a pandas.DataFrame. It includes the metadata and provides methods for manipulating the data frame. It is intended for processing the data set. Once this stage of the workflow is complete, convert the raw data to an AQGDataset instance, which represents a finalized processed dataset.

dataset = raw.to_dataset()
dataset

<AQGDataset: 20240620_163341>

Save to file

AQGDataset.to_nc() exports the dataset as NETCDF4, a binary data format. The resulting file contains the full signal of all data columns and all relevant metadata. It is also transparently compressed.

dataset.to_nc("../data/test.nc")

Custom data processing

You do not have to run AQGRawData.process(). It is merely a convenience method to run a standard sequence of steps and accepts configuration parameters for these.

For instance, you can do the workflow without any processing whatsoever and keep the original corrections.

raw = read_aqg_raw_dataset(path_to_raw_dataset)
dataset = raw.to_dataset()