Rawdata

Functions and classes to handle the AQG's raw data files

`AQGRawData`

Dataframe of AQG rawdata

This is a wrapper class for a pandas.DataFrame and a dictionary of metadata.

Attributes:

Name	Type	Description
`df`	`DataFrame`	Full raw data
`metadata`	`dict`	All metadata. Created from the .info file.
`name`	`str`	Dataset identifier.
`_modified`	`bool`	True, when columns where modified and the corrected signal has to be recalculated.
`log_entries`	`list[str]`	List of log messages explaining processing steps taken.

`altitude` `property`

Location altitude, in meters

`const_corrections` `property`

Constant gravity corrections in the metadata

`coordinate` `property`

Location latitude and longitude, in degree

`dg_syst` `property`

Const. systematic bias correction

`meter` `property`

Instrument identifier

`orientation` `property`

Measurement setup orientation

`point` `property`

Measurement point identifier

`pressure_admittance` `property`

Atmospheric pressure admittance factor

`start_time` `property`

Time when the first data was recorded

`tilt_offset` `property`

Instrument tiltmeter offset angles

`variable_corrections` `property`

List of correction columns in the dataframe

`vgg` `property`

Vertical gravity gradient

`apply_corrections()`

Calculate corrected gravity signal by applying all corrections

`calculate_dg_polar()`

Recalculate the gravity correction due to polar motion

`calculate_dg_pressure()`

Recalculate the gravity correction due to atmospheric pressure

`calculate_dg_tilt()`

Recalculate the instrument tilt gravity correction

`detect_outliers(g_true=None, g_std=None, threshold=3000, sigma_threshold=3, neighbors=15, min_gap='30s')`

Identify and mark outliers in the gravity signal

Drops that deviate from the mean g_true by more than g_std * sigma_threshold are considered outliers.

Parameters:

Name	Type	Description	Default
`g_true`	`float`	Gravity value to take as reference for calculation of outlier deviation. Pass `None` to have this value calculated automatically.	`None`
`g_std`	`float`	Gravity signal standard deviation to use for sigma-criterion. Pass `None` for automatic calculation.	`None`
`threshold`	`float`	Absolute threshold for maximum deviation.	`3000`
`sigma_threshold`	`float`	Multiplicator for standard deviation `g_std` defining outlier threshold.	`3`
`neighbors`	`int`	Number of drops neighboring an outlier to also mark as outliers.	`15`
`min_gap`	`str`	Minimum time gap between two outliers to be considered as separate events. Specify as `pandas.Timedelta` argument, e. g. '60s'.	`'30s'`

`from_files(info_path, raw_csv_paths, csv_paths, name, source=None)` `classmethod`

Read an AQGRawData instance from rawdata files

Parameters:

Name	Type	Description	Default
`info_path`	`str \| Path \| Path`	Path to AQG .info file.	required
`raw_csv_paths`	`list[str \| Path \| Path]`	Paths to AQG _raw.csv files with one row per drop.	required
`csv_paths`	`list[str \| Path \| Path]`	Paths to AQG .csv files of preprocessed data.	required
`name`	`str`	Dataset identifier.	required
`source`	`str \| Path`	Data source identifier. If unspecified, `info_path` is taken.	`None`

Returns:

Type	Description
`AQGRawData`	Raw data object.

`limit_time_period(t_from=None, t_to=None)`

Limit the data time period to a certain interval

Parameters:

Name	Type	Description	Default
`t_from`	`datetime \| str`	Start time ...	`None`
`t_to`	`datetime \| str`	... and end time to which the data should be adjusted to. If provided as a string, this has to be in a format that is automatically parsable by standard conversion functions.	`None`

`log(text)`

Add a message to the processing log

`mark_outlier_range(t_start, t_end)`

Mark a segment of drops as outliers

`mark_outlier_ranges(time_ranges)`

Mark a list of time ranges as outliers

Parameters:

Name	Type	Description	Default
`time_ranges`	`list[tuple \| str]`	List of time ranges. Elements can be a tuple of two timestamps, or a string of format `START .. END`, where `START` and `END` are inputs to `pd.to_datetime()`.	required

`process(*, change_tilt_offset=None, recalculate_dg_pressure=True, recalculate_dg_polar=True, detect_outliers=None, outlier_ranges=None, station_height_difference=None, time_period=None, **metadata)`

Apply a standard processing procedure to the dataset

Parameters:

Name	Type	Description	Default
`change_tilt_offset`	`tuple[float, float]`	Recalculate tilt angles and tilt correction for new tiltmeter offset.	`None`
`recalculate_dg_pressure`	`bool`	Whether to call `recalculate_dg_pressure()`.	`True`
`recalculate_dg_polar`	`bool`	Whether to call `recalculate_dg_polar()`.	`True`
`detect_outliers`	`dict \| False`	Parameters for outlier detection. Pass a dictionary of parameters to override defaults of `detect_outliers()`. Pass `None` to use default parameters. Set `False` to deactivate.	`None`
`outlier_ranges`	`list`	List of time ranges to mark as outliers. See `mark_outlier_ranges()`.	`None`
`station_height_difference`	`float`	Offset to apply to the measurement reference height.	`None`
`time_period`	`tuple[str \| datetime, str \| datetime]`	Tuple of start and end timestamps according to which the time series will be cut.	`None`
`**metadata`	`dict`	Parameters to pass on to `replace_metadata()`.	`{}`

Returns:

Type	Description
`AQGRawData`	Itself.

`recalculate_tilt(tilt_offset)`

Recalculate the tilt angles for a new tilt offset

Parameters:

Name	Type	Description	Default
`tilt_offset`	`tuple[float, float]`	New tilt offset angles, in radian.	required

In order to reclculate tilt angles with a new tilt offset, the original tilt offset has to be known. This is currently not part of the AQG .info files, but can be found in the global AQG configruation file settings.ini.

Set the original tilt offset manually by specifying it in the metadata:

AQGRawData.metadata["tilt_offset"] = 1.234e-3, 0.123e-3  # radians

`replace_metadata(*, default_orientation=None, orientation=None, pressure_admittance=None, syst_uncertainty=None, vgg=None, tilt_offset=None, **kwargs)`

Replace metadata parameters

Parameters:

Name	Type	Description	Default
`orientation`	`int`	Sensor head setup orientation, in degree.	`None`
`default_orientation`	`int`	Default sensor head setup orientation, if it is not yet explicitly specified.	`None`
`pressure_admittance`	`float`	Atmospheric pressure admittance factor, in nm/s²/hPa.	`None`
`vgg`	`float \| str`	Gravity gradient, in nm/s²/m.	`None`
`tilt_offset`	`(tuple, str)`	Instrument tiltmeter offset angles this dataset was recorded with. String is split by commas.	`None`
`**kwargs`	`dict`	Additional metadata parameters. Parameters starting with prefix `dg_`, such as `dg_syst`, are interpreted as constant gravity corrections.	`{}`

`set_const_correction(name, value)`

Set a constant bias correction

`set_dg_earth_tide(dg_earth_tide, name)`

Replace the earth tide gravity correction time series

The time series will be linearly interpolated to determine a correction for each drop.

Parameters:

Name	Type	Description	Default
`dg_earth_tide`	`Series`	Time series of gravity corrections.	required
`name`	`str`	Tide model identifier name.	required

`set_pressure_admittance(admittance)`

Change the admittance factor used for atmospheric pressure correction

Parameters:

Name	Type	Description	Default
`admittance`	`float \| str`	New admittance factor. Pass a float value in units of nm/s²/hPa. Or pass a string value with adequate units, that can be parsed by `parse_pressure_admittance()`.	required

`shift_reference_height(height_offset)`

Apply an offset to the measurement reference height

`to_dataset()`

Create a finalized dataset of processing results

Converts the pandas.DataFrame to an xarray.Dataset with metadata.

Returns:

Type	Description
`AQGDataset`	Dataset of processed results

`process_aqg_raw_dataset(path, *, name=None, config=None)`

Read and process an AQG raw dataset from a directory or zip-archive

Applies standard processing procesdure. See process().

Parameters:

Name	Type	Description	Default
`path`	`str \| Path \| Path`	Path to directory or zip-archive containing AQG raw data.	required
`name`	`str`	Dataset identifier. If unspecified, the name is infered from the parent directory of the data files.	`None`
`config`	`dict`	Processing configuration parameters. The relevant parameters are selected using combine_dataset_config.	`None`

Returns:

Type	Description
`AQGDataset`	Processed AQG dataset

`read_aqg_raw_dataset(path, name=None)`

Read an AQG raw dataset from a directory or zip-archive

Parameters:

Name	Type	Description	Default
`path`	`str \| Path \| Path`	Path to directory or zip-archive containing AQG raw data.	required
`name`	`str`	Dataset identifier. If unspecified, the name is infered from the parent directory of the data files.	`None`

Returns:

Type	Description
`AQGRawData`	Unprocessed AQG dataset

Rawdata

AQGRawData

altitude property

const_corrections property

coordinate property

dg_syst property

meter property

orientation property

point property

pressure_admittance property

start_time property

tilt_offset property

variable_corrections property

vgg property

apply_corrections()

calculate_dg_polar()

calculate_dg_pressure()

calculate_dg_tilt()

detect_outliers(g_true=None, g_std=None, threshold=3000, sigma_threshold=3, neighbors=15, min_gap='30s')

from_files(info_path, raw_csv_paths, csv_paths, name, source=None) classmethod

limit_time_period(t_from=None, t_to=None)

log(text)

mark_outlier_range(t_start, t_end)

mark_outlier_ranges(time_ranges)

process(*, change_tilt_offset=None, recalculate_dg_pressure=True, recalculate_dg_polar=True, detect_outliers=None, outlier_ranges=None, station_height_difference=None, time_period=None, **metadata)

recalculate_tilt(tilt_offset)

replace_metadata(*, default_orientation=None, orientation=None, pressure_admittance=None, syst_uncertainty=None, vgg=None, tilt_offset=None, **kwargs)

set_const_correction(name, value)

set_dg_earth_tide(dg_earth_tide, name)

set_pressure_admittance(admittance)

shift_reference_height(height_offset)

to_dataset()

process_aqg_raw_dataset(path, *, name=None, config=None)

read_aqg_raw_dataset(path, name=None)

`AQGRawData`

`altitude` `property`

`const_corrections` `property`

`coordinate` `property`

`dg_syst` `property`

`meter` `property`

`orientation` `property`

`point` `property`

`pressure_admittance` `property`

`start_time` `property`

`tilt_offset` `property`

`variable_corrections` `property`

`vgg` `property`

`apply_corrections()`

`calculate_dg_polar()`

`calculate_dg_pressure()`

`calculate_dg_tilt()`

`detect_outliers(g_true=None, g_std=None, threshold=3000, sigma_threshold=3, neighbors=15, min_gap='30s')`

`from_files(info_path, raw_csv_paths, csv_paths, name, source=None)` `classmethod`

`limit_time_period(t_from=None, t_to=None)`

`log(text)`

`mark_outlier_range(t_start, t_end)`

`mark_outlier_ranges(time_ranges)`

`process(*, change_tilt_offset=None, recalculate_dg_pressure=True, recalculate_dg_polar=True, detect_outliers=None, outlier_ranges=None, station_height_difference=None, time_period=None, **metadata)`

`recalculate_tilt(tilt_offset)`

`replace_metadata(*, default_orientation=None, orientation=None, pressure_admittance=None, syst_uncertainty=None, vgg=None, tilt_offset=None, **kwargs)`

`set_const_correction(name, value)`

`set_dg_earth_tide(dg_earth_tide, name)`

`set_pressure_admittance(admittance)`

`shift_reference_height(height_offset)`

`to_dataset()`

`process_aqg_raw_dataset(path, *, name=None, config=None)`

`read_aqg_raw_dataset(path, name=None)`