pipelinex.extras.transformers.mlflow package

Submodules

pipelinex.extras.transformers.mlflow.mlflow_io_time_logger module

class pipelinex.extras.transformers.mlflow.mlflow_io_time_logger.MLflowIOTimeLoggerTransformer(enable_mlflow=True, metric_name_prefix='_time_to_')[source]

Bases: kedro.io.transformers.AbstractTransformer

Log duration time to load and save each dataset.

__init__(enable_mlflow=True, metric_name_prefix='_time_to_')[source]
Parameters
  • enable_mlflow (bool) – Enable logging to MLflow.

  • metric_name_prefix (str) – Prefix for the metric names. The metric names are metric_name_prefix concatenated with ‘load <data_set_name>’ or ‘save <data_set_name>’

load(data_set_name, load)[source]

This method will be deprecated in Kedro 0.18.0 in favour of the Dataset Hooks before_dataset_loaded and after_dataset_loaded.

Wrap the loading of a dataset. Call load to get the data from the data set / next transformer.

Parameters
  • data_set_name (str) – The name of the data set being loaded.

  • load (Callable[[], Any]) – A callback to retrieve the data being loaded from the data set / next transformer.

Return type

Any

Returns

The loaded data.

save(data_set_name, save, data)[source]

This method will be deprecated in Kedro 0.18.0 in favour of the Dataset Hooks before_dataset_saved and after_dataset_saved.

Wrap the saving of a dataset. Call save to pass the data to the data set / next transformer.

Parameters
  • data_set_name (str) – The name of the data set being saved.

  • save (Callable[[Any], None]) – A callback to pass the data being saved on to the data set / next transformer.

  • data (Any) – The data being saved

Return type

None