pipelinex.extras.datasets package

Subpackages

• pipelinex.extras.datasets.httpx package
  • Submodules
  • pipelinex.extras.datasets.httpx.async_api_dataset module
• pipelinex.extras.datasets.opencv package
  • Submodules
  • pipelinex.extras.datasets.opencv.images_dataset module
• pipelinex.extras.datasets.pandas package
  • Submodules
  • pipelinex.extras.datasets.pandas.csv_local module
  • pipelinex.extras.datasets.pandas.efficient_csv_local module
  • pipelinex.extras.datasets.pandas.fixed_width_csv_dataset module
  • pipelinex.extras.datasets.pandas.histgram module
  • pipelinex.extras.datasets.pandas.pandas_cat_matrix module
  • pipelinex.extras.datasets.pandas.pandas_describe module
• pipelinex.extras.datasets.pandas_profiling package
  • Submodules
  • pipelinex.extras.datasets.pandas_profiling.pandas_profiling module
• pipelinex.extras.datasets.pillow package
  • Submodules
  • pipelinex.extras.datasets.pillow.images_dataset module
• pipelinex.extras.datasets.requests package
  • Submodules
  • pipelinex.extras.datasets.requests.api_dataset module
• pipelinex.extras.datasets.seaborn package
  • Submodules
  • pipelinex.extras.datasets.seaborn.seaborn_pairplot module
• pipelinex.extras.datasets.torchvision package
  • Submodules
  • pipelinex.extras.datasets.torchvision.iterable_images_dataset module

Submodules

pipelinex.extras.datasets.core module

class pipelinex.extras.datasets.core.AbstractDataSet

Bases: abc.ABC

AbstractDataSet is the base class for all data set implementations. All data set implementations should extend this abstract class and implement the methods marked as abstract.

Example:

from kedro.io import AbstractDataSet
import pandas as pd

class MyOwnDataSet(AbstractDataSet):
    def __init__(self, param1, param2):
        self._param1 = param1
        self._param2 = param2

    def _load(self) -> pd.DataFrame:
        print("Dummy load: {}".format(self._param1))
        return pd.DataFrame()

    def _save(self, df: pd.DataFrame) -> None:
        print("Dummy save: {}".format(self._param2))

    def _describe(self):
        return dict(param1=self._param1, param2=self._param2)

exists()

Checks whether a data set’s output already exists by calling the provided _exists() method.

Return type:

bool

Returns:

Flag indicating whether the output already exists.

Raises:

DataSetError – when underlying exists method raises error.

classmethod from_config(name, config, load_version=None, save_version=None)

Create a data set instance using the configuration provided.

Parameters:

• name (str) – Data set name.

• config (Dict[str, Any]) – Data set config dictionary.

• load_version (Optional[str]) – Version string to be used for load operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.

• save_version (Optional[str]) – Version string to be used for save operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.

Return type:

AbstractDataSet

Returns:

An instance of an AbstractDataSet subclass.

Raises:

DataSetError – When the function fails to create the data set from its config.

load()

Loads data by delegation to the provided load method.

Return type:

Any

Returns:

Data returned by the provided load method.

Raises:

DataSetError – When underlying load method raises error.

release()

Release any cached data.

Raises:

DataSetError – when underlying release method raises error.

Return type:

None

save(data)

Saves data by delegation to the provided save method.

Parameters:

data (Any) – the value to be saved by provided save method.

Raises:

DataSetError – when underlying save method raises error.

Return type:

None

class pipelinex.extras.datasets.core.AbstractVersionedDataSet(filepath, version, exists_function=None, glob_function=None)

Bases: pipelinex.extras.datasets.core.AbstractDataSet, abc.ABC

AbstractVersionedDataSet is the base class for all versioned data set implementations. All data sets that implement versioning should extend this abstract class and implement the methods marked as abstract.

Example:

from kedro.io import AbstractVersionedDataSet
import pandas as pd


class MyOwnDataSet(AbstractVersionedDataSet):
    def __init__(self, param1, param2, filepath, version):
        super().__init__(filepath, version)
        self._param1 = param1
        self._param2 = param2

    def _load(self) -> pd.DataFrame:
        load_path = self._get_load_path()
        return pd.read_csv(load_path)

    def _save(self, df: pd.DataFrame) -> None:
        save_path = self._get_save_path()
        df.to_csv(str(save_path))

    def _exists(self) -> bool:
        path = self._get_load_path()
        return path.is_file()

    def _describe(self):
        return dict(version=self._version, param1=self._param1, param2=self._param2)

__init__(filepath, version, exists_function=None, glob_function=None)

Creates a new instance of AbstractVersionedDataSet.

Parameters:

• filepath (PurePath) – Path to file.

• version (Optional[Version]) – If specified, should be an instance of kedro.io.core.Version. If its load attribute is None, the latest version will be loaded. If its save attribute is None, save version will be autogenerated.

• exists_function (Optional[Callable[[str], bool]]) – Function that is used for determining whether a path exists in a filesystem.

• glob_function (Optional[Callable[[str], List[str]]]) – Function that is used for finding all paths in a filesystem, which match a given pattern.

exists()

Checks whether a data set’s output already exists by calling the provided _exists() method.

Return type:

bool

Returns:

Flag indicating whether the output already exists.

Raises:

DataSetError – when underlying exists method raises error.

load()

Loads data by delegation to the provided load method.

Return type:

Any

Returns:

Data returned by the provided load method.

Raises:

DataSetError – When underlying load method raises error.

resolve_load_version()

Compute and cache the version the dataset should be loaded with.

Return type:

Optional[str]

resolve_save_version()

Compute and cache the version the dataset should be saved with.

Return type:

Optional[str]

save(data)

Saves data by delegation to the provided save method.

Parameters:

data (Any) – the value to be saved by provided save method.

Raises:

DataSetError – when underlying save method raises error.

Return type:

None

exception pipelinex.extras.datasets.core.DataSetAlreadyExistsError

Bases: pipelinex.extras.datasets.core.DataSetError

DataSetAlreadyExistsError raised by DataCatalog class in case of trying to add a data set which already exists in the DataCatalog.

exception pipelinex.extras.datasets.core.DataSetError

Bases: Exception

DataSetError raised by AbstractDataSet implementations in case of failure of input/output methods.

AbstractDataSet implementations should provide instructive information in case of failure.

exception pipelinex.extras.datasets.core.DataSetNotFoundError

Bases: pipelinex.extras.datasets.core.DataSetError

DataSetNotFoundError raised by DataCatalog class in case of trying to use a non-existing data set.

class pipelinex.extras.datasets.core.Version(load, save)

Bases: pipelinex.extras.datasets.core.Version

This namedtuple is used to provide load and save versions for versioned data sets. If Version.load is None, then the latest available version is loaded. If Version.save is None, then save version is formatted as YYYY-MM-DDThh.mm.ss.sssZ of the current timestamp.

exception pipelinex.extras.datasets.core.VersionNotFoundError

Bases: pipelinex.extras.datasets.core.DataSetError

VersionNotFoundError raised by AbstractVersionedDataSet implementations in case of no load versions available for the data set.

pipelinex.extras.datasets.core.generate_timestamp()

Generate the timestamp to be used by versioning.

Return type:

str

Returns:

String representation of the current timestamp.

pipelinex.extras.datasets.core.get_filepath_str(path, protocol)

Returns filepath. Returns full filepath (with protocol) if protocol is HTTP(s).

Parameters:

• path (PurePath) – filepath without protocol.

• protocol (str) – protocol.

Return type:

str

Returns:

Filepath string.

pipelinex.extras.datasets.core.get_protocol_and_path(filepath, version=None)

Parses filepath on protocol and path.

Parameters:

• filepath (str) – raw filepath e.g.: gcs://bucket/test.json.

• version (Optional[Version]) – instance of kedro.io.core.Version or None.

Return type:

Tuple[str, str]

Returns:

Protocol and path.

Raises:

• DataSetError – when protocol is http(s) and version is not None.

• Note – HTTP(s) dataset doesn’t support versioning.

pipelinex.extras.datasets.core.parse_dataset_definition(config, load_version=None, save_version=None)

Parse and instantiate a dataset class using the configuration provided.

Parameters:

• config (Dict[str, Any]) – Data set config dictionary. It must contain the type key with fully qualified class name.

• load_version (Optional[str]) – Version string to be used for load operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.

• save_version (Optional[str]) – Version string to be used for save operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.

Raises:

DataSetError – If the function fails to parse the configuration provided.

Returns:

(Dataset class object, configuration dictionary)

Return type:

2-tuple

pipelinex.extras.datasets.core.validate_on_forbidden_chars(**kwargs)

Validate that string values do not include white-spaces or ;