pipelinex.framework.context package¶

Submodules¶

pipelinex.framework.context.context module¶

pipelinex.framework.context.flexible_catalog_context module¶

class pipelinex.framework.context.flexible_catalog_context.FlexibleCatalogContext(package_name, project_path, env=None, extra_params=None)[source]¶

Bases: kedro.framework.context.context.KedroContext

Convert Kedrex’s Syntactic Sugar to pure Kedro Catalog.

pipelinex.framework.context.flexible_context module¶

class pipelinex.framework.context.flexible_context.FlexibleContext(*args, **kwargs)[source]¶

Bases: pipelinex.framework.context.flexible_parameters_context.FlexibleParametersContext, pipelinex.framework.context.flexible_catalog_context.FlexibleCatalogContext, pipelinex.framework.context.flexible_run_context.FlexibleRunContext

project_name = 'KedroProject'¶

project_version = '0.17.1'¶

class pipelinex.framework.context.flexible_context.MLflowFlexibleContext(*args, **kwargs)[source]¶

Bases: pipelinex.framework.context.flexible_context.FlexibleContext

Deprecated alias for FlexibleContext for backward-compatibility

pipelinex.framework.context.flexible_parameters_context module¶

class pipelinex.framework.context.flexible_parameters_context.FlexibleParametersContext(*args, **kwargs)[source]¶

Bases: kedro.framework.context.context.KedroContext

__init__(*args, **kwargs)[source]¶

Create a context object by providing the root of a Kedro project and the environment configuration subfolders (see kedro.config.ConfigLoader)

Raises

KedroContextError – If there is a mismatch between Kedro project version and package version.

Parameters

package_name – Package name for the Kedro project the context is created for.
project_path – Project path to define the context for.
env – Optional argument for configuration default environment to be used for running the pipeline. If not specified, it defaults to “local”.
extra_params – Optional dictionary containing extra project parameters. If specified, will update (and therefore take precedence over) the parameters retrieved from the project configuration.

property params¶

Read-only property referring to Kedro’s parameters for this context.

Return type

Dict[str, Any]

Returns

Parameters defined in parameters.yml with the addition of any: extra parameters passed at initialization.

run(*args, **kwargs)[source]¶

Runs the pipeline with a specified runner.

Parameters

tags – An optional list of node tags which should be used to filter the nodes of the Pipeline. If specified, only the nodes containing any of these tags will be run.
runner – An optional parameter specifying the runner that you want to run the pipeline with.
node_names – An optional list of node names which should be used to filter the nodes of the Pipeline. If specified, only the nodes with these names will be run.
from_nodes – An optional list of node names which should be used as a starting point of the new Pipeline.
to_nodes – An optional list of node names which should be used as an end point of the new Pipeline.
from_inputs – An optional list of input datasets which should be used as a starting point of the new Pipeline.
to_outputs – An optional list of output datasets which should be used as an end point of the new Pipeline.
load_versions – An optional flag to specify a particular dataset version timestamp to load.
pipeline_name – Name of the Pipeline to execute. Defaults to “__default__”.

Raises

KedroContextError – If the resulting Pipeline is empty or incorrect tags are provided.
Exception – Any uncaught exception will be re-raised after being passed to``on_pipeline_error``.

Returns

Any node outputs that cannot be processed by the DataCatalog. These are returned in a dictionary, where the keys are defined by the node outputs.

pipelinex.framework.context.flexible_parameters_context.import_modules(modules=None)[source]¶

pipelinex.framework.context.flexible_run_context module¶

class pipelinex.framework.context.flexible_run_context.FlexibleRunContext(package_name, project_path, env=None, extra_params=None)[source]¶: Bases: pipelinex.framework.context.save_pipeline_json_context.SavePipelineJsonContext, pipelinex.framework.context.flexible_run_context.StringRunnerOptionContext, pipelinex.framework.context.flexible_run_context.OnlyMissingOptionContext

class pipelinex.framework.context.flexible_run_context.OnlyMissingOptionContext(package_name, project_path, env=None, extra_params=None)[source]¶

Bases: kedro.framework.context.context.KedroContext

Users can override the remaining methods from the parent class here, or create new ones (e.g. as required by plugins)

run(tags=None, runner=None, node_names=None, from_nodes=None, to_nodes=None, from_inputs=None, load_versions=None, pipeline_name=None, only_missing=False)[source]¶

Runs the pipeline with a specified runner.

Parameters

tags (Optional[Iterable[str]]) – An optional list of node tags which should be used to filter the nodes of the Pipeline. If specified, only the nodes containing any of these tags will be run.
runner (Optional[AbstractRunner]) – An optional parameter specifying the runner that you want to run the pipeline with.
node_names (Optional[Iterable[str]]) – An optional list of node names which should be used to filter the nodes of the Pipeline. If specified, only the nodes with these names will be run.
from_nodes (Optional[Iterable[str]]) – An optional list of node names which should be used as a starting point of the new Pipeline.
to_nodes (Optional[Iterable[str]]) – An optional list of node names which should be used as an end point of the new Pipeline.
from_inputs (Optional[Iterable[str]]) – An optional list of input datasets which should be used as a starting point of the new Pipeline.
load_versions (Optional[Dict[str, str]]) – An optional flag to specify a particular dataset version timestamp to load.
pipeline_name (Optional[str]) – Name of the Pipeline to execute. Defaults to “__default__”.
only_missing (bool) – An option to run only missing nodes.

Raises

KedroContextError – If the resulting Pipeline is empty or incorrect tags are provided.
Exception – Any uncaught exception will be re-raised after being passed to``on_pipeline_error``.

Return type

Dict[str, Any]

Returns

Any node outputs that cannot be processed by the DataCatalog. These are returned in a dictionary, where the keys are defined by the node outputs.

class pipelinex.framework.context.flexible_run_context.StringRunnerOptionContext(package_name, project_path, env=None, extra_params=None)[source]¶

Bases: kedro.framework.context.context.KedroContext

Allow to specify runner by string.

run(*args, runner=None, **kwargs)[source]¶

Runs the pipeline with a specified runner.

Parameters

tags – An optional list of node tags which should be used to filter the nodes of the Pipeline. If specified, only the nodes containing any of these tags will be run.
runner (Union[AbstractRunner, str, None]) – An optional parameter specifying the runner that you want to run the pipeline with.
node_names – An optional list of node names which should be used to filter the nodes of the Pipeline. If specified, only the nodes with these names will be run.
from_nodes – An optional list of node names which should be used as a starting point of the new Pipeline.
to_nodes – An optional list of node names which should be used as an end point of the new Pipeline.
from_inputs – An optional list of input datasets which should be used as a starting point of the new Pipeline.
to_outputs – An optional list of output datasets which should be used as an end point of the new Pipeline.
load_versions – An optional flag to specify a particular dataset version timestamp to load.
pipeline_name – Name of the Pipeline to execute. Defaults to “__default__”.

Raises

KedroContextError – If the resulting Pipeline is empty or incorrect tags are provided.
Exception – Any uncaught exception will be re-raised after being passed to``on_pipeline_error``.

Return type

Dict[str, Any]

Returns

Any node outputs that cannot be processed by the DataCatalog. These are returned in a dictionary, where the keys are defined by the node outputs.

pipelinex.framework.context.save_pipeline_json_context module¶

class pipelinex.framework.context.save_pipeline_json_context.SavePipelineJsonContext(package_name, project_path, env=None, extra_params=None)[source]¶: Bases: kedro.framework.context.context.KedroContext