pipelinex.framework.context package

Submodules

pipelinex.framework.context.context module

pipelinex.framework.context.flexible_catalog_context module

class pipelinex.framework.context.flexible_catalog_context.FlexibleCatalogContext(package_name, project_path, env=None, extra_params=None)[source]

Bases: kedro.framework.context.context.KedroContext

Convert Kedrex’s Syntactic Sugar to pure Kedro Catalog.

pipelinex.framework.context.flexible_context module

class pipelinex.framework.context.flexible_context.FlexibleContext(*args, **kwargs)[source]

Bases: pipelinex.framework.context.flexible_parameters_context.FlexibleParametersContext, pipelinex.framework.context.flexible_catalog_context.FlexibleCatalogContext, pipelinex.framework.context.flexible_run_context.FlexibleRunContext

project_name = 'KedroProject'
project_version = '0.17.1'
class pipelinex.framework.context.flexible_context.MLflowFlexibleContext(*args, **kwargs)[source]

Bases: pipelinex.framework.context.flexible_context.FlexibleContext

Deprecated alias for FlexibleContext for backward-compatibility

pipelinex.framework.context.flexible_parameters_context module

class pipelinex.framework.context.flexible_parameters_context.FlexibleParametersContext(*args, **kwargs)[source]

Bases: kedro.framework.context.context.KedroContext

__init__(*args, **kwargs)[source]

Create a context object by providing the root of a Kedro project and the environment configuration subfolders (see kedro.config.ConfigLoader)

Raises

KedroContextError – If there is a mismatch between Kedro project version and package version.

Parameters
  • package_name – Package name for the Kedro project the context is created for.

  • project_path – Project path to define the context for.

  • env – Optional argument for configuration default environment to be used for running the pipeline. If not specified, it defaults to “local”.

  • extra_params – Optional dictionary containing extra project parameters. If specified, will update (and therefore take precedence over) the parameters retrieved from the project configuration.

property params

Read-only property referring to Kedro’s parameters for this context.

Return type

Dict[str, Any]

Returns

Parameters defined in parameters.yml with the addition of any

extra parameters passed at initialization.

run(*args, **kwargs)[source]

Runs the pipeline with a specified runner.

Parameters
  • tags – An optional list of node tags which should be used to filter the nodes of the Pipeline. If specified, only the nodes containing any of these tags will be run.

  • runner – An optional parameter specifying the runner that you want to run the pipeline with.

  • node_names – An optional list of node names which should be used to filter the nodes of the Pipeline. If specified, only the nodes with these names will be run.

  • from_nodes – An optional list of node names which should be used as a starting point of the new Pipeline.

  • to_nodes – An optional list of node names which should be used as an end point of the new Pipeline.

  • from_inputs – An optional list of input datasets which should be used as a starting point of the new Pipeline.

  • to_outputs – An optional list of output datasets which should be used as an end point of the new Pipeline.

  • load_versions – An optional flag to specify a particular dataset version timestamp to load.

  • pipeline_name – Name of the Pipeline to execute. Defaults to “__default__”.

Raises
  • KedroContextError – If the resulting Pipeline is empty or incorrect tags are provided.

  • Exception – Any uncaught exception will be re-raised after being passed to``on_pipeline_error``.

Returns

Any node outputs that cannot be processed by the DataCatalog. These are returned in a dictionary, where the keys are defined by the node outputs.

pipelinex.framework.context.flexible_parameters_context.import_modules(modules=None)[source]

pipelinex.framework.context.flexible_run_context module

class pipelinex.framework.context.flexible_run_context.FlexibleRunContext(package_name, project_path, env=None, extra_params=None)[source]

Bases: pipelinex.framework.context.save_pipeline_json_context.SavePipelineJsonContext, pipelinex.framework.context.flexible_run_context.StringRunnerOptionContext, pipelinex.framework.context.flexible_run_context.OnlyMissingOptionContext

class pipelinex.framework.context.flexible_run_context.OnlyMissingOptionContext(package_name, project_path, env=None, extra_params=None)[source]

Bases: kedro.framework.context.context.KedroContext

Users can override the remaining methods from the parent class here, or create new ones (e.g. as required by plugins)

run(tags=None, runner=None, node_names=None, from_nodes=None, to_nodes=None, from_inputs=None, load_versions=None, pipeline_name=None, only_missing=False)[source]

Runs the pipeline with a specified runner.

Parameters
  • tags (Optional[Iterable[str]]) – An optional list of node tags which should be used to filter the nodes of the Pipeline. If specified, only the nodes containing any of these tags will be run.

  • runner (Optional[AbstractRunner]) – An optional parameter specifying the runner that you want to run the pipeline with.

  • node_names (Optional[Iterable[str]]) – An optional list of node names which should be used to filter the nodes of the Pipeline. If specified, only the nodes with these names will be run.

  • from_nodes (Optional[Iterable[str]]) – An optional list of node names which should be used as a starting point of the new Pipeline.

  • to_nodes (Optional[Iterable[str]]) – An optional list of node names which should be used as an end point of the new Pipeline.

  • from_inputs (Optional[Iterable[str]]) – An optional list of input datasets which should be used as a starting point of the new Pipeline.

  • load_versions (Optional[Dict[str, str]]) – An optional flag to specify a particular dataset version timestamp to load.

  • pipeline_name (Optional[str]) – Name of the Pipeline to execute. Defaults to “__default__”.

  • only_missing (bool) – An option to run only missing nodes.

Raises
  • KedroContextError – If the resulting Pipeline is empty or incorrect tags are provided.

  • Exception – Any uncaught exception will be re-raised after being passed to``on_pipeline_error``.

Return type

Dict[str, Any]

Returns

Any node outputs that cannot be processed by the DataCatalog. These are returned in a dictionary, where the keys are defined by the node outputs.

class pipelinex.framework.context.flexible_run_context.StringRunnerOptionContext(package_name, project_path, env=None, extra_params=None)[source]

Bases: kedro.framework.context.context.KedroContext

Allow to specify runner by string.

run(*args, runner=None, **kwargs)[source]

Runs the pipeline with a specified runner.

Parameters
  • tags – An optional list of node tags which should be used to filter the nodes of the Pipeline. If specified, only the nodes containing any of these tags will be run.

  • runner (Union[AbstractRunner, str, None]) – An optional parameter specifying the runner that you want to run the pipeline with.

  • node_names – An optional list of node names which should be used to filter the nodes of the Pipeline. If specified, only the nodes with these names will be run.

  • from_nodes – An optional list of node names which should be used as a starting point of the new Pipeline.

  • to_nodes – An optional list of node names which should be used as an end point of the new Pipeline.

  • from_inputs – An optional list of input datasets which should be used as a starting point of the new Pipeline.

  • to_outputs – An optional list of output datasets which should be used as an end point of the new Pipeline.

  • load_versions – An optional flag to specify a particular dataset version timestamp to load.

  • pipeline_name – Name of the Pipeline to execute. Defaults to “__default__”.

Raises
  • KedroContextError – If the resulting Pipeline is empty or incorrect tags are provided.

  • Exception – Any uncaught exception will be re-raised after being passed to``on_pipeline_error``.

Return type

Dict[str, Any]

Returns

Any node outputs that cannot be processed by the DataCatalog. These are returned in a dictionary, where the keys are defined by the node outputs.

pipelinex.framework.context.save_pipeline_json_context module

class pipelinex.framework.context.save_pipeline_json_context.SavePipelineJsonContext(package_name, project_path, env=None, extra_params=None)[source]

Bases: kedro.framework.context.context.KedroContext