pipelinex.framework.context package¶
Submodules¶
pipelinex.framework.context.context module¶
pipelinex.framework.context.flexible_catalog_context module¶
pipelinex.framework.context.flexible_context module¶
-
class
pipelinex.framework.context.flexible_context.
FlexibleContext
(*args, **kwargs)[source]¶ Bases:
pipelinex.framework.context.flexible_parameters_context.FlexibleParametersContext
,pipelinex.framework.context.flexible_catalog_context.FlexibleCatalogContext
,pipelinex.framework.context.flexible_run_context.FlexibleRunContext
-
project_name
= 'KedroProject'¶
-
project_version
= '0.17.1'¶
-
-
class
pipelinex.framework.context.flexible_context.
MLflowFlexibleContext
(*args, **kwargs)[source]¶ Bases:
pipelinex.framework.context.flexible_context.FlexibleContext
Deprecated alias for FlexibleContext for backward-compatibility
pipelinex.framework.context.flexible_parameters_context module¶
-
class
pipelinex.framework.context.flexible_parameters_context.
FlexibleParametersContext
(*args, **kwargs)[source]¶ Bases:
kedro.framework.context.context.KedroContext
-
__init__
(*args, **kwargs)[source]¶ Create a context object by providing the root of a Kedro project and the environment configuration subfolders (see
kedro.config.ConfigLoader
)- Raises
KedroContextError – If there is a mismatch between Kedro project version and package version.
- Parameters
package_name – Package name for the Kedro project the context is created for.
project_path – Project path to define the context for.
env – Optional argument for configuration default environment to be used for running the pipeline. If not specified, it defaults to “local”.
extra_params – Optional dictionary containing extra project parameters. If specified, will update (and therefore take precedence over) the parameters retrieved from the project configuration.
-
property
params
¶ Read-only property referring to Kedro’s parameters for this context.
- Return type
Dict
[str
,Any
]- Returns
- Parameters defined in parameters.yml with the addition of any
extra parameters passed at initialization.
-
run
(*args, **kwargs)[source]¶ Runs the pipeline with a specified runner.
- Parameters
tags – An optional list of node tags which should be used to filter the nodes of the
Pipeline
. If specified, only the nodes containing any of these tags will be run.runner – An optional parameter specifying the runner that you want to run the pipeline with.
node_names – An optional list of node names which should be used to filter the nodes of the
Pipeline
. If specified, only the nodes with these names will be run.from_nodes – An optional list of node names which should be used as a starting point of the new
Pipeline
.to_nodes – An optional list of node names which should be used as an end point of the new
Pipeline
.from_inputs – An optional list of input datasets which should be used as a starting point of the new
Pipeline
.to_outputs – An optional list of output datasets which should be used as an end point of the new
Pipeline
.load_versions – An optional flag to specify a particular dataset version timestamp to load.
pipeline_name – Name of the
Pipeline
to execute. Defaults to “__default__”.
- Raises
KedroContextError – If the resulting
Pipeline
is empty or incorrect tags are provided.Exception – Any uncaught exception will be re-raised after being passed to``on_pipeline_error``.
- Returns
Any node outputs that cannot be processed by the
DataCatalog
. These are returned in a dictionary, where the keys are defined by the node outputs.
-
pipelinex.framework.context.flexible_run_context module¶
-
class
pipelinex.framework.context.flexible_run_context.
FlexibleRunContext
(package_name, project_path, env=None, extra_params=None)[source]¶ Bases:
pipelinex.framework.context.save_pipeline_json_context.SavePipelineJsonContext
,pipelinex.framework.context.flexible_run_context.StringRunnerOptionContext
,pipelinex.framework.context.flexible_run_context.OnlyMissingOptionContext
-
class
pipelinex.framework.context.flexible_run_context.
OnlyMissingOptionContext
(package_name, project_path, env=None, extra_params=None)[source]¶ Bases:
kedro.framework.context.context.KedroContext
Users can override the remaining methods from the parent class here, or create new ones (e.g. as required by plugins)
-
run
(tags=None, runner=None, node_names=None, from_nodes=None, to_nodes=None, from_inputs=None, load_versions=None, pipeline_name=None, only_missing=False)[source]¶ Runs the pipeline with a specified runner.
- Parameters
tags (
Optional
[Iterable
[str
]]) – An optional list of node tags which should be used to filter the nodes of thePipeline
. If specified, only the nodes containing any of these tags will be run.runner (
Optional
[AbstractRunner
]) – An optional parameter specifying the runner that you want to run the pipeline with.node_names (
Optional
[Iterable
[str
]]) – An optional list of node names which should be used to filter the nodes of thePipeline
. If specified, only the nodes with these names will be run.from_nodes (
Optional
[Iterable
[str
]]) – An optional list of node names which should be used as a starting point of the newPipeline
.to_nodes (
Optional
[Iterable
[str
]]) – An optional list of node names which should be used as an end point of the newPipeline
.from_inputs (
Optional
[Iterable
[str
]]) – An optional list of input datasets which should be used as a starting point of the newPipeline
.load_versions (
Optional
[Dict
[str
,str
]]) – An optional flag to specify a particular dataset version timestamp to load.pipeline_name (
Optional
[str
]) – Name of thePipeline
to execute. Defaults to “__default__”.only_missing (
bool
) – An option to run only missing nodes.
- Raises
KedroContextError – If the resulting
Pipeline
is empty or incorrect tags are provided.Exception – Any uncaught exception will be re-raised after being passed to``on_pipeline_error``.
- Return type
Dict
[str
,Any
]- Returns
Any node outputs that cannot be processed by the
DataCatalog
. These are returned in a dictionary, where the keys are defined by the node outputs.
-
-
class
pipelinex.framework.context.flexible_run_context.
StringRunnerOptionContext
(package_name, project_path, env=None, extra_params=None)[source]¶ Bases:
kedro.framework.context.context.KedroContext
Allow to specify runner by string.
-
run
(*args, runner=None, **kwargs)[source]¶ Runs the pipeline with a specified runner.
- Parameters
tags – An optional list of node tags which should be used to filter the nodes of the
Pipeline
. If specified, only the nodes containing any of these tags will be run.runner (
Union
[AbstractRunner
,str
,None
]) – An optional parameter specifying the runner that you want to run the pipeline with.node_names – An optional list of node names which should be used to filter the nodes of the
Pipeline
. If specified, only the nodes with these names will be run.from_nodes – An optional list of node names which should be used as a starting point of the new
Pipeline
.to_nodes – An optional list of node names which should be used as an end point of the new
Pipeline
.from_inputs – An optional list of input datasets which should be used as a starting point of the new
Pipeline
.to_outputs – An optional list of output datasets which should be used as an end point of the new
Pipeline
.load_versions – An optional flag to specify a particular dataset version timestamp to load.
pipeline_name – Name of the
Pipeline
to execute. Defaults to “__default__”.
- Raises
KedroContextError – If the resulting
Pipeline
is empty or incorrect tags are provided.Exception – Any uncaught exception will be re-raised after being passed to``on_pipeline_error``.
- Return type
Dict
[str
,Any
]- Returns
Any node outputs that cannot be processed by the
DataCatalog
. These are returned in a dictionary, where the keys are defined by the node outputs.
-