PipelineX Overview

PipelineX is a Python package to build ML pipelines for experimentation with Kedro, MLflow, and more

PipelineX provides the following options which can be used independently or together.

• HatchDict: Python in YAML/JSON

  HatchDict is a Python dict parser that enables you to include Python objects in YAML/JSON files.

  Note: HatchDict can be used with or without Kedro.

• Flex-Kedro: Kedro plugin for flexible config

  • Flex-Kedro-Pipeline: Kedro plugin for quicker pipeline set up

  • Flex-Kedro-Context: Kedro plugin for YAML lovers

• MLflow-on-Kedro: Kedro plugin for MLflow users

  MLflow-on-Kedro provides integration of Kedro with MLflow with Kedro DataSets and Hooks.

  Note: You do not need to install MLflow if you do not use.

• Kedro-Extras: Kedro plugin to use various Python packages

  Kedro-Extras provides Kedro DataSets, decorators, and wrappers to use various Python packages such as:

  • <PyTorch>

  • <Ignite>

  • <Pandas>

  • <OpenCV>

  • <Memory Profiler>

  • <NVIDIA Management Library>

  Note: You do not need to install Python packages you do not use.

Please refer here to find out how PipelineX differs from other pipeline/workflow packages: Airflow, Luigi, Gokart, Metaflow, and Kedro.