Pipeline components¶
Overview¶
Pipelines comprise of nodes that are implemented using components. A component typically only implements one unit-of-work, such as loading data, transforming data, training a model, or deploying a model to serve. The following depicts a basic pipeline in the Visual Pipeline Editor, which utilizes components to load a data file, split the file, truncates the resulting files, and counts the number of records in each file.
The same pipeline could be implemented using a single component that performs all these tasks, but that component might not be as universally re-usable. Consider, for example, that for another project the data resides in a different kind of storage. With fine-granular components you’d only have to replace the load data component with one that supports the other storage type and could retain everything else.
Elyra includes three generic components that allow for the processing of: Jupyter notebooks, Python scripts, and R scripts. These components are called generic because they can be used in all runtime environments that Elyra pipelines currently support: local/JupyterLab, Kubeflow Pipelines, and Apache Airflow.
Note: Refer to the Best practices topic in the User Guide to learn more about special considerations for generic components.
Custom components are commonly only implemented for Kubeflow Pipelines or Apache Airflow, but not both.
There are many example custom components available that you can utilize in pipelines, but you can also create your own. Details on how to create a component can be found in the Kubeflow Pipelines documentation and the Apache Airflow documentation. Do note that in Apache Airflow components are called operators, but for the sake of consistency the Elyra documentation refers to them as components.
Example custom components¶
For illustrative purposes the Elyra component registry includes a few custom components that you can use to get started.
Component details and demo pipelines can be found in the https://github.com/elyra-ai/examples repository:
Note that example components are provided as is. Unless indicated otherwise they are not maintained by the Elyra community.
Managing custom components¶
In this release Elyra utilizes an internal registry to store information about custom components that you can use in the Visual Pipeline Editor to create pipelines.
Adding a component to the registry¶
This feature is currently not available.
Modifying a component¶
This feature is currently not available.
Deleting a component from the registry¶
This feature is currently not available.