Runtime Configuration

Prerequisites

  • A Kubeflow Pipelines Endpoint
  • IBM Cloud Object Storage or other S3 Based Object Store (Optional)

Configuring Runtime Metadata

AI Pipelines require configuring a pipeline runtime to enable its full potential. AI Pipelines currently only support Kubeflow Pipelines with plans to expand support for other runtimes in the future.

To configure runtime metadata for Kubeflow Pipelines use the elyra-metadata install runtimes command providing appropriate options. This command will create a json file in your local Jupyter Data directory under its metadata/runtimes subdirectories. If not known, the Jupyter Data directory can be discovered by issuing a jupyter --data-dir command on your terminal.

Here’s an example invocation of elyra-metadata install runtimes --schema_name=kfp to create runtime metadata for use by Kubeflow Pipelines corresponding to the example values in the table below. Following its invocation, a file containing the runtime metadata can be found in [JUPYTER DATA DIR]/metadata/runtimes/my_kfp.json.

elyra-metadata install runtimes --schema_name=kfp \
       --name=my_kfp \
       --display_name="My Kubeflow Pipeline Runtime" \
       --api_endpoint=https://kubernetes-service.ibm.com/pipeline \
       --cos_endpoint=http://minio-service.kubeflow:9000 \
       --cos_username=minio \
       --cos_password=minio123 \
       --cos_bucket=test-bucket

This produces the following content in my_kfp.json:

{
    "display_name": "My Kubeflow Pipeline Runtime",
    "schema_name": "kfp",
    "metadata": {
        "api_endpoint": "https://kubernetes-service.ibm.com/pipeline",
        "cos_endpoint": "http://minio-service.kubeflow:9000",
        "cos_username": "minio",
        "cos_password": "minio123",
        "cos_bucket": "test-bucket"
    }
}

NOTE: In case of typing a custom bucket name using minio cloud storage, make sure the bucket name has no underscores

To validate your new configuration is available, run:

elyra-metadata list runtimes

Available metadata instances for runtimes (includes invalid):

Schema   Instance  Resource  
------   --------  -------- 
kfp      my_kfp    /Users/jdoe/Library/Jupyter/metadata/runtimes/my_kfp.json

Existing runtime metadata configurations can be removed via elyra-metadata remove runtimes --name=[runtime]:

elyra-metadata remove runtimes --name=my_kfp

Elyra depends on its runtime metadata to determine how to communicate with your KubeFlow Pipelines Server and with your chosen Object Store to store artifacts.

Parameters

api_endpoint

The KubeFlow Pipelines API Endpoint you wish to run your Pipeline.

Example: https://kubernetes-service.ibm.com/pipeline

cos_endpoint

This should be the URL address of your S3 Object Storage. If running an Object Storage Service within a kubernetes cluster (Minio), you can use the kubernetes local DNS address.

Example: https://minio-service.kubeflow:9000

cos_username

Username used to access the Object Store. SEE NOTE.

Example: minio

cos_password

Description: Password used to access the Object Store. SEE NOTE.

Example: minio123

cos_bucket

Name of the bucket you want your artifacts in. If the bucket doesn’t exist, it will be created

Example: test-bucket

NOTE: If using IBM Cloud Object Storage, you must generate a set of HMAC Credentials and grant that key at least Writer level privileges. Your access_key_id and secret_access_key will be used as your cos_username and cos_password respectively.