Deploying Kubeflow Pipelines Locally for Elyra

Elyra’s pipeline editor depends on runtimes like Kubeflow to properly execute its pipelines. In the example, we will be deploying Kubeflow Pipelines on Kubernetes using Docker Desktop. Note that these instructions will ONLY install the Kubeflow Pipelines component.

Requirements

• Docker Desktop
  • Available for MacOS and Windows
• kubectl
  • Available for MacOS
  • Note: Windows users should skip this step since Docker Desktop adds its own version of kubectl to PATH

Enabling Kubernetes on Docker Desktop

After installing our requirements, we want to enable kubernetes in Docker Desktop.

In this example, we will be performing the steps on a MacOS system

1. In the upper right corner locate the Docker Desktop Icon, Click and go to PreferencesElyra
2. Navigate to the Resources sub-menu and ensure that Docker Desktop has at least4 CPUs, 8 GB of Memory and 1 GB of Swap. If not, increase as necessary and click Apply & Restart.Elyra
3. Navigate to the Kubernetes sub-menu and click Enable Kubernetes and hit Apply & RestartElyra
4. Docker Desktop should now install a single node deployment of Kuberneteson your system and configure your kubectl to the correct local kubernetes clusterElyraElyra
5. Verify that your cluster is up and running and configured correctly by runningkubectl get all --all-namespaces and verifying that the Docker Desktop pods are present and in Running stateElyra

Install KubeFlow Pipelines

• Deploy Kubeflow

export PIPELINE_VERSION=0.5.1
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -k "github.com/elyra-ai/elyra/etc/kubernetes/kubeflow-pipelines?ref=master"

• Get status of the Kubeflow deployment and ensure all pods are running before proceeding.Deployment times vary from system to system so please be patient when the pods are starting up.

kubectl get all -n kubeflow

• Setup port forwarding to use the Kubeflow Pipelines UI / API

kubectl port-forward $(kubectl get pods -n kubeflow | grep ml-pipeline-ui | cut -d' ' -f1) 31380:3000 -n kubeflow &

• Add minio-service to your local hosts file

echo '127.0.0.1  minio-service' | sudo tee -a /etc/hosts

• Setup port forwarding to use the Minio Object Service with Kubeflow

kubectl port-forward $(kubectl get pods -n kubeflow | grep minio | cut -d' ' -f1) 9000:9000 -n kubeflow &

• Your Kubeflow Pipelines API and Minio Object Store endpoints should be respectively

UI Endpoint: http://localhost:31380
API Endpoint: http://localhost:31380/pipeline
Object Storage Endpoint: http://minio-service:9000

These endpoints will be used to configure your Elyra metadata runtime with the command below:

elyra-metadata install runtimes --replace=true \
       --schema_name=kfp \
       --name=kfp-local \
       --display_name="Kubeflow Pipeline (local)" \
       --api_endpoint=http://localhost:31380/pipeline \
       --cos_endpoint=http://minio-service:9000 \
       --cos_username=minio \
       --cos_password=minio123 \
       --cos_bucket=covid

Troubleshooting

If the kubernetes cluster fails to start you may need to analyze the container logs to figure out which component is failing.

1. To do this, go back to your Kubernetes sub-menu under Preferences
2. Check the box for Show System Containers (Advanced) and click Apply and Restart Elyra
3. Open a terminal window and run docker ps. This should show a list of containers currently running in Docker as well as the containers that are part of your kubernetes cluster. Elyra
4. To view the logs of a container, use

docker logs <CONTAINER ID>

The output of these logs should hopefully assist with any issues you may encounter.

Additional Resources and Documentation

Docker Desktop Installation DocsKubeFlow Installation Docs