Data Scientist Dashboard & Workflow

Data Scientist Menu

The Data Scientist Screen provides a GUI-based mechanism to navigate through the workflow.

_images/Data_Scientist_Menu.jpg
Dashboard Overview of the current status of the workflow
Workspaces Access and view program code
Datasets Access and view training datasets
Notebooks Experiment with models, datasets, & hyperparameters
Jobs Train model with large dataset and hyperparameters
Models Trained models, ready for Inference or transfer learning
Inferences Deployed models
Pipelines Kubeflow pipelines
Experiments Experiments using pipelines

Data Scientist Dashboard

The Data Scientist dashboard provides an overview of the current state of the workflow.

_images/Data_Scientist_Dashboard.jpg

The dashboard has the following sections:

  • Total number of Notebooks, Training Runs, Inferences, & resources, and how many are active
  • Quick access to the detailed screen for the Notebooks and Training Runs
  • Overall usage information for the user

Data Scientist Workflow

The standard workflow for a data scientist using DKube is as follows:

  • Load the workspaces and datasets into the DKube (Section Workspaces)
  • Create Notebook (Section Create Notebook)
    • Experiment with model using different hyperparameters
  • Create Job (Section Create Training Job)
    • Formal training session using larger datasets
  • Deploy to Inference (Section Deploy Model)

Shared Data

Users in a Group share models, datasets, notebooks, jobs & inferences. These are shown on the screens for each type of data. At the top right-hand of the screen there is a drop-down menu that allows the User to select what is visible: just the User’s data, just the shared data, or both.

_images/Data_Scientist_Ownership.jpg

Hyperparameter Optimization

DKube implements Katib-based hyperparameter optimization. This enables automated tuning of hyperparameters for a model, based upon target objectives.

This is described in more detail at Katib Introduction.

The section Hyperparameter Optimization provides the details on how to use this feature from within DKube.

Example Model and Dataset Locations

Some example models and datasets are available from One Convergence.

Mnist Digits Classification

This example classifies digits from images. It can be used to demonstrate:

  • Training
  • Inference
  • Hyperparameter Tuning
  • Kubeflow Pipelines

Note

The test images for this model are in a specific format, so please use the images supplied

Model https://github.com/oneconvergence/dkube-examples/tree/master/tensorflow/classification/mnist/digits/classifier/program
Training Dataset https://github.com/oneconvergence/dkube-examples/tree/master/tensorflow/classification/mnist/digits/classifier/data
Test Images https://github.com/oneconvergence/dkube-examples/tree/master/tensorflow/classification/mnist/digits/inference
Hyperparameter Tuning File http://oneconvergence.com/guide/downloads/mnist-tuning.yaml (see section Hyperparameter Tuning)
Pipeline File Included in DKube Notebook (see section Kubeflow Pipelines)

Cat & Dog Classification

This example classifies an image as either a dog or a cat. It can be used to demonstrate:

  • Training
  • Inference
  • Hyperparameter Tuning
  • Kubeflow Pipelines
Model https://github.com/oneconvergence/dkube-examples/tree/master/tensorflow/classification/resnet/catsdogs/classifier/program
Training Dataset https://github.com/oneconvergence/dkube-examples/tree/master/tensorflow/classification/resnet/catsdogs/classifier/data
Test Images Any image in jpg, png, or tiff format
Hyperparameter Tuning File http://oneconvergence.com/guide/downloads/catsdogs-tuning.yaml (see section Hyperparameter Tuning)
Pipeline File Included in DKube Notebook (see section Kubeflow Pipelines)

Pet Object Detection

This example detects pet breeds from images. It can be used to demonstrate:

  • Preprocessing
  • Training
  • Inference

The instructions are available at https://github.com/oneconvergence/dkube-examples/tree/master/tensorflow/object-detection/pets

Repos

There are 2 types of repositories in the navigation screen: Workspaces and Datasets. In order to use them, they must first be accessible by DKube. There are different ways that the repositories can be accessed by DKube. These are described in more detail in the sections that describe each input source.

  • They can be downloaded to the internal DKube storage area. In this case, the data becomes part of DKube and cannot be updated externally.
  • They can accessed as a pointer to a folder or file. In this case, the files can be updated by an external application and DKube will access the new version of the file on the next run.

Workspaces

_images/Data_Scientist_Workspaces.jpg

Workspaces contain the programs that are available within DKube to trains models using datasets.

The Workspaces screen provides the details on the programs that have been downloaded or linked by the current User, and programs that have been downloaded by other Users in the same Group.

The details on the program can be accessed by selecting the name.

_images/Data_Scientist_Workspaces_Detail.jpg

Add a Program to the Workspace

_images/Data_Scientist_Workspaces_New.png

In order to add a program to the Workspace, and use it within DKube:

  • Select the “+ Workspace” button at the top right-hand part of the screen
  • Fill in the information necessary to access the Program
  • Select the “Add Workspace” button

Fields for Adding a Program

The input fields for adding the Program depend upon the source. The program is downloaded into the internal DKube storage.

GitHub

The GitHub option copies the files in a directory into the local DKube store.

Name Unique user-chosen identification for the program
URL url of the GitHub directory that contains the program code. An example GitHub url is https://github.com/oneconvergence/dkube-examples/tree/master/tensorflow/classification/mnist/digits/classifier/program
Tag Optional detailed field to allow grouping
Access Token Auth token to use if the URL is in a private repo. The following link provides a reference on how to create the token: Create GitHub Token

K8S Volume

DKube can access a k8s persistent volume created for the user. The files are not downloaded from the volume.

Name Unique user-chosen identification for the program
Volume Persistent volume directory created for user
Tag Optional detailed field to allow grouping

DKube

The DKube input source provides a way to access the program from a specific location, rather than from a repository. The program would first be copied to the DKube storage volume, or created by an external application, and the program is then referenced by DKube. The program code is not copied into the DKube data base, but rather referenced from its location. If the program is changed by an external program, the updated program will be used by DKube in subsequent notebooks and jobs.

Name Unique user-chosen identification for the program
Path The path to the program directory. The is a relative path from the DKube user storage folder. So, for example, if the folder is located at “/root/store/users/<username>/<program>/<program name>”, the path should be specified as “<program>/<program name>”
Tag Optional detailed field to allow grouping

S3 File System

The fields are compatible with the standard S3 File System. The files can be attached remotely or copied into the local DKube store, depending upon the “remote” field selection. The files can be:

AWS Check this box if the Program is on AWS
Name Unique user-chosen identification for the program
Access Key https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html
Secret Key https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html
Bucket https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html
Prefix/Subpath Folder within the bucket. If the bucket is called “examples”, and it has a sub-folder called “mnist”, with a further sub-folder called “program”, the syntax for this field is “mnist/program”.
Tag Optional detailed field to allow grouping
Endpoint If not AWS, url of the Minio server endpoint
Use as remote Check to mount the Program (remote), otherwise copy into DKube

Workstation

This input source uploads a file from the workstation that is running the browser.

Name Unique user-chosen identification for the program
Select File Select the file to be uploaded
Extract uploaded file Check this box to extract the files from a compressed archive

Note

The Workstation input source will upload a single file. In order to upload multiple files in one step, an archive file (TAR or ZIP) can be created that combines the multiple files into one. If the uploaded archive file is a combination of multiple files, the checkbox “Extract upload file” should be selected. DKube will then uncompress the single archive file into multiple files. If the “Extract upload file” is not checked, the archive will be treated a single file, with the assumptions that the model code will use it as it is.

Other

This input source provides the ability to access a Program from any publicly available url. The Program will be downloaded to DKube.

Name Unique user-chosen identification for the program
URL Public URL for access to the directory
Tag Optional detailed field to allow grouping

Delete a Program from the Workspace

  • Select the Model to be deleted with the left-hand checkbox
  • Click the “Delete” icon at top right-hand side of screen
  • Confirm the deletion

Datasets

_images/Data_Scientist_Datasets.jpg

The Datasets contain the training data.

The details on the Model can be accessed by selecting the Model name.

_images/Data_Scientist_Datasets_Detail.jpg

Add a Dataset

_images/Data_Scientist_Datasets_New.png

In order to download a model to the Workspace, and use it within DKube:

  • Select the “+ Dataset” button at the top right-hand part of the screen
  • Fill in the information necessary to access the Dataset
  • Select the “Add Dataset” button

Fields for Adding a Dataset

The input fields for adding the Dataset are the same as they are for adding a Workspace ( Fields for Adding a Program )

Delete a Dataset

  • Select the Dataset to be deleted with the left-hand checkbox
  • Click the “Delete” icon at top right-hand side of screen
  • Confirm the deletion

Notebooks

DKube uses Jupyter Notebooks for experimentation. It includes the model and dataset information, as well as the hyperparameters for the model run.

_images/Data_Scientist_Notebooks.jpg

The status messages are described in section Status Field of Notebooks, Training Jobs, and Inference.

Special “DKube” Notebook

A special Notebook is created when DKube is installed, named “DKube” (item 1 in the screenshot above). It is used for tasks such as pipeline creation, inference, and migration.

  • The DKube Notebook is in the “stopped” state when DKube is first run. In order to use it, select the DKube Notebook instance and start it.
  • It is good practice to stop Notebook instances if they are not going to be used in order to conserve system resources.

TensorBoard

TensorBoard can be accessed from the Notebook screen.

  • When a Notebook instance is created, TensorBoard is in the “stopped” state. In order to use it, TensorBoard must be started by selecting the play icon (item 2 in the screenshot above).
  • It can take several minutes for TensorBoard to be active after being started.
  • It is good practice to stop TensorBoard instances if they are not going to be used in order to conserve system resources.

Notebook Actions

There are actions that can be performed on a Notebook instance (item 3 in the screenshot above).

  • A Jupyter notebook can be started
  • A Training Job can be submitted based on the current configuration and parameters in the Notebook.

In addition, there are actions that can be performed that are selected from the icons above the list of Notebook instances (item 4 in the screenshot). For these actions, the Notebook instance checkbox is selected, and the action is performed by choosing the appropriate icon.

  • A Notebook instance can be stopped or started
  • Notebook instances can be deleted
  • A new Notebook instance can be created by either cloning it (clone icon), or starting from a blank configuration (+ Notebook icon)

Notebook Details

More information can be obtained on the Notebook by selecting the name. This will open a detailed window.

_images/Data_Scientist_Notebook_Detail.jpg

Create Notebook

There are 2 methods to create a Notebook:

  • Create a new Notebook by selecting the “+ Notebook” button at the top right-hand side of the screen, and fill in the fields manually
    • This is typically done for the first Notebook, since there is nothing to clone
  • Clone a Notebook from an existing instance
    • This will open the same new Notebook dialog screen, but the fields are pre-loaded from the existing Notebook. This is convenient when a new Notebook will have only a few different fields, such as hyperparameters, from an existing Notebook.

In both cases, the “New Notebook” screen will appear. Once the fields have been entered or changed, select “Submit”

_images/Data_Scientist_Notebooks_New.jpg

Note

The first Job or Notebook will take extra time to start due to the image being pulled prior to initiating the task. The message might be “Starting” or “Waiting for GPUs”. Each time a new version of TensorFlow is run for the first time, the delay will occur. It will not happen after the first run.

New Notebook Fields

Notebook Name Unique user-chosen identification for Notebook
Tag Optional detailed identification, used for grouping
Container Select the Container & Framework to use
GPUs Number of GPUs to use
Hyperparameters Hyperparameters to use
Workspace Program to use
Dataset Training Dataset to use
Model Existing trained Model for transfer learning

Note

The Notebook Name should be 20 characters or less in length

Container

The Container fields allows the user to specify the docker container (standard DKube or custom), as well as the version of TensorFlow.

_images/Data_Scientist_Notebooks_Container.jpg

GPUs

_images/Data_Scientist_Notebooks_GPUs.jpg

The number of GPUs to use in the Notebook. The GPUs are shared with all Users in a Group. If there are currently not enough GPUs to satisfy the request, the Notebook will be queued until enough GPUs are freed up.

The “Select GPUs” field shows how many GPUs are currently available in the group, as well as the maximum number of GPUs in the group. This provides insight into the selection.

  • The screen shows how many GPUs available at that moment based on other Notebooks, Jobs, & Inferences that are currently using them. Since the usage of the GPUs is dynamic, and since the GPUs are being shared by other Users in the same Group, the actual number of GPUs available when the Notebook is submitted may be different.
  • Just below the GPU selection is a checkbox that allows the Notebook to start with the number of GPUs that are available upon submission. This will ensure that the Notebook does not queue.

Hyperparameters

Input the hyperparameters for the notebook run. The use of the hyperparameters is based on the program.

_images/Data_Scientist_Notebooks_Hyperparameters.jpg

There are some preset hyperparameters that are defaulted in DKube. Additional parameters can be added by selecting the “+ Add another”. This will allow an additional “Key” and “Value”. More parameters can be added by repeated use of this option.

_images/Data_Scientist_Notebooks_Hyperparameters_Add.jpg

If there are many parameters to be added, it is possible to import a JSON file with the parameters and values set by the file.

_images/Data_Scientist_Notebooks_Hyperparameters_JSON.jpg

The format of the JSON file is as follows:

 {
 "key1" : "value1"
 "key2" : "value2"
}
  • If the parameter is not currently part of DKube, it will add both the key and value
  • If the parameter already exists, the value from the file will replace the current value

Workspace

Select the model to use from the ones that have been loaded into DKube.

_images/Data_Scientist_Notebooks_Workspace.jpg

Dataset

Select the Dataset to use for training.

_images/Data_Scientist_Notebooks_Dataset.jpg

Model

Select the previously trained model from DKube.

_images/Data_Scientist_Notebooks_Model.jpg

Delete Notebook

  • Select Notebook name from the left-hand checkbox
  • Click “Delete” icon at top right-hand side of screen
  • Confirm the deletion

Jupyter Notebook

_images/Data_Scientist_Notebooks_Jupyter.jpg

The following steps will open a Jupyter window:

  • Select the Jupyter icon on the right-hand side of the Notebook instance to be edited
  • A popup will appear with the Jupyter interface

Note

The Jupyter notebook may sometimes take several minutes to load.

See also

User data can be migrated between platforms from within the Jupyter notebook, as described in section Migrating User Data Between Platforms

Jobs

The Job workflow manages training and preprocessing on a workspace program based on the dataset selected. The primary difference between the functions are:

  • Once the Training Job is complete, it creates a trained Model
  • When a Preprocessing Job is complete, it creates a Dataset ( Preprocessing Jobs)

Training Jobs

_images/Data_Scientist_Jobs.jpg

The status messages are described in section Status Field of Notebooks, Training Jobs, and Inference .

The Job workflow screen is used to:

  • Initiate Training Jobs
  • View and manage Jobs that have been initiated

There are several types of Training Jobs shown in this screen:

  • Training Jobs that have been initiated from the Jobs screen or from the Notebooks screen. Training Jobs can be run individually, or be initiated as part of the Hyperparameter Optimization feature described below.
  • Jobs that have been initiated from the Pipeline Experiments screen

TensorBoard

TensorBoard can be accessed from the Jobs screen.

  • When a Training Job instance is created, TensorBoard is in the “stopped” state. In order to use it, TensorBoard must be started by selecting the play icon (item 1 in the screenshot above).
  • It can take several minutes for TensorBoard to be active after being started.
  • It is good practice to stop TensorBoard instances if they are not going to be used in order to conserve system resources.

Training Job Actions

There are actions that can be performed on a Job instance (item 2 in the screenshot above). For these actions, the Job instance checkbox is selected, and the action is performed by choosing the appropriate icon.

  • A Job instance can be stopped or started
  • Job instances can be deleted
  • A new Job instance can be created by either cloning it (clone icon), or starting from a blank configuration (+ Training Job icon)

Hyperparameter Optimization

DKube supports Katib-based hyperparameter optimization. This enables automated tuning of hyperparameters for a program and dataset, based upon target objectives. An optimization study is initiated by uploading a configuration file during the Training Job submission as described in Hyperparameter Tuning .

The study initiates a set of trials, which run through the parameters in order to achieve the objectives, as provided in the configuration file. After all of the trial jobs have completed, DKube provides a graph of the trial results, and lists the best hyperparameter combinations for the required objectives.

The optimization study is ready for viewing when the status is shown as “complete”. That indicates that the trials associated with the study are all complete. The output results of the study can be viewed and downloaded by selecting the Katib icon at the far right hand side of the Job line.

As described in section Hyperparameter Tuning, an optimization job is initiated by providing a YAML configuration file in the Hyperparameter Optimization field when submitting a Job.

A study that has been initiated using Hyperparameter Optimization is identified by the Katib icon on the far right.

_images/Data_Scientist_Jobs_Katib.jpg

Selecting the icon opens up a window that shows the study configuration parameters and the study results. The study configuration summarizes the hyperparameters being tested and the objectives.

_images/Data_Scientist_Jobs_Hyperparameter_Config.jpg

The study results provides a graph of the trials, and lists the best trials based on the objectives.

_images/Data_Scientist_Jobs_Hyperparameter_Results.jpg

Training Job Details

More information can be obtained on the Job by selecting the name. This will open a detailed window.

  • A Hyperparameter Optimization Job is identified by the Katib icon on the far right of the screen

The Training Job details screen provides information about the Job such as status, resource utilization, and Job parameters. It is possible to save the current hyperparameters to a JSON file for later use from this screen.

  • Select “Inputs”
  • Within the “Hyperparameters” section, there is a “Custom” line
  • Putting the cursor on the right arrow will show the customer hyperparameters
  • This information can be cut and pasted into a file
_images/Data_Scientist_Jobs_Detail.jpg

Hyperparameter Optimization Job Details

A Hyperparameter Optimization Job can be identified in the Jobs screen by a Katib icon on the far right.

_images/Data_Scientist_Jobs_Katib.jpg

The detailed screen for Hyperparameter Optimization shows the trials associated with the study. Each trial can be selected to show the Job details such as status, resource utilization, and Job parameters.

_images/Data_Scientist_Jobs_Katib_Detail.jpg

Create Training Job

A Training Job can be created in the following ways:

  • Create a Job from a Notebook instance, using the “Training Job” icon on the right-hand side of the selected Notebook instance
    • This will pre-load the parameters from the Notebook instance
  • Create a new Training Job by selecting the “+ Training Job” button at the top right-hand side of the screen, and fill in the fields manually
  • Clone a Training Job from an existing instance
    • This will open the same new Training Job dialog screen, but most of the fields are pre-loaded from the existing Job. This is convenient when a new Job will have only a few different fields, such as hyperparameters, as the existing Job.
  • A Job is automatically created as part of the Pipelines capability

For the cases where a Job is created by the User, the “New Training Job” screen will appear. Once the fields have been filled in, select “Submit”

_images/Data_Scientist_Jobs_New.jpg

Note

The first Job will take additional time to start due to the image being pulled prior to initiating the task. The message might be “Starting” or “Waiting for GPUs”. Each time a new version of TensorFlow is run for the first time, the delay will occur. It will not happen after the first run.

New Job Fields

Training Name Unique user-chosen identification for Training Job
Tag Optional detailed identification, used for grouping
Container Select the Container & Framework to use
GPUs Number of GPUs to allocate to the job
Hyperparameters Program hyperparameters
Hyperparameter Tuning Optional file upload to initiate Hyperparameter Optimization
Workspace Program to use
Dataset Training Dataset to use
Model Previously trained Model for transfer learning

The Job fields are similar to the Create Notebook fields (See section New Notebook Fields). The following field definitions are the same as Notebook:

  • Training Name
  • Tag
  • GPUs
  • Hyperparameters
  • Workspace
  • Dataset
  • Model

The Jobs fields are different in the following ways:

  • There are additional options in the Container field
  • There is an additional field, Hyperparameter Tuning

Both of these are detailed below.

Training Job Container

_images/Data_Scientist_Jobs_Container.jpg

The Container for a Job has more options than Notebook.

  • A start-up script name must be provided. The script should reside in the Workspace directory. For the examples, the name of the script is “model.py”, and this needs to be entered into the field. If a different script name is used, it should be specified here.
  • Jobs can be run with GPUs distributed across the cluster. In order to enable this, the “Distributed workloads” option needs to be selected.
_images/Data_Scientist_Jobs_Container_Advanced.jpg

When distributing the workload across a number of nodes, the number of workers needs to be specified. DKube takes the number of GPUs specified in the GPU field, and requests that number of GPUs for each worker.

So, for example, if the number of GPUs is 4, and the number of workers is 1, then 8 GPUs will be requested, spread across 2 nodes.

Hyperparameter Tuning

In order to specify that the Job should be managed as a Hyperparameter Optimization study, a yaml file must be uploaded that includes the configuration for the experiment. The YAML files for the examples that are included with DKube are provided in section.

The YAML files for the examples that are included with DKube are available at the following links.

mnist http://oneconvergence.com/guide/downloads/mnist-tuning.yaml
catsdogs http://oneconvergence.com/guide/downloads/catsdogs-tuning.yaml

The YAML file should be downloaded to a file, and used when submitting the Job. The YAML file can be edited to have different configurations.

Leaving this field blank (no file uploaded) will indicate to DKube that this is a standard (non-hyperparameter optimization) Job.

The format of the configuration file is explained at Katib Introduction.

_images/Data_Scientist_Jobs_Hyperparameter_Tuning.jpg

Abort Job

  • Select the Job to be aborted with the left-hand checkbox
  • Click the “Abort” icon at the top right-hand side of the screen

Delete Job

  • Select the Job to be deleted with the left-hand checkbox
  • Click the “Delete” icon at top right-hand side of screen
  • Confirm the deletion

Preprocessing Jobs

A Preprocessing Job outputs a Dataset entry when it is complete. This is typically done in order to modify a raw dataset such that it can be used in Training.

_images/Data_Scientist_Preprocessing.jpg

An example preprocessing job can be used to go through the steps. It is available at https://github.com/oneconvergence/dkube-examples/tree/master/tensorflow/object-detection/pets

Create Preprocessing Job

A Preprocessing Job can be created in the following ways:

  • Create a new Preprocessing Job by selecting the “+ Preprocessing” button at the top right-hand side of the screen, and fill in the fields manually
  • A Job is automatically created as part of the Pipelines capability

For the cases where a Preprocessing Job is created by the User, the “New Preprocessing Job” screen will appear. Once the fields have been filled in, select “Submit”

_images/Data_Scientist_Preprocessing_New.jpg

Note

The first Job will take additional time to start due to the image being pulled prior to initiating the task. Each time a new version of TensorFlow is run for the first time, the delay will occur. It will not happen after the first run.

New Preprocessing Job Fields

Preprocessing Name Unique user-chosen identification for the job
Target Dataset Name Unique user-chosen name for the resulting dataset
Tag Optional detailed identification, used for grouping
Container Select the Container to use
Hyperparameters Program hyperparameters
Workspace Program to use
Dataset Training Dataset to use
Model Previously trained Model for transfer learning

The fields are similar to the Create Notebook fields (See section New Notebook Fields). The following field definitions are the same as Notebook:

  • Tag
  • Hyperparameters
  • Workspace
  • Dataset
  • Model

The Preprocessing fields are different in the following ways:

  • There are 2 name fields instead of one.
  • The**Container** field has different options

Preprocessing Container

_images/Data_Scientist_Preprocessing_Container.jpg
  • The Container for a preprocessing job requires a custom Docker image.
    • If the Docker image is private, the “Private” option should be selected, which will prompt for a username and password
  • A start-up script name must be provided.

Models

_images/Data_Scientist_Models.jpg

When a Job has completed, it will create a trained Model which will appear in the “Models” section of the DKube UI. The Models section shows the Models from the current User, as well as Models that have been trained by other Users in the same Group.

Add a Model

Models can also be imported into DKube for transfer learning.

  • Select the “+ Model” button at the top right-hand part of the screen
  • Fill in the information necessary to access the Dataset
  • Select the “Add Model” button

Fields for Adding a Model

The input fields for adding the Model are the same as they are for adding a Workspace (Fields for Adding a Program).

Model Details

_images/Data_Scientist_Models_Detail.png

Selecting the Model will call up a screen which provides more details on the Model. From this screen, the trained Model can be deployed or exported.

Deploy Model

_images/Data_Scientist_Models_Deploy.png

A Model can be deployed for inference testing. Once deployed, it will appear in the “Inferences” section of the DKube UI. In order to deploy the model:

  • Select the “Deploy” button at the top right-hand side of the screen
  • Fill in the Inference name

Export Model

A Model can be exported for use on another system. In order to export the model:

  • Select the Export button at the top right-hand side of the screen
  • Fill in the location of the file to create

Delete Model

  • Select the Model to be deleted with the left-hand checkbox
  • Click the “Delete” icon at top right-hand side of screen
  • Confirm the deletion

Inferences

_images/Data_Scientist_Inferences.jpg

The status messages are described in section Status Field of Notebooks, Training Jobs, and Inference .

After a trained Model has been deployed, it appears in the Inferences section. The Inferences section shows the deployed Models from the current User as well as other Users in the same Group.

The “Serving URL” can be used to test the Model with an external test program.

Testing the Inference

One Convergence has provided a UI-based application to test the inference for the examples included as part of the DKube installation (see section Example Model and Dataset Locations). It is accessed from a special notebook that gets initiated automatically after DKube is installed, called “DKube”.

Note

The DKube Notebook may not be running. If it is in the “aborted” state, select the DKube Notebook instance and start it.

Note

The inference application supplied here will not work for any programs that are outside of the examples included as part of DKube.

_images/Data_Scientist_Notebooks_Dkube.jpg

Opening a Jupyter notebook from this DKube entity exposes a file in the /tools/ folder called “dkube.html”

_images/Data_Scientist_Notebooks_Jupyter_Tools_DKube.jpg

Selecting this file opens a list of applications. Select the DKube inference app.

_images/Data_Scientist_Notebooks_Jupyter_Tools_Infapp.jpg

This will call up a screen that allows inference testing.

_images/Data_Scientist_Notebooks_Jupyter_Tools_Infapp_Screen.jpg

There are two fields that need to be filled in.

Model Serving URL url from the Inferences screen that identifies the location of the served model
Program Dropdown menu to select the model type
Testing Image Image that should be used to test the inference

An example of a model serving url from the Inferences page is shown below.

_images/Data_Scientist_Inferences_Selected.jpg

Copy and paste the url into the Model Serving url field.

Once the image has been selected, select the “Predict” button, and the model will make its prediction.

Kubeflow Pipelines

DKube supports Kubeflow Pipelines. Pipelines describe a workflow structure graphically, identifying the flow from one step to the next, and define the inputs and outputs between the steps.

An introduction to Kubeflow Pipelines can be found at Kubeflow Pipelines.

The steps of the pipeline use the underlying components of DKube in order to perform the required actions.

The following sections describe what is necessary to create and execute a Pipeline within DKube.

Create Workspace & Dataset

Prior to creating the pipeline, a Workspace and Dataset must be created that exactly matches the names expected in the example pipelines provided. An explanation of how to add a Workspace is in Section Add a Program to the Workspace, and how to add a Dataset is in Section Add a Dataset.

Create Pipeline

The Pipeline definition is managed from a Jupyter Notebook. There is a special Notebook created within DKube after installation, with the name “DKube”. This Notebook is used to create the pipeline and start an experiment.

_images/Data_Scientist_Notebooks_Dkube.jpg

Note

The DKube Notebook may not be running. If it is in the “aborted” state, select the DKube Notebook instance and start it.


Selecting the Jupyter icon will open a window that includes a number of folders. Select the folder “Pipeline”.

_images/Data_Scientist_Notebooks_DKube_Jupyter_Pipeline.jpg

Within the “Pipeline” directory, select the “ipynbs” folder, which contains the pipeline descriptions.

_images/Data_Scientist_Notebooks_DKube_Pipeline_IPYNB.jpg

Within that folder, there are a number of IPYNB files, each one describing one of the DKube examples provided.

dkube-mnist-pipeline.ipynb Classify images of digits
dkube-resnet-pipeline.ipynb Classify images of cats and dogs

Select one of the examples. This will open a window within Jupyter that can be used to compile the description, create a pipeline, and start an experiment.

_images/Data_Scientist_Notebooks_DKube_Pipeline_Compile.jpg

Run all of the cells within Jupyter. This will create the pipeline and run an experiment.

_images/Data_Scientist_Notebooks_DKube_Pipeline_RunAll.jpg

Managing Experiments and Runs

Experiments that have been created from pipelines are managed from the Experiments screen.

_images/Pipeline_Experiments_Started.jpg

The experiment that was initiated from the previous step will show up in this section. The runs within the experiment can be viewed and managed by selecting the experiment name. Within that, the user can see the pipeline, view the progress and status, and view the artifacts, input/output parameters, and logs.

The status of the Pipeline is shown on this screen. It also provides the ability to view the progress and detailed information.

_images/Pipeline_Experiments_Running.jpg

Selecting the name of the experiment brings up the current state of the Pipeline. Selecting a Pipeline box within the “current state” screen brings up window that provides more details on the configuration and status of the Run.

_images/Pipeline_Experiments_Running_Steps.jpg

Selecting “view pipeline” brings up a screen that shows the entire Pipeline. Selecting the Step box in that window brings up the details of the Run parameters.

_images/Pipeline_Experiments_Full_Pipeline.jpg

Within the full Pipeline view, the source code can also be inspected, by selecting the “Source” tab.

_images/Pipeline_Experiments_Source.jpg

Cloning a Run

Once an experiment has been created, different Runs can be initiated with varying parameters through the “Clone run” function. After the Runs have completed they will show up on the Experiments screen.

_images/Pipeline_Experiments_Clone_Run.jpg

Deleting Experiments and Runs

An Experiment can be deleted from the Experiments screen by selecting one or more of them and choosing the “Delete Experiments” function. Both Experiments and Runs can be deleted from the Runs screen. The top of the screen will delete the Experiment and all of the Runs. If only the Runs need to be deleted, they can be selected and deleted in the Run section of the screen.

_images/Pipeline_Experiments_Runs_Delete.jpg

Important

Experiments and Runs are not part of the standard Group mechanism that separate and protect Notebooks, Jobs, Models, and Inferences. Experiments and Runs are shared by all Users, regardless of what Group they are in. Because of that, it is possible for one User to delete an Experiment or Run from another User. Care must be taken when using the delete function.

Creating Experiments and Runs from a Pipeline

Experiment and Runs can be created directly from previously-added Pipelines. The Pipelines that are part of DKube are listed in the Pipelines screen. This screen lists the Pipeline examples that come with DKube, and several basic Pipelines that are part of the Kubeflow Pipelines release.

Migrating User Data Between Platforms

User data can be easily migrated between platforms through the dkubectl command. This includes:

  • Notebooks
  • Jobs
  • Workspaces
  • Datasets
  • Models
  • Inferences

After the migration, the target platform will look like the source platform, and the work can continue.

The migration within DKube is initiated from Jupyter. This should be started from the special DKube Notebook that is created automatically by the installation process.

_images/Data_Scientist_Notebooks_Dkube.jpg

Note

The DKube Notebook may not be running. If it is in the “aborted” state, select the DKube Notebook instance and start it.

Migration ini File

The dkubectl command uses an ini file to provide the context for the migration. There are source and target sections for credentials, and a section on what notebooks, jobs, and inferences should be migrated. Models, workspaces, and datasets for the selected items will also be migrated.

The ini file is accessible from the /tools/dkubectl/ folder.

_images/Notebook_Jupyter_dkubectl.jpg
_images/Notebook_Jupyter_ini_Files_Migration.jpg
_images/Notebook_Jupyter_Migration_ini.jpg

The following fields must be completed:

Migration User-chosen name to give the migration job
jwt DKube OAuth token from the Operator or Data Scientist right-hand menu (DKube OAuth Token)
dkubeurl url of the master node for the DKube installation. This is the same url that is available in the browser window used to access DKube.
jobs Names of the Notebooks, Jobs, and Inferences to migrate

Note

The fields must have quotes surrounding the data

After filling in the fields, the ini file must be saved.

_images/Notebook_Jupyter_Migration_Save.png

Important

If the file is not saved before moving ahead, the information will need to be entered again

Running the Migration Command

The command is initiated from within the Jupyter notebook.

_images/Notebook_Jupyter_New.jpg

Important

The Python command must be invoked from the home folder, as shown in the figure above. If it is started from another folder, the configuration file will not be found using the command below.

_images/Notebook_Jupyter_New_Python.jpg

The format for the command is:

!dkubectl migration start --name=<migration name> --config=./tools/dkubectl/migration.ini
migration name User-defined name for the migration
migration ini file Name of ini file for migration - normally migration.ini

_images/Notebook_Jupyter_Migration_Run.jpg

The migration will initiate, and provide status.

Once the jobs have been migrated to the target, they will be visible through the DKube url on the target platform.