How to run your python code in KnowledgeX

KnowledgeX connects data owners and data science specialists in a new gig-based platform. We foster trust by letting data owners retain control over their data. The platform allows the owner of the data to execute code without ever sharing sensitive data with the freelancer who wrote it.
Learn more about KnowledgeX.

In this article, we will go over:

1. How does the execution of a python model work in KnowledgeX?
2. How to transform your python code so the worker can run it?
3. How to deploy your program to DockerHub?
4. How to verify your program to see if the worker is able to run it?

TL;DR: Jump to section 2

How does the execution of a python model work in KnowledgeX?

In KnowledgeX, we use iExec — A blockchain-based decentralized cloud computing platform behind the scenes. The main advantage of iExec is that it allows transparent orchestration of jobs using blockchain, and most importantly, it facilitates TEEs — Trusted Execution Environment.

What is TEE? — Have you ever wondered why FaceID in Apple devices is so secure, and even though someone injected a malicious app, the app still can’t take over your phone and clean up your credit cards connected to ApplePay?

The Secure Enclave is a dedicated secure subsystem integrated into Apple:

“The Secure Enclave is isolated from the main processor to provide an extra layer of security and is designed to keep sensitive user data secure even when the Application Processor kernel becomes compromised.”

The enclave in TEE is a separate part of the memory and is not shared with other processes in the machine. iExec workers facilitate enclaves using Intel® SGX — hardware-backed enclaves which execute your program on the data in the most secure way. In simple words — nobody, including the worker and other processes in the machine, can “see” what’s inside an enclave — cannot access your sensitive data. (if you want to know more about TEEs, read this in-depth article)

In KnowledgeX, we offer execution in three ways:

• Cloud — the execution is performed on our worker.
• On-Premises — KnowledgeX will help you deploy your worker and add it to our worker pool. The data never leaves your company.
• In TEE — Execution in the enclave is the most secure way that we recommend for highly confidential data. This execution will be triggered on iExec owned worker with hardware-based Intel® SGX machines

What happens to your code and data?

1. As soon as you start execution, KnowledgeX creates a set of requests in smart contracts.
2. Deploys your code
3. Pick the worker for the execution.
4. Starts the execution
5. Generated records in the blockchain allow full auditability. Since blockchain is immutable, you can trace your data, code, execution and be sure that no one can alter this data.

Sequence diagram of Execution Job

If you want to know more about executions in KnowledgeX see this publication.

How to transform your python code so the worker will be able to run it?

It’s simple! Visit our python template repository and give us a star:
https://github.com/JadenX-GmbH/k-python-template

Clone this repository.

In most cases, python code has some input dataset and creates the output. In our template go to template_app.py and paste your code to def your_python_code(arg, input_path, output) .

• arg — an optional input parameter e.g. decryption key or some parameter of the model. For now, can be omitted.
• input_path — the path to the input file. Depends if the input file is text, video, a dataset of some recordings you will need to read it
• output_path — path to the output file

The code will be executed in the main function. The path to the input file, path to the output folder, and creation of the necessary file computed.json will be handled for you.

Ok, but what are these strange IEXEC_… variables?

When creating an execution job, input files and arguments are automatically assigned to those variables. Don’t worry, it will work, and you can test it!

To check if your code works, run the following snippet:

If you want to learn more about docker see this article

How to deploy your code to DockerHub?

A KnowledgeX worker will run your code using docker. First, it will pull the image, download the correct input files and start the execution. Your task is to deploy the code to the Docker Hub so the worker can pull it.

Fortunately, we have you covered! Just follow the instructions.

You really don’t need to be a docker expert to deploy your image!

Before we can start, you need to:

1. Install docker on your machine — visit this link and follow the steps
2. Create a free account at Docker Hub

Now we are ready to go.

First, we will build the docker image. No worries, the Dockerfile is already in the repository. Just run:

docker build . --tag k-python-template

Now similarly to the previous case, we need to run the app in the docker container.

As soon as you verified that the app is running correctly we can publish the image in the Docker Hub.

$ docker login
$ docker tag k-python-template <dockerhub-user>/k-python-template:1.0.0
$ docker push <dockerhub-user>/k-python-template:1.0.0

How to verify your program to see if the worker is able to run it?

Finally, we can test if the worker is able to run the docker image. If it created an output file correctly you are ready to start execution in KnowledgeX.