Example: Hello Weeve!

A Step-by-step guide to module creation
In this section we are going to create 'Hello weeve!' module which aims to convert the temperature of a data stream from Celsius to Fahrenheit. Such module is a ReST endpoint that returns values and it could be qualified as a Processing module. The steps that we are going to take include: creating module code, testing, containerizing and deploying it to a container registry.
We are going to use the following technologies: REST API, Containers (Docker), Postman (for testing), Python, and Bottle.
Plan of action:
1.downloading our module Boilerplate
2.creating a module with Python (requires Python3)
3.testing a module with Postman
4.testing a module with Boilerplate's automated tools
5.containerizing a module with Docker
6.creating module’s YML file
7.deploying a module to Git repository
8.deploying a module to DockerHub
9.requesting publishing to weeve
1.  Downloading our module Boilerplate
Since we can classify this module's type as Processing, git clone our Python-Processing Boilerplate and get familiar with its documentation if you have not done it yet (Boilerplate Introduction and Python-Processing Boilerplate).
2.  Creating a module with Python
As you know from the documentation, we will mostly work in src/module directory of the boilerplate. Module boilerplate logic is as follows: receive data from the previous module -> validate data with validator.py -> pass data to module.py -> output data to the next module.
Let's start with validator.py where we will change the type of allowed data. Currently, boilerplate accepts incoming data as a type of dict (JSON single object) or list (array of JSON objects). In this tutorial, we want to work with single objects only. Hence, in validator.py change line allowed_data_types = [dict, list] to allowed_data = [dict].
Next, go to src/module/module.py where we will implement the module business logic in module_main(). As you can see, module_main() has implemented try-except approach that helps to handle errors without crashing the code. We are going to work within the scope of try. 
Firstly, we need to extract temperature data from the received data. Since Python dictionary, which represents a JSON object, supports key-value approach, we can get the temperature values by accessing the right field. However, our data might have different formats, so we need to make our module as flexible as possible. For instance, the data could look like one of the following samples:
{
    "temperature": 12,
    "unit": "Celsius",
    "device": "Termometer-1"
}
or
{
    "engineTemp": 87,
    "machine": "Compressor-A12D",
    "location": "Assembly-Block-5"
}
In the first case we would like to access a temperature value assigned to field "temperature", whereas in the second case we would be interested in field "engineTemp". Hence, our module should paramiterize a key holding temperature value in the incoming data. We will store this key name in INPUT_LABEL environment variable that will be set by the user in weeve Platform as explained later in the later sections. Therefore, we can access the temperature readings with the following line:
c_temp = received_data[os.getenv("TEMPERATURE_LABEL")]
Secondly, we need to implement a separate function celsiusToFahrenheit() that is going to convert our temperature units from Celsius to Fahrenheit. We can create this function within the same file:
def celsiusToFahrenheit(c_temp):
    """ Returns: converted temperature from Celsius to Fahrenheit """
    return (c_temp * (9 / 5)) + 32
Later, we call this function in the module main logic:
f_temp = celsiusToFahrenheit(c_temp)
Thirdly, our module_main() needs to return processed data, so that the boilerplate could send them to the next module. Thus, we need to build a JSON object that can be passed with ReST API POST request as an output of our module. Composing a completely new JSON object might be a bad idea, as we could lose all the other information we received with our data like device or location in the above samples. Consequently, we are going to just swap temperature values:
received_data[os.getenv("TEMPERATURE_LABEL")] = f_temp
After implementing the above instructions, your src/module/module.py should look similar to the following example:
"""
This file implements module's main logic.
Data processing should happen here.
Edit this file to implement your module.
"""
​
import os
from logging import getLogger
​
log = getLogger("module")
​
​
def module_main(received_data: any) -> [any, str]:
    """
    Process received data by implementing module's main logic.
    Function description should not be modified.
    Args:
        received_data (any): Data received by module and validated.
    Returns:
        any: Processed data that are ready to be sent to the next module or None if error occurs.
        str: Error message if error occurred, otherwise None.
    """
​
    log.debug("Processing ...")
​
    try:
        # extract temperature data
        c_temp = received_data[os.getenv("TEMPERATURE_LABEL")]
        
        # convert units from Celsius to Fahrenheit
        f_temp = celsiusToFahrenheit(c_temp)
        
        # swap converted data
        received_data[os.getenv("TEMPERATURE_LABEL")] = f_temp
​
        return received_data, None
​
    except Exception as e:
        return None, f"Exception in the module business logic: {e}"
​
def celsiusToFahrenheit(c_temp):
    """ Returns: converted temperature from Celsius to Fahrenheit """
    return (c_temp * (9 / 5)) + 32
Finally, we need to update some remaining details in the Boilerplate code.
docker/docker-compose.yml (more on docker-compose files here)
in line 3 change boilerplate to hello-weeve
in line 8 change MODULE_NAME from boilerplate to hello-weeve
after line 12 add another environment variable TEMPERATURE_LABEL and assign some sample value like temperature
3.  Testing a module with Postman
Once our module script is ready, it's time to test it locally. For this we are going to use Postman, which you can download here.
Let's start with preparing .env file to hold our module's configuration. Duplicate example.env file, change its name to .env and add TEMPERATURE_LABEL environment variable that we use in our module's main logic. Your .env should look like this:
# REQUIRED MODULE SETTINGS
​
MODULE_NAME=hello-weeve
MODULE_TYPE=Processing
​
# REQUIRED API SETTINGS
​
LOG_LEVEL=DEBUG
INGRESS_HOST=0.0.0.0
INGRESS_PORT=80
EGRESS_URLS=https://test-module.free.beeceptor.com
​
# YOU CAN ADD REMAINING MODULE SETTINGS BELOW
​
TEMPERATURE_LABEL=temperature
Notice that we changed MODULE_NAME to hello-weeve and we assigned EGRESS_URLS to https://test-module.free.beeceptor.com. We recommend using Beeceptor as an endpoint simulator to see module's output. After navigating to Beeceptor website, enter Endpoint Name as test-module and click Create Endpoint. Later copy Your Beeceptor Endpoint link and save it as EGRESS_URLS env. Our data converted from Celsius to Fahrenheit will be output to that endpoint.
We can test the module locally or with Docker. To test locally, in your terminal navigate to your module's root folder and run make install_dev to install the module requirements like requests or bottle. Then, run make run_app to run the module.
If the following error occurs, navigate to makefile and in run_app section, change python to python3. As this error depends on how your system has installed Python3:
python image/src/main.py
Traceback (most recent call last):
  File "image/src/main.py", line 3, in <module>
    from app import create_app
  File "/Users/kuba/weeve/modules/docs-hello-weeve/image/src/app/__init__.py", line 13
    def create_app() -> Flask:
                     ^
SyntaxError: invalid syntax
make: *** [run_local] Error 1
If make run_app is executed correctly, you should see a response similar to the following:
set -a && source .env && set +a && python src/main.py
{'level': 'INFO', 'time': '2022-07-31 11:59:17,049', 'filename': 'main', 'message': 'hello-weeve running on 0.0.0.0 at port 80 with end-point set to https://test-module.free.beeceptor.com'}
This shows that your module runs successfully, but now it is time to check its functionality. The last line of log info is of the most interest to us. It contains information on which host address and port our module is running. In this case it is host 0.0.0.0 and port 80 and it is a default value that can be changed in .env file by modifying INGRESS_HOST and INGRESS_PORT values. These values are modified by weeve Agent in production, so the values you set are used only for your development and local testing.
Now, we can open Postman and navigate go to File > New Tab (⌘T on Mac or CTRL+T on Windows). You should see a window as below:
Postman Welcome
Enter request URL as http://0.0.0.0:80/ and set ReST API method to POST. Then just below that choose Body > raw > JSON and create a sample data with temperature in Celsius:
{
    "timestamp": 1605531095680,
    "rpm": 8591,
    "temperature": 5,
    "power": 473,
    "factoryId": "factoryBerlin",
    "servoId": "servoA3389"
}
Your window should now look similar as below:
Postman POST to http://0.0.0.0:80/
If you followed this guide step by step, after pressing SEND button you should receive the following response:
Postman Response
Now, let's check if our data was converged correctly by navigating to our Beeceptor endpoint page. We should see:
Beeceptor Response
As you see, the temperature 5°C changed to 41°F.
To test the module with Docker, start by changing MODULE variable in makefile to your desired Docker image name. It should be composed of your organization/username in DockerHub and followed by module name. In our case it is weevetutorial/hello-weeve with weevetutorial being our username in DockerHub and hello-weeve as a module name. In your terminal navigate to your module's root folder and run make create_image to build the Docker image. Then, run make run_image to run the module. Follow above steps to test your module with Postman.
4.  Testing a module with Boilerplate's automated tools
To speed up development process, our Boilerplates enable creating automated test cases. We will be working in /test directory.
Start with adding environment variables to test.env:
# REQUIRED MODULE SETTINGS
​
MODULE_NAME=hello-weeve
MODULE_TYPE=Processing
​
# REQUIRED API SETTINGS
​
LOG_LEVEL=DEBUG
INGRESS_HOST=0.0.0.0
INGRESS_PORT=80
EGRESS_URLS=http://listener1:9000,http://listener2:9001
​
# YOU CAN ADD REMAINING MODULE SETTINGS BELOW
​
TEMPERATURE_LABEL=temperature
Then, in the test/assets/ directory you can see two files: expected_output.json and input.json. Files' names are selfexplainatory, so we will edit these files to provide test cases for our module. Remove expected_output.json content and add the following code:
{
    "timestamp": 1605531095680,
    "rpm": 8591,
    "temperature": 41.0,
    "power": 473,
    "factoryId": "factoryBerlin",
    "servoId": "servoA3389"
}
Similarly, input.json should be:
{
    "timestamp": 1605531095680,
    "rpm": 8591,
    "temperature": 5,
    "power": 473,
    "factoryId": "factoryBerlin",
    "servoId": "servoA3389"
}
In your terminal navigate to your module's root folder and run make run_test. The boilerplate will use Docker to build your module's image. Later, it will create two additional images that will be attached to your module to simulate weeve edge application. At the end, it will pass data from input.json and compare your module's output with expected_output.json data. You should see the similar results:
Module's test results
5.  Containerizing a module with Docker
In this section we are going to work with Docker, you might want to install it before continuing.
Our module intercontainer communication is already handled by the boilerplate. Hence, we can now test containerizing our module. Our boilerplates provides us with Dockerfile that is necessary for this task and its settings are sufficient for our case (but sometimes if more complex modules are implemented you might want to modify Dockerfile accordingly). Dockerfile contains installation steps and container settings. Each line in Dockerfile is a separate command that will be executed when building the image. You can inspect docker/Dockerfile to learn what we implemented there. Moreover, you can learn more about Dockerfiles on Docker official page.
Navigate to makefile and at the very top set MODULE variable to weevetutorial/hello-weeve as this is our module name. Next, run make create_image command in the terminal to build Docker image of our container. You can learn more about the used docker build command here.
In order to check if the image was build successfully run docker images command that should output the below results:
REPOSITORY                            TAG       IMAGE ID       CREATED         SIZE
weevetutorial/hello-weeve   latest    fb0f76b4438f   3 seconds ago   135MB
You can see that weevetutorial/hello-weeve image was created on your local machine. Its TAG is latest as it is a default tag used by Docker.
Now, let's run make run_image command to run the image and test a functionality of the module. You can learn more about the used docker run command here. As a result, the terminal should return the following response:
docker run -p 80:80 --rm --env-file=./.env weevetutorial/hello-weeve:latest
[ENTRYPOINT] Entrypoint script for the module.
[ENTRYPOINT] Environment validated.
{'level': 'INFO', 'time': '2022-07-31 12:21:15,393', 'filename': 'main', 'message': 'hello-weeve running on 0.0.0.0 at port 80 with end-point set to https://test-module.free.beeceptor.com'}
You can check if your Docker container works by using Postman again. If you check Beeceptor page, you should get the same results.
6.  YAML file
YAML (YML) files are used for data-serialization and module files configuration. With YML files we can help weeve data service maintainers to set the module requirements and pass important information to the module containers. You can find our full tutorial on YAML files here, so to speed up our tutorial go to Module.yaml file and delete the content. Then copy and paste the following lines:
displayName: 'Hello Weeve'
moduleName: 'hello-weeve'
description: 'Hello weeve module that converts Celsius to Fahrenheit'
versionName: 'v1.0.0'
isPublic: true
categories:
  - 'Process'
type: 'Processing'
image:
  name: 'weevetutorial/hello-weeve'
  tag: 'v1.0.0'
  homepage: 'https://hub.docker.com/weevetutorial/hello-weeve'
repository: 'https://github.com/weeve-modules/hello-weeve'
envs:
  - name: Temperature Label
    key: TEMPERATURE_LABEL
    description: Data field storing temperature value
    value: 'temperature'
    type: 'text'
    options: []
    dependencies: []
ports: []
mounts: []
tags:
  - 'Python'
  - 'Processing'
  - 'Boilerplate'
  - 'Celsius'
  - 'Fahrenheit'
  - 'converter'
icon: 'https://amazon.com/Python.png'
You should change image/homepage and repository URLs to the ones pointing to your repositories.
YML is an easy to read file and a meaning of specific variables can be found in the YAML documentation.
7.  Deploying the module to Git repository
If everything works fine, you can commit the module to your repository, i.e.: GitHub or GitLab.
8.  Deploying the module to container registry
Now it’s time to push our module image to Docker registry (DockerHub), so we could later access it in weeve edge application.
You can either use your own Docker registry or you can store it with weeve.
If choose the first option, you would need a Docker account. To be able to access your account from the command line, you need to log in with the docker login command.
Since you should have a Docker image named weevetutorial/hello-weeve:latest stored on your local machine, you can commit it to Docker registry with the line: $ docker tag weevetutorial/hello-weeve:latest <your-docker-registry-account>/hello-weeve:latest
To publish your image to the Docker registry, use the docker push command: $ docker push <your-docker-registry-account>/hello-weeve:latest
9.  Request publishing to weeve
The last step in the process of creating a weeve module is publishing it to weeve! We are still working on automating this process, so for now you will need to contact us with the Module.yaml file and a link to module's DockerHub repo.
Congratulations! You have just deployed your first containerized weeve module! You can access hello-weeve module in our GitHub repo.
Last modified 7mo ago