Modules Boilerplates
Guide for using Python Modules Boilerplates.
In order to simplify a process of creating weeve Modules, we introduce Module Boilerplates as a standardized approach. These Boilerplates would allow developers to create a module without a need for understanding and knowing weeve ecosystem and backend configurations. Since we recognize three types of modules: Input, Processing and Output, there are three types of Boilerplates. Although our Boilerplates currently support only Python programming language, we plan to introduce other languages in the future.
Boilerplates are designed in such a way that they handle synchronization with weeve intercontainer communication specification for developers. Moreover, it simplifies the process of publishing the module to module registry like DockerHub. In the end, a developer would edit only one file in a Boilerplate to perform the module logic, as all other requirements for a module to work in a weeve ecosystem would be already implemented by a Boilerplate.
Python Input Module Boilerplate serves as a starting point for developers to build input modules for weeve Platform and Edge Applications. Input modules are responsible for inputting data to the data service. For instance, they can subscribe to some MQTT topic and once they receive data they would pass them to the next module that processes them. Or, they can be mounted with a sensor attached to a host device and gather data read by the sensor. Module developer should work only in
src/module/module.py file and we recommend getting familiar with Boilerplate's GitHub repo README.File tree for Python Input Module Boilerplate is as follows:
├── src
│ ├── api
│ │ ├── __init__.py
│ │ ├── log.py # log configurations
│ │ └── send_data.py # sends data to the next module
│ ├── module
│ │ └── module.py # [*] main logic for the module
│ └── main.py # module entrypoint
├── docker
│ ├── .dockerignore
│ ├── docker-compose.yml
│ ├── docker-entrypoint.sh
│ └── Dockerfile
├── example.env # sample environment variables for the module
├── Module.yaml # used by weeve platform to generate resource in Data Service Designer section
├── makefile
├── README.md
├── example.README.md # README template for writing module documentation
└── requirements.txt # module dependencies, used for building Docker image
However, the most important resources are:
name | description |
|---|---|
src | All source code related to the module (API and module code). |
src/main.py | Entry-point for the module. |
src/api | Code responsible for setting module's API and communication with weeve ecosystem. |
src/module | Code related to the module's business logic. This is working directory for module developers. |
docker | All resources related to Docker (Dockerfile, docker-entrypoint.sh, docker-compose.yml). |
example.env | Holds examples of environment variables for running the module. |
requirements.txt | A list of module dependencies. |
Module.yaml | Module's YAML file that is later used by weeve platform Edge Application Designer. |
A module developer should work mostly in
src/module/module.py and avoid editing other files, so that the module remains in synchronization with weeve system. src/module/module.py contains a module_main() function which is responsible for module's logic. A developer should implement all necessary functionality within this function. The received or read data should be transformed into a JSON object assigned to variable input_data that will be later sent to the next module in weeve edge application as output data. Although a module developer decides about the format of output data, any other changes to ReST API POST call should be avoided.If any specific Python packages are required by the module, they can be added to the
requirements.txt and docker/Dockerfile should be also changed accordingly if any port bindings or mounts are required.The last important element of a module is its YAML file that is later used by weeve platform Edge Application Designer. Navigate to Tutorial - Module YAML to learn more about creating
Module.yml file.Python Processing Module Boilerplate serves as a starting point for developers to build Processing modules for weeve Platform and Edge Applications. Processing modules are responsible for analyzing and processing data passing through the data service. For instance, they can filter the values based on some condition and compare value, then pass the output to the next module. Or, they can implement some machine learning model that would do prediction on received data. Module developer should work only in
src/module directory (module.py and validator.py) and we recommend getting familiar with Boilerplate's GitHub repo README.File tree for Python Processing Module Boilerplate is as follows:
├── src
│ ├── api
│ │ ├── __init__.py
│ │ ├── log.py # log configurations
│ │ ├── processing_thread.py # a separate thread responsible for triggering data processing and sending to the next module
│ │ ├── send_data.py # sends data to the next module
│ │ └── request_handler.py # handles module's API and receives data from a previous module
│ ├── module
│ │ ├── module.py # [*] main logic for the module
│ │ └── validator.py # [*] validation logic for incoming data
│ └── main.py # module entrypoint
├── docker
│ ├── .dockerignore
│ ├── docker-compose.yml
│ ├── docker-entrypoint.sh
│ └── Dockerfile
├── test
│ ├── assets
│ │ ├── input.json # input data for tests (sample module input)
│ │ └── expected_output.json # expected output data for tests (sample module output)
│ ├── boilerplate_test.py # script handling module testing
│ ├── docker-compose.test.yml
│ ├── Dockerfile.listener # dockerfile for a container used to simulate egress endpoint
│ ├── listener.py # script implementing egress endpoint
│ └── test.env # environment variables for tests
├── example.env # sample environment variables for the module
├── Module.yaml # used by weeve platform to generate resource in Data Service Designer section
├── makefile
├── README.md
├── example.README.md # README template for writing module documentation
├── requirements_dev.txt # module dependencies for testing, used for building Docker image
└── requirements.txt # module dependencies, used for building Docker image
However, the most important resources are:
name | description |
|---|---|
src | All source code related to the module (API and module code). |
src/main.py | Entry-point for the module. |
src/api | Code responsible for setting module's API and communication with weeve ecosystem. |
src/module | Code related to the module's business logic. This is working directory for module developers. |
docker | All resources related to Docker (Dockerfile, docker-entrypoint.sh, docker-compose.yml). |
test | All resources related to automating testing of the module in development process. |
example.env | Holds examples of environment variables for running the module. |
requirements.txt | A list of module dependencies. |
Module.yaml | Module's YAML file that is later used by weeve platform Edge Application Designer. |
A module developer should work mostly in
src/module/module.py and avoid editing other files, so that the module remains in synchronization with weeve system. src/module/module.py contains a module_main() function which is responsible for module's logic. After data are received by a module, they are validated (more details on validation process can be found in src/module/validator.py and more requirements can be added by a module developer accordingly) and passed to module_main() function. A developer should implement all necessary functionality within this function. The function returns [data, str]. Returned data should be in a form that can be later send with ReST API POST request to the next module, i.e. a single JSON object or an array of JSONs. Data returned by module_main() will be the module's output sent to the next module with send_data() function in src/api/send_data in an exact form as they were returned by module's main logic. String returned by module_main() is an error message that is logged if any error occurs during data processing.If any specific Python packages are required by the module, they can be added to the
requirements.txt and docker/Dockerfile should be also changed accordingly if any port bindings or mounts are required.The last important element of a module is its YAML file that is later used by weeve platform Edge Application Designer. Navigate to Tutorial - Module YAML to learn more about creating
Module.yml file.Processing Module Boilerplate supports automated testing of the module. Our automated testing pipeline builds a Docker container of the module and provides two additional Docker containers that simulate a preceding and succeeding modules, and the edge application. Then, input data provided in
test/assets/input.json are passed to the tested module and its output is compared with expected output data provided in test/assets/expected_output.json. Hence, to use our automated testing pipeline a module developer needs to provide sample data in abovementioned .json files. Please navigate to the boilerplate's README Module Testing section for more detail on running the tests.Python Output Module Boilerplate serves as a starting point for developers to build Output modules for weeve Platform and Edge Applications. Output modules are the last instance of weeve Edge Applications and they are responsible for outputting data to the final endpoint. For instance, they can output data by calling some database API and writing to it. Or, they can send notifications to Slack or WhatsApp when triggered by a specific value. Or, they can save data to some file on a device. Module developer should work only in
src/module/module.py file and we recommend getting familiar with Boilerplate's GitLab repo README.File tree for Python Output Module Boilerplate is as follows:
├── src
│ ├── api
│ │ ├── __init__.py
│ │ ├── log.py # log configurations
│ │ ├── processing_thread.py # a separate thread responsible for triggering data outputting
│ │ └── request_handler.py # handles module's API and receives data from a previous module
│ ├── module
│ │ ├── module.py # [*] main logic for the module
│ │ └── validator.py # [*] validation logic for incoming data
│ └── main.py # module entrypoint
├── docker
│ ├── .dockerignore
│ ├── docker-compose.yml
│ ├── docker-entrypoint.sh
│ └── Dockerfile
├── example.env # sample environment variables for the module
├── Module.yaml # used by weeve platform to generate resource in Data Service Designer section
├── makefile
├── README.md
├── example.README.md # README template for writing module documentation
└── requirements.txt # module dependencies, used for building Docker image
However, the most important resources are:
name | description |
|---|---|
src | All source code related to the module (API and module code). |
src/main.py | Entry-point for the module. |
src/api | Code responsible for setting module's API and communication with weeve ecosystem. |
src/module | Code related to the module's business logic. This is working directory for module developers. |
docker | All resources related to Docker (Dockerfile, docker-entrypoint.sh, docker-compose.yml). |
example.env | Holds examples of environment variables for running the module. |
requirements.txt | A list of module dependencies. |
Module.yaml | Module's YAML file that is later used by weeve platform Edge Application Designer. |
A module developer should work mostly in
src/module/module.py and avoid editing other files, so that the module remains in synchronization with weeve system. src/module/module.py contains a module_main() function which is responsible for module's logic. After data are received by a module, they are validated (more details on validation process can be found in src/module/validator.py and more requirements can be added by a module developer accordingly) and passed to module_main() function. A developer should implement all necessary functionality within this function. In a contrast to Input Boilerplate and Processing Boilerplate, this Boilerplate does not have src/api/send_data.py and its send_data() function, because it is developer's responsibility to implement functionality outputting data to the final endpoint. Hence, it is a module developer responsible for creating a module that correctly interacts with a chosen endpoint API (including correct JSON body format, API URL address, headers, etc).If any specific Python packages are required by the module, they can be added to the
requirements.txt and docker/Dockerfile should be also changed accordingly if any port bindings or mounts are required.The last important element of a module is its YAML file that is later used by weeve platform Edge Application Designer. Navigate to Tutorial - Module YAML to learn more about creating
Module.yml file.Weeve Python Module Boilerplates are supposed to simplify and speed up the creation of modules. We recognize that building a module and keeping it in synchronization with weeve system can be overwhelming for a developer. Therefore, we aimed to make our Boilerplates helpful and useful to the developers. Boilerplates make module creation so simple that it often takes a module developer up to 7 lines of code to implement a module!
We recommend reading Boilerplates' README files in GitHub along with this tutorial document.
Last modified 7mo ago