Integrations
Integrations
Breadcrumbs

SDK

Introduction


This document is a plugin development kit.

Project structure

Project object model

This is the pom.xml file containing information about the project and the configuration details used to build the project, such as the project version, description, dependencies, etc.

Example: tot-plugin-example/pom.xml

Ref. Doc.: Apache Maven


Configuration file

The way we configure is through a YAML configuration file.

Example: tot-plugin-example/src/main/resources/application.yaml

Ref. Doc.: Spring.io


Main class (mainClass)

The class with the main method needed to start the application. It is annotated with @SpringBootApplication and contains the Spring startup instruction: SpringApplication.run(...), it already includes the basic packages for the application to start and interact with tot.

Example: tot-plugin-example/src/main/java/com/anjana/tot/plugin/example/TotPluginExampleApplication.java


Configuration class

To load the related set of properties from the configuration file, a bean class is created with the @Configuration annotation. See section Tot-plugin-lib

Example: tot-plugin-example/src/main/java/com/anjana/tot/plugin/example/configuration/ExampleConfiguration.java


Controller

Once we have this, we proceed to create the Controller class annotated with @RestController to define the operations that will be carried out. See section Tot-plugin-lib

Example: tot-plugin-example/src/main/java/com/anjana/tot/plugin/example/controller/OperationsController.java


Operations that a plugin could carry out:


Structure management:

- Create dataset: The entry point for this functionality is in StructureManagementControllerInterface (See section Tot-plugin-lib). This endpoint is responsible for creating an object (whether a table, file, or anything else desired) using the metadata of a dataset in anjana. It receives the dataset metadata and its datasetfields. This is the action performed when a governed dataset is created or when one is changed from ungoverned to governed.


Permission management:

- Create group: To do this, the createGroup method is created whose endpoint is in IdentityManagementControllerInterfaceV2 (See section Tot-plugin-lib). The functionality of this endpoint is to create a group and assign permissions. To do this, a role is created in the database and that role is granted permission for some tables. The DTOs used by this endpoint are DSAOperationResultDTO, DSAFullInfoDTO (See section Tot-lib). This is the action the plugin performs when a DSA with governed datasets is created or versioned.


- Delete group: To do this, the deleteGroup method is created whose endpoint is in IdentityManagementControllerInterfaceV2 (See section Tot-plugin-lib). The functionality of this endpoint is to delete a group and unassign permissions. To do this, the role that has the permissions to be revoked is deleted. The DTOs used by this endpoint are DSAOperationResultDTO, DSAFullInfoDTO (See section Tot-lib). This is the action the plugin performs when a DSA with governed datasets expires.


- Remove object from a group: To do this, the removeObject method is created whose endpoint is in IdentityManagementControllerInterfaceV2 (See section Tot-plugin-lib). The functionality of this endpoint is to revoke the permissions a role has over some tables. The DTOs used by this endpoint are ObjectOperationResultDTO, ObjectOperationDTO (See section Tot-lib). This is the action the plugin performs when a governed dataset expires.


- Assign user: To do this, the addUser method is created whose endpoint is in IdentityManagementControllerInterfaceV2 (See section Tot-plugin-lib). The functionality of this endpoint is to modify users in a table's policies. The DTOs used by this endpoint are DSAOperationResultDTO, DSAFullInfoDTO (See section Tot-lib). This is the action the plugin performs when a user joins a DSA with governed datasets.


- Unassign user: To do this, the removeUser method is created whose endpoint is in IdentityManagementControllerInterfaceV2 (See section Tot-plugin-lib). The functionality of this endpoint is to modify users in a table's policies. The DTOs used by this endpoint are DSAOperationResultDTO, DSAFullInfoDTO (See section Tot-lib). This is the action the plugin performs when a user leaves a DSA with governed datasets.


Data sampling: To do this, the sample method is created whose endpoint is in SampleDataControllerInterface (See section Tot-plugin-lib). The functionality of this endpoint is to obtain a data sample. The DTOs used by this endpoint are SampleDataDTO, MetadataInfoDTO (See section Tot-lib). This is the action the plugin performs when a user enters the data sampling tab on a dataset with that flag set to true on a governed dataset.


Metadata extraction: In this section we distinguish between two methods: metadataList and metadataExtract.

- metadataList: Its endpoint is in ExtractMetadataControllerInterface (See section Tot-plugin-lib). Its function is to obtain a general list of objects extracted from a technology. The DTOs used by this endpoint are ResourceStructureDTO, ResourceDTO (See section Tot-lib).

- metadataExtract: Its endpoint is in ExtractMetadataControllerInterface (See section Tot-plugin-lib). Its function is the extraction of a specific object from a technology. The DTOs used by this endpoint are MetadataImportedDTO, ResourceMetadataRequestDTO (See section Tot-lib).


It is recommended to develop the metadata extraction section asynchronously as there may be some delay on the part of the technologies when returning a response.


Services

The service layer consists of classes annotated with @Service. This annotation is used to mark classes as service providers since they are the ones that provide functionalities. A specific Service class is implemented for each functionality.


Example:

Data sampling

tot-plugin-example/src/main/java/com/anjana/tot/plugin/example/service/SampleDataService.java

Metadata extraction tot-plugin-example/src/main/java/com/anjana/tot/plugin/example/service/ExtractMetadataService.java


General overview of the directory, file and class distribution:

- tot-plugin-example
    - src
        - main
            - java/com/anjana/tot/plugin/example
                  - configuration
                      ExampleConfiguration.java
                  - controller
                      OperationsController.java
                  - service
                      ExtractMetadataService.java
                      PermissionService.java
                      SampleDataService.java
                  TotPluginExampleApplication.java
            - resources
                application.yaml
        - test/resources
    pom.xml

Libraries

We have three libraries that provide implementation of some functionalities. These are: tot-lib, tot-plugin-lib and customers-lib.

Below, some functionalities of each one that may be of interest for plugin development are explained.


Tot-lib

One of the most important aspects of this library is the extensive repertoire of DTOs it offers for data mapping.

They are found in: tot-lib/src/main/java/com/anjana/tot/core/model

Other classes of interest: TotConstants.java , TotUtils.java in tot-lib/src/main/java/com/anjana/tot/core/util

Customers-lib

Like the tot-lib library, this library offers several DTOs that can be useful in customers-lib/src/main/java/com/anjana/core/customers/model as well as several classes containing enumerators to have typed values in customers-lib/src/main/java/com/anjana/core/customers/enums .

Tot-plugin-lib

One of the most important aspects of this library are the interfaces with the endpoints required for the plugin to function, which will be implemented by the plugin's operations controller; these are located in tot-plugin-lib/src/main/java/com/anjana/tot/plugin/core/interfaces .

Another class of interest is PluginConfiguration.java located in src/main/java/com/anjana/tot/plugin/core/configuration/PluginConfiguration.java .

This contains the most common properties of a plugin, which are the location (location), the server (server), the list of aris (aris) and the prefix (groupPrefix). The plugin's configuration class should extend this and additional properties to be retrieved from the configuration file can be added to it.

This library also contains other classes of interest, such as the TotPluginException.java class in tot-plugin-lib/src/main/java/com/anjana/tot/plugin/core/exception for capturing exceptions that the plugin may throw.

It contains basic utilities to read avro, parquet, excel and csv files (which are used by anjana's native plugins to extract information from those files and in data sampling). If you want to use them, you must include the package com.anjana.tot.plugin.core.container as another package to scan in the plugin's main class.


How to create a plugin

This skeleton only contains what is necessary (including example configuration) to start the project and have it register in tot.

To make a fully functional plugin, several steps are required.


Dependencies

The dependencies included in the example only serve to start a Spring Boot and have access to the utilities that the other plugins have. Those necessary for the integration you want to develop must be included. (Note: Changing the Spring version due to integration needs could affect the interaction with tot and with anjana).


Some dependencies that have not been included are the avro, parquet, excel and csv file processing dependencies used by all of anjana's native plugins. If you want to use them, the following dependencies must be included (in addition to including the com.anjana.tot.plugin.core.container package in the project's main class so that the file reading components are started).

XML
<!-- apache avro content -->
<dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro</artifactId>
  <version>1.10.1</version>
</dependency>

<!-- apache parquet content -->
<dependency>
  <groupId>org.apache.parquet</groupId>
  <artifactId>parquet-common</artifactId>
  <version>1.11.1</version>
</dependency>
<dependency>
  <groupId>org.apache.parquet</groupId>
  <artifactId>parquet-hadoop</artifactId>
  <version>1.11.1</version>
</dependency>
<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-common</artifactId>
  <version>3.3.0</version>
</dependency>

<!-- Excel content -->
<dependency>
  <groupId>org.apache.poi</groupId>
  <artifactId>poi</artifactId>
  <version>5.0.0</version>
</dependency>
<dependency>
  <groupId>org.apache.poi</groupId>
  <artifactId>poi-ooxml</artifactId>
  <version>5.0.0</version>
</dependency>

<!-- CVS Content -->
<dependency>
  <groupId>commons-validator</groupId>
  <artifactId>commons-validator</artifactId>
  <version>1.7</version>
</dependency>


Configuration

The included configuration is only for correct interaction with tot, and its values must be adapted to those needed.

YAML
totplugin:
  server:
    urls:
     - http://localhost:15000/tot/
    keep-alive-seconds: 60
    connection:
      - name: dev
        aris:
          - ari: "anja:totplugin:extract:/infra/tech/zone/"
          - ari: "anja:totplugin:sample:/infra/tech/zone/"
          - ari: "anja:totplugin:im:/infra/tech/zone/"


  • location: The access URL for the plugin, required so that tot can redirect calls to the developed plugin.

  • server.url: URL where tot is deployed, required to register, unregister and remain in tot's registries.

  • server.keep-alive-seconds: How often it will re-register in tot to remain registered.

  • aris: Set the triplets (infrastructure, technology and zone) that the plugin governs and what functionalities it offers (extract: extraction, sample: data sampling, im: active governance).

  • encryption.privateKey and encryption.tot.publicKey: These must be the same ones configured in tot so that its security works correctly.


Services

The included services are completely empty; this is where all the logic needed for the integration will be developed. The method names and structure can be freely changed, the only requirement is that they must be called from the controller, which is, ultimately, the entry point in the plugin.