Apps
What is a BigID App[edit]
BigID Applications allow you to add your own business logic and UI to a BigID system. This means that you can add dashboards, synchronize BigID with an external system, or even add entire data governance applications.
BigID applications are written as web applications. This means you can use any programming language and development environment you want. We've created samples in Typescript, Java, and Python to get you started. You also can use our partner Retool to create low-code BigID apps.
Common Use Cases[edit]
There are a variety of uses for BigID apps, but the most common are:
- Sending BigID's classification results to external systems (Alation App, Informatica App, Wiz App)
- Adding intelligence from an external system into the BigID Data Catalog (Okera App)
- Using third-party password stores inside of BigID (AWS Secrets Manager App)
- Generating proprietary reports with BigID data (Sanctions.io)
- Configuring BigID with information from other systems (AWS AutoDiscovery)
How are BigID Apps implemented?[edit]
BigID apps are web applications. In their simplest form can be implemented with 3 HTTP endpoints. More advanced applications provide their own user interface which is also written as a web page.
This means you can use your programming environment of choice to make a BigID app. As long as your programming language of choice can send and receive HTTP requests it can be used to make a BigID app.
The Simplest BigID App[edit]
BigID apps range from a few hundred lines to entire data governance suites. Below is a sample of an minimal BigID app. It consists of the manifest and an action. The specifics of manifests and actions are covered more below, but this code is a good launching point.
const app = express();
app.use(express.json());
app.get('/manifest', (req, res) => {
let manifest = {
app_name: "Training App",
description: "Test App",
vendor: "BigID",
category: "utility",
license_type: "FREE",
actions: [
{
description: "test",
params: [],
is_sync: true,
action_id: "Sync"
}
],
global_params: []
};
res.json(manifest)
});
app.post('/execute', (req, res) => {
let response = {
"statusEnum": "ERROR",
"executionId": req.body.executionId,
"progress": 0,
"message": ""
};
if (req.body.actionName !== "Sync") {
return res.json(response);
}
// Do something here
// Update status to success if we had a success
response.statusEnum = "SUCCESS";
response.progress = 1;
return res.json(response);
})
app.listen(3000);
Knowledge Check[edit]
Defining Your Application[edit]
The name, description, and capabilities of your application are defined in a JSON file called the App Manifest. The App Manifest tells us two types of information:
- Who your application is (metadata)
- What your application can do (actions)
A sample manifest is below:
{
"app_name": "Training App",
"description": "This application is a training sample",
"category": "privacy",
"license_verification_key": "",
"license_type":"FREE",
"vendor": "BigID",
"is_interactive": true,
"actions": [],
"global_params": []
}
- app_name - The name that will appear within the apps page for your application
- description - The description that will appear within the apps page for your application
- category - The section of the apps page your app will be placed in once installed. Options are "privacy", "protection", "perspective", and "utility"
- license_verification_key - A key you receive from the BigID Marketplace after your app has been reviewed and submitted. Without a key, a warning will be presented to users when installing your application.
- license_type - Whether a paid license is required for your application. Options are "paid" or "free"
- vendor - The name of the application author
- is_interactive - Boolean representing whether this application has a custom UI component. True if a custom ui component exists.
- actions - One or more App Actions that your application can perform
- global_params - global settings for your application that are sent with all action requests
Exercise: Create an App Manifest[edit]
Exposing Logs to BigID[edit]
Apps are most commonly installed within Kubernetes clusters. This makes it difficult for users other than system administrators to know what's going on in your application. To make this process easier you can publish logs to the /logs endpoint as text. The contents of this endpoint will then be accessible to users by navigating to the App Activity Log within the BigID UI.
Customizing Icons[edit]
Apps have two endpoints where they can customize icons displayed within the UI, /assets/icon sets the icon shown within the app drawer. /assets/sideBarIcon sets the icon shown within the sidebar while the app is active or pinned to the sidebar. You can return jpg, png, and svg icons at that endpoint, but we suggest svg to allow the icon to scale with your user's screens.
Defining App Actions[edit]
Actions are schedulable pieces of business logic that your application allows BigID to access. They can synchronize between BigID and other systems, modify BigID contents and more. They are defined in your manifest under actions with the following format:
{
"action_id": "dsConnections",
"description": "The action updates the data source connections in our training app",
"is_sync": true,
"action_params": [
{
"param_name": "save",
"param_type": "boolean",
"is_cleartext": true,
"param_description": "Should we save data source connections?",
"default_value": "true",
"param_priority": "primary",
"is_mandatory": true
}
]
}
This action will look like the following inside BigID:
Actions can have multiple configurations and schedules. For instance, someone can run your action once a week with settings to modify all data sources and once a day to modify a specific data source.
Exercise: Defining App Actions[edit]
Implementing App Actions[edit]
All app actions will result in a POST request to your app's /execute endpoint. This post request will have the following body:
{
"actionName": "dsConnections",
"executionId": "5f0bd3bad10a2604246ad846",
"globalParams": [],
"actionParams": [
{
"paramName": "save",
"paramValue": "true"
}
],
"bigidToken": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX25hbWUiOiJiaWdpZC13ZWIiLCJpc0FkbWluIjp0cnVlLCJyb2xlSWRzIjpbInN5c3RlbSJdLCJ0eXBlIjoiYWNjZXNzLXRva2VuIiwiaWF0IjoxNTk0NjEwNjE4LCJleHAiOjE1OTUyMTU0MTh9.0isHh5qJ1pa8rwJVLQD-wjf5Vik5-dwNtwBGM0EFQCw",
"updateResultCallback": "https://bigid.mybigid.com:443/api/v1/tpa/executions/5f0bd3bad10a2604246ad846",
"bigidBaseUrl": "https://bigid.mybigid.com:443/api/v1/",
"tpaId": "5f04b073292cf28c3bb756fb"
}
- actionName - the name of the action that you defined in your manifest that is currently being executed
- executionId - the ID of this particular call to the action. Used in BigID for auditing and tracking if an action was completed successfully.
- globalParams/actionParams - parameters defined in the the manifest and the values set for these parameters in the BigID UI.
- bigidToken - the API token that lets you query BigID APIs. You need this to get data from BigID
- updateResultCallback - URL is where you can send status information about long running tasks so BigID knows they are complete
- bigidBaseUrl - the URL of the BigID API so you know where to send API calls
- tpaId - unique ID for our application from BigID
The bigidToken is a system token that can be used to access any of the BigID API endpoints.
Your /execute endpoint should return its status with the following JSON:
{
"statusEnum": "COMPLETED/ERROR/IN_PROGRESS",
"executionId": executionId,
"progress": 1,
"message": "Successfully imported data sources"
}
This status will be displayed within the application's activity page in the BigID UI. Your app may have long running actions and need to incrementally update the user with its status. You can do that using the updateResultCallback provided in the initial request to your action.
PUT https://bigidBaseUrl/api/v1/tpa/executions/executionId HTTP/1.1
Authorization: bigidToken
{
"statusEnum": "IN_PROGRESS",
"progress": 0.5,
"message": "Almost there!"
}
Exercise: Implementing App Actions[edit]
Now that you've created an basic action, you can use API token supplied by BigID to your action to call any of the BigID API Endpoints to modify and retrieve data in BigID.
Creating an App Frontend[edit]
Apps can also have a user interface that's displayed within the BigID UI. This UI takes the form of a website. The app framework provides an SDK for both Typescript and Javascript that allows you to communicate with BigID from your app.
In your manifest, set the is_interactive property to true.
{
"is_interactive": true,
}
When you install an app with this flag set equal to true, users will be prompted for a UI URL during installation as can be seen in the below app install walkthrough. By default this will be set to <YOUR APP>/ui, so we suggest you use that endpoint to make installation easier for end users.
This UI can either be served directly to your users or BigID can function as a proxy to request the contents of the UI from a private IP address only accessible to BigID.
Exercise: Adding a UI[edit]
Framework Features[edit]
Storing Application Data[edit]
BigID applications should be multitenant. This allows your application to be used by many BigID installations. To make this possible, you need someplace to store configuration and application data that is unique to each environment. BigID allows your application to store information through a series of API calls named TPA Storage.
To store a value into TPA storage you can post the information to the TPA storage endpoint:
Then to retrieve the values you've stored, you can retrieve them with a GET request like below:
If you want to retrieve just a single key like the test key we set earlier you can use the /key endpoint like below:
Both actions and the BigID UI SDK provide you the context of the environment you're in. To make your application properly store data across environments, use the bigidBaseUrl provided by an action or the getBffUrl() function in the UI SDK to determine the base url to send requests to.
Retrieve Data Source Credentials[edit]
If enabled in the application settings, your application can retrieve data source credentials from BigID to act upon data sources without prompting the user to enter credentials again.
To enable this feature, your app needs to do the following:
- Set the APPLICATION_CREDENTIALS_KEY environment variable in your BigID installation to a SHA256 key.
- Enable "Allow Application to retrieve BigID data sources credentials" within your application settings
- Retrieve encrypted data source credentials from tpa/{tpaid}/credentials/{datasource}
- Decrypt Credentials
Supply Data Source Credentials[edit]
Your application can supply credentials to BigID. This allows BigID to interface with custom credentials stores like Amazon Secrets Manager, Azure Credentials Manager, or a homegrown solution. To do this you need to create an action in your application to receive credential requests.
When the scanner needs to contact a data source that your application is providing credentials for it will execute the action. This action will be look the same as other action executions, but will have an additional parameter credentialProviderCustomQuery that is set by the user to indicate which credential they want your app to provide. The scanner request will look like the following:
{
"actionName": "your-action",
"executionId": "executionid",
"globalParams": [],
"actionParams": [
{
"paramName": "credentialProviderCustomQuery",
"paramValue": "user set credential locator"
}
],
"bigidToken": "bigidjwt",
"updateResultCallback": "https://bigid.mybigid.com:443/api/v1/tpa/executions/executionid",
"bigidBaseUrl": "https://bigid.mybigid.com:443/api/v1/",
"tpaId": "appid"
}
In response to this your application should look up the credential and supply the username and password in the additionalData field of the response like below:
{
"executionId": "executionid",
"statusEnum": "COMPLETED",
"progress": 1,
"message": "User found and deserialized",
"additionalData": {
"username": "admin",
"password": "password"
}
}
The scanner will then use that username and password to contact the data source.
A sample password vault implementation is available at https://source.bigid.tools/training/credentials-vault
Credential Types[edit]
The above sample shows basic authentication, but there's several other types of credentials that are available.
Basic Credential Type[edit]
This credential type supplies a username and password.
{
"executionId": "executionid",
"statusEnum": "COMPLETED",
"progress": 1,
"message": "User found and deserialized",
"additionalData": {
"username": "admin",
"password": "password"
}
}
JSON Credential Type[edit]
This credential type supplies a JSON object.
{
"executionId": "executionid",
"statusEnum": "COMPLETED",
"progress": 1,
"message": "User found and deserialized",
"additionalData": {
"content_enc": "{}"
}
}
Personal Access Token Credential Type[edit]
This credential type supplies an access token for a user.
{
"executionId": "executionid",
"statusEnum": "COMPLETED",
"progress": 1,
"message": "User found and deserialized",
"additionalData": {
"personalAccessToken": "token"
}
}
Kerberos Principal Credential Type[edit]
This credential type supplies a Kerberos Principal username and password.
{
"executionId": "executionid",
"statusEnum": "COMPLETED",
"progress": 1,
"message": "User found and deserialized",
"additionalData": {
"principal": "principal",
"username": "user",
"password": "pass"
}
}
Key Credential Type[edit]
This credential type supplies a secret key.
{
"executionId": "executionid",
"statusEnum": "COMPLETED",
"progress": 1,
"message": "User found and deserialized",
"additionalData": {
"authentication_key_enc": "secret"
}
}
Account Authentication Credential Type[edit]
This credential type supplies a user account and a secret key.
{
"executionId": "executionid",
"statusEnum": "COMPLETED",
"progress": 1,
"message": "User found and deserialized",
"additionalData": {
"authentication_key_enc": "secret",
"accountName": "user"
}
}
OAuth2 Credential Type[edit]
This credential type supplies an OAuth2 client id, client secret and token url. It is used for data sources that use the OAuth2 Client Credentials flow.
{
"executionId": "executionid",
"statusEnum": "COMPLETED",
"progress": 1,
"message": "User found and deserialized",
"additionalData": {
"url": "tokenurl.com/oauth2/token",
"clientId": "clientid",
"client_secret_enc": "secret"
}
}
Role Credential Type[edit]
This credential type supplies a role name and session name.
{
"executionId": "executionid",
"statusEnum": "COMPLETED",
"progress": 1,
"message": "User found and deserialized",
"additionalData": {
"roleSessionName": "session",
"roleResourceName": "resource"
}
}
AAD Service Principal Credential Type[edit]
This credential type supplies a principal id and secret used for Azure Active Directory.
{
"executionId": "executionid",
"statusEnum": "COMPLETED",
"progress": 1,
"message": "User found and deserialized",
"additionalData": {
"principalId": "principal",
"principal_secret_enc": "secret"
}
}
Credential Credential Type[edit]
This credential type supplies an access key and secret.
{
"executionId": "executionid",
"statusEnum": "COMPLETED",
"progress": 1,
"message": "User found and deserialized",
"additionalData": {
"accessKey": "key",
"secret_key_enc": "secret"
}
}
STS Credential Type[edit]
This credential type supplies an access key, secret, and session token for use with AWS STS.
{
"executionId": "executionid",
"statusEnum": "COMPLETED",
"progress": 1,
"message": "User found and deserialized",
"additionalData": {
"accessKey": "key",
"secret_key_enc": "secret",
"session_token_enc": "000000"
}
}
Custom Permissions and Roles[edit]
Permissions in BigID allow you to control which users have access to which parts of your system. You can create custom permissions to control how users access your app. For example you could create a custom permission to make your application read only for your report writers, but give full access to admins.
Roles in BigID allow you to give a set of permissions to users. For instance you might want all report writers to be able to do the same thing. You also might want all users of your app to be able to do certain things. That's where custom roles come in.
Both custom roles and custom permissions are defined in the manifest.
A custom permission has the following format:
{
"action": "permission.action",
"label": "Label Displayed in UI",
"description": "Description about what this permissions allows the user to do"
}
A role is a collection of permissions and has the following format:
{
"name": "test",
"permissions": ["permission.action1","permission.action2"],
}
So if we wanted to add a custom permission named app.deleteData and a custom role named App Admin, our manifest would look like below:
{
"app_name": "Training App",
"description": "This application is a training sample",
"category": "privacy",
"license_verification_key": "",
"license_type":"FREE",
"vendor": "BigID",
"is_interactive": true,
"actions": [],
"global_params": []
"permissions": [
{
"action": "app.DeleteData",
"label": "Allow this user to delete data",
"description": "This allows a user to delete data in sample app"
}
],
"custom_roles": [
{
"name": "App Admin",
"permissions": ["app.DeleteData"]
}
]
}
Your application can then retrieve the permissions assigned to a user by doing a GET request to the /api/v1/roles/rbac/user-permissions endpoint using the user's token like below:
Custom permissions will be name-spaced with the permission.applications.{App Name} prefix
Certification[edit]
To become a BigID certified developer, you need to create an application showing your knowledge. There are no specific programming language requirements for this application, just like within BigID. We will not judge your code on syntax or style. We’ve designed this project to be flexible enough to focus on your specific domain while also being rigorous enough to verify your knowledge to BigID customers around the world.
Grading[edit]
The project consists of requirements and optional exercises. Your application MUST implement all requirements. Not implementing a single requirement will result in a failing score. After implementing all requirements, you may choose between optional exercises to get yourself to 5 points. You do not need to implement all requirements from one exercise. Any activity that generates points will count.
Requirements[edit]
- Your application must be packaged as a .zip file.
- Your application must respond to port 3000 over HTTP.
- Your application must have a valid Manifest file and serve that file at /manifest.
- Your application manifest must have a name and a description.
- Your application may not rely on any external servers beyond those specified in this document.
Optional Exercises[edit]
Complete as many of the below exercises to obtain 5 points.
System Integration Utility[edit]
- Your application responds to an action named “synchronize” with a status of “In Progress.” This action is also specified within your manifest. (+1 points)
- Your application validates that the BigID Token provided to the actions endpoint before performing any other action. (+1 points)
- The “synchronize” action sends a JSON array of data source IP addresses from the BigID instance to test-harness.mybigid.com. This request contains the “TESTID” environment variable in the Authorization header. (+1 points)
- If the request to test-harness.mybigid.com fails, the application will report an error status to the BigID instance callback URL. (+1 points)
- If the request to test-harness.mybigid.com succeeds, the application will report a success status to the BigID instance callback URL. (+1 points)
- Your application uploads a file named result.json as a report to BigID's action storage (https://api.bigid.com/wrappers/tpa.html#post-/executions/-executionId-/attachment). (+1 points)
Password Manager[edit]
- Your application responds to an action named “vault” with a status on “In Progress.” This action is also specified within your manifest. (+1 points)
Interactive Dashboard[edit]
- Your application returns an HTML at /ui that retrieves a token using the BigID UI SDK (+1 points)
- Using the BigID UI SDK, you store a value named "User" within your app's TPA storage. (+1 points)
- Using the BigID UI SDK, you retrieve a value named "User" from your app's TPA storage. (+1 points)
Submit Certification Project[edit]