BigID API/Add Data Source Tutorial

In this article, you'll learn:

• How to list available data source connectors within a system
• Where to find parameters required for a given data source
• How to add a data source using the BigID API

scenarioYou have a custom system that maintains an inventory of your databases and have just purchased BigID to better understand what's inside those databases for an internal AI initiative. Your executive staff says this will "bring your organization into the modern age". Use the BigID API to add a data source to start your integration with the inventory system.

In this tutorial, we'll use SAMPLE as our session token. This is unique to the training sandbox and will not work in other environments. See BigID API/Tutorial for information on authenticating with BigID.

Discovering Data Sources

You can see what data source connectors are installed in your environment through the BigID UI, but since we're focused on the API (and because all actions in the UI can be performed in the API), we are going to use the API to retrieve them.

Press Send on the request below to get a listing of the data source connectors installed on our test BigID system.

You'll see our test system has around 70 different data source connectors installed. If you don't see a data source you want to use, you can develop your own or it might already exist, just not on our system. See the BigID docs for an exhaustive list.

Getting Data Source Parameters

Each type of data source has different parameters needed to connect to it. These parameters can be as simple as a username and password or as complex as rate limiting information. BigID uses templates to display these fields to the user in the UI. We can use those same templates to determine what we need to supply when adding a data source via the API. We're going to add a MySQL database. Use the below request to get the template for an rdb-mysql data source.

{
    "name": "rdb-mysql",
    ...
    "fields": [
        {
            "type": "string",
            "name": "name",
            "apiName": "name",
            "displayName": "Data Source Name",
            "placeholder": "Type data source name",
            "mandatory": true,
            "mandatoryForTest": true,
            "validation": [
                {
                    "regex": "^[\\w\\-\\s\\(\\):]+$",
                    "errorText": "Invalid value. Please use alphanumeric characters, spaces, underscore, dash and parentheses."
                }
            ],
            "section": "connection",
            "order": 0,
            "enabled": true
        },
        {
            "type": "stringSelect",
            "name": "enabled",
            "apiName": "enabled",
            "displayName": "Status",
            "defaultValue": "yes",
            "options": [
                {
                    "value": "yes",
                    "label": "Enabled"
                },
                {
                    "value": "no",
                    "label": "Disabled"
                }
            ],
            "section": "connection",
            "order": 1,
            "enabled": true
        },
        {
            "type": "string",
            "name": "friendlyName",
            "apiName": "friendly_name",
            "displayName": "Data Source Friendly Name",
            "placeholder": "Type data source friendly Name",
            "section": "connection",
            "order": 2,
            "enabled": true
        },
        {
            "type": "string",
            "name": "dbUrl",
            "apiName": "rdb_url",
            "displayName": "DB URL",
            "placeholder": ":",
            "tooltipText": "Enter a connection string to the data source.",
            "section": "connection",
            "mandatoryForTest": true,
            "order": 0,
            "enabled": true
        },
        {
            "type": "string",
            "name": "dBSchemaName",
            "apiName": "rdb_name",
            "displayName": "DB/Schema Name",
            "placeholder": ".",
            "tooltipText": "Enter database or schema name. Note: this field may be case sensitive depending on the specific data source.",
            "isSeparatorAfter": true,
            "mandatoryForTest": false,
            "section": "connection",
            "order": 1,
            "enabled": true
        },
        {
            "type": "string",
            "name": "userName",
            "apiName": "username",
            "displayName": "User Name",
            "visibleIf": [
                {
                    "field": "useCredentialOrNamePass",
                    "value": false
                }
            ],
            "enabledIf": [
                {
                    "field": "useCredentialOrNamePass",
                    "value": false
                }
            ],
            "mandatoryForTest": true,
            "section": "connection",
            "order": 6,
            "enabled": true
        },
        {
            "type": "password",
            "name": "password",
            "displayName": "Password",
            "apiName": "password",
            "visibleIf": [
                {
                    "field": "useCredentialOrNamePass",
                    "value": false
                }
            ],
            "enabledIf": [
                {
                    "field": "useCredentialOrNamePass",
                    "value": false
                }
            ],
            "mandatoryForTest": true,
            "isSeparatorAfter": true,
            "nullifyIfNotChanged": true,
            "section": "connection",
            "order": 7,
            "enabled": true
        },

        {
            "type": "string",
            "name": "type",
            "apiName": "type",
            "mandatory": true,
            "hidden": true,
            "defaultValue": "rdb-mysql",
            "section": "connection",
            "order": 9,
            "enabled": true
        },
        {
            "type": "string",
            "name": "tableToScan",
            "apiName": "rdb_comma_sep_table_list",
            "displayName": "Table(s) to scan",
            "tooltipText": "Enter a list of tables to scan, separated by commas. If not specified, all tables will be scanned.",
            "placeholder": "table1, table2, table3",
            "section": "connectionAdvanced",
            "order": 2,
            "enabled": true
        },
        {
            "type": "stringSelect",
            "name": "location",
            "apiName": "location",
            "displayName": "Location",
            "tooltipText": "Select the country, state, province or territory where the Data Source is located.",
            "asyncSource": {
                "url": "countries",
                "fieldsMapping": {
                    "id": "code",
                    "label": "displayName",
                    "value": "name"
                },
                "enabled": true
            },
            "placeholder": "The physical location of the data source",
            "callToFill": {
                "title": "Select a Geographic Location",
                "description": "Geographic location is essential information for regulatory compliance, \n    and can be used as a query parameter in applications that use the query filter, such as Policies, Data Catalog and Inventory.",
                "iconName": "location",
                "order": 2
            },
            "section": "details",
            "order": 3,
            "enabled": true
        },
        {
            "type": "boolean",
            "name": "enableClassifiers",
            "apiName": "classification_is_enabled",
            "displayName": "Enable Classifiers",
            "section": "scanSettings",
            "order": 3,
            "enabled": true
        }
        ...
    ]
}