View source for Connectors


<html><style>
.head {
    max-width:600px;
    height:30vh;
}
header {
    display: flex;
}
.page-heading {
display: none;
}
@media only screen and (max-width: 600px) {
  header {
    flex-direction: column-reverse;
  }
  .head { width:100%; max-width:100%;}
}
</style></html>
{{Box/start}}
In this article, you'll learn:
* What do connectors add to your BigID system
* What types of connectors exist
* How to write a REST-based BigID connector
{{Box/end}}

= What is a BigID Connector =
BigID Connectors allow your BigID system to provide insights about new types of data. Whether that's a known data type like CSV from a new type of data source or something completely new to the BigID ecosystem, a connector will allow you to bring BigID's data discovery capabilities to that system.

==Why do we need connectors? ==
Every data source has its own way of communicating with third parties. Some data sources return information nicely organized, others return it as a jumbled mess. In order for BigID to give you the insights you expect, data needs to be fed to BigID in a consistent way. Connectors work as translators between the multitude of formats that data sources have adapted to the standard format BigID expects. Note that even if the data format is the same (REST JSON, REST XML, GraphQL, etc) small differences make it difficult to reuse connectors. Think of a connector as a way to interface with a single system.

== How are connectors implemented? ==

Connectors can either be implemented as a REST API or as a Java JAR file. REST connectors are bound by the limitations of HTTP connections including timeouts, size limitations and more. Java connectors are well suited for complex use cases especially those involving data sources that stream data.

==BigID Scanning Process==

While BigID has different scanning methods (snapshots, metadata scans, Hyperscan), they all depend on scanners. Scanners allow BigID to contact data sources and create the search maps that are used to power the BigID system. Depending on your deployment model you may have scanners located in the BigID cloud, on-premise, or in your organization's cloud provider accounts. Scanners take the form of a Docker container and require only outbound network access.

In a scan, the scanner will do the following:

*If correlation is enabled, load all correlation records in order to find them within data sources.
*Scan table and file metadata to determine access permissions and ownership
*Classify data streams

After a user starts a scan, the scanner will use the data in the scan request to determine what type of connection to make. In the case of REST API scans, the connector will reach out to your connector. This means your REST connector must allow inbound network access from your scanner, and your data source must allow inbound access from the connector. For Java connectors, the scanner will directly communicate with the data source.

== Connector Types ==

There are two different types of connectors supported within BigID. Which type of connector you
want to use to connect to your data source will have broad implications on setup, network security
settings, and connector installation.

=== Internal (Java-based Connectors) ===
Most of the connectors you are familiar with are Java-based connectors.

These connectors are written in the Java programming language and distributed as JAR files. To
install a new Java-based connector, an administrator must manually load the connector JAR file
into the scanner using the command-line. Thankfully, the 50+ BigID written internal connectors are
bundled in the scanner by default. The scanner directly uses these connectors’ code to connect to
your data sources.

These connectors allow large amounts of customization in the scanning process and the
connection to your data source. Due to the customization options, they are more complicated to
create and are not the recommended connector development method for BigID customers.

===External (Generic REST API Connectors)===
External connectors allow you to create a connector in your favorite programming language. The
scanner will communicate with your connector over HTTPS so as long as your programming language of choice can respond to web requests, it can be used to create an external connector.

External connectors can be hosted on any server that has a network connection to both your
scanner and your data source.

There are two different types of external connectors that you can create: unstructured and structured.

====Unstructured External Connector====

Unstructured connectors allow BigID to scan files from a given data source. An example of an
unstructured data source is Google Drive.

====Structured External Connector====

Structured connectors allow BigID to scan databases. An example of a structured connector would
be our MySQL connector.

== The Simplest BigID Connector ==
Below is sample code for the simplest REST connector you can make.
<syntaxhighlight lang="JavaScript">
const express = require('express')
const app = express()
const port = 3000;

// This is our fake data for the connector
const FAKE_DATA = {
    Customers: [
        { Id: 1, Name: "Michael", Address: "100 Osceola Parkway, Kissimmee FL" },
        { Id: 2, Name: "Bob", Address: "1 Sand Lake Rd, Orlando FL" },
        { Id: 3, Name: "Stewart", Address: "1 Sand Lake Rde, Orlando FL" }
    ],
    Orders: [
        { Id: 1, Item: "banana", customerId: 2, Price: 1 }
    ]
};

/**
 * Describes all objects inside a data source
 *
 * @async
 * @param {{ domain: string; user: string; pass: string; header: string; }} login login information from BigID
 * @param {string} object name of object
 */
async function describeObjects(login) {

    const objects = [];

    // Create a new object for each table
    for (let table in FAKE_DATA) {
        let thisObject = { objectName: table, fields: [] };

        // Get field definitions for the object using the first item in each object
        for (let field in FAKE_DATA[table][0]) {
            thisObject.fields.push({ fieldName: field, fieldType: typeof FAKE_DATA[table][0][field] });
        }
        objects.push(thisObject);
    }

    return {
        status: 'success',
        objects: objects
    };
}


/**
 * Describes a specific object in the data source.
 *
 * @async
 * @param {{ domain: string; user: string; pass: string; header: string; }} login login information from BigID
 * @param {string} object name of object
 */
async function describeObject(login, object) {

    const objectFields = [];

    // Get field definitions for the object using the first item in the object
    for (let field in FAKE_DATA[object][0]) {
        thisObject.fields.push({ fieldName: field, fieldType: typeof FAKE_DATA[table][0][field] });
    }
    objectFields.push(thisObject);

    return {
        status: "success",
        objectName: object,
        fields: objectFields
    }
}

/**
 * Get records from data source for a specific object
 *
 * @async
 * @param {{ domain: string; user: string; pass: string; header: string; }} login login information from BigID
 * @param {string} object
 * @param {number} offset
 * @returns {unknown}
 */
async function getRecords(login, object, offset, count) {
    let records = [];

    // an offset defines where we should start. Use that as the starting point
    // TODO splice only elements needed
    const recordWindow = FAKE_DATA[object];

    for(let record of FAKE_DATA['object']) {
        
        let thisResult = {
            id: record.Id, // All records *must* have a unique ID for BigID to identify them
            data: []
        };
        for(let field in record){
            thisResult.data.push({fieldName: field, fieldType: typeof record[field], fieldValue: record[field]});
        }
        records.push(thisResult);
    }

    return {
        status: "success",
        records: records,
        offset: records.length
    }

}

/**
 * Get a count of the number of records of a given object type
 *
 * @async
 * @param {{ domain: string; user: string; pass: string; header: string; }} login login information from BigID
 * @param {string} object object name
 * @returns {unknown}
 */
async function countRecords(login, object) {
    return {
        status: "success",
        count: FAKE_DATA[object].length
    };
}

/**
 * Search through objects
 *
 * @param {{ domain: string; user: string; pass: string; header: string; }} login
 * @param {*} object
 * @param {*} search
 */
function search(login, object, search) {
    //TODO
}

/**
 * Returns BASIC formatted auth data from the authorization header
 *
 * @param {*} req
 * @returns {{ user: string; pass: string; header: string; }}
 */
function getAuthData(req) {
    
    if (req.headers.authorization === undefined) {
        console.log("No Login information supplied");
        res.status(401).json({
            status: 'error'
        })
        return null;
    }

    const b64auth = (req.headers.authorization || '').split(' ')[1] || ''
    const [login, password] = Buffer.from(b64auth, 'base64').toString().split(':');

    return {
        header: 'Basic ' + Buffer.from(login + ':' + password).toString('base64')
    }
}

app.get('/objects/', async (req, res) => {
    console.log(`LISTOBJS()`);

    const login = getAuthData(req);
    if (login === null) { return; }

    const obj = await describeObjects(login);
    if (obj.status === "success") {
        return res.json(obj)
    }

    return res.status(401).json(obj);
});

app.get('/objects/:object/describe', async (req, res) => {
    console.log(`DESCRIBEOBJS(object=${req.params.object})`);

    const login = getAuthData(req);
    if (login === null) { return; }

    let result = await describeObject(login, req.params.object);

    if (result.status === "success") {
        return res.json(result);
    }
    return res.status(401).json(result);

});

app.get('/objects/:object/records', async (req, res) => {
    console.log(`GETRECORDS(offset=${offset},count=${count},object=${req.params.object})`);

    const login = getAuthData(req);
    if (login === null) { return; }

    let offset = req.query.Offset || 0;
    let count = req.query.Count || 25;
    if (count > 200) { count = 200 }

    return res.json(await getRecords(login, req.params.object, offset, count));
});

app.get('/objects/:object/count', async (req, res) => {
    console.log(`COUNTRECORDS(object=${req.params.object})`);

    const login = getAuthData(req);
    if (login === null) { return; }

    return res.json(await countRecords(login, req.params.object));
});

app.listen(port, () => {
    console.log(`Sample Connector listening on port ${port}`)
});
</syntaxhighlight>

== Knowledge Check ==
<html>
<style>
.quiz input[type=submit] {
    padding: 10px 20px;
    border-radius: 10px;
    background: var(--color-primary);
    font-weight: 600;
    color: #fff;
    border: 0px;
    cursor: pointer;
}
</style>
</html>
<quiz display=simple>

{X}

{What is the primary purpose of BigID connectors? (Select one)
|type="()"}
+ To translate various data formats into a standard format expected by BigID.
- To store large volumes of data.
- To manage user access permissions.
- To generate reports automatically.

{Which statement about REST connectors is true? (Select one)
|type="()"}
+ REST connectors are bound by HTTP limitations such as timeouts and size constraints.
- REST connectors allow unlimited data transfer.
- REST connectors do not require network access.
- REST connectors are better suited for complex use cases involving data streams.

{What are the two types of external connectors in BigID? (Select two)
|type="[]"}
+ Structured
+ Unstructured
- Metadata
- Hyperscan

{How are Java-based connectors installed in BigID? (Select one)
|type="()"}
+ By manually loading the JAR file into the scanner using the command line.
- By uploading them through the BigID web interface.
- By configuring them in the network settings.
- By downloading them directly from the BigID cloud.


{/X}

</quiz>

= Writing a Structured Connector =
</br>
{{Left|
<html>
<center>
<?xml version="1.0" encoding="UTF-8"?>
<!-- Do not edit this file with editors other than draw.io -->
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="350px" height="617px" viewBox="-0.5 -0.5 350 617"><defs/><g><g data-cell-id="0"><g data-cell-id="1"><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-1"><g><rect x="135" y="16" width="60" height="30" fill="none" stroke="none" pointer-events="all"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 58px; height: 1px; padding-top: 31px; margin-left: 136px;"><div style="box-sizing: border-box; font-size: 0; text-align: center;  "><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;;  line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><font style="font-size: 19px;"><b>Structured Connector</b></font><div><font style="font-size: 13px;"><i style="">Databases</i></font></div></div></div></div></foreignObject><text x="165" y="35" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Structured...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-2"><g><rect x="35" y="76" width="260" height="110" rx="16.5" ry="16.5" fill="none" stroke="#000000" pointer-events="all" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 258px; height: 1px; padding-top: 131px; margin-left: 36px;"><div style="box-sizing: border-box; font-size: 0; text-align: center;"><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><b>Object</b><div>Something in the world that you are storing multiple of.</div><div><br /></div><div>Examples: Customers, Products, Orders.</div><div><br /></div><div>All Objects must have the same <i>Fields</i></div></div></div></div></foreignObject><text x="165" y="135" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Object...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-5"><g><path d="M 165 236 L 165 192.37" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/><path d="M 165 187.12 L 168.5 194.12 L 165 192.37 L 161.5 194.12 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all" style="fill: light-dark(rgb(0, 0, 0), rgb(255, 255, 255)); stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-6"><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 211px; margin-left: 195px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Contains</div></div></div></foreignObject><text x="195" y="214" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="11px" text-anchor="middle">Contains</text></switch></g></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-3"><g><rect x="0" y="236" width="330" height="150" rx="22.5" ry="22.5" fill="none" stroke="#000000" pointer-events="all" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 328px; height: 1px; padding-top: 311px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><b>Record</b><div>An individual item you are storing.</div><div><br /></div><div>Examples: Your customer Michael, an apple you sell, an order Michael made for the product</div><div><br /></div><div>All Records must have a unique ID.</div></div></div></div></foreignObject><text x="165" y="315" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Record...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-8"><g><path d="M 165 436 L 165 392.37" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/><path d="M 165 387.12 L 168.5 394.12 L 165 392.37 L 161.5 394.12 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all" style="fill: light-dark(rgb(0, 0, 0), rgb(255, 255, 255)); stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-9"><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 411px; margin-left: 195px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Contains</div></div></div></foreignObject><text x="195" y="414" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="11px" text-anchor="middle">Contains</text></switch></g></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-7"><g><rect x="0" y="436" width="330" height="180" rx="27" ry="27" fill="none" stroke="#000000" pointer-events="all" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 328px; height: 1px; padding-top: 526px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><b>Fields</b><div>Pieces of information about a record.</div><div><br /></div><div>Examples: Michael's address is 100 Church St, an apple costs $1, Michael's order was placed on January 1st.</div><div><br /></div><div>Every field has a field name, a field value and a field type. Field types can be any string, but typically represent database types like string or number</div></div></div></div></foreignObject><text x="165" y="530" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Fields...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-10"><g><rect x="560" y="16" width="60" height="30" fill="none" stroke="none" pointer-events="all"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 58px; height: 1px; padding-top: 31px; margin-left: 561px;"><div style="box-sizing: border-box; font-size: 0; text-align: center;  "><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><font style="font-size: 19px;"><b>Unstructured Connector</b></font><div><font style="font-size: 14px;"><i>Files/Streams</i></font></div></div></div></div></foreignObject><text x="590" y="35" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Unstructur...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-11"><g><rect x="460" y="76" width="260" height="110" rx="16.5" ry="16.5" fill="none" stroke="#000000" pointer-events="all" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 258px; height: 1px; padding-top: 131px; margin-left: 461px;"><div style="box-sizing: border-box; font-size: 0; text-align: center;"><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><b>Container</b><div>A grouping of files or streams. Think of a file folder</div><div><br /></div><div>Examples: Invoices, 2024, LogStreamForApp</div></div></div></div></foreignObject><text x="590" y="135" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Container...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-12"><g><path d="M 590 236 L 590 192.37" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/><path d="M 590 187.12 L 593.5 194.12 L 590 192.37 L 586.5 194.12 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all" style="fill: light-dark(rgb(0, 0, 0), rgb(255, 255, 255)); stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-13"><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 211px; margin-left: 620px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Contains</div></div></div></foreignObject><text x="620" y="214" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="11px" text-anchor="middle">Contains</text></switch></g></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-14"><g><rect x="425" y="236" width="330" height="150" rx="22.5" ry="22.5" fill="none" stroke="#000000" pointer-events="all" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 328px; height: 1px; padding-top: 311px; margin-left: 426px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><b>Object</b><div>An individual file or stream of information</div><div><br /></div><div>Examples: InvoiceJan1.pdf, TaxNoticeFor2024.pdf, Kenesis Stream named LogStreamForApp</div><div><br /></div><div>All Objects must have a unique ObjectName.</div></div></div></div></foreignObject><text x="590" y="315" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Object...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-15"><g><path d="M 710 436 L 710 416 L 590 416 L 590 392.37" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/><path d="M 590 387.12 L 593.5 394.12 L 590 392.37 L 586.5 394.12 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all" style="fill: light-dark(rgb(0, 0, 0), rgb(255, 255, 255)); stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-16"><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 406px; margin-left: 651px;"><div style="box-sizing: border-box; font-size: 0; text-align: center;"><div style="display: inline-block; font-size: 11px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Contains</div></div></div></foreignObject><text x="651" y="410" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="11px" text-anchor="middle">Contains</text></switch></g></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-17"><g><rect x="620" y="436" width="180" height="90" rx="13.5" ry="13.5" fill="none" stroke="#000000" pointer-events="all" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 178px; height: 1px; padding-top: 481px; margin-left: 621px;"><div style="box-sizing: border-box; font-size: 0; text-align: center;"><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><b>Metadata</b><div>Pieces of information about an Object.</div><div><br /></div><div>Examples: Last Modified Date, Owner, Name, File Permissions</div></div></div></div></foreignObject><text x="710" y="485" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Metadata...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-20"><g><path d="M 470 436 L 470 416 L 590 416 L 590 392.37" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/><path d="M 590 387.12 L 593.5 394.12 L 590 392.37 L 586.5 394.12 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all" style="fill: light-dark(rgb(0, 0, 0), rgb(255, 255, 255)); stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-21"><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 406px; margin-left: 530px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Contains</div></div></div></foreignObject><text x="530" y="410" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="11px" text-anchor="middle">Contains</text></switch></g></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-18"><g><rect x="380" y="436" width="180" height="90" rx="13.5" ry="13.5" fill="none" stroke="#000000" pointer-events="all" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 178px; height: 1px; padding-top: 481px; margin-left: 381px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><div><b>InputStream</b></div><div>The data inside the file in UTF8 format</div></div></div></div></foreignObject><text x="470" y="485" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">InputStream...</text></switch></g></g></g></g></g></g></svg></center></html>}}{{Right|Structured connectors are used to connect to systems that organize data in a predefined format. Examples of structured data sources are databases, spreadsheets, and data warehouses. 

As we mentioned before, BigID connectors serve as translators between an external system's format and BigID's data format. The diagram to the left shows the structure of that internal data format. Structured data sources consist of '''Objects'''. You can think of objects as things in the real world like products, or invoices, or even your customers. If you're familiar with relational databases, an object most often takes the form of a table in your database. 

Inside of objects we have '''records'''. Records are an individual instance of an object. Because fields are defined at the object level, all records of the same type will have the same fields. There is only one required field for a Record and that is a unique ID. BigID will use this ID when attempting to fetch an individual record so it's important that it's unique.

Inside of records, we have '''fields and values'''. Fields and values are the actual data BigID will scan for your object. Fields will also have a datatype. The datatype of a field is shown unedited inside the BigID Catalog. This means it can be any string you want to present to users whether that be something easy to understand like "text" or something more specific like varchar(255). Keep in mind your audience when designating the data types for a given field. Many of the users of your BigID system may be concerned seeing highly technical data types.}}<html><div style="clear:both"></div></html>


While your data source may store its data differently, you will need to reorganize it into this structure for BigID to scan the data. Remember that connectors are translators. '''Improper translation from your data source's format to BigID's is the largest issue when testing and developing custom connectors.''' See the below example of a JSON file for a single user converted into this format:

<html>
<div style="display:flex; flex-direction:row; align-items:center; justify-items:space-between">
<div style="width:45%">
</html>
<syntaxhighlight lang="JSON">
{
    "user":"user@bigid.com",
    "phone": "(321) 555-5555",
    "information": [
        {"key": "name", "value": "user"},
        {"key": "favcolor", "value": "green"}
    ]
}
</syntaxhighlight>
<html>
</div>
<div style="text-align:center;padding:10px">translates to</div>
<div style="width:45%">
</html>
<syntaxhighlight lang="JSON">{
  "objectName": "User",
  "fields": [
    {
      "fieldName": "Name",
      "fieldType": "string"
    },
    {
      "fieldName": "FavColor",
      "fieldType": "string"
    },
    {
      "fieldName": "Phone",
      "fieldType": "string"
    }
  ]
}</syntaxhighlight>
<syntaxhighlight lang="JSON">{
  "id": "user@bigid.com",
  "fields": [
    {
      "fieldName": "phone",
      "fieldValue": "(321) 555-5555"
    },
    {
      "fieldName": "name",
      "fieldValue": "user"
    },
    {
      "fieldName": "favcolor",
      "fieldValue": "green"
    }
  ]
}</syntaxhighlight>
<html></div></div></html>

{{Note|If you're using code generation software, or just want it for your own purposes, the swaggerfile describing a structured connector is available [https://developer.bigid.com/images/4/45/Structured.yaml here.]}} 

== Connector Endpoints ==

BigID will request and view your connector’s hierarchy through six endpoints:

* List what '''fields''' an '''object''' has (GET /objects/<objectName>/describe)
* List what '''objects''' exist (/objects)
* List what '''records''' are inside an '''object''' (GET /objects/<objectName>/records)
* Count the number of '''records''' inside an '''object''' (GET /objects/<objectName>/count)
* Return the '''fields and values''' for a given record ID (GET /objects/<objectName>/<recordID>)
* Search for '''records''' in an '''object''' (POST /objects/<objectName>/sar)

You are not required to implement every endpoint outlined here. If you want a connector that only does DSAR requests you only need to implement the following endpoints:

* List what '''objects''' exist (/objects)
* List what '''fields''' an '''object''' has (GET /objects/<objectName>/describe)
* Search for '''records''' in an '''object''' (POST /objects/<objectName>/sar)
{{Warning|Any additional endpoints added to your connector are of no interest to BigID. The BigID system is not concerned with the implementation details of your connector as long as HTTP requests to these endpoints succeed and are in the correct format}}
=== List what '''fields''' an '''object''' has ===
Since all records within an object must contain the same fields, BigID uses an object's fields to tell users what type of information is inside your data source. BigID expects the following response by your connector to see what fields an item has:
<syntaxhighlight lang="HTTP">GET /objects/User/describe HTTP/1.1</syntaxhighlight>
<syntaxhighlight lang="JSON">
{
  "status": "success",
  "objectName": "User",
  "fields": [
    {
      "fieldName": "Name",
      "fieldType": "string"
    },
    {
      "fieldName": "FavColor",
      "fieldType": "string"
    },
    {
      "fieldName": "Phone",
      "fieldType": "string"
    }
  ]
}</syntaxhighlight>


====Exercise: List what '''fields''' an '''object''' has====

{{Scenario|You want to create a connector for your org to be able to scan for sensitive data within an ecommerce system, ShopDB. '''Create an endpoint for your connector that allows BigID to list an object's fields'''}}
{{CodeTest|connector%2F1-list-object}}

=== List what '''objects''' exist ===

Now that we have an object class created, we need to let BigID all of the objects our connector
has access to. Our object listing endpoint needs to display the fields within an object that we
implemented in the previous exercise. As we implement these endpoints, many of them rely on
one another.

<syntaxhighlight lang="HTTP">GET /objects/ HTTP/1.1</syntaxhighlight>
<syntaxhighlight lang="JSON">
{
  "status": "success",
  "objects": [
    {
      "objectName": "User",
      "fields": [
        {
          "fieldName": "Name",
          "fieldType": "string"
        },
        {
          "fieldName": "FavColor",
          "fieldType": "string"
        },
        {
          "fieldName": "Phone",
          "fieldType": "string"
        }
      ]
    }
  ]
}</syntaxhighlight>

==== Exercise: List what '''objects''' exist ====

{{Scenario|For your ShopDB connector to work, you need to tell BigID what objects are inside the database. The code for this endpoint is similar to your previous endpoint. '''Create an endpoint for your connector that allows BigID to list out all objects.'''}}

{{CodeTest|connector%2F2-list-objects}}

=== List what '''records''' are inside an object ===

Now that BigID knows what objects we have and what type of information it can find inside those
objects, we need to return the actual data BigID will scan.

<syntaxhighlight lang="HTTP">GET /objects/User/records?Count=&Offset= HTTP/1.1</syntaxhighlight>
<syntaxhighlight lang="JSON">
{
  "status": "success",
  "records": [
    {
      "id": 1,
      "data": [
        {
          "fieldName": "Name",
          "fieldValue": "Michael",
          "fieldType": "string"
        },
        {
          "fieldName": "Phone",
          "fieldValue": "3215555555",
          "fieldType": "string"
        }
      ]
    }
  ],
  "offset": 1
}</syntaxhighlight>

====Exercise: List what '''records''' are inside an object====

{{Scenario|BigID has the structure of your connector, but scans are still failing with 0 records found. '''Create an endpoint for your connector that allows BigID to list out records.'''}}
{{CodeTest|connector%2F3-records}}

=== '''Count''' the number of records inside an object ===

In order for BigID to properly paginate through your data sources, we need to know how many total records to expect.

<syntaxhighlight lang="HTTP">GET /objects/User/count HTTP/1.1</syntaxhighlight>
<syntaxhighlight lang="JSON">
{
  "status": "success",
  "count": 1
}</syntaxhighlight>

=== Record Pagination ===

In a previous exercise, you returned all records inside of an object in a single call. It's extremely common for providers not to allow all records to be returned in a single call either do to performance, or cost reasons. This is where the offset and count parameter come in. 

Data providers typically meter their APIs in one of the following ways:

* Limit the number of requests per second/hour/day
* Limit the number of records returned in a single request
* Limit the amount of data in GB processed in a given time period

To handle these use cases, BigID has two properties on the records endpoint: Count and Offset.

Count dictates the batch size. This is how many records your connector should return in a single call. This allows BigID users to throttle your connector based on how many records it returns per page.

Offset allows BigID to paginate requests.

A sample scan using the records endpoint looks like the following:
<html>
<script src="https://cdn.jsdelivr.net/npm/mermaid/dist/mermaid.min.js"></script>
<center>
<div class="mermaid">
    sequenceDiagram
      participant BigID
      participant Connector

      BigID->>Connector: Call /count
      Connector-->>BigID: Total records = 40

      BigID->>Connector: Call /records with offset = 0 and count = 20
      Connector-->>BigID: 20 Records retrieved, offset = 20

      BigID->>Connector: Call /records with offset = 21 and count = 20
      Connector-->>BigID: 20 Records retrieved, offset = 40

      BigID->>BigID: Stop as offset >= total (40 >= 40)
  </div>
</center>
<script>
if (document.documentElement.classList.contains('skin-citizen-dark')) {
  mermaid.initialize({ startOnLoad: true, theme: 'dark' });
} else {
  mermaid.initialize({ startOnLoad: true, theme: 'base' });
}
    
  </script>
</html>

{{Warning|Not implementing offset correctly will result in BigID continually scanning the data source and the scan never completing. Your connector must stop sending records when the offset supplied is larger than the total number of records.}}

==== Exercise: Pagination ====
{{Scenario|Your connector is constantly scanning and scans are not completing. '''Implement pagination so BigID knows when you are done scanning.'''}}
{{CodeTest|connector%2F3-pagination}}

===Return the fields and values for a given '''record ID'''===

There are cases when we only want to look into an individual record. This is especially true in cases
where we already know the unique ID like a DSAR or a record investigation.

<syntaxhighlight lang="HTTP">GET /objects/User/records/1 HTTP/1.1</syntaxhighlight>
<syntaxhighlight lang="JSON">
{
  "status": "success",
  "records": [
    {
      "id": 1,
      "data": [
        {
          "fieldName": "Name",
          "fieldValue": "Michael",
          "fieldType": "string"
        },
        {
          "fieldName": "Phone",
          "fieldValue": "3215555555",
          "fieldType": "string"
        }
      ]
    }
  ],
  "offset": 1
}</syntaxhighlight>

====Exercise: Return the fields and values for a given '''record ID'''====
{{Scenario|Your team has been trying to investigate sensitive data in your data source using the "Investigate" button in the data inventory, but they aren't getting results. '''Create an endpoint for your connector that allows BigID to lookup an individual record'''}}
{{CodeTest|connector%2F4-record}}

==='''Search''' for records in an object===

Our final service that our connector needs to offer is the ability to search for records. This is what
allows our connector to do DSARs. Depending on your data source, you may need to implement searching on your own. 

<syntaxhighlight lang="HTTP">POST /objects/User/records/sar HTTP/1.1
Content-Type: application/json


[{"fieldName": "Name", "fieldValue": "Michael", "isFullMatch": "false"}]
</syntaxhighlight>
<syntaxhighlight lang="JSON">
{
  "status": "success",
  "records": [
    {
      "id": 1,
      "data": [
        {
          "fieldName": "Name",
          "fieldValue": "Michael",
          "fieldType": "string"
        },
        {
          "fieldName": "Phone",
          "fieldValue": "3215555555",
          "fieldType": "string"
        }
      ]
    }
  ],
  "offset": 1
}</syntaxhighlight>

====Exercise: '''Search''' for records in an object====
{{Scenario|Your team is using BigID to comply with GDPR DSAR requests. ShopDB doesn't have any search functionality so you'll have to implement it yourself. '''Implement the SAR endpoint so BigID can properly query your data source.'''}}
{{CodeTest|connector%2F5-sar}}

=== Authentication ===

The connector you've written is allowing anyone on the internet to see the information inside of your data source which is no good. This connector's database also has no username and password which is extremely rare. Typically the data sources your connectors will access will also require authentication of some type. 

BigID has the ability to store and manage these credentials for you inside of the BigID system or use credentials from external password vaults. When a scan is started, BigID will send the credentials to your connector. 

Credentials sent to your connector will always take the form of a [https://en.wikipedia.org/wiki/Basic_access_authentication Basic authentication header]. This header is not encrypted. For this reason, you must never install a BigID connector without HTTPS.

{{Warning|Never install a BigID connector without HTTPS. Credentials are sent to them unencrypted and will be exposed if you do not secure your connector}}

While Basic auth credentials always take the form of a username and password, not all systems use that mechanism for login. Below are some examples of how you can format a username and password into the credentials your application is expecting:

* OAuth Client Credentials - Username: https://tenanturl.com/,CLIENT_ID Password: CLIENT_SECRET
* Multi Tenant SaaS application with API key - Username: tenantID Password: API_KEY
* SaaS application with just an API key - Username: <blank> Password: API_KEY

====Exercise: Authentication====

{{Scenario|Your organization is writing a connector and wants to be sure the data is only viewable to BigID. You've set up the appropriate firewalls, but also want your connector to be protected by a username and password. '''Protect your records endpoint with username password auth'''}}
{{CodeTest|connector%2F3-auth}}

{{Warning|In the real world, never hardcode credentials into a connector. Your connector should use the native auth of your data source. This will allow you to reuse the connector for multiple instances of that data source.}}

= Writing an Unstructured Connector =
</br>
{{Left|
<html>
<center>
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="450px" height="617px" viewBox="350 0 450 617"><defs/><g><g data-cell-id="0"><g data-cell-id="1"><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-1"><g><rect x="135" y="16" width="60" height="30" fill="none" stroke="none" pointer-events="all"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 58px; height: 1px; padding-top: 31px; margin-left: 136px;"><div style="box-sizing: border-box; font-size: 0; text-align: center;  "><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;;  line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><font style="font-size: 19px;"><b>Structured Connector</b></font><div><font style="font-size: 13px;"><i style="">Databases</i></font></div></div></div></div></foreignObject><text x="165" y="35" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Structured...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-2"><g><rect x="35" y="76" width="260" height="110" rx="16.5" ry="16.5" fill="none" stroke="#000000" pointer-events="all" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 258px; height: 1px; padding-top: 131px; margin-left: 36px;"><div style="box-sizing: border-box; font-size: 0; text-align: center;"><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><b>Object</b><div>Something in the world that you are storing multiple of.</div><div><br /></div><div>Examples: Customers, Products, Orders.</div><div><br /></div><div>All Objects must have the same <i>Fields</i></div></div></div></div></foreignObject><text x="165" y="135" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Object...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-5"><g><path d="M 165 236 L 165 192.37" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/><path d="M 165 187.12 L 168.5 194.12 L 165 192.37 L 161.5 194.12 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all" style="fill: light-dark(rgb(0, 0, 0), rgb(255, 255, 255)); stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-6"><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 211px; margin-left: 195px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Contains</div></div></div></foreignObject><text x="195" y="214" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="11px" text-anchor="middle">Contains</text></switch></g></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-3"><g><rect x="0" y="236" width="330" height="150" rx="22.5" ry="22.5" fill="none" stroke="#000000" pointer-events="all" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 328px; height: 1px; padding-top: 311px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><b>Record</b><div>An individual item you are storing.</div><div><br /></div><div>Examples: Your customer Michael, an apple you sell, an order Michael made for the product</div><div><br /></div><div>All Records must have a unique ID.</div></div></div></div></foreignObject><text x="165" y="315" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Record...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-8"><g><path d="M 165 436 L 165 392.37" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/><path d="M 165 387.12 L 168.5 394.12 L 165 392.37 L 161.5 394.12 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all" style="fill: light-dark(rgb(0, 0, 0), rgb(255, 255, 255)); stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-9"><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 411px; margin-left: 195px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Contains</div></div></div></foreignObject><text x="195" y="414" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="11px" text-anchor="middle">Contains</text></switch></g></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-7"><g><rect x="0" y="436" width="330" height="180" rx="27" ry="27" fill="none" stroke="#000000" pointer-events="all" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 328px; height: 1px; padding-top: 526px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><b>Fields</b><div>Pieces of information about a record.</div><div><br /></div><div>Examples: Michael's address is 100 Church St, an apple costs $1, Michael's order was placed on January 1st.</div><div><br /></div><div>Every field has a field name, a field value and a field type. Field types can be any string, but typically represent database types like string or number</div></div></div></div></foreignObject><text x="165" y="530" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Fields...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-10"><g><rect x="560" y="16" width="60" height="30" fill="none" stroke="none" pointer-events="all"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 58px; height: 1px; padding-top: 31px; margin-left: 561px;"><div style="box-sizing: border-box; font-size: 0; text-align: center;  "><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><font style="font-size: 19px;"><b>Unstructured Connector</b></font><div><font style="font-size: 14px;"><i>Files/Streams</i></font></div></div></div></div></foreignObject><text x="590" y="35" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Unstructur...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-11"><g><rect x="460" y="76" width="260" height="110" rx="16.5" ry="16.5" fill="none" stroke="#000000" pointer-events="all" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 258px; height: 1px; padding-top: 131px; margin-left: 461px;"><div style="box-sizing: border-box; font-size: 0; text-align: center;"><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><b>Container</b><div>A grouping of files or streams. Think of a file folder</div><div><br /></div><div>Examples: Invoices, 2024, LogStreamForApp</div></div></div></div></foreignObject><text x="590" y="135" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Container...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-12"><g><path d="M 590 236 L 590 192.37" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/><path d="M 590 187.12 L 593.5 194.12 L 590 192.37 L 586.5 194.12 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all" style="fill: light-dark(rgb(0, 0, 0), rgb(255, 255, 255)); stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-13"><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 211px; margin-left: 620px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Contains</div></div></div></foreignObject><text x="620" y="214" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="11px" text-anchor="middle">Contains</text></switch></g></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-14"><g><rect x="425" y="236" width="330" height="150" rx="22.5" ry="22.5" fill="none" stroke="#000000" pointer-events="all" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 328px; height: 1px; padding-top: 311px; margin-left: 426px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><b>Object</b><div>An individual file or stream of information</div><div><br /></div><div>Examples: InvoiceJan1.pdf, TaxNoticeFor2024.pdf, Kenesis Stream named LogStreamForApp</div><div><br /></div><div>All Objects must have a unique ObjectName.</div></div></div></div></foreignObject><text x="590" y="315" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Object...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-15"><g><path d="M 710 436 L 710 416 L 590 416 L 590 392.37" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/><path d="M 590 387.12 L 593.5 394.12 L 590 392.37 L 586.5 394.12 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all" style="fill: light-dark(rgb(0, 0, 0), rgb(255, 255, 255)); stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-16"><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 406px; margin-left: 651px;"><div style="box-sizing: border-box; font-size: 0; text-align: center;"><div style="display: inline-block; font-size: 11px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Contains</div></div></div></foreignObject><text x="651" y="410" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="11px" text-anchor="middle">Contains</text></switch></g></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-17"><g><rect x="620" y="436" width="180" height="90" rx="13.5" ry="13.5" fill="none" stroke="#000000" pointer-events="all" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 178px; height: 1px; padding-top: 481px; margin-left: 621px;"><div style="box-sizing: border-box; font-size: 0; text-align: center;"><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><b>Metadata</b><div>Pieces of information about an Object.</div><div><br /></div><div>Examples: Last Modified Date, Owner, Name, File Permissions</div></div></div></div></foreignObject><text x="710" y="485" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">Metadata...</text></switch></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-20"><g><path d="M 470 436 L 470 416 L 590 416 L 590 392.37" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/><path d="M 590 387.12 L 593.5 394.12 L 590 392.37 L 586.5 394.12 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all" style="fill: light-dark(rgb(0, 0, 0), rgb(255, 255, 255)); stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-21"><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 406px; margin-left: 530px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 11px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Contains</div></div></div></foreignObject><text x="530" y="410" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="11px" text-anchor="middle">Contains</text></switch></g></g></g></g><g data-cell-id="FfGzdVv6ISSjYSYJcTpr-18"><g><rect x="380" y="436" width="180" height="90" rx="13.5" ry="13.5" fill="none" stroke="#000000" pointer-events="all" style="stroke: light-dark(rgb(0, 0, 0), rgb(255, 255, 255));"/></g><g><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 178px; height: 1px; padding-top: 481px; margin-left: 381px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: &quot;Arial&quot;; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; "><div><b>InputStream</b></div><div>The data inside the file in UTF8 format</div></div></div></div></foreignObject><text x="470" y="485" fill="light-dark(#000000, #ffffff)" font-family="&quot;Arial&quot;" font-size="12px" text-anchor="middle">InputStream...</text></switch></g></g></g></g></g></g></svg></center></html>}}{{Right|Unstructured connectors allow you to connect to data sources that store data as unorganized blobs of data. Most commonly this means file stores, but data warehouses and streaming applications also fall under this type of data sources. Because unstructured data sources don't have discrete labels for each piece of data inside them, we use a different format inside of BigID for this type of data. 

Unstructured sources have '''Containers''' that represent a logical grouping of data. Typically a '''Container''' is a file folder, but it can be any label applied to a group of data.

Inside of containers you have '''Objects'''. Objects represent a single piece of data stored inside a container. The most common example of an object is a file. All objects inside a container need to have a unique name so BigID can request information about a specific object.

Objects contain two separate pieces of data. One is '''Metadata''. Metadata is information ''about'' a given file or stream. Most commonly this means information like who the owner of the file is and when it was last modified. Some types of data like photos may have additional metadata like what type of camera took the picture. BigID uses this data to analyze permissions and scan faster.

Objects also contain an '''inputStream'''. This is the raw file contents that BigID will scan.
}}

<div style="clear:both"></div>
{{Note|If you're using code generation software, or just want it for your own purposes, the swaggerfile describing an unstructured connector is available [https://developer.bigid.com/images/6/69/ConnectorGenericApiUnstructured.yaml here.]}} 
== Connector Endpoints ==

BigID will request and view your connector’s hierarchy through six endpoints:

* List what '''Containers''' exist (/objects)
* List what '''Objects''' exist (/objects/<containerName>?Offset=)
* Return the '''Metadata''' for a given object (GET /objects/<containerName>/<objectName>/describe)
* Return the '''InputStream''' for a given object (GET /objects/<containerName>/<objectName>/content-stream)
* '''Search''' for specific data inside objects (POST /objects/<containerName>/<objectName>/sar?Offset=)

Just like for structured connectors, you are not required to implement every endpoint outlined here. If you want a connector that only does DSAR requests you only need to implement the following endpoints:

* List what '''Containers''' exist (/objects)
* List what '''Objects''' exist (/objects/<containerName>?Offset=)
* '''Search''' for specific data inside objects (POST /objects/<containerName>/<objectName>/sar?Offset=)
{{Warning|Any additional endpoints added to your connector are of no interest to BigID. The BigID system is not concerned with the implementation details of your connector as long as HTTP requests to these endpoints succeed and are in the correct format}}

===List what '''containers''' exist===

The first step in BigID's scan of unstructured data sources will always be to list out the containers that exist within a data source. This allows BigID to know where to start the discovery process. Compared to structured connectors, this endpoint is extremely simple. That's because containers or folders don't really specify anything about the information that can be inside them, just that they are a storage location.

<syntaxhighlight lang="HTTP">GET /objects HTTP/1.1</syntaxhighlight>
<syntaxhighlight lang="JSON">
{
  "status": "success",
  "containers": [
    {
      "containerName": "Folder"
    },
    {
      "containerName": "Folder2"
    }
  ]
}</syntaxhighlight>

====Exercise: List what '''containers''' exist====
{{Scenario|Your team has a set of CSV files they are looking to scan on a file server. '''Implement the containers endpoint to let BigID know what containers are present on the file server'''}}
{{CodeTest|connector%2F6-containers}}

===List what '''objects''' exist inside a container===
Now that BigID knows what containers our connector has access to, we can give it insights into the objects inside those containers. This endpoint will also provide BigID with the metadata about the objects so we can do things like HyperScan and enforce permissions policies.

You'll notice that certain fields are set directly on the object, whereas others are set inside of custom fields. The field set in the object are predetermined by the BigID connector specification and cannot change. The custom fields object allows you to add your own!

{{Warning|sarCapable refers to whether an unstructured data source could be processed as structured data. This means BigID will ignore the actual file contents and only use structured fields defined here. '''It should not be set to true for any file you want to perform SARs on''', especially if you want BigID to classify the file content.}}

<syntaxhighlight lang="HTTP">GET /objects/Folder?Offset= HTTP/1.1</syntaxhighlight>
<syntaxhighlight lang="JSON">{
  "status": "success",
  "objects": [
    {
      "containerName": "Folder",
      "objectName": "1.pdf",
      "dateCreated": "26/02/2020 19:00:34",
      "lastModified": "26/02/2020 19:00:34",
      "owner": 501,
      "sizeInBytes": 177924,
      "schemaFields": "NULL",
      "sarCapable": false,
      "customFields": [
        {
          "fieldName": "file_type",
          "fieldValue": "pdf"
        }
      ]
    }
  ],
  "offset": 1
}</syntaxhighlight>
====Exercise: List what '''objects''' exist inside a container====
{{Scenario|List the objects inside of the container and their metadata so BigID can scan populate the catalog. '''Implement the objects endpoint to let BigID know what objects are present on the file server'''}}
{{CodeTest|connector%2F7-objects}}

===Return the '''metadata''' for a given object===

Just like we listed out the metadata for all objects, we need to be able to list out the metadata for a single object so BigID can inspect single objects without needing to go through potentially thousands of listings in our container.

<syntaxhighlight lang="HTTP">GET /objects/Folder/File/describe HTTP/1.1</syntaxhighlight>
<syntaxhighlight lang="JSON">{
  "status": "success",
  "containerName": "Folder",
  "objectName": "1.pdf",
  "dateCreated": "26/02/2020 19:00:34",
  "lastModified": "26/02/2020 19:00:34",
  "owner": 501,
  "sizeInBytes": 177924,
  "schemaFields": "NULL",
  "sarCapable": false,
  "customFields": [
    {
      "fieldName": "file_type",
      "fieldValue": "pdf"
    }
  ]
}</syntaxhighlight>
====Exercise: Return the '''metadata''' for a given object====
{{Scenario|We need our users to be able to know the metadata of a single file. The format is exactly the same as the container listing endpoint so feel free to reuse your code. '''Implement the object endpoint to let BigID know a single object's metadata'''}}
{{CodeTest|connector%2F8-object}}

===Return the '''inputStream''' for a given object===

In order for BigID to do classification on the contents of the file it needs access to them. The inputStream endpoint is unique in that it just returns the raw data from the file. There's no formatting required here, just return the raw data.

{{Note|What about binary data? BigID doesn't natively have the ability to process binary data from REST connectors. You'll need to use a library in your connector to convert the document to UTF8 text.}}

<syntaxhighlight lang="HTTP">GET /objects/Folder/File/content-stream HTTP/1.1

utf-8 data here</syntaxhighlight>

====Exercise: Return the '''inputStream''' for a given object====
{{CodeTest|connector%2F9-inputstream}}

==='''Search''' for specific data inside objects===
<syntaxhighlight lang="HTTP">POST /objects/Folder/File/sar?Offset= HTTP/1.1
Content-Type: application/json


[{"fieldName": "Name", "fieldValue": "Michael", "isFullMatch": "false"}]</syntaxhighlight>
<syntaxhighlight lang="JSON">{
  "status": "success",
  "records": [
    {
      "id": "container/object",
      "data": [
        {
          "searchedFieldName": "fieldName",
          "searchedFieldValue": "fieldValue",
          "fullObjectName": "container/object",
          "offset": "12"
        }
      ]
    }
  ]
}</syntaxhighlight>
===Exercise: '''Search''' for specific data inside objects===
{{CodeTest|connector%2F10-sar}}

=Distributing a BigID Connector=

<html>
<center>
<div class="mermaid">
        graph TD
            Cloud[BigID Cloud Environment]
            OnPrem[BigID On-Premise Environment]
            Connector[Connector]
            Scanner[Scanner]
            Target[Target Data Source]
            
            subgraph BigID Environment
                Cloud --- |Outbound Access| Scanner
                OnPrem --- |Outbound Access| Scanner
            end
            
            Scanner -->|Inbound Access| Connector
            Connector -->|Outbound Access| Target
</div>
</center>
</html>

BigID connectors are distributed as Docker images. This allows them to be run within the BigID app server Kubernetes cluster or on the Kubernetes clusters and Docker compose servers running an organization's scanners. As long as your scanner can access your connector and your connector can access your data source there are no other deployment requirements.

= Certification =
{{:Connectors/ConnectorDev_Certification}}