BigID API/DSAR Tutorial: Difference between revisions

From BigID Developer Portal
 
(26 intermediate revisions by the same user not shown)
Line 14: Line 14:
== Getting DSAR Profiles ==
== Getting DSAR Profiles ==


BigID uses DSAR profiles to specify which databases to look for users. You can create these using the APIs, but creating them via the UI is preferred since the UI will provide suggestions as you work. In our case, we already have a few data sources within our system.
BigID uses DSAR profiles to specify what data sources we should use to look for user data. You can create these using the APIs, but creating them via the UI is preferred since the UI will provide suggestions as you work. In our case, we already have a few profiles within our system so we can use an already created profile.
 
Add a new header named "Authorization" and paste the session token you got in the previous request to authenticate yourself.


<html>
<html>
<iframe style="border:0px; width:100%; height:400px; border-radius:10px;" src="https://apibrowser.mybigid.com/?url=ds-connections&method=GET&selectedSetting=headers"></iframe>
<iframe style="border:0px; width:100%; height:400px; border-radius:10px;" src='https://apiexplorer.bigid.tools/?url=sar/profiles&method=GET&selectedSetting=none&headers=%5B%7B%22name%22%3A%22Authorization%22%2C%22value%22%3A%22SAMPLE%22%7D%5D'></iframe>
</html>
</html>


In that API call, we can see a list of data sources and all the information for each data source.  
From this API call we can see a list of DSAR profiles. This also gives us insight into why organizations use DASR profiles. Different groups of systems can have users with the same unique ID (employee number 1 and customer number 1 are probably different people). Profiles allow us to segment those user groups. Below we see a different profile for US customers to illustrate that.


<syntaxhighlight lang="JSON" lines>
<syntaxhighlight lang="JSON" lines>
{
{
     "status": "success",
  "profiles": [
     "statusCode": 200,
     {
    "data": {
      "_id": "5d93e8431810782ce9173ae0",
        "ds_connections": [
      "name": "Default Profile",
            "<data source info here>"
      "allEnabledEs": true,
        ]
      "allEnabledDs": true,
      "scopes": [
        "root"
      ],
      "isCustom": false
    },
     {
      "_id": "614d7794dfa3fdf8bd71cacb",
      "allEnabledDs": true,
      "allEnabledEs": false,
      "name": "ALL PI for US Customer",
      "scopes": [
        "root"
      ],
      "shouldAttributesEnrichment": true,
      "isCustom": true
     }
     }
  ]
}
}
</syntaxhighlight>
</syntaxhighlight>


We now know the API calls we need and can use our programming language of choice to prepare our report. Below are some samples.
In our case, we know a user with the email [email protected] is present in the default profile. However, we need to figure out what attributes we can use to execute a DSAR with this profile so we get the names correct.
 
<html>
<iframe style="border:0px; width:100%; height:400px; border-radius:10px;" src="https://apiexplorer.bigid.tools/?url=sar%2Fattributes%3FprofileId%3D5d93e8431810782ce9173ae0&method=GET&selectedSetting=none&headers=%5B%7B%22name%22%3A%22Authorization%22%2C%22value%22%3A%22SAMPLE%22%7D%5D"></iframe>
</html>


<tabs>
We can see that there's an idsor_attribute named Email that has an identifiability of 1. This means that 100% of the sampled users have a different Email which is perfect for our DSAR. Let's initiate a DSAR on this user using that Email attribute.
<tab name="NodeJS">
<syntaxhighlight lang="Javascript" lines>
import fetch from 'node-fetch';


let credentials = { username: "bigid", password: "bigid111" };
== Initiating a DSAR ==
let env = "https://sandbox.mybigid.com/";


async function getDataSources(credentials, env) {
Because the amount of data scanned as part of a DSAR can be well into the petabytes, the DSAR API is asynchronous. The below request will start a DSAR and return immediately with an ID you can use to track its progress. Execute the below request to start your DSAR.
    // Request API Key using user/pass authentication
    const sessionRequest = fetch(env + 'api/v1/sessions', {
        method: 'POST',
        body: JSON.stringify(credentials),
        headers: { 'Content-Type': 'application/json' }
    });
    const sessionData = await request.json();


    const dsRequest = fetch(env + 'api/v1/ds-connections', {
<html>
         method: 'GET',
<iframe style="border:0px; width:100%; height:400px; border-radius:10px;" src="https://apiexplorer.bigid.tools/?url=sar/reports&method=POST&selectedSetting=body&headers=%5B%7B%22name%22%3A%22Authorization%22%2C%22value%22%3A%22SAMPLE%22%7D%5D&body=%7B%22userDetails%22%3A%7B%22attributes%22%3A%7B%22Email%22%3A%22alex.bulis%40gmail.com%22%7D%2C%22userId%22%3A%22DSAR%22%2C%22displayName%22%3A%22DSAR%22%7D%2C%22profileId%22%3A%225d93e8431810782ce9173ae0%22%7D"></iframe>
         headers: {
</html>
             'Content-Type': 'application/json',
 
             'Authorization': sessionData.auth_token
== Checking DSAR Status ==
Let's check the status of your DSAR. Replace REQUESTID in the URL of the below request with the requestId you received  above when you initiated the DSAR.
 
<html>
<iframe style="border:0px; width:100%; height:400px; border-radius:10px;" src='https://apiexplorer.bigid.tools/?url=sar/reports/REQUESTID/status&method=GET&selectedSetting=none&headers=%5B%7B%22name%22%3A%22Authorization%22%2C%22value%22%3A%22SAMPLE%22%7D%5D'></iframe>
</html>
 
When checking the status, we can see the state of our DSAR overall as well as each individual data source. The sample below has 17/18 data sources in the scan completed so the overall state is still "Started". '''Rerun the above request until you have a state of "Complete".'''
 
<syntaxhighlight lang="JSON" lines highlight="7">
{
    "status": "success",
    "statusCode": 200,
    "data": {
         "requestId": "61784c9b1a1d3e0ea3b21a53",
        "userId": "DSAR",
        "state": "Started",
         "statuses": {
             "Completed": 17,
            "Queued": 1,
             "InProgress": 1
         }
         }
     });
        ...
     return await dsRequest.json();
     },
}
     "message": null
</syntaxhighlight>
}</syntaxhighlight>
</tab>
 
<tab name="Javascript">
== Retrieving DSAR Results ==
<syntaxhighlight lang="javascript" lines>
 
let credentials = { username: "bigid", password: "bigid111" };
After your DSAR has completed, there's a variety of ways we can get and display the results. For a full view of the options, visit [https://www.docs.bigid.com/bigid/reference/reports-2#get_api-v1-sar-reports-requestid the BigID Docs]. Some common reports are:
let env = "https://sandbox.mybigid.com/";
* Full report - Includes internal information like what data source information was found in and column names. Can be JSON or CSV.
* Short Report - Includes attribute names and values. Respects profile settings for hiding columns. Useful for returning to consumers. Can be in JSON, CSV, or a PDF.
* Personal Info report - PDF report with information, consent records, purposes of processing, and data source information. Formatted as a PDF. Useful as a pretty report for consumers and is an expanded version of the short report.


async function getDataSources(credentials, env) {
Since we want to automate what we would return to consumers, we're going to retrieve the short report in JSON. You can also replace json in the URL with csv to change the format. '''Replace REQUESTID with the requestId of your DSAR that can be found above.'''
    // Request API Key using user/pass authentication
    const sessionRequest = window.fetch(env + 'api/v1/sessions', {
        method: 'POST',
        body: JSON.stringify(credentials),
        headers: { 'Content-Type': 'application/json' }
    });
    const sessionData = await request.json();


    const dsRequest = window.fetch(env + 'api/v1/ds-connections', {
<html>
        method: 'GET',
<iframe style="border:0px; width:100%; height:400px; border-radius:10px;" src='https://apiexplorer.bigid.tools/?url=sar/reports/REQUESTID/short-report%3Fformat%3Djson&method=GET&selectedSetting=none&headers=%5B%7B%22name%22%3A%22Authorization%22%2C%22value%22%3A%22SAMPLE%22%7D%5D'></iframe>
        headers: {
</html>
            'Content-Type': 'application/json',
            'Authorization': sessionData.auth_token
        }
    });
    return await dsRequest.json();
}
</syntaxhighlight>
</tab>
<tab name="Python">
<syntaxhighlight lang="python" lines>
import requests


credentials = {'username':'bigid', 'password':'bigid111'}
Now that we know the required APIs to perform a DSAR we can add this into our account portal and give users self service access to their data.
env = 'https://sandbox.mybigid.com/'


def getDataSources(credentials, env):
[[Category:Tutorial]][[Category:API]]
    sessionRequest = requests.post(env+'api/v1/sessions', data = credentials)
    sessionData = sessionRequest.json()
    dsRequest = requests.get(env+'api/v1/ds-connections', headers = {'Authorization':sessionData.get('auth_token')})
    return dsRequest.json()
</syntaxhighlight>
</tab>
</tabs>

Latest revision as of 22:49, 9 March 2022

In this article, you'll learn:

  • Find a DSAR profile using the BigID API
  • Search for individuals using attributes using the BigID API
  • Run a DSAR scan using the BigID API
  • Get a DSAR report using the BigID API


scenarioYou already have your own account management portal that also manages other tasks for your business. You already have BigID for your security and data governance teams and they've mentioned that it can perform DSARs too. Run the DSAR calls required to integrate your management portal with BigID's DSAR capabilities

In this tutorial, we'll use SAMPLE as our session token. This is unique to the training sandbox and will not work in other environments. See BigID API/Tutorial for information on authenticating with BigID.

Getting DSAR Profiles[edit]

BigID uses DSAR profiles to specify what data sources we should use to look for user data. You can create these using the APIs, but creating them via the UI is preferred since the UI will provide suggestions as you work. In our case, we already have a few profiles within our system so we can use an already created profile.

From this API call we can see a list of DSAR profiles. This also gives us insight into why organizations use DASR profiles. Different groups of systems can have users with the same unique ID (employee number 1 and customer number 1 are probably different people). Profiles allow us to segment those user groups. Below we see a different profile for US customers to illustrate that.

{
  "profiles": [
    {
      "_id": "5d93e8431810782ce9173ae0",
      "name": "Default Profile",
      "allEnabledEs": true,
      "allEnabledDs": true,
      "scopes": [
        "root"
      ],
      "isCustom": false
    },
    {
      "_id": "614d7794dfa3fdf8bd71cacb",
      "allEnabledDs": true,
      "allEnabledEs": false,
      "name": "ALL PI for US Customer",
      "scopes": [
        "root"
      ],
      "shouldAttributesEnrichment": true,
      "isCustom": true
    }
  ]
}

In our case, we know a user with the email [email protected] is present in the default profile. However, we need to figure out what attributes we can use to execute a DSAR with this profile so we get the names correct.

We can see that there's an idsor_attribute named Email that has an identifiability of 1. This means that 100% of the sampled users have a different Email which is perfect for our DSAR. Let's initiate a DSAR on this user using that Email attribute.

Initiating a DSAR[edit]

Because the amount of data scanned as part of a DSAR can be well into the petabytes, the DSAR API is asynchronous. The below request will start a DSAR and return immediately with an ID you can use to track its progress. Execute the below request to start your DSAR.

Checking DSAR Status[edit]

Let's check the status of your DSAR. Replace REQUESTID in the URL of the below request with the requestId you received above when you initiated the DSAR.

When checking the status, we can see the state of our DSAR overall as well as each individual data source. The sample below has 17/18 data sources in the scan completed so the overall state is still "Started". Rerun the above request until you have a state of "Complete".

{
    "status": "success",
    "statusCode": 200,
    "data": {
        "requestId": "61784c9b1a1d3e0ea3b21a53",
        "userId": "DSAR",
        "state": "Started",
        "statuses": {
            "Completed": 17,
            "Queued": 1,
            "InProgress": 1
        }
        ...
    },
    "message": null
}

Retrieving DSAR Results[edit]

After your DSAR has completed, there's a variety of ways we can get and display the results. For a full view of the options, visit the BigID Docs. Some common reports are:

  • Full report - Includes internal information like what data source information was found in and column names. Can be JSON or CSV.
  • Short Report - Includes attribute names and values. Respects profile settings for hiding columns. Useful for returning to consumers. Can be in JSON, CSV, or a PDF.
  • Personal Info report - PDF report with information, consent records, purposes of processing, and data source information. Formatted as a PDF. Useful as a pretty report for consumers and is an expanded version of the short report.

Since we want to automate what we would return to consumers, we're going to retrieve the short report in JSON. You can also replace json in the URL with csv to change the format. Replace REQUESTID with the requestId of your DSAR that can be found above.

Now that we know the required APIs to perform a DSAR we can add this into our account portal and give users self service access to their data.