Export Metadata Tutorial

Listing Every Object in the Catalog

Because you are synchronizing the whole catalog, start with /data-catalog/objects/stream. This endpoint streams a flat JSON array of catalog objects and pages with a cursor, so it can walk a catalog of any size.

To page through the stream:

Send the first request with offsetKey= (an empty string) to start at the beginning.
Read nextOffsetKey from the last object in the returned array.
Pass that value as offsetKey on the next request.
Stop when the array is empty or holds fewer objects than your limit.

The sort field must be identical on every request, because the cursor is derived from that field’s value.

Drilling Into a Single Object

For unstructured data, we discover information directly about and inside the files. For example, we can see that there are phone numbers within a file, or that a file matches a machine learning model for an invoice.

For structured data we discover data about the data inside of the columns as well as the columns themselves. For example we would know that a database Users has a column named State and that column contains values of the type US State Abbreviation. This means that if we want information about specific columns we will need to perform an extra API request. The below API request shows how to retrieve the information about a specific table. The column parameter follows the format of Data Source Name.schema name.table name.

For unstructured data sources, you can use the information returned from the catalog stream or use the attributes call to get more detailed information like below:

By using these three API calls you can find out information about any element being scanned by your BigID system.

Code Samples

Python

# Metadata Export Tutorial
import requests
import json

base_url = "https://developer.bigid.com/api/v1"
headers = {
    "Authorization": "Bearer SAMPLE",
    "Content-Type": "application/json"
}

# The cursor is derived from the sort field, so it must not change between pages.
CATALOG_SORT = '[{"field":"fullyQualifiedName","order":"asc"}]'

# 1. Stream every object in the catalog, one page at a time
def stream_catalog_objects(page_size=1000, filter_expr=None):
    """Yield every catalog object using cursor-based pagination."""
    url = f"{base_url}/data-catalog/objects/stream"
    params = {
        "limit": page_size,
        "sort": CATALOG_SORT,
        "offsetKey": "",  # empty string starts at the beginning
    }
    if filter_expr:
        params["filter"] = filter_expr

    while True:
        res = requests.get(url, headers=headers, params=params, timeout=300)
        res.raise_for_status()
        page = res.json()  # a flat JSON array, not an object

        if not page:
            return
        yield from page
        if len(page) < page_size:
            return  # short page means we reached the end

        # The cursor for the next page lives on the last object returned.
        next_offset_key = page[-1].get("nextOffsetKey")
        if not next_offset_key:
            return
        params["offsetKey"] = next_offset_key

count = 0
for obj in stream_catalog_objects():
    count += 1
    print(obj["fullyQualifiedName"])
print(f"Exported {count} catalog objects")

# 2. Retrieve information about a specific table
url = f"{base_url}/data-catalog/object-details/columns?object_name=DataSourceName.schemaName.tableName" # Replace "DataSourceName.schemaName.tableName with desired table information"
res = requests.get(url, headers=headers)
data = res.json()
print(json.dumps(data, indent=2))

# 3. Retrieve information about unstructured data source
url = f"{base_url}/data-catalog/object-details/attributes?object_name=Sales%20Drive.devops%40bigiddemo.com%2FEducation%2FProspects4-Restricted.docx" # Example using attribute object_name to fetch detailed information
res = requests.get(url, headers=headers)
data = res.json()
print(json.dumps(data, indent=2))

JavaScript

// Metadata Export Tutorial
const baseUrl = "https://developer.bigid.com/api/v1";
const headers = {
  "Authorization": "Bearer SAMPLE", // Replace with your token
  "Content-Type": "application/json"
};

// The cursor is derived from the sort field, so it must not change between pages.
const CATALOG_SORT = '[{"field":"fullyQualifiedName","order":"asc"}]';

// 1. Stream every object in the catalog, one page at a time
async function* streamCatalogObjects({ pageSize = 1000, filter } = {}) {
  let offsetKey = ""; // empty string starts at the beginning

  while (true) {
    const params = new URLSearchParams({
      limit: String(pageSize),
      sort: CATALOG_SORT,
      offsetKey,
    });
    if (filter) params.set("filter", filter);

    const res = await fetch(`${baseUrl}/data-catalog/objects/stream?${params}`, { headers });
    if (!res.ok) throw new Error(`Catalog stream failed: ${res.status} ${res.statusText}`);

    const page = await res.json(); // a flat JSON array, not an object

    if (page.length === 0) return;
    yield* page;
    if (page.length < pageSize) return; // short page means we reached the end

    // The cursor for the next page lives on the last object returned.
    const { nextOffsetKey } = page[page.length - 1];
    if (!nextOffsetKey) return;
    offsetKey = nextOffsetKey;
  }
}

async function exportAllCatalogObjects() {
  let count = 0;
  for await (const obj of streamCatalogObjects()) {
    count++;
    console.log(obj.fullyQualifiedName);
  }
  console.log(`Exported ${count} catalog objects`);
  return count;
}

// 2. Retrieve information about a specific structured table
async function getTableMetadata() {
  const tableFQN = "DataSourceName.schemaName.tableName"; // Replace this with your actual table name
  const url = `${baseUrl}/data-catalog/object-details/columns?object_name=${encodeURIComponent(tableFQN)}`;
  console.log(`Fetching metadata for table: ${tableFQN}`);
  const res = await fetch(url, { headers });
  const data = await res.json();
  console.log("Structured Table Metadata:\n", JSON.stringify(data, null, 2));
  return data;
}

// 3. Retrieve info about an unstructured object
async function getUnstructuredAttributes() {
  const objectFQN = "Sales [email protected]/Education/Prospects4-Restricted.docx"; // Replace as needed
  const url = `${baseUrl}/data-catalog/object-details/attributes?object_name=${encodeURIComponent(objectFQN)}`;
  console.log(`Fetching metadata for unstructured object: ${objectFQN}`);
  const res = await fetch(url, { headers });
  const data = await res.json();
  console.log("Unstructured Object Metadata:\n", JSON.stringify(data, null, 2));
  return data;
}

// Run all 3 steps
(async () => {
  await exportAllCatalogObjects();
  await getTableMetadata();
  await getUnstructuredAttributes();
})();

Terms Privacy Notice Cookies Your Privacy Choices Legal Resources Certifications Conduct & Ethics Sub-processors