Export Metadata Tutorial

In this article, you'll learn:

• What the BigID data catalog can be used for
• What is the structure of the BigID Catalog
• How to retrieve data from each part of the BigID Catalog Structure
• How to export the entire catalog using cursor-based pagination

Scenario

Your organization is using a third party cataloging tool for your data. Unfortunately there isn’t a prebuilt integration between BigID and this system relies on manual mapping and surveys. You want your data catalog to represent the real world, not what surveys say. Use the BigID API to synchronize what BigID sees within your systems with this third party cataloging tool.

Listing Every Object in the Catalog

Because you are synchronizing the whole catalog, start with /data-catalog/objects/stream. This endpoint streams a flat JSON array of catalog objects and pages with a cursor, so it can walk a catalog of any size.

Don't use skip/limit for a full export

You may also see GET /data-catalog, which returns { "results": [...], "totalRowsCounter": ... } and pages with skip and limit. On Elasticsearch-backed catalogs that endpoint is bound by the 10,000 result window: skip is clamped at 10,000 and limit shrinks to 10000 - skip. Paging past 10,000 objects returns zero rows instead of an error, so a skip-based export silently truncates. Use /data-catalog for interactive, bounded queries and for requireTotalCount; use /data-catalog/objects/stream for exports.

To page through the stream:

1. Send the first request with offsetKey= (an empty string) to start at the beginning.
2. Read nextOffsetKey from the last object in the returned array.
3. Pass that value as offsetKey on the next request.
4. Stop when the array is empty or holds fewer objects than your limit.

The sort field must be identical on every request, because the cursor is derived from that field’s value.

Note

Streaming from the catalog requires BigID v250 or later. On MongoDB-backed catalogs, cursor pagination also requires the feature flag CATALOG_TASK_OFFSET_STREAMING_FROM_MONGO_ENABLED=true. Elasticsearch-backed catalogs need no flag.

Drilling Into a Single Object

For unstructured data, we discover information directly about and inside the files. For example, we can see that there are phone numbers within a file, or that a file matches a machine learning model for an invoice.

For structured data we discover data about the data inside of the columns as well as the columns themselves. For example we would know that a database Users has a column named State and that column contains values of the type US State Abbreviation. This means that if we want information about specific columns we will need to perform an extra API request. The below API request shows how to retrieve the information about a specific table. The column parameter follows the format of Data Source Name.schema name.table name.

For unstructured data sources, you can use the information returned from the catalog stream or use the attributes call to get more detailed information like below:

By using these three API calls you can find out information about any element being scanned by your BigID system.

Code Samples

Python

# Metadata Export Tutorial
import requests
import json

base_url = "https://developer.bigid.com/api/v1"
headers = {
    "Authorization": "Bearer SAMPLE",
    "Content-Type": "application/json"
}

# The cursor is derived from the sort field, so it must not change between pages.
CATALOG_SORT = '[{"field":"fullyQualifiedName","order":"asc"}]'

# 1. Stream every object in the catalog, one page at a time
def stream_catalog_objects(page_size=1000, filter_expr=None):
    """Yield every catalog object using cursor-based pagination."""
    url = f"{base_url}/data-catalog/objects/stream"
    params = {
        "limit": page_size,
        "sort": CATALOG_SORT,
        "offsetKey": "",  # empty string starts at the beginning
    }
    if filter_expr:
        params["filter"] = filter_expr

    while True:
        res = requests.get(url, headers=headers, params=params, timeout=300)
        res.raise_for_status()
        page = res.json()  # a flat JSON array, not an object

        if not page:
            return
        yield from page
        if len(page) < page_size:
            return  # short page means we reached the end

        # The cursor for the next page lives on the last object returned.
        next_offset_key = page[-1].get("nextOffsetKey")
        if not next_offset_key:
            return
        params["offsetKey"] = next_offset_key

count = 0
for obj in stream_catalog_objects():
    count += 1
    print(obj["fullyQualifiedName"])
print(f"Exported {count} catalog objects")

# 2. Retrieve information about a specific table
url = f"{base_url}/data-catalog/object-details/columns?object_name=DataSourceName.schemaName.tableName" # Replace "DataSourceName.schemaName.tableName with desired table information"
res = requests.get(url, headers=headers)
data = res.json()
print(json.dumps(data, indent=2))

# 3. Retrieve information about unstructured data source
url = f"{base_url}/data-catalog/object-details/attributes?object_name=Sales%20Drive.devops%40bigiddemo.com%2FEducation%2FProspects4-Restricted.docx" # Example using attribute object_name to fetch detailed information
res = requests.get(url, headers=headers)
data = res.json()
print(json.dumps(data, indent=2))

JavaScript

// Metadata Export Tutorial
const baseUrl = "https://developer.bigid.com/api/v1";
const headers = {
  "Authorization": "Bearer SAMPLE", // Replace with your token
  "Content-Type": "application/json"
};

// The cursor is derived from the sort field, so it must not change between pages.
const CATALOG_SORT = '[{"field":"fullyQualifiedName","order":"asc"}]';

// 1. Stream every object in the catalog, one page at a time
async function* streamCatalogObjects({ pageSize = 1000, filter } = {}) {
  let offsetKey = ""; // empty string starts at the beginning

  while (true) {
    const params = new URLSearchParams({
      limit: String(pageSize),
      sort: CATALOG_SORT,
      offsetKey,
    });
    if (filter) params.set("filter", filter);

    const res = await fetch(`${baseUrl}/data-catalog/objects/stream?${params}`, { headers });
    if (!res.ok) throw new Error(`Catalog stream failed: ${res.status} ${res.statusText}`);

    const page = await res.json(); // a flat JSON array, not an object

    if (page.length === 0) return;
    yield* page;
    if (page.length < pageSize) return; // short page means we reached the end

    // The cursor for the next page lives on the last object returned.
    const { nextOffsetKey } = page[page.length - 1];
    if (!nextOffsetKey) return;
    offsetKey = nextOffsetKey;
  }
}

async function exportAllCatalogObjects() {
  let count = 0;
  for await (const obj of streamCatalogObjects()) {
    count++;
    console.log(obj.fullyQualifiedName);
  }
  console.log(`Exported ${count} catalog objects`);
  return count;
}

// 2. Retrieve information about a specific structured table
async function getTableMetadata() {
  const tableFQN = "DataSourceName.schemaName.tableName"; // Replace this with your actual table name
  const url = `${baseUrl}/data-catalog/object-details/columns?object_name=${encodeURIComponent(tableFQN)}`;
  console.log(`Fetching metadata for table: ${tableFQN}`);
  const res = await fetch(url, { headers });
  const data = await res.json();
  console.log("Structured Table Metadata:\n", JSON.stringify(data, null, 2));
  return data;
}

// 3. Retrieve info about an unstructured object
async function getUnstructuredAttributes() {
  const objectFQN = "Sales [email protected]/Education/Prospects4-Restricted.docx"; // Replace as needed
  const url = `${baseUrl}/data-catalog/object-details/attributes?object_name=${encodeURIComponent(objectFQN)}`;
  console.log(`Fetching metadata for unstructured object: ${objectFQN}`);
  const res = await fetch(url, { headers });
  const data = await res.json();
  console.log("Unstructured Object Metadata:\n", JSON.stringify(data, null, 2));
  return data;
}

// Run all 3 steps
(async () => {
  await exportAllCatalogObjects();
  await getTableMetadata();
  await getUnstructuredAttributes();
})();