Find Duplicate Data Tutorial

In this article, you'll learn:

• What the BigID data catalog can be used for
• How to use the has_duplicates filter
• How to use the duplicate_id filter
• When to stream catalog objects instead of paging through them

Scenario

You’re seeing increasingly high storage costs in your cloud data sources. Looking at the names of these data sources, they don’t seem to be storing anything that’s particularly large, but you suspect that they’re storing similar data which is increasing your storage costs. Use the BigID catalog to get a list of duplicated data

This means that both structured and unstructured data sources can have multiple objects within them. In a production BigID system there can be millions of objects so we need to filter.

Note

If you ever want to request information about objects with specific properties think: What filter should I use?

Picking the right endpoint

Two endpoints return catalog objects, and the choice matters once your result set is large:

• GET /data-catalog/objects/stream streams a flat JSON array and pages with an offsetKey cursor. Use it whenever the result set is unbounded, such as sweeping every object that has duplicates.
• GET /data-catalog returns { "results": [...], "totalRowsCounter": ... } and pages with skip and limit. On Elasticsearch-backed catalogs it is capped at the 10,000 result window: skip is clamped to 10,000 and limit shrinks to 10000 - skip, so paging past 10,000 objects returns zero rows rather than an error. Keep it for bounded lookups, like fetching the handful of objects that share one duplicate_id.

The filter syntax is identical on both, so a filter you try in the explorer below works unchanged against the stream.

Streaming requires BigID v250 or later. On MongoDB-backed catalogs, cursor pagination also requires the feature flag CATALOG_TASK_OFFSET_STREAMING_FROM_MONGO_ENABLED=true; Elasticsearch-backed catalogs need no flag.

Using Duplicate Filters

Let’s use the has_duplicates filter to request objects that contain duplicate files. Because any number of objects in your estate could carry duplicates, this sweep runs against the streaming endpoint.

The stream returns a flat JSON array rather than a results wrapper, and it returns far fewer objects than an unfiltered sweep of the catalog. But what are the duplicates? Each duplicate object has a duplicate_id that represents a hash of the file. We can filter objects by this ID to find all the duplicates. Since one hash only ever matches a small number of files, that lookup is bounded, so the paginated /data-catalog endpoint is a good fit. Replace DUPLICATEID in the URL of the request below with the duplicate_id of the first object above to find its duplicates.

Now you have a list of the files that are duplicated, you can delete some of your unneeded copies to save on data storage costs.

Code Samples

Python

# Duplicate Data Tutorial
import requests
import json

base_url = "https://developer.bigid.com/api/v1"
headers = {
    "Authorization": "Bearer SAMPLE",
    "Content-Type": "application/json"
}

# The cursor is derived from the sort field, so it must not change between pages.
CATALOG_SORT = '[{"field":"fullyQualifiedName","order":"asc"}]'

def stream_catalog_objects(page_size=1000, filter_expr=None):
    """Yield catalog objects via the streaming endpoint, following the cursor."""
    url = f"{base_url}/data-catalog/objects/stream"
    params = {
        "limit": page_size,
        "sort": CATALOG_SORT,
        "offsetKey": "",  # empty string starts at the beginning
    }
    if filter_expr:
        params["filter"] = filter_expr

    while True:
        res = requests.get(url, headers=headers, params=params, timeout=300)
        res.raise_for_status()
        page = res.json()  # a flat JSON array, not an object

        if not page:
            return
        yield from page
        if len(page) < page_size:
            return  # short page means we reached the end

        # The cursor for the next page lives on the last object returned.
        next_offset_key = page[-1].get("nextOffsetKey")
        if not next_offset_key:
            return
        params["offsetKey"] = next_offset_key

# 1. Walk every catalog object
total = sum(1 for _ in stream_catalog_objects())
print("Total objects in catalog:", total)

# 2. Stream only the objects that have duplicates
duplicates = list(stream_catalog_objects(filter_expr='has_duplicates="true"'))
print("Objects with duplicates:", len(duplicates))

# Get the duplicate_id of the first object (for example)
duplicate_id = duplicates[0].get("duplicate_id")

# 3. Get all objects that share the same duplicate_id.
#    This is a small, bounded lookup, so the paginated endpoint is a good fit.
response = requests.get(
    f"{base_url}/data-catalog",
    headers=headers,
    params={"filter": f'duplicate_id="{duplicate_id}"'}
)
data = response.json()
print("Objects with same duplicate_id:", json.dumps(data, indent=2))

JavaScript

// Duplicate Data Tutorial
const baseUrl = "https://developer.bigid.com/api/v1";
const headers = {
  "Authorization": "Bearer SAMPLE",
  "Content-Type": "application/json"
};

// The cursor is derived from the sort field, so it must not change between pages.
const CATALOG_SORT = '[{"field":"fullyQualifiedName","order":"asc"}]';

// Yield catalog objects via the streaming endpoint, following the cursor.
async function* streamCatalogObjects({ pageSize = 1000, filter } = {}) {
  let offsetKey = ""; // empty string starts at the beginning

  while (true) {
    const params = new URLSearchParams({
      limit: String(pageSize),
      sort: CATALOG_SORT,
      offsetKey,
    });
    if (filter) params.set("filter", filter);

    const res = await fetch(`${baseUrl}/data-catalog/objects/stream?${params}`, { headers });
    if (!res.ok) throw new Error(`Catalog stream failed: ${res.status} ${res.statusText}`);

    const page = await res.json(); // a flat JSON array, not an object

    if (page.length === 0) return;
    yield* page;
    if (page.length < pageSize) return; // short page means we reached the end

    // The cursor for the next page lives on the last object returned.
    const { nextOffsetKey } = page[page.length - 1];
    if (!nextOffsetKey) return;
    offsetKey = nextOffsetKey;
  }
}

// 1. Walk every catalog object
async function countAllCatalogObjects() {
  let total = 0;
  for await (const _ of streamCatalogObjects()) total++;
  console.log("Total objects in catalog:", total);
  return total;
}

// 2. Stream only the objects that have duplicates
async function getObjectsWithDuplicates() {
  const duplicates = [];
  for await (const obj of streamCatalogObjects({ filter: 'has_duplicates="true"' })) {
    duplicates.push(obj);
  }
  console.log("Objects with duplicates:", duplicates.length);
  return duplicates;
}

// 3. Get all objects that share the same duplicate_id.
//    This is a small, bounded lookup, so the paginated endpoint is a good fit.
async function getObjectsByDuplicateId(duplicateId) { // Use duplicate id of desired object obtained above in step 2
  console.log(`Fetching objects for duplicate_id: ${duplicateId}`);
  const params = new URLSearchParams({ filter: `duplicate_id="${duplicateId}"` });
  const res = await fetch(`${baseUrl}/data-catalog?${params}`, { headers });
  const data = await res.json();
  console.log("Objects with same duplicate_id:", JSON.stringify(data, null, 2));
  return data;
}

// Run all 3 steps
(async () => {
  await countAllCatalogObjects();
  const duplicates = await getObjectsWithDuplicates();
  if (duplicates.length > 0) {
    await getObjectsByDuplicateId(duplicates[0].duplicate_id);
  }
})();