Pagination

Explores the different pagination mechanisms offered by the DAS API.

Introduction

The DAS API methods will only return up to 1000 records at a time. If you need to retrieve more than 1000 items, you will need to paginate the records. This is achieved by making multiple API calls and crawling through multiple "pages" of data. We support two mechanisms: page-based and keyset pagination.

We recommend page-based pagination for beginners. It is the simplest and best way to get started. Keyset pagination is recommended for advanced users that need to query across large (500k+) datasets efficiently.

Sorting Options

With the DAS API you can sort the data by different fields:

id: Sort by the asset ID, in binary (default).
created: Sort by the date the asset was created.
recent_action : Sort by the date the asset was last updated (not recommended).
none: Do not sort the data (not recommended).

Disabling the sorting will yield the fastest results, but since the data is not ordered you may have inconsistent results when paginating.

Pagination Options

Page-based

Recommended for beginners.

With this method, the user specifies the page number and the number of items they want per page. To iterate to the next page, simply increment the page number. This is easy, intuitive, and fast for most use-cases.

const url = `https://mainnet.helius-rpc.com/?api-key=<api_key>`

const example = async () => {
    let page = 1;
    let items = [];
    while (true) {
        const response = await fetch(url, {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
            },
            body: JSON.stringify({
                jsonrpc: '2.0',
                id: 'my-id',
                method: 'searchAssets',
                params: {
                    grouping: ['collection', '5PA96eCFHJSFPY9SWFeRJUHrpoNF5XZL6RrE1JADXhxf'],
                    page: page,
                    limit: 1000,
                    sortBy: { sortBy: 'id', sortDirection: 'asc' },
                },
            }),
        });
        const { result } = await response.json();
        if (result.items.length == 0) {
            console.log('No items remaining');
            break;
        } else {
            console.log(`Processing results from page ${page}`);
            items.push(...result.items);
            page += 1;
        }
    }
    console.log(`Got ${items.length} total items`);
};
example();

Using pages requires the database to crawl across all items until it reaches the next page. For example, if you ask for page 100 and page size 1000, the database needs to traverse across the first 1M records before returning your data.

For this reason, page-based pagination is not recommended for large datasets. Keyset pagination is far better suited for these types of workloads.

Keyset

With this method, the user defines pages by providing conditions that filter the data set. For example, you can say "Get me all assets with an ID > X but ID < Y". The user can traverse across the entire dataset by modifying X or Y on each call. We provide two methods of keyset pagination:

Cursor-based – Easier to use, but less flexible.
Range-based – More complex, but very flexible.

Keyset pagination is only supported when sorting by id.

Cursor-based

A DAS query without any pagination parameters will return a cursor. The cursor is a pointer that you can give to the DAS API to continue from where you left off.

const url = `https://mainnet.helius-rpc.com/?api-key=<api_key>`

const example = async () => {
    let items = [];
    let cursor;
    while (true) {
        let params = {
            grouping: ['collection', '5PA96eCFHJSFPY9SWFeRJUHrpoNF5XZL6RrE1JADXhxf'],
            limit: 1000,
            sortBy: { sortBy: 'id', sortDirection: 'asc' },
        } as any;
        if (cursor != undefined) {
            params.cursor = cursor;
        }
        const response = await fetch(url, {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
            },
            body: JSON.stringify({
                jsonrpc: '2.0',
                id: 'my-id',
                method: 'searchAssets',
                params: params,
            }),
        });
        const { result } = await response.json();
        if (result.items.length == 0) {
            console.log('No items remaining');
            break;
        } else {
            console.log(`Processing results for cursor ${cursor}`);
            cursor = result.cursor;
            items.push(...result.items);
        }
    }
    console.log(`Got ${items.length} total items`);
};
example();

At the time of writing, the cursor will be the last asset id of the response, however the cursor design is flexible and can support any string.

Range-based

To query across a range, you can specify before and/or after. The query is essentially identical to "get me all assets after X but before Y". You can traverse the dataset by updating either the before or after parameter on each call.

const url = `https://mainnet.helius-rpc.com/?api-key=<api_key>`

const example = async () => {
    // Two NFTs from the Tensorian collection.
    // The "start" item has a lower asset ID (in binary) than the "end" item.
    // We will traverse in ascending order.
    let start = '6CeKtAYX5USSvPCQicwFsvN4jQSHNxQuFrX2bimWrNey';
    let end = 'CzTP4fUbdfgKzwE6T94hsYV7NWf1SzuCCsmJ6RP1xsDw';
    let sortDirection = 'asc';
    let after = start;
    let before = end;
    let items = [];

    while (true) {
        const response = await fetch(url, {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
            },
            body: JSON.stringify({
                jsonrpc: '2.0',
                id: 'my-id',
                method: 'searchAssets',
                params: {
                    grouping: ['collection', '5PA96eCFHJSFPY9SWFeRJUHrpoNF5XZL6RrE1JADXhxf'],
                    limit: 1000,
                    after: after,
                    before: before,
                    sortBy: { sortBy: 'id', sortDirection: sortDirection },
                },
            }),
        });
        const { result } = await response.json();
        if (result.items.length == 0) {
            console.log('No items remaining');
            break;
        } else {
            console.log(`Processing results with (after: ${after}, before: ${before})`);
            after = result.items[result.items.length - 1].id;
            items.push(...result.items);
        }
    }
    console.log(`Got ${items.length} total items`);
};
example();

Parallel Querying with Keysets (advanced)

Advanced users that need to query large datasets (e.g. scan entire compressed NFT collections) must use keyset-based pagination for performance reasons. The following example shows how users can parallel query by partitioning the Solana address range and leveraging the before/after parameters. This method is fast, efficient, and safe. If you have any questions or need help, don't hesitate to reach out on Discord!

In the example below we scan the entire Tensorian collection (~10k records). It partitions the Solana address space into 8 ranges and scans those ranges concurrently. You'll notice that this example is FAR faster than any of the other examples.

import base58 from 'bs58';

const url = `https://mainnet.helius-rpc.com/?api-key=<api_key>`

const main = async () => {
    let numParitions = 8;
    let partitons = partitionAddressRange(numParitions);
    let promises = [];
    for (const [i, partition] of partitons.entries()) {
        let [s, e] = partition;
        let start = bs58.encode(s);
        let end = bs58.encode(e);
        console.log(`Parition: ${i}, Start: ${start}, End: ${end}`);

        let promise: Promise<number> = new Promise(async (resolve, reject) => {
            let current = start;
            let totalForPartition = 0;
            while (true) {
                const response = await fetch(url, {
                    method: 'POST',
                    headers: {
                        'Content-Type': 'application/json',
                    },
                    body: JSON.stringify({
                        jsonrpc: '2.0',
                        id: 'my-id',
                        method: 'searchAssets',
                        params: {
                            grouping: ['collection', '5PA96eCFHJSFPY9SWFeRJUHrpoNF5XZL6RrE1JADXhxf'],
                            limit: 1000,
                            after: current,
                            before: end,
                            sortBy: { sortBy: 'id', sortDirection: 'asc' },
                        },
                    }),
                });
                const { result } = await response.json();
                totalForPartition += result.items.length;
                console.log(`Found ${totalForPartition} total items in parition ${i}`);
                if (result.items.length == 0) {
                    break;
                } else {
                    current = result.items[result.items.length - 1].id;
                }
            }
            resolve(totalForPartition);
        });
        promises.push(promise);
    }
    let results = await Promise.all(promises);
    let total = results.reduce((a, b) => a + b, 0);
    console.log(`Got ${total} total items`);
};

// Function to convert a BigInt to a byte array
function bigIntToByteArray(bigInt: bigint): Uint8Array {
    const bytes = [];
    let remainder = bigInt;
    while (remainder > 0n) {
        // use 0n for bigint literal
        bytes.unshift(Number(remainder & 0xffn));
        remainder >>= 8n;
    }
    while (bytes.length < 32) bytes.unshift(0); // pad with zeros to get 32 bytes
    return new Uint8Array(bytes);
}

function partitionAddressRange(numPartitions: number) {
    let N = BigInt(numPartitions);

    // Largest and smallest Solana addresses in integer form.
    // Solana addresses are 32 byte arrays.
    const start = 0n;
    const end = 2n ** 256n - 1n;

    // Calculate the number of partitions and partition size
    const range = end - start;
    const partitionSize = range / N;

    // Calculate partition ranges
    const partitions: Uint8Array[][] = [];
    for (let i = 0n; i < N; i++) {
        const s = start + i * partitionSize;
        const e = i === N - 1n ? end : s + partitionSize;
        partitions.push([bigIntToByteArray(s), bigIntToByteArray(e)]);
    }

    return partitions;
}

main();

PreviousGet Nft Editions NextFungible Token Extension (Beta)

Last updated 7 months ago