Checkpoint is a library for indexing data from Starknet events and making it accessible through GraphQL. Checkpoint is inspired by The Graph and focused on providing similar functionality for Starknet.

How it works

This diagram gives more detail about the flow of data once a Checkpoint instance has started processing Starknet transactions:

As highlighted in the flow diagram above:

1. A decentralized application adds data to Starknet through a transaction on a smart contract.

2. The smart contract emits one or more events while processing the transaction. Checkpoint continually scans Starknet for new blocks and checks if it contains events for your configured contracts they may contain.

4. The decentralized application queries the Checkpoint GraphQL API. Checkpoint translates the GraphQL queries into SQL queries to fetch this entity data from the database. The decentralized application displays this data in a rich UI for end-users, which they can also use to issue new transactions on Starknet, and the cycle repeats.

Installation

Checkpoint is an NPM package that can be installed through the following command:

npm install @snapshot-labs/checkpoint@beta

Guides: jump right in

Follow our handy guides to get started with Checkpoint as quickly as possible:

Core concepts: dive a little deeper

Learn the fundamentals of Checkpoint to get a deeper understanding of how it works:

Checkpoint's time travel queries feature allows to access historical data at a specific block number. By specifying a block, you can retrieve data from that exact point in the past.

{
  proposals (block: 123456) {
    id
    start
  }
}

In this guide, we will set up a project that uses Checkpoint to expose data.

Setup the starter template project

To successfully, follow this guide and run the template project, you'll need the following dependencies set up on your computer:

• Node.js (>=14.x.x)

• Yarn

• PostgreSQL (or Docker to quickly run an image).

1. Setup dependencies

First, install Node dependencies by running the following command in the project's directory:

yarn

2. Run project

Checkpoint projects (and by extension this template) require a PostgreSQL database connection to store indexed data. If you have a PostgreSQL server running, then create a copy of the .env.example file and name .env. Then update the DATABASE_URL value in the .env file to match the connection string to your database.

docker run --name checkpoint-postgres \
    -p "5432:5432" \
    -e POSTGRES_PASSWORD=default_password -d postgres

Next, start the server by running:

yarn dev

This will start the indexing process and also startup the GraphQL server for querying indexed information.

Querying the API data

Now that the server is running, open up your browser and visit http://localhost:3000. This will show a GraphiQL interface to explore the exposed data.

Try executing the following query:

query {
  posts {
    id
    author
    content
    tag
    created_at_block
    created_at
    tx_hash
  }
}

You should get a response similar to the one in this image:

The response to your queries depends on how much on-chain data Checkpoint has indexed. If the server has run long enough, you'll get more results. Checkpoint exposes a _metadata query, which can be used to track the latest block it has indexed. Example of this is:

query {
  _metadata(id: "last_indexed_block") {
    value
  }
}

Template project structure

Here is a quick description of important files within the template project and what each does.

├── src
│   ├── checkpoints.json
│   ├── config.ts
│   ├── index.ts
│   ├── schema.gql
│   ├── utils.ts
│   └── writers.ts
...

You are encouraged to try modifying the template project to understand how Checkpoint works and quickly get started using it to index your contracts data on Starknet.

It's possible to create one-to-one relations in schema. Those relations will be then available for querying via GraphQL.

Define relations in schema

To define one-to-one relation nested entity must have id field that is either String! or ID!. In parent entity create field (with any name) that is of nested entity's type (can be nullable or required).

type Space {
  id: String!
  name: String
}

type Proposal {
  id: String!
  name: String
  space: Space!
}

Create entities in writer

When creating entities in writer set Proposal's space field to the value of Space's id field.

const proposal = new Proposal('proposal_id');
proposal.name = 'Proposal name';
proposal.space = 'Space_id';

await proposal.save();

Query data via GraphQL

It's now possible to nest space entity when querying proposals. You can still filter proposals by space (limited to Space's id currently).

{
  proposals(first: 10, where: { space: "0x00b60f2a154b9aaec8e4bec8e04f86d6cd92a9c993871e904bd815962603492d" }) {
    id
    space {
      id
      name
    }
  }
}

In many cases there are contracts that should be tracked but their addresses are not known right away so they can't be configured in static config. Templates make it possible to define templates for such contracts that can be dynamically turned into sources whenever writer becomes aware of new contract's existence - for example when new contract has been deployed through factory.

Defining templates

Sample config might look like this:

const config = {
  network_node_url: 'RPC_NODE',
  sources: [
    {
      contract: '0x2121c175922cef8870b6b6b956d01a7ac75af034dfbb9135698f88644b17ea3',
      start: 345862,
      events: [
        {
          name: 'space_deployed',
          fn: 'handleSpaceCreated'
        }
      ]
    }
  ],
  templates: {
    Space: {
      events: [
        {
          name: 'proposal_created',
          fn: 'handlePropose'
        },
        {
          name: 'vote_created',
          fn: 'handleVote'
        }
      ]
    }
  }
};

Executing templates

After template is defined it can be executed in any writer to start tracking specific contract address for events defined in that template:

const handleSpaceCreated: starknet.Writer = async ({ block, helpers }) => {
  await helpers.executeTemplate('Space', {
    contract: 'some_new_contract_address',
    start: block.block_number
  });
}

Once template has been executed all events defined in that template will be tracked for given contract address.

Checkpoint initially supported indexing Starknet contracts only, but it can also index Ethereum contracts now.

Usage with Ethereum is very similar to usage with Starknet, differences are:

• Use full signature for event name in config (ProposalUpdated(uint256,(address,bytes),string) instead of ProposalUpdated)

• Your writers should be using evm.Writer instead of starknet.Writer.

• You should create indexer using new evm.EvmIndexer instead of new starknet.StarknetIndexer.

Checkpoint provides support for BigInt, Decimal, BigDecimal. It's also possible to define custom (or define existing) decimal types.

PostgreSQL mappings

BigInt and BigDecimal types are mapped to following PostgreSQL types by default:

• BigInt -> bigint

• Decimal -> decimal(10, 2)

• BigDecimal -> decimal(20, 8)

Using BigInt and BigDecimal types

To use default (or custom) BigInt and BigDecimal type it needs to be predeclared in GraphQL schema using scalar keyword. Once it's declared it can be used as field's type.

scalar BigInt

type User {
  votes_count: BigInt
}  

Custom decimal types

It's possible to define new or to redefine existing decimal types using overrides. This file should be called overrides.json and stored in your source directory.

{
  "decimal_types": {
    "BigDecimalVP": {
      "p": 60,
      "d": 0
    }
  }
}

You can then pass those to Checkpoint via options:

import overridesConfig from './overrides.json';

// ...

const checkpoint = new Checkpoint(schema, {
  overridesConfig
});

Checkpoint is a Node.js library and running the following NPM commands will install it:

npm install @snapshot-labs/checkpoint@beta

# or if using yarn
yarn add @snapshot-labs/checkpoint@beta

Two arguments can be used to initialize a Checkpoint instance. These are:

• GraphQL schema (required)

Creating a Checkpoint Configuration

• Mainnet: 0x0654e9232d5f402829755029901f69c32b423ded0f8c081e416e3b24f5a7a46e

• Sepolia: 0x03aa7630a4f9c5108bf3cd1910c7d45404cba865fc0fc0756bf9eedc073a98a9

With new versions of Checkpoint contracts on different networks can be indexed and queried from single Checkpoint instance.

import { CheckpointConfig } from "@snapshot-labs/checkpoint";

export const mainnetConfig: CheckpointConfig = {
  network_node_url: "https://starknet-mainnet.infura.io/v3/46a5dd9727bf48d4a132672d3f376146",
  sources: [
    {
      contract: "0x0654e9232d5f402829755029901f69c32b423ded0f8c081e416e3b24f5a7a46e",
      start: 639485,
      events: [
        {
          name: "new_post",
          fn: "handleNewPost",
        }
      ],
    }
  ]
};

export const sepoliaConfig: CheckpointConfig = {
  network_node_url: "https://starknet-sepolia.infura.io/v3/46a5dd9727bf48d4a132672d3f376146",
  sources: [
    {
      contract: "0x03aa7630a4f9c5108bf3cd1910c7d45404cba865fc0fc0756bf9eedc073a98a9",
      start: 65137,
      events: [
        {
          name: "new_post",
          fn: "handleNewPost",
        }
      ],
    }
  ]
};

In this case we want to index Poster contract on both Starknet mainnet and Starknet sepolia so we define two configurations.

The start block number is set to 639485 because our mainnet contract was deployed at that block. This will mean Checkpoint starts scanning from that block as opposed to starting at block 0.

Defining GraphQL entity schemas

Checkpoint requires a set of defined GraphQL Schema Objects. These schema objects will be used to create the database tables for indexing records and also generate graphql queries for accessing the indexed data.

For this guide, we will want to track a Post entity and have it exposed via the graphql API. This entity can be defined as the following schema file:

""" Entity named Post """
type Post {
  id: String!
  author: String!
  created_at_block: Int!
}

When updating your schema you should run following script to generate ORM models:

yarn checkpoint generate

Creating Data Writers

Data writers are JavaScript functions that get invoked by Checkpoint when it discovers a block containing this event. A Data writer is responsible for writing records to the database. These records will eventually be exposed via Checkpoint's GraphQL endpoint.

We have defined data writer function in our Checkpoint configuration for new_post event called handleNewPost.

Let's create these data writer functions:

import { starknet } from "@snapshot-labs/checkpoint";
import { getAddress } from '@ethersproject/address';
import { Post } from '../.checkpoint/models';

// We create createWriters function that accepts indexerName.
// This means we can reuse those for different networks (for example
// "mainnet" and "sepolia".
export const createWriters(indexerName: string) {
    // handleNewPost will get invoked when a `new_post` event
    // is found at a block
    const handleNewPost: starknet.Writer = async ({ event, block }) => {
        if (!event) return;
    
        // extract posters address from events data
        const author = getAddress(BigNumber.from(event.data[0]).toHexString());
        
        // store Post in database (note we pass indexerName here to be able
        // to tell apart Posts on each network)
        const post = new Post(`${author}/${tx.transaction_hash}`, indexerName);
        post.author = author;
        post.created_at_block = block.blockNumber;
        await post.save();
    };
    
    return { handleNewPost };
}

With the above code snippet, we have a data writer that writes new posts to the PostgreSQL database.

Starting Checkpoint

Finally, we can initialize a checkpoint instance with our arguments like these:

import fs from 'fs/promises';
import Checkpoint from "@snapshot-labs/checkpoint";
import { createWriters } from './writers.ts';
import { mainnetConfig, sepoliaConfig } from './config.ts';

...

const schemaFile = path.join(__dirname, `${dir}../src/schema.gql`);
const schema = fs.readFileSync(schemaFile, 'utf8');

...

const checkpoint = new Checkpoint(schema);

const mainnetIndexer = new starknet.StarknetIndexer(createWriters('mainnet'));
checkpoint.addIndexer('mainnet', mainnetConfig, mainnetIndexer);

const sepoliaIndexer = new starknet.StarknetIndexer(createWriters('sepolia'));
checkpoint.addIndexer('sepolia', sepoliaConfig, sepoliaIndexer);

Next, we start up checkpoint's indexer like this:

checkpoint.start();

The above code will start the checkpoint indexer, and it will begin processing each Starknet block. When a relevant contract event is found, it gets passed to the data writer for that event.

Querying data

Once Checkpoint has run for a while and has written some data to its database, you can query this data using the generated GraphQL API.

You can mount the query endpoint on any port you like using the graphql handler exported by Checkpoint's object. Like this:

import express from 'express';

const checkpoint = new Checkpoint(...);

const app = express();
app.use('/graphql', checkpoint.graphql);

const PORT = 3000;
app.listen(PORT, () => console.log(`Listening at http://localhost:${PORT}`));

This will mount a GraphQL endpoint that can be accessed at http://localhost:3000/graphql (and a GraphiQL interface at http://localhost:3000/graphql when visited from the browser).

Checkpoint exposes two types of queries:

These enables users to fetch information about indexer.

For starters, you can visit http://localhost:3000/graphql in your browser and try running the sample query generated in the graphiql UI.

Conclusion

At this juncture, you should have Checkpoint running, indexing your contracts data (on multiple networks) and serving this indexed data via graphql.

Checkpoint uses a configuration object to determine which networks and contract information it will be indexing.

// Configuration used to initialize Checkpoint
export interface CheckpointConfig {
  network_node_url: string;
  optimistic_indexing?: boolean;
  fetch_interval?: number; 
  tx_fn?: string;
  global_events?: ContractSourceConfig[];
  sources?: ContractSourceConfig[];
  templates?: { [key: string]: ContractTemplate };
  // mapping from ABI name to contract ABI
  abis?: { [key: string]: any };
}

export interface ContractEventConfig {
  // name of event in the contract
  name: string;
  // callback function in writer
  fn: string;
}

export interface ContractSourceConfig {
  // contract address
  contract: string;
  // name of ABI (if used it Checkpoint will parse events using provided ABI)
  abi?: string;
  // start block number
  start: number;
  // callback function in writer to handle deployment
  deploy_fn?: string;
  events: ContractEventConfig[];
}

export interface ContractTemplate {
  // name of ABI (if used it Checkpoint will parse events using provided ABI)
  abi?: string;
  events: ContractEventConfig[];
};

Checkpoint supports Express.js by default, but it's also possible to use Checkpoint with ApolloServer. Assuming ApolloServer is already installed in your Checkpoint project you can start ApolloServer using Checkpoint's schema with following code:

import Checkpoint, { LogLevel, createGetLoader } from '@snapshot-labs/checkpoint';
import { ApolloServer } from '@apollo/server';
import { startStandaloneServer } from '@apollo/server/standalone';

// Checkpoint initialization

async function run() {
  const server = new ApolloServer({
    schema: checkpoint.getSchema()
  });

  const { url } = await startStandaloneServer(server, {
    listen: { port: 3000 },
    context: async () => {
      const baseContext = checkpoint.getBaseContext();
      return {
        ...baseContext,
        getLoader: createGetLoader(baseContext)
      };
    }
  });

  console.log(`🚀  Server ready at: ${url}`);
}

run();

When initializing Checkpoint library there is an optional parameter that can be passed to configure some extra behavior of the library.

The type definition for the option object is:

This option can be provided as the final argument to the Checkpoint constructor like this:

See more about the configuration options below.

Database options

By default, when a Checkpoint object is started, it looks up the DATABASE_URL environment variable, but with dbConnection option parameter, you can specify a different connection string within the codebase itself and this value will override the DATABASE_URL value in the environment when connecting to the database.

Logging options

There are six (6) log levels currently supported by Checkpoint, these are:

In a non-production environment, you can set the prettifyLogs option to true and this will output a pretty version

Overrides config

Seed checkpoints manually

You can seed Checkpoint with a list of blocks, and Checkpoint will start by scanning this list of blocks first and invokes the appropriate data writers for any events founds. Once Checkpoint checks through the list of seeded blocks, it continues sequential scanning from the next block.

There is a seedCheckpoints method defined on Checkpoint instances:

This seedCheckpoints method should be called before starting the indexer, and the method can be called with as many contracts and blocks you already have.\

Exporting checkpoint blocks

You can set up a process (or external service) that periodically uses the _checkpoints query to fetch the latest blocks and export them for archiving or sharing with another instance of Checkpoint.

Using our Poster contract examples from above, to fetch all blocks where Checkpoint has encountered an event, you can run the following query:

Checkpoint enables data manipulation in your blockchain database without having to construct SQL queries directly. It features a capability to generate model objects from a GraphQL schema, and this guide will lead you through the process of creating and saving a model object into your database.

Generating Models

The first step is to generate the models from your GraphQL schema. This is done with the checkpoint generate command. Run it in your project environment like this:

yarn checkpoint generate

After running this command, your models will be created based on your GraphQL schema.

Importing Models

To use the generated models in your scripts, you need to import them. This is done with an import statement at the top of your file. For instance, if you've generated a Post model, you can import it like this:

import { Post } from '../.checkpoint/models';

This allows you to use the Post class to create new posts and save them to the database.

Defining Your Model

Each model is defined in a GraphQL schema file, such as schema.gql. Every model object will have a unique id along with other properties. Here's an example of a Post model:

scalar Text

type Post {
  id: String!
  author: String!
  content: Text!
  tag: String
  tx_hash: String!
  created_at: Int!
  created_at_block: Int!
}

Please note, this Post model is merely an example. Your actual model will have properties specific to your needs.

Creating a model Instance

To create a new instance of your model, you instantiate it with the id and indexer name. Here's how you'd create a new Post:

const post = new Post(`${author}/${tx.transaction_hash}`, "INDEXER_NAME");

You then set the other properties:

post.author = author;
post.content = content;
post.tag = tag;
post.tx_hash = tx.transaction_hash;
post.created_at = timestamp;
post.created_at_block = blockNumber;

Edit a model instance

Once the instance is defined, you can load an entity and edit it using loadEntity like that :

const post = await Post.loadEntity(id);
post.author = '0x123...';
await post.save();

Saving a model Instance

Once the instance is fully defined, you can save it into your database using the save method. This operation is asynchronous, so you should use await:

await post.save();

Checkpoint requires a set of defined GraphQL Schema Objects. The schema objects will be used to create a database structure for indexing this data and also generating GraphQL Queries for accessing the indexed data.

In Checkpoint terms, these schema objects are called Entities, an Entity can be defined as a GraphQL Object with a unique name and an id field.

For example:

""" Vote is a valid entity """
type Vote {
  id: String!
  voter: User
  space: String
  proposal: Int
  choice: Int
  vp: Int
  created: Int
}

type User {
  id: String!
  vote_count: Int
  created: Int
}

Vote and User are valid entities because their names are unique (within the schema) and contain an id field.

Query generation

Checkpoint generates two Query fields for each of the defined entities. One for querying a single entity record by its id and the second for querying multiple records of an entity.

For example, using the earlier defined User entity, Checkpoint will generate two query fields like:

type Query {
  user(id: ID): User
  users(
    first: Int
    skip: Int
    orderBy: String
    orderDirection: OrderDirection
    where: WhereUser
  ): [User]
}

type WhereUser {
  id: String
  id_in: [String]
  vote_count_gt: Int
  vote_count_gte: Int
  vote_count_lt: Int
  vote_count_lte: Int
  vote_count: Int
  vote_count_in: [Int]
  created_gt: Int
  created_gte: Int
  created_lt: Int
  created_lte: Int
  created: Int
  created_in: [Int]
}

Things to note:

• The name of the single record entity query is derived from the entity's name lowercased.

• The name of the multi-record entity query is derived from the entity's name lowercased with an s suffix.

• The generated Where* types fields for multi-record are derived based on original fields in the entity, and non-null values are treated a AND where filters when being executed against the database.

Data writers are callback functions that Checkpoint invokes when the event of a contract is discovered at a particular block.

type EvmWriter = (args: {
  blockNumber: number;
  eventIndex?: number;
  source: ContractSourceConfig;
  helpers: { executeTemplate };
  tx: Transaction;
  block: BlockWithTransactions | null;
  rawEvent?: Log;
  event?: LogDescription;
}) => Promise<void>;

Checkpoint: indexing made easy

Currently, Checkpoint exposes the following internal data queries:

1. `_metadata` and `_metadatas` query fields.

These are used to query single or multiple metadata records. Metadata records are key-value pairs of data used by Checkpoint internally to describe the state of its process. These queries are defined as:

type Query {
    """ queries a single metadata value by it's id (key) """
    _metadata(id: ID!, indexer: String): _Metadata

    """ queries multiple metadata values """
    _metadatas(
    indexer: String
    first: Int
    skip: Int
    orderBy: String
    orderDirection: OrderDirection
    where: Where_Metadata
    ): [_Metadata]
}

""" Core metadata values used internally by Checkpoint """
type _Metadata {
    """ example id: last_indexed_block """
    id: ID!
    indexer: String!
    value: String
}

For starters, you can execute the following query to see a list of all metadata values exposed by Checkpoint:

query {
    _metadatas {
        id
        _indexer
        value
    }
}

2. `_checkpoint` and `_checkpoints` query fields

Internally, Checkpoint keeps track of blocks where contracts event are found. These records are usually used by Checkpoint to speed up re-indexing when restarted. The _checkpoint(s) queries provide a way to query these blocks. The results can be exported and used to seed another Checkpoint instance running on another machine.

These queries are defined as:

type Query {
    """ queries a single _checkpoint entry by it's id"""
    _checkpoint(id: ID!, indexer: String): _Checkpoint
    
    """ queries multiple _checkpoint entires """
    _checkpoints(
    indexer: String
    first: Int
    skip: Int
    orderBy: String
    orderDirection: OrderDirection
    where: Where_Checkpoint
    ): [_Checkpoint]
}

""" Contract and Block where its event is found. """
type _Checkpoint {
    """ id computed as last 5 bytes of sha256(contract+block) """
    id: ID!
    block_number: Int!
    contract_address: String!
}

For example, you can run the following query to fetch all blocks where the event of a particular contract can be found:

query {
    _checkpoints(indexer: "mainnet", where: {
        contract_address: "0x<contract-address>"
    }) {
        block_number
    }
}

Data writers are callback functions that Checkpoint invokes when the event of a contract is discovered at a particular block.

type StarknetWriter = (args: {
  blockNumber: number;
  eventIndex?: number;
  source: ContractSourceConfig;
  helpers: { executeTemplate };
  tx: Transaction;
  block: FullBlock | null;
  rawEvent?: Event;
  event?: ParsedEvent;
}) => Promise<void>;