Celo Contract Subgraph Syncing Failure and Debugging

4 min readJan 20, 2022

Dapplooker has been selected as the winner of Celo Make Crypto Mobile Hackathon in the Web3 and Infrastructure category 🏆. We implemented Celo contract subgraph and Celo dashboard with metrics like TVL, daily/monthly volumes and much more. You can see subgraph and hackathon submission using the link below:

Celo Make Crypto Mobile Hackathon Submission

Subgraph: https://thegraph.com/hosted-service/subgraph/dapplooker/develop

Indexed Contract: 0x471EcE3750Da237f93B8E339c536989b8978a438

As you can see, subgraph is not efficient as it was tracking all the transactions of Celo contract. After the hackathon got over, we started to update subgraph with analytics entities like User, Token, TokenDayData.

Now less data will be stored, so we expect subgraph to index faster.⚡

Subgraph Indexing Issue: Non-Deterministic Error 🚩

After creating analytics entities in subgraph, we deployed subgraph on a new account.

Updated Subgraph: https://thegraph.com/hosted-service/subgraph/dapplooker/celo-tokens-analytics-subgraph?version=pending

We faced a major roadblock while syncing updated subgraph on the hosted Graph node. Please see the error below:

Subgraph failed with non-deterministic error: failed to process trigger: block #10419840 (0xdaf4…f9c0), transaction 929724430d76bdf33710d0b7319f2e7df2739b4ae41548eb1764cadda79f013e: Mapping terminated before passing in trigger: send failed because receiver is gone, retry_delay_s: 1800, attempt: 17
Processing 13 triggers, component: BlockStream

The block number at which it was failing was random and not fixed. In most cases a non-deterministic error goes away after multiple retries. We retried deploying same subgraph multiple times but to no avail. 🤦‍♂️

Debugging with TheGraph Team 💡

We started following up with TheGraph team on the issue. As the error is non-deterministic error, it was difficult to know the root cause. We got some hint from TheGraph team, where we could query for Subgraph ID and know error details. Follow the steps below to query for subgraph health. This will retrieve the error message for failed subgraphs: 📊

Find your Deployment ID (“Qm….”)
Go to https://graphiql-online.com/
Enter API https://api.thegraph.com/index-node/graphql
Run below query:

{
  indexingStatuses(subgraphs: ["Qm..."]) {
    subgraph
    synced
    health
    entityCount
    fatalError {
      handler
      message
      deterministic
      block {
        hash
        number
      }
    }
    chains {
      chainHeadBlock {
        number
      }
      earliestBlock {
        number
      }
      latestBlock {
        number
      }
    }
  }
}

As you can see in the screenshot below, we received the following query result for failed subgraph. 🧾

One issue in subgraph was that there was a version mismatch between the graph-cli version locally and apiVersion(subgraph.yaml) used for subgraph deployment.

Subgraphs with ApiVersion<=0.0.4 and graph-cli<=0.21.1 use an archaic version of AssemblyScript. So, when we use newer versions of graph-cli ( which in our case was 0.25.1), and ApiVersion >= 0.0.5 has to be used also, which uses a very recent version of AssemblyScript.

Identifying the Issue on TheGraph Node ⚡

We initiated subgraph deployment with apiVersion=0.0.5 with graph version 0.25.1. Unfortunately we received the same error but on a different block number.

failed to process trigger: block #190080 (0xac57…835d), transaction ac57ac5a96108b0f133ac6bac59e1bd0df0263d248a4fcc36393464b7c73835d: Mapping terminated before handling trigger: oneshot canceled

Finally TheGraph team informed us that the issue was with the hosted graph node for Celo contract. 📜

The issue have been identified, and a fix will be implemented.
You can track the issue here https://github.com/graphprotocol/graph-node/pull/3158

As per PR details:

This was a sneaky consequence of Transaction.from being made optional which we missed #2896. This caused a panic in the runtime here

Update as of Jan 20, 2022 📈

PR is merged but fix is not deployed on the hosted Graph node. Once the fix is deployed on the hosted graph node, we will re-deploy the subgraph again. Great thanks to TheGraph team for the help throughout the debugging process. 🙌

Update as of Feb 15, 2022 📊

The issue is resolved ✅. We have successfully deployed Celo subgraph on hosted service without any issues.

“Let’s build data driven web3”

About Dapplooker

Dapplooker is multi-chain analytics and visualization platform for blockchain networks and Dapps. Our aim is to empower people by making analysis of blockchain data simple with the aid of our easy-to-use visual editor. Users can run SQL queries on smart contract data, build charts and dashboards which can be shared with the community. Learn more: https://dapplooker.com

Website | Twitter | Discord | Medium