This is the first product report from Meteor Wallet for **BigQuery and Data Pipelines,** and we plan to make monthly report until the product is fully shipped moving forward.

1. Summary

• We have studied the existing BigQuery table and data structure

  BigQuery Schemas

• We have studied the existing python source code

• We have come up with a new architecture to maintain the BigQuery Pipeline here:

  BigQuery Proposed Architecture

• We are starting to work on the new architecture.

2. Changes We Propose:

• Instead of using near indexer framework as mentioned in the original proposal, we decided to go with FastNear Data Server as suggested by both Nearblocks and FastNear team as it is more stable and reliable.

• There is an existing ft_balances_daily table that keeps track of a list of hard coded wallet IDs on their near balances on validators.

  

  • We will replace this part using FastNear indexer for 2 reasons:
    • current_validators from validators rpc call will get only active validators in current epoch and if a validator is inactive, it will be skipped and won't be checked
    • Current method spams hundreds of RPC calls for each account within short timespan which can lead to RPC error sometimes especially during peaks.