1. Block Processor

• Fetches blocks and chunks from FastNear Data Server.
• Processes and stores data into PostgreSQL table matching the BigQuery dataset format.

2. Staking Processor

• Fetches staking info via RPC to calculate NEAR account balances.
• Updates ft_balances_daily table with calculated balances.

3. Data Packer

• Packs processed data into Avro format.
• Stores Avro files in an SSD folder.

4. Data Uploader

• Retrieves Avro files from SSD.
• Uploads Avro files to Google Cloud Storage periodically.
• Initiates BigQuery import via Google Cloud API.
• Records job_id from Google Cloud API in PostgreSQL.
• Monitors job status via API.
  • Success: Deletes Avro files and corresponding database rows.
  • Failure: Sends alert via email/Telegram for manual investigation.