Data and analytics

Introduction

As utilization of the network continues to grow, an increasing amount of valuable information will exist in the onchain data. As the volume of data rapidly increases, calculating and aggregating this information to report upon or drive a dapp can become a time and process heavy endeavor.

Leveraging existing data providers can expedite development, produce more accurate results, and reduce ongoing maintenance efforts. This will enable a team to concentrate on the core functionality their project is trying to provide.

Prerequisites

You should understand the basic concept of Block Explorers in order to better understand using them in the data analytics context. In addition, familiarize yourself with the concept of an to understand the benefits they add to a system design.

In terms of architectural fundamentals, understanding what an APIopens in a new tab and RESTopens in a new tab are, even in theory.

Block explorers

Many Block Explorers offer RESTfulopens in a new tab APIopens in a new tab gateways that will provide developers visibility into real-time data on blocks, transactions, validators, accounts, and other onchain activity.

Developers can then process and transform this data to give their users unique insights and interactions with the . For example, Etherscanopens in a new tab provides execution and consensus data for every 12s slot.

The Graph

The Graphopens in a new tab is an indexing protocol that provides an easy way to query blockchain data through open APIs known as subgraphs.

With The Graph, developers can benefit from:

Decentralized indexing: Enables indexing blockchain data through multiple indexers, thus eliminating any single point of failure
GraphQL queries: Provides a powerful GraphQL interface for querying indexed data, making data retrieval super simple
Customization: Define your own logic for transforming & storing blockchain data, and reuse subgraphs published by other developers on The Graph Network

Follow this quick-startopens in a new tab guide to create, deploy, and query a subgraph within 5 minutes.

Client diversity

Client diversity is important for the overall health of the Ethereum network because it provides resilience to bugs and exploits. There are now several client diversity dashboards including clientdiversity.orgopens in a new tab, rated.networkopens in a new tab, supermajority.infoopens in a new tab and Ethernodesopens in a new tab.

Dune Analytics

Dune Analyticsopens in a new tab pre-processes blockchain data into relational database (DuneSQL) tables, allows users to query blockchain data using SQL and build dashboards based on query results. Onchain data are organized into 4 raw tables: blocks, transactions, (event) logs and (call) traces. Popular contracts and protocols have been decoded, and each has its own set of event and call tables. Those event and call tables are processed further and organized into abstraction tables by the type of protocols, for example, dex, lending, stablecoins, etc.

SQD

SQDopens in a new tab is a decentralized hyper-scalable data platform optimized for providing efficient, permissionless access to large volumes of data. It currently serves historical on-chain data, including event logs, transaction receipts, traces, and per-transaction state diffs. SQD offers a powerful toolkit for creating custom data extraction and processing pipelines, achieving an indexing speed of up to 150k blocks per second.

To get started, visit the documentationopens in a new tab or see EVM examplesopens in a new tab of what you can build with SQD.

SubQuery Network

SubQueryopens in a new tab is a leading data indexer that gives developers fast, reliable, decentralized, and customized APIs for their web3 projects. SubQuery empower developers from over 165+ ecosystems (including Ethereum) with rich indexed data to build an intuitive and immersive experiences for their users. The SubQuery Network powers your unstoppable apps with a resilient and decentralized infrastructure network. Use SubQuery's blockchain developer toolkit to build the web3 applications of the future, without spending time building a custom backend for data processing activities.

To start, visit the Ethereum quick start guideopens in a new tab to start indexing Ethereum blockchain data in minutes in a local Docker environment for testing before going live on a SubQuery's managed serviceopens in a new tab or on SubQuery's decentralised networkopens in a new tab.

Ethernow - Mempool Data Program

Blocknativeopens in a new tab provides open access to its Ethereum historical mempool data archiveopens in a new tab. This enables researchers and community good projects to explore the pre-chain layer of Ethereum Mainnet. The data set is actively maintained and represents the most comprehensive historical record of mempool transaction events within the Ethereum ecosystem. Learn more at Ethernowopens in a new tab.

EVM Query Language

EVM Query Language (EQL) is an SQL-like language designed to query EVM (Ethereum Virtual Machine) chains. EQL's ultimate goal is to support complex relational queries on EVM chain first-class citizens (blocks, accounts, and transactions) while providing developers and researchers with an ergonomic syntax for everyday use. With EQL, developers can fetch blockchain data using familiar SQL-like syntax and eliminate the need for complex boilerplate code. EQL supports standard blockchain data requests (e.g., retrieving an account's nonce and balance on Ethereum or fetching the current block size and timestamp) and is continually adding support for more complex requests and featuresets.