Sharding: What it is and why many blockchain protocols rely on it
Credit to Author: Lucas Mearian| Date: Mon, 28 Jan 2019 03:00:00 -0800
As blockchains are being rolled out in an increasing number of pilot programs for everything from cross-border financial transactions to supply chain management, one persistent issue remains: a lack of scalability.
As more computers join the peer-to-peer network, the efficiency of the whole system typically degrades.
Scalability has already been identified as an issue with cryptocurrencies such as bitcoin and Ethereum’s Ether. If a distributed ledger is to achieve adoption by financial technology (FinTech) companies and compete with payment networks hundreds of times faster, it must find a way to boost scalability and throughput and address latency problems.
Enter “sharding.”
Sharding is one of several popular methods being explored by developers to increase transactional throughput. Simply stated, sharding is a way of partitioning to spread out the computational and storage workload across a peer-to-peer (P2P) network so that each node isn’t responsible for processing the entire network’s transactional load. Instead, each node only maintains information related to its partition, or shard.
The information contained in a shard can still be shared among other nodes, which keeps the ledger decentralized and theoretically secure because everyone can still see all ledger entries; they simply don’t process and store all the information: account balances, contract code, etc.
One of the most persistent problems with public blockchains revolves around consensus protocols – how to get an agreement from users over whether proposed transactions are authentic and should be added to a distributed ledger; consensus protocols such as the most popular proof of work (PoW) mechanism can be highly compute intensive.
In today’s blockchains, each authenticating computer or node records all the data on the electronic ledger and is part of the consensus process. In large blockchains such as bitcoin, the majority of participating nodes must authenticate new transactions and record that information if they are to be added to the ledger; that makes completing each transaction slow and arduous. Because of that, bitcoin, which is based on a PoW, can only process 3.3 to 7 transactions per second – and a single transaction can take 10 minutes to finalize. Ethereum, another popular blockchain ledger and cryptocurrency, is only able to process from 12 to 30 transactions per second.
By comparison, Visa’s VisaNet on average processes 1,700 transactions per second.
The upside to having every node record new information being added to a blockchain is that the data is indisputable and unchangeable — blockchains are write-once, append many electronic ledgers.
Ethereum and Hyperledger are the world’s leading blockchain platforms and the basis for a myriad of applications, from cryptocurrencies such as Ethereum’s Ether to “smart” or self-executing online contracts used in commercial transactions and supply chain management. Ethereum is exploring sharding while Hyperledger isn’t, according to David Huseby, Security Maven for Hyperledger at the Linux Foundation.
“Sharding is a concept that originated with horizontal database partitioning and is being adopted by Ethereum… and functions so that not every node has to process all transactions, thereby enhancing scalability,” said Avivah Litan, a Gartner vice president and distinguished analyst.
Last year, Ethereum began exploring ways to increase performance after the blockchain ledger and cryptocurrency reached more than one million transactions per day.
Ethereum settled on two proposed fixes. One was a “layer 2” mechanism – processing transactions off the chain in a standard database and only recording permanent entries on the ledger; the other solution was sharding, allowing many more transactions to be processed in parallel at the same time.
Layer 2 protocols send most transactions off-chain and only interact with the underlying blockchain in order to enter and exit the layer-2 system. Layer 2 protocols transfer data between nodes within a LAN or an adjacent WAN, lifting the burden from the P2P blockchain network.
After sharding is activated, the “state” of a blockchain is split into shards or partitions. Each unique user account equates to one shard, and accounts can only transact with other accounts in the same shard, Litan explained. “This allows lots of parallel transactions to happen at the same time,” she said. “A separate protocol chosen by Ethereum then allows cross-shard communications.”
Ethereum has been exploring using only a small subset of nodes to verify each blockchain transaction. As long as there are sufficient number of nodes verifying each transaction, the network is still secure while also being able to process many transactions in parallel.
Along with addressing scalability, some argue sharding also maintains the native security of a blockchain because it retains “most of the desired decentralization and security properties of a blockchain,” Ethereum creator Vitalik Buterin wrote in a blog post.
“In theory, the increase in transaction throughput is linear in the number of shards. Four shards? Four times the throughput, roughly. There can be any number of shards,” Hyberledger’s Huseby said.
But, the devil is in the details, Huseby pointed out. In order to maintain blockchain’s security, you have to guard against shard takeovers. Corrupting the nodes in a given shard will lead to the permanent loss of the corresponding portion of data, according to research paper published by Cornell University.
In the Ethereum network’s model, for example, nodes must be randomly assigned to a shard and at random times they get reassigned to another randomly chosen shard.
“The idea is to make it difficult for an attacker to predict, or force, which shard their (malicious) node gets assigned to. This makes it more difficult to get a Byzantine takeover of any one shard,” Huseby said.
Sharding with Hyperledger blockchains isn’t so cut and dry, Huseby explained.
“Our blockchains typically don’t deal with ‘addresses’ like cryptocurrencies do. Hyperledger blockchains focus on maintaining a global state (think database) and the consensus mechanism regulates the updates to that state while the blockchain stores state updates securely,” Huseby said.
Hyperledger networks can be sharded vertically like Ethereum, but since it does not split up an address space, it is free to try different sharding techniques.
“If I had to shard a Hyperledger network, I would first take advantage of the division between transaction validation and block construction,” Huseby said. “Transaction validation is much slower than block construction, so my first pass would be to drastically increase the number of transaction validator nodes.”
A second challenge is dealing with “thin” clients, also called SPV (Simplified Payment Verification) wallets, to ensure those nodes have the full picture of the blockchain state while it’s divided among shards. To address the visibility problem associated with sharding, thin clients communicate via separate networks and maintain local state copies for each shard.
Lastly, inter-shard communication poses a challenge because each shard appears as a separate blockchain network.
Earlier this month, start-up firm Devvio announced it had created a highly efficient distributed ledger protocol based on sharding, layer 2 protocols and efficient consensus mechanism, that can address all the major problems facing blockchain networks. Devvio said its protocol could scale up for global financial business by executing up to eight million transactions per second.
Devvio claims it scales effeciently using independent blockchains based on sharding. As additional throughput is needed, thousands of shards can be added over time to ultimately process tens of millions of transactions per second, on-chain, on a global public blockchain.
In Devvio’s “Devv” protocol, each shard represents a separate blockchain ledger; the company claims thousands of shards can be added to a global public blockchain over time to ultimately process tens of millions of transactions per second. For example, each shard is an independent blockchain node on the Devv decentralized ledger that can handle up to 3,000 transactions. Adding another node would double the number of transactions capable of being processed, according to Devvio CEO Tom Anderson.
Each shard (which is also a crypto wallet) becomes an input on a larger network, which Devvio calls the T1 network; individual shards can communicate to others via a separate transactional network, called T2.
Martha Bennett, a principal analsyst with Forrester Research, pointed out that virtually all the current blockchain frameworks that use, or are proposing to use, sharding perform the function differently.
PolyShard is a sharding solution that uses ideas from coding theory to simultaneously
achieve optimal guarantees in security, storage efficiency, and computational efficiency. The key intuition is that nodes should not store replicated data; instead, they should store coded linear combinations of data.
For example, seven universities under the auspices of a Swiss non-profit organization called the Distributed Technology Research Foundation (DTR) just announced they are developing a digital currency network that solves blockchain’s scalability and performance problems through sharding.
“A lack of scalability is holding back cryptocurrency adoption, and [our] groundbreaking research is addressing this,” said Joey Krug, a member of the DTR Foundation Council. “The Unit-e developers are turning this research into real scalable performance which will benefit a huge swath of decentralized financial applications.”
Unit-e is using “entirely new ways of sharding” called “PolyShard” – a storage and computation solution that grows more efficient with more users without sacrificing security. The key is that the PolyShard protocol mixes up data from different users and transactions in a way that still allows accurate data recovery, akin to virtualization on server and storage systems.
To date, however, sharding mechanisms are still in the development-and-testing phase – theoretical in a sense – and creating standardized methods that address not only scalability but security. That challenge must be addressed before sharding can be considered a solution.
“Sharding isn’t such a cut-and-dry solution to scalability,” Husebuy said. “There are lots of details that have to be considered and we’re going to need some empirical experimentation to go along with the theory before we can call it secure. The implementations have to be careful about assumptions so that there are no holes an attacker can use to bypass any of the security and consensus mechanism.”