Go Merkle Tree
High performance Golang Merkle Tree, supporting parallelization and OpenZeppelin sibling-sorting.
Merkle Tree
Below is a Merkle Tree data structure sample illustration generated by the set of data blocks {A, B, C, D}. Each leaf node of the tree corresponds to the hash value of a block in the data block set, whereas each branch node corresponds to the hash value of the concatenation of the child node hashes (i.e. Hash( hash_a || hash_b), or Hash(hash_a + hash_b)).This structure is particularly beneficial for proof of membership/existence. In order to demonstrate the presence of a data block, such as block C, one merely requires Hash_11, Hash_0, and the Top Hash (i.e., Merkle Root). The individual can subsequently calculate new Hash_10, Hash_1, and Top Hash, and compare the new Top Hash with the previous one to verify whether the block exists in the data block set.
Installation
go get -u github.com/txaty/go-merkletreeConfiguration
// Customizable hash function used for tree generation.
HashFunc TypeHashFunc
// Number of goroutines run in parallel.
// If RunInParallel is true and NumRoutine is set to 0, use number of CPU as the number of goroutines.
NumRoutines int
// Mode of the Merkle Tree generation.
Mode TypeConfigMode
// If RunInParallel is true, the generation runs in parallel, otherwise runs without parallelization.
// This increase the performance for the calculation of large number of data blocks, e.g. over 10,000 blocks.
RunInParallel bool
// SortSiblingPairs is the parameter for OpenZeppelin compatibility.
// If set to `true`, the hashing sibling pairs are sorted.
SortSiblingPairs bool
// If true, the leaf nodes are NOT hashed before being added to the Merkle Tree.
DisableLeafHashing boolTo define a new Hash function:
func myHashFunc(data []byte) ([]byte, error) {
sha256Func := sha256.New()
sha256Func.Write(data)
return sha256Func.Sum(nil), nil
}Critical Advisory: It's essential to ensure that the hash functions employed by parallel algorithms are concurrent-safe.
DefaultHashFunc, which is the default hash function used for tasks executed sequentially in this library, does NOT offer concurrent-safety, given its re-utilization of the same SHA256 digest.
Example
Proof generation and verification of all blocks
package main
import (
"crypto/rand"
"fmt"
mt "github.com/txaty/go-merkletree"
)
// first define a data structure with Serialize method to be used as data block
type testData struct {
data []byte
}
func (t *testData) Serialize() ([]byte, error) {
return t.data, nil
}
// generate dummy data blocks
func generateRandBlocks(size int) (blocks []mt.DataBlock) {
for i := 0; i < size; i++ {
block := &testData{
data: make([]byte, 100),
}
_, err := rand.Read(block.data)
handleError(err)
blocks = append(blocks, block)
}
return
}
func main() {
blocks := generateRandBlocks(10)
// the first argument is config, if it is nil, then default config is adopted
tree, err := mt.New(nil, blocks)
handleError(err)
// get proofs
proofs := tree.Proofs
// verify the proofs
for i := 0; i < len(proofs); i++ {
ok, err := tree.Verify(blocks[i], proofs[i])
handleError(err)
fmt.Println(ok)
}
// or you can also verify the proofs without the tree but with Merkle root
// obtain the Merkle root
rootHash := tree.Root
for i := 0; i < len(blocks); i++ {
// if hashFunc is nil, use SHA256 by default
ok, err := mt.Verify(blocks[i], proofs[i], rootHash, nil)
handleError(err)
fmt.Println(ok)
}
}
func handleError(err error) {
if err != nil {
panic(err)
}
}Build tree and generate proofs for a few blocks
blocks := generateRandBlocks(10)
// create a Merkle Tree config and set mode to tree building
config := &mt.Config{
Mode: ModeTreeBuild,
}
tree, err := mt.New(config, blocks)
handleError(err)
// get the proof for a specific data block
// method GenerateProof is only available when ModeTreeBuild or ModeProofGenAndTreeBuild
proof0, err := tree.GenerateProof(blocks[0])
handleError(err)
proof3, err := tree.GenerateProof(blocks[3])
handleError(err)Parallel run
blocks := generateRandBlocks(10)
// create a Merkle Tree config and set parallel run parameters
config := &mt.Config{
RunInParallel: true,
NumRoutines: 4,
}
tree, err := mt.New(config, blocks)
handleError(err)Benchmark
Setup:
| AWS EC2 | CPU | Memory | OS | Hash Function | Go Version |
|---|---|---|---|---|---|
| c5.4xlarge | 16 Core | 32GB | Ubuntu 22.04 LTS | SHA256 | 1.21.4 |
Benchmark tasks:
- Proof generation for all the blocks: at the end we can obtain the Merkle Root and the proofs of all the data blocks.
- Proof verification: verify a single proof.
|
|
|
Note: Please note that the size of each data block is determined by the tree depth, which is represented on the x-axis of the figures. In order to better visualize the full range of values, the y-axis is shown using a logarithmic scale. However, it's important to keep in mind that the real time difference between data points will be larger than what is depicted on the figure due to the logarithmic scale.
Benchmark implementation can be found in txaty/merkle-tree-bench.
Dependencies
This project requires the following dependencies:
-
golang.org/x/sync: This package provides
errgroupwhich is utilized to manage errors stemming from goroutines. - gomonkey: This is a versatile Go library designed to facilitate monkey patching during unit testing. Please be aware that users operating on Apple Silicon MacBooks may encounter 'permission denied' issues. Nevertheless, these potential issues do not impact the functionality or use of the Merkle Tree library.
License
Released under the MIT License.
