Mantle
Mantle is an framework for writing indexers on Terra network.
Getting Started
Writing your first indexer
You can consider the term indexer a fancy way of saying an ETL logic (as in Extract-Transform-Load). Having said that, it is no different that your indexer follows the aforementioned sequence in your code; You need to Extract some data first.
Extract
Here is how you would do extraction in mantle.
type request struct {
BlockState struct {
Height uint64
Block struct {
Header struct {
AppHash string
}
}
}
FaucetBalance struct {
Result sdk.Coins
} `query:"BankBalancesAddress(Address: $address)"`
}
func YourIndexer(query types.Query, commit types.Commit) {
// assign a memory space to hold request
req := request{}
// make the request!
err := query(&request, map[string]interface{}{
"address": "terra1h8ljdmae7lx05kjj79c9ekscwsyjd3yr8wyvdn"
})
if err != nil {
// handle any error in extract
}
// ... your logic goes here
}Notice how the request type defines all data you are requesting for this specific indexer. Mantle uses graphql for data fetching, so there is no SQL or document names involved in the request. All you need to make sure is to supply the desired entity as the type name. Mantle handles the conversion to graphql query for you.
You may want to name your request type in small letters, so it is not exported within the same package. This way you can keep naming your requests
request.
In fact, the request we just wrote after graphql query conversion is:
query(Address: String!) {
BlockState {
Height
Block {
Header {
AppHash
}
}
}
FaucetBalance: BankBalancesAddress(Address: $address) {
Result
}
}You may be wondering about the mantle:"BankBalancesAddress(Address: $address)" part. This is mantle way of specifying graphql aliases. Using this, you can query the same entity with different arguments under different names.
Here are some of the examples:
// let's assume the entity we are requesting is called `Entity`.
type Entity struct {
Data string
}
// Demonstrated here also is how you can reuse already defined type definition;
// everything defined in `Entity` will get embedded in graphql query automatically.
type request struct {
// omitting the Height parameter will resolve the entity of current block height.
Entity Entity
// specifying Height will resolve the specific entity generated at that specific height.
Entity1 Entity `query:"Entity(Height: 1000)"`
}This results in graphl query:
query {
Entity: Entity {
Data
}
Entity1: Entity(Height: 1000) {
Data
}
}Disallowed act: self-referencing
Whilst the request scheme is simple and effective, you may run into race conditions if you are not careful.
The inherent race condition is introduced by the way mantle handles cross-indexer dependencies. If you request an entity that is expected to be resolved by another indexer in the same height round, the request call will block until that specific entity is finally resolved.
For example, let's assume that you wrote the following indexer (more on the commit part later):
type Entity struct {
Data string
}
type request struct {
Entity Entity
}
func YourIndexer(query types.Query, commit types.Commit) {
req := request{}
err := query(request, nil)
// racey!
commit(Entity{
Data: "Hello World"
})
}With this code, you are basically requesting the current version of Entity that is to be resolved (committed) after your requeest call. This obviously will NEVER resolve.
Whilst mantle will yell at you with an error (with a lot of
Transform
It is 100% up to you to write the transform logic. (more to be added if any
Load (Commit)
Much like requests, you first need to write type definition for your entity. Some rules:
- All fields must be exported (start with a capital letter).
- The type name becomes entity name for later requests. For this reason, type must also be exported.
- Root type must be of
structor[]structtype; nointerfaceor primitive types allowed - Not all types of golang are supported.
- Try to use primitive golang types. For example,
BigInts can be safely converted tostring. - Some composite types (i.e.
time.Time,sdk.Coins) are supported, but don't expect full coverage. If you deem a specific type nessary, open an issue.
- Try to use primitive golang types. For example,
// declare your entity
type TrackFaucet struct {
Height uint64
BalanceUluna string
BalanceUkrw string
}
func YourIndexer(query types.Query, commit types.Commit) {
req := request{}
err := query(request, nil)
// commit!
commitError := commit(Entity{
Data: "Hello World"
})
if commitError != nil {
return commitError // only handle this error if you really know what you're doing
}
}Disallowed act: Duplicate commits
Mantle doesn't care if you call commit multiple times within your indexer (i.e. persisting different entities). However, committing the same entity more than once is disallowed.
With the commit call, mantle will automatically index (as in database indexes) the entity with the current Height. Committing the same entity more than once will:
- not only waste a db transaction for nothing,
- but also forces the underlying cross-indexer dependency resolver to emit the entity many times.
Since mantle can't tell which version of entity is the right one, this is an undefined behaviour. If you need to commit multiple instances of the same entity, consider using slice type.
Duplicate commit will result in error, and in most cases you should return that error again from your indexer. This way, mantle gets signalled of the failure, which in turn safely discards all changes in that round and does a graceful shutdown.
You may want to persist different types of entities processed by an indexer function.
Commitsbeing called multiple times this way is totally fine.
Putting it all together 🚀
package indexers
import (
"reflect"
sdk "github.com/cosmos/cosmos-sdk/types"
"github.com/terra-project/mantle-sdk/types"
)
type request struct {
BlockState struct {
Height uint64
Block struct {
Header struct {
AppHash string
}
}
}
FaucetBalance struct {
Result sdk.Coins
} `query:"BankBalancesAddress(Address: $address)"`
}
type TrackFaucet struct {
Height uint64
ProofThatThisIsReal string
BalanceUluna string `model:"index"`
BalanceUkrw string `model:"index"`
}
func InitTrackFaucet(register types.Register) {
register(
collectTrackFaucet,
reflect.TypeOf(TrackFaucet{}),
)
}
func collectTrackFaucet(query types.Query, commit types.Commit) error {
// make request
// the result output is exactly the same as the struct
// we have defined earlier (request)
req := request{}
// making request with $address parameter (faucet address in this case)
errorInQuery := query(&req, map[string]interface{}{
"address": "terra1h8ljdmae7lx05kjj79c9ekscwsyjd3yr8wyvdn",
})
// handle error
if errorInQuery != nil {
panic(errorInQuery)
}
// YOUR INDEXER LOGIC GOES HERE
// only save uluna and ukrw, in string form
var uluna string
var ukrw string
for _, balance := range req.FaucetBalance.Result {
if balance.Denom == "uluna" {
uluna = balance.Amount.String()
} else if balance.Denom == "ukrw" {
ukrw = balance.Amount.String()
}
}
// save!
commitError := commit(TrackFaucet{
Height: req.BlockState.Height,
ProofThatThisIsReal: req.BlockState.Block.Header.AppHash,
BalanceUluna: uluna,
BalanceUkrw: ukrw,
})
if commitError != nil {
return commitError
}
}Advanced Usage
Database Indexes
For efficient search request, you may use
Defining index in entity
// entity definition
type TrackFaucet struct {
// Supplying `model:"index"` will use field name as index key.
BalanceUluna string `model:"index"`
// Supplying `model:"primary"` will create a primary index key.
// Entity with primary key is unique, meaning only __ONE__ entity
// with the designated primary key can exist in database.
//
// Committing another entity with a pre-existing primary key will
// overwrite the previously committed entity.
//
// This is useful if you don't want to index by block Height.
// i.e. a model which only persists the latest state, and is keyed by account address.
AccountAddress string `model:"primary"`
}Searching by index
By Height
Mantle indexes every entity by Height. You can request any entity with height:
// define your query with Height parameter
type request struct {
faucetBalance TrackFaucet `query:"TrackFaucet(Height: $address)"`
}
// in your request call
req := request{}
reqErr := query(&req, map[string]interface{}{
"Height": 5555 // type must be uint variant
})By a specific index
// define your query with the index name
type request struct {
faucetBalance TrackFaucet `query:"TrackFaucet(Uluna: $lunaAmount)"`
}
// in your request call
req := request{}
reqErr := query(&req, map[string]interface{}{
"lunaAmount": "someLunaAmount" // type must be the same as index field type
})By range
When you do a range search, it is safe to assume that you expect multiple entities as response. It means your expected type will be a slice of the underlying type.
To search by range you MUST:
- use slice type
- use pluralized entity name
- use index parameter with the postfix
_range. e.g.) index nameukrwbecomesukrw_range. - use argument type of
indexType[]
_rangeparameters are ONLY available for pluralized queries. For singular entities, _range parameters are undefined.
// notice how we are:
// - using []TrackFaucet instead of TrackFaucet.
// - also using TrackFaucets instead of TrackFaucet.
type request struct {
faucetBalance []TrackFaucet `query:"TrackFaucets(Uluna_range: [$range_start, $range_end])"`
}
// in your request call
req := request{}
reqErr := query(&req, map[string]interface{}{
"range_start": "60000" // type must be the same as index field type
"range_end": "100000" // type must be the same as index field type
})Aggregation
TBD
Heightless Commits
TBD
