Using nested fields for indexes.

[timurkhametov]

Hi there. I see that PR #62 hasn't been merged yet. Do you know guys other workarounds to add indexes for nested fields?
Let's say I have approximately the following structure:

type MyStruct struct {
   ID int
   External ExternalDependency
   YetAnotherDependency []AnotherDependency 
}

type ExternalDependency struct {
   ExternalID string
}

type AnotherDependency struct {
  CustomID string
}

How to add indexes for ExternalID and CustomID?

[dnephin]

Hello @timurkhametov, thanks for the question! go-memdb allows anyone to implement their own indexes. The provided indexers handle common cases, but often in Consul we will write our own custom indexers. There are some advantages to using custom indexers, one of which is that it can be a bit easier to understand how index values are built from a struct.

To write a custom indexer you would need to implement two interfaces: Indexer and one of either SingleIndexer or MultiIndexer (if a single entry should be queryable at multiple index values, ex: indexing a slice of properties like YetAnotherDependency.CustomID in your example). Optionally the PrefixIndexer interface can also be implemented to support prefix queries.

As an example, to index ExternalID I might write something like this:

type externalIDIndexer struct{}

func (externalIDIndexer) FromArgs(args ...interface{}) ([]byte, error) {
	if len(args) != 1 {
		return nil, fmt.Errorf("wrong number of args %d, expected 1", len(args))
	}
	s, ok := args[0].(string)
	if !ok {
		return nil, fmt.Errorf("wrong type for arg %T, expected string", args[0])
	}
	return append([]byte(s), 0), nil
}

func (externalIDIndexer) FromObject(raw interface{}) (bool, []byte, error) {
	s, ok := raw.(MyStruct)
	if !ok {
		return false, nil, fmt.Errorf("wrong type for arg %T, expected MyStruct", raw)
	}
	if s.External.ExternalID == "" {
		return false, nil, nil
	}
	return true, append([]byte(s.External.ExternalID), 0), nil
}

If you are writing many custom indexers some helpers can be used to remove the common boilerplate that will likely be shared by all of them. For example, this is what we use in Consul, to allow a custom indexer to be implemented by two functions, without needing to check the args length in each. indexBuilder is used instead of memdb.CompoundIndex when there are multiple fields in an index.

If there are many custom indexers for a single type, functional composition can be used to remove the duplication of type asserting each time:

func myStructIndexer(f func(s MyStruct) ([]byte, error)) func(arg interface{}) ([]byte, error) {
	return func(arg interface{}) ([]byte, error) {
		s, ok := raw.(MyStruct)
		if !ok {
			return nil, fmt.Errorf("wrong type for arg %T, expected MyStruct", raw)
		}
		return f(s)
	}
}

Some part of this might be good to add to the godoc examples for this package.

[hbagdi]

Here is another implementation in case you are interested: https://github.com/Kong/deck/blob/main/state/indexers/subFieldIndexer.go

[timurkhametov]

@dnephin, @hbagdi thank you guys for a very quick reply! @dnephin thanks for the detailed documentation!