illegal_state_exception Error retrieving path nested_item inner_hits and top_hits error

[benwtrent]

Elasticsearch Version

Sometime in v8, present in current release

Installed Plugins

No response

Java Version

bundled

OS Version

any

Problem Description

Elasticsearch nested query with inner hits and top-hits aggregation will return illegal_state_exception.

Steps to Reproduce

This might even be able to reduced further, but the single doc with these fields is enough.

DELETE test

PUT test
{
    "mappings": {
        "properties": {
            "nested_item": {
                "type": "nested",
                "properties": {
                    "title": {
                        "type": "text"
                    },
                    "keyword3": {
                        "type": "keyword"
                    },
                    "content": {
                        "type": "text"
                    },
                    "number": {
                        "type": "long"
                    }
                }
            }
        }
    }
}

POST test/_doc
{
    "nested_item": [
        {
            "title": "foo"
        },
        {
            "keyword3": "foobar",
            "title": "foobar",
            "content": "foobar`foobar"
        },
        {
            "title": "baz| barz",
            "content": "foobar`foobar",
            "number": "2"
        }
    ]
}

POST test/_search
{
    "aggregations": {
        "nested_item": {
            "aggregations": {
                "nested_item_filter": {
                    "aggregations": {
                        "0": {
                            "aggregations": {
                                "01": {
                                    "top_hits": {}
                                }
                            },
                            "filter": {
                                "bool": {
                                    "should": []
                                }
                            }
                        }
                    },
                    "filter": {
                        "bool": {
                            "should": []
                        }
                    }
                }
            },
            "nested": {
                "path": "nested_item"
            }
        }
    },
    "query": {
        "nested": {
            "inner_hits": {},
            "path": "nested_item",
            "query": {
                "bool": {
                    "should": []
                }
            }
        }
    }
}

Logs (if relevant)

java.lang.IllegalStateException: Error retrieving path nested_item
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.SearchHit$NestedIdentity.extractSource(SearchHit.java:1097)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.fetch.FetchPhase.prepareNestedHitContext(FetchPhase.java:424)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.fetch.FetchPhase.prepareHitContext(FetchPhase.java:307)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:214)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:90)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:248)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:91)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.fetch.subphase.InnerHitsPhase.hitExecute(InnerHitsPhase.java:96)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.fetch.subphase.InnerHitsPhase$1.process(InnerHitsPhase.java:63)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:231)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:90)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:248)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:91)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.runFetchPhase(TopHitsAggregator.java:245)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.buildAggregation(TopHitsAggregator.java:201)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.metrics.MetricsAggregator.lambda$buildAggregations$0(MetricsAggregator.java:41)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.AggregatorBase.buildAggregations(AggregatorBase.java:343)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.metrics.MetricsAggregator.buildAggregations(MetricsAggregator.java:41)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:175)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForFixedBucketCount(BucketsAggregator.java:269)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator.buildAggregations(FiltersAggregator.java:212)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.AdaptingAggregator.buildAggregations(AdaptingAggregator.java:103)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:175)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForFixedBucketCount(BucketsAggregator.java:269)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregator.buildAggregations(FiltersAggregator.java:212)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.AdaptingAggregator.buildAggregations(AdaptingAggregator.java:103)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:175)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildAggregationsForSingleBucket(BucketsAggregator.java:307)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.bucket.nested.NestedAggregator.buildAggregations(NestedAggregator.java:129)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:162)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:48)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:380)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:414)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:403)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:367)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
  at org.apache.lucene.core@10.3.1/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
  at org.apache.lucene.core@10.3.1/org.apache.lucene.search.TaskExecutor.invokeAll(TaskExecutor.java:111)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:371)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:338)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:213)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.query.QueryPhase.executeQuery(QueryPhase.java:143)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:70)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:700)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:906)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$7(SearchService.java:739)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1076)
  at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
  at java.base/java.lang.Thread.run(Thread.java:1474)

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[benwtrent]

This code path is pretty opaque, I attached a debugger, and indeed, we somehow execute the following path where the nested object in the for loop has an offset == 1 where the nestedParsedSource.size == 1

This SMELLS like something isn't being reset correctly upstream.

public Source extractSource(Source root) {
            // Isolate the nested json array object that matches with nested hit and wrap it back into the same json
            // structure with the nested json array object being the actual content. The latter is important, so that
            // features like source filtering and highlighting work consistent regardless of whether the field points
            // to a json object array for consistency reasons on how we refer to fields
            Map<String, Object> rootSourceAsMap = root.source();
            Map<String, Object> nestedSourceAsMap = new HashMap<>();
            Map<String, Object> current = nestedSourceAsMap;
            for (SearchHit.NestedIdentity nested = this; nested != null; nested = nested.getChild()) {
                String nestedPath = nested.getField().string();
                current.put(nestedPath, new HashMap<>());
                List<Map<?, ?>> nestedParsedSource = XContentMapValues.extractNestedSources(nestedPath, rootSourceAsMap);
                if (nestedParsedSource == null) {
                    return Source.empty(root.sourceContentType());
                }
                if (nested.getOffset() > nestedParsedSource.size() - 1) {
                    throw new IllegalStateException("Error retrieving path " + this.field);
                }
                rootSourceAsMap = (Map<String, Object>) nestedParsedSource.get(nested.getOffset());
                if (nested.getChild() == null) {
                    current.put(nestedPath, rootSourceAsMap);
                } else {
                    Map<String, Object> next = new HashMap<>();
                    current.put(nestedPath, next);
                    current = next;
                }
            }
            return Source.fromMap(nestedSourceAsMap, root.sourceContentType());
        }

@romseygeek I realize this code is from YEARS ago ;). But wondering if you had any inspiration here.

[batcity]

Hello @benwtrent , was this patched in main or am I doing something wrong here, I wasn't able to recreate this bug using the main branch

Here's the script I'm running:

ELASTIC_USER="elastic"
ELASTIC_PASS=""
ES_HOST="https://localhost:9200"

# --- 1️⃣ Delete index 'test' if it exists ---
echo "Deleting index 'test'..."
curl -k -u $ELASTIC_USER:$ELASTIC_PASS -X DELETE "$ES_HOST/test"
echo -e "\n"

# --- 2 Create index 'test' with nested mappings ---
echo "Creating index 'test'..."
curl -k -u $ELASTIC_USER:$ELASTIC_PASS -X PUT "$ES_HOST/test" -H "Content-Type: application/json" -d '
{
    "mappings": {
        "properties": {
            "nested_item": {
                "type": "nested",
                "properties": {
                    "title": {
                        "type": "text"
                    },
                    "keyword3": {
                        "type": "keyword"
                    },
                    "content": {
                        "type": "text"
                    },
                    "number": {
                        "type": "long"
                    }
                }
            }
        }
    }
}'
echo -e "\n"

# --- 3 Insert a document into 'test' ---
echo "Indexing document into 'test'..."
curl -k -u $ELASTIC_USER:$ELASTIC_PASS -X POST "$ES_HOST/test/_doc" -H "Content-Type: application/json" -d '
{
    "nested_item": [
        {
            "title": "foo"
        },
        {
            "keyword3": "foobar",
            "title": "foobar",
            "content": "foobar`foobar"
        },
        {
            "title": "baz| barz",
            "content": "foobar`foobar",
            "number": "2"
        }
    ]
}'
echo -e "\n"

# --- 4 Perform nested search with aggregations ---
echo "Searching 'test' index with nested aggregations..."
curl -k -u $ELASTIC_USER:$ELASTIC_PASS -X POST "$ES_HOST/test/_search" -H "Content-Type: application/json" -d '
{
    "aggregations": {
        "nested_item": {
            "aggregations": {
                "nested_item_filter": {
                    "aggregations": {
                        "0": {
                            "aggregations": {
                                "01": {
                                    "top_hits": {}
                                }
                            },
                            "filter": {
                                "bool": {
                                    "should": []
                                }
                            }
                        }
                    },
                    "filter": {
                        "bool": {
                            "should": []
                        }
                    }
                }
            },
            "nested": {
                "path": "nested_item"
            }
        }
    },
    "query": {
        "nested": {
            "inner_hits": {},
            "path": "nested_item",
            "query": {
                "bool": {
                    "should": []
                }
            }
        }
    }
}'
echo -e "\n"

echo "All operations completed."

Here's the result I get locally:

Deleting index 'test'...
{"acknowledged":true}-e

Creating index 'test'...
{"acknowledged":true,"shards_acknowledged":true,"index":"test"}-e

Indexing document into 'test'...
{"_index":"test","_id":"PkAEIpoBVPTVynfBFPhx","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}-e

Searching 'test' index with nested aggregations...
{"took":53,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]},"aggregations":{"nested_item":{"doc_count":0,"nested_item_filter":{"doc_count":0,"0":{"doc_count":0,"01":{"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}}}}}}-e

All operations completed.

[benwtrent]

I have reconfirmed that this occurs on main.

[batcity]

Interesting, I must've done something wrong while trying to recreate it then. Apologies 😅

[mayya-sharipova]

I've studied this search request (simplified it a bit) , and I understood what is the problem.

POST test/_search
{
    "aggregations": {
        "nested_item": {
            "nested": {
                "path": "nested_item"
            },
            "aggregations": {
                "01": {
                    "top_hits": {}
                }
            }
        }
    },
    "query": {
        "nested": {
            "inner_hits": {},
            "path": "nested_item",
            "query": {
                "match_all": {}
            }
        }
    }
}

The way top_hits agg works is that it exercises query, and nested top_hits will retrieve nested docs from the one query returns. The problem is with inner_hits part of query, here with nested aggregation on top we get 2 levels of nestedness, which is incorrect.

---

One way is to treat this not as a bug, but a wrong user request. The user should either drop "inner_hits" part, or make agg not "nested":

POST test/_search
{
    "aggregations": {
        "nested_item": {
            "nested": {
                "path": "nested_item"
            },
            "aggregations": {
                "01": {
                    "top_hits": {}
                }
            }
        }
    },
    "query": {
        "nested": {
            "inner_hits": {},
            "path": "nested_item",
            "query": {
                "match_all": {}
            }
        }
    }
}


POST test/_search
{
    "aggregations": {
        "nested_item": {
            "nested": {
                "path": "nested_item"
            },
            "aggregations": {
                "01": {
                    "top_hits": {}
                }
            }
        }
    },
    "query": {
        "nested": {
            "path": "nested_item",
            "query": {
                "match_all": {}
            }
        }
    }
}

---

Another way is to fix the code not apply inner_hits part of a query if we have top_hits aggregation in already a part of nested. I've put a PR that fixes this issue.