[WIP] Add index UUID invariant check in routing node

[gmarouli]

In an effort to explore the possibility of atomic index renames, we introduced an invariant to the routing node to ensure that the unique uuids we encounter match the ones the shards are associated with.

This invariant check triggered a lot of test failures because a lot of tests fallback to _na_ as index UUID and they do not set unique UUID. In this PR:

• We update these tests to create unique uuids
• We replace the string literal _na_ with the constant when it refers to an index UUID so it's easier to track it.

[nielsbauman]

Thinking out loud: I am worried this will have a noticeable performance impact. It looks like we construct this at least on every routing allocation (code path starts here). I am worried that we have to loop over all the indices on that node again (we're also doing that on the line above). Perhaps we could combine this line with the one above and create one loop that fills both sets?

I think the memory overhead of the Set is not that bad, as it's only the set itself that we hold in memory - the UUID strings themselves are already elsewhere on the heap. What do you think?

[gmarouli]

That is a good point, what we can do is to run it in the invariant loop. This will make it slower for tests but it won't affect production code that runs without the assertions

[nielsbauman]

This is exactly what I had in mind, thanks!

[nielsbauman]

Ah, I just now noticed that we also use this field here:

elasticsearch/server/src/main/java/org/elasticsearch/indices/cluster/IndicesClusterStateService.java

Line 547 in 8dba961

} else if (localRoutingNode == null || localRoutingNode.hasIndex(index.getUUID()) == false) {

(on our WIP branch for the rename). We'll have to keep an eye out to see if we still need that, to determine whether we need this field after all.

[nielsbauman]

I scanned through all the changes and they LGTM, but there are obviously a lot of changes, so I'm also partially relying on a passing CI here.