CrateDB 3 Node Cluster: Replica Shards are missing

[BANDANA_PANDEY]

Hi,

We have cluster of three nodes with 6 shards. But on two nodes, replica shards are missing resulting in below error:

Error!

SQLParseException[Couldn't create execution plan from logical plan because of: the shard 4 of table [device] is not available:

Showing CrateDB cluster health as Critical

So, in order to resolve that we tried executing this command

 ALTER CLUSTER REROUTE RETRY FAILED

But, it didn’t do anything and still showing the same error.

Here is the output of select * from sys.allocations where current_state != 'STARTED' limit 100;

Any Help would be much appreciated.
Thanks

[proddata]

Could your share the decisions (2nd column)?

Seems like some primaries are also currently not allocated to any node.

[BANDANA_PANDEY]

@proddata here

[proddata]

Did one of the nodes crash or did you restart the cluster within the last 48h?
Could you share the logs for the nodes?