Why do I get these disk watermark exceeded errors and those gateway errors on a newly created 3-node cluster?

Hello,

Why do I get the following warnings (?) on a new 3-node CrateDB cluster? I’m just getting started and the data disks are almost empty, I didn’t create any tables yet.

I also set gateway.expected_nodes and gateway.recover_after_nodes to 3 as recommended.

loadgenerator001@eris:~$ source /home/loadgenerator001/venv/cratedb/bin/activate
(cratedb) loadgenerator001@eris:~$ crash --verbose --hosts 192.168.239.30 --username energytix
Password:
+----------------------------+--------------+---------+-----------+---------+
| server_url                 | node_name    | version | connected | message |
+----------------------------+--------------+---------+-----------+---------+
| http://192.168.239.30:4200 | crate-dn-001 | 4.6.3   | TRUE      | OK      |
+----------------------------+--------------+---------+-----------+---------+
CONNECT OK
CLUSTER CHECK OK
TYPES OF NODE CHECK OK
cr> select id, node_id, description from sys.node_checks order by id, node_id;
+----+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id | node_id                | description                                                                                                                                                                                                                                    |
+----+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|  1 | FM5C1UbYRJSyPnRQytu25Q | The value of the cluster setting 'gateway.expected_nodes' must be equal to the maximum/expected number of master and data nodes in the cluster. https://cr8.is/d-node-check-1                                                                  |
|  1 | GyekXiMsT6GKJEZ4NrdOYA | The value of the cluster setting 'gateway.expected_nodes' must be equal to the maximum/expected number of master and data nodes in the cluster. https://cr8.is/d-node-check-1                                                                  |
|  1 | ZonWXPbIToyAB8tePVErgw | The value of the cluster setting 'gateway.expected_nodes' must be equal to the maximum/expected number of master and data nodes in the cluster. https://cr8.is/d-node-check-1                                                                  |
|  2 | FM5C1UbYRJSyPnRQytu25Q | The value of the cluster setting 'gateway.recover_after_nodes' needs to be greater than half of the maximum/expected number of nodes and equal or less than the maximum/expected number of nodes in the cluster. https://cr8.is/d-node-check-2 |
|  2 | GyekXiMsT6GKJEZ4NrdOYA | The value of the cluster setting 'gateway.recover_after_nodes' needs to be greater than half of the maximum/expected number of nodes and equal or less than the maximum/expected number of nodes in the cluster. https://cr8.is/d-node-check-2 |
|  2 | ZonWXPbIToyAB8tePVErgw | The value of the cluster setting 'gateway.recover_after_nodes' needs to be greater than half of the maximum/expected number of nodes and equal or less than the maximum/expected number of nodes in the cluster. https://cr8.is/d-node-check-2 |
|  3 | FM5C1UbYRJSyPnRQytu25Q | If any of the "expected nodes" recovery settings are set, the value of 'gateway.recover_after_time' must not be zero. Otherwise the "expected nodes" setting wouldn't have any effect. https://cr8.is/d-node-check-3                           |
|  3 | GyekXiMsT6GKJEZ4NrdOYA | If any of the "expected nodes" recovery settings are set, the value of 'gateway.recover_after_time' must not be zero. Otherwise the "expected nodes" setting wouldn't have any effect. https://cr8.is/d-node-check-3                           |
|  3 | ZonWXPbIToyAB8tePVErgw | If any of the "expected nodes" recovery settings are set, the value of 'gateway.recover_after_time' must not be zero. Otherwise the "expected nodes" setting wouldn't have any effect. https://cr8.is/d-node-check-3                           |
|  5 | FM5C1UbYRJSyPnRQytu25Q | The high disk watermark is exceeded on the node. The cluster will attempt to relocate shards to another node. Please check the node disk usage. https://cr8.is/d-node-check-5                                                                  |
|  5 | GyekXiMsT6GKJEZ4NrdOYA | The high disk watermark is exceeded on the node. The cluster will attempt to relocate shards to another node. Please check the node disk usage. https://cr8.is/d-node-check-5                                                                  |
|  5 | ZonWXPbIToyAB8tePVErgw | The high disk watermark is exceeded on the node. The cluster will attempt to relocate shards to another node. Please check the node disk usage. https://cr8.is/d-node-check-5                                                                  |
|  6 | FM5C1UbYRJSyPnRQytu25Q | The low disk watermark is exceeded on the node. The cluster will not allocate new shards to the node. Please check the node disk usage. https://cr8.is/d-node-check-6                                                                          |
|  6 | GyekXiMsT6GKJEZ4NrdOYA | The low disk watermark is exceeded on the node. The cluster will not allocate new shards to the node. Please check the node disk usage. https://cr8.is/d-node-check-6                                                                          |
|  6 | ZonWXPbIToyAB8tePVErgw | The low disk watermark is exceeded on the node. The cluster will not allocate new shards to the node. Please check the node disk usage. https://cr8.is/d-node-check-6                                                                          |
|  7 | FM5C1UbYRJSyPnRQytu25Q | The flood stage disk watermark is exceeded on the node. Tables that reside on an affected disk on this node have been made read-only. Please check the node disk usage. https://cr8.is/d-node-check-7                                          |
|  7 | GyekXiMsT6GKJEZ4NrdOYA | The flood stage disk watermark is exceeded on the node. Tables that reside on an affected disk on this node have been made read-only. Please check the node disk usage. https://cr8.is/d-node-check-7                                          |
|  7 | ZonWXPbIToyAB8tePVErgw | The flood stage disk watermark is exceeded on the node. Tables that reside on an affected disk on this node have been made read-only. Please check the node disk usage. https://cr8.is/d-node-check-7                                          |
+----+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
SELECT 18 rows in set (0.016 sec)

That is strange. where is the data folder mounted to?
Does the crate user have sufficient rights?

The /data folders on all nodes are mounted on /dev/sdb1, a 500 GB disks. E.g. on the first node:

administrator@crate-dn-001:~$ df -hT | grep "/data"
/dev/sdb1      ext4      492G   73M  467G   1% /data

I ran the following command on all 3 nodes this morning:

sudo chown crate:crate /data

therefore:

administrator@crate-dn-001:~$ ls -ld /data/
drwxr-xr-x 3 crate crate 4096 Oct  1 13:49 /data/

I didn’t do anything about the crate Linux user, apparently it was created when I installed CrateDB Debian package using apt.

Is there anything else I should check?