A backup job is reported running on select * from sys.jobs. Despite killing the job via kill the job is still reported running. New backup jobs do not start, as they also report.
Backing off execute_stmt_with_backoff(...) for 0.2s (crate.client.exceptions.ProgrammingError: SQLActionException[ConcurrentSnapshotExecutionException: [bakup_repo] a snapshot is already running])
cratedb: 4.1.5
select * from sys.jobs where stmt like '%SNAPSHOT%' order by started desc limit 100;
+--------------------------------------+--------------------------------------------------------------------------------------+---------------+---------------------------------------------------------------------------------------+----------+
| id | node | started | stmt | username |
+--------------------------------------+--------------------------------------------------------------------------------------+---------------+---------------------------------------------------------------------------------------+----------+
| a4faccac-d08f-476d-a464-6ba1f613bf66 | {"id": "IU0WEqrLQOS2oeetnTSQsw", "name": "crate-f7f920b9eda840f2a3f61a6c4d83fd33-0"} | 1593077039244 | select * from sys.jobs where stmt like '%SNAPSHOT%' order by started desc limit 100 | crate |
| 0479e903-520d-4f0d-b287-4174dbc04325 | {"id": "CIdmvmHtS_KmelR32IVH4A", "name": "crate-f7f920b9eda840f2a3f61a6c4d83fd33-2"} | 1592898444707 | CREATE SNAPSHOT "backup_repo"."20200623074724" ALL | system |
| | | | WITH ("wait_for_completion" = TRUE, "ignore_unavailable" = TRUE) | |
+--------------------------------------+--------------------------------------------------------------------------------------+---------------+---------------------------------------------------------------------------------------+----------+
Trying to kill …
r> kill '0479e903-520d-4f0d-b287-4174dbc04325';
KILL OK, 0 rows affected (0.002 sec)
cr> select * from sys.jobs where stmt like '%SNAPSHOT%' order by started desc limit 100;
+--------------------------------------+--------------------------------------------------------------------------------------+---------------+---------------------------------------------------------------------------------------+----------+
| id | node | started | stmt | username |
+--------------------------------------+--------------------------------------------------------------------------------------+---------------+---------------------------------------------------------------------------------------+----------+
| 5fcc6618-b1b1-4c2d-a578-288e60f502ee | {"id": "IU0WEqrLQOS2oeetnTSQsw", "name": "crate-f7f920b9eda840f2a3f61a6c4d83fd33-0"} | 1593077169444 | select * from sys.jobs where stmt like '%SNAPSHOT%' order by started desc limit 100 | crate |
| 0479e903-520d-4f0d-b287-4174dbc04325 | {"id": "CIdmvmHtS_KmelR32IVH4A", "name": "crate-f7f920b9eda840f2a3f61a6c4d83fd33-2"} | 1592898444707 | CREATE SNAPSHOT "cloud_backup_20200618154859"."20200623074724" ALL | system |
| | | | WITH ("wait_for_completion" = TRUE, "ignore_unavailable" = TRUE) | |
+--------------------------------------+--------------------------------------------------------------------------------------+---------------+---------------------------------------------------------------------------------------+----------+
SELECT 2 rows in set (0.020 sec)
How could the stale backup be deleted/reset?
regards,
Walter