Upgraded Crate from 4.2.7 to 4.3.4, confusing issue with mapping?

CrateDB
1

Hi there,
Our team is trying to upgrade from 4.2.7 to 4.3.4, using Docker images.
We verified that there’s no breaking changes in 4.3 which would affect us, as well as that a direct upgrade path is available.

I read through Upgrade from 4.2.6 to >= 4.3.0 problem to see if this solved the answer, and the query to verify when tables were updated shows that all of our tables originated in 4.x, so no 3.x → 4.x upgrade issues.

We do have a object column which is dynamic in nature, and are getting the following error:

Mapping definition for [columnName] has unsupported parameters: [fields : \{keyword={type=keyword, ignore_above=256}}]

Has keyword been removed as a valid parameter? Reading through Elasticsearch docs, it sounds like there were some changes introduced in v7 onwards, but also this seems like it might only be tangentially related.

The SHOW CREATE TABLE for this particular top-level object shows:

"object" OBJECT(DYNAMIC) AS (
      -- other columns not shown here
      "columnName" TEXT INDEX USING FULLTEXT
) NOT NULL

Being new to Crate, I do not know if I’ve provided the level of detail needed for analysis, but hope this is enough to get the ball rolling, so to speak.

Thanks for any assistance.

2

@Azadi_Bogolubov

Could you give us some more context on how those tables were created and with which version.
We tried to replicate this (also see java - CRATE db tables are causing an error when upgrading to CRATE:4.3.4 from CRATE:4.2.7 - Stack Overflow), but with the information given this is really not that easy and until now we could not.

There was some change in the underlying architecture between 4.1 to 4.2, but I am not aware for any such changes in 4.3.

Can you replicate that behaviour with a new table?
Which version was the problematic table created with?

3

Hey there @proddata,
Thanks for the response.

When I run:
SELECT DISTINCT version['created'], version['upgraded'] FROM "information_schema"."tables" WHERE table_schema not IN ('sys','information_schema','pg_catalog') limit 100;

I get:

version[‘created’] version[‘upgraded’]
4.0.12 4.2.7
4.1.5 4.2.7
4.2.7 NULL

So I think the tables were created in 4.0, 4.1, and 4.2 respectively. Everything has migrated up to 4.2.7, and when I attempt to upgrade even to 4.3.0, it gives the same errors.

There is a note about breaking changes in the 4.3.0 release notes, but none seem to point to this.

We were able to do a workaround as follows, to quote my colleague who is also working this problem.

> I followed the steps mentioned here at: System information — CrateDB: Reference on official documentation of crate DB to re-index the table which is causing the issue, created a new snapshot of that table(snapshot name: archived_snap_einvoice_data_2021_09_21_1) in crate:4.2.7, and restored it into crate:4.3.4 which perfectly worked fine.

The common thread really does appear to be the TEXT INDEX USING FULLTEXT index.
Here’s a more complete structure from SHOW CREATE TABLE:

CREATE TABLE IF NOT EXISTS "schema_name"."table_name" (
      "topLevelObject" OBJECT(DYNAMIC) AS (
            -- other fields      
            "problemField" TEXT INDEX USING FULLTEXT,
            -- other fields
      ) NOT NULL
)

I have a request in to management of what other information I can provide, without disclosing sensitive information on a public forum as well.

I hope this is useful information.

The problem arises when loading a new Docker container with a newer version (4.3.0+), and then restoring the snapshot.
In production, as I understand, it is running in a Kubernetes pod.

Also, just realized upon looking at the username, that this SO ticket was submitted by my colleague on a different team, so sorry for the duplicate posting, as it was mentioned by my team architect to raise the issue and was unaware of the existence of the other ticket, as I did not see it in Google search results last week in spite of it’s existence.

Thanks for any further clarification.

4

Do you see in the logs, which tables are affected? i.e. can this be singled out to a specific version? e.g. 4.0.12 or 4.1.5?

As said before we tried to replicate this unsuccessfully and with the limited information this is looking for needles in a haystack.

Honestly speaking, that typically is something that would be catched and clarified within our customer success / engineering teams who would accompany upgrades

5

Hey there, thanks for the response. Our team has been discussing this, and the solution you provided on the StackOverflow article is likely the route we are gonna be taking going forward.

It appears that our issue is arising from the use of dynamic objects, and the subsequent indexing on them causing an issue. We have multiple tables causing this issue, and the log each time simply says what was provided in the initial post.

Definitely not knocking on the engineering teams about the upgrade, cause as a fellow engineer, I realize how much work goes into these types of upgrades.

Going to consider this issue resolved with the notes from the StackOverflow article.

The version we first see this in is 4.3.0, upgrading from 4.2.7.

Thanks for your assistance in this matter, and I hope you have a wonderful day!

1 Like