Logstash jdbc将外连接作为子文档(Logstash jdbc left outer join as subdocuments)

我正在使用Logstash jdbc插件将MySQL数据导入ElasticSearch。 由于左外连接,我最终得到一个'父行'的多个'子行'。 假设1个用户有1个或更多文档。 我尝试group_concat文档的文本,然后按用户ID分组以保留每个用户1行。

但是,MySQL的group_concat的长度限制为1024 ...



I'm using the Logstash jdbc plugin to get MySQL data into ElasticSearch. Due to a left outer join I end up with multiple 'child rows' for a single 'parent row'. Say 1 user has 1 or more documents. I tried to group_concat the text of the documents and then group by by the user id to retain 1 row per user.

However, MySQL's group_concat has a length limit of 1024...

Does anyone know a solution to overcome the group_concat altogether and deal with left outer joins as nested documents?




SET SESSION group_concat_max_len = 102400

或您认为必要的长度( 官方文档中的更多信息)

我在复杂的OUTER JOIN中使用它来创建许多子对象/文档,到目前为止它工作得非常好。

The default value is 1024, but you may increase the allowed length for GROUP_CONCAT using

SET SESSION group_concat_max_len = 102400

or whatever length you deem necessary (more info in the official docs)

I'm using that in complex OUTER JOINs which create many child objects/documents, and it works pretty fine so far.