0b86f04bae
fix(kafka): fix template processing for header
...
Fixes https://emqx.atlassian.net/browse/EMQX-10846
2023-08-28 17:28:05 -03:00
9bded07834
feat: cluster purge
...
Fixes https://emqx.atlassian.net/browse/EMQX-10763
2023-08-28 10:23:20 -03:00
f3467f44b2
chore: bump hstreamdb_erl driver to `0.4.5+v0.16.1`
2023-08-28 17:28:50 +08:00
84e68f8ed8
Merge pull request #11508 from paulozulato/fix-azure-msg-error-handling
...
fix(kafka): fix result handling when sending message with invalid header
2023-08-25 14:57:09 -03:00
0381ac0410
fix(kafka_producer): use correct timestamp template field
...
Fixes https://emqx.atlassian.net/browse/EMQX-10847
2023-08-24 14:20:42 -03:00
60e6217496
fix(kafka): fix result handling when sending message with invalid header
...
Fixes https://emqx.atlassian.net/browse/EMQX-10846
2023-08-24 10:41:42 -03:00
61536b5362
Merge pull request #11494 from paulozulato/fix-kinesis-validators
...
fix(kinesis): set validators for static constraints
2023-08-22 18:28:35 -03:00
20b46091f9
fix(kinesis): set validators for static constraints
...
Fixes https://emqx.atlassian.net/browse/EMQX-10833
2023-08-22 14:42:44 -03:00
45e2e687e5
fix(greptimedb_bridge): avoid double-parsing write syntax during probe and remove code duplication
...
Fixes https://emqx.atlassian.net/browse/EMQX-10843
2023-08-22 12:42:48 -03:00
13ebcd6290
Merge pull request #11478 from JimMoen/hstreamdb-tls-support
...
Hstreamdb tls support
2023-08-22 10:40:54 +08:00
bd5680db00
Merge pull request #11461 from thalesmg/bridge-dryrun-flex-timeout-r52-20230816
...
fix(bridge): make dryrun health check timeout more malleable
2023-08-21 14:49:46 -03:00
999988cab4
Merge pull request #11401 from kjellwinblad/kjell/fix/mongo_date_format/EMQX-10727
...
fix: rule SQL mongo_date function should return a string in test mode
2023-08-21 16:20:59 +02:00
f45b5fac6f
chore: unhidden hstreamdb bridge for e5.2.0
2023-08-21 10:40:51 +08:00
01c9095982
Merge pull request #11394 from zmstone/0804-upgrade-wolff-to-1.7.7
...
chore: upgrade Kafka client wolff to 1.7.7
2023-08-20 10:14:13 +02:00
afffdbbaa1
fix(kinesis): replace default payload template
...
Fixes https://emqx.atlassian.net/browse/EMQX-10766
2023-08-17 17:00:14 -03:00
ebecbd1545
fix(bridge): make dryrun health check timeout more malleable
...
Fixes https://emqx.atlassian.net/browse/EMQX-10773
- Makes the timeout for probing a bridge more malleable to account for differences between
each database.
- Increases GCP PubSub Consumer default health check timeout to account for GCP
slowness/throttling.
2023-08-17 09:21:19 -03:00
e3572eb758
Merge pull request #11459 from thalesmg/kafka-health-check-interval-r52-20230816
...
feat(kafka): add option to configure health check interval
2023-08-17 09:04:30 -03:00
7469222a17
Merge remote-tracking branch 'upstream/master' into 0817-e5.2.0-code-freeze
2023-08-17 08:16:04 +02:00
740680d54b
Merge pull request #11444 from paulozulato/fix-kinesis-msg-access-denied
...
fix(kinesis): return error message on access denied
2023-08-16 18:02:46 -03:00
a64386ef82
fix(kinesis): return error message on access denied
...
Fixes https://emqx.atlassian.net/browse/EMQX-10764
2023-08-16 16:04:23 -03:00
ffca581229
feat(kafka): add option to configure health check interval
...
Fixes https://emqx.atlassian.net/browse/EMQX-10781
2023-08-16 11:08:06 -03:00
4e80d669b0
fix(influxdb_bridge): avoid double-parsing write syntax during probe
...
Fixes https://emqx.atlassian.net/browse/EMQX-10771
2023-08-15 17:05:57 -03:00
82b8538041
feat(gcp_producer): add support for defining message attributes and ordering key
...
Fixes https://emqx.atlassian.net/browse/EMQX-10652
2023-08-14 10:33:17 -03:00
9ca9c65af2
Merge pull request #11367 from paulozulato/feat-gcp-devices
...
feat(gcp-iot): port GCP IoT Core compatibility layer from e4.4
2023-08-10 09:44:55 -03:00
a85c948e23
feat(gcp-iot): port GCP IoT Core compatibility layer from e4.4
...
Fixes https://emqx.atlassian.net/browse/EMQX-10341
2023-08-09 10:48:25 -03:00
9900a32850
feat(kafka_consumer): add mqtt topic placeholder support
...
Fixes https://emqx.atlassian.net/browse/EMQX-10678
2023-08-07 11:42:34 -03:00
19d091eef1
docs: add changelog entry
2023-08-07 16:28:39 +02:00
ff88e508a6
docs: add changelog
2023-08-07 10:39:04 +02:00
b24a9d5343
chore(ldap-authz): update apps version && changes
2023-08-04 18:14:09 +08:00
7055eafb91
fix(ldap): fix license date and some minor problems
2023-08-04 11:01:13 +08:00
b4a21b6afb
feat(ldap): update changes
2023-08-03 18:56:46 +08:00
a3be1529c6
fix(schema): ensure enterprise module is correctly loaded
...
Fixes https://emqx.atlassian.net/browse/EMQX-10654
Despite loading the application in `nodetool`, we need to invoke `:module_info()` to force
the module to be actually loaded, especially when it's called in `call_hocon` to generate
the initial configurations. Otherwise, it'll fail to list all the bridge schemas.
2023-07-28 16:27:43 -03:00
399f849f7b
feat(rabbitmq_bridge): add TLS support
...
Fixes https://emqx.atlassian.net/browse/EMQX-10605
2023-07-27 15:52:46 -03:00
5ac01c9b85
Merge pull request #11360 from id/0727-sync-release-51-to-master
2023-07-27 16:58:09 +02:00
cbfca8c043
chore: merge master into release-51
2023-07-27 15:19:57 +02:00
5abe4bed88
feat: add Azure Event Hub Producer bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-10338
2023-07-27 09:22:39 -03:00
c7bf3ece34
Merge pull request #11338 from thalesmg/fix-pgsql-error-r51-20230724
...
fix(postgres_bridge): fix table existence check and handle sync_required
2023-07-25 10:53:03 -03:00
7a16ff4f04
fix(postgres_bridge): fix table existence check and handle sync_required
...
Fixes https://emqx.atlassian.net/browse/EMQX-10629
During health checking, we check whether tables in the SQL statement
exist. Such check was done by asking the backend to parse the
statement using a named prepared statements. Concurrent health checks
could then result in the error:
```erlang
{error,{error,error,<<"42P05">>,duplicate_prepared_statement,<<"prepared statement \"get_status\" already exists">>,[{file,<<"prepare.c">>},{line,<<"451">>},{routine,<<"StorePreparedStatement">>},{severity,<<"ERROR">>}]}}
```
This could lead to an inconsistent state in the driver process, which
would crash later when a message from the backend (`READY_FOR_QUERY`, "idle"):
```
2023-07-24T13:05:58.892043+00:00 [error] Generic server <0.2134.0> terminating. Reason: {'module could not be loaded',[{undefined,handle_message,[90,<<"I">>,...
```
Added calls to `epgsql:sync/1` for functions that could return
`{error, sync_required}`.
Also, redundant calls to `parse2` were removed to reduce the number of requests.
2023-07-25 09:15:14 -03:00
69f4275871
fix(oracle): fix return error checking on table validation
...
Fixes https://emqx.atlassian.net/browse/EMQX-10622
2023-07-24 14:50:38 -03:00
5ffd7f2a73
chore: remove the hstreamdb changes due to we hide it in e5.1.1
2023-07-24 20:14:47 +08:00
ba9dcbcff0
chore: style
2023-07-24 17:58:48 +08:00
ffcd04bc9f
docs: add change log
2023-07-24 17:58:48 +08:00
31cf05ec66
Merge pull request #11316 from paulozulato/fix-oracle-pool-size
...
fix(oracle): fix Pool Size parameter retrieval
2023-07-21 17:51:21 -03:00
85ab97970f
fix(oracle): fix Pool Size parameter retrieval
...
Fixes https://emqx.atlassian.net/browse/EMQX-10599
2023-07-21 11:02:27 -03:00
243b8f5b67
chore: merge 'upstream/master' into v5.1.2
2023-07-21 13:25:46 +02:00
e76ba760c9
fix(oracle): discard nested tokens when checking table
...
Fixes https://emqx.atlassian.net/browse/EMQX-10597
2023-07-19 15:03:49 -03:00
b7f0f7b32d
chore: merge master into release-51
2023-07-19 09:28:05 +02:00
4f4a813f04
fix(tdengine): update the default template and the changelog
2023-07-18 10:31:18 +00:00
484519dcf9
feat(kinesis): implement Amazon Kinesis Producer bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-10474
Fixes https://emqx.atlassian.net/browse/EMQX-10475
2023-07-17 11:51:42 -03:00
c746c85739
chore: update changes/ee/fix-11266.en.md
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-07-14 09:18:30 +08:00
66baea2ba0
chore: update changes
2023-07-13 08:02:36 +00:00
541d03a0ba
refactor(schema_registry): refactor schema registry app and modules
...
Fixes https://emqx.atlassian.net/browse/EMQX-10361
- Moves `lib-ee/emqx_ee_schema_registry` to `apps/emqx_schema_registry`.
- Removes the `_ee_` segment from module names.
- Exceptions are the table names which are kept to avoid backwards incompatibilities.
2023-07-12 09:24:15 -03:00
cacfb51280
fix(emqx_machine): add emqx_ee_schema_registry to the reboot apps list
...
As emqx_ee_schema_registry uses Mria tables (schema_registry_shard),
a node joining a cluster needs to restart this application in order to
restart relevant Mria shard processes.
2023-07-12 09:23:28 -03:00
ae59a26659
Merge pull request #11223 from JimMoen/fix-influxdb-float-value-trans
...
fix: influxdb float serialization error
2023-07-10 23:04:01 +08:00
e9f1d7f2bf
fix: influxdb float serialization error
2023-07-10 13:53:11 +08:00
7de26a1776
feat(authz): use extensible map format for actions in authz rules
...
* support authorization on retain, qos fields
* refactored authz tests heavily
2023-07-07 19:40:34 +03:00
9bb159cf1e
fix(rebalance): fix changelog and version
2023-07-07 18:04:11 +03:00
d01eee7fe4
chore: update changes && app version
2023-07-07 18:22:54 +08:00
1587f038a5
feat: hstreamdb bridge with batch query
2023-07-07 12:25:36 +08:00
afe6989625
chore: update changes
2023-07-07 11:04:18 +08:00
353cc065db
Merge pull request #11175 from thalesmg/fix-mysql-start-error
...
fix(mysql_bridge): make nxdomain a 400 API error
2023-07-06 08:26:12 -03:00
33cb29efdb
Merge pull request #11169 from kjellwinblad/kjell/improved_sparkplugb/EMQX-10351
...
feat: add Sparkplug encode and decode functions to the rule engine
2023-07-04 07:24:06 +02:00
b5e91422de
fix: changelog file ending
2023-07-03 18:21:39 +02:00
c216dfd96b
fix(mysql_bridge): make nxdomain a 400 API error
...
Fixes https://emqx.atlassian.net/browse/EMQX-10460
2023-06-30 16:10:09 -03:00
3c8f591cc4
fix(mongodb): hide `topology.pool_size` and fix it at 1
...
Fixes https://emqx.atlassian.net/browse/EMQX-10408
From an old conversation with @kjellwinblad:
> There are 3 pool_sizes
> - The buffer workers pool size, just exposed here: https://github.com/emqx/emqx/pull/9742
> - The topology.pool_size, which controls the pool size for the poolboy_pool in Kjell's
diagram (on mongodb's side).
> - The pool_size from emqx_connector_mongo:mongo_fields that controls the ecpool pool
size (on EMQX's side).
> So we actually want to set topology.pool_size = 1 and hide it from users.
2023-06-30 10:17:33 -03:00
4d45de2939
docs: changelog entry for new sparkplug decode and encode functions
2023-06-30 10:28:23 +02:00
515a7d0c80
docs: add changelog entries for MySQL refactoring
2023-06-27 17:23:04 +02:00
0a95749f86
docs: add changelog entries for Redis refactoring
2023-06-27 10:24:44 +02:00
f4f068faaa
Merge pull request #11079 from paulozulato/feat-kafka-headers
...
feat(kafka): add support for kafka headers and ext headers
2023-06-22 12:04:32 -03:00
2f00cf7f84
Merge pull request #11107 from thalesmg/fix-mongo-health-check-reason-master
...
fix(mongo): return health check failure reason
2023-06-22 09:30:34 -03:00
7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer
...
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
2023-06-22 09:17:45 -03:00
f47cc2a458
feat(kafka): add support for kafka headers and ext headers
...
Fixes https://emqx.atlassian.net/browse/EMQX-9175
2023-06-21 18:27:39 -03:00
18f0510353
fix(mongo): return health check failure reason
...
Fixes https://emqx.atlassian.net/browse/EMQX-10335
2023-06-21 15:09:37 -03:00
8430ec673c
feat(kafka): check whether target topic exists
...
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-21 15:00:35 -03:00
b442910ff1
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-10281
2023-06-19 16:04:12 -03:00
e42cc58694
Merge pull request #11050 from emqx/release-51
...
chore: sync release-51 back to master
2023-06-15 09:45:31 +08:00
36d6350880
Merge pull request #11031 from paulozulato/fix-influx-wrong-username
...
fix(influxdb): check authentication
2023-06-14 14:41:50 -03:00
a573b9b38c
fix(influxdb): check authentication
...
Checks authentication on bridge start and get status. Also, handle
authentication error when sending message.
Fixes https://emqx.atlassian.net/browse/EMQX-10213
2023-06-14 11:27:39 -03:00
f57ff802e4
Merge pull request #11038 from thalesmg/fix-pulsar-get-status-r51
...
fix(pulsar_producer): do not return `disconnected` when checking status (r5.1)
2023-06-14 08:32:25 +08:00
25b0e31035
fix(kafka_producer): do not return disconnected when checking status (r5.1)
...
Fixes https://emqx.atlassian.net/browse/EMQX-10279
Related: https://github.com/emqx/emqx/pull/11038
Since wolff client has its own replayq that lives outside the management of the buffer
workers, we must not return disconnected status for such bridge: otherwise, the resource
manager will eventually kill the producers and data may be lost.
2023-06-13 13:54:57 -03:00
b609792a90
fix(pulsar_producer): do not return `disconnected` when checking status (r5.1)
...
Fixes https://emqx.atlassian.net/browse/EMQX-10278
Since Pulsar client has its own replayq that lives outside the management of the buffer
workers, we must not return disconnected status for such bridge: otherwise, the resource
manager will eventually kill the producers and data may be lost.
2023-06-13 11:44:45 -03:00
415424654b
chore: update changes
2023-06-13 16:13:40 +08:00
c11011d857
Merge pull request #11024 from thalesmg/pulsar-cosmetic-connecting-check-r51
...
feat(pulsar): retry health check a bit before returning (r5.1)
2023-06-12 13:00:54 -03:00
741c1f091e
fix(pulsar): update pulsar -> 0.8.3
...
Fixes https://emqx.atlassian.net/browse/EMQX-10229
See https://github.com/emqx/pulsar-client-erl/pull/59
Fixes a case_clause error that could arise from race conditions.
2023-06-12 10:29:40 -03:00
db5d14d5bf
feat(pulsar): retry health check a bit before returning (r5.1)
...
Fixes https://emqx.atlassian.net/browse/EMQX-10228
This is a cosmetic fix for the Pulsar Producer bridge health check status.
Pulsar connection process is asynchronous, therefore, when a bridge of this type is
created or updated (which is the same as stopping and re-creating it), the immediate
status will be connecting because it’s indeed still connecting. The bridge will connect
very soon afterwards (assuming there are no true network/config issue), but having to
refresh the UI to see the new status and/or seeing the resource alarm might annoy users.
This workaround adds a few retries to account for that effect to reduce the probability of
seeing the `connecting` state on such happy-paths.
2023-06-12 10:26:07 -03:00
12b73ba1c0
Merge remote-tracking branch 'origin/master' into release-51
2023-06-12 10:35:47 +02:00
e8ccdb8d0f
Merge pull request #10998 from zmstone/0609-no-batch-for-mongodb
...
fix(mongodb): hide batch_size for mongodb resource
2023-06-11 21:26:12 +02:00
654a9dcab5
Merge pull request #11006 from zmstone/0610-fix-quic-default-certs-path
...
fix(quic): environment variables as cert file prefix for quic listener
2023-06-11 11:52:56 +02:00
ddef751527
fix(mongodb): hide batch_size for mongodb resource
...
MongoDB connector currently does not support batching
so the batch_size option has no effect.
However we cannot remove the field, so we choose to hide it from
schema
2023-06-11 11:08:58 +02:00
97850de524
Merge remote-tracking branch 'origin/release-51' into 0610-merge-release-51-to-master
2023-06-10 12:23:55 +02:00
9f135d1f2b
fix(quic): environment variables as cert file prefix for quic listener
2023-06-10 12:23:20 +02:00
48ac942807
feat(emqx_bridge_kafka): add tcp keepalive option
...
Closes: EMQX-8725
2023-06-09 22:10:17 +03:00
720a84d48b
fix(kafka): fix timeout field for some kafka fields
...
Change type of fields 'Partition Count Refresh Interval' and 'Offset
Commit Interval' to avoid accepting values larger than allowed.
Fixes https://emqx.atlassian.net/browse/EMQX-10196
Fixes https://emqx.atlassian.net/browse/EMQX-10199
2023-06-09 13:38:04 -03:00
c6eb1af82c
docs: add changelog entry
2023-06-09 16:17:13 +02:00
b60dbbc792
docs: add change log entry for Kafka query mode parameter
2023-06-08 16:27:04 +02:00
c1cf2365c2
Merge remote-tracking branch 'origin/master' into release-51
2023-06-08 12:30:02 +02:00
0b95bc1c1b
Merge pull request #10907 from HJianBo/refactor-on-stop-simple
...
feat: refactored some bridges to avoid leaking resources part.2
2023-06-08 09:37:31 +08:00
54f7941329
fix(rebalancing): add changelog
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-06-07 21:37:08 +03:00
260fae296b
feat(gcp_pubsub): generate jwt tokens on demand without workers (5.1)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9603
Rather than relying on a JWT worker to produce and refresh tokens, we
could just produce then on demand when pushing the messages to GCP
PubSub. That can generate a bit of extra work (as multiple processes
might realize it’s time to refresh the JWT and do so), but that
shouldn’t be much. In return, we avoid any possibility of not having
a fresh JWT when pushing messages.
2023-06-06 13:19:24 -03:00
0f808345e6
Merge pull request #10924 from JimMoen/refactor_influxdb_on_stop
...
feat: refactor influxdb connector to to avoid resources leaking
2023-06-05 18:02:15 +08:00
Paulo Zulato