04adb65c09
refactor(ds): zone -> shard
...
Also bump erlang-rocksdb version
2023-06-22 10:20:18 +02:00
8ac0bba958
test(ds): verify preserve / restore works with stored iterators
2023-06-22 10:20:18 +02:00
d2065e0c1b
feat(ds): allow to preserve / restore iterators in the db
...
So that we could guarantee replay consistency / availability under
the assumption that nodes may be restarted or even lost occasionally.
2023-06-22 10:20:18 +02:00
cbded9af39
feat(ds): Make `create_generation` safer against bad input
2023-06-22 10:20:18 +02:00
83d1606d00
feat(ds): Make iteration fully generation-aware
2023-06-22 10:20:18 +02:00
bf9d57f6a9
chore(ds): Simplify iteration-related typespecs
2023-06-22 10:20:18 +02:00
cf6a5e1643
feat(ds): Allow to create new storage generations
2023-06-22 10:20:18 +02:00
f80e906553
fix(ds): Correct typespec
2023-06-22 10:20:18 +02:00
f5a7b49f57
feat(ds): Wire iteration options up to the app config
2023-06-22 10:20:18 +02:00
2bf8a07b05
test(ds): Split unit tests off into a full-fledged suite
2023-06-22 10:20:18 +02:00
cf70184ebd
test(ds): Provide more general `keymapper_info/1`
2023-06-22 10:20:18 +02:00
2f385022cd
chore(ds): Rename testsuite to reflect test subject better
2023-06-22 10:20:18 +02:00
d875fa49d3
feat(ds): Enable periodic iterator refresh
...
This might be helpful during replays taking multiple tens of seconds so
that underlying iterators won't hold onto in-memory / on-disk data
structures for too long, preventing rocksdb from recycling them.
2023-06-22 10:20:18 +02:00
7f408da251
refactor(ds): Use inline functions instead of macros
2023-06-22 10:20:07 +02:00
a11e75d189
fix(ds): clear bitmask of topic filter tail containing wildcards
2023-06-22 10:16:55 +02:00
ac0935ef91
test(ds): Proptest that iteration is exhaustive
...
Compare iteration results against what an extremely simplified model
produces.
2023-06-22 10:16:45 +02:00
60e3070328
test(ds): Scale up number of messages per topic in proptests
2023-06-22 10:16:41 +02:00
82a7292851
test(ds): Use `_build/test/proper` as a scratch dir for testruns
2023-06-22 10:16:32 +02:00
440fecd268
test(ds): Add a proptest on iterator preserve / restore
...
Which verifies that preservation and restoration of iterators does not
affect the outcome of an iteration (under the precondition that the
state of database is constant during an iteration).
2023-06-22 10:16:23 +02:00
1f5425975f
feat(ds): add an ability to preserve and restore iterators
...
This will allow to persist iteration state and to periodically recreate
iterators during long replays.
2023-06-22 10:16:20 +02:00
94e8dbfe7f
test(ds): Move proptests into a separate module
...
Following conventions. Also add few proptests on keyspace filters.
2023-06-22 10:16:16 +02:00
4c643aa7d5
refactor(ds): Introduce keyspace filter concept
...
So we could conveniently test it separately.
2023-06-22 10:16:10 +02:00
c5826cee56
refactor(ds): rename `compute_hash_*` → `compute_topic_*`
2023-06-22 10:16:05 +02:00
85d3783ff1
chore(ds): Drop few unused macrodefinitions
2023-06-22 10:15:56 +02:00
a0bcdb5104
chore(ds): Attempt to make `compute_next_seek`'s logic clearer
2023-06-22 10:15:49 +02:00
f1e39d977a
fix(ds): Fix dialyzer warnings
2023-06-22 10:15:47 +02:00
adcbf40d27
refactor(ds): tau -> epoch
2023-06-22 10:15:44 +02:00
120d4e66ae
refactor(ds): Factor out configuration to a separate module
2023-06-22 10:15:42 +02:00
0bbc5ecb32
refactor(ds): Introduce bitwise_concat function
2023-06-22 10:15:40 +02:00
b5bb77dd58
feat(ds): Add metadata storage and supervision tree
2023-06-22 10:15:37 +02:00
5e30a5d3dd
feat(ds): Implement keyspace partitioning across time
2023-06-22 10:15:32 +02:00
8c9c9cc669
feat(ds): Allow to specify message store options
...
* Keymapper
* Column family name + DB options
* DB write / read options
2023-06-22 10:15:25 +02:00
5b6c4d766b
test(ds): Add basic property tests
2023-06-22 10:15:20 +02:00
a1e29e4e24
refactor(ds): Improve function naming
2023-06-22 10:15:03 +02:00
cd12338c3f
feat(ds): Smoke tests for wildcard iterator scans
2023-06-22 10:14:46 +02:00
9c1cd4911d
feat(ds): Implement iterator next
2023-06-22 10:14:43 +02:00
27b925405b
test(ds): Add testcase
2023-06-22 10:14:41 +02:00
691a27883d
feat(ds): Initial commit
2023-06-22 10:14:37 +02:00
f47cc2a458
feat(kafka): add support for kafka headers and ext headers
...
Fixes https://emqx.atlassian.net/browse/EMQX-9175
2023-06-21 18:27:39 -03:00
62d3766726
Merge pull request #10645 from paulozulato/data-bridge-target-unavailable
...
Data bridge target unavailable
2023-06-21 18:19:23 -03:00
62b832be45
Merge pull request #11118 from fix/EMQX-9964/bump-hocon
...
chore: bump hocon to 0.39.10
2023-06-21 23:13:35 +02:00
86d787eced
chore: bump hocon to 0.39.10
...
Which comes with a fix for slightly more user-friendly validation error
messages.
2023-06-21 21:25:43 +02:00
18f0510353
fix(mongo): return health check failure reason
...
Fixes https://emqx.atlassian.net/browse/EMQX-10335
2023-06-21 15:09:37 -03:00
8430ec673c
feat(kafka): check whether target topic exists
...
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-21 15:00:35 -03:00
decfd6df2b
feat(buffer_worker): log expired message count
...
Fixes https://emqx.atlassian.net/browse/EMQX-10165
```
iex(emqx@127.0.0.1)38> 2023-06-21T11:09:35.569404-03:00 [info] msg: buffer_worker_dropped_expired_messages, mfa: emqx_resource_buffer_worker:log_expired_messge_count/1, line: 982, expired_count: 900, resource_id: <<"bridge:webhook:webhook">>, worker_index: 3
```
2023-06-21 14:38:51 -03:00
7cf8a6c892
chore: bump app vsns
2023-06-21 16:36:51 +02:00
5fa87091c4
Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master
2023-06-21 16:08:41 +02:00
7382c6e6e1
Merge pull request #11103 from savonarola/0620-update-erlcloud
...
chore(deps): update erlcloud and remove unnecessary overrides
2023-06-21 15:07:45 +03:00
f50d7334d6
Merge pull request #11077 from zhongwencool/fix-crash-ip-port-type
2023-06-21 17:36:59 +08:00
5f47649d4d
chore: downgrade log severity level of 'not_ready' nodes to info
2023-06-21 09:45:45 +02:00
045ed340dd
fix(emqx_conf_app): fix release version detect during cluster conf sync
2023-06-21 09:28:46 +02:00
0463828e84
refactor(gcp_pubsub): transform connector into opaque client
2023-06-20 15:27:42 -03:00
1d791d7a8c
fix(resource): validate maximum worker pool size
...
Fixes https://emqx.atlassian.net/browse/EMQX-10297
2023-06-20 14:26:42 -03:00
ffce6fefa8
docs: note that connector state must be immutable to avoid errors
2023-06-20 13:50:53 -03:00
dddccfdc5c
chore: bump to v5.1.0 official cut
2023-06-20 17:09:17 +02:00
728656d4f6
chore: bump to e5.1.0 official cut
2023-06-20 17:08:18 +02:00
ba96c725e5
feat: add time-based pull for safety
2023-06-20 11:36:35 -03:00
57460e83b9
test(fix): rm unused var warnings
2023-06-20 11:15:37 -03:00
2ac2d4c037
refactor: addressing review comments
2023-06-20 11:15:13 -03:00
54d96488a0
chore(deps): update erlcloud and remove unnecessary overrides
2023-06-20 16:46:23 +03:00
b588b88a60
chore: change autoclean interval default value to 24h
...
There is currently a bug in OTP causing external backends to
fail when merge schema (mnesia_schema:change_storage_type/3).
To avoid forcing users to clean up the DB after the node is
stopped for a short while (5m before this change), we increase
the default auto-clean interval to 24 hours.
This PR also has the autoclean config removed from examples.
2023-06-20 15:43:37 +02:00
8f595cbeca
Merge pull request #11098 from lafirest/feat/doc-example
...
chore: add examples of how to configure
2023-06-20 20:09:44 +08:00
f95787d229
test: make possible to test core/replica locally
2023-06-20 12:42:00 +02:00
e5803a61f1
fix(management): Don't wait for an empty shard
2023-06-20 10:08:41 +02:00
bb2bf059ac
chore: modify the build to copy examples into the release
2023-06-20 14:53:14 +08:00
36ee872361
chore: hide plugins from conf load cli
2023-06-20 10:34:09 +08:00
856de78698
chore: remove the is_list guard
2023-06-20 08:39:23 +08:00
093cdab838
chore: to_integer to make sure integer is converted
2023-06-20 08:39:23 +08:00
07172e42f0
test: integer CI check failed
2023-06-20 08:39:23 +08:00
a4be9b8281
chore: bump hocon to 0.39.9
2023-06-20 08:39:23 +08:00
45963b6a95
fix: ip_port schema type crash
2023-06-20 08:39:23 +08:00
ead511a128
chore: bump to 5.1.0-rc.1
2023-06-19 23:19:27 +02:00
86b78981c4
Merge pull request #11092 from ieQu1/mria-0.5.5
...
chore(mria): Bump version to 0.5.5
2023-06-19 23:13:36 +02:00
13746c2cdf
fix(resource): check status when (re)starting a resource
...
Fixes https://emqx.atlassian.net/browse/EMQX-10290
2023-06-19 18:01:02 -03:00
01d758e4c0
fix(bpapi): fix `query_mode` type(spec)
...
The previous spec was wrong due to a mismatch between type declaration and usage in specs:
it would never be `dynamic`, and it was lacking the `simple_async`, `simple_sync` and
`no_queries` constructors.
2023-06-19 16:04:12 -03:00
b442910ff1
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-10281
2023-06-19 16:04:12 -03:00
bb49482529
refactor(gcp_pubsub): move logging from connector to bridge
2023-06-19 15:59:00 -03:00
dabefdb01b
fix(gcp_pubsub): redact request when errors occur
2023-06-19 15:59:00 -03:00
7f850f7499
fix(resource): fix `query_mode/0` type and usage
2023-06-19 15:59:00 -03:00
a1690b3684
chore(gcp_pubsub): add stricter compile/xref flags
2023-06-19 15:59:00 -03:00
5375421954
refactor(gcp_pubsub): split connector into producer and reusable parts
2023-06-19 15:59:00 -03:00
5f10936091
feat(oracle): check whether target table exists
...
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 15:22:27 -03:00
c9a2ddf98c
feat(mysql): check whether target table exists
...
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 15:22:27 -03:00
f82c56ae16
chore(mria): Bump version to 0.5.5
2023-06-19 19:28:17 +02:00
cc939d2bda
Merge pull request #11087 from zhongwencool/bump-v5.1.0.alpha.8
...
chore: bump to v5.1.0.alpha.8
2023-06-19 23:00:53 +08:00
b0a6c18917
chore: bump vsn to 5.1.0-alpha.8
2023-06-19 22:59:05 +08:00
9454af9a8b
feat(postgresql): check whether target table exists
...
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 11:12:10 -03:00
af7b7ebf7c
Merge pull request #11088 from fix/EMQX-10068/nicer-doc-tags
...
fix(ft-api): make schema tags look nicer in the docs
2023-06-19 16:00:04 +02:00
7f1e8baa76
Merge pull request #11078 from thalesmg/fix-pmap-alias-master
...
fix(nolink_apply): avoid sending late replies to caller
2023-06-19 10:32:43 -03:00
ae5846be29
fix(ft-api): make schema tags look nicer in the docs
2023-06-19 13:35:21 +02:00
1f1d61cef6
Merge pull request #11084 from zhongwencool/hide-plugins-from-cli
...
chore: hide plugins from conf load cli
2023-06-19 19:11:52 +08:00
64612cab14
chore: bump to v5.1.0.alpha.8
2023-06-19 18:22:51 +08:00
6efb07f6eb
fix: crash on emqx_ctl listeners
2023-06-19 18:12:33 +08:00
a578d6fa1d
chore: release e5.1.0-alpha.8
2023-06-19 17:19:03 +08:00
8b20a70344
chore: hide plugins from conf load cli
2023-06-19 15:57:40 +08:00
803cd6c812
Merge pull request #11082 from emqx/release-51
...
sync release-51 to master
2023-06-19 15:12:37 +08:00
30ac6c4e48
Merge pull request #11074 from JimMoen/fix-share-sub-nl-protocol-error
...
fix: shared-sub with nl sub-option should cause protocol error
2023-06-19 10:39:08 +08:00
59ed8798c3
Merge pull request #11026 from kjellwinblad/kjell/fix/rule_engine/div_mod_strange/EMQX-10216
...
fix: rule engine different behavior for div and mod
2023-06-17 07:20:37 +02:00
13d9f5c3e7
Merge pull request #11053 from fix/EMQX-10231/cluster-view
...
fix(ft-fs): use `emqx:running_nodes()` as default cluster view
2023-06-17 00:54:24 +02:00
50e7d5d2ec
fix(nolink_apply): avoid sending late replies to caller
...
Due to race conditions, it's possible that the caller to `pmap`/`nolink_apply` might
receive a late reply.
e.g. when a timeout occurred while resource manager was checking a resource's health:
```
19:18:23.084 [error] [data: ..., event_data: {#Reference<0.3247872820.3887857670.131018>, {:normal, [false, true, true, true, true, true]}}, event_type: :info, msg: :ignore_all_other_events, state: :connected]
```
Using an alias and also checking for the race condition in the `after` block (like
[`gen`](a76bf63197/lib/stdlib/src/gen.erl (L270-L277)
2023-06-16 17:03:39 -03:00
8801ae9e5a
Merge pull request #11004 from paulozulato/fix-rewrite-wildcard
...
fix(rewrite): avoid wildcards on destination topic on publish
2023-06-16 09:57:46 -03:00
178711f742
fix: shared-sub with nl sub-option should cause protocol error
2023-06-16 18:19:23 +08:00
5f7e938b20
Merge pull request #11070 from JimMoen/use-mria-autoclean
...
fix: use mria.cluster_autoclean
2023-06-16 14:56:52 +08:00
c4e7129cb6
fix: use mria.cluster_autoclean and mria.cluster_autoheal
2023-06-16 14:04:24 +08:00
b49b80d36c
fix(emqx_schema): use non negative integer type for 'depth' SSL option
...
Closes: EMQX-10276
2023-06-16 10:59:39 +08:00
b2f4dbd232
Merge pull request #11067 from paulozulato/chore-bump-influx
...
chore(influxdb): bump version for influxdb client
2023-06-15 20:16:28 -03:00
e127df3912
chore(influxdb): bump version for influxdb client
2023-06-15 17:53:08 -03:00
9b56f116f4
Merge pull request #11062 from zhongwencool/log-file-rename
2023-06-15 20:51:52 +08:00
3406a5d916
Merge pull request #11063 from zhongwencool/bump-to-alpha.7
...
chore: bump to e5.1.0-alpha.7
2023-06-15 19:22:37 +08:00
fc1d78134a
test: replace 'to' with 'path'
2023-06-15 19:13:51 +08:00
9060a2dc05
chore: bump to e5.1.0-alpha.7
2023-06-15 19:05:13 +08:00
e6c2da8d19
feat: rename log.file.to to log.file.path
2023-06-15 18:24:21 +08:00
84a5d0c3c1
fix: delete a default listener then recreate it, it's disabled
2023-06-15 17:19:11 +08:00
0adbb0deec
feat: try to save config when post_config_update failed not in init mfa status
2023-06-15 17:19:11 +08:00
c160301dbe
fix: don't log enoent error
2023-06-15 17:06:09 +08:00
7dade3a52c
fix: tls_certfile_gc notice log don't print abspath
2023-06-15 17:06:09 +08:00
64bbe21209
fix(emqx_schema): use non negative integer type for 'depth' SSL option
...
Closes: EMQX-10276
2023-06-15 11:22:58 +03:00
903d8d899e
fix: .js files are required by dashboard as static
...
revert parts of 8c1334c421
2023-06-15 16:10:16 +08:00
c28346d67b
chore: bump ecql for obfuscate sensitive data to avoid leakage
2023-06-15 10:35:18 +08:00
e42cc58694
Merge pull request #11050 from emqx/release-51
...
chore: sync release-51 back to master
2023-06-15 09:45:31 +08:00
8c7c0877fd
fix(rewrite): avoid wildcards on destination topic on publish
...
Fixes https://emqx.atlassian.net/browse/EMQX-9247
2023-06-14 20:02:03 -03:00
6dc281345c
test(ft-fs): run API tests with replicant nodes
2023-06-15 00:50:18 +03:00
acd6e5635b
fix(ct): adapt cluster setup helper to replicant nodes
...
Before this change, trying to start cluster consisting of cores +
replicants resulted in error joining replicant node to the cluster.
2023-06-15 00:50:15 +03:00
52354bf58a
fix(ft-fs): use `emqx:running_nodes()` as default cluster view
...
When iterating over complete exports cluster-wide.
2023-06-14 22:54:56 +03:00
36d6350880
Merge pull request #11031 from paulozulato/fix-influx-wrong-username
...
fix(influxdb): check authentication
2023-06-14 14:41:50 -03:00
41004e86da
Merge pull request #10959 from qzhuyan/dev/william/remove-depre-quic-config
...
chore: remove deprecated QUIC listener config
2023-06-14 23:01:00 +08:00
bd269e259e
Merge pull request #11047 from zhongwencool/dashboard-listeners-deprecate-enable
...
feat: deprecated dashboard.listeners.name.enable
2023-06-14 22:58:39 +08:00
7882755114
chore: bump cowboy to 2.9.2
...
cowboy 2.9.2 uses ranch 1.8.1-emqx that fixes incompatible TLS options
2023-06-14 17:35:17 +03:00
4425942030
fix(emqx_schema): move tls version gap validation to emqx_schema
2023-06-14 17:29:56 +03:00
6d88cd7a20
fix(tls): fix incompatible tls options and issue with tls version gap
2023-06-14 17:29:56 +03:00
a573b9b38c
fix(influxdb): check authentication
...
Checks authentication on bridge start and get status. Also, handle
authentication error when sending message.
Fixes https://emqx.atlassian.net/browse/EMQX-10213
2023-06-14 11:27:39 -03:00
9209e0bed5
test: delete dashboard.listeners.xx.enable
2023-06-14 20:51:17 +08:00
4b82d4d927
feat: deprecated dashboard.listeners.name.enable
2023-06-14 20:51:17 +08:00
267053cc35
Merge pull request #11045 from zhongwencool/hide-zones-authn-in-listeners
...
feat: hide zone/authn in listeners and remove listeners's authn api
2023-06-14 20:32:27 +08:00
bf62cdc3e3
Merge pull request #11037 from sstrigler/EMQX-10174-e-5-1-0-reconnect-does-not-show-any-errors-for-apache-io-tdb-or-webhook-or-kafka
...
fix(emqx_connector): report errors in on_start handler
2023-06-14 13:20:47 +02:00
f6df52020c
test: conf_cli failed with replace
2023-06-14 18:43:41 +08:00
1beb588207
fix(ft): give more failproof example in README
...
Without that `ssl` defaults to being disabled, and interacting
with AWS S3 with such configuration consistently times out.
2023-06-14 13:15:00 +03:00
9a0856fd6b
fix: authn/authz's keys should be binary
2023-06-14 18:00:36 +08:00
bd33dcb049
feat: remove zone from /clients api
2023-06-14 17:11:20 +08:00
0d6d441f4c
test(emqx_connector): start/stop test for webhook bridge
2023-06-14 09:56:50 +02:00
121cd9124a
chore: improve show_keys output
2023-06-14 15:53:55 +08:00
7668753f50
feat: hide zone/authn in listeners and remove /listeners/:id/authentication api
2023-06-14 15:17:12 +08:00
80e2c2c955
fix(http): avoid crash if the old sample data dont contain `live_connections`
...
In version 5.1.0, a new metric called "live_connections" was added. However, we try to merge
some data from the Mria disk table, which may include old data formats.
Therefore, maps:get/3 is needed to avoid crashes.
2023-06-14 15:14:20 +08:00
8ce0380704
chore: bump vsn to e5.1.0-alpha.6
2023-06-14 12:49:02 +08:00
86d3b702d7
Merge pull request #11042 from zhongwencool/max-connections-string
...
fix: api_listener crash when setting max_connnections as string
2023-06-14 12:46:40 +08:00
feb08b1181
chore: update 11042 changelog
2023-06-14 11:40:20 +08:00
091c49277f
fix: api_listener crash when setting max_connnections as string
2023-06-14 11:26:41 +08:00
26ad06b9ca
chore: update changes
2023-06-14 10:49:17 +08:00
c4c32f5032
chore: cover stomp mountpoint tests
2023-06-14 10:49:17 +08:00
4212d90672
fix(stomp): unmount topic prefix in delivering proccess
2023-06-14 10:49:17 +08:00
ieQu1