Commit Graph

47 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi
ea8d54fd8b test: ensure atom exists in test module 2023-09-27 12:58:06 +02:00
firest
9e55ae240a feat(sso): add role into the result of login endpoints 2023-09-22 13:48:07 +08:00
Zaiming (Stone) Shi
99b60849b4 test: fix flaky test emqx_falpping_SUITE:t_conf_update_timer 2023-09-18 20:44:38 +02:00
Zaiming (Stone) Shi
7c2f87fabe test: merge broker and router boot modules 2023-09-06 21:36:16 +02:00
Zaiming (Stone) Shi
876d539336 Merge remote-tracking branch 'andrew/ft/EMQX-10713/unified-route-tab' into 0831-make-use-of-new-routing-table 2023-08-31 15:46:33 +02:00
Paulo Zulato
ee77976424 fix(kafka): avoid producer name collision between Kafka and AEH bridges
Fixes https://emqx.atlassian.net/browse/EMQX-10860
2023-08-30 14:48:10 -03:00
Paulo Zulato
0b86f04bae fix(kafka): fix template processing for header
Fixes https://emqx.atlassian.net/browse/EMQX-10846
2023-08-28 17:28:05 -03:00
Andrew Mayorov
e85789306b
chore(router): drop test-only has_routes/1
Seems that it's too much support work only for test purposes, where
`lookup_routes/1` is nearly as usable.
2023-08-28 14:07:46 +04:00
Paulo Zulato
84e68f8ed8
Merge pull request #11508 from paulozulato/fix-azure-msg-error-handling
fix(kafka): fix result handling when sending message with invalid header
2023-08-25 14:57:09 -03:00
Paulo Zulato
cb1e105e19
Merge pull request #11516 from paulozulato/fix-kafka-aeh-client-unique-id
Fix Kafka/AEH ClientId uniqueness
2023-08-25 10:14:44 -03:00
Paulo Zulato
535c7f8b43 fix(kafka): avoid ClientId collision between Kafka and Azure bridges
Fixes https://emqx.atlassian.net/browse/EMQX-10860
2023-08-24 18:46:22 -03:00
Thales Macedo Garitezi
0381ac0410 fix(kafka_producer): use correct timestamp template field
Fixes https://emqx.atlassian.net/browse/EMQX-10847
2023-08-24 14:20:42 -03:00
Paulo Zulato
60e6217496 fix(kafka): fix result handling when sending message with invalid header
Fixes https://emqx.atlassian.net/browse/EMQX-10846
2023-08-24 10:41:42 -03:00
Thales Macedo Garitezi
ffca581229 feat(kafka): add option to configure health check interval
Fixes https://emqx.atlassian.net/browse/EMQX-10781
2023-08-16 11:08:06 -03:00
Thales Macedo Garitezi
b6decf9592
Merge pull request #11402 from thalesmg/dynamic-kconsu-mqtt-topics-20230807
feat(kafka_consumer): add mqtt topic placeholder support
2023-08-07 15:10:29 -03:00
Thales Macedo Garitezi
5c8dc092a1 fix(http_bridge): don't attempt to convert headers to atoms
Fixes https://emqx.atlassian.net/browse/EMQX-10653
2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi
9900a32850 feat(kafka_consumer): add mqtt topic placeholder support
Fixes https://emqx.atlassian.net/browse/EMQX-10678
2023-08-07 11:42:34 -03:00
Thales Macedo Garitezi
5abe4bed88 feat: add Azure Event Hub Producer bridge
Fixes https://emqx.atlassian.net/browse/EMQX-10338
2023-07-27 09:22:39 -03:00
Thales Macedo Garitezi
6cd503865b fix(machine_boot): ensure emqx_bridge starts after its companion apps
We need to reverse the dependency of `emqx_bridge` and `emqx_bridge_*`, because the former
loads and starts bridges during its application startup.  If the individual bridge
application being loaded has not started with its dependencies, the supervision tree will
not be ready for that.
2023-07-20 13:11:44 -03:00
JimMoen
b089fba100
refactor: rm ee_bridge and ee_connector application 2023-07-07 12:25:37 +08:00
Paulo Zulato
f47cc2a458 feat(kafka): add support for kafka headers and ext headers
Fixes https://emqx.atlassian.net/browse/EMQX-9175
2023-06-21 18:27:39 -03:00
Paulo Zulato
62d3766726
Merge pull request #10645 from paulozulato/data-bridge-target-unavailable
Data bridge target unavailable
2023-06-21 18:19:23 -03:00
Paulo Zulato
8430ec673c feat(kafka): check whether target topic exists
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-21 15:00:35 -03:00
zhongwencool
07172e42f0 test: integer CI check failed 2023-06-20 08:39:23 +08:00
Thales Macedo Garitezi
73fe671dba test: rm unused var warnings 2023-06-12 15:19:47 -03:00
Zaiming (Stone) Shi
97850de524 Merge remote-tracking branch 'origin/release-51' into 0610-merge-release-51-to-master 2023-06-10 12:23:55 +02:00
Serge Tupchii
48ac942807 feat(emqx_bridge_kafka): add tcp keepalive option
Closes: EMQX-8725
2023-06-09 22:10:17 +03:00
Kjell Winblad
6f2271e9f0 test: add Kafka producer bridge test case for query_mode parameter 2023-06-08 16:27:04 +02:00
zhongwencool
8d8efe449e
Merge pull request #10896 from zhongwencool/api-keys-hot-conf-update
feat: api_key conf support hot conf
2023-06-02 15:58:45 +08:00
Thales Macedo Garitezi
83a7c800cc test(kafka_consumer): reduce flakiness
The crash reason can also be `{error, not_found}` depending on
timing...

```
=ERROR REPORT==== 31-May-2023::12:17:31.461490 ===
    exception: error
    key_path: [bridges,kafka_consumer,
               't_resource_manager_crash_before_subscriber_started-576460752303419630']
    module: emqx_conf_schema
    msg: change_config_crashed
    reason: {badmatch,{error,not_found}}
    stacktrace: [{emqx_resource_manager,create_and_return_data,5,
                     [{file,
                          "/emqx/apps/emqx_resource/src/emqx_resource_manager.erl"},
                      {line,119}]},
                 {emqx_bridge_resource,create,4,
                     [{file,
                          "/emqx/apps/emqx_bridge/src/emqx_bridge_resource.erl"},
                      {line,171}]},
                 {emqx_bridge,'-perform_bridge_changes/2-fun-1-',5,
                     [{file,"/emqx/apps/emqx_bridge/src/emqx_bridge.erl"},
                      {line,367}]},
                 {maps,fold_1,3,[{file,"maps.erl"},{line,411}]},
                 {emqx_bridge,perform_bridge_changes,2,
                     [{file,"/emqx/apps/emqx_bridge/src/emqx_bridge.erl"},
                      {line,354}]},
                 {emqx_bridge,post_config_update,5,
                     [{file,"/emqx/apps/emqx_bridge/src/emqx_bridge.erl"},
                      {line,229}]},
                 {emqx_config_handler,call_post_config_update,7,
                     [{file,"/emqx/apps/emqx/src/emqx_config_handler.erl"},
                      {line,398}]},
                 {emqx_config_handler,do_post_config_update,8,
                     [{file,"/emqx/apps/emqx/src/emqx_config_handler.erl"},
                      {line,336}]},
                 {emqx_config_handler,check_and_save_configs,7,
                     [{file,"/emqx/apps/emqx/src/emqx_config_handler.erl"},
                      {line,281}]},
                 {emqx_config_handler,handle_update_request,4,
                     [{file,"/emqx/apps/emqx/src/emqx_config_handler.erl"},
                      {line,195}]},
                 {emqx_config_handler,handle_call,3,
                     [{file,"/emqx/apps/emqx/src/emqx_config_handler.erl"},
                      {line,124}]},
                 {gen_server,try_handle_call,4,
                     [{file,"gen_server.erl"},{line,1149}]},
                 {gen_server,handle_msg,6,
                     [{file,"gen_server.erl"},{line,1178}]},
                 {proc_lib,init_p_do_apply,3,
                     [{file,"proc_lib.erl"},{line,240}]}]
    update_req: {{update,#{<<"authentication">> => <<"none">>,
                           <<"bootstrap_hosts">> =>
                               <<"toxiproxy.emqx.net:9292">>,
                           <<"connect_timeout">> => <<"5s">>,
                           <<"enable">> => true,
                           <<"kafka">> =>
                               #{<<"max_batch_bytes">> => <<"896KB">>,
                                 <<"max_rejoin_attempts">> => 5,
                                 <<"offset_commit_interval_seconds">> => 3,
                                 <<"offset_reset_policy">> => <<"latest">>},
                           <<"key_encoding_mode">> => <<"none">>,
                           <<"metadata_request_timeout">> => <<"5s">>,
                           <<"min_metadata_refresh_interval">> => <<"3s">>,
                           <<"ssl">> =>
                               #{<<"enable">> => false,
                                 <<"server_name_indication">> => <<"auto">>,
                                 <<"verify">> => <<"verify_none">>},
                           <<"topic_mapping">> =>
                               [#{<<"kafka_topic">> =>
                                      <<"t_resource_manager_crash_before_subscriber_started-576460752303419662">>,
                                  <<"mqtt_topic">> =>
                                      <<"mqtt/topic/-576460752303422911">>,
                                  <<"payload_template">> => <<"${.}">>,
                                  <<"qos">> => 0}],
                           <<"value_encoding_mode">> => <<"none">>}},
                 #{override_to => cluster}}
```
2023-05-31 10:19:55 -03:00
Zhongwen Deng
5729ea0426 feat: api_key conf support hot conf 2023-05-31 18:23:58 +08:00
Thales Macedo Garitezi
9c3f838e14
Merge pull request #10841 from thalesmg/kafka-validate-key-v50
feat({kafka,pulsar}_producer): add validation for empty message key when strategy = key_dispatch
2023-05-30 09:37:15 -03:00
Thales Macedo Garitezi
9b7e473cf6 feat(kafka_producer): add validation for empty message key when strategy = key_dispatch
Fixes https://emqx.atlassian.net/browse/EMQX-9979
2023-05-26 12:02:58 -03:00
Thales Macedo Garitezi
32e6213ce3 fix(resource_manager_sup): use one_for_one instead of simple_one_for_one
Using `simple_one_for_one` has a potential race condition issue where
we read the PID of the resource manager before trying to remove a
resource, and then that PID changes because it was either dead at
first, or it crashed and changed, and later we use this stale PID to
try to remove it from the supervisor.  Under such circumstances, the
restarting child might linger in the supervisor, leaking resources.

By using the resource ID itself as a child ID (and using `one_for_one`
restart strategy), we ensure the child is truly removed.
2023-05-25 18:07:43 -03:00
Thales Macedo Garitezi
cb34bc5c46 test(kafka_consumer): attempt to stabilize cluster tests
Example failure:
https://github.com/emqx/emqx/actions/runs/5070096710/jobs/9105822319#step:7:515

The attempt here is to setup the spy as early as possible, before the
bridge starts, so we avoid missing rebalancing events.
2023-05-25 16:38:09 -03:00
Thales Macedo Garitezi
0ca3f51503 fix(kafka): improve shutdown and health check logs during shutdown 2023-05-25 16:38:09 -03:00
Thales Macedo Garitezi
5df7314255 feat(kafka): ensure allocated resources are removed on failures 2023-05-25 16:38:09 -03:00
Zaiming (Stone) Shi
40e8d5d039 refactor: rename lib-ee/emqx_ee_conf to apps/emqx_enterprise 2023-05-22 14:51:27 +02:00
某文
6056b0e7a8 fix: ensure atom key for emqx_config:get 2023-05-11 17:40:03 +08:00
Thales Macedo Garitezi
79cf5cad19
Merge pull request #10378 from thalesmg/pulsar-producer-e50
feat: implement Pulsar Producer bridge (e5.0)
2023-04-25 18:01:43 -03:00
Thales Macedo Garitezi
e9fde12913 test: attempt to fix flaky test
Example failure: https://github.com/emqx/emqx/actions/runs/4789177314/jobs/8517116154#step:7:503
2023-04-24 15:19:33 -03:00
Thales Macedo Garitezi
26883eec02 test(kafka): fix innocuous test assertion 2023-04-24 09:28:36 -03:00
Stefan Strigler
53871e3a2c fix: stale ref to emqx_json after rebase 2023-04-14 16:30:27 +02:00
Stefan Strigler
062ce5f819 refactor: rename emqx_map_lib to emqx_utils_maps 2023-04-14 13:41:34 +02:00
Stefan Strigler
9c11bfce80 refactor: rename emqx_misc to emqx_utils 2023-04-14 13:41:27 +02:00
Stefan Strigler
f8e9e54393 refactor: move emqx_json to emqx_utils_json 2023-04-14 13:31:27 +02:00
Thales Macedo Garitezi
871ee90b3e refactor(kafka_bridge): move kafka bridge into its own app
Fixes https://emqx.atlassian.net/browse/EMQX-9481
2023-04-12 13:54:45 -03:00