From b50d6bf1fd59d04b3219b98ad75229c76b66ca9b Mon Sep 17 00:00:00 2001 From: ieQu1 <99872536+ieQu1@users.noreply.github.com> Date: Thu, 1 Feb 2024 00:10:48 +0100 Subject: [PATCH 1/2] chore(ds): Change the license to Apache 2.0 Due to technicalities parts of the original code were licensed under BSL. In preparations for the public release of the feature, the license has been changed to Apache 2.0 --- apps/emqx_durable_storage/BSL.txt | 94 ------------------- apps/emqx_durable_storage/src/emqx_ds_app.erl | 12 +++ .../src/emqx_ds_bitmask_keymapper.erl | 14 ++- .../src/emqx_ds_builtin_db_sup.erl | 12 +++ .../src/emqx_ds_storage_layer_sup.erl | 12 +++ apps/emqx_durable_storage/src/emqx_ds_sup.erl | 12 +++ .../emqx_ds_storage_bitfield_lts_SUITE.erl | 12 +++ .../emqx_ds_message_storage_bitmask_shim.erl | 14 ++- .../test/props/payload_gen.erl | 16 ++++ 9 files changed, 102 insertions(+), 96 deletions(-) delete mode 100644 apps/emqx_durable_storage/BSL.txt diff --git a/apps/emqx_durable_storage/BSL.txt b/apps/emqx_durable_storage/BSL.txt deleted file mode 100644 index f0cd31c6f..000000000 --- a/apps/emqx_durable_storage/BSL.txt +++ /dev/null @@ -1,94 +0,0 @@ -Business Source License 1.1 - -Licensor: Hangzhou EMQ Technologies Co., Ltd. -Licensed Work: EMQX Enterprise Edition - The Licensed Work is (c) 2023 - Hangzhou EMQ Technologies Co., Ltd. -Additional Use Grant: Students and educators are granted right to copy, - modify, and create derivative work for research - or education. -Change Date: 2028-01-26 -Change License: Apache License, Version 2.0 - -For information about alternative licensing arrangements for the Software, -please contact Licensor: https://www.emqx.com/en/contact - -Notice - -The Business Source License (this document, or the “License”) is not an Open -Source license. However, the Licensed Work will eventually be made available -under an Open Source License, as stated in this License. - -License text copyright (c) 2017 MariaDB Corporation Ab, All Rights Reserved. -“Business Source License” is a trademark of MariaDB Corporation Ab. - ------------------------------------------------------------------------------ - -Business Source License 1.1 - -Terms - -The Licensor hereby grants you the right to copy, modify, create derivative -works, redistribute, and make non-production use of the Licensed Work. The -Licensor may make an Additional Use Grant, above, permitting limited -production use. - -Effective on the Change Date, or the fourth anniversary of the first publicly -available distribution of a specific version of the Licensed Work under this -License, whichever comes first, the Licensor hereby grants you rights under -the terms of the Change License, and the rights granted in the paragraph -above terminate. - -If your use of the Licensed Work does not comply with the requirements -currently in effect as described in this License, you must purchase a -commercial license from the Licensor, its affiliated entities, or authorized -resellers, or you must refrain from using the Licensed Work. - -All copies of the original and modified Licensed Work, and derivative works -of the Licensed Work, are subject to this License. This License applies -separately for each version of the Licensed Work and the Change Date may vary -for each version of the Licensed Work released by Licensor. - -You must conspicuously display this License on each original or modified copy -of the Licensed Work. If you receive the Licensed Work in original or -modified form from a third party, the terms and conditions set forth in this -License apply to your use of that work. - -Any use of the Licensed Work in violation of this License will automatically -terminate your rights under this License for the current and all other -versions of the Licensed Work. - -This License does not grant you any right in any trademark or logo of -Licensor or its affiliates (provided that you may use a trademark or logo of -Licensor as expressly required by this License). - -TO THE EXTENT PERMITTED BY APPLICABLE LAW, THE LICENSED WORK IS PROVIDED ON -AN “AS IS” BASIS. LICENSOR HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS, -EXPRESS OR IMPLIED, INCLUDING (WITHOUT LIMITATION) WARRANTIES OF -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND -TITLE. - -MariaDB hereby grants you permission to use this License’s text to license -your works, and to refer to it using the trademark “Business Source License”, -as long as you comply with the Covenants of Licensor below. - -Covenants of Licensor - -In consideration of the right to use this License’s text and the “Business -Source License” name and trademark, Licensor covenants to MariaDB, and to all -other recipients of the licensed work to be provided by Licensor: - -1. To specify as the Change License the GPL Version 2.0 or any later version, - or a license that is compatible with GPL Version 2.0 or a later version, - where “compatible” means that software provided under the Change License can - be included in a program with software provided under GPL Version 2.0 or a - later version. Licensor may specify additional Change Licenses without - limitation. - -2. To either: (a) specify an additional grant of rights to use that does not - impose any additional restriction on the right granted in this License, as - the Additional Use Grant; or (b) insert the text “None”. - -3. To specify a Change Date. - -4. Not to modify this License in any other way. diff --git a/apps/emqx_durable_storage/src/emqx_ds_app.erl b/apps/emqx_durable_storage/src/emqx_ds_app.erl index dcf353a99..552d6e044 100644 --- a/apps/emqx_durable_storage/src/emqx_ds_app.erl +++ b/apps/emqx_durable_storage/src/emqx_ds_app.erl @@ -1,5 +1,17 @@ %%-------------------------------------------------------------------- %% Copyright (c) 2020-2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. %%-------------------------------------------------------------------- -module(emqx_ds_app). diff --git a/apps/emqx_durable_storage/src/emqx_ds_bitmask_keymapper.erl b/apps/emqx_durable_storage/src/emqx_ds_bitmask_keymapper.erl index a3b65c7e6..11dcba3ea 100644 --- a/apps/emqx_durable_storage/src/emqx_ds_bitmask_keymapper.erl +++ b/apps/emqx_durable_storage/src/emqx_ds_bitmask_keymapper.erl @@ -1,5 +1,17 @@ %%-------------------------------------------------------------------- -%% Copyright (c) 2022-2023 EMQ Technologies Co., Ltd. All Rights Reserved. +%% Copyright (c) 2022-2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. %%-------------------------------------------------------------------- -module(emqx_ds_bitmask_keymapper). diff --git a/apps/emqx_durable_storage/src/emqx_ds_builtin_db_sup.erl b/apps/emqx_durable_storage/src/emqx_ds_builtin_db_sup.erl index 9df07eb18..68aa0ee90 100644 --- a/apps/emqx_durable_storage/src/emqx_ds_builtin_db_sup.erl +++ b/apps/emqx_durable_storage/src/emqx_ds_builtin_db_sup.erl @@ -1,5 +1,17 @@ %%-------------------------------------------------------------------- %% Copyright (c) 2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. %%-------------------------------------------------------------------- %% @doc Supervisor that contains all the processes that belong to a diff --git a/apps/emqx_durable_storage/src/emqx_ds_storage_layer_sup.erl b/apps/emqx_durable_storage/src/emqx_ds_storage_layer_sup.erl index 424b35133..136669ed2 100644 --- a/apps/emqx_durable_storage/src/emqx_ds_storage_layer_sup.erl +++ b/apps/emqx_durable_storage/src/emqx_ds_storage_layer_sup.erl @@ -1,5 +1,17 @@ %%-------------------------------------------------------------------- %% Copyright (c) 2022-2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. %%-------------------------------------------------------------------- -module(emqx_ds_storage_layer_sup). diff --git a/apps/emqx_durable_storage/src/emqx_ds_sup.erl b/apps/emqx_durable_storage/src/emqx_ds_sup.erl index e863e74ce..ed2c83c62 100644 --- a/apps/emqx_durable_storage/src/emqx_ds_sup.erl +++ b/apps/emqx_durable_storage/src/emqx_ds_sup.erl @@ -1,5 +1,17 @@ %%-------------------------------------------------------------------- %% Copyright (c) 2022-2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. %%-------------------------------------------------------------------- -module(emqx_ds_sup). diff --git a/apps/emqx_durable_storage/test/emqx_ds_storage_bitfield_lts_SUITE.erl b/apps/emqx_durable_storage/test/emqx_ds_storage_bitfield_lts_SUITE.erl index 03d86dd88..5878e9f7a 100644 --- a/apps/emqx_durable_storage/test/emqx_ds_storage_bitfield_lts_SUITE.erl +++ b/apps/emqx_durable_storage/test/emqx_ds_storage_bitfield_lts_SUITE.erl @@ -1,5 +1,17 @@ %%-------------------------------------------------------------------- %% Copyright (c) 2022-2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. %%-------------------------------------------------------------------- -module(emqx_ds_storage_bitfield_lts_SUITE). diff --git a/apps/emqx_durable_storage/test/props/emqx_ds_message_storage_bitmask_shim.erl b/apps/emqx_durable_storage/test/props/emqx_ds_message_storage_bitmask_shim.erl index ddc199a44..97b90fcb7 100644 --- a/apps/emqx_durable_storage/test/props/emqx_ds_message_storage_bitmask_shim.erl +++ b/apps/emqx_durable_storage/test/props/emqx_ds_message_storage_bitmask_shim.erl @@ -1,5 +1,17 @@ %%-------------------------------------------------------------------- -%% Copyright (c) 2020-2023 EMQ Technologies Co., Ltd. All Rights Reserved. +%% Copyright (c) 2020-2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. %%-------------------------------------------------------------------- -module(emqx_ds_message_storage_bitmask_shim). diff --git a/apps/emqx_durable_storage/test/props/payload_gen.erl b/apps/emqx_durable_storage/test/props/payload_gen.erl index b969c0043..d4d19b403 100644 --- a/apps/emqx_durable_storage/test/props/payload_gen.erl +++ b/apps/emqx_durable_storage/test/props/payload_gen.erl @@ -1,3 +1,19 @@ +%%-------------------------------------------------------------------- +%% Copyright (c) 2023-2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. +%%-------------------------------------------------------------------- + %% @doc This module provides lazy, composable producer streams that %% can be considered counterparts to Archiver's consumer pipes and %% therefore can facilitate testing From 139f5dc3bd3c4b58d536abfe578aafce7ccee71a Mon Sep 17 00:00:00 2001 From: ieQu1 <99872536+ieQu1@users.noreply.github.com> Date: Thu, 1 Feb 2024 00:14:08 +0100 Subject: [PATCH 2/2] chore(ds): Remove an obsolete document; superceded by ./README.md --- apps/emqx_durable_storage/IMPLEMENTATION.md | 119 -------------------- 1 file changed, 119 deletions(-) delete mode 100644 apps/emqx_durable_storage/IMPLEMENTATION.md diff --git a/apps/emqx_durable_storage/IMPLEMENTATION.md b/apps/emqx_durable_storage/IMPLEMENTATION.md deleted file mode 100644 index 33f02dfc4..000000000 --- a/apps/emqx_durable_storage/IMPLEMENTATION.md +++ /dev/null @@ -1,119 +0,0 @@ -# General concepts - -In the logic layer we don't speak about replication. -This is because we could use an external DB with its own replication logic. - -On the other hand, we introduce notion of shard right here at the logic layer. -This is because shared subscription logic needs to be aware of it to some extend, as it has to split work between subscribers somehow. - - -# Modus operandi - -1. Create a draft implementation of a milestone -2. Test performance -3. Test consistency -4. Goto 1 - -# Tables - -## Message storage - -Data is written every time a message matching certain pattern is published. -This pattern is not part of the logic layer spec. - -Write throughput: very high - -Data size: very high - -Write pattern: append only - -Read pattern: pseudoserial - -Number of records: O(total write throughput * retention time) - - -# Push vs. Pull model - -In push model we have replay agents iterating over the dataset in the shards. - -In pull model the client processes work with iterators directly and fetch data from the remote message storage instances via remote procedure calls. - -## Push pros: -- Lower latency: message can be dispatched to the client as soon as it's persisted -- Less worry about buffering - -## Push cons: -- Needs pushback logic -- It's not entirely justified when working with external DB that may not provide streaming API - -## Pull pros: -- No need for pushback: client advances iterators at its own tempo - -## Pull cons: -- 2 messages need to be sent over network for each batch being replayed. -- RPC is generally an antipattern - -# Invariants - -- All messages written to the shard always have larger sequence number than all the iterators for the shard (to avoid missing messages during replay) - - -# Parallel tracks - -## 0. Configuration - -This includes HOCON schema and an interface module that is used by the rest of the code (`emqx_ds_conf.erl`). - -At the early stage we need at least to implement a feature flag that can be used by developers. - -We should have safety measures to prevent a client from breaking down the broker by connecting with clean session = false and subscribing to `#`. - -## 1. Fully implement all emqx_durable_storage APIs - -### Message API - -### Session API - -### Iterator API - -## 2. Implement a filter for messages that has to be persisted - -We don't want to persist ALL messages. -Only the messages that can be replayed by some client. - -1. Persistent sessions should signal to the emqx_broker what topic filters should be persisted. - Scenario: - - Client connects with clean session = false. - - Subscribes to the topic filter a/b/# - - Now we need to signal to the rest of the broker that messages matching `a/b/#` must be persisted. - -2. Replay feature (separate, optional, under BSL license): in the configuration file we have list of topic filters that specify what topics can be replayed. (Lower prio) - - Customers can do this: `#` - - Include minimum QoS of messages in the config - -## 3. Replace current emqx_persistent_session with the emqx_durable_storage - -## 4. Garbage collection - -## 5. Tooling for performance and consistency testing - -At the first stage we can just use emqttb: - -https://github.com/emqx/emqttb/blob/master/src/scenarios/emqttb_scenario_persistent_session.erl - - -Consistency verification at the early stages can just use this test suite: - -`apps/emqx/test/emqx_persistent_session_SUITE.erl` - -## 6. Update rocksdb version in EMQX (low prio) - -## 9999. Alternative schema for rocksdb message table - -https://github.com/emqx/eip/blob/main/active/0023-rocksdb-message-persistence.md#potential-for-future-optimizations-keyspace-based-on-the-learned-topic-patterns - -Problem with the current bitmask-based schema: - -- Good: `a/b/c/#` - -- Bad: `+/a/b/c/d`