Define configuration architecture #37

New Issue

glyph · 2021-11-25T09:49:02Z

glyph commented

2021-11-25 09:49:02 +00:00

As mentioned briefly in PR#36.

Our web application has a configuration file (Rocket.toml) and we have also been considering implementing a Peach.toml configuration file for all other Peach-related settings.

The path to an alternate Rocket configuration file (.toml) can be defined using the ROCKET_CONFIG environment variable (Configuration: Default Provider docs).

@notplants raised the idea of using Figment to manage our configuration landscape. I'd like to learn more about Figment and how it works. My initial thought is to avoid it if possible and aim for One Config To Rule Them All.

As mentioned briefly in [PR#36](https://git.coopcloud.tech/PeachCloud/peach-workspace/pulls/36). Our web application has a configuration file (`Rocket.toml`) and we have also been considering implementing a `Peach.toml` configuration file for all other Peach-related settings. The path to an alternate Rocket configuration file (`.toml`) can be defined using the `ROCKET_CONFIG` environment variable ([Configuration: Default Provider docs](https://rocket.rs/v0.5-rc/guide/configuration/#default-provider)). @notplants raised the idea of using [Figment](https://docs.rs/crate/figment/0.10.6) to manage our configuration landscape. I'd like to learn more about Figment and how it works. My initial thought is to avoid it if possible and aim for One Config To Rule Them All.

glyph commented

2021-12-10 09:22:37 +00:00

Here's an example toml file as a starting point for further config discussions:

# PeachCloud Configuration

title = "PeachCloud Configuration Settings"

# Enable this setting to disable all non-sbot-related services
standalone_pub_mode = false

# go-sbot settings
sbot_enabled = true
sbot_port = 8008
sbot_network_key = "1KHLiKZvAvjbY1ziZEHMXawbCEIM6qwjCDm3VYRan/s="

# Web user-interface settings
# Can be "light" or "dark"
web_ui_mode = "dark"

# Web server settings
web_server_ip = "127.0.0.1"
web_server_port = 3000
deployment_mode = "prod"

# JSON-RPC server settings
jsonrpc_ip = "127.0.0.1"
jsonrpc_port = 5110
jsonrpc_network = true
jsonrpc_oled = true

# OLED menu system settings
menu_enabled = true

# Network interface settings
wifi_interface = "wlan0"
ap_interface = "ap0"

# Network data traffic thresholds
data_warning_threshold = 0
data_cutoff_threshold = 0
enable_traffic_warning = false
enable_traffic_cutoff = false

Here's an example `toml` file as a starting point for further config discussions: ```toml # PeachCloud Configuration title = "PeachCloud Configuration Settings" # Enable this setting to disable all non-sbot-related services standalone_pub_mode = false # go-sbot settings sbot_enabled = true sbot_port = 8008 sbot_network_key = "1KHLiKZvAvjbY1ziZEHMXawbCEIM6qwjCDm3VYRan/s=" # Web user-interface settings # Can be "light" or "dark" web_ui_mode = "dark" # Web server settings web_server_ip = "127.0.0.1" web_server_port = 3000 deployment_mode = "prod" # JSON-RPC server settings jsonrpc_ip = "127.0.0.1" jsonrpc_port = 5110 jsonrpc_network = true jsonrpc_oled = true # OLED menu system settings menu_enabled = true # Network interface settings wifi_interface = "wlan0" ap_interface = "ap0" # Network data traffic thresholds data_warning_threshold = 0 data_cutoff_threshold = 0 enable_traffic_warning = false enable_traffic_cutoff = false ```

glyph commented

2021-12-14 09:10:50 +00:00

Further thoughts about this:

We can either ship the .toml file as part of the config or we could hardcode the default config and write it to file if not found (I'm leaning towards the second option...it also gives us the option of exposing a "restore defaults" action).

Further thoughts about this: We can either ship the `.toml` file as part of the config or we could hardcode the default config and write it to file if not found (I'm leaning towards the second option...it also gives us the option of exposing a "restore defaults" action).

notplants commented

2021-12-18 14:40:50 +00:00

Note currently we have Rocket.toml in /usr/share/peach-web/Rocket.toml (because /usr/share/peach-web was set as the working directory for Rocket),

and all other peach-configurations are in /var/lib/peachcloud.

I still think it will be good to ensure all configurations are in one folder, so we should aim to combine these in some way (either through a new config management scheme, as part of this issue, or by putting the peach-web working directory inside of /var/lib/peachcloud)

Note currently we have Rocket.toml in /usr/share/peach-web/Rocket.toml (because /usr/share/peach-web was set as the working directory for Rocket), and all other peach-configurations are in /var/lib/peachcloud. I still think it will be good to ensure all configurations are in one folder, so we should aim to combine these in some way (either through a new config management scheme, as part of this issue, or by putting the peach-web working directory inside of /var/lib/peachcloud)

👍 1

glyph commented

2022-01-17 08:35:43 +00:00

notplants' config comments from PR #70:

PEACH_STANDALONE_MODE

my suggestion is a reiteration of One Config To Rule Them All, which I don't mean configs instead of env vars, what I mean is one way of handling configurations, so that for any configuration (whether peach_standalone_mode or something else) you know there is one system and one config file which handles all of them, and you dont have to arbitrarily remember which configs are handled in which ways.

if we want the ability to set env var, instead of as a a value in config.yml, I would change to using figment to handle all configurations and use a hierarchy (env > config)

and then I would have PEACH_STANDALONE_MODE as one config thats handled in the same way as all the others.

figment looks pretty easy to setup a hierarchy like this (https://docs.rs/figment/0.10.6/figment/). and then we would just need to change all the functions in config_manager to get values from figment instead of with the home-made yaml reading.

one question I've been thinking about is how this would interact with rocket's own config system (which we need to use for the rocket authentication env). I wouldn't just use rocket's env system for everything, because we also want to have access to configurations for processes running outside of a rocket context (e.g. peach-config). I have a feeling, from some reading, that there might be a way to set the file rocket reads its configuration from during the build phase, and then set this file to be /var/lib/peachcloud/config.toml , that way this config file can be read by rocket, and also read by figment from processes that are not running rocket.

but all that said, we can also keep this multiple configs now, and it could be a separate issue at some point if desired to replace config_manager with figment and then make these changes at that point.

...

If looking deeper into using figment instead of the current config_manager,
some other things to consider:

at what point in the rocket cycle to load the configs from config (normally in rocket, I think it loads them during the build phase... but in our web app, it reads and writes configs... how do we ensure that when we've written/updated a config that the new value will be known the next URL that the user goes to... aka the config should be reloaded for every view)

best way to serialize and write figment configs (since we read and write them from the app)

notplants' config comments from [PR #70](https://git.coopcloud.tech/PeachCloud/peach-workspace/pulls/70): > 2. PEACH_STANDALONE_MODE > > my suggestion is a reiteration of One Config To Rule Them All, which I don't mean configs instead of env vars, what I mean is one way of handling configurations, so that for any configuration (whether peach_standalone_mode or something else) you know there is one system and one config file which handles all of them, and you dont have to arbitrarily remember which configs are handled in which ways. > > if we want the ability to set env var, instead of as a a value in config.yml, I would change to using figment to handle all configurations and use a hierarchy (env > config) > > and then I would have PEACH_STANDALONE_MODE as one config thats handled in the same way as all the others. > > figment looks pretty easy to setup a hierarchy like this (https://docs.rs/figment/0.10.6/figment/). and then we would just need to change all the functions in config_manager to get values from figment instead of with the home-made yaml reading. > > one question I've been thinking about is how this would interact with rocket's own config system (which we need to use for the rocket authentication env). I wouldn't just use rocket's env system for everything, because we also want to have access to configurations for processes running outside of a rocket context (e.g. peach-config). I have a feeling, from some reading, that there might be a way to set the file rocket reads its configuration from during the build phase, and then set this file to be /var/lib/peachcloud/config.toml , that way this config file can be read by rocket, and also read by figment from processes that are not running rocket. > > but all that said, we can also keep this multiple configs now, and it could be a separate issue at some point if desired to replace config_manager with figment and then make these changes at that point. ... > If looking deeper into using figment instead of the current config_manager, some other things to consider: > > - at what point in the rocket cycle to load the configs from config (normally in rocket, I think it loads them during the build phase... but in our web app, it reads and writes configs... how do we ensure that when we've written/updated a config that the new value will be known the next URL that the user goes to... aka the config should be reloaded for every view) > - best way to serialize and write figment configs (since we read and write them from the app)

glyph referenced this issue

2022-01-17 09:34:43 +00:00

Wide range of web improvements #70

notplants commented

2022-01-17 14:20:41 +00:00

Another reason in support of figment is that Rocket already depends on Figment, so using Figment for our non-rocket configuration doesn't add an extra dependency.

I also don't think our current setup is that confusing, we would just need to document we have configurations stored in three places:

/var/lib/peachcloud.config.yml (configs which are read-writeable by the application)
PEACH_STANDALONE_MODE env variable
disable_auth which is a rocket config which can be set by ROCKET_DISABLE_AUTH env variable or as a value in Rocket.toml

But there would be some gained extra clarity by combining these in some way.

Another reason in support of figment is that Rocket already depends on Figment, so using Figment for our non-rocket configuration doesn't add an extra dependency. I also don't think our current setup is that confusing, we would just need to document we have configurations stored in three places: - /var/lib/peachcloud.config.yml (configs which are read-writeable by the application) - PEACH_STANDALONE_MODE env variable - disable_auth which is a rocket config which can be set by ROCKET_DISABLE_AUTH env variable or as a value in Rocket.toml But there would be some gained extra clarity by combining these in some way.

notplants commented

2022-01-17 14:22:58 +00:00

another possibility short of using figment for everything (and thus having to also figure out how to make figment read/writeable), perhaps could also make PEACH_STANDALONE_MODE a rocket config, so there are two types of configurations instead of three:

those that can be set with env_vars which are rocket configs
and those which are read-writeable and go in config.yml

the main strange thing about this would be that I think rocket configs require ROCKET_ before the env variable, which is kind of weird.

another possibility short of using figment for everything (and thus having to also figure out how to make figment read/writeable), perhaps could also make PEACH_STANDALONE_MODE a rocket config, so there are two types of configurations instead of three: - those that can be set with env_vars which are rocket configs - and those which are read-writeable and go in config.yml the main strange thing about this would be that I think rocket configs require ROCKET_ before the env variable, which is kind of weird.

glyph commented

2022-01-17 14:53:38 +00:00

@notplants

make PEACH_STANDALONE_MODE a rocket config, so there are two types of configurations instead of three

This sounds like a neat option to me; all web-app-related config goes in the Rocket config file and all other peach-related config goes in /var/lib/peachcloud.config.yml

We could even store the Rocket config in /var/lib/Rocket.toml so that it's alongside the peachcloud config file (if we wanted to keep them together), and then set the ROCKET_CONFIG env var to point to the file path.

I personally don't mind having more than one config file if there is clear separation of intent / purpose and good documentation.

how do we ensure that when we've written/updated a config that the new value will be known the next URL that the user goes to... aka the config should be reloaded for every view

I believe this is a great usecase for Rocket's Managed State feature. So we'd load the config during the building phase to set the initial values (reading them into managed state and retrieving them via the &State request guard) and then each runtime change in state results in a) a write to the config file, and b) a write to the managed state.

I'm not 100% sure that it's possible to write to managed state but my guess is that it must be possible if the initial state value is defined as mut.

@notplants > make PEACH_STANDALONE_MODE a rocket config, so there are two types of configurations instead of three This sounds like a neat option to me; all web-app-related config goes in the Rocket config file and all other peach-related config goes in `/var/lib/peachcloud.config.yml` We could even store the Rocket config in `/var/lib/Rocket.toml` so that it's alongside the `peachcloud` config file (if we wanted to keep them together), and then set the `ROCKET_CONFIG` env var to point to the file path. I personally don't mind having more than one config file if there is clear separation of intent / purpose and good documentation. > how do we ensure that when we've written/updated a config that the new value will be known the next URL that the user goes to... aka the config should be reloaded for every view I believe this is a great usecase for Rocket's [Managed State](https://rocket.rs/v0.5-rc/guide/state/#managed-state) feature. So we'd load the config during the building phase to set the initial values (reading them into managed state and retrieving them via the `&State` request guard) and then each runtime change in state results in a) a write to the config file, and b) a write to the managed state. I'm not 100% sure that it's possible to write to managed state but my guess is that it must be possible if the initial state value is defined as `mut`.

👍 1

notplants commented

2022-01-17 15:38:57 +00:00

interesting about Managed State. it sounds like close, but I have trouble imagining exactly what it would look like, and how this would also interact with peach-config which is interacting with the configs outside of a Rocket context.

the other approach I was thinking about was to modify the load_peach_config and save_peach_config functions in config_manager, to use figment. Then theoretically all of the other code that currently interacts with configs, might not need to change. then, the RocketBuild phase could also call load_peach_config and pass this same config into Rocket (so that disable_auth and peach_standalone_mode are also there as needed). in other words, in this approach, load_peach_config gets called once during the rocket build phase, and these initially loaded values are not read-writeable (aka to change a value of PEACH_STANDALONE_MODE or disable_auth... you have to restart rocket after changing the value... which is fine), and all the other config values are read-writeable, but they get re-loaded whenever they are used, as load_peach_config gets called whenever they are accessed.

however, one could argue there is even a downside of the read-writeable configs being able to be alternatively set with ENV variables... its maybe a bit counter-intuitive... another argument to keep it as two configs... one for env variables which require a rocket restart to change (currently disable_auth and peach_standalone_mode)... and everything else which is read-writeable without a rocket restart in config.yml. In this two config model we don't really need to change anything, other than possibly moving peach_standalone_mode to be a rocket config like disable_auth.

interesting about Managed State. it sounds like close, but I have trouble imagining exactly what it would look like, and how this would also interact with peach-config which is interacting with the configs outside of a Rocket context. the other approach I was thinking about was to modify the load_peach_config and save_peach_config functions in config_manager, to use figment. Then theoretically all of the other code that currently interacts with configs, might not need to change. then, the RocketBuild phase could also call load_peach_config and pass this same config into Rocket (so that disable_auth and peach_standalone_mode are also there as needed). in other words, in this approach, load_peach_config gets called once during the rocket build phase, and these initially loaded values are not read-writeable (aka to change a value of PEACH_STANDALONE_MODE or disable_auth... you have to restart rocket after changing the value... which is fine), and all the other config values are read-writeable, but they get re-loaded whenever they are used, as load_peach_config gets called whenever they are accessed. however, one could argue there is even a downside of the read-writeable configs being able to be alternatively set with ENV variables... its maybe a bit counter-intuitive... another argument to keep it as two configs... one for env variables which require a rocket restart to change (currently disable_auth and peach_standalone_mode)... and everything else which is read-writeable without a rocket restart in config.yml. In this two config model we don't really need to change anything, other than possibly moving peach_standalone_mode to be a rocket config like disable_auth.

glyph commented

2022-01-18 12:42:49 +00:00

keep it as two configs... one for env variables which require a rocket restart to change (currently disable_auth and peach_standalone_mode)... and everything else which is read-writeable without a rocket restart in config.yml. In this two config model we don't really need to change anything, other than possibly moving peach_standalone_mode to be a rocket config like disable_auth.

This sounds like the simplest approach to me 👍

> keep it as two configs... one for env variables which require a rocket restart to change (currently disable_auth and peach_standalone_mode)... and everything else which is read-writeable without a rocket restart in config.yml. In this two config model we don't really need to change anything, other than possibly moving peach_standalone_mode to be a rocket config like disable_auth. This sounds like the simplest approach to me 👍

glyph referenced this issue

2022-01-18 15:13:03 +00:00

Load values from Rocket.toml into managed state #74

Sign in to join this conversation.