coop-cloud/backup-bot-two
6
0
Fork 6
Code Issues 16 Pull Requests 4 Packages Projects Releases Wiki Activity

RESTIC_REPOSITORY_FILE will not be read if RESTIC_REPOSITORY is not set. #51

New Issue
Closed
opened 2024-04-16 16:28:19 +00:00 by cgalo5758 · 7 comments
cgalo5758 commented 2024-04-16 16:28:19 +00:00
Copy Link
Copy Source

RESTIC_REPOSITORY_FILE will not be read if RESTIC_REPOSITORY is not set because the click option repository is required.

Additionally, even if you managed to pass in a dummy value for RESTIC_REPOSITORY, anticipating it to be overwritten, it still wouldn't work unless you pass in the restic password (redundantly):

raise restic.errors.ResticFailedError(
restic.errors.ResticFailedError: Restic failed with exit code 1: Resolving password failed: Fatal: /var/run/secrets/restic_password does not exist

edit:
forget the strikethrough part.

`RESTIC_REPOSITORY_FILE` will not be read if `RESTIC_REPOSITORY` is not set because the click option `repository` is required. ~~Additionally, even if you managed to pass in a dummy value for `RESTIC_REPOSITORY`, anticipating it to be overwritten, it still wouldn't work unless you pass in the restic password (redundantly):~~ ~~raise restic.errors.ResticFailedError(~~ ~~restic.errors.ResticFailedError: Restic failed with exit code 1: Resolving password failed: Fatal: /var/run/secrets/restic_password does not exist~~ edit: forget the strikethrough part.
moritz added the bug label 2024-04-17 07:34:15 +00:00
moritz commented 2024-04-17 07:39:09 +00:00
Owner
Copy Link
Copy Source

Hey @cgalo5758 thank you for your bug report. Can you give some more information. Can you post your .env file, which secrets have you set and a debug output (passing --log DEBUG).
From the error you posted it looks like you haven't set the restic_password secret. Did you commented SECRET_RESTIC_PASSWORD_VERSION in your .env file?

Hey @cgalo5758 thank you for your bug report. Can you give some more information. Can you post your .env file, which secrets have you set and a debug output (passing `--log DEBUG`). From the error you posted it looks like you haven't set the `restic_password` secret. Did you commented `SECRET_RESTIC_PASSWORD_VERSION` in your .env file?
cgalo5758 commented 2024-04-18 19:07:29 +00:00
Author
Copy Link
Copy Source

The only envs I have setup are CRON_SCHEDULE and RESTIC_REPOSITORY_FILE. If I understand the following section correctly, you'd only need to set up RESTIC_REPOSITORY_FILE so I don't have RESTIC_PASSWORD_FILE nor RESTIC_REPOSITORY because they'd be redundant as I am already passing the password through RESTIC_REPOSITORY_FILE

Restic REST server Storage

You can simply set the RESTIC_REPOSITORY variable to your REST server URL rest:http://host:8000/.
If you access the REST server with a password rest:https://user:pass@host:8000/ you should hide the whole URL containing the password inside a secret.
Uncomment these lines:

SECRET_RESTIC_REPO_VERSION=v1
COMPOSE_FILE="$COMPOSE_FILE:compose.secret.yml"

Add your REST server url as secret:

`abra app secret insert <backupbot_name> restic_repo v1 "rest:https://user:pass@host:8000/"`

The secret will overwrite the RESTIC_REPOSITORY variable.

But basically, it won't pass this line in backupbot.py:

@click.option('repository', '--repo', '-r', envvar='RESTIC_REPOSITORY', required=True) # <---
def cli(loglevel, service, repository, machine_logs):

E.g.:

224adf88c7b9:/# backup --log DEBUG create
Usage: backup [OPTIONS] COMMAND [ARGS]...
Try 'backup --help' for help.

Error: Missing option '--repo' / '-r'.

And then if I pass repository:

224adf88c7b9:/# backup --log DEBUG -r poop create 
Read secret value: rest:https://example-user:example-password@restic.evilcorp.com:42069/evilcorp
Uncaught exception
Traceback (most recent call last):
  File "/usr/bin/backup", line 365, in <module>
    cli()
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1685, in invoke
    super().invoke(ctx)
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/backup", line 65, in cli
    init_repo()
  File "/usr/bin/backup", line 80, in init_repo
    raise error
  File "/usr/bin/backup", line 74, in init_repo
    restic.cat.config()
  File "/usr/lib/python3.11/site-packages/restic/internal/cat.py", line 17, in config
    return self.run(cmd)
           ^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/restic/internal/cat.py", line 20, in run
    return json.loads(command_executor.execute(cmd))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/restic/internal/command_executor.py", line 25, in execute
    raise restic.errors.ResticFailedError(
restic.errors.ResticFailedError: Restic failed with exit code 1: Resolving password failed: Fatal: /var/run/secrets/restic_password does not exist
The only envs I have setup are `CRON_SCHEDULE` and `RESTIC_REPOSITORY_FILE`. If I understand the following section correctly, you'd only need to set up `RESTIC_REPOSITORY_FILE` so I don't have `RESTIC_PASSWORD_FILE` nor `RESTIC_REPOSITORY` because they'd be redundant as I am already passing the password through `RESTIC_REPOSITORY_FILE` >### Restic REST server Storage > >You can simply set the `RESTIC_REPOSITORY` variable to your REST server URL `rest:http://host:8000/`. >If you access the REST server with a password `rest:https://user:pass@host:8000/` you should hide the whole URL containing the password inside a secret. >Uncomment these lines: >``` >SECRET_RESTIC_REPO_VERSION=v1 >COMPOSE_FILE="$COMPOSE_FILE:compose.secret.yml" >``` >Add your REST server url as secret: >``` >`abra app secret insert <backupbot_name> restic_repo v1 "rest:https://user:pass@host:8000/"` >``` >The secret will overwrite the `RESTIC_REPOSITORY` variable. But basically, it won't pass this line in `backupbot.py`: ``` @click.option('repository', '--repo', '-r', envvar='RESTIC_REPOSITORY', required=True) # <--- def cli(loglevel, service, repository, machine_logs): ``` E.g.: ``` 224adf88c7b9:/# backup --log DEBUG create Usage: backup [OPTIONS] COMMAND [ARGS]... Try 'backup --help' for help. Error: Missing option '--repo' / '-r'. ``` And then if I pass repository: ``` 224adf88c7b9:/# backup --log DEBUG -r poop create Read secret value: rest:https://example-user:example-password@restic.evilcorp.com:42069/evilcorp Uncaught exception Traceback (most recent call last): File "/usr/bin/backup", line 365, in <module> cli() File "/usr/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/click/core.py", line 1685, in invoke super().invoke(ctx) File "/usr/lib/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/bin/backup", line 65, in cli init_repo() File "/usr/bin/backup", line 80, in init_repo raise error File "/usr/bin/backup", line 74, in init_repo restic.cat.config() File "/usr/lib/python3.11/site-packages/restic/internal/cat.py", line 17, in config return self.run(cmd) ^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/restic/internal/cat.py", line 20, in run return json.loads(command_executor.execute(cmd)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/restic/internal/command_executor.py", line 25, in execute raise restic.errors.ResticFailedError( restic.errors.ResticFailedError: Restic failed with exit code 1: Resolving password failed: Fatal: /var/run/secrets/restic_password does not exist ```
3wordchant commented 2024-04-19 16:27:33 +00:00
Owner
Copy Link
Copy Source

Thanks both!

Drive-by comment on this:

I don't have RESTIC_PASSWORD_FILE nor RESTIC_REPOSITORY because they'd be redundant as I am already passing the password through RESTIC_REPOSITORY_FILE

AFAIU the "restic password" is the encryption password, whereas the password included in the restic repo when using HTTPS is the HTTP password.

So basically I think RESTIC_PASSWORD_FILE and RESTIC_REPOSITORY_FILE would both be required when using HTTPS.

Thanks both! Drive-by comment on this: > I don't have `RESTIC_PASSWORD_FILE` nor `RESTIC_REPOSITORY` because they'd be redundant as I am already passing the password through `RESTIC_REPOSITORY_FILE` AFAIU the "restic password" is the encryption password, whereas the password included in the restic repo when using HTTPS is the HTTP password. So basically I think `RESTIC_PASSWORD_FILE` and `RESTIC_REPOSITORY_FILE` would both be required when using HTTPS.
👍 2
cgalo5758 commented 2024-04-19 22:13:38 +00:00
Author
Copy Link
Copy Source

AFAIU the "restic password" is the encryption password, whereas the password included in the restic repo when using HTTPS is the HTTP password.

So basically I think RESTIC_PASSWORD_FILE and RESTIC_REPOSITORY_FILE would both be required when using HTTPS.

Oh wait that's correct. My bad 😅. Forget about this part then:

Additionally, even if you managed to pass in a dummy value for RESTIC_REPOSITORY, anticipating it to be overwritten, it still wouldn't work unless you pass in the restic password (redundantly):

    raise restic.errors.ResticFailedError(
restic.errors.ResticFailedError: Restic failed with exit code 1: Resolving password failed: Fatal: /var/run/secrets/restic_password does not exist

The other issue still persists. Passing in a dummy value for RESTIC_REPOSITORY so that RESTIC_REPOSITORY_FILE can be read is kinda odd.

> AFAIU the "restic password" is the encryption password, whereas the password included in the restic repo when using HTTPS is the HTTP password. > > So basically I think RESTIC_PASSWORD_FILE and RESTIC_REPOSITORY_FILE would both be required when using HTTPS. Oh wait that's correct. My bad 😅. Forget about this part then: > Additionally, even if you managed to pass in a dummy value for `RESTIC_REPOSITORY`, anticipating it to be overwritten, it still wouldn't work unless you pass in the restic password (redundantly): > > ``` > raise restic.errors.ResticFailedError( > restic.errors.ResticFailedError: Restic failed with exit code 1: Resolving password failed: Fatal: /var/run/secrets/restic_password does not exist > ``` The other issue still persists. Passing in a dummy value for `RESTIC_REPOSITORY` so that `RESTIC_REPOSITORY_FILE` can be read is kinda odd.
👍 1
3wordchant commented 2024-04-21 23:44:08 +00:00
Owner
Copy Link
Copy Source

OK yeah I would agree, if RESTIC_REPOSITORY_FILE is set then neither RESTIC_REPOSITORY nor -r should be required. @cgalo5758 do you have a preferred fix?

OK yeah I would agree, if `RESTIC_REPOSITORY_FILE` is set then neither `RESTIC_REPOSITORY` nor `-r` should be required. @cgalo5758 do you have a preferred fix?
cgalo5758 commented 2024-04-24 02:19:36 +00:00
Author
Copy Link
Copy Source

I tried just removing the required=True from the click option, but that doesn't fix it entirely. When export_secrets() sets RESTIC_REPOSITORY from RESTIC_REPOSITORY_FILE, both environment variables are set, leading to the following error:

restic.errors.ResticFailedError: Restic failed with exit code 1: Fatal: Options -r and --repository-file are mutually exclusive, please specify only one

And so to fix this, I suggest also editing export_secrets()

def export_secrets():
    for env in os.environ:
        if env.endswith('FILE') and not "COMPOSE_FILE" in env:
            logger.debug(f"exported secret: {env}")
            with open(os.environ[env]) as file:
                secret = file.read()
                os.environ[env.removesuffix('_FILE')] = secret

            if env == 'RESTIC_REPOSITORY_FILE':
                # RESTIC_REPOSITORY_FILE and RESTIC_REPOSITORY are mutually exclusive
                logger.info("RESTIC_REPOSITORY set to RESTIC_REPOSITORY_FILE. Unsetting RESTIC_REPOSITORY_FILE.")
                del os.environ['RESTIC_REPOSITORY_FILE']

This is not elegant, but it is the least invasive way of doing this as RESTIC_REPOSITORY is the only variable of these two that is used by the script directly.

I tried just removing the `required=True` from the click option, but that doesn't fix it entirely. When `export_secrets()` sets RESTIC_REPOSITORY from RESTIC_REPOSITORY_FILE, both environment variables are set, leading to the following error: ``` restic.errors.ResticFailedError: Restic failed with exit code 1: Fatal: Options -r and --repository-file are mutually exclusive, please specify only one ``` And so to fix this, I suggest also editing `export_secrets()` ``` def export_secrets(): for env in os.environ: if env.endswith('FILE') and not "COMPOSE_FILE" in env: logger.debug(f"exported secret: {env}") with open(os.environ[env]) as file: secret = file.read() os.environ[env.removesuffix('_FILE')] = secret if env == 'RESTIC_REPOSITORY_FILE': # RESTIC_REPOSITORY_FILE and RESTIC_REPOSITORY are mutually exclusive logger.info("RESTIC_REPOSITORY set to RESTIC_REPOSITORY_FILE. Unsetting RESTIC_REPOSITORY_FILE.") del os.environ['RESTIC_REPOSITORY_FILE'] ``` This is not elegant, but it is the least invasive way of doing this as RESTIC_REPOSITORY is the only variable of these two that is used by the script directly.
moritz commented 2024-04-29 14:26:52 +00:00
Owner
Copy Link
Copy Source

@cgalo5758 thank you for your input, this should be fixed now.

@cgalo5758 thank you for your input, this should be fixed now.
❤️ 1
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 wontfix
This won't be fixed
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
3wordchant aadil (Aadil Ayub) abra-bot (Abra Bot) ammaratef45 amras (Sarma) Apfelwurm appletalk arjan basebuilder BornDeleuze Brooke carla cas (Cassowary) codegod100 coopcloud cyrnel decentral1se (d1) dede devydave fauno (fauno) flancian Frando iexos jade (Jade Ambrose) javielico (Javielico) jjsfunhouse jmakdah2 (Jackie Makdah) joe-irving (Joe Irving) kawaiipunk (KawaiiPunk) knoflook kolaente lambdabundesverband linnealovespie (April) marlon (marlon) mayel mirsal moosemower moritz nicksellen (Nick Sellen) notplants oxaliq (sorrel) p4u1 pau pharaohgraphy (Andrew 🐦‍🔥❤️‍🔥✴️) PhiNatalie renovate-bot (Comrade Renovate Bot) ripclap rix rscmbbng sef (sef) simon sixsmith (Sixsmith) stevensting tobias trav (Trav Fryer) val (val (he/him)) vaznasty virtualboys wolcen (Chris Thompson) wykwit xynosis yksflip
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: coop-cloud/backup-bot-two#51