coop-cloud/backup-bot-two
coop-cloud
/
backup-bot-two
6
0
Fork 3
Code Issues 12 Pull Requests 1 Packages Projects Releases Wiki Activity

RESTIC_REPOSITORY_FILE will not be read if RESTIC_REPOSITORY is not set. #51

New Issue
Closed
opened 2024-04-16 16:28:19 +00:00 by cgalo5758 · 7 comments
cgalo5758 commented 2024-04-16 16:28:19 +00:00
Copy Link

RESTIC_REPOSITORY_FILE will not be read if RESTIC_REPOSITORY is not set because the click option repository is required.

Additionally, even if you managed to pass in a dummy value for RESTIC_REPOSITORY, anticipating it to be overwritten, it still wouldn't work unless you pass in the restic password (redundantly):

raise restic.errors.ResticFailedError(
restic.errors.ResticFailedError: Restic failed with exit code 1: Resolving password failed: Fatal: /var/run/secrets/restic_password does not exist

edit:
forget the strikethrough part.

`RESTIC_REPOSITORY_FILE` will not be read if `RESTIC_REPOSITORY` is not set because the click option `repository` is required. ~~Additionally, even if you managed to pass in a dummy value for `RESTIC_REPOSITORY`, anticipating it to be overwritten, it still wouldn't work unless you pass in the restic password (redundantly):~~ ~~raise restic.errors.ResticFailedError(~~ ~~restic.errors.ResticFailedError: Restic failed with exit code 1: Resolving password failed: Fatal: /var/run/secrets/restic_password does not exist~~ edit: forget the strikethrough part.
moritz added the
bug
 label 2024-04-17 07:34:15 +00:00
moritz commented 2024-04-17 07:39:09 +00:00
Member
Copy Link

Hey @cgalo5758 thank you for your bug report. Can you give some more information. Can you post your .env file, which secrets have you set and a debug output (passing --log DEBUG).
From the error you posted it looks like you haven't set the restic_password secret. Did you commented SECRET_RESTIC_PASSWORD_VERSION in your .env file?

Hey @cgalo5758 thank you for your bug report. Can you give some more information. Can you post your .env file, which secrets have you set and a debug output (passing `--log DEBUG`). From the error you posted it looks like you haven't set the `restic_password` secret. Did you commented `SECRET_RESTIC_PASSWORD_VERSION` in your .env file?
cgalo5758 commented 2024-04-18 19:07:29 +00:00
Author
Copy Link

The only envs I have setup are CRON_SCHEDULE and RESTIC_REPOSITORY_FILE. If I understand the following section correctly, you'd only need to set up RESTIC_REPOSITORY_FILE so I don't have RESTIC_PASSWORD_FILE nor RESTIC_REPOSITORY because they'd be redundant as I am already passing the password through RESTIC_REPOSITORY_FILE

Restic REST server Storage

You can simply set the RESTIC_REPOSITORY variable to your REST server URL rest:http://host:8000/.
If you access the REST server with a password rest:https://user:pass@host:8000/ you should hide the whole URL containing the password inside a secret.
Uncomment these lines:

SECRET_RESTIC_REPO_VERSION=v1
COMPOSE_FILE="$COMPOSE_FILE:compose.secret.yml"

Add your REST server url as secret:

`abra app secret insert <backupbot_name> restic_repo v1 "rest:https://user:pass@host:8000/"`

The secret will overwrite the RESTIC_REPOSITORY variable.

But basically, it won't pass this line in backupbot.py:

@click.option('repository', '--repo', '-r', envvar='RESTIC_REPOSITORY', required=True) # <---
def cli(loglevel, service, repository, machine_logs):

E.g.:

224adf88c7b9:/# backup --log DEBUG create
Usage: backup [OPTIONS] COMMAND [ARGS]...
Try 'backup --help' for help.

Error: Missing option '--repo' / '-r'.

And then if I pass repository:

224adf88c7b9:/# backup --log DEBUG -r poop create 
Read secret value: rest:https://example-user:example-password@restic.evilcorp.com:42069/evilcorp
Uncaught exception
Traceback (most recent call last):
  File "/usr/bin/backup", line 365, in <module>
    cli()
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1685, in invoke
    super().invoke(ctx)
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/backup", line 65, in cli
    init_repo()
  File "/usr/bin/backup", line 80, in init_repo
    raise error
  File "/usr/bin/backup", line 74, in init_repo
    restic.cat.config()
  File "/usr/lib/python3.11/site-packages/restic/internal/cat.py", line 17, in config
    return self.run(cmd)
           ^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/restic/internal/cat.py", line 20, in run
    return json.loads(command_executor.execute(cmd))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/restic/internal/command_executor.py", line 25, in execute
    raise restic.errors.ResticFailedError(
restic.errors.ResticFailedError: Restic failed with exit code 1: Resolving password failed: Fatal: /var/run/secrets/restic_password does not exist
The only envs I have setup are `CRON_SCHEDULE` and `RESTIC_REPOSITORY_FILE`. If I understand the following section correctly, you'd only need to set up `RESTIC_REPOSITORY_FILE` so I don't have `RESTIC_PASSWORD_FILE` nor `RESTIC_REPOSITORY` because they'd be redundant as I am already passing the password through `RESTIC_REPOSITORY_FILE` >### Restic REST server Storage > >You can simply set the `RESTIC_REPOSITORY` variable to your REST server URL `rest:http://host:8000/`. >If you access the REST server with a password `rest:https://user:pass@host:8000/` you should hide the whole URL containing the password inside a secret. >Uncomment these lines: >``` >SECRET_RESTIC_REPO_VERSION=v1 >COMPOSE_FILE="$COMPOSE_FILE:compose.secret.yml" >``` >Add your REST server url as secret: >``` >`abra app secret insert <backupbot_name> restic_repo v1 "rest:https://user:pass@host:8000/"` >``` >The secret will overwrite the `RESTIC_REPOSITORY` variable. But basically, it won't pass this line in `backupbot.py`: ``` @click.option('repository', '--repo', '-r', envvar='RESTIC_REPOSITORY', required=True) # <--- def cli(loglevel, service, repository, machine_logs): ``` E.g.: ``` 224adf88c7b9:/# backup --log DEBUG create Usage: backup [OPTIONS] COMMAND [ARGS]... Try 'backup --help' for help. Error: Missing option '--repo' / '-r'. ``` And then if I pass repository: ``` 224adf88c7b9:/# backup --log DEBUG -r poop create Read secret value: rest:https://example-user:example-password@restic.evilcorp.com:42069/evilcorp Uncaught exception Traceback (most recent call last): File "/usr/bin/backup", line 365, in <module> cli() File "/usr/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/click/core.py", line 1685, in invoke super().invoke(ctx) File "/usr/lib/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/bin/backup", line 65, in cli init_repo() File "/usr/bin/backup", line 80, in init_repo raise error File "/usr/bin/backup", line 74, in init_repo restic.cat.config() File "/usr/lib/python3.11/site-packages/restic/internal/cat.py", line 17, in config return self.run(cmd) ^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/restic/internal/cat.py", line 20, in run return json.loads(command_executor.execute(cmd)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/restic/internal/command_executor.py", line 25, in execute raise restic.errors.ResticFailedError( restic.errors.ResticFailedError: Restic failed with exit code 1: Resolving password failed: Fatal: /var/run/secrets/restic_password does not exist ```
3wordchant commented 2024-04-19 16:27:33 +00:00
Owner
Copy Link

Thanks both!

Drive-by comment on this:

I don't have RESTIC_PASSWORD_FILE nor RESTIC_REPOSITORY because they'd be redundant as I am already passing the password through RESTIC_REPOSITORY_FILE

AFAIU the "restic password" is the encryption password, whereas the password included in the restic repo when using HTTPS is the HTTP password.

So basically I think RESTIC_PASSWORD_FILE and RESTIC_REPOSITORY_FILE would both be required when using HTTPS.

Thanks both! Drive-by comment on this: > I don't have `RESTIC_PASSWORD_FILE` nor `RESTIC_REPOSITORY` because they'd be redundant as I am already passing the password through `RESTIC_REPOSITORY_FILE` AFAIU the "restic password" is the encryption password, whereas the password included in the restic repo when using HTTPS is the HTTP password. So basically I think `RESTIC_PASSWORD_FILE` and `RESTIC_REPOSITORY_FILE` would both be required when using HTTPS.
👍 2
cgalo5758 commented 2024-04-19 22:13:38 +00:00
Author
Copy Link

AFAIU the "restic password" is the encryption password, whereas the password included in the restic repo when using HTTPS is the HTTP password.

So basically I think RESTIC_PASSWORD_FILE and RESTIC_REPOSITORY_FILE would both be required when using HTTPS.

Oh wait that's correct. My bad 😅. Forget about this part then:

Additionally, even if you managed to pass in a dummy value for RESTIC_REPOSITORY, anticipating it to be overwritten, it still wouldn't work unless you pass in the restic password (redundantly):

    raise restic.errors.ResticFailedError(
restic.errors.ResticFailedError: Restic failed with exit code 1: Resolving password failed: Fatal: /var/run/secrets/restic_password does not exist

The other issue still persists. Passing in a dummy value for RESTIC_REPOSITORY so that RESTIC_REPOSITORY_FILE can be read is kinda odd.

> AFAIU the "restic password" is the encryption password, whereas the password included in the restic repo when using HTTPS is the HTTP password. > > So basically I think RESTIC_PASSWORD_FILE and RESTIC_REPOSITORY_FILE would both be required when using HTTPS. Oh wait that's correct. My bad 😅. Forget about this part then: > Additionally, even if you managed to pass in a dummy value for `RESTIC_REPOSITORY`, anticipating it to be overwritten, it still wouldn't work unless you pass in the restic password (redundantly): > > ``` > raise restic.errors.ResticFailedError( > restic.errors.ResticFailedError: Restic failed with exit code 1: Resolving password failed: Fatal: /var/run/secrets/restic_password does not exist > ``` The other issue still persists. Passing in a dummy value for `RESTIC_REPOSITORY` so that `RESTIC_REPOSITORY_FILE` can be read is kinda odd.
👍 1
3wordchant commented 2024-04-21 23:44:08 +00:00
Owner
Copy Link

OK yeah I would agree, if RESTIC_REPOSITORY_FILE is set then neither RESTIC_REPOSITORY nor -r should be required. @cgalo5758 do you have a preferred fix?

OK yeah I would agree, if `RESTIC_REPOSITORY_FILE` is set then neither `RESTIC_REPOSITORY` nor `-r` should be required. @cgalo5758 do you have a preferred fix?
cgalo5758 commented 2024-04-24 02:19:36 +00:00
Author
Copy Link

I tried just removing the required=True from the click option, but that doesn't fix it entirely. When export_secrets() sets RESTIC_REPOSITORY from RESTIC_REPOSITORY_FILE, both environment variables are set, leading to the following error:

restic.errors.ResticFailedError: Restic failed with exit code 1: Fatal: Options -r and --repository-file are mutually exclusive, please specify only one

And so to fix this, I suggest also editing export_secrets()

def export_secrets():
    for env in os.environ:
        if env.endswith('FILE') and not "COMPOSE_FILE" in env:
            logger.debug(f"exported secret: {env}")
            with open(os.environ[env]) as file:
                secret = file.read()
                os.environ[env.removesuffix('_FILE')] = secret

            if env == 'RESTIC_REPOSITORY_FILE':
                # RESTIC_REPOSITORY_FILE and RESTIC_REPOSITORY are mutually exclusive
                logger.info("RESTIC_REPOSITORY set to RESTIC_REPOSITORY_FILE. Unsetting RESTIC_REPOSITORY_FILE.")
                del os.environ['RESTIC_REPOSITORY_FILE']

This is not elegant, but it is the least invasive way of doing this as RESTIC_REPOSITORY is the only variable of these two that is used by the script directly.

I tried just removing the `required=True` from the click option, but that doesn't fix it entirely. When `export_secrets()` sets RESTIC_REPOSITORY from RESTIC_REPOSITORY_FILE, both environment variables are set, leading to the following error: ``` restic.errors.ResticFailedError: Restic failed with exit code 1: Fatal: Options -r and --repository-file are mutually exclusive, please specify only one ``` And so to fix this, I suggest also editing `export_secrets()` ``` def export_secrets(): for env in os.environ: if env.endswith('FILE') and not "COMPOSE_FILE" in env: logger.debug(f"exported secret: {env}") with open(os.environ[env]) as file: secret = file.read() os.environ[env.removesuffix('_FILE')] = secret if env == 'RESTIC_REPOSITORY_FILE': # RESTIC_REPOSITORY_FILE and RESTIC_REPOSITORY are mutually exclusive logger.info("RESTIC_REPOSITORY set to RESTIC_REPOSITORY_FILE. Unsetting RESTIC_REPOSITORY_FILE.") del os.environ['RESTIC_REPOSITORY_FILE'] ``` This is not elegant, but it is the least invasive way of doing this as RESTIC_REPOSITORY is the only variable of these two that is used by the script directly.
moritz commented 2024-04-29 14:26:52 +00:00
Member
Copy Link

@cgalo5758 thank you for your input, this should be fixed now.

@cgalo5758 thank you for your input, this should be fixed now.
❤️ 1
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
feature/dockerfile
bb2-classic
feature/selective_paths
enable-label
backupbot_revolution
backup_volumes
multi_path
0.2.0+1.0.0
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: coop-cloud/backup-bot-two#51
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?