abra assumes cluster --listen-ip is always public routable #317

New Issue

Ghost · 2022-04-29T15:15:28Z

Ghost commented

2022-04-29 15:15:28 +00:00

i have an interesting use-case in that my environment is mixed metal/vps mesh model over wireguard or nebula.

when a cluster is spun up, the --listen-addr and --advertise-addr are set on docker swarm init so that the rest of the nodes in the proposed mesh are able to communicate on the encrypted mesh network, which is then utilised via overlay proxy network in docker.

to dive deeper, swarm nodes listen on private network ie: 100.64.0.0/10 which are non-routable via public internet.

during a standar abra app new traefik and following abra app config traefik.mydomain.com an error message is noted:

app domain traefik.mydomain.com does not appear to resolve to app server 100.65.100.10?

i would propose being able to optionally pass an arg that allows operator to specify the public routed ip address thus dismissing the advertise and listen address on the swarm cluster itself.

note: this is a unique use-case since many operators from my observation run in public-routed cidr blocks on vps/paas vendors.

if there is interest in this, i likely can work on a feature spike of it.

i have an interesting use-case in that my environment is mixed metal/vps mesh model over wireguard or nebula. when a cluster is spun up, the `--listen-addr` and `--advertise-addr` are set on `docker swarm init` so that the rest of the nodes in the proposed mesh are able to communicate on the encrypted mesh network, which is then utilised via overlay proxy network in docker. to dive deeper, swarm nodes listen on private network ie: `100.64.0.0/10` which are non-routable via public internet. during a standar `abra app new traefik` and following `abra app config traefik.mydomain.com` an error message is noted: `app domain traefik.mydomain.com does not appear to resolve to app server 100.65.100.10?` i would propose being able to optionally pass an arg that allows operator to specify the public routed ip address thus dismissing the advertise and listen address on the swarm cluster itself. note: this is a unique use-case since many operators from my observation run in public-routed cidr blocks on vps/paas vendors. if there is interest in this, i likely can work on a feature spike of it.

Ghost added the

enhancement

abra

labels 2022-04-29 15:15:28 +00:00

decentral1se commented

2022-04-30 16:40:14 +00:00

Hey @mutefall, that's an interesting one alright!

and following abra app config traefik.mydomain.com an error message is noted: app domain traefik.mydomain.com does not appear to resolve to app server 100.65.100.10?

Hmmmmm 🤔 that seems like a bug that abra app config ... is failing with domain name resolution 😱 Was this not an output instead from abra app deploy ...? Btw I was also curious if deploy --no-domain-checks was a work-around for now?

i would propose being able to optionally pass an arg that allows operator to specify the public routed ip address thus dismissing the advertise and listen address on the swarm cluster itself ... if there is interest in this, i likely can work on a feature spike of it.

Sounds legit! Probably an arg on the abra server add then? I guess it comes down to some threading / re-working of the logic in 574d556bb9/cli/server/add.go (L330-L333)

Hey @mutefall, that's an interesting one alright! > and following abra app config traefik.mydomain.com an error message is noted: app domain traefik.mydomain.com does not appear to resolve to app server 100.65.100.10? Hmmmmm 🤔 that seems like a bug that `abra app config ...` is failing with domain name resolution 😱 Was this not an output instead from `abra app deploy ...`? Btw I was also curious if `deploy --no-domain-checks` was a work-around for now? > i would propose being able to optionally pass an arg that allows operator to specify the public routed ip address thus dismissing the advertise and listen address on the swarm cluster itself ... if there is interest in this, i likely can work on a feature spike of it. Sounds legit! Probably an arg on the `abra server add` then? I guess it comes down to some threading / re-working of the logic in https://git.coopcloud.tech/coop-cloud/abra/src/commit/574d556bb96570e878726fb4147d42a73ea9d5f7/cli/server/add.go#L330-L333

Ghost commented

2022-05-06 11:08:40 +00:00

@decentral1se let me give it a test with the --no-domain-checks today and report back. i've been a way for a bit and just now catching up :-)

@decentral1se let me give it a test with the `--no-domain-checks` today and report back. i've been a way for a bit and just now catching up :-)

👍 1

Ghost commented

2022-05-15 13:53:25 +00:00

@decentral1se i apologise for the latency, have been busy with daytime activities.

i did try deployment with --no-domain-checks and it appears to be a workaround.

i'm curious if it would make sense to add a bool arg to EnsureDomainsResolveIPv4 function. which could be called upstream which would take into account those running upstream load balancers for on-premise situations which would then take the yourapp.domain.com deployment and compare the ipv4 addr from the upstream load balancer to ensure parity with the a record

per also upon adding a server, we could introduce a similar optional arg but string type to declare the ipv4 addr up the upstream load balancer which would then satisfy the the ability for the dns package to do its work properly.

@decentral1se i apologise for the latency, have been busy with daytime activities. i did try deployment with `--no-domain-checks` and it appears to be a workaround. i'm curious if it would make sense to add a `bool` arg to [EnsureDomainsResolveIPv4](https://git.coopcloud.tech/coop-cloud/abra/src/commit/3b8893502a871ca25937201a90cb9fb254b2575e/pkg/dns/common.go#L34) function. which could be called `upstream` which would take into account those running upstream load balancers for on-premise situations which would then take the `yourapp.domain.com` deployment and compare the ipv4 addr from the upstream load balancer to ensure parity with the `a record` per also upon [adding](https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/cli/server/add.go) a server, we could introduce a similar optional arg but `string` type to declare the `ipv4` addr up the upstream load balancer which would then satisfy the the ability for the `dns` package to do its work properly.

decentral1se commented

2022-05-15 22:02:43 +00:00

No worries @mutefall, thanks for making time for it again! Glad you have that work-around for working around 🙂

Your proposal makes sense but i think the bit where it falls down is the additional arg on server add.

abra is basically stateless when it comes to the DNS logic, it just does some resolving and sees what it finds. So, where do we store the "remember this is the upstream load balancer and use that IP" for abra to look up later on when doing deployments?

I'm a bit cautious to add a command-line flag to the interface that potentially isn't being used by a wide audience (niche use case). Of course, it's hard to say that this is really niche since a lot of folks use load balancers!

There was talk of doing a config file in #303, maybe this is one for that? We could use the config file as a staging area for experimental options.

So, you have some option in the config file (have a proposal for how it can look?) and then when deploying, if abra sees that option in the config file, it does something else at deploy time.

No worries @mutefall, thanks for making time for it again! Glad you have that work-around for working around 🙂 Your proposal makes sense but i think the bit where it falls down is the additional arg on `server add`. `abra` is basically stateless when it comes to the DNS logic, it just does some resolving and sees what it finds. So, where do we store the "remember this is the upstream load balancer and use that IP" for `abra` to look up later on when doing deployments? I'm a bit cautious to add a command-line flag to the interface that potentially isn't being used by a wide audience (niche use case). Of course, it's hard to say that this is really niche since a lot of folks use load balancers! There was talk of doing a config file in https://git.coopcloud.tech/coop-cloud/organising/issues/303, maybe this is one for that? We could use the config file as a staging area for experimental options. So, you have some option in the config file (have a proposal for how it can look?) and then when deploying, if `abra` sees that option in the config file, it does something else at deploy time.

Ghost commented

2022-05-16 17:15:55 +00:00

@decentral1se

Your proposal makes sense but i think the bit where it falls down is the additional arg on server add.

nothing like lack of sleep to think too shallow.

abra is basically stateless when it comes to the DNS logic, it just does some resolving and sees what it finds. So, where do we store the "remember this is the upstream load balancer and use that IP" for abra to look up later on when doing deployments?

There was talk of doing a config file in #303, maybe this is one for that? We could use the config file as a staging area for experimental options.

So, you have some option in the config file (have a proposal for how it can look?) and then when deploying, if abra sees that option in the config file, it does something else at deploy time.

this looks to be a better approach. since i can workaround with the original suggestion i will close this and focus energy on #303

@decentral1se >Your proposal makes sense but i think the bit where it falls down is the additional arg on server add. nothing like lack of sleep to think too shallow. >abra is basically stateless when it comes to the DNS logic, it just does some resolving and sees what it finds. So, where do we store the "remember this is the upstream load balancer and use that IP" for abra to look up later on when doing deployments? >There was talk of doing a config file in #303, maybe this is one for that? We could use the config file as a staging area for experimental options. >So, you have some option in the config file (have a proposal for how it can look?) and then when deploying, if abra sees that option in the config file, it does something else at deploy time. this looks to be a better approach. since i can workaround with the original suggestion i will close this and focus energy on #303

❤️ 1

Ghost closed this issue

2022-05-16 17:15:58 +00:00

Sign in to join this conversation.

No Milestone

No project

No Assignees

2 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: coop-cloud/organising#317