coop-cloud/organising
coop-cloud
/
organising
7
0
Fork 0
Code Issues 155 Pull Requests Projects 2 Releases Activity

3rd party abra integration thread #106

New Issue
Closed
opened 2021-07-22 11:25:10 +00:00 by decentral1se · 18 comments
decentral1se commented 2021-07-22 11:25:10 +00:00
Owner
Copy Link

VPS providers

DNS providers

(Feel free to edit this post to update!)

## VPS providers - [x] [Capsul](https://capsul.org/) / [Servers.coop](https://servers.coop) - [x] Hetzner - [ ] Digital Ocean - [ ] Linode ## DNS providers - [x] Gandi - [ ] Porkbun - [ ] freedns.afraid.org - [ ] https://njal.la/ - [ ] https://www.freenom.com - [ ] https://www.opennic.org/ (p. cool, a DNS root alternative to ICANN's one with all domains being free) (Feel free to edit this post to update!)
❤️ 1
decentral1se added this to the (deleted) milestone 2021-07-22 11:25:10 +00:00
decentral1se added the
enhancement
help wanted
 labels 2021-07-22 11:25:10 +00:00
decentral1se commented 2021-07-29 09:19:33 +00:00
Author
Owner
Copy Link

abra server new capsul [--instance/-i <instance>] [--token/-t <token>] ... where if any flags are missing, we prompt for interactive input. We can implement this as a plugin within the go-abra source tree. Each sub-command such as capsul can specify its own flags to match required inputs.

`abra server new capsul [--instance/-i <instance>] [--token/-t <token>] ...` where if any flags are missing, we prompt for interactive input. We can implement this as a plugin within the `go-abra` source tree. Each sub-command such as `capsul` can specify its own flags to match required inputs.
decentral1se commented 2021-07-29 10:34:46 +00:00
Author
Owner
Copy Link

maybe also abra server dns gandi --domain foo.com --ipv4 x.x.x.x --wildcard is also a way we can automagically wire up a top-level A record and a wildcard *.foo.com in one command?

> maybe also `abra server dns gandi --domain foo.com --ipv4 x.x.x.x --wildcard` is also a way we can automagically wire up a top-level A record and a wildcard `*.foo.com` in one command?
knoflook commented 2021-08-03 11:31:15 +00:00
Owner
Copy Link

I can't edit this post but I would love to see freedns.afraid.org as well as porkbun.com as DNS providers :) fixed!

~~I can't edit this post but I would love to see freedns.afraid.org as well as porkbun.com as DNS providers :)~~ fixed!
❤️ 1
forest commented 2021-08-19 18:45:54 +00:00
Copy Link

When I was doing this, I decided to base it on Terraform. Terraform has a ton of already supported providers, w/ a nice and easy (open) way to add more. And it's easy-ish to use terraform modules to make it so that apps can require specific records to be created without knowing what provider they will use.

https://git.sequentialread.com/forest/rootsystem/src/branch/master/terraform-modules/gateway-dns
https://git.sequentialread.com/forest/rootsystem/src/branch/master/terraform-modules/dns-gandi

I wrote code that generates the root terraform code file based on the modules:

https://git.sequentialread.com/forest/rootsystem/src/branch/master/automation/terraformCodeGeneration.go#L72

func WriteTerraformCodeForTargetedModules(...) (...) {

.... blahblahblahblah....

	terraformCodeFilepath := filepath.Join(terraformFolder, "main.tf")
	terraformCodeFile, err := os.OpenFile(terraformCodeFilepath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0700)
	if err != nil {
		return []string{}, errors.Wrapf(err, "can't initializeTerraformProject because can't os.OpenFile(\"%s\")", terraformCodeFilepath)
	}

	fmt.Fprintf(
		terraformCodeFile,
		`
	  terraform {
	    backend "http" {
	      address = "http://localhost:%d/%s"
	    }
	  }
%s
%s
%s
%s
%s
	  `,
		configuration.TERRAFORM_STATE_SERVER_PORT_NUMBER,
		terraformConfig.TerraformProject,
		remoteState,
		strings.Join(providerStanzas, "\n"),
		strings.Join(variableStanzas, "\n"),
		strings.Join(moduleStanzas, "\n"),
		strings.Join(outputStanzas, "\n"),
	)

    ...

}

I'm not saying my code is the best way to do this or you should use it, just wanted to share the approach I took.

When I was doing this, I decided to base it on Terraform. Terraform has a ton of already supported providers, w/ a nice and easy (open) way to add more. And it's easy-ish to use terraform modules to make it so that apps can require specific records to be created without knowing what provider they will use. https://git.sequentialread.com/forest/rootsystem/src/branch/master/terraform-modules/gateway-dns https://git.sequentialread.com/forest/rootsystem/src/branch/master/terraform-modules/dns-gandi I wrote code that generates the root terraform code file based on the modules: https://git.sequentialread.com/forest/rootsystem/src/branch/master/automation/terraformCodeGeneration.go#L72 ``` func WriteTerraformCodeForTargetedModules(...) (...) { .... blahblahblahblah.... terraformCodeFilepath := filepath.Join(terraformFolder, "main.tf") terraformCodeFile, err := os.OpenFile(terraformCodeFilepath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0700) if err != nil { return []string{}, errors.Wrapf(err, "can't initializeTerraformProject because can't os.OpenFile(\"%s\")", terraformCodeFilepath) } fmt.Fprintf( terraformCodeFile, ` terraform { backend "http" { address = "http://localhost:%d/%s" } } %s %s %s %s %s `, configuration.TERRAFORM_STATE_SERVER_PORT_NUMBER, terraformConfig.TerraformProject, remoteState, strings.Join(providerStanzas, "\n"), strings.Join(variableStanzas, "\n"), strings.Join(moduleStanzas, "\n"), strings.Join(outputStanzas, "\n"), ) ... } ``` I'm not saying my code is the best way to do this or you should use it, just wanted to share the approach I took.
❤️ 1
forest commented 2021-08-19 18:51:59 +00:00
Copy Link

I also built a UI for the terraform plan and apply process. It generates an SVG diagram of the terraform modules & resources using graphviz dot and post-processes the SVG to make it style-able within HTML. then the HTML-based client GUI application applies CSS animations to the SVG elements based on thier current build/apply status.

https://git.sequentialread.com/forest/rootsystem/src/branch/master/automation/terraformSvg.go

https://git.sequentialread.com/forest/seedpacket/src/branch/master/webcontent/wizard/launch/infrastructure-provisioning.js#L234

STEP 2: update the css classes on the svg based on the terraform apply status to animate it.

Demo video of this is here: https://picopublish.sequentialread.com/files/automated-server-garden-demo1.mp4

The specific part im talking about starts at 18 minutes in:

image

Yes this whole process was terribly hacky in more than 1 way, but I did believe in it, and I still do--

I think that building a GUI for these awesome and powerful "enterprise automation" apps like terraform might be decent way to bridge the gap between the "graybeards" and the "soccer moms".

I also built a UI for the terraform plan and apply process. It generates an SVG diagram of the terraform modules & resources using graphviz `dot` and post-processes the SVG to make it style-able within HTML. then the HTML-based client GUI application applies CSS animations to the SVG elements based on thier current build/apply status. https://git.sequentialread.com/forest/rootsystem/src/branch/master/automation/terraformSvg.go https://git.sequentialread.com/forest/seedpacket/src/branch/master/webcontent/wizard/launch/infrastructure-provisioning.js#L234 > STEP 2: update the css classes on the svg based on the terraform apply status to animate it. Demo video of this is here: https://picopublish.sequentialread.com/files/automated-server-garden-demo1.mp4 The specific part im talking about starts at 18 minutes in: ![image](/attachments/127d5e74-9b9a-48f0-b339-c823e8da64e7) Yes this whole process was terribly hacky in more than 1 way, but I did believe in it, and I still do-- I think that building a GUI for these awesome and powerful "enterprise automation" apps like terraform might be decent way to bridge the gap between the "graybeards" and the "soccer moms".
image.png
403 KiB
decentral1se added this to the Portability testing (software/hardware) milestone 2021-09-09 14:27:30 +00:00
decentral1se commented 2021-09-22 07:24:21 +00:00
Author
Owner
Copy Link

Refs:

Refs: - https://github.com/hashicorp/terraform/issues/16648 - https://learn.hashicorp.com/tutorials/terraform/automate-terraform - modules e.g. https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs - https://cloudinit.readthedocs.io/en/latest/topics/examples.html - https://www.digitalocean.com/community/tutorials/how-to-use-cloud-config-for-your-initial-server-setup - `TF_IN_AUTOMATION=1 terraform apply -input=false -auto-approve -json -var="x=y"` - runs terraform and spits out everything as JSON which is super easy to parse from `abra` - https://www.terraform.io/docs/internals/machine-readable-ui.html
decentral1se self-assigned this 2021-09-22 07:24:34 +00:00
decentral1se added this to the Beta release (software) project 2021-09-29 13:48:16 +00:00
decentral1se commented 2021-09-30 13:29:56 +00:00
Author
Owner
Copy Link

Next steps for the Terraform investigation:

  • can secrets be kept out of the state files? if not, this might not be usable
  • we should put the state files into the ~/.abra/servers/foo/... folders alongside the existing env files. this fits in with our existing shared git repository workflow
Next steps for the Terraform investigation: - can secrets be kept out of the state files? if not, this might not be usable - we should put the state files into the `~/.abra/servers/foo/...` folders alongside the existing env files. this fits in with our existing shared git repository workflow
decentral1se removed their assignment 2021-10-05 10:05:37 +00:00
decentral1se commented 2021-10-05 10:08:17 +00:00
Author
Owner
Copy Link

I liked how simple it was to spec out creating a Hetzner VPS:

terraform {
  required_providers {
    hcloud = {
      source = "hetznercloud/hcloud"
      version = "1.31.1"
    }
  }
}

variable "hcloud_token" {}

provider "hcloud" {
  token = "${var.hcloud_token}"
}

resource "hcloud_server" "web" {
  name        = "terraform-test"
  image       = "debian-10"
  server_type = "cx11"
}

then terraform init, terraform plan and terraform apply basically.

I liked how simple it was to spec out creating a Hetzner VPS: ``` terraform { required_providers { hcloud = { source = "hetznercloud/hcloud" version = "1.31.1" } } } variable "hcloud_token" {} provider "hcloud" { token = "${var.hcloud_token}" } resource "hcloud_server" "web" { name = "terraform-test" image = "debian-10" server_type = "cx11" } ``` then `terraform init`, `terraform plan` and `terraform apply` basically.
knoflook commented 2021-10-05 10:40:54 +00:00
Owner
Copy Link

terraform looks really cool as a tool and has a pretty big library of providers from what I see but I have some concerns about hashicorp closing it down at some point, also this opens the door for using AWS, GCP, Azure and other questionable services. I also want to refer to this issue: #91 and the question "how can we keep this anti-capitalist af?".
Even though terraform is unquestionably a cool tool I think we might want to avoid incorporating it in this project. I'm not saying "absolutely no!" though and if it saves us enough time and we can make good use of it i'm ok with going that way

terraform looks really cool as a tool and has a pretty big library of providers from what I see but I have some concerns about hashicorp closing it down at some point, also this opens the door for using AWS, GCP, Azure and other questionable services. I also want to refer to this issue: https://git.coopcloud.tech/coop-cloud/organising/issues/91 and the question "how can we keep this anti-capitalist af?". Even though terraform is unquestionably a cool tool I think we might want to avoid incorporating it in this project. I'm not saying "absolutely no!" though and if it saves us enough time and we can make good use of it i'm ok with going that way
👀 1
decentral1se commented 2021-10-18 18:37:25 +00:00
Author
Owner
Copy Link

FWIW I like your argumentation @knoflook and am moving on with implementing as-thin-as-possible integrations and leaving Terraform. https://github.com/libdns/libdns seems like a solid general DNS library and I'll get the Gandi implementation done shortly.

FWIW I like your argumentation @knoflook and am moving on with implementing as-thin-as-possible integrations and leaving Terraform. https://github.com/libdns/libdns seems like a solid general DNS library and I'll get the Gandi implementation done shortly.
knoflook commented 2021-10-19 08:22:40 +00:00
Owner
Copy Link

awesome! that means we can contribute our code to other projects which is always cool 🖤

awesome! that means we can contribute our code to other projects which is always cool 🖤
❤️ 1
knoflook commented 2021-10-19 08:48:41 +00:00
Owner
Copy Link

I'll get the Gandi implementation done shortly.

are you going to use your private github account for it or is there a co-op cloud github account that we should use for maintaining our providers?

> I'll get the Gandi implementation done shortly. are you going to use your private github account for it or is there a co-op cloud github account that we should use for maintaining our providers?
decentral1se commented 2021-10-19 10:03:38 +00:00
Author
Owner
Copy Link

@knoflook I've made a start in:

https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/cli/domain

And storing library code in:

https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/pkg/dns

If you're rocking Gandi, abra domain list -p gandi <zone> works right now

If you want to add a separate provider, follow https://github.com/libdns/libdns#implementing-new-providers and host it in a separate repo? Can be under the coop-cloud/... namespace or? Like we do with tagcmp.

@knoflook I've made a start in: > https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/cli/domain And storing library code in: > https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/pkg/dns If you're rocking Gandi, `abra domain list -p gandi <zone>` works right now If you want to add a separate provider, follow https://github.com/libdns/libdns#implementing-new-providers and host it in a separate repo? Can be under the `coop-cloud/...` namespace or? Like we do with `tagcmp`.
knoflook commented 2021-10-19 11:00:50 +00:00
Owner
Copy Link

If you want to add a separate provider, follow https://github.com/libdns/libdns#implementing-new-providers and host it in a separate repo? Can be under the coop-cloud/... namespace or? Like we do with tagcmp.

ah yeah i forgot that you can fork from gitea to github. I started working on porkbun provider but I have to educate myself on how go packages work - i have some library code written but i don't know how to execute it 🤦

> If you want to add a separate provider, follow https://github.com/libdns/libdns#implementing-new-providers and host it in a separate repo? Can be under the coop-cloud/... namespace or? Like we do with tagcmp. ah yeah i forgot that you can fork from gitea to github. I started working on porkbun provider but I have to educate myself on how go packages work - i have some library code written but i don't know how to execute it 🤦
forest commented 2021-10-19 13:52:28 +00:00
Copy Link

Go packages (modules image ) are typically git repos -- the package ID is the repo URL.

So you would 1st create the repo, say https://git.coopcloud.tech/coop-cloud/my-thingy

Say that the go source files are located at the root of the repository. Then you would go there and run
go mod init git.coopcloud.tech/coop-cloud/my-thingy

If the source files are in a sub folder, say, pkg instead, it would be
go mod init git.coopcloud.tech/coop-cloud/my-thingy/pkg

From there you should be able to run go run, go build, go test, and you should be good to go. Once you push the go.mod and go.sum files that go mod init created, other folks should be able to pull in your package by running
go get git.coopcloud.tech/coop-cloud/my-thingy

Go packages (modules ![image](/attachments/d2ab0383-fa9a-4deb-8767-9c63ad35d4b9) ) are typically git repos -- the package ID is the repo URL. So you would 1st create the repo, say https://git.coopcloud.tech/coop-cloud/my-thingy Say that the go source files are located at the root of the repository. Then you would go there and run `go mod init git.coopcloud.tech/coop-cloud/my-thingy` If the source files are in a sub folder, say, `pkg` instead, it would be `go mod init git.coopcloud.tech/coop-cloud/my-thingy/pkg` From there you should be able to run `go run`, `go build`, `go test`, and you should be good to _go_. Once you push the `go.mod` and `go.sum` files that `go mod init` created, other folks should be able to pull in your package by running `go get git.coopcloud.tech/coop-cloud/my-thingy`
image.png
1010 B
decentral1se commented 2021-10-19 20:59:25 +00:00
Author
Owner
Copy Link
mkdir myprovider && cd myprovider
go mod init coopcloud.tech/myprovider
git init && git add . && git commit -m "init"
go mod tidy # when you add an import or go install

Some other stuff but that basically it 👌

We setup coop-cloud/coopcloud.tech#20 so we can go install coopcloud.tech/... stuff because Fux Github.

``` mkdir myprovider && cd myprovider go mod init coopcloud.tech/myprovider git init && git add . && git commit -m "init" go mod tidy # when you add an import or go install ``` Some other stuff but that basically it 👌 We setup https://git.coopcloud.tech/coop-cloud/coopcloud.tech/issues/20 so we can `go install coopcloud.tech/...` stuff because Fux Github.
decentral1se commented 2021-10-20 15:00:59 +00:00
Author
Owner
Copy Link

The Gandi implementation is now out on latest, I've just ran:

abra record create -p gandi autonomic.zone A zigzag 192.168.178.24
abra record list -p gandi autonomic.zone
abra record remove -p gandi autonomic.zone A zigzag

Probably needs a UI/UIX pass but anyway, functional for now.

I'm gonna switch this issue over to spicing abra up as we've prototyped integrations with 3rd party providers now and people can add their own as needed or convince us to do it if they really need them ;)

I'm moving on.

The Gandi implementation is now out on latest, I've just ran: ``` abra record create -p gandi autonomic.zone A zigzag 192.168.178.24 abra record list -p gandi autonomic.zone abra record remove -p gandi autonomic.zone A zigzag ``` Probably needs a UI/UIX pass but anyway, functional for now. I'm gonna switch this issue over to [spicing abra up](https://git.coopcloud.tech/coop-cloud/organising/projects/9) as we've prototyped integrations with 3rd party providers now and people can add their own as needed or convince us to do it if they really need them ;) I'm moving on.
decentral1se removed this from the Beta release (software) project 2021-10-20 15:01:08 +00:00
decentral1se added this to the Spicing abra up project 2021-10-20 15:01:11 +00:00
decentral1se changed title from 3rd party go-abra integration thread to 3rd party abra integration thread 2021-11-13 22:20:05 +00:00
decentral1se commented 2023-01-18 17:46:04 +00:00
Author
Owner
Copy Link

Maybe gonna close off this mega thread now and we can open specific tickets for specific integrations as needed.

Maybe gonna close off this mega thread now and we can open specific tickets for specific integrations as needed.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
Labels
Clear labels   
abra

Everything to do with abra

  
abra-gandi

Abra Gandi plugin

  
awaiting-feedback

Ping/pong on comms

  
backups

   
bug

Something is not working

  
build

Go build related issues

  
ci/cd

Getting the robots into the mix

  
community organising

Opening this thing up

  
contributing

Contributors stuff

  
coopcloud.tech

Our main website

  
democracy

Democratic decision making

  
design

Design thinking required

  
documentation

Let's write things together

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
finance

Money things

  
funding

Anything related to grant funding

  
good first issue

Easy start with development

  
help wanted

Need some help

  
installer

Installation related issues

  
kadabra

   
performance

Performance related

  
proposal

Large change which requires feedback & decisin making

  
question

More information is needed

  
recipes.coopcloud.tech

Our recipes catalogue

  
security

Securing our shit

  
test

Unit or integration test suite

  
wontfix

This won't be fixed

No Label
abra
abra-gandi
awaiting-feedback
backups
bug
build
ci/cd
community organising
contributing
coopcloud.tech
democracy
design
documentation
duplicate
enhancement
finance
funding
good first issue
help wanted
installer
kadabra
performance
proposal
question
recipes.coopcloud.tech
security
test
wontfix
Milestone
Clear milestone
No items
No Milestone
Portability testing (software/hardware)
Projects
Clear projects
No project
Spicing abra up
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: coop-cloud/organising#106
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?