Docker image, & local development docker-compose.yml (#2)

Adds:
- a Docker image, which can be used in both development and production
- a `docker-compose.yml` file for local development (could probably be adapted for production deployments)

Testing:
- `git checkout docker`
- `docker-compose up`
- pray 🙏
- go to http://localhost:5000

## App architecture

I added the ability to load secret config variables (`HUB_TOKEN`, `STRIPE_SECRET_KEY` etc) from files, to support [Docker Secrets](https://docs.docker.com/engine/swarm/secrets/) natively. The code should probably log an error if a specified `HUB_TOKEN_FILE` doesn't exist instead of failing silently..

## Docker architecture

This uses a multi-stage build to reduce the size of the final image -- having pipenv install to a predefined virtualenv, and then copying that over.

The compose file doesn't include a definition for a cron runner service, and I haven't tested running one yet. Here be dragons!

You can rebuild the image locally using `docker-compose build`, but this isn't required for changes to the app code, only if you edit the `Dockerfile`, or want to publish your image for use on a swarm server (in which case you will need to edit the image name to put in your own Docker hub credentials).

Currently, the image is rebuilt (should set up auto-tagging..) and published with every push to this 3wordchant/capsul-flask fork.

Reviewed-on: https://git.autonomic.zone/3wordchant/capsul-flask/pulls/2
Co-authored-by: 3wordchant <3wordchant@noreply.git.autonomic.zone>
Co-committed-by: 3wordchant <3wordchant@noreply.git.autonomic.zone>

.gitignore

@@ -1,5 +1,6 @@
 notes.txt
 .env
+.env.bak
 .vscode
 
 *.pyc

capsulflask/console.py

@@ -199,6 +199,14 @@ def create():
   capacity_avaliable = current_app.config["HUB_MODEL"].capacity_avaliable(512*1024*1024)
   errors = list()
 
+  affordable_vm_sizes = dict()
+  for key, vm_size in vm_sizes.items():
+    # if a user deposits $7.50 and then creates an f1-s vm which costs 7.50 a month, 
+    # then they have to delete the vm and re-create it, they will not be able to, they will have to pay again.
+    # so for UX it makes a lot of sense to give a small margin of 25 cents for usability sake
+    if vm_size["dollars_per_month"] <= account_balance+0.25:
+      affordable_vm_sizes[key] = vm_size
+
   if request.method == "POST":
     if "csrf-token" not in request.form or request.form['csrf-token'] != session['csrf-token']:
       return abort(418, f"u want tea")
@@ -209,6 +217,8 @@ def create():
       errors.append("Size is required")
     elif size not in vm_sizes:
       errors.append(f"Invalid size {size}")
+    elif size not in affordable_vm_sizes:
+      errors.append(f"Your account must have enough credit to run an {size} for 1 month before you will be allowed to create it")
 
     if not os:
       errors.append("OS is required")
@@ -260,13 +270,6 @@ def create():
       
       return redirect(f"{url_for('console.index')}?created={id}")
   
-  affordable_vm_sizes = dict()
-  for key, vm_size in vm_sizes.items():
-    # if a user deposits $7.50 and then creates an f1-s vm which costs 7.50 a month, 
-    # then they have to delete the vm and re-create it, they will not be able to, they will have to pay again.
-    # so for UX it makes a lot of sense to give a small margin of 25 cents for usability sake
-    if vm_size["dollars_per_month"] <= account_balance+0.25:
-      affordable_vm_sizes[key] = vm_size
 
   for error in errors:
     flash(error)

capsulflask/hub_api.py

@@ -178,4 +178,4 @@ def on_create_claimed(payload, host_id):
     network_name=payload['network_name'],
     public_ipv4=payload['public_ipv4'],
     ssh_authorized_keys=list(map(lambda x: x["name"], payload['ssh_authorized_keys'])),
-  )
+  )

capsulflask/hub_model.py

@@ -227,13 +227,12 @@ class CapsulFlaskHub(VirtualizationInterface):
       except:
         # no need to do anything here since if it cant be parsed then generic_operation will handle it.
         pass
-
-    if error_message != "":
-      raise ValueError(f"create capsul operation {operation_id} on {assigned_hosts} failed with {error_message}")
       
     if number_of_assigned != 1:
       assigned_hosts_string = ", ".join(assigned_hosts)
       raise ValueError(f"expected create capsul operation {operation_id} to be assigned to one host, it was assigned to {number_of_assigned} ({assigned_hosts_string})")
+    if error_message != "":
+      raise ValueError(f"create capsul operation {operation_id} on {assigned_hosts_string} failed with {error_message}")
       
 
   def destroy(self, email: str, id: str):

capsulflask/shell_scripts/capacity-avaliable.sh

@@ -3,7 +3,7 @@
 # check available RAM and IPv4s
 
 ram_bytes_to_allocate="$1"
-ram_bytes_available="$(($(grep Available /proc/meminfo | grep -o '[0-9]*') * 1024))"
+ram_bytes_available=$(grep -E "^(size|memory_available_bytes)" /proc/spl/kstat/zfs/arcstats | awk '{sum+=$3} END {printf "%.0f", sum}')
 ram_bytes_remainder="$((ram_bytes_available - ram_bytes_to_allocate))"
 
 if echo "$ram_bytes_to_allocate" | grep -vqE "^[0-9]+$"; then
@@ -11,8 +11,8 @@ if echo "$ram_bytes_to_allocate" | grep -vqE "^[0-9]+$"; then
   exit 1
 fi
 
-# 0.25GB
-if [ "$ram_bytes_remainder" -le $((1 * 1024 * 1024 * 1024 / 4)) ]; then
+# 20GB
+if [ "$ram_bytes_remainder" -le $((20 * 1024 * 1024 * 1024)) ]; then
   echo "VM is requesting more RAM than $(hostname -f) has available."
   echo "Bytes requested: $ram_bytes_to_allocate"
   echo "Bytes available: $ram_bytes_available"

capsulflask/shell_scripts/create.sh

@@ -6,7 +6,6 @@
 
 vmname="$1"
 template_file="/tank/img/$2"
-qemu_tank_dir="/tank"
 vcpus="$3"
 memory="$4"
 pubkeys="$5"
@@ -51,40 +50,40 @@ if echo "$public_ipv4" | grep -vqE "^[0-9.]+$"; then
   exit 1
 fi
 
-disk="$vmname.qcow2"
-cdrom="$vmname.iso"
-xml="$vmname.xml"
+disk="/tank/vm/$vmname.qcow2"
+cdrom="/tank/vm/$vmname.iso"
+xml="/tank/vm/$vmname.xml"
 
 if [ -f /tank/vm/$vmname.qcow2 ]; then
     echo "Randomly generated name matched an existing VM! Odds are like one in a billion. Buy a lotto ticket."
     exit 1
 fi
 
-cp "$template_file" "/tank/vm/$disk"
+cp "$template_file" "$disk"
 cp /tank/config/cyberia-cloudinit.yml /tmp/cloudinit.yml
 echo "$pubkeys" | while IFS= read -r line; do
   echo "      - $line" >> /tmp/cloudinit.yml
 done
 
-cloud-localds "/tank/vm/$cdrom" /tmp/cloudinit.yml
+cloud-localds "$cdrom" /tmp/cloudinit.yml
 
-qemu-img resize "/tank/vm/$disk" "$root_volume_size"
+qemu-img resize "$disk" "$root_volume_size"
 virt-install \
     --memory "$memory" \
     --vcpus "$vcpus" \
     --name "$vmname" \
-    --disk "$qemu_tank_dir/vm/$disk",bus=virtio \
-    --disk "$qemu_tank_dir/vm/$cdrom",device=cdrom \
+    --disk "$disk",bus=virtio \
+    --disk "$cdrom",device=cdrom \
     --os-type Linux \
     --os-variant generic \
     --virt-type kvm \
     --graphics vnc,listen=127.0.0.1 \
-    --network network=$network_name,model=virtio \
+    --network network=$network_name,filterref=clean-traffic,model=virtio \
     --import \
-    --print-xml > "/tank/vm/$xml"
+    --print-xml > "$xml"
 
-chmod 0600 "/tank/vm/$xml" "/tank/vm/$disk" "/tank/vm/$cdrom"
-virsh define "/tank/vm/$xml"
+chmod 0600 "$xml" "$disk" "$cdrom"
+virsh define "$xml"
 virsh start "$vmname"
 
 echo "success"

capsulflask/static/style.css

@@ -1,8 +1,8 @@
 html {
-  color: #241e1e;
+  color: #bdc7b8;
   font: calc(0.40rem + 1vmin) monospace;
   overflow-y: scroll;
-  background-color: #bdc7b8;
+  background-color: #241e1e;
 }
 
 body {
@@ -19,8 +19,8 @@ body {
 }
 
 a {
-  color:#00517a;
-  text-shadow: 1px 1px 0px #eee;
+  color:#6CF;
+  text-shadow: 1px 1px 0px #000c;
 }
 
 a.no-shadow {
@@ -28,7 +28,7 @@ a.no-shadow {
 }
 
 a:hover, a:active, a:visited {
-  color: #323417;
+  color: #b5bd68;
 } 
 
 .nav-links  a {
@@ -59,11 +59,11 @@ h1, h2, h3, h4, h5 {
   margin: initial;
   padding: initial;
   text-transform: uppercase;
-  text-shadow: 2px 2px 0px #eee;
+  text-shadow: 2px 2px 0px #0007;
 }
 
 main {
-  border: 1px dashed #241e1e;
+  border: 1px dashed #bdc7b8;
   padding: 1rem;
   margin-bottom: 2em;
 
@@ -143,7 +143,7 @@ input, textarea, select, label {
 input, select, textarea {
   outline: 0;
   padding: 0.25em 0.5em;
-  color: #241e1e;
+  color: #bdc7b8;
   background-color: #bdc7b805;
 }
 

capsulflask/templates/account-balance.html

@@ -49,6 +49,8 @@
           {% if btcpay_enabled %}
           <li><a href="/payment/btcpay">Add funds with Bitcoin/Litecoin/Monero (btcpay)</a></li>
           {% endif %}
+    
+          <li>Cash: email <a href="mailto:treasurer@cyberia.club">treasurer@cyberia.club</a></li>
         </ul>
       </li>
     </ul>

capsulflask/templates/base.html

@@ -13,7 +13,7 @@
 <nav>
   <div class="row justify-space-between half-margin">
     <div>
-      🦉 <a href="/"><b>YOLOCOLO</b></a>
+      <a href="/"><b>Capsul</b></a>💊
     </div>
     <div>
       &nbsp;
@@ -27,6 +27,7 @@
   <div class="row justify-center half-margin wrap nav-links">
     <a href="/pricing">Pricing</a>
     <a href="/faq">FAQ</a>
+    <a href="/changelog">Changelog</a>
 
     {% if session["account"] %} 
       <a href="/console">Capsuls</a>
@@ -46,12 +47,11 @@
 </main>
 {% block subcontent %}{% endblock %}
 <footer>
-  This server runs <a
-    href="https://giit.cyberia.club/~forest/capsul-flask">capsul-flask</a> by
-  Cyberia Computer Club, available under the <a
-    href="https://creativecommons.org/licenses/by-sa/4.0/">Attribution-ShareAlike
-    4.0 International</a> licence.<br/><br/>
-  <a href="https://git.autonomic.zone/3wordchant/capsul-flask/src/branch/yolocolo/capsulflask{% block pagesource %}{% endblock %}">View page source</a>
+  (c) Attribution-ShareAlike 4.0 International <br/>
+  &nbsp;&nbsp;&nbsp;&nbsp;A service by Cyberia Computer Club 2020-<span class="bigtext">∞</span> <br/>
+  <br/>
+  <br/>
+  <a href="https://giit.cyberia.club/~forest/capsul-flask/tree/master/capsulflask{% block pagesource %}{% endblock %}">View page source</a>
 </footer>
 </body>
 </html>

capsulflask/templates/faq.html

@@ -10,32 +10,82 @@
 <p>
  <ul>
    <li>
-     What is this?
-     <p>
-      This is a <strong>technical demo</strong> of <a
-       href="https://giit.cyberia.club/~forest/capsul-flask">Capsul</a>, for the
-      as-yet-untitled <a href="https://coops.tech">Cotech</a> server hosting
-      initiative, which you can <a
-      href="https://community.coops.tech/t/call-for-input-v2-co-op-vps-survey/2802/9">read
-     about on the Cotech forum</a>.
-     </p>
+     Which instance type should I buy?
+     <p>There are no hard rules for this sort of thing, but here are some guidelines:</p>
+     <p>f1-xs: blog, vpn, bot, cgit</p>
+     <p>f1-s: a bot, owncloud, gitea, popular blog</p>
+     <p>f1-m: docker host, build system</p>
+     <p>f1-l: large webservice, rotund java app</p>
+     <p>f1-x: gitlab (wow such memory very devops</p>
+     <p>f1-xx: something gargantuan</p>
    </li>
    <li>
-     What do you mean, "technical demo"?
-     <p>No backups</p>
-     <p>No service level agreement</p>
-     <p>"Best effort" support</p>
+     How do I log in?
+     <p>ssh to the ip provided to you using the cyberian user.</p>
+     <pre class='code'>$ ssh cyberian@1.2.3.4</pre>
+     <p>For more information, see <a href="/about-ssh">Understanding the Secure Shell Protocol (SSH)</a>.</p>
    </li>
    <li>
-     Where can I get this, but, more reliable?
-     <p>Cyberia, the authors of this platform, run the canonical instance, <a
-      href="https://capsul.org">Capsul.org</a>, on hardware they own. Please
-     send them your money! (cash, crypto, or card accepted).</p>
+     How do I change to the root user?
+     <p>The cyberian user has passwordless sudo access by default. This should work:</p>
+     <pre class='code'>
+# Linux
+$ sudo su -
+
+# OpenBSD
+$ doas su -</pre>
    </li>
    <li>
-     How do I use this system?
-     <p>Please see <a href="https://capsul.org/faq">the official Capsul FAQ
-       page</a>.</p>
+     Do you offer reverse DNS?
+     <p>We do, but right now it's a manual process. Shoot us an email and we'll get it done.</p>
+   </li>
+   <li>
+    What if I don't pay / don't maintain my payments?
+    <p>Your VM will eventually be deleted.
+       Capsul will send you a few inoffensive reminders as that termination date approaches.
+    </p>
+   </li>
+   <li>
+    Besides my virtual machines and payments, what information do you keep about me?
+     <p>We associate an email address with every VM so that we can track payment and respond to support requests.</p>
+     <p>If you pay with a credit card, Stripe stores some additional details about you that we literally cannot delete.</p>
+   </li>
+   <li>
+     What can I do with my VM?
+     <p>Make it into a mailserver, a tor relay, a VPN host, whatever you'd like - we do have one small request, though.</p>
+     <p>Crypto mining on capsul is currently considered obnoxious behavior, because the hashrates on our CPUs is so low and because mining crypto consumes entire processor cores that could have otherwise been shared between many dozens of other users.</p>
+     <p>In the future, if we have plentiful CPU resources, we may come out with a tier more suitable for mining - maybe a high cpu tier or similar, where each VM gets a full dedicated core and sharing them is not anticipated.</p>
+     <p>We will never snoop on your traffic or inspect what's going on inside of our customer virtual machines - we don't want to. We hope that you'll extend us a similar courtesy and try not to use too much of our shared CPU resources. Capsul is currently a shared (resource-wise) world, and we all must live in it together!</p>
+     <p>Also, mandatory: our systems exist within the USA, and as such those systems are bound by US law.</p>
+   </li>
+   <li>
+     Can you recover my passwords/insert new keys?
+     <p>Can we? Technically yes. Will we? No, never. It would violate the trust that our users have in us.
+     We have no interest in touching client VMs after they're running.
+     We promise to keep your machines running smoothly.
+     If you lose access to your VM, that's on you.</p>
+   </li>
+   <li>
+     Do you offer refunds?
+     <p>Not now, but email us and we can probably figure something out.</p>
+   </li>
+   <li>
+    Where do the VMs run? Is it on a machine that you guys own/control?
+     <p>Capsul runs on a server named Baikal which Cyberia built from scratch & mailed to a datacenter
+      in Georgia called CyberWurx. CyberWurx staff installed it for us in a rack space that
+      Cyberia pays for. </p>
+   </li>
+   <li>
+     Do you offer support?
+     <p>Yep, see <a href="/support">our support page</a>.</p>
+   </li>
+   <li>
+     Do you have an SLA?
+     <p>No, but we normally respond pretty quickly.</p>
+   </li>
+   <li>
+    Will you implement feature X?
+    <p>Maybe! Email <a href="mailto:ops@cyberia.club">ops@cyberia.club</a> and ask us about it.</p>
    </li>
  </ul>
 </p>

capsulflask/templates/index.html

@@ -1,26 +1,31 @@
 {% extends 'base.html' %}
 
-{% block content %}
-  <h1>
-  <pre>
-             _                 _
- _   _  ___ | | ___   ___ ___ | | ___
-| | | |/ _ \| |/ _ \ / __/ _ \| |/ _ \
-| |_| | (_) | | (_) | (_| (_) | | (_) |
- \__, |\___/|_|\___/ \___\___/|_|\___/
- |___/
 
+{% block content %}
+  <h1>CAPSUL</h1>
+  <pre>
+       .-.  
+      /:::\ 
+     /::::/ 
+    / `-:/  
+   /    /   
+   \   /    
+    `"`     
   </pre>
-  <span>Co-operative hosting using <a href="https://cyberia.club">Cyberia</a>'s Capsul</span>
+  <span>Simple, fast, private compute by <a href="https://cyberia.club">cyberia.club</a></span>
 {% endblock %}
 
 {% block subcontent %}
 <p>
  <ul>
-   <li>Sign up for an account!</li>
-   <li>Add some funds!</li>
-   <li>Create a VPS!</li>
-   <li>Give your feedback!</li>
+   <li>Low friction: simply log in with your email address and fund your account with Credit/Debit or Cryptocurrency</li>
+   <li>All root disks are backed up at no charge</li>
+   <li>All storage is fast, local, and solid-state</li>
+   <li>All network connections are low latency</li>
+   <li>Supported by amazing volunteers from Cyberia</li>
+   <li>Upfront prices, no confusing billing</li>
+   <li>Operated by a Minnesota non-profit organization that will never exploit you</li>
+   <li>We donate a portion of our proceeds to likeminded hacker groups around the globe</li>
  </ul>           
 </p>
 {% endblock %}

capsulflask/templates/pricing.html

@@ -7,17 +7,42 @@
     <h1>CAPSUL TYPES & PRICING</h1>
   </div>
   <div class="row half-margin">
-    <p>
-      Rates for this service aren't set yet. You can see Cyberia's Capsul pricing
-      on <a href="https://capsul.org/pricing">their website</a>.
-    </p>
-  </div>
-  <div>
-    <pre>
-      SUPPORTED OPERATING SYSTEMS:
-
-      {% for os_id, os in operating_systems.items() %}   - {{ os.description }} 
+    <table>
+      <thead>
+        <tr>
+          <th>type</th>
+          <th>monthly*</th>
+          <th>cpus</th>
+          <th>mem</th>
+          <th>ssd</th>
+          <th>net</th>
+        </tr>
+      </thead>
+      <tbody>
+      {% for vm_size_key, vm_size in vm_sizes.items() %}
+        <tr>
+          <td>{{ vm_size_key }}</td>
+          <td>${{ vm_size['dollars_per_month'] }}</td>
+          <td>{{ vm_size['vcpus'] }}</td>
+          <td>{{ vm_size['memory_mb'] }}</td>
+          <td>25G</td>
+          <td>{{ vm_size['bandwidth_gb_per_month'] }}</td>
+        </tr>
       {% endfor %}
-    </pre>
+      </tbody>
+    </table>
+  </div>
+  <div class="row half-margin">
+    <pre>
+    * net is calculated as a per-month average
+    * vms are billed for a minimum of 24 hours upon creation
+    * all VMs come standard with one public IPv4 address
+    
+    SUPPORTED OPERATING SYSTEMS:
+
+
+    {% for os_id, os in operating_systems.items() %}   - {{ os.description }} 
+    {% endfor %}
+  </pre>
   </div>
 {% endblock %}

capsulflask/templates/support.html

@@ -7,14 +7,20 @@
     <h1>SUPPORT</h1>
   </div>
   <div class="row half-margin">
-    <a href="mailto:yolocolo@doesthisthing.work?subject=Please%20help!">yolocolo@doesthisthing.work</a> 
+    <a href="mailto:support@cyberia.club?subject=Please%20help!">support@cyberia.club</a> 
   </div>
 {% endblock %}
 
 {% block subcontent %}
 <p>
-  You can also find us on Matrix: <a
-    href="https://matrix.to/#/#untitled-hosting.public:autonomic.zone">#untitled-hosting.public:autonomic.zone</a>.
+  Note: We maintain a searchable archive of all support emails at 
+  <a href="https://lists.cyberia.club/~cyberia/support">https://lists.cyberia.club/~cyberia/support</a>
+</p>
+<p>
+  If you do not want your mail to appear in a public archive, email <a href="mailto:capsul@cyberia.club?subject=Please%20help!">capsul@cyberia.club</a> instead.
+</p>
+<p>
+  Please describe your problem or feature request, and we will do our best to get back to you promptly. Thank you very much.
 </p>
 {% endblock %}