toolshed/docs.coopcloud.tech
3
3
Fork 32
Code Issues 12 Pull Requests 7 Packages Releases Activity

shorter folder name, hex/nlnet add
continuous-integration/drone/push Build is passing
Details

Browse Source
This commit is contained in:
decentral1se
2026-10-29 15:44:53 +01:00
parent 37064cf4d2
commit 058dfe75de
10 changed files with 574 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/federation/funding-applications/culture-of-solidarity.md → docs/federation/funding/culture-of-solidarity.md
View File
docs/federation/funding-applications/ford-foundation.md → docs/federation/funding/ford-foundation.md
View File
docs/federation/funding-applications/index.md → docs/federation/funding/index.md
View File
docs/federation/funding/nlnet-codesupply.md
+567
View File
@@ -0,0 +1,567 @@
---
title: NLNet CodeSupply
---
# NLNet CodeSupply
> Status: **pending** ([Link to open call](https://nlnet.nl/propose/))
## System3: High Performance Scanner
## Reference
- [`0x29a/hestia`](https://codeberg.org/0x29a/hestia)
## Project Summary
System3 is a generalization of a docker SBOM generation and security scanning
tool called Hestia. It is intended to create a higher performance and more
extensible alternative to ScanCode.io.
Hestia is a proof of concept high speed, stand alone scanner for large numbers
of docker images currently built to scan for vulnerabilities in the Co-op Cloud
recipe catalog. This project will merge functionality from AboutCode to provide
a more scalable alternative architecture to ScanCode.io and the scancode cli.
The results of this project will be a Software Composition Analysis (SCA) tool
that analyzes docker containers, produce SBOMs, and use those to identify
vulnerable software. This vulnerability information will be enriched using open
sources (such as EPSS). This proof of concept will be pluggable, making it easy
to port scancode.io pipelines over to it.
System3 is a reference to part of the cyberneticist Stafford Beer's Viable
System Model.
## Proposed effort
- Total cost: €30030.00
## Explanation
### Which tasks will you undertake, and (roughly) how much time/effort will each take?
1.5 engineer weeks: Currently Hestia uses Syft and Trivy to build SBOMs and
scan them for vulnerabilities. Over the course of this project, Syft will be
replaced by AboutCode's container-inspector. Custom code fetching will be
replaced with skopeo (what AboutCode's fetchcode-container plugin uses under
the hood) to help identify missing or incorrectly specified docker images.
AboutCode's packageurl-python will be used to generate PURLs for vulnerability
identification. Where useful, AboutCode's commoncode will replace existing
utilities.
(SPDX generation will use spdx-tools, rather than adopting ScanCode's custom
implementation.)
1.5 engineer weeks: PURLdb and Vulernable code databases will be used, if
possible, to find vulnerabilities. If this works, they will replace Trivy as
the scanner (though other tools may also be used to identify vulnerabilities).
This accounts for both the exploration time and implementation (given
feasibility). If this is not feasible, this time will also include a problem
analysis.
6 engineer weeks: Turning Hestia into System3 will also mean changing the
underlying architecture. Currently a pipeline architecture passes messages
between workers (analogous to ScanCode.io's pipes). This will be abstracted
with a framework where worker functions advertise their capabilities using
decorators, and interfaces and pipelines are specified using YAML. The actual
queue management functionality will be abstracted so that worker developers
simply need to understand the data model of their worker's domain and can
ignore all other details of the system.
This time accounts for framework design, implementation, usability experiments,
and improvements based on feedback.
Additional work: Interspersed through the development process, I will be adding
unit testing, property testing, and integration testing. This will all be run
through continuous integration to detect breaking changes in upstream
dependencies and bugs induced by code changes.
### What hourly/daily/monthly rate did you use?
€65/hr x 40 hour/week x 11 weeks = 28600 + 5% margin of error = 30030.00.
### Will there be any expenses?
There are no additional expenses.
### Next Steps and Strech Goals:
If I'm able to complete other tasks, I will add additional workers such as:
- A worker that connects to matrix and reports vulnerabilities to a chat
- A worker that files security issues based on SCA results against Gitea, Forjeo, or GitHub repositories
As a fast follow, I also plan to build a tool that turns pipeline definitions
into cloud pipelines using n8n and Terraform.
### Comparison
System3 will be most similar to ScanCode.io and the ScanCode toolkit CLI
application.
The ScanCode toolkit schedules jobs against the ScanCode.io server. The System3
cli will simply be a scaled and stripped down version of the architecture,
allowing it to run independently from the representation layer.
ScanCode.io is a django web service that exposes a REST API and a web UI. The
underlying pipeline data model tightly couples multiple unrelated concepts
(such as django presentation). Failure to separate concerns degrades
performance by increasing the context needed to execute operations, increasing
the minimum lines of code involved in any pipeline, and by preventing efficient
horizontal scaling (that is, scaling by component not scaling the entire
platform).
Hestia already presents more clean separation of concerns that allows it to
scale any individual component simply by increasing the number of worker
threads. A queuing oriented design makes it possible to transition from threads
in a single application, as it currently is, to a server-less architecture with
minimal changes. With appropriate abstraction, workers can run locally as
threads within a single application, or can be deployed to Scaleway Serverless
Functions, OVHcloud managed Knative, AWS Lambda, Google Cloud Functions,
Beam.cloud, Kubernetes with Knative/OpenFaaS/Fission, or other frameworks,
without being completely rearchitected. When deployed to serverless platforms,
horizontal scaling can be handled by the platform. Serverless bound to queues
(such as RabbitMQ or SQS) is already a common architecture for systems that
must support horizontal scaling.
The failure to separate concerns also forces pipeline developers to understand
the concept of a project and a pipeline, in order to implement a pipe. Hestia
separates these concerns so that a worker simply needs to take input from a
queue and write to another queue.
System3 will further abstract this so that pipelines can be written using a
configuration language, like yaml or json, while workers are simply functions
and classes that focus on input and output. Presentation is completely
separated, reducing developer context and enabling more complexity in the way
that views can be managed. The System3 proof of concept will implement the
vulnerable code pipeline for docker containers.
An enterprise, especially one focused on technology such as Microsoft, Amazon,
or Google, may have thousands or millions of repositories, container images,
and servers to assess. Downloading each container is simply infeasible, even
for a project as small as Co-op Cloud with 2k docker images to scan. Hestia
automates the work of downloading, caching SBOM artifacts, and cleaning up so
that it doesn't run out of disk space. As I understand it, this requires
additional external scripts to emulate in ScanCode.
The failure to separate representation from the rest of the platform means that
data processing workloads can impact the performance of the web interface.
But the monolithic nature of ScanCode.io represents a larger missed opportunity
at generalization. Scanning for vulnerabilities in repositories or containers
can provide important security signals. These signals may be more appropriately
processed within an open source SIEM/XDM such as Wazuh or an existing internal
stack. Security signals don't need to be handled as a single atomic pipeline,
but can be processed as a stream. Streaming events can allow continuous
autonomous data enrichment, such as a vulnerability detected in a repository
triggering a search against a data lake to identify potential deployed
endpoints, then triggering dynamic vulnerability scanning against those
endpoints.
Why should a "project" be limited to things that can be represented as files?
Why not a list of IP addresses used by workers running NMAP with NSE
vulnerability detection scripts, producing vulnerability identification
results, which could be enriched using the very same enrichment as SCA? Why
should it not be possible to correlate vulnerabilities identified in SCA, SAST,
build time DAST, or IAST reports, with each other to find gaps in the security
process?
With the right architecture, these problems are solved by implementing workers
and wiring them into pipelines or writing new visualizations over data produced
by these workers. That is the vision of System3.
### Technical (and other) Challenges
There are three patterns my system needs to support:
- Queue with a single action. A worker consumes an item from the queue, processes the item, in serial.
- Pub/Sub. Work items are broadcast in parallel with no built-in mechanism for deduplication, allowing different types of work items to execute in parallel.
- Aggregation/Correlation. A work item may have several subtasks. Results of subtasks need to be collected up and correlated back to original projects. For example, a "recipe" (or application) has multiple versions, each version has multiple images, each image has multiple pieces of software, each piece of software may have multiple vulnerabilities. A worker needs to correlate vulnerabilities back to "recipes" at specific versions.
There are two modes:
- Serial
- Batch
There are two data source/sinks:
- Messages
- Data Lake
There are two target platforms:
- Local
- (host your own) Cloud
Each of these, and the interaction between them, creates multiple opportunities
for failure. The experience of the developer will be determined by how I
balance exposing vs hiding detail. The more I obscure, the easier the system
will be to use but the more limited it will be. This is often the key
difference between enterprise tooling (where everything must be adapted to a
wide range of scenarios) and small business tooling (where everything must be
as simple as possible).
Due to character limitations, I've enumerated these challenges in more detail
in the attached doc titled "Technical Challenges" (see below).
### Ecosystem (max. 2000 characters)
The tool will immediately be used by the Co-op Cloud project. Long term
operational funding and management will be handled by the federation.
Federation decisions are made democratically, via a resolution process. I'm
actively coordinating with Co-op Cloud stakeholders and have attached a draft
of the resolution 0XX (see below for draft) that describes the larger ecosystem
within which this fits.
One missing element is the security advisory announcement element (Co-op Cloud
Security Advisory, or CCSA). Security advisory management is also a gap within
the [AboutCode
ecosystem](https://aboutcode.org/docs/getting_started/software-security/#manage-risk-with-aggregated-vulnerability-data).
I will be writing a proposal for this as well, as I also have a proof of
concept for it.
Since I'm primarily using stable AboutCode libraries, coordination with
AboutCode should not be necessary. If it is, I can use GitHub issues and the
gliter channel (#aboutcode-org_discuss:gitter.im). During development I will
probably join the VulnerableCode meeting to make sure I'm tracking the
development of that project.
However, the main users will likely be the Co-op Cloud project at the
beginning. Once I can demonstrate that the system is stable and maintainable,
it will make sense to expand to influencing AboutCode to consider integration
opportunities.
## Background
### Personal
I have been a security engineer at Amazon (both the store and AWS) for almost
10 years, and was a consultant to companies including Amazon and Microsoft for
several years before that. During my time in AWS, I designed, built,
operationalized, and drove adoption for a security testing framework now used
by every AWS service and dozens of adjacent security teams. I designed a single
framework that was easy for security engineers to develop, developers to
onboard to and extend, and operators to maintain. The project was able to gain
so much momentum, because I started by pulling in multiple stakeholders to
understand the scope of the problem.
I have extensive experience with systems that must be highly performant. The
tool I replaced was a DAST scanner. The scanner my tooling replace scanned a
few dozen endpoints with 7 tests. My scanner ran dozens (and later hundreds) of
tests against 16k endpoints, and did so in hours instead of days. I have
implemented multiple orders of magnitude performance improvements to tooling
more than once.
I have developed other tooling as well, always bridging the gap between
developers and security, always focused on making using the same toolkit for
endpoint testing as build/integration testing. I have lead multiple consecutive
successful programs to drive security improvement at scale at AWS. AWS is a
highly technologically diverse environment. The combination of operating at
scale against heterogeneous systems and previous work as a consultant has given
me the diverse background I need to address similar problems globally.
## Other Funding (max. 1000 characters)
There is currently no other funding for this project.
## Attachments
* [`technical_challenges.pdf`](./technical_challenges.pdf)
* [`security_bus_design.pdf`](./security_bus_design.pdf)
## Resolution 0XX Draft
### Resolution 0XX: Security Event Management
- Deadline: Date
- Size: Medium
#### Summary
Create a federation wide Security Event Management process to address the
following concerns:
- End users and security researchers need a mechanism to report security
vulnerabilities in Co-op Cloud maintained recipes
- Operators need to know if the version they are deploying or have deployed is
vulnerable, or if there are security related actions they need to take
- Maintainers needed to know if a recipe they maintain needs a(n emergency)
security update
#### Details
##### Background
We have created a Co-op Cloud Security room on Matrix to coordinate on security
improvements, and have discussed security concerns during kite flying on
[2026-08-27](https://docs.coopcloud.tech/federation/kite-flying-pad-archive/#2026-08-27-12-utc),
[2026-08-06-19](https://docs.coopcloud.tech/federation/kite-flying-pad-archive/#2026-08-06-19-utc),
and
[2026-07-30](https://docs.coopcloud.tech/federation/kite-flying-pad-archive/#2026-07-30-12-utc)
(the later related to maturity models). We have also worked on at least one
[document](https://cloud.klasse-methode.it/s/aNitMQjeo5QpHs4?dir=/&editing=false&openfile=true)
outlining the security domains we are working within and to help focus on the
most critical areas.
Over the course of the past month, we have identified the most critical gaps as
described by the following user stories.
##### User Stories
- US1: As a Security Researcher, I have found a vulnerability in a Co-op Cloud
recipe. I would like to responsibly disclose this vulnerability to a Co-op
Cloud Security Contact so that users can be protected.
- US2: As an operator, I would like to be actively informed as soon as possible
when an exploitable vulnerability is detected in a recipe I am using so that
I can keep my end users safe.
- US3: As an operator, I would like to be able to find out if any recipes I'm
currently using have known vulnerabilities so that I can upgrade them in
order to keep my customers safe.
- US4: As a maintainer, I would like to be informed of vulnerabilities in
recipes I maintain so that I can update recipe definitions and tell consumers
to upgrade.
- US5: As an operator or maintainer, I would like to be able to run my own
vulnerability scans to identify new vulnerabilities so that I can keep ahead
of security team latency. Vulnerability scanners tend to produce a lot of
noise, so I'd like to be able to leverage security team pre-filtering for all
*known* issues so that I can focus on only *net new* issues.
##### Co-op Cloud Security Advisories (CCSA) feed
Security advisories are the foundation of any project's security program. They
allow a project to announce security events, including, but not limited to,
vulnerabilities that require upgrades, security incidents such as compromised
hosts, and security program changes.
The primary function of an advisory is to fulfill US2 by announcing
vulnerabilities as soon as there are actions that an end user can take to
mitigate the issue.
Feeds generally come in the form of an email list. Often the email list is
replicated to an RSS for easier tracking.
Within the Co-op Cloud recipe set, there are currently known issues with
recipes. Unfortunately the current way to alert users is to post in the general
channel. This is easy to miss and is not easily searchable.
The creation of a CCSA email list and RSS feed allows operators to quickly
learn about issues, as information becomes available, without needing to filter
through other chatter. With an RSS feed, additional automation (such as a
Matrix bot) becomes significantly easier.
##### Co-op Cloud Security Advisories (CCSA) catalog and Abra Integration
While email/RSS can fulfill US2, a catalog can help fulfill UC3. CCSA can be
written using a simple format such as yaml, toml, or json (so they can be
searched based on recipe, CVE, or other metadata), and output to email, RSS,
and a json vulnerability catalog. Such a catalog can be easily searched using
automated tooling, can be rendered into a searchable page, or can be integrated
into the abra deployment tool.
One specific request that multiple operators have repeated is for some kind of
"abra vulnerability check." This is a sequence diagram for populating feeds and
catalogs:
```
,--. ,---------. ,-----. ,-------. ,----.
|SE| |CCSA tool| |email| |catalog| |feed|
`-+' `----+----' `--+--' `---+---' `--+-'
| security advisory tml | | | |
|- - - - - - - - - - - ->| | | |
| | | | |
| | send from ccsa@security.coopcloud.tech | | |
| |---------------------------------------->| | |
| | | | |
| | build security.coopcloud.tech/ccsa.json | |
| |------------------------------------------------->| |
| | | | |
| | publish to security.coopcloud.tech/rss.xml| |
| |---------------------------------------------------------->|
| | | | |
,-+. ,----+----. ,--+--. ,---+---. ,--+-.
|SE| |CCSA tool| |email| |catalog| |feed|
`--' `---------' `-----' `-------' `----'
```
Once in a catalog, a user could run abra to test their own servers:
```
,----. ,----. ,----------. ,---------.
|user| |abra| |swarm host| |ccsa.json|
`--+-' `--+-' `-----+----' `----+----'
| abra sec_check | | |
|- - - - - - - - - - - - ->| | |
| | | |
| | get recipes | |
| |- - - - - - - - - - - - - - >| |
| | | |
| | deployed recipe/image list | |
| |< - - - - - - - - - - - - - -| |
| | | |
| | download if not cached |
| |- - - - - - - - - - - - - - -|- - - - - >|
| | | |
| | check against recipes |
| |- - - - - - - - - - - - - - -|- - - - - >|
| | | |
| | vulnerable matches| |
| |< - - - - - - - - - - - - - -|- - - - - -|
| | | |
| list of vulnerable apps | | |
|< - - - - - - - - - - - - | | |
| | | |
,--+-. ,--+-. ,-----+----. ,----+----.
|user| |abra| |swarm host| |ccsa.json|
`----' `----' `----------' `---------'
```
Here are command line mockups to help communicate the UX..
Check one host, find vulnerabilities:
```
$ abra ccsa_check example.coopcloud.tech
Downloading latest Co-op Cloud Security Advisories
|================================================| 100%
Checking........ complete!
[thisapp.example.coopcloud.tech]
CRITICAL: RCE in ThisApp 1.2.3+3.4.5 -> ThisApp 1.2.4+3.4.6
CVE-4321-12345, CVE-2033-12346, CVE-2033-12347
https://security.coopcloud.tech/CCSA-20260914-2.html
[thatapp.example.coopcloud.tech]
WARNING: Denial of Service ThatApp 3.2+2.2.3
CVE-4321-12111
https://security.coopcloud.tech/CCSA-20260914-6.html
```
Check for vulnerabilities across all managed hosts:
```
$ abra server ls -m
[{"host":"foo.coopcloud.tech","name":"foo"},
{"host":"bar.coopcloud.tech","name":"bar"}]
$ abra ccsa_check example.coopcloud.tech --all
Using cached Co-op Cloud Security Advisories
Checking...
foo
bar
...complete!
[thisapp.foo.coopcloud.tech]
CRITICAL: RCE in ThisApp 1.2.3+3.4.5 -> ThisApp 1.2.4+3.4.6
CVE-4321-12345, CVE-2033-12346, CVE-2033-12347
https://security.coopcloud.tech/CCSA-20260914-2.html
[thatapp.foo.coopcloud.tech]
WARNING: Dangerous configuration in OtherApp 332.1+1.2.3
Read advisory to make sure this deployment isn't vulnerable
https://security.coopcloud.tech/CCSA-20260914-6.html
[bar.coopcloud.tech]
No vulnerabilities detected!
```
CCSA can provide a mechanism to alert users about issues outside of just CVEs.
Supply chain attacks could poison docker images, requiring upgrades or
downgrades for images downloaded during certain times. Compromise of member
accounts could result in malicious code in abra or other tools. A CCSA can
provide a key outlet for incident response actions that require public
reporting.
##### Maintainer
In order to be able to provide updates to fulfill UC2 and UC3, we first need to
have actionable information. It isn't enough to tell an operator that an
application has a vulnerability if they can't do anything about it.
Unactionable information manifests apathy. We must have a clear path from
vulnerable to (more) secure.In most cases, this will require working with a
recipe maintainer to upgrade a recipe to a version that isn't vulnerable.
Unfortunately, there is no automated mechanism to identify a recipe maintainer
point of contact. There are few acceptable options for this:
1. Include a maintainer contact (email or matrix id) in the recipe.
2. Include a link to a maintainer support matrix channel.
3. Create a matrix channel in the coop cloud space for maintainers.
4. create a mailing list for maintainers.
Option 1 is optimal since it provides the highest level of ownership and
accountability. Option 2 makes a support rotation easier and more accessible,
but has no clear and specific point of contact. Option 3 is not optimal since
it would be a bit noisy. Option 4 would also make a support rotation easier (as
option 2) but would make it harder to find a specific person during a security
event.
##### Reporting Contact
Today we are able to identify new security issues. Once a new vulnerability is
identified within a recipe, someone needs to triage the vulnerability. This
includes reading the description, understanding how it does or does not impact
the specific image and associated network as deployed via the recipe, and
determining the level of additional actions.
For important vulnerabilities, maintainers need to be alerted. Someone needs to
coordinate with those maintainers to make sure there's a safe upgrade path.
Someone needs to write up a security advisory
([demo](https://0x29a.codeberg.page/CCSA-DEMO/), publish that advisory, and,
optimally, answer questions related to the advisory.
This is part of the work of a Security Analyst in the field of Vulnerability
Management. This needs to be staffed in accordance with our event response
time. This doesn't need to be a task that takes up all someone's time, but
there do need to be contacts in various time zones to be able to respond in
case something comes up.
##### VEX
Triage work can be recorded using the Vulnerability Exploitability eXchange
format (VEX). By also publishing this information as we review findings, we can
coordinate with operators who may wish to run their own scans, or who are doing
internal security work. We may also want to consume VEX from operators or other
analysis teams. Hestia does not currently support VEX ingestion, but Trivy
does. Code and infrastructure adjustments to hestia could allow authorized
users to coordinate on vulnerability management.
##### Recommendations
1. Create a "Security@" email mailing list to which coop members and authorized
security engineers can subscribe.
2. Repurpose the existing "Security" room for public (operator, maintainer, and
external) facing security support.
3. Create a "Security Team" room in the "Co-op Cloud" matrix space specifically
for people working on security problems (triaging, security engineering,
etc).
4. Allocate infrastructure (two swarm hosts, one subdomain) for security team
related work and testing.
5. Create a "scanner" bot ([PoC here](https://codeberg.org/0x29a/hestia)) to
find new vulnerabilities for triage. Run this daily, increase to hourly if
needed.
6. Create a matrix bot for the "Security Team" room to alert on new
vulnerabilities.
7. Create a "Recipe Maintainer" room specifically for identifying maintainers
to coordinate with.
8. Create a "support" object for each recipe entry in "recipes.json" a. Add a
(mandatory) "maintainer" field to this support object to identify a
maintainer contact (could be a mailing list maintainers are on). b. Add an
(optional) SLA field to the "support" field that lists the maintainer
response time. c. Decrease the "status" to "basic" for all recipes that lack
the support object. d. Decrease the "status" to "ok" for all recipes that
lack an SLA or for which the SLA has been missed in the last month.
9. Create a security.coopcloud.tech subdomain for security related information
b. link from docs.coopcloud.tech to security.coopcloud.tech a. list security
contact information here
##### Follow up
1. Consider compensating maintainers of recipes with low SLAs and high scores
2. Determine how to staff Triage team
3. Apply for funding from nlnet to staff the security team for the first year.
4. Create generic incident response plans for different incident types
5. Identify an appropriate security contact method
```
// Security contact - email, matrix chat, list on website
// Security feed - Co-op Cloud Security Advisory
// -- mailing list, RSS feed
//
// Security management team
```
docs/federation/funding-applications/private-funder.md → docs/federation/funding/private-funder.md
View File
docs/federation/funding/security_bus_design.pdf
BIN
View File
Binary file not shown.
docs/federation/funding-applications/sovereign-tech-fund.md → docs/federation/funding/sovereign-tech-fund.md
View File
docs/federation/funding/technical_challenges.pdf
BIN
View File
Binary file not shown.
docs/federation/funding-applications/user-operated-internet.md → docs/federation/funding/user-operated-internet.md
View File
mkdocs.yml
+7 -5
View File
@@ -160,11 +160,13 @@ nav:
- "Surveys":
- federation/surveys/usage-2026.md
- "Funding applications":
- federation/funding-applications/culture-of-solidarity.md
- federation/funding-applications/ford-foundation.md
- federation/funding-applications/private-funder.md
- federation/funding-applications/sovereign-tech-fund.md
- federation/funding-applications/user-operated-internet.md
- federation/funding/culture-of-solidarity.md
- federation/funding/ford-foundation.md
- federation/funding/private-funder.md
- federation/funding/sovereign-tech-fund.md
- federation/funding/user-operated-internet.md
- federation/funding/sovereign-tech-fund.md
- federation/funding/nlnet-codesupply.md
- "Abra":
- abra/index.md
- "Install": abra/install.md