Problem: When we use `ensureConnection()`, it doesn't have any handling
for when the database is being closed, and therefore can keep the
database up and running forever (which fails the tests).
Solution: Use `cooler.open()`, which has some database close handling
built in, and doesn't keep the tests open forever (causing them to time
out and fail).
Problem: Unfortunately `setImmediate()` doesn't actually ensure that
`server !== null`, so we can get fun race condition errors that have
been reported on Windows.
Solution: Replace `setImmediate()` hack with event emitter that actually
ensures that the server is listening (and hopefully isn't `null)`.
Fixes: https://github.com/fraction/oasis/issues/401
Problem: When proxying Oasis, the CSRF + DNS rebind security precautions
will respond with HTTP 400 if you use a hostname that Oasis doesn't
know about. For example, if Oasis is listening on `localhost` and you
use Caddy/Nginx/etc to proxy that to `oasis.example.com`, then Oasis
will see GET requests as DNS rebind attacks and POST requests as CSRF
attacks.
Solution: Add `--allow-host` command-line flag so that you can
`--allow-host oasis.example.com` and ensure that the host is allowed by
the security measures.
Problem: During a refactor the SSB connection management was changed to
be more conservative, so it only ensures that we have a connection once
the server is started. This isn't good, because it means `oasis
--no-open` no longer starts an SSB service in the background.
Solution: Run `ensureConnection()` to ensure that we have a connection
to the SSB service regardless of regardless of whether we've received
any requests over HTTP.
Problem: When replying to a private thread you only see the root post
because it's hiding all private comments. Instead, it should be hiding
all *encrypted* comments that we can't decrypt.
Solution: Use `isNotEncrypted()` instead of `isNotPrivate()`.
Problem: The Threads view advertises that it only shows public posts,
but is showing some private posts as well. This is not a security
concern, it's just a UI inconsistency.
Solution: Ensure that private posts are filtered from the view before
rendering. This also integrates a refactor that uses functions to check
for basic info like whether a message is a post, whether it's
encrypted/decrypted, whether it has a root, whether it has a fork, etc.
Problem: The Windows tests seem to be failing, it looks like a timeout
problem? Also there are a handful of debug statements around the code
that can be safely removed.
Solution: Double the timeout length and remove debug statements.
Problem: The test suite isn't closing the database because `app.close()`
only affects the HTTP server. This means that tests don't exit cleanly
and sockets remain open and all sorts of really fun stuff that we don't
want while writing tests.
Solution: Refactor `src/ssb.js` so that we can exit cleanly and have
less rope to hang ourselves with. Add a small lifecycle test that can
help us ensure that the bare minimum lifecycle events are working
correctly, plus now the previous tests are passing on my machine too.
Problem: ESLint and TypeScript help catch some types of regressions, but
they don't protect us against obvious stuff like "the server won't
start". This means that humans need to test a bunch of stuff manually,
and that can be really tedious and exhausting.
Solution: Yesterday someone invented this cool concept called "testing"
where you write automated tests for your software to ensure it actually
works the way you expect. It might have beeen invented before yesterday,
I don't know. Anyway, this solution adds a bunch of tests that send HTTP
GET requests to a bunch of endpoints to make sure the server is at least
returning HTTP 200 responses. It also fixes a race condition where HTTP
server was available before the readme / version strings were loaded.
Problem: I made a bad merge in 2836c80 which broke the server in some
environments. I had issues running the globablly installed binary,
whereas `npm start` seemed to work fine. Anyway, there's an error about
some missing config options because we weren't importing SSB-Config into
the server configuration. Instead, the full config was just:
```json
{ "conn": { "autostart": true } }
```
This lacks important properties like `shs`, which means that the server
can't start. Fun!
Solution: Pass SSB-Config first and then overlay our custom config on
top. I've also added comments and changed the variable names so that
this is harder to miss in the future.
Problem: Footer buttons have the normal button background hover state
that makes them difficult to read. This was meant to be fixed in another
PR but I think I got the CSS order wrong.
Solution: Reorder the CSS hierarchy to fix the bug. For real this time.
- Instead of having a custom on hover effect, just add the Liked by
message to the title of the heart.
- When there are > 16 likes on a post, show +X more to convey this to
users.
Previous query incorrectly pulled and sorted posts, often showing years
old posts from newly followed people. This now behaves more consistently
as "recent threads from people in your extended network".
- Adds a hover popup that shows the names of everyone who liked a post
when hovering of the heart.
- Add new call to post.get that retrieves the names of all voters and
returns them instead of their ID's.
Problem: A `null` relationship represents when it's your profile, which
isn't very intuitive. It causes an error when we try to check for the
`blocking` and `following` properties, which don't exist on `null`.
Solution: Instead of `null`, set a `me` property to tell whether this
relationship is our own profile.
Problem: We have a way to follow and unfollow, but there's no way to
block or unblock. Also if you go to the profile of a blocked peer, their
posts still show up.
Solution: Add block and unblock and hide messages from blocked peers on
their profile page.
Fixes: https://github.com/fraction/oasis/issues/370
Problem: Messages are being fetched with `post.get()`, which runs the
`transform()` function to decorate them with Markdown/etc, but then
they're being passed through `transform()` a second time at the end of
the function. This is inefficient and applies side-effects (like adding
channels to the post) twice.
Solution: Remove the final `transform()` so that these posts are only
decorated once.
Fixes: https://github.com/fraction/oasis/issues/358
Problem: The workaround is no longer necessary now that the new version
of @types/koa has been published. Thanks to @peterblazejewicz,
@harryparkdotio, and @orta for the quick upstream fix.
Solution: Run `npm update` and remove the `@ts-ignore` comment.
Problem: We still have heaps of implicit 'any' types that our linters
can't make sense of.
Solution: Keep on adding type documentation! Slowly but surely I'm
hoping we can reduce the amount of untyped JavaScript in the repo.
Problem: Since we don't validate the referer hostname or the HTTP host
header, an attacker who can trick an Oasis user into clicking a
malicious link can submit HTTP requests to localhost, potentially
publishing messages or extracting information.
Solution: Ensure that all requests have valid hostnames in the referer
to protect against CSRF attacks, ensure that all requests have valid
hostnames in the HTTP header to protect against DNS rebind attacks, and
ensure that errors are only output in the HTTP response when we're sure
that the request is valid.
Problem: After some fantastic new pull requests, there are some
inconsistencies between how each author implemented different features.
Solution: Fix regressions and iron out inconsistencies. This fixes the
heart highlight bug, uses the same indentation for Summaries + Threads +
the thread view, tightens up the CSP now that we don't need
unsafe-inline styles, and uses a neutral tone for indents with the same
width as the blockquote border (so far violet has been reserved for
private messages).
Problem: We're only attempting connection via Unix sockets, but some SSB
services may choose to only listen on TCP sockets.
Solution: Start by attempting connection to Unix socket and fallback to
the TCP socket remote if that doesn't work, before just trying to start
our own SSB service.
While reading long conversations, it's easy to get a bit lost as to whom
is saying what. With some basic CSS, we can keep the header of the post
sticky so you can see more context above the message.
This allow someone to pass in the theme as a command line arg or set the
theme in defaults.json, while still allowing someone to set the theme on
a per-browser basis as a cookie.
Problem: The peer list shows peers that are connected and in the process
of establishing a new connection, which has lots of turbulence. It's
confusing to see dozens of "connections" that only exist for a few
milliseconds when they're actually just connection attempts.
Solution: Only show peers with the state "connected".
Problem: When publishing a root post, other clients give you the option
of adding a `channel` property, which is basically just a hashtag. We
show this at the end of the message, but since replies often copy the
`channel` property then it gets appended to every reply.
Solution: Only append the channel as a hashtag for root posts.
Problem: We have TypeScript enabled on the project but have been missing
lots of definition files, so there are like 500 errors when you enable
`--noImplicitAny`.
Solution: Add definitions and fix the bugs that they point out. This
reduces the number of errors with `--noImplicitAny` to 285.
Problem: When publishing a root post, other clients give you the option
of adding a `channel` property, which is basically just a hashtag. We
show this at the end of the message, but since replies often copy the
`channel` property then it gets appended to every reply.
Solution: Only append the channel as a hashtag for root posts.
Problem: Our dependencies are falling behind and it'd be nice to have
them up-to-date. The only big update it that Common-Good upgraded to the
latest version of Prettier.
Solution: Update them!
Problem: By default we only connect to people that we're directly
following, which often don't have public IP addresses.
Solution: Automatically connect to all available peers within our
replication hops (default 2) to make peering more reliable.
Problem: Trying to view a private message crashes the server and leaks a
bunch of memory. That's bad. This problem is caused by faulty handling
in the function that finds thread ancestors. There's some code that says
"if the next ancestor is a private message, just return the ancestors
that we know about", which returns nothing when we're looking up a
private post (because we can't identify *any* ancestors).
Solution: Ensure that we're only resolving the promise once in the
function by chaining `else if`s and ensure that we only return the
ancestor list if there are actually ancestor in it. If the ancestor list
is empty, we can just return the single message that we know about and
pass it off.
Problem: Since the `errorView()` has a refresh we can probably just call
it `indexingView()` and add the indexing-specific code.
Solution: Rename `errorView()` to `indexingView()` and add a progress
bar for it.
Problem: The indexing message is being thrown as an error, which is
pasted as plaintext, so we can't do a page refresh. This is frustrating
when you're waiting for the indexes to finish because you have to
manually refresh a bunch while you wait.
Solution: Use the prototype from @justinabrahms to add an HTML message
for the indexing error and automatically refresh the page every ten
seconds.
Problem: When our views are still indexing the database they apparently
don't respond over MuxRPC, which means that we're just waiting forever
until they finish. This means that people who are indexing the database
get an HTTP response that might take an hour to finish.
Solution: If we have more than 1 mebibyte of backlog, present an error
message explaining the situation and asking for a bit of patience.
Problem: The SSB-Markdown library has some SSB-flavored quirks, like
messing with newline behavior, and is inappropriate for rendering plain
Markdown like the readme.
Solution: Since SSB-Markdown uses Markdown-It under the hood, we can use
the same library and just render our Markdown *without* the SSB-flavored
quirks. This gives us the ability to wrap text and avoid SSB-Markdown
problems without having to rewrite all of our Markdown.
Problem: A big chunk of both the readme and the startup output is
dedicated to configuration, which feels to me to be focused on
developers and advanced users rather than beginners who just want to try
using Oasis.
Solution: Move readme configuration info to its own file and hide all
config output (except one line) behind `--debug`. While doing this I
noticed that we're `require()`ing a few modules that we don't need
before setting `process.env.DEBUG`, which I've reorganized so that we
don't accidentally disable debug mode for those modules.
Problem: I have debug mode manually enabled and have `console.log()`
calls sprinkled through the code and I'm doing `require("crypto")` too
many times.
Solution: Remove debugging cruft so that we don't have a perma-debug
mode.
Problem: We can set our name and profile description but profile images
are expected by most people and supported by most clients and we don't
have them. A profile without an image can sometimes lack the intimacy
you'd get if you let people upload profile images that they can use as a
visual avatar.
Solution: Add profile image upload to the Edit Profile page and add a
bunch of plumbing for `ssb.blobs.add()` to add the blob and publish a
message setting it as a profile image.
Problem: Other clients render a `channel` property, which is basically a
way to add a single hashtag to your post, but Oasis doesn't support
that. This means that someone might post "this is fun!" under the
channel #running and people using Oasis would just see "this is fun!"
without any context.
Solution: Add the hashtag to the bottom of the post, which visually
looks the same as someone adding two newlines and a hashtag to the end
of the text in their post.
Problem: The Extended page is sorting by received timestamp, not
asserted timestamp.
Solution: Find the `Date.now` and replace it with `Date.now()`, which
refers to the current date instead of `undefined`, which is what you get
if you try to `JSON.stringify(Date.now)`.
Problem: Searching for a hashtag should bring you to the hashtag page,
and the hashtag page should have some useful information about which
page you're on instead of just showing you the messages.
Solution: Add code so that if you search for a hashtag you're brought to
that page, and display some useful help text at the top of the page
explaining the hashtag page. While ensuring that the hashtag page showed
posts from around the network (instead of only people you're following)
I fixed the popular page so that it has the same behavior again (fixing
a regression) and filtered out gatherings (fixing *another* regression)
on the popular page. These probably should've happened in another commit
but I got carried away. :/
Happy to split this commit into two if it hurts to bundle the popular
fixes with the hashtag fix, but I'm low on energy and want to at least
open a PR in case it doesn't bother anyone.
Problem: The search results weren't hiding results from blocked feeds.
Solution: Add `socialFilter()` to ensure that all blocked feeds are
removed from search results as the first step in the stream processing.
Problem: Sometimes you try to run Oasis twice and the second time you
run the program it blows up in your face about the port being taken.
Most other software just focuses or reopens the window instead of
throwing errors at you.
Solution: Implement `/.well-known/oasis` as a sanity check to see
whether the thing listening on the HTTP port. If it replies "oasis" then
we can open the tab (unless `config.open === false`) and avoid throwing
the error.
Problem: An SSB client should allow you to declare your own name, but
Oasis didn't support that behavior at all.
Solution: Add a basic 'Edit Profile' page that lets you set your name
and description. This doesn't implement profile images because I had
limited time and didn't want to think about encoding formats, but it's
worth mentioning that I remember something about binary uploads with the
default form encoding actually send 3 times as much data because of
escapes or something? This might not effect us because we're on
localhost, but I want to make a note that this isn't implemented yet.
This also makes a small change regarding the `<label>` element --
previously we were writing them as siblings to the input and using the
`for` attribute to target the input, but while messing with the CSS I
tried putting them directly in the `label()` and it ends up having the
same effect with less code.
Problem: Trying to use the DTA index when using SSB-Query doesn't work,
because that index only exists on SSB-Backlinks.
Solution: Change the filter to ensure that we're sorted by the asserted
timestamp and ignoring messages from the future. The SSB-Backlinks
plugin was taking the minimum between `.value.timestamp` and `timestamp`
but since we're not using SSB-Backlinks in this query (or maybe at all?)
we get to make our own little hacky query. Cel taught me that the
`$sort` option causes problems here because it buffers the entire result
in memory before doing the sort. Boo. Instead we just need to reference
the property that we want to be sorted by with some operation, even if
it does nothing (like `{ $gt: null }`), but since we're filtering time
travelers we actually need `{ $lge: Date.now() }` there anyway.
Problem: We load the default config with `require()`, which would allow
arbitrary JavaScript execution, are producing debug output everywhere
with `console.log()` statements, and don't have instructions for how to
write the config file. The config file is also `.config/**/config.json`,
and I think it would be helpful to specify that this is the **default**
config rather than any other kind of config.
Solution: Rename the config to `~/.config/oasis/default.json`, replace
the `require()` with `fs.readFile()`, remove `console.log()` for
debugging, and add some example JSON for how to use the config file.
This should give pretty much expected behaviour. Each config value
can be set by three sources:
1. By command-line argument. If it is not given, then
2. By config file. Or, lastly
3. By default value in the source code.
I can't test that the config file is searched and read from the right
place on windows or macOS, but on linux it works.
Problem: We're doing tons of unnecessary cryptography by encrypting the
connection between the "client" and "server", which are often running in
the same process.
Solution: Instead of connecting to the SSB service over TCP and
encrypting the stream, just connect over a socket (supported on Windows,
macOS, and Linux) and don't bother encrypting anything. This is what
Patchwork and Patchbay do already, and since our secret is at
`~/.ssb/secret` then we should be comfortable with `~/.ssb/socket` being
a trusted file where access implies authentication.
Local tests suggest that when sodium-native is available, this commit
reduces the time to render the 'Popular (Day)' page by 17%, but when we
have to fall back to JavaScript cryptography the same page now takes 30%
less time to render. My intuition is that this improvement is more
dramatic on mobile, but requires further testing before we can pat
ourselves on the back too much. :)
Problem: We've been using a workaround to filter the results of
SSB-Mentions, but that's just been fixed upstream and so we don't need
the workaround anymore.
Solution: Update the SSB-Mentions package and remove the workaround.
Problem: During the SSB-Query refactor I used `{ private: false }` in an
attempt to hide private messages, but apparently this option is only
available via SSB-Backlinks, so it was ignored without any warning (!).
This allowed private messages to be displayed in public views.
Solution: Remove the unused option and add a filter to public views that
ensures that private messages are always hidden.
