docs: improve documentation

README.md

@@ -1,18 +1,16 @@
-source files are in this format:
+# Mapping cameras in the city
 
-i.e. contents of source-data/by-district/01-lekkerwijk:
-```
-astraat, bstraat, cweg, dstraat, theo de vstraat.
-```
+Rotterdam has a lot of streets with live view cameras. The list of these streets is published in Gemeenteblad for the city and this is an attempt to put this data on a map. The specific locations of cameras aren't published but this is a good start.
 
-they have useful comments and shit. for instance the rotterdam central station needs to be marked manually on the map, and there's some streets which i had to rename because they were misspelled in the gemeenteblad. this is all documented in the source files as comments starting with #
+![grayscale map with a bunch of red lines][docs/partial-map.png]
 
-this is how they are formatted in the gemeenteblad.
+this is a WIP, the current structure of repo is as follows:
+- `docs/` - self-explanatory
+- `source-data`
+  - `all-streets` - what I copied from the OG 2022 gemeenteblad with the camera plan. Doesn't include further changes. Not very useful
+  - `by-district/` - the document that describes the camera surveillance divides the city into 11 districts. So there's 11 files in that directory, each containing an annotated list of streets in that district, with further updates from the gemeente. Also includes spelling fixes (surprising amount of misspelled streets in the source documents, but sometimes also the streets are just called differently in OSM. For example Overpass doesn't like "1° Pijnackerstraat" but "1e Pijnackerstraat" works fine. Or the municipality put A. Wellingpad in the document they published but the street is actually called Alberta Wellingpad on OSM.)
+  - `query-header` and `query-footer` - don't worry about it too much, this is used by `scripts/03-build-query-for-district` which includes the header and footer in every file it generates. You'll need to play with this if you choose to run these scripts for a different city, because that's where Rotterdam is coded in.
+- `scripts/` - the heart of this repo. A set of bash scripts using some basic unix utils and the Overpass API to parse the source data and return jsons that can be converted into geojson. They should be ran in order, starting from `01-sed-to-single-line`, till `05-show-missing-streets`. Each of them will take data from the previous one and output to `working-data/its-own-number` so you can inspect the data at every step of the process.
+- `geojson-files/` - QGIS, which I use to assemble the maps accepts data in geojson format. This is where those files should land. Think of it as the output directory of this program. This is not populate by any script. You have to run `for i in working-data/04/*; do osmtogeojson "$i" > "geojson-files/$(basename $i)"; done` from the root dir of this repo to get those files, and `osmtogeojson` is a node.js program that you obtain by running `npm install osmtogeojson`
 
-you run `./scripts/01-sed-to-single-line` this removes all the unnecesary metadata from source files and spits out a single-line file into working-data/01/ for each source file
 
-then `./scripts/02-way-queries` builds queries for every single street, grouping them in files of 5 queries to not overload the API. then it spits them out into `working-data/02/` for `./scripts/03-build-query-for-district` which adds all the header and footer for the actual request to work properly.
-
-then `./scripts/04-call-overpass-api` POST's all these files into the overpass API and stores the resulting JSONs in `working-data/04` these then need to be converted into GeoJSON for QGIS to be able to understand them. I use https://github.com/tyrasd/osmtogeojson which you need to install via npm.
-
-`for i in working-data/04/*; do osmtogeojson "$i" > "geojson-files/$(basename $i)"; done`

docs/README.md

@@ -0,0 +1,18 @@
+source files are in this format:
+
+i.e. contents of source-data/by-district/01-lekkerwijk:
+```
+astraat, bstraat, cweg, dstraat, theo de vstraat.
+```
+
+they have useful comments and shit. for instance the rotterdam central station needs to be marked manually on the map, and there's some streets which i had to rename because they were misspelled in the gemeenteblad. this is all documented in the source files as comments starting with #
+
+this is how they are formatted in the gemeenteblad.
+
+you run `./scripts/01-sed-to-single-line` this removes all the unnecesary metadata from source files and spits out a single-line file into working-data/01/ for each source file
+
+then `./scripts/02-way-queries` builds queries for every single street, grouping them in files of 5 queries to not overload the API. then it spits them out into `working-data/02/` for `./scripts/03-build-query-for-district` which adds all the header and footer for the actual request to work properly.
+
+then `./scripts/04-call-overpass-api` POST's all these files into the overpass API and stores the resulting JSONs in `working-data/04` these then need to be converted into GeoJSON for QGIS to be able to understand them. I use https://github.com/tyrasd/osmtogeojson which you need to install via npm.
+
+`for i in working-data/04/*; do osmtogeojson "$i" > "geojson-files/$(basename $i)"; done`