update readme

README.md

@@ -1,22 +1,29 @@
 
-# directorate for applied fediverse research
+# doing frequent applied fediverse research
 
-* independently tries to verify fediverse statistics
+* hands-on approach to verify statistics on fediverse from fediverse.network and the-federation.info
 * draws conclusions from that
 
 ## methodology 
 
-Currently the script starts from <https://post.lurk.org> and queries `/api/v1/instance/peers` to find servers it is peering with. For each of the peering servers it hasn't seen before it does the same and in addition it tries to query `/api/v1/instance` for meta data.
+### Mapping the network
+Currently the script starts from <https://post.lurk.org> and queries `/api/v1/instance/peers` to find servers it is peering with. For each of the peering servers it hasn't seen before it does the same. This from the assumption that getting peer lists from Mastodon & Pleroma gives enough of a view of 'known fediverse' to work with.
 
-This method is a bit lacking because providing `/api/v1/instance` is voluntary and specific to later versions of mastodon/activitypub fediverse. We should study the methodology of [fediverse.network](https://fediverse.network/info) for better results.
+  > initial peer list
+  >> all peers of the initial peers
+  >>> all peers of the peers of the inintial peers
+  ------------------------------------------------- +
+  the known fediverse?
 
-When the request fails on a given instance it just logs it as 'error' now.
+### Instance metadata
+We try to query `/.well-known/nodeinfo` for instance meta-data such as software type etc. This is what both fediverse.network and the-federation.info do
+
+When any request fails on a given instance it logs the raised `Exception`, if it is a HTTP error instead we currently log the answer.
 
 Latest scrape results can be found in `instance_scrape.json`
 
 ## TODO FIXME
-* add detailed error message to json when we get one
+* ~~add detailed error message to json when we get one~~ 
 * ~~abstract the functions so we can multithread them~~
-* find a way to also scrape for instances that don't announce themselves