more info on methodology and where it is lacking

README.md

@@ -4,7 +4,17 @@
 * independently tries to verify fediverse statistics
 * draws conclusions from that
 
-## infos / resources
+## methodology 
 
-methodology of [fediverse.network](https://fediverse.network/info)
+Currently the script starts from <https://post.lurk.org> and queries `/api/v1/instance/peers` to find servers it is peering with. For each of the peering servers it hasn't seen before it does the same and in addition it tries to query `/api/v1/instance` for meta data.
+
+This method is a bit lacking because providing `/api/v1/instance` is voluntary and specific to later versions of mastodon/activitypub fediverse. We should study the methodology of [fediverse.network](https://fediverse.network/info) for better results.
+
+When the request fails on a given instance it just logs it as 'error' now.
+
+Latest scrape results can be found in `instance_scrape.json`
+
+## TODO FIXME
+* add detailed error message to json when we get one
+* abstract the functions so we can multithread them