Use async module to speed up pad processing #6

Closed
opened 5 years ago by decentral1se · 3 comments
Owner

We could also batch the pad IDs into groups and then use the core async library to run them asynchronously. This could drastically speed things up if you have a bunch of cores. Might need some investigation to realise first though.

Combined with #5, we could really have a faster feedback loop.

We could also batch the pad IDs into groups and then use the core async library to run them asynchronously. This could drastically speed things up if you have a bunch of cores. Might need some investigation to realise first though. Combined with https://git.vvvvvvaria.org/varia/etherpump/issues/5, we could really have a faster feedback loop.
Poster
Owner
loop = asyncio.get_event_loop()

tasks = []
url = "http://localhost:8080/{}"

for i in range(5):
    task = asyncio.ensure_future(hello(url.format(i)))
    tasks.append(task)

loop.run_until_complete(asyncio.wait(tasks))
* https://aiohttp.readthedocs.io/en/stable/ * https://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html ```python loop = asyncio.get_event_loop() tasks = [] url = "http://localhost:8080/{}" for i in range(5): task = asyncio.ensure_future(hello(url.format(i))) tasks.append(task) loop.run_until_complete(asyncio.wait(tasks)) ```
rra commented 5 years ago

You could also check what I did here with the multiprocessing library to use different threads for doing HTTP requests.

Comes down to the same I guess..

You could also check what I did [here](https://git.vvvvvvaria.org/rra/dafr/src/branch/master/fedicrawler.py) with the multiprocessing library to use different threads for doing HTTP requests. Comes down to the same I guess..
Poster
Owner

OK, that's in. I used the trio async library and asks HTTP client.

OK, that's in. I used the trio async library and asks HTTP client.
decentral1se closed this issue 4 years ago
Sign in to join this conversation.
No Label
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.