optimize io module #9
Comments
|
Using the benchmark described here: http://www.artificialworlds.net/blog/2017/06/12/making-100-million-requests-with-python-aiohttp/ to run this program #!/usr/bin/env python3.6
from aiohttp import ClientSession
from pypeln import io
import asyncio
import sys
async def fetch(url, session):
async with session.get(url) as response:
return await response.read()
async def main():
r = int(sys.argv[1])
url = "http://localhost:8080/{}"
async with ClientSession() as session:
data = range(r)
await io.each(lambda i: fetch(url, session), data, workers=1000, run = False)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())It runs much slower than the async client described in that article: |
|
This is still the case this morning even though I ran at about 2018-09-21 08:00 UTC. Comparison with the numbers from that article: |
Here is the raw profile: prf.txt |
|
@andybalaam thanks for looking into this. I am not able to reproduce your published results. I am getting the following numbers in which all 3 clients have similar performance and are consistent with the client-async-sem Uses more more memory as expected. client-async-as-completed Uses a lot of CPU, this might be because of the use of client-pypeln-io By the numbers its pretty good balanced. As I said before on a comment, your original code had to be slightly modified because now https://github.com/cgarciae/pypeln/tree/develop/benchmarks/100_million_downloads I am using |
|
@andybalaam I don't know if its a coincidence but I am getting times in the 17-20 seconds range on all 3 clients with Maybe you added an extra |
|
@andybalaam figured it out! It was the limit in the |
|
Awesome! By the way I am usually referred to as "Andy". |
|
Woohoo! I updated my blog post to refer to yours. Fantastic work! |
|
Excellent @andybalaam ! |
|
Thank you for improving on what I tried to do :-) |
http://www.artificialworlds.net/blog/2017/06/12/making-100-million-requests-with-python-aiohttp/comment-page-1/#comment-186106
The text was updated successfully, but these errors were encountered: