python

[Python Patterns] Fun with Asyncio in Python 3.8

Here are some examples of using the Asyncio module in Python 3.8

Here are some examples of using the Asyncio module in Python 3.8
[Note: I am now on Python 3.9.5]

The testing is done hitting the SWAPI as it is public and I can just run through a range() of numbers. Go to https://swapi.dev/documentation for more info.

First the OG way with the Requests module

This is my bread-and-butter-go-out-and-get-things-done way of working with API's on the Internet.  As you can see I go and grab a character in a serial fashion one by one.

It averages about 6 seconds calling the API individually and then printing out the information.

CODE

    import requests
    import time


    def main():
        start_time = time.time()

        print('OG requests')

        # We want to get people 1 through 15
        PEOPLE = 16

        url = 'https://swapi.co/api/people/{}'

        for i in range(1, PEOPLE):
            response = requests.get(url=url.format(i))
            data = response.json()
            print("Character: {0}, Birth Year: {1}".format(
                data['name'], data['birth_year']))

        print('The script took {0} second !'.format(time.time() - start_time))


    if __name__ == '__main__':
        main()

Async: As Completed

This code here creates a list of tasks to run, then pushes them to the event loop, then it gives the results as they complete.

It averages about 0.8 seconds calling the API, Nifty!

CODE

import aiohttp
import asyncio
import time


async def fetch(session, url):
    # Get the goods
    async with session.get(url) as response:
        return await response.json()


async def main():
    start_time = time.time()

    # ##################################################
    # This prints results as they complete

    print('Async as_completed')

    url = 'https://swapi.dev/api/people/{}'

    tasks = []
    async with aiohttp.ClientSession() as session:
        for i in range(1, 10):
            tasks.append(fetch(session, url.format(i)))

        for future in asyncio.as_completed(tasks):
            result = await future
            print(f'{result["name"]}')
            print(f'{result["birth_year"]}')
            print(f'{result["gender"]}')

    script_time = time.time() - start_time
    print(f'The script took {script_time} second !')


if __name__ == '__main__':
    asyncio.run(main())

Async: Gather

This code does the same as_completed, but the difference is that it waits for all the tasks to complete, then iterates through the data to provide the data we'd like

It averages about 0.9 seconds calling the API, Good!

CODE

import aiohttp
import asyncio
import time


async def fetch(session, url):
    # Get the goods
    async with session.get(url) as response:
        return await response.json()


async def main():
    start_time = time.time()

    # ##################################################
    # This prints results after they are done

    print('Async gather')

    url = 'https://swapi.dev/api/people/{}'

    tasks = []
    async with aiohttp.ClientSession() as session:
        for i in range(1, 10):
            tasks.append(fetch(session, url.format(i)))

        results = await asyncio.gather(*tasks)
        # print(results)
        for result in results:
            print(f'{result["name"]}')
            print(f'{result["birth_year"]}')
            print(f'{result["gender"]}')

    script_time = time.time() - start_time
    print(f'The script took {script_time} second !')


if __name__ == '__main__':
    asyncio.run(main())

Async: Pools

This code creates a Semaphore pool to get the requests, and finally uses the gather pattern to return the data.

It averages about 1 second calling the API, Cool!

CODE

# semaphore gather

import aiohttp
import asyncio
import time


async def fetch(sema, url, session):
    async with sema, session.get(url) as response:
        return await response.json()


async def main():
    start_time = time.time()

    # ##################################################
    # This prints results after they are done
    # Using a Semaphore to create a pool of threads

    print('Semaphore gather')

    url = 'https://swapi.dev/api/people/{}'

    tasks = []
    # create instance of Semaphore
    sema = asyncio.Semaphore(5)

    async with aiohttp.ClientSession() as session:
        for i in range(1, 10):
            # pass Semaphore and session to every GET request
            task = fetch(sema, url.format(i), session)
            tasks.append(task)

        results = await asyncio.gather(*tasks)
        # print(results)
        for result in results:
            print(f'{result["name"]}')
            print(f'{result["birth_year"]}')
            print(f'{result["gender"]}')

    script_time = time.time() - start_time
    print(f'The script took {script_time} second !')


if __name__ == '__main__':
    asyncio.run(main())

Conclusion

As you can see the OG way of using requests is slower by a wide margin. It looks like Async: Gather might be the best way to go if you want to write some Async code.

Personally though, it all depends on what you are trying to get out of your code.  I can see using any of the ways described above from a quick and dirty script, to a daily time sensitive program.

Code Repo

GitHub - BitsofJeremy/AsyncTesting
Contribute to BitsofJeremy/AsyncTesting development by creating an account on GitHub.
GitHubBitsofJeremy

REFERENCES

https://www.integralist.co.uk/posts/python-asyncio/
https://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html
https://stackoverflow.com/questions/56523043/using-python-3-7-to-make-100k-api-calls-making-100-in-parallel-using-asyncio
https://stackoverflow.com/questions/40836800/python-asyncio-semaphore-in-async-await-function

My blog posts tagged with "Python Patterns" are designed to be a quick look reference for some Python code snippets I use a lot.  They are written to be a quick starting point for future projects so I do not need to type as much.

Read next