Writing Asynchronous Python
I broke into JavaScript at the beginning of last year for a million reasons: money, front end development experience, money, back end development... money... but one of the things that immediately gave me issues was the fact that everything seemed to be done asynchronously! From React hooks to axios requests and just about everything in between, it seemed like I needed to become way more familiar with asynchronous programming.
For those that aren't familiar, asynchronous programming is basically running a piece of code and just letting that part of your program run while you do other things. Think of it like putting a pot on the stove to boil and moving on to cut the veggies and prepare the rest of the dish. You could wait for the pot to boil to move on to the rest of the dish, right? Sure, but you'd be cooking your food a lot slower. Instead, you let the pot heat up while you work on everything else. Letting the pot boil while you prepare the rest of the food is cooking asynchronously and just like in cooking, asynchronous programming can really make your solutions a lot faster.
Python has recently added native support for asynchronous programming, and I've taken a bit of time to implement it in a lot of my projects. I'll talk a little bit about how I've implemented asynchronous Python code and then we'll dive into a couple of more simple examples.
What can work asynchronously
If you've got a type of operation that just needs to run but the rest of your code doesn't really depend on the output, chances are that's a good candidate for asynchronous programming. One of the best examples is logging: if something happens and I need to write to a log file, I usually won't care to wait for that file input/output to run. I can call a logging function that outputs a message to a file and immediately move on with the rest of my program while that file I/O runs in the background. That may seem like a fairly trivial application, because file I/O is pretty fast now, but if you're writing to the log file a lot, this can have a pretty big effect at scale. The effect can be even more dramatic when you're writing to a database: if a certain database write is very stable and the rest of my code doesn't immediately need to read that data, it's fairly safe to implement that write asynchronously.
Now, you'll notice there is a fairly important "if" there...
What does not work asynchronously
If your code fairly immediately depends on the output of a function, it's probably a dangerous function to implement asynchronously. If I'm writing to a database and then I immediately need to read the value I wrote to that database, I may enter a race condition if the write happens asynchronously: I may be trying to read a value from the database before the value is actually written, because I've moved on to more code without waiting for the write to occur. Following our cooking analogy, that would be like setting some meat on a cool pan to cook and then immediately needing to put the cooked meat into the next step of the recipe. The meat probably isn't ready to move on to the next step because the pan and the meat are still cool.
Another example would be web requests. If I am making an asynchronous request to an external web server and then immediately need to use the value that is returned from that web server to do some other thing, I'm going to run into problems with race conditions: if the web server is running slow or there is some other cause of latency, now I'm depending on a value that likely won't exist yet when the dependent function runs.
This can cause anything from minor, annoying logical bugs to serious security issues depending on the implementation. It really is best to use caution when implementing asynchronous code in your applications, even though they can be very powerful. Let's dive into some examples to show how to properly implement asynchronous functions in Python.
Examples
Below is a very simple asynchronous code example.
We have a very simple function, printMessage(message), that just prints a message out and uses the asyncio library's build in sleep function. We then have a main() function that calls the gather function in the asyncio library, which allows us to call several functions at once and wait for them all to finish executing and return. In our example, we run it 5 times with 5 different messages. Now, if we ran this synchronously, it would naturally take at least 5 seconds, since each function call would print its message and wait for one second to move on to the next call. But if we look at our output here, it runs in just over one second.
Luckily, Python guardrails you fairly well from making major mistakes in your code. Generally speaking, you always have to await any asynchronous requests. However, you do have the capability to run a create_task() function call which does not require the main thread to wait, meaning if you used this on a database write function or a function that writes to a global variable, you may have some issues.
This is a bit of a weird example, but it kinda shows what can happen, even if I had to force the error here.
In this code, we read a value out of a text file that we've written to and save it in GLOBALVAR. We then print out the original value. Then, we asynchronously call the setGlobal(g) function, which writes out a new number to the file (in this case, 6) but because it's called using the create_task() function, our thread immediately moves on to the next line, where we sleep for a second, open the text file again and expect the value written to be the same... but it isn't, because the function we called before has already written to that file. If this were unintentional, it could come with some serious consequences. Luckily, the only bugs I get are intentional ones. 😉
If this example seems a bit contrived, it is. It's fairly difficult for my sleep-deprived brain to think of a good way to simulate race conditions realistically in Python here, but hopefully I've explained the problem enough for you to semi-understand how it can cause issues. Hopefully I've also explained how it can be super helpful as well!
Like the info? Learn something cool? Sign up to my weekly research newsletter, Valhalla Research Weekly, to keep up to date with all of my research!