There should be one – and preferably only one – obvious way to do it.
There are multiple ways to manage resources with Python, but only one of them
is save, reliable and Pythonic.
Before we dive in, let's examine what resources can mean in this context. The
most obvious examples are open files, but the concept is broader: it includes
locked mutexes, started client processes, or a temporary directory change
using os.chdir(). The common theme is that all of these require some sort
of cleanup that must reliably be executed in the future.
The file must be closed, the mutex unlocked, the process terminated, and the
current directory must be changed back.
So the core question is: how to ensure that this cleanup really happens?
Failed solutions
Manually calling the cleanup function at the end of a code block is the most obvious solution:
f = open('file.txt', 'w')
do_something(f)
f.close()
The problem with this is that f.close() will never be executed if
do_something(f) throws an exception. So we'll need a better solution.
C++ programmers see this and try to apply the C++ solution: RAII,
where resources are acquired in an object's constructor and released in the
destructor:
class MyFile(object):
def __init__(self, fname):
self.f = open(fname, 'w')
def __del__(self):
self.f.close()
my_f = MyFile('file.txt')
do_something(my_f.f)
# my_f.__del__() automatically called once my_f goes out of scope
Apart from being verbose and a bit un-Pythonic, it's also not necessarily
correct. __del__() is only called once the object's refcount reaches zero,
which can be prevented by reference cycles or leaked references.
Additionally, until Python 3.4 some __del__() methods were not called during
interpreter shutdown.
A workable solution
The way to ensure that cleanup code is called in the face of exceptions is the
try ... finally construct:
f = open('file.txt', 'w')
try:
do_something(f)
finally:
f.close()
In contrast to the previous two solutions, this ensures that the file is closed
no matter what (short of an interpreter crash). It's a bit unwieldy, especially
when you think about try ... finally statements sprinkled all over a large
code base. Fortunately, Python provides a better way.
The correct solution™
The Pythonic solution is to use the with statement:
with open('file.txt', 'w') as f:
do_something(f)
It is concise and correct even if do_something(f) raises an exception. Nearly
all built-in classes that manage resources can be used in this way.
Under the covers, this functionality is implemented using objects known as
context managers, which provide __enter__() and __exit__()
methods that are called at the beginning and end of the with block. While
it's possible to write such classes manually, an easier way is to use the
contextlib.contextmanager decorator.
from contextlib import contextmanager
@contextmanager
def managed_resource(name):
r = acquire_resource(name)
try:
yield r
finally:
release_resource(r)
with managed_resource('file.txt') as r:
do_something(r)
The contextmanager decorator turns a generator function (a function with a
yield statement) into a context manager. This way it is possible to make
arbitrary code compatible with the with statement in just a few lines of
Python.
Note that try ... finally is used as a building block here. In contrast
to the previous solution, it is hidden away in a utility resource manager
function, and doesn't clutter the main program flow, which is nice.
If the client code doesn't need to obtain an explicit reference to the
resource, things are even simpler:
@contextmanager
def managed_resource(name):
r = acquire_resource(name)
try:
yield
finally:
release_resource(r)
with managed_resource('file.txt'):
do_something()
Sometimes the argument comes up that this makes it harder to use those
resources in interactive Python sessions – you can't wrap your whole session in
a gigantic with block, after all. The solution is simple: just call
__enter__() on the context manager manually to obtain the resource:
cm_r = managed_resource('file.txt')
r = cm_r.__enter__()
# Work with r...
cm_r.__exit__(None, None, None)
The __exit__() method takes three arguments, passing None here is fine
(these are used to pass exception information, where applicable). Another option
in interactive sessions is to not call __exit__() at all, if you can live
with the consequences.
Wrap Up
Concise, correct, Pythonic. There is no reason to ever manage resources in any
other way in Python. If you aren't using it yet - start now!