Testing functions isn't hard, but how do you test user input and output?

In my last article, I started looking at "pytest", a framework for testing Python programs that's really changed the way I look at testing. For the first time, I really feel like testing is something I can and should do on a regular basis; pytest makes things so easy and straightforward.

One of the main topics I didn't cover in my last article is user input and output. How can you test programs that expect to get input from files or from the user? And, how can you test programs that are supposed to display something on the screen?

So in this article, I describe how to test input and output in a variety of ways, allowing you to test programs that interact with the outside world. I try not only to explain what you can do, but also show how it fits into the larger context of testing in general and pytest in particular.

User Input

Say you have a function that asks the user to enter an integer and then returns the value of that integer, doubled. You can imagine that the function would look like this:

def double():
    x = input("Enter an integer: ")
    return int(x) * 2

How can you test that function with pytest? If the function were to take an argument, the answer would be easy. But in this case, the function is asking for interactive input from the user. That's a bit harder to deal with. After all, how can you, in your tests, pretend to ask the user for input?

In most programming languages, user input comes from a source known as standard input (or stdin). In Python, sys.stdin is a read-only file object from which you can grab the user's input.

So, if you want to test the "double" function from above, you can (should) replace sys.stdin with another file. There are two problems with this, however. First, you don't really want to start opening files on disk. And second, do you really want to replace the value of sys.stdin in your tests? That'll affect more than just one test.

The solution comes in two parts. First, you can use the pytest "monkey patching" facility to assign a value to a system object temporarily for the duration of the test. This facility requires that you define your test function with a parameter named monkeypatch. The pytest system notices that you've defined it with that parameter, and then not only sets the monkeypatch local variable, but also sets it up to let you temporarily set attribute names.

In theory, then, you could define your test like this:

def test_double(monkeypatch):
    monkeypatch.setattr('sys.stdin', open('/etc/passwd'))
    print(double())

In other words, this tells pytest that you want to open /etc/passwd and feed its contents to pytest. This has numerous problems, starting with the fact that /etc/passwd contains multiple lines, and that each of its lines is non-numeric. The function thus chokes and exits with an error before it even gets to the (useless) call to print.

But there's another problem here, one that I mentioned above. You don't really want to be opening files during testing, if you can avoid it. Thus, one of the great tools in my testing toolbox is Python's StringIO class. The beauty of StringIO is its simplicity. It implements the API of a "file" object, but exists only in memory and is effectively a string. If you can create a StringIO instance, you can pass it to the call to monkeypatch.setattr, and thus make your tests precisely the way you want.

Here's how to do that:

from io import StringIO
from double import double

number_inputs = StringIO('1234\n')

def test_double(monkeypatch):
    monkeypatch.setattr('sys.stdin', number_inputs)
    assert double() == 2468

You first create a StringIO object containing the input you want to simulate from the user. Note that it must contain a newline (\n) to ensure that you'll see the end of the user's input and not hang.

You assign that to a global variable, which means you'll be able to access it from within your test function. You then add the assertion to your test function, saying that the result should be 2468. And sure enough, it works.

I've used this technique to simulate much longer files, and I've been quite satisfied by the speed and flexibility. Just remember that each line in the input "file" should end with a newline character. I've found that creating a StringIO with a triple-quoted string, which lets me include newlines and write in a more natural file-like way, works well.

You can use monkeypatch to simulate calls to a variety of other objects as well. I haven't had much occasion to do that, but you can imagine all sorts of network-related objects that you don't actually want to use when testing. By monkey-patching those objects during testing, you can pretend to connect to a remote server, when in fact you're just getting pre-programmed text back.

Exceptions

What happens if you call the test_double function with a string? You probably should test that too:

str_inputs = StringIO('abcd\n')
def test_double_str(monkeypatch):
    monkeypatch.setattr('sys.stdin', str_inputs)
    assert double() == 'abcdabcd'

It looks great, right? Actually, not so much:

E   ValueError: invalid literal for int() with base 10: 'abcd'

The test failed, because the function exited with an exception. And that's okay; after all, the function should raise an exception if the user gives input that isn't numeric. But, wouldn't it be nice to specify and test it?

The thing is, how can you test for an exception? You can't exactly use a usual assert statement, much as you might like to. After all, you somehow need to trap the exception, and you can't simply say:

assert double() == ValueError

That's because exceptions aren't values that are returned. They are raised through a different mechanism.

Fortunately, pytest offers a good solution to this, albeit with slightly different syntax than you've seen before. It uses a with statement, which many Python developers recognize from its common use in ensuring that files are flushed and closed when you write to them. The with statement opens a block, and if an exception occurs during that block, then the "context manager"—that is, the object that the with runs on—has an opportunity to handle the exception. pytest takes advantage of this with the pytest.raises context manager, which you can use in the following way:

def test_double_str(monkeypatch):
    with pytest.raises(ValueError):
        monkeypatch.setattr('sys.stdin', str_inputs)
        result = double()

Notice that you don't need an assert statement here, because the pytest.raises is, effectively, the assert statement. And, you do have to indicate the type of error (ValueError) that you're trying to trap, meaning what you expect to receive.

If you want to capture (or assert) something having to do with the exception that was raised, you can use the as part of the with statement. For example:

def test_double_str(monkeypatch):
    with pytest.raises(ValueError) as e:
        monkeypatch.setattr('sys.stdin', str_inputs)
        results = double()
    assert str(e.value) == "invalid literal for int()
     ↪with base 10: 'abcd'"

Now you can be sure that not only was a ValueError exception raised, but also what message was raised.

Output

I generally advise people not to use print in their functions. After all, I'd like to get some value back from a function; I don't really want to display something on the screen. But at some point, you really do actually need to display things to the user. How can you test for that?

The pytest solution is via the capsys plugin. Similar to monkeypatch, you declare capsys as a parameter to your test function. You then run your function, allowing it to produce its output. Then you invoke the readouterr function on capsys, which returns a tuple of two strings containing the output to stdout and its error-message counterpart, stderr. You then can run assertions on those strings.

For example, let's assume this function:

def hello(name):
    print(f"Hello, {name}!")

You can test it in the following way:

def test_hello(capsys):
    hello('world')
    captured_stdout, captured_stderr = capsys.readouterr()
    assert captured_stdout == 'Hello, world!'

But wait! This test actually fails:

E   AssertionError: assert 'Hello, world!\n' == 'Hello, world!'
E     - Hello, world!
E     ?              -
E     + Hello, world!

Do you see the problem? The output, as written by print, includes a trailing newline (\n) character. But the test didn't check for that. Thus, you can check for the trailing newline, or you can use str.strip on stdout:

def test_hello(capsys):
    hello('world')
    captured_stdout, captured_stderr = capsys.readouterr()
    assert captured_stdout.strip() == 'Hello, world!'

Summary

pytest continues to impress me as a testing framework, in no small part because it combines a lot of small, simple ideas in ways that feel natural to me as a developer. It has gone a long way toward increasing my use of tests, both in general development and in my teaching. My "Weekly Python Exercise" subscription service now includes tests, and I feel like it has improved a great deal as a result.

In my next article, I plan to take a third and final look at pytest, exploring some of the other ways it can interact with (and help) write robust and useful programs.

Resources

• The pytest website.
• An excellent book on the subject is Brian Okken's Python testing with pytest, published by Pragmatic Programmers. He also has many other resources, about pytest and code testing in general, at http://pythontesting.net.
• "Testing Your Code with Python's pytest" by Reuven M. Lerner