Non-Existent Python Mock Convenience Methods: A step toward sanity
Several months ago, I spent some time researching the potential dangers of incorrectly using the convenience methods provided by Python's mock library. To summarize briefly what I learned, it's really easy for our human brains to make mistakes and try to utilize non-existent methods on mock objects. When we do this, the mock object will kindly assume we meant what we said, and assign the non-existent method as a new mock method on the mock object (not what we intended at all). So, for example, we might intend to use assert_called_once_with but type assert_called_once. Such a tiny mistake, but now any test that relies solely on this assertion will always pass. This is because assert_called_once is not one of the convenience methods actually provided by the mock library. Here’s a generic example illustrating this problem:
As tends to be the case, research only goes so far. After a few weeks, I had to admit that I hadn't taken any concrete action based on what I’d learned (besides being more mindful when writing tests). So, when I found myself with a little spare time, I decided to try out one of many potential solutions—a simple blacklist check that would not only point out mistakes to developers on their local machines, but would cause our continuous build system to indicate a failure in the case that any blacklisted assertions were in use.
Note: Blacklist solutions, as with all solutions, have their shortcomings. Most notably, only predictable mistakes are caught and prevented. There are many ways to protect against non-existent convenience methods; this is merely one.
What I did
I chose my team’s own Python client library as a guinea pig for my new mock checker. The client is a reasonably small code base with just over 300 unit tests at the time of this writing. Having spent a considerable amount of time in our test suite, I had doubts that the checker would find anything—the human ego is a funny thing. (We all know where this is going).
The starting point for my mock checker script was a gist provided by Tyler R. in his article assert_called_once: Threat or Menace. Here it is:
I wanted developers to always get fast feedback when running their tests locally.This gist is actually a git pre-commit hook written by Tyler’s colleagues Cheng C. and Nicholas N. I didn't feel great about preventing developers from committing, though. I had slightly different goals:
I wanted our continuous integration tool to quickly report a broken build if the check failed.
- Even if they broke the build, I didn't want to reject bad commits (in light of the use of feature branches).
As well, there were a few fundamental changes needed in the script itself:
The blacklist needed to expand to include some suggestions from colleagues (as well as a few others I had in mind).
Because Python’s compiler module is deprecated and the use of Abstract Syntax Trees is (arguably) a bit complex for such a simple operation, I opted to check files with ordinary file-reading methods (see method check_file - no need to understand ast here).
- My simple use case didn't require passing or parsing of arguments.
A humbling surprise
After writing the script and updating our Makefile, I discovered that my team’s client was making use of two non-existent mock convenience methods in five locations. Worse, a git blame revealed that my name was all over these lines (whether I originally wrote the tests or moved them to a new location without properly checking the validity of the code I was moving is beside the point).
The silver lining was that this provided pull-request reviewers with an example of what our continuous integration tool’s output would look like in the case that errors were found by the new script (though I assured everyone that I'd fix the tests prior to pushing my changes):
Non-existent mock method check:
tests/test_client_use.py:136: you may have called a non-existent method (assert_not_called) on mock
tests/test_files.py:166: you may have called a non-existent method (called_once_with) on mock
tests/test_files.py:175: you may have called a non-existent method (called_once_with) on mock
tests/test_common.py:544: you may have called a non-existent method (assert_not_called) on mock
tests/test_common.py:558: you may have called a non-existent method (assert_not_called) on mock
make: *** [unit] Error 1
In very short order, my team’s Python client library gained some protection against commonly used non-existent convenience methods on mocks. Is this solution perfect? No. Was this my best coding effort? No. Did I learn a lot about Abstract Syntax Trees without even realizing I wanted to? Yes. Will you be inspired to run a script such as mock_checker.py against your code base or implement your own solution? Hopefully. Feel free to modify my script for your own purposes and leave a comment if you try something out. You might be surprised what you find.
Copyright 2016 Workiva Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.