Examples, Acceptance Criteria and Acceptance Tests
Acceptance criteria vs. Acceptance tests
Recently, i had an interesting discussion with my good friend Xu Yi. We discovered an interesting difference in our way of thinking about acceptance criteria and acceptance tests.
In his thinking, acceptance criteria and acceptance tests have one-to-many relationship; while in my thinking, acceptance criteria and acceptance tests have many-to-many relationship.
Let me illustrate with a story about "cancel reservation".
Acceptance criteria could be:
- cancel 1-day before travel begins
- charge 10% for normal user, while no charge for vip user
- email notice about success or failure
His list of Acceptance tests is something like this:
- fail when user tries to cancel on the same day as travel begins
- succeed when user tries to cancel 2-day before travel begins
- normal user is charged with 10% for successful cancellation
- vip user is not charged for successful cancellation
- email with success notice is sent for successful cancellation
- email with failure notice is sent for failed cancellation
My list of Acceptance tests is something like this:
- a normal user cancels a reservation 2-day before travel begins, succeeds with 10% charge, and success notice is sent
- a vip user cancels a reservation 2-day before travel begins, succeeds with no charge, and success notice is sent
- any user cancels a reservation on the same day as travel begins, fails and failure notice is sent
[note] the above tests are still not concrete enough, but this is not particularly relevant for this discussion.
He likes his flavor because that makes each test very focused, if it fails, it fails exactly one thing. I like my flavor because that makes each test a user task, it creates more understandable specification.
Then, the discussion went to what examples are, are they acceptance tests, acceptance criteria or something else? Examples are concrete, while acceptance criteria are more abstract rules. We tend to think that examples are not the same as acceptance criteria, while both examples and acceptance tests are concrete, thus they are more similar. Are they the same? If we ask users to give an example, they would usually not go to the granularity in Xu Yi's list, while possibly similar to the granularity in my list. From here, i realize that while we talk about examples, there are two different roles they play.
Examples as discovery vehicle
Examples are powerful vehicle for the discovery. In big discovery to creating stories, you tell customer journeys, while in small discovery to eliciting acceptance criteria, you tell stories. They both are examples. Starting from some examples, we apply heuristics or play "what about" to get variations and alternatives, then rules emerge, then more examples. We switch between abstract thing and concrete thing, and iterate for discovery. Examples in this context are not specification.
Examples as Acceptance tests
Eventually, examples are refined into specification, and they are also called Acceptance tests. There is some tradeoff during refinement. It is more understandable from business and user perspective once the granularity matches their natural size. It is easier to debug from development and testing perspective with smaller granularity. It is also possible to supplement very small thus very focused examples, as illustrated in Xu Yi's list, with more integrated big-picture examples.
In short, here's my current way of thinking about examples, acceptance criteria and acceptance tests. We start discovery from rough or un-refined concrete examples, derive abstract acceptance criteria from those, then, refine examples into acceptance tests, which are still concrete but refined ones.
