Note: although this post focusses on mobile app development using Xamarin it also applies to other native mobile apps built in Swift, Java or even web apps. it’s 2017! whatever you are building get started with Test Automation!
As a consultant working for Xpirit i get to see a lot of different customers which I help with my expertise in building mobile applications to improve their mobile apps. Something I noticed in the previous year is that continuous delivery is a hot topic and companies and teams focus on deploying apps automatically to their testers through hockeyapp or even to the stores in beta and / or production.
In agile scenario’s (and come on who isn’t doing that currently? Every company or project I visit is saying they are agile or doing Scrum although some only do dailies and call that scrum 😉 ) In the current world it is really important to be able to release often because you want to be able to adapt to customer needs which are almost always changing and evolving.
Implementing a Shift left Quality Model
Test Automation is a process that does not belong to the developers or testers alone. It’s something that has to be in everyone’s mind from Product Owner to Developer and Tester. Automated tests can help you lower regression test effort but investing in Test Automation can really help you make a shift left focussing on quality earlier in your application development process.
Traditional Quality Model
In the traditional quality model most attention to quality happens in the phases AFTER development. This is a pattern that originates from the waterfall era and does not work that well with short iterations. When will you test or verify that you’re matching the expectations of your stakeholders? Quality issues will arise in the last few days of the sprint and there will be barely time to fix them in the current sprint. This is what happens at a lot of companies i visit so lets look at what we should do to improve this.
Shift left Quality Model
So how do we make this shift to the left and get earlier feedback on the software quality that we are building?
A common pitfall i see at companies is that they focus on the deployments only and then test manually which often ends up in testing a lot of functionality a sprint after the actual development happend. instead of having an Agile Test Pyramid in place I often come across the Software Testing Cupcake.
Agile test pyramid
The Agile Test Pyramid is a concept created by Mike Cohn. It gives a good description of how your automated test landscape should look like for any type of application. The Software Testing Cupcake anti pattern is often a Agile Test Pyramid turned up side down or just 3 different teams not working together building their own tests. Often this starts with plain old manual testing of the application. after a couple of releases and regression testing taking more and more time teams start to invest in test automation and invest in automating their manual tests by writing automated UI tests.
This may work for a while but automated UI tests are really expensive to build, break often and give very little information of what exactly is going wrong. Keeping these tests up to date can be hard whenever the UI is changing often and also take quite some time to execute.
The Agile Test Pyramid focusses on a solid base of Unit tests. Although unit tests do not focus on the end to end scenarios that the manual testers might be executing they do have a lot of advantages over Automated UI tests: They are cheap to build and maintain, give really exact error messages of where some piece of code is not working as expected and most of all they are (or should be) executing fast.
Another benefit of unit tests is that they ensure proper isolation of dependencies leading to a better cleaner software architecture. I’ve witnessed to many projects where was asked to help to write unit tests for a certain piece of software which was full of spaghetti code that made it nearly impossible to write tests. Writing unit tests immediately from the start while writing your app code (preferably even in TDD style) will certainly lead to cleaner and better testable code.
As an example of fast feedback take a look at the live unit testing feature of Visual Studio 2017 where you immediately see feedback of your unit tests while editing your code.
In my role as consultant I’m often ask to come up with some guidelines and rules around unit testing. My approach is that just setting certain rules around unit testing is not going to work it makes developers focus to much on these rules and not on the actual purpose: building better software. An example of this is setting rules on code coverage. A question that is often raised is: “Shall we set a rule for code coverage on 80 / 90 / 100%?” Doing this often leads in developers aiming for this number of code coverage and makes them think they are finished when reaching this “goal”. If you can reach the 80% code coverage with just testing all the simple code and leave 20% of complex code untested you are clearly missing the point of why we are writing unit tests.
“Shall we set a rule for code coverage on 80 / 90 / 100%?” Doing this often leads in developers aiming for this number of code coverage and makes them think they are finished when reaching this “goal”.
The only way to make unit testing work for your team is to have the full team believe writing unit tests is good for them as a team and it gives them the ability to make changes easier in existing code saving time and money.
If you are building a stable base of the testing pyramid out of unit tests make sure these tests run fast and don’t have any external dependencies. if tests become slow only 1 thing will happen: people will stop running them and all it’s value will be lost. If you set up a good software architecture that focusses on loosely coupling for example using dependency injection your code will be a lot easier to unit test because it’s also easier to create mocks for your dependencies. So what kind of tools do i prefer?
Unit testing Tools
In the Microsoft landscape there are a few flavors of unit testing frameworks. MSTest, NUnit and XUnit are the frameworks that are used the most. There are a lot of similarities between them but i think currently most people either use NUnit or XUnit. My personal preference leans a bit more to XUnit but there is no real winner here.
Next to a unit testing framework it’s also convenient to have a mocking framework. There are several mocking frameworks available. i prefer MOQ here but if you are used to other frameworks just stick with those. Other popular mocking frameworks are: FakeItEasy or NSubstitute.
Stacy Gay wrote a nice article on these tools described above called “The holy Trinity of C# Unit Testing“.
Another thing you might want to take a look at is Fluent assertions to create very nice readable assertions in your tests
Service tests are often called “Integration test”. I dont like the name Integration test because people only think about testing APIs or webservices. Service tests can be tests that integrate multiple units within a compontent or do actual integration between different components in your application landscape. The first kind of service tests can just use the same tools as your Unit tests. Doing integration testing through all your application layers from app to API to backend can be quite complicated because of connections to systems and test data. An advice could be to experiment with containers running your test backend so it’s easier to reset it’s state between tests.
When you want to create tests for your UI components “specification by example” can really help you building the right application your product owner really wanted. When building Xamarin apps the MVVM pattern is the design pattern used for building your UI Logic in 99% of all the apps i come across. I love the MVVM pattern because it makes testing your UI logic in your viewmodels really easy. If you have no experience with MVVM check out this episode of the Xamarin show by James Montemagno.
Specification By Example
Specification by example is a technique used in Behaviour driven development that focusses on building the right thing. How often did you show a new piece of functionality and your product owner asked you: “can it also do this?”, “what happens in scenario X?”. Oops you didn’t think about that edge case.. This is where specification by example comes in. Specification by Example says you should create your specifications in a way that you describe all the scenario’s in examples. These samples should not be created by the product owner or tester. this is a team effort. If you’re doing agile you’ve probably heard of the 3 amigos (BA, Tester & Developer). During refinement they should work together to work out all the different scenarios. These scenarios can then be written down in a certain DSL called Gherkin. The cool thing about this is that you can create automated tests that will execute your specifications. Your product owner will see the texts he helped write as specifications turn into green tests. So what about those questions i explained earlier? well the simple answer should be you’ll have to add another scenario in either case. if you start thinking in examples you’ll do a good job in covering most scenarios the first time but it can always happen that you missed some. just add another scenario and thus another test and you’re ready to go!
Below is a sample that is written in Gherkin:
within .Net you can use Specflow to turn Gherkin into automated tests. What specflow does is create a test method for each line of your specification file called a step. within this step you can execute the actions you would expect the app to do. The Given When Then syntax really works great together with Arrange, Act Assert that you are used to use in Unit Tests. In the Given steps you’ll do your arrangements, in the When steps you do your Actions and then in the Then step you can do your assertions. This works really well with Xamarin executing these tests on your viewmodels. You can also take it a step further writing Automated UI tests in combination with Specification by example.
Automated UI tests
We’re almost reaching the top of the test pyramid and the next step is automated UI testing. A common mistake here is to focus to much on your UI tests while not having a good foundation of Unit tests. Automated UI tests look like a really good solution to automated your manual test efforts but it is always a combination of this + Unit tests. Automated UI tests often do require quite some maintenance if you aren’t stubbing away your backends and UI’s can be quite heavy to change and then your tests might also have to be changed often. UI tests also give very poor details when things are not working as intended. You’ll just get the feedback a user would also get for example: “If i press this button it should reload the list”. If something is not working right you’ll still have to find where exactly your code is not working properly: Hello Unit tests!
Automated UI testing does have some really good benefits especially in the mobile world. The mobile device landscape is really diverse with 1000’s of different devices all with different combinations of OS versions & screen sizes. Xamarin test cloud is a really good way to test if your app is really working on all those devices.
Xamarin test cloud also works great in combination with Behaviour driven development and specification by example. you can easily create your specifications in Gherkin and hook up Specflow together with Xamarin UI tests. I did a Xamarin university talk on this last december.
Xamarin test cloud together with Specification by example gives you great feedback on how your specifications look on actual devices. Take a look at the specifications on the left and then you’ll get a screenshot of how the app looks during each step.
Manual exploratory testing
The top of the pyramid is about Exploratory testing. This is something that isn’t something you can automate. Although these are tests manual tests these tests aren’t about writing a test script and following all the steps every time you’re executing these tests (that is just plain old manual testing). It’s also not clicking around like crazy to see what happens. This is called Monkey testing :). Exploratory testing is about looking for edge cases and try to see what happens there. An example could be: What happens if you kill the cellular connection halfway loading the list of items, or what happens when i get a phone call when i’m using my app. These tests are really hard to automate but do give great value to your development team.
Wrap up: It’s 2017. Start implementing some test automation in your apps!
Hopefully this post inspired you to start automating your tests within your app development lifecycle. If you have any comments about tools or practises i’ve missed in this post (of course I could not list them all) Hopefully after 2017 we can look back and see a change in more and more people embracing the test automation spirit.
Geert van der Cruijsen