You are here

Black Box

Choose Your Test Approach with Cynefin Help

Testing is a complex activity, involving many decision points and multiple possible approaches to the same end goal, that of ensuring the most useful information is provided about the tested software.

At the same time, nature is also complex, and an inspiration source for some very useful models, such as Cynefin. This sense making and context determining model is quite popular in the agile practices world, and can also be a good candidate for testing.

Is test automation the best approach? Should automation alternatives be considered? Should I build or buy my testing solution? These are some of the questions that people involved in testing are facing many times, and that can be made easier to answer to with the help of the Cynefin framework.

My session reveals how Cynefin can be used to make sense of the testing context, thus helping determining the most suitable approach for testing.

At the same time, since Cynefin is not a static model, the session highlights how the testing context is or can be changed, updating accordingly the testing approach.

Key takeaways:

  • Awareness of the context challenges for testing
  • Introduction of the Cynefin framework
  • Mapping testing strategies to Cynefin complexity domains
  • An example for supporting the “no-automation” decision

Let’s Share the Testing!

In the world of DevOps and Continuous Delivery how can we, as Software Testers, adapt to continue to add value within our teams?

Within my cross-functional Agile team, the testing activity had become the bottleneck. The ‘To Do’ cards on my Kanban board were piling up. As the sole test specialist within my team I felt as if I was preventing us from being able to release code to our live environment. Frustrated, we got together as a team to discuss how we could fix this problem.

Our solution? I was going to share my exploratory and automated testing knowledge with the team. We were going to test throughout the development process. We could design test plans and discuss technical challenges together. We were going to collaborate on the testing effort. My role as the test specialist was going to evolve, which made me nervous.

In this talk I’m going to share how my team removed the testing bottleneck, increased productivity and started to become true cross-functional team members.

Key takeaways:

  • How sharing your testing knowledge can increase your team’s productivity.
  • How to encourage non test specialists to get involved in the testing effort.
  • How sharing your testing knowledge can improve communication within your team.

Dude, Why Don't You Test UX?

I noticed, that most of companies have no UX designers and products are just created by developers without any clear idea of how it should look and how it should work. I want to remind, that quality is not only accurate numbers in tables or correctly filled forms. Quality is also overall look and feel of the product, so QA should also work on making product look and feel good (or at least better).

I will ask my audience some question e.g:

  • How many of you know what is UX? And will give a definition of UX and also explain the difference between UI and UX.
  • How many of you have UX designer on board?


Firstly i would like to tell my story of how I used to work with UX designer and how i handled my work while having no UX designer in the company at all. I will tell some tricks on how to convince your developers and team leads to start changing the UX and how I managed to get a UX designed on board, including:

  • Learn about good UI/UX;
  • Start writing UI/UX related bugs;
  • Do some usability labs;
  • Use paint (crop and drag!);
  • Question new features;
  • UI/UX hall of shame/fame;
  • Stop talking about business to business applications, we all want to use "Facebook" on our daily basis!
  • Don't get used to bad UX!
  • Grow little UX designer in every developer!
  • Don't stop, even if you have UX designer, as you are the one who uses the product every day!


Direct UX impact for your product:

  • For apps UX usually impacts rating in app store;
  • For webpages UX can impact the sales of products on website, likability to use your web against concurrent product and more;
  • For desktop application it might take a lot of time to train your users if UX is not intuitive and users are more likely to choose competitors product against yours.


Key takeaways:

  • UX is important and can affect your product ratings in app store and overall user satisfaction.
  • If you have no dedicated UX designer, then QA should also cover this role.
  • Ways, how to drag your colleagues in to UX, so it would not be only your wish, but all teams purpose and target.
  • What you can do to make your product better on a UX side.

Vision Boards - Project Your Goals

How do teams share their understanding on the common goals? It is either audio or visual. Recording each talk and storing them (tagged) is not the most effective way to share common knowledge. Sketching is not new to agile teams. We are taking it a step forward in the form of Vision Boards. Vision Board – is creative visualization of your goals. While our focus in this talk, remains on- how teams could use the board, Individuals use these in order to make their life goals into reality. There are pictures or sketches of what they want – all pasted together on one board – so they constantly remind themselves of their ultimate goals in the bigger scheme of things. These goals may not be achievable with one task. They may need a series of tasks which do not directly seem to be connected with the goal. But these visualizations captured - are very good indicators of what success means to one.

We used Vision Boards to visualize our customer experience, their reactions and expected patterns of use for our application. This board single handedly kept all our teams aligned and as many changes happened – the teams knew their true north when they were discussing how to design the screens and which features to build on (priority). Our already agile teams were constantly looking at the short term goals of prioritised features, but vision board helped them reduce chaos and clutter and saved lot of time on understanding the overall requirement  - it also served as the basis for User Stories.

Key takeaways:

  1. How to capture the common vision, effectively?
  2. What is really important? What qualifies to go into the vision board?
  3. How do teams relate with the vision board. How do we to build it? And continue to refer to it.
  4. Benefits of using Vision board in a volatile environment.

The Age of Virtual Reality is Upon Us

The age of virtual reality is upon us. Just around the corner from now, the market is going to explode with companies geared to providing their customers with customized VR experiences. Already we have Oculus, HTC Vive, Google Cardboard/Daydream, amongst others. And Apple has finally joined the race introducing their VR development kit. Are you ready for this? In this talk, I will go over different automated testing techniques, the challenges we face, and how you can validate your VR content with existing technologies such as Selenium, Appium and image comparisons. I will provide a live demo with code examples and cover the different tools to achieve this.

Key takeaways:

VR is a tricky thing to validate in a deterministic way. The techniques I cover will show you how this can be done in an automated way without having to manually test your VR application. With image comparison technologies, and libraries such as FFMPEG, Selenium/Appium we can utilize these tools to validate our VR applications. 

Attendees will come away with:

  • The difficulty of testing VR in a deterministic way.
  • What tools are available to us to help test VR applications.
  • How they can test VR applications in a deterministic automated way.
  • Code examples to get them started automating VR applications. 

Doubt Builds Trust

In an uncertain world, your team wants answers. Project managers want to know when you can ship. Project owners want testing to be done. Developers want to know that you’ve caught all the bugs. Testers can find jobs getting paid to assure people of a product’s quality. But I don’t trust testers who always have confident answers to their team’s questions. Eventually a bug gets through, a deadline is missed, or a commitment is broken. Testing is not quality assurance. I trust the tester who expresses doubt. Doubt builds trust.

In my talk, I’ll explore how safety language, specificity, and nuance should color everything about the way we work. Testing software means engaging with uncertainty, and our communication should reflect that. Saying “I don’t know” can spark the beginning of a dialogue. Being able to admit the possibility of an unexplored path, a unknown interaction, or a fallible memory makes the difference between a team that moves forward and a team that stagnates by digging up evidence of mistaken certainty. We’ll get thinking about why it’s most important to say “I don’t know” in an interview and how admitting doubt can help a tester find an environment where they can thrive.

Key takeaways:

I want to give testers the power to be vulnerable at work. Rather than staying silent, testers can start admitting what they don’t know to get better explanations for themselves. If they’re already a pro at this themselves, they can encourage their teammates to voice their questions and foster an environment where this is encouraged. When attendees get back to work, I want them to be able to:

  • Approach situations with humility and a desire to learn.
  • Refrain from mocking or judging those who know less than they do.
  • Consider the privileges that allow them to gain knowledge others haven’t.


A story of building a unicorn in a no-QA organisation, told from the quality assurance perspective.

In my IT career, I transitioned from manual tester to test automation and performance testing engineer. Then to software engineer, tech lead and engineering manager. On that journey, I have witnessed and experimented with various approaches to the QA. In the end I found my happiness and zen in product organisation setup that has no formal QA at all. No QA role, no testers, no testing managers, no one particular responsible for the QA. Sounds a bit chaotic, right? That's how we do it at

But no worries, because in product engineering the day to day assurance of quality is actually extremely granular and goes way beyond software. I am keen to share our experience and learnings on:

  • How we measure the product quality. Because having no bugs in the application does not mean that product is of great quality.
  • How we track the quality of our product engineering teams, their deployments, releasing, pull requests, services etc.
  • Our methodologies for assessing the product impact of every move we make, every step we take. What we track about customer behavior. How we assess the price of our real and theoretical mistakes.
  • System design quality. Journey from monolith to microservice architecture.
  • Tough life of engineers when they have no testers to rely on. And how we empower them. 
  • Blameless postmortems. 

Key takeaways: 

  • The only sustainable no-QA org setup is probably all-QA
  • Quality in product engineering is very granural - data-driven approach helps to navigate
  • The system needs to be self-aware in terms of quality and test itself 24/7

Opening Keynote: Why Should Exploratory Testing Even Be the Subject of a Keynote?

Exploratory testing isn’t a new concept. It’s been around for ages, it’s a generally well-known approach, and its details have been well described. So why on earth could it be the subject of a keynote?

Because we are, as an industry, still not doing enough to understand, teach and promote the power of exploratory testing. We use it and talk about it (some more confidently and effectively than others), but we are – ironically – often only scratching the surface.

In this keynote, it is my aim to rejuvenate your interest and passion for doing and teaching exploratory testing. I’ll look at why it’s so powerful and necessary, and why it could be our savior in the future. Through stories, examples and new names for things that we’ve called “intuition” for too long, I’ll get you thinking about how you can explore your exploration more – to deliver teams and projects the valuable information they need.

Ending Keynote: How Come Testers Are so Incredibly Successful

It was so mind puzzling how much of my daily life was not contributing to my success and I was just doing things regardless if they improved my life quality or not. It really bothered me. I was not really improving as a tester anymore and not my life either.

I started to use my testing mindset in my life deliberately and challenge it more and more. I found while testing my life I could pinpoint bugs in it and I could get to the root causes of how I was stuck. This in return made me a better tester and so has the loop continued.

Moving abroad with my family was one of the things I did. The experience of living in a different environment has made me realize the value of the testing mindset to a new extent.

I have experienced many controversial methods work out like zero bug policy or transition from developers' and testers' organization to dev ops model as more specific examples.

This story-filled keynote will take you through the personal examples of how I improved my life quality and how that relates to becoming better in the craftmanship of engineering.

Key takeaways:

  • How testing skills hugely improve life quality
  • The value of living abroad and how it contributes to success
  • Key experiences changing from a tester and a developer organization to dev ops and how zero bug policy has helped greatly

Driving Higher Quality with Devops Initiatives

Plenty of things have been written about the roles of testers and QA in the ever changing world of developers, devops, SRE et al. It is a fact that devops practices are here to stay, and the demands for time to market is getting ever so small. Still, high quality is what will keep your product relevant. 

Sigge will describe how devops initiatives can affect the culture of quality amongst development teams, driving high quality software delivery. 

Key takeaways: 

  • How devops and quality work fits in the bigger development context
  • How to get developers more aware and caring about quality
  • The importance of all kinds of automation in a fast paced future

OPENING KEYNOTE: Machine Learning from System Quality Perspective

Machine learning systems are among the most complex ones out there. For now, machine learning and data science become a must when a company grows big enough. But with time, it gets adopted in ever more places, big and small. Is it the future? Most probably. For us, at Bolt, it's present. We have grown in 5 years to become a global ride-hailing company operating in tens of countries, with millions of monthly rides. How to handle and optimize everything that's going on? Data science and machine learning was the answer. But creating some models and algorithms is just the first part of the adventure. Each call to this 'artificial intelligence' returns results to some real person in the world, so the quality must be top-notch. And this field is young, the technology changes blazingly fast and there are few established QA practices for it. Solving quality challenges at scale is not easy, but it's rewarding. And fun too.

Machine learning is the cherry on the cake of our technology. I will tell you a story of how we grew as a company, team and an intermix of ever more complex systems.

Key takeaways:

  • If you don't intend to have machine learning in your organization, you might feel happy.
  • But if you do want it, you will be prepared, and happy too.
  • And if you never gave it a thought, you might want to try it out!

Testing Backend API from Mobile QA Perspective Using Rest Assured

From the position title "Mobile QA" you may think that you only have to deal with testing of the applications themselves. This is true sometimes, when you have a purely offline app or your app works solely with 3rd party API. My experience shows that most of the apps deal with the backend, and there will be your responsibility to make sure it works as expected. There are numerous examples of tools for API testing, and Rest Assured is only one of them, and the one that I currently use. I will tell you about challenges that may occur during backend API testing and what result mobile apps developers expect to get from you.

Key takeaways: 

  • Why mobile QA should test backend
  • What are key points to check when writing API tests
  • Using Rest Assured for API testing

Security by Stealth

Security isn’t very fun for development teams to think about. It’s complex and something that isn’t brought to mind when considering requirements. Too often, it neglected by teams and left to the end for penetration testers to consider. But, it doesn’t have to be. Security can be considered early in the development cycle. How can we encourage this behaviour? How can you get development teams interested?

Security is an important skill to possess while delivering quality software. The cost of not having security skills within teams is now more obvious than ever. Security should be in the forefront of development teams minds. Even with these risks, data leaks and denial of service attacks are in the headlines often. How do we stop our companies being another statistic?

Learning should not be compulsory. Especially if you want something to become part of the culture. Beginning with a simple workshop to expanding to a security guild, people were eager to be involved. This lead to further workshops which included the basics of threat modelling using STRIDE to the complexity of automated checks. Security at Sky became not only fun but cool. Security was no longer a rarely thought about requirement but a fun, oft thought about need.

Accessibility Assumptions and Arguments

There is a massive assumption in software development that accessibility = disability. I’ll dispel that myth with information, examples and practical tips of how our assumptions are potentially costing us, customers, making interactions harder and that the whole population has accessibility ‘issues’ with the applications we are building.

Key takeaways: 
Attendees will learn about accessibility assumptions, mostly false and how they cloud the small amount of attention we give it.That accessibility actually affects approx. 90%, yes 90% of all the ‘users’ who visit your site or use your app.Some common mistakes we make when designing sites and applications.

Arguments against accessibility and how you can counter them if you encounter them, and you shouldn't because considering accessibility is the right thing to do! Accessible to All Quadrants that put accessibility in a new light of inclusion looking at both readability, inclusive language, usability and compliance to the web content accessibility guidelines as only a part rather than the goal.

Why using inclusive design to ensure your site or app is accessible is the right thing to do, did I mention that?  Because it is!

Quite a few tips the attendees can take away and apply the next day to improve the reach and impact their sites and applications can make. The testing they can do immediately to help ask questions to improve their quality the very next day. 

Listen Beyond the Pass/Fail - Analysing Test Results over Time

Does this sound familiar? Each morning when I got into my office and opened my mailbox I would have received a report of the nightly test run, mainly Selenium and TestStack White run through Team City on dedicated test agents. Most days at least something had failed and most days we didn’t worry too much about it. After I while, a voice in the back of my head started nagging me about there being a pattern but I couldn’t pinpoint it when looking at each individual test run.

When I put all the historic data into a model and started twisting and turning it, a number of patterns stood out clear as a day and by using them I could start working on long-term continuous improvement instead of putting out fires.

During this talk we will look at a number of perspectives you can explore and how they might give you important insights:  

  • Tests that are always green: Do we need them? Why are they never failing?
  • Tests that are always failing: Do they add value? Can we remove them?
  • Tests that fail a lot: Is there an underlying issue? Are we addressing the problem or just re-run them locally and blame the environment?
  • Are the tests run when we need them, or are we running them nightly because we see no other option? Can they be sped up to a point where they can provide a faster feedback loop? No? Are we sure?


We will also address the problem of trying to use the same collection of tests to solve multiple competing needs: fast feedback to developers about recent changes, feedback to testers on where to exlore, feedback to team about releasing to production and feedback to managers who want to know they are spending their money in the right places. Can it be done? Should it? Are there alternatives?

Key takeaways: 

  • Looking at your test results over a longer time period will give you additional insights
  • What you can look for
  • Trends can be analyzed for continuous improvement

Utilizing Component Testing for Ultra Fast Builds

A best practice of software architecture is to design your applications into independent modules or components, with a published contract for interaction between components. This is a principal of the popular microservice style architecture, but also it applies to components created in a large monolith or with the out of the box patterns that front end libraries promote.

If we are able to test the functionality of the component independently, and apply a level of trust that those components work, this opens the door to rethinking, Continuous integration and Continuous delivery strategy, potentially reducing the need for long test suites and many environments. It will also cause a rethinking of how to unit testing strategy and the test pyramid.

In this talk Tim Cochran, will talk through the different kinds of component testing, show working examples and advice when to apply them. He will also cover what this might mean for your organization's broader testing strategy.

Key takeaways: 

  • Learn about component testing strategy
  • Worked examples of testing front-end and back-end components
  • Adjusting your CI/CD pipelines to take advantage
Subscribe to Black Box