Automated validation toll gates for a big data platform

The challenge

At first, the client focused on extending the data platform with new data sources and adding new functionalities and reports to the web application. Our team noticed that the root cause of most bugs and time spent on maintenance were issues in the underlying data: providers not following the agreed format, missing values, non-matching identifiers. The responsibility for executing data-related tests was spread across four teams, and their outcomes were never gathered together.

The two business goals:

• Build an extensible data platform that gathers information about companies worldwide and connects the data relevant to each company from dozens of different data sources.
• Design and implement a configurable web UI, for internal and external customers, that supports analytical activities, building the decision funnel, browsing company details and composing detailed comparison reports.

What we did

• Conducted workshops with all the teams in the program to determine existing test catalogs, level of automation, execution models and reporting approaches, which resulted in “a golden source of truth”.
• For data-specific checks, ran a series of workshops, recommended the tool with the highest ROI for the project, and mentored relevant stakeholders.
• Applied a “fail fast” approach to get immediate feedback about the changes introduced.
• Implemented an engine able to execute tests gathered in many suites, written using different test frameworks.
• Integrated six different test sources developed by internal and external teams.
• Built a single source of truth and control center for storage, maintenance, execution and reporting of test case results.
• Introduced an Allure-based visual tool for business stakeholders to monitor the health of all application environments.
• Set up automated toll gates at different stages of data processing that measure quality constraints and stop the process if they are not addressed.

How it works

Based on our experience from earlier projects, the team recommended a new approach, which the client accepted. Quality checks now sit where most of the bugs started: in the data, as it moves through processing.

1. One engine A distributed end-to-end testing solution executes tests gathered in many suites and written in different test frameworks, from six sources built by internal and external teams, all run from CI/CD pipelines.
2. Control center Test cases and their results are stored, maintained, executed and reported in one place, the single source of truth for testing.
3. Toll gates Automated and unified data validations at different stages of data processing measure quality constraints and stop the process if they are not addressed. Crucial data defects surface early, and the client escalates them with the respective data vendors, avoiding more thorough manual testing of the data sets.
4. Visibility for decisions An Allure-based visual tool gives business stakeholders an aggregated, holistic view of the health of all application environments, a solid base for educated release Go/NoGo decisions.

Alongside test automation, CodiLime contributed data engineering, DevOps and UX/UI work to the project.