Technology Insights

6 Ways Quality Drives Product Development at AppDirect

By Ideas @ AppDirect / November 30, 2022

At AppDirect, our commitment to providing the best software possible is one of AppDirect's defining characteristics and we believe that ensuring the highest quality for our products is everyone’s responsibility—not just the job of our quality engineers.

We use the Net Promoter Score (NPS) as our baseline metric to measure the quality of service we deliver to our customers and how quality affects their level of satisfaction. Used widely as a key metric, the NPS score is based on a survey that asks customers to rate how likely they are to recommend a product.

Quality is intricately woven into every product

With quality and customer satisfaction as key drivers in our product development approach, the AppDirect Quality Center of Excellence (QCoE) team is committed to advancing AppDirect's quality culture and fostering continuous quality improvements. The team establishes quality standards and trains all our engineers to ensure a consistent understanding and application of the standards across the company. We use a combination of industry standard testing practices and cutting-edge test tooling and processes for all teams to follow. This also allows us to scale consistent quality across the entire organization.

In this blog

In this blog, we’ll share the 6 primary mechanisms our Quality Center of Excellence (QCoE) team uses to ensure our SaaS products meet the highest quality standards:

1. Automation-first mindset

2. Three-environment deployment approach

3. AppDirect testing pyramid

4. Quality-driven software development lifecycle process

5. CI/CD (Continuous Integration/Continuous Deployment)

6. Feature release

1. Automation-first mindset

AppDirect utilizes an automation-first approach.

• For product developers, this means ensuring unit and component testing is in place before code is merged.

• For quality engineers, it means that integration and end-to-end testing has to be planned and implemented, and the definition of done needs to be determined at the outset.

Having a robust automation suite ensures that manual testing is focused on corner cases, sanity testing, and the user experience.

2. Three-environment deployment approach

AppDirect uses a three-environment approach in its release pipeline—integration, staging, and production. Each environment has inbound and outbound testing requirements, ensuring that all code changes flowing through pass our test automation coverage and product acceptance criteria.

As the code builds, unit, component, narrow integration, and contract tests are run as part of continuous integration.

1. Once these tests are passed, the code is deployed to the integration environment, where end-to-end and broad integration tests are run.

2. The code is then promoted to the staging environment where end-to-end, broad integration tests and system integration tests are run.

3. The code is finally promoted to production behind the feature flag where final sanity checks are performed.

3. AppDirect testing pyramid

The AppDirect testing pyramid follows industry standards closely. It outlines exactly which tests should be automated and where they should be run.

The testing pyramid aligns teams and processes across the entire AppDirect organization by defining supported testing types, as well as when, where, and how to execute them.

The specific definitions are as follows:

• Unit and component tests—The most numerous tests because they run the fastest, these tests verify that each method produces the expected results and that the collection of methods work together to provide the needed product functionality.
• Narrow integration tests—Conducted in CI using test doubles, these tests validate the interactions and integrations with its dependencies.
• Contract tests—Contract tests are specialized cross-team component tests that verify if the provider interface matches the consumer expectations.
• Broad integration tests—Run in a test environment using real services to validate the system, these tests provide the desired functionality with all dependencies and integrations working together.
• End-to-end UI tests—Often referred to as E2E tests, these cover critical workflows through the AppDirect platform. The E2E tests ensure a collection of services work together to accomplish a workflow goal.

On-demand tests

In addition to the testing we conduct using the standard test pyramid, we also run the following additional on-demand tests on AppDirect products, as required:

• Performance tests—Performance tests measure how well a service operates under production-like data volume and web traffic. They’re run on a dedicated load environment where services are measured for availability and p95, p99 response times.
• Security tests—AppDirect adheres to the Security Development Lifecycle (SDL) process to ensure security is integrated into each stage of Software Development Lifecycle (SDLC).
  • The SDL process helps developers build more secure software and address security compliance requirements while reducing development cost.
  • The information security team (InfoSec) security review consists of three major stages:

    • Security design review (SDR)

    • Static application security testing (SAST)

    • Dynamic application security testing (DAST)

• System integration testing—Often referred to as SIT tests, these cover critical user journeys through the AppDirect platform. SIT ensures the entire stew of services, databases, UIs, third party services, and background jobs function in a production-like environment.
• Accessibility testing—Often called A11Y tests, these define A11Y best practices that are followed during SDLC. This testing ensures the AppDirect platform conforms to WCAG AA compliance standards.
• Cross-browser and visual regression testing—AppDirect platform leverages both cross browser and visual regression testing (VRT). Cross-browser testing ensures web pages are functional across different browsers and free of HTML, CSS, and JavaScript issues. VRT validates that changes made to the platform do not negatively affect the visual appearance of the UI.

4. Quality-driven software development lifecycle process

At AppDirect, we know that for most customers, quality is one of the top features they're looking for. That's why it's an integral and essential consideration at every step of the software development lifecycle (SDLC). The following diagram showcases how we put this into practice—from planning to feature development, through to feature release.

Let's look into each section of the diagram in depth.

Feature planning

New features are prioritized by the product owner before being presented to the scrum team for discussion. The planning team discusses features to give the team a better idea of the functionality required. Together, developers and quality engineers define the acceptance criteria and discuss and document design considerations, integrations, functional and non-functional testing (performance, security), and potential risks associated with the feature.

Next, stories for feature development, test automation, and non-functional testing (if necessary) are created and estimated in the feature development ticket. The feature ticket is not closed until all stories are complete.

Sprint planning

At AppDirect we follow SAFe agile practices, with sprint planning being an essential component. During the sprint planning meeting, the features that are ready to be worked on are added to the sprint plan, and the developers and quality engineers are given stories to work on.

Development

Developers are responsible for writing unit, component and narrow integration tests. These tests are run as part of the continuous integration workflow whenever code is committed to the branch. Developers are encouraged to maintain 80 percent or higher unit test coverage.

Quality engineers are responsible for writing contract, broad integration and E2E UI tests, along with creating the stories for performance testing. A suite of smoke tests is also created and run as part of a continuous deployment workflow. Smoke tests verify that a feature is working and ensure there are no show stoppers in the build being tested. In essence, it’s a small and rapid regression test suite of the major functionalities.

The security team (InfoSec) is responsible for the SDL process.

• The scrum team submits an initial risk assessment, after which the Infosec team starts the security design review (SDR), which is an in-depth analysis of application design.

• The team then conducts dynamic application security testing (DAST) to identify security flaws and vulnerabilities through penetration testing.

• Static application security testing (SAST) happens as part of a continuous integration workflow, with automated deployment blocking in the event that security vulnerabilities with a certain severity threshold are detected.

In order to reinforce and improve the security and quality of the software we deliver, all new services must successfully pass the SDL security review process before being deployed in production.

5. CI/CD (continuous integration/continuous deployment)

At AppDirect, we use continuous integration and continuous deployment (CI/CD) to ensure we can deliver the highest-quality products quickly, securely, and efficiently.

Continuous integration

A continuous integration workflow allows us to implement both quality and security checks on every code check-in. For quality checks, unit, component, contract and narrow integration tests are run to provide rapid feedback to the developer. For security checks, static code analysis is done on the pull request and blocks the merge if security vulnerabilities are detected.

Continuous deployment

AppDirect follows continuous deployment practices to deliver small incremental changes to production as quickly as possible. Deployment happens sequentially through integration and staging environments before changes are deployed to production. Smoke, E2E UI, and broad integration tests provide the necessary quality checks for each environment before code is promoted to the next environment. Using feature flags helps ensure these automatic deployments do not impact the customer experience unexpectedly.

6. Feature release

Once the feature is built, the next steps are to deploy the feature to the marketplace.

After the code for a feature is deployed, we use feature flags to quickly and easily enable or disable it, with no need for additional code changes. With feature flags, AppDirect delivers every new feature to production incrementally by releasing it in three phases: preview, early adopter (EA) or general audience (GA). New features are enabled in test environments for quality sign-off before they’re enabled in production.

Release sign-off

Once a week, we schedule the next set of feature flags to be enabled. Even though most code changes are deployed into production automatically once they pass all automated quality checks, feature flags allow us to control when the changes are made available to customers. Prior to enabling a feature flag for everyone, quality engineers and product managers run tests in production, by enabling the features first in the production sandbox environment.

Monitoring

At AppDirect, engineering teams leverage real user monitoring (RUM) and application performance monitoring (APM) to view and resolve factors affecting platform performance. These tools enable us to maintain system availability, enhance the user experience, and improve service performance and response times. For each service, monitors are set up to continuously check metrics and scrum teams can be alerted when critical shifts occur, allowing them to take action quickly.

Conclusion

The approaches we’ve described provide AppDirect with a strong quality foundation. Our QCoE approach and processes play a crucial role in defining the quality vision and maintaining a quality culture. QCoE ensures scrum teams adhere to quality processes, it produces in-house training to upskill quality engineers, and it defines quality key performance indicators (KPIs) to gauge whether quality is trending in the right direction.

This article was written through a collaboration between AppDirect’s Brian Albright, Darpan Shah, Madhav Bhogaraju, and Rahul Jain