Building QA Infrastructure (Part 1): Establishing Upstream Visibility

Context and starting baseline

When you are brought onto a project as the sole QA engineer handling User Acceptance Testing (UAT), your immediate instinct is to dive straight into the sprint backlog. You want to verify incoming fixes, execute regressions, and ensure nothing breaks before deployment. On my current project — a Salesforce-based order management system (OMS) stream with fast-paced, biweekly release cycles — that was exactly how things started.

Although other QA engineers supported different areas of the wider enterprise account, I was the only QA responsible for this specific order management stream. I work directly within a small, cross-functional team alongside a Project Manager (PM), a Business Analyst (BA), and developers. In this stream, the BA focuses primarily on testing within the Integrate environment, while my role is to verify functionality on UAT and drive the regression testing before production deployments.

It didn’t take long to realize that while daily testing activity was happening, the infrastructure around our QA process needed more structure. Crucial information was scattered across Jira, Xray, Confluence, and-worst of all-locked inside people's heads.

Addressing this required stepping back from daily test execution to build a resilient QA infrastructure, a process I’m sharing across a three-part series. In this first post, I’ll cover how we established live operational visibility across our delivery pipeline. In Part 2, we’ll look at preserving historical traceability through structured weekly snapshots. Finally, in Part 3, I’ll share how we turned raw test results into context-aware release decisions.

Testing activity vs. QA infrastructure

There is an important difference between executing tests and building QA infrastructure.

• Testing activity often answers immediate, ticket-level questions: Is this ticket working? Did this bug get fixed? Can this item move to the next column?
• QA infrastructure answers broader operational questions: What workload is approaching from development? What failed UAT and is blocking the sprint? Where are the current risks or bottlenecks for this week's release?

When you are the only QA on a stream, you cannot scale your hours, but you can scale your processes. Without a proper infrastructure, you quickly become a communication bottleneck. You spend more time answering repetitive status questions than actually checking the software.

QA depends heavily on development readiness, clear requirements, and stable deployment windows. If the information surrounding these dependencies is fractured, QA becomes entirely reactive. You discover blockers late, ask the same questions repeatedly, and turn release syncs into repeated clarification sessions instead of evidence-based decisions. Senior QA work often starts in these exact gaps — not because they are dramatic failures, but because they quietly create confusion across planning, testing, and release confidence.

The gaps in the system

The project wasn't lacking effort, but it lacked a unified operational picture. As our biweekly release cadence intensified, a few structural friction points became glaringly obvious:

• Scattered project telemetry. Jira held the workflows, Xray tracked executions, and Confluence housed the documentation. Because these pieces didn't talk to each other cleanly, understanding the actual state of a release required manual, time-consuming investigation.
• Ambiguous status usage. There was lingering confusion regarding when and how the team actually intended to use specific workflow states like "Ready for Smoke Test", "Fixed", and "Removed".
• Board and dashboard misalignment. At one point, we discovered that the team had been working from an older Jira board. Because our automated dashboards relied on specific JQL filters tied to the active board, parts of the dashboard logic no longer reflected the actual workflow.

This wasn’t just a minor administrative cleanup task. If a team relies on dashboards for planning, QA readiness, and release discussions, outdated logic can create misleading confidence. The data no longer matches reality, and trust in the project's visibility begins to erode.

Building dashboards as information assets

To resolve this, I focused on building three dedicated Jira dashboards designed to map directly to our delivery pipeline. The goal was to turn scattered data into a clear visibility layer, ensuring that the team could answer their own high-level questions without needing to run manual query filters or trace ticket history.

1. DEV intake & readiness

This dashboard looks upstream. It shows what is currently in development, what is approaching the QA gate, and what requirements might need deeper analysis before they hit UAT. For a solo QA, upstream visibility is essential; it gives you the lead time to review requirements, identify unclear areas, design test scenarios, and prepare complex test data before the code lands in your environment.

2. QA intake & upcoming work

This view organizes the immediate horizon, helping shape the short-term QA plan. In an order management system, ticket counts are deceptive. A small number of tickets affecting core payments, invoicing, fulfillment, stock, or exception handling requires significantly more analysis than a long list of minor UI adjustments. This dashboard made that upcoming weight visible early for sprint balancing.

3. QA execution (UAT → PROD)

The final layer serves as a live release status checkpoint. It displays exactly what is currently sitting in UAT, what has passed, what has failed, what needs retesting, and what is moving closer to production readiness. It gave the team an accurate, real-time view of how the sprint was coming along and provided a clear data baseline to flag risks before they jeopardized the biweekly release.

Driving workflow clarity from the board level

Building dashboards is useless if the underlying board data is flawed. I initiated direct conversations with the team to explicitly clarify status usage, board ownership, and process expectations. We had to ensure the process supported the actual work instead of silently working around the gaps.

As part of this alignment, we tackled a major visibility issue on the Jira board: when work failed UAT, it had a tendency to blend back into the backlog or get buried in ticket comments and status history. To fix this, we introduced clear board-level visibility for a "UAT Test Failed" state.

The operational impact

This workflow improvement significantly reduced the need for repeated status checks, because the developers could immediately see which items required bugfixing or retesting. One look at the board gave the team instant clarity on what was blocked, what was ready for a turnaround, and what could move toward release.

To ensure our metrics remained reliable, I built a centralized filter-mapping spreadsheet to document the JQL behind every gadget on our dashboards. This allowed us to systematically identify which filters referenced old tracking structures, update them to point to our single active board, and safely transition the team without losing historical data integrity.

Sanitized snippet of the centralized JQL filter structure

Setting the stage for traceability

The final realization from this initial infrastructure phase was that live operational visibility and historical traceability serve two different reporting needs.

The dashboards helped solve the real-time visibility problem. They gave the team a clearer view of what was happening in the moment: what was approaching QA, what was currently in UAT, what had failed, what needed retesting, and what was moving toward production.

But a live dashboard is fluid. It changes every time a ticket moves, a bug gets fixed, or a filter is updated. It cannot preserve the specific context of what happened during a defined reporting period.

Live metrics alone also do not fully capture the engineering groundwork behind QA delivery: upstream analysis, technical clarification, test case maintenance, process refinement, and preparation work that does not always appear as a simple pass/fail result.

We had established clearer real-time visibility, but we still needed a reliable historical record. That gap became the catalyst for the next phase of the infrastructure: shifting from active tracking to structured weekly traceability.

I’ll cover how we built that historical layer in the next part of the series.

Lessons learned

• Clarify gaps, don't work around them. It is tempting for a solo tester to silently bypass an ambiguous workflow status just to keep testing. A senior engineer stops, flags the gap, and aligns the team so the process reflects reality.
• Make failures actionable. A highly functional project board shouldn't just track upward progress; it must clearly highlight failure so the team can self-heal and resolve blockers without constant alignment meetings.
• Dashboards are assets, not widgets. Automated charts are only as accurate as the query filters underneath them. Caring for your JQL logic is a fundamental part of maintaining project visibility.

Key takeaways

When you are the only QA engineer on a development stream, your ultimate value isn't measured solely by how many hours you spend executing test scripts manually. It’s measured by the systems you put in place to keep the project moving safely and predictably. By cleaning up our board visibility, mapping out our dashboard logic, and defining workflow expectations, we turned a scattered testing effort into a clear, reliable system.

But while dashboards give you clearer real-time visibility for the "now," they do not preserve history. In part two of the series, I will explore how we took this foundational visibility and established a bounded weekly reporting layer to ensure long-term historical traceability across the project.