NNaatOps
Start free

Test cases and test runs

Write and import test cases, group them in suites, run them and file bugs from failures.

The Tests tab of each project holds its test cases and test runs. Cases are linked to the stories and bugs they prove, so coverage and results show on the issue, on the release and in the client report.

Test casesTest cases

Write a test case

  1. Choose New test case.
  2. Fill in:
    • Title: what is checked, such as Reject an expired card.
    • Suite: the group it belongs to, or No suite.
    • Preconditions: what must be true first, such as Signed in with a saved card.
    • Steps: for each step, the action and the expected result.
    • Priority and status (Draft, Ready or Deprecated).
    • Covers: the stories or bugs this case proves.
    • Automated: tick it if a script already runs this check.
  3. Save.

You can also write a case from a story: open the story and choose + Test case in its Quality section.

Suites

The list on the left groups cases into suites such as Login and security or UPI payments. Type a name under the list and choose + to add one. Deleting a suite keeps its cases; they move to No suite.

Import from Excel or CSV

  1. Choose Import.
  2. Drop an .xlsx or .csv file, up to 5,000 rows. Exports from TestRail, Zephyr and Xray work.
  3. Check which column fills each field: title (required), steps, expected result, preconditions, suite, priority, status, covers, automated, test data and original ID. Steps can be in one cell per case (numbered) or one row per step joined by the case ID.
  4. Look at the first cases in the preview, then import.

Story keys in the Covers column, such as MOB-12, are linked to those stories. Old .xls files can't be read: open them in Excel and save as .xlsx or CSV first.

Run tests

  1. Choose New test run. To run only some cases, tick them first; the button becomes Run N selected.
  2. Name the run, and choose the sprint, the environment (staging, iOS 18…), the release and which cases: every ready case or one suite.
  3. Choose Start run.

Starting a test runStarting a test run

Record results

The run shows the cases on the left and the selected case on the right, with its preconditions and steps.

  • Mark Passed, Failed, Blocked or Skipped. Keys: P, F, B, S. Move with J (next) and K (previous).
  • Write the actual result, device and data in What happened. It saves when you leave the field.
  • Filter the list to To run or Failed or blocked.
  • The bar at the top shows how many cases have run and the pass rate.
  • When everything is done, choose Complete run.

Recording resultsRecording results

File a bug from a failure

After marking a case Failed, choose File a bug. NaatOps creates a bug in the same sprint, with the case, its steps and your notes, linked back to the case. The bug shows in My work › Bugs and on the board, and the bug's Quality section says which case and run found it.

Send automated test results from CI

Your automated tests can report into NaatOps after every build, next to the manual runs.

  1. Make an API token in Settings › My account and save it in your CI as a secret called NAATOPS_TOKEN.
  2. Have your tests write a JUnit XML report. JUnit, pytest, Jest (with jest-junit), Playwright, Cypress, Go, PHPUnit and .NET can all do this.
  3. On Tests › Test runs, choose Send results from CI. Copy the step for GitHub Actions, GitLab CI or Any CI (a curl command) and add it after your tests, so results arrive even when tests fail.

Setting up CI resultsSetting up CI results

Each upload becomes a completed run marked From CI, with a View build link back to the pipeline.

Runs sent from CI in the listRuns sent from CI in the list

  • Matching: a test counts against a test case when its name contains the case key, such as SBM-T12, or matches the case title.
  • New tests: tests that match nothing become automated test cases, grouped by their test suite. Add &createMissing=false to the address to record only the cases you already have.
  • Failures: the failure message goes into the result's What happened notes, ready for File a bug.
  • Optional details you can add to the address: name (the run name), environment, releaseId, sprintId and buildUrl.

CI results count toward coverage and the traceability matrix like any other run.

See coverage

  • On a story, Quality lists its test cases with their last results, and API checks.
  • In the case list, Covers and Last result show the same from the other side.
  • On a release, Tests passed counts results from runs linked to that release.

Traceability matrix

Open Tests › Traceability for the whole picture in one table: each story, task and request, the test cases and API checks linked to it with their latest results, and any bugs those tests found. Clients and auditors often ask for this before UAT sign-off.

Each requirement is marked:

  • Passing: everything linked to it passed last time.
  • Failing: a linked test or check failed, or a bug it found is still open.
  • Not run yet: it has tests, but none has been run.
  • Not covered: nothing is linked to it yet. Add a test case that covers it.

The cards at the top show how much is covered, passing, failing and not covered. Narrow the table by Release, Sprint, Type, Coverage or a key or title. Epics appear only when you pick Epics as the type.

The traceability matrixThe traceability matrix

  • Download Excel saves the rows you're looking at as an .xlsx file.
  • PDF opens a printable A4 landscape page. Choose Print or save as PDF to send it to the client.

Plan limits

The Free plan holds 100 test cases and Team holds 1,000. Delivery and the Firm Pack have no limit.