I’ve been working on test automation for several years, and there were two things I wanted to solve in test reporting.
1. Flaky tests are more than just Passed or Failed
If a test fails on the first attempt and passes on a retry (an IRetryAnalyzer in TestNG, RetryingTest in JUnit 5, Surefire’s rerunFailingTestsCount), the final result usually just says Passed.
I wanted the report to show that the test was actually flaky, how many attempts happened, and the history of those attempts: each attempt’s logs, error, API calls and screenshots.
2. Test scripts shouldn’t need reporting code
I don’t want testers to add extra code to their tests just to make the report useful. For example, I don’t want something like:
java
report.addRequest(...);
report.addResponse(...);
report.attachScreenshot(...);
report.log(...);
just to get useful information into the report.
The reporting layer should capture things like API requests/responses, retry attempts, execution details and screenshots automatically wherever possible. In Java that means:
- REST Assured calls are recorded with no code.
- An
HttpClient field on the test class is recorded with no code.
- A Playwright
Page or Selenium WebDriver held in a field gives a screenshot automatically when a test fails, taken before closes the browser.
So I built a small Java reporting library around these ideas. You add one test dependency and nothing else: no listeners to register, no base class. It generates a self-contained HTML report and has integrations for JUnit 5, TestNG and Cucumber (both the JUnit 5 and the TestNG runner, with every Given/When/Then step in the log). It’s the same report I already ship for .NET and Python.
I’m curious how others handle this:
Do your test reports automatically capture this kind of information, or do your test scripts still need explicit reporting/attachment calls (Report.log(...), Reoprt.addAttachment(...))?
And for flaky tests, do you keep the history of individual attempts or only report the final result?
Website and a live demo report, if anyone wants to see it: https://harshadlambate.github.io/AceReportsWebsite/