The penetration test report is the document in the compliance file most often ticked “present” and least often questioned. A test with a wrongly drawn scope, unwritten rules or findings never retested is weak evidence in front of an auditor, and it doesn’t measure the real risk either. The report sits on the shelf. Below we look, through the eyes of the technical team, at choosing the test type, the scope and rules document, operational safety during the test, how to read report quality, and verification after remediation.
The difference between vulnerability scanning and penetration testing
Vulnerability scanning looks for known vulnerability signatures across a broad set of assets with automated tools and is repeated continuously. A penetration test, on the other hand, is manual work within a defined scope and a limited time, in which an expert tries to show the real impact by linking findings together. Scanning says, “this server has a missing patch.” A penetration test puts on the table, “this missing patch, combined with a weak service account password, lets you reach domain admin rights.”
Neither replaces the other. If most findings in a penetration test report could have been found by scanning, either scanning isn’t done regularly or the test didn’t go deep. For the continuous scanning and prioritization cycle, see the vulnerability management program guide; the real value of a penetration test lies where automated tools can’t see: privilege escalation, business logic and chained attack paths.
Test types: external, internal, web, social engineering
Compliance requirements usually say “regular security testing” without naming the type. The asset you want to protect and the threat scenario determine the right type:
| Test type | The question it answers | Typical scope |
|---|---|---|
| External network test | Can someone get in from the internet? | Public IPs, VPN, email, published services |
| Internal network test | How far does an attacker inside get? | From the user VLAN to AD, servers, segments |
| Web application test | Are the application logic and authorization sound? | Authentication, sessions, authorization, input handling, API |
| Social engineering | Do the human-layer controls work? | Phishing, phone, physical entry attempts |
| Wireless and cloud configuration | Do wireless network and cloud settings create leaks? | SSIDs, guest network separation, cloud identity and storage settings |
The level of information is a separate decision. In a black box test, the test team is given only the target addresses; it’s realistic but may stay on the surface in a limited time. In a gray box test, a standard user account and basic architecture information are provided; for compliance testing, it’s usually the most efficient option in terms of the balance between time and depth. A white box test includes access to source code or configuration. In an internal network test, starting with a “compromised user” scenario, that is, a standard domain account and a corporate laptop, imitates a common attack starting point. So that the test team doesn’t spend its first hours on gaps you already know about, close the known weaknesses on the AD side in advance; the Active Directory security controls guide was written for this.
Preparing the scope and rules document
The rules of engagement document is signed by both sides before the test starts and draws the legal and technical boundaries of the test. An incomplete document leads to the test being cut short or to outages in production. It should include at least:
- Scope: IP ranges, domain names, application URLs, API endpoints and the user roles to be tested. Verify that the in-scope assets really belong to the organization; an IP block given up years ago may now belong to someone else.
- Out of scope: Excluded systems and the reason. Leaving a critical system out of scope is a risk decision and should appear in the report.
- Third-party permissions: For systems running in the cloud or at a hosting company, check the provider’s testing policy; permission may be needed on shared infrastructure.
- Permitted and prohibited techniques: Denial of service, deleting data, leaving persistence mechanisms, physical entry and social engineering are written out explicitly.
- Time windows and test source IPs: At what times and from which source addresses testing will be done; that’s the only way the SOC can tell test traffic from a real attack.
- Communication and emergency stop: People reachable 24/7 on both sides, the notification method for critical findings and the procedure for stopping the test.
- Data handling: Not copying personal or confidential data accessed during the test, and how captured password hashes will be stored and destroyed.
In Türkiye, some sector regulations may require the test to be carried out by a company or staff holding specific competence certificates. Verify your regulator’s current requirement before signing the contract.
Operational safety during the test
The test also puts the organization’s own security controls to the test. That’s why preparation goes both ways:
- Critical in-scope systems should have a current backup whose restore has been tested.
- The account lockout policy should be known; password attempts should be kept below the lockout threshold. Otherwise the help desk is flooded with locked-account calls in the morning.
- Will the SOC be told about the test, or will detection capability be measured too? In the second case, only a limited number of managers are informed and a code word is set to confirm “test or real?”
- The accounts the test team uses and the objects it creates (files, users, scheduled tasks) are recorded; at the end of the test, it’s verified that they have been cleaned up.
- Test traffic should leave a trace in the central logs. Looking at which alarms were raised during the test is a way to measure your detection coverage at no extra cost.
Assessing report quality
The report is the test’s only lasting output; the value of the compliance evidence depends directly on the report’s quality. Read the report you receive with these questions:
- Does the executive summary explain the most important risk chains and business impact without drowning in technical detail?
- Are the scope, test dates, source IPs, test type and information level written in the report?
- Is the methodology stated? For example, a reference to a framework such as the OWASP Web Security Testing Guide for web applications, or NIST SP 800-115 or PTES for the general testing process.
- For each finding, are there the affected asset, reproduction steps, request/response or screen evidence, a severity rating and the rationale for the rating (for example a CVSS vector)?
- Were the findings verified manually, or was scanner output copied? Unfiltered false positives and raw scanner output are signs of low quality.
- Are attack chains described separately from the individual findings?
- Are the remediation recommendations actionable and specific to the environment? Recommendations at the level of “apply the patches” are useless.
- Are the areas that couldn’t be tested and the reason (time, access, scope) stated?
A practical test: pick a random finding from the report and have your own team follow the reproduction steps. If the steps don’t hold up, neither does the report.
Remediation tracking and retesting
From a compliance standpoint, the step most often missing is verifying the remediation. Every finding in the report should be moved into the vulnerability management system or ticketing system with its owner and target date. When remediation is complete, the finding is closed with a retest. The retest can be a limited verification by the test company; for simple findings, the internal team can also repeat it and record it:
# Verifying from the user VLAN that the SMB and RDP access reported as closed is closed
nmap -Pn -p 445,3389 10.10.20.0/24 -oA retest-smb-rdp-2026-11
# Web finding: checking that the HSTS header has been added
curl -sI https://app.example.com | grep -i strict-transport-security
# Checking that the old TLS version mentioned in the report has been disabled
nmap -p 443 --script ssl-enum-ciphers app.example.com
The retest should use the same method as the finding’s original evidence. A “clean” result obtained with a different method may not show that the finding is really closed; for example, a port closed on the server side may still be reachable from another VLAN. After findings related to internet-facing assets, the external attack surface management guide is a complement for continuously monitoring the external surface. For findings that can’t be closed, a time-limited exception and a compensating control record are opened.
Technical records to keep for compliance
Collect the evidence that will be asked for in the next audit before the test ends. Putting the report in the archive isn’t enough:
| Record | What it proves |
|---|---|
| The signed scope and rules document | That the test was planned and authorized |
| The final report and version information | That the test was done, and the findings |
| A finding–ticket mapping table | That every finding has an owner and a target date |
| The retest report or internal verification outputs | The effectiveness of the remediation |
| Exception records | Conscious acceptance of risk that can’t be closed |
| SOC alarms raised during the test | That detection capability was measured |
Keeping these records together with the test report produces direct evidence for ISO/IEC 27001:2022 Annex A 8.8 (Management of technical vulnerabilities) and for NIS2’s area on assessing the effectiveness of measures. The retention period of logs generated during the test also affects their evidential value; this topic is covered in the central log retention architecture guide. If you need to plan scoping, testing and retesting as a single process, take a look at our penetration testing and security validation service.
Frequently asked questions
How often should a penetration test be done?
The frequency depends on the compliance framework, contracts and the pace of change. Common practice is to test at least once a year and after significant infrastructure or application changes. For example, PCI DSS v4.0 requires internal and external penetration testing at least once a year and after significant changes.
Should you choose a black box or a gray box test?
A black box test realistically imitates an external attacker but may stay on the surface in a limited time. In a gray box test, a standard user account and basic architecture information are provided; it produces deeper findings in the same time and is usually more efficient for compliance testing.
How do I know a penetration test report is of good quality?
The scope, dates and methodology are written down; every finding is backed by reproduction steps and evidence; the findings have been verified manually; attack chains are described separately; and the recommendations are specific to the environment. Reports dominated by raw scanner output are low quality.
Is a retest mandatory?
A retest is the strongest way to show in an audit that findings have been closed. A limited verification by the test company, or a check by the internal team using the same method as the original evidence and recorded, is accepted; closing the ticket alone isn’t enough.