Maximizing value: How companies and pentesters can achieve more together

March 30, 2025 17 mins

Maximizing value: How companies and pentesters can achieve more together

A closer look at the collaborative efforts between Aon and Fullstory reveals the strategies they employ to maximize success on pentesting engagements.

In this joint reflection, Aon and Fullstory share key takeaways on how security consultancies and their clients can get the most out of pentesting assessments. This article presents a behind-the-scenes look at the entire process, featuring an interview-style format that highlights the perspectives of both parties following a successful engagement.

Can you walk us through what specific criteria Fullstory prioritizes when short-listing and selecting a pentesting vendor?

Fullstory: There are a number of criteria that Fullstory requires when selecting our penetration testing services partner to ensure great results that will be trusted by our customers:

The consulting firm must conduct assessments with a combination of source code review, static, and dynamic testing approaches. We often hear about firms that automate all testing, which highlights a lack of security expertise and maturity.
Engagements performed for Fullstory must have resources assigned that are familiar with in-scope programming languages and technologies, as many consulting firms will assign any available resource even without needed skills.
We prioritize consulting companies that tend to have full-time employees on staff, rather than sub-contracting our work out to save money—this ensures higher quality control of the work and a longer-term professional relationship with us.
Because we want our customers to gain the most value possible from any engagement, we focus on vendors that will provide a "customer-facing deliverable" (CFD) that can appropriately communicate the scope and findings.
Many of our global customers require that we leverage CREST-accredited penetration testing vendors, which helps speak to overall assessment rigor.

Defining success criteria is crucial for both parties involved. How do you collaborate to establish clear objectives and metrics for success at the onset of an engagement?

Fullstory: Expectation setting is crucial in large-scope penetration testing purchases as the amount of time it takes to source a vendor, define a scope, schedule appropriate resources, conduct the work, and receive a final report can easily take four months, end-to-end. To avoid any issues within this commitment, we ensure that our vendor understands the driving force behind the work (e.g., customer-facing report), any explicit deadlines we have to hit, the technologies/programming languages in scope we need coverage on, and will provide up-front walkthroughs as necessary of our architecture and product. As we communicate these details, we check in at each phase of scoping and drafting of the Statement of Work (SoW) to avoid any misunderstandings that could hurt our success.

We will even share previous examples of reports, contracts, or other documentation that gives them a clearer picture of what we want, or what we do not want to see from them. Every vendor has a different way of working—as does every customer—so spending a couple hours of time discussing the project in advance of a contract is absolutely critical.

Pentesting engagements are typically time-boxed, making time management a critical component to the overall success of the assessment. What is the best way to maximize success given the limited time available for a pentest?

Aon: Every organization that has had to conduct a penetration test knows that at the crossroads of scale and complexity is a budget that is fractionally sufficient to adequately meet all security requirements. While time-boxing serves as a pragmatic, budget-conscious approach to achieve breadth and depth, it is hardly a panacea. Pentesters typically have days to weeks to complete an assessment, while threat actors theoretically have months to years—and that's just to plan! Organizations with mature security teams understand this dichotomy, and seek to maximize the utility of their budgets not only through time, but also through passive and active support of their pentesting teams. In Aon's experience, attempts at efficiency gains and support have come in many forms, including but not limited to:

Access to full source code. On heavily time-boxed application assessments, pentesters typically don't have sufficient time to reverse-engineer how functionality was implemented or infer whether security controls were ineffective or absent. Source code helps to short-circuit that process, while also expanding pentesting coverage through a combination of dynamic and static analysis.
Run local builds. Collaborating with development teams to run local builds allows pentesters to debug application functionality and can aid in crafting targeted proof-of-concept exploits.
Kickoff calls. Live client demos coupled with Q&A sessions provide an opportunity for penetration testers to perform a "mini threat model" by asking developers pertinent questions relating to: application business logic, security controls, newly deployed functionality, functionality that is highest risk / historically least reviewed / should be prioritized, etc.
Group chats. Whether your organization uses Slack, Teams, etc., having a place to collaborate in real-time, bi-directionally ask questions, communicate interesting observations, report vulnerabilities, contemplate remediation strategies (while keeping sensitive information secure) is key. In this case, Fullstory's development and security teams and Aon's pentesters openly collaborated to determine exploitability, perform root cause analysis, and brainstorm possible remediation strategies.
Documentation. Network diagrams, architecture diagrams, API documentation, and previous penetration test reports all help to provide a foundational understanding of the current state security, allowing pentesting teams to build off of the existing knowledge for the upcoming assessment. Less time spent uncovering knowledge that is already known means more productive time testing. In particular, Fullstory provided Aon with in-depth documentation, including a spreadsheet with a list of all microservices and a Neo4j database with authorization definitions.

Fullstory: A perplexing, antiquated belief is that somehow an opaque "black-box" assessment is more valuable to request because it mimics how so-called "real" attackers would go after your product or services. For us, the goal is to find any and all material security issues in our product and services as efficiently as possible so that we can reduce potential harm to our customers and business. That means it's in our interest to give as much code, documentation, and support to the talented penetration testers we hired to find as many potential issues as possible. High-quality work costs a non-trivial amount of money, so we get more ROI this way.

Communication is key during the course of any engagement. How do you structure communication channels and processes to ensure transparency, responsiveness, and clarity throughout the engagement

Aon: Active communication prior to and during a pentesting engagement is a necessary, mutual responsibility for both pentesting vendors and their customers. Oftentimes, when things go awry on an assessment it is due to a lack of communication on one or both sides. At Aon, we use, among others, a combination of email, secure file transfer, and chat (e.g., Slack, Teams) to securely communicate with clients and their security teams, with the latter having the added benefit of real-time collaboration. This has had several desirable benefits:

Scope / Timeline Clarity. Assessments are dynamic in nature, and sometimes the scope expands and contracts as the assessment progresses. Frequent touch points along the way allow both sides to re-calibrate on the scope and delivery timelines.
Verification / Troubleshooting. Prior to the start of an assessment, we request and confirm all prerequisites have been provided and confirm everything is working as expected. Any issues are documented and communicated to the client after reasonable troubleshooting, allowing the client enough time to investigate any issues while the pentesting team starts covering working functionality.
Critical/High Vulnerability Reporting. If you're an organization, there is nothing worse than getting to the end of an assessment and having your pentest vendor surprise you with several critical / high severity vulnerabilities. At Aon, we typically communicate critical / high severity vulnerabilities along the way. We also use private chat to communicate with internal client teams giving them an opportunity to triage, ask questions, and begin remediation as quickly as possible.
Regular Exchange of Information. Rabbit holes are every pentester's worst nightmare. A healthy exchange of observations, questions, etc., can help a client guide their pentesting vendors to possible avenues worth investigating, as well as help to avoid ones that may lead down the dreaded rabbit hole. For example, during the Fullstory assessment, Aon encountered a vulnerability that may have only been exploitable in a particular circumstance, but further investigation would not have been an efficient use of time without the appropriate context. In the end, communication with Fullstory allowed Aon to quickly understand the full risk profile, and subsequently move on to other portions of the assessment.
Notify, Pivot, Circle Back. Good pentesting teams don't simply stop testing once they've run into an obstacle. After troubleshooting, we communicate the issue to the client and subsequently shift priorities to other functionality that can be tested. We can circle back to completing the task once the issue is resolved.

Fullstory: In our experience with Aon, they took great care to identify blockers early in our engagement that would have prevented them from providing the best work possible. Notably, we heard from them early on to determine the cause of issues they had deploying our development environment locally, collaborating in real-time with our engineers to find solutions that would further unlock the depth they could provide on dynamic testing and validation of findings. With past consultancies, these sorts of issues can linger silently and only be brought up at a mid-point check-in, wasting consultant time and focus that could have provided a better outcome for us. This proactive approach to the engagement fostered more alignment and trust early on.

Some organizations have adversarial relationships between development and security teams. How does your team approach building and maintaining a collaborative relationship with security consulting, security engineering, development, and DevOps teams throughout the engagement?

Fullstory: At Fullstory, we consider our engineers our customers, in a similar manner to a consulting firm where we provide a combination of advisory services and technology platforms. We prioritize taking on the "pain" of security first, triaging vulnerability classes and finding high-leverage spots to resolve issues, rather than expecting them to whack-a-mole the problems in a vacuum. This approach means that sometimes our engineers never see "what we are doing," but that's fine because it's vastly more efficient and less disruptive if we can do our best to avoid them ever feeling that pain in their day-to-day workflow.

It's also crucial that you understand how your engineers actually work so that when you need to get their attention, you can do so in the most optimal place possible. A lot of security teams will use their native technologies to track work—and then force engineers into those systems. We always attempt to move that work into an expected place, such as a Jira task or GitHub comment, so that the engineers’ flow is more seamless with security.

With Aon's assessment of services, they always showed our engineers the most respect possible, which is critical to avoid having anyone feel targeted for a vulnerability finding. Unfortunately, not all consultancies bring this sort of empathy and maturity to the table, which can cause friction between engineers, the security team, and the engagement consultants. When people feel bad about their work, they are more likely to shut down conversations and partnership. Aon always brought the best spirit forward in communications with all team members, staying objective, considerate, and detailed to help bridge the gap between a finding and our future resolution.

Aon: Aon encounters a large number of clients who have contentious relationships between security and development teams, with our pentesting team frequently positioned right in the middle. In these organizations, development teams treat security (and particularly pentesting) as an obstacle to feature releases, bug fixes, and productivity. Not surprisingly, these teams are slower to respond to pentesting inquiries or may be unwilling to help. In these organizations, we also tend to see a high recurrence of the same class of vulnerabilities year-over-year—a sign that the client is only fixing reported vulnerabilities and not addressing the root cause that led to the vulnerability in the first place.

In contrast, Fullstory's security culture felt very inclusive and collaborative. Starting with the development team, the code is filled with detailed comments of security considerations they've taken at every step, indicating security is a key part of the development lifecycle. Further, as Aon discovered issues in the application, Fullstory's security team modified their static code analysis tool to detect additional instances of this and other related classes of vulnerabilities. There were also several in-depth discussions between Fullstory's development teams, security teams, and Aon around how we discovered certain risks, our opinion on severity, and options for remediation; conversations that are indicative of a mature security posture.

In rare cases and due to a number of reasons (e.g., limited scope, limited time, etc.), assessments may not have material findings. What measures can be taken to ensure value is still derived from the assessment?

Aon: When there are a limited number of findings on an assessment, the first step pentesting vendors should take is to perform extra diligence by re-reviewing their vulnerability and best practice checklists (e.g., checks derived from OWASP ASVS). Next, it is important to threat model the application for business logic-specific vulnerabilities that may not be typically found by application / network scanners, static analysis tools, manual shallow code review, etc. This step should involve conversations with the client teams to collaborate and permute on test cases that may not have been covered in a previous pentest.

It is possible that even after executing the steps above, the results may still be underwhelming. However, it is important to note that a limited finding assessment does not necessarily mean the assessment yields limited value. Demonstrating to the internal teams the various test cases that were performed, as well as detailed observations and results for each test case, can help them understand the pentesting approach, the overall level of coverage, how well their infrastructure / application stood up to a variety of attacks, and may even lead them to investigate other soft spots where the attacks would have otherwise been successful given more time. As a result, when there are a limited number of findings, thoroughly documenting test cases in an appendix can be immensely valuable for clients, while also helping to substantiate due diligence on the part of the pentesting vendor.

Closeout calls mark the end of an engagement and provide an opportunity for the pentesting team to present their findings and for both teams to discuss the risk and align on remediation. What steps do you take to ensure a smooth and informative closeout call, and how do you facilitate productive discussions about assessment findings and recommendations? How do you handle findings from the pentesting team that do not present an immediate risk?

Aon: Closeout calls allow for pentesting companies and clients to align on findings and proposed remediations. At Aon, our primary focus at this stage of the process is making an effort to ensure the client understands the technical and business risk associated with the vulnerability, how we discovered the vulnerability (and thus how the client can reproduce it), evidence that we found the vulnerability (e.g., via screenshots, code snippets, etc.), and perhaps most importantly, how should the client fix it.

These calls should result in healthy discussions around risk, which may differ when viewed through the lens of both parties. If, after several rounds of discussion, both teams settle on a risk, updates to the report are issued where necessary. In cases where the two teams don't necessarily see eye-to-eye on the risk, the report can typically be updated with a footnote from the client that captures their perspective but with all other details remaining the same. This approach allows us to remain an independent third-party pentesting firm, ensuring our vulnerability and risk descriptions are presented objectively while acknowledging the client's perspective.

Fullstory: In receiving results from a consultancy partner, many security teams may spend an inordinate amount of time "defending" themselves or trying to squash every finding from being seen on a report by their customers. There is great value in alignment on those findings, especially if what was provided was not the scope originally requested, as some firms may do when they run out of primary attack surface.

In a relationship with a team like Aon, our goal is to ensure we understand the entire context of the finding foremost, as it can be easy to assume you get the complete risk after a quick read. This sometimes leads to additional validation testing and calls to have both teams fully aligned on the potential for harm and likelihood of abuse. In cases where there may be potential harm to Fullstory's revenue, for instance, we may decide that the likelihood of that is low enough that it's prohibitive to focus on fixing for now. That can be easily communicated to our customers as not posing any risk to them.

Aon spent extra time fielding our questions, thoughts, threat model details, proof-of-concept requests, and other inputs so that by the time our report draft was finalized, both Fullstory and Aon felt like we had reached the most accurate version of what was assessed. Their willingness to consider our perspectives allowed us to feel heard and respected while still ensuring that conclusions are unbiased and fact-based, developed through observations gathered. From scoping to delivery, Aon was in lockstep with our security team on framing risk in a complete manner, allowing for our customers to have an accurate summary of our application security, while also ensuring that our company had the utmost clarity on findings in context of our business operation and profile.

Building — Support

Contact our cyber response teams for urgent breach assistance.

Contact Us

In summary, this article presents the viewpoints of Aon and Fullstory, two companies collaborating on a pentesting engagement from different perspectives—one as the pentesting provider and the other as the client. The success of this collaboration was driven by a structured approach that included clearly defined objectives, thorough planning, regular communication, active collaboration, and a joint commitment to tackling challenges together throughout the assessment. These elements combined to create a productive partnership that highlights the importance of alignment and teamwork in achieving effective security outcomes.

Explore More

Capability Overview

Cyber Resilience
Product / Service

Penetration Testing Services

Authors

Elijah Seymour

Technical Director, Security Testing, Cyber Solutions
Mark Stanislav

Fullstory
Rohit Kapur

Director, Security Testing, Cyber Solutions
Stephen Komal

Managing Director, Security Testing Practice Leader, Cyber Solutions

Cyber security services are offered by Stroz Friedberg Inc., its subsidiaries and affiliates. Stroz Friedberg is part of Aon’s Cyber Solutions which offers holistic cyber risk management, unsurpassed investigative skills, and proprietary technologies to help clients uncover and quantify cyber risks, protect critical assets, and recover from cyber incidents.

Fullstory is on a mission to help technology leaders make better, more informed decisions by injecting behavioral data into their analytics stack. The company’s patented technology unlocks the power of quality behavioral data at scale by transforming every digital visit into actionable data and insights. With Fullstory, enterprises can get closer to their customers’ true sentiment and intentions to predict what they want, create personalized experiences, and drive conversion, loyalty, and revenue.

This material has been prepared for informational purposes only and should not be relied on for any other purpose. You should consult with your own professional advisors or IT specialists before implementing any recommendation, following any of the steps or guidance provided herein. Although we endeavor to provide accurate and timely information and use sources that we consider reliable, there can be no guarantee that such information is accurate as of the date it is received or that it will continue to be accurate in the future.

The contents herein may not be reproduced, reused, reprinted or redistributed without the expressed written consent of Aon, unless otherwise authorized by Aon. To use information contained herein, please write to our team.