Why are we here? Let’s get the mission clear

• What's the one outcome you'd be disappointed if we didn't address in the next 90 days? Options: Reduce false positives, Show developer adoption, Prove low build impact, Surface exploitable vulnerabilities, Meet audit/compliance ask, Other
• Who is the executive sponsor we should align to, and who in engineering will be our day-to-day counterpart?
• What triggered this evaluation—select all that apply and tell us the specific incident or mandate that prompted action Options: Log4j-style incident, Customer or third-party audit, Production breach tied to dependency, CISO board mandate, Technical debt/legacy tool fatigue, Other
• How many active repositories and teams would you realistically want us to consider for an initial evaluation? Options: 1–3, 4–10, 11–50, 51–200, 200+
• Briefly describe a recent example (repo and short context) where you felt blind to a vulnerability or dependency risk

Are we really seeing everything that matters?

• If you had to guess right now: what percentage of your repositories are covered by your current scanning and inventory tools? Options: <25%, 25–50%, 51–75%, 76–95%, Nearly 100%
• How often do you find gaps between inventory reports and what teams actually have in source control or CI? Options: Very often, Sometimes, Rarely, Never, Not sure
• Which of these sources do you currently scan (select all that apply)? Options: Source code (SAST), Open-source dependencies (SBOM/Dedup), Container images, Infrastructure-as-code, IDE/Developer workstations, CI/CD pipelines
• Tell us about a time when a scan missed something critical, or conversely produced noise that masked urgent findings—what happened and what was the impact?
• Who on your team currently owns inventory accuracy and how do they verify it? Options: AppSec, Platform/SRE, Developer teams, Security operations, Other

When developers stop listening, security stops working—what's their experience today?

• How would you describe your developers’ current attitude toward security scan findings? Options: Trust and act on them, Skeptical but sometimes act, Ignore most findings, We don’t know / no feedback loop
• Estimate the average number of findings a developer sees per pull request from your current tools (rough number is fine) Options: 0–5, 6–20, 21–50, 51–200, 200+
• How much extra build time (CI minutes) would you consider acceptable to keep—per pipeline run—if findings were reliably actionable? Options: <30s, 30s–2min, 2–5min, 5–10min, 10+min
• Share an example where a developer asked for clearer fix guidance—what information were they missing?
• Which developer-facing channels do you use today to surface findings (select all that apply)? Options: PR comments, IDE annotations/plugins, Slack/Teams alerts, Email, Dashboards, Ticketing system
• How do you currently measure developer engagement with security findings? Options: Fix rate within X days, Comments/responses on PRs, Time-to-fix metric, Manual audits, We don’t measure it

What would ‘trusted, low-noise security’ actually look and feel like here?

• If we said your appetite was to land at <5% false positives in pilot repos, how realistic does that feel and what would it take to get there? Options: Very realistic, Possible with tuning, Unlikely, Impossible / not a priority
• Which three success signals matter most for you in a 30-day pilot (pick top 3)? Options: Developer engagement rate, False-positive rate, Fix rate (closed vulnerabilities), Build-time impact, Number of high-severity exploitable findings, Time-to-fix
• What specific targets would make you declare the pilot a success (please include numeric or percent goals where possible)?
• Describe the ideal developer experience for triaging and fixing a finding in one sentence
• How would you prefer we present evidence of quality during the pilot—live demo in a repo, weekly dashboards, raw data exports, or recorded sessions? Options: Live walkthroughs in repo, Weekly dashboards, Raw CSV/JSON exports, Recorded sessions with commentary, Combination

What’s standing between a quick win and another stalled pilot?

• What roadblocks have derailed pilots or tool evaluations in the past? Options: Lack of CI access, Integration complexity, Too much noise, Stakeholder turnover, Procurement delays, Other
• Do you have constraints around data sharing, telemetry, or external access we need to respect for the pilot? Options: Strict (no external access), Controlled (approved accounts only), Open with NDA, No constraints / flexible
• How comfortable are your teams with granting short-term service accounts and repo access for a 30-day evaluation? Options: Very comfortable, Some skepticism, Unlikely without exec approval, Not possible
• If we need to suppress noise during initial tuning, who should own the suppression rules and ongoing tuning? Options: AppSec, Platform/SRE, Dev team leads, Shared responsibility
• Tell us about a recent security change that caused friction—what was the organizational reaction and how was it resolved?

Who truly needs to say yes, and how will they judge it?

• Who are the must-have approvers for a pilot (people/roles), and who are the optional stakeholders we should keep in the loop?
• What acceptance criteria will each stakeholder group use to sign off at pilot completion? Options: Technical validation (SRE/Dev), Reduced noise (Dev leads), Risk metrics (CISO/AppSec), Compliance evidence (GRC), Business impact (VPs)
• How will procurement, legal, or security review affect timeline—are there SLA or contractual expectations we should know about? Options: Fast-track (days), Standard review (weeks), Extended (months), Unknown
• What escalation or rollback paths do you require if an integration negatively impacts CI or builds?
• What timeline does leadership expect for a decision after the pilot wraps (immediate, 2–4 weeks, longer)? Options: Immediate, 2–4 weeks, 1–2 months, Undetermined

Designing a pilot we'd both bet on (practical decisions)

• We recommend three active repositories for a 30-day run—which repos would you nominate (please name them and explain why each is a good test case)?
• Which pilot scope would you prefer for each repo: IDE-only, CI-only, full IDE+CI, or incremental? Options: IDE-only, CI-only, Full IDE+CI, Start IDE then CI
• Which success metrics should we track weekly (select up to 5)? Options: Developer engagement rate, False-positive rate, Fix rate within 7 days, Avg time-to-fix, Build time delta, Number of actionable high-risk findings
• Who will be the day-to-day owners for triage, tuning, and communications during the pilot (names/roles)?
• How frequently would you like checkpoints and in what format (live working session, status report, dashboard review)? Options: Twice-weekly working sessions, Weekly review, Bi-weekly, One kickoff + final review, Dashboards only
• What noise-suppression strategy would you prefer early on (auto-suppress certain categories, human review gating, threshold-based, or customized rules)? Options: Auto-suppress categories, Human gating before developer alerts, Threshold-based releases, Custom rule set per repo, Combination

Practical readiness checklist — what do we need before day one?

• Can you grant short-lived service accounts, repository read access, and CI integration tokens for the pilot? Options: Yes — ready now, Yes — requires approval, No — will need alternatives, Unsure
• What CI/CD systems, build platforms, languages, and frameworks should we expect to integrate with (list all that apply)?
• Do you have existing suppression rules, allowlists, or reachability annotations we should import? Options: Yes — we can share, Some — partial, No — none exist, Unsure
• Who will own incident response or remediation during the pilot if we surface an exploitable vulnerability (role/contact)?
• Are there legal, privacy, or regulatory concerns about scanning code or exporting telemetry we must respect? Options: Yes — strict constraints, Some constraints, No concerns, Unsure
• What would a minimally acceptable rollback plan look like if an integration caused build failures?

• Solution Experience Preparation (Prework & Alignment)
• Incident Context Review & Impact Mapping
• Live Code Scenario Walkthrough — Diagnosis, Proof & Validation
• Developer Fix Validation & Experience Feedback
• Noise Tuning, Reachability Calibration & Pilot Acceptance

• Agree on the telemetry and instrumentation required to measure engagement and fix rates during the pilot.
• Seller to prepare scenario playbooks that map each demo step to the specific consequence it eliminates.
• Session Setup & Scope Reminder
• Prove that findings are exploitable (reachability) and that each proof ties directly to the incident or consequence previously described.
• Demonstrate that fix guidance is specific to the customer's code and can be applied by developers within their workflow.
• Obtain explicit validation from engineering owners that the demos represent real problems and acceptable fixes.
• Seller to record each scenario with annotated steps that map every screen/action back to the customer's defined consequence.
• Engineer champions to attempt an immediate confirmatory change in a sandbox branch and report whether the guidance was sufficient.
• Capture and log any false-positive or irrelevant findings observed during the walkthrough for tuning.
• Recap Metrics & Desired Developer Outcomes
• Demonstrate a developer can implement a fix using provided guidance within an acceptable time window.
• Collect concrete developer feedback on clarity of guidance and perceived noise/false positives.
• Introductions & Objectives
• Engineer champions to create the test PRs in the agreed sandbox and link them for metrics capture.
• Seller to enable telemetry hooks (IDE plugin logs, PR annotations, CI check results) for the duration of the pilot.
• Both teams to agree on a short developer feedback survey to run after each fix attempt.
• Present Observed Noise & False-Positive Examples
• A concrete tuning plan is agreed with specific suppression rules and reachability settings.
• Quantitative pilot acceptance criteria and gates are documented and agreed upon.
• Owners for tuning, reporting cadence, and escalation/rollback paths are assigned.
• Seller to implement agreed suppression rules and reachability thresholds in the pilot configuration.
• Customer to confirm pilot start date and provide final authorization for repository/CI access.
• Both teams to schedule the first three checkpoint meetings and define the report template for acceptance metrics.
• Customer provides a one-sentence current state that all participants agree on.
• Consequences of the current state are quantified with at least one metric or example.
• A one-sentence, measurable future-state success statement is agreed for the experience.
• All prework items (repos, access, sample PRs, baseline metrics) are listed and owners assigned.
• Customer to supply three target repositories (names and owners) and one representative problematic PR or commit for each.
• Customer to provide baseline metrics export (current false-positive rate, average build time, current fix rate) for the selected repos.
• Seller to provide a preconfigured sandbox branch and instructions for minimal access needed for live demos.
• Assign engineering and security champions who will attend live walkthroughs and validate outcomes.
• Recap One-Sentence Current State & Future State
• Everyone shares a common, evidence-backed understanding of the incident and affected scope.
• A prioritized list of 2–3 real scenarios to demonstrate during the live walkthrough is agreed.
• Measurement baselines and success metrics for the experience are confirmed.
• Customer to deliver redacted incident artifacts, dependency manifests, and any exploit proof-of-concept (if available).
• Customer to confirm repository owners and add engineer champions to the demo invite.
• Pair-Programming: Engineer Applies Fix #1
• Define Tuning & Suppression Rules
• One-Sentence Current State
• Incident Timeline & Root Cause Evidence
• Scenario 1 — Dependency Vulnerability with Reachability Proof
• Set Pilot Acceptance Criteria & Metrics
• Consequence Quantification
• Scenario 2 — Exploitable Application Code Path
• Review PR Commenting and CI Feedback
• Map Affected Services, Repos & Owners
• Collect Qualitative Feedback
• Scenario 3 — Container/Image or IaC Misconfiguration
• Define Future-State Success Statement
• Translate Incident to Developer Pain Points
• Assign Owners, Escalation & Rollback Paths
• Agree on Instrumentation for Pilot
• Prework & Access Checklist
• Baseline Metrics Confirmation
• Agree Checkpoint Cadence & Reporting
• Validation Checkpoint & Forced Confirmation
• Agree Scenario Priorities for Live Walkthrough
• Schedule & Roles for Live Sessions

• Install IDE security plugin
• Integrate scanner into CI/CD pipelines
• Deploy container image scanning in builds
• Activate infrastructure-as-code scanning
• Enable dependency SBOM generation and monitoring
• Configure reachability analysis for repositories
• Set up pull request bot with inline fixes
• Generate and attach contextual fix patches
• Apply noise filtering and suppression rules
• Enforce PR merge gates for high-risk findings
• Enable continuous vulnerability monitoring and alerts
• Activate vulnerability prioritization engine
• Integrate findings into issue tracker workflows

Install IDE security plugin

• Which IDEs are used by the developers who will receive the plugin? Options: VS Code, JetBrains (IntelliJ/PyCharm/WebStorm), Visual Studio, Eclipse, Other
• Approximately how many developers should receive the plugin during the 30-day pilot? Options: < 10, 10-50, 51-200, 200+
• Will plugin installation be managed centrally (MDM) or should it be end-user installable? Options: Managed centrally (IT/MDM), Self-install by developers, Combination of both
• Are there policy or security restrictions that prevent installing third-party plugins on developer machines? Options: No restrictions, Some restrictions - requires approval, Strict no-third-party policy
• Do you require the plugin to suggest inline fixes or only to surface findings and links to guidance? Options: Inline fixes suggested, Findings + guidance only, Configurable per repo/team
• Do you want the IDE plugin telemetry and usage data (e.g., clicks on fixes) to be collected for pilot success metrics? Options: Yes, No, Only anonymized/aggregated

Integrate scanner into CI/CD pipelines

• Which CI/CD systems host the target repositories' pipelines? Options: GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure DevOps, Other
• For the pilot repos, how many builds per day (approx) will include the scanner run? Options: < 10, 10-50, 51-200, 200+
• What is the acceptable build time overhead for adding the scanner (per build)? Options: < 30s, 30s-2min, 2-5min, 5+ min
• Should scanner runs be gated (fail build on high-risk findings) or non-blocking for the pilot? Options: Non-blocking (report only), Soft gate (warn, do not block), Hard gate (fail build)
• Who will provide credentials/service accounts for CI integration and who is responsible for pipeline configuration?
• Are there existing secrets scanning, artifact retention, or compliance constraints the integration must respect? Options: No, Yes - list constraints

Deploy container image scanning in builds

• Do your builds produce container images for the pilot repositories? Options: Yes, No
• Which container registries are used (for credentials and scan integrations)? Options: Docker Hub, Amazon ECR, Google Container Registry, Azure Container Registry, Private Registry, Other
• Should image scanning run at build time, push time, or both? Options: Build time, Push time, Both
• Do you require scanning of base images and layered analysis (e.g., identify vulnerable packages inside image layers)? Options: Yes - layered analysis, No - only final image manifest, Configurable
• If a high-severity image vulnerability is found, what action should the pipeline take during pilot? Options: Fail build, Create ticket and continue, Notify owners only
• Are there licensing or SBOM requirements for container images we should include? Options: Yes - provide details, No

Activate infrastructure-as-code scanning

• Which IaC technologies are present in the targeted repositories? Options: Terraform, CloudFormation, ARM templates, Kubernetes manifests/Helm, Pulumi, Other
• Where should IaC scanning run: pre-commit, CI, or during deployment pipelines? Options: Pre-commit, CI, Deployment pipelines, Multiple locations
• Do you want policy-as-code checks (e.g., deny public S3, require MFA) enforced automatically or reported for manual remediation? Options: Enforce automatically, Report only, Hybrid
• Are there specific cloud accounts/regions or modules to exclude from scanning?
• Who owns IaC remediation in your org (DevOps, AppSec, Cloud Engineers)? Options: DevOps/Platform, Application Teams, Cloud Security/Infra, Other
• Do you require detection of secrets or misconfigurations in IaC as part of the pilot? Options: Yes - include secrets detection, No - misconfig only, No

Enable dependency SBOM generation and monitoring

• Which languages and package managers are used in the pilot repositories? Options: npm/Yarn, Maven/Gradle, pip/Poetry, Go modules, Rust Cargo, Other
• Do you require SBOM generation for every build or on a schedule (e.g., daily)? Options: Every build, Daily, Weekly, On-demand
• Should SBOMs be stored in your registry/artifact store or pushed to a central monitoring service? Options: Stored in artifact registry, Pushed to central monitoring, Both, Undecided
• Do you need monitoring/alerting for newly disclosed vulnerabilities affecting SBOM components? Options: Yes - critical only, Yes - high+medium, No
• Do you use private package registries or mirrors that require credentialed access for SBOM resolution? Options: Yes - will provide credentials, No
• Are license compliance checks part of the SBOM monitoring requirements for the pilot? Options: Yes, No, Maybe - discuss

Configure reachability analysis for repositories

• Do the pilot repositories contain the source and build artifacts needed for static reachability analysis (e.g., full repo history, compiled artifacts)? Options: Yes - full source, Partial - limited artifacts, No - additional access required
• Which languages and frameworks should reachability analysis prioritize?
• Are there specific entry points or services (e.g., REST endpoints, message handlers) we should focus reachability on? Options: Yes - list entry points, No - analyze all
• Do you want reachability analysis to exclude test code, generated code, or third-party vendor modules? Options: Exclude tests, Exclude generated code, Exclude vendor modules, None of the above
• What false-positive tolerance do you expect for reachability (acceptable FP rate)? Options: Very low (<5%), Low (5-15%), Moderate (15-30%), Higher (>30%)
• Who will validate and triage reachability results during the pilot (AppSec, engineering leads, dedicated SRE)? Options: AppSec, Engineering leads, SRE/Platform, Cross-functional team

Set up pull request bot with inline fixes

• Should the bot open comments, create fix-commits, or open suggested-change PRs for findings? Options: Open comments only, Create suggested-change PRs, Create fix commits directly, Configurable per repo
• Which branches and repositories should the PR bot operate on during the pilot?
• What confidence threshold should the bot use before proposing inline fixes (to limit noise)? Options: Very high, High, Medium, Low
• Should bot actions be restricted by code owners or require an approval workflow before merge? Options: Require code owner approval, Auto-apply with optional review, Always request human approval
• Who will receive bot notifications and who is responsible for addressing bot-created PRs?
• Do you require the bot to include tests or CI validation in suggested fixes? Options: Yes - include tests/CI check, No - fixes only, Prefer but optional

Generate and attach contextual fix patches

• Do you want fix patches to be language-specific and follow project linting/formatting rules? Options: Yes - enforce formatting, No - minimal patch, Configurable per repo
• Should generated patches include explanatory comments and links to rationale/resources? Options: Yes - include comments and links, No - minimal patch
• Do you require the patches to include unit or integration test updates where applicable? Options: Yes - include tests where possible, No - patches only, Prefer but optional
• What is the maximum acceptable size or scope for an automated patch (e.g., single-file, multi-file)? Options: Single-file only, Multi-file allowed up to 5 files, No strict limit - review required
• Who signs off on automated patches before merge during pilot (author, reviewer, security owner)? Options: Author/PR owner, Designated security reviewer, Code owner, Automated sign-off allowed
• Do you want patches delivered as direct commits to branch or as suggestion comments/PRs for manual merge? Options: Direct commits, Suggestion comments/PRs, Configurable

Apply noise filtering and suppression rules

• Which suppression scopes do you require: organization-wide, repository, path, file, or line-level? Options: Organization, Repository, Path, File, Line
• Do you prefer temporary suppressions (auto-expire) or permanent suppressions with audit trail? Options: Temporary (with expiry), Permanent with audit log, Combination
• Should noise filters be pre-populated during pilot (based on historical data) or built iteratively? Options: Pre-populate from historical data, Build iteratively during pilot, Hybrid
• What team or role should be able to create suppression rules (AppSec, repo owners, SRE)? Options: AppSec only, Repo owners, SRE/Platform, Cross-functional
• Do you require approval workflows or reviews for creating suppression rules? Options: Yes - approval required, No - immediate creation, Only for org-level suppressions
• How should suppressed findings be surfaced (hidden entirely, visible in audit view, reported as suppressed)? Options: Hidden entirely, Visible in audit view only, Reported as suppressed with reason

Enforce PR merge gates for high-risk findings

• Which severity levels are considered high-risk for merge gating during the pilot? Options: Critical only, Critical + High, High + Medium, Custom mapping
• Should merge gates be applied automatically or require a manual override process? Options: Automatic block, Manual override allowed, Notify and do not block
• Who may approve overrides or exemptions for blocked PRs? Options: AppSec, Repo owner, Team leads, Designated approvers
• Do you want gating integrated as CI status checks, branch protection rules, or both? Options: CI status checks, Branch protection, Both
• Should gating be rolled out progressively (e.g., advisory mode first) across target repos? Options: Yes - start advisory, No - enforce from day one, Staged per repo
• What is the escalation path if merge gates are blocking critical releases during pilot?

• Non-Disclosure Agreement (NDA)
• Master Services Agreement (MSA)
• Statement of Work (SOW)
• Pilot Acceptance Criteria
• Roles & Responsibilities (RACI)
• Timeline & Milestones
• Data Access & Privacy Agreement (DPA)
• Security & Access Authorization
• Escalation and Rollback Plan
• Licensing, Pricing & Billing Schedule
• Service Level & Pilot Support Agreement (SLA)
• Change Order & Scope Modification
• Termination, Exit & Handover

• Pilot Results & Outcomes Review
• Operational Rollout Planning
• Shared Backlog & Continuous Improvement Workshop
• Executive Decision & Funding Review

• Opening & Objectives
• Create a short list of tuning and detection gap tickets (with owners and target dates) for items raised in the Noise & Failure Modes section.
• If acceptance is conditional, schedule a 2-week follow-up validation scan and assign owner to run it.
• Assign platform engineering owner to implement build-time mitigations and report pre-rollout test results.
• Recap: Pilot Acceptance & Future State
• Define a concrete, phased rollout plan with dates, scope per phase, and measurable success targets.
• Identify and assign operational owners and support model for rollout execution and escalation.
• Agree on technical mitigations to preserve acceptable build impact and low false-positive rates at scale.
• Establish clear rollback criteria and executive escalation paths to manage risk during expansion.
• Draft a rollout project plan (milestones, owners, risks) and circulate for sign-off.
• Document required access changes and create onboarding checklist for teams in phase 1 of rollout.
• Pre-work Review
• Establish a single shared backlog with categories, prioritization rules, and owners.
• Agree on triage cadence and SLAs to ensure continuous improvement and responsiveness to developer feedback.
• Identify immediate quick wins to reduce noise and increase developer trust within 30 days.
• Populate the agreed backlog in the chosen tracking tool and assign owners to the top 10 items.
• Schedule recurring backlog triage meetings and invite relevant engineering and AppSec reps.
• Deliver a short 'what changed' update to pilot developers documenting quick wins and expected behavior changes.
• Executive Summary: Current State & Consequence
• Obtain executive approval to fund and mandate the phased organizational rollout.
• Ensure executives understand measurable monitoring and reporting that will track rollout success.
• Secure an executive sponsor and define the cadence for executive updates post-rollout.
• Produce a one-page decision memo (pilot results, ROI, ask) and circulate for executive signatures.
• If approved, authorize finance/procurement to execute contractual changes and allocate budget.
• Assign an executive sponsor and schedule the first executive status review three weeks after rollout starts.
• Ensure all stakeholders share a single, explicit statement of current state, consequence, and future state.
• Validate pilot metrics against success signals and get explicit acceptance or list of gaps to close.
• Agree on concrete tuning actions and owners for any identified false positives or missed detections.
• Decide immediate disposition: proceed to rollout planning, extend pilot, or remit to technical remediation.
• Prepare and circulate an acceptance report that maps each success signal to measured results and stakeholder sign-offs.
• Pilot Outcomes vs Success Signals
• One-sentence Current State
• Rollout Phasing & Scope
• Backlog Structure & Taxonomy
• Consequence Summary
• Prioritization Criteria
• ROI & Risk Reduction Case
• Success Targets for Rollout
• Requested Commitment
• One-sentence Future State
• Triage Process & Cadence
• Integration & Access Requirements
• Roadmap Mapping & Quick Wins
• Decision & Next Steps
• Metrics Dashboard Walkthrough
• Build-Time & Reachability Tuning Plan
• Operational Model & Roles
• Developer Communication & Feedback Loop
• Proof: Code-level Examples
• Noise & Failure Modes
• Risks, Rollback & Escalation Paths
• Timeline, Milestones & Approval Gates
• Validation Checkpoints
• Decision & Next Steps