Quick Intro — Who's in the Room?

• Who should we consider part of the formal decision team for this evaluation (names and roles)? Options: Program officer / funder staff, Nonprofit executive director, Program director at grantee, Research/monitoring lead, Board member or trustee, Policy or legislative liaison, Other
• Which single person is expected to sign the contract or authorize budget for this work? Options: Program officer, Executive director / CEO, Board chair, Grants manager, Government procurement officer, Unknown / needs confirmation
• Who needs to be informed but is unlikely to influence the design (e.g., communications, beneficiaries, partner agency)? Options: Communications lead, Beneficiary representatives, Partner organization leads, IRB/ethics office, Fund finance team, Other
• How stable is the decision team between now and the expected project timeline (turnover, political cycles)? Options: Very stable — no expected changes, Some turnover possible, High turnover risk, Unclear
• Tell us anything about these people — personalities, prior experience with evaluations, or recent events — that will shape how decisions are made.

Who's Really Driving the Decision?

• If the person with the loudest voice wins, who would lose — and what would that cost your program or the funder?
• Which stakeholders have formal veto power over budget, methodology, or report release? Options: Primary decision-maker (budget), Legal or procurement, Board chair, Program leadership, Partner organization leadership, IRB/ethics committee, No formal veto
• Who in the group is most likely to prioritize reputational or political risk over methodological trade-offs? Options: Funders / program officers, Executive leadership, Board members, Government policy staff, Partner leaders, None / unknown
• How have disagreements between key stakeholders affected past decisions on evaluations or program changes?
• How long has the core decision-making team worked together, and how does that tenure affect their willingness to accept critique or change?

What 'Good Evidence' Really Means to Them

• When you hear the phrase 'rigorous evidence,' whose definition of rigor matters most—and how do they describe it?
• For each key stakeholder, what concrete findings, metrics, or thresholds would make them conclude the evaluation succeeded (list person/role → evidence needed)?
• Do any stakeholders require specific statistical thresholds or power for decisions (e.g., p-values, minimum effect size, subgroup power)? Options: Yes — strict statistical thresholds, Yes — approximate benchmarks, No, qualitative evidence is sufficient, Unsure
• If the answer above is yes, please specify the thresholds, who requires them, and why they matter.
• How important is demonstrating equitable impacts across subgroups to your decision-makers (e.g., race, income, geography)? Options: Primary decision criterion, Important but secondary, Nice-to-have, Not a priority
• Which forms of evidence will carry the most weight with each stakeholder: quantitative outcomes, qualitative stories, cost-effectiveness, implementation fidelity, or something else? Options: Quantitative outcomes, Qualitative narratives / case studies, Cost-effectiveness / ROI, Implementation fidelity / process data, Mixed evidence package, Other

Timing, Triggers, and Funding Windows — What Breaks the Stalemate?

• If there’s one date or deadline that makes this project happen — or causes it to collapse — what is that date and why?
• Which external deadlines are tied to funding or reporting (select all that apply)? Options: Grant renewal cycle, Board meeting / decision cycle, Federal reporting / Evidence Act deadline, Budget appropriations deadline, Contractual deliverable date, None / no external deadline, Other
• What funding triggers or milestones (e.g., match funding, enrollment targets, partner agreements) will unlock payment or continuation of the program?
• How flexible are timelines if the evaluation needs more time for IRB approval, data access, or partner coordination? Options: Very flexible, Somewhat flexible, Not flexible — fixed deadline, Unsure
• Are there internal checkpoints where interim findings could influence decisions even before the final report (e.g., pilot adjustments, funding reallocation)? If so, what are they?

What's Getting Hidden Under the Surface?

• What inconvenient truth about the initiative would stakeholders prefer the evaluation not fully expose?
• Are there known data gaps, quality concerns, or partner conflicts that could limit credible conclusions? Options: Missing outcome data, Low sample size, No comparison group, Partners unwilling to share data, Inconsistent measures across sites, Other
• Which organizations or leaders are likely to resist independent findings, and what does their resistance usually look like?
• Have previous evaluations led to backlash, funding changes, or reputational harm? Briefly describe what happened and the fallout.
• How comfortable are stakeholders with public dissemination versus internal use only for the final report? Options: Full public release, Public with caveats or embargo, Internal distribution only, Unsure / conditional

How Will People Actually Use the Findings?

• If this report produced one concrete decision six months from now, what would that decision be—and who would act on it?
• Which audiences need digestible summaries for non-research stakeholders (select all that apply)? Options: Board members / trustees, Funders / program officers, Policymakers / legislators, Community or beneficiary groups, Media / public, Partner agencies
• Do any stakeholders require access to raw data, technical appendices, or replications rights? Options: Raw data access, Technical appendix only, Right to commission replication, No special access required, Unsure
• How will you measure whether recommendations were acted on and had the intended effect (short, medium, long-term indicators)?
• Who will be responsible for communicating findings externally, and are there communication constraints or approvals we must plan for?

Deal-Breakers, Red Lines, and Non-Negotiables

• What would make you walk away from this evaluation before it begins?
• Are there legal, ethical, or independence constraints that must be honored (select all that apply)? Options: IRB / ethics approval required, Non-disclosure agreements, Publication restrictions by funder, Data ownership limitations, Independence clause prohibiting program staff analysis, None
• Are there stakeholders who must approve the final wording or visuals before release (name/role and scope of approval)?
• Which budget or payment terms are non-negotiable for you (select all that apply)? Options: Fixed-price contract, Milestone-based payments, Cost-reimbursable model, Upfront deposit required, Procurement-specific terms, Other
• If a dispute arises about interpretation or acceptance of findings, what process should we use to resolve it (mediation, steering committee, funder final say)? Options: Steering committee review, Independent arbitrator / mediator, Funder has final decision, Joint revision process, Other

Confidence & Next Steps — Are We Ready to Move?

• On a risk scale, how risky is moving forward now versus waiting to clarify more? Options: Low risk — proceed, Moderate risk — proceed with mitigations, High risk — clarify before proceeding, Unsure
• What are the top three unanswered questions that would need resolving before you could commit to a scoping or contract?
• Who should be in the next meeting (name and role), and what would success look like at the end of that meeting? Options: Program officer / funder rep, Executive director, Program director / site lead, Research or M&E lead, Partner representative, Other
• Please list contact details (name, role, email) for the people to invite and any calendar constraints we should know about.
• Realistically, when could you give us a commitment to begin scoping (select one)? Options: Immediately (within 2 weeks), Within 1 month, 1–3 months, 3+ months, Unsure

Tell Us the Story Behind Your Program

• In one sentence, how would you describe the core purpose of this program?
• What are the primary activities or services you deliver (select all that apply)? Options: Direct services to participants, Training/capacity-building, Cash or in-kind assistance, Policy or advocacy work, Technical assistance to other organizations, Other
• Who is the program intended to serve? (population, age ranges, geography, eligibility criteria)
• Briefly summarize the logic linking your activities to the outcomes you expect.
• Which core assumptions about participants or context does your theory of change rely on?
• How long has the program operated at its current model or scale? Options: < 1 year, 1–2 years, 3–5 years, 6–10 years, 10+ years

What's Really Driving Results (or Not)?

• Which part of this program do you suspect is creating the biggest impact—and why might that assumption be incorrect?
• Which outcome measures do you currently monitor on a regular basis? Options: Enrollment/participation, Completion/retention, Short-term skill/knowledge gains, Behavioral changes, Economic outcomes (income, employment), Health outcomes, Client satisfaction, Other
• Of the outcomes you track, which three do decision-makers treat as most persuasive?
• How confident are you that your tracked outcomes accurately reflect real change (not just administrative artifacts)? Options: Very confident, Somewhat confident, Unsure, Not confident
• Can you share recent trends or numbers for 1–3 priority metrics (briefly list values, time periods, and any caveats)?
• Which outcomes have stakeholders repeatedly asked for but you currently cannot measure well?

Where the Numbers Come From

• If your metrics were misleading, where in your data sources or collection would you expect the problem to originate?
• Which data sources feed your metrics (select all that apply)? Options: Client/participant records, Administrative/government data, Program surveys, Phone/SMS data, Case management software, Partner databases, Third-party vendor data, Sensors/devices, Paper records, Other
• Who technically owns each major data source, and are there formal agreements governing access? Options: We own and control, Owned by partner with shared access, Owned by funder, Third-party vendor controls, No clear ownership, Unsure
• How frequently are the main data sources updated? Options: Daily, Weekly, Monthly, Quarterly, Annually, Ad-hoc/irregular, Unsure
• Do you have documented codebooks, data dictionaries, or field guides for key datasets? Describe their completeness. Options: Comprehensive and up-to-date, Partial / outdated, Informal notes only, None exist, Unsure
• Which data gaps or persistent missing fields cause the most analysis headaches? Please provide examples.

Who's at the Table—and Who's Missing?

• Who makes final decisions about program design, data sharing, and evaluation priorities—and are those roles aligned?
• Which partners regularly contribute to implementation, data collection, or interpretation? Options: Service delivery partners, Local government, National agency, Funders, Academic partners, Independent evaluators, Community organizations, Data vendors, Other
• Do any partners or internal stakeholders have incentives that could bias how outcomes or data are reported? Options: Yes, No, Unsure
• If you answered Yes or Unsure, describe the incentives, conflicts, or pressures and who they involve.
• How involved are frontline staff and program participants in shaping metrics or interpreting findings? Options: Heavily involved, Occasionally consulted, Rarely consulted, Not involved, Unsure
• Which voices (e.g., community members, frontline staff, local leaders) do you feel are underrepresented in evidence discussions?

Data Plumbing: How Information Flows (and Leaks)

• What is the weakest point in your data flow—the place most likely to silently corrupt a finding?
• Which tools and processes do you use to collect and store data (select all that apply)? Options: Paper forms, Excel/spreadsheets, Dedicated case management system, Custom database, Cloud storage (Google/Drive, S3), EHR/clinical system, Survey platform (Qualtrics, ODK), APIs/data pipelines, Other
• What quality assurance practices are in place (training, double-entry, automated checks, audits)? Options: Routine training for collectors, Automated validation rules, Field audits/spot checks, Double data entry, No formal QA, Other
• Have you experienced data loss, corruption, or prolonged access delays in the last 12 months? Options: Yes—significant, Yes—minor, No, Unsure
• How accessible are de-identified raw data and metadata to an independent evaluator, practically speaking? Options: Easily accessible, Accessible with approvals, Technically possible but time-consuming, Restricted / not accessible, Unsure
• Who is responsible for maintaining the technical systems (title/role), and how quickly do they resolve issues?

What Keeps You Up at Night About Evidence?

• What is the single greatest risk you fear will invalidate or weaken any evaluation findings?
• Which of the following evidence risks have you encountered (select all that apply)? Options: Selection bias, High participant attrition, Measurement error/inconsistent instruments, Small sample sizes, Key confounders unmeasured, Missing data, Data access/legal constraints, Low survey response rates, Other
• Which of these risks would you prioritize resolving before an evaluator begins analysis? Options: Selection bias, Attrition, Measurement error, Sample size, Legal/data access, None of the above, Unsure
• Have you previously used statistical or design strategies (e.g., matching, instrumental variables, randomized assignment) to address bias? Please describe.
• How do concerns about equity (differential effects across groups) affect which risks you worry about most?

Past Evidence: Wins, Holes, and Hard Truths

• When past evaluations fell short, what was the real root cause—even if it wasn't written in the final report?
• Have you commissioned external evaluations previously? Which study types have you used? Options: Randomized controlled trial (RCT), Quasi-experimental (matching, regression discontinuity), Pre-post without control, Process/implementation evaluation, Qualitative evaluation only, Mixed-methods, None
• Which recommendations from prior evaluations were implemented and what changed as a result?
• Were there evaluation findings that stakeholders contested or disregarded? What made them contested?
• How satisfied have you been with previous evaluators on clarity, timeliness, and independence? Options: Very satisfied, Somewhat satisfied, Neutral, Somewhat dissatisfied, Very dissatisfied
• What would you have wanted previous evaluators to do differently?

Decision Timelines and Success Signals

• If leadership asks tomorrow 'Did this program work?', what answer do they expect—and what evidence would satisfy them?
• List critical decision dates or funding milestones we should align to (e.g., board meeting, grant renewal deadline).
• How quickly do you realistically need preliminary findings or interim signals? Options: Within 1 month, 1–3 months, 3–6 months, 6–12 months, No urgency / flexible
• Which specific success signals would trigger scale-up, redesign, or termination of the program?
• How should equity shape success criteria (select all that apply)? Options: Disaggregate results by race/ethnicity, Disaggregate by income, Disaggregate by geography, Measure differential access/benefit, Apply equity-weighted outcomes, Equity not prioritized, Other
• Who must formally accept the evaluation findings for them to be acted upon (roles/titles)?

Practical Readiness: Capacity, Resources, and Legal

• If we tried to launch a rigorous evaluation next month, what single missing resource or approval would block us from starting?
• What internal staff time and skills can you allocate to support evaluation activities (select all that apply)? Options: Data manager/analyst, Program manager, Frontline data collectors, IT/system administrator, Legal/compliance, None available, Other
• Are there existing data-sharing agreements, MOUs, or legal constraints we should know about? Options: Fully executed agreements, Draft agreements in progress, No agreements, Unsure
• Is IRB or equivalent human subjects review required, and what is its status? Options: Not required, Required and approved, Required and pending, Required and not yet submitted, Required and denied, Unsure
• What is the realistic budget range you have available for an external evaluation? Options: <$50k, $50k–$150k, $150k–$300k, $300k–$600k, >$600k, Unsure
• If the budget were limited, what elements of an evaluation are non-negotiable for you (e.g., independent analysis, sample size, disaggregation)?

Next Steps — What Would Success Look Like in This Partnership?

• If you could guarantee one thing from an external evaluator, what would it be—and why hasn't that happened before?
• How involved would you like to be in design and analysis decisions? Options: Deep collaboration (joint design/analysis), Advisory role (regular check-ins), Minimal involvement (approvals only), Prefer evaluator-led, Unsure
• Which deliverables matter most to you (select top 3)? Options: Technical report with methods appendix, Short executive summary, Policy brief, Presentation to leadership/board, Interactive dashboard, Raw de-identified datasets, Implementation recommendations, Community-facing summary
• How should findings be communicated to different audiences (e.g., funders, frontline staff, participants)? Please describe tone, format, or channels you prefer.
• What would make you decide to move forward with an evaluator in the next 30 days?
• Are there any immediate concerns, sensitivities, or political dynamics we should address before proceeding?

Starting With Confidence: A Quick Snapshot

• In one sentence, what would make you feel confident we can start fieldwork next week?
• Which of these milestones is already fully complete for this project? Options: Funded/contract signed, IRB approval, Data access agreements signed, Partner permissions granted, Staffing confirmed, Sampling frame finalized, None of the above
• Who is the single person on your team we should contact first if something threatens the start date?
• Realistically, how soon are you hoping fieldwork begins? Options: Within 2 weeks, Within 1 month, 1–2 months, 3+ months, Timing is flexible
• What keeps you up at night about launching this study?

What Could Stop This Study From Starting?

• If the study failed to launch on the planned date, what would likely be the single biggest reason? Options: IRB delay, Data access blocked, Partner withdrawal, Insufficient staff, Sampling/screening issues, Budget/payment hurdles, Other
• For the risk you selected, how long has this been an unresolved concern? Options: It surfaced this week, 1–4 weeks, 1–3 months, Longer than 3 months, I’m not sure
• Who would be the decision-maker to resolve that issue, and what would convince them to approve moving forward?
• What fallback or mitigation have you already considered or put in place for this risk?
• On a scale between 'we can absorb a short delay' and 'a delay kills the study,' where are you? Options: Absorb a short delay (1–2 weeks), Manageable but costly (2–6 weeks), Risk of significant harm (6–12 weeks), Delay would likely cancel the project

Who Must Say Yes — And What Would Shake Them?

• Who are the external or internal stakeholders whose approval is required before fieldwork begins? Options: IRB/Ethics board, Data-owning agency, Partner organizations, Funders, Host institution leadership, Legal/compliance, Other
• For each stakeholder you selected, what specific evidence or assurance do they expect before granting permission?
• Which stakeholder is most likely to request changes that would materially alter the study design? Options: IRB/Ethics board, Data-owning agency, Partner organizations, Funders, Legal/compliance, None likely
• Describe a recent example (even from another project) where a stakeholder request created a major shift — what happened and how was it resolved?
• How would you like us to surface and package materials to get a quicker 'yes' from these stakeholders? Options: Concise memo + Q&A, Formal technical appendix, Executive summary + visuals, Pre-meeting with stakeholder, Other

Is Your Data Truly Ready to Tell the Story?

• If we attempted preliminary analyses tomorrow, which data sources would we actually be able to access? Options: Administrative records, CMS/ED/Health agency data, Program enrollment lists, Survey platform exports, Partner-held spreadsheets, None available yet, Other
• For each accessible source, what is the current access level? Options: Direct DB access (read-only), Secure file transfer (SFTP/CSV), Aggregate reports only, Requires new agreement, No access
• Are there known data quality issues (missing fields, inconsistent IDs, time lags) that would affect primary outcomes? Options: Yes — significant, Yes — manageable, Minor/expected issues, No major issues, Don’t know yet
• How long would it take your team or data partners to provide a cleaned, analysis-ready extract if requested? Options: <1 week, 1–2 weeks, 2–4 weeks, 1+ month, Unknown
• Please list any unique data linking or de-identification requirements we should know about.

Permissions & Paperwork: Where Do Bottlenecks Live?

• Which legal or administrative approvals remain outstanding? Options: IRB/Ethics approval, Data sharing agreement (DSA), Memorandum of Understanding (MOU), Business associate agreement (BAA), Agency data use agreement, None outstanding, Other
• For each outstanding item, who is responsible for advancing it and what is their planned completion date?
• Have any partners signaled they will require contract amendments or additional clauses before they permit data access? Options: Yes — major changes, Yes — minor edits, No, Unsure
• What IRB category do you expect this study to fall under? Options: Exempt, Expedited, Full review, Not human subjects / IRB not required, Unsure
• If the IRB issues conditional approval, what conditions would be unacceptable to you?

Are Your Partners Ready to Be Partners?

• Which partner organizations are essential for fieldwork operations (e.g., recruitment, access, local staff)?
• How would partners describe their current bandwidth for supporting this study? Options: High — ready to support, Moderate — some capacity, Low — stretched thin, Unavailable right now, Unsure
• Have you documented partners’ responsibilities and commitments in writing? Options: Yes — fully documented, Partially documented, No, verbal only, Not yet discussed
• Tell us about any previous collaborations with these partners that succeeded or failed — what patterns should we watch for?
• What incentives or reassurances would increase partner engagement and reliability? Options: Clear timelines & checkpoints, Reimbursement for staff time, Simpler data collection tools, Joint communications to stakeholders, Capacity-building support, Other

Can We Actually Staff This With Confidence?

• Which core roles are already staffed for deployment (e.g., field lead, data manager, survey enumerators, qualitative interviewers)? Options: Field lead/project manager, Data manager/analyst, Survey enumerators, Qualitative interviewers, Community liaisons, None staffed yet
• For roles not yet staffed, what is the main hiring or contracting constraint? Options: Budget, Availability of qualified candidates, Clear job specs, Security/clearance requirements, Time to onboard, Other
• How long does onboarding typically take for field staff in your context (training, background checks, IRB certifications)? Options: <1 week, 1–2 weeks, 2–4 weeks, 1+ month
• What training or competency checks are non-negotiable before staff begin interacting with participants? Options: IRB/ethics training, Data security training, Cultural competency/equity training, Protocol-specific mock interviews, Other
• If a key staff person becomes unavailable after launch, what's the contingency for immediate replacement?

Sampling & Field Readiness — Is the Frame Real?

• Do you have an actionable sampling frame (list of participants/units) that we can use to draw our sample? Options: Complete and usable, Partial — needs cleaning, Exists but requires permissions, No usable frame
• What percentage of the frame do you estimate is contactable and valid today? Options: >90%, 70–90%, 50–70%, <50%, Unknown
• Are there known subgroups or strata we must oversample to address equity priorities? Options: Yes — clearly defined, Yes — to be defined, No, Unsure
• What approvals (consent, guardian permission, agency sign-off) are required to reach or recruit from the frame?
• If sampling must be adjusted in the field (low response, unreachable clusters), which trade-offs are acceptable to you (timeline extension, reduced power, additional waves)? Options: Extend timeline, Increase recruitment resources, Accept lower power/sample size, Adjust sampling strategy, Pause and reconvene

If Something Breaks — What’s the Real Plan?

• When deployment hits a significant obstacle (e.g., lost access to a data source), who has final authority to approve the fix or change? Options: Project PI, Funder representative, Partner lead, Joint steering committee, Other
• What decision-making timeline is acceptable for critical fixes (hours, days, weeks)? Options: Within hours, 1 business day, 2–5 business days, More than a week
• Describe one contingency you’d accept that still preserves the study’s core questions.
• How do you prefer we escalate problems to you (email, phone, scheduled huddle, emergency contact)? Options: Email + daily updates, Phone call then email summary, Scheduled rapid-response huddle, Platform ticketing system, Other
• What would feel like a reasonable compensation or remedy to partners/participants if fieldwork requires extra burden due to adjustments? Options: Modest stipend, Additional staff support, Delayed reporting credit, No compensation, Other

Commitments, Timelines, and Final Sign-offs

• What is the non-negotiable earliest date we must finish deployment by (e.g., for funding, policy cycles)?
• Which deliverables or milestones must be completed before payment or fund release? Options: IRB approval, Data transfer complete, Fieldwork initiation, Interim report, Final dataset delivered, Other
• Who will provide final sign-off that fieldwork is acceptable to proceed (name and role)?
• Are there pre-agreed acceptance criteria for fieldwork quality (response rates, data completeness thresholds, protocol adherence)? Options: Yes — fully defined, Partially defined, No, not defined, Want help defining them
• What final evidence or artifacts would make you comfortable to greenlight the first wave of fieldwork? Options: IRB approval letter, Signed DSAs/MOUs, Staff roster and trainings complete, Sample draw and contact list, Pilot test results, Other

Agreeing on Small Next Steps (Practical, Concrete)

• Given everything above, what are the top three actions you want our team to take in the next 7 days? Options: Confirm IRB status and next steps, Request/validate data extracts, Draft partner communications, Recruit/contract field staff, Run a small pilot, Other
• Which of those actions should we treat as urgent and why?
• Who on your side will be our touchpoint for these 7‑day actions (name, role, best contact)?
• How would you prefer we report progress on these items (daily checklist, brief status update, shared dashboard)? Options: Daily checklist email, Twice-weekly brief, Shared live dashboard, Ad-hoc as issues arise, Other
• Is there anything else — a concern, context, or boundary — we haven’t asked about that would change how we approach readiness?

Starting Bright: Quick introductions and what's on the table

• Briefly describe who you are, your role, and the program or grant that would be evaluated.
• Which of these best describes the primary customer group requesting this evaluation? Options: Foundation program officer, Nonprofit executive director, Government program manager, Grantmaking committee/Board, Other
• Who within your organization is the final decision-owner for commissioning and accepting an evaluation? Options: Program officer/manager, Executive Director/CEO, Board or committee, Granting agency official, Other
• What triggered the idea for an external evaluation now? Options: Grant renewal/renewal decision, Board or leadership request, Federal reporting requirement, Program expansion/pivot, Donor request, Other
• Roughly when would findings need to exist to influence that trigger (month/quarter and year)? Options: Within 3 months, 3–6 months, 6–12 months, 12–18 months, Longer than 18 months, Unsure
• Who else should be in the room for discovery conversations (names or roles)?

Are we asking the right question — or just proving what we already believe?

• What is the central question you expect this evaluation to answer—and why might that be the wrong question?
• Which outcome(s) do you currently treat as the most important? Options: Participant outcomes (e.g., test scores, employment), Service delivery metrics (e.g., visits served), Equity indicators (e.g., subgroup gaps), Cost/efficiency measures, Policy or systems change, Other
• How were those outcomes chosen—board mandate, funder priority, historical practice, or something else? Options: Board/leadership request, Funder requirement, Historical tracking, Stakeholder input, Other
• What would surprise you in the results—i.e., what finding would make you rethink how the program works?
• How confident are you that existing measures truly capture the change you care about? Options: Very confident, Somewhat confident, Not confident, Unsure

Where the evidence is thin — what keeps you up at night about data?

• What single piece of information do you wish you had right now to make a better decision about this program?
• Which of these data sources are available or potentially available for analysis? Options: Administrative program data, Client surveys, Partner records, State/federal administrative data, Qualitative interviews/focus groups, Existing evaluation reports, None of the above / unsure
• Who owns or controls access to the most important data—and how easy would it be to request access? Options: We control it and access is easy, We control it but access requires approvals, Partner controls it, State/federal agency controls it, Unknown/unsure
• Have you previously commissioned evaluations or analyses for this program? If so, what were the key findings and limitations?
• Thinking about data quality, what specific issues do you anticipate (missing data, inconsistent definitions, small samples, access delays, etc.)?

Who really holds the keys — influence, approval, and credibility

• If the evaluation found that the program needed major changes, who could block implementation or reinterpret the results?
• Which stakeholders should explicitly approve study design, interim findings, and final acceptance? Options: Funder/program officer, Executive leadership, Program staff, Partner organizations, Beneficiary representatives, IRB/ethics committee, Other
• How do different stakeholders define 'good' evidence—rigorous causal proof, credible descriptive trends, or something else? Options: Causal impact (RCT/quasi-experimental), Comparative outcome trends, In-depth qualitative understanding, Mixed-methods triangulation, Policy-relevant syntheses, Unsure
• What political or reputational pressures might shape how findings are received or shared?
• Are there equity priorities or communities we must center when designing questions and interpreting results? Please list and explain.

Imagine a decision that actually changes things — who benefits and how?

• Describe a concrete decision you want this evaluation to enable (e.g., renew funding, scale program, pivot model, inform policy), and why it matters.
• When that decision is made, what specific evidence threshold would make leaders feel comfortable (effect size, qualitative consensus, stakeholder endorsement, cost threshold)? Options: Statistically significant effect, Meaningful practical improvement, Strong qualitative evidence, Cost-effectiveness demonstration, Broad stakeholder agreement, Unsure
• Who will use the report and how—internal strategy, funder decisions, public reporting, or advocacy? Options: Internal leadership decisions, Funder/board decisions, Public dissemination/advocacy, Academic publication, Regulatory reporting
• How would different outcomes (positive, null, negative) ideally be communicated to funders, staff, and communities?
• What deliverables would be most useful (technical report, one-page brief, slide deck, data appendix, public summary)? Options: Technical report, One-page executive brief, Presentation slides, Data appendix/reproducible code, Community-facing summary, Infographic/visuals

Design reality check — tradeoffs we're willing to live with

• Which tradeoff would you prioritize if constrained: speed of results, methodological rigor, or breadth of coverage? Options: Speed of results, Methodological rigor, Breadth/representativeness, Balance of all three, Unsure
• What sample sizes or subgroup analyses are non-negotiable for you (specific populations, geographies, or program cohorts)?
• Would you consider experimental or quasi-experimental designs (randomization, regression discontinuity, matched comparison)? Options: Yes, open to randomization, Yes, open to quasi-experimental, Prefer observational/mixed methods, Not comfortable with experimental designs, Unsure
• Are there legal, ethical, or IRB constraints we must plan for (consent, vulnerable populations, data-sharing restrictions)? Options: IRB required, Confidentiality constraints, State/federal data restrictions, Partner consent needed, No known constraints, Unsure
• What budget range is realistic for this scope (ballpark), and are there milestones/payments tied to specific approvals or deliveries? Options: <$50k, $50–150k, $150–300k, $300–600k, >$600k, Unsure

Risk radar — what's most likely to derail us and how would it feel?

• Name the single most realistic risk that would make this evaluation unusable or irrelevant.
• How long has that risk been present, and what attempts have been made to mitigate it? Options: New issue (last 3 months), Recent (3–12 months), Persistent (1+ year), Always been a concern, Unsure
• Which operational constraints would you be willing to invest in to reduce that risk (additional data collection, partner coordination, incentives, longer timeline)? Options: Additional data collection, Partner coordination meetings, Participant incentives, Extended timeline, Increased budget, Not willing to invest
• If we find major data integrity problems mid-study, what decision-rule should we use about whether to pause, redesign, or stop?
• Who on your team should be responsible for rapid escalation when issues arise (name or role)?

What winning looks like — beyond a report

• When this evaluation is done, what concrete change would make you say it was a clear success?
• Which success signals matter most (policy change, funding decision, program redesign, stakeholder buy-in, improved equity outcomes)? Options: Policy change, Funding renewal/increase, Program redesign/scale, Stakeholder buy-in/acceptance, Documented equity improvements, Other
• How should we measure follow-through on recommendations—who owns tracking and what cadence is realistic? Options: Quarterly progress reports, Semi-annual reviews, Annual check-ins, No tracking planned, Other
• What level of transparency do you want around interim findings during fieldwork (regular check-ins, embargoed memos, public pre-briefs)? Options: Weekly/biweekly check-ins, Milestone-based memos, Pre-briefs with select stakeholders, Hold until final report, Other
• Who needs to be convinced of the findings for action to follow—list roles and what would persuade each of them.

Small, fast commitments that show forward motion

• What is the smallest, least controversial commitment you could make today to get this work started? Options: Share contacts for data owners, Approve a short scoping budget, Designate a point person, Agree to a high-level timeline, None of the above
• Who will sign off on a mutual scope and budget (name or role), and what timeline do they need for review?
• Which documents or agreements must be in place before fieldwork (IRB, data sharing agreement, partner MOUs)? Options: IRB approval, Data Sharing Agreement (DSA), Partner Memoranda of Understanding, Independence/safeguard agreement, None/unsure
• Please provide the contact information (name, role, email) for the person who manages legal/data agreements.
• Realistically, when could you commit to a scoping call to finalize design options? Options: Within 1 week, Within 2–4 weeks, Within 1–2 months, Longer than 2 months, Unsure