Industrial & Manufacturing
›
Energy, Utilities & Sustainability
›
Utility Regulation & Rate Cases
Utility Performance Rates
Long-cycle programs where regulation, capital, and grid reliability define the pace.
Navigant (Guidehouse)
Brattle Group
Analysis Group
Concentric Energy Advisors
Inside this journey
-
Pre-Discovery
Align the room on outcomes, decision process, and constraints before deeper discovery.
-
Stakeholder Alignment
Confirm decision roles, timelines, political sensitivities, and what ‘good’ looks like for the utility, commission staff, and intervenors.
Alignment Questions
Start Here: Who's In Our Conversation?
- Which organization best describes your role in this effort?
- What's your job title or functional area?
- What brought you to explore performance-based regulation right now? (pick the top two)
- How soon do you expect a decision or formal filing timeline to materialize?
- What do you want to get out of today's discovery conversation? (one concrete outcome)
- Who else should be at the table for follow-ups (by function/team)?
Who Really Holds the Keys—and Why It Matters
- If the PBR conversation becomes politically heated, who do you think will face the most scrutiny—and why?
- Which parties have formal decision authority in your process?
- Who wields informal influence (staff leaders, commissioners, external advocates) even if they don't have a formal vote?
- How would you describe the political sensitivity of this topic inside your organization?
- Has your organization faced internal pushback on past regulatory initiatives? If yes, what typically triggered it?
- What timeline constraints from external stakeholders (commission schedules, legislative windows, elections) must we design around?
How Does Today Limit Tomorrow? (The Data & System Reality)
- What if the performance metrics you’re asked to track can’t be produced reliably from your existing systems—what happens next?
- Which core systems currently hold the data you'd need for common PBR metrics?
- How complete and auditable is historical performance data for reliability, customer satisfaction, and cost-efficiency?
- Who currently owns and signs off on the data that would feed performance reports?
- Describe any recent examples where data quality or reporting gaps affected a regulatory filing or stakeholder discussion.
- What level of effort are you willing to commit to close data gaps before a filing? (staff hours, budget, timeline)
When Targets Move, What Breaks? (Unintended Consequences & Risk)
- Which unintended outcome worries you most if a PBR metric is poorly designed?
- Have you seen an example—inside your utility or elsewhere—where incentives produced perverse results? Tell the story briefly.
- What financial exposure is the organization willing to accept for performance incentives or penalties as a percent of revenue?
- Which operational functions would likely need changes if metrics shifted behavior (select all that apply)?
- How do you currently monitor and mitigate risks tied to new regulatory mechanisms?
- What would be an acceptable escalation path and governance cadence to catch and correct unintended consequences early?
Who Needs to Say 'This Works'?
- Which stakeholder’s approval would be the hardest to secure—and what concerns would they raise first?
- For each of these groups—utility leadership, commission staff, and intervenors—what does ‘success’ look like to them?
- What evidence or analyses would convince commission staff that a metric is robust and fair?
- What are non-negotiable acceptance criteria for the utility (e.g., minimal earnings risk, implementation feasibility)?
- How important is transparency versus simplicity for stakeholder acceptance?
- Are there specific external comparators or precedents you want to mirror (states/utilities/commissions)? If so, list them.
Are Our Measures Truly Measuring What Matters?
- What if a high score on a metric turns out to be luck—how would you detect and correct that?
- Which performance dimensions do you consider core to customer outcomes (select top three)?
- Which external factors must be controlled for in metric design (weather, economic activity, DER penetration, fuel prices)?
- How comfortable are you with metrics that require complex normalization or adjustment formulas?
- What trade-offs between precision and stakeholder understandability would you accept?
- Describe one metric you think must be included and one you think should be avoided—why for each.
If We Could Snap Our Fingers: What Would Success Feel Like?
- Imagine a PBR your leadership would champion—what is the single visible outcome that would make them proud?
- Which of these outcomes would you prioritize if you had to pick one?
- What realistic trade-offs would you accept to achieve that outcome (e.g., smaller incentives, phased implementation, limited metric scope)?
- How would success be tracked in year one versus years two and three? What cadence and format would satisfy stakeholders?
- What lessons from past regulatory wins or losses would you want us to apply immediately?
- On a scale from 1–10, how important is piloting before committing to a full-plan filing?
What Would Make You Take Action Now?
- What single missing thing—data, internal alignment, budget, legal comfort—would trigger you to start a formal engagement today?
- What budget or resource range could you realistically allocate for a first-phase PBR design effort?
- Who would be the day-to-day project owner from your side and who needs to be informed/consulted?
- What governance cadence would make you comfortable (steering committee, monthly checkpoints, ad hoc escalations)?
- What are the non-negotiable go/no-go criteria you would use before filing a proposed PBR?
- Would you be open to a small pilot or demonstrated proof-of-concept before committing to a full filing?
-
Current State Mapping
Document historical performance, prior orders, data systems, reporting gaps, and operational constraints that will shape metric design.
Current State
Quick Snapshot — Who We’re Working With
- Tell us the jurisdiction and utility type we’ll be mapping (state/province and investor-owned, municipal, co-op, etc.).
- Which internal teams will be partners on this work (select all that apply)?
- What is the primary business goal that motivated you to explore performance-based regulation now?
- Who in your organization will be the day‑to‑day contact for data and modeling access, and how senior are they?
- Briefly describe your preferred target timeline for initial modeling, filing, and any pilot (months or quarter).
If the Data Could Talk, Would You Trust It?
- Where do you suspect the single largest data reliability risk lives today?
- Which source systems currently feed your performance reporting (select all that apply)?
- Who formally owns the datasets we would need for metric design and modeling (role/title)?
- How often do you run reconciliations between source systems and published metrics?
- Tell us about a time data surprised you—what was revealed and what did that make you change?
What the Numbers Say — And the Story They’re Not Telling
- If you had to be blunt: which published metric consistently misrepresents operational reality?
- How many years of cleaned, auditable historical performance data are readily available for modeling?
- Which performance dimensions does management treat as reliable versus aspirational (list specific metrics and why)?
- What known external factors routinely drive noise in your measures (weather, fuel prices, DERs, policy changes, customer behavior)?
- Describe one historical event or period that would skew baseline targets if not accounted for, and how long its effect lasted.
Reading the Orders: What Commissions Keep Coming Back To
- What commission directives or prior orders most constrain the way metrics can be defined or used?
- When was the last rate case or order that explicitly addressed PBR or incentive structure in your jurisdiction?
- Which elements of past testimony or filings drew the strongest pushback from commission staff or intervenors?
- Are there legacy compliance obligations or unresolved docket items we need to model or avoid triggering?
- Share an example where a prior order produced an unintended operational consequence—what happened and who bore the cost?
Where Reporting Breaks Down — Tell Us the Awkward Truth
- Which audiences complain most about current reports—internal execs, commission staff, intervenors, or others?
- How regularly do you receive questions or data challenge requests after you publish a performance report?
- In plain terms: what’s the single most common correction or rework you have to do post‑reporting?
- Do your current reports include clear audit trails and version histories that would satisfy commission review?
- Describe a recent reporting episode that eroded stakeholder confidence and how long it took to restore trust.
Operational Realities That Will Drive Metric Design
- If we designed a metric that required a daily operational change, would your crews be able to deliver it?
- What are your current staffing or capacity bottlenecks that would limit data collection or new reporting?
- Are there planned capital projects or system upgrades that will materially change baselines within the next 1–3 years?
- How quickly can operations change documented processes when a new performance metric requires it?
- Give an example of an operational trade‑off you’d be unwilling to make for better scores (safety, reliability, cost, customer service).
Where Politics and Stakeholders Bend the Numbers
- Which stakeholder positions would force you to redesign or drop a proposed metric?
- How much public transparency are you willing to commit to for early pilot results (full public release, redacted, internal only)?
- Describe a past stakeholder engagement that changed the design of a metric—what convinced you to change course?
- Which external groups do you expect to be the most skeptical about PBR-related metrics in your next proceeding?
- How important is political acceptability versus technical precision when choosing a metric (rank or explain)?
Data Governance & Trust — Who Signs Off and How?
- Do you have a formal data governance framework that covers definitions, ownership, and approval for published metrics?
- What role or committee currently approves changes to metrics or reporting logic?
- How long does it take to approve a change to a report or metric definition from proposal to publication?
- Are there external audit or third‑party validation requirements we must design for?
- If a metric was disputed after filing, what is your preferred remediation path (reopen, recalibrate, offer exceptions, other)?
Modeling Readiness — Could We Start Tomorrow?
- If we asked for anonymized, record‑level datasets (customer, outage, billing), how ready are those extracts today?
- Which modeling platforms or formats do you prefer we use or accept outputs in (select all that apply)?
- Are there legal, privacy, or contractual constraints on sharing data with consultants that we should know about?
- What is the shortest realistic timeline for us to receive production‑quality datasets for early scenario runs?
- List any specific fields or granularities that are mandatory for your analysis (examples: 15‑minute meter reads, per‑feeder outage logs, work order timestamps).
Fast Wins, Dealbreakers, and Red Lines
- What would count as a 'quick win' — a small change we could model that would materially improve confidence or buy‑in?
- Are there any absolute dealbreakers or metrics you refuse to accept under any design (e.g., direct customer penalties, certain reliability measures tied to weather)?
- What minimum reporting, governance, or audit features must be present for you to support a metric publicly?
- Who are the three people (roles/titles) whose approval would be required to move from modeling to filing?
- If we could only deliver one clear dataset or analysis in the first 30 days, which would be most useful to your internal decision makers?
-
-
Outcome Discovery
Define targeted regulatory outcomes, measurable success signals, stakeholder constraints, and acceptance criteria for proposed PBR mechanisms.
Discovery Questions
Setting the Table: Who’s Holding the Baton?
- Who in your organization is the primary sponsor for PBR work right now?
- What single outcome would your leadership point to if asked ‘why pursue PBR’?
- What timeline is leadership expecting for a filing or Commission decision on PBR?
- Who must be in the room to approve design choices (list by role/team)?
If Targets Were People: Which Demands Would Break the Room?
- Which regulatory outcome would be politically costly to get wrong for your utility?
- Which three outcomes should be prioritized in the proposed mechanism? (pick up to 3)
- Which outcome do you believe commissioners and commission staff will most scrutinize?
- How do intervenors (consumer advocates, environmental groups, large customers) currently frame ‘success’ in this jurisdiction?
- What trade-offs between prioritized outcomes are you most worried will be politically unacceptable?
Where the Numbers Meet Reality: What Can We Trust?
- Which of these metrics does your utility already track reliably?
- How would you rate the overall data readiness for PBR metric calculation?
- Which data quality issues are most common and how long have they persisted?
- Which normalization or adjustment levers do you expect to apply to metrics (choose all that apply)?
- What confidence threshold would you require before accepting a metric for regulatory use (e.g., statistical confidence, auditability)?
What ‘Good’ Actually Looks Like — Beyond the Headline Numbers
- Describe two specific, observable signs a commissioner would point to and say ‘this is working’ after year one.
- What’s an acceptable range of year-to-year variation for key metrics before it becomes politically contentious?
- Should targets be framed as stretch goals, achievable baselines, or a mix? Why?
- Who beyond leadership must sign off on acceptance criteria and what would be their likely non-negotiable conditions?
- Which non-financial signals (e.g., transparency, reporting cadence, stakeholder engagement) are essential to make outcomes credible?
Who’s Likely to Push Back — And What Will Satisfy Them?
- Which external stakeholder groups do you expect to be most skeptical of the proposed PBR design?
- What are the top three concerns those stakeholders will raise in public comment or hearings?
- What forms of evidence have historically reduced opposition in your jurisdiction?
- Are there stakeholders whose acceptance must be secured before filing? If so, who and what would they require?
- Has your utility previously faced a regulatory or stakeholder rejection of a performance mechanism? Briefly describe what occurred.
Draw the Worst-Case — Then Tell Us How You’d Fix It
- Which unintended consequences from PBR keep you up at night?
- If one of those outcomes happened in year one, which immediate corrective actions would you consider acceptable to propose to the Commission?
- What governance structure — internal and external — would make you comfortable that issues will be caught early and resolved?
- Operationally, how quickly could your teams implement corrective changes (people, processes, systems)?
- What transparency or audit provisions would you accept to reduce political risk while protecting sensitive information?
Money, Risk, and How We’ll Measure Trade-offs
- What percentage of allowed revenue do you consider an acceptable maximum exposure for at-risk incentives/penalties?
- What upside/downside range on earnings is politically tolerable for your leadership and board?
- Which modeling scenarios should we prioritize to demonstrate trade-offs to stakeholders?
- How do you believe risk should be shared between the utility and customers (e.g., symmetrical incentives, caps, sharing bands)?
- Which safeguards do you view as essential to include in the mechanism (select all that would be required for you)?
The Data-Workload Test: Can Your Team Deliver?
- Which internal teams would own ongoing metric calculation, verification, and reporting?
- How many full-time equivalents could realistically be allocated to ongoing PBR reporting and governance?
- What systems or data feeds must be integrated to produce timely, auditable metrics?
- Where do you expect to need external support (choose all that apply)?
- What is your realistic timeline for producing an initial validated metric report suitable for filing?
What Would Make You Say Yes Today?
- What are the non-negotiable criteria you need met before endorsing a proposed PBR mechanism?
- Which deliverables would you require from a consulting engagement to feel comfortable moving to filing?
- Who within your organization must sign the final approval to proceed with filing?
- What budget constraints or procurement steps could block an immediate engagement?
- How would you want the first year’s success communicated internally and externally to reduce political friction?
A Small Pilot to Defuse Risk — What Would It Take?
- If we proposed a limited pilot to demonstrate metric validity and political acceptability, what must it prove to you?
- What pilot scope would be acceptable — duration, geographic footprint, and customer sample size?
- Which stakeholders should be included in pilot oversight to ensure credibility?
- What go/no-go decision criteria would you want at pilot close (e.g., metric thresholds, data quality benchmarks, stakeholder support)?
- How quickly could you commit the people and budget needed to run an initial pilot?
-
Solution Experience
Use the customer’s data and scenarios to demonstrate how candidate metric sets and incentive designs produce measurable outcomes and trade-offs.
Experience Meetings
- Current State & Consequence Alignment
- Data Readiness & Scenario Setup
- Metric Candidate Modeling Workshop (Live)
- Sensitivity, Equity & Stakeholder Impact Review
- Confirm Preferred Metrics, Incentives & Next Steps
- Consulting team to run extended sensitivity ranges for flagged metrics and deliver updated outputs.
- Assign technical contact for automated data pulls or manual handoffs for the modeling team.
- Context Recap & Execution Rules
- Produce validated model outputs for baseline and each candidate metric/incentive combination.
- Quantify financial and performance trade-offs across candidate sets with visuals stakeholders can interpret.
- Secure stakeholder confirmations (or clear objections) on which metric behaviors reflect operational reality.
- Identify which candidates require sensitivity testing or alternative normalization.
- Consolidate modeled outputs, snapshots, and parameter sets for each candidate and circulate within 24 hours.
- Customer to confirm or correct any operational assumptions surfaced during validation.
- Consulting team to schedule targeted sensitivity tests for the items flagged during validation.
- Recap of Selected Candidates & Open Issues
- Identify metrics that are overly sensitive to exogenous factors and agree mitigation approaches.
- Quantify stakeholder-specific impacts and confirm whether results are politically acceptable.
- Agree on guardrails, deadbands, or normalization rules needed for regulatory robustness.
- Document remaining risks and the data or analysis required to close them.
- Introductions & Meeting Objectives
- Customer to review distributional impacts with legal/finance and provide commentary on political acceptability.
- Draft proposed mitigation language (deadbands, caps, normalization) for inclusion in the Solution Scope package.
- One-line Readbacks (Current, Consequence, Future)
- Obtain formal confirmation of the preferred metric set and incentive formulas to advance.
- Agree acceptance criteria and list of deliverables to be produced in the Solution Scope stage.
- Assign owners and deadlines for outstanding analysis, legal review, and stakeholder messaging.
- Schedule the Solution Scope kickoff and confirm required attendees and pre-reads.
- Stakeholders to provide written approval (email or e-sign) of the selected metric/incentive package.
- Consulting team to finalize the Solution Scope draft (metrics, formulas, scenarios, deliverables) and circulate within 3 business days.
- Assign owners for filings, data ops, and stakeholder communications and record them in the decision log.
- Schedule the Solution Scope kickoff meeting and distribute required pre-reads.
- Achieve single-sentence alignment on Current State that all stakeholders accept.
- Make the Consequence explicit and, where possible, quantified (cost, risk, timelines).
- Agree a one-sentence Future State outcome that the solution experience must demonstrate.
- Define the minimum model outputs and acceptance criteria required to prove the Future State.
- Circulate finalized one-sentence Current State, Consequence, and Future State to attendees.
- Customer to assign data owners and provide required datasets and data owner contacts within 3 business days.
- Consulting team to draft initial modeling scope and list of required scenarios tied to the Future State.
- Recap of Required Model Outputs
- Confirm the model-ready dataset and assign owners to remediate gaps.
- Agree explicit normalization rules to isolate utility performance from exogenous factors.
- Finalize the scenario matrix (baseline, upside, downside, sensitivity cases) with defined assumptions.
- Set deadlines and validators for data handoffs so modeling can proceed without ambiguity.
- Customer to deliver cleaned dataset extracts and updated data dictionary by agreed date.
- Consulting team to document and publish normalization rules and scenario definitions for sign-off.
- Readback: Current State (one sentence)
- Baseline Model Run
- Data Inventory & Definitions
- Recommended Metric Set & Incentive Formula
- Sensitivity Tests: Exogenous Factor Cases
- Data Gaps & Normalization Rules
- Candidate Metric Set A (Proof)
- Acceptance Criteria & Operational Implications
- Surface Consequence (explicit & quantified)
- Distributional Impact Analysis
- Scenario Matrix Agreement
- Candidate Metric Set B & C (Comparative Trade-offs)
- Equity & Affordability Checks
- Regulatory Framing & Stakeholder Messaging
- Define Future State (one sentence of operational outcome)
- Decision & Action Plan into Solution Scope
- Modeling Governance & Deliverables
- Tie Outputs Back to Problems
- Implications for Modeling & Scope
-
Solution Scope
Define selected metrics, incentive formulas, modeling scenarios, deliverables, responsibilities, and acceptance criteria for filings and implementation.
Scope Configuration
- Define metric calculations and reporting protocols
- Design incentive and penalty formulas
- Build financial impact and scenario model
- Draft regulatory testimony and exhibits
- Prepare regulatory filing attachments
- Develop data collection schema and ETL specs
- Implement automated performance dashboard
- Configure IT data pipelines for metrics
- Run performance risk and sensitivity simulations
- Draft multi-year rate plan tariff language
- Construct earnings-sharing mechanism calculations
- Produce first annual performance report
- Train utility staff on reporting and governance
Scope Questions
Define metric calculations and reporting protocols
- Are you selecting from standard industry metrics, creating custom metrics, or a hybrid approach?
- What baseline period should be used to set targets and baselines?
- Should metrics be normalized for weather, economic activity, or external factors?
- List the primary data sources that will feed metric calculations (e.g., AMI, SCADA, OMS, billing systems).
- Who within the utility will own calculation logic, updates, and sign-off (role or team)?
- What acceptance criteria should be used to validate metric calculations (e.g., statistical significance, stakeholder agreement)?
Design incentive and penalty formulas
- Should the mechanism provide rewards only, both rewards and penalties, or a different orientation?
- What is the desired incentive magnitude (expressed as % of revenue, ROE impact, or dollar cap)?
- Should the formula include deadbands, collars, or tiered bands to limit volatility?
- Which stakeholder protection features are required (e.g., affordability caps, intervenor review, phase-in)?
- Which financial flows should be affected by the incentive (base rates, trackers, reconciliation accounts)?
- What acceptance tests or review gates should the formula pass (e.g., stress test, legal review, simplicity threshold)?
Build financial impact and scenario model
- What modeling horizon is required for financial impacts (1, 3, 5 years or custom)?
- Which scenarios should be included by default?
- Do you require probabilistic methods (Monte Carlo) or deterministic scenario runs?
- Which outputs are required from the model (revenue requirement, customer bill impacts, earnings forecast, cash flow)?
- Describe current availability of financial inputs and assumptions (complete, partial, minimal).
- Who within the utility or advisors will validate key model assumptions and outputs?
Draft regulatory testimony and exhibits
- Which parties require tailored testimony or supporting exhibits (utility, commission staff, intervenors)?
- What level of technical detail do you want in testimony and exhibits?
- Is an independent expert witness or third-party validation required for testimony?
- Which exhibit types are expected (workpapers, model outputs, data tables, charts)?
- What is the filing timeline for testimony and exhibits (target filing date / lead time)?
- Who will be the sign-off authority for testimony (role/team)?
Prepare regulatory filing attachments
- Which attachments are required for the filing package?
- Which file formats are preferred for attachments?
- Are there confidentiality or redaction requirements (public vs. confidential versions)?
- Do attachments require audit trails, version control, or signed affidavits?
- Who will assemble, QA, and submit the attachments (roles or vendor)?
- What acceptance criteria should attach files meet (commission checklist, internal QA)?
Develop data collection schema and ETL specs
- Which existing systems will provide source data (AMI, SCADA, OMS, billing, HR)?
- What data frequency is required for each metric (real-time, daily, monthly, quarterly)?
- Who will be the owner(s) of the ETL pipeline and schema (team/role)?
- Which data quality rules are required (validation rules, completeness thresholds, outlier handling)?
- Preferred ETL tooling or tech stack (SQL jobs, Python pipelines, cloud ETL, vendor tools)?
- Is data lineage, governance documentation, and access control required as part of specs?
Implement automated performance dashboard
- Who are the primary dashboard users and audiences?
- What key visualizations and interactions are required (trend lines, drilldowns, maps)?
- What refresh cadence is required for the dashboard (real-time, daily, monthly)?
- What access controls and sharing model are needed (internal only, external stakeholder access, public portal)?
- Which dashboard platform or preference do you have (Tableau, Power BI, Looker, custom)?
- What acceptance criteria must the dashboard meet (UAT signoff, accessibility, performance SLAs)?
Configure IT data pipelines for metrics
- Will pipelines be hosted on-premises, in the cloud, or hybrid?
- Estimate expected data volumes and velocity for metrics (low/medium/high).
- Are there specific security or compliance requirements (NERC CIP, state privacy, ISO)?
- What SLAs for data latency and availability are required?
- Should data pipelines integrate with existing CI/CD and monitoring systems?
- Identify the IT stakeholders or teams responsible for pipeline implementation and operations.
Run performance risk and sensitivity simulations
- Which risk drivers should be prioritized in simulations (weather, DER, load growth, policy, data error)?
- How many sensitivity runs or scenario permutations are desired (low/medium/high)?
- Do you want full Monte Carlo probabilistic outputs, scenario-tree analysis, or deterministic stress cases?
- What reporting outputs are required from simulations (probability distributions, worst-case, regulatory risk metrics)?
- What tolerance for model complexity do stakeholders accept (simple transparent models vs. complex but precise)?
- Who will review and approve simulation assumptions and results (roles or external reviewers)?
Draft multi-year rate plan tariff language
- What multi-year plan length is desired?
- Which rate-adjustment mechanisms should be included (annual true-up, formula rates, trackers)?
- Is legal review and tariff redline by counsel required before filing?
- Do you need a crosswalk tying new tariff language to existing tariff sections and schedules?
- What stakeholder engagement approach is planned for tariff negotiation (workshops, settlement talks, standard filing)?
- Who will draft, review, and sign tariff language within the organization?
-
Mutual Commit
Finalize commercial terms, governance cadence, filing responsibilities, and escalation paths tied to milestone deliverables.
Agreement Modules
- Non-Disclosure Agreement (NDA)
- Master Services Agreement (MSA)
- Statement of Work (SOW)
- Commercial Terms & Payment Schedule
- Governance & Escalation Plan
- Filing Responsibilities & Regulatory Deliverables Schedule
- Data Access & Security Agreement (DPA)
- Acceptance Criteria & Success Metrics
- Risk Allocation & Indemnity Terms
- Change Order & Scope Amendment Process
- Implementation RACI & Operational Responsibilities
- Performance Assurance & Financial Reconciliation
- Approval Signatures & Authorized Filers
- Transition, Closeout & Recalibration Plan
-
Deployment
Operationalize rollout with readiness checks, enablement, and outcome validation.
-
Pre-Deployment Readiness
Confirm data pipelines, owners, access, reporting templates, and risk controls needed to support modeling and regulatory filings.
Readiness Questions
Quick Snapshot: Where We Stand Today
- Briefly describe the program or filing we're preparing, the current target filing date, and the primary outcome you need from this deployment.
- How ready is your underlying data for modeling and filing?
- Which performance metrics (by name) must be demonstrably supported by data for the filing to be credible?
- Which tools or platforms are you currently using for modeling and reporting?
- What is your internal timeline to produce the first complete model run for review?
- What concerns do you already have about meeting the filing deadline or regulatory expectations?
Who Really Owns the Data?
- If we had to pull every dataset tomorrow, which internal groups or third parties would push back—and what would they say?
- Which teams currently hold the primary data sources we’ll need?
- What describes your data governance posture today?
- What typical permission, security, or legal hurdles slow dataset access?
- How long does it typically take to obtain an approved extract or formal access to a new dataset?
- When owners decline or delay sharing, what is their most common reason (capacity, risk, unclear benefit, etc.)?
What's Failing Behind the Scenes?
- What recurring data or reporting failures have previously undermined confidence in filings or testimony?
- How often do you encounter errors or inconsistencies in the datasets used for regulatory analysis?
- Have you had post‑submission corrections or challenges in past filings?
- Which root causes best explain past problems?
- How are anomalies and outliers currently detected and resolved?
- Tell us about the most recent instance where a report was questioned: what happened, who raised it, and what was the downstream impact?
If the Regulator Asked Today...
- If the commission demanded your metric definitions, source code, and raw data this afternoon, would you be ready to hand them over?
- What documentation exists for each metric (data dictionary, transformation logic, acceptance criteria)?
- Do you have end‑to‑end data lineage and version control for key datasets and models?
- How do you record and communicate changes to metric calculations or baselines over time?
- How confident are you that external stakeholders (commission staff, intervenors) would accept current evidence without demanding further validation?
- What would be the immediate operational or reputational impact if regulators asked for a public data dump of the filing inputs?
What Would No‑Surprises Look Like?
- Imagine a filing that produced zero unexpected follow‑up requests—what invisible systems and controls had to be in place for that to happen?
- Which reporting cadence would meaningfully reduce risk for stakeholders?
- In what formats should evidence be delivered to minimize friction with commission staff and intervenors?
- Which controls must be in place before submission (pick non‑negotiables)?
- How would you prefer residual risk and uncertainty be presented to commissioners (narrative, quantitative sensitivity, RAG status, etc.)?
- What monitoring or alerting would make you feel comfortable after the filing is accepted?
Bridging People, Tools, and Process
- If an operational team says 'that’s not our job,' who will actually close the gap and how?
- Which function should lead pre‑deployment readiness?
- What governance cadence would keep momentum without overburdening teams?
- What resources (people, tools, budget) would close your top three gaps?
- What handoffs and training will operations, finance, or reporting teams need to own ongoing monthly/quarterly reports?
- Which legacy systems must remain connected to the reporting chain?
Ready or Not—Next Moves
- If we were asked to produce a formal 'readiness certificate' in two weeks, what would clearly fail and what would clearly pass?
- What are the single biggest blockers we should address first?
- Which mitigation activities could realistically be completed within two weeks?
- What level of residual risk is acceptable to proceed to filing (choose the best fit)?
- Who must sign the final go/no‑go for deployment?
- What would you like our consulting team to do first to move readiness forward (top three priorities, in order)?
-
Deployment Enablement
Execute modeling, prepare testimony and stakeholder materials, schedule filings, and coordinate operational changes with clear owners.
-
Validation Checklist
Run sensitivity tests, validate assumptions, confirm reporting processes, and document readiness for submission and first-year tracking.
Validation Questions
Getting Comfortable — Quick Introductions
- Who are you on this project (role and primary decision authority)?
- What prompted you to explore performance-based regulation now?
- What’s your target timeline for filing or resolving a new framework?
- Which outcomes matter most to you right now (select up to 3)?
- In one sentence, what feels most urgent about this work?
What If Your Metrics Are Rewarding The Wrong Behavior?
- Which performance metrics are currently part of your regulatory conversations, and why were they chosen?
- Where do you believe the current metrics fail to distinguish between utility action and external factors?
- Give an example where a metric produced an outcome you didn’t expect—what happened and who pushed back?
- How would stakeholders describe the fairness of current incentives—do they see winners and losers?
- If we kept your current metric set but tightened targets, what unintended consequences should we anticipate?
If You Had to Name the Single Thing That Derails Progress, What Is It?
- Thinking back to prior rate cases or reforms, what single issue most often delayed or derailed agreement?
- How often have settlement talks broken down over metric definitions or incentives?
- Which internal group typically raises the loudest objections—finance, operations, legal, or executives—and why?
- When things get political, what do you notice happens to technical proposals (tone, timing, or scope)?
- How long has that sticking point been a recurring theme for your team?
How Confident Are You That Your Data Would Survive Cross-Examination?
- Which systems house the primary data we’d need to model metrics and incentives?
- How complete and auditable is that data—would you call it 'commission-ready' today?
- What specific reporting gaps or quality issues have caused the most pain in past proceedings?
- Who owns the data end-to-end (team or title), and how quickly can they produce historical extracts?
- Would you be willing to provide sample datasets under a confidentiality arrangement for preliminary modeling?
- Describe one instance where data issues changed the outcome of a regulatory discussion—what did you learn?
Imagine The Commission Loved The Package — What Did You Give Up For That To Happen?
- What measurable signals would convince you and the commission that the mechanism is succeeding in year one?
- What level of financial upside or downside is politically acceptable versus a deal-breaker?
- What compromises on metric granularity or frequency would you accept to gain faster implementation?
- Who must publicly endorse the package for it to be durable (titles, organizations)?
- If success required trade-offs between cost and reliability, which direction would your leadership lean?
Who Wins—and Who Loses—When This Works?
- Which external stakeholders are most likely to oppose a proposed mechanism, and why?
- Who inside your organization would need the most convincing, and what keeps them from being early champions?
- Have you previously negotiated compromises that brought opponents on board? Share an example and the tactic that worked.
- How comfortable are you with a public-facing narrative that links incentives to customer outcomes?
- What channels would you prefer for stakeholder engagement—written briefs, workshops, joint modeling sessions, or public hearings?
If We Proposed Operational Changes, Would You Lean In or Push Back?
- How much operational change are you willing to accept to support credible metrics—minor tweaks, moderate process changes, or major system upgrades?
- What internal capabilities do you have today for ongoing performance management (owners, cadence, tools)?
- What budget or resource constraints would most limit your ability to implement recommended changes?
- Who would be the operational owner for implementing data pipelines and reporting (title/team)?
- How do you want governance to look—monthly scorecards, quarterly executive reviews, or something else?
- If implementation required temporary manual reporting while systems are upgraded, would you accept that trade-off?
What Would Make You Feel Safe To Move Forward?
- What’s the smallest deliverable from us that would meaningfully de-risk the next decision for you?
- Who signs off on pursuing formal filings—what’s the internal approval path and expected timeline?
- Do you require external validations (e.g., independent auditor, commission staff review) before filing?
- What legal or compliance checks must occur before we can share modeling publicly?
- Realistically, when could you commit to an internal kickoff if you saw the right evidence?
- Any final concerns or constraints you want us to know before we build a tailored proposal?
-
-
Success
Review initial performance versus targets, capture lessons learned, and maintain a shared channel for issues, recalibration, and continuous improvement.
Success Reviews
- Initial Performance Review (Customer Data)
- Lessons Learned & Process Improvement Workshop
- Metric Recalibration Decision Session
- Governance, Reporting & Escalation Cadence
- Continuous Improvement Roadmap & Annual Planning
Issues & Enhancements
- Document triage and escalation procedures so incidents are resolved within agreed timelines.
- Capture a prioritized list of lessons and improvement initiatives with clear owners.
- Identify at least 2 quick-win changes that can be implemented within 60 days to improve measurement or lower risk.
- Agree a documentation and distribution plan so lessons inform future filings and operations.
- Create a prioritized improvement backlog with descriptions, owners, estimated effort, and target dates.
- Implement at least one quick-win data or process change within 30 days and report completion.
- Publish a 'Lessons Learned' brief to the shared channel and regulatory stakeholders.
- Goals & Decision Criteria
- Select and document a recalibration approach (or confirm no change) for each metric under review.
- Ensure every selected change is backed by modeled evidence and an agreed public rationale for transparency.
- Assign drafting and filing responsibilities with clear deadlines.
- Produce a Decision Memorandum summarizing chosen recalibration(s), evidence, modeled impacts, and recommended filing language.
- Update metric specification documents and the public-facing metric guide to reflect approved changes.
- Notify commission staff and core intervenors of the intent to amend (if applicable) and share the Decision Memorandum.
- Review Proposed Governance Model
- Stand up a shared channel with clear access rules and response SLAs.
- Agree reporting cadence and standard templates so all parties receive consistent, timely information.
- Opening & Objectives
- Confirm owners and RACI for ongoing operations and change approvals.
- Provision the shared channel (e.g., Teams/Slack workspace or ticketing queue) and assign channel admins.
- Publish reporting templates and schedule the recurring reporting meetings.
- Create an incident response playbook with severity definitions and contact lists.
- Recap Priorities from Workshop & Decisions
- Agree a sequenced, funded 6–12 month roadmap with owners and milestones.
- Secure resource commitments or identify funding paths for key initiatives.
- Establish calendarized validation and recalibration checkpoints for the coming year.
- Finalize and distribute the Continuous Improvement Roadmap with Gantt-style milestones and owners.
- Obtain any required budget approvals or MOU signatures for prioritized work.
- Schedule the next validation review and add recurring governance meetings to calendars.
- Establish a single, agreed factual account of observed performance vs targets.
- Identify the top variance drivers with evidence and agree on which require further investigation.
- Make the financial and regulatory consequences explicit so urgency is aligned across stakeholders.
- Secure owners and timelines for required deep-dive analyses.
- Produce a concise variance report (one-page executive summary + appendices) documenting drivers, assumptions, and evidence.
- Assign owners for 2–3 deep-dive analyses (data quality, operational cause, financial impact) with due dates.
- Circulate validated dashboard snapshots and meeting minutes within 48 hours.
- Recap of Findings (5 mins)
- Evidence Recap (one slide per metric)
- Breakout: What Worked / What Didn't
- Define Roadmap Themes & Milestones
- Agree Reporting Cadence & Templates
- Confirm Current State (one-sentence)
- Data & Systems Retrospective
- Recalibration Options Presented
- Shared Channel Protocols
- Resource & Budget Alignment
- Performance vs Targets (Dashboard Walkthrough)
- Variance Drivers & Root Causes
- Modeled Financial & Operational Impacts
- Schedule Validation & Recalibration Checkpoints
- Issue Triage & Escalation Paths
- Process & Governance Retrospective
- Financial & Regulatory Consequence Assessment
- Roles, RACI & Approvals
- Success Metrics & Acceptance Criteria
- Prioritization Exercise (Impact x Effort)
- Stakeholder Risk & Acceptability Assessment
- Validation Check — Is the Diagnosis Correct?
- Decision & Documentation
- Pilot Period & Success Criteria