Decision making under uncertainty with structured expert judgment
Learn how risk, expected utility and structured expert judgment fit together, with a worked example and a way to weigh experts against your own criteria.
Contents
- Risk and uncertainty are not the same
- How decision theory and expected utility frame a choice
- Your attitude towards risk changes the answer
- What structured expert judgment adds to the model
- A worked example of a decision under uncertainty
- Balancing expert opinion with your own criteria
- How electricity markets handle uncertain prices and demand
- Books, courses and PDF sources for further study
- Questions about expert judgment under uncertainty
- What does an everyday uncertain decision look like?
- What is the difference between risk and uncertainty in decisions?
- How do you know which expert to trust most?
- Can expert judgment replace my own criteria?
- Is structured expert judgment worth the effort for small decisions?
- Do I need advanced math to apply these methods?
Decision making under uncertainty with structured expert judgment means turning expert knowledge into calibrated probabilities before you choose. Experts are scored on how well their estimates match known answers, and their views are weighted accordingly. The weighted probabilities then feed a decision model alongside your own criteria and your tolerance for risk.
- Under risk the odds are known, while under uncertainty you must estimate them.
- Structured expert judgment weights experts by tested accuracy, not by seniority or confidence.
- Expected utility combines probabilities with how much each outcome matters to you.
- Your attitude toward risk can change which option ranks first.
- Keep expert probabilities and your own criteria in separate columns before combining them.
Risk and uncertainty are not the same
Uncertainty in decision making means you must choose before you know which outcome will happen. Economists separate two cases that look similar but call for different tools. Under risk, the probabilities of each outcome are known or can be measured from data. Under uncertainty, those probabilities are unknown and must be estimated or argued for.
A dice game is a clean case of risk, since every face has a known chance. A new product launch is closer to uncertainty, because no record tells you how buyers will respond. Real decisions mix both, with some quantities measured well and others barely known. Sorting the known parts from the unknown ones opens structured decision making, a stepwise way to compare options.
Decisions under uncertainty still deserve numbers, even when the numbers are judgments rather than measurements. A probability you state openly can be checked, challenged and updated as evidence arrives. A vague feeling of 'likely' cannot be tested, and different readers will hear it differently. Writing estimates down is what makes expert input usable later.
How decision theory and expected utility frame a choice
Decision theory gives a formal way to choose when outcomes are uncertain. The core idea is expected utility, which weighs each outcome by its probability and its value to you. You list the options, the possible outcomes of each, and how likely each outcome is. Then you score how much each outcome matters and pick the option with the highest weighted total.
Utility differs from money because the same gain can mean more or less to different people. Losing a large sum may hurt a small firm far more than an equal gain would help. Expected utility captures that imbalance, while a plain average of payoffs hides it. The distinction matters whenever one outcome could be severe or hard to reverse.
Moving from theory to application means choosing a model that a person or computer can solve. Decision trees suit a few sequential choices, while Markov decision processes handle long chains of actions. A Markov decision process is a model where each action moves the system between states with set probabilities. Such models appear in aircraft collision avoidance, robotics, medical planning and energy operations.
Your attitude towards risk changes the answer
Attitude toward risk describes how much certainty you will trade for a chance at a better result. A risk-averse decision maker prefers a sure outcome over a gamble with the same average value. A risk-neutral one cares only about the average, and a risk-seeking one prefers the gamble. Decision theory encodes these attitudes in the shape of the utility curve.
Two people with identical probabilities can still reach opposite choices. The difference comes from their risk attitudes, not from their beliefs about outcomes. Keeping beliefs and attitudes apart lets a group see whether it disagrees on facts or on values. Experts should supply the beliefs, while the person who bears the consequences sets the attitude.
Risk attitude can be estimated by asking which sure amount you would accept instead of a gamble. That sure amount is called the certainty equivalent of the gamble. A certainty equivalent well below the average payoff signals strong risk aversion. Repeating the question at several stakes shows whether your caution grows as amounts get larger.
What structured expert judgment adds to the model
Structured expert judgment is a formal method for turning several experts' views into one probability distribution. A probability distribution is a range of possible values with the chance attached to each part. The method treats experts as sources of evidence whose reliability can be measured, not assumed. Expert elicitation, the process of drawing out these estimates, follows a written protocol to limit bias.
Calibration questions are the core of the Classical Model developed by Roger Cooke. Experts estimate quantities whose true values the analyst knows but the experts do not. Each expert gets a calibration score for accuracy and an information score for how narrow the ranges are. Weighting then gives more influence to experts who proved both accurate and informative on those questions.
Informal expert advice suffers from traps that a structured process is designed to reduce. Confident experts can dominate a room even when their estimates are poorly calibrated. Group discussion can also pull everyone toward the first number spoken, an effect called anchoring. Separate written estimates and performance-based weights reduce both effects without silencing anyone.
- Define the uncertain quantities the decision depends on
- Select experts and brief them on the protocol
- Ask calibration questions with answers known to the analyst
- Collect each expert's estimates separately, as ranges
- Score and weight the experts on calibration results
- Combine weighted estimates into one distribution
- Feed that distribution into the decision model
A worked example of a decision under uncertainty
Consider a manufacturer choosing among three suppliers for a part with uncertain delivery reliability. No supplier has a long record, so the chance of serious delays is unknown. The firm invites several logistics experts to estimate delay probabilities for each supplier. Calibration questions about past shipments with known outcomes are used to score each expert first.
The weighted expert estimates then become inputs to a simple expected utility comparison. Supplier A has the lowest price but the widest range of possible delays. Supplier B costs more but shows a narrow, low delay range across the weighted estimates. Supplier C sits in between on both price and delay risk.
The ranking of the three suppliers depends on how severe a delay would be for the firm. A firm that can absorb late parts may favor Supplier A for its savings. A firm whose production line stops without the part may rank Supplier B first. The expert probabilities stay the same in both cases, and only the firm's values change.
| Supplier | Price position | Weighted delay estimate | Suits a buyer who |
|---|---|---|---|
| A | Lowest | Wide range, higher upper end | Can absorb late deliveries |
| B | Highest | Narrow range, low values | Cannot tolerate stoppages |
| C | Middle | Moderate range | Wants a balance of cost and reliability |
Balancing expert opinion with your own criteria
Expert input and personal criteria work together when each one has a clearly separate job. Experts are strong at estimating what is likely to happen in their field. You are the authority on what matters to you, such as cost, timing or comfort with risk. Mixing the two in one score hides whether a ranking comes from facts or from preferences.
A side-by-side table keeps expert estimates and your own criteria visible as separate columns. One set of columns holds the probabilities or ranges that the experts supplied. Another set holds your weights, such as how much you value price against reliability. When you combine them, you can trace every rank in the shortlist back to its source.
Overriding an expert estimate is reasonable when you hold information the experts did not see. Record that override as a separate note rather than quietly changing the number. Disagreeing with expert values is different, since your values are yours to set. A sensitivity check, which tests how results shift as inputs change, shows when the top option flips.
How electricity markets handle uncertain prices and demand
Electricity markets are a demanding test case because prices, demand and renewable output all shift unpredictably. Producers must commit to offers before they know the price that will clear the market. Retailers sign fixed-price deals with customers while buying power at prices that move. Large consumers weigh fixed contracts against spot prices, which are set for near-immediate delivery.
Stochastic programming is an optimization method that plans across many scenarios, each with a probability. A generator might model scenarios for wind output, fuel cost and demand together. The resulting plan performs reasonably across the scenarios rather than perfectly in just one. Conditional value at risk, the average loss in the worst scenarios, lets planners cap downside exposure.
Expert judgment fills gaps in electricity models where historical data says little about the future. New market rules, new technologies and extreme weather can make past prices a weak guide. Structured elicitation lets planners convert expert views on these shifts into scenario probabilities. The model then shows how sensitive each contract or investment is to those expert inputs.
Books, courses and PDF sources for further study
Books on uncertain decisions range from algorithmic textbooks to energy economics and expert elicitation. 'Decision Making Under Uncertainty: Theory and Application' by Mykel Kochenderfer focuses on computational methods for real systems. 'Decision Making Under Uncertainty in Electricity Markets' by Conejo, Carrión and Morales applies stochastic programming to power systems. Roger Cooke's 'Experts in Uncertainty' presents the Classical Model of structured expert judgment in depth.
University courses on uncertain decisions teach probability, utility theory, Markov decision processes and reinforcement learning. Reinforcement learning is a method where a system learns good actions from trial and feedback. Some universities publish lecture videos and slides openly, while others limit materials to enrolled students. Short professional courses on expert elicitation also exist, aimed at analysts in risk and policy work.
A PDF copy of a textbook is legitimate only when the publisher or author offers it. Some authors post free drafts or companion books on their own pages, so check there first. University libraries often provide electronic access to these titles for registered readers. Unauthorized PDF sites can carry outdated editions, missing chapters or malware.
Questions about expert judgment under uncertainty
What does an everyday uncertain decision look like?
An everyday uncertain decision is choosing whether to carry an umbrella on a cloudy morning. You must act on a rough sense of the chance of rain. That chance gets weighed against the nuisance of carrying it or the cost of getting wet.
What is the difference between risk and uncertainty in decisions?
Risk applies when outcome probabilities are known, and uncertainty applies when they must be estimated. Many real choices fall under uncertainty, which is where expert probability judgments become useful. Structured methods turn those judgments into numbers that a decision model can use.
How do you know which expert to trust most?
Trust goes to the experts who score well on calibration questions with known answers. Experts whose ranges captured the true values, while staying narrow, receive more weight. Seniority, title and confidence play no direct role in the weighting.
Can expert judgment replace my own criteria?
No, expert judgment covers beliefs about outcomes, not what those outcomes are worth to you. Your criteria and risk attitude set the values in the decision model. Keeping both visible lets you see which input drives the final ranking.
Is structured expert judgment worth the effort for small decisions?
Not always, since the full method suits decisions with high stakes and thin data. For smaller choices, asking a few experts for written ranges brings some of the same discipline. Even a light version reduces anchoring and lets you compare estimates side by side.
Do I need advanced math to apply these methods?
No, the basic logic needs only probabilities and a weighted sum of outcome values. Larger models, such as stochastic programs or Markov decision processes, call for software and training. For high-stakes engineering or financial decisions, involve a trained analyst to build and check the model.