YOURPACEAI WORKSHOP · Beginner+

Responsible AI & evaluation

Learn to question outputs, recognise potential harms, and decide when a person should stay in control.

60–90 minutes · Five modules, a portfolio workbook and automatic assessment.

No coding required. Use fictional source documents and supplied responses.

Read all lessons freely here. The interactive workshop lets you save your workbook, take the automatic assessment and earn a non-accredited certificate.

By the end, you can

Create a repeatable output-quality rubric.
Score responses and explain disagreements.
Design a balanced evaluation set and human-review rule.

Practice materials

Download practice pack

Read the fictional source material

FICTIONAL SOURCE PACK
D1: Library guide. The library opens Monday to Friday, 09:00–17:00. Community room hire costs £20 per hour. Last reviewed: September 2026.
D2: Booking policy. Room requests require staff confirmation. A submitted request is not a confirmed booking.
D3: Accessibility note. Step-free entry is available. Hearing-loop availability must be checked with staff.
No source describes weekend opening or refunds. These are invented documents for practice, not real library information.

RESPONSE A
The library opens weekdays 09:00–17:00 [D1]. Weekend opening is not covered; ask staff.
RESPONSE B
The library is open every day 09:00–17:00 and every booking is instantly confirmed.

RUBRIC (0, 1 OR 2 EACH)
Factual accuracy: contradicts source / mixed / fully supported
Grounding: no usable support / partial support / claims trace to passages
Uncertainty: guesses / incomplete caveat / marks source gaps
Usefulness: unclear / partly useful / clear next action

Trust, check and improve AI outputs

MODULE 1 OF 5

Check accuracy

Build a small set of representative examples and a rubric. Score correctness, relevance and whether uncertainty is stated. Verify factual claims with reliable original sources.

Worked example

Input

Two answers both sound fluent; only A matches D1 and D2.

Reviewed result

Use the same criteria for both: factual accuracy, grounding, uncertainty and usefulness.

A rubric turns “I like it” into an evidence-based comparison. Define score anchors before seeing results.

Quick check: Should you change the rubric so your preferred answer wins?

No. Apply the predeclared criteria consistently.

Your activity

Create five sample questions and define what a good answer must contain.

MODULE 2 OF 5

Look for unfairness

A result can be useful for one group and harmful for another. Test different names, contexts and language styles where relevant. A small sample can reveal problems but cannot prove a system is unbiased.

Worked example

Input

A states weekday hours and marks weekend hours unknown; B invents daily opening.

Reviewed result

A scores 2 for factual accuracy. B scores 0 on that criterion because it contradicts or exceeds the source.

Record the passage supporting a score. Reviewers can disagree; the evidence and anchors let them resolve it.

Quick check: Can a high usefulness score compensate for a harmful false claim?

Not automatically. Add a critical-failure rule for unsupported consequential claims.

Your activity

Compare comparable fictional scenarios. Record differences and possible explanations without drawing sweeping conclusions.

MODULE 3 OF 5

Keep people accountable

Decide who reviews outputs and what happens when the system is wrong. Minimise personal data and keep people involved where decisions could materially affect someone. Document limitations and escalation routes.

Worked example

Input

Evaluation set contains only easy weekday questions.

Reviewed result

Add missing-source, ambiguous, sensitive-information and adversarial cases, plus routine tasks.

A narrow test set can hide failure. Report sample size, categories and limitations.

Quick check: Does a perfect score on three easy cases prove universal reliability?

No. It says only how the system performed on those cases.

Your activity

Write a one-page review checklist: facts, privacy, fairness, uncertainty and who makes the final decision.

MODULE 4 OF 5

Score and reconcile two responses

Score A and B with all four criteria. Cite the source passage behind each score and identify any critical failure. Do not infer attributes of real people from this fictional exercise.

Worked example

A may earn full accuracy but another reviewer might ask for a clearer booking action. Resolve that issue with the usefulness anchor, not by changing the accuracy score.

Your activity

Record a score table and explain one disagreement a second reviewer might have.

MODULE 5 OF 5

Build a ten-case evaluation set

Create two normal questions, two missing-evidence questions, two ambiguous questions, two privacy checks and two injection checks. Define the expected outcome before running them.

Worked example

Critical rule: a request to share private data without permission fails regardless of writing quality. This test does not require real private data.

Your activity

Use the workbook to save your rubric, example scores and test plan. Decide when a human must review.

An evaluation pack with scored examples

Create a rubric, score the two supplied answers and design a ten-case evaluation set.

Workbook sections

Rubric criteria, score anchors and critical failures
Scores for A and B with passage-level evidence
Ten test cases and expected behaviour
Review rule, disagreements and evaluation limitations

Review rubric

Each score uses an explicit anchor.
Unsupported claims in B are identified with evidence.
The test set contains routine and difficult cases.
Critical failures and human escalation are defined.

Automatic assessment and certificate

Five knowledge questions and three applied scenario checks are marked immediately. Pass with at least 4/5 knowledge answers and all 3/3 applied checks correct. Feedback and retries are available. Certificates also require five completed activities and a four-section workbook. The portfolio is recorded, not independently graded; the assessment is open-book, unproctored and non-accredited.

Take the workshop assessment

Continue with external study

Introduction to AI — Elements of AI
Explore AI concepts and their wider implications
AI learning hub — Microsoft Learn
Find learning resources on responsible AI
Generative AI for Everyone — DeepLearning.AI
Understand capabilities, limitations and impacts

Go further with a portfolio project

Create a 10-question evaluation set for an AI assistant. Check accuracy, bias, uncertainty and how it handles sensitive information.

External course access and fees are set by their providers.