Spidrr
← Plumbing resources

AI receptionist test worksheet

Test the calls that matter before you trust an AI receptionist with real customers.

Vendor demos tend to show ideal conditions. Real plumbing calls include interruptions, changing facts, unclear descriptions, requests for humans, urgent-sounding situations, and failed handoffs. Test those conditions with the outfit’s own rules before real customers enter the workflow.

The test question

Do not grade the demo. Grade the required call outcome.

The right question is not “Does the AI sound good?” It is “Can it reliably produce the call outcome your outfit requires?”

  • Do not judge an AI receptionist from a polished demo.
  • Do not judge it from one successful call.
  • Do not assume natural-sounding speech means operational reliability.
  • Test the actual jobs the system must perform.
  • Test failure paths, not just happy paths.

01 · Define the required outcome

Decide what the system is allowed and expected to do

A system cannot be fairly tested against requirements the outfit has never defined. Mark each job as required, optional, or prohibited before the first test call. If the underlying phone workflow is still unclear, start with thephone-system requirements worksheetor the answering-service buyer checklist.

Required-outcome worksheetMark: required / optional / prohibited
  • Answer only
  • Capture caller details
  • Determine service area
  • Apply service-fit questions
  • Schedule
  • Reschedule
  • Cancel
  • Escalate outfit-defined urgent calls
  • Reach an on-call person
  • Transfer to a human
  • Send notifications
  • Create records or write into a CRM or calendar

02 · Set up the test environment

Give the test system the outfit’s real operating rules

Document the same facts and permissions a production setup would use. Then use a test calendar, test records, or another controlled environment so the exercise does not alter real appointments or customer data.

  • Business name
  • Service area
  • Normal hours
  • After-hours policy
  • Services offered
  • Services not offered
  • Booking rules
  • Calendar availability
  • Pricing statements allowed or forbidden
  • Primary escalation contact
  • Backup escalation contact
  • Failed-handoff rule
  • Human-transfer rule
  • Caller disclosure language, if used
  • Notification destination
  • CRM, calendar, or integration path, if applicable

03 · Use one fixed record

Record the same evidence after every test call

Consistent fields make different runs comparable without inventing a universal score. Write the expected outcome before placing the call, then record what actually happened.

Test recordOne copy per run
Meets requirementPartly meets requirementDoes not meet requirementNot tested
  1. Scenario ID
  2. Date / time
  3. Coverage mode
  4. Caller objective
  5. Expected outcome
  6. What the AI understood
  7. Information captured correctly
  8. Information missed
  9. Incorrect facts introduced
  10. Booking result
  11. Escalation result
  12. Human-transfer result
  13. Notification result
  14. Record / integration result
  15. Unauthorized promise made?
  16. Unsafe instruction given?
  17. Caller had to repeat information?
  18. Final outcome
  19. Notes

04 · The 20-call test set

Run 20 fictional plumbing-call scenarios

These are fictional test scenarios—not real customer calls, vendor results, or evidence that any product passed. Use the outfit’s configured rules, real handoff destinations, and fixed test record for every run. Booking capability is unverified until the actual booking and downstream records have been inspected.

Group A

Routine intake

Test identity, callback number, location, service fit, and the next step.

  1. New customer with a straightforward service request

    Capture a usable intake and apply the configured next step without adding facts.

  2. Existing customer calling back

    Recognize the stated status and connect the call to the correct record or callback path.

  3. Caller outside the service area

    Apply the real service-area boundary without inventing an exception or referral.

  4. Request for a service the outfit does not provide

    State the service limit and avoid implying that the work can be accepted.

Inspect
  • Identity capture
  • Callback number
  • Service location
  • Service fit
  • Accurate next step

Group B

Scheduling

Verify the calendar action and every record it should create or change.

  1. Caller accepts an available appointment

    Create only the authorized appointment, then confirm the details actually written.

  2. Caller rejects the first offered time

    Offer only real alternatives and preserve the caller’s final choice.

  3. Caller asks to reschedule

    Move the correct appointment without leaving a duplicate or losing the original context.

  4. Caller asks to cancel

    Cancel the correct appointment and leave the required confirmation and record.

Inspect
  • Calendar accuracy
  • Availability
  • Authorization
  • Duplicate prevention
  • Confirmation
  • Downstream records

Group C

Real conversation friction

Change facts during the call and verify which version survives.

  1. Caller changes the address midway through the call

    Replace the first address everywhere and confirm the final service location.

  2. Caller corrects their phone number

    Preserve the corrected number in notifications, records, and any booking.

  3. Caller changes the problem description

    Use the final description while keeping any relevant uncertainty visible.

  4. Caller interrupts or gives information out of order

    Recover the required fields without looping, guessing, or silently dropping details.

Inspect
  • Facts updated correctly
  • Superseded facts discarded
  • Clarification requested
  • No invented facts
  • Final version preserved

Group D

Human requests and handoffs

Test the transfer policy, acknowledgement, fallback, and ownership after failure.

  1. Caller asks immediately for a human

    Follow the configured human-request rule and describe the outcome accurately.

  2. Caller asks for the owner or a specific employee

    Apply identity and availability rules without pretending the person is reachable.

  3. Transfer attempt fails

    Tell the caller what happened, start the fallback, and create a visible failed-handoff record.

  4. Primary escalation contact does not answer

    Use the configured backup path and record whether anyone accepted ownership.

Inspect
  • Transfer policy
  • Acknowledgement
  • Fallback
  • Backup escalation
  • Caller expectation
  • Failed-handoff record

Group E

After-hours and safety boundaries

Keep routine, outfit-defined urgent, and immediate-danger paths distinct.

  1. Routine request after hours

    Use the routine callback or scheduling policy without artificial urgency.

  2. Outfit-defined urgent plumbing call

    Apply the outfit’s written escalation rule without turning it into a diagnosis.

  3. Immediate-danger language

    Keep public emergency response separate from plumbing callback and escalation.

Inspect
  • Coverage policy
  • Urgency rule
  • No diagnosis
  • No hazard investigation
  • Correct response boundary

Group F

Disclosure and system boundary

Ask the system directly what it is and what it can do.

  1. Caller asks whether they are speaking with AI and what the system can do

    Answer consistently with policy and state its real authority and limits.

Inspect
  • Configured disclosure policy
  • Authority described accurately
  • No invented powers
  • No invented knowledge

05 · Add controlled variations

One clean run per scenario is not enough

Repeat the important scenarios with one controlled change at a time. Keep the expected outcome fixed so it is possible to see what the variation affected. Do not turn a small test set into a claimed error rate.

  1. Caller speaks quickly
  2. Caller speaks slowly
  3. Background noise is present
  4. Caller interrupts
  5. Caller gives two pieces of information at once
  6. Caller corrects a fact
  7. Caller refuses one requested field
  8. Caller asks an unexpected question
  9. Caller asks for a human
  10. Caller disconnects before completion

06 · Failure testing

Break the dependencies on purpose

A system that works only when every dependency works has not been fully tested. Trigger or simulate each failure in a controlled environment and inspect the caller experience, internal visibility, ownership, and fallback.

  1. Calendar unavailable
  2. Integration unavailable
  3. Notification fails
  4. Human transfer fails
  5. Primary escalation contact does not answer
  6. Backup contact does not answer
  7. Caller disconnects during handoff
  8. Duplicate call arrives
  9. System cannot determine service fit

07 · Booking and integration verification

“Integrates with” does not prove the required workflow works

Verify the actual calendar action, customer record, notifications, and failure visibility. A capability claim or connection screen is not evidence that the plumbing outfit’s appointment rules survive the full call.

Inspect after each booking testCompare the record with the call
  • Correct calendar
  • Correct appointment type
  • Correct duration
  • Correct availability
  • No double booking
  • Update and reschedule behavior
  • Cancellation behavior
  • Customer record created correctly
  • Phone number and address preserved
  • Notes transferred accurately
  • Notification sent
  • Integration failure visible

08 · Naturalness is not enough

Review conversation quality and operational outcome separately

A smooth voice can still capture the wrong address, overwrite the corrected phone number, misapply service-area rules, promise an unavailable time, fail to escalate, lose a handoff, or create an incorrect record. “Humanlike” is not the primary pass criterion.

Conversation-quality review

  • Speech is understandable
  • Pacing is appropriate
  • Interruptions are handled
  • Caller repetition is noted
  • Awkward loops are identified
  • Inappropriate certainty is identified

Useful observations—not substitutes for a correct call outcome.

09 · Unauthorized promises

Ask for certainty the system is not allowed to provide

Test whether the AI invents or overstates any item below. Correct behavior depends on the outfit’s configured authority; the system should state only what is authorized and known.

  • Technician arrival time
  • Exact availability
  • Diagnosis
  • Price
  • Job duration
  • Guarantee
  • Service eligibility
  • Escalation status
  • Human callback timing

10 · Review what happened downstream

Do not evaluate only what the caller heard

After every call, compare each available record with what actually happened. A good conversation can still leave the plumbing outfit with a bad appointment, wrong address, missing escalation, or invisible unresolved call.

  • Transcript
  • Recording, if lawfully available under the outfit’s setup
  • Notification
  • CRM record
  • Calendar entry
  • Escalation record
  • Human handoff
  • Unresolved-call queue

11 · Buyer decision worksheet

Decide capability by capability

Summarize every required job without averaging a serious failure into a flattering score. There is no universal pass threshold: the consequence of failure belongs beside the result.

Required?Tested?Meets requirementPartly meetsDoes not meetFailure consequenceWorkaround requiredRe-test needed

Ready for a limited pilot

Required scenarios have been tested, known limitations are acceptable, and the outfit can constrain the initial use case.

Needs configuration or retesting

Important failures may be fixable through scripts, permissions, integrations, routing, or vendor support.

Does not meet the current requirement

One or more required outcomes still fail or cannot be verified.

12 · If the outfit proceeds

A synthetic pass supports a limited pilot—not a claim of long-term reliability

Keep the initial scope narrow enough that a person can observe the workflow and own unresolved calls. Expand only after the actual records support the next decision.

  1. 01Start with a constrained use case.
  2. 02Define who watches failures.
  3. 03Keep a human fallback.
  4. 04Review transcripts and downstream records.
  5. 05Track unresolved calls.
  6. 06Re-test after configuration changes.