Detecting Unfamiliar Signs (US Patent 10,928,828)

Granted by the United States Patent and Trademark Office on February 23, 2021.
Inventors: Zhinan Xu, Maya Kabkab, Chen Wu, and Woojong Koh.
Assignee: Waymo LLC, Mountain View, CA.

Bibliographic Overview #

Abstract #

Aspects of the disclosure relate to determining a sign type of an unfamiliar sign. The system includes one or more onboard processors configured to receive an image generated by a perception sensor system of an autonomous vehicle and identify image data corresponding to a traffic sign within the image. The image data corresponding to the traffic sign is evaluated by a sign type model. When the sign type model is unable to identify a type of the traffic sign with sufficient confidence, the processors extract one or more visual and physical attributes of the traffic sign (such as geometry, color palette, retroreflectivity, placement, and text). These extracted attributes are compared against known regulatory and physical attributes of standard traffic signs stored in a relational knowledge structure. Based on this comparison, the vehicle determines the sign type and content, arbitrates whether the sign requires vehicle actuation or human operator intervention, and controls the vehicle in an autonomous driving mode.

The Engineering Challenge: The Infinite Long Tail of Roadway Signs #

Self-driving vehicles rely heavily on high-definition (HD) maps that contain centimeter-accurate spatial priors of static road geometry, lane networks, and permanent traffic control assets (such as traffic lights and standard regulatory signs). In nominal conditions, the autonomous vehicle’s onboard perception system cross-references observed visual objects with these map priors to confirm its world state.

However, real-world roads are dynamic and prone to constant modification:

• Temporary Construction & Work Zones: Portable orange signs, handmade arrow boards, and variable message trailers are frequently deployed without prior inclusion in HD map databases.
• Dynamic & Time-Dependent Signals: School zone signs featuring flashing LED beacons, time-of-day restrictions (“No Left Turn 7 AM–9 AM Except Buses”), and electronic variable speed signs change meaning based on clock time and physical lighting.
• Rare & Regional Variations: State park markers, historic district signs, hand-painted advisory notices, and novel municipal designs create an endless long tail of out-of-distribution (OOD) visual stimuli.

The Softmax Failure Mode #

In classical deep neural network perception pipelines, a sign classifier is trained on a closed set of predefined sign classes using a softmax output layer. When presented with an unseen, ambiguous, or novel sign:

1. Overconfident Misclassification: The model forces an out-of-distribution input into one of its pre-existing discrete classes, potentially interpreting a benign informational sign as an emergency stop or vice-versa.
2. Underconfident Rejection: The model yields low confidence across all classes, marking the object as unidentifiable.

The Teleoperation Bottleneck #

When an autonomous vehicle encounters an unidentifiable roadside object that might be a traffic sign, the safest naive response is to stop or slow down and place a call to a remote fleet assistance operator (teleoperation).

At fleet scale (thousands of autonomous robotaxis serving commercial rides daily), unfiltered remote assistance requests create severe operational bottlenecks:

• If every non-critical, novel sign (such as a brown recreational sign for a local trail or a blue billboard for highway lodging) halts the vehicle and triggers human review, fleet latency spikes and phantom traffic jams occur.
• Conversely, failing to recognize a genuine regulatory stop or detour sign threatens physical safety.

This patent was developed to solve this fundamental trade-off: how to reason about novel, unfamiliar signs in real time without human intervention whenever possible, and how to intelligently triage remote assistance when human confirmation is genuinely required.

Technical Architecture: Multi-Tiered Attribute Fallback Perception #

The patented architecture introduces a multi-tier fallback pipeline that bridges deep learning with rule-based regulatory ontology:

[Onboard Cameras / Perception Sensors]
                  │
                  ▼
   [Traffic Sign Detection & Cropping]
                  │
                  ▼
      [Primary Sign Type Model]
                  │
        ┌─────────┴─────────┐
        │ Confidence >= τ   │ Confidence < τ (Unfamiliar)
        ▼                   ▼
[Standard Action]   [Attribute Extraction Engine]
                    ├─ Geometric Shape (Octagon, Triangle, Diamond, Rectangle)
                    ├─ Color Palette (Red, Orange, Yellow, Green, Blue, Brown)
                    ├─ Retroreflective Coefficient & Elevation Mounting
                    └─ Optical Character Recognition (OCR) & Keyphrase Parsing
                            │
                            ▼
                    [Relational Ontology Matching]
                    (MUTCD Regulatory Knowledge Base)
                            │
        ┌───────────────────┴───────────────────┐
        ▼                                       ▼
[Instructive / Safety-Critical]       [Informational / Non-Critical]
(e.g., Regulatory Red, Detour Orange) (e.g., Recreation Brown, Rest Area Blue)
        │                                       │
        ├─ High-Priority Autonomous Action     ├─ Deprioritized / Ignored by Planner
        └─ Priority Teleoperation Escalation    └─ No Remote Teleoperation Dispatches

Key Figures from the Patent #

1. Vehicle Platform & Perception Sensor Suite (Figure 3) #

The autonomous vehicle platform (illustrated on the Chrysler Pacifica Hybrid autonomous fleet) incorporates a 360-degree sensor suite consisting of roof-mounted primary LiDAR domes, perimeter LiDARs, high-resolution color perception cameras, and radar sensors connected to high-performance onboard inference computers.

2. Sign Identification & Ground Truth Categorization (Figure 7) #

The primary perception model is trained across foundational sign families established under regulatory frameworks such as the Manual on Uniform Traffic Control Devices (MUTCD).

3. Relational Attribute Decomposition (Figure 8) #

When the primary sign model detects an unfamiliar sign that does not match known training exemplars with sufficient confidence, the system decomposes the sign into constituent attributes rather than attempting whole-object classification.

Attributes extracted by the vision pipeline include:

• Geometric Primitives: Octagonal, triangular (equilateral down/up), circular, diamond (rhombus), vertical rectangle, and horizontal rectangle.
• Photometric & Colorimetric Data: Chromaticity coordinates matched against standard highway color specifications (MUTCD 23 CFR 655):
  • Red: Exclusively assigned to regulatory commands requiring immediate driver cessation or right-of-way concession (Stop, Yield, Wrong Way, Do Not Enter).
  • Fluorescent Pink / Orange: Exclusively assigned to temporary traffic control, construction zones, and emergency detours.
  • Yellow: Exclusively assigned to general physical hazards and road geometry warnings (sharp curves, lane merges, crossings).
  • White / Black: Regulatory speed and lane assignments.
  • Green: Highway navigational guidance and directional mileage.
  • Blue: Motorist services, rest stops, and medical facilities.
  • Brown: Public recreation, historic cultural landmarks, and national parks.
• Optical Character Recognition (OCR): Text extraction classified into instructive commands (e.g., "DETOUR", "STOP", "ONE WAY", "ROAD CLOSED") vs. informative content (e.g., "SCENIC OVERLOOK", "PICNIC AREA", "YOSEMITE NATIONAL PARK").

4. Real-Time Autonomous Sign Control Pipeline (Figure 12) #

The complete end-to-end procedural workflow executing on the vehicle’s onboard compute system:

Representative Patent Claim: Independent Claim 1 #

The legal scope of the granted invention is defined by independent Claim 1:

1. A method of controlling an autonomous vehicle in response to detecting and analyzing an unfamiliar traffic sign, the method comprising:

• receiving, by one or more processors, an image generated by a perception system of the autonomous vehicle, the perception system including one or more sensors;
• identifying, by the one or more processors, image data corresponding to a traffic sign in the image generated by the perception system of the autonomous vehicle;
• inputting, by the one or more processors, the identified image data corresponding to the traffic sign into one or more system software modules running on the one or more processors;
• determining, by the one or more processors, that the traffic sign in the image is an unfamiliar traffic sign that cannot be identified by the one or more system software modules;
• identifying, by the one or more processors, one or more attributes of the unfamiliar traffic sign;
• comparing, by the one or more processors, the identified one or more attributes of the unfamiliar traffic sign to known attributes of other traffic signs;
• determining, by the one or more processors, a category of the unfamiliar traffic sign based on the comparing; and
• controlling, by the one or more processors, the vehicle in an autonomous driving mode based on the determined category of the unfamiliar traffic sign,
• wherein the one or more processors are further configured to associate the image with a label, and categorize the label as unidentifiable or unknown when the label indicates that the determined category of the unfamiliar traffic sign has a confidence level which fails to satisfy a predetermined threshold.

Impact & Retrospective #

During my tenure at Waymo (2016–2021), scaling autonomous driving from geofenced testing to fully driverless commercial operations required solving problems where pure end-to-end deep learning frequently fell short: edge cases in real-world infrastructure.

By decoupling the detection of novel objects from discrete class predictions and grounding them in regulatory semantics and physical attributes, this system delivered critical capabilities:

1. Unblocking New Geographic Service Areas: Rapidly deploying into new municipalities without needing prior human mapping of every unique municipal sign variant.
2. Graceful Degradation: When encountering a damaged, weather-occluded, or non-standard sign, the vehicle reliably deduced whether the sign demanded safety-critical deceleration (e.g. an obscured red octagonal sign) or could be safely passed without abrupt braking.
3. Teleoperation Efficiency: Dramatically minimized unnecessary dispatch calls to human remote assistants, protecting fleet uptime while maintaining strict safety standards.

Archival Links & Resources #

• Local Full-Text Document: US Patent 10,928,828 B2 (PDF)
• USPTO Patent Grant: Google Patents US10928828B2
• Continuation Application: Google Patents US20240054772A1
• Citation Index: Google Scholar Profile Record