Skip to main content

Regular Expressions in Python

Regular expressions describe patterns in text. Use them for bounded lexical tasks such as extracting identifiers or validating a deliberately small format; use a parser for nested structure or a complete language grammar.

Write patterns as raw strings so Python string escaping does not obscure regex escaping:

import re

event = re.compile(
    r"^(?P<level>INFO|WARN|ERROR)\s+user=(?P<user>[\w.-]+)$"
)

match = event.fullmatch(line)
if match:
    level = match["level"]
    user = match["user"]

Choose the matching contract​

  • search finds the first match anywhere.
  • match checks only at the beginning.
  • fullmatch requires the entire input to match and is usually clearest for validation.
  • finditer streams non-overlapping match objects; findall constructs a list whose shape changes with capturing groups.
  • sub replaces matches; a callable replacement can make context-sensitive changes without a second parsing pass.

Use named groups for fields with domain meaning and non-capturing groups (?:...) when grouping is only structural.

Meaning and escaping​

For str patterns, classes such as \w and \d use Unicode semantics by default. Add re.ASCII only when the format is explicitly ASCII. Case-insensitive matching is not locale-aware human-language comparison.

Escape untrusted text that must be treated literally:

literal_pattern = re.compile(re.escape(user_supplied_prefix))

Do not interpolate arbitrary text directly into a pattern or replacement template. Pattern escaping and replacement escaping have different rules.

Performance and validation boundaries​

Ambiguous nested repetitions and overlapping alternatives can cause extreme backtracking on adversarial input. Keep patterns simple, bound input size, test near misses, and isolate or replace regex work when a hard time limit matters. The standard re API does not provide a general per-match timeout.

A regex can check syntax but not ownership, deliverability, authorization, or business validity. For URLs, email addresses, dates, and structured formats, prefer a standard parser plus domain validation.

Use re.VERBOSE and comments when a pattern is complex enough to require maintenance. At some point a named parser is the more honest abstraction.

Source​