Stage 0 Draft / June 8, 2022

Regular Expression Atomic Operators for ECMAScript

Introduction

See the proposal repository for background material and discussion.

1 Text Processing

1.1 RegExp (Regular Expression) Objects

1.1.1 Patterns

The RegExp constructor applies the following grammar to the input pattern String. An error occurs if the grammar cannot interpret the String as an expansion of Pattern.

Syntax

Pattern[UnicodeMode, N] :: Disjunction[?UnicodeMode, ?N] Disjunction[UnicodeMode, N] :: Alternative[?UnicodeMode, ?N] Alternative[?UnicodeMode, ?N] | Disjunction[?UnicodeMode, ?N] Alternative[UnicodeMode, N] :: [empty] Alternative[?UnicodeMode, ?N] Term[?UnicodeMode, ?N] Term[UnicodeMode, N] :: Assertion[?UnicodeMode, ?N] Atom[?UnicodeMode, ?N] Atom[?UnicodeMode, ?N] Quantifier Assertion[UnicodeMode, N] :: ^ $ \ b \ B ( ? = Disjunction[?UnicodeMode, ?N] ) ( ? ! Disjunction[?UnicodeMode, ?N] ) ( ? <= Disjunction[?UnicodeMode, ?N] ) ( ? <! Disjunction[?UnicodeMode, ?N] ) Quantifier :: QuantifierPrefix QuantifierPrefix ? QuantifierPrefix + QuantifierPrefix :: * + ? { DecimalDigits[~Sep] } { DecimalDigits[~Sep] , } { DecimalDigits[~Sep] , DecimalDigits[~Sep] } Atom[UnicodeMode, N] :: PatternCharacter . \ AtomEscape[?UnicodeMode, ?N] CharacterClass[?UnicodeMode] ( GroupSpecifier[?UnicodeMode]opt Disjunction[?UnicodeMode, ?N] ) ( ? : Disjunction[?UnicodeMode, ?N] ) ( ? > Disjunction[?UnicodeMode, ?N] ) SyntaxCharacter :: one of ^ $ \ . * + ? ( ) [ ] { } | PatternCharacter :: SourceCharacter but not SyntaxCharacter AtomEscape[UnicodeMode, N] :: DecimalEscape CharacterClassEscape[?UnicodeMode] CharacterEscape[?UnicodeMode] [+N] k GroupName[?UnicodeMode] CharacterEscape[UnicodeMode] :: ControlEscape c ControlLetter 0 [lookahead ∉ DecimalDigit] HexEscapeSequence RegExpUnicodeEscapeSequence[?UnicodeMode] IdentityEscape[?UnicodeMode] ControlEscape :: one of f n r t v ControlLetter :: one of a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I J K L M N O P Q R S T U V W X Y Z GroupSpecifier[UnicodeMode] :: ? GroupName[?UnicodeMode] GroupName[UnicodeMode] :: < RegExpIdentifierName[?UnicodeMode] > RegExpIdentifierName[UnicodeMode] :: RegExpIdentifierStart[?UnicodeMode] RegExpIdentifierName[?UnicodeMode] RegExpIdentifierPart[?UnicodeMode] RegExpIdentifierStart[UnicodeMode] :: IdentifierStartChar \ RegExpUnicodeEscapeSequence[+UnicodeMode] [~UnicodeMode] UnicodeLeadSurrogate UnicodeTrailSurrogate RegExpIdentifierPart[UnicodeMode] :: IdentifierPartChar \ RegExpUnicodeEscapeSequence[+UnicodeMode] [~UnicodeMode] UnicodeLeadSurrogate UnicodeTrailSurrogate RegExpUnicodeEscapeSequence[UnicodeMode] :: [+UnicodeMode] u HexLeadSurrogate \u HexTrailSurrogate [+UnicodeMode] u HexLeadSurrogate [+UnicodeMode] u HexTrailSurrogate [+UnicodeMode] u HexNonSurrogate [~UnicodeMode] u Hex4Digits [+UnicodeMode] u{ CodePoint } UnicodeLeadSurrogate :: any Unicode code point in the inclusive range 0xD800 to 0xDBFF UnicodeTrailSurrogate :: any Unicode code point in the inclusive range 0xDC00 to 0xDFFF

Each \u HexTrailSurrogate for which the choice of associated u HexLeadSurrogate is ambiguous shall be associated with the nearest possible u HexLeadSurrogate that would otherwise have no corresponding \u HexTrailSurrogate.

HexLeadSurrogate :: Hex4Digits but only if the MV of Hex4Digits is in the inclusive range 0xD800 to 0xDBFF HexTrailSurrogate :: Hex4Digits but only if the MV of Hex4Digits is in the inclusive range 0xDC00 to 0xDFFF HexNonSurrogate :: Hex4Digits but only if the MV of Hex4Digits is not in the inclusive range 0xD800 to 0xDFFF IdentityEscape[UnicodeMode] :: [+UnicodeMode] SyntaxCharacter [+UnicodeMode] / [~UnicodeMode] SourceCharacter but not UnicodeIDContinue DecimalEscape :: NonZeroDigit DecimalDigits[~Sep]opt [lookahead ∉ DecimalDigit] CharacterClassEscape[UnicodeMode] :: d D s S w W [+UnicodeMode] p{ UnicodePropertyValueExpression } [+UnicodeMode] P{ UnicodePropertyValueExpression } UnicodePropertyValueExpression :: UnicodePropertyName = UnicodePropertyValue LoneUnicodePropertyNameOrValue UnicodePropertyName :: UnicodePropertyNameCharacters UnicodePropertyNameCharacters :: UnicodePropertyNameCharacter UnicodePropertyNameCharactersopt UnicodePropertyValue :: UnicodePropertyValueCharacters LoneUnicodePropertyNameOrValue :: UnicodePropertyValueCharacters UnicodePropertyValueCharacters :: UnicodePropertyValueCharacter UnicodePropertyValueCharactersopt UnicodePropertyValueCharacter :: UnicodePropertyNameCharacter DecimalDigit UnicodePropertyNameCharacter :: ControlLetter _ CharacterClass[UnicodeMode] :: [ [lookahead ≠ ^] ClassRanges[?UnicodeMode] ] [ ^ ClassRanges[?UnicodeMode] ] ClassRanges[UnicodeMode] :: [empty] NonemptyClassRanges[?UnicodeMode] NonemptyClassRanges[UnicodeMode] :: ClassAtom[?UnicodeMode] ClassAtom[?UnicodeMode] NonemptyClassRangesNoDash[?UnicodeMode] ClassAtom[?UnicodeMode] - ClassAtom[?UnicodeMode] ClassRanges[?UnicodeMode] NonemptyClassRangesNoDash[UnicodeMode] :: ClassAtom[?UnicodeMode] ClassAtomNoDash[?UnicodeMode] NonemptyClassRangesNoDash[?UnicodeMode] ClassAtomNoDash[?UnicodeMode] - ClassAtom[?UnicodeMode] ClassRanges[?UnicodeMode] ClassAtom[UnicodeMode] :: - ClassAtomNoDash[?UnicodeMode] ClassAtomNoDash[UnicodeMode] :: SourceCharacter but not one of \ or ] or - \ ClassEscape[?UnicodeMode] ClassEscape[UnicodeMode] :: b [+UnicodeMode] - CharacterClassEscape[?UnicodeMode] CharacterEscape[?UnicodeMode] Note

A number of productions in this section are given alternative definitions in section A.1.1.

1.1.2 Pattern Semantics

1.1.2.1 Runtime Semantics: CompileSubpattern

The syntax-directed operation CompileSubpattern takes argument direction (forward or backward) and returns a Matcher.

Note 1

This section is amended in B.1.2.5.

It is defined piecewise over the following productions:

Disjunction :: Alternative | Disjunction
  1. Let m1 be CompileSubpattern of Alternative with argument direction.
  2. Let m2 be CompileSubpattern of Disjunction with argument direction.
  3. Return a new Matcher with parameters (x, c) that captures m1 and m2 and performs the following steps when called:
    1. Assert: x is a State.
    2. Assert: c is a Continuation.
    3. Let r be m1(x, c).
    4. If r is not failure, return r.
    5. Return m2(x, c).
 Note 2

The | regular expression operator separates two alternatives. The pattern first tries to match the left Alternative (followed by the sequel of the regular expression); if it fails, it tries to match the right Disjunction (followed by the sequel of the regular expression). If the left Alternative, the right Disjunction, and the sequel all have choice points, all choices in the sequel are tried before moving on to the next choice in the left Alternative. If choices in the left Alternative are exhausted, the right Disjunction is tried instead of the left Alternative. Any capturing parentheses inside a portion of the pattern skipped by | produce undefined values instead of Strings. Thus, for example,

/a|ab/.exec("abc")

returns the result "a" and not "ab". Moreover,

/((a)|(ab))((c)|(bc))/.exec("abc")

returns the array

["abc", "a", "a", undefined, "bc", undefined, "bc"]

and not

["abc", "ab", undefined, "ab", "c", "c", undefined]

The order in which the two alternatives are tried is independent of the value of direction.

 Alternative :: [empty]
  1. Return a new Matcher with parameters (x, c) that captures nothing and performs the following steps when called:
    1. Assert: x is a State.
    2. Assert: c is a Continuation.
    3. Return c(x).
 Alternative :: Alternative Term
  1. Let m1 be CompileSubpattern of Alternative with argument direction.
  2. Let m2 be CompileSubpattern of Term with argument direction.
  3. If direction is forward, then
    1. Return a new Matcher with parameters (x, c) that captures m1 and m2 and performs the following steps when called:
      1. Assert: x is a State.
      2. Assert: c is a Continuation.
      3. Let d be a new Continuation with parameters (y) that captures c and m2 and performs the following steps when called:
        1. Assert: y is a State.
        2. Return m2(y, c).
      4. Return m1(x, d).
  4. Else,
    1. Assert: direction is backward.
    2. Return a new Matcher with parameters (x, c) that captures m1 and m2 and performs the following steps when called:
      1. Assert: x is a State.
      2. Assert: c is a Continuation.
      3. Let d be a new Continuation with parameters (y) that captures c and m1 and performs the following steps when called:
        1. Assert: y is a State.
        2. Return m1(y, c).
      4. Return m2(x, d).
 Note 3

Consecutive Terms try to simultaneously match consecutive portions of Input. When direction is forward, if the left Alternative, the right Term, and the sequel of the regular expression all have choice points, all choices in the sequel are tried before moving on to the next choice in the right Term, and all choices in the right Term are tried before moving on to the next choice in the left Alternative. When direction is backward, the evaluation order of Alternative and Term are reversed.

 Term :: Assertion
  1. Return CompileAssertion of Assertion.
 Note 4

The resulting Matcher is independent of direction.

 Term :: Atom
  1. Return CompileAtom of Atom with argument direction.
 Term :: Atom Quantifier
  1. Let m be CompileAtom of Atom with argument direction.
  2. Let q be CompileQuantifier of Quantifier.
  3. Assert: q.[[Min]] ≤ q.[[Max]].
  4. Let parenIndex be CountLeftCapturingParensBefore(Term).
  5. Let parenCount be CountLeftCapturingParensWithin(Atom).
  6. Return a new Matcher with parameters (x, c) that captures m, q, parenIndex, and parenCount and performs the following steps when called:
    1. Assert: x is a State.
    2. Assert: c is a Continuation.
    3. If q.[[Atomic]] is true, then
      1. Let d be a new Continuation with parameters (y) that performs the following steps when called:
        1. Assert: y is a State.
        2. Return y.
      2. Let r be RepeatMatcher(m, q.[[Min]], q.[[Max]], true, x, d, parenIndex, parenCount).
      3. If r is failure, return failure.
      4. Return c(r).
    4. Return RepeatMatcher(m, q.[[Min]], q.[[Max]], q.[[Greedy]], x, c, parenIndex, parenCount).

1.1.2.2 Runtime Semantics: CompileQuantifier

The syntax-directed operation CompileQuantifier takes no arguments and returns a Record with fields [[Min]] (a non-negative integer), [[Max]] (a non-negative integer or +∞), [[Greedy]] (a Boolean), and [[Atomic]] (a Boolean). It is defined piecewise over the following productions:

Quantifier :: QuantifierPrefix
  1. Let qp be CompileQuantifierPrefix of QuantifierPrefix.
  2. Return the Record { [[Min]]: qp.[[Min]], [[Max]]: qp.[[Max]], [[Greedy]]: true, [[Atomic]]: false }.
 Quantifier :: QuantifierPrefix ?
  1. Let qp be CompileQuantifierPrefix of QuantifierPrefix.
  2. Return the Record { [[Min]]: qp.[[Min]], [[Max]]: qp.[[Max]], [[Greedy]]: false, [[Atomic]]: false }.
 Quantifier :: QuantifierPrefix +
  1. Let qp be CompileQuantifierPrefix of QuantifierPrefix.
  2. Return the Record { [[Min]]: qp.[[Min]], [[Max]]: qp.[[Max]], [[Greedy]]: true, [[Atomic]]: true }.

1.1.2.3 Runtime Semantics: CompileAtom

The syntax-directed operation CompileAtom takes argument direction (forward or backward) and returns a Matcher.

Note 1

This section is amended in B.1.2.7.

It is defined piecewise over the following productions:

Atom :: PatternCharacter
  1. Let ch be the character matched by PatternCharacter.
  2. Let A be a one-element CharSet containing the character ch.
  3. Return CharacterSetMatcher(A, false, direction).
 Atom :: .
  1. Let A be the CharSet of all characters.
  2. If DotAll is not true, then
    1. Remove from A all characters corresponding to a code point on the right-hand side of the LineTerminator production.
  3. Return CharacterSetMatcher(A, false, direction).
 Atom :: CharacterClass
  1. Let cc be CompileCharacterClass of CharacterClass.
  2. Return CharacterSetMatcher(cc.[[CharSet]], cc.[[Invert]], direction).
 Atom :: ( GroupSpecifieropt Disjunction )
  1. Let m be CompileSubpattern of Disjunction with argument direction.
  2. Let parenIndex be CountLeftCapturingParensBefore(Atom).
  3. Return a new Matcher with parameters (x, c) that captures direction, m, and parenIndex and performs the following steps when called:
    1. Assert: x is a State.
    2. Assert: c is a Continuation.
    3. Let d be a new Continuation with parameters (y) that captures x, c, direction, and parenIndex and performs the following steps when called:
      1. Assert: y is a State.
      2. Let cap be a copy of y's captures List.
      3. Let xe be x's endIndex.
      4. Let ye be y's endIndex.
      5. If direction is forward, then
        1. Assert: xe ≤ ye.
        2. Let r be the Range (xe, ye).
      6. Else,
        1. Assert: direction is backward.
        2. Assert: ye ≤ xe.
        3. Let r be the Range (ye, xe).
      7. Set cap[parenIndex + 1] to r.
      8. Let z be the State (ye, cap).
      9. Return c(z).
    4. Return m(x, d).
 Atom :: ( ? : Disjunction )
  1. Return CompileSubpattern of Disjunction with argument direction.
 Atom :: ( ? > Disjunction )
  1. Let m be CompileSubpattern of Disjunction with argument direction.
  2. Return a new Matcher with parameters (x, c) that captures m and performs the following steps when called:
    1. Assert: x is a State.
    2. Assert: c is a Continuation.
    3. Let d be a new Continuation with parameters (y) that performs the following steps when called:
      1. Assert: y is a State.
      2. Return y.
    4. Let r be m(x, d).
    5. If r is failure, return failure.
    6. Return c(r).
 AtomEscape :: DecimalEscape
  1. Let n be the CapturingGroupNumber of DecimalEscape.
  2. Assert: n ≤ NcapturingParens.
  3. Return BackreferenceMatcher(n, direction).
 Note 2

An escape sequence of the form \ followed by a non-zero decimal number n matches the result of the nth set of capturing parentheses (22.2.2.1). It is an error if the regular expression has fewer than n capturing parentheses. If the regular expression has n or more capturing parentheses but the nth one is undefined because it has not captured anything, then the backreference always succeeds.

 AtomEscape :: CharacterEscape
  1. Let cv be the CharacterValue of CharacterEscape.
  2. Let ch be the character whose character value is cv.
  3. Let A be a one-element CharSet containing the character ch.
  4. Return CharacterSetMatcher(A, false, direction).
 AtomEscape :: CharacterClassEscape
  1. Let A be CompileToCharSet of CharacterClassEscape.
  2. Return CharacterSetMatcher(A, false, direction).
 AtomEscape :: k GroupName
  1. Let matchingGroupSpecifiers be GroupSpecifiersThatMatch(GroupName).
  2. Assert: matchingGroupSpecifiers contains a single GroupSpecifier.
  3. Let groupSpecifier be the sole element of matchingGroupSpecifiers.
  4. Let parenIndex be CountLeftCapturingParensBefore(groupSpecifier).
  5. Return BackreferenceMatcher(parenIndex, direction).

A Additional ECMAScript Features for Web Browsers

A.1 Additional Syntax

A.1.1 Regular Expressions Patterns

Term[UnicodeMode, N] :: [+UnicodeMode] Assertion[+UnicodeMode, ?N] [+UnicodeMode] Atom[+UnicodeMode, ?N] Quantifier [+UnicodeMode] Atom[+UnicodeMode, ?N] [~UnicodeMode] QuantifiableAssertion[?N] Quantifier [~UnicodeMode] Assertion[~UnicodeMode, ?N] [~UnicodeMode] ExtendedAtom[?N] Quantifier [~UnicodeMode] ExtendedAtom[?N] Assertion[UnicodeMode, N] :: ^ $ \ b \ B [+UnicodeMode] ( ? = Disjunction[+UnicodeMode, ?N] ) [+UnicodeMode] ( ? ! Disjunction[+UnicodeMode, ?N] ) [~UnicodeMode] QuantifiableAssertion[?N] ( ? <= Disjunction[?UnicodeMode, ?N] ) ( ? <! Disjunction[?UnicodeMode, ?N] ) QuantifiableAssertion[N] :: ( ? = Disjunction[~UnicodeMode, ?N] ) ( ? ! Disjunction[~UnicodeMode, ?N] ) ExtendedAtom[N] :: . \ AtomEscape[~UnicodeMode, ?N] \ [lookahead = c] CharacterClass[~UnicodeMode] ( GroupSpecifier[~UnicodeMode]opt Disjunction[~UnicodeMode, ?N] ) ( ? : Disjunction[~UnicodeMode, ?N] ) ( ? > Disjunction[~UnicodeMode, ?N] ) InvalidBracedQuantifier ExtendedPatternCharacter InvalidBracedQuantifier :: { DecimalDigits[~Sep] } { DecimalDigits[~Sep] , } { DecimalDigits[~Sep] , DecimalDigits[~Sep] } ExtendedPatternCharacter :: SourceCharacter but not one of ^ $ \ . * + ? ( ) [ | AtomEscape[UnicodeMode, N] :: [+UnicodeMode] DecimalEscape [~UnicodeMode] DecimalEscape but only if the CapturingGroupNumber of DecimalEscape is ≤ CountLeftCapturingParensWithin(the Pattern containing DecimalEscape) CharacterClassEscape[?UnicodeMode] CharacterEscape[?UnicodeMode, ?N] [+N] k GroupName[?UnicodeMode] CharacterEscape[UnicodeMode, N] :: ControlEscape c ControlLetter 0 [lookahead ∉ DecimalDigit] HexEscapeSequence RegExpUnicodeEscapeSequence[?UnicodeMode] [~UnicodeMode] LegacyOctalEscapeSequence IdentityEscape[?UnicodeMode, ?N] IdentityEscape[UnicodeMode, N] :: [+UnicodeMode] SyntaxCharacter [+UnicodeMode] / [~UnicodeMode] SourceCharacterIdentityEscape[?N] SourceCharacterIdentityEscape[N] :: [~N] SourceCharacter but not c [+N] SourceCharacter but not one of c or k ClassAtomNoDash[UnicodeMode, N] :: SourceCharacter but not one of \ or ] or - \ ClassEscape[?UnicodeMode, ?N] \ [lookahead = c] ClassEscape[UnicodeMode, N] :: b [+UnicodeMode] - [~UnicodeMode] c ClassControlLetter CharacterClassEscape[?UnicodeMode] CharacterEscape[?UnicodeMode, ?N] ClassControlLetter :: DecimalDigit _

B Copyright & Software License

Copyright Notice

© 2022 Ron Buckton, Ecma International

Software License

All Software contained in this document ("Software") is protected by copyright and is being made available under the "BSD License", included below. This Software may be subject to third party rights (rights from parties other than Ecma International), including patent rights, and no licenses under such third party rights are granted under this license even if the third party concerned is a member of Ecma International. SEE THE ECMA CODE OF CONDUCT IN PATENT MATTERS AVAILABLE AT https://ecma-international.org/memento/codeofconduct.htm FOR INFORMATION REGARDING THE LICENSING OF PATENT CLAIMS THAT ARE REQUIRED TO IMPLEMENT ECMA INTERNATIONAL STANDARDS.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
  3. Neither the name of the authors nor Ecma International may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE ECMA INTERNATIONAL "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL ECMA INTERNATIONAL BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.