debian._deb822_repro.tokens module

class debian._deb822_repro.tokens.Deb822CommaToken

Bases: Deb822SeparatorToken

Used by the comma-separated list value parsers to denote a comma between two value tokens.

class debian._deb822_repro.tokens.Deb822CommentToken(text: str)

Bases: Deb822Token

property is_comment: bool
class debian._deb822_repro.tokens.Deb822ErrorToken(text: str)

Bases: Deb822Token

Token that represents a syntactical error

property is_error: bool
class debian._deb822_repro.tokens.Deb822FieldNameToken(text: str)

Bases: Deb822Token

property text: _CaseInsensitiveString
class debian._deb822_repro.tokens.Deb822FieldSeparatorToken

Bases: Deb822SeparatorToken

class debian._deb822_repro.tokens.Deb822NewlineAfterValueToken

Bases: Deb822SemanticallySignificantWhiteSpace

The newline after a value token.

If not followed by a continuation token, this also marks the end of the field.

class debian._deb822_repro.tokens.Deb822PipeToken

Bases: Deb822SeparatorToken

Used in some dependency fields as OR relation

class debian._deb822_repro.tokens.Deb822SemanticallySignificantWhiteSpace(text: str)

Bases: Deb822WhitespaceToken

Whitespace that (if removed) would change the meaning of the file (or cause syntax errors)

class debian._deb822_repro.tokens.Deb822SeparatorToken(text: str)

Bases: Deb822Token

property is_separator: bool
class debian._deb822_repro.tokens.Deb822SpaceSeparatorToken(text: str)

Bases: Deb822SemanticallySignificantWhiteSpace

Whitespace between values in a space list (e.g. “Architectures”)

property is_separator: bool
class debian._deb822_repro.tokens.Deb822Token(text: str)

Bases: Locatable

A token is an atomic syntactical element from a deb822 file

A file is parsed into a series of tokens. If these tokens are converted to text in exactly the same order, you get exactly the same file - bit-for-bit. Accordingly ever bit of text in a file must be assigned to exactly one Deb822Token.

_parent_element: ReferenceType['Deb822Element'] | None
_text: str
_token_size: Range | None
_verify_token_text() → None
clear_parent_if_parent(parent: Deb822Element) → None
convert_to_text() → str
property is_comment: bool
property is_error: bool
property is_separator: bool
property is_whitespace: bool
property parent_element: Deb822Element | None
size() → Range

Describe the objects size as a continuous range

property text: str
class debian._deb822_repro.tokens.Deb822ValueContinuationToken(text: str)

Bases: Deb822SemanticallySignificantWhiteSpace

The whitespace denoting a value spanning an additional line (the first space on a line)

class debian._deb822_repro.tokens.Deb822ValueDependencyToken(text: str)

Bases: Deb822Token

Package name, architecture name, a version number, or a profile name in a dependency field

class debian._deb822_repro.tokens.Deb822ValueDependencyVersionRelationOperatorToken(text: str)

Bases: Deb822Token

class debian._deb822_repro.tokens.Deb822ValueToken(text: str)

Bases: Deb822Token

A field value can be split into multi “Deb822ValueToken”s (as well as separator tokens)

class debian._deb822_repro.tokens.Deb822WhitespaceToken(text: str)

Bases: Deb822Token

The token is a kind of whitespace.

Some whitespace tokens are critical for the format (such as the Deb822ValueContinuationToken, spaces that separate words in list separated by spaces or newlines), while other whitespace tokens are truly insignificant (space before a newline, space after a comma in a comma list, etc.).

property is_whitespace: bool
debian._deb822_repro.tokens._value_line_tokenizer(func: Callable[[str], Iterable[Deb822Token]]) → Callable[[str], Iterable[Deb822Token]]
debian._deb822_repro.tokens.comma_split_tokenizer(v: str) → Iterable[Deb822Token]
debian._deb822_repro.tokens.tokenize_deb822_file(sequence: Iterable[str | bytes], encoding: str = 'utf-8') → Iterable[Deb822Token]

Tokenize a deb822 file

Parameters:

  • sequence – An iterable of lines (a file open for reading will do)

  • encoding – The encoding to use (this is here to support Deb822-like APIs, new code should not use this parameter).

debian._deb822_repro.tokens.whitespace_split_tokenizer(v: str) → Iterable[Deb822Token]