<commit_before>---
title: 'Blazingly fast parsing, part 1: optimizing the scanner'
author: '<NAME> ([@tverwaes](https://twitter.com/tverwaes)), scandalous optimizer'
avatars:
  - 'toon-verwaest'
date: 2019-03-25 13:33:37
tags:
  - internals
  - parsing
---
To run a JavaScript program, the source text needs to be processed so V8 can understand it. V8 starts out by parsing the source into an abstract syntax tree (AST), a set of objects that represent the program structure. That AST gets compiled to bytecode by Ignition. The performance of these parse + compile phases is important: V8 cannot run code before compilation is done. In this series of blog posts, we focus on parsing, and the work done in V8 to ship a blazingly fast parser.

In fact, we start the series one stage before the parser. V8’s parser consumes ‘tokens’ provided by the ‘scanner’. Tokens are blocks of one or more characters that have a single semantic meaning: a string, an identifier, an operator like `++`. The scanner constructs these tokens by combining consecutive characters in an underlying character stream.
<commit_msg>Add tweet<commit_after>---
title: 'Blazingly fast parsing, part 1: optimizing the scanner'
author: '<NAME> ([@tverwaes](https://twitter.com/tverwaes)), scandalous optimizer'
avatars:
  - 'toon-verwaest'
date: 2019-03-25 13:33:37
tags:
  - internals
  - parsing
tweet: '1110205101652787200'
---
To run a JavaScript program, the source text needs to be processed so V8 can understand it. V8 starts out by parsing the source into an abstract syntax tree (AST), a set of objects that represent the program structure. That AST gets compiled to bytecode by Ignition. The performance of these parse + compile phases is important: V8 cannot run code before compilation is done. In this series of blog posts, we focus on parsing, and the work done in V8 to ship a blazingly fast parser.

In fact, we start the series one stage before the parser. V8’s parser consumes ‘tokens’ provided by the ‘scanner’. Tokens are blocks of one or more characters that have a single semantic meaning: a string, an identifier, an operator like `++`. The scanner constructs these tokens by combining consecutive characters in an underlying character stream.

{'fraction_non_alphanumeric': 0.06845637583892618, 'fraction_numerical': 0.026398210290827742, 'mean_word_length': 4.5073891625615765, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5174716', 'n_tokens_mistral': 634, 'n_tokens_neox': 574, 'n_words': 331}