Login Preferences
About News Publications Statistics Downloads API Guide License How to use Preferences
bn:17765289n
Noun Concept
Categories: Parsing, Programming language implementation, Compiler construction, Articles with short description, Lexical analysis
EN
lexical analysis  lexer  automatic semicolon insertion  lexeme  lexer generator
EN
Lexical tokenization is conversion of a text into meaningful lexical tokens belonging to categories defined by a "lexer" program. Wikipedia
Definitions
Examples
Relations
Sources
EN
Lexical tokenization is conversion of a text into meaningful lexical tokens belonging to categories defined by a "lexer" program. Wikipedia
The process of converting a sequence of characters into a sequence of tokens Wikipedia Disambiguation
Computing process of parsing a sequence of characters into a sequence of tokens Wikidata
The conversion of a stream of characters to a stream of meaningful tokens; normally to simplify parsing. Wiktionary
Conversion of a stream of characters to a stream of meaningful tokens. Wiktionary (translation)
EN
While it's often not difficult to identify tokens while parsing, having a separate stage for lexical analysis simplifies the structure of your compiler. Wiktionary
PART OF
compilation
HAS KIND
JLex
planetary scanner
PRODUCT OR MATERIAL PRODUCED
sequence
USES
source code
Wikipedia
EN
lexical analysis
Wikidata
EN
lexical analysis
Wiktionary
EN
lexical analysis
Wikipedia Redirections
EN
automatic semicolon insertion, lexeme (computer science), lexer, lexer generator, lexers, Lexical analyser, Lexical Analysis, lexical analyzer, Lexical parser, lexical syntax, lexical token, Lexical tokenization, lexing, List of lexer generators, semicolon insertion, token (parser), Token scanner, Token splitting, Tokenise, Tokeniser, tokenize, tokenized, tokenizer, tokenizing
Wikidata Alias
EN
lexing, tokenization

Please note that, in order to improve your browsing experience on this website, BabelNet® uses various types of cookies, including: browsing functionality, performance and statistical cookies.

By continuing to browse the site you are agreeing to our use of cookies. OK

License