C# Token Description

Token. What exactly is a token in the C# programming language?

The specification defines a token using the C# grammar. Tokens are generally any unit that is not whitespace or a comment. They are part of the text of a program.

Info: This article covers the concept of tokens at the level of the C# language. It helps us understand how the language is compiled.

Comment

Example. Let's explore the concept of tokens by looking at this C# program. We see the program and then break it up into individual tokens. These include keywords (class), identifiers (Program), literals (2) and operators (=).

Program: C#

class Program
{
    static void Main()
    {
	int test = 2;
	float item = 5.5f;
	char unit = 'e';
	string basic = "c#";
    }
}

Tokens

class
Program
{
static
void
Main
(
)
{
int
test
=
2
;
float
item
=
5.5f
;
char
unit
=
'e'
;
string
basic
=
"c#"
;
}
}

Token types

identifier:

Identifiers are arbitrary names in your C# program.
This includes Program, Main, test, item, unit and basic.

keyword:

Many keywords are reserved by the C# language.
These include class, static, void, int, float, char and string.

integer-literal:
real-literal:
character-literal:
string-literal:

Literals are constant values you can use in your program.
These include 2, 5.5f, 'e' and "c#".
Literals include the enclosing quotation marks.
For the literal 5.5f, the numeric suffix is included in the literal.
This is a real-literal.

operator-or-punctuator:

These are characters in your program that are part of the scope
structure, or assignment and arithmetic statements.
Some operators, such as && are considered a single token.
The characters () are separate tokens.

Specification. For more material like this on the C# language, the specification is available. It goes into painful detail on almost every issue relating to this language. This material was adapted from page 61 of the C# annotated specification.

Summary. We now know what a token is in the C# language. This can be helpful in understanding how the language is parsed. Any time you see more closely how the compiler approaches a language, you gain insight into how it should be used.

Review: This description provides insight into how the C# compiler views programs.

Related Links