What does Tokens do and why they need to be created in C programming

0 votes

I am now reading a book (Programming Principles and Practice by Bjarne Stroustrup).

Tokens are introduced in this episode:

"A token is a string of letters that represents a unit, such as a number or an operator." 

That's how a C++ compiler handles its source code. 

In fact, "tokenizing" in some form or another is how most text analysis begins."

class Token {
public:
    char kind;
    double value;
};

I understand what they are, but he never explains it in depth, which makes it difficult for me.


Aug 1, 2022 in C++ by Nicholas
• 7,760 points
633 views

1 answer to this question.

0 votes

Tokenization is essential in determining what a programme does. 

What Bjarne is referring to in respect to C++ code is how tokenization rules alter the meaning of a programme. 

We need to know what the tokens are and how they are determined. 

Specifically, how can we recognise a single token when it comes among other characters, and how should tokens be delimited if  there is ambiguity?

Consider the prefix operators ++ and +, for example. Assume we have just one token + to deal with. 

What does the following excerpt mean?

int i = 1;
++i;

Is the above going to apply unary + on i twice with + only? Or will it only increase it once? Naturally, it's vague. 

We require an additional token, thus ++ is introduced as its own "word" in the language.

But there is now another (though minor) issue. 

What if the programmer just wants to use unary + twice without incrementing? 

Rules for token processing are required. 

So, if we discover that a white space is always used as a token separator, our programmer may write:

int i = 1;
+ +i;

A C++ implementation begins with a file full of characters, converts them to a series of tokens ("words" with meaning in the C++ language), and then tests to see if the tokens occur in a "sentence" with some valid meaning.

answered Aug 2, 2022 by Damon
• 4,960 points

Related Questions In C++

0 votes
0 answers

When and why do I need to use cin.ignore() in C++?

In C++, I developed a simple application that requested the user to enter a number and then a string.  Surprisingly, when I ran the application, it never paused to ask for the string.  It simply ignored it.  After conducting some research on StackOverflow, I discovered that I needed to include the following line: cin.ignore(256, '\n'); before the line with the string input  That addressed the problem and allowed the software to run.  My issue is why C++ need the cin.ignore() line, and how can I forecast when I will need to use it. Here's the software I created: #include <iostream> #include <string> using namespace std; int main() { ...READ MORE

Jul 4, 2022 in C++ by Nicholas
• 7,760 points
522 views
0 votes
0 answers

What does the C++ standard state the size of int, long type to be?

I'm seeking for specific information on the sizes of basic C++ types.  I understand that it is determined by the architecture (16 bits, 32 bits, or 64 bits) and the compiler. But are there any C++ standards? On a 32-bit architecture, I'm using Visual Studio 2008.  This is what I get: char : 1 byte short : 2 ...READ MORE

Jul 5, 2022 in C++ by Nicholas
• 7,760 points
228 views
0 votes
0 answers

What does the C++ standard state the size of int, long type to be?

I'm seeking for specific information on the sizes of basic C++ types.  I understand that it is determined by the architecture (16 bits, 32 bits, or 64 bits) and the compiler. But are there any C++ standards? On a 32-bit architecture, I'm using Visual Studio 2008.  This is what I get: char : 1 byte short : 2 ...READ MORE

Jul 7, 2022 in C++ by Nicholas
• 7,760 points
179 views
0 votes
0 answers

What does '&' do in a C++ declaration?

As a C person, I'm attempting to comprehend some C++ code.  The declaration of my function is as follows: int foo(const string &myname) { cout ...READ MORE

Aug 17, 2022 in C++ by Nicholas
• 7,760 points
198 views
0 votes
0 answers

Is hiding implementation detail Encapsulation or Abstraction?

I have seen some people defining abstraction ...READ MORE

May 6, 2022 in Java by narikkadan
• 63,420 points
2,451 views
0 votes
1 answer

Why would anyone use set instead of unordered_set?

Unordered sets must compensate for their O(1) ...READ MORE

answered Jun 1, 2022 in C++ by Damon
• 4,960 points
2,221 views
0 votes
1 answer

What is a smart pointer and when should I use one?

A smart pointer is similar to a ...READ MORE

answered Jun 2, 2022 in C++ by Damon
• 4,960 points
260 views
0 votes
1 answer

The new syntax "= default" in C++11

A defaulted default function Object() { [native code] } is defined as a user-defined default function Object() { [native code] } with an empty compound statement and no initialization list. I'll give you an example to demonstrate the difference: #include <iostream> using namespace std; class A { public: ...READ MORE

answered Jun 7, 2022 in C++ by Damon
• 4,960 points
302 views
0 votes
1 answer

What is compile-time polymorphism and why does it only apply to functions?

"Compile time polymorphism" used to signify function overloading.  It only applies to functions because that's all you can overload. Templates in modern C++ modify this.  One example has previously been provided by Neil Butterworth.  Another technique is template specialisation.  As an example: #include <iostream> #include <string> template <class T> struct my_template { ...READ MORE

answered Jun 20, 2022 in C++ by Damon
• 4,960 points
465 views
0 votes
1 answer

why are copy constructors needed and what are the cases where they are very helpful?

A copy constructor is a member function ...READ MORE

answered May 31, 2022 in C++ by Damon
• 4,960 points
450 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP