collageCreating A Programming Language From Scratch
Key Concepts & What You Need to Know
What?
In this article, we’ll go through the most important parts and concepts of designing a new language. We will analyze the most important points, and the logic behind them, but not the exact lines of code that make them do what they do. This way, you can follow along using the language you feel most comfortable with.
Why?
There are approximately 700 programming languages being used today by the world-wide code community. Some of them are made specifically to run on your web-browser and others are made to run on satellites that orbit the earth. With so many languages available, the question that remains is: why would we want to create a new one?
It doesn’t matter if you want to create a new language to solve a specific problem, or just to learn the inner workings of an interpreter or compiler – In both cases, you will need to understand the major steps of how source code gets translated into a set of instructions for a computer to execute.
How?
In this article, we will go through, together, the three main steps that transform text into instructions for a machine to compute. We will do this by analyzing the theory behind each step of creating an interpreted language, using examples from the source code of the ZAP! interpreter.
The ZAP! language is a javascript interpreted language with a specific set of keywords. If you want to have a taste for the language before we begin, feel free to check out the online interpreter at https://jzsiggy.github.io/ZAP/ .
The source code can be found at: https://github.com/jzsiggy/ZAP/tree/master/jzap
Now that you’ve had a look at what we can create, let’s dive in!
Step 1: The Lexer
The first step in creating our interpreter is to receive an input stream and separate it into tokens recognized by our language’s semantics.
A token is a data structure that transforms a string that has an underlying value to our language into something our interpreter can comprehend. The following image might make things clearer:
In ZAP!, the source code enters the interpreter as a single string. The Lexer analyzes the input character by character, creating a list of the tokens that our interpreter will evaluate.
Token Types
Step 2: Evaluating Expressions
The two major players in any language are expressions and statements. first we’ll tackle expressions and then move on to parsing full statements.
In fact, expressions are a special type of statement. Expressions are at any part of the source code where we must return a single value. In the image below, we will analyze some examples.
When we initialize a variable and set its value to 3 + 5, we also use an expression within a variable declaration statement.
The Evaluator class will be invoked any time our interpreter finds an expression. Since our evaluator, as in Python, Javascript, C and others, follows the PEMDAS order of operations, we must build our code accordingly.
The logic behind the Evaluator is to create a tree of operations. It will iterate through all tokens in the expression from right to left, searching for the operators with lowest precedence. This will break the expression into two other expressions to be evaluated beforehand.
The expressions can be of four types:
Binary Expressions:
These expressions are composed by a left expression node, an operator and a right expression node. Once executed, this expression will return the result of the operation induced by the operator with the values of the two child nodes.
Unary Expressions
These expressions are represented by a single operator and a child expression. The operator can be ‘-’ (Negative) or ‘!’ (Not). The resulting value will be the value of the child expression joined by the operator.
IMPORTANT NOTE! — The not operator always returns a bool value.
Groups
Group expressions are the expressions with highest precedence. They are the first ones to be executed. Groups are formed by expressions within parenthesis.
Literals
These are the smallest bits of our interpreter. they represent primary values, and can be strings, bools or numbers.
Our evaluator will iterate through a raw expression breaking it down into smaller and smaller sub expressions until it works with only primaries.
To follow the PEMDAS order of operations, the bigger expression is always split at the operators of lowest precedence first. The tree is then executed from the bottom up, guaranteeing that multiplications always happen before additions for example.
Step 3: Parsing Statements
Now that we’ve gone through the process of evaluating expressions, we can start to parse statements. This is where our programming language will start looking less like an algebraic calculator and more like a multi-purpose, functional language.
To begin, we’ll admit a new rule to our language semantics: all statements must be followed by semicolons.
This way, we can divide our token list, generated by our Lexer, at every semicolon. Each subdivision will be parsed as a statement.
Our language will have 5 different types of statements:
Expression Statements
These statements have no output. Nothing gets printed to the StdOut, however, our interpreter does some work behind the curtains. The semantics of this statement is very simple: no keywords, just an expression. The Interpreter will then call the Evaluator to evaluate the expression.
Variable Declarations
These statements make it possible for us to reuse and manipulate data. In ZAP!, we initialize a variable with the ‘@’ keyword. This way, When our Parser class encounters an ‘@’ at the beginning of a statement it knows the structure the rest of the statement has to follow: First, the name of the variable: an Identifier; Then an equals sign and finally an expression.
When we look at Environments, we will find out how our interpreter stores these values!
Show Statements
These statements make it possible for ZAP! to show information on the StdOut. For the interpreter to understand it’s dealing with a show, the statement must begin with the ‘show’ keyword, followed by the expression of which it must show the value.
Block Statements
Block statements are nothing more than a series of statements incapsulated in braces. They are important because as we will soon see, each block statement has its own scope; that is, variables declared in a nested block statement cannot be accessed from outside of it.
Conditional Statements
This is where our language takes a big step towards Turing Completeness. Using If / Else Statements, we can choose what part of our source code we want to execute depending on the state of our interpreter. A conditional statement starts with the ‘if’ keyword, followed by an expression and a block statement.
The block statement may or may not be followed by the ‘else’ keyword, and another block statement.
If the expression following the ‘if’ keyword is truthy, the code in the following block statement will be executed, else, the code in the ‘else’ statement is executed.
Loops
Loop statements are key in any programming language. With loops, The number of instructions our interpreter executes can start becoming way larger than the size of our source code.
In ZAP!, we only allow while loops, defined by the while keyword, followed by an expression. We then open a block statement with the instructions of the body of our loop. While the expression evaluates to a truthy value, the body statement will be executed.
Function Declarations
We’ve already found ways to reuse and store data; variables. Now it’s time we teach our interpreter to reuse and store code. Using functions, we can write a set of instructions for our interpreter to follow with the statements available to us. When we call our function, we can pass in arguments on which the instructions will be applied.
In ZAP! A function statement starts with the ‘fn’ keyword, followed by an identifier specifying the function’s name. The interpreter will then look for a list of arguments enclosed by two bars (i.e. ‘|’). After the arguments have been specified, we must follow with a fat arrow (i.e. ‘=>’) and a block statement.
The block statement is the function’s body. It specifies the instructions to be applied on the arguments.
To call the functions, we must type the function’s name, followed by the list of arguments (if any) enclosed in bars.
IMPORTANT NOTE! — Our interpreter will evaluate a function call as an expression, not a statement! — But we will look deeper into this soon.
That’s it for the steps in crafting our interpreter, however we must still clarify some *VERY* important concepts!
Environments
When we defined all of our statements, two of them required storing something in our interpreters memory — variable and function declarations.
Now we’ll look deeper into how our interpreter can store this data, and how these structures will be scoped.
Let’s start by defining ‘scope’. The scope of a variable or function is defined by who can access these values. When we talk about scoping, two words instantly come to mind: ‘local’ and ‘global’. These words have everything to do with scope. A globally scoped variable will be accessible to any statement that tries to call it, independent of where it’s located in the source code. Meanwhile, locally scoped data can only be accessible by statements in there scope.
In ZAP!, scopes are defined by block statements. When we nest block statements one inside the other, variables and functions defined in the child blocks cannot be accessed by statements in parent blocks. To understand why, we must understand how we store these values: Environments.
To store values in ZAP!, and in many other programming languages, we must instantiate the Environment class. This data structure will have a values attribute, and some methods to define, fetch and call variables and functions.
Every time we run into a variable declaration, we will call the define method of our current environment passing our variable name, and it’s value.
Another important attribute of the Environment class is the ‘enclosing’ Attribute. This defines the scope of our environment. We can have multiple environments enclosed in one another. This happens when we nest block statements. When we start a new block statement, we create a new Environment, and set it’s enclosing environment to the previous.
When we try to get a variable value from our environment, our interpreter first searches our directly enclosed environment, and then goes to the latter environments to try to fetch the value.
This way, children statements can access variables defined in parents, but the contrary will throw an error.
Calling Functions
We have reached the most complicated, but also most beautiful part of our interpreter: calling functions.
In our environment, it is pretty straight forward to save a variable: we pass it’s name and value, and voilà, its saved. However, how would we go about doing the same thing for functions? Enter a new class — ZapFunction.
Our ZapFunction class will have a body and arguments attribute, and a call method.
When we stumble upon a function declaration in ZAP!, we divide the statement into name, arguments, and body. The name is stored as a key in our environment’s values, while the arguments and body is used to instantiate a new ZapFunction. This instance is then referenced as the value pair to our functions name in the environment.
When the function gets called, the interpreter will look for the function name in the environment and check if its value is a ZapFunction instance. If so, it will create a new environment, just for the function, and map the expected arguments as variables who’s values are equivalent to the arguments passed by the user. Then, the function’s body will be executed within this new environment. This is all defined inside the call method of the ZapFunction instance.
phew! That’s a lot of stuff! Now we’re almost done fully understanding our interpreter!
Handling Errors
Throughout this article, many times I cited what keywords or tokens our interpreter was looking for when parsing a statement or evaluating an expression, but what happens when it doesn’t find what it was looking for?
Following good practices, we will create a class just for that — Error handling. This way, we can handle all of our errors in the same organized way.
This class will have a ‘throw’ method that exits the current process and throws an error message we pass in as an argument.
That’s it for handling errors!
Summing up & crafting our interpreter
Let’s recap to see if everything is crystal clear…
We’ll start building our interpreter by the Lexer class, who’s job is to divide the source code into pieces that our interpreter can understand. These pieces are called tokens and have types and values.
The second step is to design the Evaluator class that evaluates expressions and returns their values. This is a complicated task seeing that we have many types of operators and the order of execution must follow PEMDAS. The evaluator will be heavily used by our Parser.
Yes, that’s the third step — Building the Parser. The Parser will separate our list of tokens at the semicolons to create a more significant list: The list of statements. By analyzing each statement, it is also the Parser’s job to classify the statements into the types we saw earlier.
In the middle of all this, we also saw how our Environment class will help us store values, how the ErrorHandler will help throw errors in an organized manner and how to call functions the right way!
That’s it! We have passed through all the basic concepts of creating an interpreted function-based language.
How to Create a Programming Language using Python?
- Difficulty Level : Hard
- Last Updated : 10 Jul, 2020
In this article, we are going to learn how to create your own programming language using SLY(Sly Lex Yacc) and Python. Before we dig deeper into this topic, it is to be noted that this is not a beginner’s tutorial and you need to have some knowledge of the prerequisites given below.
Prerequisites
- Rough knowledge about compiler design.
- Basic understanding of lexical analysis, parsing and other compiler design aspects.
- Understanding of regular expressions.
- Familiarity with Python programming language.
Getting Started
Install SLY for Python. SLY is a lexing and parsing tool which makes our process much easier.
Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.
To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning – Basic Level Course
pip install sly
Building a Lexer
The first phase of a compiler is to convert all the character streams(the high level program that is written) to token streams. This is done by a process called lexical analysis. However, this process is simplified by using SLY
First let’s import all the necessary modules.
- Python3
from sly import Lexer |
Now let’s build a class BasicLexer which extends the Lexer class from SLY. Let’s make a compiler that makes simple arithmetic operations. Thus we will need some basic tokens such as NAME, NUMBER, STRING. In any programming language, there will be space between two characters. Thus we create an ignore literal. Then we also create the basic literals like ‘=’, ‘+’ etc., NAME tokens are basically names of variables, which can be defined by the regular expression [a-zA-Z_][a-zA-Z0-9_]*. STRING tokens are string values and are bounded by quotation marks(” “). This can be defined by the regular expression ”.*?”.
Whenever we find digit/s, we should allocate it to the token NUMBER and the number must be stored as an integer. We are doing a basic programmable script, so let’s just make it with integers, however, feel free to extend the same for decimals, long etc., We can also make comments. Whenever we find “//”, we ignore whatever that comes next in that line. We do the same thing with new line character. Thus, we have build a basic lexer that converts the character stream to token stream.
- Python3
class BasicLexer(Lexer): tokens = { NAME, NUMBER, STRING } ignore = 't ' literals = { '=', '+', '-', '/', '*', '(', ')', ',', ';'} # Define tokens as regular expressions # (stored as raw strings) NAME = r'[a-zA-Z_][a-zA-Z0-9_]*' STRING = r'".*?""' # Number token @_(r'd+') def NUMBER(self
|
collage