Book Image

Build Your Own Programming Language - Second Edition

By : Clinton L. Jeffery
Book Image

Build Your Own Programming Language - Second Edition

By: Clinton L. Jeffery

Overview of this book

There are many reasons to build a programming language: out of necessity, as a learning exercise, or just for fun. Whatever your reasons, this book gives you the tools to succeed. You’ll build the frontend of a compiler for your language and generate a lexical analyzer and parser using Lex and YACC tools. Then you’ll explore a series of syntax tree traversals before looking at code generation for a bytecode virtual machine or native code. In this edition, a new chapter has been added to assist you in comprehending the nuances and distinctions between preprocessors and transpilers. Code examples have been modernized, expanded, and rigorously tested, and all content has undergone thorough refreshing. You’ll learn to implement code generation techniques using practical examples, including the Unicon Preprocessor and transpiling Jzero code to Unicon. You'll move to domain-specific language features and learn to create them as built-in operators and functions. You’ll also cover garbage collection. Dr. Jeffery’s experiences building the Unicon language are used to add context to the concepts, and relevant examples are provided in both Unicon and Java so that you can follow along in your language of choice. By the end of this book, you'll be able to build and deploy your own domain-specific language.

Related Content you might be interested in

Current Title:

Small book image
Build Your Own Programming Language - Second Edition
Clinton L. Jeffery
Jan, 2024 | 556 pages
No titles found
Table of Contents (27 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
1
Section I: Programming Language Frontends
Section I: Programming Language Frontends
Free Chapter
2
Why Build Another Programming Language?
Why Build Another Programming Language?
Motivations for writing your own programming language
Types of programming language implementations
Organizing a bytecode language implementation
Languages used in the examples
The difference between programming languages and libraries
Applicability to other software engineering tasks
Establishing the requirements for your language
Case study – requirements that inspired the Unicon language
Summary
Questions
3
Programming Language Design
Programming Language Design
Determining the kinds of words and punctuation to provide in your language
Specifying the control flow
Deciding on what kinds of data to support
Overall program structure
Completing the Jzero language definition
Case study – designing graphics facilities in Unicon
Summary
Questions
4
Scanning Source Code
Scanning Source Code
Technical requirements
Lexemes, lexical categories, and tokens
Regular expressions
Using UFlex and JFlex
Writing a scanner for Jzero
Regular expressions are not always enough
Summary
Questions
5
Parsing
Parsing
Technical requirements
Syntax analysis
Context-free grammars
Using iyacc and BYACC/J
Writing a parser for Jzero
Summary
Questions
6
Syntax Trees
Syntax Trees
Technical requirements
Using GNU Make
Learning about trees
Creating leaves from terminal symbols
Building internal nodes from production rules
Forming syntax trees for the Jzero language
Debugging and testing your syntax tree
Summary
Questions
7
Section II: Syntax Tree Traversals
Section II: Syntax Tree Traversals
8
Symbol Tables
Symbol Tables
Technical requirements
Establishing the groundwork for symbol tables
Creating and populating symbol tables for each scope
Checking for undeclared variables
Finding redeclared variables
Handling package and class scopes in Unicon
Testing and debugging symbol tables
Summary
Questions
9
Checking Base Types
Checking Base Types
Technical requirements
Type representation in the compiler
Assigning type information to declared variables
Determining the type at each syntax tree node
Runtime type checks and type inference in Unicon
Summary
Questions
10
Checking Types on Arrays, Method Calls, and Structure Accesses
Checking Types on Arrays, Method Calls, and Structure Accesses
Technical requirements
Checking operations on array types
Checking method calls
Checking structured type accesses
Summary
Questions
11
Intermediate Code Generation
Intermediate Code Generation
Technical requirements
What is intermediate code?
An intermediate code instruction set
Annotating syntax trees with labels for control flow
Generating code for expressions
Generating code for control flow
Summary
Questions
12
Syntax Coloring in an IDE
Syntax Coloring in an IDE
Writing your own IDE versus supporting an existing one
Downloading the software used in this chapter
Adding support for your language to Visual Studio Code
Integrating a compiler into a programmer’s editor
Avoiding reparsing the entire file on every change
Using lexical information to colorize tokens
Highlighting errors using parse results
Summary
Questions
13
Section III: Code Generation and Runtime Systems
Section III: Code Generation and Runtime Systems
14
Preprocessors and Transpilers
Preprocessors and Transpilers
Understanding preprocessors
Code generation in the Unicon preprocessor
The difference between preprocessors and transpilers
Transpiling Jzero code to Unicon
Summary
Questions
15
Bytecode Interpreters
Bytecode Interpreters
Technical requirements
Understanding what bytecode is
Comparing bytecode with intermediate code
Building a bytecode instruction set for Jzero
Implementing a bytecode interpreter
Writing a runtime system for Jzero
Running a Jzero program
Examining iconx, the Unicon bytecode interpreter
Summary
Questions
16
Generating Bytecode
Generating Bytecode
Technical requirements
Converting intermediate code to Jzero bytecode
Comparing bytecode assembler with binary formats
Linking, loading, and including the runtime system
Unicon example – bytecode generation in icont
Summary
Questions
17
Native Code Generation
Native Code Generation
Technical requirements
Deciding whether to generate native code
Introducing the x64 instruction set
Using registers
Converting intermediate code to x64 code
Generating x64 output
Summary
Questions
Leave a review!
18
Implementing Operators and Built-In Functions
Implementing Operators and Built-In Functions
Implementing operators
Writing built-in functions
Integrating built-ins with control structures
Developing operators and functions for Unicon
Summary
Questions
19
Domain Control Structures
Domain Control Structures
Knowing when a new control structure is needed
Scanning strings in Icon and Unicon
Rendering regions in Unicon
Summary
Questions
20
Garbage Collection
Garbage Collection
Grasping the importance of garbage collection
Counting references to objects
Marking live data and sweeping the rest
Summary
Questions
21
Final Thoughts
Final Thoughts
Reflecting on what was learned from writing this book
Deciding where to go from here
Exploring references for further reading
Summary
22
Section IV: Appendix
Section IV: Appendix
23
Answers
Answers
24
Other Books You May Enjoy
Other Books You May Enjoy
25
Index
Index
Appendix: Unicon Essentials
Appendix: Unicon Essentials
Syntactic shorthand
Running Unicon
Using Unicon’s declarations and data types
Evaluating expressions
Debugging and environmental issues
Function mini-reference
Selected keywords

Motivations for writing your own programming language

Sure, some programming language inventors are rock stars of computer science, such as Dennis Ritchie or Guido van Rossum! Becoming a rock star in computer science was easier back in the previous century. In 1993, I heard the following report from an attendee of the second ACM History of Programming Languages Conference: “The consensus was that the field of programming languages is dead. All the important languages have been invented already.” This was proven wildly wrong a year or two later when Java hit the scene, and perhaps a dozen times since then when important languages such as Go emerged. After a mere six decades, it would be unwise to claim our field is mature and that there’s nothing new to invent that might make you famous.

In any case, celebrity is a bad reason to build a programming language. The chances of acquiring fame or fortune from your programming language invention are slim. Curiosity and a desire to know how things work are valid reasons, so long as you’ve got the time and inclination, but perhaps the best reason to build your own programming language is necessity.

Some folks need to build a new language, or a new implementation of an existing programming language, to target a new processor or compete with a rival company. If that’s not you, then perhaps you’ve looked at the best languages (and compilers or interpreters) available for some domain that you are developing programs for, and they are missing some key features for what you are doing, and those missing features are causing you pain. This is the stuff Master’s theses and PhD dissertations are made of. Every once in a blue moon, someone comes up with a whole new style of computing for which a new programming paradigm requires a new language.

While we are discussing your motivations for building a language, let’s also talk about the different kinds of languages, how they are organized, and the examples this book will use to guide you.

Previous Section
End of Section 2
Next Section