Book Image

Build Your Own Programming Language - Second Edition

By : Clinton L. Jeffery
Book Image

Build Your Own Programming Language - Second Edition

By: Clinton L. Jeffery

Overview of this book

There are many reasons to build a programming language: out of necessity, as a learning exercise, or just for fun. Whatever your reasons, this book gives you the tools to succeed. You’ll build the frontend of a compiler for your language and generate a lexical analyzer and parser using Lex and YACC tools. Then you’ll explore a series of syntax tree traversals before looking at code generation for a bytecode virtual machine or native code. In this edition, a new chapter has been added to assist you in comprehending the nuances and distinctions between preprocessors and transpilers. Code examples have been modernized, expanded, and rigorously tested, and all content has undergone thorough refreshing. You’ll learn to implement code generation techniques using practical examples, including the Unicon Preprocessor and transpiling Jzero code to Unicon. You'll move to domain-specific language features and learn to create them as built-in operators and functions. You’ll also cover garbage collection. Dr. Jeffery’s experiences building the Unicon language are used to add context to the concepts, and relevant examples are provided in both Unicon and Java so that you can follow along in your language of choice. By the end of this book, you'll be able to build and deploy your own domain-specific language.

Related Content you might be interested in

Current Title:

Small book image
Build Your Own Programming Language - Second Edition
Clinton L. Jeffery
Jan, 2024 | 556 pages
No titles found
Table of Contents (27 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
1
Section I: Programming Language Frontends
Section I: Programming Language Frontends
Free Chapter
2
Why Build Another Programming Language?
Why Build Another Programming Language?
Motivations for writing your own programming language
Types of programming language implementations
Organizing a bytecode language implementation
Languages used in the examples
The difference between programming languages and libraries
Applicability to other software engineering tasks
Establishing the requirements for your language
Case study – requirements that inspired the Unicon language
Summary
Questions
3
Programming Language Design
Programming Language Design
Determining the kinds of words and punctuation to provide in your language
Specifying the control flow
Deciding on what kinds of data to support
Overall program structure
Completing the Jzero language definition
Case study – designing graphics facilities in Unicon
Summary
Questions
4
Scanning Source Code
Scanning Source Code
Technical requirements
Lexemes, lexical categories, and tokens
Regular expressions
Using UFlex and JFlex
Writing a scanner for Jzero
Regular expressions are not always enough
Summary
Questions
5
Parsing
Parsing
Technical requirements
Syntax analysis
Context-free grammars
Using iyacc and BYACC/J
Writing a parser for Jzero
Summary
Questions
6
Syntax Trees
Syntax Trees
Technical requirements
Using GNU Make
Learning about trees
Creating leaves from terminal symbols
Building internal nodes from production rules
Forming syntax trees for the Jzero language
Debugging and testing your syntax tree
Summary
Questions
7
Section II: Syntax Tree Traversals
Section II: Syntax Tree Traversals
8
Symbol Tables
Symbol Tables
Technical requirements
Establishing the groundwork for symbol tables
Creating and populating symbol tables for each scope
Checking for undeclared variables
Finding redeclared variables
Handling package and class scopes in Unicon
Testing and debugging symbol tables
Summary
Questions
9
Checking Base Types
Checking Base Types
Technical requirements
Type representation in the compiler
Assigning type information to declared variables
Determining the type at each syntax tree node
Runtime type checks and type inference in Unicon
Summary
Questions
10
Checking Types on Arrays, Method Calls, and Structure Accesses
Checking Types on Arrays, Method Calls, and Structure Accesses
Technical requirements
Checking operations on array types
Checking method calls
Checking structured type accesses
Summary
Questions
11
Intermediate Code Generation
Intermediate Code Generation
Technical requirements
What is intermediate code?
An intermediate code instruction set
Annotating syntax trees with labels for control flow
Generating code for expressions
Generating code for control flow
Summary
Questions
12
Syntax Coloring in an IDE
Syntax Coloring in an IDE
Writing your own IDE versus supporting an existing one
Downloading the software used in this chapter
Adding support for your language to Visual Studio Code
Integrating a compiler into a programmer’s editor
Avoiding reparsing the entire file on every change
Using lexical information to colorize tokens
Highlighting errors using parse results
Summary
Questions
13
Section III: Code Generation and Runtime Systems
Section III: Code Generation and Runtime Systems
14
Preprocessors and Transpilers
Preprocessors and Transpilers
Understanding preprocessors
Code generation in the Unicon preprocessor
The difference between preprocessors and transpilers
Transpiling Jzero code to Unicon
Summary
Questions
15
Bytecode Interpreters
Bytecode Interpreters
Technical requirements
Understanding what bytecode is
Comparing bytecode with intermediate code
Building a bytecode instruction set for Jzero
Implementing a bytecode interpreter
Writing a runtime system for Jzero
Running a Jzero program
Examining iconx, the Unicon bytecode interpreter
Summary
Questions
16
Generating Bytecode
Generating Bytecode
Technical requirements
Converting intermediate code to Jzero bytecode
Comparing bytecode assembler with binary formats
Linking, loading, and including the runtime system
Unicon example – bytecode generation in icont
Summary
Questions
17
Native Code Generation
Native Code Generation
Technical requirements
Deciding whether to generate native code
Introducing the x64 instruction set
Using registers
Converting intermediate code to x64 code
Generating x64 output
Summary
Questions
Leave a review!
18
Implementing Operators and Built-In Functions
Implementing Operators and Built-In Functions
Implementing operators
Writing built-in functions
Integrating built-ins with control structures
Developing operators and functions for Unicon
Summary
Questions
19
Domain Control Structures
Domain Control Structures
Knowing when a new control structure is needed
Scanning strings in Icon and Unicon
Rendering regions in Unicon
Summary
Questions
20
Garbage Collection
Garbage Collection
Grasping the importance of garbage collection
Counting references to objects
Marking live data and sweeping the rest
Summary
Questions
21
Final Thoughts
Final Thoughts
Reflecting on what was learned from writing this book
Deciding where to go from here
Exploring references for further reading
Summary
22
Section IV: Appendix
Section IV: Appendix
23
Answers
Answers
24
Other Books You May Enjoy
Other Books You May Enjoy
25
Index
Index
Appendix: Unicon Essentials
Appendix: Unicon Essentials
Syntactic shorthand
Running Unicon
Using Unicon’s declarations and data types
Evaluating expressions
Debugging and environmental issues
Function mini-reference
Selected keywords

The difference between programming languages and libraries

Unless you are in it for the “fun” or the intellectual experience, building a programming language is a lot of work that might not be necessary. If your motives are strictly utilitarian, you don’t have to make a programming language when a library will do the job. Libraries are by far the most common way to extend an existing programming language to perform a new task. A library is a set of functions or classes that can be used together to write applications for some hardware or software technology. Many languages, including C and Java, are designed almost completely to revolve around a rich set of libraries. The language itself is very simple and general, while much of what a developer must learn to develop applications consists of how to use the various libraries.

The following is what libraries can do:

  • Introduce new data types (classes) and provide public functions (an API) to manipulate them
  • Provide a layer of abstraction on top of a set of hardware or operating system calls

The following is what libraries cannot do:

  • Introduce new control structures and syntax in support of new application domains
  • Embed/support new semantics within the existing language runtime system

Libraries do some things badly, so you might end up preferring to make a new language:

  • Libraries often get larger and more complex than necessary.
  • Libraries can have even steeper learning curves and poorer documentation than languages.
  • Every so often, libraries have conflicts with other libraries.
  • Applications that use libraries can become broken if the library changes incompatibly in a later version.

There is a natural evolutionary path from a library to a language. A reasonable approach to building a new language to support an application domain is to start by making or buying the best library available for that application domain. If the result does not meet your requirements in terms of supporting the domain and simplifying the task of writing programs for that domain, then you have a strong argument for a new language.

This book is about building your own language, not just building your own library. It turns out that learning about tools and techniques to implement programming languages is useful in many other contexts.

Previous Section
End of Section 6
Next Section