Book Image

Build Your Own Programming Language - Second Edition

By : Clinton L. Jeffery
Book Image

Build Your Own Programming Language - Second Edition

By: Clinton L. Jeffery

Overview of this book

There are many reasons to build a programming language: out of necessity, as a learning exercise, or just for fun. Whatever your reasons, this book gives you the tools to succeed. You’ll build the frontend of a compiler for your language and generate a lexical analyzer and parser using Lex and YACC tools. Then you’ll explore a series of syntax tree traversals before looking at code generation for a bytecode virtual machine or native code. In this edition, a new chapter has been added to assist you in comprehending the nuances and distinctions between preprocessors and transpilers. Code examples have been modernized, expanded, and rigorously tested, and all content has undergone thorough refreshing. You’ll learn to implement code generation techniques using practical examples, including the Unicon Preprocessor and transpiling Jzero code to Unicon. You'll move to domain-specific language features and learn to create them as built-in operators and functions. You’ll also cover garbage collection. Dr. Jeffery’s experiences building the Unicon language are used to add context to the concepts, and relevant examples are provided in both Unicon and Java so that you can follow along in your language of choice. By the end of this book, you'll be able to build and deploy your own domain-specific language.

Related Content you might be interested in

Current Title:

Small book image
Build Your Own Programming Language - Second Edition
Clinton L. Jeffery
Jan, 2024 | 556 pages
No titles found
Table of Contents (27 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
1
Section I: Programming Language Frontends
Section I: Programming Language Frontends
Free Chapter
2
Why Build Another Programming Language?
Why Build Another Programming Language?
Motivations for writing your own programming language
Types of programming language implementations
Organizing a bytecode language implementation
Languages used in the examples
The difference between programming languages and libraries
Applicability to other software engineering tasks
Establishing the requirements for your language
Case study – requirements that inspired the Unicon language
Summary
Questions
3
Programming Language Design
Programming Language Design
Determining the kinds of words and punctuation to provide in your language
Specifying the control flow
Deciding on what kinds of data to support
Overall program structure
Completing the Jzero language definition
Case study – designing graphics facilities in Unicon
Summary
Questions
4
Scanning Source Code
Scanning Source Code
Technical requirements
Lexemes, lexical categories, and tokens
Regular expressions
Using UFlex and JFlex
Writing a scanner for Jzero
Regular expressions are not always enough
Summary
Questions
5
Parsing
Parsing
Technical requirements
Syntax analysis
Context-free grammars
Using iyacc and BYACC/J
Writing a parser for Jzero
Summary
Questions
6
Syntax Trees
Syntax Trees
Technical requirements
Using GNU Make
Learning about trees
Creating leaves from terminal symbols
Building internal nodes from production rules
Forming syntax trees for the Jzero language
Debugging and testing your syntax tree
Summary
Questions
7
Section II: Syntax Tree Traversals
Section II: Syntax Tree Traversals
8
Symbol Tables
Symbol Tables
Technical requirements
Establishing the groundwork for symbol tables
Creating and populating symbol tables for each scope
Checking for undeclared variables
Finding redeclared variables
Handling package and class scopes in Unicon
Testing and debugging symbol tables
Summary
Questions
9
Checking Base Types
Checking Base Types
Technical requirements
Type representation in the compiler
Assigning type information to declared variables
Determining the type at each syntax tree node
Runtime type checks and type inference in Unicon
Summary
Questions
10
Checking Types on Arrays, Method Calls, and Structure Accesses
Checking Types on Arrays, Method Calls, and Structure Accesses
Technical requirements
Checking operations on array types
Checking method calls
Checking structured type accesses
Summary
Questions
11
Intermediate Code Generation
Intermediate Code Generation
Technical requirements
What is intermediate code?
An intermediate code instruction set
Annotating syntax trees with labels for control flow
Generating code for expressions
Generating code for control flow
Summary
Questions
12
Syntax Coloring in an IDE
Syntax Coloring in an IDE
Writing your own IDE versus supporting an existing one
Downloading the software used in this chapter
Adding support for your language to Visual Studio Code
Integrating a compiler into a programmer’s editor
Avoiding reparsing the entire file on every change
Using lexical information to colorize tokens
Highlighting errors using parse results
Summary
Questions
13
Section III: Code Generation and Runtime Systems
Section III: Code Generation and Runtime Systems
14
Preprocessors and Transpilers
Preprocessors and Transpilers
Understanding preprocessors
Code generation in the Unicon preprocessor
The difference between preprocessors and transpilers
Transpiling Jzero code to Unicon
Summary
Questions
15
Bytecode Interpreters
Bytecode Interpreters
Technical requirements
Understanding what bytecode is
Comparing bytecode with intermediate code
Building a bytecode instruction set for Jzero
Implementing a bytecode interpreter
Writing a runtime system for Jzero
Running a Jzero program
Examining iconx, the Unicon bytecode interpreter
Summary
Questions
16
Generating Bytecode
Generating Bytecode
Technical requirements
Converting intermediate code to Jzero bytecode
Comparing bytecode assembler with binary formats
Linking, loading, and including the runtime system
Unicon example – bytecode generation in icont
Summary
Questions
17
Native Code Generation
Native Code Generation
Technical requirements
Deciding whether to generate native code
Introducing the x64 instruction set
Using registers
Converting intermediate code to x64 code
Generating x64 output
Summary
Questions
Leave a review!
18
Implementing Operators and Built-In Functions
Implementing Operators and Built-In Functions
Implementing operators
Writing built-in functions
Integrating built-ins with control structures
Developing operators and functions for Unicon
Summary
Questions
19
Domain Control Structures
Domain Control Structures
Knowing when a new control structure is needed
Scanning strings in Icon and Unicon
Rendering regions in Unicon
Summary
Questions
20
Garbage Collection
Garbage Collection
Grasping the importance of garbage collection
Counting references to objects
Marking live data and sweeping the rest
Summary
Questions
21
Final Thoughts
Final Thoughts
Reflecting on what was learned from writing this book
Deciding where to go from here
Exploring references for further reading
Summary
22
Section IV: Appendix
Section IV: Appendix
23
Answers
Answers
24
Other Books You May Enjoy
Other Books You May Enjoy
25
Index
Index
Appendix: Unicon Essentials
Appendix: Unicon Essentials
Syntactic shorthand
Running Unicon
Using Unicon’s declarations and data types
Evaluating expressions
Debugging and environmental issues
Function mini-reference
Selected keywords

Summary

In this chapter, you learned the difference between inventing a programming language and inventing a library API to support whatever kinds of computing you want to do. Several different forms of programming language implementations were considered. This first chapter allowed you to think about functional and non-functional requirements for your own language.

These requirements might be different for your language than the example requirements discussed for the Java subset Jzero and the Unicon programming language, which were both introduced.

Requirements are important because they allow you to set goals and define what success will look like. In the case of a programming language implementation, the requirements include what things will look and feel like for the programmers that use your language, as well as what hardware and software platforms it must run on. The look and feel of a programming language include answering both external questions regarding how the language implementation and the programs written in the language are invoked, as well as internal issues such as verbosity: how much the programmer must write to accomplish a given compute task.

You may be keen to get straight to the coding part. Although the classic build-and-fix mentality of novice programmers might work on scripts and short programs, for a piece of software as large as a programming language, we need a bit more planning first. After this chapter’s coverage of the requirements, Chapter 2, Programming Language Design, will prepare you to construct a detailed plan for the implementation, which will occupy our attention for the remainder of this book!

Previous Section
End of Section 10
Next Section