Programatica Tools

Directory tools/base/parse2

The README file is displayed below the directory index.

Name	Last modified	Size
Parent Directory		-
HuMakefile	2003-09-30 19:11	79
LexUtil.hs	2001-11-16 23:35	1.4K
Lexer/	2009-01-04 11:20	-
LexerGen/	2009-01-04 11:21	-
LexerNotes.html	2003-05-08 20:37	3.1K
LexerSpec/	2009-01-04 11:34	-
Makefile	2009-01-04 11:18	940
NumberNames.hs	2001-12-08 02:18	216
ParseMonad.hs	2006-06-09 22:39	2.1K
ParsedSyntax.hs	2001-12-08 01:03	127
Parser/	2005-06-01 23:00	-
ParserOptions.hs	2005-10-11 01:09	969
PosName.hs	2004-04-02 23:34	238
SourceNames.hs	2009-01-04 11:18	2.4K
lexgen	2001-11-30 01:55	145
pfe.cgi	2018-03-29 23:19	6.3K

Extensible Haskell front-end for the Programatica project - directory base/parse2

Extensible Haskell front-end for the Programatica project

What is in this directory (base/parse2)

This directory contains a new version of the base language lexer and parser. There are two main differences from the old version.

Changes to the parser

The new parser produces a variant av the abstract syntax where every identifier is paired with its position in the source file. This variant of the syntax is defined in module PosSyntax. It reuses the names of all types from the plain abstract syntax, to minimize the number of changes required in the grammar file. The type used for identifiers with source position is called SN and defined in module SourceNames.

Changes to the lexer

(See also the slides from the talk A Lexer for Haskell in Haskell).

Instead of being handwritten in Haskell, the new lexer is generated from a lexical syntax specification. The advantages with this approach are that

it allows the lexer to be implemented in a modular way, closely resembling the specification in the Haskell report, so it should be much easier to verify that the implementation agrees with what the Haskell report specifies. (The old lexer was buggy.)
it should be much easier to adapt the implementation to changes of the Haskell report (and there has been subtle changes in the lexical syntax in every new version of the Haskell report, I believe). (The old lexer was outdated.)
it is still as efficient as the old handwritten, monolithic lexer.

The specification is expressed in Haskell, using simple regular expression combinators, and then compiled to a DFA using standard text book algorithms. The regular expression compiler is implemented in Haskell, and the DFAs it generates are output in the form of Haskell source code.

Two pieces of handwritten code accompany the automatically generated code:

A small function to recognize nested comments, since these can't be described using regular expressions.
Functions to implement Haskell's layout convention. The structure of the implementation of these closely follow the specifications in the Haskell report (appendix B.3).

Subdirectories

Lexer: Handwritten and automatically generated code for the lexer.
LexerGen: The regular expression compiler.
LexerSpec: The lexical syntax specification for Haskell 98 (based on appendix B.2).
Parser: The Happy parser for the context free grammar (based on appendix B.4).