2015-02-08 01:52:26 +00:00
cpp-peglib
==========
C++11 header-only [PEG ](http://en.wikipedia.org/wiki/Parsing_expression_grammar ) (Parsing Expression Grammars) library.
2015-02-13 02:08:58 +00:00
*cpp-peglib* tries to provide more expressive parsing experience in a simple way. This library depends on only one header file. So, you can start using it right away just by including `peglib.h` in your project.
2015-02-08 01:52:26 +00:00
2015-02-16 01:22:34 +00:00
The PEG syntax is well described on page 2 in the [document ](http://pdos.csail.mit.edu/papers/parsing:popl04.pdf ). *cpp-peglib* also supports the following additional syntax for now:
2015-02-18 23:00:11 +00:00
* `<` ... `>` (Anchor operator)
* `$<` ... `>` (Capture operator)
2015-03-04 03:08:18 +00:00
* `$name<` ... `>` (Named capture operator)
2015-02-18 23:00:11 +00:00
* `~` (Ignore operator)
2015-03-04 03:08:18 +00:00
* `\x20` (Hex number char)
2015-02-08 01:52:26 +00:00
2015-03-03 02:52:09 +00:00
This library also supports the linear-time parsing known as the [*Packrat* ](http://pdos.csail.mit.edu/~baford/packrat/thesis/thesis.pdf ) parsing.
2015-02-27 03:40:00 +00:00
2015-02-08 01:52:26 +00:00
How to use
----------
2015-02-15 22:52:39 +00:00
This is a simple calculator sample. It shows how to define grammar, associate samantic actions to the grammar and handle semantic values.
2015-02-08 01:52:26 +00:00
```c++
2015-02-15 22:52:39 +00:00
// (1) Include the header file
2015-02-12 02:04:08 +00:00
#include <peglib.h>
2015-03-11 17:53:24 +00:00
#include <assert.h>
2015-02-12 02:04:08 +00:00
using namespace peglib;
using namespace std;
int main(void) {
2015-02-15 22:52:39 +00:00
// (2) Make a parser
2015-02-13 02:08:58 +00:00
auto syntax = R"(
# Grammar for Calculator...
Additive < - Multitive ' + ' Additive / Multitive
Multitive < - Primary ' * ' Multitive / Primary
Primary < - ' ( ' Additive ' ) ' / Number
Number < - [ 0-9 ] +
)";
2015-02-14 15:13:10 +00:00
peg parser(syntax);
2015-02-13 02:08:58 +00:00
2015-02-15 22:52:39 +00:00
// (3) Setup an action
2015-02-13 02:08:58 +00:00
parser["Additive"] = {
2015-03-11 17:53:24 +00:00
nullptr, // Default action
2015-02-22 00:38:30 +00:00
[](const SemanticValues& sv) {
2015-03-11 17:53:24 +00:00
return sv[0].get< int > () + sv[1].get< int > (); // "Multitive '+' Additive"
2015-02-13 02:08:58 +00:00
},
2015-03-11 17:53:24 +00:00
[](const SemanticValues& sv) { return sv[0]; } // "Multitive"
2015-02-13 02:08:58 +00:00
};
2015-02-27 03:40:00 +00:00
parser["Multitive"] = [](const SemanticValues& sv) {
switch (sv.choice) {
2015-03-11 17:53:24 +00:00
case 0: // "Multitive '+' Additive"
2015-03-09 18:58:43 +00:00
return sv[0].get< int > () * sv[1].get< int > ();
2015-03-11 17:53:24 +00:00
default: // "Multitive"
2015-03-09 18:58:43 +00:00
return sv[0].get< int > ();
2015-02-27 03:40:00 +00:00
}
2015-02-13 02:08:58 +00:00
};
2015-03-09 18:58:43 +00:00
parser["Number"] = [](const char* s, size_t n) {
return stoi(string(s, n), nullptr, 10);
2015-02-13 02:08:58 +00:00
};
2015-02-15 22:52:39 +00:00
// (4) Parse
2015-03-11 17:53:24 +00:00
parser.packrat_parsing(true); // Enable packrat parsing.
2015-02-13 02:08:58 +00:00
int val;
2015-03-09 18:58:43 +00:00
parser.parse("(1+2)*3", val);
2015-02-13 02:08:58 +00:00
2015-02-16 03:21:18 +00:00
assert(val == 9);
2015-02-12 02:04:08 +00:00
}
```
2015-02-08 01:52:26 +00:00
2015-02-15 22:52:39 +00:00
Here is a complete list of available actions:
```c++
2015-02-22 00:38:30 +00:00
[](const SemanticValues& sv, any& dt)
[](const SemanticValues& sv)
2015-03-09 18:58:43 +00:00
[](const char* s, size_t n)
2015-02-15 22:52:39 +00:00
[]()
```
2015-02-22 00:38:30 +00:00
`const SemanticValues& sv` contains semantic values. `SemanticValues` structure is defined as follows.
2015-02-19 03:28:57 +00:00
```c++
2015-02-22 00:38:30 +00:00
struct SemanticValue {
peglib::any val; // Semantic value
std::string name; // Definition name for the sematic value
const char* s; // Token start for the semantic value
2015-03-09 18:58:43 +00:00
size_t n; // Token length for the semantic value
2015-03-11 18:10:59 +00:00
2015-04-08 16:02:26 +00:00
// Utility method
2015-03-11 18:10:59 +00:00
template < typename T > T& get();
template < typename T > const T& get() const;
std::string str() const;
2015-02-19 03:28:57 +00:00
};
2015-02-22 00:38:30 +00:00
struct SemanticValues : protected std::vector< SemanticValue >
{
2015-02-27 03:40:00 +00:00
const char* s; // Token start
2015-03-09 18:58:43 +00:00
size_t n; // Token length
2015-02-27 03:40:00 +00:00
size_t choice; // Choice number (0 based index)
2015-03-11 18:10:59 +00:00
using std::vector< T > ::size;
using std::vector< T > ::operator[];
using std::vector< T > ::begin;
using std::vector< T > ::end;
// NOTE: There are more std::vector methods available...
2015-04-08 16:02:26 +00:00
// Transform the semantice values vector to another vector
2015-03-11 18:10:59 +00:00
template < typename F > auto map(size_t beg, size_t end, F f) const -> vector< typename std::remove_const < decltype ( f ( SemanticValue ( ) ) ) > ::type>;
template < typename F > auto map(F f) const -> vector< typename std::remove_const < decltype ( f ( SemanticValue ( ) ) ) > ::type>;
template < typename T > auto map(size_t beg = 0, size_t end = -1) const -> vector< T > ;
2015-02-22 00:38:30 +00:00
}
2015-02-19 03:28:57 +00:00
```
2015-02-22 00:38:30 +00:00
`peglib::any` class is very similar to [boost::any ](http://www.boost.org/doc/libs/1_57_0/doc/html/any.html ). You can obtain a value by castning it to the actual type. In order to determine the actual type, you have to check the return value type of the child action for the semantic value.
2015-03-09 18:58:43 +00:00
`const char* s, size_t n` gives a pointer and length of the matched string. This is same as `sv.s` and `sv.n` .
2015-02-22 00:38:30 +00:00
`any& dt` is a data object which can be used by the user for whatever purposes.
2015-02-19 03:28:57 +00:00
The following example uses `<` ... ` >` operators. They are the *anchor* operators. Each anchor operator creates a semantic value that contains `const char*` of the position. It could be useful to eliminate unnecessary characters.
2015-02-16 01:11:02 +00:00
```c++
auto syntax = R"(
ROOT < - _ TOKEN (',' _ TOKEN )*
TOKEN < - < [ a-z0-9 ] + > _
_ < - [ \t\r\n]*
)";
peg pg(syntax);
2015-03-09 18:58:43 +00:00
pg["TOKEN"] = [](const char* s, size_t n) {
2015-02-18 03:35:07 +00:00
// 'token' doesn't include trailing whitespaces
2015-03-09 18:58:43 +00:00
auto token = string(s, n);
2015-02-16 01:11:02 +00:00
};
auto ret = pg.parse(" token1, token2 ");
```
2015-02-18 23:00:11 +00:00
We can ignore unnecessary semantic values from the list by using `~` operator.
```c++
peglib::peg parser(
" ROOT < - _ ITEM (',' _ ITEM _ )* "
" ITEM < - ( [ a-z ] ) + "
" ~_ < - [ \t]* "
);
2015-02-22 00:38:30 +00:00
parser["ROOT"] = [& ](const SemanticValues& sv ) {
assert(sv.size() == 2); // should be 2 instead of 5.
2015-02-18 23:00:11 +00:00
};
auto ret = parser.parse(" item1, item2 ");
```
2015-02-15 22:52:39 +00:00
Simple interface
----------------
*cpp-peglib* provides std::regex-like simple interface for trivial tasks.
2015-02-18 03:35:07 +00:00
`peglib::peg_match` tries to capture strings in the `$< ... >` operator and store them into `peglib::match` object.
2015-02-15 22:52:39 +00:00
```c++
peglib::match m;
auto ret = peglib::peg_match(
R"(
2015-02-18 03:35:07 +00:00
ROOT < - _ ( ' [ ' $ < TAG_NAME > ']' _)*
2015-02-15 22:52:39 +00:00
TAG_NAME < - ( ! ' ] ' . ) +
_ < - [ \t]*
)",
" [tag1] [tag:2] [tag-3] ",
m);
assert(ret == true);
assert(m.size() == 4);
assert(m.str(1) == "tag1");
assert(m.str(2) == "tag:2");
assert(m.str(3) == "tag-3");
```
There are some ways to *search* a peg pattern in a document.
```c++
using namespace peglib;
auto syntax = R"(
2015-02-18 03:35:07 +00:00
ROOT < - ' [ ' $ < [ a-z0-9 ] + > ']'
2015-02-15 22:52:39 +00:00
)";
auto s = " [tag1] [tag2] [tag3] ";
// peglib::peg_search
peg pg(syntax);
size_t pos = 0;
2015-03-09 18:58:43 +00:00
auto n = strlen(s);
2015-02-15 22:52:39 +00:00
match m;
2015-03-09 18:58:43 +00:00
while (peg_search(pg, s + pos, n - pos, m)) {
2015-02-15 22:52:39 +00:00
cout < < m.str ( ) < < endl ; / / entire match
cout < < m.str ( 1 ) << endl ; // submatch #1
pos += m.length();
}
// peglib::peg_token_iterator
peg_token_iterator it(syntax, s);
while (it != peg_token_iterator()) {
cout < < it- > str() < < endl ; / / entire match
cout < < it- > str(1) < < endl ; // submatch #1
++it;
}
// peglib::peg_token_range
for (auto& m: peg_token_range(syntax, s)) {
cout < < m.str ( ) < < endl ; / / entire match
cout < < m.str ( 1 ) << endl ; // submatch #1
}
```
2015-02-09 22:12:59 +00:00
Make a parser with parser operators
-----------------------------------
2015-02-08 01:52:26 +00:00
2015-02-09 22:12:59 +00:00
Instead of makeing a parser by parsing PEG syntax text, we can also construct a parser by hand with *parser operators* . Here is an example:
2015-02-08 01:52:26 +00:00
```c++
using namespace peglib;
using namespace std;
2015-02-09 22:12:59 +00:00
vector< string > tags;
2015-02-08 14:43:49 +00:00
Definition ROOT, TAG_NAME, _;
2015-02-14 15:38:15 +00:00
ROOT < = seq(_, zom(seq(chr('['), TAG_NAME, chr(']'), _)));
2015-03-09 18:58:43 +00:00
TAG_NAME < = oom(seq(npd(chr(']')), dot())), [& ](const char* s, size_t n ) {
tags.push_back(string(s, n));
2015-02-14 15:38:15 +00:00
};
_ < = zom(cls(" \t"));
2015-02-08 01:52:26 +00:00
auto ret = ROOT.parse(" [tag1] [tag:2] [tag-3] ");
```
The following are available operators:
2015-02-20 03:27:47 +00:00
| Operator | Description |
| :------- | :------------------ |
| seq | Sequence |
| cho | Prioritized Choice |
| zom | Zero or More |
| oom | One or More |
| opt | Optional |
| apd | And predicate |
| npd | Not predicate |
| lit | Literal string |
| cls | Character class |
| chr | Character |
| dot | Any character |
| anc | Anchor character |
| cap | Capture character |
| usr | User defiend parser |
2015-04-08 16:02:26 +00:00
Predicate control
-----------------
2015-06-12 22:57:58 +00:00
* TODO
2015-04-08 16:02:26 +00:00
2015-02-20 03:51:04 +00:00
Adjust definitions
------------------
2015-02-20 03:27:47 +00:00
2015-02-20 03:51:04 +00:00
It's possible to add and override definitions with parser operaters.
2015-02-20 03:27:47 +00:00
```c++
auto syntax = R"(
ROOT < - _ 'Hello' _ NAME '!' _
)";
Rules rules = {
{
2015-03-09 18:58:43 +00:00
"NAME", usr([](const char* s, size_t n, SemanticValues& sv, any& c) {
2015-02-20 03:27:47 +00:00
static vector< string > names = { "PEG", "BNF" };
for (const auto& n: names) {
2015-03-09 18:58:43 +00:00
if (n.size() < = n & & !n.compare(0, n.size(), s, n.size())) {
2015-02-20 03:27:47 +00:00
return success(n.size());
}
}
return fail(s);
})
},
{
"~_", zom(cls(" \t\r\n"))
}
};
peg g = peg(syntax, rules);
assert(g.parse(" Hello BNF! "));
```
2015-02-08 01:52:26 +00:00
2015-02-08 01:58:25 +00:00
Sample codes
------------
* [Calculator ](https://github.com/yhirose/cpp-peglib/blob/master/example/calc.cc )
2015-02-22 04:23:59 +00:00
* [Calculator (with parser operators) ](https://github.com/yhirose/cpp-peglib/blob/master/example/calc2.cc )
* [Calculator (AST version) ](https://github.com/yhirose/cpp-peglib/blob/master/example/calc3.cc )
2015-02-13 02:08:58 +00:00
* [PEG syntax Lint utility ](https://github.com/yhirose/cpp-peglib/blob/master/lint/peglint.cc )
2015-02-08 01:58:25 +00:00
2015-02-08 01:52:26 +00:00
Tested Compilers
----------------
* Visual Studio 2013
* Clang 3.5
TODO
----
* Optimization of grammars
* Unicode support
License
-------
MIT license (© 2015 Yuji Hirose)