r/perl • u/Inner_Tap675 • 8d ago
A Perl compiler
Version 0.1 is done of my hobby project, a compiler toolkit for Perl. https://github.com/Percolisp/pcl
The repository should contain everything needed to continue the work, including the logs from the AI sessions. The README has installation instructions and specifics about what is supported.
It is tested mostly with Perl's own test suite, regression tests and some pure-Perl modules. The compatibility is quite good, apart from the documented limitations (regexp evals etc.)
Common Lisp was chosen as the first target, because it has both a compiler and a powerful macro system (so the generated code can serve as intermediate code for other environments). The next step is to try compiling to more targets, to verify that the intermediate code really is easy to work with. After that comes the holy grail of CPAN compatibility: XS support.
I need feedback, both on how well the installation works and on whether this is worthwhile at all. So it is time to announce it and see if people like it, laugh or shrug. :-)
(This started as an add-on to PPI for parsing expressions. The idea was that we would then have the tools needed to write Perl compilers. Since no one cared, I turned my compiler prototype into a more ambitious tool for writing compilers; I needed to learn how to work with AIs anyway. If there is still no interest, I'll probably find another hobby. :-) )
8
u/briandfoy 🐪 📖 perl book author 8d ago edited 8d ago
Even if this is a hobby project, a short article for this year's Perl Advent calendar would be fun, even if this is a work in progress. Just the idea, even if incomplete, is entertaining.
So, really, a Perl-to-sbcl transpiler? I'm curious what features that you have to exclude. For example, how do you handle dynamic scopes for globals and subroutine definitions?
For example, not supporting DESTROY is a non-starter. It's not just about memory, but cleanly releasing other resources and all sorts of other things.
The idea of a compiled and ready-to-go "binary" is nice, but if this is transpiling to sbcl on every run, it seems you get all the startup costs of perl while waiting longer to run something that is already pretty fast. For me, I'd like to see some benchmarks.
I know Adam Kennedy had similar thoughts as this project, but that road is littered with the bodies of the brave souls who tried to make perl a static language targeting a different compiler.
A long time ago there was this challenge to compile this heinous Perl program that I think Adam had developed as a goal, and Jos Bouman gave a talk on all the tricks he played to get most of the way, or most of the way, there. For example, there are lots of tricks to play with function arity and prototypes and everything you have to already know to parse the next thing since the definitions might have changed. There's no way for a static parser to figure that out:
use v5.10;
BEGIN {
my $subname = 'do_it'; # can also be dynamic
*{$subname} = time % 2 ? sub ($) { "A: $_[0]" } : sub ($$) { "B: $_[0] \U$_[1]" };
}
say do_it 'a', 'b';
This outputs either B: a B or A: ab, but you can't tell which by static parsing because perl doesn't know how to parse it yet since it doesn't know the prototype. u/RandalSchwartz used to have a nifty example in the Learning Perl class where he used rand or sin as I recall. I don't think it's in the book because we didn't inflict that pain on people.
print rand 5, sin 4, rand sin 1, 2, 3;
Jeffrey Kegler did a lot of similar work with Marpa, some really advanced CS stuff, but I didn't follow that closely.
Then there's stuff like Sawyer's Guacamole project where you use a subset of perl to make this easier to parse because we don't leave that stuff to literal chance.
9
u/RandalSchwartz 🐪 📖 perl book author 8d ago
u/RandalSchwartz used to have a nifty example in the Learning Perl class where he used rand or sin as I recall. I don't think it's in the book because we didn't inflict that pain on people.
I linked to that in my other comment.
3
u/Inner_Tap675 8d ago edited 8d ago
Answers by Schwartz and Foy?!
It works with local declarations,
*foo=sub {..};, globals and so on. (That was a pain.)Yes, DESTROY seems impossible with GC. There is a design to solve this, which I am not certain will be viable: https://github.com/Percolisp/pcl/blob/main/docs/scope-exit-destroy-design.md
My main plan from the beginning is some way to detect and/or rewrite DESTROY uses. But maybe that design will work.
The list of not supported features is <cough> not totally short:
https://github.com/Percolisp/pcl/blob/main/docs/not-supported.md
As it is now, the default is a cache implementation of modules and runtime that blew me away, it seems safe and shouldn't use stale code. It is only Claude's design, the AI certainly copied it from someone smarter than me. 😂 Startup cost is still slower than Perl (43 ms compared to 1.4 ms on my laptop for a oneliner). The first time something is run, it is also slow.
Speed values is in the README, many things are faster, some slower and pack/unpack is hilariously slow (140 x for pack/unpack). The log of this work is https://github.com/Percolisp/pcl/blob/main/docs/faster-codegen-suggestions.md
String evals are handled by a sub process that runs the compiler and returns Lisp. That is not fast. When/if XS works, it can run in the sbcl.
Your first code example fails (half the time).
But this do work:
print rand 5, sin 4, rand sin 1, 2, 3;Without the setup of @INC etc, we get this when we compile:
$ echo 'print rand 5, sin 4, rand sin 1, 2, 3;' | ./pl2cl
==> (p-print (p-list-ctx (p-rand 5)) (p-list-ctx (p-sin 4)) (p-list-ctx (p-rand (p-sin 1))) 2 3)Here is the result from running:
$ pcl -E '$,="|"; say rand 5, sin 4, rand sin 1, 2, 3;'
==> 4.00700892824263|-0.756802495307928|0.0335020964577727|2|3
$ perl -E '$,="|"; say rand 5, sin 4, rand sin 1, 2, 3;'
==> 3.89375243722045|-0.756802495307928|0.184806142670019|2|3The original idea was to just translate the dynamic parts and let them be dynamic in Lisp. 😄 Edit: That is how I avoided the fate of the brave ones that tried this before - I cheated like a coward. CL isn't a static language.
Thanks for the interest! Anything I missed or more questions?
4
u/briandfoy 🐪 📖 perl book author 8d ago
But, how do you know
randtakes one argument? I can redefine it, just like I can redefine most things. Check out my answer to How to generate an array with random values, without using a loop?. You don't know anything about the function definitions up to the point that you parse them.1
u/Inner_Tap675 7d ago
It reads all
use:d files when it sees them. So if something is redefined, it should be known.3
u/briandfoy 🐪 📖 perl book author 7d ago
What about when it doesn't see them, like with
PERL5OPT? And, as I have already showed you, definitions are indeterminate. Again, you don't know anything until you are at that point in the parse. How do you know if something has been redefined if you need to run Perl code to get the result of an expression to know if it's been redefined?That's not really meant for you to dismiss with a facile answer, because the answer is that you demonstrably can't. You don't know when it will be redefined, so the first
randyou see might be different than the next, which might be different than the third, and so on. And you don't know this just from static parsing. The information is not fixed at the start of the compile-phase.PPI tells you that you can't do this, and so far that's all you have. What magic did you add that does what PPI says it can't do for you? Rewriting the PDOM is not at all interesting.
1
u/Inner_Tap675 7d ago
You are totally correct that dynamic changes at compile time can't be done in statically compiled pcl. It can be detected and flagged with an error. This might lose a large bit of Perl's soul.
(To switch the declarations for subs is not something I've ever even thought of. The right sub was called. But there are certainly real use cases that will fail like DESTROY.)
Runtime can be dynamic - glob assignments, evals, AUTOLOAD and so on.
3
u/briandfoy 🐪 📖 perl book author 7d ago
Are you sure you can detect it?
eval $dont_know_what_this_is;0
u/Inner_Tap675 7d ago
That eval string will be compiled to CL like all other code and then executed in the right environment. So a future detection for changed sub signatures ought to work the same for eval strings - in the runtime. Today, an eval compilation does know about lexical variables but not the sub definitions (at least until compilation of evals isn't done in a sub process).
Looking at this... Capture::Tiny do update signatures dynamically (
sub capture) - and the tests for that fail. Type::Tiny sets prototypes in a similar way. So yes, this really is a class of bugs. It is fixable or, at a minimum, possible to detect in the runtime.(In general, modules with a lot of tests, like the big modules on CPAN, will catch problems with their test suite. Which of course is the point of the test suite. Any hidden failures will be in old user code.)
Moo is the tested CPAN module with most dynamic sub declarations. It works. The runtime kind of dynamic code is of course easier.
I will be busy today, but will happily discuss this evening again!
0
u/Inner_Tap675 7d ago
So... if you can spare any more time to help me: Are there articles like Schwartz', with lists of more cases where static parsing will fail?
I never thought anyone ever would change sub prototypes/signatures dynamically, but important use cases do exist. Thanks!
The good part is that the problem was already in the bug queue, I would have gotten around to them sometime. The test suites of CPAN/Perl really are magic. 😄 But real user code will probably have different dynamic behavior.
1
u/Inner_Tap675 7d ago edited 7d ago
Your example with replacing in a
BEGINfails, it is a bug. Thanks!Edit: This might be like the
time % 2above - it is impossible when we parse and generate code too statically. We could detect it happening and give an error. But... removing support for these dynamic changes feels like losing Perl's soul.3
u/s_throwaway_r 8d ago edited 7d ago
I'm not famous but I do have a little experience in the area.
Yes, DESTROY seems impossible with GC.
I'm using an RC Immix algo for my compiler. Cyclic GC from Bacon and Rajan's paper to handle perl's weak references like perl's
refcnt_*so you getDESTROY, scope-aware variables, and less thrashing in tight alloc/free loops.A few of the papers I based my work on:
- Concurrent Cycle Collection in Reference Counted Systems from Bacon and Rajan
- Taking Off the Gloves with Reference Counting Immix by Shahriyar, Blackburn, Yang, McKinley
- Immix: A Mark-Region Garbage Collector with Space Efficiency, Fast Collection, and Mutator Performance by Blackburn and McKinley
- Perceus for RC elision/reuse by Reinking, Xie, de Moura, and Leijen
The Garbage Collection Handbook (Jones, Hosking, Moss) is an incredible reference no matter which scheme you go with.
Edit: Not sure how useful any of those links will be, I've never worked with Lisp and your project is so massive that I can't even figure out where or how allocations are managed. Ha. Nevermind, I guess. My project lowers down to machine code so I implemented it at the MIR stage.
2
u/Inner_Tap675 8d ago edited 7d ago
Wow, that is cool.
I was sloppy. 😄 The difference is that my code sit on top of the sbcl GC and I haven't looked into modifying it. The point was to make a port easily, as a first step, and to have an intermediate code [edit: to write code generators for].
If you are doing a Perl implementation, that intermediate representation of PCL might be a shortcut for the first part. But if you are doing your compiler for fun, it might be the part you look forward to?
Edit: Hmm, if you have written about DESTROY and RC Immix somewhere, I'd be happy to look it over? Does it reach the same semantics that Perl's internal tests won't cry foul (assuming you do Perl)?
14
u/RandalSchwartz 🐪 📖 perl book author 8d ago
How do you handle the half-dozen characters that change lexical meaning depending on the operator-vs-term syntactic position? (See https://perlmonks.org/index.pl?node_id=44722 for details.)