First version of new post

Compiling Parfait, better mom, builtin etc
2019-08-20 22:06:37 +03:00 · 2019-08-20 22:06:37 +03:00 · 90977d1a7c
commit 90977d1a7c
parent 6e8864b238
2 changed files with 204 additions and 0 deletions
--- a/app/assets/images/1600_tests.png
+++ b/app/assets/images/1600_tests.png
--- a/app/views/posts/_2019-08-20-parsing-parfait-and-other-goodies.haml
+++ b/app/views/posts/_2019-08-20-parsing-parfait-and-other-goodies.haml
@ -0,0 +1,204 @@
 %p
  As we rubyx will eventually need to parse and compile itself, i am very happy to
  report success on the first steps on that journey. Also benchmarks, better design
  and another conference are on the list.
 %h2 Compiling parfait
 %p
  As a recap, Parfait is that part of the core library that we need already during
  compilation. Ie the compiler creates Parfait objects during compilation and uses
  Parfait code to do this. This off course is a conundrum, which is solved by using the
  Parfait Code as is in the compiler (and some ruby module magic to avoid name clashes)
 %p
  Naturally any meaningful program that the compiler generates will use Parfait
  and so Parfait must be available at run-time, ie parsed and compiled. Since i have been
  busy doing the basics, this has been on the ToDo for a long while.
 %p
  Now, finally, most the basics are inplace and i have started what feels like a tremendous
  task. In fact i have succefully compiled
  %em three files.
  Object, DataObject and Integer, to be precise.
 %p
  The significance of this is actually much greater (especially since there are no tests)
  yet. Parfait, as part of rubyx, is what one may call ideomatic ruby, ie real world ruby.
  Off course i have to smoothen out a few bugs before compiling actually worked, but
  surprisingly little. In other words the compiler is functional enough to
  compile larger more feature ritch ruby programs, but more on that below.
 %h2 Design improvements
 %p
  The overall design has been like in the picture below for a while already.
  Alas, the implementation oof this architecture was slightly lacking.
  To be precise, when mom code was generated, it was immediately converted to risc.
  In other words the layer only existed conceptually, or in transit.
 %p.center.three_width
  = image_tag "architecture.png" ,  alt: "Architectural layers"
 %p
  Now the code works
  %em exactly
  as advertised. Ruby comes in from the top and binary code out at the bottom.
  But more than that, every layer is a distinct step, in fact there are methods on the
  topmost compiler object to create every level down from ruby. This is obviously
  very handy for testing.
 %h2 Automated binary tests
 %p
  Speaking of testing, we are at over 1600 tests, which is more than 200 up from before
  the design rewrite. At over 15000 assertions this is still 95%, in other words everything
  apart from a few fails. And with parallel execution still fast.
 %p.center.full_width
  = image_tag "1600_tests.png" ,  alt: "Lots of test, never boring"
 %p
  But the main achievement a couple of weeks ago was the integration of binary testing
  int the automated test flow. Specifically on
  %em Travis.
 %p
  This uses a feature of Qemu that i had not know before, namely that one can get qemu
  to run binaries from a different target on a machine, by simply calling it with
  qemu-arm.
 %p
  I had done previous testing of binaries via ssh, usually to an qemu emulated pi on my
  machine. This setup is vastly more complicated, as described
  =ext_link "here" , "/arm/qemu.html"
  and i had shied away from that. Meaning they would happen irregularily and all that.
  My only consolation was that the test would run on the interpreter, but off course that
  does not test the arm and elf genertion.
 %p
  The actual tests that i am talking about are a growing number of "mains" tests, found in
  the tets/mains directory.
  These are actal programs that calculate or output stuff. They are complete system
  tests in the sense that we only test their output (system output).
 %p
  As we usually link to "a.out" files (thus overwriting and avoiding cleanup), the actual
  invocation of qemu for a binary is really simple:
  %pre
    %code
      qemnu-arm ./a.out
  but that still leaves you to generate that binary. This can be done by using the
  rubyxc compiler and linking the resulting object file (see bug #13). But sine i too am a
  lazy programmer i have automated these steps into the rubyxc compiler, and so one
  can just compile/link/execute a source file like this:
  %pre
    %code
      :preserve
        ./bin/rubyxc execute test/mains/source/fibo__8.rb
  This will compile, link and execute this specific fibonachi test. This output
  of this test will be 8, as encoded in the file name.
  So this and 20 others will be tested as binaries now every time travid does its thing.
 %p
  BTW, i have also created arubyxc command to execute a file via the interpreter.
  This can sometimes yield better errors when things go wrong.
  %pre
    %code
      :preserve
        ./bin/rubyxc interpret test/mains/source/puts_Hello-there_11.rb
  And as a second btw, i also added an option to the compiler, so one can control the
  Parfait factory size with the option --parfait.
 %h2 Misc other news
 %h3 Microbenchmarks
 %p
  At the last conference in Hamburg, someone asked the fair question: So how fast is it?
  It's been so long that i did tests, that i could only mumble.
  Now i finished updating the tests, but it will be a while before i can answer the
  question more fully.
 %p
  So for starters, because the functionality of the compiler is limited, i did very small
  benchmarks. Very small means 20 lines or less, loops, string output, fibonacchi, both
  linear and recursive. I realized too late, that that will tell most about integer
  performance.
 %p
  Now because of the early days, i will not go into detail here. In general speed was not
  as fast as i had hoped from by 4 year old benchmarks, about the same as mri. I
  will have to do some work on the calling convention and probably some on integer
  handling too. I think i can quite easily shave 30-50% off, and that alone should
  verify the saying that all benchmarks are lies. Like the one where rubyx is doing
  "hello world" faster than C. Yes, C, not mri. But only because i switched the buffering
  off, because also rubyx does not buffer (apples and oranges ...)
 %p
  So i will take this round as inspiration to do some optimisation and performance
  measuring. And come back to it later.
 %h3 Implicit returns
 %p
  As part of parsing Parfait, i implemented a first version of implicit returns.
  Low hanging fruits, and in fact most common use cases, included constants and calls.
  So when a method ends in a simple variable, constant, or a call, a return will be added.
  More complex rules like returns for if's or while will ave to wait, but i found that i
  personally don't tend to use them anyway.
 %p
  Since class methods are basically methods (of the meta class), adding the unified
  return handling to them was easy too.
 %h3 Improved Block handling
 %p
  Block handling, at least the simple implicit kind, has worked for a while, but was in
  several ways too complicated. The block was unneccessarily assigned to a local, and
  compiling was handled by picking them out.
 %p
  This all stemmed from a misunderstanding, or lack of understanding: Blocks, or should
  i say Lambdas, are constants. Just like a string or integer. They are created once at
  compile time and can not change identity. In fact Methods and Classes are also contants,
  and i reflected this in the Vool level by calling them Expressions, instead of
  before Statements.
 %p
  So now the Lambda Expression is created and just added as an argument to the send.
  Compiling thee Lambda is triggered by the constant creation, ie the step down from
  vool to mom, and the block compiler added to method compiler automatically.
 %h3 Vool coming into focus
 %p
  I've been saying ruby without the fluff, to descibe vool. And while that is true,
  it is quite vague. Two major things have become clear about vool through the work above.
 %p
  Firstly, Vool has no complex or recursive send statements. Arguments must be variables or
  constants. Calls are executed before and assigned to a temporary variable. In effect
  recursive calls are flattened into a list, and as such the calling does not rely on a
  stack as in ruby.
 %p
  Secondly, Vool distinguishes between expressions and statements. Like other lower level,
  but not ruby. As a rule of thumb, Statements do things, Expression are things. In other
  words, only expressions have value, statements (lke if or while) do not.
 %h2 Plans
 %h4 GrillRB conference
 %p
  I will speak in
  =ext_link "Wrazlaw" , "https://grillrb.com/"
  in about a week. The plan is to make a comparison with rails and focus on the
  possibilities, rather than technical detail.
 %h4 Calling
 %p
  The Calling can do with work and i noticed two mistakes i did. One is that creating
  a new message for every call is unneccessarily complicated. Its is only in the
  special case that a Proc is created that the return sequence (a mom instruciton) needs
  to keep the message alive.
 %h4 Integers
 %p
  I still want to hang on to Integers being objects, though creation is clealy costly.
  In the future a full escape analysis will help off course, but for now it should be easy
  enough to figure out wether an int is passed down. If not loops can be
  destructively change the int. A simple special case is a the times method.
 %h4 Mom instruction invocation
 %p
  I have this idea of being able to code more stuff higher up. To make that more
  efficient i am thinking of macros or instruction invocation at the vool level.
  Only inside Parfait off course. The basic idea would be to save the call/return
  code, and have eg X.return_jump map to the Mom::ReturnJump Instruction. "Just" have
  to figure out the passing semantics, or how that integrates intot the vools code.
 %h4 Better Builtin
 %p
  The generation of the current builtin methods has always bothered me a bit.
  It is true that some things just can not be expressed as ruby and so some
  alternative mechanism is needed (even in c one can embed assembler).
 %p
  The main problem i have is that those methods don't check their arguments and as such
  may cause core dumps. So they are to high level and hopefully all we really need is
  that previous idea of being able to integrate Mom code into vool. As Mom is extensible
  that should take care of any possible need. And we could code the methods normally as
  part of Parfait, make them safe, and just use the lower level inside them. Lets see!.