move compiler into virtual

and associated changes in code, requires tests etc

lib/virtual/compiler.rb

@@ -0,0 +1,44 @@
+module Virtual
+  module Compiler
+
+    # Compiling is the conversion of the AST into 2 things:
+    # - code (ie sequences of Instructions inside Blocks) carried by CompiledMethods
+    # - an object graph containing all the Methods, their classes and Constants
+    #
+    # Some compile methods just add code, some may add structure (ie Blocks) while
+    # others instantiate Class and Method objects
+    #
+    # Everything in ruby is an expression, ie returns a value. So the effect of every compile
+    # is that a value is put into the ReturnSlot of the current Message.
+    # The compile method (so every compile method) returns the value that it deposits which
+    # may be unknown Mystery value.
+    #
+    # The Compiler.compile uses a visitor patter to dispatch according to the class name of
+    # the expression. So a NameExpression is delegated to compile_name etc.
+    # This makes the dispatch extensible, ie Expressions may be added by external code,
+    # as long as matching compile methods are supplied too.
+    #
+    def self.compile expression , method
+      exp_name = expression.class.name.split("::").last.sub("Expression","").downcase
+      #puts "Expression #{exp_name}"
+      begin
+        self.send "compile_#{exp_name}".to_sym , expression, method
+      rescue NoMethodError => e
+        puts "No compile method found for " + exp_name
+        raise e
+      end
+    end
+
+  end
+end
+
+require_relative "compiler/basic_expressions"
+require_relative "compiler/callsite_expression"
+require_relative "compiler/compound_expressions"
+require_relative "compiler/if_expression"
+require_relative "compiler/function_expression"
+require_relative "compiler/module_expression"
+require_relative "compiler/operator_expressions"
+require_relative "compiler/return_expression"
+require_relative "compiler/while_expression"
+require_relative "compiler/expression_list"

lib/virtual/compiler/README.md

@@ -0,0 +1,84 @@
+### Compiling
+
+The Ast (abstract syntax tree) is created by salama-reader gem and the classes defined there
+
+The code in this directory compiles the AST to the virtual machine code.
+
+If this were an interpreter, we would just walk the tree and do what it says.
+Since it's not things are a little more difficult, especially in time.
+
+When compiling we deal with two times, compile-time and run-time.
+All the headache comes from mixing those two up.*
+
+Similarly, the result of compiling is two-fold: a static and a dynamic part.
+
+- the static part are objects like the constants, but also defined classes and their methods
+- the dynamic part is the code, which is stored as streams of instructions in the CompiledMethod
+
+Too make things a little simpler, we create a very high level instruction stream at first and then
+run transformation and optimization passes on the stream to improve it.
+
+Each ast class gets a compile method that does the compilation.
+
+#### Compiled Method and Instructions
+
+The first argument to the compile method is the CompiledMethod.
+All code is encoded as a stream of Instructions in the CompiledMethod.
+Instructions are stored as a list of Blocks, and Blocks are the smallest unit of code,
+which is always linear.
+
+Code is added to the method (using add_code), rather than working with the actual instructions.
+This is so each compiling method can just do it's bit and be unaware of the larger structure
+that is being created.
+The genearal structure of the instructions is a graph
+(with if's and whiles and breaks and what), but we build it to have one start and *one* end (return).
+
+
+#### Messages and frames
+
+Since the machine is virtual, we have to define it, and since it is oo we define it in objects.
+
+Also it is important to define how instructions operate, which is is in a physical machine would
+be by changing the contents of registers or  some stack.
+
+Our machine is not a register machine, but an object machine: it operates directly on objects and
+also has no separate stack, only objects. There are a number of objects which are accessible,
+and one can think of these (their addresses) as register contents.
+(And one wouldn't be far off as that is the implementation.)
+
+The objects the machine works on are:
+
+- Message
+- Frame
+- Self
+- NewMessage
+
+and working on means, these are the only objects which the machine accesses.
+Ie all others would have to be moved first.
+
+When a Method needs to make a call, or send a Message, it creates a NewMessage object.
+Messages contain return addresses and arguments.
+
+Then the machine must find the method to call.
+This is a function of the virtual machine and is implemented in ruby.
+
+Then a new Method receives the Message, creates a Frame for local and temporary variables
+and continues execution.
+
+The important thing here is that Messages and Frames are normal objects.
+
+And interestingly we can partly use ruby to find the method, so in a way it is not just a top
+down transformation. Instead the sending goes back up and then down again.
+
+The Message object is the second parameter to the compile method, the run-time part as it were.
+Why? Since it only exists at runtime: to make compile time analysis possible
+(it is after all the Virtual version, not Parfait. ie compile-time, not run-time).
+Especially for those times when we can resolve the method at compile time.
+
+
+*
+As ruby is a dynamic language, it also compiles at run-time. This line of thought does not help
+though as it sort of mixes the seperate times up, even they are not.
+Even in a running ruby programm the stages of compile and run are seperate.
+Similarly it does not help to argue that the code is static too, not dynamic,
+as that leaves us with a worse working model.

lib/virtual/compiler/basic_expressions.rb

@@ -0,0 +1,101 @@
+module Virtual
+  # collection of the simple ones, int and strings and such
+
+  module Compiler
+
+    # Constant expressions can by definition be evaluated at compile time.
+    # But that does not solve their storage, ie they need to be accessible at runtime from _somewhere_
+    # So we view ConstantExpressions like functions that return the value of the constant.
+    # In other words, their storage is the return slot as it would be for a method
+
+    # The current approach moves the constant into a variable before using it
+    # But in the future (in the one that holds great things) we optimize those unneccesay moves away
+
+  #    attr_reader :value
+      def self.compile_integer expression , method
+        int = Virtual::IntegerConstant.new(expression.value)
+        to = Virtual::Return.new(Virtual::Integer , int)
+        method.add_code Virtual::Set.new( to , int)
+        to
+      end
+
+      def self.compile_true expression , method
+        value = Virtual::TrueConstant.new
+        to = Virtual::Return.new(Virtual::Reference , value)
+        method.add_code Virtual::Set.new( to , value )
+        to
+      end
+
+      def self.compile_false expression , method
+        value = Virtual::FalseConstant.new
+        to = Virtual::Return.new(Virtual::Reference , value)
+        method.add_code Virtual::Set.new( to , value )
+        to
+      end
+
+      def self.compile_nil expression , method
+        value = Virtual::NilConstant.new
+        to = Virtual::Return.new(Virtual::Reference , value)
+        method.add_code Virtual::Set.new( to , value )
+        to
+      end
+
+      #    attr_reader  :name
+      # compiling name needs to check if it's a variable and if so resolve it
+      # otherwise it's a method without args and a send is issued.
+      # whichever way this goes the result is stored in the return slot (as all compiles)
+      def self.compile_name expression , method
+        return Virtual::Self.new( Virtual::Mystery ) if expression.name == :self
+        name = expression.name
+        if method.has_var(expression.name)
+          # either an argument, so it's stored in message
+          if( index = method.has_arg(name))
+            method.add_code MessageGet.new(name , index)
+          else # or a local so it is in the frame
+            method.add_code FrameGet.new(name , index)
+          end
+        else
+          call = Ast::CallSiteExpression.new(expression.name , [] ) #receiver self is implicit
+          Compiler.compile(call, method)
+        end
+      end
+
+
+      def self.compile_module expression , method
+        clazz = Virtual::BootSpace.space.get_or_create_class name
+        raise "uups #{clazz}.#{name}" unless clazz
+        to = Virtual::Return.new(Virtual::Reference , clazz )
+        method.add_code Virtual::Set.new( to , clazz )
+        to
+      end
+
+  #    attr_reader  :string
+      def self.compile_string expression , method
+        value = Virtual::StringConstant.new(expression.string)
+        to = Virtual::Return.new(Virtual::Reference , value)
+        Virtual::BootSpace.space.add_object value
+        method.add_code Virtual::Set.new( to , value )
+        to
+      end
+
+      #attr_reader  :left, :right
+      def self.compile_assignment expression , method
+        raise "must assign to NameExpression , not #{expression.left}" unless expression.left.instance_of? Ast::NameExpression
+        r = Compiler.compile(expression.right , method )
+        raise "oh noo, nil from where #{expression.right.inspect}" unless r
+        index = method.has_arg(name)
+        if index
+          method.add_code Virtual::Set.new(Virtual::Return.new , Virtual::MessageSlot.new(index , r,type , r ))
+        else
+          index = method.ensure_local(expression.left.name)
+          method.add_code Virtual::Set.new(Virtual::Return.new , Virtual::FrameSlot.new(index , r.type , r ))
+        end
+        r
+      end
+
+      def self.compile_variable expression, method
+        method.add_code Virtual::InstanceGet.new(expression.name)
+        Virtual::Return.new( Virtual::Mystery )
+      end
+  end
+end

lib/virtual/compiler/callsite_expression.rb

@@ -0,0 +1,28 @@
+module Virtual
+  module Compiler
+  # operators are really function calls
+
+#    call_site - attr_reader  :name, :args , :receiver
+
+    def self.compile_callsite expession , method
+      me = Compiler.compile( expession.receiver , method )
+      method.add_code Virtual::NewMessage.new
+      method.add_code Virtual::Set.new(Virtual::NewSelf.new(me.type), me)
+      method.add_code Virtual::Set.new(Virtual::NewName.new(), Virtual::StringConstant.new(expession.name))
+      compiled_args = []
+      expession.args.each_with_index do |arg , i|
+        #compile in the running method, ie before passing control
+        val = Compiler.compile( arg , method)
+        # move the compiled value to it's slot in the new message
+        to = Virtual::NewMessageSlot.new(i ,val.type , val)
+        # (doing this immediately, not after the loop, so if it's a return it won't get overwritten)
+        method.add_code Virtual::Set.new(to , val )
+        compiled_args << to
+      end
+      method.add_code Virtual::MessageSend.new(expession.name , me , compiled_args) #and pass control
+      # the effect of the method is that the NewMessage Return slot will be filled, return it
+      # (this is what is moved _inside_ above loop for such expressions that are calls (or constants))
+      Virtual::Return.new( method.return_type )
+    end
+  end
+end

lib/virtual/compiler/compound_expressions.rb

@@ -0,0 +1,16 @@
+module Virtual
+  module Compiler
+
+#    attr_reader :values
+    def self.compile_array expession, context
+      to.do
+    end
+#    attr_reader :key , :value
+    def self.compile_association context
+      to.do
+    end
+    def self.compile_hash context
+      to.do
+    end
+  end
+end

lib/virtual/compiler/expression_list.rb

@@ -0,0 +1,10 @@
+module Virtual
+  module Compiler
+#    list - attr_reader  :expressions
+    def self.compile_list expession , method
+      expession.expressions.collect do |part|
+        Compiler.compile(  part , method )
+      end
+    end
+  end
+end

lib/virtual/compiler/function_expression.rb

@@ -0,0 +1,71 @@
+module Virtual
+  module Compiler
+#    function attr_reader  :name, :params, :body , :receiver
+    def self.compile_function  expression, method
+      args = expression.params.collect do |p|
+        raise "error, argument must be a identifier, not #{p}" unless p.is_a? Ast::NameExpression
+        p.name
+      end
+      r = expression.receiver ? Compiler.compile(expression.receiver, method ) : Virtual::Self.new()
+      new_method = Virtual::CompiledMethod.new(expression.name , args , r )
+      new_method.class_name = r.is_a?(Virtual::BootClass) ? r.name : method.class_name
+      clazz = Virtual::BootSpace.space.get_or_create_class(new_method.class_name)
+      clazz.add_instance_method new_method
+
+      #frame = frame.new_frame
+      return_type = nil
+      expression.body.each do |ex|
+        return_type = Compiler.compile(ex,new_method  )
+        raise return_type.inspect if return_type.is_a? Virtual::Instruction
+      end
+      new_method.return_type = return_type
+      new_method
+    end
+    def scratch
+      args = []
+      locals = {}
+      expression.params.each_with_index do |param , index|
+        arg = param.name
+        register = Virtual::RegisterReference.new(Virtual::RegisterMachine.instance.receiver_register).next_reg_use(index + 1)
+        arg_value = Virtual::Integer.new(register)
+        locals[arg] = arg_value
+        args << arg_value
+      end
+      # class depends on receiver
+      me = Virtual::Integer.new( Virtual::RegisterMachine.instance.receiver_register )
+      if expression.receiver.nil?
+        clazz = context.current_class
+      else
+        c = context.object_space.get_or_create_class expression.receiver.name.to_sym
+        clazz = c.meta_class
+      end
+
+      function = Virtual::Function.new(name , me , args )
+      clazz.add_code_function function
+
+      parent_locals = context.locals
+      parent_function = context.function
+      context.locals = locals
+      context.function = function
+
+      last_compiled = nil
+      expression.body.each do |b|
+        puts "compiling in function #{b}"
+        last_compiled = b.compile(context)
+        raise "alarm #{last_compiled} \n #{b}" unless last_compiled.is_a? Virtual::Word
+      end
+
+      return_reg = Virtual::Integer.new(Virtual::RegisterMachine.instance.return_register)
+      if last_compiled.is_a?(Virtual::IntegerConstant) or last_compiled.is_a?(Virtual::ObjectConstant)
+        return_reg.load function , last_compiled if last_compiled.register_symbol != return_reg.register_symbol
+      else
+        return_reg.move( function, last_compiled ) if last_compiled.register_symbol != return_reg.register_symbol
+      end
+      function.set_return return_reg
+
+      context.locals = parent_locals
+      context.function = parent_function
+      function
+    end
+  end
+end

lib/virtual/compiler/if_expression.rb

@@ -0,0 +1,42 @@
+module Virtual
+  module Compiler
+#    if - attr_reader  :cond, :if_true, :if_false
+
+    def self.compile_if expression , method
+      # to execute the logic as the if states it, the blocks are the other way around
+      # so we can the jump over the else if true ,and the else joins unconditionally after the true_block
+      merge_block = method.new_block "if_merge"    # last one, created first
+      true_block =  method.new_block "if_true"     # second, linked in after current, before merge
+      false_block = method.new_block "if_false"    # directly next in order, ie if we don't jump we land here
+
+
+      is = Compiler.compile(expression.cond, method )
+      # TODO should/will use different branches for different conditions.
+      # just a scetch : cond_val = cond_val.is_true?(method) unless cond_val.is_a? Virtual::BranchCondition
+      method.add_code Virtual::IsTrueBranch.new( true_block )
+
+      # compile the true block (as we think of it first, even it is second in sequential order)
+      method.current true_block
+      last = is
+      expression.if_true.each do |part|
+        last = Compiler.compile(part,method  )
+        raise part.inspect if last.nil?
+      end
+
+      # compile the false block
+      method.current false_block
+      expression.if_false.each do |part|
+        puts "compiling in if false #{part}"
+        last = Compiler.compile(part,method )
+        raise part.inspect if last.nil?
+      end
+      method.add_code Virtual::UnconditionalBranch.new( merge_block )
+
+      puts "compiled if: end"
+      method.current merge_block
+
+      #TODO should return the union of the true and false types
+      last
+    end
+  end
+end

lib/virtual/compiler/module_expression.rb

@@ -0,0 +1,24 @@
+module Virtual
+  module Compiler
+#    module attr_reader  :name ,:expressions
+    def self.compile_module expression , context
+      return clazz
+    end
+
+    def self.compile_class expression , method
+      clazz = ::Virtual::BootSpace.space.get_or_create_class expression.name
+      puts "Created class #{clazz.name.inspect}"
+      expression.expressions.each do |expr|
+        # check if it's a function definition and add
+        # if not, execute it, but that does means we should be in salama (executable), not ruby. ie throw an error for now
+        raise "only functions for now #{expr.inspect}" unless expr.is_a? Ast::FunctionExpression
+        #puts "compiling expression #{expression}"
+        expression_value = Compiler.compile(expr,method  )
+        clazz.add_instance_method(expression_value)
+        #puts "compiled expression #{expression_value.inspect}"
+      end
+
+      return clazz
+    end
+  end
+end

lib/virtual/compiler/operator_expressions.rb

@@ -0,0 +1,9 @@
+module Virtual
+  module Compiler
+#    operator attr_reader  :operator, :left, :right
+    def self.compile_operator expression, method
+      call = Ast::CallSiteExpression.new(expression.operator , [expression.right] , expression.left )
+      Compiler.compile(call, method)
+    end
+  end
+end

lib/virtual/compiler/return_expression.rb

@@ -0,0 +1,24 @@
+module Virtual
+  module Compiler
+
+#    return attr_reader  :expression
+    def self.compile_return expression, scope ,method
+      Virtual::Reference.new
+    end
+    def old
+      into = context.function
+      puts "compiling return expression #{expression}, now return in return_regsiter"
+      expression_value = expression.compile(context)
+      # copied from function expression: TODO make function
+
+      return_reg = Virtual::Integer.new(Virtual::RegisterMachine.instance.return_register)
+      if expression_value.is_a?(Virtual::IntegerConstant) or expression_value.is_a?(Virtual::ObjectConstant)
+        return_reg.load into , expression_value
+      else
+        return_reg.move( into, expression_value ) if expression_value.register_symbol != return_reg.register_symbol
+      end
+      #function.set_return return_reg
+      return return_reg
+    end
+  end
+end

lib/virtual/compiler/while_expression.rb

@@ -0,0 +1,27 @@
+module Virtual
+  module Compiler
+
+#    while- attr_reader  :condition, :body
+    def self.compile_while expression, method
+      start = Virtual::Label.new("while_start")
+      method.add_code start
+      is = expression.condition.compile(method )
+      branch = Virtual::IsTrueBranch.new "while"
+      merge = Virtual::Label.new(branch.name)
+      branch.other = merge   #false jumps to end of while
+      method.add_code branch
+      last = is
+      expression.body.each do |part|
+        last = part.compile(method  )
+        raise part.inspect if last.nil?
+      end
+      # unconditionally brnach to the start
+      merge.next = method.current.next
+      method.current.next = start
+      # here we add the end of while that the branch jumps to
+      #but don't link it in (not using add)
+      method.current =  merge
+      last
+    end
+  end
+end

lib/virtual/machine.rb

@@ -35,7 +35,7 @@ module Virtual
     def initialize
       @parser  = Parser::Salama.new
       the_end = Halt.new
-      @message = Message.new(the_end , the_end , :Object)
+#      @message = Message.new(the_end , the_end , :Object)
     end
     attr_reader :message
 
@@ -62,24 +62,6 @@ module Virtual
       main = Virtual::CompiledMethod.main
       Compiler.compile( parts , main )
     end
-
   end
+
 end
-
-require_relative "instruction"
-require_relative "compiled_method"
-require_relative "frame"
-require_relative "message"
-require_relative "slots/slot"
-require_relative "type"
-require_relative "object"
-require_relative "constants"
-require_relative "boot_space"
-# the passes _are_ order dependant
-require_relative "passes/send_implementation"
-require_relative "passes/get_implementation"
-require_relative "passes/enter_implementation"
-require_relative "passes/frame_implementation"
-
-Sof::Volotile.add(Virtual::Block ,  [:method])
-Sof::Volotile.add(Virtual::CompiledMethod ,  [:current])