Metaprogramming Ruby: Core Concepts - Eval Family I

In ruby, block is the cornerstone of metaprogramming with a surprisingly powerful scope controlling capability. Based on this definition, much brilliant interfaces are introduced, and help to implement many valuable features, like callable objects, eval and its huge family.

1. blocks

Block represents a snippet of code, it permits to be called immediately or lately depends on different cases. But compared with some similar concepts in functional programming languages, like Lisp, block still has more limits, and more readability respectively.

In commonly, block can be used like this:

1
2
3
4
5
6
7
8
9
10
11
def show_hello
  yield 'hello' if block_given?
end

show_hello { |say| puts say }

show_hello do |say|
  file = File.open('/tmp/output.txt', 'w')
  file.write say
  file.close
end

‘yield’ is a ruby keyword used to call blocks sent to the current method. block_given? is an instance method from Kernel to probe whether there is a block for current method.

One of the most useful aspect of block is closure, it captures bindings where it’s defined, and avoid impact of scope changing by connecting to those bindings, like flat scopes, and shared scopes.

Frankly speaking, block is actually not responsible for running code, but only representation (Except yield, which indeed means running code immediately). There are more powerful tools to help enhance block usage.

2. callable objects

Block is just like a package of code, and you need to use yield to execute it if you like. However, there are more ways to package code in ruby, including Proc and lambda.

Proc

We already know that block is not an object in ruby which is really quite a few, but Proc is basically a block turned object, can be seen as a consistent form of block, and you do not have to use yield to run it immediately, it will be running later as you want (Deferred Evaluation). you can define a Proc like this:

1
2
3
4
inc = Proc.new {|x| x + 1}
inc.call(1) #2
dec = proc {|x| x - 1}
inc.call(2) #1

lambda

Except for Proc, lambda can also be used for transferring blocks, but with simpler and a little different way:

1
2
3
4
inc = lambda {|x| x + 1}
inc.call(1) #2
dec = ->(x) {x - 1}
dec.call(2) #1

& operator

Once you have defined a block for a method, there is a way to convert it to a Proc inner method by using &. For example:

1
2
3
4
5
6
7
8
9
10
11
12
def bar
  yield 'ruby'
end

def foo(&operation)
  bar(&operation)
  operation.call('world')
end

foo {|x| puts "hello #{x}"}
# hello ruby
# hello world

& can be seen like transferring a block into Proc, but you need to remove & if you want to use it as a Proc.

Comparison between procs and lambdas

You may notice that lambda is also a Proc, but you can still use Proc#lambda? to see the actual type of the target. Besides there are two important differences: keyword ‘return’ and arguments checking.

a. return

As plain blocks, the program will return out of scope where block defined if return statement is set inner block, so does Proc (which may mean that the return may cause exception when the block is called in nonreturnable scope, like top-level). While for lambda, the return only runs out of block, not even farther.

b. arity

Arity means the number of arguments. In real case, lambda has less tolerance than Proc, which means that lambda requires correct number of arity, which is no need for procs.

Method objects

All methods are objects of Method. You can use Kernel#method or Kernel#singleton_method to get the object of method, then use Method#call to run the method. Also you may use Method#to_proc and define_method to convert method and proc to each other. Eventhough the method still has scope of it’s defined.

Unbound Methods

Method object can also be unbound from original scopes by using Method#unbind, or Module#instance_method. Generated UnboundMethod object can not be called directly, but you can still call it after bind it to another scope by using UnboundMethod#bind. The only notice is that UnboundMethod object can only be bound to same class or sub if it’s from class, and no such limitation when it’s from module.

Comments

Metaprogramming Ruby: Core Concepts - Open Classes & Refinements

Opening Classes

Opening classes brings much flexibility for ruby. It permits you to modify existed classes or modules permanently, without having to implement whole world in case you just need to give a simple patch for current tool. For example:

1
2
3
4
5
class String
  def to_alphanumeric
    gsub(/[^\w\s]/, '')
  end
end

to_alphanumeric is new interface to return only alpha and numeric part of a string. However, opening class may not work well, especially it’s commonly dangerous to opening a frequently used target which would cause unexpected error in code. Due to this reason someone calls opening classes monkey patch and just leaves far from this feature.

While whatever, monkey patch continues conciseness in ruby, sometimes you are able to save your life gracefully with such powerful tool, except the trap behind it.

Refinements

From ruby 2.0, there is a more advanced form of monkey patch which is called refinements. Refinements are providing a way to extend classes only under specific scope, but not including modules. For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
class C
  def foo
    puts "C#foo"
  end
end

module M
  refine C do
    def foo
      puts "C#foo in M"
    end
  end
end

Module#refine creates an anonymous module that contains the changes or refinements to the class (C in the example). self in the refine block is this anonymous module similar to Module#module_eval.

Scope

We use using to activate refinements defined in any places, but you may only activate refinements at top-level, not inside any class, module or method scope. You may activate refinements in a string passed to Kernel#eval that is evaluated at top-level. Refinements are active until the end of the file or the end of the eval string, respectively.

Refinements are lexical in scope. When control is transferred outside the scope the refinement is deactivated. This means that when you require or load a file or call a method that is defined outside the current scope the refinement will be deactivated.

If a method is defined in a scope where a refinement is active the refinement will be active when the method is called.

When defining multiple refinements in the same module, inside a refine block all refinements from the same module are active when a refined method is called.

You may also activate refinements in a class or module definition, in which case the refinements are activated from the point where using is called to the end of the class or module definition.

Method lookup under refinement

When looking up a method for an instance of class C Ruby checks:

1. If refinements are active for C, in the reverse order they were activated:

The prepended modules from the refinement for C

The refinement for C

The included modules from the refinement for C

2. The prepended modules of C
3. C
4. The included modules of C

If no method was found at any point this repeats with the superclass of C.

Note that methods in a subclass have priority over refinements in a superclass. For example, if the method / is defined in a refinement for Integer 1 / 2 invokes the original Fixnum#/ because Fixnum is a subclass of Integer and is searched before the refinements for the superclass Integer.

If a method foo is defined on Integer in a refinement, 1.foo invokes that method since foo does not exist on Fixnum.

Super

When super is invoked method lookup checks:

The included modules of the current class. Note that the current class may be a refinement.
If the current class is a refinement, the method lookup proceeds as in the Method Lookup section above.
If the current class has a direct superclass, the method proceeds as in the Method Lookup section above using the superclass.

Note that super in a method of a refinement invokes the method in the refined class even if there is another refinement which has been activated in the same context.

Refinements and module inclusion

Refinements are inherited by module inclusion. That is, using activates all refinements in the ancestors of the specified module. Refinements in a descendant have priority over refinements in an ancestor.

After all, refinement is still an experiment feature in ruby 2.x series, there is a detailed specification in official website, which should cover most aspect of it.

Comments

Metaprogramming Ruby: Core Concepts - Scope & Self

Scope defines the circumstance around any ruby statement, which basically limits your accessibility to resources. But ruby also has implicit and complicate Constants/Methods lookup strategy to make things not so easy at all. This article will clarify the concepts about scope and self firstly, then deep into the Constants/Methods lookup strategy. Finally, let’s try to understand top-level which may be the most special case in this topic.

1. Scope

In most cases, the ‘Scope’ we say in ruby commonly refers to variable scope, which has a little bit different from other popular languages. For example, there are four basic variable scopes including local, global, instance and class level, none of them has business with each other. We’ve already know the naming specification below:

Name begins with Variable scope
[a-z] or _ local variable
$ global variable
@ instance variable
@@ class variable

Actually the tricky is not naming, it’s about default scope changing. As we know that ruby has lexical scope, but not like others, it does not support nested scope, like inner scope shares outer scope in some languages. Each of four variable scopes in above figure is purely independent to each other.

message & scope resolution operator

Dot . is message operator in ruby, mainly used for message receiving, like method call. Double colon :: is scope resolution operator, commonly used for getting constant in specific scope, Like Foo::Bar, means searching ./foo/bar with relative path. If Foo is undefined in current scope, you should always use ::Foo::Bar to figure it out using absolute path.

scope changing

There are four basic cases which may create and change the scope lexically: Class definitions (class), Module definitions (module), Methods (def) and Blocks. Block scope is a little bit different from others and more interesting, which would create its own scope after the block defined, but also have closure capability, like code below:

1
2
3
4
5
foo = ''
bar = lambda { foo = 'hi' if foo == 'hello' }
foo = 'hello'
bar.call
foo #hi

For sure we can break this very easily:

1
2
3
4
5
foo = ''
bar = lambda { |;foo| foo = 'hi' if foo == 'hello' }
foo = 'hello'
bar.call
foo #hello

Notice that |;foo| defines foo as a local variable in its block, this kind of feature was introduced from ruby 1.9.

constants scope

The scope we’ve talked until now basically only means variable scope. In ruby, constant is totally different type of data compared with other languages like C++ or Java, any constant name must be started with uppercase character. And none of them can be assigned dynamically. Like:

1
2
3
def foo
  Bar = 'hello'
end #should give SyntaxError for dynamic constant assignment

Constants defined within a class or module can be accessed within its parent. And scope operator we talked before should be used when it’s outside the class or module. Constants defined outside any class or module are global scope, can be accessed directly or by using the scope operator ‘::’ with no prefix.

2. Self

As we know that the grail in ruby world would be conciseness. However, such conciseness is built on many bright solutions like keyword self.

if you just type a name in ruby console, it will show you an error:

1
foo #NameError: undefined local variable or method `foo' for main:Object

Indeed it has a huge gap between ruby and other languages in such case. But the expression with only name as a statement in ruby really means asking for local variable first, if found nothing then go on for searching method defined by current object. Here ‘main’ is a special object which represents the top-level, will have more discussion in the last part of this article.

calling methods

When you want to call a method, there should always be a receiver pointed to target object. Commonly that would be in this form:

1
some_object.some_method

If some_object is omitted, some_method will be called on the object where the current scope belongs to. self is one of the keywords referring to the current object. Note that private is an exception in this case, we will show an example here which has already been introduced in last several articles talking about private and public.

1
2
3
4
5
6
def foo
  'Hello'
end

foo #Hello
self.foo #NoMethodError: private method `foo' called for main:Object

The most usage of self would be defining a class method, or getting singleton class of current class.

3. Method lookup

The order of name lookup should be local variable/constant, then method. We has discussed about variable and constant scope in the first part. Here let’s have a look at method lookup.

When a message call happens, ruby will found its receiver firstly which should be an object. Since all methods are stored in classes and modules so method lookup walks these, not the objects themselves.

Here is the order of method lookup for the receiver’s class or module R:

i. The prepended modules of R in reverse order.

ii. For a matching method in R.

iii. The included modules of R in reverse order.

If R is a class with a superclass, this is repeated with R‘s superclass until a method is found. And once a match is found method lookup stops. If no match is found this repeats from the beginning, but looking for method_missing. The default method_missing is BasicObject#method_missing which raises a NameError when invoked.

4. Understanding top-level

May be you’re not interested in top-level mechanism as a rails developer (You have to use top-level one day even you’re only develop rails application), but you need to notice that ruby is also regarded as a powerful scripting language and already as a built-in tool in many popular OS distributions. We talk about top-level because it behaves different from any ruby official documents.

I’ve found a nice explanation about top-level here, but there are some newer update in latest ruby, we’ll go through this.

Most of us may know that top-level refers to main, which is an object of Object. You can prove it very easily like this:

1
2
self #main
self.class #Object

But how about define a top-level method?

1
2
3
4
5
6
7
def foo
  'hello'
end
foo #hello
method(:foo).owner #Object
Object.private_method_defined?(:foo) #true
self.class.private_instance_methods(false) #[:foo]

Does that look like a class definition? Even for the constant definition:

1
2
Foo = 'hello'
Object.const_defined?(:Foo, false) #true

It just looks like you’re operating on class Object. Not only for this, we found that there can also use public, private, or include method, they are shadow in singleton_class of main:

1
singleton_class.private_instance_methods(false) #[:public, :private, :include, :using, :define_method]

Based on our analyst above, we find a strange dual-nature of ruby top-level with some unexpected definitions in main object in purpose. Matz gives a explanation about design of top-level. You can also get more discussion here. From my view, such unusual design exactly helps conciseness of this language, but also gains our confusion and curiosity.

Comments

Metaprogramming Ruby: Core Concepts - Object Oriented Hierarchies

Since last several articles, we’ve gone through all basic but the most important internal definitions in Ruby lang. They are BasicObject, Object, Module, and Kernel, don’t forget Class. The five composite the root of Object Oriented Hierarchies in Ruby world. One of their most useful benefit is incredible metaprogramming ability, which already provides people revolutionary tools like Ruby on Rails. Frankly speaking, the word metaprogramming makes itself too mysterious to be accepted by most people. Conciseness and Expressiveness - are the only purpose of this feature.

1. Class

Classes in Ruby are first-class objects, each class in Ruby is an instance of class Class. Typically, you create a new class by using: Class.new. When a new class is created, an object of type Class is initialized and assigned to a global constant (Name in this case). When Name.new is called to create a new object, the new method in Class is run by default. Classes, modules, and objects are interrelated, we will discuss more in next chapter.

Public Class Methods

new

Creates a new anonymous (unnamed) class with the given superclass (or Object if no parameter is given). You can give a class a name by assigning the class object to a constant. If a block is given, it is passed the class object, and the block is evaluated in the context of this class using class_eval.

Public Instance Methods

allocate

Allocates space for a new object of class’s class and does not call initialize on the new instance. The returned object must be an instance of class.

new

Calls allocate to create a new object of class’s class, then invokes that object’s initialize method, passing it args. This is the method that ends up getting called whenever an object is constructed using .new.

superclass

Returns the superclass of class, or nil. Returns nil when the given class does not have a parent class.

Private Instance Methods

inherited

Callback invoked whenever a subclass of the current class is created.

2. Hierarchy of Five Kings

For more clearly, here we give a figure to cover all five items listed in preface. (We also list two guests in this figure to represent custom codes in the hierarchy)

More advance Ruby OO hierarchy

From the figure, we can see that class can be completely separated with objects under well design. Ruby uses this kind of hierarchy to implement its internal Object Oriented design. Here we would like to introduce the responsibilities on metaprogramming for each in above figure, and have deeper discussion on Module and Class, which are directly related to custom codes.

BasicObject

BasicObject is the root of all objects in Ruby through its subclass Object. The only class method in BasicObject is ‘new’, which is used to allocate memory to new object, and do initialization. For the public instance methods, mainly including object_id, send, and instance_eval/instance_exec to provide basic operations on any objects in Ruby. Also, BasicObject figures out the method_missing mechanism, which is very important for Ruby DSL definition, and singleton methods hook — which actually connects classes and objects in Ruby.

Object

If any class is defined by using keyword ‘class’ without other specific inheritance, it should be inherited from Object by default. Object offers the top useful Constants in Ruby world, like ARGV, ENV, RUBY_VERSION, STDIN/STDOUT/STDERR, etc. Also for public instance methods, it provides objects comparison, clone/dup, freezing, and respond_to? checking, etc.

In metaprogramming field, Object mainly focus on instance level usage, like getting class of object, extend, instance_of?, methods, retrieving singleton_class/singleton_methods of object. etc.

Kernel

Kernel is very special in the hierarchy, because it’s absolutely not class but an object. On the other hand, Kernel is mixed into class Object, making it become the earliest members in the whole hierarchy and also play very important role in the process.

Unlike Object, Constants in Kernel do not refer to some parameters of internal Ruby lang, they’re more like an utility or tool kit for subclasses, especially the reference to popular classes. Kernel also plays hard in Process operations(abort, fork, spawn), Source file loading(auto loading, require), Exception(catch, fail, raise, throw), String(chomp, chop, format, gets, gsub, sprintf, sub), Controlling(loop, sleep, rand, srand, trace/untrace), and IO(open, print, put, readlines, select).

Kernel also enhances the metaprogramming, like binding, block_given?, caller, eval, lambda, and proc.

3. Modules and Classes

Module

Basically, Modules serve two purposes in Ruby: namespacing and mix-in functionality.

A namespace can be used to organize code by package or functionality that separates common names from interference by other packages. Mix-in functionality allows sharing common methods across multiple classes or modules. Ruby comes with the Enumerable mix-in module which provides many enumeration methods based on the each method and Comparable allows comparison of objects based on the <=> comparison method.

A module is created using the module keyword.

Class

Every class is also a module, but unlike modules a class may not be mixed-in to another module (or class). Like a module, a class can be used as a namespace. A class also inherits methods and constants from its superclass. Use the class keyword to create a class. Any method defined on a class is callable from its subclass, the same is true for constants. You can override the functionality of a superclass method by redefining the method. If you wish to invoke the superclass functionality from a method use super.

If you do not supply a superclass your new class will inherit from Object. You may inherit from a different class using < followed by a class name.

When used without any arguments super uses the arguments given to the subclass method. To send no arguments to the superclass method use super(). To send specific arguments to the superclass method provide them manually like super(2). super may be called as many times as you like in the subclass method.

Common properties

Note that there are many similarities between modules and classes. Besides the ability to mix-in a module, the description of modules also applies to classes.

Reopening

Reopening classes is a very powerful feature of Ruby, but it is best to only reopen classes you own. Otherwise it may lead to naming conflicts or difficult to diagnose bugs.

Nesting

Modules may be nested.

Packaging

Many packages create a single outermost module (or class) to provide a namespace for their functionality. You may also define inner modules using :: provided the outer modules (or classes) are already defined.

Self

self refers to the object that defines the current scope. And it will change when entering a different method or when defining a new module.

Constants

Accessible constants are different depending on the module nesting (which syntax was used to define the module). if you use :: to define A::B without nesting it inside A a NameError exception will be raised because the nesting does not include A. If a constant is defined at the top-level you may preceded it with :: to reference it.

Methods

Class methods may be called directly(This is slightly confusing, but a method on a module is often called a “class method” instead of a “module method”. See also Module#module_function which can convert an instance method into a class method.). When a class method references a constant it uses the same rules as referencing it outside the method as the scope is the same. Instance methods defined in a module are only callable when included. These methods have access to the constants defined when they were included through the ancestors list.

Visibility

Ruby has three types of visibility. The default is public. A public method may be called from any other object.

The second visibility is protected. When calling a protected method the sender must be a subclass of the receiver or the receiver must be a subclass of the sender. Otherwise a NoMethodError will be raised. Protected visibility is most frequently used to define == and other comparison methods where the author does not wish to expose an object’s state to any caller and would like to restrict it only to inherited classes.

The third visibility is private. A private method may not be called with a receiver, not even self. If a private method is called with a receiver a NoMethodError will be raised.

Alias and Undef

You may also alias or undefine methods, but these operations are not restricted to modules or classes.

4. Singleton classes

The singleton class (also known as the metaclass or eigenclass) of an object is a class that holds methods for only that instance. You can access the singleton class of an object using class << object like this:

1
2
3
4
5
6
7
8
9
class Foo
  class << self
    # self is the singleton class here
  end
end

class << Foo
  # self is the singleton class here
end

This allows definition of methods and attributes on a class (or module) without needing to write def self.my_method.

Since you can open the singleton class of any object this means that this code block:

1
2
3
4
5
bar = Object.new

def bar.bar_method
  1 + 1
end

is equivalent to this code block:

1
2
3
4
5
6
7
bar = Object.new

class << bar
  def bar_method
    1 + 1
  end
end

Understanding singleton class in ruby is very important to investigate internal ruby, especially for its OO design. In next article, we’ll introduce the scope mechanism in ruby.

Comments

Metaprogramming Ruby: Core Concepts - Kernel

The Kernel is the object of Module, thus it has all instance methods from the parent. Also it’s available as instance methods in every Ruby object since the Kernel is included by class Object. Kernel has widespread usage including but not limit to internal Constant references, Processes operations, Loading Strategies, Exceptions, Controlling, IO, and Basic string operations. This article will focus on metaprogramming on instance level.

Kernel

All Constants and instance methods are public in Kernel, thus they all can be used in any objects as callee.

binding

Returns a Binding object, describing the variable and method bindings at the point of call. This object can be used when calling eval to execute the evaluated command in this environment.

eval

Evaluates the Ruby expression(s) in string. If the binding is given, which must be a Binding object, the evaluation is performed in its context. If the optional filename and lineno parameters are present, they will be used when reporting syntax errors.

block_given? / iterator?

Returns true if yield would execute a block in the current context. The iterator? form is mildly deprecated.

callcc{|cont| block}

Generates a Continuation object, which it passes to the associated block. You need to require ‘continuation’ before using this method. Performing a cont.call will cause the callcc to return (as will falling through the end of the block). The value returned by the callcc is the value of the block, or the value passed to cont.call. In general, Continuation object is used to be analogous to C setjmp/longjmp, or a thread. cont will return itself, and cont.call will jump to end of callcc block.

caller

Returns the current execution stack—an array containing strings in the form file:line or file:line: in ‘method’. Returns nil if start is greater than the size of current execution stack. Optionally you can pass a range, which will return an array containing the entries within the specified range.

caller_locations

Returns the current execution stack—an array containing backtrace location objects. Returns nil if start is greater than the size of current execution stack. Optionally you can pass a range, which will return an array containing the entries within the specified range.

eval

Evaluates the Ruby expression(s) in string. If binding is given, which must be a Binding object, the evaluation is performed in its context. If the optional filename and lineno parameters are present, they will be used when reporting syntax errors.

global_variables

Lookup global variables.

lambda

Equivalent to Proc.new, except the resulting Proc objects check the number of parameters passed when called.

proc

Equivalent to Proc.new

Comments

Metaprogramming Ruby: Core Concepts - Module

For internal ruby, Module is mainly used for holding class methods for descendants. Suppose each object has class of A, A always has class of Class, and Class inherits from Module, thus class A should hold all class methods from instance methods in Module. On the other hand, user can mix in self defined Module to provide more instance methods for their class, or use module function directly.

Module

Module provides two kinds of class level metaprogramming methods, either public or private. Each of them has different usage for building descendants.

Public instance methods

ancestors

Returns a list of modules included/prepended in current module.

class_eval

Evaluates the string or block in the context of module, except that when a block is given, constant/class variable lookup is not affected. This can be used to add methods to a class. class_eval returns the result of evaluating its argument.

class_exec

Evaluates the given block in the context of the class/module. The method defined in the block will belong to the receiver. Any arguments passed to the method will be passed to the block. This can be used if the block needs to access instance variables.

class_variable_defined?

Returns true if the given class variable is defined in object.

class_variable_get

Returns the value of the given class variable (or throws a NameError exception). The @@ part of the variable name should be included for regular class variables.

class_variable_set

Sets the class variable named by symbol to the given object.

class_variables

Returns an array of the names of class variables in current module. This includes the names of class variables in any included modules, unless the inherit parameter is set to false.

const_defined?

Says whether module or its ancestors have a constant with the given name: If module is a Module, additionally Object and its ancestors are checked. In each of the checked classes or modules, if the constant is not present but there is an autoload for it, true is returned directly without autoloading. If the constant is not found the callback const_missing is not called and the method returns false. If inherit is false, the lookup only checks the constants in the receiver. In this case, the same logic for autoloading applies. If the argument is not a valid constant name a NameError is raised with the message “wrong constant name name”.

const_get

Checks for a constant with the given name in module. If inherit is set, the lookup will also search the ancestors (and Object if module is a Module). The value of the constant is returned if a definition is found, otherwise a NameError is raised. This method will recursively look up constant names if a namespaced class name is provided.

const_missing

Invoked when a reference is made to an undefined constant in module. It is passed a symbol for the undefined constant, and returns a value to be used for that constant. If found, it returns the loaded class. It therefore implements an autoload feature similar to Kernel#autoload and #autoload.

const_set

Sets the named constant to the given object, returning that object. Creates a new constant if no constant with the given name previously existed. If symbol or string is not a valid constant name a NameError will be raised with a warning “wrong constant name”.

constants

Returns an array of the names of the constants accessible in module. This includes the names of constants in any included modules, unless the inherit parameter is set to false.

instance_method

Returns an UnboundMethod representing the given instance method in module.

instance_methods

Returns an array containing the names of the public and protected instance methods in the receiver. For a module, these are the public and protected methods; for a class, they are the instance (not singleton) methods.

method_defined?

Returns true if the named method is defined by module (or its included modules and, if module is a class, its ancestors). Public and protected methods are matched. String arguments are converted to symbols.

module_eval

Evaluates the string or block in the context of module, except that when a block is given, constant/class variable lookup is not affected. This can be used to add methods to a class. module_eval returns the result of evaluating its argument.

module_exec

Evaluates the given block in the context of the class/module. The method defined in the block will belong to the receiver. Any arguments passed to the method will be passed to the block. This can be used if the block needs to access instance variables.

name

Returns the name of the module module. Returns nil for anonymous modules.

prepend

Invokes Module.prepend_features on each parameter in reverse order.

private_class_method

Makes existing class methods private. Often used to hide the default constructor new.

private_constant

Makes a list of existing constants private.

private_instance_methods

Returns a list of the private instance methods defined in module. If the optional parameter is false, the methods of any ancestors are not included.

private_method_defined?

Returns true if the named private method is defined by _mod_ (or its included modules and, if module is a class, its ancestors).

protected_instance_methods

Returns a list of the protected instance methods defined in module. If the optional parameter is false, the methods of any ancestors are not included.

protected_method_defined?

Returns true if the named protected method is defined by module (or its included modules and, if module is a class, its ancestors).

public_class_method

Makes a list of existing class methods public.

public_constant

Makes a list of existing constants public.

public_instance_method

Similar to instance_method, searches public method only.

public_instance_methods

Returns a list of the public instance methods defined in module. If the optional parameter is false, the methods of any ancestors are not included.

public_method_defined?

Returns true if the named public method is defined by module (or its included modules and, if module is a class, its ancestors).

remove_class_variable

Removes the definition of the symbol, returning that constant’s value.

singleton_class?

Returns true if module is a singleton class or false if it is an ordinary class or module.

Private instance methods

Private instance methods in Module are mainly used in internal classes level.

alias_method

Makes new_name a new copy of the method old_name. This can be used to retain access to methods that are overridden.

append_features

When this module is included in another, Ruby calls append_features in this module, passing it the receiving module in module. Ruby’s default implementation is to add the constants, methods, and module variables of this module to module if this module has not already been added to module or one of its ancestors.

define_method

Defines an instance method in the receiver. The method parameter can be a Proc, a Method or an UnboundMethod object. If a block is specified, it is used as the method body. This block is evaluated using instance_eval, a point that is tricky to demonstrate because define_method is private.

extend_object

Extends the specified object by adding this module’s constants and methods (which are added as singleton methods). This is the callback method used by Object#extend.

extended

The equivalent of included, but for extended modules.

included

Callback invoked whenever the receiver is included in another module or class. This should be used in preference to Module.append_features if your code wants to perform some action when a module is included in another.

method_added

Invoked as a callback whenever an instance method is added to the receiver.

method_removed

Invoked as a callback whenever an instance method is removed from the receiver.

method_undefined

Invoked as a callback whenever an instance method undefined but called from the receiver.

module_function

Creates module functions for the named methods. These functions may be called with the module as a receiver, and also become available as instance methods to classes that mix in the module. Module functions are copies of the original, and so may be changed independently. The instance-method versions are made private. If used with no arguments, subsequently defined methods become module functions. String arguments are converted to symbols. prepend_features When this module is prepended in another, Ruby calls prepend_features in this module, passing it the receiving module in module. Ruby’s default implementation is to overlay the constants, methods, and module variables of this module to mod if this module has not already been added to mod or one of its ancestors. See also Module#prepend.

prepended

The equivalent of included, but for prepended modules.

refine

Refine klass in the receiver. Returns an overlaid module.

remove_const

Removes the definition of the given constant, returning that constant’s previous value. If that constant referred to a module, this will not change that module’s name and can lead to confusion. remove_method Removes the method identified by symbol from the current class. For an example, see Module.undef_method.

undef_method

Prevents the current class from responding to calls to the named method. Contrast this with remove_method, which deletes the method from the particular class; Ruby will still search superclasses and mixed-in modules for a possible receiver. String arguments are converted to symbols.

Comments

Metaprogramming Ruby: Core Concepts - Object

As the default root of all Rubric objects, Object holds quite a lot valuable instance methods for common usage. It mixes in the Kernel module to provide global accessibility of Kernel APIs for all inherited classes. This article will introduce Object definition.

Object

Object only holds a few constants and instance methods for objects of descendants, all these methods are only used for instance level metaprogramming.

class

Gets class Constant of object.

define_singleton_method / new_method

Defines a singleton method in the receiver. The parameters can be a Proc, a Method or an UnboundMethod object. If a block is specified, it is used as the method body.

extend

Enhance object with instance methods from modules given as a parameter.

instance_of?

Check if obj is an instance of the given class(Except ancestors).

instance_variable_defined?

Check if the given instance variable is defined in obj.

instance_variable_get

Gets the value of the given instance variable, or nil if the instance variable is not set. Just note that the @ part of the variable name should be included for regular instance variables.

instance_variable_set

Sets the instance variable named by symbol to the given object. The variable does not have to exist prior to this call.

instance_variables

Returns array of instance variables

is_a? / kind_of?

Returns true if class is the class of object, or if class is one of the superclasses of obj or modules included in obj.

itself

returns object itself

method

Looks up the named method as a receiver in object, returning a Method object. The Method object acts as a closure in object’s object instance, so instance variables and the value of self remain available.

methods

Returns a list of the names of public and protected methods of object. This will include all the methods accessible in obj’s ancestors. If the optional parameter is false, it returns an array of object’s public and protected singleton methods, the array will not include methods in modules included in object.

private_methods

Returns the list of private methods accessible to object.

protected_methods

Returns the list of protected methods accessible to object.

public_method

Similar to method, searches public method only.

public_methods

Returns the list of public methods accessible to obj.

public_send

Invokes the method identified by symbol, passing it any arguments specified. Unlike send, #public_send calls public methods only.

remove_instance_variable

Removes the named instance variable from object, returning that variable’s value.

respond_to_missing?

Hook method to return whether the object can respond to id method or not.

send / __send__

Invokes the method identified by symbol, passing it any arguments specified. You can use __send__ if the name send clashes with an existing method in obj.

singleton_class

Returns the singleton class of object. This method creates a new singleton class if obj does not have one. If object is nil, true, or false, it returns NilClass, TrueClass, or FalseClass, respectively. If object is a Fixnum or a Symbol, it raises a TypeError.

singleton_method

Similar to #method, searches singleton method only.

singleton_methods

Returns an array of the names of singleton methods for object. Only public and protected singleton methods are returned.

Comments

Metaprogramming Ruby: Core Concepts - BasicObject

Metaprogramming is one of core features for ruby lang, it’s widely used from internal ruby to standard library, also endless magic gems. In next a few articles we will focus on those core stuff. I recommend for Metaprogramming ruby 2 as a reference, except a little verbose and may be only for primer usage. For more ascent knowledge, perhaps you need to track the trunk of ruby. This essay will concentrate on BasicObject, which is parent for all classes in ruby.

BasicObject

As the object oriented hierarchy in ruby core, each type of class is also object of Class except BasicObject, and All the objects of Class has root superclass BasicObject, the one is also object of Class. Thus we’ll have whole picture below.

Ruby OO hierarchy

i.class methods

BasicObject only has single public class method, names “new”.

ii.instance methods

logical operators, any instance of BasicObject is almost empty, with only a few methods, like logical operators !, !=, ==. Here == has same meaning with equal? for object level equality, The difference is that == is recommended for descendant overriding, but equal? should always be same as it is defined in BasicObject.

__id__ / object_id

An integer identifier of an object, should be unique for a given object. But some objects of builtin classes are reused for optimization, like immediate values (not passed by reference but value, like nil, true, false, Fixnums, Symbols, and some Floats) and frozen string literals.

__send__ / send

Double underscore gives a backup for pure alphabet version which may be overrode through other ways. Invokes the method, and passes all arguments using dynamic number of arguments.

instance_eval / instance_exec

instance_eval support passing a string of ruby source code, with file path and line number when compilation errors happen, also given block, within the context of receiver. Thus eval code will have access to instance variables and private methods.

instance_exec only support using block as an argument, but allows one more argument to be passed from outer scope into receiver.

iii.private instance methods

method_missing

In BasicObject, method_missing belongs to private instance methods, any call for this method will raise an error. But descendant can override this method to do dynamic processing, according to user’s input like symbol name and arguments. If overrode method does not want to do any processing, super should be called to pass message bottom from up.

singleton_method_added / singleton_method_removed / singleton_method_undefined

Callback for singleton method operations. As we already know, ruby will not create the whole clone of the class for any objects. Only with a middle layer between object and class - singleton class. Singleton class is generated when object comes out, and any messages sent to object should be transferred to its singleton class first, then its own class if not matches in singleton class.

class’s singleton class is different to object’s singleton class. That’s because singleton class of a class should be inherited from its parent’s singleton class, until BasicObject’s singleton class, then it can reach non singleton class. But for object, singleton class should always have parent class of its own class.

Comments

Microsevices陷阱: 本质与陷阱

最近两年,“微服务”这个词从最初的争议,到随后的逐渐接受,如今已经成为许多时髦的技术人员常挂在嘴边的词汇。在他们看来,似乎你不“微服务”就落伍了——因为你看,好像全世界都在“微服务”。然而当一些自诩为技术先驱的人开始尝试将其落地,并且后来发现自己当初根本就不了解到底“微服务”真正意味着什么时,我们才发现,要想革现代软件工程的命,远没发几篇文章、或者侃侃而谈来的容易。

1. 微服务的本质

可能你见过或者听过某些自称正在构建“微服务架构”的人,坦白的说,“微服务架构”真的不是指什么具体的软件架构,和大多数与之相关的早期技术一样,也没有一个标准告诉你什么是微服务、如何通过以下几个步骤构建微服务、或是让你参加培训,结束之后取得XXX认证,就证明你掌握了微服务架构。

其实微服务更像是一系列前沿软工思想的汇集,至于它为什么叫“微服务”,本系列的第一篇文章就已经提到过(虽然至今都有争议),但你可能会觉得它有点标题党——这也影响不了它拥有这个名字的既定事实。撇开名字,我们来看看微服务所承继的几个软件领域:

基于业务概念建模

从稳定性的角度考虑,围绕业务边界上下文所构建的接口要比来源于技术概念的接口更加不易变更,从而降低风险。这部分来源于领域驱动设计的主张。

自动化

自动化涉及方方面面,包括自动化测试、自动部署、自动配置管理乃至整个持续交付流程,都被认为是微服务落地的重要途径。

隐藏内部实现细节

隐藏实现的好处在于,一方面数据隐藏能够显著降低数据访问代码的耦合性;另外,服务内可以采用异构技术栈来实现,无论使用何种技术,你都可以用REST、RPC来实现服务间通信。

一切去中心化

去中心化的核心是组织自服务团队。近年来我们已经尝过自动化带来的好处,但鉴于Conway法则,设计开始来自领域而非现有的技术架构,而为了保证每个团队能够独立开发、测试和部署,按照服务划分现有团队是一个趋势(而非现实)。自服务团队能显著降低实施自动化的复杂度,更加贴近业务领域,并且形成一个分散的系统架构。与ESB或类似的编配系统完全相反的是,去中心化强调采用编排以及哑管道,并且在服务边界上采用智能终端来表现关联逻辑和数据,从而实现低耦合。

独立部署

强调每个服务都必须是可独立部署的,甚至当发生重大变更时,也倾向于采用版本共存终端来保证各自的独立性。独立部署能够带来更快的feature发布、强化自动化部署流程的一致性(而非经常需要人工编配部署管线)。推荐采用单服务-单主机的部署架构,从而避免部署单个服务时带来的边际效应;采用蓝绿部署/金丝雀发布来降低出错的风险;并且使用消费者驱动契约测试来保证集成接口的一致性。

错误隔离

分布式系统能够做到比单一系统更加可靠——前提是做好应对各种错误的万全准备(这也是人类毕生追求的目标…)。如果我们不能保证下游服务的正确性,系统就有可能发生级连式的灾难,且比单一系统脆弱的多。因此要时刻考虑网络的不可靠性、服务间通信的各种潜在问题:超时、隔板、断路器等在应对灾难时的作用、理解网络分区带来的后果、以及权衡一致性和可用性两种方案。

完备监控

传统意义上的监控,恐怕只是一个简单的dashboard,列出所有运行中节点的状态,并且当某个节点的指标超过阈值时产生警告。但随着去中心化的深入,传统监控方式根本不能应对复杂架构出现的各种问题。因此,采用语义化监控、综合化监控去模拟用户的真实行为,聚合用户日志和统计数据,采用关联ID跟踪消息在系统内的传递路径。

严格的说,不具备上述特征的系统就不能被称作微服务。事实上,当你开始尝试上述某个或某几个新的领域时,多数情况下你的系统无疑蕴涵了极大的风险——因为只有整体采用才能实现微服务的闭环,否则只能是一个摇摇欲坠的怪胎。有的人可能认为可以从简单模型开始演进——但注意,这里所谓的演进并不是从某个独立的方面出发,而是要在一开始就要做通盘设计,并且以简单方式先落地而已。

那么企业费尽千辛万苦、从事如此巨大的迁移工程——效果真的能够立竿见影吗?

2. 发觉陷阱

一些谨慎的开发者认为,可以先从单一系统入手,再逐渐向微服务过渡。对此,微服务的拥簇者声称:“你将无法(或很难)精炼出微服务架构,如果不在项目的最开始就设计成那样”。

“这无疑又在扯过早做大型设计的蛋(That’s BDUF Baloney)。”

Uncle Bob更倾向于“迫使无知”,即好的架构应尽量保证组件对部署和内部通信方式的透明,这样就可以根据实际情况选择部署成微服务,或多个单一系统+应用服务器,亦或纯粹的单一系统。这种“实际情况”有时可能是需要大规模扩展的、只需要少量部署的、以及单个节点足矣的。

一旦打破了“迫使无知”,则很有可能会造成过度设计,使得太多因素过早被考虑进来,最终却形成一堆烂摊子。对于微服务强调的其它几个方面,例如跨语言的服务实现——确实,采用REST的通信架构使得服务间实现强解耦,包括语言、数据库等。但是,什么样的业务需求会导致项目初期就要切割成不同语言、甚至采用不同的数据库技术呢?这种高配置复杂度的架构随之带来的好处又能有多少?

而对于“单一系统”的说法,Uncle Bob更认为这种广告式的宣传,使得采用非微服务形式的系统听起来就像是一块又大又丑的“石头”,造成受众对微服务架构的盲目崇拜,进而为布道者带来利益。

比起微服务,Uncle Bob更加倾向于一种“整洁架构”的风格,后者是对近年来一些流行系统架构的全面总结,焦点主要集中在代码架构方面,而对微服务强调的独立部署、异构数据库等持开放的观点。

3. 稳步向前

当然,无论是微服务,还是整洁架构,能最终轻松玩转的仍然是极少数。例如,从源头的设计出发,对领域的理解会影响边界上下文的稳定性,你要确信自己真的“彻底认识”了该领域,否则后面的一切都是空谈,还不如老老实实把单一系统做好。

另外,如果任何环节存在未知领域,就尽量不要采取过于激进的做法——这是遏制风险的关键。

对于任何一种架构,只有当整体规模增加到一定限值时,才能够真正考验架构的有效性,而面向未知又十分危险——因此强烈不建议如此激进地改变根本架构。毕竟在整个生态圈中,技术团队只是起到实现和保障作用,过早、过度引入风险是完全没有必要的。

那么何时才能选择向前呢?如今声称自己正或多或少实现微服务架构的团队包括Amazon、Netflix、REA和Gilt等,无一不是从现有实践中持续发掘存在的问题,并为此开发了一系列工具、平台甚至整体实践,其中的部分已经成为流行的开源项目——可见,对于某些技术能力领先世界的公司来说,微服务之路尚且存在各种困难,而剩下积累不深却大谈微服务的团队,要么是盲目追新,要么就是醉翁之意不在酒了。

Microsevices陷阱: 层级演进(下)

8. 服务发现

随着服务数量的增加,如何获取某个服务的信息则成为挑战,同时它也是部署监控的前提条件。另一些开发中遇到的例子,如:一、想获取当前环境中运行的服务(也就是采集监控对象);二、想获取当前系统中的API列表从而避免重新发明轮子;这些需求其实都隐含了一种新的需求:针对“服务”的发现机制,在近年来发展非常迅速,已经形成了几个相当成熟的解决方案。它们基本上都遵循了如下工作模式:首先提供一种针对服务的注册过程,当启动新实例时就将其注册进系统;然后,提供一种发现系统内已注册服务的途径。我们先来看一种历史最悠久也最简单的解决方案:

DNS

DNS是解决服务发现问题最直接、简便和有效的手段。因为借助DNS,尽管实例IP可能随时发生变化,但域名是不变的。因此在早期,大多数应用都会采用类似<service-name>-<environment>.company.com这样的二级域名来指定服务地址。有的甚至去掉了environment,直接搭建不同的DNS服务器并设置不同的映射表,从而实现配置管理的一致性。后者看起来很方便,但成熟的解决方案不多,亚马逊的Route53服务是少数优秀的代表。

然而采用DNS也存在一定风险性。在DNS协议中存在TTL的概念,也就是说主机内部会维护一个DNS表,当TTL未失效时系统仍将采用本地的DNS映射表,这种设置让实例的更新带来一定风险。为了解决这一问题,DNS并不直接指向实例IP,而是若干实例前设置的反向代理(这个通常是固定不变的)。

DNS方案由于其易用性而得到了大量的应用,但随着微服务架构规模的增加,这种方法的有效性和可维护性将逐渐降低,特别是DNS服务器节点的存在,于是就出现了更多动态服务发现方案。

9. 服务注册

由于DNS本身的局限性,许多基于服务注册的发现方案被提出并得到了广泛应用。

Zookeeper

Zookeeper最初是Hadoop项目的一部分,其用途非常广泛,包括配置管理、同步服务间数据、选举Leader、消息队列以及命名服务(这才是本文重点)。Zookeeper首先是高可用的,其后台是一个集群并且实现数据Replica;Zookeeper的核心提供一种存储信息的层级命名空间,客户端可以在该层级插入、修改和查询节点;此外,它们能够添加针对节点的监视以观察该节点是否发生变化;也就是说,我们可以把服务的地址信息保存在这种存储结构,并且当其中的信息发生改动时执行其它变更操作,例如关闭上游服务。

Zookeeper是一个非常通用的系统,你可以只把它看作是一个能够保存信息树,并且能在任何变化时发出警告的高可用服务。由于其底层性和单一性,目前针对Zookeeper的扩展库也非常之多。从现在的时间来看,Zookeeper可能有些过时——因为与更多后起之秀相比,它看起来不是那么开箱即用,但不可否认其曾经十分流行。必须承认,由于Zookeeper底层算法PAXOS实现起来非常困难,而该产品又饱经考验,任何企图重新发明轮子的组织都在此吃过不少苦头。这正是分布式协调系统复杂性的体现。

Consul

Consul同样支持配置管理和服务发现,但提供了更多高级功能,从而更加易用。例如针对服务发现的HTTP接口,以及一个简单的DNS服务器,甚至还提供SRV记录,能够同时提供给定名字的IP和端口信息。这就使得曾经采用DNS服务的公司迁移到Consul变得非常容易。

Consul还提供了节点健康检查功能,这就包含了一部分监控系统的能力,并且由于其高容错设计和依赖临时节点机制的系统的针对性,使其有能力在某些情况下完全替代Nagios和Sensu。

Consul的RESTful HTTP接口覆盖了从服务注册、查询信息、以及健康检查等大部分功能,使其与现有系统集成变的十分简便。其底层依赖的Serf能够检测集群内节点,进行错误管理以及发出警告。良好的设计使Consul如今成为该领域极具竞争力的明星之一。

Eureka

Eureka是由Netflix开源的、只关注服务发现领域的系统。Eureka提供基本的负载均衡功能,基于REST的集成终端,以及基于Java的客户端库。但是,由于Eureka是基于JVM并且实现了所有客户端,这对于多语言环境来说可能会是一个挑战。

自己动手

在某些公有云服务器中,服务发现可以借助一些内置功能来实现,当然这要建立在你对该产品非常熟悉,且系统规模较小的情况之下,通常会与采用上述重型框架相比省去很多麻烦。另一方面,服务发现除了是对分布式系统内部提供信息,任何现有的report和dashboard都可以借此提供实时的节点信息,从而允许人工检查。

10. 服务文档化

随着微服务架构的实施,更多时候我们希望能够以API的形式暴露一部分服务缝隙,从而允许其他开发者接入更多类型的服务。但是构建开放API的过程是非常复杂的,特别是在采用微服务架构时,如何保证服务文档的实时性将成为挑战。这里介绍目前流行的两个服务文档技术:Swagger和HAL。

Swagger

Swagger能够通过一组Web界面展示当前的API信息,而你只需要定义一套POST模版即可。Swagger要求相关服务暴露一个符合标准的文件(可以通过Swagger提供的标准库实现),例如对于Java你可以使用方法标注描述相关API。

HAL和HAL浏览器

HAL即Hypertext Application Language,是一种超媒体资源的描述标准,HAL同样提供用户界面和标准库。通过Web界面,用户不仅可以查看API信息,还能直接调用相关API。不同的是,HAL中所有的文档信息都基于超媒体控制,因此很容易就能够将信息对外展示。这就限制了用户只能使用超媒体描述系统内资源,而从现有系统迁移则远没那么容易。

11. 自描述系统

在SOA的发展初期,类似全局描述、发现以及集成UDDI标准被用于帮助人们理解当前运行着什么服务。事实证明,UDDI对大多数项目来说都非常重,从而导致更多替代产品的出现。

不可否认,获取系统的整个框图并理解其架构对于团队成员来说十分重要。通过关联ID跟踪下游服务并理解调用链、从而让我们更快理解系统原理。采用类似Consul的服务发现技术,能够让我们查看当前运行的微服务。HAL能展示出当前终端上包含的所有服务能力,同时状态监控页面和相关系统能够使我们整体和局部的状态。上述信息都能够用于为进一步设计提供线索,且比只有一个简单到随时会过期的wiki页面强得多。尽管多数情况下我们都是从后者开始的,但不要忘了把上述信息逐渐引入到系统中,随着系统复杂性增加,描述信息的演化能让我们更好的理解系统。