软件设计与架构笔记(3)

模块化编程(Modular Programming):信息隐藏与职责分割

上世纪60年代起,人们意识到实现复杂系统的前提是把系统合理分割为相互独立的部分,这些独立的部分被称作模块。与前文提到的结构化编程和过程式编程的区别是,一个模块可包含若干个子程,也允许组装不同模块以实现子程或程序。D.L. Parnas把这种编程技术称为模块化编程[DLP72],其中模块意味着任务职责,而模块化设计则表示一系列的跨模块的“系统级”设计决策。自此,模块化成为软件设计领域的重要主题之一。

针对模块化的研究包含两个基本组成部分:

  1. 一个良好的模块化系统(设计)应具备哪些特征?

  2. 一个良好的模块化系统(设计)应如何实现?

信息隐藏(Information Hiding)

最初的软件设计方法论认为,组织应当建立统一的文档管理系统,软件由设计人员设计好后开放给全体人员,从而让每个人都尽量了解设计背后包含的一系列决策。1971年,Parnas首次提出信息隐藏的概念,反驳了前述的传统设计“广播”实践[DLP71]。

从软件结构的角度看,软件设计包含了对模块自身特征以及模块间的连接(connetion)的描述,其中连接意味着设计对模块间作用的假设。而我们已经知道软件结构最重要的两个目标:系统变更正确性检验,而一个好的软件结构应使上述目标变得更加容易。以简化系统变更为例,如何使针对当前模块的变更不会传递到其它模块呢?答案当然是应尽量使针对当前模块的变更不至于打破其它模块对其所做的假设,即连接的稳定性。那么如何保证连接的稳定呢?直观来看当然是尽量减少假设的规模,即减少连接所包含的信息。

再以软件文档系统为例,实践证明,保证系统设计文档和代码的一致性需要花费可观的成本,这在大多数组织来说都难以实现。同时为了保证文档自身的可理解性,一个好的实践是建立组织统一的标准和术语,但实践证明这也很难做到,因为假设总会根据需求发生变更,而一旦新的假设违反了组织统一标准,则会引起标准的误用,而反过来扩展标准又有可能造成对已有文档假设的破坏。上述复杂性意味着,在一个实践统一文档标准的组织内,标准会尽量维持系统设计的最小假设,而这又会与文档本身的知识传递作用相违背。

Parnas认为设计人员应尽量“控制”信息的传播,例如在设计中只使用外部接口描述该模块,从而避免细节过早暴露,对外部隐藏那些尚待决定、不稳定或不应被外部了解的具体实现。

职责分割(Responsibility Segment)

人们普遍认为应根据功能职责划分系统模块,但缺少统一的划分方法,导致具体划分时会出现不同结果,原因在于实际问题域的复杂性。以比较简单的经典KWIC系统为例,考虑如下两种模块化设计方案:

  1. 系统被划分成5个部分:分别是输入模块I、循环移位(circular shift)模块C、字母排序(Alphabetizing)模块A、输出模块O和主控制模块M。具体来说,I接受行格式的数据输入,把每个单词用四个字母进行压缩表示,其余字母作为单词的结尾,然后将其存储到系统核心(core);当I读完所有数据,C先对核心中每行数据进行循环移位处理,并记录每条新数据到原始数据的索引,最后把数据存储回核心;然后,A从核心中读取数据,把C中生成的数据按字母进行排序并存储回核心;最后O把A中排序好的数据结合I中获得的原始数据进行匹配和格式化输出;主控制模块M负责控制其余模块的调用顺序,进行错误处理和进行一些必要的空间分配等操作。从实践角度考虑,该方案具备良好的职责分割和接口设计。

  2. 系统被划分成6个部分:行存储模块L、输入模块I’、循环移位器C’、排序器A’、输出模块O’ 和主控制模块M’。具体来说,L负责提供对行数据进行操作的功能,例如常见的增删改查等;I’负责读入数据并调用L写入数据;C’用来计算并返回所有的循环移位索引。A’的功能是返回给定索引序号的字母序序号;O’用于输出L或C’中包含的数据;M’与方案1中M的功能类似。该方案也具有良好的职责分割和接口设计。

为了进一步比较这两种方案的区别,首先来看两者在分析该问题时所作出的设计决策及其可能影响:

  1. 输入格式。

  2. 存储介质,例如把所有行数据存储在core中,那么假设数据集较大,则该决策就会面临挑战。

  3. 存储压缩,例如对每个单词进行压缩,假设数据集不大,处理时间反而会因为不必要的压缩而增加。

  4. 为循环移位器创建索引,而非直接存储所有数据,对于较小的数据集而言,后一选项可能更加合适。

  5. 为所有数据进行一次性按字母序排序,而非只在需要时进行搜索或只进行部分排序。在某些场景中,可能更希望把索引计算量分配至不同时间的字母序操作中。

以下分别使用是否容易变更是否可独立开发是否便于理解三个具有共识的软件设计目标分析和比较上述方案。

易变更性,由于都拥有唯一的输入模块,那么决策1的任何潜在变化都不会导致输入模块以外的变化;对决策2和3来说,由于涉及数据的格式表示,方案1中由于多个模块都需要直接读写core中的数据,一旦存储格式因假设发生变化就会导致所有关联模块的修改,相应的方案2由于独立出了行存储功能,因此依旧把改动影响限制在了一个模块之内。同样,决策4的变更可能导致存储格式的变化,方案2中的C’模块只用于计算而非存储,因此变更影响小于方案1;决策5的情况则与决策4类似,方案2具有更好的易变更性。

可独立开发,方案1的模块间接口实际上是通过数据存取间接实现的,其实质是数据格式和表结构的设计,在这种情况下,所有相关模块的接口设计都存在一定联系。而方案2通过若干个函数及其参数就实现了模块间的接口,因此对模块分别可独立开发有更好的支持。

可理解性,根据方案1,为了理解模块职责,需要至少理解模块I、C、A内部的实现,特别是数据存取的设计和实现,才能了解模块间的相互关系;相反方案2只需要通过接口函数的定义就可以了解模块职责了。

从职责分割的角度来说,上述两种方案都给出了初看相当合理的划分,但经过分析我们的结论显然是方案2比方案1更好,那么如何实现诸如本例中更好的职责分割呢?

一种用于模块分割的标准与系统设计方法

实际经验表明,人们直观上会倾向于作出符合上节提到的方案1的设计,这是因为方案1更加显而易见:例如借助流程图(flowchart)工具描述系统功能和流程,再自然映射在模块的划分上。而方案2的核心在于每个模块都努力隐藏其设计决策,包括接口和定义也都以较少的信息呈现,Panars认为这反映了一种以隐藏自身设计决策为目标的模块间分割标准,与传统的流程式思考模式有显著区别。

由于构建计算机系统的复杂性,设计人员在60年代起开始采用一些系统模拟语言(Simulation languages)辅助系统设计。显然,当系统需求越复杂,模拟语言也就会变得更复杂,就越难以满足设计人员的目标,因此模拟语言最初的应用并不成功。1968年,沃森研究院的F. W. Zurcher描述了一种迭代式的多层建模方法,通过在不同的抽象层级(Levels of abstraction)上安排设计决策,为设计人员提供了有效的系统思考工具。

Zurcher提出在一个模型中构建系统的多重表示,即抽象层级。以计算机系统为例,在最上层,该模型只表示系统的若干基本任务,并且给定这些功能所达到的目标,即始终优先回答what;进一步,在下一层引入CPU、存储层级和文件系统的概念,并指定连接上层每个任务的程序和数据的划分;然后,再下一层级描述更加细节的系统表示,直到完整描述了整个系统设计,即回答how。迭代在实现上述方法中同样重要。在采用模拟语言时,先实现上一层的系统设计描述程序,程序应包括本层所含的所有抽象定义;在进入下一层时,构建一个独立的可被上层操作的实现本层抽象意义的模拟程序,从而实现迭代式的层次设计结构。

这种层级建模方法中,每一层都仅包含该层定义范围内的设计决策,即令设计人员更容易理解模型及其具体行为,并且把针对设计的修改限定在本层级范围内,降低了设计变更的影响范围。当进入正式实现阶段时,编程人员可以用具体的算法和数据结构实现替换最底层的模拟程序,从而构建完整系统。

结论

如果说结构化编程奠定了现代编程语言的基础,那么模块化编程则为软件设计提供了应对复杂问题的有效工具。与结构化编程和过程式编程几乎一锤定音相比,模块化编程在过去50年间历经了长期演进,虽然70年代开始大量编程语言开始引入模块(module)的概念,但抽象表达本身的复杂性使整个软件设计和开发过程经历了飞速的变革,而这一切源于模块化的设计思想。

引用

[DLP72], On the Criteria to be Used in Decomposing Systems into Modules

[DLP71], Information distribution aspects of design methodology

[FWZ68], ITERATIVE MULTI-LEVEL MODELLING - A METHODOLOGY FOR COMPUTER SYSTEM DESIGN

Comments

软件设计与架构笔记(2)

结构化编程(Structured Programming):计算语言的突破

上世纪50-60年代,人类的计算能力实现了迅猛发展,各界对计算机的应用也有很高期许,越来越多的领域希望得到强大的计算赋能从而实现飞跃。然而当面临的问题越多、越复杂时,人们在解决问题的道路上发现了一条巨大的鸿沟,即以现有的软件构建理论和方法难以应对这些挑战。机遇与挑战并存,这场软件危机(Software Crisis)最终促成了软件工程作为一门独立的学科从计算机科学的襁褓中成长起来。

软件危机这个词最早在1968年的北约组织软件工程会议上被诸多与会者提出[NATO68],由此引发的技术创新和组织行为思辨至今依然活跃。而更现实的影响是,科学家们首先在编程语言本身找到了突破口——结构化编程

发明于上世纪50年代的ALGOL语言,首次用begin…end语句引入了代码块的概念,通过限定其中变量声明的词法作用域,提高程序的可读性,从此引起了围绕代码块的研究。1966年,论文[Bohm66]证明使用三种基本的程序结构就能表达任何可计算函数:顺序执行、条件选择和循环迭代,这为随后针对结构化编程的讨论提供了理论依据。1968年,Dijkstra发表了著名的”GOTO语句有害“的观点,并且肯定了如条件选择、循环等语句的应用,同时称GOTO语句应该在所有“高级语言”(这里指除了机器码之外的语言)中被废除[EWD68]。Dijkstra认为应当尽可能减少静态程序和动态运行进程之间的差距,而GOTO语句造成了大量程序难以被理解,即人很难从混乱的静态代码中认识程序的真正意图。这一废除GOTO语句的言论激起旷日持久的争论,反对者认为GOTO所具有的灵活性能满足持续的系统优化工作,但争论两方基本同意应当对GOTO限制使用。于是,结构化编程开始被广泛接受。

伴随着结构化编程的普及,过程式编程(Procedural programming)也在60年代起被许多流行语言采纳,如COBOL和BASIC。这种编程方法以代码块为基础,允许使用子过程(也称子程或函数)编写程序单元,并且可以被程序随时调用。使得来自不同程序员甚至不同组织的代码变得简单可复用,为随后代码库的流行奠定基础。

结构化程序设计与分析

结构化编程实现了编程语言的巨大进步,作为首席布道者,Dijkstra发表了很多关于程序的可理解性以及结构化编程实践的原则性观点[EWD70],但如何设计结构化程序还需要进一步说明。1971年,在计算机教育领域功勋卓著的Niklaus Wirth详细解释了一种自顶而下逐步求精的程序设计方法,并以数学中经典的八皇后问题(把这个著名问题作为编程案例,原因之一是尚无该问题的已知解析解)为例演示了程序设计从问题分析到实现的过程[NW71]。

简单分析可以得到八皇后问题的直观解法:对于全体候选解的集合A,其中每个解元素x满足条件函数p,即(x ∈ A) ∧ p(x),则:

1
2
3
repeat Generate the next element of A and call it x
until p(x) ∨ (no more elements in A);
if p(x) then x = solution

由排列组合知识可知,集合A的空间可达232,枚举算法效率较低。通过对问题进一步的分析,使用回溯法解决该问题的算法效率较高,即:

1
2
3
4
j := l;
repeat trystep j;
if successful then advance else regress
until (j < 1) ∨ (j > n) 

以上述程序分析结果为基础构建程序,按照回溯算法的基本思想,首先依照specification给出初步实现:

1
2
3
4
5
6
7
variable board, pointer, safe;
considerfirstcolumn;
repeat trycolumn;
  if safe then
  begin setqueen; considernextcolumn
  end else regress
until lastcoldone ∨ regressoutoffirstcol

根据现有结构化编程语言的表达能力,对如下指令进一步分解:

trycolumn:

1
2
3
procedure trycolumn;
repeat advancepointer; testsquare
until safe ∨ lastsquare 

regress:

1
2
3
4
5
6
7
8
9
10
11
procedure regress;
begin reconsiderpriorcolumn
  if ¬ regressoutoffirstcol then
  begin removequeen;
      if lastsquare then
      begin reconsiderpriorcolumn;
          if ¬ regressoutoffirstcol then
              removequeen
      end
  end
end

截至目前,如需对上述程序中的指令做进一步分解,就需要设计额外的数据表示了。通过分析待分解语句,可知需要设计一个记录每位皇后位置的数据表示,例如使用二维数组表达棋盘上的每个方块。这里给出一个优化的数据表示方式:

1
2
integer j (0 ≤ j ≤ 9)
integer array x[1:8] (0 ≤ x[i] ≤ 8) 

其中j表示当前被检查的列序号,一维数组x用于存储上一次被检查方块的坐标,程序的部分指令可以被进一步细化为:

1
2
3
4
5
6
7
8
9
10
11
12
13
procedure considerfirstcolumn;
  begin j := 1; x[1] := 0 end
procedure considernextcolumn;
  begin j := j + 1; x[j] := 0 end
procedure reconsidetpriorcolumn; j := j - 1
procedure advancepointer;
  x[j] := x[j] + 1
Boolean procedure lastsquare;
  lastsquare := x[j] = 8
Boolean procedure lastcoldone;
  lastcoldone := j > 8
Boolean procedure regressoutoffirstcol;
  regressoutoffirstcol := j < 1 

接下来考虑剩余指令testsquare、setqueen和removequeen。

指令testsqaure需要验证是否满足问题条件,通过已知的x数组应不难通过计算进行判定,问题是可能导致较高的计算量,同时考虑到testsquare的调用频次较高,这里采用额外数据表示进行优化,设计三个Boolean型数组,其意义如下:

1
2
3
a[k] = true : no queen is positioned in row k
b[k] = true : no queen is positioned in the /-diagonal k
c[k] = true : no queen is positioned in the \-diagonal k 

那么testsquare就可以用简单的布尔运算表示,其余指令也可以通过上述结构完成:

1
2
3
4
5
6
procedure testsquare;
  safe := a[x[j]] ∧ b[j+x[j]] ∧ c[j-x[j]]
procedure setqueen;
  a[x[j]] := b[j+x[j]] := x[j-x[j]] := false
procedure removequeen;
  a[x[j]] := b[j+x[j]] := c[j-x[j]] := true 

此时发现上述实现的x[j]调用次数过多,为了进一步优化,把x[j]用变量i表示,从而有:

1
2
3
4
5
6
7
8
9
10
11
12
13
procedure testsquare;
  safe := a[i] ∧ b[i+j] ∧ c[i-j]]
procedure setqueen;
  a[i] := b[i+j] := c[i-j] := false
procedure removequeen;
  a[i] := b[i÷j] := c[i-j] := true
procedure considerflrstcolumn ;
  begin j:= 1; i:= 0 end
procedure advancepointer; i := i + l
procedure considernextcolumn
  begin x[j] := i; j:=j+l; i := 0 end
Boolean procedure lastsquare;
  lastsquare := i = 8 

通过inline替换程序中的部分指令,其余采用过程调用,从而最终实现如下程序:

1
2
3
4
5
6
7
8
9
j := 1; i := 0;
repeat
  repeat i := i + 1 ; testsquare
  until safe ∨ (i = 8);
  if safe then
  begin setqueen; x[j] := i; j := j + 1; i := 0
  end else regress
until (j > 8) ∨ (j < 1);
if i > 8 then PRINT(x) else FAILURE 

前述过程清晰解释了逐步求精这种非常经典的结构化程序的分析和设计过程,从早期分析确定适用算法,然后利用基本的结构化编程元素描述初步程序,对复杂过程进一步分解,同时考虑额外必要的数据表示和程序运行效率优化,最终使用目标编程语言实现程序。这是一种具有普遍适用意义的编程方法论,也呼应了Wirth的那句名言:程序=算法+数据结构。

结论

50年前的软件危机所揭露的问题成为今天软件工程研究的基石。GOTO语句的争论直至今天,从历史发展看,更多人选择支持Dijkstra的GOTO有害论,许多90年代以后出现的编程语言并没有在应用层面设计GOTO语句。但是,GOTO争论背后有关编程语言灵活和统一的争辩还远未结束。另一方面,结构化编程促成了一套良好的编程方法论,迄今Wirth的逐步求精方法还被采用于程序设计课程,为计算机教育的普及和广泛应用打下了坚实基础。同时,软件设计所要解决的问题也得以提升到更高的复杂度水平。

引用

NATO68, NATO Software Engineering Conference

Bohm66, Flow Diagrams, Turing Machines and Languages with Only Two Formation Rules

EWD68, Go-to statement considered harmful

EWD70, Notes on structured programming

NW71, Program Development by Stepwise Refinement

Comments

软件设计与架构笔记(1)

《软件设计与架构笔记》系列,是笔者对自上世纪60年代末至今在工业界和学术界皆有一定影响的软件设计方法的学习和记录,期望通过历史的时间轴把握相关技术发展的脉络,尝试理解推动了这一领域中概念、方法、原则、模式、实践不断演进的若干基本动机,倚靠巨人的肩膀,但求一条少些人云亦云的实践之路。

自诞生之日起,软件设计就同时在工业界和学术界探索和实践着,然而二者的动机和方法大相径庭。例如计算机科学家Edsger W. Dijkstra,一生就致力于对计算的简洁性和精确性的探索,其工作背后蕴含了严谨的数学美学;而工业界则侧重于使用由计算衍生的自动化方法解决传统生产的问题,根本目的是追求经济利益的最大化。有趣的是,二者的偶然交汇就迸发出这一系列文章的主题——软件设计,而软件架构——作为稍晚出现的buzz word,有时也被本文采用以和行业用语保持一致。

THE Multiprogramming system:早期探索

1965年,Dijkstra在埃因霍温科技大学领导了一支团队在Electrologica(EL) X8上开发多道程序系统,该系统的主要目标是能够平滑地处理持续的用户程序流,并将其作为服务提供给学校。该系统的设计目标是:1.降低短程序运行的周转时间; 2.更经济地使用外设; 3.结合后备存储器的自动控制和中央处理器的经济使用; 4.验证一种经济可行性,即将EL X8用于运行在不依赖容量和计算能力,仅需要通用计算机灵活性的应用程序。

出于多道程序系统的复杂性,实时触发中断的偶然性和不可复现性使系统开发的debug面临挑战。为此,团队决定在系统构建之初就重视对debug能力的设计,从而在具体实现前就能证明系统的逻辑可靠性,并显著降低了实际bug数量。在论文EWD68中,为了提高可测试性,设计者采用层级结构划分整个系统,并以不同的职责区分系统层级:

0级,负责把逻辑可用的进程分配给处理器。为了防止任何进程独占处理器,该层实现了一个实时时钟中断功能。

1级,实现“段控制器”,通过中断与上层的顺序进程保持同步,负责从自动后备存储器中记录数据。

2级,实现“消息解释器”,负责在控制键盘的输入时产生中断,并且联接系统对话的操作员和特定的目标进程。

3级,实现与输入流缓冲和输出流解缓冲相关的顺序进程,通过逻辑通信单元实现对具体外设的抽象,并按照资源限制采用同步方法限制外设运行的数量。

4级,实现独立用户程序。

根据上述层级划分,团队制定了需求规格说明书,并依此实现系统。在验证阶段,在添加下一层级前,需要对前一层进行充分测试,例如针对0级中实现的实时中断和处理器分配,首先设计一个完整的测试状态空间,然后依次进行测试。而当对1级的“段控制器”进行测试时,可以在0级时制定的测试状态空间的基础上,通过引入“请求页”操作,实现状态空间的扩展,只需引入少量新的测试就可以满足当前层的测试需求,直至完成整个系统。

Dijkstra认为,虽然在概念和设计阶段花费了较长时间,但是该过程为系统贡献了良好的设计,避免传统非层级实现可能面临的测试状态空间“爆炸”问题,从而对系统质量提供保证。

虽然缺少定量的研究方法,发布于1968年的THE Multiprogramming System可以说是首次定性地证明了结构在软件设计中的重要作用,并且以系统的可测试性为例进行了深入阐释。

结论

系统的复杂性引出了软件设计问题。Dijkstra把软件开发过程划分成三个阶段:概念、构建和验证,并且由一个基于层级划分的设计案例指出结构因素在软件设计中的重要性。虽然工业界可能面临更多的问题(例如成本、人员、规模、业务复杂程度等),但是概念阶段产出的良好设计,能使验证阶段受益,从而实现整体的系统质量保证(笔者注:某种程度上也起到控制成本的作用),是THE多道程序系统的一项重要结论,也启发后人对软件概念阶段本身和其边际效应的进一步研究。

值得一提的是,按照不同职责划分层级,底层能够对上层隐藏其核心概念和具体实现,例如0级隐藏了处理器操作,1级隐藏了“页存储”机制,2级隐藏了电传打印控制台等。但是“信息隐藏”作为一个基本设计概念被明确提出,则是若干年以后了。

引用

[EWD68] EW Dijkstra, The structure of the ‘THE’-multiprogramming system.

Comments

Working With Legacy Engineering

Looking back to the past five years, my common experience for joining a project is always looking like this:

Learning business knowledge -> Revealing technical context -> Launching a new project and instilling good practices into the team to make better software as usual.

While it’s not always true for software engineers, I rarely got chance to work on the pre-existing codebase, not to mention a whole legacy engineering.

Things have changed since I stepped on a new journey. My very first challenge is taking over a legacy engineering. I call it legacy engineering because all I have to care is far more than only the codebase, although it is absolutely focal to tackle with the legacy code. The rest of this article will introduce the steps and thoughts I’ve taken during this process.

Communication

A good teacher can at least half the work. Everyone who ever worked in the team can teach and benefit, but the following question is how to learn from people more effectively. My answer is no doubt through good communication.

For those who are interested, I would suggest some great learnings like Nonviolent Communication and Crucial Conversations. No follow-up discussion since we’re not talking about psychology here, but remember it’s indeed the first priority soft skill as a professional.

Documentation

For a hand-over stage, documentation in any form is less effective than a good communication. But it’s worthy to keep valuable knowledge and make it more efficient to spread those to broader stakeholders. Again, it always depends on requirement at the moment for determining the scope of what should be documented. My personal inclination is several topics listed as follows:

  1. Team specific knowledge that everyone should be aware of.
  2. Any confusing/complicated part.
  3. The reason for counter-intuitive decisions made by former team members.

It finally leads to three basic documents for me:

  1. Feature list (Rough is good).
  2. Team engineering practice, like workflow definition and release cycle.
  3. Crucial code design.

Of course, the Evernote in hand to prepare for any unknown technologies and skills.

Codebase

Being familiar with codebase is always the top priority for a newbie, but former topics we just discussed would influence the effectiveness of codebase learning. As software engineering today has adopted many common practices like CI/CD and DevOps, those automation practices can help to guide people along with the code or script, rather than outdated document and lengthy tutorial.

Build (Pipeline)

Build automation is the most matured practice in nowadays software engineering. For a newbie, the build script provides panorama and coarse-grained view of system modularity and dependency, it would be even clearer if the team is maintaining an effective CI pipeline.

A undoubted fact is that the more matured for build automation, the more efficiency for a newbie to get first wave things done. Otherwise, it would take weeks to set up a local environment by referring to an far outdated guide and consulting different team members!

Test

It’s not very common to see “effective tests” in the codebase, not to mention the practice of the whole test pyramid or sandwich. Automation test is a proven practice to assure quality, sometimes it is promoted to be the specification of the codebase. However, bad practice for automation test also does harm other than keeping high quality.

For a newbie, in most cases, the tests especially the bad tests cannot help for specification at all, it shows less value than a good naming practice. My personal experience is that the test is not so good for early learning if it’s not well organized and properly implemented. That’s not saying that it doesn’t play vital role to indicate the consistency when any changes happen.

It’s usually easy for everyone to see the value of automation test, while it also makes this practice be one of the most confusing parts in modern software engineering as lack of deep understanding.

Operation

As DevOps is more adopted, it turns to be more necessary for a newbie to understand operation knowledge. Must-have topics include configuration management and specification of cloud platform and core systems.

Design (Structural) and Refactor

The core of codebase learning is about design. A design could be regarded as a collection of decisions along the software development. For a newbie, it’s more important to concentrate on the crucial design points than knowing every corners.

Although it varies for the different roles, as a backend newbie, the top concern comes into the structural design. To learn software structure,a good starting point could be any selected feature from both business and technical perspective, it can help dive into the deeper codebase to understand the structure, then forms into whole picture of modularity design.

A good approach is to elaborate design with some tools, CRC even Naked CRC is agiler, but UML is also very cool and common, it again depends on team background for these tools selection. More important is that whichever tool is used, only the leading design rules need to be considered, rather than drawing a full system map. During this process, design pattern may benefit but also cause restriction.

It’s not exactly same for design of software structure and logical data model, but the latter could also be learned via any cleaner ways (like E-R diagram), not necessary to dig into physical data model too early unless there is significant reason to do that.

Feature Enhancement

I wouldn’t suggest making big change or refactor on the legacy codebase which is just taken over unless the change is too small to cause any visible influence. Before doing any work alone, a newbie still needs to learn from core team members about:

  1. Architecture Roadmap.
  2. Non-functional requirements.
  3. Technical debts and roadmap of resolution.

Despite the following work would be feature enhancement in most cases. I’d take different strategy depends on the potential influence of the change.

Here I’m not going to discuss the case of a tiny change, that wouldn’t be a big deal anyway. But if the change is going to happen on the higher level design rule but not touch leading yet (which usually implies a broader influence across code), I could try to add new features via extensional way (rather than editing as is code and making feature changes directly, methods metioned in Working effectively with legacy code are good guidance), and extra feature toggles may help for QA and CD.

The worst case is the desired change must happen on leading design rule, which means it won’t work with just simple extension, a big change is definitely required. Even in this case, I wouldn’t edit any as is code in an early stage, and purely technical refactor without guidance from business requirement is also not a good option. Instead, I’d create a separate package or namespace as the neighbor of the code which is going to die, and naming it “experiment” or anything makes sense. Using least effort to make experiment code work as well as automation test, without breaking any as is feature or code. Once new feature and refactor direction is confirmed, the experiment code can be transformed into formal and replace dead one.

The benefit of this “experimental duplication” is that I can continue looking at two implementations and making a trade-off when thinking of the design for following refactor until the new feature is completed and confirmed. Later on, refactor and clean-up can be done to completely remove the deprecation.

Conclusion

In this article, we discussed the strategies I would take while working with legacy engineering. Someone calls it reverse engineering to imply the digestion for an incredibly overdue but complicated legacy codebase or engineering. While in most cases it does not have to make upfront “reverse engineering” to reach critical decision for future, nor do a hasty change on codebase to cause risk. We can certainly take actions to delay that decision making, by mitigating waste and respecting the economic model of our beloved software industry.

Comments

测试三明治和雪鸮探索测试

测试金字塔理论被广泛应用于计划和实施敏捷软件开发所倡导的的测试自动化,并且取得了令人瞩目的成就。本文尝试从产品开发的角度出发,结合Kent Beck最近提出的3X模型和近年来迅速发展的自动化测试技术,提出并讨论一种新的测试层级动态平衡观:三明治模型。同时,为了应对端到端测试在实践中面临的种种挑战,设计并实现了一种面向用户旅程的端到端自动化测试框架——雪鸮。实际项目经验表明,雪鸮能够显著提升端到端测试的可维护性,减少不确定性影响,帮助开发人员更快定位和修复问题,对特定时期的产品开发活动更具吸引力。

背景

测试金字塔

按照自动化测试的层级,从下至上依次为单元测试集成测试端到端测试,尽量保持数量较多的低层单元测试,以及相对较少的高层端到端测试,这就是测试金字塔理论。随着敏捷软件开发的日益普及,测试金字塔逐渐为人所知,进而得到广泛应用。Mike CohnMartin Fowler以及Mike Wacker等先后对测试金字塔进行了很好的诠释和发展,其主要观点如下:

  • 测试层级越高,运行效率就越低,进而延缓持续交付的构建-反馈循环。
  • 测试层级越高,开发复杂度就越高,如果团队能力受限,交付进度就会受到影响。
  • 端到端测试更容易遇到测试结果的不确定性问题,按照Martin Fowler的说法,这种结果不确定性的测试毫无意义。
  • 测试层级越低,测试的代码隔离性越强,越能帮助开发人员快速定位和修复问题。

3X模型

2016年起,敏捷和TDD先驱Kent Beck开始在个人facebook主页撰写系列文章,阐述产品开发的三个阶段——Explore、Expand和Extract,以及在不同阶段中产品与工程实践之间的关系问题,即3X模型。近二十年软硬件技术的飞速发展,使得软件开发活动面临敏捷早期从未遇到的市场变革,而根据在facebook工作的经历,Kent Beck把产品开发总结为三个阶段:

  • 探索(Explore),此时的产品开发仍处于非常初期的阶段,仍然需要花费大量时间寻找产品和市场的适配点,收益也是最低的阶段。
  • 扩张(Expand),一旦产品拥有助推器(通常意味着已经找到了市场的适配点),市场需求就会呈现指数级上升,产品本身也需要具备足够的伸缩性以满足这些需求,由此收益也会快速上升。
  • 提取(Extract),当位于该阶段时,公司通常希望最大化产品收益。但此时收益的增幅会小于扩张阶段。

3X

Kent Beck认为,如果以产品是否成功作为衡量依据,那么引入自动化测试在探索阶段的作用就不大,甚至会延缓产品接受市场反馈循环的速度,对产品的最终成功毫无用处,还不如不引入;当位于扩张阶段时,市场一方面要求产品更高的伸缩性,另一方面也开始要求产品保证一致的行为(例如质量需求),那么此时就该引入自动化测试来保证产品的行为一致性;当产品最终处于提取阶段时,任何改动都应以不牺牲现有行为为前提,否则由此引发的损失可能远高于改动带来的收益,此时自动化测试就扮演了非常重要的角色。

测试工具爆炸式增长和综合技能学习曲线陡升

根据SoftwareQATest网站的历史数据,2010年记录的测试工具有440个,共划分为12个大类。这个数字到2017年已经变为560个,共15个大类,且其中有340个在2010年之后才出现。也就是说,平均每年就有50个新的测试工具诞生。

面对测试工具的爆炸式增长,一方面所支持的测试类型更加完善,更加有利于在产品开发过程中保证产品的一致性;另一方面也导致针对多种测试工具组合的综合技能学习曲线不断上升。在实践中,团队也往往对如何定义相关测试的覆盖范围感到不知所措,难以真正发挥测试工具的效用,也很难对产品最终成功作出应有的贡献。

从金字塔到三明治

作为敏捷在特定时期的产物,测试金字塔并不失其合理性,甚至还对自动化测试起到了重要推广作用。但是,随着行业整体技术能力的不断提升,市场需求和竞争日趋激烈,在项目中具体实施测试金字塔时往往遭遇困难,即便借助外力强推,其质量和效果也难以度量。

此外,随着软件设计和开发技术的不断发展,低层单元测试的传统测试技术和落地,因前、后端技术栈的多样化而大相径庭;同时,在经历过覆盖率之争,如何确保单元测试的规范和有效,也成为工程质量管理的一大挑战;高层的端到端测试则基本不受技术栈频繁更替的影响,随着不同载体上driver类技术的不断成熟,其开发复杂度反而逐渐降低。

这里讨论一种新的测试层级分配策略,我们称之为三明治模型 。如下图所示,该模型允许对不同测试层级的占比进行动态调整,说明了倒金字塔形、沙漏形以及金字塔形分配对特定产品开发阶段的积极作用。

Sandwich

产品开发的自动化测试策略

根据3X模型,在探索初期往往选择避开自动化测试。一旦进入扩张期,产品的可伸缩性和行为一致性就成为共同目标,但此时也常会发生大的代码重构甚至重写,如果沿用测试金字塔,无论补充缺失的单元测试,还是只对新模块写单元测试,都既损害了产品的快速伸缩能力,也无法保证面向用户的产品行为一致性。因此,如果在探索后期先引入高层的端到端测试,覆盖主要用户旅程,那么扩张期内所产生的一系列改动都能够受到端到端测试的保障。

需要注意的是,用户旅程在产品即将结束探索期时通常会趋于稳定,在扩张期出现颠覆性变化的概率会逐渐减少,端到端测试的增长率会逐步下降。

除此以外,随着扩张期内不断产生的模块重构和服务化,团队还应增加单元测试和集成测试的占比。其中,单元测试应确保覆盖分支场景(可以在CI中引入基于模块的覆盖率检测);集成测试和某些团队实践的验收测试,则需进一步覆盖集成条件和验收条件(在story sign-off和code review时验收)。

许多新兴的测试技术和工具擅长各自场景下的验收测试,但更重要的仍是识别产品阶段和当前需求,以满足收益最大化。

Sandwich-3x

由此我们认为,随着产品开发的演进,测试层级的分配应参考三明治模型,动态调整层级占比,更加重视运营和市场反馈,致力于真正帮助产品走向成功。

端到端测试的机遇和挑战

与其他测试层级相比,端到端测试技术的发展程度相对滞后。一方面,作为其基础的driver工具要在相应载体成熟一段时间之后才能趋于稳定,web、mobile无不如是。另一方面,端到端测试偏向黑盒测试,更加侧重描述用户交互和业务功能,寻求硬核技术突破的难度较高,于是较少受开发人员青睐。但是,由于端到端测试更接近真实用户,其在特定产品开发活动中的性价比较高,有一定的发展潜力。

然而,当前实践中的端到端测试,普遍存在如下问题:

  • 低可维护性。一般实践并不对测试代码质量作特别要求,而这点在端到端测试就体现得更糟。因为其涉及数据、载体、交互、功能、参照(oracle)等远比单元测试复杂的broad stack。虽然也有Page Object等模式的广泛应用,但仍难以应对快速变化。
  • 低运行效率。如果拿单次端到端测试与单元测试相比,前者的运行效率肯定更低。因此只一味增加端到端测试肯定会损害构建-反馈循环,进而影响持续交付。
  • 高不确定性。同样因为broad stack的问题,端到端测试有更高的几率产生不确定测试,表现为测试结果呈随机性成功/失败,进一步降低运行效率,使得真正的问题很容易被掩盖,团队也逐渐丧失对端到端测试的信心。
  • 难以定位问题根因。端到端测试结果很难触及代码级别的错误,这就需要额外人工恢复测试环境并尝试进行问题重现。其中所涉及的数据重建、用户交互等会耗费可观的成本。

方法

为了解决传统端到端测试遇到的种种挑战,本文设计了一种面向用户旅程的端到端自动化测试框架——雪鸮(snowy_owl),通过用户旅程优先、数据分离、业务复用和状态持久化等方法,显著提高了端到端测试的可维护性,降低不确定性的影响,并且能够帮助团队成员快速定位问题。

用户旅程驱动

端到端测试应尽量贴近用户,从用户旅程出发能保证这一点。在雪鸮中,用户旅程使用被称作play books的若干yaml格式的文件进行组织,例如下列目录结构:

play_books/
  core_journey.yml
  external_integration.yml
  online_payment.yml

其中每个play book由若干plots所组成,plot用于表示用户旅程中的“情节”单位,其基本特征如下:

  • 单一plot可以作为端到端测试独立运行,例如发送一条tweet的情节:
1
2
3
4
5
6
7
8
9
10
11
12
SnowyOwl::Plots.write 'send a plain text tweet' do
  visit '/login'
  page.fill_in 'username', with: 'username'
  page.fill_in 'password', with: 'password'
  page.find('a', text: 'Sign In').click
  # verify already login?
  page.find('a', text: 'Home').click
  # verify already on home page?
  page.fill_in 'textarea', with: 'Hello World'
  page.find('a', text: 'Send').click
  # verify already sent?
end
  • 单一plot应是紧密关联的一组用户交互,并且具备体现一个较小业务价值的测试参照。
  • plot可以被play book引用任意次从而组成用户旅程,play book同时定义了所引用plots之间的顺序关系,基本语法如下所示:
1
2
3
4
---
- plot_name: send a plain text tweet
  digest: 2aae6c35c94fcfb415dbe95f408b9ce91ee846ed
  parent: d6b0d82cea4269b51572b8fab43adcee9fc3cf9a

其中plot_name表示情节的标题,digest和parent分别表示当前情节引用在整个端到端测试过程中的唯一标识和前序情节标识,初期开发人员可以通过各个情节的引用顺序定义用户旅程,大多数情况下digest和parent将由系统自动生成并维护。

整个play books集合将是一个以plots为基础组成的森林结构,而端到端测试的执行顺序则是针对其中每棵树进行深度遍历。

通用业务复用

由于plot本身必须是一个独立可运行的端到端测试,那么plots之间通常会共享一部分交互操作,例如用户登录。雪鸮允许把高度可复用的交互代码进行二次抽取,称作determination:

1
2
3
4
5
6
7
8
SnowyOwl::Determinations.determine('user login') do |name, user_profile|
  # return if already login
  visit '/login'
  page.fill_in 'username', with: user_profile[:username]
  page.fill_in 'password', with: user_profile[:password]
  page.find('a', text: 'Sign In').click
  # verify already login?
end

这样,plot的代码就可以简化成:

1
2
3
4
5
6
7
8
SnowyOwl::Plots.write 'send a plain text tweet' do
  determine_user_login({username: 'username', password: 'password'})
  page.find('a', text: 'Home').click
  # verify already on home page?
  page.fill_in 'textarea', with: 'Hello World'
  page.find('a', text: 'Send').click
  # verify already sent?
end

这里应注意Determination和Page Object的区别。看似使用Page Object可以达到相同的目的,但是后者与Page这一概念强绑定。而Determination更加侧重描述业务本身,更符合对用户旅程的描述,因此比Page Object在plot中更具适用性。当然,在描述更低层的组件交互时,Page Object仍然是最佳选择。

测试数据分离

合理的数据设计对描绘用户旅程非常重要,雪鸮对测试逻辑和数据进行了进一步分离。例如用户基本数据(profile),同样是使用yaml文件进行表示:

data/
  tweets/
    plain_text.yml
  users/
    plain_user.yml

那么在plot的实现中,就可以使用同名对象方法替代字面值:

1
2
3
4
5
6
7
8
SnowyOwl::Plots.write 'send a plain text tweet' do
  determine_user_login({username: plain_user.username, password: plain_user.password})
  page.find('a', text: 'Home').click
  # verify already on home page?
  page.fill_in 'textarea', with: plain_text.value
  page.find('a', text: 'Send').click
  # verify already sent?
end

情节状态持久化

雪鸮的另一个重要功能是情节状态的持久化和场景复原。为了启用情节状态持久化,开发人员需要自己实现一个持久化脚本,例如对当前数据库进行dump,并按照雪鸮提供的持久化接口把dump文件存储至指定位置。

当端到端测试运行每进入一个新的情节之前,系统会自动执行持久化脚本。也就是说,雪鸮支持保存每个情节的前置运行状态。

当端到端测试需要从特定的情节重新开始运行时,雪鸮同样会提供一个恢复接口,通过用户自定义的数据恢复脚本把指定位置的dump文件恢复至当前系统。

该功能有两处消费场景:

  • 由于broad stack的问题,端到端测试不确定性的技术因素一般较为复杂。实际经验表明,测试的随机失败率越低,就越难以定位和修复问题,而通过不断改进测试代码的方式消除这种不确定性的成本较高,效果也不好。但是,可以尽量消除不确定性带来的影响。例如,不确定测试导致的测试失败,通常会导致额外人工验证时间,完全可以选择让系统自动重试失败的测试。另一方面,重试会造成测试运行效率降低,特别是针对端到端测试。当一轮端到端测试结束后,雪鸮只会自动重试失败的情节测试,同时利用该情节对应的数据dump文件保证场景一致性,这就减少了重试整个端到端测试带来的运行效率下降问题。
  • 当团队成员发现端到端测试失败,通常需要在本地复现该问题。而借助测试dump文件,可以直接运行指定plot测试,从而避免额外的人工设置数据和交互操作,加快问题定位和解决。

实践

雪鸮在笔者所在的项目有超过6个月的应用时间。该项目在产品开发方面长期陷入困境,例如过程中同时兼具了3X每个阶段的特点,不仅缺少清晰的产品主线,还背负了接棒遗留系统的包袱。这种状况对工程质量管理提出了更大挑战。

项目采用雪鸮对已有端到端测试进行了重构,生成了一个核心用户旅程和三个涉及外部系统集成的重要用户旅程,包含24个plots,9个determinations,使端到端测试实现了长期稳定运行。在本地相同软硬件环境下,不确定性导致的随机失败从原有10%降低至1%以内,部署至云环境并采用headless模式后,连续15天测试失败记录为零,运行效率的损失可以忽略不计。同时,当用户旅程产生新分支时,可以引入新的情节测试节点,并且根据业务需求将其加入现有play book树,从而实现端到端测试的快速维护。

持续集成与常态化运行

项目完整的端到端测试的平均运行时间保持在19分钟左右,为了不影响现有持续集成节奏,CI每30分钟自动更新代码并运行端到端测试,结果在dashboard同步显示,一旦发生测试失败,第一优先级查找失败原因并尝试在本地复现和修复。

常态化运行端到端测试的另一个好处是,能够以低成本的方式实现24小时监控系统各个组件的功能正确性,有助于更早发现问题:一次,产品即将上线的支付功能发生异常,查看CI记录发现端到端测试在晚上9:15左右出现了首次告警。通过及时沟通,确认是海外团队在当时擅自改动了支付网关的一个配置,造成服务不可用的问题,并迅速得以解决。

结论与展望

Kent Beck的3X模型,提出了从不同产品开发阶段看待工程实践的新视角。而敏捷一贯推崇的TDD等实践,更多体现在个人技术专长(Expertise)方面,与产品是否成功并无必然联系。然而,程序员的专业主义(Professionalism)的确同时涵盖了技术专长和产品成功两个方面,二者相辅相成。因此,如何通过平衡众多因素并最终提高整体专业性,这才是软件工程面临的经典问题。本文给出的测试三明治模型,目的就是帮助思考产品开发过程中测试层级间的平衡问题。

为了应对现有端到端测试面临的挑战,本文设计并实现了一种新的面向用户旅程的端到端测试框架,通过职责隔离、业务复用和状态持久化等手段,构建了易于维护且更加有效的端到端测试。同时,基于上述方法构建的测试代码,更易于和自动化测试的其他研究领域相结合,在诸如测试数据构建、用例生成、随机测试和测试参照增强等方向有进一步的应用潜力。

Comments

微情境混合现实

从前有一池清潭,在附近的居民眼里它和普通池塘并没有什么区别。某天一位富绅来到池边,搬起一块巨石砸进了池中心,泛起的沉渣顿时把池水变成了浊潭。富绅见人就说,突然变浑浊的潭水里其实是出土了宝藏,价值万贯。消息传开,好奇的人们不远万里赶到潭边,有的确实想发笔横财,有的只是为了观光。富绅自己则在潭边围起一段栅栏,向来访者销售门票,不日累积万贯。不久,别处的清潭都变浊了。

近年火爆的VR/AR在大众眼中是“黑科技”,其实早在50年前(The Sword of Damocles,1965)就被人玩剩了,他的创作者是图形学界祖师之一Ivan Sutherland。但可能是发现自己已经走得太远(实际上当时现代图形学基础还没有完全确立),于是萨大爷就带着他一票学生和下属回去后方铺路,后者们先后发明了Smalltalk编程语言、Gouraud着色算法、图形反走样算法,创办创意设计巨头Adobe以及高性能计算鼻祖SGI,甚至还联合乔帮主创办了Pixar(现任Pixar/Disney动画总裁的Edwin Catmull也是图形学巨擘,发明了灰常实用的Catmull-Rom样条函数,直到现在还被用于关键帧插值和渲染头发丝)。

有意思的是,VR/AR在理论上的研究早已从感知器官上升到哲学范畴……多年来在实践上却仍然停留在科技馆和特殊行业应用领域。最终,geek们期待的科技大跃进并没有真正到来,尘埃褪去,呈现的却是更加亲民的头显(HMD)和各种奇思妙想的控制器(Controller),无不体现着以人为本的时代理念。而如果你能尝试SteamVR上的应用、特别是体验下Proto奖的获奖作品时,你会欣慰地发现,原来真正有梦想、有创造力的人还在不断涌现,你就会从各种捞钱、炫富的纸金现实中暂时解脱出来了。

如果避开硬科幻,从终极关怀的角度审视VR/AR,我们所处时代的主题原来并非“黑科技”,而是“低门槛”和“大众化”。这里,本文提出一种特定场景下的混合现实产品思路,最后以一个真实的开源Demo为例,希望能为许多有想法的人打开一扇窗,鼓励更多人能参与到这一领域中来。

C22-Full

为什么是特定场景?

门槛是首要考虑因素。以Daydream为例,在如今诸多VR/AR平台中,Daydream的优势是非常易见的:它太容易接近大众了。对于其它竞品平台,则更多聚焦在高端和行业应用上,不利于小团队做快速原型开发和内容创新(就是说只适合憋大招)。

而Daydream的劣势目前也很明显:3-DOF的控制器无法捕捉用户的全身动作,反馈缺失导致了沉浸感不足(Vive的这一优势在市面上目前还无人匹敌);另外,移动设备受限的性能、参差不齐的规格也导致潜在的用户体验问题,这恐怕也是Google率先携Pixel发布的初衷。

因此在设计VR/AR产品时,需要首先考虑平台对应用场景的限制。

微情境(Micro-Situation)

理论上说,VR/AR技术几乎可以被用于任何情境。基于现实考量,在“微情境”中实现的性价比则更高。

例如非厨师职业学习做饭,传统上可能是一对一的学徒体系;后来发展成书籍报刊、电视传播。前者属于文字模式更多得靠个人天赋,后者的声音+图像模式就更容易被大众所接受。

但有人可能会质疑为什么不真的去尝试做一顿饭?其实这相当于混淆了学习和实践两种活动,“学做饭”和“做饭”本身不是一回事(当然你也可以说在实践中学习,但你也不能证明你是在完全未经过学习的情况下做了一顿饭,况且如果真的是这样,那么满足各位食客的可能性恐怕微乎其微)。

再来看“虚拟学做饭”这一活动的价值,最大的劣势就是你无法从外观之外获得任何的有效反馈,包括直接吃掉自己的学习成果。而优势是节省钱、食材、时间,切身经历每一道工序,这是现有的学习形式都不具备的。

从这一情境本身来说,用户只需要一间整洁的厨房,摆放整齐的食材和调味品,再加上面前一座灶台即可。而交互内容仅需要双手参与。那么即将进入寻常百姓家的Daydream平台完全满足需求,于是皆大欢喜。

其实这里需要考虑的重点在“微”上,什么样的情境才算是“微情境”呢?我们不尝试进一步去做更多的文字游戏,对于“微情境”的发现要更多依赖对情境本身、以及对VR/AR技术的理解和运用,这也是不可避免的过程。

这里再延伸到一个稍“大”一些的微情境:产品说明书

相信大家对各式各样的产品说明书都不陌生,但从现实的生活经验来看,在大多数情况下,要么产品自身完全做到了自说明,要么很难快速(或者根本无法)从说明书中找到自己需要的答案。多年来,尽管平面设计不断进步,多媒体新招层出不穷,也很难断言一个真正有效的用户-产品沟通方式,而VR/AR的普及将为此带来全新的维度。

例如你网购了一套书柜,然而需要自己动手安装,原本自信满满的你花了很长时间,终于无奈地瘫坐在沉甸甸的零件旁,边满头大汗地研究起貌似手绘的安装说明……在当前情境中,家具的零件都可以被虚拟化,用户也不需要真的在场景中走来走去,而是直接上手了解零件功能和安装步骤即可——可以想见,当平台越廉价,“微情境”中的VR/AR应用所带来的附加价值就会越高(例如直接嵌入产品官网Web,并借助移动设备进行VR/AR显示,具体实现请参考文末Demo)。

或许,人们会在不久之后的春晚上体验到VR/AR抢红包之类的全民娱乐呢?

为什么是混合现实(MR)?

在这波VR行情之前,就有许多AR应用运行在智能手机上,此外还包括不久前的google glass,却始终没有杀手级应用出现(酝酿多年也才蹦出一只Pokemon Go…)。原因甚多,其中市场太小、远离大众是一个非常重要的因素。难道VR就没有市场的疑问?否则巨头们为何集体押注在后者身上?由此可猜想投资VR的回报率明显要更高一筹,却万不能沉迷于只做游客。

值得一提的是Hololens展示了当前无与伦比的MR技术,无奈距离大众太过遥远,注定只会面向专业领域。实际上基于现有的VR设备要想直接实现接近Hololens的MR并不困难,毕竟只需要一个透明显示和空间位置标定,如果要做进一步的场景感知会稍复杂(但也已经不是壁垒),不过由此发展的成本肯定比一步到位要低许多。

(这里不得不说句题外话,平台大战可以造就繁荣的假象,但也可能扰乱了内容创造者们的思想,从而延缓技术的最终普及,这种风险在当前看来是不可避免的)

然而如果单纯比拼想象力,MR更胜VR/AR,虽然前者只是模糊了后者之间的界限。在接受程度上,人们总会倾向于那些更加抽象、简洁且表现力更强的形式,MR无疑将会最终胜出。

真正的困难之处在于,MR不仅仅在于两幅人眼画面的结合,也意味着更加复杂的应用场景,以及更具挑战的交互模式,而不是一个看起来更高级的HMD。

来自22世纪的程序员(C22)

C22是一个基于传统PC+Cardboard的微情境应用。它展示了未来的程序员,能够摆脱固定显示器的束缚,使用HMD、键鼠就可以编写代码,并且在自身环境周围展示各种信息的技术。

C22-Right

如上图所示,PC中的应用信息被独立传送至HMD中创建的每个虚拟屏幕上,程序员可以在三维场景中浏览各种信息。

除了信息浏览外,C22允许程序员在盲打的情况下使用键盘、鼠标在对应的应用上进行操作,并且能够实时同步。

由于当前C22无法显示在一块透明质的HMD上(类似Hololens),因此也无法真正让普通用户感受到在现实中操作应用。此外由于分辨率的限制,C22目前能实现的用户体验也比较受限

C22-Left

为了快速实现产品原型,C22全部使用JavaScript编写,并且采用了许多尚未广泛兼容的技术(详见项目github主页),由此带来的好处是核心部分搭建只用了两个晚上的时间,后期为了提高体验可能会进一步扩展当前技术栈。

同样由于兼容问题,C22目前只支持Android设备,并计划在2017年第一季度支持Daydream,届时,用户可以在周围自由设置信息布局。并且由于WebVR将得到浏览器原生支持,C22的渲染能力也将得到显著提升。

2015年举行的Proto颁奖大会上,年近八旬的萨大爷获虚拟现实“创始人奖”,大神激励台下一帮天才后辈说:“内容意味着一切,摄像机(指技术)并不创造内容,只有最伟大的创造者才得以使用技术把现实带给人们”。

闻者深受共鸣。

mr
Comments

Metaprogramming Ruby: Core Concepts - Extensions

As we already know, ruby provides several ways to extend our classes and objects including but not limit to OO inheritance, Module mix-in, Dynamic Methods, Monkey patching / Refinement, Eval family, Ghost methods, Callable objects, Singleton classes / methods, etc. But there are still some important tips deserve to be mentioned, before we moving forward.

1. Include vs Extend

Firstly let’s see an example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
module Foo
  def self.foo
    p 'Hello world'
  end

  def say_hi
    p 'Hello world'
  end
end

class Bar
  include Foo
end

Bar.foo #NoMethodError: undefined method `foo' for Bar:Class
Bar.new.say_hi #Hello world

It seems that module mix-in can only involve instance methods but not others like class methods. But how to do similar thing at class level? You must hear about extend:

1
2
3
4
5
6
7
8
9
10
11
module Foo
  def foo
    p 'Hello world'
  end
end

class Bar
  extend Foo
end

Bar.foo #Hello world

Extend is used in object level, so we are sure it can also be used for any class in ruby. But in previous article, we know that class method is actually saved as a singleton method of the original class, also instance method of its singleton class. So that should also happen on include:

1
2
3
4
5
6
7
8
9
10
11
12
13
module Foo
  def foo
    p 'Hello world'
  end
end

class Bar
  class << self
    include Foo
  end
end

Bar.foo #Hello world

Thus we can regard extend as a kind of special usage on include through above examples.

2. Method wrappers

Method wrapper means wrapping an existing method inside a new method, which is very useful when you want to make extension without changing the source, like code in standard library or other cases.

There are several ways to implement method wrappers in ruby, and they are all in composite form of primitives which’ve already been introduced in previous articles. We’ll go through below.

Around Alias

Module#alias_method (also the key word ‘alias’) is used to give another name to ruby methods. It will involve more accessibility if an usual method could have different domain names(e.g. size, length and count). Also more flexibilities if you want, like code below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
class Foo
  def say_hi
    p 'Hello'
  end
end

class Foo
  alias_method :say_hello, :say_hi

  def say_hi
    say_hello
    p 'World'
  end
end

foo = Foo.new
foo.say_hi #Hello\nWorld\n

This is just a kind of wrapper using open class, alias_method, and method redefinition.

Refinement

We talked about refinement and suggested using refine instead of monkey patch. Actually refinement is even more powerful than that.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
class Foo
  def say_hi
    p 'Hello'
  end
end

module FooRefinement
  refine Foo do
    def say_hi
      super
      p 'World'
    end
  end
end

using FooRefinement

Foo.new.say_hi #Hello\nWorld\n

Only thing you need to notice is that the key word ‘using’ may not work well with your IRB environment, which means you couldn’t get result in mind for some versions of ruby if you run those code in IRB. See more information here.

The benefit of refinement wrapper is controllable scope of wrapper unlike around alias which affects globally. However, accessibility to original method is also lower than alias way.

Prepending

Module#prepend is the simplest way to implement method wrapper without scope configurability like refinement. But much more clear than other two.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
class Foo
  def say_hi
    p 'Hello'
  end
end

module FooPrepending
  def say_hi
    super
    p 'World'
  end
end

Foo.class_eval do
  prepend FooPrepending
end

Foo.new.say_hi #Hello\nWorld\n

3. Class Macros

Ruby objects have no attributes - May this won’t surprise or confuse you too much. Indeed we’ve hear about instance variables or class variables, but you can not access them directly from outside. That means getter or writer can not be avoided:

1
2
3
4
5
6
7
8
9
10
11
12
13
class Foo
  def bar=(value)
    @bar = value
  end

  def bar
    @bar
  end
end

foo = Foo.new
foo.bar = 'Hello'
foo.bar #Hello

It’s just not ruby style! Ruby provides series accessors for class definition using metaprogramming api, like attr_accessor, attr_reader and attr_writer, they are quite intuitive to use:

1
2
3
4
5
6
7
class Foo
  attr_accessor :bar
end

foo = Foo.new
foo.bar = 'Hello'
foo.bar #Hello

attr_* come from Module as private instance methods, thus they all can be used in module or class definitions.

Comments

Metaprogramming Ruby: Core Concepts - Eval Family II

In previous article several basic scope controlling tools are introduced, like block, proc, lambda, and method object. All of them should work well when you organize them in your code reasonably. There are more flexible way to mix code and bindings in ruby - eval, which may not be used broadly than others, so that considerable traps about it are still unclear for us.

3. instance_eval / instance_exec

BasicObject#instance_eval can be used almost everywhere in ruby, you can use it like this:

1
2
3
foo = Object.new
foo.instance_eval { @bar = 'hello' }
foo.instance_variable_get('@bar') #hello

In the block of instance_eval, the scope changes to the instance foo, thus any operations inner it should bind to the instance, except closure variables.

BasicObject#instance_exec has similar feature to eval one, but with arguments support. This benefit shows as below:

1
2
3
4
5
6
7
8
9
class Foo
  def initialize
    @bar = 'hello'
  end
end
output = 'world'
foo = Foo.new
foo.instance_exec(output){|output| output = "#{@bar} #{output}"}
output #world

4. class_eval / class_exec

Module#class_eval works for evaluating a block in the context of an existing class. It seems to be similar to instance_eval which will have scope changes to instance self, but also changes current class (excluding singleton class).

class_eval can be used to open a existing class, without using keyword ‘class’. That means it has no scope changing issue compared with keyword way. Actually it even does not need to know the constant name of the target, while keyword way indeed needs.

Also Module#class_exec plays same role like instance_exec for instance_eval.

Think about the thing what we’ve discussed several weeks ago, in ruby all items are instance of class, including classes no matter internal or customized. Which means instance_eval and class_eval both can work for classes in many cases, but indeed have different self meaning.

You may also notice that there is also Module#module_eval / Module#module_exec methods, but they are just alias for class_eval / class_exec without any other changes.

5. Kernel#eval

Unlike instance_eval/class_eval, Kernel#eval only accept a string of ruby code as its argument, run the code and give the result. Even it’s so powerful, using string of code is not a good choice if you have other ways. The most issue for string of code is security.

Suppose you have eval expression in your code, which means others can evaluate any code using such method, including get/set operations. Ruby defines safe levels, which actually limits the evaluation of code from outside. By default, ruby will mark potentially unsafe objects which many come from external sources. And user could config a global variable $SAFE, it’s range from 0 to 3(0 by default) with more strict.

1
2
3
$SAFE = 1
eval "p 'hello #{gets()}'"
world #SecurityError: Insecure operation - eval

By default, eval can accept almost any code from outside. However, it will not be permitted for code from I/O device(tainted object) if $SAFE is none zero. Below gives more details about $SAFE:

$SAFE Constraints
0 No checking of the use of externally supplied (tainted) data is performed. This is Ruby’s default mode.
>= 1 Ruby disallows the use of tainted data by potentially dangerous operations.
>= 2 Ruby prohibits the loading of program files from globally writable locations.
>= 3 All newly created objects are considered tainted.
Comments

Metaprogramming Ruby: Core Concepts - Eval Family I

In ruby, block is the cornerstone of metaprogramming with a surprisingly powerful scope controlling capability. Based on this definition, much brilliant interfaces are introduced, and help to implement many valuable features, like callable objects, eval and its huge family.

1. blocks

Block represents a snippet of code, it permits to be called immediately or lately depends on different cases. But compared with some similar concepts in functional programming languages, like Lisp, block still has more limits, and more readability respectively.

In commonly, block can be used like this:

1
2
3
4
5
6
7
8
9
10
11
def show_hello
  yield 'hello' if block_given?
end

show_hello { |say| puts say }

show_hello do |say|
  file = File.open('/tmp/output.txt', 'w')
  file.write say
  file.close
end

‘yield’ is a ruby keyword used to call blocks sent to the current method. block_given? is an instance method from Kernel to probe whether there is a block for current method.

One of the most useful aspect of block is closure, it captures bindings where it’s defined, and avoid impact of scope changing by connecting to those bindings, like flat scopes, and shared scopes.

Frankly speaking, block is actually not responsible for running code, but only representation (Except yield, which indeed means running code immediately). There are more powerful tools to help enhance block usage.

2. callable objects

Block is just like a package of code, and you need to use yield to execute it if you like. However, there are more ways to package code in ruby, including Proc and lambda.

Proc

We already know that block is not an object in ruby which is really quite a few, but Proc is basically a block turned object, can be seen as a consistent form of block, and you do not have to use yield to run it immediately, it will be running later as you want (Deferred Evaluation). you can define a Proc like this:

1
2
3
4
inc = Proc.new {|x| x + 1}
inc.call(1) #2
dec = proc {|x| x - 1}
inc.call(2) #1

lambda

Except for Proc, lambda can also be used for transferring blocks, but with simpler and a little different way:

1
2
3
4
inc = lambda {|x| x + 1}
inc.call(1) #2
dec = ->(x) {x - 1}
dec.call(2) #1

& operator

Once you have defined a block for a method, there is a way to convert it to a Proc inner method by using &. For example:

1
2
3
4
5
6
7
8
9
10
11
12
def bar
  yield 'ruby'
end

def foo(&operation)
  bar(&operation)
  operation.call('world')
end

foo {|x| puts "hello #{x}"}
# hello ruby
# hello world

& can be seen like transferring a block into Proc, but you need to remove & if you want to use it as a Proc.

Comparison between procs and lambdas

You may notice that lambda is also a Proc, but you can still use Proc#lambda? to see the actual type of the target. Besides there are two important differences: keyword ‘return’ and arguments checking.

a. return

As plain blocks, the program will return out of scope where block defined if return statement is set inner block, so does Proc (which may mean that the return may cause exception when the block is called in nonreturnable scope, like top-level). While for lambda, the return only runs out of block, not even farther.

b. arity

Arity means the number of arguments. In real case, lambda has less tolerance than Proc, which means that lambda requires correct number of arity, which is no need for procs.

Method objects

All methods are objects of Method. You can use Kernel#method or Kernel#singleton_method to get the object of method, then use Method#call to run the method. Also you may use Method#to_proc and define_method to convert method and proc to each other. Eventhough the method still has scope of it’s defined.

Unbound Methods

Method object can also be unbound from original scopes by using Method#unbind, or Module#instance_method. Generated UnboundMethod object can not be called directly, but you can still call it after bind it to another scope by using UnboundMethod#bind. The only notice is that UnboundMethod object can only be bound to same class or sub if it’s from class, and no such limitation when it’s from module.

Comments

Metaprogramming Ruby: Core Concepts - Open Classes & Refinements

Opening Classes

Opening classes brings much flexibility for ruby. It permits you to modify existed classes or modules permanently, without having to implement whole world in case you just need to give a simple patch for current tool. For example:

1
2
3
4
5
class String
  def to_alphanumeric
    gsub(/[^\w\s]/, '')
  end
end

to_alphanumeric is new interface to return only alpha and numeric part of a string. However, opening class may not work well, especially it’s commonly dangerous to opening a frequently used target which would cause unexpected error in code. Due to this reason someone calls opening classes monkey patch and just leaves far from this feature.

While whatever, monkey patch continues conciseness in ruby, sometimes you are able to save your life gracefully with such powerful tool, except the trap behind it.

Refinements

From ruby 2.0, there is a more advanced form of monkey patch which is called refinements. Refinements are providing a way to extend classes only under specific scope, but not including modules. For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
class C
  def foo
    puts "C#foo"
  end
end

module M
  refine C do
    def foo
      puts "C#foo in M"
    end
  end
end

Module#refine creates an anonymous module that contains the changes or refinements to the class (C in the example). self in the refine block is this anonymous module similar to Module#module_eval.

Scope

We use using to activate refinements defined in any places, but you may only activate refinements at top-level, not inside any class, module or method scope. You may activate refinements in a string passed to Kernel#eval that is evaluated at top-level. Refinements are active until the end of the file or the end of the eval string, respectively.

Refinements are lexical in scope. When control is transferred outside the scope the refinement is deactivated. This means that when you require or load a file or call a method that is defined outside the current scope the refinement will be deactivated.

If a method is defined in a scope where a refinement is active the refinement will be active when the method is called.

When defining multiple refinements in the same module, inside a refine block all refinements from the same module are active when a refined method is called.

You may also activate refinements in a class or module definition, in which case the refinements are activated from the point where using is called to the end of the class or module definition.

Method lookup under refinement

When looking up a method for an instance of class C Ruby checks:

1. If refinements are active for C, in the reverse order they were activated:

The prepended modules from the refinement for C

The refinement for C

The included modules from the refinement for C

2. The prepended modules of C
3. C
4. The included modules of C

If no method was found at any point this repeats with the superclass of C.

Note that methods in a subclass have priority over refinements in a superclass. For example, if the method / is defined in a refinement for Integer 1 / 2 invokes the original Fixnum#/ because Fixnum is a subclass of Integer and is searched before the refinements for the superclass Integer.

If a method foo is defined on Integer in a refinement, 1.foo invokes that method since foo does not exist on Fixnum.

Super

When super is invoked method lookup checks:

The included modules of the current class. Note that the current class may be a refinement.
If the current class is a refinement, the method lookup proceeds as in the Method Lookup section above.
If the current class has a direct superclass, the method proceeds as in the Method Lookup section above using the superclass.

Note that super in a method of a refinement invokes the method in the refined class even if there is another refinement which has been activated in the same context.

Refinements and module inclusion

Refinements are inherited by module inclusion. That is, using activates all refinements in the ancestors of the specified module. Refinements in a descendant have priority over refinements in an ancestor.

After all, refinement is still an experiment feature in ruby 2.x series, there is a detailed specification in official website, which should cover most aspect of it.

Comments