Meta


Augmenting and Unifying
Language Paradigms




Wade Holst

Problem

There exist language-level barriers to developer productivity.

  • Language limitations:

    • missing traits: first-class objects, keyword params, ...
    • problematic syntax: redundant method signatures, separate tests, ...
    • slow evolution: closures in C++, class syntax in Javascript, ...

  • Expertise does not cross language boundaries:

    • syntax: is different.
    • semantics: can be inconsistent.
    • libraries: have differing interfaces.
    • tools: have differing interfaces.

Solution

Meta allows developers to be more productive.

  • Language limitations:

    • missing traits: Meta adds missing traits to languages.
    • problematic syntax: Meta reduces syntactic burden.
    • slow evolution: Meta can evolve rapidly.

  • Expertise does not cross language boundaries:

    • syntax: is similar.
    • semantics: is consistent.
    • libraries: have identical interfaces.
    • tools: have identical interfaces.

Meta is...

  • A means of augmenting existing languages.
    • Meta<C++> augments C++.
    • Meta<Haskell> augments Haskell.
    • Meta<TeX> augments TeX.
    • Meta<X> augments arbitrary formal language X.
  • A means of unifying languages within a paradigm.
    • Meta(Oopl) unifies object-oriented programming languages.
    • Meta(Fun) unifies functional programming languages.
    • Meta(Doc) unifies document markup languages.
    • Meta(P) unifies languages belonging to paradigm P.
  • A means of increasing productivity and eudaimonia.
    • Developers produce more and write less.
    • Expertise in Meta<X> transfers to expertise in Meta<Y>.
  • Implemented in Meta.

Meta<Python>

  • An augmented version of Python:
    • Meta<Python> can do everything Python can, and more.

  • Syntax
    • Defines new syntax for concepts above the level of statements
    • Uses Python for statement level syntax (e.g. method scopes).

  • Benefits
    • A more consistent, intuitive, expressive way to develop in Python.
    • Unifies Python with other OO languages.

Meta<C++>

  • An augmented version of C++:
    • Meta<C++> can do everything C++ can, and more.

  • Syntax
    • Defines new syntax for concepts above the level of statements
    • Uses C++ for statement level syntax (e.g. method scopes).

  • Benefits
    • A more consistent, intuitive, expressive way to develop in C++.
    • Unifies C++ with other OO languages.

Meta<X>

  • An augmented version of X:
    • Meta<X> can do everything X can, and more.

  • Syntax
    • Defines new syntax for concepts above the level of statements
    • Uses X for statement level syntax (e.g. method scopes).

  • Benefits
    • A more consistent, intuitive, expressive way to develop in X.
    • Unifies X with other OO languages.

Example: Meta<Python>

Program 1: person.meta

Compiling person.meta into Python:

% metac compile -L oopl -b python person.meta
% metac compile -b py person.meta
% metac compile person.meta
% metac person.meta
% metac person
Program 1b: person.meta compiled into Python
meta: 1 file     18 lines    313 bytes
base: 4 files   115 lines   2265 bytes

Example: Meta<C++>

Program 2a: person.metacc

Compiling person.metacc into C++:

% metac compile -L oopl -b C++ person.metacc
% metac compile -b cc person.metacc
% metac compile person.metacc
% metac person.metacc
% metac person
Program 2b: person.metacc as C++
meta: 1 file     18 lines    315 bytes
base: 8 files   159 lines   3397 bytes





Meta-Languages


Meta(Oopl)



Wade Holst

Meta-languages

A Meta-language augments/unifies languages within a paradigm.

  • e.g. Meta(Oopl), Meta(Doc), Meta(Viz), Meta(Fun), etc.

Each Meta-language consists of:

  • Base Languages: the set of languages to augment/unify.
    • For each base language X, an augmented language Meta<X> is defined.
  • Contract: the set of traits defining the paradigm.
    • Meta<X> is guaranteed to support all traits in this contract even if X does not.
  • Schema: the set of legal syntactic Meta constructs
    • Provides syntactic access to all desired traits of the paradigm.
  • Compiler: program that converts Meta<X> source into base language X.
    • Provides all missing traits guaranteed by the contract.
  • Language Hierarchy: a collection of progressively more expressive languages
    • The most expressive introduces statement-level Meta constructs.

Meta-languages

A Meta-language augments/unifies languages within a paradigm.

  • e.g. Meta(Oopl), Meta(Doc), Meta(Viz), Meta(Fun), etc.

Each Meta-language consists of:

  • Base Languages: the set of languages to augment/unify.
    • For each base language X, an augmented language Meta<X> is defined.
  • Contract: the set of traits defining the paradigm.
    • Meta<X> is guaranteed to support all traits in this contract even if X does not.
  • Schema: the set of legal syntactic Meta constructs
    • Provides syntactic access to all desired traits of the paradigm.
  • Compiler: program that converts Meta<X> source into base language X.
    • Provides all missing traits guaranteed by the contract.
  • Language Hierarchy: a collection of progressively more expressive languages
    • The most expressive introduces statement-level Meta constructs.

The Meta(Oopl) Base Languages

Existing

Pythonhas the most support and largest code base (250k lines)
C++has basic support, being actively developed
Javascripthas basic support, being actively developed

Planned

Javawhen the first three are robust, will be easy to add.
C#"
Objective-C"
Ruby"
Perl"
Eiffel"
Swift"

Possible

Gomay not be sufficiently object-oriented to justify inclusion
Rustmay not be sufficiently object-oriented to justify inclusion
CLOSmore exploration needed (multi-method dispatch!)
Delphimore exploration needed
Dartmore exploration needed
Squeakmore exploration needed (depends on UI requirements)

Meta-languages

A Meta-language augments/unifies languages within a paradigm.

  • e.g. Meta(Oopl), Meta(Doc), Meta(Viz), Meta(Fun), etc.

Each Meta-language consists of:

  • Base Languages: the set of languages to augment/unify.
    • For each base language X, an augmented language Meta<X> is defined.
  • Contract: the set of traits defining the paradigm.
    • Meta<X> is guaranteed to support all traits in this contract even if X does not.
  • Schema: the set of legal syntactic Meta constructs
    • Provides syntactic access to all desired traits of the paradigm.
  • Compiler: program that converts Meta<X> source into base language X.
    • Provides all missing traits guaranteed by the contract.
  • Language Hierarchy: a collection of progressively more expressive languages
    • The most expressive introduces statement-level Meta constructs.

The Meta(Oopl) Contract

All of these exist in augmented language Meta<X> whether or not they exist in base language X.

Uplifted Traits (good ideas from baselangs):

  • C++: type system, bit fields, operator overloading
  • Java: synchronized, throws
  • Python: keyword params, indentation, repl
  • Javascript: variant types
  • Go: goroutines
  • C#: override
  • Perl: regexp syntax
  • Smalltalk: metaclasses, categories
  • Eiffel: pre/post conditions
  • Beta: sub-extend semantics
  • Tcl: upvar

Language Interoperability

  • The Meta(Oopl) Syntax
    • incremental multi-language code in same file
  • The Meta(Oopl) Language Hierarchy
    • incremental use of Meta-level statements
  • The Meta(Oopl) Type System
    • as concise as Java, as expressive as C++
  • The Meta(Oopl) Library
    • classes defined in all base languages

Augmentations

  • Auto-generated testing infrastructure
  • Auto-generated reflexivity/meta-programming
  • Auto-generated build system
  • Auto-generated object serialization
  • Auto-generated UML visualization
  • Auto-generated native code interface
  • Auto-generated accessors and service methods
  • Source code customization/canonicalization
  • Threads, fibers, parallelization
  • Multi-method dispatch
  • Aspect-oriented programming support
  • Profiling, Test coverage
  • Method extensions: Implementation variants, ...
  • Field extensions: bit, optional, transient
  • Augmented inheritance paradigms, visibility
  • Explicit data dependencies
  • Units of measurement
  • Literal string variable interpolation

Meta-languages

A Meta-language augments/unifies languages within a paradigm.

  • e.g. Meta(Oopl), Meta(Doc), Meta(Viz), Meta(Fun), etc.

Each Meta-language consists of:

  • Base Languages: the set of languages to augment/unify.
    • For each base language X, an augmented language Meta<X> is defined.
  • Contract: the set of traits defining the paradigm.
    • Meta<X> is guaranteed to support all traits in this contract even if X does not.
  • Schema: the set of legal syntactic Meta constructs
    • Provides syntactic access to all desired traits of the paradigm.
  • Compiler: program that converts Meta<X> source into base language X.
    • Provides all missing traits guaranteed by the contract.
  • Language Hierarchy: a collection of progressively more expressive languages
    • The most expressive introduces statement-level Meta constructs.

The Meta(Oopl) Schema

namespacea space within which other constructs can be defined.
  classa collection of state and behavior representing an abstract data type.
    methodan named piece of code that operates on an instance of some class.
      testxa child within a tests: block, for defining multiple tests per method.
    fielda single element of state within a class.
      accessora method that provides get/set/ref/etc. access to a field.
    lifecyclea concise encapsulation of functionality for initializing/finalizing a class instance.
    categorya syntactic means of grouping related methods/fields within a class together.
    assoca formal mechanism for identfying external code needed by a specific class.
    resourcea formal mechanism for identifying external resources needed by a program.
    commanda mechanism for defining a named entry point into a program.
      flaga mechanism for allowing users to pass information into a program.
  behaviordefines a multi-method (supports dynamic dispatch on more than one receiver).
    receiverthe code associated with a receiver tuple within a behavior.
nativea mechanism for dumping raw base-language code into a compilation stream.

Statement-level constructs in Meta(Oopl)*:
vara variable declaration.
blocka statement-level generalizatioan of a lexical scope.
loopa statement-level generalization of all looping structures.
switcha statement-level generalization of all branching structures.
more to come...

Meta-languages

A Meta-language augments/unifies languages within a paradigm.

  • e.g. Meta(Oopl), Meta(Doc), Meta(Viz), Meta(Fun), etc.

Each Meta-language consists of:

  • Base Languages: the set of languages to augment/unify.
    • For each base language X, an augmented language Meta<X> is defined.
  • Contract: the set of traits defining the paradigm.
    • Meta<X> is guaranteed to support all traits in this contract even if X does not.
  • Schema: the set of legal syntactic Meta constructs
    • Provides syntactic access to all desired traits of the paradigm.
  • Compiler: program that converts Meta<X> source into base language X.
    • Provides all missing traits guaranteed by the contract.
  • Language Hierarchy: a collection of progressively more expressive languages
    • The most expressive introduces statement-level Meta constructs.

The Meta(Oopl) Compiler

The Meta(Oopl) compiler is (currently) implemented in Meta<Python>.

  • 10 meta files produce 43 python files.
  • 46.3k meta lines produce 66.4k python lines.
  • 1.72M meta bytes produce 2.14M python bytes.
  • 10 namespaces, 115 classes, 828 methods, 329 fields, ...
  • Currently supports Python, C++, and Javascript, with Java next, then many more.
Group/File      Files files   %F   Lines   lines   %L    Bytes    bytes   %B
----------------------------------------------------------------------------
kernel              7    29 75.9   39090   55506 29.6  1467320  1815278 19.2
  cli                     4         1646    2523  1.5    59502    76654  1.3
  fs                      4         1316    2460  1.9    40190    62340  1.6
  logs                    4          171     451  2.6     4852    10215  2.1
  parser                  5        32879   46318  1.4  1253936  1555431  1.2
  root                    4          344     466  1.4    13598    15780  1.2
  shell                   4          428     586  1.4    16297    19195  1.2
  test                    4         2306    2702  1.2    78945    75663  1.0
lib                 3    14 78.6    7232   10872 33.5   252683   323209 21.8
  markdown                5         3979    5962  1.5   140749   180483  1.3
  shell                   4         1920    2643  1.4    65549    77611  1.2
  units                   5         1333    2267  1.7    46385    65115  1.4
TOTAL              10    43 76.7   46322   66378 30.2  1720003  2138487 19.6
  

Meta-languages

A Meta-language augments/unifies languages within a paradigm.

  • e.g. Meta(Oopl), Meta(Doc), Meta(Viz), Meta(Fun), etc.

Each Meta-language consists of:

  • Base Languages: the set of languages to augment/unify.
    • For each base language X, an augmented language Meta<X> is defined.
  • Contract: the set of traits defining the paradigm.
    • Meta<X> is guaranteed to support all traits in this contract even if X does not.
  • Schema: the set of legal syntactic Meta constructs
    • Provides syntactic access to all desired traits of the paradigm.
  • Compiler: program that converts Meta<X> source into base language X.
    • Provides all missing traits guaranteed by the contract.
  • Language Hierarchy: a collection of progressively more expressive languages
    • The most expressive introduces statement-level Meta constructs.

The Meta(Oopl) Language Hierarchy


  • Tier 0: X
    • missing traits, problem syntax, slow evolution.
    • syntax/semantics/libraries/tools differ between languages.

  • Tier 1: Meta<X>
    • guaranteed traits, uniform expressive syntax.
    • syntax/semantics/libraries/tools consistent.
    • every programmer more productive.
    • gives expressive power.
    • removes tedium.
    • eudaimonia.

  • Tier 2: Meta<X|Y>
    • syntactic language interoperability.
    • easy incremental conversion of X to Y.
    • makes native code optimizations easy.
    • addresses legacy code problem.

  • Tier 3: Meta(Oopl)*
    • Makes cross-language libraries possible.
    • One impl. compiles to every baselang.





Meta(Oopl) Showcase



A Sampling of Guranteed Traits

Syntactic Conciseness

Program 3a: person3.metapy
Program 3b: person3.metapy in Python

Auto-generated Unit Testing Framework

Program 4a: person4.metapy
Program 4b: person4.metapy in Python

Language Unification (Meta<Python|C++>)

Program 6: person5.meta in Meta<Python|C++>

UML Class Diagrams









 % metac uml shapes.meta

 % metac uml -mf shapes.meta
Program 6: shapes.meta

UML Class Diagrams...

Behaviors and Multi-Method Dispatch

  • Suppose we have a Shape hierarchy.
    • Each class implements an interface
    • Method signatures repeated in each class
    • Can we do better?

  • behavior: reverse class and method
    • A method signature
    • Documentation
    • Multiple per-class implementations.
  • Supports multi-method dispatch
    • Call-sites can have multiple receivers.
    • Dispatch based on dynamic types of all receivers.
    • Meta generates code to optimize dispatch.
    • cf. CLOS, Cecil, Dylan, Julia

Source Code Configuration/Canonicalization

  • Configuration
    • Every attribute-key/feature-value can customized:
      • one or more aliases can be added
      • syntax hightlighting font and color can be specified.
      • customization can occur at any lexical scope.
    • Use Cases:
      • improved individual readability
      • multi-lingual programs

  • Canonicalization
    • A canonical representation of every Meta(Oopl) program exists.
    • Meta can convert between personal and canonical
      • and thus between one personalized syntax and another.
    • Use Cases:
      • implicit style guide compliance.
      • sharing code with others.
      • storing the canonical version in a repository.

Meta(Oopl) Library

A library implemented in every base language.

  • Identical interface.

  • Implementations optimized to use per-language features.

  • Supported by:
    • Meta Type System
    • Shared Meta Syntax
    • Meta(Oopl)*

  • Includes:
    • String classes
    • Array classes
    • Dictonary classes
    • Meta types
    • Units of Measurement
    • Streams
    • Dates
    • Regexps
    • ...

Meta(Oopl) Tools Interfaces


Common Actions
helpshow all available commands
compileparse/expand/import/translate/compile
testinvoke a unified test harness
runexecute a program
debuginvoke a base-language debugger
replinvoke base-language repeat-execute-print loop

Less Common Actions
profileperform memory/time profiling
coverageperform test coverage analysis
canonicalcanonicalize .meta files (auto-lint)
htmlproduce .html version of .meta files
umlgenerate uml class diagrams from .meta source.
shellinvoke an interactive shell (to explore a program, etc.)
reversereverse compile base-language code into .meta source
configview/modify metac configuration values

Esoteric Actions
setupperform bootstrapping and configuration of metac
emacsgenerate a customized emacs major-mode for a Meta-language schema
vimgenerate a customized vim plug-in for a Meta-Language schema
schemagenerate an .html description of a Meta schema
cleanremove code from repository
indexprint out index of namespaces/classes/methods/fields
mirrorproduce a self-contained copy of code








Summary

Problem

There exist language-level barriers to developer productivity.

  • Language limitations:

    • missing traits: first-class objects, keyword params, ...
    • problematic syntax: redundant method signatures, separate tests, ...
    • slow evolution: closures in C++, class syntax in Javascript, ...

  • Expertise does not cross language boundaries:

    • syntax: is different.
    • semantics: can be inconsistent.
    • libraries: have differing interfaces.
    • tools: have differing interfaces.

Solution

Meta allows developers to be more productive.

  • Language limitations:

    • missing traits: Meta adds missing traits to languages.
    • problematic syntax: Meta reduces syntactic burden.
    • slow evolution: Meta can evolve rapidly.

  • Expertise does not cross language boundaries:

    • syntax: is similar.
    • semantics: is consistent.
    • libraries: have identical interfaces.
    • tools: have identical interfaces.

Meta Hypotheses

  1. There exist language-level barriers to developer productivity.
    • missing traits, problem syntax, slow evolution, limited knowledge transfer

  2. Meta allows developers to be more productive.
    • guaranteed traits, related code adjacent, rapid evolution, common syntax/semantics/interface/tools.

  3. Libraries and tools are more important than syntax.
    • Unification of tools/libraries interfaces is more beneficial than a new syntax is detrimental.

  4. Statement-level syntax is ubiquitous, high-level syntax is used less.
    • Introducing new high-level syntax is not as intrusive as new statement-level syntax.

  5. Meta<Y> from Meta<X> is much easier than Y from X.
    • Incremental, adjacent, easy migration.
    • Learning curve significantly reduced.

  6. An efficient language-neutral Meta Library is possible and desirable.
    • Expertise comes with knowing the library. One library to rule them all!

  7. Someone proficient in X can become proficient in Meta<X> in 10 hours.
    • Syntax is very consistent, every program is fully self documenting.

Code Auto-Generated By Meta(Oopl)

  • for every base language
    • bazel WORKSPACE
    • for every namespace:
      • a test namespace
      • a BUILD file
      • for every class
        • a meta class
        • a test class
        • a mock class
        • a protobuf message
        • numerous build rule in various BUILD files
        • for every method
          • test method(s)
        • for every field
          • private state
          • a getter
          • a setter
          • many other accessors depending on type, etc.
        • implicit methods:
          • printMeta(): human readable reprsentation of an instance.
          • serializeMeta(): serialization to protobuf.
          • bytesMeta(): calculates memory used by an instance.

Existing Meta Code Bases

Some stats on existing Meta code bases:
  • Source: 111 Meta<Python> files, 218k lines, 8.1M bytes.
  • Produces: 504 Python files, 318k lines, 10.1M bytes
  • Contains: 115 namespaces, 858 classes, 4014 methods, 1514 fields
Group/File    Files files   %F   Lines   lines   %L    Bytes    bytes   %B
--------------------------------------------------------------------------
metac            10    43 76.7   46921   65729 28.6  1741779  2123189 18.0
dnd5             18    72 75.0   32584   44276 26.4  1170255  1369594 14.6
games             5    25 80.0    9866   18751 47.4   340122   511672 33.5
wae-life          5    21 76.2    3826    6251 38.8   162578   216517 24.9
wae-story        11    62 82.3   29457   41312 28.7  1140434  1381423 17.4
wae-metaxy        2     8 75.0     257     822 68.7     9294    21221 56.2
wmh              60   273 78.0   95187  141248 32.6  3537118  4469323 20.9
TOTAL           111   504 78.0  218098  318389 31.5  8101580 10092939 19.7

Research Directions

  • Apples-to-apples comparison of memory and performance across languages

  • The Meta(Oopl) Library
    • Is a general-purpose cross-language library possible?
    • Can statement level Meta constructs be defined that are syntactic desirable?

  • Is Go sufficiently object-oriented to be included in Meta(Oopl)?

  • Mocks/Stubs/Fakes/Dummies/Spies

  • Reflection

  • Multi-Methods

  • Interactions between Meta-Languages (Meta-Meta-Language?)

Things To Do

  • Breadth: Add more base languages to Meta(Oopl)
    • C++ (basics done, more work on typing and unittests needed)
    • Javascript (basiscs done, need to write more code to test)
    • Java (not started)
    • Perl (not started)

  • Depth: Add more constructs
    • statement level!
    • bit fields, optional fields, etc.

  • Meta(Oopl)*
    • Build out the classes across all languages.
    • Implement Meta in Meta(Oopl)*

  • More Meta-Languages

Meta-Languages

  • Meta(Oopl): augments and unifies object-oriented programming languages.
    • Java, C++, Python, C#, Javascript, Swift, Objective-C, Perl, Delphi, Ruby, Kotlin, ...

  • Meta(Fun): augments and unifies functional programming languages.
    • Lisp, R, Scheme, Haskell, Elm, Miranda, Dylan, ML, Rust, Scala, ...

  • Meta(Proc): augments and unifies procedural programming languages.
    • C, Go, Pascal, Fortran, COBOL, BASIC, ...

  • Meta(Logic): augments and unifies logic-based programming languages.
    • Prolog, Mercury, Fril, ...

  • Meta(Doc): augments and unifies markup languages.
    • HTML, XHTML, Markdown, MathML, SVG, LaTeX, Wiki markup, X3D, ...

  • Meta(Viz): augments and unifies data visualization languages.
    • gnuplot, dygraph, matplotlib, graphviz, D3, ...

  • Meta(Family): augments and unifies genealogy languages.
    • GEDCOM, Gramps, ELF, ...







Questions?

Meta Syntax: Constructs and Attributes

In Meta, there is exactly one self-contained syntactic entity, a construct.

<metafile>   ::- <construct>+
<construct>  ::- <attribute>+ <terminator>?
<attribute>  ::- <attribute_key>? <attribute_value>
<terminator> ::- ';' | 'end' (<kind> <id>?)? ';'

Attributes come in three flavors:

  • feature attributes
    • appear before the primary attribute (zero or more).
    • values are limited to a pre-defined fixed set of tokens.
    • keys are always optional.
    • all feature attributes are optional and have default values.
  • primary attribute
    • the key is the name of a construct, and is always present.
    • the value is a unique identifier for the construct (some constructs allow optional primary values).
    • there is exactly one occurrence of the primary attribute in a construct.
  • secondary attributes
    • appear after the primary attribute (zero or more).
    • keys are usually required but sometimes optional.
    • there is an attribute value type associated with each value.
    • values can vary from a simple identifier to a complex block.

An Example Meta Program

Program 1: person.meta, in Meta<Python>
  • A namespace demo.
  • A class Person.
  • A field name.
  • A lifecycle subsumes constructor(s), destructor, copy/move semantics, etc. within a single syntactic construct.

  • Simple Blocks:
    • lexical scope containing arbitrary text
    • Meta performs no validation on these lines
    • ex: comment: in line 1, #: in lines 7,11, scope: in line 14.

  • Complex Blocks:
    • lexical scope containing nested constructs.
    • Meta fully validates syntax of complex blocks
    • ex: scope: in lines 3 and 5.

The Meta Compiler: metac

Program 1: person.meta, in Meta<Python>
  • When using metac, one specifies:
    • the Meta-Language to use (--metalang, -L)
    • the target base language (--baselang, -b)
    • the source files
    • the action to perform (compile, test, etc.)

  • Compilation Phases:
    • parse: parse Meta source into constructs
    • expand: implement missing traits
    • import: parse dependent code
    • translate: generate base code
    • compile: compile base code

  • Compiler Actions: (subset)
    • compile: convert Meta code to base code.
    • test: invoke test harness.
    • reverse: convert base code to Meta.
    • uml: generate UML class diagram
    • html: general HTML from Meta source.
    • shell: interactive shell to explore Meta src.

The Meta(Oopl) Contract: Uplifting

Base languages often have useful traits missing in other languages.

  • C++:
    • type system:
      • value/ref/ptr types
      • const types
      • move semantics
      • templates
    • const methods
    • operator overloading
    • bit fields
  • Java:
    • synchronized methods
    • throws in signature
  • Python:
    • indentation
    • keyword params
    • interactive shell
  • Javascript:
    • Variant types
  • C#:
    • override keyword
  • Perl:
    • regexp support
  • Smalltalk:
    • metaclasses
    • class vars and methods
    • categories
  • Eiffel:
    • pre and post conditions
  • Beta:
    • sub-extend semantics
    • monads
  • Tcl:
    • upvar
    • wish

The Meta(Oopl) Contract: Augmentations

  • Namespaces:
    • can be arbitrarily nested.
  • Classes:
    • nested and inner classes are supported
    • auto-generated meta-class, test-class, mock-class, protobuf description, etc.
    • auto-generated methods (summary, printing, serializing, size, etc.).
  • Methods:
    • can appear within namespaces, within classes, and within methods
    • support for closures, instance/class/static methods, multi-methods, operator overloading, etc.
  • Fields:
    • accessors: auto-generated getters, setters, mutators, incrementers, adders, etc.
    • object serialization: auto-generated methods for writing to protobuf and human-readable streams
    • uml diagrams: auto-generated class diagrams from the source code.
    • optional fields: no memory if not present (at the expense of more memory and slower access for such fields).
    • bit fields: C++ bit fields are provided in all base languages.
    • support for instance/class/static fields

The Meta(Oopl) Contract: Augmentations...

  • Behaviors:
    • methods containing class tuples instead of classes containing methods
    • provides support for multi-method dispatch
  • Associations:
    • formalized means of capturing all code dependencies
    • are first-class objects
  • Resources:
    • formalized means of capturing all data dependencies (external files, etc.)
    • are first-class objects
  • Command Line Interface:
    • formalized entry points supporting nested commands, flags and args.
    • are first-class objects
  • Aspects:
    • The ability to write code that inspects/inserts/deletes/modifies code elsewhere in the program.
    • Meta's consistent syntax makes AOP both easy and efficient to implement.
  • Augmented Inheritance Paradigms:
    • replace vs extend vs sub-extension

The Meta(Oopl) Contract: Augmentations...

  • Auto-generated Testing Infrastructure:
    • The Meta(Oopl) Library defines metax.test.TestCase
    • Every user-defined class produces an auto-generated test class
    • Every user-defined method produces one or more test methods.
    • The code being tested is adjacent to the tests for that code.
  • Auto-generated Reflexivity and Meta-programming Support:
    • Each user-defined class has an associated auto-generated meta-class.
    • Provides support for class variables and class methods.
    • Provides multiple levels of reflection and meta-programming.
  • Auto-generated Build System:
    • Bazel workspaces and BUILD files are auto-generated.
    • Provides all the benefits of Bazel without any need to know what it does.
  • Auto-generated Object Serialization:
    • metaxStr: auto-generated one-line string representation of class instance
    • metaxPrint: auto-generated human-readable printing of instances.
    • metaxWrite: auto-generated machine-readable protobuf serialization
    • metaxSize: auto-generated computation of memory usage of instance.
  • Auto-generated UML visualization:
    • UML class diagrams can be automatically generated at arbitrary levels of detail.
  • Native Code Support:
    • Meta programs can provide implementations of specific methods in other languages.
    • The compiler adds all the glue needed to make the native code work.

The Meta(Oopl) Contract: Augmentations...

  • Source Code Customization and Canonicalization:
    • Users can rename all attribute keys and feature values, thus controlling the syntax of a program.
    • Programs can be canonicalized and personalized, allowing conversion between very different syntatic appearances.
  • Profiling:
    • Suppoort for analyzing cpu/memory usage per method.
    • Uniform means of invoking profiling, and standrdize output format.
  • Test Coverage:
    • Suppoort for performing test coverage analysis.
    • Uniform means of invoking coverage, and standrdize output format.
  • Implementation Variants:
    • Support for providing multiple implementations of a method.
    • Tools for assessing which implementation performs better in varied conditions.
  • Literal String Variable Interpolation:
    • programs can use special syntax (even within base code) for formatting strings based on variable interpolation.
          val = "Person ? has ? and weight ?"
        produces (in Python)
          val = 'Person %s has height=%1.5f and weight %5.2f' % (
            self._name, self.height(), weight)
    
  • Meta-Base Line Mapping:
    • Meta maintains a mapping from every line in all generated base-language files to corresponding line in Meta source file.
    • Allows Meta to report line numbers relative to Meta source code.

The Meta(Oopl) Contract: Language Interoperability

  • The Meta(Oopl) Language Hierarchy:

    • Programs can be written in Meta<Xi> then easily migrated to Meta<Xj>
    • Programs can embed native code
    • Programs can be implemented in a base-language-independent syntax

  • The Meta(Oopl) Syntax:

    • Meta<Xi> and Meta<Xj> share identical syntax above statement level
    • Users can learn new language syntax much more quickly.

  • The Meta(Oopl) Type System:

    • Augmented primitive types: int, uint, float, double, char, bool, units, etc.
    • Native types: types that have one-to-one mapping to baselang equivalents.
    • Class types:
    • Variant types: a type that is one of an explicit set of types.
    • Weak references
    • By-value, by-reference, and by-pointer.
    • Mutable and immutable
    • Template/generic types

  • The Meta(Oopl) Library:

    • A collection of classes defined in every base language with exactly the same interface and semantics.
    • The difference between a novice and a master programmer in a specific language is about depth of knowledge of libraries.

Auto-generated Unit Testing Framework

Program 4: person4.meta
  • Every user class has an auto-generated test class.
  • Every user method has one or more auto-generated test stubs.
  • The test: attribute in a method is added to the associated test stub.
  • The setup: attribute of lifecycle constructs is added as a setUp method in the test class.

Invoking the test harness:

 # Test all methods in a single class using bazel
 % metac test -b py person4.meta demo.Person

 # Test a single method using bazel
 % metac test person4.meta demo.Person.bodyMassIndex

 # Test all methods in a namespace without bazel
 % metac -r person4 demo
 Running raw tests (without bazel)
  % cd <meta_repo>/oopl/python
  % python demo_test/__init__.py | special_filtering
 =======================================================
 Person.Person                            ...     365 us   
 Person.bodyMassIndex                     ...      28 us   
 demo                :   2 tests in 0.002 seconds: OK    

Language Unification: Code Migration

Given a Meta<Python> program, let's convert it to Meta<C++>.

  • We know that the programs differ only at statement level, so:
    1. Make a copy of the Meta<Python> program.
    2. For every method scope:, convert the Python code to C++.
  • Issues
    • the Meta<C++> program won't compile until it is fully converted.
    • maintaining separate files for the two implementations poses a burden.
  • Can we do better?
Program 5a: person.metapy in Meta<Python>
Program 5b: person.metacc in Meta<C++>

Language Unification...

What if Meta syntax allowed us to limit attributes to base languages?
  • In Meta, an attribute key can have an optional base language qualifier.
    • Base languages are identified by two-letter suffix in the interests of conciseness.
    • Although especially useful for method scopes, qualifiers can be used on any attribute.

  • When Meta compiles a program:
    • It knows the destination base language.
    • It can ignore any attribute that specifies a non-matching base language.
    • Compare line 14 of Program 5c with line 14 of Program 5d. Note the scope<py>: vs scope<cc>:

  • Now we can merge implementations ...
Program 5c: person.meta in Meta<Python>
Program 5d: person.meta in Meta<C++>

Meta-Level Benefits Of Meta

  • Adding a feature to Meta(Oopl) benefits every OO user (compared with OO users of a specific language).
  • Adding a feature to Meta benefits every user of any formal language
  • Meta(Oopl) can compile to different versions of a base language
    • python2 or python3
    • C++11 or C++14 or C++17
    • ES2015, ES2016, ES2017, ES2018
    • Meta(Oopl)* can generate appropriate code in various versions.
  • ...

Details We No Longer Need To Know

  • outside code
    • how to compile a file
    • how to test a namespace/class/method
    • how to profile a program for memory/cpu usage
    • how to perform test coverage analysis
    • how to invoke a repeat-execute-print loop
    • how to generate uml class diagrams

  • inside code
    • how to define a class/method/field/constructor/etc.
    • how to serialize or print an object
    • how to define a setup/teardown method

Command Line Interface

Program 5: person6.meta

Program 5 showcases Meta's CLI support:

  • top-level command defines person binary
    • defines --name flag
    • defines subcommand bmi
      • defines two positional args
      • invokes the auto-generated metaxPrint.

  • Invoking the person binary:
        % person --name="Cal Jones" bmi 87 1.87
        Person 10ff0b4d0:
          name   = Name 10ff0b490:
            given   = Bob
            surname = Smith
            middle  = None
          height = 1.87
          weight = 87
        Bob has BMI 24.88
    

Meta Syntax: High-level grammar rules

<metafile>        ::- <construct>+
<construct>       ::- <attributes> <terminator>

<attributes>      ::- <feature_attr>* <primary_attr> <secondary_attr>*
<feature_attr>    ::- <attr_key>? <feature_value>
<primary_attr>    ::- <construct_name> <construct_id>?
<secondary_attr>  ::- <attr_key>? <secondary_value>

<terminator>      ::- ';' | 'end' (<construct_name> <construct_id>?)? ';'

<attr_key>        ::- <id>
<construct_name>  ::- <id>
<construct_id>    ::- <word>

<secondary_value> ::- <id> | <xid> | <word> | <lit_expr> | <type> |
                      <callsite> | <simple_block> | <complex_block>
<simple_block>    ::- ':' <newline> (<indentation> <line>)*
<complex_block>   ::- ':' <newline> (<indentation> <construct>)*

<lit_expr>        ::- <literal> | <expr>
<literal>         ::- <number> <unit>? | <string> | <list> | <map>
<expr>            ::- <op> <id> | <id> <op> <id> | '(' <expr> ')'

Meta Syntax: Terminal grammar rules

<id>              ::- [a-zA-Z_][a-zA-Z0-9_]*
<word>            ::- [^\s]+
<string>          ::- '"' [^"]+ '"' | "'" [^']+ "'"
<number>          ::- <int> | <float>
<list>            ::- '[' (<expr> (',' <expr)*)? ']'
<map>             ::- '[' (<string> ':' <expr> ',')*
                  ::-     (<string> ':' <expr> [','])? ']'

<callsite>        ::- '@' <id> ('.' | '!') <id>
                      ( '(' ( <lit_expr> (',' <lit_expr>)* )? ')' )?

<type>            ::- <type_qualifiers>? <type_name> <type_params>?
<type_qualifiers> ::- @#? | &?(#?[usw]?\*)*#?
<type_name>       ::- <id>
<type_params>     ::- '<' <type> ( ',' <type> )* '>'

Psychology

  • Most programmers learn just enough of a language to get by1
  • The more difficult it is to do something, the less likely it is to be used.
    • If one needs to do a lot of work before a method can be tested, it is less likely to be tested.
    • If one needs to learn how to set up profiling infrastructure, it is less likely that profiling will be used.
    • If one doesn't know the debugger, the programmer may just use print statements (wasting time in finding bugs)
  • point 3
  • point 4

Title

  • point 1
  • point 2
    • sub a
    • sub b
    • sub c
  • point 3
  • point 4

Title

  • point 1
  • point 2
    • sub a
    • sub b
    • sub c
  • point 3
  • point 4

Terminology

  • paradigm: a means of classifying languages based on their traits.
    • Ex: object-oriented, procedural, functional, markup, data visualization, ...
  • trait: a capability/feature/aspect/ability of a language.
    • Ex: dynamic dispatch, meta-classes, strong typing, inheritance, keyword parameters.
  • base language: a pre-existing language augmented by Meta.
    • C++, Go, Haskell, TeX, Gnuplot, ...
  • meta-language: an augmented language "above" a base language.
    • Meta extends C++, Meta extends Go, Meta extends TeX, ...
  • Metadigm: augments and unifies languages within a paradigm.
    • Meta(Oopl) for object-oriented languages, Meta(Doc) for document markup, ...
  • augment: to make better by adding to.
    • Meta is a superset of C++, Meta is a superset of TeX, ...
  • unify: to make more uniform.
    • If X and Y have the same paradigm, Meta and Meta share most syntax.

Solution...

  • Augment: Increase Expressivity (do more while coding less).

    • Remove the repetitive tedious parts of writing code
      • Remove redundancy and automate boilerplate.
      • Meta involves 3-6 times less code than X

    • Keep related code together
      • unit tests adjacent to code being tested
      • native code embedded within parent language
      • alternate implementations kept adjacent

    • Remove psychological barriers to doing things
      • Unit testing is easy
      • Native code is easy.
      • Migrating to a new language is easy.
      • Exploring alternative implementations is easy.
      • ... many many more ...

  • Unify: Support Cross Language Interactions and Interoperability

    • Knowledge in Meta transfers to Meta.
    • Rapid prototyping and rapid migration to production.
    • Address the Legacy Code Problem.
    • Support for native code

Meta In Meta


The Meta implementation contains:
  • 10 namespaces, 155 classes, 819 methods, 325 fields, 26 flags, ...
Group/File      nmsp  cls life meth  fld  res  beh  rcv  cmd flag  ntv
----------------------------------------------------------------------
cli                1    9    5   32   16         1    3              2
fs                 1    9    8   59   23    1                        1
logs               1    2    2    9    5                              
parser             1   96   40  492  184   12   40  128   36   23    2
root               1    3    3    2    6                             1
shell              1    1    1   11    1                              
test               1    3    5   72   22    2                        2
markdown           1   17   24   69   37    5              1    3     
shell              1    9   11   48   19                             1
units              1    6    8   25   12                   1          

TOTAL             10  155  107  819  325   20   41  131   38   26    9

Meta Code Bases

Group/File      Files files   %F   Lines   lines   %L    Bytes    bytes   %B
----------------------------------------------------------------------------
metac              10    43 76.7   46322   64960 28.7  1720003  2098057 18.0
dnd5               18    72 75.0   32584   44143 26.2  1170255  1368003 14.5
games               5    25 80.0    9866   18586 46.9   340122   509642 33.3
wae-life            5    21 76.2    3826    6184 38.1   162578   215668 24.6
wae-story          11    62 82.3   29457   41093 28.3  1140434  1378728 17.3
wae-metaxy          2     8 75.0     256     797 67.9     9216    20853 55.8
wmh                60   273 78.0   95187  140688 32.3  3537118  4462028 20.7
TOTAL             111   504 78.0  217498  316451 31.3  8079726 10052979 19.6

Group/File      nmsp  cls life meth  fld  res  beh  rcv  cmd flag  ntv
----------------------------------------------------------------------
metac             10  155  107  819  325   20   41  131   38   26    9
dnd5              18   75  106  629  169    0    0    0    0    0   39
games              5   78   57  269  177    0    7   15    8    4    7
wae-life           5   31   19   49    9    1    0    0    4    2    6
wae-story         15  109   98  451  117    5    0    0   15   15    8
wae-metaxy         2    8    7    8    0    0    0    0    0    0    1
wmh               60  402  439 1780  713   36    4   16  147  131   79
TOTAL            115  858  833 4005 1510   62   52  162  212  178  149

The Meta(Oopl) Language Hierarchy

  • Meta extends X (new syntax above statement level)
  • Every attribute can be qualified with a base language.
  • Meta and Meta can be expressed in Meta.
    • Meta
    • Meta
    • Meta.

  • Can implement code in Meta
    • i.e. implementation in every base language supported by Meta(Oopl).
    • Unfortunately, the code quickly gets very messy.
    • Can we do better?

  • This leads to Meta
    • i.e. implementation in every base language supported by Meta(Oopl).
    • Unfortunately, the code quickly gets very messy.
    • Can we do better?

  • What if Meta introduced its own statement-level syntax?
    • A program written in such a language:
      • could be compiled to any base language
      • wouldn't contain any base language code.
    • This is Meta(Oopl)*, the most general language in the Meta(Oopl) Language Hierarchy.

  • Meta(Oopl)* has specialized use-cases.
    • Useful for creating consistent cross-language libraries.
    • Meta statement syntax is not as concise as base languages.