October 6, 2016

Pedagogical Learning with the Inkling Programming Language | Bonsai

The Root of Bonsai: Inkling and its Compiler

Pedagogical Programming

In the early days of forming the company, our founders Mark and Keen had a unified vision: make the implementation of artificial intelligence accessible to every developer. Current state of the art tools for machine learning and artificial intelligence are extremely complicated and difficult to understand. Eventually, they identified the core question: “What exactly is it that developers want to do?” The answer was that they wanted to teach a computer, and the only way to do that is to create tools for teaching a computer. The first thing they would need a high-level programming language to express the core concepts of teaching, an academic discipline known as pedagogy.

Pedagogy is the discipline that deals with the theory and practice of education; it thus concerns the study and practice of how best to teach.              Wikipedia

I interviewed our language specification and compiler expert, Megan Adams, and got an in depth view of the process of creating a new language to harness the power of AI. With two decades of experience in language and compiler design at HP Labs, Amaret, and others, she's an unequivocal expert on the subject.

Why a new language?

The fundamental ideas that the language needed to express, and its fundamental objectives, were so different from any other programming language, that a new language was required... Like SQL, it reflects very closely the technological characteristics of a system. In SQL, it's the database - for Inkling, it's providing support for artificial intelligence and machine learning.

The language is designed to solve problems around machine training and teaching... Inkling has the concept of a concept, and no other language has that. Inking has the concept of a curriculum, and no other language has that.

Introducing a language and getting buy-in is risky. However it's a very strong value proposition for us because we're replacing, with forty lines of code, the highly complicated software required for TensorFlow (or other  machine learning platform). We insulate users from that complexity in a way that parallels how high level languages insulate users from the complexities of machine code. Inkling provides an enormous simplification, effectively an order of magnitude. Also we provide advanced machine learning capabilities, such as multi-concept, at a high level of abstraction. That is a powerful feature that will show great value over time, I believe.

What challenges are involved in designing a language?

The truth is, coming up with a language specification is very hard. It's not hard in the way that quantum mechanics is hard. There are many different elements that you need to balance. The expressiveness and the power of the language, its utility, your [...] understanding of the technology as well as the benefits of the technology which you're trying to make accessible...

My favorite professor used to say this one thing over and over, and that was that “Computer science is the science of trade-offs.” That sums up the process of designing a language - you must carefully examine those trade-offs and come up with the proper balance.

It also reminds me of that quality in mathematics which is called “elegance”.  You know how poetry is a very compressed form, if there's a syllable in the wrong place, it doesn't work... it has to have balance, and freedom, and expressiveness, and a great deal of structural integrity. Writing a language is difficult technically, but it also has those fundamental aspects. It isn't just a technical exercise, there are qualities that make it a beautiful, expressive tool. There's been a tremendous amount of thought about Inkling. Designing a beautiful tool is kind of artistic in a sense.

The implementing part is a challenge, and the design part is fun.

Tell us about the features of the language.

In terms of dominant software design paradigms, it's using the Functional paradigm, but it's got a few Imperative statements.  It is statically typed... but to understand what Inkling offers it is more relevant to note that it offers  base statements that don't exist in other languages. Inkling has the notion of curriculum, concept, and lessons to train for concepts and to teach curriculums. When the stream statement is fully implemented, it will have a full architecture of concepts and streams, which will support the LINQ syntax, which is a Turing-complete language in and of itself. When you're training machines, you train with lots of data — as you saw with your simulation. As these systems are being built out, with multiple concepts, you're streaming data through your runtime architecture, and this  allows you to define operations on your streams. While in other languages you define operations on objects or variables, in this language you operate on streams. One attribute of this which is not immediately  obvious is the flexibility that streams and stream operations provide for input to lessons. The workflow input can be sliced and operated on in many ways as appropriate to the particular lesson learning task. This is a capability unique to Inkling.

What are the major benefits of stream processing?

Stream processing will make it much easier to use data in our systems, because data isn't always pristine, and you still need to leverage it and abstract from it. There are a lot of capabilities that are tried and true in the field to make this straightforward - streams allow engineers to clean and normalize data on the fly. This is one area of Inking that parallels other languages — for example C# supports operations on streams — unlike other areas of Inkling, where there is no parallel at all. There's no language that has the concept of a curriculum, but many languages support of the use of streams. Although stream processing has developed a lot over the last decade, for example with LINQ and ReactiveX.

How was the compiler built?

The compiler is template-metaprogrammed in C++, using a Boost library called Boost.Spirit. Template metaprogramming is like a whole other language. It's a member of the C++ family of languages. You use templates, and then the compiler will generate source code based on those templates that then get compiled with the rest of your source code. So in a way, it's like an engine that produces source code according to the characteristics of the template. Because it's a form of programming that happens at compile time, it allows you to do things in your code that you would not otherwise be able to do. Another thing that's interesting about it is that some of the logic that normally happens at runtime gets offloaded to compile time, which means that you can end up with faster and smaller code because some of the decision trees and logic that would take place at runtime are handled at compile time. The Inklng compiler is small and fast, which gives us flexibility in our tool chain.

What were the most challenging parts of building the compiler?

Template metaprogramming is a difficult topic to master, and I found it to be a satisfying challenge. If you make a mistake, you can get these error messages called “novels” that are hundreds of lines long... Once I figured out how to implement the grammar I wanted, the thing about this tool that's actually fantastic is that the resulting parser is very small, fast, and reliable. It hasn't been hard to maintain once I got the hang of it. There's been a lot of good features added to C++ 11 and 14, as well as the forthcoming C++ 17 , so the language is a lot of fun to work with. C++ is a new language - actually, several languages.

I think if I were to write another language from scratch, I would use this tool because I really like the results.

What's your favorite general purpose language, C++?

Y'know, I tend to like the one I'm working with the most. Right now, I'm back in C++ land because I'm working in it, and I think the latest C++ is the bomb, it's really great! Before that, I did some work in Python, and although there are certain things about it that make my teeth hurt, there are other things about Python that really ease the development of new applications. Before that, I worked in Java, which I liked a lot, and Clojure, which was very interesting. I worked in Ruby, that was okay. I'd say right now, my favorite languages are C++ and Python.

Which programming languages inspired you in this work?

Speaking for myself, I would say the languages I think about most are C# and SQL, but that said, the reason Inkling exists is that it's doing something very different. It's needed in this problem domain. There needs to be a new language to express and solve the problems that exist. One of the things that Inkling allows you do is represent your machine learning problem in a way that's independent of whatever technology is being used by the backend. Today it's TensorFlow, but if  Microsoft or Amazon come out with something new the next month, you won't have to change your Inkling code.

One of the things that's desirable is that Inkling enables users who are not experts in the underlying algorithms and apis of machine learning to reap the benefits of AI. If our goal is to enable people who don't know machine learning to implement  machine learning, then we don't want them to need to know the differences between the underlying algorithms and apis.

Tune in next week!

We'll dive into the BRAIN and how it tunes hyperparameters with our Control Systems expert Ruofan Kong! Read more about template metaprogramming at the Gentle Introduction to TMP, or learn more about Boost.Spirit at Dr. Dobb's.

Always. Be. Learning.

Stay up-to-date on our latest product news, AI industry highlights, and more!