POP, What it Seeks to Solve¶
Plugin Oriented Programming seeks to bring together concepts and wisdom from the history of computing in new ways to solve modern computing problems. To introduce Plugin Oriented Programming, it is necessary to first cover the problems in computing today and where they came from. It is also important to answer the question of “why”. Why is introducing a new programming paradigm today the answer to these problems?
Software development operates on a much larger scale today than ever in the history of the practice. In fact, software development operates on a scale much larger than any engineering practice in the history of the world. Our economy, applications, devices and more, run on top of innumerable years of engineering effort.
The magnitude of what has been created makes the great wonders of the world that have come before seem small in comparison. The pyramid of Giza, the international road network, the power grid, and many more feats, pale in comparison to the vast collection of software engineering that has been and continues to be written day by day.
The result is countless tracks of software, lines of code beyond human reckoning.
The vast array of software available today creates new problems and challenges that cannot be addressed with existing engineering techniques. The nature of software engineering has shifted dramatically over the last 50 years, beginning with academia and government, shifting to large corporate endeavors, to small companies and teams, to globally distributed software engineering.
With these shifts, it is now an excellent time to fundamentally revisit programming paradigms. The systems used today were developed in the distant past, in a world where today’s challenges did not exist.
A Brief History¶
Software engineering started in a theoretical space as the first computers were developed with the primary purpose of executing math; breaking and creating codes and encryption. As time moved on, databases became a critical component of computing. Storing and crunching datasets became the territory of large organizations.
This time period saw computers move from rooms of vacuum tubes into mainframes. Applications were all driven on command-line interfaces and written in procedural and functional code. Languages were close to the system and designed for math dominated software engineering. Some of the largest and most complex endeavors were Operating Systems and Databases.
Multics, and Unix¶
In the 1960s, the goal of creating an Operating System became the dream of many organizations. MIT, GE and Bell Labs collaborated to create an OS called “Multics”. Many of the ideas around Operating systems were developed, but in the end Multics did not take root.
In 1969 a new Operating System was developed by Ken Thompson and Dennis Ritchie. This system was called Unix and was inspired in large part by Multics. The Unix Operating System lives on today in newer incarnations, primarily in Linux, but the Unix OS concepts are also at the core of almost all major modern Operating Systems (such as MacOS).
The Unix Philosophy¶
Ken Thompson and Dennis Ritchie knew that Multics was too complicated and inadvertently created a system that was too difficult to maintain. So they created a simple philosophy of software development for applications on Unix.
I will use Peter H. Salus’ version of the Unix Philosophy to explain it here:
- Write programs that do one thing and do it well.
- Write programs to work together.
- Write programs to handle text streams, because that is a universal interface.
This philosophy created the ecosystem of software that is the foundation for most of the world’s software today. However, we will see that it was not followed into the 1990s when computing hit consumers and the world changed.
Windows and the GUI¶
In the early 1990s, we saw the emergence of a new kind of computer, a Personal Computer. A computer that fit on a desk and was meant to solve the problems of an individual rather than the problems of banks, governments, and corporations.
The personal computer changed how applications were made. Now the Graphical User Interface, or GUI, became king. The GUI mandated that a new type of application needed to be developed, one that is more familiar to modern computer users.
GUIs and Windows diverged from the Unix philosophy and introduced more than traditional text streams. Data became more complex, as did applications.
In this era, a programming paradigm became dominant, one that worked very well at creating GUIs, Object Oriented Programming (also called OOP). The ideas around OOP had existed for many years, but the GUI gave them new life! It became much easier to build graphical applications when the aspects of the applications could be represented as objects.
Many traditional software developers railed against OOP, saying that it would create dirty and difficult to manage code. But OOP brought many merits to Graphical applications such as making it easier to reuse other developers’ code in new ways.
Rather than transferring data between programs on a command-line shell using text streams, GUIs needed to be able to manage much more complicated datasets. This gave rise to new ways to store data outside of databases. The days of XML began to take hold, and transferring large, structured datasets between applications grew.
Object Oriented Programming¶
Object Oriented Programming quickly became the dominant form of software development in the 1990s. With the rise of Windows and the GUI this only made sense.
OOP created a new set of rules and ideas around software development. What was once simply creating small compact functions that are called in sequence to execute data management and mathematical tasks, became object oriented programming. OOP changed the landscape, introducing many new development models and concepts, such as inheritance and polymorphism. Code became all about making objects that represented standalone entities.
But what is the origin of OOP? Where did these ideas come from? OOP is deeply rooted in how programming languages are written, and the core ideas are very simple. OOP is all about bringing data together with functionality. An object expresses one of the core concepts inside of programming languages, exposing a type. A type is used to represent something, like a string of text or a number. But with OOP the developer can make their own types, constructing an application from a vast collection of types.
OOP has since become the dominant way to write software, but the world has changed once again.
The Rise of the Web, SaaS, and Open Source¶
The days of personal computing has evolved into the current world, but software development remains in large part a combination of the concepts of the past.
Since the 1990’s, the internet has become the dominant stage for software development. The worlds of back-end computation and graphical environments have merged together. The code that used to be relatively thin in desktop applications, the back-end code, has become vast; on a scale that was once unimaginable, with millions of servers running massive volumes of code. The GUIs have changed as well, with the new canvas being a web browser and phone apps.
This entire structure is also run, in huge part, by Open Source code. Code that is freely given away, and whose development has enabled the modern cell phone market and the modern data center.
This new combination has also given rise to a new way of software development. When code is developed in the open, it no longer needs to be developed in an office. It is developed with the world as its artist.
The World of Today¶
The world of today presents many unique challenges. Gone are the days of the desktop application’s dominance. The back-ends of the web run almost entirely on the dominant, modern incarnation of Unix called Linux. The current world only exists because of Open Source and free software. With a giant collection of free software to work with, it becomes viable to express new concepts and ideas with code in cost effective ways.
Rather than seeing a revolution in software from new concepts, like we saw in the 1990s, the world of today is a combination of almost all of the innovations that the world has seen over the last 50 years.
But the shoulders of past giants that we all stand on today present us with new problems.
The Problems of Today¶
Much of the history of computing was skipped in this brief review, but it aims to set up the issues we face today. There are many areas that are worth addressing.
OOP was not made for Today’s Problems¶
OOP was created from the bottom up. It has many brilliant concepts, but it was created to solve issues with writing programming languages. It came of age with Java, C++ and the rise of the desktop computer and desktop GUI.
The objections of older Unix era developers (whose code we still use) ring more true today than ever before. Most code is overly complex and brittle. Monolithic and large software discards the wisdom of the Unix Philosophy, rather than seeking to extend it.
Today’s problems are a combination of back-end issues, distributed computing, distributed development, and a new class of GUI that is not driven by the old rules of the personal computer.
Open Source Scalability¶
Open Source creates new and unique challenges. We all rely on software that is not being supported monetarily. The economics of Open Source have not panned out in ways that allow it to be developed with the original goals in mind. Open Source has also diverged from the Unix Philosophy. Early on it was all about making small applications, programs that can be maintained by just a few engineers in their spare time. But recently, Open Source software has been driven by corporate interest and venture capital, creating large, monolithic application stacks.
Finally, there is the problem of the Open Source developer. Many of the best and most capable developers in the world today have written Open Source platforms that power the world. But they so often only become slaves to their creations. More often than not, they are not proportionally rewarded for the work they have done and become bogged down in maintaining communities of contributors rather than creating more software.
The Pull Request¶
Senior software developers quickly become enveloped in code review, rather than code creation. The best developers’ jobs turn from brilliance to gatekeepers. Linus Torvalds said that he no longer writes code. Rather his job is to say “no”. He is one of the most brilliant developers in the world, but his success has taken him from his profession and from his art.
Modern code contributions have become so easy that they flood core developers and mandate that they review others’ code rather than work on it themselves. This drives project leaders away from their creations and robs the world of their gifts and robs them of the joy they once found in the act of creation.
Testing Slows Down Development¶
Code needs to be tested and verified. This becomes even more critical in distributed development teams and Open Source projects, as the reliability of the contributors’ code and willingness to maintain that code falls under question.
Testing becomes more and more onerous as codebases become rapidly larger. Code bases also need to expose complex interfaces that are often deeply difficult to test quickly and efficiently.
Complex Code Deployment¶
Most of the modern code available today is complex to deploy and manage. Many applications needing to work in concert in a brittle infrastructure creates issues with infrastructure management. These issues are so bad that we have spent the years between 2010 and 2020 reinventing how code should be deployed time and time again, only introducing more and more complicated code management models.
The Mythical Man Month¶
Back in 1975 Fred Brooks published a pivotal book called The Mythical Man Month. This book backed up the Unix Philosophy by stating that just hiring more engineers did not yield proportional returns.
As codebases get larger and more monolithic, more engineers are needed to maintain them. But those codebases get harder and harder to improve! The large team produces less and less new product – stagnation rules. The best engineers get bogged down in holding up monoliths, rather than building new code.
Plugin Oriented Programming¶
Plugin Oriented Programming seeks to solve all of these issues by learning from the past and applying that wisdom to the issues of today.
We start by taking the Unix Philosophy, and modernizing it:
- Write programs that do one thing and do it well.
- Write programs to work together.
- Write programs to handle text streams, because that is a universal interface.
Becomes…
- Write programs that do one thing and do it well.
- Write programs to work together.
- Write programs to expose interfaces that can be easily merged together.
It hardly changes, only the definition of interfaces needs to be updated to deal with more complex data than what is found in a text stream.
Let’s evaluate these concepts one by one from a POP perspective.
Write programs that do one thing and do it well¶
POP is built to allow developers to write units of code in smaller, more manageable applications. But those units still function as fully capable applications.
This overcomes the Mythical Man Month by making sure that code is always broken up into small units that can be managed by small developer teams. This also makes handing off code to new developers much easier. Learning a new codebase is exponential relative to the size of the codebase. Something that is only 5-20 thousand lines of code can be handed off in days, but it can take years to transfer code that is closer to a hundred thousand lines or more.
This also makes rewrites viable. It is a regular thing to see a codebase reach its limits, or to realize that old assumptions no longer make sense. When the code is broken up into smaller chunks, those chunks can be more easily replaced or refactored.
Finally, plugin based development allows for code to be externally extended. This means that a small application can be extended easily through external plugins. This solves the Pull Request problem. External developers can write code that runs in the context of another application, but can be maintained and tested separately.
This allows codebases to be developed in such a way that small teams can operate in an isolated way while extending another team’s code.
Write Programs The Work Together¶
Plugin Oriented Programming builds applications that can be universally merged together. This means that multiple applications can be merged together into one larger application. We write monolithic code for good reasons, it is easier to deploy, and some problems are large problems to solve. Plugin Oriented Programming allows for monolithic code to be developed in pluggable ways and then merged into the monolith for easy use and deployment.
This means that the purpose of Plugin Oriented applications is so that they can work together through Plugin Oriented Programming’s concept of app merging.
Write programs to expose interfaces that can be easily merged together¶
Plugin Oriented Programming allows applications to be written in such a way that they can run alone or as a part of a larger application. This is done by building apps that expose simple interfaces. Sometimes these interfaces are simple function calls, or they are data queues that receive output data.
These simple interfaces allow for app merging and the creation of larger applications that are themselves just small bridges between collections of small applications.
Finally, POP applications can all be deployed as single binary deployments. This solves the complex deployment problem – always making code easy to distribute.
How Does Plugin Oriented Programming Do This?¶
This book is designed to introduce you to this new and growing way of thinking and developing. Plugin Oriented Programming introduces new concepts like hierarchical namespaces, app merging, configuration unification, dynamic code discovery and more to build a paradigm and an ecosystem that drives towards a future of more efficient and rapid development.
Plugin Oriented Programming has been implemented in a project called pop. The pop project is written in Python, is easy to learn and extends the already vast Python ecosystem.
Plugin Oriented Programming can theoretically be implemented on multiple programming languages or as a central hallmark of a programming language.
Read on to discover the new world of application development, Plugin Oriented Programming.