Tackling Integration Challenges: Centralised Ontology vs. Proto-Based Service-Oriented Architecture

There are only two problems in enterprise software: integration and the next integration.

Data about customers, transactions, locations, assets, processes, and service offerings are scattered across various business units, databases, and instances of internal and external applications. This fragmentation poses a challenge when other systems, processes, and data scientists need to access the data for analysis and AI implementation.

Two fundamental approaches can address this issue: a centralised ontology and decentralised, proto-based, service-oriented architecture.

Centralised ontology systems, such as Palantir Foundry or ETL processes that culminate in a central data warehouse, are designed to provide a centralised repository of metadata and data integration services. They are typically used in large organisations where data is stored in various disparate systems and need to be integrated and analysed. These systems provide a way to define common data models and integrate data from multiple sources.

Consider centralised ontologies as the "one schema to rule them all" - a meta-database that is the canonical truth into and through which all data must flow. Like the CCP, these systems solve integration challenges with an authoritarian imposition of order, structure and ideological dogma. The advantage of this approach is that disputes are arbitrated centrally, allowing strict control over data governance and security. This centralisation, however, comes at the expense of flexibility and speed of innovation and can make these architectures costly to develop and maintain.

Service Oriented Architectures, popularised by Amazon and Netflix, on the other hand, are like Switzerland. Each Canton has complete autonomy but has a strict contract in how it interfaces with the rest of the country. In the case of a Service Oriented Architecture, each system must expose its data and services via a well-defined set of APIs - in effect, a contract with the rest of the enterprise. This approach allows for a more flexible and adaptable architecture that can be easily extended to support new data sources and services.

Protocol buffers (Protos) are often used to write this contract. Let's use an example from Netflix's world. You can read more on their engineering blog: https://netflixtechblog.com/tagged/protocol-buffers

At Netflix, the team responsible for managing the process of making a movie defines a resource object called Production that contains all the information relevant to that movie as it moves through the production process. Some of that data, however, needs to come from other teams. The scripts team supports that integration by providing that data via a clean and contractually defined API.

// Contains Production-related information  
// Note: in the film and TV industry, the term production refers to the process of making a movie, not the environment to run a software
message Production {
  string id = 1;
  string title = 2;
  ProductionFormat format = 3;
  repeated CastMember cast_members = 4; // Data coming from Scripts team
  ProductionSchedule schedule = 5;
  // ... more fields
}

// The Script & Casting team has independently built and deployed a Scripts service that other teams can use to integrate script data
service ScriptsService {
  // Create a report template method.
    rpc GetScript (GetScriptRequest) returns (Script) {}
}

// The scripts team has a well defined contract for their Script resource that also defines the cast
message Script {
  string id = 1;
  repeated CastMember cast_members = 2;
  message CastMember {
    string character = 1;
    string actor = 2;
  }
}

Proto-based service-oriented architectures are designed to provide a more decentralised approach to data integration. They use a protocol buffer (Proto) schema to define data models, which can then be used to generate code for client and server applications. This approach allows for a more flexible and adaptable architecture that can be easily extended to support new data sources and services.

A proto-based service-oriented architecture offers flexibility, scalability, autonomy, and support for cross-functional teams due to its schema-driven approach and loose coupling of services. This allows for rapid adaptation to changing business needs, independent development and deployment of services in various languages, and improved collaboration among different functional areas. The architecture also enhances scalability, performance, fault tolerance, and error handling by enabling horizontal scaling across multiple servers or cloud instances, graceful error management, and independent service monitoring. Lastly, it ensures better security and data governance through the implementation of security protocols, access controls, and distributed ledger management for greater transparency and accountability.

At Alis Exchange, we've experienced how adopting a protobuf-based Service Oriented Architecture unlocks speed and innovation. We developed our Alis Build platform to make this approach to solving the integration process as simple and accessible as possible.

The platform makes it easy for developers, actuaries, business analysts and data scientists to rapidly Define high-quality API contracts, Build the underlying service and business logic and Deploy these services to the cloud.