What is a Data Platform?

Over time, organizations need to go beyond a single DB for querying and storing data to a set of DBs that cater to different business requirements. A Data Platform might comprise:

  • Search Index
  • A relational DB
  • NoSQL DB
  • Data Warehouse

Why a Data Warehouse?

It is a subject of interest to understand how the application uses the DB. The inspection can happen with a set of queries to know the DB usage. But it might affect your primary workload, so you can create isolated replica nodes for such purpose.
However, there is a time when the schema of DB data is not suitable for querying that global view of the DB. So using an ETL pipeline, data is stored in the desired schema in a data warehouse such as S3.

Why a Search Index

Used for allowing applications to search the DB. Primarily Lucene based solution such as ES, Solr. The index is mostly eventually consistent with the DB. It is expensive to update index in the write path.


Simplifying go-kit toolkit for Microservices – Part I


go-kit is one of the most complete and flexible toolkits for developing microservices in Go language. At the same time, the learning curve of go-kit is steep. In this post, I’m trying to explain go-kit fundamental components using general purpose client-server model of Linux.

General Purpose Client-Server Architecture

A server in Linux binds & listens on a port, accepts connections and server requests. A client creates its own socket, makes a request to the server (IP: PORT). The request is just a buffer. The server spawns a thread or runs the handler in the same process. The handler is a function that understands request parameters and typecast it to its expected type.
The client receives a response from the server. The response is read into a buffer and then interpreted.

go-kit in Client-Server Paradigm

go-kit helps create a service. A service listens on a well-known port and IP. It is the server equivalent of Linux server. The server defines handlers for different types of request (PUT, GET, POST). A handler is a function called for a service request. The handler is a generic wrapper over service specific business logic implementation. A handler for GET call may eventually call get_status_reservation() in a sample reservation system.

go-kit intends to keep the core logic away from go-kit influence. The core logic (set of functions that your service implements) stays in a file called service.go. The remaining go-kit code tries to access these functions in an abstract manner. There are entities called endpoints, transport and server. Each of these allows a generic interaction of service functions through HTTP (or gRPC).

The overall objective is to expose service functionality through go-kit toolchain.


It defines the send and receiver buffer structure. Each service API could expect and send different data. The transport defines structures that define request and response for each API

In the next post, I will share endpoints, the most interesting part of go-kit.

Istio: A Novice Explanation

Istio: A Novice Explanation

What is ISTIO

  • A microservices manager
  • A service mesh based system. Service mesh means a system built with many microservices 🙂
  • Manages traffic, policies for authorization, encryption, load balancing, tracing, logging (all repetitive tasks are clubbed in ISTIO)
  • It is another layer on a microservice. ISTIO is hosted on the same container/VM of your microservice.

How ISTIO work

  • It needs a separate cluster to function.
  • It has four components in its architecture
    • Envoy
    • Pilot
    • Citadel
    • Galley
  • It uses a proxy server called Envoy to monitor TCP/IP traffic and pass them to another component called Mixer.
  • Citadel enforces communication security.
  • Galley takes care of user authorization part


  • At a high scale of microservices, ISTIO eases managing common tasks
  • In my opinion, the USP is that ISTIO allows these tasks without any code modification to services.

ISTIO & Kubernetes

  • Istio configurations are merged with Kubernetes service deployment YAML.
  • ISTIO adds a new section to the service or deployment YAML.
  • There are sections suchs as:
    • VirtualService: It exposes a service to public IPs through a load balancer.
    • Gateway: A public gateway with ingress config (who can contact a service)

ISTIO Commands

$ kubectl get svc -n istio-system
$ kubectl get pods -n istio-system


Architecture Pattern: CQRS

Command and Query Responsibility Segregation provides excellent decoupling for shared data at a nominal price of higher latency for latest data. It fits very well in micro service architecture for cases of data sharing among services where one service is a reader and another a writer.


Suppose there are two serviceA & B that need to share a database. We do not want to create a DB dependency. So, we can create two DBs and purpose them for writing only and reading only.

Service A always writes to its DB. And after every write it sends and event to a pub/sub. B is subscribed to the pub/sub. B get all write events from the pub/sub and update its DB. All clients of service B always read B’s DB.

Opera Browser: Architecture

Opera is built on Chromium project. Chromium is an open  source project that uses WebKit, a free and open source rendering engine. WebKit is open sourced by Apple.

So, Google Chrome and Opera resemble a lot. They use process for each tab and look similar. The benefit of using processes to handle tabs is:

  1. Better security for browser as processes can’t circumvent security rules easily.
  2. Identification of inactivated tabs and swapping them out of memory, to keep browser light-weight.

To understand more about rendering and Chromium architecture, please visit the following:

  1. https://www.chromium.org/developers/design-documents/multi-process-architecture
  2. https://www.chromium.org/developers/design-documents/displaying-a-web-page-in-chrome