Why Redis Pipelining is a Good Idea?

What is PipeLining?

Pipelining is a form of asynchronous task execution. A Pipeline is a task that is composed of many subtasks. Each subtask may be dependent on its previous subtasks.

t1 => t2 => t3
            ^
        t4 =|

t2 depends on t1. t3 is dependent on t2 & t4.

While we are executing subtask t3 for primary task S1, we can also execute subtask t1 for another primary task S2. This is a pipelined execution.

How Redis Pipelining helps in Client-Server communication?
Each network request between the client and server involves a latency named Round-Trip-Time. Redis server reads a request from a client and client waits till the server writes the response.
We can rather ask the client to send many requests at once and collectively wait for the responses.

This helps us achieve:

  • Ability to counter RTT of a network.
  • Better throughput for client
  • Saves multiple reads from server to client.
  • The server sends single write with all responses

Classic Usage

  • POP3 uses pipelining

References

Advertisements

What is a Data Platform?

Over time, organizations need to go beyond a single DB for querying and storing data to a set of DBs that cater to different business requirements. A Data Platform might comprise:

  • Search Index
  • A relational DB
  • NoSQL DB
  • Data Warehouse

Why a Data Warehouse?

It is a subject of interest to understand how the application uses the DB. The inspection can happen with a set of queries to know the DB usage. But it might affect your primary workload, so you can create isolated replica nodes for such purpose.
However, there is a time when the schema of DB data is not suitable for querying that global view of the DB. So using an ETL pipeline, data is stored in the desired schema in a data warehouse such as S3.

Why a Search Index

Used for allowing applications to search the DB. Primarily Lucene based solution such as ES, Solr. The index is mostly eventually consistent with the DB. It is expensive to update index in the write path.

Why Docker is a Long Term Future for Platform?

What make Docker so popular and long-lasting?

  • Container is essentially OS level virtualization. Each application gets illusion of its own OS, having almost absolute control over it. Another advantage is that host OS knows about the container processes and hence can share its resources among hosted containers.
  • The concept of containers was started by FreeBSD, refined by Solaris and re-implemented by Linux.
  • Containers are better than two other levels of virtualization:
    • ABI/platform level, where application integrates with the platform (Google App Engine), doesn’t scale well.
    • Hardware level, where a virtual hardware runs the OS (e.g. virtual machines, hypervisors).
  • Docker containers run close to the real hardware, and host OS has knowledge of resource usage. Hence it’s an optimal sweet spot for virtualization.
  • Joyent SmartOS is built on OpenSolaris and provides Solaris features to Linux like Docker containers. It acheives that by allowing Linux APIs translated to Solaris APIs. Everything runs on the bare metal hence.
  • SmartOS containers get ZFS, Dtrace by default 🙂
  • SmartOS containers are very secure as they run in zones.

I hope to cover Smart OS design internals, Docker container on Linux details in next posts.

Reference

Linux Memory Management Tricks

Tips to Improve Dynamic Memory Performance

Instead of using memset() to initialize malloc()’ed memory, use calloc(). Because when you call memset(), VM system has to map the pages in to memory in order to zero initialize them. It’s very expensive and wasteful if you don’t intend to use the pages right away.

calloc() reserves the needed address space but does not zero initialize them unless memory is used. Hence it postpones the need to load pages in to memory. It also lets the system initialize pages as they’re used, as opposed to all at once.

  • Lazy allocation: A global(normal variable or a buffer) can be replaced with a static and a couple of functions to allow its access.

  • memcpy() & memmove() needs both blocks to be memory resident. Use them if size of blocks is small(>16KB), you would be using the blocks right away, if blocks are not page aligned, blocks overlap.
    But if you intend to postpone the use, you would increasing the working set of the application. For small amount of data, use memcpy().

  • To check heap dysfunctional behavior: $ MALLOC_CHECK_=1 ./a.out
    It’ll give an address related to each violation of dynamic memory routines.

  • Electric fence : Works very well with gdb

  • Libsafe: for libc routines

Internet Not working eh? How Ubuntu Resolves DNS?

Problem

I upgraded my Ubuntu from 16.04 to 18. And the Internet stopped working. 😦

I tried rebooting interface, adding a new interface to ifconfig. But it did not help.

What helped was an answer on AskUbuntu

How it worked?

The answer is in how Ubuntu lookup domains like http://www.google.com.

Step 1
Check the domain in the local DNS, also called a DNS stub.

Step 2
If Step 1 fails (local DNS lookup is a miss), it contacts a public DNS such as 8.8.8.8 also called Google Public DNS.

Since after the upgrade public DNS was not set, the internet seemed inaccessible!

How to Update Public DNS?

open /etc/resolv.conf
Add DNS=8.8.8.8

Done 🙂

Written with StackEdit.I

Notes on Go Structure Tags

Go has a feature to create a user-defined type with struct. Go also allows the program to specify meta-information with structure field.
This meta-information is called Tags. A tag usually helps in packaging/unpacking a user-defined type structure.

  package main

  import (
      "encoding/json"
      "fmt"
  )

  type Test struct {
      // json tags indicate the key name to use in json data
      Name    string `json:"myname"`
      Country string `json:"region"`
  }

  func main() {
      // The order of json need not match the order of structure fields.
      p := map[string]string{"region": "earth", "myname": "hello"}

      marshalled, err := json.Marshal(p)
      fmt.Println(marshalled, err)

      // get the var back from marshalled data
      var m Test
      json.Unmarshal(marshalled, &m)
      fmt.Printf("name=%v\n", m.Name)
      fmt.Printf("name=%v\n", m.Country)
  }

Why should I use json tags

  • The benefit of json tags is the flexibility to marshal with either a JSON or a struct.
  • JSON input for marshaling is convenient and flexible with elements ordering.

Reference