In this post, I am going to build a simple distributed key-value store using JAVA and sockets for networking. I am going to show how to use Zookeeper as a coordination service in a cluster with multiple partitions and replication. Following are the functions of Zookeeper service in this system: …

“If the only tool you have is a hammer, it is tempting to treat everything as if it were a nail.” — Abraham Maslow In the field of software engineering, every choice you make requires making one or more tradeoffs. …

This post is a gist of our efforts in migrating from managed Spark in Synapse and Databricks to Kubernetes for our production workloads. Without the nitty gritty of migration efforts, lets get into the details of how did we setup our Spark cluster and how our Spark application runs on…

In a distributed hash table, a hash table with N keys is partitioned into M partitions or nodes. This is done in order to handle large number of keys and queries that exceeds the limitations of a single machine CPU and RAM. For e.g. if the size of a hash…

Having worked on variety of problems over the past 10 years or so, one of the most challenging aspect that still remains for me is how to tackle inter-dependency problems in OOPs efficiently? The confusions that arises when dealing with problems that require dealing with inter-dependency in an OOPs are…

Chat sessions are maintained by the chat server which coordinates between multiple user sessions like managing user identities, accepting messages from sender, forwarding messages to receipient, tracking user login/logout info etc. While a full-fledged chat server requires database connections to persist user and session informations. This could be a simple…

There were several instances where I wished that my file I/O gets a bit more faster because there were times when they required to be almost real time. Sometimes we can get around this problem by using an in-memory cache (hash tables, trees etc.) …

As a ML and a data engineer, I am quite often required to download lots of images (for Computer Vision projects) from different datasets available over the internet. Using Python requests.get() to download thousands of images each on an average 50KB in size can be very time consuming. For a…

Having worked extensively with Python across multiple projects in my career, I have used multithreading and multiprocessing in many places wherever I felt that there is a need for one. One of the biggest advantage of multithreading and multiprocessing is that it is quite straightforward to understand where you will…