Unified Payment Interface(UPI) Architecture

---

Unified Payment Interface(UPI) Architecture

In the digital age, where convenience and speed are very very important, the Unified Payments Interface (UPI) has emerged as a game-changer in the realm of financial transactions in India. UPI, introduced by the National Payments Corporation of India (NPCI), has revolutionized the way people send and receive money, making it as simple as a few taps on their smartphones. With its seamless integration across multiple banking platforms and widespread adoption by various payment apps, UPI has become the go-to solution for instant, secure, and cashless transactions.

In this tech blog, we will delve into the world of UPI and explore its inner workings, advantages, and the impact it has had on the Indian digital payments landscape. Whether you are a tech enthusiast, a curious user, or a business owner looking to embrace UPI for your enterprise, this comprehensive guide will provide you with the insights you need to navigate the exciting world of UPI.

Transactions stats for Year 2023:

Transactions in Million

If you compare month on month between May 2022 vs 2023 — (9415–5955/5955) * 100 ~ 59% increase, this is really huge.

9415*1000k/24*60*60*30 ~ 4k transactions/s, this is very high QPS. How does UPI system is able to handle such load?

First let’s start with the basic flow diagram of any UPI transaction and later we will deep dive into the detailed architecture of it.

---

UPI high level transaction flow

---

Here are the entities involved in above transaction:

1. User :Initiates a transaction for another UPI ID after login to his/her PSP app account.
2. PSP — Payer: Validate the user(with the help of NPCI) and forwards request to NPCI for the further processing of the transaction.
3. NPCI UPI ecosystem: Forwards the request to PSP(Payee) for further address resolution and processing.
4. PSP — Payee: Uses NPCI APIs to find out the account details/address of the Payer’s bank/Remitter Bank.
5. Remitter Bank: Validate and confirms the account deduction request raised by the Beneficiary Bank. Sends the confirmation back to NPCI.
6. Beneficiary Bank: Validate and confirms the account credit request raised by the Remitter Bank. Sends the confirmation back to NPCI.

Finally NPCI notifies about the successful or failed transaction request back to all involved PSP. This notification may be push/pull type based on the configuration.

References:

How Does UPI Work? - GeeksforGeeks
A Computer Science portal for geeks. It contains well written, well thought and well explained computer science and…www.geeksforgeeks.org

Monthly Metrics
UPI Daily Product Statistics Trended Aug 2021 No data available for the selected filters UPI Monthy Product Statistics…www.npci.org.in

Statistics of NPCI - National Payments Corporation of India
Know more about the Statistics of NPCI - National Payments Corporation of India.www.npci.org.in

UPI: Unified Payments Interface - Instant Mobile Payments | NPCI
UPI - Unified Payments Interface is an instant real-time payment system developed by NPCI to facilitate inter-bank…www.npci.org.in

https://www.youtube.com/watch?v=QpLy0_c_RXk

Message Acknowledgement types and different delivery methods in a Distributed Message Queuing system

In today's world of communication, message queues are ubiquitous.

In the world of distributed systems, message queue plays a vital role in enabling reliable and asynchronous communication between components.

One critical aspect of this communication is message acknowledgement. 

Acknowledgement ensures that messages are delivered successfully and processed by the intended recipients. 

In this blog post, we will explore the various types of message acknowledgements and the different delivery methods employed in a distributed message queuing system.

Popular Open APIs available for developers

There are several open APIs (Application Programming Interfaces) available for developers across various domains. Here are a few popular ones:

1. OpenWeatherMap API: Provides access to weather data and forecasts worldwide.

   Website: https://openweathermap.org/api
   Website: https://developers.google.com/maps/documentation
   Website: https://developer.twitter.com/en/docs/twitter-api
   Website: https://docs.github.com/en/rest
   Website: https://stripe.com/docs/api
   Website: https://www.twilio.com/docs/quickstart
   Website: https://api.nasa.gov/
   Website: https://platform.openai.com/docs/quickstart
   Website: https://developer.spotify.com/documentation/web-api/
   Website: https://wiki.openstreetmap.org/wiki/API

2. Google Maps API: Enables integration with Google Maps services, including mapping, geocoding, and routing.

3. Twitter API: Allows developers to interact with the Twitter platform, access tweets, post tweets, and perform various other actions.

4. GitHub API: Enables developers to interact with GitHub repositories, manage issues, pull requests, and perform various Git operations.

5. Stripe API: Provides a platform for developers to integrate online payment processing into their applications.

6. Twilio API: Allows developers to add SMS, voice, and video communication capabilities to their applications.

7. NASA API: Provides access to various NASA data sets, including astronomy images, Mars Rover photos, and more.

8. OpenAI API: Offers access to powerful natural language processing models and capabilities, including text generation and language translation.

9. Spotify API: Enables developers to integrate with the Spotify music platform, access playlists, search for tracks, and manage user libraries.

10. OpenStreetMap API: Provides access to crowd-sourced map data, geocoding, and routing information.

Let's start with OpenAI APIs:

First of you would need to know what are the different models which is available, and, then, find a API which fits your use case.

I tried Image APIs, which can be useful for designers to create/update images in different format.

Example:

curl https://api.openai.com/v1/images/generations \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "prompt": "A cute baby sea otter",
    "n": 2,
    "size": "1024x1024"
  }'

Interesting part is Prompt, you can pass simple conversation type command similar to the way you interact with your designer while building a product.

Another API which I have tried is extracting tags from a list of product attributes. 

This is the API:

curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Extract Tags from this product description: You can get your clothes thoroughly cleaned in the LG FHM1006SDW 6 kg Front-loading Washing Machine. Its 6 Motion Direct Drive technology pushes the wash drum in numerous directions when you select a wash programme. This ensures that your garments are gently yet thoroughly washed. In addition, this washing machine produces less noise, vibration, and wear and tear as its motor is directly attached to the drum. Moreover, this washing machine removes pet hair, pollen allergens, and house dust mites from delicate garments that require special care, thanks to its Steam Wash programme"}]
  }'

Response:

"choices": [
        {
            "message": {
                "role": "assistant",
                "content": "LG, FHM1006SDW, 6 kg, Front-loading, Washing Machine, 6 Motion Direct Drive, wash programme, gentle washing, less noise, vibration, wear and tear, motor, drum, pet hair, pollen allergens, house dust mites, delicate garments, special care, Steam Wash programme."
            }
]

Reference: https://platform.openai.com/docs/api-reference/chat/create

 

Self vs Team's visibility

Today, I am going to discuss about the self vs team's visibility.

I hope most of you would have seen this Bollywood movie "Gangaajal", in one of the scene when the hero of the movie (Ajay Devgan) was getting the president medal because of a successful crackdown in one case.

He refused to take the credit, instead he said that it's his team who has done this, but, the guy who was giving his medal said"Sainik to kaam karte heen hain, lekin, credit to finally senapati ko heen milta hai, directly ya indirectly".

I was the firm believer of the above thought, if my team's work or team members are visible in the system, this means, I am visible by default.

As a leader of your team, it's your responsibility that you need to make sure that the last line of people in your team should get the due credit and right visibility.

What about you? It's equally important for you also to be on top everything and to be shown as leading from the front.

Getting things done is not longer seen as adding very much value, because, that is just one part of your role. There are other parts, like adding value, right delegation etc.....

So, how to build the self visibility? Here are a few steps which you should do, I am also trying to do the same:

1. Any collaboration opportunity outside your team, you should lead it.

2. Product ideation, first level of thought should come from you and make sure that's visible to your boss 🙂.

3. Start giving valuable inputs in any cross team discussion, this will establish you as a thought leader.

4. Start giving internal/external teach talk.

5. Collaborate with the guy in your team who is afraid of doing presentation, this will help him/her in building visibility.

6. Talk, over communicate about self and team's achievement.

7. Write Blog, organise tech meet-up....

This list is very long, pick the top three and start doing it today.

One of my friend keep quoting this "Career is a hard fought battle, you need to fight very hard". True, very, very true.

Happy learning...

Data Observability Platform

I am just back from my long vacation, last week we had a tech meetup on "Data Observability".

Please follow the details here, will be publishing a detailed blog soon on "Data Observability". Watchout this space.

Last week Tech Meetup details: Data Observability tech meetup

Scaling lessons from building BBDaily.com(Bigbasket.com) and shiskha.com?

In this post, I will talk about a few scaling lessons from my own experience.

Below are a high level architectures of any web based application which can fairly scale up to a few million requests/day. 

This we have accomplished while scaling out shiksha.com and BBDaily(Bigbasket.com), every application goes though this cycle. 

Architecture 1(VM based local cloud):

Architecture 1(AWS based):

Let's talk about the lessons learned in the journey:

Scaling FE tier:

1. Front End Tier can be fine tuned and scale out horizontally if you have a common Session Management Service in place. You can easily add an ALB(Application Load Balancer) on top of the front end tier and route the request based on any built in algorithm provided by ALB like Round Robin, Weighted Round Robin, IP hash etc.. more.

2. Serve Static Content (JS, Images, CSS, Fonts etc...) through any CDN provider, there are many CDN providers which you can consider. 

3. While using CDN, you need to take care of versioning of the static contents every time a content is updated. This is required so that you don't need to ask your client to delete the local client side cache after a new release of your application. We faced this issue multiple times where we had to go to our CND providers and manually delete the old cache when versioning was not in place.

4. Static Asset Versioning is a simple technique which you can accomplish by writing a simple script which creates a new version of the asset, store the version no in a Config/DB/Cache. This script you can easily plug in your build/deployment pipeline.

5. You should configure Static Domains (no need to send cookies to get static content) for image, CSS, JS etc.... separately. This is the key learning we had while doing the FE optimization's, because there are limits on no of parallel connections/domain a web browser can make. Also, there are limits on max no of connections. 

6. If you are serving different assets through different domain, this will speedup the content loading/parallelization by the web browser/client.

7. No need to send cookies for fetching the static content this was happening because we were serving static contents from the sub-domain not the static domain. It was increasing the HTTP request size which was unnecessary.

8. Use FE profiling tools like Lighthouse or Pagespeed or Chrome Developer Tool to analyze and fix/improve the performance.

Scaling BE tier:

1. Separate target groups based on different applications for example: Consumer app, OPS app, Admin and Crons/Batch Jobs.

2. Within each target group you can set different autoscaling policy based on the type of application for example CPU/Memory/Disk IO intensive or a combination.

3. APIs profiling and optimizations, use any APM tool (New Relic, Rakuten SixthSense).

4. Separate DB replicas for different applications and reports, this is required to scale out read requests. Typically applications are mostly read heavy as compared to write.

5. Used managed services of the cloud provider where your application is hosted (AWS, GCP, Azure etc..). This helps in reducing the overhead of managing the infra/DevOps cost, also you can focus on budling the features for your customer/business. There is a tradeoff here, but, we can discuss that in other post.

6. DB archival/partitioning, you need to do this once your data grows so much that quiring/managing has become difficult. You can think about using any cluster based solution where data is distributed across multiple nodes along with replicas.

7. VividCortex can used for DB monitoring/query analysis. 

8. Other monitoring/alerting tools CloudWatch, Opsgenie, Datadog, Apptuit etc.., use these tools for monitoring/alerting purpose.

Here is the detailed tech talk, which I did. 

Video link

Medium: https://medium.com/@aditya0987654321/scaling-lessons-from-building-bbdaily-com-bigbasket-com-and-shiskha-com-35c2afe74c59

 

Application/Platform cost optimization's series - Part Three

Let's finish this post with an example of "Back-Of-The-Envelope Calculation or Fermi problems", this is a useful technique while you are building a new platform from scratch or scaling out your existing platform.

Example: Let's say you want to build a photo-sharing app, below are the high-level features:

1. User should be able to register with the app
2. Upload a photo/photos
3. Get a sharable link after the photo is uploaded
4. User should be able to view/browse through the uploaded photos

Non-functional requirement(assumptions):
1. No of active users/month ~ 1M
2. Average photo upload/day ~10 photos
3. Total new users/day ~ 1k
4. Average photo size ~2MB

Based on the above nos, let's do some calculations for the user and their photos data.

N = no of years you want to hold the data

1. Let's say the average user profile size (basic details and profile image) is 1MB. The total data size of profile DB = N*1000*365*1MB = N*365GB
2. Photos storage requirements: N*10*33000*365*2MB = N*.25TB
3. Let's say you want to replicate the data to m no of replicas

So the total data storage requirement would be:
Profile DB size ~ m*N*365GB
Photos DB size ~ m*N*.25TB

Similarly, we can do the calculation for the no of read-to-write requests (QPS) and provision our server based on that.

Previous post link(just to connect the dot): http://bitly.ws/D6P2

Happy weekend