0

Manager’s role in code review

by Deepak Dhakal 16. June 2022 03:04

The engineering manager's role in the code review process

Ultimately, the job of engineering leaders is not to code—it is, instead, to remove obstacles so their teams are able to spend more time working on valuable solutions and so their work output has the reach, impact, and visibility it deserves. So it would be easy enough to designate code review as the engineers’ domain, where managers need not tread.

But that makes for a massive missed opportunity.

Without guidance and a healthy review culture, the PR process can disintegrate into unproductive, though understandable, behaviors. Some developers see code review as pulling them away from their true work, and it’s impossible to deny that review is a somewhat subjective process that, unshepherded, can lead to disagreements, stalled commits, and even outright hostility.

Many individuals also have not experienced the art of accepting and giving criticism, and therefore haven’t learned it. Unintentional communication breakdowns can lead to both social and technical frustration, which of course gums up the works too.

These sorts of PR environments are not sustainable or healthy. That’s why managerial support is critical, if reviews are to become opportunities for teams to learn from each other and work toward more effective, more creative solutions.

This holds true with engineers of all levels. Engineers new to the organization (or new to the industry) glean the team’s culture, pace of work, style, and implicit coding standards through involvement in the PR process. Senior developers are able to coach more junior ones in their domain expertise, and for engineers of all levels, code review is a chance to identify strengths and weaknesses (both individually and organizationally). And for co-located and distributed teams alike, code review is perhaps the richest opportunity for work-centric socialization and team-building.

So where do managers take part in this process?

Where training was once at the heart of management, code review is now the prime way of improving an engineering team’s output. By participating in and observing code review, managers are able to track the health and productivity of the team, which provides insight into where to intervene and where to encourage progress.

In other words—the team is responsible for creating code and for peer reviewing that code. Managers are responsible for the behavioral trends exhibited in their teams’ code reviews.

In our work, we’ve learned to recognize common patterns exhibited by software engineering teams—both successful patterns that can be nourished, and problematic ones that an aware manager can remedy. Here, we’ve assembled eight of those dynamics that demonstrate the behaviors common to developers and to engineering teams, how to recognize them using GitPrime metrics, and what managers can do to bolster their teams’ health and productivity.

Common dynamics to identify in the code review process

Long-running PRs

Long-running pull requests have been open for an unusually long period of time. Organizations ship at different rates, so a PR that stays open for 5 hours could be long-running for one organization, where 24 hours will long-running for another. Sometimes, PRs will stay open for several days.

There are a number of reasons why a PR might stay open for an extended period of time:

  • There’s uncertainty or disagreement about the code (which can reveal itself with a few back-and-forth comments earlier in the PR followed by silence)
  • There are large spaces of time between comments and responses in the review
  • The PR is massive (think: multiple days’ or even weeks’ worth of work) and team members are avoiding having to review all of that code
  • The PR was submitted at 5pm on a Friday, so the review didn’t start until the following Monday at best

Apart from being symptomatic of possible disagreement or confusion within the team, long-running PRs are also themselves a problem. A PR that is a week old can quickly become irrelevant, especially in fast-moving teams. In short, long-running PRs are bottlenecks to a release.

How to recognize them: Long-running PRs can quickly be identified in the team’s Review Workflow report, filtered by “PR Status: Open” and sorted by “oldest PRs.” Select the number of PRs you’d like to see in one view, then hover over those that have been open for more than a day.

If you see a few back-and-forth comments with signs of uncertainty or disagreement in the communication, followed by silence, it’s worth checking in to see how you can move the conversation forward.

What to do:

  • If there are signs of disagreement or confusion in the discussion: It’s usually best to first check in with the Submitter. It’s their responsibility to get their work across the line, so they should be encouraged to bubble up disagreements or uncertainties as they arise. If there is a disagreement, get their read on it and offer advice to move it forward. Depending on the situation, get the Reviewer’s read on it as well — ideally when everyone is together in a room or on a call. Make a decision, and ask anyone who disagrees to “disagree and commit” for the sake of the team’s progress.
  • To manage this pattern in the long-term, or if there are large spaces between comments and responses in the review: Set expectations or targets around Time to First Comment and Time to Resolve. (Both metrics can be found in GitPrime’s PR Resolution report.) It can also be helpful to communicate best practices around responding to colleagues in a timely manner. When it takes someone a day to respond to a comment, that can mean there’s a lot of time spent waiting on others, and the communication isn’t timely enough to be as effective as it could be.
Heroing

Heroing is the reoccurring tendency to fix other people’s work at the last minute. Right before each release, the Hero finds some critical defect and makes a diving catch to save the day.

Of course, attention to detail is essential and a good save is usually better than no save. But regular Heroing leads to the creation of unhealthy dynamics within the team or otherwise encourages undisciplined programming. Some team members even learn to expect Heroes to jump in on every release.

Heroing can be a symptom of micro-management or poor delegation. It also points to trust issues on a number of levels. Heroing will ultimately undermine growth by short-circuiting feedback loops and, over time, can foster uncertainty and self-doubt in otherwise strong engineers. At its worst, Heroing feeds a culture of laziness: everyone knows the Hero will “fix” the work anyway, so why bother. Ironically, those last-minute fixes are the genesis of a lot of technical debt.

How to recognize it: The Hero typically dominates GitPrime’s Help Others metric, particularly in the form of late arriving check-ins. They’re also distinguishable in the review process, where they may be self-merging PRs (and typically right before the deadline), or they will show very low Receptiveness in the review process (meaning either others aren’t providing substantial feedback or the Hero isn’t incorporating it).

It can be hard to disagree with their changes—especially with these changes being made so late in the sprint. This is partly why the Hero’s PRs usually show a very low level of engagement in the review process (see the Review and Collaboration metrics).

What to do

  • Rather than managing the “saves,” manage the code review process. Ideally, team members are making small and frequent commits and requesting interim reviews for larger projects. If that’s not the case, consider working toward that goal first. Getting the Hero’s feedback early, even before the code is done, will help improve the problematic tendencies.
  • When the team is in the habit of getting feedback early and often throughout a project, as opposed to submitting massive PRs all at once, the barrier to participating in the review process is lower. This can make it easier to promote healthier collaboration patterns and get everyone—especially the Hero—to give and be receptive to feedback in reviews. Coach the Hero to turn their “fixes” into actionable feedback for their teammates to implement with time to spare.
Over-helping

Collaboration among teammates is to be expected, as it is a natural part of the development process. However, “Over-helping” can occur if one developer spends an unnatural amount of time helping another developer get their work across the line.

Engineer One submits. Engineer Two cleans it up, over and over again. This behavior can be normal on small project-based teams. But when that 1-2-1-2 pattern doesn’t taper off, it’s a signal that can draw your attention.

The problem is threefold: (1) always cleaning someone else’s work takes away from one’s own assignments, (2) it impairs the original author’s efforts toward true independent mastery, (3) it can overburden the helper and leave the original author in a continuous unnatural waiting state.

How to recognize it: You’ll notice this common dynamic in the same way you’d realize Heroing in GitPrime’s Review and Collaboration reports and the Help Others metric. Look for reoccurring, last-minute corrections between the same two people.

In the Review and Collaboration and Operational reports, you’ll notice these two engineers consistently review each other’s work. One engineer will have a high Help Others, but it’s not reciprocated. The load-bearing engineer will also show high levels of Influence and Review Coverage. The other engineer will not. One engineer will have a high Impact; the other won’t.

This behavior can be perfectly healthy and expected when in a mentorship-type situation. But beyond a certain point, rotation is in order.

What to do:

  • Bring additional engineers into the code review process. A side effect of this solution is that by increasing the distribution of reviews, you’re strengthening the team’s overall knowledge of the codebase.
  • Cross-train and assign both engineers to different areas of the codebase.
  • Assign the senior engineer a very challenging project. The idea here is to give them challenging projects where they don’t have the time or energy to review their colleague’s work.
  • Lastly, the stronger of the two is showing natural leadership and coaching tendencies. Look for opportunities to feed these skills more broadly to the whole team.

One note of caution: be mindful when the two engineers are friends or were colleagues at a former employer. Making light of a friendship or teasing them can be incredibly damaging and hurtful. Go the extra mile to keep it professional. And, as always, be transparent. You’re not trying to split up friendships. It’s the manager’s job to ensure that knowledge of the codebase is distributed evenly across the team and to ensure that people are honing their craft and growing their careers.

Over-engagement

It may seem counterintuitive that over-communicating in code review can be a problem. After all, we want our teams to be excited about—and engaged in—their work. However, since over-commenting comes at the expense of an engineer’s own deliverables, such over-engagement can affect team morale because some engineers are not achieving what is, ultimately, their responsibility to accomplish.

The problem arrises when over-engaged engineers spend a disproportionate amount of their time providing feedback (helpful or not) on other people’s work and too little time working on their own projects. The healthy mix of reviewing and contributing is thrown out of whack.

Furthermore, while gregarious teammates often ignite interesting and creative conversations, this problem emerges when their PR contributions tip from beneficial commentary to comment spam. Their contributions may make them appear busy, but the data shows this behavior is not constructive.

How to recognize it: As a manager, you may miss that this is happening, because most of what you see appears to be sound engagement. But the teammates know it’s going on. They’re on the receiving end of that over-engagement, and they’re acutely aware of it—they just might be unable to talk to you about it.

You may first spot this dynamic in the Review Workflow when you notice a team member is frequently engaging in the PR process as a Reviewer. Perhaps they’re commenting, providing feedback, or otherwise showing a strong level of engagement with other people’s work. This behavior is only something worth noticing when the feedback that’s being provided is going far beyond what’s “good enough” for that specific project. If you notice a highly engaged Reviewer, you can go to their Player Card to see their level of Involvement and Influence—and evaluate that in context with their Code Fundamentals (“Is this team member spending more time in review than on their own work, and is that expected?”), and with the team’s averages for that time period.

What to do:

  • It helps to remain aware that over-engaged engineers may have come to believe through learned behavior that commenting in this way allows them to contribute most helpfully to the team. Helping others is helpful, right? So as a leader, you need to ensure that you’re setting proper expectations and guiding these engineers to correct course.
  • It can be challenging to tell these engineers not to comment so much, because they often feel they are contributing positively. A more constructive approach is to add projects to their bucket of work over the next couple sprints. As they feel the pressure of hitting their deliverables, they’ll naturally tend to scale back on their commenting activity. That’s when you give them an out.
  • By doing this, you can let them know, “You clearly have a lot of work on your plate. If that needs to come at the expense of some of the code review you’ve been doing, that’s totally okay.” Just giving them that little out is usually all you need to restore balance on the team. That engineer is hitting objectives for the sprint, and the team feels overall happier and more harmonious.
Just one more thing

Just One More Thing refers to a pattern of late-arriving pull requests. A team submits work, but then—right before the deadline—they jump in and make additions to that work.

Sometimes only one or two individual contributors will show this pattern, and that generally points to behaviors that require an individual. But when the majority of the team is submitting PRs right before a deadline, it can mean there are larger process or even cultural issues that are causing an unpredictable workflow. This pattern can occur for a wide range of reasons, including last minute requests, poor planning or estimates, and too much work in progress.

How to recognize it: Just One More Thing, when appearing across a team, is characterized by a spike in PRs being submitted near the end of a sprint after the main PR was approved. These engineers will also show a high level of New Work.

What to do:

  • Late-arriving PRs are a sign that work is being rushed and given less review. Even when the work is submitted by engineers who are very familiar with the code, the PRs should be treated as riskier than other equally sized PRs that are submitted earlier in the sprint.
  • When you notice a spike in PRs being submitted, it can be helpful to review the work submitted and decide whether it should be given an extra day’s review.
  • Longer-term, consider working with the team to identify any bottlenecks or process issues that could be eliminated or improved.
  • If the team’s estimates or deadlines are causing last-minute stress, consider setting different internal deadlines for projects. Another framework that some teams use is to consider the three levers in setting a deadline: the external deadline (if any), the scope of the project, and the resources available. It’s typically not realistic to change one without having to change the others, so it can help the planning process to take all three variables into account.
  • If last-minute requests are coming in from outside the team, talking to the stakeholders or managers whose groups are regularly causing the problem can give you the opportunity to show the impact of the problem and understand what’s going on from their perspective.
Knowledge silos

Knowledge Silos are usually experienced between departments in traditional organizational structures, but they also form within teams when information is not passing freely between individuals. They form when a group of engineers review only each others’ work.

Imagine two or three engineers who review all of each others’ PRs, and don’t review anyone else’s PRs on their team. These engineers learn about each other’s work and techniques, and the areas of the code that they’re working in, while other engineers on the team who aren’t part of the silo don’t have that same level of information.

There are plenty of reasons why engineers will get into a cycle of reviewing only each other’s work — figuring out the reasons why, through discussions with the team and by reviewing the Team Collaboration metrics, can sometimes point you toward the broader team dynamics at play. For example, if these engineers want to work together because everyone else on the team is slow to review their code, you can consider setting expectations around Time to First Comment and Reaction Time.

Whatever the cause, reviewing a select group of engineers’ work for a long time can lead to less substantial reviews simply because the engineers trust that each others’ work is good enough. When that happens, these situations can turn into bug factories. Work is being approved and pushed forward without adequate evaluation.

How to recognize it: When team members are co-located, a basic understanding of where people sit in an office along with an awareness of any other social bonds can be helpful indicators as to where silos may form.

You can also use the Knowledge Sharing report to visualize how knowledge is being distributed across a team in the review process and to identify knowledge silos. If there are two or three people who review only each others’ code, the team’s Knowledge Sharing Index will trend toward 0. If the majority of the team reviews each others’ code, the Index will trend toward 1.

You can then drill down into specific team dynamics with the Review Radar. When there are silos, there will be a small group of engineers who review only each others’ work across multiple sprints.

What to do:

  • Bring in the outsiders! Look for outliers and stranded engineers and get those individuals involved in the review process. You can also see whether there’s anyone who could be cross-trained or onboarded on a specific area of the code that an engineering within the silo is working on.
  • Assign other engineers to review the work of the individuals that make up the silo, and have the individuals within that tight-knit group review the work of others outside their group.
Self-merging PRs

Self-merging pull requests refers to when engineers open a pull request and then approve it themselves. This means no one else reviewed the work and it’s headed to production.

As a general rule, engineers should never merge their own code. In fact, most companies don’t permit them to: self-merging bypasses any form of check on the code, as well as skipping the opportunity for improvement and learning.

If the code is worth putting into the main code branch, it is worth having somebody review it. Self-merging represents a material security risk to the company, no matter how talented an engineer is. Yet as a practical matter, unreviewed pull requests happen a lot, for any number of reasons.

How to recognize it: Self-merging is easy to see because the submitter and the reviewer are the same people. In GitPrime, these instances will show up in the team’s Unreviewed PRs metric as well as in the Review Workflow.

What to do:

  • Many organizations prevent self-merging PRs by configuring their build systems to reject them. Enforced review is most common among companies that work under regulatory compliance, like Fintech or Biotech companies.
  • Even in organizations that don’t enforce review, managers should be in the know when these situations do happen. Reviewing these PRs on a case-by-case basis, even though they’re being reviewed after they’ve have been merged, will help ensure that any bugs or problems are not going to get buried.
  • If the commit was trivial, you might be able to give QA a heads-up to take a close look at it. If the unreviewed pull requests are non-trivial, walk those back if the circumstances allow and require a code review.
  • Reducing the frequency of unreviewed and self-merged pull requests is a best practice (Unreviewed PRs should be 0%, or close to it). If engineers are in the habit of self-merging without review, it may be helpful to have an informal conversation with them to ensure that they understand the why behind the review process or that they are at least clear on expectations. If they’re more senior, encourage them to follow the best practice of getting code thoroughly reviewed by others, so other engineers will model that behavior.
Ramping up

We cannot overstate how helpful the PR process can be in encouraging the team to consistently learning from each other. Code review is always a chance for cross-pollinating information and expertise between team members. So using PRs in the process of onboarding and ramping up engineers new to the organization is a great way to build connections between developers who will be working together.

A key component of successful engineer onboarding is ensuring that new arrivals are learning both the code, as well as how to engage meaningfully with their new teammates.

If you ask your engineers—and particularly recent hires—how things are going, you’re bound to get the initial “Great” answer. Things may be great, and they often are. But sometimes they may be going off track like Han Solo in a botched princess rescue, bluffing their way by saying, “Everything’s fine, we’re all fine here now, thank you…How are you?”

So you can rely on the data to show you how things are truly going.

What to do:

  • Use the data to understand the dynamics of a new hire’s review engagement. By looking at PRs Submitted and PRs Reviewed, you can identify where new hires are interacting with their teammates. If you’ve paired engineers together, you can see where those interactions are going well, as well as where new hires have jumped in on other engineers’ work too. After all, commenting on multiple people’s work will build rapport within the team over time.
  • The data can also help you understand the quality of their comments in the review process by examining the responsiveness and review coverage of both the new hires and the experienced developers. This will demonstrate the traction the new hires are getting within the team, and where engineers may be dragging their feet, you can use this opportunity to remind them about the importance of welcoming new team members into the fold.

Remember that PRs are about so much more than finding bugs. While ramping up, as well as throughout engineers’ time with the team, code review is a fantastic way to reinforce cultural values. By encouraging engineers to participate more in the PR process, not only are you encouraging best practices, you’re also empowering them to speak their minds and use their voices. “You were brought onto the team for good reasons,” you’re saying. “Your team wants to hear what you have to contribute.”

Participation as a newcomer also reinforces the notion for the established engineers that they need to be open to receiving and accepting feedback. “This is an important part of who you are,” you’re saying. “And this is an important value for this organization.”

In short, PRs are one of the best ways not to just talk about your values but to actually live them, every day.

And that’s true not just in the onboarding process, but in your day-to-day, too. Developing software is no longer about individual engineers cranking out code (if it ever was). It’s a team effort. By focusing on these dynamics of the code review process and utilizing the data available to you, you can improve the value your team gleans from the code review process—and start guiding better engineers and better products as a result.

Conclusion

Ultimately, code review is about so much more than just catching errors. It’s a place where the team can work together to create even better solutions for customers. The review process is where team members can share knowledge, provide feedback, learn from one another, and build a culture that supports healthy collaboration patterns.

And managers can provide the most high-value contributions not by participating in those reviews, but by looking at the process as a whole and noticing quick wins and areas where the team could work better together. A leader’s role is to remove obstacles that block developers from doing their best work, and to coach team members toward healthy work and collaborate patterns. They work on the process, rather than in the process.

Tags:

0

Software Engineer interview Prep. Master list

by Deepak Dhakal 2. July 2021 00:46

System Design Interview Prep Master Doc

Compiled by - Pooja Biswas

pooja biswas | LinkedIn

All system design youtube channels

  1. https://www.youtube.com/user/dimakorolev/videos

  2. https://www.youtube.com/c/ByteByByte/videos

  3. https://www.youtube.com/channel/UC_n-A84J0UcU5uq4sEh2CnQ

  4. https://www.youtube.com/c/DefogTech/videos

  5. https://www.youtube.com/c/HusseinNasser-software-engineering/videos

  6. https://www.youtube.com/c/SystemDesignInterview/videos

  7. https://www.youtube.com/c/sudoCODE/videos

  8. https://www.youtube.com/c/codeKarle/videos

  9. https://www.youtube.com/c/CMUDatabaseGroup/videos

  10. https://www.youtube.com/c/interviewingio/videos

  11. https://www.youtube.com/c/GauravSensei/videos

  12. https://www.youtube.com/c/EngineeringwithUtsav/videos

  13. https://www.youtube.com/channel/UClB4KPy5LkJj1t3SgYVtMOQ/videos

  14. https://www.youtube.com/c/ExponentTV/videos

  15. https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39_tb2fErI4-WkMbsvGQk9_UB

  16. https://www.youtube.com/c/TheInterviewSage/videos

  17. https://www.youtube.com/c/ThinkSoftware/videos

  18. https://www.youtube.com/c/SuccessinTech/videos

  19. https://www.youtube.com/c/TechDummiesNarendraL/videos

                  20. https://www.youtube.com/c/OktaDev/videos



Important Books :

  1. Amazon.com: System Design Interview – An Insider's Guide eBook: Xu, Alex: Kindle Store

  2. Amazon.com: Operating Systems: Three Easy Pieces eBook: Arpaci-Dusseau, Remzi, Arpaci-Dusseau, Andrea: Kindle Store

  3. Web Scalability for Startup Engineers: Ejsmont, Artur: 9780071843652: Amazon.com: Books

  4. Understanding Distributed Systems: What every developer should know about large distributed applications: Vitillo, Roberto: 9781838430207: Amazon.com: Books

  5. Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems: Kleppmann, Martin: 9781449373320: Amazon.com: Books

  6. Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale: 9781491936160: Computer Science Books @ Amazon.com

  7. Buy Distributed Algorithms – An Intuitive Approach 2e (The MIT Press) Book Online at Low Prices in India | Distributed Algorithms – An Intuitive Approach 2e (The MIT Press) Reviews & Ratings - Amazon.in

  8. Elasticsearch: The Definitive Guide: A Distributed Real-Time Search and Analytics Engine eBook: Gormley, Clinton, Tong, Zachary: Amazon.in: Kindle Store

  9. Buy Cassandra – The Definitive Guide, 3e: Distributed Data at Web Scale Book Online at Low Prices in India | Cassandra – The Definitive Guide, 3e: Distributed Data at Web Scale Reviews & Ratings - Amazon.in

Imp keywords

 

  1. Write ahead logging

  2. SSL passthrough / termination on load balancers

  3. Clock vectors

  4. Hinted handoff

  5. Bloom filters

  6. Gossip protocols

  7. Merkle trees

  8. Two phase commit

  9. Two phase locking

  10. Consistent hashing

  11. Data replication

  12. Total order broadcast

  13. Isolation levels (read uncomitted, read comitted, repeatable read, serializable)

  14. Quad trees (GeoHashin)

  15. Inverse indexing - Google search/any search indexing

  16. Gaming ranking - rank players based on score and faster.

  17. Word search in trie - auto suggestions system design

  18. Sort 5gb data in 1gb memory - merge sort

  19. Paxos algo

  20. Merkle Tree

  21. Backpressure

  22. Circuit breaker

  23. Raft

  24. Service discovery

  25. Saga

  26. Hyperloglog

 

 

 

Important algorithms

https://github.com/resumejob/system-design-algorithms

  • Frugal Streaming

  • Geohash / S2 Geometry

  • Leaky bucket / Token bucket

  • Loosy Counting

  • Operational transformation

  • Quadtree / Rtree

  • Ray casting

  • Reverse index

  • Rsync algorithm

  • Trie algorithm















Cloud design patterns




https://docs.microsoft.com/en-us/azure/architecture/patterns/index-patterns

Cloud arch pattern

 

Ambassador

Create helper services that send network requests on behalf of a consumer service or application.

Design and Implementation,


Operational Excellence

Anti-Corruption Layer

Implement a façade or adapter layer between a modern application and a legacy system.

Design and Implementation,


Operational Excellence

Asynchronous Request-Reply

Decouple backend processing from a frontend host, where backend processing needs to be asynchronous, but the frontend still needs a clear response.

Messaging

Backends for Frontends

Create separate backend services to be consumed by specific frontend applications or interfaces.

Design and Implementation

Bulkhead

Isolate elements of an application into pools so that if one fails, the others will continue to function.

Reliability

Cache-Aside

Load data on demand into a cache from a data store

Data Management,


Performance Efficiency

Choreography

Let each service decide when and how a business operation is processed, instead of depending on a central orchestrator.

Messaging,


Performance Efficiency

Circuit Breaker

Handle faults that might take a variable amount of time to fix when connecting to a remote service or resource.

Reliability

Claim Check

Split a large message into a claim check and a payload to avoid overwhelming a message bus.

Messaging

Compensating Transaction

Undo the work performed by a series of steps, which together define an eventually consistent operation.

Reliability

Competing Consumers

Enable multiple concurrent consumers to process messages received on the same messaging channel.

Messaging

Compute Resource Consolidation

Consolidate multiple tasks or operations into a single computational unit

Design and Implementation

CQRS

Segregate operations that read data from operations that update data by using separate interfaces.

Data Management,


Design and Implementation,


Performance Efficiency

Deployment Stamps

Deploy multiple independent copies of application components, including data stores.

Reliability,


Performance Efficiency

Event Sourcing

Use an append-only store to record the full series of events that describe actions taken on data in a domain.

Data Management,


Performance Efficiency

External Configuration Store

Move configuration information out of the application deployment package to a centralized location.

Design and Implementation,


Operational Excellence

Federated Identity

Delegate authentication to an external identity provider.

Security

Gatekeeper

Protect applications and services by using a dedicated host instance that acts as a broker between clients and the application or service, validates and sanitizes requests, and passes requests and data between them.

Security

Gateway Aggregation

Use a gateway to aggregate multiple individual requests into a single request.

Design and Implementation,


Operational Excellence

Gateway Offloading

Offload shared or specialized service functionality to a gateway proxy.

Design and Implementation,


Operational Excellence

Gateway Routing

Route requests to multiple services using a single endpoint.

Design and Implementation,


Operational Excellence

Geodes

Deploy backend services into a set of geographical nodes, each of which can service any client request in any region.

Reliability,


Operational Excellence

Health Endpoint Monitoring

Implement functional checks in an application that external tools can access through exposed endpoints at regular intervals.

Reliability,


Operational Excellence

Index Table

Create indexes over the fields in data stores that are frequently referenced by queries.

Data Management,


Performance Efficiency

Leader Election

Coordinate the actions performed by a collection of collaborating task instances in a distributed application by electing one instance as the leader that assumes responsibility for managing the other instances.

Design and Implementation,


Reliability

Materialized View

Generate prepopulated views over the data in one or more data stores when the data isn't ideally formatted for required query operations.

Data Management,


Operational Excellence

Pipes and Filters

Break down a task that performs complex processing into a series of separate elements that can be reused.

Design and Implementation,


Messaging

Priority Queue

Prioritize requests sent to services so that requests with a higher priority are received and processed more quickly than those with a lower priority.

Messaging,


Performance Efficiency

Publisher/Subscriber

Enable an application to announce events to multiple interested consumers asynchronously, without coupling the senders to the receivers.

Messaging

Queue-Based Load Leveling

Use a queue that acts as a buffer between a task and a service that it invokes in order to smooth intermittent heavy loads.

Reliability,


Messaging,


Resiliency,


Performance Efficiency

Retry

Enable an application to handle anticipated, temporary failures when it tries to connect to a service or network resource by transparently retrying an operation that's previously failed.

Reliability

Scheduler Agent Supervisor

Coordinate a set of actions across a distributed set of services and other remote resources.

Messaging,


Reliability

Sequential Convoy

Process a set of related messages in a defined order, without blocking processing of other groups of messages.

Messaging

Sharding

Divide a data store into a set of horizontal partitions or shards.

Data Management,


Performance Efficiency

Sidecar

Deploy components of an application into a separate process or container to provide isolation and encapsulation.

Design and Implementation,


Operational Excellence

Static Content Hosting

Deploy static content to a cloud-based storage service that can deliver them directly to the client.

Design and Implementation,


Data Management,


Performance Efficiency

Strangler Fig

Incrementally migrate a legacy system by gradually replacing specific pieces of functionality with new applications and services.

Design and Implementation,


Operational Excellence

Throttling

Control the consumption of resources used by an instance of an application, an individual tenant, or an entire service.

Reliability,


Performance Efficiency

Valet Key

Use a token or key that provides clients with restricted direct access to a specific resource or service.

Data Management,


Security

 

 

 

 

Data Management patterns

  • 06/23/2017

  • +3

Data management is the key element of cloud applications, and influences most of the quality attributes. Data is typically hosted in different locations and across multiple servers for reasons such as performance, scalability or availability, and this can present a range of challenges. For example, data consistency must be maintained, and data will typically need to be synchronized across different locations.

Additionally data should be protected at rest, in transit, and via authorized access mechanisms to maintain security assurances of confidentiality, integrity, and availability. Refer to the Azure Security Benchmark Data Protection Control for more information.

Pattern

Summary

Cache-Aside

Load data on demand into a cache from a data store

CQRS

Segregate operations that read data from operations that update data by using separate interfaces.

Event Sourcing

Use an append-only store to record the full series of events that describe actions taken on data in a domain.

Index Table

Create indexes over the fields in data stores that are frequently referenced by queries.

Materialized View

Generate prepopulated views over the data in one or more data stores when the data isn't ideally formatted for required query operations.

Sharding

Divide a data store into a set of horizontal partitions or shards.

Static Content Hosting

Deploy static content to a cloud-based storage service that can deliver them directly to the client.

Valet Key

Use a token or key that provides clients with restricted direct access to a specific resource or service.

DATA MANAGEMENT PATTERNS

 

 


Design and Implementation patterns

  • 06/23/2017

  • 2 minutes to read

Good design encompasses factors such as consistency and coherence in component design and deployment, maintainability to simplify administration and development, and reusability to allow components and subsystems to be used in other applications and in other scenarios. Decisions made during the design and implementation phase have a huge impact on the quality and the total cost of ownership of cloud hosted applications and services.

Pattern

Summary

Ambassador

Create helper services that send network requests on behalf of a consumer service or application.

Anti-Corruption Layer

Implement a façade or adapter layer between a modern application and a legacy system.

Backends for Frontends

Create separate backend services to be consumed by specific frontend applications or interfaces.

CQRS

Segregate operations that read data from operations that update data by using separate interfaces.

Compute Resource Consolidation

Consolidate multiple tasks or operations into a single computational unit

External Configuration Store

Move configuration information out of the application deployment package to a centralized location.

Gateway Aggregation

Use a gateway to aggregate multiple individual requests into a single request.

Gateway Offloading

Offload shared or specialized service functionality to a gateway proxy.

Gateway Routing

Route requests to multiple services using a single endpoint.

Leader Election

Coordinate the actions performed by a collection of collaborating task instances in a distributed application by electing one instance as the leader that assumes responsibility for managing the other instances.

Pipes and Filters

Break down a task that performs complex processing into a series of separate elements that can be reused.

Sidecar

Deploy components of an application into a separate process or container to provide isolation and encapsulation.

Static Content Hosting

Deploy static content to a cloud-based storage service that can deliver them directly to the client.

Strangler Fig

Incrementally migrate a legacy system by gradually replacing specific pieces of functionality with new applications and services.

DESIGN AND IMPLEMENTATION PATTERNS

 

 

Messaging patterns

+4

The distributed nature of cloud applications requires a messaging infrastructure that connects the components and services, ideally in a loosely coupled manner in order to maximize scalability. Asynchronous messaging is widely used, and provides many benefits, but also brings challenges such as the ordering of messages, poison message management, idempotency, and more.

Pattern

Summary

Asynchronous Request-Reply

Decouple backend processing from a frontend host, where backend processing needs to be asynchronous, but the frontend still needs a clear response.

Claim Check

Split a large message into a claim check and a payload to avoid overwhelming a message bus.

Choreography

Have each component of the system participate in the decision-making process about the workflow of a business transaction, instead of relying on a central point of control.

Competing Consumers

Enable multiple concurrent consumers to process messages received on the same messaging channel.

Pipes and Filters

Break down a task that performs complex processing into a series of separate elements that can be reused.

Priority Queue

Prioritize requests sent to services so that requests with a higher priority are received and processed more quickly than those with a lower priority.

Publisher-Subscriber

Enable an application to announce events to multiple interested consumers asynchronously, without coupling the senders to the receivers.

Queue-Based Load Leveling

Use a queue that acts as a buffer between a task and a service that it invokes in order to smooth intermittent heavy loads.

Scheduler Agent Supervisor

Coordinate a set of actions across a distributed set of services and other remote resources.

Sequential Convoy

Process a set of related messages in a defined order, without blocking processing of other groups of messages.

MESSAGING PATTERNS

 

 

 

Book https://learning.oreilly.com/library/view/cloud-architecture-patterns/9781449357979/




LEETCODE all system design problems

  1. Web Crawler']

https://leetcode.com/discuss/interview-question/system-design/124657/Facebook-or-System-Design-or-A-web-crawler-that-will-crawl-Wikipedia

  1. Detect web crawler https://leetcode.com/discuss/interview-question/system-design/548816/Amazon-or-System-Design-or-Web-Crawler-Detector

  2. Yelp

  3. Distributed file system

  4. URL shortening and Pastebin https://leetcode.com/discuss/interview-question/system-design/124804/Design-Pastebin

https://leetcode.com/discuss/interview-question/system-design/124658/Design-URL-Shortening-service-like-TinyURL

  1. Instagram

https://leetcode.com/discuss/interview-question/system-design/124802/Design-Instagram

https://leetcode.com/discuss/interview-question/system-design/586749/design-Instagram

https://leetcode.com/discuss/interview-question/system-design/719253/Design-Facebook-%3A-System-Design-Interview

  1. Dropbox.

  2. Twitter

https://leetcode.com/discuss/interview-question/system-design/124689/Design-twitter

  1. Redis https://leetcode.com/discuss/interview-question/system-design/125751/Design-a-distributed-cache-system

  2. Youtube or Netflix https://leetcode.com/discuss/interview-question/system-design/733520/Design-YouTube-Very-detailed-design-with-diagrams

https://leetcode.com/discuss/interview-question/system-design/144287/Design-Recommendation-System-for-Amazon-Videos

https://leetcode.com/discuss/interview-question/system-design/600861/System-Design-Youtube-add-click-counts

https://leetcode.com/discuss/interview-question/system-design/557250/Design-a-video-streaming-service-to-support-playback-video-from-different-devices

https://leetcode.com/discuss/interview-question/system-design/496042/Design-video-sharing-platform-like-Youtube

https://leetcode.com/discuss/interview-question/system-design/158698/Distributed-database%3A-Netflix

https://leetcode.com/discuss/interview-question/system-design/124565/Design-Netflix-recommendation-engine

https://leetcode.com/discuss/interview-question/system-design/150607/Design-youtube

  1. Ticketmaster

https://leetcode.com/discuss/interview-question/system-design/124803/Design-BookMyShow

https://leetcode.com/discuss/interview-question/system-design/315763/System-Design-or-Seat-reservation-application-like-Ticket-Master-or-BookMyShow

  1. Facebook Messenger or WhatsApp https://leetcode.com/discuss/interview-question/system-design/585930/Amazon-or-System-Design-or-Design-a-Chat-Service

https://leetcode.com/discuss/interview-question/system-design/220073/How-would-you-design-WhatsApp

https://leetcode.com/discuss/interview-question/system-design/124613/Amazon-or-System-Design-or-A-scalable-chat-application-on-phone-browsing

  1. Typeahead suggesions.

  2. Twitter search.

  3. Newsfeed ranking

https://leetcode.com/discuss/interview-question/system-design/349627/How-do-you-design-a-meta-data-for-a-news-feed

https://leetcode.com/discuss/interview-question/system-design/153871/Design-a-News-Feed-system-(like-Facebook-Linkedin-etc.)

  1. Web search.

  2. LinkedIn "you may know ..."., https://leetcode.com/discuss/interview-question/system-design/1036762/Google-Onsite-System-Design-How-to-do-it

https://leetcode.com/discuss/interview-question/system-design/153941/Design-the-%22People-You-May-Know%22-feature-on-LinkedIn-or-Facebook.

  1. Uber / Luxe (anti-uber)?

  2. Freight / delivery orchestration? (edited) 

  3. Botnet/decentralized web crawler/torrent 

https://leetcode.com/discuss/interview-question/system-design/594844/System-design-question-Help-needed

https://leetcode.com/discuss/interview-question/system-design/464997/Design-a-P2P-file-sharing-application-like-BitTorrent

  1. Coupon redeeming system

https://leetcode.com/discuss/interview-question/system-design/353302/Design-a-couponvoucher-management-system-or-DellEMC

https://leetcode.com/discuss/interview-question/system-design/459593/Facebook-or-System-Design-or-E-commerce-Apply-discount-on-every-nth-order

  1. Message queue kafka, service bus

https://leetcode.com/discuss/interview-question/system-design/124761/Deciding-which-queue-to-send-a-post-to

https://leetcode.com/discuss/interview-question/system-design/206134/Amazon-or-System-Design-or-Design-a-Distributed-Message-queue

https://leetcode.com/discuss/interview-question/system-design/734303/Microsoftor-Design-an-Enterprise-Service-Bus

  1. Rate limiter https://leetcode.com/discuss/interview-question/system-design/637402/Design-a-efficient-client-side-rate-limit-handler

https://leetcode.com/discuss/interview-question/system-design/124558/Uber-or-Rate-Limiter

  1. Design leetcode - asked in amazon and fb. https://leetcode.com/discuss/interview-question/system-design/649021/Design-Leetcode

  2. https://leetcode.com/discuss/interview-question/system-design/409736/Facebook-or-System-Design-or-Hacker-Rank-LeetCode-Contest-Leadership-Board-System

https://leetcode.com/discuss/interview-question/system-design/308452/System-Design-or-Programming-contest-platform-like-LeetCode

  1. Large log data collection and processing system 

https://leetcode.com/discuss/interview-question/system-design/124603/Amazon-or-Phone-screen-or-How-to-handle-large-log-data
https://leetcode.com/discuss/interview-question/system-design/128037/How-would-you-parse-a-huge-log-file

https://leetcode.com/discuss/interview-question/system-design/189030/Design-a-system-which-can-report-frequently-occurring-exceptions-on-a-dashboard

https://leetcode.com/discuss/interview-question/system-design/196142/Copy-coredump-files-from-millions-of-system-to-single-Storage-server-like-S3

https://leetcode.com/discuss/interview-question/system-design/431023/Google-or-Onsite-or-Get-all-logs-between-times

https://leetcode.com/discuss/interview-question/system-design/440546/Facebook-or-System-Design-Onsite-or-Compute-Percentile-Metrics-Over-Time-Series

https://leetcode.com/discuss/interview-question/system-design/124603/Amazon-or-Phone-screen-or-How-to-handle-large-log-data

https://leetcode.com/discuss/interview-question/system-design/1133962/Service-which-will-download-data-from-multiple-sources-and-ingests-it-in-the-system

https://leetcode.com/discuss/interview-question/system-design/942087/System-Design%3A-Design-a-system-to-process-data-in-different-formats-from-different-sources

https://leetcode.com/discuss/interview-question/system-design/852238/Need-help-with-System-Design-problem-asked-in-a-real-interview

https://leetcode.com/discuss/interview-question/system-design/820877/Bloomberg-System-Design

https://leetcode.com/discuss/interview-question/system-design/778868/Facebook-oror-Onsite-oror-System-Design-Aggregation-click-events

https://leetcode.com/discuss/interview-question/system-design/725364/System-Design-or-IOT-sensor-data-aggregator

https://leetcode.com/discuss/interview-question/system-design/202946/Design-a-system-to-aggregate-metrics-from-large-cluster(800%2B)-of-web-servers

  1. Realtime stock price monitoring system/ live score update cricbuzz, realtime gaming score

https://leetcode.com/discuss/interview-question/system-design/625918/Amazon-or-System-Design-or-Design-a-real-time-gaming-ranking-system

https://leetcode.com/discuss/interview-question/system-design/431712/Bloomberg-or-Design-a-system-to-give-prices-of-a-stock

  1. Stock trading system https://medium.com/@narengowda/stock-exchange-system-design-answered-ad4be1345851

https://leetcode.com/discuss/interview-question/system-design/820877/Bloomberg-System-Design

https://leetcode.com/discuss/interview-question/system-design/124794/Design-a-Multicurrency-trading-system

https://leetcode.com/discuss/interview-question/system-design/490034/FAANG-or-Onsite-or-Intern-or-System-Design-Stock

  1. Kill switch for stopping stock trading https://leetcode.com/discuss/interview-question/system-design/124553/Kill-Switch

  2. Design network fail over

https://leetcode.com/discuss/interview-question/system-design/124598/Design-network-fail-over

  1. Design AB testing framework https://leetcode.com/discuss/interview-question/system-design/124595/AB-Testing

https://leetcode.com/discuss/interview-question/system-design/228661/Design-a-Data-Experimentation-platform

  1. Design parking lot system https://leetcode.com/discuss/interview-question/system-design/124576/Design-a-parking-lot-system.

https://leetcode.com/discuss/interview-question/system-design/575186/Design-a-Parking-Spot-System

https://leetcode.com/discuss/interview-question/system-design/598634/Microsoft-or-Onsite-or-System-Design-or-SDE-2

https://leetcode.com/discuss/interview-question/system-design/850712/System-Design-Amazon-2020-(SDE-2)

https://leetcode.com/discuss/interview-question/system-design/765686/System-Design-Interview-Question%3A-Parking-Lot-or-Low-Level-Design

https://leetcode.com/discuss/interview-question/system-design/125260/Parking-Lots-Design

  1. Reccomendation Engine https://leetcode.com/discuss/interview-question/system-design/124565/Design-Netflix-recommendation-engine

  2. Smart voice assistant like siri, alexa

https://leetcode.com/discuss/interview-question/system-design/124566/Design-AlexaSiriGoogle-Home-Architecture

https://leetcode.com/discuss/interview-question/system-design/848252/Amazon-System-Design

  1. Nearest store location, another variation of topk https://leetcode.com/discuss/interview-question/system-design/124567/Nearest-Store-Locators

https://leetcode.com/discuss/interview-question/system-design/533061/How-to-implement-nearest-location-kind-of-functionality-in-a-google-map-type-application

https://leetcode.com/discuss/interview-question/system-design/154172/Design-google-map-database

  1. Job scheduling sytem https://leetcode.com/discuss/interview-question/system-design/124697/Walmartlabs-onsite

https://leetcode.com/discuss/interview-question/system-design/124786/Google-Scheduling-Job-Involving-both-RAM-and-CPU

https://leetcode.com/discuss/interview-question/system-design/692996/Microsoft-System-Design-Please-help

https://leetcode.com/discuss/interview-question/system-design/553563/Googleor-Distributed-SystemorPerformance

https://leetcode.com/discuss/interview-question/system-design/344524/Amazon-or-Design-a-JobTask-Scheduler

https://leetcode.com/discuss/interview-question/system-design/124672/Implement-a-task-scheduler

  1. Elevator system

https://leetcode.com/discuss/interview-question/system-design/149264/Design-an-Elevator-system

  1. Malware detection system https://leetcode.com/discuss/interview-question/system-design/1019028/FB-or-System-Design-or-Multi-Engine-Malware-Analyzer

https://leetcode.com/discuss/interview-question/system-design/150610/Design-a-malware-detection-system

  1. Garbage collector

  2. Google docs https://leetcode.com/discuss/interview-question/system-design/148187/System-Design-or-Google-Docs

https://leetcode.com/discuss/interview-question/system-design/148187/System-Design-or-Google-Docs

https://leetcode.com/discuss/interview-question/system-design/208207/Design-a-Google-Sheet-System

https://leetcode.com/discuss/interview-question/system-design/349669/Google-SWE-L5-or-Onsite-or-Design-Google-Docs-Versioning-System

https://leetcode.com/discuss/interview-question/system-design/322448/Content-Management-System-Design

https://leetcode.com/discuss/interview-question/system-design/194402/Design-a-file-sharing-system

  1. Ecommerce Price checker system https://leetcode.com/discuss/interview-question/system-design/140742/E-commerce-(Amazon)Website-looking-into-other-competitor-Website-products-prices-and-update

  2. Notification system https://leetcode.com/discuss/interview-question/system-design/138097/Design-Notification-Service-for-Amazon-Alexa

  3. Online ludo game 

  4. metric monitoring service

  5. Ecommerce site ,Shopping cart, product catalog, payment gateway 

https://leetcode.com/discuss/interview-question/system-design/211415/Interview-Question-Ecommerce-System-design-(-Eg-%3A-Amazon-)%3A-Concurrency-issues-handling

https://leetcode.com/discuss/interview-question/system-design/589546/Amazon-or-System-Design-or-Amazon-Order-System

https://leetcode.com/discuss/interview-question/system-design/675539/System-Design-question-asked-in-interview

https://leetcode.com/discuss/interview-question/system-design/666792/Microsoft-or-System-design-or-Please-help

https://leetcode.com/discuss/interview-question/system-design/1124722/System-Design-or-Shopping-Cart-or-Payment-Gateway-or-Product-Catalog

https://leetcode.com/discuss/interview-question/system-design/886390/Design-Recommendation-API-or-Akamai-Interview

https://leetcode.com/discuss/interview-question/system-design/776927/Design-an-accountpayment-system

https://leetcode.com/discuss/interview-question/system-design/706038/System-Design-Payment-System-Wallet-system-Payment-gateway

  1. Seller summary page https://leetcode.com/discuss/interview-question/system-design/124612/Phone-Interview-Question%3A-Design-an-Seller-Summary-Page

  2. Customer who bought this also bought https://leetcode.com/discuss/interview-question/system-design/124557/Amazon's-%22Customers-who-bought-this-item-also-bought%22-recommendation-system

  3. Distributed key value store, https://leetcode.com/discuss/interview-question/system-design/1120468/Design-Assignment-or-Implement-a-distributed-Key-Value-(KV)-store-or-SE-Role-Avalara

https://leetcode.com/discuss/interview-question/system-design/747591/Amazon-or-Onsite-or-System-design-or-Please-help

  1. Facebook live commenting

https://leetcode.com/discuss/interview-question/system-design/583184/FBInstagram-'Live-Comments'-System-design

  1. Facebook status search

  2. Image editing ( asked in fb 2021) https://leetcode.com/discuss/interview-question/system-design/1077411/Facebook-or-Onsite-2021-or-System-Design-or-Design-image-editing

  3. File download application system https://leetcode.com/discuss/interview-question/system-design/1071562/Design-a-File-Download-Application-System

  4. Proximity server https://leetcode.com/discuss/interview-question/system-design/923677/Facebook-or-System-Design

  5. Top N songs, another top k problem

https://leetcode.com/discuss/interview-question/system-design/124702/Design-a-service-to-calculate-the-top-k-listened-songs-in-past-24-hours

https://leetcode.com/discuss/interview-question/system-design/243604/Design-a-real-time-dashboard-showing-the-most-played-songs

  1. Privacy setting at facebook

  2. Distributed configuration management system

  3. Design gmail https://leetcode.com/discuss/interview-question/system-design/1014986/Google-or-Onsite-or-System-Design%3A-Design-an-Email-system-like-GMAIL

  4. News reading feature in alexa https://leetcode.com/discuss/interview-question/system-design/1014181/Amazon-or-System-Design-or-SDE2

  5. Ads click visualisation system https://leetcode.com/discuss/interview-question/system-design/1002923/Facebook-or-Online-or-Real-time-data-visualization-for-ads-clicks

  6. IoT devices management system https://leetcode.com/discuss/interview-question/system-design/974890/Design-a-system-for-management-of-IOT-devices

  7. Timer service https://leetcode.com/discuss/interview-question/system-design/973207/System-Design-or-Timer-service

  8. Service monitoring and alerting system like pagerduty, azure monitor etc
    https://leetcode.com/discuss/interview-question/system-design/958919/System-Design-Interview-or-Service-Health-Monitoring-and-Alerting-Service

https://leetcode.com/discuss/interview-question/system-design/287678/Design-a-monitoring-or-analytics-service-like-Datadog-or-SignalFx

  1. Load balancer https://leetcode.com/discuss/interview-question/system-design/943352/Facebook-or-E5-System-Design-Interview-Question-or-Menlo-Park

  2. Design undergeound system 

  3. Leader board table design https://leetcode.com/discuss/interview-question/system-design/892083/Leaderboard-table-system-design-for-online-game

  4. Slot booking system for playarena etc https://leetcode.com/discuss/interview-question/system-design/880581/Event-Booking-for-playarenas-Low-level-design

https://leetcode.com/discuss/interview-question/system-design/423613/Amazon-or-Phone-Screen-or-Design-Restaurant-Reservation-System

  1. Food delivery app https://leetcode.com/discuss/interview-question/system-design/874074/Food-Delivery-App-or-Low-Level-Design-or-Interview-Question

  2. Online gaming lobby service https://leetcode.com/discuss/interview-question/system-design/874074/Food-Delivery-App-or-Low-Level-Design-or-Interview-Question

  3. URL fishing varifier https://leetcode.com/discuss/interview-question/system-design/896312/Google-system-design

  4. Design whatsapp/instagram story

https://leetcode.com/discuss/interview-question/system-design/388222/Snapchat-or-System-Design-or-Instagram-Story-Feature

  1. File sharing with collaborative editing https://leetcode.com/discuss/interview-question/system-design/838085/File-sharing-service-with-collaborative-editing-or-Amazon

https://leetcode.com/discuss/interview-question/system-design/824659/Intuit-or-Long-Poll-vs-Web-socket-vs-Server-send-Events

  1. Stack overflow tags https://leetcode.com/discuss/interview-question/system-design/838025/Design-a-tagging-system-like-tags-used-in-stack-overflow

https://leetcode.com/discuss/interview-question/system-design/307558/Design-Stack-Overflow

  1. Tinyurls https://leetcode.com/discuss/interview-question/system-design/838012/URL-Shortener-or-MD5-or-How-to-deal-with-collisions-or-FinTech-startup

  2. Github like cloud repo https://leetcode.com/discuss/interview-question/system-design/837383/System-design-of-code-repository-like-github 

  3. Job posting site https://leetcode.com/discuss/interview-question/system-design/811840/Job-listing-storage-and-search

  4. S3/cloud object store https://leetcode.com/discuss/interview-question/system-design/811503/System-design-Object-store-design-like-S3GCS

  5. Celebrity timeline generation https://leetcode.com/discuss/interview-question/system-design/810561/Timeline-generation-for-celebrities-or-System-Design-or-Google

  6. Kindle service 

  7. Bidding system https://leetcode.com/discuss/interview-question/system-design/792060/Bidding-System%3A-System-Design-Interview

  8. Billing system - asked in fb interview

  9. RPC system for client server comm https://leetcode.com/discuss/interview-question/system-design/790034/Client-Server-Communication%3A-System-Design-Interview

  10. Autonomous driving system https://leetcode.com/discuss/interview-question/system-design/789961/Design-a-cloud-based-simulationvisualization-platform-for-a-self-driving-cars-company

  11. Github code search https://leetcode.com/discuss/interview-question/system-design/789015/Github-%3A-Design-search-feature-in-Github-scale-code-repository

  12. Car showroom https://leetcode.com/discuss/interview-question/system-design/785960/Amazon-System-Design-Question

  13. Airport boarding gate security https://leetcode.com/discuss/interview-question/system-design/785960/Amazon-System-Design-Question

  14. Social graph https://leetcode.com/discuss/interview-question/system-design/782906/Design-a-social-graph

  15. Place of interest https://leetcode.com/discuss/interview-question/system-design/777945/Design-a-system-to-source-store-and-display-places-of-interest

  16. Tinder https://leetcode.com/discuss/interview-question/system-design/774870/Tinder-System-Design-or-Online-Dating-App-System-Design

  17. Grocery store https://leetcode.com/discuss/interview-question/system-design/769578/Amazon-orSystem-Design-or-Amazon-Go-or-suggestion-on-solution-welcome

https://leetcode.com/discuss/interview-question/system-design/467655/Amazon-Onsite-or-System-Design-Pickup-Delivery-System-For-Groceries

  1. Display ads https://leetcode.com/discuss/interview-question/system-design/761814/Design-number-of-ads-to-show-to-users-on-a-google-search

  2. Add badge to peoples spotify account https://leetcode.com/discuss/interview-question/system-design/748408/How-to-design-a-system-to-add-badges-to-people's-Spotify-account

  3. Xml to json conversation https://leetcode.com/discuss/interview-question/system-design/743624/System-design-question%3A-Amazon-SDE2%3A-Large-xml-files-to-json-conversion

  4. Ocr web app https://leetcode.com/discuss/interview-question/system-design/741676/System-Design%3A-OCR-web-app

  5. High scale otp generation system https://leetcode.com/discuss/interview-question/system-design/728464/Microsoft-or-Onsite-or-Modify-an-OTP-generation-system-to-handle-more-requests

  6. Distributed counter https://leetcode.com/discuss/interview-question/system-design/685310/Microsoft-virtual-or-Design-distributed-counter

https://leetcode.com/discuss/interview-question/system-design/277606/Design-a-performance-counter

  1. Scan for viruses in uploaded file https://leetcode.com/discuss/interview-question/system-design/659875/Design-a-system-where-client-can-upload-a-file-and-viruses-need-to-be-scanned

  2. Log processing at scale https://leetcode.com/discuss/interview-question/system-design/622704/Design-a-system-to-store-and-retrieve-logs-for-all-of-eBay

  3. London travel card system https://leetcode.com/discuss/interview-question/system-design/617408/Marshall-Wace-or-Onsite-or-How-would-you-design-Oyster-(London-Travel-Card-system)tion

  4. Payment system for newyork MTA https://leetcode.com/discuss/interview-question/system-design/305388/Design-a-transportation-payment-System.

  5. Ad click counter https://leetcode.com/discuss/interview-question/system-design/584458/Facebook-or-System-Design-or-Ad-Click-Counter

  6. Design slack https://leetcode.com/discuss/interview-question/system-design/582975/Design-Slack

https://leetcode.com/discuss/interview-question/system-design/339849/System-Design-or-Slack

  1. Wikipedia https://leetcode.com/discuss/interview-question/system-design/574872/Wikipedia-or-DBsystem-design-thoughts

https://leetcode.com/discuss/interview-question/system-design/174380/Uber-design-question-Design-Wikipeida

  1. ML related system design https://leetcode.com/discuss/interview-question/system-design/566057/Machine-Learning-System-Design-%3A-A-framework-for-the-interview-day

  2. Windows update https://leetcode.com/discuss/interview-question/system-design/560512/System-Design-Question

  3. People also searched for https://leetcode.com/discuss/interview-question/system-design/559481/Amazon-or-System-design-or-SDE-2-India smiliar to linkedin's you may also know

  4. Cashback processing system https://leetcode.com/discuss/interview-question/system-design/543041/Design-cashback-processing-system

  5. Pub sub arch

  6. Google or amazob book preview https://leetcode.com/discuss/interview-question/system-design/538295/Design-Google-Books-preview-Amazon-Books-look-inside

  7. Design copy right detection https://leetcode.com/discuss/interview-question/system-design/530031/FAANG-Interview-Question-Design-a-copyright-detection-system

  8. Reddit https://leetcode.com/discuss/interview-question/system-design/469900/Netflix-or-System-Design-Web-App-Like-Reddit

  9. Facebook nearby friends https://leetcode.com/discuss/interview-question/system-design/430926/Design-Nearby-Friends

  10. Google photos home page https://leetcode.com/discuss/interview-question/system-design/398523/System-Design-Google-photos-homepage

https://leetcode.com/discuss/interview-question/system-design/396949/System-Design-Google-Photos

  1. Image upload system https://leetcode.com/discuss/interview-question/system-design/391183/Ebay-System-Design-Question

https://leetcode.com/discuss/interview-question/system-design/390503/Google-or-System-Design

  1. Facebook translator service https://leetcode.com/discuss/interview-question/system-design/386322/Design-a-translator-service-for-facebook

https://leetcode.com/discuss/interview-question/system-design/318811/Google-or-System-design-or-Design-a-translation-service-like-Google-Translate

  1. System for health care data https://leetcode.com/discuss/interview-question/system-design/368245/Design-Service-to-Interface-with-Healthcare-Data

  2. Health score app https://leetcode.com/discuss/interview-question/system-design/366754/Amazon-or-System-Design-for-health-score-app

  3. Railway reservation system https://leetcode.com/discuss/interview-question/system-design/364965/Railway-Reservation-System

  4. Treadmill system https://leetcode.com/discuss/interview-question/system-design/362168/Google-or-Onsite-or-Tread-Mill-System-Design

  5. Addressed of entire planet https://leetcode.com/discuss/interview-question/system-design/341980/Amazon-or-System-Design-or-System-to-capture-unique-addresses-in-the-entire-world

  6. Gofundme https://leetcode.com/discuss/interview-question/system-design/336089/System-Design-or-GoFundMe

  7. Shipping fullfilment https://leetcode.com/discuss/interview-question/system-design/320719/Design%3A-Scalable-Shipping-Fulfillment-Center

  8. Flight search API https://leetcode.com/discuss/interview-question/system-design/309853/JSON-structure-for-Flight-search-API

  9. Splitwise https://leetcode.com/discuss/interview-question/system-design/306519/System-Design-or-Splitwise

  10. Google calendar https://leetcode.com/discuss/interview-question/system-design/305654/System-Design-or-Google-Calendar

  11. Fb popular/trending pages https://leetcode.com/discuss/interview-question/system-design/305505/Design-a-most-populartrending-profiles-page

  12. Courier service https://leetcode.com/discuss/interview-question/system-design/301423/Design-a-UPS-style-mail-delivery-system

  13. Hr portal https://leetcode.com/discuss/interview-question/system-design/289092/Design-an-HR-web-portal-for-Amazon's-recruiting-team

  14. Outlook recurring meeting https://leetcode.com/discuss/interview-question/system-design/286891/Design-Outlook-recurring-meeting-system-with-variable-input

  15. Communication system for ecom sites https://leetcode.com/discuss/interview-question/system-design/286457/Design-a-communication-platform.

  16. Imdb https://leetcode.com/discuss/interview-question/system-design/270416/Design-a-movies-reviews-aggregator-system

  17. Realtime event aggregator https://leetcode.com/discuss/interview-question/system-design/270412/Design-a-Real-Time-Event-Aggregation-System

  18. Gpay https://leetcode.com/discuss/interview-question/system-design/270406/Design-a-Payment-System-like-Google-Pay

  19. Top shared post https://leetcode.com/discuss/interview-question/system-design/258398/Design-top-shared-post-system-in-5mins1-hour1-day1-week

  20. Market place analytics https://leetcode.com/discuss/interview-question/system-design/227797/System-Design-E-Commerce-Marketplace-analytics

  21. Top 10 most liked articles https://leetcode.com/discuss/interview-question/system-design/225609/Design-system-which-will-show-top-10-most-liked-articles-within-1524-hours.

  22. Auth for multi tenant https://leetcode.com/discuss/interview-question/system-design/225331/Design-authentication-system-to-multi-tenant-environment

https://leetcode.com/discuss/interview-question/system-design/202958/Multi-Tenant-Saas-Architecture

  1. Design Changefeed https://leetcode.com/discuss/interview-question/system-design/208888/Design-a-system-to-keep-track-of-changes-in-an-SQL-database

  2. Track runners in marathon https://leetcode.com/discuss/interview-question/system-design/200342/Bloomberg%3A-Implement-a-system-to-track-runners-in-a-marathon

  3. Hotel booking page this many people visiting https://leetcode.com/discuss/interview-question/system-design/163204/Design-%22How-Many-people-currently-viewing-the-property%22-for-a-E-Commerce-Hotel-Booking-Site

  4. Text line editor https://leetcode.com/discuss/interview-question/system-design/124679/Implement-a-Text-Line-Editor

  5. Location sharing service https://leetcode.com/discuss/interview-question/system-design/124673/Design-a-Location-Sharing-Android-Application

  6. Build system https://leetcode.com/discuss/interview-question/system-design/124807/Design-a-build-system

  7. Hourly backup from mobile phone https://leetcode.com/discuss/interview-question/system-design/124792/Design-a-system-that-can-handle-hourly-backups-for-mobile-phones

  8. Google help system https://leetcode.com/discuss/interview-question/system-design/125191/Design-the-Google-help-system

 

 

 

 

 

 

 

 

Other interesting post  about specific component design

  1. https://leetcode.com/discuss/interview-question/system-design/136140/Write-a-class-which-is-hard-to-test

  2. Interesting variation of ecart problem where first one who click buys will buy https://leetcode.com/discuss/interview-question/system-design/498895/Startup-interview-or-Designing-tricky-e-shop

  3. Back of envelope calculation https://leetcode.com/discuss/interview-question/system-design/357656/Experience-with-back-of-the-envelope-calculations

  4. Drawing tool https://leetcode.com/discuss/interview-question/system-design/1148896/System-Design-Drawing-Tool-Recos

  5. Very good list https://leetcode.com/discuss/general-discussion/670355/Experienced-Interview-Preparation-Guide-All-Resources

  6. https://leetcode.com/discuss/interview-question/1002218/Facebook-or-Google-or-Top-System-Design-Interview-Questions-(Part-1)

  7. https://github.com/donnemartin/system-design-primer

  8. https://github.com/binhnguyennus/awesome-scalability

  9. Json parser https://leetcode.com/discuss/interview-question/system-design/1052608/Design-a-JSON-Parser-or-SDE2

  10. https://leetcode.com/discuss/interview-question/system-design/1043657/MakeMyTrip-Backend-Developer-or-System-DesignMultithreading

  11. https://leetcode.com/discuss/interview-question/system-design/1042229/Facebook-or-Google-or-Top-System-Design-Interview-Questions-(Part-2)

  12. https://leetcode.com/discuss/interview-question/system-design/1038585/How-do-you-scale-up-an-Application-to-serve-thousands-of-request-per-second

  13. User data access policy design https://leetcode.com/discuss/interview-question/system-design/895268/Google-or-System-Design

  14. https://leetcode.com/discuss/interview-question/system-design/829466/Amazon-or-Phone-or-Seattle-or-Column-Store-vs-Row-Store

  15. https://leetcode.com/discuss/interview-question/system-design/808216/Phone-book-and-search - suffix tree

  16. https://hackernoon.com/scaling-websockets-9a31497af051

  17. Online whiteboard drawing like draw.io

  18. https://leetcode.com/discuss/interview-question/system-design/799474/Virtual-onsite-at-DocuSign

  19. https://leetcode.com/discuss/interview-question/system-design/795890/How-many-transactions-does-Oracle-DB-handle-Read-and-write

  20. How to store Recently viewed item https://leetcode.com/discuss/interview-question/system-design/775139/Amazon-System-Design-customer's-recently-viewed-items

  21. Depth an interviewer can go https://leetcode.com/discuss/interview-question/system-design/773980/Watch-this-before-System-Design-Interview-the-details-an-interviewer-can-go-to-evaluate-candidate

  22. Privacy api https://leetcode.com/discuss/interview-question/system-design/727474/System-design-critique-request-for-below-question

  23. LLD https://leetcode.com/discuss/interview-question/system-design/692383/Google-or-Onsite-or-Design-a-organization-pharmacy-shop-with-managers

  24. T9 predicitve system https://leetcode.com/discuss/interview-question/system-design/685338/Microsoft-or-Onsite-or-Design-the-T9-predictive-text-algorithm-and-system

  25. Storsge of millions of subscriber https://leetcode.com/discuss/interview-question/system-design/680047/How-will-you-store-millions-of-subscribers-list-(assume-it-as-email-id)

  26. Flipkart warehouse portal 

https://leetcode.com/discuss/interview-question/system-design/663037/Amazon-SystemDesign-Flipkart-Suggestions-Design-Warehouse-Portal

  1. https://leetcode.com/discuss/interview-question/system-design/632537/Design-an-algorithm-to-efficiently-transfer-required-bytes-of-data-to-a-single-PC-on-the-network.

  2. https://leetcode.com/discuss/interview-question/system-design/581804/Facebook-System-Design-Preparation - different type of design interviews at facebook

  3. Count current active user on the page, https://leetcode.com/discuss/interview-question/system-design/557603/Postman-or-OA-or-System-Design

https://leetcode.com/discuss/interview-question/system-design/532889/Design-a-counter-for-a-website-which-tells-how-many-visits-happened-to-that-website

  1. Csv parsing at scale https://leetcode.com/discuss/interview-question/system-design/545664/CSV-parsing-at-scale

  2. API to get best selling book https://leetcode.com/discuss/interview-question/system-design/535162/Design-read-api-to-get-best-selling-books

  3. Fb mutual friend https://leetcode.com/discuss/interview-question/system-design/533810/FB-API-System-design

  4. https://leetcode.com/discuss/interview-question/system-design/532089/Update-System-design-of-Amazon-to-handle-10x-times-more-traffic-than-what-it-currently-receives

  5. https://leetcode.com/discuss/interview-question/system-design/513374/FAANG-system-design-interview-question

  6. Sql scalability https://leetcode.com/discuss/interview-question/system-design/507833/SQL-Scalability

  7. Backup from dc1 to dc2 https://leetcode.com/discuss/interview-question/system-design/502522/System-design-to-backup-datacenter1-to-datacenter-2

  8. Order within next 1 hour to get early delivery feature https://leetcode.com/discuss/interview-question/system-design/499558/Amazon-System-Design-Question

  9. https://leetcode.com/discuss/interview-question/484956/design-airport-luggage-handling-system

  10. https://leetcode.com/discuss/interview-question/system-design/483959/Google-onsite-(Theoretical-System-Design)

  11. https://leetcode.com/discuss/interview-question/system-design/421969/Twitch-or-System-Design-Onsite-or-Design-Twitch-Analytics-Use-Case

  12. Good discussion on designing SAAS https://leetcode.com/discuss/interview-question/system-design/385363/Design-Task-Executor-as-Saas

  13. Track down bad request https://leetcode.com/discuss/interview-question/system-design/313117/Design-a-system-capable-of-tracking-bad-request-down-quickly

  14. Facebook https://leetcode.com/discuss/interview-question/system-design/311825/Facebook-System-design

  15. Flight landing with onerunway  https://leetcode.com/discuss/interview-question/system-design/303745/Handling-flight-landing-requests-with-one-runway

  16. Good read on data modelling https://leetcode.com/discuss/interview-question/system-design/295671/Amazon-or-Data-Engineer-Role-or-Database-Design-Question

  17. Traffic light controller https://leetcode.com/discuss/interview-question/system-design/291233/Design-a-traffic-light-controller

  18. Live streaming view count https://leetcode.com/discuss/interview-question/system-design/284232/Livestreaming-view-count.

  19. Random number generator for a slot mschine https://leetcode.com/discuss/interview-question/system-design/281762/Microsoft-or-Design-a-random-number-generator-for-a-slot-machine

  20. Company badge system https://leetcode.com/discuss/interview-question/system-design/279371/Design-a-badge-system

  21. API gateway with graphql https://leetcode.com/discuss/interview-question/system-design/255282/Design-API-gateway

  22. Good read on scaling https://leetcode.com/discuss/interview-question/system-design/250803/Large-amount-of-query-request-in-seconds.

  23. https://leetcode.com/discuss/interview-question/system-design/249988/How-to-Transfer-1-GB-of-data-from-one-microservice-to-another

  24. https://leetcode.com/discuss/interview-question/system-design/234311/Celebrity-Twitter-hack

  25. Amazon locker https://leetcode.com/discuss/interview-question/system-design/233869/Design-Amazon-Locker-system

  26. Upload large file https://leetcode.com/discuss/interview-question/system-design/224398/Algorithm-for-upload-large-file

  27. Currency civersation https://leetcode.com/discuss/interview-question/system-design/216912/Design-an-app-that-converts-USD-to-another-country-currency.

  28. Geohash https://leetcode.com/discuss/interview-question/system-design/203364/How-to-retrieve-addresses-based-on-Latitude-and-Longitude

https://leetcode.com/discuss/interview-question/system-design/124568/How-to-design-a-system-to-retrieve-address-information-longitude-and-latitude-information.

  1. Strategy to update sattelite firmware https://leetcode.com/discuss/interview-question/system-design/198761/Updating-Satellite-Frimware

  2. Design card deck https://leetcode.com/discuss/interview-question/system-design/194663/Design-a-class-that-represents-a-deck-of-cards

  3. Send large files https://leetcode.com/discuss/interview-question/system-design/193953/Distribute-binary-file-(daily)-for-thousands-of-servers

  4. Extract json from a text file https://leetcode.com/discuss/interview-question/system-design/191944/How-to-extract-JSON-object-from-a-text-file-of-size-100-GB

  5. https://leetcode.com/discuss/interview-question/system-design/177823/How-to-track-overall-activity-time-of-every-user-efficiently

  6. Design video pause functionality https://leetcode.com/discuss/interview-question/system-design/177327/Video-pause-functionality-for-multiple-devices

  7. https://leetcode.com/discuss/interview-question/system-design/172809/Sync-front-end-with-back-end-in-realtime

  8. Push/pull API for producer consumer https://leetcode.com/discuss/interview-question/system-design/158701/Design-push-and-pull-APIs-for-a-producer-consumer-system

  9. HA for webproxy https://leetcode.com/discuss/interview-question/system-design/134797/HA-for-HTTP-proxy

  10. Alarm app https://leetcode.com/discuss/interview-question/system-design/133426/Microsoft-Mobile-app-for-alarm

  11. Snake and ladder https://leetcode.com/discuss/interview-question/system-design/132140/Design-online-multiplayer-snakeandladder-game

  12. https://leetcode.com/discuss/interview-question/system-design/124858/First-non-repeating-word-in-a-file-File-size-can-be-100GB.

  13. Store extremely large parse matrix https://leetcode.com/discuss/interview-question/system-design/125306/How-to-store-extremely-large-sparse-matrices

  14. Implement mine swipper

 

 

 

OOP design https://github.com/tssovi/grokking-the-object-oriented-design-interview

https://github.com/savitansh/SystemDesignInterview

 

 

https://leetcode.com/discuss/interview-question/system-design/943886/Facebook-Product-Design-Questions

 

https://github.com/checkcheckzz/system-design-interview

GitHub - donnemartin/system-design-primer: Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.

 

 

 

Drawing tool
https://leetcode.com/discuss/interview-question/system-design/758105/Remote-system-design-diagram-drawing-tool

 

Other informstive posts

 

I hope it will be helpful if you have mentioned your overall experience.

If you are looking for interview perspective, You need to gain knowledge about distributed systems.

To start with, to simplify it,

For a Simple Application

Input --> Processing --> DataStore --> Processing/Output

For Distributed Systems [I wrote considering for System Design interview]

Following contains quick overlook about the list that need to have for basic understanding.

  1. Input/Output
    Communication[Rest, grpc, WebSocket, tcp/udp, webRTC], API Gateway, Proxy Server, Load Balancing -- [Reverse Proxy]
    Message Queue -- [Queue for sending/receiving information] Kafka/RabitMQ

  2. Processing
    Computation -- Micro-service [Python, Springboot/Java, Go...], MapReduce
    Service Discovery/Registry -- [Finding and Redirecting the load, Saga Pattern] -- Eg: Eureka

  3. Datastore/Distributed Consensus
    Cache -- [Simple Cache to reduce database hits] -- Eg: Redis
    SQL -- [SQL database for Relational Data -- Transactions based like payments, Horizontal Partition/Vertical Partition, Shading ] Eg: MySQL
    NOSQL -- [NOSQL for querying documents like product, product details, CAP] Eg: MongoDB/Apache Cassandra
    Concurrency -- 2 Phase Commit, 3 Phase Commit, Saga Pattern[Choreography saga, Orchestrator Saga], Split Brain problem.
    FileSystem -- Hadoop File System

  4. Security
    AAA -- Authentication, Authorisation, Auditing

Misc -- Logging/Notification

Once you have overall picture in your mind, you can start with YouTube videos like Gaurav Sen or Tech Dummies.

 

 

 

 

Product Design

The product design interview at Facebook will involve designing a product or API to support an end-user experience. Here's a list of concepts that Facebook recommends you review before your interview:

-Scalability

-Design patterns

-Data ownership

-Protocols

-Data formats

-Client-server design

-Designing for long term vs. complexity

-Accommodating possible product changes

Some example questions involve designing a product API or an email server.

 

 

System Design

The system design interview at Facebook will ask you to weigh design considerations for complex problems. Here's a list of concepts that Facebook recommends you review before your interview:

-Concurrency (threads, deadlock, starvation, consistency, coherence)

-Caching

-Database partitioning, replication, sharding, CAP Theorem

-Networking (IPC, TCP/IP)

Real-world performance (relative performance RAM, disk, your network, SSD)

-Availability and reliability (types of failures, failure units, how failures may manifest, mitigations, etc.)

-Data storage and data aggregation

-QPS capacity/machine estimation (back of the envelope estimates), byte size estimation

Some example questions involve architecting a video distribution system or designing a mobile image search client.

 

 

Tags:

0

Software Engineer interview Prep. Master list

by Deepak Dhakal 2. July 2021 00:46

System Design Interview Prep Master Doc

Compiled by - Pooja Biswas

pooja biswas | LinkedIn

All system design youtube channels

  1. https://www.youtube.com/user/dimakorolev/videos

  2. https://www.youtube.com/c/ByteByByte/videos

  3. https://www.youtube.com/channel/UC_n-A84J0UcU5uq4sEh2CnQ

  4. https://www.youtube.com/c/DefogTech/videos

  5. https://www.youtube.com/c/HusseinNasser-software-engineering/videos

  6. https://www.youtube.com/c/SystemDesignInterview/videos

  7. https://www.youtube.com/c/sudoCODE/videos

  8. https://www.youtube.com/c/codeKarle/videos

  9. https://www.youtube.com/c/CMUDatabaseGroup/videos

  10. https://www.youtube.com/c/interviewingio/videos

  11. https://www.youtube.com/c/GauravSensei/videos

  12. https://www.youtube.com/c/EngineeringwithUtsav/videos

  13. https://www.youtube.com/channel/UClB4KPy5LkJj1t3SgYVtMOQ/videos

  14. https://www.youtube.com/c/ExponentTV/videos

  15. https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39_tb2fErI4-WkMbsvGQk9_UB

  16. https://www.youtube.com/c/TheInterviewSage/videos

  17. https://www.youtube.com/c/ThinkSoftware/videos

  18. https://www.youtube.com/c/SuccessinTech/videos

  19. https://www.youtube.com/c/TechDummiesNarendraL/videos

                  20. https://www.youtube.com/c/OktaDev/videos



Important Books :

  1. Amazon.com: System Design Interview – An Insider's Guide eBook: Xu, Alex: Kindle Store

  2. Amazon.com: Operating Systems: Three Easy Pieces eBook: Arpaci-Dusseau, Remzi, Arpaci-Dusseau, Andrea: Kindle Store

  3. Web Scalability for Startup Engineers: Ejsmont, Artur: 9780071843652: Amazon.com: Books

  4. Understanding Distributed Systems: What every developer should know about large distributed applications: Vitillo, Roberto: 9781838430207: Amazon.com: Books

  5. Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems: Kleppmann, Martin: 9781449373320: Amazon.com: Books

  6. Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale: 9781491936160: Computer Science Books @ Amazon.com

  7. Buy Distributed Algorithms – An Intuitive Approach 2e (The MIT Press) Book Online at Low Prices in India | Distributed Algorithms – An Intuitive Approach 2e (The MIT Press) Reviews & Ratings - Amazon.in

  8. Elasticsearch: The Definitive Guide: A Distributed Real-Time Search and Analytics Engine eBook: Gormley, Clinton, Tong, Zachary: Amazon.in: Kindle Store

  9. Buy Cassandra – The Definitive Guide, 3e: Distributed Data at Web Scale Book Online at Low Prices in India | Cassandra – The Definitive Guide, 3e: Distributed Data at Web Scale Reviews & Ratings - Amazon.in

Imp keywords

 

  1. Write ahead logging

  2. SSL passthrough / termination on load balancers

  3. Clock vectors

  4. Hinted handoff

  5. Bloom filters

  6. Gossip protocols

  7. Merkle trees

  8. Two phase commit

  9. Two phase locking

  10. Consistent hashing

  11. Data replication

  12. Total order broadcast

  13. Isolation levels (read uncomitted, read comitted, repeatable read, serializable)

  14. Quad trees (GeoHashin)

  15. Inverse indexing - Google search/any search indexing

  16. Gaming ranking - rank players based on score and faster.

  17. Word search in trie - auto suggestions system design

  18. Sort 5gb data in 1gb memory - merge sort

  19. Paxos algo

  20. Merkle Tree

  21. Backpressure

  22. Circuit breaker

  23. Raft

  24. Service discovery

  25. Saga

  26. Hyperloglog

 

 

 

Important algorithms

https://github.com/resumejob/system-design-algorithms

  • Frugal Streaming

  • Geohash / S2 Geometry

  • Leaky bucket / Token bucket

  • Loosy Counting

  • Operational transformation

  • Quadtree / Rtree

  • Ray casting

  • Reverse index

  • Rsync algorithm

  • Trie algorithm















Cloud design patterns




https://docs.microsoft.com/en-us/azure/architecture/patterns/index-patterns

Cloud arch pattern

 

Ambassador

Create helper services that send network requests on behalf of a consumer service or application.

Design and Implementation,


Operational Excellence

Anti-Corruption Layer

Implement a façade or adapter layer between a modern application and a legacy system.

Design and Implementation,


Operational Excellence

Asynchronous Request-Reply

Decouple backend processing from a frontend host, where backend processing needs to be asynchronous, but the frontend still needs a clear response.

Messaging

Backends for Frontends

Create separate backend services to be consumed by specific frontend applications or interfaces.

Design and Implementation

Bulkhead

Isolate elements of an application into pools so that if one fails, the others will continue to function.

Reliability

Cache-Aside

Load data on demand into a cache from a data store

Data Management,


Performance Efficiency

Choreography

Let each service decide when and how a business operation is processed, instead of depending on a central orchestrator.

Messaging,


Performance Efficiency

Circuit Breaker

Handle faults that might take a variable amount of time to fix when connecting to a remote service or resource.

Reliability

Claim Check

Split a large message into a claim check and a payload to avoid overwhelming a message bus.

Messaging

Compensating Transaction

Undo the work performed by a series of steps, which together define an eventually consistent operation.

Reliability

Competing Consumers

Enable multiple concurrent consumers to process messages received on the same messaging channel.

Messaging

Compute Resource Consolidation

Consolidate multiple tasks or operations into a single computational unit

Design and Implementation

CQRS

Segregate operations that read data from operations that update data by using separate interfaces.

Data Management,


Design and Implementation,


Performance Efficiency

Deployment Stamps

Deploy multiple independent copies of application components, including data stores.

Reliability,


Performance Efficiency

Event Sourcing

Use an append-only store to record the full series of events that describe actions taken on data in a domain.

Data Management,


Performance Efficiency

External Configuration Store

Move configuration information out of the application deployment package to a centralized location.

Design and Implementation,


Operational Excellence

Federated Identity

Delegate authentication to an external identity provider.

Security

Gatekeeper

Protect applications and services by using a dedicated host instance that acts as a broker between clients and the application or service, validates and sanitizes requests, and passes requests and data between them.

Security

Gateway Aggregation

Use a gateway to aggregate multiple individual requests into a single request.

Design and Implementation,


Operational Excellence

Gateway Offloading

Offload shared or specialized service functionality to a gateway proxy.

Design and Implementation,


Operational Excellence

Gateway Routing

Route requests to multiple services using a single endpoint.

Design and Implementation,


Operational Excellence

Geodes

Deploy backend services into a set of geographical nodes, each of which can service any client request in any region.

Reliability,


Operational Excellence

Health Endpoint Monitoring

Implement functional checks in an application that external tools can access through exposed endpoints at regular intervals.

Reliability,


Operational Excellence

Index Table

Create indexes over the fields in data stores that are frequently referenced by queries.

Data Management,


Performance Efficiency

Leader Election

Coordinate the actions performed by a collection of collaborating task instances in a distributed application by electing one instance as the leader that assumes responsibility for managing the other instances.

Design and Implementation,


Reliability

Materialized View

Generate prepopulated views over the data in one or more data stores when the data isn't ideally formatted for required query operations.

Data Management,


Operational Excellence

Pipes and Filters

Break down a task that performs complex processing into a series of separate elements that can be reused.

Design and Implementation,


Messaging

Priority Queue

Prioritize requests sent to services so that requests with a higher priority are received and processed more quickly than those with a lower priority.

Messaging,


Performance Efficiency

Publisher/Subscriber

Enable an application to announce events to multiple interested consumers asynchronously, without coupling the senders to the receivers.

Messaging

Queue-Based Load Leveling

Use a queue that acts as a buffer between a task and a service that it invokes in order to smooth intermittent heavy loads.

Reliability,


Messaging,


Resiliency,


Performance Efficiency

Retry

Enable an application to handle anticipated, temporary failures when it tries to connect to a service or network resource by transparently retrying an operation that's previously failed.

Reliability

Scheduler Agent Supervisor

Coordinate a set of actions across a distributed set of services and other remote resources.

Messaging,


Reliability

Sequential Convoy

Process a set of related messages in a defined order, without blocking processing of other groups of messages.

Messaging

Sharding

Divide a data store into a set of horizontal partitions or shards.

Data Management,


Performance Efficiency

Sidecar

Deploy components of an application into a separate process or container to provide isolation and encapsulation.

Design and Implementation,


Operational Excellence

Static Content Hosting

Deploy static content to a cloud-based storage service that can deliver them directly to the client.

Design and Implementation,


Data Management,


Performance Efficiency

Strangler Fig

Incrementally migrate a legacy system by gradually replacing specific pieces of functionality with new applications and services.

Design and Implementation,


Operational Excellence

Throttling

Control the consumption of resources used by an instance of an application, an individual tenant, or an entire service.

Reliability,


Performance Efficiency

Valet Key

Use a token or key that provides clients with restricted direct access to a specific resource or service.

Data Management,


Security

 

 

 

 

Data Management patterns

  • 06/23/2017

  • +3

Data management is the key element of cloud applications, and influences most of the quality attributes. Data is typically hosted in different locations and across multiple servers for reasons such as performance, scalability or availability, and this can present a range of challenges. For example, data consistency must be maintained, and data will typically need to be synchronized across different locations.

Additionally data should be protected at rest, in transit, and via authorized access mechanisms to maintain security assurances of confidentiality, integrity, and availability. Refer to the Azure Security Benchmark Data Protection Control for more information.

Pattern

Summary

Cache-Aside

Load data on demand into a cache from a data store

CQRS

Segregate operations that read data from operations that update data by using separate interfaces.

Event Sourcing

Use an append-only store to record the full series of events that describe actions taken on data in a domain.

Index Table

Create indexes over the fields in data stores that are frequently referenced by queries.

Materialized View

Generate prepopulated views over the data in one or more data stores when the data isn't ideally formatted for required query operations.

Sharding

Divide a data store into a set of horizontal partitions or shards.

Static Content Hosting

Deploy static content to a cloud-based storage service that can deliver them directly to the client.

Valet Key

Use a token or key that provides clients with restricted direct access to a specific resource or service.

DATA MANAGEMENT PATTERNS

 

 


Design and Implementation patterns

  • 06/23/2017

  • 2 minutes to read

Good design encompasses factors such as consistency and coherence in component design and deployment, maintainability to simplify administration and development, and reusability to allow components and subsystems to be used in other applications and in other scenarios. Decisions made during the design and implementation phase have a huge impact on the quality and the total cost of ownership of cloud hosted applications and services.

Pattern

Summary

Ambassador

Create helper services that send network requests on behalf of a consumer service or application.

Anti-Corruption Layer

Implement a façade or adapter layer between a modern application and a legacy system.

Backends for Frontends

Create separate backend services to be consumed by specific frontend applications or interfaces.

CQRS

Segregate operations that read data from operations that update data by using separate interfaces.

Compute Resource Consolidation

Consolidate multiple tasks or operations into a single computational unit

External Configuration Store

Move configuration information out of the application deployment package to a centralized location.

Gateway Aggregation

Use a gateway to aggregate multiple individual requests into a single request.

Gateway Offloading

Offload shared or specialized service functionality to a gateway proxy.

Gateway Routing

Route requests to multiple services using a single endpoint.

Leader Election

Coordinate the actions performed by a collection of collaborating task instances in a distributed application by electing one instance as the leader that assumes responsibility for managing the other instances.

Pipes and Filters

Break down a task that performs complex processing into a series of separate elements that can be reused.

Sidecar

Deploy components of an application into a separate process or container to provide isolation and encapsulation.

Static Content Hosting

Deploy static content to a cloud-based storage service that can deliver them directly to the client.

Strangler Fig

Incrementally migrate a legacy system by gradually replacing specific pieces of functionality with new applications and services.

DESIGN AND IMPLEMENTATION PATTERNS

 

 

Messaging patterns

+4

The distributed nature of cloud applications requires a messaging infrastructure that connects the components and services, ideally in a loosely coupled manner in order to maximize scalability. Asynchronous messaging is widely used, and provides many benefits, but also brings challenges such as the ordering of messages, poison message management, idempotency, and more.

Pattern

Summary

Asynchronous Request-Reply

Decouple backend processing from a frontend host, where backend processing needs to be asynchronous, but the frontend still needs a clear response.

Claim Check

Split a large message into a claim check and a payload to avoid overwhelming a message bus.

Choreography

Have each component of the system participate in the decision-making process about the workflow of a business transaction, instead of relying on a central point of control.

Competing Consumers

Enable multiple concurrent consumers to process messages received on the same messaging channel.

Pipes and Filters

Break down a task that performs complex processing into a series of separate elements that can be reused.

Priority Queue

Prioritize requests sent to services so that requests with a higher priority are received and processed more quickly than those with a lower priority.

Publisher-Subscriber

Enable an application to announce events to multiple interested consumers asynchronously, without coupling the senders to the receivers.

Queue-Based Load Leveling

Use a queue that acts as a buffer between a task and a service that it invokes in order to smooth intermittent heavy loads.

Scheduler Agent Supervisor

Coordinate a set of actions across a distributed set of services and other remote resources.

Sequential Convoy

Process a set of related messages in a defined order, without blocking processing of other groups of messages.

MESSAGING PATTERNS

 

 

 

Book https://learning.oreilly.com/library/view/cloud-architecture-patterns/9781449357979/




LEETCODE all system design problems

  1. Web Crawler']

https://leetcode.com/discuss/interview-question/system-design/124657/Facebook-or-System-Design-or-A-web-crawler-that-will-crawl-Wikipedia

  1. Detect web crawler https://leetcode.com/discuss/interview-question/system-design/548816/Amazon-or-System-Design-or-Web-Crawler-Detector

  2. Yelp

  3. Distributed file system

  4. URL shortening and Pastebin https://leetcode.com/discuss/interview-question/system-design/124804/Design-Pastebin

https://leetcode.com/discuss/interview-question/system-design/124658/Design-URL-Shortening-service-like-TinyURL

  1. Instagram

https://leetcode.com/discuss/interview-question/system-design/124802/Design-Instagram

https://leetcode.com/discuss/interview-question/system-design/586749/design-Instagram

https://leetcode.com/discuss/interview-question/system-design/719253/Design-Facebook-%3A-System-Design-Interview

  1. Dropbox.

  2. Twitter

https://leetcode.com/discuss/interview-question/system-design/124689/Design-twitter

  1. Redis https://leetcode.com/discuss/interview-question/system-design/125751/Design-a-distributed-cache-system

  2. Youtube or Netflix https://leetcode.com/discuss/interview-question/system-design/733520/Design-YouTube-Very-detailed-design-with-diagrams

https://leetcode.com/discuss/interview-question/system-design/144287/Design-Recommendation-System-for-Amazon-Videos

https://leetcode.com/discuss/interview-question/system-design/600861/System-Design-Youtube-add-click-counts

https://leetcode.com/discuss/interview-question/system-design/557250/Design-a-video-streaming-service-to-support-playback-video-from-different-devices

https://leetcode.com/discuss/interview-question/system-design/496042/Design-video-sharing-platform-like-Youtube

https://leetcode.com/discuss/interview-question/system-design/158698/Distributed-database%3A-Netflix

https://leetcode.com/discuss/interview-question/system-design/124565/Design-Netflix-recommendation-engine

https://leetcode.com/discuss/interview-question/system-design/150607/Design-youtube

  1. Ticketmaster

https://leetcode.com/discuss/interview-question/system-design/124803/Design-BookMyShow

https://leetcode.com/discuss/interview-question/system-design/315763/System-Design-or-Seat-reservation-application-like-Ticket-Master-or-BookMyShow

  1. Facebook Messenger or WhatsApp https://leetcode.com/discuss/interview-question/system-design/585930/Amazon-or-System-Design-or-Design-a-Chat-Service

https://leetcode.com/discuss/interview-question/system-design/220073/How-would-you-design-WhatsApp

https://leetcode.com/discuss/interview-question/system-design/124613/Amazon-or-System-Design-or-A-scalable-chat-application-on-phone-browsing

  1. Typeahead suggesions.

  2. Twitter search.

  3. Newsfeed ranking

https://leetcode.com/discuss/interview-question/system-design/349627/How-do-you-design-a-meta-data-for-a-news-feed

https://leetcode.com/discuss/interview-question/system-design/153871/Design-a-News-Feed-system-(like-Facebook-Linkedin-etc.)

  1. Web search.

  2. LinkedIn "you may know ..."., https://leetcode.com/discuss/interview-question/system-design/1036762/Google-Onsite-System-Design-How-to-do-it

https://leetcode.com/discuss/interview-question/system-design/153941/Design-the-%22People-You-May-Know%22-feature-on-LinkedIn-or-Facebook.

  1. Uber / Luxe (anti-uber)?

  2. Freight / delivery orchestration? (edited) 

  3. Botnet/decentralized web crawler/torrent 

https://leetcode.com/discuss/interview-question/system-design/594844/System-design-question-Help-needed

https://leetcode.com/discuss/interview-question/system-design/464997/Design-a-P2P-file-sharing-application-like-BitTorrent

  1. Coupon redeeming system

https://leetcode.com/discuss/interview-question/system-design/353302/Design-a-couponvoucher-management-system-or-DellEMC

https://leetcode.com/discuss/interview-question/system-design/459593/Facebook-or-System-Design-or-E-commerce-Apply-discount-on-every-nth-order

  1. Message queue kafka, service bus

https://leetcode.com/discuss/interview-question/system-design/124761/Deciding-which-queue-to-send-a-post-to

https://leetcode.com/discuss/interview-question/system-design/206134/Amazon-or-System-Design-or-Design-a-Distributed-Message-queue

https://leetcode.com/discuss/interview-question/system-design/734303/Microsoftor-Design-an-Enterprise-Service-Bus

  1. Rate limiter https://leetcode.com/discuss/interview-question/system-design/637402/Design-a-efficient-client-side-rate-limit-handler

https://leetcode.com/discuss/interview-question/system-design/124558/Uber-or-Rate-Limiter

  1. Design leetcode - asked in amazon and fb. https://leetcode.com/discuss/interview-question/system-design/649021/Design-Leetcode

  2. https://leetcode.com/discuss/interview-question/system-design/409736/Facebook-or-System-Design-or-Hacker-Rank-LeetCode-Contest-Leadership-Board-System

https://leetcode.com/discuss/interview-question/system-design/308452/System-Design-or-Programming-contest-platform-like-LeetCode

  1. Large log data collection and processing system 

https://leetcode.com/discuss/interview-question/system-design/124603/Amazon-or-Phone-screen-or-How-to-handle-large-log-data
https://leetcode.com/discuss/interview-question/system-design/128037/How-would-you-parse-a-huge-log-file

https://leetcode.com/discuss/interview-question/system-design/189030/Design-a-system-which-can-report-frequently-occurring-exceptions-on-a-dashboard

https://leetcode.com/discuss/interview-question/system-design/196142/Copy-coredump-files-from-millions-of-system-to-single-Storage-server-like-S3

https://leetcode.com/discuss/interview-question/system-design/431023/Google-or-Onsite-or-Get-all-logs-between-times

https://leetcode.com/discuss/interview-question/system-design/440546/Facebook-or-System-Design-Onsite-or-Compute-Percentile-Metrics-Over-Time-Series

https://leetcode.com/discuss/interview-question/system-design/124603/Amazon-or-Phone-screen-or-How-to-handle-large-log-data

https://leetcode.com/discuss/interview-question/system-design/1133962/Service-which-will-download-data-from-multiple-sources-and-ingests-it-in-the-system

https://leetcode.com/discuss/interview-question/system-design/942087/System-Design%3A-Design-a-system-to-process-data-in-different-formats-from-different-sources

https://leetcode.com/discuss/interview-question/system-design/852238/Need-help-with-System-Design-problem-asked-in-a-real-interview

https://leetcode.com/discuss/interview-question/system-design/820877/Bloomberg-System-Design

https://leetcode.com/discuss/interview-question/system-design/778868/Facebook-oror-Onsite-oror-System-Design-Aggregation-click-events

https://leetcode.com/discuss/interview-question/system-design/725364/System-Design-or-IOT-sensor-data-aggregator

https://leetcode.com/discuss/interview-question/system-design/202946/Design-a-system-to-aggregate-metrics-from-large-cluster(800%2B)-of-web-servers

  1. Realtime stock price monitoring system/ live score update cricbuzz, realtime gaming score

https://leetcode.com/discuss/interview-question/system-design/625918/Amazon-or-System-Design-or-Design-a-real-time-gaming-ranking-system

https://leetcode.com/discuss/interview-question/system-design/431712/Bloomberg-or-Design-a-system-to-give-prices-of-a-stock

  1. Stock trading system https://medium.com/@narengowda/stock-exchange-system-design-answered-ad4be1345851

https://leetcode.com/discuss/interview-question/system-design/820877/Bloomberg-System-Design

https://leetcode.com/discuss/interview-question/system-design/124794/Design-a-Multicurrency-trading-system

https://leetcode.com/discuss/interview-question/system-design/490034/FAANG-or-Onsite-or-Intern-or-System-Design-Stock

  1. Kill switch for stopping stock trading https://leetcode.com/discuss/interview-question/system-design/124553/Kill-Switch

  2. Design network fail over

https://leetcode.com/discuss/interview-question/system-design/124598/Design-network-fail-over

  1. Design AB testing framework https://leetcode.com/discuss/interview-question/system-design/124595/AB-Testing

https://leetcode.com/discuss/interview-question/system-design/228661/Design-a-Data-Experimentation-platform

  1. Design parking lot system https://leetcode.com/discuss/interview-question/system-design/124576/Design-a-parking-lot-system.

https://leetcode.com/discuss/interview-question/system-design/575186/Design-a-Parking-Spot-System

https://leetcode.com/discuss/interview-question/system-design/598634/Microsoft-or-Onsite-or-System-Design-or-SDE-2

https://leetcode.com/discuss/interview-question/system-design/850712/System-Design-Amazon-2020-(SDE-2)

https://leetcode.com/discuss/interview-question/system-design/765686/System-Design-Interview-Question%3A-Parking-Lot-or-Low-Level-Design

https://leetcode.com/discuss/interview-question/system-design/125260/Parking-Lots-Design

  1. Reccomendation Engine https://leetcode.com/discuss/interview-question/system-design/124565/Design-Netflix-recommendation-engine

  2. Smart voice assistant like siri, alexa

https://leetcode.com/discuss/interview-question/system-design/124566/Design-AlexaSiriGoogle-Home-Architecture

https://leetcode.com/discuss/interview-question/system-design/848252/Amazon-System-Design

  1. Nearest store location, another variation of topk https://leetcode.com/discuss/interview-question/system-design/124567/Nearest-Store-Locators

https://leetcode.com/discuss/interview-question/system-design/533061/How-to-implement-nearest-location-kind-of-functionality-in-a-google-map-type-application

https://leetcode.com/discuss/interview-question/system-design/154172/Design-google-map-database

  1. Job scheduling sytem https://leetcode.com/discuss/interview-question/system-design/124697/Walmartlabs-onsite

https://leetcode.com/discuss/interview-question/system-design/124786/Google-Scheduling-Job-Involving-both-RAM-and-CPU

https://leetcode.com/discuss/interview-question/system-design/692996/Microsoft-System-Design-Please-help

https://leetcode.com/discuss/interview-question/system-design/553563/Googleor-Distributed-SystemorPerformance

https://leetcode.com/discuss/interview-question/system-design/344524/Amazon-or-Design-a-JobTask-Scheduler

https://leetcode.com/discuss/interview-question/system-design/124672/Implement-a-task-scheduler

  1. Elevator system

https://leetcode.com/discuss/interview-question/system-design/149264/Design-an-Elevator-system

  1. Malware detection system https://leetcode.com/discuss/interview-question/system-design/1019028/FB-or-System-Design-or-Multi-Engine-Malware-Analyzer

https://leetcode.com/discuss/interview-question/system-design/150610/Design-a-malware-detection-system

  1. Garbage collector

  2. Google docs https://leetcode.com/discuss/interview-question/system-design/148187/System-Design-or-Google-Docs

https://leetcode.com/discuss/interview-question/system-design/148187/System-Design-or-Google-Docs

https://leetcode.com/discuss/interview-question/system-design/208207/Design-a-Google-Sheet-System

https://leetcode.com/discuss/interview-question/system-design/349669/Google-SWE-L5-or-Onsite-or-Design-Google-Docs-Versioning-System

https://leetcode.com/discuss/interview-question/system-design/322448/Content-Management-System-Design

https://leetcode.com/discuss/interview-question/system-design/194402/Design-a-file-sharing-system

  1. Ecommerce Price checker system https://leetcode.com/discuss/interview-question/system-design/140742/E-commerce-(Amazon)Website-looking-into-other-competitor-Website-products-prices-and-update

  2. Notification system https://leetcode.com/discuss/interview-question/system-design/138097/Design-Notification-Service-for-Amazon-Alexa

  3. Online ludo game 

  4. metric monitoring service

  5. Ecommerce site ,Shopping cart, product catalog, payment gateway 

https://leetcode.com/discuss/interview-question/system-design/211415/Interview-Question-Ecommerce-System-design-(-Eg-%3A-Amazon-)%3A-Concurrency-issues-handling

https://leetcode.com/discuss/interview-question/system-design/589546/Amazon-or-System-Design-or-Amazon-Order-System

https://leetcode.com/discuss/interview-question/system-design/675539/System-Design-question-asked-in-interview

https://leetcode.com/discuss/interview-question/system-design/666792/Microsoft-or-System-design-or-Please-help

https://leetcode.com/discuss/interview-question/system-design/1124722/System-Design-or-Shopping-Cart-or-Payment-Gateway-or-Product-Catalog

https://leetcode.com/discuss/interview-question/system-design/886390/Design-Recommendation-API-or-Akamai-Interview

https://leetcode.com/discuss/interview-question/system-design/776927/Design-an-accountpayment-system

https://leetcode.com/discuss/interview-question/system-design/706038/System-Design-Payment-System-Wallet-system-Payment-gateway

  1. Seller summary page https://leetcode.com/discuss/interview-question/system-design/124612/Phone-Interview-Question%3A-Design-an-Seller-Summary-Page

  2. Customer who bought this also bought https://leetcode.com/discuss/interview-question/system-design/124557/Amazon's-%22Customers-who-bought-this-item-also-bought%22-recommendation-system

  3. Distributed key value store, https://leetcode.com/discuss/interview-question/system-design/1120468/Design-Assignment-or-Implement-a-distributed-Key-Value-(KV)-store-or-SE-Role-Avalara

https://leetcode.com/discuss/interview-question/system-design/747591/Amazon-or-Onsite-or-System-design-or-Please-help

  1. Facebook live commenting

https://leetcode.com/discuss/interview-question/system-design/583184/FBInstagram-'Live-Comments'-System-design

  1. Facebook status search

  2. Image editing ( asked in fb 2021) https://leetcode.com/discuss/interview-question/system-design/1077411/Facebook-or-Onsite-2021-or-System-Design-or-Design-image-editing

  3. File download application system https://leetcode.com/discuss/interview-question/system-design/1071562/Design-a-File-Download-Application-System

  4. Proximity server https://leetcode.com/discuss/interview-question/system-design/923677/Facebook-or-System-Design

  5. Top N songs, another top k problem

https://leetcode.com/discuss/interview-question/system-design/124702/Design-a-service-to-calculate-the-top-k-listened-songs-in-past-24-hours

https://leetcode.com/discuss/interview-question/system-design/243604/Design-a-real-time-dashboard-showing-the-most-played-songs

  1. Privacy setting at facebook

  2. Distributed configuration management system

  3. Design gmail https://leetcode.com/discuss/interview-question/system-design/1014986/Google-or-Onsite-or-System-Design%3A-Design-an-Email-system-like-GMAIL

  4. News reading feature in alexa https://leetcode.com/discuss/interview-question/system-design/1014181/Amazon-or-System-Design-or-SDE2

  5. Ads click visualisation system https://leetcode.com/discuss/interview-question/system-design/1002923/Facebook-or-Online-or-Real-time-data-visualization-for-ads-clicks

  6. IoT devices management system https://leetcode.com/discuss/interview-question/system-design/974890/Design-a-system-for-management-of-IOT-devices

  7. Timer service https://leetcode.com/discuss/interview-question/system-design/973207/System-Design-or-Timer-service

  8. Service monitoring and alerting system like pagerduty, azure monitor etc
    https://leetcode.com/discuss/interview-question/system-design/958919/System-Design-Interview-or-Service-Health-Monitoring-and-Alerting-Service

https://leetcode.com/discuss/interview-question/system-design/287678/Design-a-monitoring-or-analytics-service-like-Datadog-or-SignalFx

  1. Load balancer https://leetcode.com/discuss/interview-question/system-design/943352/Facebook-or-E5-System-Design-Interview-Question-or-Menlo-Park

  2. Design undergeound system 

  3. Leader board table design https://leetcode.com/discuss/interview-question/system-design/892083/Leaderboard-table-system-design-for-online-game

  4. Slot booking system for playarena etc https://leetcode.com/discuss/interview-question/system-design/880581/Event-Booking-for-playarenas-Low-level-design

https://leetcode.com/discuss/interview-question/system-design/423613/Amazon-or-Phone-Screen-or-Design-Restaurant-Reservation-System

  1. Food delivery app https://leetcode.com/discuss/interview-question/system-design/874074/Food-Delivery-App-or-Low-Level-Design-or-Interview-Question

  2. Online gaming lobby service https://leetcode.com/discuss/interview-question/system-design/874074/Food-Delivery-App-or-Low-Level-Design-or-Interview-Question

  3. URL fishing varifier https://leetcode.com/discuss/interview-question/system-design/896312/Google-system-design

  4. Design whatsapp/instagram story

https://leetcode.com/discuss/interview-question/system-design/388222/Snapchat-or-System-Design-or-Instagram-Story-Feature

  1. File sharing with collaborative editing https://leetcode.com/discuss/interview-question/system-design/838085/File-sharing-service-with-collaborative-editing-or-Amazon

https://leetcode.com/discuss/interview-question/system-design/824659/Intuit-or-Long-Poll-vs-Web-socket-vs-Server-send-Events

  1. Stack overflow tags https://leetcode.com/discuss/interview-question/system-design/838025/Design-a-tagging-system-like-tags-used-in-stack-overflow

https://leetcode.com/discuss/interview-question/system-design/307558/Design-Stack-Overflow

  1. Tinyurls https://leetcode.com/discuss/interview-question/system-design/838012/URL-Shortener-or-MD5-or-How-to-deal-with-collisions-or-FinTech-startup

  2. Github like cloud repo https://leetcode.com/discuss/interview-question/system-design/837383/System-design-of-code-repository-like-github 

  3. Job posting site https://leetcode.com/discuss/interview-question/system-design/811840/Job-listing-storage-and-search

  4. S3/cloud object store https://leetcode.com/discuss/interview-question/system-design/811503/System-design-Object-store-design-like-S3GCS

  5. Celebrity timeline generation https://leetcode.com/discuss/interview-question/system-design/810561/Timeline-generation-for-celebrities-or-System-Design-or-Google

  6. Kindle service 

  7. Bidding system https://leetcode.com/discuss/interview-question/system-design/792060/Bidding-System%3A-System-Design-Interview

  8. Billing system - asked in fb interview

  9. RPC system for client server comm https://leetcode.com/discuss/interview-question/system-design/790034/Client-Server-Communication%3A-System-Design-Interview

  10. Autonomous driving system https://leetcode.com/discuss/interview-question/system-design/789961/Design-a-cloud-based-simulationvisualization-platform-for-a-self-driving-cars-company

  11. Github code search https://leetcode.com/discuss/interview-question/system-design/789015/Github-%3A-Design-search-feature-in-Github-scale-code-repository

  12. Car showroom https://leetcode.com/discuss/interview-question/system-design/785960/Amazon-System-Design-Question

  13. Airport boarding gate security https://leetcode.com/discuss/interview-question/system-design/785960/Amazon-System-Design-Question

  14. Social graph https://leetcode.com/discuss/interview-question/system-design/782906/Design-a-social-graph

  15. Place of interest https://leetcode.com/discuss/interview-question/system-design/777945/Design-a-system-to-source-store-and-display-places-of-interest

  16. Tinder https://leetcode.com/discuss/interview-question/system-design/774870/Tinder-System-Design-or-Online-Dating-App-System-Design

  17. Grocery store https://leetcode.com/discuss/interview-question/system-design/769578/Amazon-orSystem-Design-or-Amazon-Go-or-suggestion-on-solution-welcome

https://leetcode.com/discuss/interview-question/system-design/467655/Amazon-Onsite-or-System-Design-Pickup-Delivery-System-For-Groceries

  1. Display ads https://leetcode.com/discuss/interview-question/system-design/761814/Design-number-of-ads-to-show-to-users-on-a-google-search

  2. Add badge to peoples spotify account https://leetcode.com/discuss/interview-question/system-design/748408/How-to-design-a-system-to-add-badges-to-people's-Spotify-account

  3. Xml to json conversation https://leetcode.com/discuss/interview-question/system-design/743624/System-design-question%3A-Amazon-SDE2%3A-Large-xml-files-to-json-conversion

  4. Ocr web app https://leetcode.com/discuss/interview-question/system-design/741676/System-Design%3A-OCR-web-app

  5. High scale otp generation system https://leetcode.com/discuss/interview-question/system-design/728464/Microsoft-or-Onsite-or-Modify-an-OTP-generation-system-to-handle-more-requests

  6. Distributed counter https://leetcode.com/discuss/interview-question/system-design/685310/Microsoft-virtual-or-Design-distributed-counter

https://leetcode.com/discuss/interview-question/system-design/277606/Design-a-performance-counter

  1. Scan for viruses in uploaded file https://leetcode.com/discuss/interview-question/system-design/659875/Design-a-system-where-client-can-upload-a-file-and-viruses-need-to-be-scanned

  2. Log processing at scale https://leetcode.com/discuss/interview-question/system-design/622704/Design-a-system-to-store-and-retrieve-logs-for-all-of-eBay

  3. London travel card system https://leetcode.com/discuss/interview-question/system-design/617408/Marshall-Wace-or-Onsite-or-How-would-you-design-Oyster-(London-Travel-Card-system)tion

  4. Payment system for newyork MTA https://leetcode.com/discuss/interview-question/system-design/305388/Design-a-transportation-payment-System.

  5. Ad click counter https://leetcode.com/discuss/interview-question/system-design/584458/Facebook-or-System-Design-or-Ad-Click-Counter

  6. Design slack https://leetcode.com/discuss/interview-question/system-design/582975/Design-Slack

https://leetcode.com/discuss/interview-question/system-design/339849/System-Design-or-Slack

  1. Wikipedia https://leetcode.com/discuss/interview-question/system-design/574872/Wikipedia-or-DBsystem-design-thoughts

https://leetcode.com/discuss/interview-question/system-design/174380/Uber-design-question-Design-Wikipeida

  1. ML related system design https://leetcode.com/discuss/interview-question/system-design/566057/Machine-Learning-System-Design-%3A-A-framework-for-the-interview-day

  2. Windows update https://leetcode.com/discuss/interview-question/system-design/560512/System-Design-Question

  3. People also searched for https://leetcode.com/discuss/interview-question/system-design/559481/Amazon-or-System-design-or-SDE-2-India smiliar to linkedin's you may also know

  4. Cashback processing system https://leetcode.com/discuss/interview-question/system-design/543041/Design-cashback-processing-system

  5. Pub sub arch

  6. Google or amazob book preview https://leetcode.com/discuss/interview-question/system-design/538295/Design-Google-Books-preview-Amazon-Books-look-inside

  7. Design copy right detection https://leetcode.com/discuss/interview-question/system-design/530031/FAANG-Interview-Question-Design-a-copyright-detection-system

  8. Reddit https://leetcode.com/discuss/interview-question/system-design/469900/Netflix-or-System-Design-Web-App-Like-Reddit

  9. Facebook nearby friends https://leetcode.com/discuss/interview-question/system-design/430926/Design-Nearby-Friends

  10. Google photos home page https://leetcode.com/discuss/interview-question/system-design/398523/System-Design-Google-photos-homepage

https://leetcode.com/discuss/interview-question/system-design/396949/System-Design-Google-Photos

  1. Image upload system https://leetcode.com/discuss/interview-question/system-design/391183/Ebay-System-Design-Question

https://leetcode.com/discuss/interview-question/system-design/390503/Google-or-System-Design

  1. Facebook translator service https://leetcode.com/discuss/interview-question/system-design/386322/Design-a-translator-service-for-facebook

https://leetcode.com/discuss/interview-question/system-design/318811/Google-or-System-design-or-Design-a-translation-service-like-Google-Translate

  1. System for health care data https://leetcode.com/discuss/interview-question/system-design/368245/Design-Service-to-Interface-with-Healthcare-Data

  2. Health score app https://leetcode.com/discuss/interview-question/system-design/366754/Amazon-or-System-Design-for-health-score-app

  3. Railway reservation system https://leetcode.com/discuss/interview-question/system-design/364965/Railway-Reservation-System

  4. Treadmill system https://leetcode.com/discuss/interview-question/system-design/362168/Google-or-Onsite-or-Tread-Mill-System-Design

  5. Addressed of entire planet https://leetcode.com/discuss/interview-question/system-design/341980/Amazon-or-System-Design-or-System-to-capture-unique-addresses-in-the-entire-world

  6. Gofundme https://leetcode.com/discuss/interview-question/system-design/336089/System-Design-or-GoFundMe

  7. Shipping fullfilment https://leetcode.com/discuss/interview-question/system-design/320719/Design%3A-Scalable-Shipping-Fulfillment-Center

  8. Flight search API https://leetcode.com/discuss/interview-question/system-design/309853/JSON-structure-for-Flight-search-API

  9. Splitwise https://leetcode.com/discuss/interview-question/system-design/306519/System-Design-or-Splitwise

  10. Google calendar https://leetcode.com/discuss/interview-question/system-design/305654/System-Design-or-Google-Calendar

  11. Fb popular/trending pages https://leetcode.com/discuss/interview-question/system-design/305505/Design-a-most-populartrending-profiles-page

  12. Courier service https://leetcode.com/discuss/interview-question/system-design/301423/Design-a-UPS-style-mail-delivery-system

  13. Hr portal https://leetcode.com/discuss/interview-question/system-design/289092/Design-an-HR-web-portal-for-Amazon's-recruiting-team

  14. Outlook recurring meeting https://leetcode.com/discuss/interview-question/system-design/286891/Design-Outlook-recurring-meeting-system-with-variable-input

  15. Communication system for ecom sites https://leetcode.com/discuss/interview-question/system-design/286457/Design-a-communication-platform.

  16. Imdb https://leetcode.com/discuss/interview-question/system-design/270416/Design-a-movies-reviews-aggregator-system

  17. Realtime event aggregator https://leetcode.com/discuss/interview-question/system-design/270412/Design-a-Real-Time-Event-Aggregation-System

  18. Gpay https://leetcode.com/discuss/interview-question/system-design/270406/Design-a-Payment-System-like-Google-Pay

  19. Top shared post https://leetcode.com/discuss/interview-question/system-design/258398/Design-top-shared-post-system-in-5mins1-hour1-day1-week

  20. Market place analytics https://leetcode.com/discuss/interview-question/system-design/227797/System-Design-E-Commerce-Marketplace-analytics

  21. Top 10 most liked articles https://leetcode.com/discuss/interview-question/system-design/225609/Design-system-which-will-show-top-10-most-liked-articles-within-1524-hours.

  22. Auth for multi tenant https://leetcode.com/discuss/interview-question/system-design/225331/Design-authentication-system-to-multi-tenant-environment

https://leetcode.com/discuss/interview-question/system-design/202958/Multi-Tenant-Saas-Architecture

  1. Design Changefeed https://leetcode.com/discuss/interview-question/system-design/208888/Design-a-system-to-keep-track-of-changes-in-an-SQL-database

  2. Track runners in marathon https://leetcode.com/discuss/interview-question/system-design/200342/Bloomberg%3A-Implement-a-system-to-track-runners-in-a-marathon

  3. Hotel booking page this many people visiting https://leetcode.com/discuss/interview-question/system-design/163204/Design-%22How-Many-people-currently-viewing-the-property%22-for-a-E-Commerce-Hotel-Booking-Site

  4. Text line editor https://leetcode.com/discuss/interview-question/system-design/124679/Implement-a-Text-Line-Editor

  5. Location sharing service https://leetcode.com/discuss/interview-question/system-design/124673/Design-a-Location-Sharing-Android-Application

  6. Build system https://leetcode.com/discuss/interview-question/system-design/124807/Design-a-build-system

  7. Hourly backup from mobile phone https://leetcode.com/discuss/interview-question/system-design/124792/Design-a-system-that-can-handle-hourly-backups-for-mobile-phones

  8. Google help system https://leetcode.com/discuss/interview-question/system-design/125191/Design-the-Google-help-system

 

 

 

 

 

 

 

 

Other interesting post  about specific component design

  1. https://leetcode.com/discuss/interview-question/system-design/136140/Write-a-class-which-is-hard-to-test

  2. Interesting variation of ecart problem where first one who click buys will buy https://leetcode.com/discuss/interview-question/system-design/498895/Startup-interview-or-Designing-tricky-e-shop

  3. Back of envelope calculation https://leetcode.com/discuss/interview-question/system-design/357656/Experience-with-back-of-the-envelope-calculations

  4. Drawing tool https://leetcode.com/discuss/interview-question/system-design/1148896/System-Design-Drawing-Tool-Recos

  5. Very good list https://leetcode.com/discuss/general-discussion/670355/Experienced-Interview-Preparation-Guide-All-Resources

  6. https://leetcode.com/discuss/interview-question/1002218/Facebook-or-Google-or-Top-System-Design-Interview-Questions-(Part-1)

  7. https://github.com/donnemartin/system-design-primer

  8. https://github.com/binhnguyennus/awesome-scalability

  9. Json parser https://leetcode.com/discuss/interview-question/system-design/1052608/Design-a-JSON-Parser-or-SDE2

  10. https://leetcode.com/discuss/interview-question/system-design/1043657/MakeMyTrip-Backend-Developer-or-System-DesignMultithreading

  11. https://leetcode.com/discuss/interview-question/system-design/1042229/Facebook-or-Google-or-Top-System-Design-Interview-Questions-(Part-2)

  12. https://leetcode.com/discuss/interview-question/system-design/1038585/How-do-you-scale-up-an-Application-to-serve-thousands-of-request-per-second

  13. User data access policy design https://leetcode.com/discuss/interview-question/system-design/895268/Google-or-System-Design

  14. https://leetcode.com/discuss/interview-question/system-design/829466/Amazon-or-Phone-or-Seattle-or-Column-Store-vs-Row-Store

  15. https://leetcode.com/discuss/interview-question/system-design/808216/Phone-book-and-search - suffix tree

  16. https://hackernoon.com/scaling-websockets-9a31497af051

  17. Online whiteboard drawing like draw.io

  18. https://leetcode.com/discuss/interview-question/system-design/799474/Virtual-onsite-at-DocuSign

  19. https://leetcode.com/discuss/interview-question/system-design/795890/How-many-transactions-does-Oracle-DB-handle-Read-and-write

  20. How to store Recently viewed item https://leetcode.com/discuss/interview-question/system-design/775139/Amazon-System-Design-customer's-recently-viewed-items

  21. Depth an interviewer can go https://leetcode.com/discuss/interview-question/system-design/773980/Watch-this-before-System-Design-Interview-the-details-an-interviewer-can-go-to-evaluate-candidate

  22. Privacy api https://leetcode.com/discuss/interview-question/system-design/727474/System-design-critique-request-for-below-question

  23. LLD https://leetcode.com/discuss/interview-question/system-design/692383/Google-or-Onsite-or-Design-a-organization-pharmacy-shop-with-managers

  24. T9 predicitve system https://leetcode.com/discuss/interview-question/system-design/685338/Microsoft-or-Onsite-or-Design-the-T9-predictive-text-algorithm-and-system

  25. Storsge of millions of subscriber https://leetcode.com/discuss/interview-question/system-design/680047/How-will-you-store-millions-of-subscribers-list-(assume-it-as-email-id)

  26. Flipkart warehouse portal 

https://leetcode.com/discuss/interview-question/system-design/663037/Amazon-SystemDesign-Flipkart-Suggestions-Design-Warehouse-Portal

  1. https://leetcode.com/discuss/interview-question/system-design/632537/Design-an-algorithm-to-efficiently-transfer-required-bytes-of-data-to-a-single-PC-on-the-network.

  2. https://leetcode.com/discuss/interview-question/system-design/581804/Facebook-System-Design-Preparation - different type of design interviews at facebook

  3. Count current active user on the page, https://leetcode.com/discuss/interview-question/system-design/557603/Postman-or-OA-or-System-Design

https://leetcode.com/discuss/interview-question/system-design/532889/Design-a-counter-for-a-website-which-tells-how-many-visits-happened-to-that-website

  1. Csv parsing at scale https://leetcode.com/discuss/interview-question/system-design/545664/CSV-parsing-at-scale

  2. API to get best selling book https://leetcode.com/discuss/interview-question/system-design/535162/Design-read-api-to-get-best-selling-books

  3. Fb mutual friend https://leetcode.com/discuss/interview-question/system-design/533810/FB-API-System-design

  4. https://leetcode.com/discuss/interview-question/system-design/532089/Update-System-design-of-Amazon-to-handle-10x-times-more-traffic-than-what-it-currently-receives

  5. https://leetcode.com/discuss/interview-question/system-design/513374/FAANG-system-design-interview-question

  6. Sql scalability https://leetcode.com/discuss/interview-question/system-design/507833/SQL-Scalability

  7. Backup from dc1 to dc2 https://leetcode.com/discuss/interview-question/system-design/502522/System-design-to-backup-datacenter1-to-datacenter-2

  8. Order within next 1 hour to get early delivery feature https://leetcode.com/discuss/interview-question/system-design/499558/Amazon-System-Design-Question

  9. https://leetcode.com/discuss/interview-question/484956/design-airport-luggage-handling-system

  10. https://leetcode.com/discuss/interview-question/system-design/483959/Google-onsite-(Theoretical-System-Design)

  11. https://leetcode.com/discuss/interview-question/system-design/421969/Twitch-or-System-Design-Onsite-or-Design-Twitch-Analytics-Use-Case

  12. Good discussion on designing SAAS https://leetcode.com/discuss/interview-question/system-design/385363/Design-Task-Executor-as-Saas

  13. Track down bad request https://leetcode.com/discuss/interview-question/system-design/313117/Design-a-system-capable-of-tracking-bad-request-down-quickly

  14. Facebook https://leetcode.com/discuss/interview-question/system-design/311825/Facebook-System-design

  15. Flight landing with onerunway  https://leetcode.com/discuss/interview-question/system-design/303745/Handling-flight-landing-requests-with-one-runway

  16. Good read on data modelling https://leetcode.com/discuss/interview-question/system-design/295671/Amazon-or-Data-Engineer-Role-or-Database-Design-Question

  17. Traffic light controller https://leetcode.com/discuss/interview-question/system-design/291233/Design-a-traffic-light-controller

  18. Live streaming view count https://leetcode.com/discuss/interview-question/system-design/284232/Livestreaming-view-count.

  19. Random number generator for a slot mschine https://leetcode.com/discuss/interview-question/system-design/281762/Microsoft-or-Design-a-random-number-generator-for-a-slot-machine

  20. Company badge system https://leetcode.com/discuss/interview-question/system-design/279371/Design-a-badge-system

  21. API gateway with graphql https://leetcode.com/discuss/interview-question/system-design/255282/Design-API-gateway

  22. Good read on scaling https://leetcode.com/discuss/interview-question/system-design/250803/Large-amount-of-query-request-in-seconds.

  23. https://leetcode.com/discuss/interview-question/system-design/249988/How-to-Transfer-1-GB-of-data-from-one-microservice-to-another

  24. https://leetcode.com/discuss/interview-question/system-design/234311/Celebrity-Twitter-hack

  25. Amazon locker https://leetcode.com/discuss/interview-question/system-design/233869/Design-Amazon-Locker-system

  26. Upload large file https://leetcode.com/discuss/interview-question/system-design/224398/Algorithm-for-upload-large-file

  27. Currency civersation https://leetcode.com/discuss/interview-question/system-design/216912/Design-an-app-that-converts-USD-to-another-country-currency.

  28. Geohash https://leetcode.com/discuss/interview-question/system-design/203364/How-to-retrieve-addresses-based-on-Latitude-and-Longitude

https://leetcode.com/discuss/interview-question/system-design/124568/How-to-design-a-system-to-retrieve-address-information-longitude-and-latitude-information.

  1. Strategy to update sattelite firmware https://leetcode.com/discuss/interview-question/system-design/198761/Updating-Satellite-Frimware

  2. Design card deck https://leetcode.com/discuss/interview-question/system-design/194663/Design-a-class-that-represents-a-deck-of-cards

  3. Send large files https://leetcode.com/discuss/interview-question/system-design/193953/Distribute-binary-file-(daily)-for-thousands-of-servers

  4. Extract json from a text file https://leetcode.com/discuss/interview-question/system-design/191944/How-to-extract-JSON-object-from-a-text-file-of-size-100-GB

  5. https://leetcode.com/discuss/interview-question/system-design/177823/How-to-track-overall-activity-time-of-every-user-efficiently

  6. Design video pause functionality https://leetcode.com/discuss/interview-question/system-design/177327/Video-pause-functionality-for-multiple-devices

  7. https://leetcode.com/discuss/interview-question/system-design/172809/Sync-front-end-with-back-end-in-realtime

  8. Push/pull API for producer consumer https://leetcode.com/discuss/interview-question/system-design/158701/Design-push-and-pull-APIs-for-a-producer-consumer-system

  9. HA for webproxy https://leetcode.com/discuss/interview-question/system-design/134797/HA-for-HTTP-proxy

  10. Alarm app https://leetcode.com/discuss/interview-question/system-design/133426/Microsoft-Mobile-app-for-alarm

  11. Snake and ladder https://leetcode.com/discuss/interview-question/system-design/132140/Design-online-multiplayer-snakeandladder-game

  12. https://leetcode.com/discuss/interview-question/system-design/124858/First-non-repeating-word-in-a-file-File-size-can-be-100GB.

  13. Store extremely large parse matrix https://leetcode.com/discuss/interview-question/system-design/125306/How-to-store-extremely-large-sparse-matrices

  14. Implement mine swipper

 

 

 

OOP design https://github.com/tssovi/grokking-the-object-oriented-design-interview

https://github.com/savitansh/SystemDesignInterview

 

 

https://leetcode.com/discuss/interview-question/system-design/943886/Facebook-Product-Design-Questions

 

https://github.com/checkcheckzz/system-design-interview

GitHub - donnemartin/system-design-primer: Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.

 

 

 

Drawing tool
https://leetcode.com/discuss/interview-question/system-design/758105/Remote-system-design-diagram-drawing-tool

 

Other informstive posts

 

I hope it will be helpful if you have mentioned your overall experience.

If you are looking for interview perspective, You need to gain knowledge about distributed systems.

To start with, to simplify it,

For a Simple Application

Input --> Processing --> DataStore --> Processing/Output

For Distributed Systems [I wrote considering for System Design interview]

Following contains quick overlook about the list that need to have for basic understanding.

  1. Input/Output
    Communication[Rest, grpc, WebSocket, tcp/udp, webRTC], API Gateway, Proxy Server, Load Balancing -- [Reverse Proxy]
    Message Queue -- [Queue for sending/receiving information] Kafka/RabitMQ

  2. Processing
    Computation -- Micro-service [Python, Springboot/Java, Go...], MapReduce
    Service Discovery/Registry -- [Finding and Redirecting the load, Saga Pattern] -- Eg: Eureka

  3. Datastore/Distributed Consensus
    Cache -- [Simple Cache to reduce database hits] -- Eg: Redis
    SQL -- [SQL database for Relational Data -- Transactions based like payments, Horizontal Partition/Vertical Partition, Shading ] Eg: MySQL
    NOSQL -- [NOSQL for querying documents like product, product details, CAP] Eg: MongoDB/Apache Cassandra
    Concurrency -- 2 Phase Commit, 3 Phase Commit, Saga Pattern[Choreography saga, Orchestrator Saga], Split Brain problem.
    FileSystem -- Hadoop File System

  4. Security
    AAA -- Authentication, Authorisation, Auditing

Misc -- Logging/Notification

Once you have overall picture in your mind, you can start with YouTube videos like Gaurav Sen or Tech Dummies.

 

 

 

 

Product Design

The product design interview at Facebook will involve designing a product or API to support an end-user experience. Here's a list of concepts that Facebook recommends you review before your interview:

-Scalability

-Design patterns

-Data ownership

-Protocols

-Data formats

-Client-server design

-Designing for long term vs. complexity

-Accommodating possible product changes

Some example questions involve designing a product API or an email server.

 

 

System Design

The system design interview at Facebook will ask you to weigh design considerations for complex problems. Here's a list of concepts that Facebook recommends you review before your interview:

-Concurrency (threads, deadlock, starvation, consistency, coherence)

-Caching

-Database partitioning, replication, sharding, CAP Theorem

-Networking (IPC, TCP/IP)

Real-world performance (relative performance RAM, disk, your network, SSD)

-Availability and reliability (types of failures, failure units, how failures may manifest, mitigations, etc.)

-Data storage and data aggregation

-QPS capacity/machine estimation (back of the envelope estimates), byte size estimation

Some example questions involve architecting a video distribution system or designing a mobile image search client.

 

 

Tags:

0

Api design .. remember these things

by Deepak Dhakal 27. January 2020 17:06

API Design

 Statelessness – There’s one place you don’t want your API to be storing state, and that’s in your application servers. Always keep application servers state-free so that they can be easily and painlessly scaled.

Content Negotiation – If you want to support multiple representations of your resources, you can use content negotiation (eg. Accept headers), or differing URLs for different representations (eg. …?format=json), or you can combine both and have your content negotiation resources redirect to specific formats.

 URI Templates – URI Templates are a well-defined mechanism for providing URL composition capabilities to your clients, or for documenting your URL access patterns to your end-user.

Design for Intent – Don’t just expose your internal business objects through your API. Design your API to have semantic meaning and to match the use-cases that your users will have. Darrel Miller wrote a great post on API Craft describing this better than I could. (Edit: I tried my best in an article entitled Stop Designing Fragile Web APIs.)

 Versioning – Theoretically, if you designed a really great API up front, you might never need to create incompatibilities in your API. For the pragmatists in us, put versioning in your API URLs (eg. a /v1/ path), so that you have a safety net in case the API doesn’t work out like you wanted. (Edit: An expanded justification is my follow-up article: Ain’t Nobody Got Time For That: API Versioning)

 Authorization – Remember when designing your API that not all users will have access to all objects in the system. It’s great if you use or build an API framework with some form of declarative security so that it’s easy to assign and modify authorization rights on read and write access to resources.

 Bulk Operations – Most clients will perform better if they can issue fewer requests to fetch or modify more data. It’s a good idea to build bulk operations into your API to support this kind of use case.

 Pagination – Pagination serves two big purposes in an API; it reduces the amount of unneeded data delivered to the client, and it reduces the amount of unneeded computation on your application servers. There are many different patterns used for making paginated collection resources; if you don’t know where to start, browse through Stackoverflow for some hints. If you want to be my personal hero, implement consistent pagination by providing links to additional pages that are timestamped or versioned, such that you will never see duplicate results in pagination requests even if the objects involved change.

 Unicode – It’s pretty obvious these days that you need to support more than English characters in a web service; just remember to keep this in mind when designing and testing your API. In particular, Unicode characters can be interesting if you use them as natural keys in a URL (eg. /users/jimbob/ becomes /users/싸이/).

 Error Logging – Be sure you design how you want your API to perform error logging, rather than just throwing it together. In particular, I find it extremely valuable to distinguish between errors that are caused by the client’s input, and errors that are caused by your software. Keep these in two separate logs.

Tags:

0

Broaden your knowledge with this well known company's blog

by Deepak Dhakal 11. August 2019 01:04

Tags: ,

0

Consistent Hashing : Save Your Weekends

by Deepak Dhakal 10. August 2019 06:01

Consistent Hashing

Consistent hashing is one of the techniques used to bake in scalability into the storage architecture of your system from grounds up.

In a distributed system, consistent hashing helps in solving the following scenarios:

  1. To provide elastic scaling (a term used to describe dynamic adding/removing of servers based on usage load) for cache servers.
  2. Scale out a set of storage nodes like NoSQL databases.

It is a very useful concept that frequently comes up in System Design Interviews. You might need to apply the concept while designing the backend of a system to alleviate bottlenecks . You could also be directly asked to design and implement  a consistent hashing algorithm. In this article, we'll look at:

Why do we need Consistent Hashing ?

Imagine that you want to create a scalable database backend with "n" database servers for your web application as depicted by the diagram below. For our simple example, we'll assume that we're just storing a key:value pair like "Country:Canada"in the DBs.

Figure 1: A distributed system with a cluster of database servers

Our goal is to design a database storage system such that:

  1. We should be able to distribute the incoming queries uniformly among the set of "n" database servers
  2. We should be able to dynamically add or remove a database server
  3. When we add/remove a database server, we need to move the minimal amount of data between the servers

So essentially we need to send each piece of incoming query to a  specific server. A simple approach is as follows:

  1. Generate a hash of the key from the incoming data :  hashValue = HashFunction(Key)"
  2. Figure out the server to send the data to by taking the modulo ("%") of the hashValue using the number of current db servers, n : "serverIndex = hashValue % n"

Let's walk through a simple example.

  • Imagine we have 4 database servers
  • Imagine our hashFunction returns a value from 0 to 7
  • We'll assume that "key0" when passed through our hashFunction, generates a hashvalue or 0, "key1" generates 1 and so on.
  • The serverIndex for "key0" is 0, "key1" is 1 and so on.

The situation assuming that the key data is unfirmly distributed, is depicted in the image below. We receive 8 pieces of data and our hashing algorithm distributes it evenly across our four database servers.

Figure 2: Sharding/ Distributing data across several database servers

Problem solved, right ?  Not quite – there's two major drawbacks with this approach, namely, Horizontal Scalability and Non-Uniform data distribution across servers.

Horizontal Scalability

This scheme is not horizontally scalable. If we add or remove servers from the set, all our existing mappings are broken. This is because the value of "n" in our function that calculates the serverIndex changes. The result is that all existing data needs to be remapped and migrated to different servers. This might be a herculean task because it'll either require a scheduled system downtime to update mappings or creating read replicas of the existing system which can service queries during the migration. In other words, a lot of pain and expenditure.

Here's a quick illustration of what happens when we add another server (server 5)  to the mix. Please refer back to figure 1 for the original key distribution. Notice that we'll need to update 3 out of the original 4 servers – i.e. 75% of servers needs to be updated!

Figure 3: Effect of adding a database server to the cluster

The effect is more drastic when a server goes down as depicted below. In this case, we'll need to update ALL servers, i.e.,100% of servers needs to be updated !

Figure 4: Effect of removing a server from the database cluster

Data Distribution – Avoiding  "Data Hot Spots" in Cluster

We cannot expect uniform distribution of data coming in all the time. There may be many more keys whose hashValue maps to server number 3 than any other servers , in which case server number 3 will become a hotspot for queries.

Consistent hashing allows up to solve both these problems. Read on to find out how !

How does Consistent Hashing Work ?

Consistent hashing facilitates the distribution of data across a set of nodes in such a way that minimizes the re-mapping/ reorganization of data when nodes are added or removed. Here's how it works:

1. Creating the Hash Key Space: Consider we have a hash function that generates integer hash values in the range [0,  2^32-1)

We can represent this as an array of integers with 2^32 -1 slots. We'll call the first slot x0 and the last slot xn – 1

Figure 5: A Hash Key Space 

2. Representing the hashSpace as a Ring: Imagine that these integers generated in step # 2 is placed on a ring such that the last value wraps around.

Figure 6: Visualizing the hash key space as a Ring

3. Placing DB Servers in Key Space (HashRing): We're given a list of database servers to start with. Using the hash function, we map each db server to a specific place on the ring. For example, if we have 4 servers, we can use a hash of their IP addressed to map them to different integers using the hash function. This simulates placing the four servers into a different place on the ring as shown below.

Figure 7: Placing database servers on a hash ring

  1. Determining Placement of Keys on Servers: To find which database server an incoming key resides on (either to insert it or query for it ), we do the following:
  • Run the key through the same hash function we used to determine db server placement on the ring.
  • After hashing the key, we'll get an integer value which will be contained in the hash space, i.e., it can be mapped to some postion in the hash ring. There can be two cases:
  1. The hash value maps to a place on the ring which does not have a db server. In this case, we travel clockwise on the ring from the point where the key mapped to untill we find the first db server. Once we find the first db server travelling clockwise on the ring, we insert the key there. The same logic would apply while trying to find a key in the ring.
  2. The hash value of the key maps directly onto the same hash vale of a db server – in which case we place it on that server.

Example: Assume we have 4 incoming keys : key0, key1, key2, key3 and none of them directly maps to the hash value of any of the 4 servers on our hash ring. So we travel clockwise from the point these keys maps to in our ring till we find the first db server and insert the key there. This is depicted in Figure 7 below.

Figure 8: Key placements on database servers in a hash ring

5. Adding a server to the Ring: If we add another server to the hash Ring, server 4, we'll need to remap the keys. However, ONLY the keys that reside between server 3 and server 0 needs to be remapped to server 4. On an average , we'll need to remap only k/n keys , where k is the number of keys and n is the number of servers. This is in sharp contrast to our modulo based placement approach where we needed to remap nearly all the keys.

The figure below shows the effect of inserting a new server4 – since server 4 now resides between key0 and server0, key0 will be remapped from server0 to server4.

Figure 9: Effect of adding a server to the hash ring

6. Removing a server from the ring: A server might go down in production and our consistent hashing scheme ensures that it has minimal effect on the number of keys and servers affected.

As we can see in the figure below, if server0 goes down, only the keys in between server3 and server 0 will need to be remapped to server 1( the area is circled in yellow). The rest of the keys are unaffected .

Figure 10: Effect of removing a server from the hash ring

At this point, consistent hashing has  successfully solved the horizontal scalability problem by ensuring that every time we scale up or down, we DO NOT have to re-arrange all the keys or touch all the database servers !

But what about the distribution of data across the various database servers? We can run into a situation where our server distribution across the hash ring is non uniform , i.e., the size of partitions each server is responsible for is not the same. But you might ask how will that happen ? Well, imagine that we started off with 3 servers (server0, server1, server2) that were more or less evenly distributed across the ring. If one of the servers fail, then the load seen by the server immediately following the failed server will be higher. This is assuming all the data that comes in has a uniform key distribution. In reality , the issue is more complicated because data does not have uniform distribution in most cases. So these two things coupled together can lead to a situation like the one shown below. Here, server0 is seeing a very high load because :

  1. Data was non-uniformly distributed to start with – so server2 was having a lot of hot spots
  2. Server2 eventually fails and had to be removed from the hash ring. (note that server 0 now gets all of server2's keys)

Figure 11: Keys can be non-uniformly distributed across servers in a hash ring

So how do we solve this ?

It turns out that there is a pretty standard solution to the problem. It involves the introduction of a number of replicas or virtual nodes for each server across the ring. For example,

Server 0 might have two replicas placed at different points across the ring.

Figure 12: Using Virtual Nodes to assign increase the key space covered by each server

But how does using replicas make the key distribution more uniform ? Here's a visual example – Figure 13 shows the key distribution with two serevers in the hash ring WITHOUT replicas. We can observe that server 0 is handling 100% of the keys.

Figure 13: Non-uniform key distribution in absence of replication of nodes in a hash ring

If we introduce one more replica of each server on the ring , then the key distribution looks like the one in figure 14. Now server0 is responsible for 50% ( 2 out of 4) keys and server 1 is responsible for the other 50% of the keys.

Figure 14: Using virtual nodes/ replication to create better key distribution in a hash ring

As the number of replicas or virtual nodes in the hash ring increase, the key distribution becomes more and more uniform. In real systems, the number of virtual nodes / replicas is very large (>100) .

At this point, Consistent Hashing has successfully solved the problem of non-uniform data distribution (hot spots) across our database server cluster.

Key things to remember about Consistent Hashing for System Design Interviews

SCENARIOS WHERE TO USE CONSISTENT HASHING

  1. You have a cluster of databases and you need to elastically scale them up or down based on traffic load. For example, add more servers during Christmas to handle the the extra traffic.
  2. You have a set of cache servers that need to elastically scale up or down based on traffic load.

BENEFITS OF CONSISTENT HASHING:

  1. Enables Elastic Scaling of cluster of database/cache servers
  2. Facilitates Replication and partitioning of data across servers
  3. Partitioning of data enables uniform distribution which relieves hot spots
  4. Points a-c enables higher availability of the system as a whole.

Implementation Consistent Hashing

Please note that this is for purely illustrative purposes only. There are no guarantees for robustness or stability if used in production code.

There are three key pieces we need to implement:

  1. A Hash table like data structure which can simulate the key space or the hash Ring. In our case, we'll use a SortedDictionary in C#
  2. A hash function that can generate a integer value for the servers ip address and incoming keys we need to map to the hash ring
  3. The server object themselves.

First we define a server class which basically encapsulates an ip address and represent a physical server.

  1. using System.Collections.Generic;
  2. using System.Linq;
  3. using System.Text;
  4. using System.Threading.Tasks;
  5. namespace ConsistentHashing
  6. {
  7. class Server
  8. {
  9. public String ipAddress;
  10. public Server(String ipAddress)
  11. {
  12. this.ipAddress = ipAddress;
  13. }
  14. }
  15. }

Next we define the hash function which will return an integer value for server ips and the keys.

  1. using System;
  2. using System.Collections.Generic;
  3. using System.Linq;
  4. using System.Text;
  5. using System.Threading.Tasks;
  6. /*
  7. * This code is taken from the stackoverflow article:
  8. * https://stackoverflow.com/questions/12272296/32-bit-fast-uniform-hash-function-use-md5-sha1-and-cut-off-4-bytes
  9. */
  10. namespace ConsistentHashing
  11. {
  12. public static class FNVHash
  13. {
  14. public static uint To32BitFnv1aHash(string toHash, bool separateUpperByte = false)
  15. {
  16. IEnumerable<byte> bytesToHash;
  17. if (separateUpperByte)
  18. bytesToHash = toHash.ToCharArray()
  19. .Select(c => new[] { (byte)((c - (byte)c) >> 8), (byte)c })
  20. .SelectMany(c => c);
  21. else
  22. bytesToHash = toHash.ToCharArray()
  23. .Select(Convert.ToByte);
  24. //this is the actual hash function; very simple
  25. uint hash = FnvConstants.FnvOffset32;
  26. foreach (var chunk in bytesToHash)
  27. {
  28. hash ^= chunk;
  29. hash *= FnvConstants.FnvPrime32;
  30. }
  31. return hash;
  32. }
  33. }
  34. public static class FnvConstants
  35. {
  36. public static readonly uint FnvPrime32 = 16777619;
  37. public static readonly ulong FnvPrime64 = 1099511628211;
  38. public static readonly uint FnvOffset32 = 2166136261;
  39. public static readonly ulong FnvOffset64 = 14695981039346656037;
  40. }
  41. }

Finally, we define the consistent hash class which enacpsulates the logic for :

  1. Creating the hash ring
  2. Adding a server to the hash ring
  3. Removing a server from the hash ring
  4. Getting the location of the server on the hash ring where a key needs to be added / retrieved from.
  1. using System;
  2. using System.Collections.Generic;
  3. using System.Linq;
  4. using System.Text;
  5. using System.Threading.Tasks;
  6. namespace ConsistentHashing
  7. {
  8. class ConsistentHash
  9. {
  10. private SortedDictionary<uint, Server> hashRing;
  11. private int numberOfReplicas; // The number of virtual nodes
  12. public ConsistentHash(int numberOfReplicas, List<Server> servers)
  13. {
  14. this.numberOfReplicas = numberOfReplicas;
  15. hashRing = new SortedDictionary<uint, Server>();
  16. if(servers != null)
  17. foreach(Server s in servers)
  18. {
  19. this.addServerToHashRing(s);
  20. }
  21. }
  22. public void addServerToHashRing(Server server)
  23. {
  24. for(int i=0; i < numberOfReplicas; i++)
  25. {
  26. //Fuse the server ip with the replica number
  27. string serverIdentity = String.Concat(server.ipAddress, ":", i);
  28. //Get the hash key of the server
  29. uint hashKey = FNVHash.To32BitFnv1aHash(serverIdentity);
  30. //Insert the server at the hashkey in the Sorted Dictionary
  31. this.hashRing.Add(hashKey, server);
  32. }
  33. }
  34. public void removeServerFromHashRing(Server server)
  35. {
  36. for (int i = 0; i < numberOfReplicas; i++)
  37. {
  38. //Fuse the server ip with the replica number
  39. string serverIdentity = String.Concat(server.ipAddress, ":", i);
  40. //Get the hash key of the server
  41. uint hashKey = FNVHash.To32BitFnv1aHash(serverIdentity);
  42. //Insert the server at the hashkey in the Sorted Dictionary
  43. this.hashRing.Remove(hashKey);
  44. }
  45. }
  46. // Get the Physical server where a key is mapped to
  47. public Server GetServerForKey(String key)
  48. {
  49. Server serverHoldingKey;
  50. if(this.hashRing.Count==0)
  51. {
  52. return null;
  53. }
  54. // Get the hash for the key
  55. uint hashKey = FNVHash.To32BitFnv1aHash(key);
  56. if(this.hashRing.ContainsKey(hashKey))
  57. {
  58. serverHoldingKey = this.hashRing[hashKey];
  59. }
  60. else
  61. {
  62. uint[] sortedKeys = this.hashRing.Keys.ToArray();
  63. //Find the first server key greater than the hashkey
  64. uint firstServerKey = sortedKeys.FirstOrDefault(x => x >= hashKey);
  65. // Get the Server at that Hashkey
  66. serverHoldingKey = this.hashRing[firstServerKey];
  67. }
  68. return serverHoldingKey;
  69. }
  70. }
  71. }

Finally, here's a test program which exercises the functionality of the above code.

  1. using System;
  2. using System.Collections.Generic;
  3. using System.Linq;
  4. using System.Text;
  5. using System.Threading.Tasks;
  6. using System.Security.Cryptography;
  7. namespace ConsistentHashing
  8. {
  9. class Program
  10. {
  11. static void Main(string[] args)
  12. {
  13. List<Server> rackServers = new List<Server>();
  14. rackServers.Add(new Server("10.0.0.1"));
  15. rackServers.Add(new Server("10.0.0.2"));
  16. int numberOfReplicas = 1;
  17. ConsistentHash serverDistributor = new ConsistentHash(numberOfReplicas, rackServers);
  18. //add a new server to the mix
  19. Server newServer = new Server("10.0.0.3");
  20. serverDistributor.addServerToHashRing(newServer);
  21. //Assume you have a key "key0"
  22. Server serverForKey = serverDistributor.GetServerForKey("key0");
  23. Console.WriteLine("Server: " + serverForKey.ipAddress + " holds key: Key0");
  24. // Now remove a server
  25. serverDistributor.removeServerFromHashRing(newServer);
  26. // Now check on which server "key0" landed up
  27. serverForKey = serverDistributor.GetServerForKey("key0");
  28. Console.WriteLine("Server: " + serverForKey.ipAddress + " holds key: Key0");
  29. }
  30. }
  31. }

OUTPUT:

  1. Server: 10.0.0.3 holds key: Key0
  2. Server: 10.0.0.2 holds key: Key0

Consistent Hashing in Action in Production Systems

There are a number of live systems which use consistent hashing including:

Further Reading On Consistent Hashing

1. Tom White's article on Consistent Hashing is the one i used to initially learn about this technique. The C# implementation in this article is loosely based on his java implementation.

2. Tim Berglund's Distributed System in One Lesson is a fantastic resource to learn about read replication, sharding and consistent hashing. Unfortunately, you'll need a safari membership for this.

3. David Karger and Eric Lehman's original paper on Consistent Hashing

4. David Karger and Alex Sherman's paper on Web Caching with Consistent Hashing

If you have any feedback, please add it to the comment section below. And if you enjoyed the article, please share it on your favorite social media platform 🙂

Tags: ,

Powered by BlogEngine.NET 1.5.0.7
Original Design by Laptop Geek, Adapted by onesoft