Subrat's Technical Blog

  Introduction In this post we explore the shifting focus of Data Engineering from  Data Infrastructure  and  Data Integration  to  Accessible Analytics , in Meta and the industry at large. However the tools that we use are largely focused on the former two which presents a major challenge for the function. We will showcase challenges this mismatch created during the evolution of Data Engineering at Meta and will formulate the properties of a next-generation data platform. In subsequent posts we will review how Data Engineering at Meta is evolving to address these challenges. But first let’s define what we mean by Accessible Analytics. We are not using the term Accessible Analytics as it is commonly understood, i.e., the act of making data accessible for extracting insights through tools.  Instead, we are also drawing attention to the fact that for data to  truly  be deemed accessible it will have to be self-describing to an extent that it do...

  AWS Architecture in Motion: Creating Animated GIFs Several days ago, I saw on LinkedIn a  great article by Ankit Jodhani  concerning one of his projects, and his architecture diagram impressed me. There was one common question in his post:  “How did you do it?” . All the people were referring to their diagram not being static, it had animations. In this guide, I’ll try to provide a step-by-step guide on how to make your own animated AWS architecture diagrams using PowerPoint and GIFs. GIFs make the diagrams more interactive and help visualize how infrastructure behaves. I hope this will encourage you to learn and develop more skills. Some key benefits of documenting architecture with diagrams are: Visualize infrastructure clearly and organized. Identify how components communicate with each other. Detect failure points or bottlenecks in the design. Explain architecture to new team members. Make changes and updates to infrastructure. Let’s get started! Prerequisites ...