Subrat's Technical Blog

import pandas as pd import os import sqlalchemy from sqlalchemy import create_engine from sqlalchemy import inspect SOURCE_SERVER = os.environ.get( 'MS SQL SERVER NAME' ) SOURCE_DRIVER = os.environ.get( 'MS SQL SERVER DRIVER' ) source_database_name = 'olympics_database' destination_database = os.environ.get( 'postgresdbpass' ) source_database_connection = sqlalchemy.create_engine( f'mssql:// {SOURCE_SERVER} / {source_database_name} ?driver= {SOURCE_DRIVER} ' ) destination_database_connection = sqlalchemy.create_engine( f'postgresql://postgres: {postgresdbpass} @localhost:5432/olympics_db' ) df = pd.read_sql( "select * from table_name" , source_database_connection) df.to_sql( "table_name" , destination_database_connection, if_exists = append) inspector = inspect(source_database_connection) table_names_list = inspector.get_table_names() #Above code generates a list of all tables in the database def extract_and_l...

  Important Resources and concepts to prepare A pache Spark is most used parallel processing engine used in Data Engineering and Databricks is the best platform providing efficient version of Apache Spark with extra features. So, This Certification will help you a lot as a Data Engineering to crack interviews and in your Career. In this blog, I have prepared complete guide mentioning Topics to prepare for the Exam and Resource I followed to crack this Exam! Important Note: First of all, I want to mention,   It’s not an easy exam to crack , So you need to learn so many concepts in Databricks and Apache Spark and   I would Highly suggest   you to complete   Databricks Certified Data Engineer Associate Certification. Associate Certification is   not an mandatory/pre-requisites ,   but you need all the skills of Associate Certification.   Personally I did not attended for Associate Certification but I had learned all the skills of Associate and then s...