I'm not a data engineer or an analyst although data analytics was my specialization in university. I simply picked this specialization as it's the only comp sci program in the scholarship that I applied to. I had no special interest in data analytics. Additionally, I've not yet had to put my data analytics knowledge to the test in a real life scenario, mainly due to lack of interest in the field but also because the demand (compared to web development) for this skill is quite low.
Recently in my workplace, the need arose for a data warehouse. We have dozens of separate database that store data of their respective applications. With the data being decentralized, it was difficult to analyze and join data across all the applications. Additionally, connecting to live databases was risky due to potential performance and security issues.
The solution I decided, was to create a data warehouse. Despite there being numerous data warehousing solutions available both paid and free, I wanted to take on the challenge of creating my own data warehousing solution from scratch. With the limited knowledge I got from university that I vaguely remember, I began this task. Note that I was not assigned this task, this is just something I wanted to do.
Most of the data warehousing solutions are intimidating to set up and implement. I wanted this to be straightforward with minimal configuration. The application is a Go program which can compile to a single binary on any platform. The configuration is a single JSON file which contains all the required information that the program needs. This information includes the database connection strings, the tables to be copied, the columns to be copied, the primary keys, etc.
to be continued...