Data Engineering Trends | Saxon AI
Data Analytics, Data Architecture / January, 30 2023

8 Interesting Data Engineering Trends in 2023

In this digital era, we have zettabytes of data across industries, AI for insights, and access to intelligent automation tools. Data scientists, analysts, data engineers, and the data operations team play a crucial role in this value chain. To extract quality insights data engineering practices are the first step. So, let us look at a few data engineering trends in 2023. 

According to Forbes, 80% of work in AI and analytics projects is mostly data engineering and around 20% goes to generating insights. With evolving AI and ML usage, many experts argue that data engineering efforts will reduce drastically in the next ten years. But the reality may be different. Let us find out. 

Data growth  

By 2025, around 463 exabytes of data will be generated globally daily. Smart homes, IoT, 5G, blockchain, and wearables contribute to this exponential growth in our lives. Also, you can expect the data in multiple formats, structures, and volumes. This further complicates data engineering. 

Business growth 

Global GDP is expected to grow by 40% between 2016-30, as reported by WHO. Businesses will continue to enter new markets and introduce new products and services in line with this economic growth. And M&A to fuel this growth trend too. The main success factor in M&A is about a well though data integration strategy. 

Data regulations 

The regulations related to data privacy expanded over the past few years across the globe. Most of the world’s population prefers to get covered under these regulations. Data privacy concerns lead organizations to improve their investments in data engineering. 

In the wake of all these changes, data engineering always remains the focus area for businesses. Hence let us look at a few data engineering trends in 2023. Most organizations tend to leave short-term trends and focus on long-term needs. But it is important to understand the changes in the upcoming 6-12 months too. 

Data Engineering Trends in 2023 

More Focus on Data Cloud Cost Optimization 

Cloud became the front-runner in data monetization in recent years and organizations are looking for cloud operations. The transition to the cloud also accelerated in the wake of the pandemic. With organizations anticipating recession in the future, the focus is shifting to cost optimization around the cloud. 

Data engineering teams had to work around speed and agility with the high expectations set for them. Though people work at a rapid pace, best practices are relatively new. Most of the time was spent on developing new queries and handling large data volumes instead of optimizing complex queries.  

Elastic storage and massive compute options have accelerated cloud adoption but costs also increased for organizations. Organizations need to focus on data cloud cost optimization in the near future. You can also see large vendors like Snowflake, and BigQuery highlighting options for data cloud cost optimization. This seems to be one of the critical data engineering trends in 2023. 

Data Catalog and Observability 

Data pipelines have been playing a major role in the data engineering processes. So far, major practices were set to extract, transform and load the data. But it is still sensitive to changes in the data structure or values. Usually, it may lead to failures and unavailability.  

The data catalog will facilitate the access and discovery of data by internal and external users. Users can manage and organize the data better so that any changes in the structure and format may not impact the data pipelines. 

Observability on the other hand understands the data flows and processes to diagnose any issues that may arise and impact the data engineering process. Both of them play a significant role in maintaining and optimizing data pipelines.  

Multi-model vs Unistore database 

Usually, the data needs to be moved between transactional and analysis-specific databases to draw insights. But the latest introduction from Snowflake, the Unistore, blurred the lines between OLTP and OLAP systems. You no longer need to copy or move the data between systems with the new concept from Snowflake. 

It is not very far before we see similar moves in the data engineering space by other vendors.  

What are multi-model databases?  

Are you dealing with relational, graph, and object-oriented data models for your needs? The line between the different databases is also blurred. The new multi-model database management system supports different data structures and interactions with data. Users can choose an appropriate data model as per their specific needs. Multi-model DBMS are supposed to reduce the total cost of ownership in data engineering. 

Central Data Platforms continue but Data Mesh Adoption Rises 

Data Mesh has been under consideration for several years by many organizations. The domain architectures, data as a product, and self-service options in data mesh became the hottest topic for discussion in the past few years. Do you see value in it? A few organizations already embarked on this journey and few others have it on the road map. But experts contemplate the journey to data mesh continues and the powerful central platform may also retain. The underlying thought is mostly due to consistent standards of centralized teams. 

Data Contracts – Early Adoption 

Unexpected schema changes impact the data quality for many reasons. And organizations are facing this challenge in addressing trust issues. Data contracts are like delivery contracts, part of the metadata-driven ingestion framework. Usually, stored in a centrally managed metastore, they help in data pipeline execution, data types validation, and missing data default rules. Applications that access data from different enterprises always face coupling challenges. Any change in data structure will have a cascading effect on the larger ecosystem. Data contracts will help organizations manage this challenge effectively. 

New Roles in Data Teams 

So far we have seen data engineers, architects, and data scientists in the data teams. But additional roles may emerge as the business objectives from data change constantly. DataOps engineers, reliability engineers, and data product managers seem to be the new roles emerging in the new future. 

Data Democratization continues 

Organizations see larger value if they make data readily available and accessible to different stakeholders. This data democratization is more valuable among the data engineering trends in 2023 because of the cost savings organizations can witness. When data is accessible, you can reduce the duplication efforts when different stakeholders need the same underlying data. You can also reduce the investment in IT and technical teams as you lower the expertise needed to access the data. Most importantly, organizations can make better use of their data assets. An exciting opportunity like a data marketplace impacts the top line too. 

Faster resolution for Data Anomalies 

A survey from Wakefield research revealed that 40% of the data professionals’ time is spent on data quality. If you can detect the issue faster you can also resolve the issue faster. As per thought leadership from Monte Carlo, organizations are constantly focusing on reducing the time for the detection of data issues. Another interesting fact from Wakefield research states that it takes around 4 hours to detect and 9 hours to resolve the issue. We can expect reduced data downtime in the near future. 

At Saxon, we continue to learn from the data engineering trends in 2023 as stated by experts. Do you prefer a mindshare with our experts? Get in touch with us. 

Follow us on LinkedIn and Medium to stay updated about the latest enterprise technology trends. 

Get in Touch


Stay up-to-date with our latest news, updates, and promotions by subscribing to our newsletter.

Copyright © 2008-2023 Saxon. All rights reserved | Privacy Policy

Address: 1320 Greenway Drive Suite # 660, Irving, TX 75038

We Help Enterprises Achieve Their Transformation Goals

Request a callback

Saxon AI

Address:  1320 Greenway Drive Suite # 660, Irving, TX 75038 United States.
Phone: +1 972 550 9346

Sija Kuttan

Sija Kuttan

Vice President - Sales

Sija.V. K is a distinguished sales leader with a remarkable journey that spans over 15 years across diverse industries. Her expertise is a fusion of capital expenditure (CAPEX) machinery sales and the intricacies of cybersecurity.

Currently serving as the Vice President of Sales at Saxon AI, Sija adeptly navigates market dynamics, client acquisition, and channel management. Her distinguished track record of nurturing strong relationships, leading diverse teams, and driving growth underscores her as an adaptable and seasoned sales professional.

Gopi Kandukuri

Gopi Kandukuri

Chief Executive Officer

Gopi is the President and CEO of Saxon Inc since its inception and is responsible for the overall leadership, strategy, and management of the Company. As a true visionary, Gopi is quick to spot the next-generation technology trends and navigate the organization to build centers of excellence.

As a digital leader responsible for driving company growth and ROI, he believes in a business strategy built upon continuous innovation, investment in core capabilities, and a unique partner ecosystem. Gopi has served as founding member and 2018 President of ITServe, a non-profit organization of all mid-sized IT Services organization in US.

Vineesha Karri

Vineesha Karri

Associate Director - Marketing

Meet Vineesha Karri, the driving force behind our marketing endeavors. With over 12+ years of experience and a robust background in the B2B landscape across the US, EMEA, and APAC regions, she is pivotal in setting up high-performance marketing teams that drive business growth through a transformation based on new-age marketing practice.

Beyond her extensive experience driving business success across Digital, Data, AI, and Automation technologies, Vineesha’s diverse skill set shines as she collaborates with varied stakeholders across hierarchies, cultivating a harmonious and results-driven workspace.

Sridevi Edupuganti

Sridevi Edupuganti

Vice President – Cloud Solutions

Sridevi Edupuganti is an innovative leader known for strategically enhancing business opportunities through technology planning, orchestrating roadmaps, and guiding technology architecture choices. With a rich career spanning over two decades as a Senior Business and Technology Executive, she has driven teams to empower customers for digital transformation.

Her leadership fosters democratized digital experiences across enterprises. She has successfully expanded service portfolios globally, including major roles at Microsoft, NTT Data, Tech Mahindra. Proficient in diverse database technologies and Cloud platforms (AWS, Azure), she excels in operational excellence. Beyond her professional achievements, Sridevi also serves as a Health & Wellness coach, impacting IT professionals positively through engaging sessions.

Joel Jolly

Joel Jolly

Vice President – Technology

Joel has over 18 years of diverse global experience and multiple leadership assignments across Big 4 consulting, IT services and product engineering. He has distinguished himself by providing strategic vision and leadership for solving common industry problems on cutting-edge technologies.

As a leader surfacing and operationalizing next-generation ideas, he was responsible for exploring new technology directions, articulating a long-term technical vision, developing effective engineering processes, partnering with key stakeholders to build a strong internal and external brand and recruiting, mentoring, and growing great talent.

Haricharan Mylaraiah

Haricharan Mylaraiah

Senior Vice President - Strategy, Offerings & Sales Enablement

Hari is a Digital Marketer and Digital transformation specialist. He is adept at cultivating strong executive and customer relationships, utilizing data across all interactions (customers, employees, services, products) to lead cross-functionally as a strategic thought partner to install discipline, process, and methodology into a scalable company-wide customer-centric model.

He has 18+ years experience in Customer Acquisition, Product Strategy, Sales & Pre-Sales Management, Customer Success, Operations Management He is a Mechanical Engineering Graduate with MBA in International Business and Information Technology.