To the untrained, or uninformed, perspective, the “data experts” in an organization are one-size-fits-all — as if data scientists and data engineering have the same skill sets and roles. Or, if there is some level of understanding of the different teams where these data experts operate, and the responsibilities they’re tasked to fill, you may have different functions trying to do the same tasks without proper communication.
Data and technology skills have never been more relevant nor more important. Areas such as artificial intelligence (AI) and machine learning (ML) are in increasing demand by businesses across industries — from consumer packaged goods (CPG) to financial services — hungry to extract value from the ever-expanding volume of data being generated by the technology that facilitates our lives today.
As the field of machine intelligence continues to expand, evolving the economy in tandem, new roles are being created and existing ones are expanding. Organizations need to fully understand what these roles bring to the table to ensure they keep the right people at the heart of every business initiative.
Understand the Requirements Needed for Different Data Roles
Data scientists and data engineers are often viewed interchangeably to those not in the industry, but their roles are quite distinct.
At a high level, data engineers are at the front lines of data discovery. They typically are responsible for building and refining the infrastructure that enables data generation, and they test, integrate, manage and optimize data from a variety of sources. Their core competencies include data production-level programming, distributed systems, data pipelines, data transformation and data analytics. They understand how to implement machine learning algorithms chosen by data scientists and make them impactful for the entire enterprise. Data engineers often come from a programming background, possibly as a result of a computer science degree — and generally have competency in languages like Python, Java or Scala.
Meanwhile, data scientists lead the charge on the analysis of data, once it’s been generated. They understand how to model data and drive meaning from unstructured data. Instead of building or maintaining data infrastructure — typically the purview of data engineers — data scientists focus on choosing appropriate machine learning algorithms, training them and testing their accuracy using various methods. They also work more closely than data engineers with business domain leaders to understand their needs and ultimately communicate complex findings in a digestible manner. Data scientists have traditionally been sourced from the disciplines of Statistics, Computer Science and Engineering. We’re also seeing an increasing trend of universities offering Data Science degree courses.
Both roles of data scientist and data engineers continue to appear near the top various lists of the best jobs to seek, and it’s easy to see why. But, although closely related, they are different disciplines that demand different professionals, accordingly.
Related Article: Data Scientists vs. BI Analysts: What's the Difference?
Approach AI-Focused Projects Holistically and Maximize Responsibilities
Understanding the traditional roles, responsibilities and skill sets of data engineers and scientists is crucial — but more important is aligning them in practice.
As a starting point, any enterprise embarking on an AI-focused data project should lay out the project lifecycle. This will include many different elements: data discovery and extraction (ETL) that respects the entitlements of the data sets; data wrangling, investigation, feature engineering, experimental model creation, result generation, model deployment in production, support and MLOps.
Each of these items can be owned by either IT (Data Engineering), Product or Data Science. And without adequate support or clear direction, each function tries to do the others' task — clearly a non-optimal situation.
This is where centralizing data, and decoupling data storage from compute, can play a crucial role. It’s critical to have a robust software architecture that allows for this, as one should not have to reinvent the ML framework each time a new data set is ingested and cataloged in the data lake or storage. By allowing easy extraction of data via APIs, experimentation via tools that data scientists prefer, and freedom to choose ML frameworks, choice of compute and guardrails around it, different teams can work on their assignments with speed. Subsequently, deployment and publishing of the finished output for sharing and collaboration join the entire process in a syndicated manner. MLOps, or DevOps for ML competency, is needed to make this efficient and self-serviceable.
Related Article: How to Leverage Data Science to Capture the Fickle Customer
Support the Specialists, Don't Force the Unicorns
Many organizations make the mistake of trying to force people to be unicorns — doing the jobs of both data scientists and data engineers. Yes, these people do exist and provide considerable value, but focusing on separating these functions and allowing people to excel in their specific disciplines will allow for more clear delineation of roles and enhanced efficiency with less risk of overlap.
Ultimately, data science and data engineering are complementary, not contradictory or redundant. When these professionals are instructed and deployed within a synergistic organizational framework, and with the right tools at their disposal, they are at the forefront of driving value for the entire organization.
0 Comments
Email This
Stumble This