Image by Editor
Are you an aspiring data scientist? If so, chances are you’ve seen or heard of many who have successfully pivoted to a data science career. And you’re hoping to make the switch someday, too.
There are several things exciting about working as a data scientist. You can:
- Build hard and soft skills transferable across domains
- Tell stories with data
- Answer business questions with data
- Build impactful solutions to business problems
And much more. As exciting as all of this sounds, being a data scientist is equally challenging if not more. But what are some of those challenges?
Let’s dive in.
When you’re working on your coding and technical skills, you’ll probably get comfortable working all on your own. But as a data scientist, you should prioritize collaboration and communication. Because data science is not about wrangling data and crunching numbers in isolation.
You need to collaborate with other professionals—not just on the same team but often across multiple teams. So your ability to collaborate with diverse teams and stakeholders is just as important as your technical skills.
Further, you should also be able to communicate your findings and insights to non-technical stakeholders, including business leaders.
Nisha Arya Ahmed, a data scientist and technical writer, shares:
“In a data science team, you will collaborate with other data science professionals on each task, their responsibility and how it all works hand in hand. This is important as you don’t want to repeat work that has already been done and use up more time and resources. Also, data professionals are not the only people you will have to collaborate with, you will be part of a cross-functional team including product, marketing, and also other stakeholders.”
– Nisha Arya Ahmed, Data Scientist and Technical Writer
If you’re someone who enjoys working on projects, completing them, and shipping them to production, you may not find data science a rewarding career.
Though you start a project with a set of objectives—refined and improved iteratively—you’ll often have to change the scope of the projects as the organization’s business goals change. Perhaps, stakeholders see a new promising direction.
So you’ll have to effectively reprioritize and modify the scope of projects. And in the worst case, abandon your project if required.
Also, at an early stage startup, you’ll often have to wear multiple hats. So your job doesn’t end with model building. Even if you manage to deploy a machine learning model to production, you have to monitor your model’s performance, look out for drifts, regress and retrain the model as needed.
Abid Ali Awan, Writer, Editor, and Data Scientist at KDnuggets, shares:
“If you work at a company, you may often have to switch between multiple teams and work on different projects simultaneously. However, most of the projects you work on may not even make it to production.
Because the company’s priorities may change or the impact of the projects may not have been significant enough. Continuously switching between teams and projects can be exhausting, and you may feel clueless as to what you are contributing towards.”
– Abid Ali Awan, Writer, Editor, and Data Scientist at KDnuggets
So working on data science projects is not a linear start-to-finish process where you finish a project and move on to the next.
A day in the life of a data scientist at two different organizations may be completely different. The roles of a data scientist, machine learning engineer, and MLOps engineer often have a lot of overlapping functionality.
Say you’re a data scientist who is very much interested in building predictive models. And you’ve landed the role of a data scientist in an organization of your interest.
However, don’t be surprised if you spend your whole day crunching numbers in spreadsheets and making reports. Or pulling data from databases using SQL. You may think wrangling data with SQL and finding answers to business questions will better fit the role of data analyst.
While in some other cases, you may be in charge of building and deploying models to production, monitoring drifts, and retraining the model as needed. In this case, you’re a data scientist who also wears the hat of an MLOps engineer.
Let’s hear what Abid has to say about role fluidity in a data career:
“I am always confused about being called a “Data Scientist”. What does it even mean? Am I a Data Analyst, Business Intelligence Engineer, Machine Learning Engineer, MLOps Engineer, or all of the above? Your role within a company is fluid if you are working at a smaller company or startup. However, larger organizations may have a clearer distinction between roles. But that doesn’t guarantee that the role is completely defined. You might be a data scientist; but a lot of the work you do will perhaps be creating analysis reports that align with business goals.”
– Abid Ali Awan, Writer, Editor, and Data Scientist at KDnuggets
As a data scientist, you should direct efforts towards projects that have the most significant impact on the business rather than pursuing technically interesting but less relevant projects. To this end, understanding business objectives is key for the following reasons:
- Understanding business objectives allows you to adapt and reprioritize your projects based on the changing needs of the organization.
- The success of a data science project is often measured by its impact on the business. So a good understanding of business objectives provides a clear framework for evaluating the success of a project, linking technical aspects to tangible business outcomes.
Matthew Mayo, Editor-in-Chief and Data Scientist at KDnuggets, shares the cost of indifference to business outcomes:
“As a data scientist, if you are indifferent to business objectives you might as well be a cat chasing a laser pointer—you will find yourself overactive and aimless, likely accomplishing nothing of much value. Understanding business goals and being able to translate them from business to data speak are crucial skills, without which you could find yourself investing time in building the most sophisticated, irrelevant models. A decision tree that works beats a state-of-the-art failure every day!”
– Matthew Mayo, Editor-in-Chief and Data Scientist, KDnuggets
Here’s what Nisha has to say in this regard:
“With anything you do, you need a reason behind it. This is your intention, which comes before your action. When it comes to the world of data, understanding the business and the challenges is imperative. Without this, you will just be confused through the process. With every step you take in a data science project, you will want to refer to the objectives that motivate the project.”
– Nisha Arya Ahmed, Data Scientist and Technical Writer
Data science, therefore, is not just about crunching numbers and building complex models. It’s more about leveraging data to drive business success.
Without a solid understanding of the business objectives, your projects may deviate from the business problems they are meant to solve—diminishing both their value and impact.
Building models is exciting. However, the road leading up to that may not be as interesting.
You can expect to spend large chunks of your time:
- Collecting data
- Identifying the most relevant subset of data to use
- Cleaning the data to make it suitable for the analysis
Now this is work that’s not super exciting. Often, you don’t even need to build the machine learning models. Once you have the data in a database, you can use SQL to answer questions. In which case you don’t even need to build a machine learning model.
Here’s Abid sharing his views on how important work is often not interesting:
“It can be tedious to do the same thing repeatedly. Often, you may be assigned the task of cleaning data, which can be quite difficult, especially when working with diverse datasets. Additionally, tasks like data validation and writing unit tests may not be as exciting but are necessary.”
– Abid Ali Awan, Writer, Editor, and Data Scientist at KDnuggets
So you’ve got to enjoy the process of working with data—including the good, the bad, and the ugly—to have a successful data science career. Because data science is all about deriving value from data. Which often is not about building the fanciest models.
As a data scientist, you’ll (probably) never be able to reach a point where you can say that you have learned it all. What you need to learn and how much depends on what you’re working on.
It could be a fairly simple task like learning and using a new framework going forward. Or something more tedious such as migrating the existing codebase to a language such as Rust for enhanced security and performance. Besides being technically strong, you should be able to learn and ramp up quickly on frameworks, tools, and programming languages as needed.
In addition, you should be willing to learn more about the domain and the business if required. It’s not very likely that you’ll work in a single domain throughout your data science career. For example, you may start out as a data scientist in healthcare, then move to fintech, logistics, and more.
During grad school, I had the opportunity to work on machine learning in healthcare—on a disease prognosis project. I’d never read Biology beyond high school. So the first few weeks were all about exploring the technicalities of specific biomedical signals—their properties, features, and much more. These were super important before I could even proceed to preprocessing the records.
Kanwal Mehreen, a technical writer shares her experience with us:
“You know that feeling when you finally learn a new skill and think, “Ah, this is it, I’m good”? Well, in data science, that moment never really comes. This field is ever-evolving with new technologies, tools, and methodologies emerging frequently. So if you are someone who prefers reaching a certain point where learning takes a backseat, then a data science career may not be the best match.
Moreover, data science is a beautiful blend of statistics, programming, machine learning, and domain knowledge. If the idea of exploring different domains, from healthcare to finance to marketing, doesn’t excite you, you may feel lost in your career.”
– Kanwal Mehreen, Technical Writer
So as a data scientist you should never shy away from constant learning and upskilling.
We have already outlined several challenges of being a data scientist including:
- Going beyond the technical skills of coding and model building
- Understanding the domain and business objectives
- Continuously learning and upskilling to stay relevant
- Being proactive without worrying about finishing projects in the literal sense
- Being ready to reprioritize, regress, and make changes
- Doing the work that is boring but necessary
Like any other tech role, the difficult part is not landing a job as a data scientist. It’s building a successful data science career.
Mathew Mayo aptly summarizes how you should embrace these challenges as a data scientist:
“Looking for a laid back career, where you can quit learning the moment you start your job and never be worried about the latest tools, tricks and techniques? Well, forget about data science! Expecting a quiet career as a data professional is akin to expecting a dry stroll through a monsoon, armed only with a cocktail umbrella and an optimistic attitude.
This field is a non-stop roller coaster of technical puzzles and non-technical enigmas: one day you’re deep-diving into algorithms, and the next you’re trying to explain your findings to someone who thinks regression is a retreat into a child-like state of behavior. But the thrill lies in these challenges, and it’s what keeps our caffeine-addled brains entertained.
If you’re allergic to challenges, you might find more solace in knitting. But if you’ve yet to back away from a confrontation with a data deluge, data science might just be your cup of… coffee.”
– Matthew Mayo, Editor-in-Chief and Data Scientist, KDnuggets
Let’s hear Kanwal’s thoughts on this:
“Let’s face this fact: data science isn’t always a smooth sail. Data doesn’t always come in neat and organized packages. Your data may look like it’s been through a storm, which might be incomplete, inconsistent, or even inaccurate. Cleaning and preprocessing this data to ensure its relevance for analysis can be challenging.
While working in a multidisciplinary field, you may have to interact with non-technical stakeholders. Explaining technical concepts to them and how they align with their objectives can be really challenging.
Therefore , if you are someone who prefers a clear, straightforward career path, a data science career might be full of roadblocks to you.”
– Kanwal Mehreen, Technical Writer
So data science is not just about math and models; it’s about going from data to decisions. And in the process, you should be always willing to learn and upskill, understand business objectives and market dynamics, and much more.
If you are looking for a challenging career that you’d like to navigate with resilience, data science is indeed a good career option for you. Happy exploring!
I thank Matthew, Abid, Nisha, and Kanwal for sharing their insights on several aspects of a data science career. And for making this article a much more interesting and enjoyable read!
Bala Priya C is a developer and technical writer from India. She likes working at the intersection of math, programming, data science, and content creation. Her areas of interest and expertise include DevOps, data science, and natural language processing. She enjoys reading, writing, coding, and coffee! Currently, she’s working on learning and sharing her knowledge with the developer community by authoring tutorials, how-to guides, opinion pieces, and more.
