The Importance of SQL in Practicing Data Science

The Importance of SQL in Practicing Data Science

A number of times last year I was asked “What do you think is the most important skill in data science?” I always replied “SQL”. Although this response was always met with a nod of agreement, I was often told that wasn’t the typical response. Understandable, the Python vs. R debate is apparently more sexy. But on day 1 of your first data job they’re going to introduce you to their data warehouse. This is the data you’ll utilize to analyze data, by writing SQL queries.

I’ve used SQL at multiple jobs throughout my career, but I wanted to make sure that other companies were doing the same. Here is a current job listing at Google for a Data Scientist, and they want experience with SQL:

Data Scientist Qualifications.png

The major cloud providers are now offering relational databases in the cloud:

Amazon RDS Database Engines .png

Also, Google Cloud SQL and Azure Database for PostgreSQL. The data is getting bigger, but SQL is here to stay (and scale).

If you read my article on data science FAQs, we saw that 51% of job openings titled “Data Scientist” in the US were asking for SQL.

Companies are using it, there is demand for the skill, and it’s here to stay. Even if for some reason you do not need SQL at your first job, I’m sure you’ll need it at some point during your career if you’re in data science.

What about Unstructured Data?

Yes, I sometimes work with unstructured data in the Big Data environment. But if I’m in there and find a variable that would be relevant for repeated use, we’ll typically have the big data team make it available in the data warehouse.

I pull some grouped data (using Hive which is quite similar to SQL) onto my local machine and do an analysis.

At some point in the future (length of time depending on the current priorities) I’ll have that data available for me to use in one of the tables in the database. And life keeps moving on.

I could spend more time in the big data environment, but queries run much faster in the relational database.

Maybe Someone Other than Yourself Pulls your Data for You

We all know that it is important to understand how the tables are related and the logic behind the data. I want to write the query that builds the model I’m putting my name on. Understanding all of the intricacies and nuances of the fields. Having a full understanding of the potential bias and caveats that will need to go along with my model allows you to communicate these caveats with the business. I also like to think I’m pretty creative when thinking about variables. This is partly due to having a good understanding of the different tables in the relational database.

There are often questions you’ll want to be able to answer by yourself. If something doesn’t seem right with your data, you’ll want to be able to dig into the discrepancy to find out what is going on. You don’t want to be blindly following data that someone else provided, and you don’t want to get held up if the data doesn’t seem quite right. I want to dig in and look for answers immediately.

Maybe You Just Want to Use Python or R

Cool, I pull data from the database into Python and R too. However, I start my query in SQL. I find that for complex queries where I am joining multiple tables it makes sense to write my query in SQL first. The errors when I misspell something are much easier to catch and track down when I’m directly in SQL rather than when I write a query directly in Python and then find that it doesn’t run for some reason. Python just let’s me know that there is an error, it’s not going to give me hints about what the problem was like I’d get in SQL.

Although you can use Python or R syntax that is not SQL to speak to the database, you still have to understand the schema and how relational databases work to be successful querying this way. It’s fairly easy to learn, even for total newbies.

Summary

The learning curve is quite easy, so you’ll be writing queries in no time if you decide to learn.

Learn it once, use it again and again in your career.

A version of this article first appeared here

Share this article

Leave your comments

Post comment as a guest

0
terms and condition.
  • Michael Nolan

    Fantastic insight.

  • Will Carroll

    Thank you for the clear explanation

  • Sean Murphy

    Well said. Thanks a bunch.

  • Gareth Thompson

    Very informative

  • Richard Klause

    Kristen, a great article !

    For someone starting out to build database / SQL skills I find the following of great use !!!
    (1) Rashid Khan’s course -- http://bit.ly/2Y0c6eS-- “SQL Server 101 : Microsoft SQL Server for Absolute Beginners” — deeply discounted video course, for the price of a lunch, and in less than 4 Hours – extremely concise and clear video lectures :)

    (2) Khan Academy -- https://www.khanacademy.org/ – free online courses covering a wide range of disciplines, including the Intro to SQL course

  • mc110

    Great article, and something I've mentioned to data scientists in the past.

    One useful thing that some databases support is running R/python/language of your choice embedded within SQL, so you never have to actually extract data from the database, pull it back to your local machine, and run your data science algorithms there. This is great not only in terms of data auditing, but also in allowing you to use your powerful database platform to run complex analysis, instead of doing data transfer to allow that analysis to be run on a much less powerful system.

    There is a good blog post about using this approach at https://kognitio.com/blog/sentiment-analysis-amazon-reviews-pt1/ to do sentiment analysis on 160 million Amazon Customer Reviews.

    Regards,
    Mark

  • Prajwal

Share this article

Kristen Kehrer 

Tech Guru

Kristen is #8 LinkedIn Global Top Voice 2018 – Data Science & Analytics. She started her journey to becoming a Data Scientist without knowing it. She finished a BS in Mathematics in 2004 but wasn’t sure how she would be applying this education in industry. She then returned to academia for a Master’s Degree in Statistics, and knew that she wanted to be analyzing data, building models, and helping to guide business decisions. Since 2010, she has utilized Data Science across multiple industries, including the utilities, healthcare, and eCommerce. Prior to attaining her Master’s Degree, she was a high school math teacher, and have always enjoyed tutoring, coaching, and mentoring. Her passion is leveraging her experience with the end-to-end job search for Data Science positions to help others effectively market their skills to land a job in this field. The coaching she has personally received was invaluable to how she approached the job search and how she presented herself, she is so thrilled to share this expertise with others. Kristen is also a member of the Data Science Office Hours YouTube channel. When she is not doing Data Science, she can be found with her husband and two young children. She is also a mechanical keyboard hobbyist and aerialist.

   

Latest Articles

View all
  • Science
  • Technology
  • Companies
  • Environment
  • Global Economy
  • Finance
  • Politics
  • Society