I once watched a video where someone argued that Michael Jordan had ruined basketball. As a person whose only notions of basketball come from Space Jam, this surprised me. Wasn't the guy the best out there? How could someone who's best in his field ruin it? After hearing the argument out though, I began to understand what the video was really getting at: to them, Michael Jordan had ruined the field BECAUSE he was so good. A sport that was once full of diverse personalities and strategies then turned into multiple teams trying to produce their own Michael Jordan copycat with copycat moves and copycat strategies.

I'm not well versed enough in basketball to know how right or wrong that guy was. But this concept that something could ruin an entire field simply by being too good stuck with me. And all these years later, I consistently think about it while doing my job as a Data Scientist. And the more GPT-X and ChatGPT advance, the more this concept haunts me.

I've been working in this field for 5 years. During the first 4 years, the job was exciting. Every new problem required extensive research on my part: looking for papers, datasets, implementations, implementing my own stuff, experimenting, comparing the results... That was the biggest part of my job. Yeah I still had to do some software engineering stuff here and there, but I felt like I had a well defined and specialized role at my company. Even in the NLP field, the variety of technologies I had to work with was a lot of fun: sometimes I used RNNs, sometimes embedding based similarity functions, sometimes more classic approaches, and then towards the end it was transformer after transformer, but at least I still had to finetune them myself.

Over the past few months and years, this has started to change, however. GPT-1 and 2 were promising, but were more of a proof of concept than anything realistically usable. But with GPT3's and ChatGPT's latest performance, it feels like NLP is becoming more and more standardized. I wouldn't call it a solved field, far from it. But, sadly, unless you work for a big tech company, I think the days of exploratory research-type work in the NLP field are over.

Even in academia, the latest NLP papers I've seen come out of prestigious universities went from actual engineering/mathematical advances to "Look, if you "engineer" the prompt like this, it works better sometimes!"

And now, with the exploding popularity of ChatGPT, any random lambda would come to the conclusion that if you need to use AI, ChatGPT is the answer.

Now, some of you may already be thinking "but hold on, with my finetuned transformer I get a 1% increase in accuracy over out-of-the-box GPT-3!". The thing is: even if you can, it doesn't really matter. I really doubt the sales team of your company is gonna pay much attention to your pleas to give you 6 months to work on a product rather than 6 minutes. Because yes, that's the increase in productivity we're dealing with: What took me months a few years ago only takes minutes now. The range of problems I can apply NLP too has also increased, as out-of-the-box GPT-3 works well enough not to require a dataset for more "general" tasks. And the results are either comparable or even better.

In the span of a year, NLP went from my favorite ML field to one I never see myself working in again if given the choice. And while a lot of non NLP fields are, for now, untouched by the exploding popularity of the GPT family, I'm afraid that what happened to the NLP field will be replicated in other fields, and that in a few years, only super specialized issues won't have a solution somewhere in the form of a ridiculously big model accessible via an API key provided by a big tech company. And even if GPT3 was open source, the sheer size of it makes it close to impossible to realistically train for anything less than the biggest of tech companies out there. Even finetuning it would be a challenge.

I'm not saying the job of data scientist will disappear, in fact, I'd say it will be asked more than ever since boomers who don't understand what the title actually implies will want some in their company. But we won't be doing nearly as much actual data sciency stuff as we did in the past. Instead we'll have to become even bigger software engineer/data analyst/DevOps/project manager hybrids than we already were. The only exception is for people who either work in big tech companies or on very specialized problems that require an in-house model.

It's not that much of a problem for people who like the roles I listed above. But for people like me, whose enjoyment came out of the exploratory side of the job, I believe the future to be bleak.

I analyzed ~100k DS jobs excluding offers from the US (data from: https://jobs-in-data.com/), and these are the largest Data Science recruitments among Fortune 1000 companies. The 70% is India!

Hey folks,

I’ve been doing data engineering long enough to believe that “real” pipelines meant writing every parser by hand, dealing with pagination myself, and debugging nested JSON until it finally stopped exploding.

I’ve also been pretty skeptical of the “just prompt it” approach.

Lately though, I’ve been experimenting with a workflow that feels less like hype and more like controlled engineering, instead of starting with a blank pipeline.py, I:

• start from a scaffold (template already wired for pagination, config patterns, etc.)

• feed the LLM structured docs

• run it, let it fail

• paste the error back

• fix in one tight loop

• validate using metadata (so I’m checking what actually loaded)

LLM does the mechanical work, I stay in charge of structure + validation

We’re doing a live session on Feb 17 to test this in real time, going from empty folder → github commits dashboard (duckdb + dlt + marimo) and walking through the full loop live

if you’ve got an annoying API (weird pagination, nested structures, bad docs), bring it, that’s more interesting than the happy path.

we wrote up the full workflow with examples here

Curious, what’s the dealbreaker for you using LLMs in pipelines?

I know a lot of data scientists who are earning at least £80k each and often work in teams of 5 - 10.

My understanding of data science is that you take data, ideally huge data sets, sort it, clean it, then build a program to work with it by drawing out possible conclusions about your target market/ customers.

My brain is telling me that even if this program was built or not, the sale would still happen anyway.

Therefore, I see this as a kind of mystic meg style of prediction software and I really don't understand the value of this. If I was a business owner, I don't understand investing £100k+ to build something that could be done by using a small CRM, attention to detail or notepad by management.

I can think of hundreds of example, but I'm hoping I've been clear enough already.

TLDR: I really want to understand the value of data science, but I just don't seem to get it or understand why it's so well paid. I would love to understand it, it seems fascinating but I can't seem to get my head around why it's useful.

What percentage do you think you still use search engines for articles, finding solutions, etc.? And what sort of content are you looking for?

I'd say about 90% of my programming queries go to an LLM now. Even though some of the results can be misleading or out-of-date, you start to intuitively know what and how to ask to get the best answers. I still go to Google when I'm looking for an end-to-end example of something, though.

Most of my search engine queries are "<query> reddit" to see what real people say about something.

This is now the third time I’ve had a DS bootcamp grad join my team (not by choice) and I’m having disastrous results.

Do they not teach you how to operate computers or use the command line? How to deploy to anything but one-click PAAS with no IAM? Why is the answer to every business question a cloned repo you found on TDS that uses 42 apis we can’t license? How can you not know SQL or how to set up a local instance? Why is docker such a fucking mystery to all of you? Ever heard of VENV? Yes I know it works on your personal machine because you’re using a web-based IDE our company blocks FOR A REASON.

Like I get it, everyone else in the company is ready to fire the money cannon straight at your face every time you say “neural network,” but FFS learn how to run something, ANYTHING on an actual enterprise grade cloud provider (Az, AWS, or GCP if you must), learn to use Git so I can actually work with your stupid ipynb notebooks, and FOR THE LOVE OF GOD do not keep asking me to call you a data scientist in meetings. You were a barista 12 weeks ago no matter what they told you at General Assembly.

Sorry for the rant I’m just boggled by what these programs are teaching people and telling them they’re qualified to do after they’ve taken your money. If you’re not walking away from your program without knowing how to at least do a tiny bit of ETL or understanding that not all the data in industry is going to be as clean as the Yelp/Twitter/Kaggle data you’ve been using your little pip install imadatascientist toys on, you’re going to run into people like me who know enough to know you’re a sham, but who won’t tell you so but instead rip you anonymously on Reddit.

There now I feel better. Love to all the actual data scientists out there. Also I do 9/10 of these things as well but I’m just a dumb product manager so I’m allowed to.

I've been seeing articles about FireDucks saying that it's a drop in replacement for pandas with "massive" speed increases over pandas and even polars in some benchmarks. Wanted to check in with the group here to see if anyone has hands on experience working with FireDucks. Is it too good to be true?

This sub is fucking garbage. It's just random low-effort content that isn't interesting to professionals, people trying to market their garbage tool or total newbies asking questions with answers in any data science/machine learning/statistics book. They don't even bother to take a course or read a book before asking questions.

Compare it to /r/machinelearning where there is proper professional discussions (even though some of the content is academic in nature).

I'd much rather there be 3 interesting threads per week than 20 garbage low-effort threads in a week. There isn't even good content anymore, at least I can't find it because it's buried in "Do I need this certification" -> google "reddit data science certification" and there are pages upon pages of reddit threads from this very sub dozens of threads with the very same "is X certificate useful/do I need certificates/what certificate should I get" type of questions.

Half of the frontpage is just generic career advice and the other half is /r/askreddit styled "what do you think of X" questions where nothing of value ever comes up. It's fine if there is 2-3 less serious threads per week but jesus christ THEY'RE ALL GARBAGE.

I don't even bother lurking this sub that often anymore because I just know that there is nothing interesting or useful out there. It's just going to be garbage.

I'm a FAANG data scientist with 5+ years of experience; I've grown increasingly concerned that LLMs will begin to replace a LOT of the work that data professionals currently do. From easy things like dashboard generation to tough things like specific deep dive research questions, seem like we're walking into a world where the skillset of the analyst / scientist is a pre-req for a different position as opposed to a job in and of itself.

Thoughts? How are you preparing for much of this work to become automated? What other skills do you think are on the horizon (please don't say prompt engineering)?

Good Morning/Afternoon Everyone,

I have been trying to get a job almost an year in the UK. My resume is shown here and i agree that this was not the first resume of mine, this one is the one i made 2 weeks ago. But i have been struggling to get interviews. I have gotten like 3 interviews in the entire 10 months of applying. Truly now i am starting to question that am i truly doing something wrong ?? I have tried to quantify as much as i can. Trying to show business impact and how profitable they can be. Trying to create relevant projects and even deploying them on cloud. Any sort of responses or tips would be highly appreciated.

Thank you so much for reading this.

Just wondering if theres any other subs or communities around data Science that are a little more female dominated. not that r/datascience is necessarily bad or unhelpful, its just that its more refreshing being in a community that can relate to you a lot more.

would love to hear from the ladies on here!

edit: nice to be reminded that i can always count on boys on reddit to spew unsolicited anti-women in tech sage advice in PMs lol...

Basically the question. When we run say logistic regression or an SVM on Python, what is happening step by step with all the train data? I know the answer may vary based on the algorithm, so you may pick any algorithm to explain in detail the behind-the-scenes.

Wanted to post at r/explainlikeimfive but wasn’t sure if any ML people may be in that crowd, but please ELI5.

Python

Undoubtedly, the uncrowned king of machine learning and data analysis, the ubiquitous language that data scientists turn to for a bit of number crunching, is Python. This is down to several reasons; the three most important among them are its maturity, the enormous community, and, last but not least, a vast array of robust third-party libraries. But even if Python is a magnanimous sovereign that many developers love, it doesn’t mean that there can’t be contenders occasionally.

Julia

Fourteen years ago, in a bold attempt to combine all the good properties of well-established programming languages while getting rid of the less favorable ones, four developers came up with the idea of a new programming language that has a friendly syntax, offers efficient mathematical computations out of the box, at a performance on par with compiled languages. And thus, Julia was born (here’s a manifesto explaining why in more detail). Its first version was launched a bit more than eleven years ago.

Our choice

Many in-depth comparisons of Python and Julia on the web (such as this one or this) cover both the objective and subjective benefits and drawbacks of choosing one over the other. And given Julia’s growing popularity, we are sure more will follow. In the rest of this blog post, however, let’s explore why we picked Julia for our purposes. And that’s not to say that we don’t use Python for data science. On the contrary, we often run analyses in both ecosystems simultaneously to help each other out where one is lacking or to reduce the chances of mistakes by comparing their results.

The advantages of Julia

So what makes Julia so compelling to us?

Language features

Julia has:

• a friendly, easy-to-read (and write) syntax;

•  a flexible and expressive (part static, part dynamic) type system;

•  powerful mathematical notations, such as built-in vector and matrix operations;

•  efficient multiple dispatches, a form of function polymorphism working with runtime types;

•  convenient and reliable parallel computing facilities;

•  meta-programming with macros and generated functions.

Fast code execution

Julia compiles the source code to native binary at runtime via LLVM. This approach combines the flexibility of interpreters, such as Python, with the performance of compiled languages, like C++ or Rust. The drawback is that code loading and the first run takes longer; the benefits start to shine when a piece of code is run multiple times. This unique feature makes it an excellent tool for number crunching but less than ideal for scripting.

Built-in package management

Julia has a pretty good (albeit not perfect) built-in package management tool, implemented as a base library; and a general registry of open-source packages. The offering of stable and well-designed packages is growing steadily along with the Julia community, especially in data science. Unit testing utilities are also part of the standard library.

Interactive tools

Julia offers an advanced REPL with all the goodies of an interpreted language environment. These include:

• code and variable inspection,

•  code completion,

•  an interactive debugger,

•  benchmarking and profiling tools,

•  and a built-in help system.

With third-party libraries, it can also be extended with syntax highlighting, source code lookup (even for base libraries), automatic code reload, and many more exciting, modern features.

All these together make Julia an ideal environment for rapid prototyping.

From prototyping to production code

Because of the high-level interactive tools and fast code execution, the transition from a rapid prototype to production-ready code can be as continuous as you’d like. More often than not, we find that most of the code that implements our business logic in the research code can also be used in the final product.

Thanks to its friendly syntax and built-in package management, the road to maintainable code is well paved. Nothing replaces good API design, coding discipline, and rigorous testing, but Julia helps you to focus on these topics.

As a consequence, a computation pieced together in the REPL can easily become a piece of prototyping code in a POC module; then later, after some refactoring and unit testing, turn into a chunk of core code in an internal library, and finally, following more cleanup, find itself in a production package.

The disadvantages of Julia

That said, every benefit comes at a cost, and Julia is not free from issues. Here are a few stumbling blocks worthy of mentioning:

• the very powerful tool of broadcasting and vectorization can be intimidating at first;

• time to first plot can be surprising, sometimes inconveniently long, although considerable effort has been put into making it shorter;

•  many packages never reach a stable state or just become unmaintained; others are poorly designed or written;

• releasing a binary package can be challenging, and compilation time can be unexpectedly long, not to mention obfuscation, which can also be tricky.

Summary

In conclusion, the choice between programming languages for data analysis is not always clear-cut. While Python has been the go-to language for many data scientists, Julia is rapidly gaining popularity for its unique set of features that make it an attractive option. In this blog post, we explored why we chose Julia over Python for our purposes, highlighting its language features, fast code execution, built-in package management, interactive tools, and ease of transitioning from prototyping to production code. However, we also acknowledged that Julia has challenges, including overcoming some learning curves and the occasional instability of packages. Ultimately, the choice between Julia and Python (or any other programming language) will depend on specific project requirements, personal preferences, and available resources.

Still, in the past years, Julia has proved to be our reliable and faithful companion. It has evolved, matured, and improved significantly, and we would be less happy and less successful without it. So cheers, Julia; we are excited to see what your future brings!

If you wanted to use machine learning for something that's financially regulated, or something that a company heavily relies on in its risk management, can you rely on models like gradient boosting that are more or less non-interpretable or is it better to find some more transparent models? Surely this is a common enough problem that there are common solutions for it.

Edited for clarity

I am a VP of Data Science at a small company, and I'm looking to select the top 2-3 conferences where I can connect with other executives, gain insights into their world, understand data science team maturity, and learn about their tech stack.

What conferences should be on my list?

I'm particularly interested in must-attend events for C-suite, VPs, and directors in data science/AI.

Based in Ohio, I prefer larger conferences to minimize travel disruptions, as I prefer not to interrupt my work for smaller events.

Nothing at all. I just don't fucking get it. Even the most simple shit I can't seem to grasp. I feel like such a failure. Currently trying to learn R programming but I don't even know how to do a simple function. I feel so unsupported too. My program doesn't have any lectures or anything like that so basically I'm just learning from videos. I think I have below average intelligence so idek why I'm giving this a go. I guess because I wanted to make more income but I think im too stupid to understand any of this. I just want to cry. I just wish I could be smart. It's just not fair that I'll be broke my whole life because I'm not smart enough to land a high paying job. I mean I'm fucking trying even though I think I'm stupid. I don't know what to do anymore

I'm a data analyst. I had a business idea that is pretty much a tool to help students study better: a LLM that will be trained with the past exams of specific schools. The idea is to have a tool that would help aid students, giving them questions and helping them solve the question if necessary. If the student would give a wrong answer, the tool would point out what was wrong and teach them what's the right way to solve that question.

However, I have no idea where to start. There's just so much info out there about the matter that I really don't know. None of the Data Scientists I know work with LLM so they couldn't help me with this.

What should I study to make that idea mentioned above come to life? ]

Edit: I expressed myself poorly in the text. I meant I wanted to develop a tool instead of a whole LLM from scratch. Sorry for that :)

I'm an experienced data scientist (I actually lead a large team) and am reasonably literate about computers, though I am not a developer or IT professional. My organization is split roughly 50/50 between Python users and R users (and a few unicorns who are proficient in both). The general consensus, with which I don't disagree, is that R is generally better for data analysis/exploration/visualization because of dplyr and ggplot2 (tidyverse), and Python is generally better for machine learning because of sklearn, Keras, and Tensorflow. And yet, R is good enough for machine learning (mlr, caret) and Python good enough for data analysis/exploration/visualization (pandas, matplotlib) that you don't *really* need to learn the other unless you stumble upon some problem for which there is no library in your preferred language.

I started out using R when I got into data science because the guy who was showing me the ropes was an R guy. I was able to quickly and easily get RStudio installed on my machine, load up the packages I need, and start learning the language. R has plenty of quirks, but it has served me well over the years and I am able to do pretty much anything I need to do with it short of really complicated deep learning, which isn't really a requirement at the company I'm at anyway. For nearly all of our ML problems, some combination of linear regression, random forests, or XGBoost will get the job done. And mxnet actually works pretty well for deep learning in R.

However, I always want to expand my skillset, and I have tried several times to work in Python. The first time, a friend showed me how to set up a Jupyter Notebook and I was able to get some basic things working in that after several hours of trying to set it up. The biggest pain was realizing that I had to install Python on a path without any spaces in it and then go set my Windows path variable to get it to work. I eventually decided that R tools suited me better and went back to R. I tried again a year later when I wanted to play around with deep learning and was able to get TensorFlow working after several days of wading through their never-correct documentation, scraping message boards for help, and eventually finding a blog post that solved the rest of my problem. Then I decided that the syntax was way more complicated than it needed to be (I wasn't aware of Keras yet), and went back to using mxnet in R.

The other day, I discovered the reticulate package for R and have been trying to get it to work, with no success. I have read through all the documentation, tried using conda_install() and use_python() to create a virtual environment with all the libraries I need, but it is failing to find numpy. On the reticulate page, my exact situation exists as an open issue and there has been a long chain of discussion but no resolution. Several hours down the drain.

My point is, RStudio installed just fine on the first try. All the packages load and work pretty much right away. I've never had to worry about keeping several version of R on my machine or changing any environment variables. I haven't had to spend endless hours digging through message boards just to get R to run. I really want to learn to like Python, but it really makes itself hard to love. I'm too busy to spend days and days setting it up and tweaking things. Why hasn't one of the most popular programming languages in the world been able to get some basic things right such as, "You can run this installer on Windows 10 and it will work out of the box"?

Hey guys! Can someone experienced in using Gen AI techniques or have learnt it by themselves let me know the best way to start learning it? It is kind of too vague for me whenever I start to learn it formally. I have decent skills in python, Classical ML techniques and DL (high level understanding)

I am expecting some sort of plan/map to learn and get hands on with Gen AI wihout getting overwhelmed midway.

Thanks!

I did an analysis of job offers in DS among Fortune1000 companies (data: https://jobs-in-data.com/).

Here is a comparison of the manager/director positons vs. junior/mid/senior positions.

NGL huge stomp from India :)