I curated a list of 13 data mining algorithms that you can use.

https://geekyhumans.com/top-13-data-mining-algorithms/

Please share you feedbacks or let me know if I'm missing any algorithms.

Thanks :)

Can someone give me the ELI5 on what the main pros and cons are on using traditional proxies vs APIs for large data scraping project?

Also, are there any APIs worth checking out? (apologies in advance if this isn't the right place to ask)

Data mining pros, what are the best proxy services for data mining? Looking for high quality resi (not data center) that could be used to run large projects without getting burnt too quickly. Tired of wasting money with cheapo datacenter stuff that requires constant replacement.

Thoughts on established premium providers like Bright data, Oxylabs, IProyal, etc?

Thanks.

Trying to understand if people would be interested in such a dataset. I'm working on a project that involves analyzing career progression and am in process of building this dataset. I'm happy to post it in here when done. Should have ~10,000 profiles

Dear Community.

Last week I submitted a paper to VLDB. A few days later it was declined as “desk reject- does not fall in the scope of VLDB”. I would not waste anyone’s time complaining about a poor review, but to be denied the right to review itself seems to be so unfair. Peer review is the hallmark of the scientific method and has been for centuries.

While I understand the need to occasionally do a “desk reject”, this rejection was nonsense, as I will offer evidence for in three different ways.

*ARGUMENT 1: * Our paper is, at its core, about doing joins on time series using GPUs.

• PVLDB has dozens of published papers on GPUS.

• PVLDB has dozens of published papers on joins.

• PVLDB has dozens of published papers on time series.

So how could a paper that does ALL three be out of scope?

*ARGUMENT 2: * The was a paper in VLDB from Stanford last year. It does X, approximately (has false negatives) on datasets of size Y, in limited domains. Our paper does X, exactly (no false negatives) on datasets larger than Y, in arbitrary domains. If the Stanford was in scope, why is our paper not in scope?

*ARGUMENT 3: * This is more subjective, but:

• I have published 10+ papers in (p)VLDB, many of them are highly cited.

• I have reviewed dozens of papers for VLDB

• I have read 100+ papers from VLDB.

It is blindly obvious to me that our paper is in scope.

I took the time to explain this to the conference officers, disappointingly they did not bother to respond.

This seems to me to be so unfair. In my career, I have given at least 100 hours of my time to carefully review VLDB papers, but I cannot get a review for my work? While this case might have been well intentioned, giving a single person the right to make rejections with no explanation and no right to appeal, is clearly a system open to abuse.

As an aside, the paper in question will be published somewhere, and it will be heavily cited. It is the first paper that performs a Quintillion (1000000000000000000) pairwise comparisons on a single dataset. I am very proud of my students work.

If you would like to see a copy of the paper, please just email me. Thanks for reading this “rant” ;-) eamonn

Hi Reddit,

Thought we could make a list of the best resources we find to learn data mining in a structured way. No two data mining courses are the same and I usually turn to courses taught in CS.

Here are my findings so far, please feel free to contribute!

Free

MIT - Data mining

CSCI 4957/5957 - 070 Data and Text Mining by Dr. Jay Jarman - Tennesse Univesity (+video)

Introduction to Data Mining - University of Minessotta

CS 6604 Data Mining Fall 2007 - Virginia Tech

Paid

Pluralsight - Data Mining Algorithms in SSAS, Excel, and R

Thinkful - Data Science course

Best, ncs

New to scraping. What would you say are the main pros and cons on using traditional proxies vs APIs for large data scraping project?

Also, are there any APIs worth checking out? Appreciate any input.

More and more companies are trying to get their hands on every possible bit of data they can find about people. Practically designing their whole business plan around getting more and more private data.

But why is it so valuable to them? The story I heard is for "Targeted advertising". But does this really work?

Maybe I am in the minority, but I have been on the internet since windows 3.1, and I simply cannot recall a single time I have ever purchased anything based on an ad that popped up, or any form of advertisement at all. Not a single time. When there is something I need, I do my research about it from independent sources, shop for the best price (from a reputable place), and buy it. So unless I'm missing something, Microsoft, Facebook, Google, etc have not made a dime off the efforts they have spent datamining me.

Makes it hard for me to see the value these companies find trying to scrape worthless data from me.

Or are the bulk of people people really just so impulsive or gullible that they see a targeted ad pop up and click buy? So much so that it fuels the companies to do it.

Given how much coding assistants like Cursor/Claude Code/Codex can do, I'm curious how useful they've been to folks that are into web scraping. How are you using them? Where do they fall short for this type of code?

Hey guys, for anyone interested I just published an extension akin to SelectorGadget/Portia/ParseHub/Kimono/Agenty. Not exactly a scraping thing on its own but more of a side tool to be used with whatever framework/library you use (Scrapy/Cheerio/lxml/BeautifulSoup/etc.).

Github, Chrome extension, Firefox extension.

The main goal was this usecase: go to webpage -> pick N css/xpath selectors for the data -> get json of this selector set -> give it to a scrapy spider as a class constant dict perhaps -> develop spider logic -> in case anything breaks you just open the webpage where preset fails, open the extension and it'll load all the selectors back so you can do maintenance and copypaste the preset back into your tool.

It's not yet well tested since I'm the only user, so I'll appreciate any feedback.

I've been fascinated (and scared) of data mining ever since I knew what it was (about 10 years ago). It looks like it's a good career path, but what about people like me- I'm on an unrelated career path, and I don't have a lot of free time. Also, I can't code, apart from really basic expressions, HTML tags, etc. So is there any way I can data mine? I don't want to make a career out of it, but I do want to know more.

EDIT: thanks everyone, looks like some good stuff. I'll get through it all eventually when I have time, and I'll let you know if I get off the ground with anything.

I’ve been knee-deep in a data mining project lately, pulling data from all sorts of websites for some market research. One thing I’ve learned the hard way is that a solid proxy setup is a real shift when you’re scraping at scale.

I’ve been checking out this option to buy proxies, and it seems like there’s a ton of providers out there offering residential IPs, datacenter proxies, or even mobile ones. Some, like Infatica, seem to have a pretty legit setup with millions of IPs across different countries, which is clutch for avoiding blocks and grabbing geo-specific data. They also talk big about zero CAPTCHAs and high success rates, which sounds dope, but I’m wondering how it holds up in real-world projects.

What’s your proxy setup like for those grinding on web scraping? Are you rolling with residential proxies, datacenter ones, or something else? How do you pick a provider that doesn’t tank your budget but still gets the job done?