Post

Conversation

Click to Subscribe to Noahpinion
Here's someone who claims that processing just 60,000 rows of data overheated their hard drive 😂
Quote
DataRepublican (small r)
@DataRepublican
Replying to @DataRepublican @JuddLegum and @elonmusk
In my initial run, which processed the first 60,000 rows, I did not find these awards—my hard drive overheated long before I could complete a full pass through the database. In a later run, which I referenced in another post, I did identify two such awards. That discrepancy is a
Show more
David Watson 🥑
Post your reply

That whole account is a dumb person’s idea of what a smart person sounds like. I’ve read their threads. They say so much and yet nothing at all
Most data scientists are bad enough programmers that I would need to see the code first before making any judgements instead of just believing she's lying
I don’t know anything about that account, but you’re clearly misreading. Person isn’t saying the processing caused the overheating. They’re saying it overheated due to environmental reasons after processing 60k rows. Come on.
A “data expert” who thinks 60,000 is a Huge enough number people will buy the “overheated my hard drive”. It’s layers upon layers of half-assed grift, manipulation and the stupid scamming the stupid.
i had never heard of this person until today and it blew my mind. any laptop in the last ~10 years could load 60k rows into memory with python. lmao
You aren't reading carefully. The claim is that the hard drive overheated before it could complete a full pass. The claim isn't that the hard drive failed during the initial run when processing only the first 60,000 rows.
Reading comprehension dropped a lot for you on that one. They said they did a pass on the first 60,000 records no problem. They did not claim that they overheated their hard drive doing so..
This is the “analyst” who demonstrated (falsely) all the people who were supposedly funded by USAID. This person need not be taken remotely seriously. They do get a lot of Elon RTs tho!
Oh I love it when non devs think they know stuff. Your laptop could overheat after 100 rows if the query is complex enough. To mock and laugh without knowing details is some serious midwit shit.
Pretty clear it says 60k rows were processed in initial run. Separately, the hard drive overheated long before she completed a full pass through the database
My guess is that only a technologically ignorant and/or exceedingly disingenuous person would think this is necessarily a condemning burn. I do appreciate that it exposes the irrational bias though.
That's not what they said; they said they scanned the first 60k rows and didn't find any awards; then they said their hard drive overheated well before being able to scan the whole thing, which would be the entirety of the database (probably a monumental amount of data).
oh no, someone notify my cloud provider, their hard drives are on fire after processing exactly 18,997,421,456 rows of data 🔥😱
Image
i’ve run spreadsheets smaller than this that have caused thermal throttling - although i am uncertain whether it was the CPU or the SSD (both of which often have throttles)
One of the apps I wrote that I'm genuinely proud of produced a statistics report for keyword usage over a world-spanning, decentralized, distributed NoSQL database. It had an intermediate table with a billion rows. Took about four hours to run on 2010 hardware.
Yeah I have melted hardware on less data than that because my sql was bad. In an unrestricted homebrewed environment it isn't even hard.
It may very well be depending on what the processing is. Hashing each row 100 trillion times rereading from disk each time would get it toasty indeed
I feel like they meant the query generated (as opposed to processed) 60k rows? Even intro level Kaggle datasets have more than 60k rows but don't need db processing
No, she doesn't. She says that while processing the whole dataset her harddrive overheated, having completed the processing for the first 60k rows. Don't think she has specified exactly how she does her processing, but given it involves finding multiple layers of connections
60k result set
Quote
DataRepublican (small r)
@DataRepublican
Replying to @molly0xFFF
The US Spending database is multiple terabytes large. It literally does not fit in any of the standard Macbook Air builds.
“Processing” and “rows” aren’t static terms. What operations are applied to the rows? Are the operations iterative and considering a single row? My day job is pulling and manipulating data in applications, and I work with datasets anywhere from a few thousands records to
depends on the row size, object sizes in those rows, and compute per cell. are you retarded?
I've gotten better performance querying databases of 60 million rows on a ten year old city owned computer. It really sucks how many liars and frauds get ahead. In this decade, it seems more reliable than talent or skill.
The reason most 7-figure founders stay stuck? They refuse to fire themselves from operations. You don’t need another funnel. You need to: •Document •Delegate •Delete
My father in law is a data scientist. It is difficult to get his attention when he's lost in wonder in a spreadsheet. We were looking at data together years ago and I asked him if we could go over 60,000 rows. I will never forget his answer "We can't, we don't know how anymore"
If you want to look for a weakness in your ideological opponent, I wouldn't start with technical incompetence. I could blow up your computer with 5k rows if you let me specify the data set and the computation.
I mean, that might make sense... if the rows were 60 quintillion columns wide. And the hard drive was being used as virtual memory. And they were working on a laptop. In a sauna.
How does she even know that her hard drive overheated? Is this happening on a laptop, a desktop, or a cloud server? At work I process large data sets using either Hadoop or Flume. At home, I use my gaming rig with a RAID NAS for handling large data sets.
The responses to this from MAGA show how polarization / dunking emotionally prime people to lose sight of the big picture. The big picture is that a person who can't query the database successfully shouldn't be so influential on Musk's thinking.
“Data expert” running a hard drive from like 80’s? This is either a demonstrable lie or for whatever reason the most decrepit hardware ever is being used here.
Doesn't this depend on what processing is being done on those rows? In the quant world even 10k rows can thrash my local hardware with all the feature extraction applied to it.
People running software that actually does a lot of complex stuff, you read the problems they have and it's not "this melted my computer," you absolutely NEVER hear that, it's "Why is this only using 12% of my CPU and none of my GPU, what did I buy all this for?"
I thought this was pretty funny, too, but when I dug in it didn’t actually seem that crazy. She apparently had the entire huge dataset on her external hard drive and was querying it locally (there are pros/cons) vs sending a request to a remote server.
That is not what he is saying. He says in a sample of 60k rows he did not find any awards. Before finish running a full pass, his hard drive overheated.
doesn’t matter how many rows, nor how big those rows are (and a row could theoretically be of any size). hard drives don’t overheat.
you should read the rest of her comments. she knows more about databases and how they work than nearly any analyst i have ever met. (db specialist here)
Can hard drives even overheat? Or did they mean the cpu. Also those don’t really overheat either. They do run slower to reduce heat, and have fans. Usually.
I’ve worked a lot with government data. Whether things look messy or not, the first thing I would suspect is not technical incompetence on the part of the analyst, it’s a lack of subject matter knowledge.
Macro Data Refinement: Where processing rows of data never ends, even in the case of hard drive immolation.
Complaining about hdd overheating while processing 60000 rows is literally incompetence even Excel handles that like a champ it's over the top on mem but still works flawlessly
The problem with Twitter is that both of you are making such short statements about complex things that it's impossible for either you or the person you are commenting on to fully explain what they mean, so we end up with this pointless tribal mudslinging
I hate to be fair to such scum, but… They successfully processed 60k (a subset of all data), and didn’t find what they were looking for. But that’s as far as the “fair” goes. Their hard drive “overheated” processing the full data set??? 🐂💩
GIF
Here's someone who "thinks" that processing 60,000 rows means scanning 60,000 rows 😂 as opposed to, say, an obviously more appropriate scenario of running some complex query which produces or updates 60,000 rows... Cheers!🥂
holds a patent for creating an innovative database architecture while working at Ebay. The processing involved in each of the rows that you mentioned probably exceeds our wildest imaginations.

Discover more

Sourced from across X
Oh wow
Image
Quote
Magills
@magills_
Image
Image
This is the same method Clinton used to get Robert Reich on the chopper
It's like watching 2 trains on the same track coming together
Image
Quote
Will
@BushidoToken
FBI has warned about a new type of cybercrime campaign whereby "free online document converter tools" are used to load malware onto victims’ computers, leading to incidents such as ransomware. ...No IOCs though 🙃 fbi.gov/contact-us/fie
This breakthrough research was funded by a grant from the NIH, a major target of the Trump budget cuts.
Quote
vittorio
@IterIntellectus
holy shit MIT researchers just turned skin cells directly into neurons without stem cell intermediate, 100-fold efficiency boost, and they actually worked when transplanted into mouse brains 1/
Show more
Image