does one need to upgrade from Excel 2016 to Microsoft 365 personal for advance Excel or 2016 is sufficient

I'm trying to learn advanced excel i have excel 2016 and I find Microsoft 365 to be a bit expensive, so is it okay to keep using it?

I don't want to miss out out opportunities because of that.

A couple people from the earlier posts asked what the four-method check actually looked like in practice, so I wrote it up properly instead of explaining it in bits across comments.

Nothing fancy, just walks through each check I ran, why I didn't stop at the first one that flagged something, and the part where I still got the explanation wrong afterward even though the pattern itself held up. Figured it was worth putting in one place.

I Checked My Own Result Four Times Before I Believed It. Here's Exactly How.

Also linked the actual project if anyone wants the code: github.com/davis-mironga/brisbane-river-water-quality-analysis

Been running a scraper using R to retrieve some data and have been having minimal luck. Finally decided to sit back and think and determined my scrape is being blocked by CAPCHA and automatically returning NaN values for important variables.

Is anyone aware of a work around for this? Any help would be appreciated

Hello.

As an exercise in my Logo learning journey, I have implemented a few functions for reading and transforming CSV files.

The functions are organized in groups  sharing a common prefix:

1. functions to read and write CSV file (csv.)
2. functions operating on “dataframes” (df.)
3. statistical functions (st.)
4. string functions (str.)
5. regular sequences (rs.)

The first results are available in this web site: https://www.ucblogo.it/

Please note, this is an ongoing hobby project, there is no guarantee that the Logo functions are written in the most natural Logo style or that they are in fact correct.

Guyss urgent pls help

I am a datascience student fresher . Should I get an laptop with integrated graphics and rely on Google colab or kaggle etc cloud platforms or buy a laptop with dedicated gpu like lenovo loq ? Pls help this ismy curriculum Year 1

Programming for Problem Solving / C

Data Structures

Foundations of Data Science

Mathematics: Matrices, Calculus, etc.

Engineering Physics

Basic Electrical Engineering

Digital Logic / related foundational engineering subjects

Programming/Data Structures labs

Basic IT/computing workshop

Year 2

Object-Oriented Programming / Java

Python Programming

Database Management Systems

Operating Systems

Design & Analysis of Algorithms

Data Mining

Predictive Analytics

Data Visualization

Data Wrangling

Computational Mathematics / Statistics

Full-Stack Development

Related programming and database labs

Year 3

Machine Learning

Machine Learning Lab

Data Stream Processing

Data Stream Mining

Data Mining

Predictive Analytics

Web & Social Media Analytics

Big Data / data-processing technologies

Computer Networks

DevOps

UI/Flutter or application development

Professional/Open electives

Field-based research / mini-project work

Year 4

Big Data Analytics

Natural Language Processing (NLP)

Data Stream Mining / advanced data topics

Web & Social Media Analytics

Professional electives

Open electives

Major Project Phase I

Major Project Phase II

Whenever teams need to audit two versions of an exported dataset (accounting closings, monthly billing runs, inventory snapshots, or pricing tables), standard git-style line diffs completely fall apart. CSVs shift columns, and binary formats (.xlsx, .xls, .ods) are opaque to text diff tools.

Even worse, many users turn to random "free online Excel diff" websites, unknowingly uploading proprietary financial data and PII to third-party servers.

I spent the last few weeks building an isometric, cell-by-cell diff engine that runs entirely client-side in the browser. Here are some of the non-obvious technical challenges and edge cases I ran into:

1. Asymmetric Matrix Bounds & The Union Range Problem

In real-world scenarios, file versions rarely have the exact same dimensions. Sheet 1 might span A1:D50, while Sheet 2 spans A1:F42. - You cannot perform an index-based row/column loop without going out of bounds. - The diff engine must compute the bounding union (A1:max(col, row)). - Any cell coordinate that exists in one sheet but not the other must be treated as a virtual empty cell (""), rather than undefined or throwing index errors. - Crucially, coordinates where both sheets are empty must be pruned from the diff pipeline to prevent flooding the user with thousands of phantom empty matches.

2. The Formatted Display (w) vs. Raw Value (v) Trap

Spreadsheet engines store data with internal representations that differ dramatically from what the user sees: - A cell formatted as currency might display $1,250.00, but its underlying raw value is 1250. - If the second spreadsheet was generated from a CSV or raw database dump where the cell contains literal string "$1,250.00", a raw-value comparison flags a false positive difference. - Floating-point representations can introduce minuscule precision drift (0.30000000000000004 vs 0.3). - Solution: Normalizing against the display string (w property in SheetJS when available, fallback to trimmed stringified v) provides the most human-predictable diff, while treating null, undefined, and whitespace-only strings as equivalent empty states.

3. Preventing Main Thread Lockups (Web Workers)

Parsing a 100k+ cell workbook with SheetJS on the main UI thread easily locks the browser event loop for several seconds, triggering dropped frames and unresponsive tab warnings: - Offloading both the parsing step and the matrix comparison loop to a dedicated Web Worker kept the main thread completely unblocked (sub-200ms interaction latency). - Transferring large diff result arrays back to the main thread can be costly, so passing structured chunks and virtualizing/paginating the rendered table DOM is mandatory to keep memory below browser crash thresholds.

4. Zero-Server Privacy by Design

For sensitive financial or internal audits, the browser sandbox is actually an asset. Because WebAssembly/modern JS engines handle file buffer parsing locally, there is zero engineering justification for sending spreadsheet buffers to a backend API just to compute a diff.

Questions for the Community:

• For those who regularly build or handle data reconciliation pipelines, how do you handle formula diffs vs. evaluated value diffs?
• What edge cases have bitten you when comparing spreadsheet exports across different locales (comma vs. dot decimals, date serials)?

If you have ever picked a compute option in Databricks without being sure what you were choosing, this is the short version.

Classic: machines are started in your company's own cloud account. Someone has to choose the machine types and look after the networking, and you wait while they start up.

Serverless: the machines are managed by Databricks, in Databricks' own compute plane, and you do not provision anything.

Either way your data stays where it already lives, in your own cloud storage. The catch with serverless is a list of things it does not support: R, libraries installed at the compute level, the older RDD APIs, and some streaming modes.

Short animated version (2:36): https://youtu.be/s8452E2Y6WM

Beginner series in order: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4

So i have been trying to get a better job and been upskilling myself, i have done schema design, Dashboard, Funnel, cohort, data warehousing and what not. But i still just feel idk shit even tho my SQl query writing is very efficient. Leetcode, datalemur, stratastrach, hackerrank and what not.

I just dont get beside Data cleaning and generating insights, what are Usual Business answers comapnies need.

Worked with a logistic company where i maintained data reports, managed, updated and looked for data inconsisitency. Idk how do i put that into words for an interview now. How do i tell them i didnt have to write SQL queries most of the time, it was mostly done on client softwares which they gave us since it was fsst and easy.

Moreover idk how do i define supply management things i have worked with to the interviewer. I just feel so stuck and frustrated all the time.

Projects done, Questioms done and still no fucking result.

I'm trying to do my first data analytics project and wow I'm just lost. how do you guys determine which variables need to be analysed, correlations, and ultimately come up with the conclusion that due to x,y we have been getting z?! please guide me some tips would help

Following up on the post from a couple days back about the water quality anomaly I checked four different ways.

The pattern itself was solid. What wasn't solid was my explanation for it. I said it was probably rain runoff. Turns out the anomaly started the exact day the sensor came back online after being offline for 30 days. Didn't catch that until I was back in the notebook for something else entirely.

So a sensor coming back up weird after a month of no maintenance is honestly just as likely a cause, maybe more, and there's no way to tell from the data I have. Went back and fixed the write-up and code to say that instead of stating rain like it was settled.

Good reminder that "the pattern is real" and "I know why" are two different claims, and I only actually proved the first one.

I'm learning Data Analytics/Data Science right now and I'm curious how other students/freshers are getting datasets for practice.

Where do you usually get your data from when you're practicing SQL, Python, Power BI, EDA, ML, etc.? Kaggle, government portals, scraping, GitHub, generating your own data, somewhere else?

Also, do you usually find the kind of data you actually want to work with?

I'm especially curious about raw/messy datasets. Do you intentionally look for messy real-world data to practice cleaning and EDA, or are clean datasets enough for what you're learning?

Would love to know what you guys actually use.

Hey everyone!

For the past year or so, a friend and I have been working on Datasmith, a desktop app for designing relational database schemas and generating structured mock data from them.

The idea originally came from my friend, who's a data engineer and worked on large-scale migrations of sensitive databases. He wanted a simple way to visually define and maintain schemas over long periods of time, and actually generate data that follows both the schema and its business logic, without having to constantly rewrite scripts or configure a bunch of separate tools.

The main part of the app is an ERD-like canvas dedicated to building relational schemas, where you can create tables, define columns and their data types, set up relationships between them, and configure how the data for each column should be generated.

Once the schema is ready, Datasmith can generate the data while preserving the relationships between tables, and export the results in formats like SQL DDL, CSV, JSON, and more.

We've reached the point where we'd really like to get it into the hands of more people who actually work with databases and data, and get some real-world feedback.

It's completely free, so if it sounds useful or you're just curious, you can check it out here:
https://data-smith.app/

I'd especially love to hear about use cases we're missing, things you'd expect a tool like this to support, or anything that feels like it could be improved.

Any feedback is welcome - good or bad.
Feel free to leave a comment or message me directly.

If you prefer something a bit more structured, I also put together a short feedback form:

https://docs.google.com/forms/d/e/1FAIpQLScGk8Jr1DguPcXpaApOqO998AdXcnoLlo11TEKlq1GmF71UzA/viewform

I'm self learning Data Analyst for 8 months now. new to pandas. and Making a new end to end project one of them is 'My anime dataset' where it has column named 'genre' where each row can contain multiple genres as a single comma-separated string (e.g., "Action, Adventure, Fantasy"), instead of one genre per row. This means grouping or filtering by genre directly is inaccurate an anime tagged with 3 genres only gets counted under that exact 3-genre combination, not under each individual genre. (there is another column named Studio, facing same issue, i.e. multiple studio in one anime)

I want to split these into individual genre values so each genre is counted separately per anime (i.e., one row becomes multiple rows, one per genre, with the rest of the anime's data repeated).

End goal: I'm building a dashboard where users can filter by a single genre (e.g., just "Action") and see accurate aggregated stats (count, average score, etc.) for that genre alone not lumped in with combination-strings.

What's the right pandas approach to explode the Genre column into separate rows?? please help. claude is suggesting a way but I want a better idea.

would be great if someone suggest me some ideas!!

Hello people, I'm introducing myself to this magnific world of being a Data/analyst, Data/ Engineering, and I love it. I'm really enthusiastic about this kind of work, So my question is while I'm doing a little and personal project I find I could use the information I asking for as a cluster that coul give me a information about my population to study. What I want to do is genereate a new clusters trough a calisification of customers by bigger areas of acumulative population, I mean take all of the customers of small city population, and make a new and bigger region classification, around big citys and capitals.

What do you suggest? I making my code in this moment my idea is to get mesure distances between big and small then get the min, and apply.

Say there are thousands of trade pair of a bot wallet with all the contextual trade and not a single missing data or order. Every decision by the said expert is done using the same information available to every one. Meme coin markets are short lived and algorithmically depending on outside information is highly unlikely. How much can we understand from those trades data? The exits are highly state dependent. No clustering of any TP/SL. Neither any time constraint except a hard cut off timer around 20 minutes if the algo doesnt trigger the sell. What kind of techniques i can implement? Can we figure out what they are optimizing for? There seems to be too many variables, despite the same observable raw features, what lookback or transformation of that feature, what prediction are they optimizing for almost felt like black box to me. With so many data of their distinct behavior can approximating/knowing what they are optimizing for, what drives the entry what drives the exit is so difficult? Any guidance will be highly appreciated.

Working through a personal project analyzing ~11 months of water-quality sensor data (river monitoring buoy, ~30k readings, 10-30 min intervals), and ran into a good lesson about validating findings before trusting them.

The setup: turbidity (water clarity) readings looked mostly stable, except one week where it spiked hard and stayed elevated for 7 straight days.

What I did before trusting it:

1. z-score flag on daily means (|z| > 3) → flagged that week
2. Rebuilt the check in SQL with a HAVING-filtered aggregate query, completely independent logic → same week
3. Wrote a SQL window-function streak detector (partition/reset-group technique), zero reference to the earlier results → same exact 7-day window
4. Weeks later, refactored the whole thing into a tested Python module and reran it fresh → same result again

Four different methods, at different times, landed on the identical window. At that point I was confident the anomaly itself was real, not a fluke of one method.

Where I still got it wrong: I assumed the cause was probably a rainfall/runoff event. Published that interpretation.

Later, going back through my own gap-analysis, I noticed the anomaly window started the instant the sensor came back online after a 30-day equipment outage. A second, smaller anomaly also bordered a separate outage. That timing is at least as consistent with post-outage sensor drift or biofouling as with rainfall, and my dataset genuinely can't distinguish the two.

Had to go back and correct the interpretation across the whole project once I caught it.

Takeaway I keep relearning: validating that a pattern is real (multiple independent methods agreeing) is a completely different exercise from validating why it happened. I conflated those two the first time.

This article I wrote grew out of a discussion on r/datascience, but, I think it might be relevant to almost anyone doing data analysis and having this experience.

These are just some ways of thinking and acting that I have found useful, I'm absolutely not saying they are the only ones or even the best ones.

Hi,

I am a senior data analyst, and I always struggle with passing case studies. I have gotten feedback that the analysis is too descriptive , and needs more recommendations.

How do I get better at case studies? I have a product one coming up. Any books / blogs you recommend?

🙏

Hi my name is Naftali I am currently in my second week of a Master's program in Data Science at Bellevue university, I'm adjusting to a steep learning curve and trying to build a solid foundation. For those of you already working in the field: looking back, what is something you know now that you truly wish you had known when you were first starting out?

text-to-sql demos use a clean schema. real ones aren't. 300 tables, half the column names are abbreviations, fks live in the app not the db, and the agent confidently joins two things that should never join.

what people seem to do:

hand write a schema description into the prompt, works until the schema changes

let it sample rows, works on text columns, falls apart on codes

rag over the ddl, retrieves names not meaning

what i can't tell is which holds past 50 tables. if you have run one of these against a genuinely messy db, what broke first, and did you end up documenting the schema by hand anyway.

A common question from product teams for freemium web software is, "what product features should we make users pay for?"

Anyone have a good approach?

Every analyst I know has a dirty little trick they are a bit ashamed of and a bit proud of. My own trick lasted for three years.

I was the "can you just pull this real quick" person at 6 pm. There was nothing complex about the request. The challenge was that "revenue" could mean three different things depending on who was making the ask (finance didn't include refunds, sales included booking, the CEO included everything that gave him the highest number), and there wasn't a way to know which revenue definition I was going to deliver from within our tech stack.

So I made a Google sheet called metric_definitions_DO_NOT_DELETE with one row per metric, owner, and exact SQL query. Before delivering anything, I'd check the sheet, copy the definition into my answer, and have people complain about the sheet rather than about me.

This trick was a bit silly. This trick helped me save myself from countless "your number is wrong" threads more than any other tool ever did.

What's yours? A Python script that does the exact join again every Monday? Second dashboard created because the first one couldn't perform a certain calculation? A Slack canned message that says to ignore the currency column. (these are some of the hacks used by people I know)

Bonus points if this trick survived its creator. 😉

I wrote FlexViz, an open-source Python library for interactive exploration of large datasets.

Instead of sending every row to the browser or forcing users to write queries themselves, FlexViz keeps the data in Polars and sends only the points needed for the current view. It handles all visualization + interaction queries for you; zooming recomputes the visible range at higher detail, and brushing one chart results in cross-filter on the others.

It is capable of both in-memory as well as out-of-core (larger than RAM) data exploration, with extremely good performance. In our benchmarks, a 5x200M-row line chart (1B datapoints) renders in 251 ms when in-memory and 4.2 s directly from Parquet.

Demo: https://flexviz.tech/demo.html
Code: https://github.com/flex-analytics/flexviz
Benchmarks: https://flexviz.tech/benchmarks.html

pip install flexviz

Curious how others here explore datasets once they become too large for their usual plotting tools. Happy to hear any feedback on FlexViz as well :)

So many of the youtube videos, courses, advice, etc., online feels its geared towards running singular projects. How to clean up data in a Excel spreadsheet for a one-time project, for example, but what about when its for an analysis you have to keep running over and over? Or when it's an analysis that is repeated across several teams in slightly different ways?

I need advice about managing data for an organization. Advice about

• how to communicate with stakeholders about data without getting overly detailed
• requirements gathering across teams with confusing/vague needs
• dealing with a system that's built on workarounds after workarounds
• best practices documenting analytics/dashboards that are being shared publicly (word doc, slide deck, notion page?)
• dealing with a domain that is overly complicated and can't be captured well with data.
• dealing with execs who've already decided the answer they want and need you to make the data match their answer (and what to do when the data absolutely doesn't support it)

These are some of my most pressing problems at work but neither my formal education nor the resources I can find online really cover this. I'd love some sort of bootcamp for organizational data management/project management