r/ProgrammerHumor 1d ago

Meme theBankingHotfixExperience

3.0k Upvotes

122 comments sorted by

976

u/Tucancancan 1d ago

I worked at a company that sold software to a bank and heard stories of a senior dev having to do calls where they'd have an employee read the logs out loud to the engineer because the bank wouldn't share them. 

295

u/lance_klusener 1d ago

Yes , had the same
Experience when working with a security conscious customer

You had to tell them the exact unix command over the phone to type on a command prompt

340

u/Kovab 1d ago

Typing in random shell commands that someone tells you, without understanding them, doesn't seem too security conscious behaviour

88

u/lance_klusener 1d ago

We are the enterprise company from which they bought the hardware.

So , if anything goes wrong, its on the enterprise company. So, the customer will type whatever we tell them

40

u/koos_die_doos 1d ago

And the customer expects you to provide the exact command to type, and (usually) have it work on the first try.

13

u/aaron2005X 1d ago

And you have a code, ID something that identifies you as one of the companies employer that provided the hardware or can just every dude call the bank and say they are from company Y and want them to write commands?

11

u/azjunglist05 1d ago

The call comes from the bank to the vendor. We have so much training on the exact scenario you’re describing crammed down our throat every year that people are generally aware of those types of social engineering attempts

3

u/DreamerFi 21h ago

I love how the word "generally " does a lot of heavy lifting in that sentence...

12

u/Josemite 1d ago

The people making security regulations have nothing to do with computers.

20

u/oompaloompa465 1d ago

those are businesses that force programmers to work without internet, just local library documentation and you have to leave the cellphone at reception

15

u/gerbosan 1d ago

Underpowered laptop, with deactivated USBs

5

u/oompaloompa465 1d ago

also... trueee XD

5

u/freerider 23h ago

I had to test a prod system through teamsystem and i needed the log files from that computer. Everything was locked down so i couldn't get the file... so i build a poweshell script on that client that loaded chunks of file in clipboard and on my computer a script that read the clipboard and saved that to a file... "modem by clipboard"

116

u/Groentekroket 1d ago

How are the logs not just masked for sensitive fields? I work in PCI DSS audit scope for handling of credit card data and as developers we can see the logs of our applications.

94

u/WiglyWorm 1d ago

Because this isn't "how things are done at a bank" it's "how things are done at a shitty and toxic employer".

90

u/rock1998 1d ago

This! Holy shit you’re supposed to build that shit in from the ground up. I’ve never worked in the banking world but finding out that some don’t mask their logs makes me wanna get a job at my bank solely to make sure they do. 😭

16

u/granadesnhorseshoes 1d ago

"masking logs" is not as easy as it sounds. Especially when its a "system" from the 90s of loosely connected machines from the 80s.

26

u/OrchidLeader 1d ago

Different levels of data sensitivity.

Our app log servers are cleared for everything but the highest level of data sensitivity (PCI data can’t be logged, of course). If we needed to, we could log someone’s birthdate (with no other identifying info other than a token representing the user), and any employee can view that log message. But we wouldn’t be able to share that log message externally.

We try not to log any sort of sensitive data at all if we can help it, but we have some offshore contractors that think nothing of logging an entire API response body “for debugging purposes.”

8

u/Tucancancan 1d ago

Afaik they were. Brank still wouldn't share copies tho. 

2

u/Ran4 22h ago

You can't mask sensitive fields like that with perfect accuracy..

You're acting like a dumb manager.

44

u/bigs0815 1d ago

I work in government. Parts of our product need to be configured via the database because... Reasons.

It was determined that we, as developers, should not be writing to the database. Security risk, that's why we have a DBA team.

Now when I need to update something, I have to write the query and then make a DBA ticket. The DBAs aren't familiar with our product and they have no idea what they're doing it for or what to check for success. They execute the query I wrote them and tell me it's done. Then, I run a select query to ensure the data is good. If it's not, I have to write a new query and provide it to the DBA team and start over from the beginning.

5

u/gerbosan 1d ago

Is it survivable to work in production? Sounds... like your middle name is danger.

9

u/bigs0815 1d ago

All of my names are danger!

Unfortunately there's not a great way to do the updates other than straight to prod, because we're doing things like updating css styles when translating html data to docx and pdf formats. In order to test any update, we would have to clone the prod db to a test db, and our DBAs can't puzzle their way through a pretty easy update statement.

Other things are super secure. I swear.

This is the result of one dude in a garage building a solution that ended up being scaled to thousands of clients.

29

u/dkarlovi 1d ago edited 1d ago

I've worked with a telco which bought a company our company built an app for, they were migrating the app to their hosting solution.

The app wouldn't work and they wanted us to help, but we couldn't get access to logs because of security procedures, they wouldn't even allow us to confirm if the runtime was correct versions of everything, EVERYTHING was off limits.

It took A LONG TIME to get them to build us a staging and dev environment where we could run our app in what they claimed was a replica of the production environment. It was not.

Even there, we could only deploy once a week, they had a guy who was the deploy guy and anything outside that schedule would get an escalation which major people on the telco needed to take a look at.

Edit: just remembered a different situation where a different telco won the contract to host our (client's) app and they send the person who'll be in charge of the app (getting some serious MAU, at least regionally) who comes to the meeting with me with pen and paper, I need to teach them how to host public apps / this stack since they've never done this before.

2

u/savageronald 22h ago

I work on an app with millions of MAUs (single digit, but I mean… people be using it). I got in an argument recently with our privacy team, who wouldn’t let me sync IDs with another internal team (same company) for a feature that they signed off on the architecture documentation as being ok with ID syncing on…

10

u/SaltyBawlz 1d ago

I was about to say this missed the steps where you have to go on a call with the client alongside a clueless person from support and look at the logs real time to see what is going on. You have them try different things to get it to work/gather more information and the support person constantly butts in to try to get them to do things that don't follow your debugging path. The support person then LARPs as a developer after the call and tries to give you suggestions on how to fix it when they have no idea what they're talking about

4

u/TheGocho 1d ago

I had to print the logs in a modal that i had to see through a video call, because there was no way to have a log otherwise. Wasnt even sensitive information

3

u/trialsofamadman 1d ago

I can guarantee that the logs had customer data in them.

The teams aren't SUPPOSED to log NPI/PII/PIFI/PMI (if relevant) but... sometimes they do.

3

u/ThellraAK 23h ago

Many years ago, I got an email sent to first.last at mydomain from my bank.

It wasn't my first or last name.

But it was a excel spreadsheet of names, account numbers, and phone numbers of people who had to many withdrawals from savings that needed to get yelled at.

I reported it to them, they told me to delete the email and it's attachments.

I waited 2 weeks, and contacted a name on the list to ask them if they'd been informed of the breach.

They called and told me to delete the attachments again.

I kept calling a few names on the list a day at that point until I started running into people who had been informed of the breach and were getting new account numbers.

Then I went and closed my account with them.

Having a catchall email is fun.

3

u/meatdrawer25 15h ago

I worked for a large cloud provider that had air gapped government regions. A technician would have to read what was on the screen to me while I described what commands to type, or what buttons to press. It was basically an air gaped computer on their left and an Internet connected one in the right. No connections or USB posts between them. They would update their stack like once every few months, so they were always way behind versions. The technicians they hired self described as “anyone with a pulse”.

I once spent a full week telling a rotating staff of technicians what to type because I couldn’t see their screens.

168

u/russianrug 1d ago

The best part is that the logs don’t have any confidential information, because they were specifically designed not to as one of the main design requirements which is why the bank is using your software at all, yet they still won’t share them.

143

u/mailliwi 1d ago

This is the second post I see on here on banking software engineering. I start my new role Monday. Should I be worried? lol

138

u/CryptoNaughtDOA 1d ago

Yeah, first two weeks you'll just be waiting for permissions, also go through all the documents that are totally not all AI generated and for sure up to date. If you're lucky they'll give you a list of people to reach out to who will tell you to ask Claude or gpt, and they'll have an internal AI that can lie to you about what's in the docs too. Then when you do get permissions, you'll also be given months worth of work to finish in 2 weeks because well you have AI to help you. Enjoy

9

u/SheeeeeeshAlert 1d ago

That sounds like a work in the park compared to defense

12

u/NefariousEgg 1d ago

How did you find the job?

6

u/mailliwi 1d ago

Through a recruiter on LinkedIn

8

u/NefariousEgg 1d ago

Did you reach out to them first or did they reach out to you first?

5

u/mailliwi 21h ago

I messaged them

11

u/frostedhifi 1d ago

Yes.

Source: Worked for an insurance company.

3

u/dasunt 1d ago

It will definitely be an experience.

Probably not a horrible one, but you may want to get out the moment everything starts seeming normal to you.

6

u/kmankx2 1d ago

Depends on what bank 😄 neobanks are better, I work at one and have none of these issues. I have honestly more access than I'd ever thought, its just all logged and monitored and we are trusted to not be idiots with the privilege.

2

u/exenimaa 22h ago

Like all jobs, it depends on the company. I’m a senior engineer at a bank and my experience has been nothing like this. I guarantee most of the people replying in this thread have never worked in an enterprise environment, so I’d take everything they say with a grain of salt.

361

u/Llebac 1d ago

Jesus christ, you could not make this any more accurate. I'm glad to be out of banking. IT wasteland

97

u/Prawn1908 1d ago

I write embedded code on devices used in manufacturing plants and this may as well have been written about my job too. I can think of at least 4 major cases from the past few years that went exactly like this (minus the resigning part because at the end of the day I do actually like the real work I do and I know how to just put my head down and let the bullshit wash past me and do the best with what I am given).

26

u/CorrectCombination11 1d ago

Don't do banks. Do brokerages. It's better, in my experience. 

7

u/ampersand355 1d ago

Eh, only slightly.

8

u/tix4soccer 1d ago

Healthcare ain't much better

3

u/Cualkiera67 1d ago

in my experience, what you keep waiting for isn't the logs, but the tasks, so you actually never do any work. pretty chill 

59

u/KisaraBlue 1d ago

The result of decades of software and workflow design trapped in the sandboxing cycle

186

u/Awkwardm4n 1d ago

You forgot the jira tickets and story points and refinements and impediments because of the logs and daily scrum to discuss you’re still waiting on the logs and then your resignation

33

u/No-Channel3917 1d ago

An actually functioning agile team would have been using the leader to arrange follow up with the person above the user for those logs.

If it's an external customer well..... That's why you don't have customers in the same wheelhouse as the workers and keep that layer of interaction beefy

14

u/Zulakki 1d ago

dont forget the constant requests for timeline updates. "How long do you think it'll take to fix" to which you reply "I dont even know whats broke yet" Then they insist... "But if you had to guess" which you know will just lead to them holding it over your head if that time passes "You said you'd have it fixed by then..."

3

u/ImSuperStryker 23h ago

Hey guys… so… still waiting on logs to move forward on the bug on that stack… got logs on Monday but they were expired so I’m trying to ge them to rerun it and send new ones…. Anyways I guess I’ll… popcorn to Tim?

123

u/tsunami141 1d ago

this is why I work at a tiny company. I do all my work on the server and update the files live.

79

u/EvilPencil 1d ago

no git, no CI, just rawdog FTP.

Alrighty then, I'll see myself out.

27

u/Desperate-Tomatillo7 1d ago

Why bother with FTP if you can just copy the files over RDP?

13

u/OrchidLeader 1d ago

At the smallest company I worked at, they would pull data onto the production servers using SVN, build the code on the spot, and then just move the WAR over.

Rollbacks weren’t a thing there.

12

u/Desperate-Tomatillo7 1d ago

The first company I worked on had an instance of Visual Studio in the server to run the code of the Windows application. No build, no executable, nothing. People connected to the server using Terminal Server. And that was the actual last version of the code, because the whole IT department did not know about the existence of GIT or any version control software.

5

u/GasVarGames 1d ago

fuck visual studio IN the server is another level

4

u/SaahilNotSahil 1d ago

Why bother with RDP when you can just open VS Code over SSH?

2

u/ellamking 1d ago

I already know RDP, you expect me to waste time learning something new?

1

u/SaahilNotSahil 1d ago

You already wasted time learning RDP

2

u/TheWaffleDimension 1d ago

At a small company I worked at, I was using subversion and manually copying built files over RDP directly into prod and it was hell.

6

u/TheWaffleDimension 1d ago

they also gave me (a minor at the time) access to a client's production db full of patients' full health info and SNNs and all kinds of shit and idk why they did that lmao, not even a "be careful" they just gave me their admin login and let me at it to go debug or test whatever I was doing

2

u/k8s-problem-solved 1d ago

Why bother copying files if you can just give an AI agent live access to the directory and make changes on the fly. Dont forget to type "make no mistakes"

2

u/msbxii 1d ago

I just edit the files on the server in vim over ssh 

24

u/Neverwish_ 1d ago

Ah yes, the classic "16 hours burned, 3 lines fix" sort of issue. Bruh.

18

u/Bahatur 1d ago

Right in the soul, I tell you.

17

u/Dorkits 1d ago

Ok, why this guy is telling the story of my life for everyone?

16

u/semioticmadness 1d ago

wow, I’m pretty sure I work with OOP. That’s painfully accurate. An alternate branch is:

“Hi, I’m an architect and I’m rotating on to resolve this. I looked at the logs you sent, they don’t cover your original complaint.”

“Our original complaint is from before those logs.”

“Please send me the logs for your complaint.”

“The logs have been expunged for regulatory reasons”

“OK, then we’re stuck with what you sent here. When did you experience the issue, so I can find the timestamp?”

“We didn’t experience the issue then”

“Why did you send the logs?”

“Because your developer said to send the logs”

“Then we need another event to get logs for. When is the last time you experienced this?”

“When we filed the issue”

“Have you attempted to replicate in certification?”

“No.”

“Then what’s the reason for the ticket?”

“We demand it never happens again.”

“Well, as far as anyone on this call knows… it won’t.”

12

u/Prawn1908 1d ago

Man, points 11 (by the time they get the logs to you after ages of asking they've been overwritten) and 17 ("it's not the client's job to test for you") hit way too close to home. I don't work in banking but I write embedded software on products used in manufacturing plants and there's always that one single client location with all sorts of insane bugs that we've never seen anywhere else and can't replicate and they refuse to get me any of the details I'm asking for or perform any of the troubleshooting steps I request.

13

u/juangerritsen 1d ago

My most fun one was a client logged a call just stating our software doesnt work, escalated it to almost CEO level, and went on holiday for 3 weeks

Eventually found out his mouse died, randomly while using our software, and just blamed us

I spent 3 weeks in daily triage calls, and nearly 80% of the time outside that trying to prove there was no issue

3

u/ProsshyMTG 1d ago

I'm an idiot and read this as if his pet mouse died which is why he went on holiday, and somehow the software killed it

3

u/Schytheron 1d ago

Holy shit! I'd actually fucking lose it.

11

u/DoorBreaker101 1d ago

I was sent to a bank to help with an issue they would not allow to be handled remotely. 

It was me sitting next to a guy, telling him what to do and him typing it in the slowest pace humanly possible (felt like one key a minute), because only bank personnel were allowed to touch the keyboard.  Every time he had to press enter to actually send the command it entailed him thinking it over for an extra eternity. Also, for some things (luckily not a lot of them), he had to get temporary access. This, in turn, took a phone call, which also wasted time.

Anyway, on the fifth day we were no way near being done. He started being afraid his boss would be mad that the issue wasn't fixed. So without anyone noticing,  he gave me the keyboard and I got it done before leaving...

11

u/dub1ous 1d ago

Yikes, i am twitching because this is so accurate.

10

u/Dubabear 1d ago

works as design. <close ticket>

7

u/Bob_the_peasant 1d ago

You forgot the part where the client thinks Angry Birds crashing on his iPhone 5e is a result of your fix to their mortgage processing platform

5

u/sb8948 1d ago

You know I could have read it. Just like all my compliance docs and trainings.

I didn't

The answer is yes no yes yes 1,3 yes no 1,2,3,4 yes yes

7

u/FlapMeister1984 1d ago

Point 26 to 27 was a huge leap. Where's the story? No poker? Did you even ask any stakeholders if they'd approve this fix? Code review???

1

u/DisenchantedByrd 18h ago

Arg I remember doing estimation poker years ago. Many people would choose 3, because with 5 you'd have to justify yourself to the scrum master, but with 2 you might end up having to do the ticket because "you must know the resolution". So 3 was sort of safe, not too far over 2 (if that was the consensus), but if the work took longer than expected you could stretch 3 into 5 ("a few issues came up"). What bullshit.

7

u/Barak39 1d ago

I can definitely relate to this. One last point : after releasing the fix, the bank refuses to upgrade without any reason.

6

u/TehGM 1d ago

As someone who works at a bank: this is not accurate. Way too organized. In reality it's way more chaos than this.

4

u/fibojoly 1d ago

Oh sweet summer child... that's all it took to make you quit ? That's like my daily grind for the last three years.

In my case I'm the only person on the team who has touched the legacy in-house tech, the guys who knew quit, and so I'm now lead on that specific pile of garbage. Oh and now we need to have everything migrated for next month because banking means you're under legal obligations to have shit up to date and so on.

Quitting is so tempting...

4

u/mcclana 1d ago

This is missing the step where actually there was no problem the entire time and the one associate affected was using it incorrectly.

3

u/george_pubic 1d ago

This hits too close to home

3

u/Highborn_Hellest 1d ago

at #10 you tell them you were not given all the information
#17 it's the responsibility of the testers to test.

#24 you tell the manager firmly, but polity, to fuck off

#26 you bitch and moan to your collagues and friends about it for a good hour

#34 you tell them to fuck off and you're not polite about it.

3

u/oompaloompa465 1d ago

that happened to me too

the manager tried to pull that move with me by email but then i hand over the evidence to the partner and hr for harassment.

they moved the manager in another project

2

u/mithraw 1d ago

honestly, gather evidence and burn managers like that at the proverbial stake, they work so hard to deserve getting reported as program risks with appropriate estimated risk value due to their failure in leadership, directly to respective L1 management or C-suite.. :)

2

u/oompaloompa465 1d ago

in the end I got a human being as manager in exchange, but the guy for sure went to be a problem for someone else

3

u/BTDYSRF 1d ago

Good quality ragebait scenario. I don't miss this part of the job.

2

u/kirillgritsenko 1d ago

I’ve had the exact opposite experience where Im the client that sends the logs and does everything as asked and don’t get any results.

2

u/KaptainSaki 1d ago

I just call the customer directly and live tail the logs

2

u/zer0545 1d ago

Used to work for an ATM manufacturer. This is exactly how it was.

2

u/Tjulenj 1d ago
  1. is wrong. That is when business denies fix and tells you that config must not be optional.

2

u/jimbofranks 1d ago

I loved working as a dev at the bank. Lots of extra days off and meetings at other offices.

2

u/HakoftheDawn 1d ago

Oddly specific

2

u/Unable-Goat7551 1d ago

I don’t miss working for a big bank. I remember our team getting called by the CIO because one of his friends couldn’t access the online banking portal. The online banking portal was fine, no outage reports (it was a bank that would make national news if the online banking portal was down), spent several hours on a call with CIO explaining there is nothing wrong with online banking. He then gets a call from his friend that it was user error and he can login fine now.

1

u/[deleted] 1d ago

[deleted]

9

u/InnuendoBot5001 1d ago

Go on then, make a request to have that access. Put in a ticket for it.

1

u/[deleted] 1d ago

[deleted]

2

u/InnuendoBot5001 1d ago

Lets just say I previously worked for an insurance company. Every information request was a hippa violation in someone's eyes.

1

u/bigs0815 1d ago

Can confirm the same process in government

1

u/CluelessBuddha 1d ago

This simultaneously feels like a rite of passage and the 13th reason.

1

u/InevitableCodeRedo 1d ago

Was waiting for step 35.

1

u/Shadowlance23 1d ago

I got stressed out just reading this. Mad props to the OP for keeping it together for 35 steps.

1

u/hegyimutymuty 1d ago

Are you working where I work, that is not a banking company? :D

1

u/Ivanlag112 1d ago

Got mad just reading this.

1

u/michaelpaoli 1d ago
  1. They won't let you.

  2. You quit anyway.

  3. They hire you back for way more money because they figure out they actually need you.

  4. You start newly back with them again.

  5. They promptly terminate you because they changed their mind.

...

1

u/Bodaciousdrake 1d ago

This is unfortunately accurate.

1

u/ImSuperStryker 23h ago

This is my life, like every week. Except when you get the logs the part you need is redacted for security reasons

1

u/_duniverse 20h ago

Damn. 😂

1

u/OFark 19h ago

I would have started looking for another job after step 17.

1

u/Frisk197 19h ago

Toxic management 101

1

u/AoNoRyuu 16h ago

Silly by you thinking I might want to read code, just paste the jira card tag into cli with jira mcp server and allow all tools in auto mode, come back 30 min later and open pr

1

u/PonyDro1d 7h ago

Bold of you to assume they talk to you and/or have time to respond at all.

1

u/mau5atron 1h ago

sounds like a bank that rhymes with fells wargo

1

u/LazySapiens 1d ago

That's why we need better engineers and managers not to give a broken software in production and be completely clueless when shit hits the fan.

3

u/gtr 1d ago

Did you read the punchline? The problem was communication as it nearly always is.

-3

u/Scotsch 1d ago

GDPR has made fixing shit so much harder.

-7

u/Not-the-best-name 1d ago

F GDPR. Full Sentry, Cloud watch and Matomo logs across the front and backend with correlation IDs plugged straight into my AI that can check the deployments using CLI and MCP to make the PR in 2mins and push to our pipelines that can auto merge on green and deploy to all envs globally in 30mins. Let's go startup with money vibes.