Linux nerd and consultant. Sci-fi, comedy, and podcast author. Former Katsucon president, former roller derby bouncer. http://punkwalrus.net

  • 0 Posts
  • 56 Comments
Joined 1 year ago
cake
Cake day: June 22nd, 2023

help-circle
  • “Decline.”

    Working in IT, I have learned that a lot of meetings are by people who gain “respect and notoriety” by having large meetings. It doesn’t matter who shows up, it’s the number, that makes them seem popular. “Get the engineers in here, this is serious business!” You begin to learn which PMs do this, and can respond (or not) accordingly. If they ping you “where are you?” you can say, “I am in an [client] audit call. I cannot leave this call while the audit is taking place.” Or whatever your industry equivalent is. YMMV, some toxic environments I have been in, this was not possible.

    I remember one PM was frozen in indecision. I had to tell him, “I can fix the problem, or having a meeting about it. Pick one.”

    “Well, both–”

    “No. I can fix the problem, or having a meeting about it. Pick one or the other.”

    “I need you in this meeting!”

    “When we explain to the customer that the fix was delayed by an hour, I can use YOUR name, as having a meeting about it instead of fixing it, correct?”

    “The meeting is to be about fixing it!”

    “No. I can fix the problem, or having a meeting about it. Pick one or the other.”

    “… we can have the meeting in your office, then.”

    Eventually, my boss shooed him away.


  • Probably HR (or the NCS equivalent) never told the right people. I am not saying this is actually what happened, but a lot of IT bemoan the fact they are never told some rando employee was fired because HR neglects to inform them. Sometimes it takes months to discover, and even with a 90 day password/login lockout, some halfway decent admin could get around this by secretly building a back door, and using the messed up communication and politics between departments to hide this. Even in the 1990s, I saw people put in “time bombs” in their code that “if such and such is not updated in 6 months, run destructo-script A.”

    But imagine someone like Kandula Nagaraju here. Worked in QA, probably did a great jobs with some skills, but had the personality of swallowing broken glass. He was terminated in October 2022 due to “poor work performance,” which could mean anything. “Not a team player.” Or maybe he really was an idiot: I mean, a smart person would have a conniption, but get employed elsewhere and then slam his former company at parties. “Those NCS folks didn’t know what they had with me!” But this guy was probably someone with some anger management issues, probably a jerk, and possibly stupid. He might have had revenge fantasies, and set up a small virtual server posing as a backup code mirror. But outside the audits, it allowed ssh from the outside, and hid it through a knockd daemon. Or maybe only launched ssh at certain hours before shutting it down again. Silently working away in a sea of virtual servers with little to no updated documentation. He gets in, has internal access, and runs a script with admin credentials because they don’t rotate their AWS keys/secrets quickly enough. Or didn’t even know he was let go.

    After Kandula’s contract was terminated and he arrived back in India, he used his laptop to gain unauthorised access to the system using the administrator login credentials. He did so on six occasions between Jan 6 and Jan 17, 2023.

    That’s embarrassing to the company. Not only did he get in, but SIX TIMES after he was let go. he probably knew what order to run the delete commands (like, say, an aws “terminate-instances” cli command from a primary node), and did so one by one, probably during hours with the least amount of supervision, where the first few alerts would take hours to get someone in the monitoring chain to wake an admin. Given his last day was in November, and he got back in January, the admins probably thought their 90 access credential rotation was “good enough,” but he got in on his 80th day or whatever.

    I know this because I have had to do triage when a former contractor did this to a company I worked for. But instead of wiping out instances, he opened a new set of cloud accounts from the master account, put them in an unmonitored region (in this case, Asia), and spun up thousands of instances to run bitcoin mining. Only because AWS notified us of “unusual traffic” were we made aware at all, and this guy knew his shit and covered his tracks very well. He did it at a speed that could have only been automated. Thankfully, AWS did not charge us the seven figure amount that this activity amassed in just three days.


    1. Things like CNC machines and proprietary interfaces to TOL equipment, like bus fare systems, message boards, etc.
    2. Don’t connect them to the Internet (most can’t, anyway, but some systems use a run-of-the-mill PC, so…)
    3. Don’t install anything on them that wasn’t supposed to be installed, even wallpaper as this could fuck up the resolution of a small 240 x 180 screen


  • I was burned afoul by a former admin who, instead of diagnosing why a mail service was failing, labeled a script as a /etc/cron.d file entry as “…” (three dots) which, unless you were careful, you’d never notice in an "ls " listing casually. The cron job ran a script with a similar name which he ran once every 5 minutes. It would launch the mail service, but simultaneous services were not allowed to run on the same box, so if it was running, nothing would happen, although this later explained hundreds of “[program] service is already running” errors in our logs. It was every 5 minutes because our solarwinds check would only notice if the service had been down for 5 minutes. The reason why the service was crashing was later fixed in a patch, but nobody knew about this little “helper” script for years.

    Until one day, we had a service failover from primary to backup. Normally, we had two mail servers servers behind a load balancer. It would serve only the IP that was reporting as up. Before, we manually disabled the other network port, but this time, that step was forgotten, so BOTH IPs were listening. We shut down the primary mail service, but after 5 minutes, it came back up. The mail software would sync all the mail from one server to the other (like primary to backup, or reversed, but one way only). With both up, the load balancer just sent traffic to a random one.

    So now, both IPs received and sent mail, along with web interface users could use. But now, with mail going to both, it created mass confusion, and the mailbox sync was copying from backup to primary. Mail would appear and disappear randomly, and if it disappeared, it was because backup was syncing to primary. It was slow, and the first people to notice were the scant IMAP customers over the next several days. Those customers were always complaining because they had old and cranky systems, and our weekend customer service just told them to wait until Monday. But then more and more POP3 customers started to notice, and after 5 days had passed, we figured out what had happened. And we only did Netbackups every week, so now thousands of legitimate emails were lost for good over 3000 customers. A lot of them were lawyers.

    Oof.


  • Having bridged both worlds, here’s how they are viewed as described by a few people that stuck with me all these years.

    The first one I “go to thought” was more than one person is “awkward.” Some even describing them “out of step, socially.” Imagine a clock that is running fast or slow, but you have mentally compensated because generally, you can adapt depending on other clues. But they are always off, and you might have to warn others ahead of time.

    Another comment was how autistic personalities are in that “uncanny valley of behavior” where people notice something is off, and it can be frightening but they are not sure why. Since autism is a spectrum of behaviors, which approach depends subjectively on the viewer. Kids, for example, can target autistic kids, and because they are developing socially, will group in “us” and “not us.” Autistic kids are “not us,” and the target of bullying. A lot of teachers know autistic kids just by how they are treated by others. “You’re too weird,” was something a lot of kids might say with developing language skills. The may not know WHY they hate a certain kid, but know that they DO. And “something is wrong.”

    Personally, I see autism as some kind of evolutionary response to a civilization that is growing faster that humans can compensate. In order to get actual insight, one has to be “out of step,” lest they just end up trapped in the normal static of everyday compulsion. Like any other evolutionary advance, nature is “trying out” various things. Most will lead to dead ends. A few will adapt in other ways, and some will flourish in a new niche with new types of diversity. I have no proof of this, but I think it’s more than “well, we define autism differently now.” Yes, there were always people who were “touched by fae” or whatever convention was explained back in the day, but something has really changed. I personally think this and gender fluidity is a positive sign of things to come.



  • Not just LinkedIn profiles: there was a case out here near DC a while ago where a well known company leased out their function space for training meetings. Using a compromised company account, a set of scammers set up some fake recruitment profiles, leased out the meeting space for “software training,” and did some “mass hiring” where 30 individuals had their credentials scanned and duplicated. The effect was someone from the recruiting company was contacting you, you had a face-to-face where you got offered an in-person, you showed up to their offices, and got a “job offer pending a background check,” with a date of hire in official-looking emails. You sent in your SSN, copies of your passport and driver’s licence, and after a few weeks, they tell you to show up for orientation. Only, the day these people showed up, the company was confused and had never heard of you. The people you supposedly spoke to had never heard of you. And your identity was stolen, and huge loans and charges started showing up in your credit report.

    Yikes.


  • I hate to be honest, but I used Amazon Prime a lot because:

    1. I cannot drive. Thus, getting to the store is difficult.
    2. I must bring in 3-4 items a week, so yeah, I save on shipping.
    3. Auto-subscriptions save a little.
    4. I have priced a lot of stuff over the years, and while Amazon is not always the best, the convenience is impressive.
    5. They have, multiple times, been incredibly helpful with customer service. Like above and beyond.
    6. COVID and nobody masks around here. I have an autoimmune condition, so it’s important that I not leave unless it’s a medical appointment or similar need.
    7. They just have stuff I can’t find anywhere. Yes, as some have said, caveat emptor, but that’s true for all the stores.

    I also save a shit ton of money. When I used to browse Walmart or Target, I used to buy a lot of shit I didn’t need. I don’t get as distracted with focused buying. I also order from Aliexpress if I can wait 30 days, and I have only been ripped off three times in several years, for a total of maybe $35.

    I’m not saying my way is better, and certainly not if it’s better for you, but it’s been a godsend to the house-bound.


  • I had an issue with a UK supplier shipping to me in the US because UK mail is owned by DHL. Two shipments made it to the US, made it to New York, made it through customs and then… “Attempted Delivery, Kein Standort ausgewiesen, US.” Never seen again.

    DHL kept trying to convince me and the seller that “Your German address is refusing shipment.”

    “Last delivery hop was Flushing, NY.”

    “No, last delivery hop was in Kein Standort ausgewiesen. That company refused shipment, it went back to the UK.”

    “‘Kein Standort ausgewiesen’ is German for ‘No location reported,’ dumbass. Where is my package?”

    “… We don’t know. Germany somewhere. Call them.”

    Thankfully my vendor has refunded my money. Sadly, he no longer will ship to the US because this keeps happening to his customers. :(











  • I have my mom’s old Smith Corona electric typewriter, possibly from the 1960s, but I haven’t researched it deeply yet. It comes with a hard suitcase with a 3-digit combination lock, and a novelty dust cover that says “Typewriter!” in a bold font. My mom died in 1987, and I just took it. Last used in 1998 when my mother in law borrowed it, and it caused a fight because she died unexpectedly and her estate executor wanted to sell it, not believing it to be mine. He was under the impression it was an antique he could get good money for, but I stole it back. Last appraisal was mid 2000s, and in new working condition, I could MAYBE sell it for $30. But it’s one of the few scant possessions I have of my moms, so I keep it.