• 0 Posts
  • 35 Comments
Joined 1 year ago
cake
Cake day: July 1st, 2023

help-circle
  • jemikwa@lemmy.blahaj.zonetoAsklemmy@lemmy.mlCrowdstrike Cockup
    link
    fedilink
    English
    arrow-up
    21
    ·
    edit-2
    16 days ago

    I want to clarify something that you hinted at in your post but I’ve seen in other posts too. This isn’t a cloud failure or remotely related to it, but a facet of a company’s security software suite causing crippling issues.

    I apologize ahead of time, when I started typing this I didn’t think it would be this long. This is pretty important to me and I feel like this can help clarify a lot of misinformation about how IT and software works in an enterprise.

    Crowdstrike is an EDR, or Endpoint Detection and Response software. Basically a fancy antivirus that isn’t file signature based but action monitoring based. Like all AVs, it receives regular definition updates around once an hour to anticipate possible threat actors using zero-day exploits. This is the part that failed, the hourly update channel pushed a bad update. Some computers escaped unscathed because they checked in either right before the bad update was pushed or right after it was pulled.
    Another facet of AVs is how they work depends on monitoring every part of a computer. This requires specific drivers to integrate into the core OS, which were updated to accompany the definition update. Anything that integrates that closely can cause issues if it isn’t made right.

    Before this incident, Crowdstrike was regarded as the best in its class of EDR software. This isn’t something companies would swap to willy nilly just because they feel like it. The scale of implementing a new security software for all systems in an org is a huge undertaking, one that I’ve been a part of several times. It sucks to not only rip out the old software but also integrate the new software and make sure it doesn’t mess up other parts of the server. Basically companies wouldn’t use CS unless they are too lazy to change away, or they think it’s really that good.
    EDR software plays a huge role in securing a company’s systems. Companies need this tech for security but also because they risk failing critical audits or can’t qualify for cybersecurity insurance. Any similar software could have issues - Cylance, Palo Alto Cortex XDR, Trend Micro are all very strong players in the field too and are just as prone to having issues.
    And it’s not just the EDR software that could cause issues, but lots of other tech. Anything that does regular definition or software updating can’t or shouldn’t be monitored because of the frequency or urgency of each update would be impractical to filter by an enterprise. Firewalls come to mind, but there could be a lot of systems at risk of failing due to a bad update. Of course, it should fall on the enterprise to provide the manpower to do this, but this is highly unlikely when most IT teams are already skeleton crews and subject to heavy budget cuts.

    So with all that, you might ask “how is this mitigated?” It’s a very good question. The most obvious solution “don’t use one software on all systems” is more complicated and expensive than you think. Imagine bug testing your software for two separate web servers - one uses Crowdstrike, Tenable, Apache, Python, and Node.js, and the other uses TrendMicro, Qualys, nginx, PHP, and Rust. The amount of time wasted on replicating behavior would be astronomical, not to mention unlikely to have feature parity. At what point do you define the line of “having redundant tech stacks” to be too burdensome? That’s the risk a lot of companies take on when choosing a vendor.
    On a more relatable scale, imagine you work at a company and desktop email clients are the most important part of your job. One half of the team uses Microsoft Office and the other half uses Mozilla Thunderbird. Neither software has feature parity with the other, and one will naturally be superior over the other. But because the org is afraid of everyone getting locked out of emails, you happen to be using “the bad” software. Not a very good experience for your team, even if it is overall more reliable.

    A better solution is improved BCDR (business continuity disaster recovery) processes, most notably backup and restore testing. For my personal role in this incident, I only have a handful of servers affected by this crisis for which I am very grateful. I was able to recover 6 out of 7 affected servers, but the last is proving to be a little trickier. The best solution would be to restore this server to a former state and continue on, but in my haste to set up the env, I neglected to configure snapshotting and other backup processes. It won’t be the end of the world to recreate this server, but this could be even worse if this server had any critical software on it. I do plan on using this event to review all systems I have a hand in to assess redundancy in each facet - cloud, region, network, instance, and software level.
    Laptops are trickier to fix because of how distributed they are by nature. However, they can still be improved by having regular backups taken of a user’s files and testing that Bitlocker is properly configured and curated.

    All that said, I’m far from an expert on this, just an IT admin trying to do what I can with company resources. Here’s hoping Crowdstrike and other companies greatly improve their QA testing, and IT departments finally get the tooling approved to improve their backup and recovery strategies.







  • I have been addicted to making ramen eggs (ajitama) for lunches lately. I eat one or two eggs over rice with some furikake or toasted seaweed and that’s all I need to power through the day. You could pair the dish with more veggies or a miso soup if you’re feeling fancy. The nice part is making half a dozen eggs squares me away for the week, so I hardly have to think about what to do.

    Another dish I like is Korean steamed eggs (gyeranjjim). It takes not even 10 mins to cook on the stove. Making rice takes longer, and you can make a lot of rice to reheat later in the week. I would cook the eggs fresh each day though , I’m not sure how reheating them would go. The broth that goes with the eggs keeps me fuller than I ever expect.

    Baba ghanoush is so tasty when you make it yourself. This requires more effort up front to roast the eggplant, but the dip is good all week. I eat it with carrots, cucumber, cauliflower, and some pita chips.

    Regular tuna salad or this chickpea “tuna” salad is always easy to whip up. I always have celery, pickles, and bread on hand so if I’m feeling up for it, I crack open a can of tuna or chickpeas for an easy lunch.





  • FFXIV handles it “okay”, in that you get a large portion of glam in game and the cash shop stuff is largely excess. There are a few cases where it would have been better to have the reward in game, but for the most part I feel like I can play the game without needing to buy anything.
    The impression the community gets is the cash shop is a begrudging feature that SE higher ups mandate to keep cash flow going (because XIV is funding most of SE’s other projects)



  • Thanks for your reply! Those are all really good points and I range from agreeing completely or understanding wholeheartedly why you feel the way you do about the issue.

    Japan’s approach to romance tropes, especially with school age people, is problematic at best. Maybe as a teen I would have fallen head over heels for it and loved most of it (except Kawakami’s plot, yuck), but I definitely had to mentally distance myself from it because it felt weird to even interact with as an adult. At least most of them progress as friends and only turn into relationships at the end (most… bleh). I don’t even remember which romance I might have picked for the final companion step, maybe I skipped the commitment altogether. A part of me fully expects P6 to do something similar and I’m less enthusiastic about that part of the game as a result.

    I never thought of the UI being difficult to read and concentrate with, but I totally get it now that you’ve mentioned it. There were definitely times I had a harder time navigating menus because of how much the background moved during transitions. The design itself was committed in every aspect, but the devs could have done more to normalize some of the text or add options to tweak the animations to be less dizzying while still being visually diverse.

    I need to try P3 and P4 still, they are in my backlog and I never feel the initiative to start one of them. Your remark about P5 dumbing down some of the features of those games is a little inspiring because I did like the combat and options from P5. If it’s even better in past games, I may like those even more.



  • I used to avoid turn based for the same sentiment, but have found some really compelling games that change the formula that have changed my mind. Not every game will be a winner, but there are still some good ones out there.

    Strict turn based used to always seem simple to me, and I don’t find it appealing all the time. Pokemon has unit variety, but the strictness of each turn can get really stale.

    Games that improve turn based combat are my preference in this category. Persona 5 changes the flow of combat depending on how each unit/character performs and exploits type weaknesses (chain/group attacks). It also takes Pokemon typing and unit diversity and makes a cool fusion/inheritance system out of it.
    Older Final Fantasy games with Active Time Battle also scratch this itch where the timing of using skills and specific character order still somewhat matters, you don’t always mash A and spam abilities. FF also does really well with unit customization - materia, GFs, Sphere Grid, etc. mean consecutive playthroughs won’t always feel the same.
    Chrono Trigger takes ATB and adds geometry in a physical dimension to attacks which is really unique, but still feels turn based at its core.

    Like you said, full tactical games are fine because the quantity of units or structure of the arena make the turn based mode interesting. BG3/Divinity, Fire Emblem, Triangle Strategy/FF Tactics, and Gloomhaven fall in this category and I love games like these.

    I realize now I kinda hit the points in the article, oops. Sorry if this was repetitive lol



  • Short answer, likely yes. It’s not definitive, you could still slip by after sending enough mail, but you are also very likely to get whacked because that VPS IP doesn’t have an email sending reputation.

    Longer answer, email gateways like Google, Microsoft, and Proofpoint don’t really care who owns what IP. Well, they might, but they’re more concerned about the sending habits of an IP. While you might send good mail from that IP, there’s no reputation for it, so you could be whacked for having a neutral reputation (the ol’ credit score dilemma but for email).
    In order to have a good reputation, you have to send a large volume of messages very gradually over several weeks to “warm” your IP as a reputable sender. I went over this slightly more in detail in another reply, but this article is pretty concise on how an enterprise accomplishes this with a dedicated IP at a provider like SendGrid: https://docs.sendgrid.com/ui/sending-email/warming-up-an-ip-address


  • It’s about sample size. Mail gateways won’t designate an IP as a reputable sending IP until it assesses a large volume of mail sent over a long period of time. You can’t send the quantity it wants all at once or even in a short window because then you’ll be designated as a spammer. So you start small with a few a day and gradually ramp up sending over multiple weeks or months to eventually send several thousands of messages in that period.

    Spammers and malicious actors too often spin up new IPs for sending mail, so gateway patterns already implicitly mandate that email should come from IPs it’s already judged reputable.

    You as an individual can’t reasonably warm your own IP. This is why services like Amazon SES or Sendgrid exist because they have huge IP pools that are ready to go. Plus, those services are very concerned with reputation and have bounce/complaint metrics defined to warn customers that abuse or poorly configure their sending habits.

    This next example is what I’m most familiar with, but I’m sure there are other services like this. If you’re a big enterprise and want your own dedicated sending IP because you’re concerned about using a shared pool, you could use something like Amazon Pinpoint which allocate IPs for your org to use in SES, but they have to be warmed before you switch your production workloads over to it full-time. It automates some of the gradual-ness of warming so you use a mix of SES plus your Pinpoint IPs to keep mail flowing for your product.

    It looks like Sendgrid also does dedicated IP warming guard rails too. This article is pretty decent for understanding how it works - https://docs.sendgrid.com/ui/sending-email/warming-up-an-ip-address The per-day warming limits give you an idea of what scale this kind of process is used for.


  • Definitely listen to this. IP Warming is a very real problem and you have to send thousands of messages at a very gradual rate for most email gateways to 1) mark you as a proper email sender, and 2) classify you as a reputable one that isn’t sending spam. Using a public/private cloud IP isn’t enough, it should be a service already used for mail sending.

    If you self host sending email and ignore using a service for outbound, make sure it isn’t at home. ISPs often block SMTP traffic to keep people from spamming others from their home. A lot of IP blocklists also auto block home IPs so you may not ever get your messages delivered.

    Make sure to set up SPF/DKIM/DMARC. At the very least SPF, DKIM if the platform supports it, and ideally all three or SPF+DMARC. It’s not that hard to configure if you do it as you go instead of years down the line after you have a dozen services sending mail as your domain.


  • Turo for sharing cars. It’s car rentals but weird since you’re borrowing other people’s cars.

    Kyte is in a similar vein for on demand cars via app, but it functions similar to a traditional car rental in every other way. Though they’re a baby company so they might not be a good Uber comparison yet.
    My personal experience has been pretty good and the prices are very competitive, but we’ll see how they survive when the VC money runs out.