After two years of automating our SOC, we decreased the alerts requiring human interaction by 40% and the time spent on alerts by 70%.

I will analyze 5 pitfalls we experienced internally or with customers to help you maximize your investments.

1. Not using other products at their fullest first

2. Starting with SecOps automation for the wrong reasons or assumptions

3. Not having the right people or focus

4. Focusing on containment orchestrations too soon

5. Not knowing how to measure success

Over the past 8 years, I’ve helped cybersecurity service companies boost their revenue and profits while enhancing security for 60+ organizations worldwide.

I share business strategy insights on SecOps and cybersecurity that will take your company to new heights. Subscribe for free to get my latest ideas first!

Sign me up!

My Automation Journey

I work for a company that offers 24x7 SOC as a Service and Incident Response.

When talking about security automation, it’s impossible not to mention SOAR. Whether open source or proprietary, both involve significant efforts and challenges.

In 2022, we decided to add a SOAR to our stack. We had automations based on Python and containers. Maintaining them was becoming complicated.

We were handling around 600-800 alerts per month from different customers. None of our analysts experienced 'alert fatigue' or burnout. Still, we felt there was room for improvement in our processes.

We had two foundational needs that aimed to move the company to the next level:

• Skyrocket efficiency: We were transitioning our customers to a fixed-price model for our SOC service. Costs can spiral out of control when managing alerts and incidents in a 24x7 operation. We needed a new way of working and a stack to support it.

• Increase technical flexibility: Our previous technology stack wasn’t the best at working with cloud-native applications and APIs.

We started our quest to find the perfect SOAR. After an extensive POC with three products, we chose a winner and implemented it in December 2022.

And we suffered. We wasted time. Until we got better.

Nowadays, the SOAR is the backbone of our operation.

Let’s analyze the pitfalls and key lessons that we learned along the way.

1. Not Using Other Products at Their Fullest First

The world is full of tales of companies that bought technology and didn’t deploy it. I'm sure you experienced something similar at least once in your career.

Organizations of all sizes underestimate the technologies they have running in their environment. After a basic implementation, they become a story of the past, and we start looking for the next shiny thing.

You prefer to invest in other tools rather than in expertise on the existing ones.

This trap is especially damaging for one single reason:

The value return of automations depends on your knowledge and capabilities of other technologies.

But there’s a more pressing issue. A mirage around SOAR that makes companies believe that is the key to fixing their security issues.

“You are able to contain users and workstations.”

“SOAR removes the pain of phishing attacks.”

SOAR is not the problem, but the misinterpretation of its promise.

You don’t need automation for your firewall to block traffic. Nor for your WAF to apply certain rate limiting, or for your EDR to block a malicious file.

If you bought a SOAR for that, I’m sorry to tell you, you’ve wasted your budget.

You don’t need anything new to do most of the containment and prevention actions you have in mind. You need to spend more time increasing the potential of your existing solutions.

You will be surprised how much you can do with your current M365 licenses, anti-phishing, EDR, Next-Gen firewall, and cloud application.

Focus on the features that are not enabled. Once you hit a roadblock, before thinking that a SOAR will save you, make sure that it’s not a product or license limitation.

You must maximize containment within a product before adding an extra layer.

2. Starting With SecOps Automation for the Wrong Reasons or Assumptions

You are using your existing technologies at their best, but there’s still a high chance that you are doing SecOps automation for the wrong reasons.

Consider a scenario: you work for Company X, a fintech company looking into SOAR because they believe it will help you analyze phishing cases faster.

But… are you actually having this problem? How many cases do you analyze daily? How many hours are you losing to this issue? Is it really a bottleneck for your team?

And… do you understand the implications of analyzing phishing cases with external tools? You might need additional, and potentially paid, threat intelligence.

Let’s take another example: Company Y, an MSSP offering 24x7 SOC services, wants to automate its level 1 operations.

First, what does that even mean? Every organization I’ve met has a different definition for level 1 and its corresponding responsibilities.

How much are you spending on your first line? What processes do you want to automate? Is it even possible? Do you have the required access with your customers to trigger certain automations?

Or… do you just want to resell the tool because your customers are asking for it? Be honest about what you expect.

Yes, SecOps automation can have amazing returns, but only if you identify your true constraints and understand the potential effort and time required to address them.

The best advice I can give you is to identify your top three bottlenecks and do a proof of concept around them before investing in new technology or profiles.

For us, it was only two, and it was hard enough.

The journey takes time, lots of iterations, and money.

3. Not Having the Right People or Focus

This was our first challenge.

We'd been doing automations for two years before formalizing our SOAR investment and realigning our SOC strategy.

We had people who knew how to code and make APIs dance.

Still, progress was slow during the first months. After various discussions with my team, we found that there was a lack of focus.

We automated tasks on an ad-hoc basis whenever necessary. These team members also handled alerts, consultancy and performed incident response.

Things improved when we decided to formalize the role of Automation Engineering. We allowed team members to focus and we set clear accountability and goals.

This is the first mistake I see in different teams. I get it, you don't have a big team and try to concentrate capacities as much as possible. But it doesn’t work. Security automation is a complex domain that requires plenty of focus.

Don't just take my word for it. Google's course on Modern Security Operations (2024) states the significance of this role in a SOC team.

Allow at least one team member to focus on automation engineering.

The second mistake I see is the underestimation of knowledge when automating processes.

A SOC team may be proficient in SIEM, incident response, EDR and detection. But it's not going to be enough.

SecOps automation requires a different set of skills and mentality. An almost impossible conjunction of knowledge in different domains.

A solid security automation initiative requires harmony between four key domains:

• Blue team knowledge: Most SOC teams already have this. You need proficiency in detections, alert handling, and the associated workload challenges. After all, what are you planning to automate?

• Automation knowledge: You need people who are able to code, use git, and be proficient with the SOAR itself. Especially, they need to know how to think automations. What if you push a playbook that completely breaks your organization?

• End-Product knowledge: It's not only about knowing your SIEM and SOAR. You need deep product knowledge of other systems within your company: Ticketing systems, EDR, firewalls, cloud applications, and so on.

• End-Product API knowledge: Often overlooked. Being proficient with a technology doesn't necessarily mean understanding its API. Deploying a product and working with its API require different skill sets. Failing to understand End-Product APIs can lead to limitations and production impacts. We'll explore this further in the next chapter.

It's nearly impossible to have all this knowledge in one individual. Still, various teams try to make it happen and fail.

The challenge is that it's hard to distribute these domains among different collaborators. You need a certain background in more than one domain if you want to be successful.

If you're starting, stack blue team and automation knowledge into one collaborator. Your engineer will need to work with other product specialists to ensure the validity and safety of your ideas.

4. Focusing On Containment Orchestrations Too Soon

This was our second challenge.

Orchestration refers to the coordination and management of multiple automated tasks or processes working together to achieve a larger goal.

We wanted what every vendor promises:

The End-To-End orchestration containment experience. To detect a threat and automatically block users in Active Directory and in every other application. All while pushing rules firewalls to block traffic from compromised users. And of course, trigger an automatic threat hunt to find more indicators.

From the MSSP perspective, the premise is beautiful. Containment orchestration can offload your SOC tremendously.

For instance, even if your SOC operates 24/7, your customers might not. No one wants to receive a call at 3 AM. That’s why one of our main KPIs is the percentage of alerts that involve contacting the customer. With End-To-End orchestration, you could take automatic containment to mitigate risks outside business hours, so you don’t wake your customers in the middle of the night.

Our customers wanted it, and so did we. We assigned part of our team to work on these playbooks, but we started to hit roadblocks.

Buy-In From Stakeholders

You are removing control and adding a point of failure.

Everyone agreed, but in the end, customers didn’t want to commit to its implications. Even with guardrails and controls, giving admin rights to an external automation that you don’t control isn’t easy to digest.

End-Product API Limitations & Knowledge

First, none of the promised built-in containment playbooks worked out of the box. We ended up creating our own from scratch.

We encountered various API limitations from other products. For example, in Check Point firewalls, you need to push policies to apply rulebase changes. This can increase CPU usage of the appliance or apply unwanted changes.

Your SOAR will use the APIs of a target product. You might know how to use your stack, but you can’t expect your team to know the API of each product and its implications.

Increased Complexity

The complexity factor plays a big role. As soon as you need to stitch two technologies together, complexity skyrockets. More than 2... well...

Let’s say that you make it work. Who’s going to check for changes in the API? Or the product modifications that might affect your automations? Your integrations might not update with the same frequency.

And what if, by overlooking this, you start affecting core operations?

That’s why maximizing your current technologies and End-Product knowledge is so important.

Automated containment is a logical evolution. It required a high organizational and SecOps maturity to make it work.

Instead, starting with what I call facilitation orchestrations is a better approach.

Facilitation orchestrations improve your operational efficiency with almost no risk of affecting production.

These orchestrations also use external APIs, though not for containment purposes.

We decided to switch our focus to using SOAR to streamline the process and make the lives of our analysts easier.

Our first eureka moment with the SOAR was when we were able to enforce processes. An alert is always handled in the same way with a consistent communication template. This approach allowed us to decrease the time spent on alerts by more than 70% in one year.

The second big leap came. We were able to close alerts without human intervention based on enrichment.

Finally, we overhauled our SOC analytics by applying context in our ticketing system.

Your first months should focus on facilitation. It's not an easy task. You will need to tinker with your ticketing system, threat intelligence tools, SOAR, and so on.

Leave containment as much as possible to the End-Product.

5. Not Knowing How to Measure Success

Justifying technology investment and defending your hard-earned budget is a complex undertaking.

Most organizations are not metric-driven. Some want to be but don't have the necessary systems in place to track what matters.

Once you start, It's easy to get lost in the endless KPIs of cybersecurity. You might convince your stakeholders of effectiveness with a pie chart of detection.

But sooner or later, management will challenge your return on investments.

You need to define between 2 and 4 measurements to assess the success of your initiatives.

In my experience, there are two key performance indicators to achieve this:

• Average time per alert (Goal: Decrease): This is a great measurement to assess alert handling optimization in your SOC. It's a basic metric for every SOC. If you don't have it, don't start with SecOps automation.

• Alerts without human intervention (Goal: Increase): The golden metric. It encompasses alerts that were filtered, auto-closed, or handled without your analyst even seeing them. This metric will help you assess your progress and identify areas where you need to focus in terms of processes.

Less is better. You will see that getting these metrics right is hard enough. Don't invest further if you don't have the means to gather these measurements.

After getting these metrics you will need to establish safety indicators. Be aware that these are more challenging to measure. Some examples are:

• Number of failed playbook executions.

• Number of missed true positives due to automations.

• Production incidents due to automations.

Define which metrics represent operational efficiency improvements in your organization. Then, ensure you have the means to track them.

Conclusions

The described pitfalls stem from my experience working with different SecOps teams.

Security automation is a mandatory quest for every SOC. Sadly, Everyone talks about the benefits and not the downfalls.

I encourage you to start taking the first steps towards automation. The sole effort of setting the foundations right will already benefit your organization.

Wasting some effort will be inevitable; after all, you must improve through iterations. I hope that my ideas allow you to find a strategy that maximizes returns.

Feel free to reach out if you want to discuss further. I'm always up for meeting interesting people with cool ideas.

My Check Point days are in the past, at least for the moment.

But…

This write-up and research hold a special place in my heart. It played a pivotal role in my career:

It was featured in Tim Hall’s book, “Check Point Firewall Performance Optimization - Third Edition”

It allowed me to present at my first international conference on one of the main stages: Check Point CPX 2020 - New Orleans

And so on.

Keep reading below.

Hello everyone,

I'm looking to share my experience and concerns regarding the current state of SSL Inspection in general, and with Check Point in particular. I'm also interested in hearing your approaches to this matter.

Important: This post focuses strictly on outbound SSL Inspection; my experience with inbound is limited to Check Point.

I'll start with a vendor-agnostic statement: One of the things that bothers me most about this subject is that many people claim to have this technology implemented in their organizations without any issues. However, upon meeting those organizations, I found one of the following:

1. They have at most 10 users.

2. Their SSL Inspection technology is shady: SSL Inspection is enabled, but there is a working and HUGE fail-open at the bottom of the engine. If something may cause issues, the connection is accepted, the log shows that the packet was "inspected," and everyone is happy.

3. They think they have SSL Inspection but don’t actually have it enabled; instead, they have a feature similar to categorizing HTTPS sites.

My Personal Experience

We all know the importance of inspecting HTTPS traffic in our network for visibility and security. However, at times, I feel like this way of thinking is my InfoSec persona speaking—one that only gives "woo woo" advice to the business regarding security and doesn’t even know how to install antivirus software. Why? Because properly implementing this solution is just painful. I also know that most customers don't want to dive into this matter and prefer to rely on their endpoint solutions for this layer of security. Even if it's not the same, I respect that.

Among our customers, we have one that is particularly tech-savvy with 900+ users. They like to enable all features in their NGFW, and one of the requirements was to enable full outgoing SSL inspection in the Check Point firewall. We started with this customer years ago with R77.30 in gateways, so you can imagine that I've been through all kinds of fun experiences with SSL Inspection.

We have fully tested SSL Inspection in the following versions: R77.30, R80.10, and R80.30.

During this journey, we went through a lot of information. There are many awesome posts here in Check Mates, SKs, SRs with the TAC—you name it.

Issues We Faced

Here is a summarized list of issues we encountered:

• Heavy performance issues in R77.30 (fixed in R80.10+)

• Many pages fail to load or load intermittently.

• Some pages work, but specific sections (like login pages) fail.

• Sophos Antivirus solution doesn't work for installation or updates: There are some posts about this issue.

• AWS Connectors failing (solved in R80.30).

The Issue with Check Point and Outbound SSL Inspection

I really like Check Point firewalls, but sometimes I feel that they take control away from the administrator. One of the first things we did was disable all options related to dropping connections that don't follow the RFC line by line, and allowing connections to non-trusted certificates. We tried everything, yet we still faced many issues.

Bypassing the connection? Good luck with that. Check Point firewalls always inspect the first packet, and because of this, many connections fail.

Probe bypass to mitigate the previous issue? Sure, but be prepared to face other issues due to SNI verification.

Fail-open in probe bypass? This was a huge surprise after it was changed in Take 189, but we still had a lot of issues even after enabling this flag.

WSTLSD debug? Many times, but good luck not taking down the firewall with it and be prepared to wait a long time for the TAC to inspect it. It's not their fault; it's just really hard to troubleshoot these issues.

The only way we found to properly bypass connections was to exclude them COMPLETELY from the SSL policy. For example, let's say you have two network segments and you only want to inspect traffic in one of them:

What Most People Do

In this example, all traffic from 10.0.0.0 will be inspected. However, you will probably have some issues in the 192.168.0.0 network as well since the bypass action enforces inspecting the first packet of the SSL handshake.

The Only Way to Do Nothing with the Connections

In our research, we found that the only way to properly bypass a connection was to exclude it completely from the policy. Obviously, this approach is not scalable and somewhat utopian in a big network.

The Most Stable Scenario We Reached

We reached a state of stability in R80.10 with the JHF prior to 189 by enabling the following flags and features:

• appi_urlf_ssl_cn_allow_not_rfc_ssl_protocols=1 (Don’t know where I got this; also, there is no documentation about it)

• enhanced_ssl_inspection=1 (Probe bypass)

• bypass_on_enhanced_ssl_inspection=1 (Fail-open probe bypass)

• Almost all features that drop packets turned off in SSL Inspection.

• HTTPS categorization turned on.

Sophos antivirus worked fine, and issues with web pages were minimized.

The Journey to R80.30

We decided to migrate one of the cluster members to R80.30 to test the new SSL Inspection engine and solve some issues we had with UserCheck. After deployment, we had issues with Proxy ARP:

https://community.checkpoint.com/t5/Enterprise-Appliances-and-Gaia/Proxy-ARP-after-upgrade-to-R80-30...

Some inspection settings started to cause issues in R80.30 that didn't previously.

We sorted them all, and the first impression was great:

• AWS Connectors worked flawlessly without enabling any of the previously stated kernel flags.

• Sophos Antivirus could get updated without enabling any of the previously stated kernel flags.

• All the services detailed in the preliminary testing document worked great.

The next day, a waterfall of user complaints started to appear:

• Sophos Antivirus could not be installed: Updates worked fine, but installation failed. After looking into the logs, no traffic was dropped (logs and output from fw ctl zdebug). The only log was a Detect regarding untrusted certificates, which we configured to accept in the SSL settings. We tried the flags, setting up an FQDN object just for *.sophos.com in the SSL Inspection policy, and it still failed.

• The main billing service of the company stopped working. Again, no logs or possible leads as to why. We even looked at PCAPs, and everything seemed fine from the firewall’s perspective. As soon as we routed this traffic through pfSense, everything worked flawlessly.

• Another invoice service stopped working: Again, no leads whatsoever. After we routed this traffic through pfSense, everything started to work.

• Web pages that did not load properly or had some functions affected.

We tried everything: performed captures by turning off SecureXL, but even the bypass flags could not solve these issues in R80.30. We had these issues in R80.10, and after turning on the different flags, everything worked, but not in this new version.

At this point, there were many issues impacting production. We blocked one hole, and another 10 appeared. It was just impossible to properly troubleshoot each issue, so we had no other option but to revert to our most stable version.

Future Plans

There is no way to deploy SSL Inspection without issues; the problem is that these issues will probably heavily affect your production environment. There is no way you can test all your organization’s use cases, and there is no way to properly assure functionality in a lab environment.

Our main concern now is the remote possibility that we will have to stick with R80.10 for life. We know we will have to update sooner or later, which is why we are now implementing a parallel CHKP Frontier similar to our failback pfSense. This new gateway will have R80.30 with the same features. Think of it as a hybrid testing/production environment.

The main idea is to route certain subnets to the R80.30 gateway and study their behavior and troubleshoot without all the user complaints.

Concerns Regarding the Current State of SSL Inspection

• Check Point firewalls don't provide a proper solution to bypass desired SSL traffic, making it hard to deploy this solution in a large environment.

• Lab tests are not representative; the only way to test is in production.

• Troubleshooting is really difficult: Many times there are no leads, and everything seems fine on the firewall, forcing you to perform PCAPs on different parts of the network and debugging.

• Check Point’s current approach to SSL Inspection impacts the brand’s image: I hear it all the time—“I have a friend who inspects SSL traffic with YYY and has no issues.” Most people don’t know that it’s more of a technology issue in general regarding SSL/TLS rather than a Check Point fault. However, other vendors offer this functionality with failback mechanisms that work without the user knowing. It’s less secure, but at the end of the day, the main metric is functionality and not security in most cases.

Hope this post helps you implement this feature in a harmless way.

Link to the original post: Check Mates - Outbound SSL Inspection: A War Story