There are at least three different ways to ask "is this Microsoft 365 tenant configured securely?", and they don't agree with each other nearly as much as you'd hope. I've got a test tenant I use for poking at M365 tooling deliberately left close to Microsoft's defaults, no hardening pass applied and it seemed like a good excuse to actually run all three side by side instead of just trusting whichever one I happened to reach for first:
- Prowler - the open-source CSPM, which added an M365 provider a while back alongside its AWS/Azure/GCP checks.
- Microsoft Secure Score - the recommendation engine baked into the Microsoft 365 Defender portal.
- ScubaGear - CISA's tool for assessing a tenant against its own M365 Secure Configuration Baselines.
Since a test tenant like this one is basically Microsoft's out-of-the-box configuration with a handful of users dropped in, it's a decent stand-in for "what does a tenant look like before anyone's gone through and hardened it," which turns out to be exactly the gap each of these tools is trying to close, just from different angles.
Throughout this post I'll reference specific checks by name or ID (things like MS.AAD.3.1v1 or entra_identity_protection_sign_in_risk_enabled) without re-explaining what each one does inline. If you want to look any of them up directly instead of digging through each tool's source or docs, here's the generic, tenant-agnostic reference data behind this post: every Prowler check ID/title, every ScubaGear control ID/requirement, every Secure Score recommendation name, and the cross-tool mapping behind the "where they agree" sections below.
Getting ScubaGear running #
If you haven't used it before, ScubaGear is a PowerShell module, and setup is refreshingly small:
Install-Module -Name ScubaGear -Scope CurrentUser
Import-Module ScubaGear
Initialize-SCuBA
Initialize-SCuBA pulls down the PowerShell modules ScubaGear depends on (Microsoft Graph, Exchange Online Management, MicrosoftTeams, powershell-yaml) and installs Open Policy Agent, which is what actually evaluates the Rego policies behind each baseline control, into a .scubagear folder in your home directory.
It only runs on Windows PowerShell 5.1. That's a hard requirement tied to some of its underlying libraries, not a version they haven't gotten around to bumping yet, so there's no point fighting it into PowerShell 7.
From there, kicking off a full scan is one line:
Invoke-SCuBA -ProductNames * -M365Environment commercial
That pops interactive sign-in windows one at a time (Graph first, then Exchange Online, Teams, and SharePoint) and drops an HTML report plus raw CSV/JSON results into a timestamped folder. Worth noting: whatever account you sign in with needs enough read access to reach every product you're asking about. I didn't have Exchange/Security Reader on the account I used, and the run silently skipped the Exchange Online and Defender baselines rather than erroring, so if a product's missing from your output folder, that's the first thing to check.
The headline numbers #
Three tools, three different ways of counting, so these aren't directly comparable, but they're a useful gut-check on how out-of-the-box this tenant really was.
| Tool | Scope this run | Result |
|---|
| Prowler | 164 checks across Entra ID, SharePoint, admin center, Defender | 78 failing (1 critical, 33 high, 42 medium, 2 low) |
| ScubaGear | 67 CISA baseline controls across Entra ID, SharePoint, Teams, Power Platform | 29 failing, 12 warnings, 14 passing |
| Secure Score | 4 category exports (identity, apps/mail, data, devices) | 38 open, tenant-policy-level recommendations |
Roughly half of everything either tool checked was still sitting on a default. That's the expected shape for a tenant nobody's gone through with a checklist yet, which is sort of the point of running it this way.
One caveat before the numbers get more granular: Prowler evaluates several conditional access checks once per policy, not once per tenant. If you've got six conditional access policies that all forget to require token protection, that's six failing rows in Prowler's CSV for one underlying gap. Once you collapse duplicates, those 78 failing rows are actually 58 distinct checks. That's the number I used for everything below, so a "Prowler: 5" isn't secretly hiding a single check repeated five times.
Breaking it down by category #
Instead of "78 vs. 29 vs. 38," here's the same findings sorted into the themes each tool actually targets. The three tools don't share a check library, so this is a best-effort crosswalk rather than an exact match, but it gives a much better sense of where each one is looking than the topline counts.
| Category | What it covers | Prowler | ScubaGear | Secure Score |
|---|
| Risk-based conditional access | Detecting and blocking risky sign-ins and risky users automatically | 3 | 2 of 3 fail | 2 |
| MFA, auth methods & credential hygiene | MFA enforcement, phishing-resistance, stale/leaked credentials | 5 | 5 fail, 3 warn of 9 | 2 |
| CA: session, device & token controls | Sign-in frequency, managed-device rules, token/session-theft protection | 21 | N/A | N/A |
| App registration & consent governance | Who can register apps, what they can ask users to consent to | 5 | 2 of 3 fail | 0 (zero-weight) |
| Privileged role management (PIM) | Time-bound, approved, alerted-on admin roles vs. standing access | 2 | 6 fail, 1 warn of 9 | N/A |
| Device registration & management | Entra-joined device defaults: local admin, LAPS, BitLocker, join limits | 6 | Not a baseline product | N/A |
| Group, guest & tenant creation governance | Who can create groups, guest accounts, or tenants unapproved | 5 | 0 of 3 fail | N/A |
| Hybrid identity & directory sync | On-prem/cloud identity sync configuration | 1 | N/A | N/A |
| Admin portal & license hygiene | Admin account licensing, who can reach admin portals | 3 | N/A | N/A |
| SharePoint external sharing & guest permissions | What external users/guests can access or re-share | 4 | 3 of 3 fail | N/A |
| SharePoint link, session & auth settings | Default link permissions, expiration windows, modern auth | 1 | 5 of 5 fail | 2 |
| Teams external access & governance | Federation, anonymous meeting access, app/email integrations | Not assessed | 3 fail, 5 warn of 20 | 3 |
| Power Platform governance | Who can create environments, whether DLP policies exist | Not assessed | 3 of 9 fail | N/A |
| Exchange Online & Defender for Office | Anti-phishing, anti-spam, mail flow, mailbox auditing | 1 | Not in this run | 26 |
| Endpoint threat visibility | Defender for Endpoint visibility into credential exposure | 1 | Not a baseline product | N/A |
| Data classification / Purview | Sensitivity labeling and classification policy coverage | Not assessed | Not a baseline product | 3 |
A few things jump out just from the shape of that table. Prowler's single biggest bucket (21 unique checks) is conditional access session/device/token controls, an area ScubaGear's baseline barely touches; CISA cares whether risk-based CA and MFA exist at all, not the finer-grained policy hygiene Prowler goes after. Flip it around and ScubaGear is the only one of the three with real depth on privileged role lifecycle (PIM) and the only one that looks at Power Platform at all. And Secure Score's 26-item Exchange/Defender for Office bucket dwarfs everything else in this table; it's clearly the deepest lens on mail security, for the obvious reason that Microsoft owns both the product and the scoring engine.
Where all three actually agree #
Out of everything, only two themes got flagged by every single tool. I'd treat unanimous findings like these as the highest-confidence, do-first items on any tenant, test or not:
Identity Protection risk policies aren't turned on. Prowler failed three separate checks around sign-in risk and user risk conditional access policies; ScubaGear's MS.AAD.2.1v1 and MS.AAD.2.3v1 failed for the same reason; and it's the single largest point value in the whole Secure Score export (+10.6% each for enabling sign-in risk and user risk policies).
MFA is enabled, but not the phishing-resistant kind. All three tools noticed a gap between "MFA exists" and "MFA is actually good." Prowler flagged the absence of a conditional access policy requiring phishing-resistant MFA for admins; ScubaGear's MS.AAD.3.1v1/3.2v1/3.6v1 failed on the same requirement; and Secure Score only gave partial credit (4.2 out of 9 points) for the MFA recommendation it does track.
Where two out of three agree #
A step down in confidence, but still solid: Prowler and ScubaGear lined up almost exactly on SharePoint and OneDrive. Prowler failed five separate checks (external sharing unrestricted, guest re-sharing allowed, OneDrive sync permitted from unmanaged devices, and modern auth not required), while ScubaGear failed every single one of its eight SharePoint controls (MS.SHAREPOINT.1.1 through 3.3): external sharing, default link permissions, and link expiration/reauth windows. Secure Score touches the edges of this (inactive session sign-out, modern auth for SharePoint apps) but doesn't score external sharing directly, so it shows up as a partial rather than a third confirmation.
Same story with app registration governance: non-admins can register apps, user consent isn't locked down, and there's no admin consent workflow. Prowler and ScubaGear both catch it; Secure Score lists a related item but at zero score weight, since it already credits a related, already-completed control elsewhere.
This is the part that actually justifies running more than one of these:
Only Prowler flagged that Microsoft Defender for Endpoint wasn't enabled at all, which meant Defender XDR's Security Exposure Management had no visibility into whether any privileged user's credentials showed up on a risky device. Worth being precise about what this is: it's not a confirmed exposure, it's a blind spot; the check fails because there's nothing to evaluate without MDE turned on in the first place. Still the tenant's single "critical" severity finding, and neither of the other two tools does this kind of device-threat correlation at all.
Only ScubaGear went deep on Entra Privileged Identity Management, recording six separate CISA "Shall" failures around role activation approval, alerting, and permanent vs. time-bound privileged assignments. Neither Prowler nor Secure Score has an equivalent check. It's also the only one of the three that assesses Power Platform at all (environment creation restrictions, DLP policy), which is easy to forget even exists until a baseline tool goes and checks it.
Only Secure Score surfaces Purview sensitivity labeling, and it's not a small miss. Publishing a data classification policy alone is worth +22% in that export, and the full label rollout (auto-labeling, extending it into the Purview data map) is worth +44% combined, more points than anything else in the entire dataset. Neither CSPM tool looks at Purview at all. Secure Score also carries a long tail (26 items) of Defender for Office anti-phishing detail: impersonation protection, quarantine actions, mailbox auditing, Safe Links. ScubaGear would probably mirror some of that if it had Exchange access on this run, but doesn't right now.
If you're choosing between Secure Score and ScubaGear specifically #
Prowler needs its own setup and a security-engineering mindset to run at all, so for a lot of readers the real decision isn't "which of three tools," it's "Secure Score, which I already have, or ScubaGear, which is free but one more thing to run." I went control-by-control through both tools' output (the control mapping linked above covers the shared themes) to see how much they actually substitute for each other. Short answer: not much, and the overlap that exists is lopsided in ScubaGear's favor.
- Of ScubaGear's 67 controls, only around 13-17 have any Secure Score counterpart at all, and almost all of them sit inside Entra ID: legacy auth, identity protection risk policies, phishing-resistant MFA, app consent/registration, password expiration, admin-count and role hygiene. Everything else ScubaGear checks, including all 9 Power Platform controls, all 6 Entra PIM lifecycle controls, most of its SharePoint link/sharing-scope detail, and most of its Teams meeting and external-access governance, has no Secure Score equivalent, period.
- Going the other direction, roughly half of Secure Score's comparable recommendations are Exchange/Defender for Office mail-hygiene items that ScubaGear's baseline could, in principle, check (it has an Exchange/Defender baseline), but didn't in this run because the scanning account lacked Exchange/Security Reader. That's a permissions gap, not a capability gap. Strip those out and Secure Score still has a genuine, ScubaGear-independent slice: leaked-credential remediation, SSPR, custom banned-password lists, admin sign-in-frequency/session-persistence controls, and Purview sensitivity labeling, none of which ScubaGear's baseline touches at all.
- Where the two do check the same thing, ScubaGear is consistently the higher-resolution tool: specific day thresholds, named Conditional Access conditions, exact role requirements, instead of Secure Score's coarser "enable this feature" framing. If they disagree on something they both claim to check, I'd trust ScubaGear's verdict over the score.
What that adds up to for picking one:
- Entra ID / MFA / Conditional Access hygiene: either works, but ScubaGear's answer is the more precise one.
- Power Platform governance or Entra PIM lifecycle: ScubaGear is your only option between these two; Secure Score doesn't go there.
- Purview data classification or deep Defender for Office/Exchange mail-hygiene detail: Secure Score is your only option here, at least until you re-run ScubaGear with full Exchange/Security Reader access, which would close a real chunk of this gap.
- A single number to put in front of leadership: that's what Secure Score is built for. ScubaGear's Pass/Fail/Warning table was never designed to roll up into one score, and trying to force it to is more effort than it's worth.
They're not substitutes for each other so much as two tools with a narrow, mostly-Entra-ID overlap and very different blind spots on either side of it. But if you're asking which to run first, run ScubaGear first. It's free, it's more precise everywhere the two overlap, and the ground it alone covers (standing privileged access and alerting in Entra PIM, ungoverned Power Platform environments) is a live privilege-escalation and data-exfiltration surface, not a polish item. Secure Score's exclusive territory (Purview labeling, mail-hygiene tuning) matters, but it's already running in your tenant for free, so there's no cost to coming back to it second. Fix what ScubaGear flags as a Shall-failure, then layer in whatever Secure Score still has open.
Takeaways #
None of these three tools is wrong, exactly; they're just answering different questions with different depth. Prowler and ScubaGear largely agree where they overlap, which is reassuring: an independent CSPM and a government baseline converging on the same failing controls is a decent signal that those controls are actually worth fixing first. But each one also has a real blind spot the others don't cover (such as Prowler's device-threat correlation, ScubaGear's Entra PIM and Power Platform depth, or Secure Score's Purview and Defender for Office detail), so picking just one and calling it done leaves real gaps.
If you're doing this for real: fix what's unanimous first (risk-based conditional access, phishing-resistant MFA), then go through the two-out-of-three items (external sharing, app consent), and don't skip re-running ScubaGear with full Exchange/Security Reader access; that one gap in this run's permissions is hiding an entire baseline's worth of results.
Published