What is Maester?

Maester is an open-source PowerShell-based test automation framework designed to help organizations monitor and maintain the security configuration of their Microsoft 365 environment. It's built on top of Pester and can be deployed as a GitHub Action, Azure DevOps pipeline, or run standalone.

I've been using Maester for a while to validate Entra ID and M365 security configurations. But one area that was completely missing was Azure DevOps. Given how central Azure DevOps is to many organizations' software supply chains, I felt this was a gap worth filling.

Building the Azure DevOps Test Suite

I wrote 37 security tests for Azure DevOps that cover the key areas you'd want to validate in any organization:

  • Authentication & Authorization: OAuth scope restrictions, SSH policy, AAD Conditional Access enforcement
  • Access Control: External user policies, guest access, invitation restrictions
  • Pipeline Security: Job authorization scope, stage chooser settings, shell task argument validation
  • Token Management: PAT creation policies, token lifespan limits, full-scope token restrictions
  • Audit & Monitoring: Audit logging enabled, log streaming, event collection
  • Resource Limits: Project counts, work item tags, storage usage thresholds

The severity breakdown: 1 Critical, 30 High, 3 Medium, and 3 Info level tests.

Making it optional

Not everyone uses Azure DevOps, and Maester already has optional test suites for Exchange, Teams, and Azure resources. I followed the same pattern. The tests gracefully skip if the ADOPS PowerShell module isn't installed or if there's no active Azure DevOps connection.

# Tests gracefully skip if ADOPS is not available
BeforeEach {
    if (-not (Get-Command 'Get-ADOPSOrganization' -ErrorAction SilentlyContinue)) {
        Set-ItResult -Skipped -Because "ADOPS module is not installed"
        return
    }
    if (-not (Get-ADOPSConnection -ErrorAction SilentlyContinue)) {
        Set-ItResult -Skipped -Because "Not connected to Azure DevOps"
        return
    }
}

What's next

The tests, documentation and a blog post for the Maester website are all in place. I'm working on getting this merged into the main Maester repository so everyone can benefit from Azure DevOps security monitoring.

If you're already using Maester, keep an eye out, Azure DevOps tests are coming.

//Sebastian