Automate Microsoft 365 Certification with ACAT

The App Compliance Automation Tool (ACAT) can be used to meet a specific set of required controls for Microsoft 365 Certification. This article outlines how to implement ACAT in Partner Center and use the compliance assessments to expedite the Microsoft 365 Certification.

Note

ACAT is currently in public preview and only supports applications built on Azure. Future updates will include functionality for apps built on non-Microsoft hosted cloud services. For any feedback on ACAT public preview, please complete this form. An ACAT product team specialist will follow up with you as soon as possible.

Create your first compliance report to onboard ACAT

ACAT gives added visability into the compliance of an application via custom reporting. Users can create reports based on the cloud infrastructure or a specific environment of an app -- production, staging, etc.

  • Search and launch App Compliance Automation Tool for Microsoft 365 in Azure portal.
  • Click Reports on the left hand side of the screen.

Create compliance report

  • Select Create new report to create your first compliance report.

    • Basics
      • Report name: Naming a compliance report is required and can't be duplicated within the tenant. It can be a combination of numbers, letters, and underscore. It's recommended to include the specific app name or environment in the name of the report.
      • Trigger time: ACAT creates compliance reports on a daily basis. This option allows you to specify what time the report wil be generated. As default, it's set to the time zone of your local machine.
      • Select resources: Select resources based on your cloud infrastructure to define the report parameters. You can also use the provided filters to search by subscription, resource group, tag, etc.

    Basic configuration

    • Microsoft 365 Certification
      • Offer GUID: The offer GUID is a unique identifier for marketplace offers in Microsoft Partner Center. Select Learn more to get your app's offer GUID. This step is optional during the initial report creation and can be configured once you begin publishing your app.

    Microsoft 365 Certification configuration

Once you confirm the configuration and create the compliance report, ACAT collects the compliance related data automatically from the following sources:

Note

Please allow 24 hours for ACAT to create the first report based on your preferences.

Audit the compliance assessments with your compliance report

Review the run-time status of the compliance reports and audit your assessments.

  • Go to Reports on the left for a summary of existing compliance reports.

  • Run-time status shows the state of the last run report with three possilbe outcomes: - Active: The compliance report ran successfully. - Failed: ACAT failed to create the assessments for the most recently run report. Failed states could be caused by an incorrect configuration in your subscription, or a system failure with ACAT. - Disabled: The compliance report had been disabled (paused) manually by the user. This feature is not currently enabled in public preview.

    • Last trigger time and Next trigger time: ACAT generates the assessments for compliance reports on a daily basis. Last trigger time indicates when the last generation was triggered, and Next trigger time indicates the time the next report will be generated.
    • Microsoft 365 Certification: Review the compliance status of controls specific to Microsoft 365 Certification.

    Compliance report list

Besides learning the high-level summary of the existing compliance reports, you can also audit the compliance assessment details in ACAT. Get the assessment details for the specific report by clicking the report name. ACAT shows the following details:

Compliance report toolbar

  • Settings: Change the configuration of compliance reports.

    • Edit basic information: Edit the basic configuration of the report, including trigger time.
    • Edit resources: Add or remove resources based on the current cloud infrastructure.
    • Edit Microsoft 365 Certification configuration: Configure offer GUIDs to associate the report with a marketplace offer in Microsoft Partner Center.
  • Download report: Download assessments of the compliance report that can be shared with partners for collaboration.

    • Cloud infrastructure inventory: This file contains the resource details of this compliance report to describe the cloud inventory of your application.
    • Microsoft 365 Certification control assessments summary (PDF): The PDF file contains a high level control assessment of the application, including the compliance state for control families and controls.
    • Microsoft 365 Certification control assessments details (CSV): This CSV file contains the control assessments details of the application, including assessments of controls and resources.
  • Notifications: Get notifications of the compliance report settings change or control assessments state change. Learn how to receive notifications via webhook here

  • Essentials indicates the status and the configuration of the compliance report.

    Compliance report essentials

  • Control assessments

    • Control assessments are organized by Microsoft 365 Certification security domains, control families and compliance.
      • Learn more about compliance controls and how to collect evidence by clicking the control name.
        • For fully manual controls, collect evidence manually in Control description.
        • For partially automated manual controls, after completing the automated customer responsibilities, collect remaining evidence manually in Evidence collection.
      • By expanding both the fully automated and partially automated manual controls, you can review the compliance states by customer responsibility at the individual control level.
      • Under customer responsibility, click Actions to find the compliance state of related resources and remediation steps for failed resources.
        • Unhealthy resources: Follow up on remediation steps to fix unhealthy resources.
        • Not applicable resources: Navigate to any resource marked N/A. Provide a reason to set up the resources so ACAT can collect the compliance assessments.

    Compliance report assessments

Use search and filters to find specific controls based on your needs.

  • Search the controls by control name or customer responsibility name.
  • Use Control family to filter by security domain or control family.
  • Use Control state to filter for current compliance failures.

Use your first compliance report with Microsoft 365 Certification audit

Before using the compliance report with Microsoft 365 Certification, you will need to configure the offer GUID to associate it with your marketplace offer. There are two options:

  • During the creation process of the compliance report, configure the offer GUID in Microsoft 365 Certification tab.
  • If the compliance report is already created, go to Settings of this compliance report to configure the offer GUID.

Once the offer GUID is configured, go to the Microsoft Partner Center to begin Microsoft 365 Certification. In Initial Documentation click Yes to confirm you are using ACAT. Select the most up to date active compliance report for the audit. The Microsoft 365 Certification submits the compliance assessments of fully automated and partial automated manual controls to the certification auditors automatically, saving you time and effort.

Note

Only use the active compliance report for Microsoft 365 Certification review. If the compliance report run-time status shows failed, ACAT had an issue creating the most recent assessment.

Note

ACAT only covers certain controls that can be automatically pulled from Azure. Microsoft 365 Certification will require additional evidence that will need to be manually provided to auditors during the Control Requirements phase of certification.

##Get a high level overview of your compliance reports

Overview provides a high level status for your compliance reports.

  • Compliance Report Run-time Status: Shows metrics of the run-time status for your compliance reports.
    • Active: The compliance report is running continuously and successfully.
    • Failed: ACAT has failed to create the compliance assessments for the most recently generated compliance report. Failed states could occur due to an incorrect configuration in your subscription creating a block in the compliance data routed to ACAT, or a system failure or outage with ACAT.
    • Disabled: The compliance report had been disabled (paused) manually by the user. This feature is not enabled in public preview.

Overview of run-time status

  • Active Regulatory Compliance Reports: This overview gives you the compliance status for each active report.

Overview of compliance status

Troubleshooting

  • Report status shows as 'Failed' once generated A report has been created successfully but the report's status is set to Failed. Usually this is caused by missing permissions for the related subscription(s). Click solution.1 to add the proper permissions.

  • Report status shows as 'Failed' after initial 24 hours A report has been created successfully but the report's status changed to Failed after 24 hours. Go to solution.2 to check the status of the ACAT service.

  • Create report failed due to registration failure The report did not run and the status is showing as registration failed. Follow this solution.3

  • Assessments do not show in the report after 24 hours If there are no assessments in a report after 24 hours, and the status is Active. Check this solution.4

  • The report is not found A report has been created successfully by another user but you cannot acces it. Follow this solution.5

  • Can't modify the report resource list A report can be opened but you cannot modify the report resource list to another subscription. Follow this solution.6

Solution list

1. Assign report creation needed permission for your subscriptions

When creating a compliance report, ACAT sets up an environment within your subscription to collect the compliance data automatically. This action requires specific permissions of the subscription. Check your subscription permission as below.

  • Search and launch the Subscriptions in Azure portal.
  • Go to the subscription that you want to use to create the compliance report.
  • Go to the Access control (IAM) on the left.
  • Select View my access to check your permission.
  • Confirm the below resources are registered under your subscription(s):

    • Microsoft.AppComplianceAutomation
    • Microsoft.Security
    • Microsoft.ResourceGraph
      Without these registration under your subscription(s), our service can't work as expected. So you need to register them for all the related subscription(s) again.
  • Check whether there's an automation under each subscription

    • Query all the subscription(s) in your report, confirm each subscription has a resource group named acat-management-rg-web
    • There is one hide resource named acat-automation
      If the resource group does not exist, or the hidden resource does not exist, please contact support team for help
  • Check whether the required policy assignment

    • Query all the subscription(s) in your report, check whether the below policy assignment is there: (Method)
      • acat custom policy set
      • acat built-in policy set
      • Azure Security Benchmark
        If miss any one of the above policy assignment, that will cause the report can't generate accurate And please contact support team for help

3. Assign the subscription registration permission

When you create a new report, if prompts creation failed due to the registration failed. Ask all the subscription(s) owners to grant the registration permission to you for these subscription(s). Without that you can't finish the report creation.

4. Check whether your report resource is applicable for the Microsoft 365 certification criteria

Review the Microsoft 365 certification criteria, for all the needed checked resource type for its criteria. Compare whether your report's resource type matches the related resource requirement. If not, the assessment report will be blank even if your report is showing Active status.

5. Check whether you have permission to access the report

The report includes one or more subscriptions in it. If you want to access this report you will need permission to all of subscriptions mentioned in the report. Otherwise you can't access this report. To get access you need to ask the related subscription owner to grant the subscription(s) access permission to you.

6. Grant the subscription owner permission to you

If you want to modify/delete the target report, and the report's subscriptions requires owner permission.