Automate Microsoft 365 Certification with ACAT
The App Compliance Automation Tool (ACAT) can be used to meet a specific set of required controls for Microsoft 365 Certification. This article outlines how to implement ACAT in Partner Center and use the compliance assessments to expedite the Microsoft 365 Certification.
Note
ACAT is currently in public preview and only supports applications built on Azure. Future updates will include functionality for apps built on non-Microsoft hosted cloud services. For any feedback on ACAT public preview, please complete this form. An ACAT product team specialist will follow up with you as soon as possible.
Create your first compliance report to onboard ACAT
ACAT gives added visability into the compliance of an application via custom reporting. Users can create reports based on the cloud infrastructure or a specific environment of an app -- production, staging, etc.
- Search and launch App Compliance Automation Tool for Microsoft 365 in Azure portal.
- Click Reports on the left hand side of the screen.
Go to Reports to create a new compliance report
Select Create new report to create your first compliance report.
- Basics
- Report name: Naming a compliance report is required and can't be duplicated within the tenant. It can be a combination of numbers, letters, and underscore. It's recommended to include the specific app name or environment in the name of the report.
- Trigger time: ACAT creates compliance reports on a daily basis. This option allows you to specify what time the report wil be generated. As default, it's set to the time zone of your local machine.
- Select resources: Select resources based on your cloud infrastructure to define the report parameters. You can also use the provided filters to search by subscription, resource group, tag, etc.
Basic configuration for the compliance report
- Microsoft 365 Certification
- Offer GUID: The offer GUID is a unique identifier for marketplace offers in Microsoft Partner Center. Select Learn more to get your app's offer GUID. This step is optional during the initial report creation and can be configured once you begin publishing your app.
- Basics
Once you confirm the configuration and create the compliance report, ACAT collects the compliance related data automatically from the following sources:
- Enable the Microsoft Defender for Cloud (free tier) for your subscription.
- Enable custom policies for your subscription.
Note
Please allow 24 hours for ACAT to create the first report based on your preferences.
Audit the compliance assessments with your compliance report
Review the run-time status of the compliance reports and audit your assessments.
Go to Reports on the left for a summary of existing compliance reports.
Run-time status shows the state of the last run report with three possilbe outcomes: - Active: The compliance report ran successfully. - Failed: ACAT failed to create the assessments for the most recently run report. Failed states could be caused by an incorrect configuration in your subscription, or a system failure with ACAT. - Disabled: The compliance report had been disabled (paused) manually by the user. This feature is not currently enabled in public preview.
- Last trigger time and Next trigger time: ACAT generates the assessments for compliance reports on a daily basis. Last trigger time indicates when the last generation was triggered, and Next trigger time indicates the time the next report will be generated.
- Microsoft 365 Certification: Review the compliance status of controls specific to Microsoft 365 Certification.
Besides learning the high-level summary of the existing compliance reports, you can also audit the compliance assessment details in ACAT. Get the assessment details for the specific report by clicking the report name. ACAT shows the following details:
Toolbar for the compliance report.
Settings: Change the configuration of compliance reports.
- Edit basic information: Edit the basic configuration of the report, including trigger time.
- Edit resources: Add or remove resources based on the current cloud infrastructure.
- Edit Microsoft 365 Certification configuration: Configure offer GUIDs to associate the report with a marketplace offer in Microsoft Partner Center.
Download report: Download assessments of the compliance report that can be shared with partners for collaboration.
- Cloud infrastructure inventory: This file contains the resource details of this compliance report to describe the cloud inventory of your application.
- Microsoft 365 Certification control assessments summary (PDF): The PDF file contains a high level control assessment of the application, including the compliance state for control families and controls.
- Microsoft 365 Certification control assessments details (CSV): This CSV file contains the control assessments details of the application, including assessments of controls and resources.
Notifications: Get notifications of the compliance report settings change or control assessments state change. Learn how to receive notifications via webhook here
Essentials indicates the status and the configuration of the compliance report.
Control assessments
- Control assessments are organized by Microsoft 365 Certification security domains, control families and compliance.
- Learn more about compliance controls and how to collect evidence by clicking the control name.
- For fully manual controls, collect evidence manually in
Control description. - For partially automated manual controls, after completing the automated customer responsibilities, collect remaining evidence manually in
Evidence collection.
- For fully manual controls, collect evidence manually in
- By expanding both the fully automated and partially automated manual controls, you can review the compliance states by customer responsibility at the individual control level.
- Under customer responsibility, click
Actionsto find the compliance state of related resources and remediation steps for failed resources.- Unhealthy resources: Follow up on remediation steps to fix unhealthy resources.
- Not applicable resources: Navigate to any resource marked N/A. Provide a reason to set up the resources so ACAT can collect the compliance assessments.
- Learn more about compliance controls and how to collect evidence by clicking the control name.
- Control assessments are organized by Microsoft 365 Certification security domains, control families and compliance.
Use search and filters to find specific controls based on your needs.
- Search the controls by control name or customer responsibility name.
- Use
Control familyto filter by security domain or control family. - Use
Control stateto filter for current compliance failures.
Use your first compliance report with Microsoft 365 Certification audit
Before using the compliance report with Microsoft 365 Certification, you will need to configure the offer GUID to associate it with your marketplace offer. There are two options:
- During the creation process of the compliance report, configure the offer GUID in
Microsoft 365 Certificationtab. - If the compliance report is already created, go to
Settingsof this compliance report to configure the offer GUID.
Once the offer GUID is configured, go to the Microsoft Partner Center to begin Microsoft 365 Certification. In Initial Documentation click Yes to confirm you are using ACAT. Select the most up to date active compliance report for the audit. The Microsoft 365 Certification submits the compliance assessments of fully automated and partial automated manual controls to the certification auditors automatically, saving you time and effort.
Note
Only use the active compliance report for Microsoft 365 Certification review. If the compliance report run-time status shows failed, ACAT had an issue creating the most recent assessment.
Note
ACAT only covers certain controls that can be automatically pulled from Azure. Microsoft 365 Certification will require additional evidence that will need to be manually provided to auditors during the Control Requirements phase of certification.
##Get a high level overview of your compliance reports
Overview provides a high level status for your compliance reports.
- Compliance Report Run-time Status: Shows metrics of the run-time status for your compliance reports.
- Active: The compliance report is running continuously and successfully.
- Failed: ACAT has failed to create the compliance assessments for the most recently generated compliance report. Failed states could occur due to an incorrect configuration in your subscription creating a block in the compliance data routed to ACAT, or a system failure or outage with ACAT.
- Disabled: The compliance report had been disabled (paused) manually by the user. This feature is not enabled in public preview.
Overview of compliance report run-time status.
- Active Regulatory Compliance Reports: This overview gives you the compliance status for each active report.
Overview of compliance status for active compliance reports.
Troubleshooting
Report status shows as 'Failed' once generated A report has been created successfully but the report's status is set to
Failed. Usually this is caused by missing permissions for the related subscription(s). Click solution.1 to add the proper permissions.Report status shows as 'Failed' after initial 24 hours A report has been created successfully but the report's status changed to
Failedafter 24 hours. Go to solution.2 to check the status of the ACAT service.Create report failed due to registration failure The report did not run and the status is showing as
registration failed. Follow this solution.3Assessments do not show in the report after 24 hours If there are no assessments in a report after 24 hours, and the status is
Active. Check this solution.4The report is not found A report has been created successfully by another user but you cannot acces it. Follow this solution.5
Can't modify the report resource list A report can be opened but you cannot modify the report resource list to another subscription. Follow this solution.6
Solution list
1. Assign report creation needed permission for your subscriptions
When creating a compliance report, ACAT sets up an environment within your subscription to collect the compliance data automatically. This action requires specific permissions of the subscription. Check your subscription permission as below.
- Search and launch the Subscriptions in Azure portal.
- Go to the subscription that you want to use to create the compliance report.
- Go to the Access control (IAM) on the left.
- Select View my access to check your permission.
- If your organization is using Azure built-in roles, your role assignments should include at least one of the following roles:
- Resource Policy Contributor and Security Admin
- Other role assignment that has higher permission (for example, Owner, etc.)
- If your organization is using Azure built-in roles, your role assignments should include at least one of the following roles:
2. Check if all ACAT related services are still available
Confirm the below resources are registered under your subscription(s):
- Microsoft.AppComplianceAutomation
- Microsoft.Security
- Microsoft.ResourceGraph
Without these registration under your subscription(s), our service can't work as expected. So you need to register them for all the related subscription(s) again.
Check whether there's an automation under each subscription
- Query all the subscription(s) in your report, confirm each subscription has a resource group named
acat-management-rg-web - There is one hide resource named
acat-automation
If the resource group does not exist, or the hidden resource does not exist, please contact support team for help
- Query all the subscription(s) in your report, confirm each subscription has a resource group named
Check whether the required policy assignment
- Query all the subscription(s) in your report, check whether the below policy assignment is there: (Method)
acat custom policy setacat built-in policy setAzure Security Benchmark
If miss any one of the above policy assignment, that will cause the report can't generate accurate And please contact support team for help
- Query all the subscription(s) in your report, check whether the below policy assignment is there: (Method)
3. Assign the subscription registration permission
When you create a new report, if prompts creation failed due to the registration failed. Ask all the subscription(s) owners to grant the registration permission to you for these subscription(s). Without that you can't finish the report creation.
4. Check whether your report resource is applicable for the Microsoft 365 certification criteria
Review the Microsoft 365 certification criteria, for all the needed checked resource type for its criteria. Compare whether your report's resource type matches the related resource requirement. If not, the assessment report will be blank even if your report is showing Active status.
5. Check whether you have permission to access the report
The report includes one or more subscriptions in it. If you want to access this report you will need permission to all of subscriptions mentioned in the report. Otherwise you can't access this report. To get access you need to ask the related subscription owner to grant the subscription(s) access permission to you.
6. Grant the subscription owner permission to you
If you want to modify/delete the target report, and the report's subscriptions requires owner permission.