Conditional Access report-only: how to validate a policy before enforcement

1. What Conditional Access report-only establishes

Report-only evaluates a policy during sign-in without enforcing its grant and session controls. Someone being able to sign in therefore does not establish that the same person could work normally after enforcement. The current configuration may permit access while the proposed policy would add a requirement.

This distinction matters when introducing an authentication requirement, requiring a compliant device or restricting a particular access path. Evaluation data supports a hypothesis about impact. Operational testing establishes whether the user has the necessary credential or device and can complete the intended task.

Do not switch an enforced policy to report-only simply to test a change: that removes its enforcement. Preserve the configuration protecting the environment and document the proposed change separately, with a meaningful name and version. The subsequent change procedure must decide which configuration will remain in service.

This article complements our Microsoft Entra Conditional Access baseline policies. Its purpose is to decide whether an individual change is ready for production, rather than define the entire identity security strategy again.

A useful decision record distinguishes three statements: the policy was evaluated, the proposed requirement is achievable, and the business process was successfully tested. These are related findings, but each needs its own evidence. Treating the first statement as proof of the other two creates false confidence.

2. Prepare scope, permissions and licensing

Before opening the portal, write a testable sentence: “Pilot users accessing the selected resource through the included clients must satisfy this control.” Identify who is excluded and why. If the team cannot describe the condition and expected outcome, it does not yet have a test to execute.

Check the Conditional Access entitlement in your subscription. Microsoft Entra ID P1 supports general Conditional Access use; risk-related capabilities can require P2. The insights and reporting workbook also requires the Azure Monitor and Log Analytics components documented by Microsoft. Having a Microsoft 365 subscription alone does not establish that this reporting setup exists.

Separate configuration from analysis. Policy administrators need an appropriate role, such as Conditional Access Administrator. Analysts require sufficient access to both sign-in data and policy information. Avoid assigning Global Administrator simply to make a review convenient. Missing policy details may reflect permissions rather than an absence of evaluation.

Record the policy identifier, groups, resources, conditions, controls and exclusions. Include the application owner and support contact. A short configuration record prevents later arguments about whether a screenshot belongs to the version that was actually approved.

Emergency access accounts should follow the tenant’s emergency access design and remain monitored. Every exclusion needs a named purpose and owner. Excluding whole departments indefinitely is an unresolved part of the scope, rather than an acceptance criterion.

Also record the proposed change window and how users can obtain help. Security testing is easier to interpret when support staff know which policy is being evaluated. Otherwise, unrelated incidents can be attributed to the pilot and genuine readiness problems can be dismissed as ordinary support noise.

3. Interpret policy results separately from availability

Read the result of the specific policy within the sign-in event. Overall sign-in status and report-only status answer different questions. A successful sign-in can coexist with a proposed requirement that remains unresolved. A failed sign-in may be caused by a different enforced policy.

Do not present these categories as a contractual service availability rate. One person may appear under different outcomes during the observation period. Decide whether the report counts unique users, events or business processes, then keep that unit consistent.

Investigate “Not applied” when the policy should cover the scenario. It may reveal incorrect group membership, a different resource or a network assumption that does not hold. Conversely, the same result for an intentionally excluded identity may be precisely the evidence the test requires.

“Success” also does not prove that a person completed a fresh authentication challenge. Evaluation may rely on information in an existing session. When the change is meant to alter authentication behaviour, test that behaviour deliberately instead of drawing conclusions from repeated access in one session.

Keep expected rejection cases visible in the acceptance record. Removing failures simply because they look unfavourable makes the result less useful. The key question is whether the outcome matches the intended security decision and whether the business has an approved alternative where one is required.

4. Build a small matrix that represents real work

A useful matrix combines identities, resources and conditions that change the security decision. It need not enumerate every theoretical combination. Start with critical workflows, known exceptions and user groups likely to need help when authentication requirements change.

• Identity: a regular employee, a dedicated administrator, an in-scope guest and a justified excluded account.
• Device: a managed compliant device, a managed noncompliant device and an unmanaged device where the design addresses that access.
• Client: the browsers, desktop applications and mobile applications actually used.
• Resource: the requested application and related resources required to finish the task.
• Situation: routine access, a new session and the user’s recovery scenario.

For each row, record the expected outcome, its reason and the person responsible for execution. Use a concrete task, such as opening a shared document, entering an administrative portal or using an application function. “Access Microsoft 365” is too broad to validate a workflow involving multiple resources.

What If helps identify policies affecting selected sign-in conditions. Use it to review the design and investigate scenarios that are difficult to reproduce. Microsoft states that it does not test service dependencies: a simulated Teams sign-in is not a substitute for testing access to a dependent resource.

Retain both What If output and actual sign-in evidence where each adds value. If they disagree, examine the parameters, policy version and resource that the application really requested. Do not rewrite the expected matrix after the fact merely to make every scenario appear accepted.

Record untested combinations explicitly. A browser test on one managed laptop does not establish mobile readiness, and an employee test does not establish guest behaviour. A small honest matrix is more useful than a long checklist whose boxes lack traceable evidence.

5. Collect evidence that supports a decision

Open Microsoft Entra sign-in logs and select a representative event. Review the identity, timestamp, application, resource, client and Conditional Access details. Locate the proposed policy in its report-only evaluation and compare the conditions with the matrix.

Portal navigation and labels can change. The important evidence is a traceable reference to the event and policy identifier, rather than a generic screenshot of the dashboard. Include time zones and avoid combining different reporting windows without explanation.

When using the workbook, filter to the policies and period relevant to the change. Establish that log integration was collecting data during that period. An empty chart is not proof of zero impact; it may mean the required information has not yet been collected.

An acceptance record should include the test case, expected result, observed event, actual outcome, discrepancy cause and decision. Restrict access to the evidence. Identity logs can contain personal information and internal details that do not belong in a public report or unrestricted support attachment.

Look for patterns as well as individual examples. If a mobile client accounts for many unresolved requirements, examine its readiness before expanding the group. If a dependent resource behaves differently, involve its owner. One favourable administrator screenshot does not represent people using other devices and workflows.

The observation window should cover relevant business cycles. MSAdvance recommends naming periodic tasks that were not observed and assigning an additional test. There is no universal number of days that automatically turns evaluation into a safe deployment.

Make the evidence understandable to the person approving the change. Explain what each unresolved case means for work, who will address it and when it will be retested. A technical export alone can obscure the difference between intended enforcement and an unintended interruption.

6. Move from evaluation to an enforced pilot

Once discrepancies are understood, prepare a small representative group for enforcement. Users should know the requirement, have the approved method or device ready and know how to obtain support. Include important scenarios instead of selecting only members of the technical team.

Before enforcement, confirm that pilot scope does not accidentally alter existing protection. Document the relationship between the proposal and active policies. Review the change with another authorised person where your change management process calls for it. This recommendation does not imply that a named expert has reviewed this article.

During the pilot, execute the matrix tasks and observe permitted access and intended rejection. Support staff should be able to explain a correct block. Record incidents by cause and affected workflow, so repeated attempts by one person are not mistaken for multiple independent users.

Suggested acceptance criteria

• Critical matrix rows have evidence and an acceptance owner.
• Intended blocks occur within the correct scope.
• Permitted workflows work with approved methods.
• Exclusions have an owner, reason and review date.
• Support has tested diagnosis and recovery instructions.
• Remaining discrepancies have an explicit decision before expansion.

These are MSAdvance operational recommendations. Adapt them to service criticality and internal obligations. Do not claim “no impact” when only some clients have been observed or when a monthly process remains untested.

A specific rollback

Define which pilot change would be reversed, who can reverse it and how the result would be confirmed. Preserve the previous configuration and identifiers. Rollback should address the problematic change within its agreed scope; indiscriminately disabling Conditional Access may create more risk than the initial incident.

After expansion, continue observing enforced policies and retire temporary exceptions according to the plan. A successful first morning does not eliminate the need to examine less frequent clients, new starters and replacement devices.

Keep the final approval connected to the exact configuration deployed. If someone changes a group, exclusion or grant requirement after the pilot, evaluate whether the evidence still applies. Acceptance is a conclusion about a defined change, not permanent approval for every later variation of the policy.

7. Hypothetical example: managed-device access

An organisation wants to require compliant devices for a resource used by office staff and field sales. It retains existing protection and prepares an evaluation policy for a pilot group. Its matrix includes a corporate browser, a mobile application and external access that should be rejected.

Evaluation shows satisfied office access and unresolved requirements on several mobile devices. The team neither treats those events as product defects nor removes them from the report. It checks enrolment, compliance and the client used. It also finds one row without events because that task normally runs at month end.

The decision is to postpone that part of the expansion, complete mobile preparation and run a controlled test of the periodic workflow. The enforced pilot then confirms allowed tasks and the intended unmanaged-device rejection. The acceptance record states what was approved and what remains outside scope.

This example is not a real project or a claim about measured MSAdvance outcomes. It illustrates why technical data needs business context: a majority of favourable events can coexist with a critical workflow that has no evidence yet.

8. Mistakes that invalidate the conclusion

Common mistakes include reading only overall sign-in status, combining user and event counts, allowing exclusions to accumulate and using What If instead of actual testing. Changing several policies at once also makes it harder to identify which modification caused a problem.

If overlapping policies need to be clarified first, a Microsoft 365 and Azure security review can define deployment scope and responsibilities. The useful commercial outcome is a defensible validation plan, rather than a promise that every configuration can be enabled without incidents.

Official references and limitations

• Microsoft Learn: report-only policy evaluation.
• Microsoft Learn: What If tool and limitations.
• Microsoft Learn: insights and reporting workbook.
• Microsoft Learn: applied policies in sign-in logs.
• Microsoft Learn: requiring device compliance.

Check current licensing, permissions and tenant options before implementation. This guide does not assess your environment or replace internal change approval.