Windows Copilot Runtime overview
Important
Self-contained apps are not supported.
Windows Copilot Runtime provides a variety of AI-powered features available via APIs, allowing you to utilize AI capabilities without the need to find, run, or optimize your own Machine Learning (ML) model. The models that power Windows Copilot Runtime on Copilot+ PCs run locally and continuously in the background.
When utilizing AI features, we recommend that you review: Developing Responsible Generative AI Applications and Features on Windows.
Windows Copilot Runtime features and APIs for Windows apps
Windows Copilot Runtime includes the following features and AI-backed APIs (in the Windows App SDK) powered by models running locally on the Windows device.
Phi Silica: The Phi Silica APIs will ship in the Windows App SDK. Similar to OpenAI's GPT Large Language Model (LLM) that powers ChatGPT, Phi is a Small Language Model (SLM) developed by Microsoft Research to perform language-processing tasks on a local device. Phi Silica is specifically designed for Windows devices with a Neural Processing Unit (NPU), allowing text generation and conversation features to run in a high performance, hardware-accelerated way directly on the device.
Text Recognition with OCR: The Text Recognition APIs (also referred to as Optical Character Recognition or OCR) will ship in the Windows App SDK. These APIs enable the recognition of text in an image and the conversion of different types of documents (such as scanned paper documents, PDF files, or images captured by a digital camera) into editable and searchable data on a local device.
Imaging APIs: The AI-enhanced Imaging APIs will ship in the Windows App SDK. These APIs perform a variety of actions such as intelligently scaling images and identifying objects within images.
Studio Effects: Windows devices with compatible Neural Processing Units (NPUs) integrate Studio Effects into the built-in device camera and microphone settings. Apply special effects that utilize AI, including: Background Blur, Eye Contact correction, Automatic Framing, Portrait Light correction, Creative Filters, or Voice Focus for filtering out background noise.
Recall: Recall enables users to quickly find things from their past activity, such as documents, images, websites and more. Developers can enrich the user's Recall experience with their app by adding contextual information to the underlying vector database with the User Activity API. This integration will help users pick up where they left off in your app, improving app engagement and user's seamless flow between Windows and your app.
Live Caption Translations help everyone on Windows, including those who are deaf or hard of hearing, better understand audio by viewing captions of spoken content (even when the audio content is in a language different from the system's preferred language).
Content Moderation: Learn how Windows Copilot Runtime moderates content and how to adjust sensitivity filters.
Integrate AI in enterprise apps using Windows Copilot Runtime APIs
Watch the demo session Integrate AI in Enterprise apps using Windows Copilot Runtime APIs from the November 2024 Ignite Conference.
Additional resources
- Get started with AI on Windows: Windows Copilot Runtime implements a Text Content Moderation API to flag and filter out potentially harmful content. Learn more about this feature and how to adjust the filter sensitivity.
- Developing Responsible Generative AI Applications and Features on Windows: Guidance for responsibly developing apps that incorporate AI.
- AI on Windows Sample Gallery: A collection of samples that demonstrate a variety of ways to enhance your Windows apps using AI.