Microsoft365R author here. I've just updated the package on GitHub with the following features:
Add support for shared mailboxes to get_business_outlook(). To access a shared mailbox, supply one of the arguments shared_mbox_id, shared_mbox_name or shared_mbox_email specifying the ID, displayname or email address of the mailbox respectively.
Add support for Teams chats (including one-on-one, group and meeting chats).
Use the list_chats() function to list the chats you're participating in, and the `get_chat()` function to retrieve a specific chat.
A chat object has class ms_chat, which has similar methods to a channel: you can send, list and retrieve messages, and list and retrieve members/attendees. One difference is that chats don't have an associated file folder, unlike channels.
Updated the app permissions for Sharepoint to fix an access issue
However, at the moment I don't have access to a suitable tenant for testing these. This is a request for anyone whose tenant has approved the Microsoft365R app registration to install the development version and try using these features.
If you have previously used OneDrive, Teams and Outlook, everything should still work the same as before.
If you previously tried to use Sharepoint and got an authentication error, you should see a screen requesting you to grant Microsoft365R permission to access your Sharepoint site. You may or may not need an admin to approve the request, depending on your tenant's policies.
Please try it out and let me know of any problems! You can install the package from GitHub with
This is a summary of the updates to AzureR family of packages in May and June 2021.
AzureAuth
Change the default caching behaviour to disable the cache if running inside Shiny.
Update Shiny vignette to clean up redirect page after authenticating (thanks to Tyler Littlefield).
Add a create_AzureR_dir function to create the caching directory manually. This can be useful not just for non-interactive sessions, but also Jupyter and R notebooks, which are not technically interactive in the sense that they cannot read user input from a console prompt.
AzureGraph
Add enhanced support for the paging API. Many Graph calls that return lists of objects do so in pages: the first response contains a subset of the full list, along with a link to the next subset. AzureGraph now features a new ms_graph_pager R6 class, which is an iterator for the pages in the result. All list_* R6 methods have been rewritten to use this class, and have filter and n arguments to filter the result set and cap the number of results.
Also add support for the batch request feature: you can pass multiple calls to the API with the graph_request R6 class and call_batch_endpoint function, and get the results back in a single response.
Add list_users(), list_groups(), list_apps() and list_service_principals() methods to the main ms_graph client class.
There is also a new Authentication vignette to guide users through the various ways to authenticate to Microsoft Graph.
AzureRMR
Similar to AzureGraph, AzureRMR has a new Authentication vignette that details how to authenticate to Resource Manager. This replaces the old “Service principal” vignette.
Update Resource Manager API version to “2021-04-01”; also update az_subscription$list_locations to handle the new response format in this API version.
AzureStor
Add support for generating a service SAS. There is a new S3 generic get_service_sas with methods for az_storage and storage_endpoint objects, and a similar R6 method for az_storage objects. See ?sas for more information.
Fix storage_save_rds and storage_load_rds to handle compression correctly. In particular, storage_load_rds should now correctly load files saved with saveRDS (#83).
Fix a bug that caused list_blobs to fail when leases were present.
Use a raw connection instead of a raw vector when calling readr::read_delim and read_csv2. This works around an issue introduced in readr 1.4.0 (#85, #86).
Update client API version to “2020-04-08”. In particular, this allows specifying resource_type="d" when creating a service or user delegation SAS for blob storage.
Add an optional service argument to storage_endpoint, to specify the service in question: blob, file, ADLS2, queue or table. This allows use of the generic endpoint function with URLs that don’t fit the usual pattern where the service is part of the hostname, eg custom domain names, IP addresses, etc.
For the same reason, remove the warning about an unrecognised endpoint URL from blob_endpoint, file_endpoint and adls_endpoint.
Microsoft365R
Microsoft365R has some significant new features in the latest version:
OneDrive/SharePoint
Add a list_shared_items() method for the ms_drive class to access files and folders shared with you (#45).
Allow getting drives for groups, sites and teams by name. The first argument to the get_drive() method for these classes is now drive_name; to get a drive by ID, specify the argument name explicitly: get_drive(drive_id=*)
Add a by_item argument to the delete_item() method for drives and the delete() method for drive items (#21). This is to allow deletion of non-empty folders on SharePoint sites with data protection policies in place. Use with caution.
Outlook
Add a search argument to the ms_outlook_folder$list_emails() method. The default is to search in the from, subject and body of the emails.
Teams
Add list_members() and get_member() methods for teams and channels.
Add support for @mentions in Teams channel messages (#26).
Other
All list_* class methods now have filter and n arguments to filter the result set and cap the number of results, following the pattern in AzureGraph. The default values are filter=NULL and n=Inf. If n=NULL, an ms_graph_pager iterator object is returned instead to allow manual iteration over the results. Note that support for filtering in the underlying Graph API is somewhat uneven at the moment.
Experimental read-only support for plans, contributed by Roman Zenka.
Add get_plan() and list_plans() methods to the az_group class. Note that only Microsoft 365 groups can have plans, not any other type of group.
To get the plan(s) for a site or team, call its get_group() method to retrieve the associated group, and then get the plan from the group.
A plan has methods to retrieve tasks and buckets, as well as plan details.
I’m happy to announce that Microsoft365R 2.1.0 is now on CRAN with Outlook email support! Here’s a quick summary of the new features:
Send, reply to and forward emails, optionally composed with blastula or emayili
Copy and move emails between folders
Create, delete, copy and move folders
Add, remove and download attachments
Here’s a sample of how to write an email using blastula:
library(Microsoft365R)
# 1st one is for your personal Microsoft account# 2nd is for your work & school account
outl <- get_personal_outlook()
outlb <- get_business_outlook()
# compose an email with blastulalibrary(blastula)
bl_body <- "## Hello!
This is an email message that was generated by the blastula package.
We can use **Markdown** formatting with the `md()` function.
Cheers,
The blastula team"
bl_em <- compose_email(
body=md(bl_body),
footer=md("sent via Microsoft365R")
)
em <- outl$create_email(bl_em, subject="Hello from R",
to="bob@example.com")
# add an attachment and send it
em$add_attachment("mydocument.docx")
em$send()
And on the other side, here’s a sample of how to work with the emails in your inbox:
# list the most recent emails in your inbox
emlst <- outl$list_emails()
# get the most recent email
em <- emlst[[1]]
# list and download attachments
em$list_attachments()
em$download_attachment("mydatafile.csv")
# reply to it
em$create_reply("Replying from R")$send()
# list the folders in your account
outl$list_folders()
# move the move recent email to a given folder
folder <- outl$get_folder("My project folder")
em$move(folder)
In addition, this release fixes a bug in the list_files() method for OneDrive/Sharepoint drives and drive items, and adds the ability to create nested drive folders in one call.
Please note that if you’re using one of the workarounds mentioned in the authentication vignette, they won’t work with Outlook. You’ll need to get the Microsoft365R app approved for your tenant, or alternatively, if you have admin rights you can create your own tenant with the required permissions. (This applies if you’re using Microsoft365R at work; if you’re using it at home on your personal account, you shouldn’t have any problems.)
If you have any feedback or comments, you can email me or open an issue at the repo.
This is an announcement that a beta Outlook email client is now part of the Microsoft365R package. You can install it from the GitHub repository with:
devtools::install_github("Azure/Microsoft365R")
The client provides the following features:
Send, reply to and forward emails, optionally composed with blastula or emayili
Copy and move emails between folders
Create, delete, copy and move folders
Add, remove and download attachments
The plan is to submit this to CRAN sometime next month, after a period of public testing. Please give it a try and give me your feedback: either via email or by opening an issue at the repo.
Here’s a small sample of the client in action (taken from the README):
library(Microsoft365R)
# 1st one is for your personal Microsoft account
# 2nd is for your work & school account
outl <- get_personal_outlook()
outlb <- get_business_outlook()
# compose an email with blastulalibrary(blastula)
bl_body <- "## Hello!
This is an email message that was generated by the blastula package.
We can use **Markdown** formatting with the `md()` function.
Cheers,
The blastula team"
bl_em <- compose_email(
body=md(bl_body),
footer=md("sent via Microsoft365R")
)
em <- outl$create_email(bl_em, subject="Hello from R",
to="bob@example.com")
# add an attachment and send it
em$add_attachment("mydocument.docx")
em$send()
# list the most recent emails in your inbox
emlst <- outl$list_emails()
# reply to the most recent email
emlst[[1]]$
create_reply("Replying from R")$
send()
Please note though, that if you’re using one of the workarounds mentioned in the authentication vignette, they won’t work with Outlook. You’ll need to get the Microsoft365R app approved for your tenant, or alternatively, if you have admin rights you can create your own tenant with the required permissions. (This applies if you’re using Microsoft365R at work; if you’re using it at home on your personal account, you shouldn’t have any problems.)
To access a team in Microsoft Teams, use the get_team() function and provide the team name or ID. You can also list the teams you’re in with list_teams(). These return objects of R6 class ms_team, which has methods for working with channels and drives.
list_teams()
team <- get_team("My team")
# list the channels in a team (including your private channels)
team$list_channels()
# get the primary channel for a team
team$get_channel()
# get a specific channel
team$get_channel("My channel")
# drives for a team
team$list_drives()
team$get_drive()
A drive is an ms_drive object, so if you’re already using Microsoft365R to interface with OneDrive and SharePoint document libraries, you already know how to use a team’s drives. Each team will generally have at least one drive, and possibly two: the default “Shared Documents” drive, which is where uploaded files are stored, and the “Teams Wiki Data” drive, if the team has a wiki. Each team channel will usually also have an associated folder in each drive.
drv <- team$get_drive()
# one folder per channel
drv$list_files()
# upload will appear in Files tab of "My channel" in the Teams client
drv$upload_file("myfile.csv", "My channel/myfile.csv")
Channels
A team object has methods for listing, retrieving, creating and deleting channels. However you should not create and delete channels unnecessarily, since Teams tracks all channels ever created, even after you delete them. In turn, a channel object has methods for listing and sending messages, and uploading and deleting files.
Channel messages
Teams channels are semi-threaded. Getting the list of messages for a channel retrieves only the first message in each thread; to get an entire thread, you get the starting message and then retrieve the replies to it. Note that channels don’t have nested replies, so you can’t reply to a reply—only to the starting message.
The body of a message is part of the list of properties returned from the host, and can be found in the properties field of the object. Other properties include metadata such as the author, date, list of attachments, etc.
chan <- team$get_channel()
# retrieve most recent messages from the server
msgs <- chan$list_messages()
# get the latest message by ID
msg <- chan$get_message(msgs[[1]]$properties$id)
# body of the message
msg$properties$body
# 10 most recent replies
repl_list <- msg$list_replies(n=10)
# body of an individual reply
repl_list[[1]]$properties$body
You can send a message to a channel as plain text (the default) or HTML. A message can also include attachments and inline images.
# sending messages to a channel
chan$send_message("Hello from R")
chan$send_message(
"<div>Hello from <em>R</em></div>", content_type="html")
# attachments and inline images
chan$send_message("Hello with attachments",
attachments=c("intro.md", "myfile.csv"))
chan$send_message("",
content_type="html", inline="graph.png")
# send a reply to a message
msg <- chan$send_message("Starting a new thread in R")
msg$send_reply("Reply from R")
Currently, Microsoft365R only supports messaging in channels. Support for chats between individuals may come later.
Channel files
Uploading a file to a channel will place it in the channel’s drive folder. The channel object itself provides convenience functions to list, upload and download files. It also provides a get_folder() method to retrieve the folder for the channel, as an ms_drive_item object; this object has more general methods for working with files.
# files for the channel
chan$list_files()
# upload a file to the channel
chan$upload_file("myfile.docx")
# open the uploaded document for editing in Word Online
chan_folder <- chan$get_folder()
item <- chan_folder$get_item("myfile.docx")
item$open()
# download it again
item$download(overwrite=TRUE)
Providing Feedback
Let us know how this works for you! You can provide feedback and make feature requests by opening an issue at the repo, or by emailing me at hongooi73 (@) gmail.com.
I’m very happy to announce Microsoft365R, a package for working with the Microsoft 365 (formerly known as Office 365) suite of cloud services. Microsoft365R extends the interface to the Microsoft Graph API provided by the AzureGraph package to provide a lightweight yet powerful interface to SharePoint and OneDrive, with support for Teams and Outlook soon to come.
Microsoft365R is now available on CRAN, or you can install the development version from GitHub with devtools::install_github("Azure/Microsoft365R").
Authentication
The first time you call one of the Microsoft365R functions (see below), it will use your Internet browser to authenticate with Azure Active Directory (AAD), in a similar manner to other web apps. You will get a dialog box asking for permission to access your information.
Microsoft365R is registered as an app in the “aicatr” AAD tenant. Because it needs read/write access to groups and SharePoint sites, you’ll need an admin to grant it access to your tenant. Alternatively, if the environment variable CLIMICROSOFT365_AADAPPID is set, Microsoft365R will use its value as the app ID for authenticating; or you can specify the app ID as an argument when calling the functions below. See also this issue at the GitHub repo for some possible workarounds.
OneDrive
To access your personal OneDrive, call the personal_onedrive() function, and to access OneDrive for Business call business_onedrive(). Both functions return an R6 client object of class ms_drive, which has methods for working with files and folders. Note that OneDrive for Business is technically part of SharePoint, and requires a Microsoft 365 Business subscription.
od <- personal_onedrive()
odb <- business_onedrive(tenant="mycompany")
# use the device code authentication flow in RStudio Server
od <- personal_onedrive(auth_type="device_code")
# list files and folders
od$list_items()
od$list_items("Documents")
# upload and download files
od$download_file("Documents/myfile.docx")
od$upload_file("somedata.xlsx")
# create a folder
od$create_folder("Documents/newfolder")
You can open a file or folder in your browser with the open_item() method. For example, a Word document or Excel spreadsheet will open in Word or Excel Online, and a folder will be shown in OneDrive.
od$open_item("Documents/myfile.docx")
You can get and set the metadata properties for a file or folder with get_item_properties() and set_item_properties(). For the latter, provide the new properties as named arguments to the method. Not all properties can be changed; some, like the file size and last modified date, are read-only. You can also retrieve an object representing the file or folder with get_item(), which has methods appropriate for drive items.
od$get_item_properties("Documents/myfile.docx")
# rename a file -- version control via filename is bad, mmkay
od$set_item_properties("Documents/myfile.docx", name="myfile version 2.docx")
# alternatively, you can call the file object's update() method
item <- od$get_item("Documents/myfile.docx")
item$update(name="myfile version 2.docx")
SharePoint
To access a SharePoint site, use the sharepoint_site() function and provide the site URL or ID.
site <- sharepoint_site("https://myaadtenant.sharepoint.com/sites/my-site-name")
The client object has methods to retrieve drives (document libraries) and lists. To show all drives in a site, use the list_drives() method, and to retrieve a specific drive, use get_drive(). Each drive is an object of class ms_drive, just like the OneDrive clients above.
# list of all document libraries under this site
site$list_drives()
# default document library
drv <- site$get_drive()
# same methods as for OneDrive
drv$list_items()
drv$open_item("teamproject/plan.xlsx")
To show all lists in a site, use the get_lists() method, and to retrieve a specific list, use get_list() and supply either the list name or ID.
site$get_lists()
lst <- site$get_list("my-list")
You can retrieve the items in a list as a data frame, with list_items(). This has arguments filter and select to do row and column subsetting respectively. filter should be an OData expression provided as a string, and select should be a string containing a comma-separated list of columns. Any column names in the filter expression must be prefixed with fields/ to distinguish them from item metadata.
# return a data frame containing all list items
lst$list_items()
# get subset of rows and columns
lst$list_items(
filter="startsWith(fields/firstname, 'John')",
select="firstname,lastname,title"
)
Finally, you can retrieve subsites with list_subsites() and get_subsite(). These also return SharePoint site objects, so all the methods above are available for a subsite.
Future plans
Currently, Microsoft365R supports OneDrive and SharePoint Online; future updates will add the ability to post to Teams channels and send emails via Outlook. You can also provide feedback and make feature requests by opening an issue at the repo, or by emailing me at hongooi73 (@) gmail.com.
Last week, I announced AzureCosmosR, an R interface to Azure Cosmos DB, a fully-managed NoSQL database service in Azure. This post gives a short rundown on the main features of AzureCosmosR.
Explaining what Azure Cosmos DB is can be tricky, so here’s an excerpt from the official description:
Azure Cosmos DB is a fully managed NoSQL database for modern app development. Single-digit millisecond response times, and automatic and instant scalability, guarantee speed at any scale. Business continuity is assured with SLA-backed availability and enterprise-grade security. App development is faster and more productive thanks to turnkey multi region data distribution anywhere in the world, open source APIs and SDKs for popular languages. As a fully managed service, Azure Cosmos DB takes database administration off your hands with automatic management, updates and patching. It also handles capacity management with cost-effective serverless and automatic scaling options that respond to application needs to match capacity with demand.
Among other features, Azure Cosmos DB is notable in that it supports multiple data models and APIs. When you create a new Cosmos DB account, you specify which API you want to use: SQL/core API, which lets you use a dialect of T-SQL to query and manage tables and documents; MongoDB; Azure table storage; Cassandra; or Gremlin (graph). AzureCosmosR provides a comprehensive interface to the SQL API, as well as bridges to the MongoDB and table storage APIs. On the Resource Manager side, AzureCosmosR extends the AzureRMR class framework to allow creating and managing Cosmos DB accounts.
AzureCosmosR is now available on CRAN. You can also install the development version from GitHub, with devtools::install_github("Azure/AzureCosmosR").
SQL interface
The meat of AzureCosmosR is a suite of methods to work with databases, containers (tables) and documents (rows) using the SQL API.
library(dplyr)
library(AzureCosmosR)
# endpoint object for this account
endp <- cosmos_endpoint(
"https://myaccount.documents.azure.com:443/",
key="mykey"
)
# all databases in this account
list_cosmos_databases(endp)
# a specific database
db <- get_cosmos_database(endp, "mydatabase")
# create a new container and upload the Star Wars dataset from dplyr
cont <- create_cosmos_container(db, "mycontainer", partition_key="sex")
bulk_import(cont, starwars)
query_documents(cont, "select * from mycontainer")
# an array select: all characters who appear in ANH
query_documents(cont,
"select c.name
from mycontainer c
where array_contains(c.films, 'A New Hope')")
You can easily create and execute JavaScript stored procedures and user-defined functions:
Aggregates take some extra work, as the Cosmos DB REST API currently only has limited support for cross-partition queries. Set by_pkrange=TRUE in the query_documents call, which will run the query on each partition key range (physical partition) and return a list of data frames. You can then process the list to obtain an overall result.
# average height by sex, by pkrange
df_lst <- query_documents(cont,
"select c.gender, count(1) n, avg(c.height) height
from mycontainer c
group by c.gender",
by_pkrange=TRUE
)
# combine pkrange results
df_lst %>%
bind_rows(.id="pkrange") %>%
group_by(gender) %>%
summarise(height=weighted.mean(height, n))
Full support for cross-partition queries, including aggregates, may come in a future version of AzureCosmosR.
Other client interfaces
MongoDB
You can query data in a MongoDB-enabled Cosmos DB instance using the mongolite package. AzureCosmosR provides a simple bridge to facilitate this.
A good introduction to Azure Cosmos DB can be found here, or you can browse the official documentation. If you have any questions or feedback about the AzureCosmosR package, you can open an issue or email me at hongooi73 (@) gmail.com.
This is an update on what’s been happening with the AzureR suite of packages. First, you may have noticed that just before the holiday season, the packages were updated on CRAN to change their maintainer email to a non-Microsoft address. This is because I’ve left Microsoft for a role at Westpac bank here in Australia; while I’m sad to be leaving, I do intend to continue maintaining and updating the packages.
To that end, here are the changes that have recently been submitted to CRAN, or will be shortly:
AzureAuth now allows obtaining tokens for the “organizations” and “consumers” generic tenants, in addition to “common”. This should make it easier to authenticate users who are using a personal account, as opposed to a work & school account. This is now live on CRAN.
AzureStor gains convenience functions for transferring data in several commonly-used formats: RDS (as created by saveRDS/readRDS), RData (as created by save/load), CSV, CSV2, and tab-delimited. These work via connections and so don’t create temporary files on disk.
AzureRMR makes it easier to retrieve subresources by adding a get_subresource method to the az_resource class. For example, if res is a storage account resource, you can obtain the subresource for a particular blob container with res$get_subresource(type="blobServices/default/containers", name="containername"). It now authenticates using AAD v2.0 by default.
Like AzureRMR, it also now authenticates using AAD v2.0 by default, making it more convenient to use with personal accounts (which require AAD v2.0).
get_graph_login is much more flexible, letting you choose authentication parameters like the permission scope(s), app ID, and authorisation method.
The internals have undergone a refactoring to enhance extensibility.
All of these changes are put to good use in the new Microsoft365R package, which is intended to be an interface to Microsoft 365 (formerly known as Office 365). Currently, it enables you to access your data stored in OneDrive (personal or business) and SharePoint Online, using the Microsoft Graph API. This will be released to CRAN in the next few weeks; later versions may add support for other Microsoft 365 services.
The new AzureCosmosR package is an interface to Azure Cosmos DB. Like the other interface packages in AzureR, it provides both management plane and data plane access, in the latter case to the SQL (core) API. You can easily create and manage Cosmos DB accounts, run queries, and create and execute stored procedures. Like Microsoft365R, this should be released to CRAN in the next few weeks.
If you have any comments or feedback, feel free to drop me a line at hongooi73@gmail.com.
Azure Functions is a cloud service that allows you to deploy “serverless” microservices that are triggered by events (timers, HTTP POST events, etc) and automatically scale to serve demand while minimizing latency. The service natively supports functions written in C#, Java, JavaScript, PowerShell, Python and TypeScript, and now supports other languages as well thanks to the launch last week of custom handlers for Azure Functions.
A new tutorial walks you through the process of creating a custom handler for a “hello world” R function. The process is fairly straightforward: use a couple of Azure CLI commands to set up a project on your local machine and create Azure resources, write a “handler” script in R to provide a Web service, and push a Docker container with the Azure Functions runtime, R engine, and packages needed to implement your Function. Then, when you want to update your Function all you need to do is push a new version of the container image with the updated R code. The video below shows a brief demo of the process in action:
The tutorial uses the httpuv package to implement a stripped-down Web server to implement the Function, but you can make things easier for yourself (at a small cost to performance) by using the plumber package. With plumber, you can easily annotate an R function you already have, and make it into a web service suitable for Azure Functions.
In this GitHub repository you’ll find code that implements an Azure Function to predict from a GLM model trained with the caret package, and also a Shiny app to consume the function. Follow the instructions in the README.md file to deploy your Function, and then launch the associated Shiny app. As you adjust the parameters on the left side of the application, the chart on the right updates in real time with an estimate of the probability that a car accident with those parameters would be fatal.
It’s important to note that the model prediction is not being generated by the Shiny app: rather, it’s being generated by an Azure Function running R in the cloud. That means you could integrate the model estimate into any application written in any language: a mobile app, or an IoT service, or anything that can call an HTTP endpoint. Furthermore, you don’t need to worry how many apps are running or how often estimates will be requested by the app: Azure Functions will automatically scale to meet the demand as needed.
Thanks to plumber, the “handler function” providing the Web service is very simple. All I needed to do was to decorate an existing function that made a prediction based on parameters with a couple of comments to make it into an HTTP POST endpoint compatible with Azure Functions.
#* Predict probability of fatality from params in body
#* @post /api/accident
function(params) {
model_path <- "."
model <- readRDS(file.path(model_path, "model.rds"))
method <- model$method
message(paste(method, "model loaded"))
prediction <- predict(model, newdata=params, type="prob")[,"dead"]
return(prediction)
}
The only other trick is to create a container containing R and the packages to support plumber and your model, but if you’re familiar with Dockerfiles this should be quite straightforward.
If you’d like to try it out yourself, you’ll find complete instructions at the repository linked below.
I was my great pleasure yesterday to be a presenter in the "Why R Webinar" series, on the topic R at Microsoft. In the talk (which you can watch below) I recounted the history of Microsoft's acquisition of Revolution Analytics, and the various way the Microsoft supports R: its membership of the R Consortium, integration with many products (including demos of Azure ML Service with GitHub Actions, and Azure Functions), and how Microsoft has driven adoption of R in large organizations by making R "IT approved". Many thanks to the Why R Foundation for hosting the talk, and to everyone who asked questions and prompted a lively discussion at the end:
You can also find links and resources associated with the talk, including the slides, at the GitHub repository linked below.
