Custom connector to rest application - Sharepoint access broken

[scoobertismycat]

Guys,

I created a custom http connector to a python (REST in its most simplified form) web application, that accesses a few Sharepoint sites. It was working fine for about 3 years until recently. The application was broken due to new MFA requirement, and I was no longer able to make read and update requests to Sharepoint from python, specifically from code utilizing the office365 REST library.

I originally set the authorization up 3 years ago w old MS Azure ACS antiquated access.  I knew that I needed to make some Auth changes, I just didn't know exactly what I needed to do to get it back to normal. I initially thought I would need to try to re-write the whole application in Power Automate by paying for access to a third party connector, or possibly just recoding the SP utility class I have with standard http request code, but I was able to get it to work finally. I just want to share with you what I learned in case someone else runs into the same issue.

Cheers,

Scoobert

[scoobertismycat]

Admittedly, I for sure am no MS Azure expert. I really wasn't exactly sure what I need to do to fix my issue, but I knew that I needed to make changes in MS Azure.

This post is for people like me that don't know anything about MS Azure, just want to get things working again, and are confused about the differences inherent in MS Azure ACS, MS Azure AD or Entrata.

I researched for a while and tried some things from the MS Azure forum, and was not able to get anything to work.

Each failed attempt got me a little closer to what I needed to do though. I found out that I originally configured my application access to Sharepoint through MS Azure ACS. And I found out that MS Azure ACS has been retired, and that MFA was being pushed as the application access standard. That standard in relation to MS TEAMS and Sharepoint was now being handled by MS Azure AD or even more recently MS Entrata.

According to MS: 

"MS Azure ACS has been retired, and Azure AD is the recommended replacement for managing authentication and authorization in Azure-based applications." 

I'll go over the old way that I used initially to setup application access, and then go into the most helpful resources I found that got it fixed. BTW, all of this stuff regarding MS Azure AD assumes you or someone you know has admin access to MS Azure AD.

OLD WAY TO SET PERMISSIONS – MS AZURE ACS 

Within MS Azure ACS there were 2-3 older MS Azure ACS admin pages that were required to be filled out to make the external application using username-password be able to access a Sharepoint site's files and lists (keep in mind contoso is just the MS test domain, you would replace it with your sharepoint domain, if you were to actually do this); 

1. Register a new application (principal) – with this link below you could register a new application with MS Azure ACS. What this means is that you are creating a representation of your application, within MS Azure ACS with the aim of ultimately providing this application registration with Authentication and Authorization to MS Resources. 

https://contoso.sharepoint.com/_layouts/15/appregnew.aspx 

1. Grant permissions to the newly created application (principal) – with this link below you would paste in some xml based permissions to grant the access needed to the application (principal). 

https://contoso-admin.sharepoint.com/_layouts/15/appinv.aspx 

1. Trust the application (principal) - Once you click on "Create" in the previous screen, you will be taken to another screen that will ask you to "Trust" this application by clicking the "Trust" button.  

For more detailed information on these urls used with MZ Azure ACS, which again has been retired, please go to; 

https://learn.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azureacs 

In this way, you would have created access for the application to a MS Azure or in my case a Sharepoint recource, and the way it was done here was with UID and PWD, which of course is not ideal, but anyway... 

Even though this is no longer the way we do this, it is worth mentioning, because when you search for documents and sites to help you setup application access to a Sharepoint resource from an external application, sometimes people mix up the old MS Azure ACS way and the new MS Azure AD (Entrata) way, and if you aren't careful, you may have a very difficult time getting it to work. 

NEW WAY TO SET PERMISSIONS – MS AZURE AD (ENTRATA)

This article below is a good starting point, and as you will see is important in how to access specifically selected Sharepoint sites, thank you very much to the Author of this article, RoboEase, its super detailed and excellent; 

Seamless Uploads and Downloads of files on SharePoint with Site.Selected API Permission via Microsof... 

This MS article has the script to create a self-signed certificate, thanks go to Microsoft for including the script to create the self-signed cert; 

Granting access via Azure AD App-Only | Microsoft Learn 

For even more information please watch the following youtube video, Thank You @GracielaPowerGI (twitter), this is excellent and she makes it very easy; 

SharePoint API set up via Azure AD App-Only & Connection from Power Automate 

Mostly Everything you need to know is in the above video and linked articles, but here is a short synopsis of setting up access with MA Azure AD to a Sharepoint resource from an external python application using a certificate for Authentication; 

1. Login to the MS Azure AD (Entrata) Portal 
2. Register a new application
3. Create a self-signed certificate for your organization - https://learn.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread - create a self-signed certificate using the script from this link by opening Windows Powershell and choose run as administrator. This will create a .cer certificate file and a .pfx file which is a combination of a public and private key file.  
4. Create private key (.pem) from the .pfx file – With MS Powershell still open as an administrator, install the program PSPKI (command is Install-Module -Name PSPKI). You will need to import this program into Powershell (Import-Module PSPKI). This module has a program called Convert-PfxToPem (command is Convert-PfxToPem -InputFile ./<cert_name>.pfx -Outputfile ./<cert_name>.pem), run the command to create a .pem file. You need to make sure to provide the location of this file to your application. Keep in mind, this PSPKI application is not native to Powershell, if you do not want a third party app to create this cert, google that. 
5. In MS Azure AD go to Request API Permissions – in addition to Microsoft Graph permissions, make sure to also specifically select Sharepoint like Graciela does in the video on the permissions page and add Sites.FullControl.All just for now. If you don't add this Permission, nothing you do, or at least nothing I did after will allow you to access Sharepoint Resources from an external app. I was not able to access anything in Sharepoint, until I added Sharepoint specific permissions, maybe others have not run into this problem. Keep in mind this Sites.FullControl.All permission is only temporary, we will configure the permissions later to access only the sites we want to access, NOT all sites. 
6. In MS Azure AD go to Certificates and Secrets, add certificate – upload the .cer certificate file you just created. Verify this is done by inspecting the Manifest (link on the left side in MS Azure AD), and look for the "keyCredentials" section exists. 
7. Inform MS Azure AD (Entrata) Principal (application) which specific Sharepoint sites we need access to – using this Article Seamless Uploads and Downloads of files on SharePoint with Site.Selected API Permission via Microsof..., walk through the process using the postman application (an HTTP request based application) to submit requests to gain Read and Write access to only the Sharepoint sites you need. There are 3 main http requests that need to be made for this, and you'll need to execute them for each Sharepoint site you need access to. In postman, I created a collection and added all of the requests to the collection so I'll have them organized going forward. 
8. In MS Azure AD (Entrata) Change API Permissions to be Sites.Selected NOT Sites.FullControl.All - Now one of the important things we need to do is scale back the permissions to access only the sites the application needs to access. Go back to MA Azure AD, select the registered principal (application), choose API Permissions, and revoke access to Sites.FullControl.All under MS Graph and Sharepoint. Now make sure each of those sections has Sites.Selected granted with Admin Consent. 
9. Now, test the permissions, to see if you can access the Sharepoint sites you need. For python you could run something like what you see below. The thumbprint value seen below can be found in MS Azure AD. As you can see I am setting the site_url directly to the site or sites I need to access.

As a last thing, Ideally I want to only access 1 list and 1 folder in each site with read and write, so If I could restrict my access within those sites to only those resources, that would be even better. But I am not there yet. If anyone has any clue how to do that, please let me know!

As always if you see anything I forgot, or is wrong let me know and I will fix it.

Cheers,

Scoobert

[scoobertismycat]

Ok, today is another day!  Now its broken again. Yesterday my test code worked perfectly, everything was great. Now today, getting this, I added the "zzz" for obfuscation:

"ValueError: {'error': 'invalid_resource', 'error_description': 'AADSTS500011: The resource principal named https://zzz.sharepoint.com/zzz/zzz was not found in the tenant named zzzzz.. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You might have sent your authentication request to the wrong tenant. Trace ID: blah Correlation ID: blah Timestamp:
2024-07-21 00:30:39Z', 'error_codes': [500011], 'timestamp': '2024-07-21 00:30:39Z', 'trace_id': 'blah', 'correlation_id': 'blah', 'error_uri': 'https://login.microsoftonline.com/error?code=500011'}"

Again, it worked great yesterday. I did remove the Sites.FullControl.All permission and left only Sites.Selected, and again it was fine.

I just put back the Sites.FullControl.All permission and still it failed, but maybe it needs time to propogate?

If anyone has any clue why this happened and what I can do, please let me know, this is driving me crazy!

[scoobertismycat]

Again, today is another day. Its working again. I'm not sure what is going on with this stuff. I'll update again if something changes, weird.