Upgraded with Azure Functions, Runbooks, and Webhooks!
These fucking vendors, man.
So some vendor saw this original article dated AUGUST OF 2022 and were apparently driven to some kind of epiphany on how they could use this concept to swindle as many gullible MSPs out of as many dollarie doos as possible. Well, we don’t stand for that nincompoopery in the HaloPSA community.
I have, admittedly, failed to protect you (rubes?) from predatory Kaseya wannabees by not updating the original script as Halo development has marched on. I sincerely apologize for my transgressions. So, as penance for my sins, I am updating the original script to be faster, better, stronger, prettier, and simple to deploy!
On with the show!
Old and Busted: HaloPSA Automation driven by Azure Runbooks
If you hang out in the HaloPSA Discord or on any of the MSP subreddits, you’ve probably seen me shill unapologetically for Azure Runbooks. I do this for good reason – Azure Automation Runbooks are highly accessible and very easy to work with. They are the best way to break into powerful automation techniques because they have a very low barrier to entry and very low skill ceiling. As you master basic multi-tenant automation, a few problems eventually become clear:
- They are slow to spin up on activation, sometimes taking multiple minutes to start, import modules, and run.
- Dependency management kind of sucks ass, despite runtime environments being released not long ago that were meant to overhaul dependency management.
- They have a maximum run duration of 3 hours. 3 hours sounds like a long time for a script to run, but some complicated delta syncs could take that long or much longer (see: Halo Integrator attempting to sync an RMM or CSP integration with thousands of entries).
- They have a maximum allocation of 400 MB of RAM, and the memory limit is what really kills it when you get into advanced automation. 400 MB of memory is not very much in the grand scheme of things and it’s really easy to exceed that even on simple tasks that have to iterate through large datasets or perform transformations.
New (old?) Hotness: HaloPSA automation driven by Azure Functions
These days I’m primarily a C# .NET Blazor developer and have largely neglected by PowerShell skills in favor of .NETs more powerful lower level languages. PowerShell is hands down the most accessible and powerful (I see what you did there, Microsoft) scripting language that is purpose built for IT professionals to automate IT processes with. The fact that it rides on top of .NET and is semantically very similar to C# is what made PowerShell a springboard for me to launch deeper into actual software development.
With that expanding software development experience came the expectation for my automations to be FAST AS FUCK BOIIIIII. So as I’ve had to go back and fix or update previous runbook based automations, I’ve found myself converting them to Azure Functions instead. Azure Functions are the perfect mix of PowerShell, webhook triggers, and always-on run speed. While a little bit more difficult to configure than a runbook, they offer a huge number of benefits:
- Can run on an app service, serverless compute, or a number of other microservice options in Azure. This means they don’t need to allocate, spin up, import modules, and then start running your script every time they execute. They are always on with dedicated compute, waiting for the webhook and ready to execute immediately on receipt. These options range from super cheap or free to crazy expensive, compared to the pennies most runbooks cost to run monthly.
- If you run your function on a Windows app service or function app, you can use automatic dependency management. This equates to just giving the app a list of modules and their major versions to include and the function app will handle the rest. For a Linux app service or function app, you can bundle your modules in the function when you deploy it. Either of these options is a massive improvement to the OG module management and new runtime environment dependency management in Azure Automation Runbooks.
- Extensive logging and log streaming options. Being able to watch the log stream in real time is a huge quality of life improvement over runbooks chunking data back to a log every 10-30 seconds. There’s also a really well fleshed out integrated testing console in Azure that allows you to simulate triggers and debug them right in the cloud.
- The ability to deploy multiple functions to the same app service means you can maximize the cost of that deployment by running all your PowerShell automation there in an easier to manage hierarchy.
Have I convinced you yet? No? Well, enjoy paying $30k a year to run your SMS verification in Rewst I guess.
Prerequisites
- An Azure Subscription
- A Twilio Account
- GDAP permissions correctly configured in your tenant
- Your users must have SMS set up as either a backup authentication method OR as an SSPR method (Entra is configured this way by default, so unless you’ve specifically configured your client tenants to not allow this, you don’t need to make any changes)
- Cyberdrain Improved Partner Portal (CIPP) – optional, but makes consenting our app registration much easier
Step 1: Configure the App Registration in Entra
- Head to your home (MSP) tenant in Entra where you have GDAP admin relationships associated to your client tenants.
- On the left bar click Applications -> App Registrations -> New Registration
- Name it anything you like, but keep in mind it will be visible in your client tenants, too. I don’t recommend calling it “KASEYA SUCKS” but you totally can, if that’s your thing.
- Set the type to “Accounts in any organizational directory (Any Microsoft Entra ID tenant – Multitenant).”
- Leave the redirect boxes blank and click “Register”
- Next, click on “Add MPN ID to verify publisher” which will take you to the branding and properties screen where you can again click “Add MPN ID to verify publisher.” You can get this information from partner.microsoft.com -> click the gear in the top right corner -> account settings -> identifiers -> Microsoft Cloud AI Partner Program -> Global partner code. This makes your app registration show up as verified. While Microsoft isn’t enforcing the text in the orange box currently (despite saying it went into effect in 2020), someday they are going to figure out they forgot to do it and lots of app registrations will break. Don’t be that guy.
*Note: The first time you do this you may receive an error. Give it a Microsoft Minute and try again in about an hour. It should work.
- Then we’ll head over to API Permissions on the left and add the two we specified above. Remove the delegated User.Read permission that comes out of the box. Add Microsoft Graph -> Application Permissions -> User.Read.All and UserAuthenticationMethod.Read.All. Then Grant Admin Consent.
Optional Sub Step: Consent the App Registration in your client tenants with CIPP (Thanks Kelvin <3)
If you use CIPP (and if you don’t, what are you waiting for?), you can navigate to the Overview tab of your new app registration, copy the Application ID, and go over to CIPP -> Tools -> Tenant Tools -> Application Approval. Leave the selection on all tenants -> click next -> paste in your App ID -> check the toggle to copy permissions from the current app -> click next -> click submit. Your application registration will be installed and consented in all your client tenants.
If you’re a heathen and don’t use CIPP, you will need to manually consent this application registration in all your client tenants for it to work. That’ll be annoying as fuck. Enjoy.
Step 2: Set up certificate authentication
Why certificate authentication? Because Microsoft fucking said so, that’s why. In order to authenticate correctly against our app registration and invoke the full power of our GDAP permissions, we need to connect to Graph using certificate authentication. This is not only the most secure method, it’s also smart and convenient. Let’s walk through the process to create a self signed key-pair to be able to authenticate with.
Fire up PowerShell on your local computer and paste the following command, updating the subject name and years to your liking:
$Certificate=New-SelfSignedCertificate –Subject automation.kaseyasucks.com -CertStoreLocation Cert:CurrentUserMy -NotAfter (Get-Date).AddYears(5) Next, we need to export TWO different types of certificates from this. One is a .CER that contains only the public key – this one will be uploaded to Entra. The second will be a PFX including the private key – that will be uploaded to our Key Vault later.
Export-Certificate -Cert $Certificate -FilePath "C:certname.cer" Next, lets export the PFX:
$Pwd = ConvertTo-SecureString -String "FredVCanGargleMuhBallz" -Force -AsPlainText
Export-PfxCertificate -Cert $Certificate -FilePath "C:certname.pfx" -Password $Pwd Then we head over to our App Registration in Entra -> Certificates and Secrets -> Certificates -> Upload Certificate
Step 3: Set up the HaloPSA Runbook and Webhook
The HaloPSA setup is rather straightforward. It consists of two components. An outgoing webhook to trigger the Azure function, and an incoming webhook for HaloPSA to receive the data from our function on the way back in. We’ll start by setting up the incoming runbook.
Incoming Runbook Configuration
First, navigate to Configuration -> Integrations -> Custom Integrations -> Custom Integration Runbooks -> New. Name it anything you like and save it. Then edit again and click the “Import from JSON” button. Paste the below:
{
"id": null,
"name": "SMS User Verification for Twilio (MSP Automator)",
"type": 1,
"content_type": "application/json",
"authentication_type": 0,
"method": 0,
"certificate_id": 0,
"certificate_name": "",
"active": true,
"events": [],
"last_status": 1,
"systemuse": "",
"runbook_start_type": 1,
"inbound_authentication_type": 0,
"algorithm": 0,
"digest": 0,
"custom_payload": false,
"payload_type": 0,
"library_licence_name": "",
"major_version_number": 0,
"minor_version_number": 0,
"patch_version_number": 0,
"version_number": "0.0.0",
"note": "MSPAutomator.com",
"steps": [
{
"step_id": 1,
"flow_id": 0,
"chatprofile_id": null,
"name": "Step 1",
"isstart": true,
"isend": false,
"islaststep": false,
"stage_number": 0,
"pipeline_stage_name": "",
"actions": [
{
"id": null,
"flow_id": 0,
"chatprofile_id": null,
"start_step": 1,
"end_step": 2,
"action_type": 18,
"action_id": -18,
"action_name": "Successful",
"action_outcome": "",
"use_work_hours": true,
"time_limit_action_name": "",
"automation_action_name": "",
"automation_runbook_name": "",
"seq": 1,
"approval_result": 1,
"restricted": false,
"conditions": [],
"conditions_exec": [],
"restrictions": [],
"todo_group_name": "",
"chat_selection_order": 1
},
{
"id": null,
"flow_id": 0,
"chatprofile_id": null,
"start_step": 1,
"end_step": 3,
"action_type": 18,
"action_id": -18,
"action_name": "Unsuccessful",
"action_outcome": "",
"use_work_hours": true,
"time_limit_action_name": "",
"automation_action_name": "",
"automation_runbook_name": "",
"seq": 2,
"approval_result": 0,
"restricted": false,
"conditions": [],
"conditions_exec": [],
"restrictions": [],
"todo_group_name": "",
"chat_selection_order": 1
}
],
"steptype": 2,
"message": "{n "ticket_id": <<SMSTicketID>>,n "outcome": "Private Note",n "who": "Your MSPs Automation Services",n "hiddenfromuser": true,n "note_html": "<<VerificationResponse!>>"n}n",
"auto_action": 8,
"auto_action_type": 3,
"input_field_id": 0,
"chat_image_type": 0,
"newticket_service_id": 0,
"start_new_chat_flow_id": "",
"iteration_type": 0,
"iteration_batch_size": 1,
"output_variables": [],
"runbook_variable_mappings": []
},
{
"step_id": 2,
"flow_id": 0,
"chatprofile_id": null,
"name": "Step 2",
"isstart": false,
"isend": true,
"islaststep": false,
"stage_number": 0,
"pipeline_stage_name": "",
"actions": [],
"steptype": 3,
"auto_action": 0,
"input_field_id": 0,
"chat_image_type": 0,
"newticket_service_id": 0,
"start_new_chat_flow_id": "",
"iteration_type": 0,
"iteration_batch_size": 1
},
{
"step_id": 3,
"flow_id": 0,
"chatprofile_id": null,
"name": "Step 3",
"isstart": false,
"isend": true,
"islaststep": false,
"stage_number": 0,
"pipeline_stage_name": "",
"actions": [],
"steptype": 3,
"auto_action": 1,
"input_field_id": 0,
"chat_image_type": 0,
"newticket_service_id": 0,
"start_new_chat_flow_id": "",
"iteration_type": 0,
"iteration_batch_size": 1
}
],
"input_variables": [
{
"method_id": 0,
"type": 3,
"data_type": 3,
"key": "SMSTicketID",
"value": "<<request^TicketID!>>",
"value_mappings": [],
"extra_process": 0,
"step_id": 0,
"step_name": "",
"mapping_type": 0
},
{
"method_id": 0,
"type": 3,
"data_type": 2,
"key": "VerificationResponse",
"value": "<<request^VerificationResponse!>>",
"value_mappings": [],
"extra_process": 0,
"step_id": 0,
"step_name": "",
"mapping_type": 0
}
],
"disabled": false,
"log_retention_policy_days": 30,
"batch_method": 0,
"batch_delay_seconds": 30,
"batch_limit": 0,
"infinite_loop_threshold": 5,
"access_control": null
}Important: When you save this JSON, a URL will appear that can be used to start the runbook externally. This is the webhook URL for the runbook that you will be required to enter in Step 4 when deploying the function app.
HaloPSA Outgoing Webhook Configuration
Next, we’ll create the outgoing webhook that triggers the SMS verification. You won’t have a URL yet, so just type anything into the URL box for now. Set up the custom payload exactly as depicted here:
Create an action in HaloPSA to trigger this webhook
Mosey on over to Configuration -> Tickets -> Actions and create a new action. Set it up like this:
Add this action to any workflow(s) you want to make it accessible to your agents, and now we can move on to the fun part!
Step 4: Deploy the Function App
You have two choices – deploy the function app manually using VSCode and the Azure Functions Extension (Find the solution and deployment instructions here in my Github)
Or, just Deploy to Azure using my template by clicking the shiny blue button below
The ARM template will ask you to fill in the following information:
- A name for the Keyvault that will be created
- A name for the function app that will be created
- Your MSP Name (used in the outbound SMS message)
- Your Twilio SID
- Your Twilio Token
- Your Twilio send from number in E.164 format (e.g. +18008675309)
- Your Halo Runbook Webhook URL
- The name you plan to upload the PFX to the keyvault as (we will do this in the next step – just fill something in here and remember what you entered)
- The App Registration ID for the application we made in Step 1
Step 5: Upload the PFX to the Keyvault
Part of the deployment in step 4 created a keyvault to hold secrets and prepopulated a bunch of secrets from the information you entered as part of the template. One thing it didn’t do is upload the PFX. Remember when I told you to remember what you entered in the template for certificate name? This is where that matters.
Navigate to the Azure portal -> the new Keyvault that was created -> Objects -> Certificates and click “Generate/Import” on the top bar.
Select “Import” from the dropdown menu instead of “Generate” and select your PFX we made earlier. Be sure to name the certificate whatever you entered in the template during deployment or nothing we have done will work.
Step 6: Retrieve the Function App URL and populate the HaloPSA Webhook
Navigate to the function app in Azure that was created as part of deployment. On the Overview screen you’ll see a URL. This is the base URL for your function app.
Now that we know the default domain of our function app, we can go back to the webhook we set up previously in HaloPSA and enter it in the URL box.
https://<yourfunctionappname>.azurewebsites.net/api/HttpTrigger1?
Setup is now complete. Now the fun part.
Testing the function app in HaloPSA
Click the button you added to the workflow and watch the magic happen.
The original incarnation of this script when operated via Azure Runbooks took up to 1-3 minutes to send the SMS and return the data to Halo. This new function app takes about 3-5 seconds from start to finish. I would say that’s a pretty damn good improvement. In fact, I’d wager that this is significantly faster than any of the paid products on the market.
So please folks, don’t pay some scumbag vendor for SMS identity verification. Use this fully functional, extremely low cost, and totally fast option instead.
As always, happy automating!