Azure Virtual Machine Scale Set Agents - Azure Pipelines (2023)

Azure DevOps-Dienste

Azure Virtual Machine Scale Set Agents, hereafter referred to as Scale Set Agents, are a form of self-hosted agents that can be automatically scaled to meet your needs. This flexibility reduces the need to constantly run dedicated agents. Unlike Microsoft hosted agents, you have flexibility in the size and image of the machines running the agents.

If you like Microsoft hosted agents but are limited by what they offer, consider scaling agents. Here are some examples:

• You need more memory, more processor, more storage space or more I/O than what we offer in native Microsoft hosted agents.
• You need an NCv2 VM with specific machine learning instruction sets.
• You need to deploy a private Azure App Service in a private VNet with no inbound connectivity.
• You must open the corporate firewall to specific IP addresses in order for Microsoft-hosted agents to communicate with your servers.
• You must restrict agent computers' network connectivity and allow them to reach only approved sites.
• You cannot get enough agents from Microsoft to meet your needs.
• Your jobs on the Microsoft hosted agent are timing out.
• You cannot separate Microsoft-hosted parallel jobs into individual projects or teams in your organization.
• You want to run multiple consecutive jobs on an agent to take advantage of the machine-level and source incremental package caches.
• You want to perform a cache configuration or warm-up before an agent starts accepting jobs.

If you like self-hosted agents but want to simplify managing them, consider scale set agents. Here are some examples:

• You don't want to run dedicated agents all day. You want to deprovision agent machines that are not being used to run jobs.
• You are running untrusted code in your pipeline and want to reimage agent machines after each job.
• You want to make it easy for your agents to regularly update the base image.

Create the scale set

To prepare to create scale set agents, you must first create a virtual machine scale set in the Azure portal. You must create the virtual machine scale set in a specific way so that Azure Pipelines can manage it. In particular, you mustdeactivate automatically scaledto allow Azure Pipelines to determine how to scale based on the number of incoming pipeline jobs. We recommend that you follow the steps below to create the scale set.

The following example creates a new resource group and a new virtual machine scale set with Azure Cloud Shell using the UbuntuLTS VM image.

1. navigate toblue cloud shellInhttps://shell.azure.com/.

2. Run the following command to verify your default Azure subscription.

   List of accounts az -o table

   If the subscription you want isn't listed as the default, select the subscription you want.

   az account set -s <your subscription ID>

3. Create a resource group for your virtual machine scale set.

   az group create \--location westus \--name vmssagents

4. Create a virtual machine scale set in your resource group. This example specifies the UbuntuLTS virtual machine image.

   az vmss create \--name vmssagentspool \--resource-group vmssagents \--image UbuntuLTS \--vm-sku Standard_D2_v3 \--storage-sku StandardSSD_LRS \--authentication-type SSH \--generate-ssh-keys \ --instance-count 2 \--disable-overprovision \--upgrade-policy-mode manual \--single-placement-group false \--platform-fault-domain-count 1 \--load-balancer ""

   Use

   Azure Pipelines doesn't support a scale setoversupplyjautomatically scaled. Make sure both features are disabled for your scale set.

   Because Azure Pipelines manages the scale set, the following configurations are required or recommended:

   • --disable-overprovision- necessary
   • --update-policy-manual-Modus- necessary
   • --load balancer ""- Azure Pipelines does not require a load balancer to forward jobs to agents in the scale set agent group, but configuring a load balancer is a way to get an IP address for the scale set agents that you can use for firewall rules. Another option to get an IP address for your scale set agents is to create your scale set using--public IP addressOptions For more information about configuring your scale set with a load balancer or public IP address, seeVirtual Machine Scale Sets documentationjaz vmss crear.
   • --instance count 2- This configuration is not required, but gives you an opportunity to verify that the scale set is fully functional before creating an agent group. The creation of the two virtual machines can take several minutes. Later, when you create the agent group, Azure Pipelines deletes those two VMs and creates new ones.

   Important

   If you are running this script using the Azure CLI on Windows, you must use the""In--load balancer ""with single quotes like this:--load balancer '""'

   If your virtual machine size is compatibleephemeral operating system disks, the following parameters for enabling ephemeral operating system disks are optional but recommended to reduce virtual machine image creation times.

   • --ephemera-os-disk true
   • --os-disk-caching Read only

   Important

   Volatile OS disks are not supported for all VM sizes. For a list of supported virtual machine sizes, seeVolatile operating system disks for Azure VMs.

   Choose any Linux or Windows image, either from the Azure Marketplace or your own custom image to create the scale set. Do not preinstall the Azure Pipelines agent in the image. Azure Pipelines automatically installs the agent when you provision new VMs. In the example above we use a simple oneUbuntuLTSPicture. For instructions on creating and using a custom image, seefrequently asked Questions.

   (Video) EVERYTHING About the Scale Set Agents for Azure Pipelines (VMSS)

   Choose any VM SKU and Storage SKU.

   Use

   Licensing considerations limit us in distributing Microsoft hosted images. We cannot provide you with these images for use in your scale set agents. Butdashesthat we use to generate these images are open source. You can use these scripts and create your own custom images.

5. After creating your scale set, navigate to your scale set in the Azure portal and verify the following settings:

   • Update Policy - Manual

     

     You can also verify this setting by running the following Azure CLI command.

     See Also

     Networking for Azure Virtual Machine Scale Sets - Azure Virtual Machine Scale SetsAzure Virtual Machine Scale Sets FAQ - Azure Virtual Machine Scale SetsAzure virtual machine scale set orchestration modes - Azure Virtual Machine Scale Sets

     az vmss show --resource-group vmssagents --name vmssagentspool --output tableName ResourceGroup Ubicación Zones Capacidad Sobreaprovisionamiento UpgradePolicy-------------- ------------- -- ---------- ------- ---------- ---------- ------ ---------vmssagentspool vmssagents westus 0 Falso Handbuch

   • Scale - Manual Scale

     

Create the agent group of the scale set

1. Navigate to your Azure DevOpsproject settings, chooseagent groupslowpipingand selectAdd groupto create a new agent group.

   

   Important

   You can create your scale set group inproject settingsÖOrganization Settings, but if you delete a scale set group, you must delete it fromOrganization Settings, and notproject settings.

2. ChooseAzure virtual machine scale setfor the type of pool. Chooseblue subscriptionthat contains the scale set, select itAuthorizeand select the virtual machine scale set you want from that subscription. If you have an existing oneservice connection, you can select it from the list instead of the subscription.

   Important

   • To configure an agent group for scale sets, you needownerÖuser access managerPermissions for the selected subscription. However, if you have one of these permissions, you will receive an error message when you select itAuthorize, verTroubleshooting.

   • The only currently supported service connection is an Azure Resource Manager (ARM) service connection based on a service principal key. ARM service connections based on certificate attestation or managed identity fail. If you try to list the existing scale sets in your subscription, you'll get an error like this:

     Invalid service endpoint with id <guid> and scope <guid>

3. Select the virtual machine scale set you want from this subscription.

4. Provide a name for your agent group.

5. Set the following options:

   • Automatically destroy virtual machines after each use- A new VM instance is used for each job. The VM goes offline after running a job and is reimaged before another job continues.
   • Storage of an unhealthy agent for research- Whether to savefailed agent virtual machinesto fix problems instead of eliminating them.
   • Maximum number of virtual machines in the scale set- Azure Pipelines automatically scales the number of agents, but does not exceed this limit.
   • Number of agents to hold- Azure Pipelines automatically scales the number of agents, but ensures there are always enough agents available to run new jobs. if you hireNumber of agents to holdA0For example, to keep costs down for a low volume of work, Azure Pipelines only starts a VM when it has a work order.
   • Delay in minutes before excess active ingredients are removed- To account for variations in build load throughout the day, Azure Pipelines waits the specified duration before removing an excess idle agent.
   • Configure virtual machines to run interactive tests(Windows Server OS only): Windows agents can be configured to run without privileges with automatic login and an interactive user interface, or they can be configured to run with elevated privileges. Check this box to run the interactive user interface without elevated layers. In both cases, the agent user is a member of the Administrators group.

6. When your settings are configured, selectCreateto create the agent group.

Use the agent group of the scale set

Using an agent group for scale sets is similar to using any other agent group. You can use it in classic build, release, or YAML pipelines. User permissions, pipeline permissions, approvals, and other checks work the same as any other agent group. For more information, seeagent groups.

How Azure Pipelines manages the scale set

After the agent group for the scale set is created, Azure Pipelines automatically scales the agent machines.

Azure Pipelines checks the health of the agents in the pool and the VMs in the scale set every 5 minutes. The decision to scale up or out is based on the number of currently idle agents. An agent is considered idle when it is online and not running a pipeline job. Azure Pipelines performs a scale-out operation when any of the following conditions are true:

• The number of idle agents is less than the number of waiting agents you specified
• No idle agents to service pipeline jobs waiting in queue

If any of these conditions are true, Azure Pipelines increases the number of virtual machines. Horizontal scaling occurs in increments of a specified percentage of the maximum pool size. Wait 20 minutes for machines to be created for each step.

Azure Pipelines escalates to agents when the number of idle agents exceeds the standby count for more than 30 minutes (configurable withDelay in minutes before excess active ingredients are removed).

To illustrate all of this in an example, consider a scale set agent group configured with two agents in standby mode and at most four agents. Suppose you want to shut down the virtual machine after each use. Also, assume that there are no VMs in the scale set to start with.

• Because the idle agent count is 0 and the idle agent count is less than the standby count of 2, Azure Pipelines is scaled out and two VMs are added to the scale set. Once these agents are online, two agents are idle.

• Suppose a pipeline job arrives and is assigned to one of the agents.

• Currently, the number of idle agents is 1, which is less than the number of standby agents, which is 2. Therefore, Azure Pipelines is scaled out and two more VMs are added (the increment size used in this example). At the moment the group has three free agents and one employed agent.

• Let's say the first agent's job is complete. Azure Pipelines takes this agent offline to reimage this machine. After a few minutes he comes back with a fresh picture. We currently have four Down agents.

• If no other order arrives for 30 minutes (configurable viaDelay in minutes before excess active ingredients are removed), Azure Pipelines determines that more agents are down than necessary. So he escalates the group to two agents.

During this process, the goal of Azure Pipelines is to reach the desired number of idle waiting agents. Clusters slowly scale up and down. Over the course of a day, the pool grows as requests are queued in the morning and grows as the load eases in the evening. You may see more idle agents than you want at different times, which is to be expected as Azure Pipelines begins to meet the limits you specify.

For maximum stability, scale set operations run sequentially. For example, if the pool needs to be scaled out and also unhealthy machines need to be removed, Azure Pipelines scales out the pool first. Once the pool has been scaled to reach the desired number of idle standby agents, the failed machines are removed based onStorage of an unhealthy agent for researchAttitude. For more information, seeunhealthy means.

Because of the 5-minute sample size, it's possible that all agents will be pipelined for a short period of time and not scale out.

Customize pipeline agent configuration

You can customize the Azure Pipelines agent configuration by setting environment variables in your custom OS image for your scale set. For example, the default working directory of the scale set agent is C:\a for Windows and /agent/_work for Linux. If you want to change the working directory, set an environment variable called VSTS_AGENT_INPUT_WORK to the desired working directory. For more information, see theUnattended configuration of the pipe agentDocumentation. Some examples are:

• VSTS_AGENT_INPUT_WORK
• VSTS_AGENT_INPUT_PROXYURL
• VSTS_AGENT_INPUT_PROXYUSERNAME
• VSTS_AGENT_INPUT_PROXYPASSWORD

Customize virtual machine startup via custom script extension

Users may want to run startup scripts on their scaleset agent computers before those computers start running pipeline jobs. Some common use cases for startup scripts are installing software, warming up caches, or checking out repositories. You can run startup scripts by using theCustom script extension for WindowsÖCustom script extension for Linux. This extension runs on all VMs in the scale set immediately after the image is created or rebuilt. The custom script extension runs before the Azure Pipelines agent extension runs.

Here is an example of creating a custom script extension for Linux.

az vmss extension set \--vmss-name <nombre de escalado> \--resource-group <group de recursos> \--name CustomScript \--version 2.0 \--publisher Microsoft.Azure.Extensions \--settings ' { \ "fileUris\":[\"https://<myGitHubRepoUrl>/myScript.sh\"], \"commandToExecute\": \"bash ./myScript.sh /myArgs\" }'

Here's an example of creating a custom script extension for Windows.

az vmss extension set \--vmss-name <Name der Escalado> \--resource-group <Rekursgruppe> \--Name CustomScriptExtension \--Version 1.9\--Publisher Microsoft.Compute \--settings '{ \ "FileUris \":[\"https://<myGitHubRepoUrl>/myscript.ps1\"], \"commandToExecute\": \"Powershell.exe -ExecutionPolicy Unrestricted -File myscript.ps1 -myargs 0 \" }'

Life cycle of a scale set agent

This is the flow of operations for an Azure Pipelines virtual machine scale set agent

1. The Azure DevOps scale set agent pool sizing job finds that the pool has too few idle agents and needs to be scaled out. Azure Pipelines invokes Azure scale sets to increase scale set capacity.

2. Azure Scale Set starts creating the new VMs. After the virtual machines are running, Azure Scale Sets runs all installed virtual machine extensions one at a time.

3. If the custom script extension is installed, it runs before the Azure Pipelines agent extension. If the custom script extension returns a non-zero exit code, the VM creation process is aborted and deleted.

4. The Azure Pipelines agent extension is running. This extension downloads the latest version of the Azure Pipelines agent along with the latest version of the setup script. Configuration scripts can be found under URLs in the following formats:

   • Linux:https://vstsagenttools.blob.core.windows.net/tools/ElasticPools/Linux/<script_version>/enableagent.sh, For example,version 14
   • Window:https://vstsagenttools.blob.core.windows.net/tools/ElasticPools/Windows/<script_version>/enableagent.ps1, For example,version 16

5. The configuration script creates a local user namedAzDevOpsif the operating system is Windows Server or Linux. For the Windows 10 client operating system, the agent runs as LocalSystem. The script unpacks, installs, and then configures the Azure Pipelines agent. As part of the configuration, the agent registers with the Azure DevOps agent group and appears in the agent group list in the offline state.

6. In most scenarios, the setup script immediately starts the agent to run as a local user.AzDevOps. The agent connects and is ready to run pipeline jobs.

   If the pool is configured for an interactive user interface, the virtual machine restarts after the agent is configured. After a restart, the local user is automatically logged in and the pipeline agent starts. The agent then connects and is ready to run pipeline jobs.

   (Video) Learning How to Set Up an Azure Virtual Machine Scale Set Agents for Azure DevOps

Create a scale set with a custom image, software, or disk size

If you just want to create a scale set using the standard 128GB OS disk with a publicly available Azure image, skip to step 10 and use the public image name (UbuntuLTS, Win2019DataCenter, etc.) to create the scale set . If not, follow these steps to customize your virtual machine image.

1. Create a virtual machine with the operating system image of your choice and optionally expand the operating system disk size from 128 GB<miTamañoDiscoGb>.

   • If you start with an available Azure image, e.g. B. <myBaseImage> = (Win2019DataCenter, UbuntuLTS):

     az vm create --resource-group <myResourceGroup> --name <MyVM> --image <myBaseImage> --os-disk-size-gb <myDiskSize> --admin-username meinBenutzername --admin-password meinPasswort

   • If you start with a generalized VHD:

     1. First create the virtual machine with an unmanaged disk of the desired size, then convert it to a managed disk:

        az vm create --resource-group <myResourceGroup> --name <MyVM> --image <myVhdUrl> --os-type windows --os-disk-size-gb <myDiskSizeGb> --use-unmanaged-disk -- Admin-Benutzername <miNombreDeUsuario> --admin-password <miContraseña> --storage-account <miCuentaAlmacenamientoVhd>

     2. Shut down the virtual machine

        az vm stop --resource-group <myresourcegroup> --name <myVM>

     3. Detach the virtual machine

        az vm desalocate --resource-group <myResourceGroup> --name <MyVM>

     4. Convert to managed disk

        az vm convert --resource-group <myresourcegroup> --name <myVM>

     5. Restart the virtual machine

        az vm start --resource-group <myresourcegroup> --name <myVM>

2. Remote Desktop (or SSH) with the VM's public IP address to customize the image. You may need to open ports in the firewall to unblock the RDP (3389) or SSH (22) ports.

   1. Window- And<MiTamañoDiscoGb>is larger than 128GB, please extend the OS disk size to fill the disk size you specified<MiTamañoDiscoGb>.

      Open DiskPart tool as administrator and run these DiskPart commands:

      1. Listenvolumen(see volumes)
      2. Select volume 2(depends on which volume is the operating system drive)
      3. extended size 72000(to expand drive by 72GB, from 128GB to 200GB)

3. Install any additional software on the virtual machine.

4. To customize the pipeline agent user's permissions, you can create a user nameAzDevOpsand give this user the required permissions. This user is created by the scaleset agent startup script if it doesn't already exist.

5. Restart the VM when you're done making adjustments

6. Generalize the VM.

   • Window- From a management console window:

     C:\Windows\System32\sysprep\sysprep.exe/generalize/oobe/shutdown

   • Linux:

     sudo waagent -deprovisioning+user -force

   Important

   Wait for the virtual machine to complete generalization and shutdown. Do not continue until the virtual machine has stopped. Schedule 60 minutes.

7. Detach the virtual machine

   az vm desalocate --resource-group <myResourceGroup> --name <MyVM>

8. Mark the virtual machine as generalized

   az vm generalize --resource-group <myresourcegroup> --name <myVM>

9. Create a virtual machine image based on the generalized image. If you follow these steps to update an existing scaleset image, note the image ID URL in the output.

   az image create --resource-group <myResourceGroup> --name <MyImage> --source <MyVM>

10. Create the scale set based on the custom virtual machine image

    az vmss create --resource-group <myResourceGroup> --name <myScaleSet> --image <myimage> --admin-username <myUsername> --admin-password <myPassword> --instance-count 2 --disable-overprovision --actualización manual del modo de política --equilibrador de carga '""'

11. Ensure that both VMs created in the scale set are online, have different names, and reach a healthy status.

You can now create an agent group with this scale set.

Update an existing scale set with a new custom image

To update the image in an existing scaleset, follow the steps in the previous stepCreate a scale set with a custom image, software, or disk sizeSection up through thecreate image azStep to create the custom operating system image. Note the URL of the ID property generated fromcreate image azDomain. Then update the scaling set with the new image as shown in the example below. After the scale set image is updated, all future VMs in the scale set will be created with the new image.

az vmss update --resource-group <myResourceGroup> --name <myScaleSet> --set virtualMachineProfile.storageProfile.imageReference.id=<id url>

Supported Operating Systems

Scale set agents currently support Ubuntu Linux, Windows Server/DataCenter 2016/2019 and Windows 10 clients.

known problems

• Debian or RedHat Linux are not supported. Only Ubuntu is.
• The Windows 10 client does not support running the pipeline agent as a local user and therefore the agent cannot interact with the UI. Instead, the agent runs as a local service.

Troubleshooting

Navigate to your Azure DevOpsproject settings, chooseagent groupslowpipingand select your agent group. Select the tab marked withDiagnose.

The Diagnostics tab shows all actions taken by Azure DevOps to create, delete, or reimage virtual machines in your Azure scale set. The diagnostics also logs any errors encountered while attempting to perform these actions. Check the errors to ensure your scaleset has enough resources to scale out. If your Azure subscription has reached resource limits on VMs, CPU cores, disks, or IP addresses, these errors will appear here.

unhealthy means

When agents or VMs fail to start, fail to connect to Azure DevOps, or disconnect unexpectedly, Azure DevOps logs errors in the agent pool.Diagnosetab and tries to delete the associated virtual machine. Network settings, image customization, and pending reboots can cause these issues. Connecting to the VM for debugging and collecting logs can help with investigation.

If you want Azure DevOps to save a failed agent VM for investigation and not automatically delete it when it detects the failed state, go to your Azure DevOpsproject settings, chooseagent groupslowpipingand select your agent group. ChooseIdeas, select the optionStorage of an unhealthy agent for research, and chooseSave on computer.

Now, when a failed agent is detected in the scale set, Azure DevOps saves that agent and its associated VM. The saved agent is displayed in theDiagnoseAgent group UI tab. Navigate to your Azure DevOpsproject settings, chooseagent groupslowpiping, select your agent group, chooseDiagnoseand write down the agent's name.

Locate the associated virtual machine in your Azure virtual machine scale set using the Azure portal in theinstancesList.

Select the instance, chooseConnectand research.

To delete the stored agent, after your investigation is complete, go to your Azure DevOpsproject settings, chooseagent groupslowpipingand select your agent group. Select the labeled tabDiagnose. Find the agent in theAgents on hold for researchmap and chooseExtinguish. This removes the agent from the pool and removes the associated virtual machine.

frequently asked Questions

• Where can I find the images used for Microsoft hosted agents?
• How do I configure scale set agents to run UI tests?
• How can I remove agents?
• Can I configure the agent group of the scale set so that no agents are in standby mode?
• How much do Scale Set Agents cost?
• What are some common problems and their solutions?
  • See more idle agents than you want at different times
  • VMSS upgrade does not work in the expected five minute interval
  • Azure DevOps Linux virtual machine scale set often fails to start the pipeline
  • You enable the option to automatically take virtual machines offline after each use for the agent pool, but you see that the virtual machines are not regenerated as intended and simply pick up new jobs as they are queued.
  • VMSS shows the agent offline when the virtual machine restarts
  • You can see different tags like _AzureDevOpsElasticPoolTimeStamp for VMSS in cost management
  • You cannot create a new agent group for the scale set and get an error that a group with the same name already exists
  • VMSS maintenance job not running on agents or retrieving logs
  • If you specify AzDevOps as the primary administrator in your VMSS script, you might encounter issues with agent configurations on scale set instances.
  • Agent extension installation fails on scale set instances due to network security and firewall settings

Where can I find the images used for Microsoft hosted agents?

Licensing considerations limit us in distributing Microsoft hosted images. We cannot provide you with these images for use in your scale set agents. Butdashesthat we use to generate these images are open source. You are free to use these scripts and create your own custom images.

How do I configure scale set agents to run UI tests?

Create a scale set with a Windows Server operating system and select the Configure virtual machines to run interactive tests option when creating the agent set.

How can I remove agents?

Navigate to your Azure DevOpsproject settings, chooseagent groupslowpipingand select your agent group. Select the tab marked withAgents.Click the Enabled toggle button to disable the agent. The disabled agent completes the currently running pipeline and does no further work. A few minutes after your current pipeline job completes, the agent will be removed.

Can I configure the agent group of the scale set so that no agents are in standby mode?

yes if you specifyNumber of agents to holdto zero, for example to keep costs down for a low volume of jobs, Azure Pipelines only starts a VM when it has a job.

How much do Scale Set Agents cost?

Scale set agents are priced similar to other self-hosted agents. You provide the infrastructure on which the agent software and jobs run, and pay for the number of jobs you want to run concurrently by purchasing parallel jobs.

For scale set agents, the infrastructure to run the software and agent jobs is Azure Virtual Machine Scale Sets and pricing is described inVirtual Machine Scale Set Pricing.

For information on buying side jobs, go toSet up and pay for parallel jobs.

What are some common problems and their solutions?

See more idle agents than you want at different times

To better understand why this happens, seeHow Azure Pipelines manages the scale set. Throughout the scaling process, the goal of Azure Pipelines is to reach the desired number of idle waiting agents. Clusters slowly scale up and down. For a day, the pool grows as requests are queued in the morning, and grows as the load eases in the evening. This is the expected behavior as Azure Pipelines gradually approaches the constraints you specified.

VMSS upgrade does not work in the expected five minute interval

The scaling job runs every five minutes, but if only one operation is being processed, you may find that the scaling does not complete within five minutes. This is currently the intention.

Azure DevOps Linux virtual machine scale set often fails to start the pipeline

The first place to look when encountering issues with scale set agents is the diagnostics tab in the agent set.

Also consider saving the messed up VM for debugging purposes. For more information, seeunhealthy means.

Saved agents exist unless you delete them. If the agent does not connect within 10 minutes, it will be marked as failed and saved if possible. Only one virtual machine is kept in a saved state. If the agent goes offline unexpectedly (due to a virtual machine restart or an image problem), it will not be saved for investigation.

Only virtual machines for which agents cannot be started are saved. If a virtual machine has an error status during creation, it is not saved. In this case, the message on the Diagnostics tab is "removing bad machine" instead of "failed to start".

He enables the option to automatically detach virtual machines after each use for the agent pool, but finds that the virtual machines are not properly recreated and simply accept new jobs when they are queued.

The option to uninstall the virtual machine after each build only works for Windows Server and supported Linux images. Windows client images are not supported.

VMSS shows the agent offline when the virtual machine restarts

Showing agents offline when the virtual machine restarts is the expected behavior. The agent service only runs in the systemd context. However, if the machine reboots for any reason, it will be considered as a corrupted virtual machine and will be deleted. For more information, seeunhealthy means.

When agents or VMs fail to start, fail to connect to Azure DevOps, or disconnect unexpectedly, Azure DevOps logs errors in the agent pool.Diagnosetab and tries to delete the associated virtual machine. Network settings, image customization, and pending reboots can cause these issues. As a workaround, disable software update for the image. You can also connect to the VM for debugging and collect logs to investigate the issue.

You can see different tags like _AzureDevOpsElasticPoolTimeStamp for VMSS in cost management

When the pool is created, a tag is added to the scale set to mark the scale set as used (to prevent two pools from using the same scale set), and another tag is added for the timestamp, which is updated on each setup job runs. (every two hours).

You cannot create a new agent group for the scale set and get an error that a group with the same name already exists

You may get an error message likeThis virtual machine scale set is already in use by set <set name>since the tag still exists in the scale set even after deletion. When an agent group is deleted, it attempts to delete the tag from the scale set, but this is a best effort attempt and gives up after three retries. Also, there can be a maximum interval of two hours during which a VMSS that is not being used by any agent group cannot be reassigned to a new one. The solution to this is to wait for this time interval to elapse or to manually remove the scale set tag from the Azure portal. When viewing the scale set in the Azure portal, select thehang tagsLink to the left and remove the highlighted label_AzureDevOpsElasticPool.

VMSS maintenance job not running on agents or retrieving logs

The maintenance job runs once every 24 hours. Virtual machines can be full before this time. Consider increasing the disk size on the virtual machine and adding a script in the pipeline to remove the contents.

if you specifyAzDevOpsAs the primary administrator in your VMSS script, you might see issues with agent configurations on scale set instances

if you specifyAzDevOpsAs a primary admin in your VMSS script, you might see issues with agent configurations on scale set instances (the user's password will be changed if it already exists).

This problem occurs because the agent extension scripts try to create the userAzDevOpsand change your password.

Agent extension installation fails on scale set instances due to network security and firewall settings

The extension must be able to download the build agent files fromhttps://vstsagentpackage.azureedge.net/agentand the build agent must be able to register with Azure DevOps Services. Ensure that this URL and the associated Azure DevOps Services URLs and IP addresses are open on the instance. For information about IPs and URLs that need to be unblocked on your firewall, seeAllowed IP addresses and domain URLs.