Regression Suite Automation Tool, a day in a live

We have been working for a while now with RSAT at various customers, it is stable, but we see sometime small glitches in user experience. In general, what is happening, if you haven’t maintained/used it for a while (for instance when during implementation RSAT is configured and used (for instance as provisioning tool), but then not actively monitored for some time) and a consultant or customer starts using it does not work (as expected). So, let us address some topics you should check before (re)using RSAT. Below is a list based on using RSAT for the last 2 years.

LCS (Lifecycle Services) Project Azure trust expires (every year)

There is a trust between DevOps and LCS, the maximum trust time is one year.
Getting a new token at: https://XXX.visualstudio.com/_usersSettings/token

In case the access token is expired, update it from LCS

For more information about integration of LCS and Dev Ops please read Is your Business Process Modeler updated? – Kaya Consulting (kaya-consulting.com)

Are the soap names still, correct?

Unbelievably, there are a lot of incorrect solutions in the MS communities, with all distinct kinds of suggestions, starting from adding only soap or at the wrong place in the URL. The correct location is:
• Tier 2 XXXXXaossoap.sandbox.operations.dynamics.com
• Tier 1 XXXXXaossoap.cloudax.dynamics.com

Limitation on the test user (the consultant test partner account)

The test user must be active in the tenant domain of the customer. In general, always try to avoid using non-customer tenant users as also uploading the task recording directly into LCS, has similar issues. The tenant domain is visual in D365 at right top, click the question mark and about, tab page licenses.

Validate the certificates

And those are the nasty ones, I have seen a lot of Non-RTFM issues here. So, let us start addressing them by environment type. We will first start Microsoft Management Console for validating the certificates.
• Start mmc.exe

• We add the snap ins for certificates; for the current user and local computer.

It is wise to save this console setup after validating and fixing!

Tier 1 dev box

When you create a certificate for a Tier 1 and look at the pulldown, we can agree, no clue, what is meant here?

Adding a user-friendly meaningful name to the certificate will help you, this can be done by pressing the right mouse and selecting properties.

Meaningful names will result in clear names on the pull down for the Thumbprints.

Wif.config

There is a widespread misconception that pressing New in the RSAT tool also adapts the config file on the remote dev box. Sorry, technical consultants can perform miracles, but this is not one of them. So please open your remote desktop to the Dev Box and validate it!

Tier 2 environments

Be aware that platform updates also update the RSAT certificates. Last week I checked all on Monday and on Friday constant complaints, it is not working. That they did a platform update during that week, they forgot to tell me 🙁

Maintain Certificates RSAT Tool

Working with the Maintain Certificates RSAT Tool. It is not perfect yet. The Tier 1 and production should not appear on the list.

The user-friendly meaningful names are missing (but you know how to fix this now)

So, you change to the meaningful name. But it is still not working. You get the next error.

It took me some time to figure out this one. Please look at the next pictures on how to resolve this. I think MS will fix this in their next release.

And if we compare it with the certificate I imported the old way, we see a difference. With below certificate RSAT can pull a token.

Microsoft is on the right track with the Maintain Certificates RSAT Tool. So, we follow Frank Dahl and support him with valuable feedback.

Conclusion

Despite the above, I am still a happy user of RSAT!

For more details about RSAT please look at.

Regression suite automation tool installation and configuration – Finance & Operations | Dynamics 365 | Microsoft Docs

DataALM in een Dynamics 365 Finance & SCM omgeving – Deel 2 van 2

Deze blog bespreekt de mogelijkheden voor de automatisering van post-refresh cleanup taken (na een database restore) via scripting.

DataALM in een Dynamics 365 Finance & SCM omgeving – Deel 1 van 2

Deze blog bespreekt Microsoft’s LCS ”Self-Service Database Refresh” voor het terugzetten van een productiedatabase in een sandboxomgeving.

Size does matter (or not?)

It is a general issue with D365FO development on those development VMs (virtual machines) when it is not working as planned, in the end we throw it away and start over. In the next blog I will explain some tricks to solve issues on the disk space. Look at the next example, we use a disk of 32GB and need 20 of them so we have 640 GB of disk space.

Of course, check the prices: Pricing – Managed Disks | Microsoft Azure.

Besides the disk size I also use DS13 V2, so I can upgrade to Managed Disks: Performance tuning development VM (Virtual Machine) – Kaya Consulting (kaya-consulting.com)

Why not 10 * 64 GB or 5 * 128 GB? It is all about IOPS!

Later, in Azure it will look like below.

When we look at the VM with Disk Management we see a striped disk that has partitions for database and ‘AOT (Application Object Tree) drive.’

So far so good, but the developer needs a good database and when restoring, scenarios like the next picture may occur.

We are stuck, no space on the G drive. And plenty of space on those other drives. So, let us adept, we start with shrinking the other disks. NOTE: K drive needs 15 GB free space for installations started from LCS (Life Cycle Services).

After shrinking we add the created free space to the G Drive.

• In case you still run into disk space issues, use the next SQL script to compress the database:

EXECUTE sp_msForEachTable ‘ Print ”?”;SET QUOTED_IDENTIFIER ON ;IF EXISTS (Select * from sys.indexes Where object_id = object_id(”?”) And type=6) ALTER INDEX ALL ON ? REBUILD ELSE ALTER INDEX ALL ON ? REBUILD WITH (data_compression=PAGE,FILLFACTOR=99)’

• Another trick is dropping the retail database, it is in any installation package, the script name is DropAllRetailChannelDbObjects.
• Set the database recovery model to simple will reduce the database log disk size.
• Check the SQL report “Disk Usage by Top Tables.”
• Add an additional disk:
  • Attach a managed data disk to a Windows VM – Azure – Azure Virtual Machines | Microsoft Docs

Of Course, I would love to give you a golden rule for size of the bacpac file in relation to the needed disk space. I can give you a refence a 9 Gb Bacpac file results in a 90 Gb Database MDF file. So  Bacpac file size multiply by 10. It is still a guesstimate 🙁

Bana Regression Suite Automation Tool(RSAT) satabilir misiniz?

Microsoft D365 FO’nun yeni sürümlerinin mevcut hızlı görünümü ile test yapmak daha fazla zaman alabilir (yılda 10 kez). Bu durumda bazı kararlar vermenizde fayda var. Örneğin;

1. Güncellemeleri atlayın ve örneğin yalnızca çift veya tek platform güncelleme numaralarını alın
2. MS’i test etmeyin ve güvenin. Ancak lütfen MS’te sadece insan olduklarını ve insanların da hata yaptığını unutmayın.
3. Daha kısa sürede daha fazla test yapılabilmesi için test ekibinizi artırın.
4. Regression Suite Automation Tool kullanmaya başlayın.
   Geçmişte 1. seçeneği seçiyorduk. Yine de her ay release çıkardık, ancak MS sürümüne ISV ile sürümler ve müşteri için özel çözümler dahil edilmedi.
   

Nasıl kullanabiliriz?

Müşterilerimizle RSAT hakkında konuşmaya başladığımızda, her zaman bütçe olmadığını söylüyorlar. Bu doğru değil, her zaman test maliyeti vardır. Sadece mevcut test maliyeti, dahili testler ve / veya üretim ortamlarında ortaya çıkan sorunlar şeklinde o müşterinin organizasyonunda gizlidir. Gördüğümüz şey; müşterilerin BT projeleri için bir bütçesi olduğu. Müşterinin / anahtar kullanıcının test ettiği saatler bu bütçenin parçası değildir.
Sonra başka bir nokta daha görüyoruz ki mevcut görev kayıtları (task recorder) çoğu zaman güncelliğini yitirmiştir (bkz. İş Süreci Modelleyiciniz (BPM) neden güncel olmalı? bloguna). RSAT kullanarak BPM kitaplığınızı otomatik olarak güncellemiş olursunuz.
RSAT, bir sağlama aracı gibi kullanarak daha da güçlü hale gelebilir. Aşağıda, veri hattı sürümünde kullandığımız bir örnek verilmiştir. İlk örnek, bir veri tabanını 1. katman (MS barındırılan veya müşteri tarafından barındırılan) üzerine geri yüklediğimiz zamandır.

Mevcut RSAT çözümünün bir bulut hizmeti olarak çalıştırmak yerine gerçek bir makineye kurulması gerekiyor. Bunu başarmak için, onu bulutta barındırılan bir Devbox’a kurduk ve maliyetleri yönetmek için otomatik kapatmayı açtık. Bu nedenle eğitimde söz konusu ortamı hem başlatan hem de durduran adımlar görüyorsunuz.
Devops’taki mevcut RSAT, yalnızca kurulu olduğu gerçek bir makinede çalışır. Bu yüzden onu bulutta barındırılan Devbox’a kurduk, Azure maliyetini düşük tutmak için bu sanal makineyi talep üzerine başlatıyoruz. RSAT provizyonunun parametrelerinde; veri tabanını yeni geri yüklediğimiz(restore) ortama işaret eden RSAT ayarları dosyasını seçiyoruz.

RSAT’ı kullandığımız görevler şunlardır:

• Kullanıcıları etkinleştirin
• Toplu işleri (Batch) başlatın
• Exchange ayarlarını güncelleyin
• Diğer tüm entegrasyon seçenekleri
Ancak RSAT, en son özelleştirilmiş müşteri kodunuzla yeni bir release yayınladığınızda da tekrar kullanılabilir. RSAT’ı bu şekilde kullanmamın nedeni; 1. Katman üzerinde yapılan testlerin dünyayı temsil etmemesi, paketi geliştirme alanında değil operasyonel alanda test etmek istiyorum.

Ve son durum, Microsoft’un güncellemeleri içindir. Yukarıdaki gibi benzerdir, RSAT görevini yalnızca MS versiyon aday ortamında çalıştırırsınız.
Şimdi iş durumu açıklandığına göre, yapı taşlarına kısaca bir göz atalım.
VM’yi başlatmak ve durdurmak için Azure CLI kullanıyoruz

Agent pools

Kullandığımız adımlar farklı agent pool’a ihtiyaç duyar. Ben her zaman sadece azure pipeline’ı kullanmayı tercih ediyorum. Ancak bunların bir sınırlaması var. Adımın sanal makinede bulunan programlara veya dosyalara ihtiyaç duyması halinde, sanal makinenin de bir agent pool’u olması gerekir. Aşağıdaki liste, bir müşteri uygulamasında ihtiyaç duyduğumuz agent pool’u göstermektedir:

Ayrıca, mevcut bir bulutta veya hatta yerel bir sanal makinede çalışan bir building pool’a ihtiyacınız olduğunu fark ettiğinizde, onu yine de kurabilirsiniz. Https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops ile ilgili bilgilere bakabilirsiniz.
Toparlamak gerekirse; RSAT’ın bir satış hikayesine ihtiyacı yoktur. Nasıl kullanılacağını bilirseniz, kendisini satmaya başlayacaktır.
Diğer DEVOPS ipuçları için https://kaya-consulting.com/category/lcs/ adresine bir göz atın.

Restore Azure database to Tier-one

In my current project I get a high demand on request for restoring Azure databases to Tier-one. Out of the box there is no support on it from LCS point of view. But that does not imply that it is not possible.
At this moment we create an additional DEVOPS release pipeline that is manually triggered. The steps of that pipeline are shown in the next picture.

The Steps

The steps we are taking are in general

• Request access token to LCS
• Download the bacpac file from the LCS asset library
• Import the bacpac file into a new database
• Stop all services
• Swap the new database with the old database
• Synchronize the database (there could be new tables & fields)
• Reset Financial Reporter
• Start all services

With the current security on the MS hosted Tier 1 boxes, it is getting more complex to completing all these steps. The complexity is related to less privileges. You do not have administrator access anymore.
When you want to perform actions that requiring administrator access, you can host a D365 environment hosted on your own azure subscription. Here you can open Pandora’s box, but Microsoft does not have any responsibility or privileges on this environment.
We prefer to have all PowerShell inline, in case you prefer to use local PowerShell, below example shows you how to transfer the parameters to PowerShell files

Now let us go through the steps one by one.

Lets Download the Database

Below is an example for downloading and restoring the Bacpac file. For more details please read https://docs.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/database/dbmovement-scenario-exportuat#import-the-database

cd C:\temp\ <# asset list #> $refreshUrl = “https://lcsapi.lcs.dynamics.com/databasemovement/v1/databases/project/$(LCSPRPOJID)” $refreshHeader = @{ Authorization = “Bearer $(TOKEN)” “x-ms-version” = ‘2017-09-15’ “Content-Type” = “application/json” } $refreshResponse = Invoke-RestMethod $refreshUrl -Method ‘GET’ -Headers $refreshHeader $DatabaseAssets = $refreshResponse.DatabaseAssets; <# find latest BACKUP on LCS #> $cstzone = [System.TimeZoneInfo]::ConvertTimeBySystemTimeZoneId( (Get-Date), ‘W. Europe Standard Time’) $filedate = Get-Date( $cstzone).AddDays($(DAYS)) -f “yyy-MM-dd” $BackupName = “GOLD-$filedate” Write-Output $BackupName $url = $DatabaseAssets | Where-Object {$_.Name -eq $BackupName} | Select-Object -Property FileLocation $output= $DatabaseAssets | Where-Object {$_.Name -eq $BackupName} | Select-Object -Property FileName Write-Output $url.FileLocation Write-Output $output.FileName <#remove old downloades#> <#Remove-Item –path C:\temp* -include *.bacpac -whatif -force> <# start download #> Import-Module BitsTransfer Start-BitsTransfer -Source $url.FileLocation -Destination $output.FileName $importFile = $output.FileName Write-Host “##vso[task.setvariable variable=BACKUPNAME]$BackupName” Write-Host “##vso[task.setvariable variable=FILENAME]$importFile”

The next step is creating the database

<#import Bacpak file#> cd C:\temp\ $fileExe = “C:\Program Files (x86)\Microsoft SQL Server\140\DAC\bin\SqlPackage.exe” & $fileExe /a:import “/sf:$(FILENAME)” /tsn:localhost “/tdn:$(BACKUPNAME)” /p:CommandTimeout=1200

Stop environment

The step for start or stopping the service are not so complex, you can even run them inline on the release pipeline (For start change the stop to start)

<#stop environment#> net stop W3SVC net stop DynamicsAxBatch net stop Microsoft.Dynamics.AX.Framework.Tools.DMF.SSISHelperService.exe net stop MR2012ProcessService net stop ReportServer

Swap Databases

Now all systems are down so we can swap a database.

<#rename Database#> $server = “.” Write-Output Drop AXDBOLD Invoke-Sqlcmd -ServerInstance $server -Query “DROP DATABASE IF EXISTS AXDBOLD” <#rename Database#> $server = “.” Write-Output Drop AXDBOLD Invoke-Sqlcmd -ServerInstance $server -Query “DROP DATABASE IF EXISTS AXDBOLD” Write-Output renaim AXDB to AXDBOLD Invoke-Sqlcmd -ServerInstance $server -Query “ALTER DATABASE AXDB MODIFY NAME = AXDBOLD” Write-Output renaim new DB to AXDB Invoke-Sqlcmd -ServerInstance $server -Query “ALTER DATABASE [$(BACKUPNAME)] MODIFY NAME = AXDB” Write-Output create technical users Invoke-Sqlcmd -ServerInstance $server -Database axdb -Query “the latest scripts MS”

I will not distribute the latest scripts MS statements, because it can change depended on latest updates from MS. In general, it should be in line with https://docs.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/database/dbmovement-scenario-exportuat#update-the-database

Lets Synchronize

The next step is about synchronizing the database, there are several ways to do it

• Start Visual studio and synchronize manually
• Scripts that need admin privileges that you do not have
• Use the next one 😊

<#sync database#> $command = Join-Path “K:\AosService\PackagesLocalDirectory” “\Bin\SyncEngine.exe” $connectionString = “Data Source=””.””;Initial Catalog=AXDB;Integrated Security=True” $arguments = “-syncmode=fullall -binfolder=K:\AosService\PackagesLocalDirectory -metadatabinaries=K:\AosService\PackagesLocalDirectory -connect=””$connectionString”” -fallbacktonative=False -verbosity=Diagnostic -continueOnError=false” Start-Process -FilePath $command -ArgumentList $arguments -Wait -PassThru -RedirectStandardError “SortError.txt” -RedirectStandardOutput “log.txt”

Management Reporter

And finally reset the database mart, normally you get again that admin privilege issue , but by adding the option -scope local, we bypass this requirement

<#reset management reporter#> K: cd K:\MROneBox\MRInstallDirectory Import-Module .\Server\MRDeploy\MRDeploy.psd1 -scope local Reset-DatamartIntegration -Reason OTHER -ReasonDetail “” -force

Please be aware the Bacpac to Tier1 is still a work in process, we must update the scripts regularly based on the requirements from MS

Additional we have on the release pipeline an RSAT provisioning step running, for more details please read XXXX

Please read the other related articles on this link https://kaya-consulting.com/category/lcs/

Which Agile methodology to use during ERP implementations

Agile SCRUM vs KANBAN

Do not use Agile SCRUM on customer implementations. With this one liner we start this blog. Why not and what should you use then?
In general, we see these days that all ERP implementations are done in an Agile way; the waterfall methodology has become an exception. But there are various Agile methodologies, the most common ones are SCRUM and KANBAN.

Customer projects

The method to use is important at Customer projects in our role of implementation partner (ERP/CRM implementations) but also on the ISV products we maintain ourselves as Kaya. Leading factor to determine which methodology to pick is not the methodology itself but how an organization is working and organized. Beside that the necessary skillset for building and releasing software plays a role too. Those skills are usually less present at a customer compared to a Microsoft, or other software, partner.

ISV products

In case you are in full control of your resources & requirements, planning fixed (release) dates is working fine. We see this at the implementation of ISV products. So, for those using Agile SCRUM would be a good choice.
However, on Customer implementations, we as a partner are not always in full control. Some examples are:

• Decisions that requirements are out of scope (read not before go-live)
• Capacity of customer resources are hard to plan. A partner has no control over it.
• Customer must accept the solution and usually they need support on the testing as they are not used to this.

So, the planning of a customer implementation needs much more flexibility. This flexibility can be better arranged with KANBAN. To have a closer look at what KANBAN is use following link https://www.scrum.org/resources/kanban-guide-scrum-teams.

Differences between Kanban and Scrum

Most important differences between KANBAN and SCRUM are:

	Scrum	Kanban
Cadence Regular	fixed length sprints	Continuous flow
Release methodology	At the end of each sprint	Continuous delivery
Roles	Product owner, scrum master, development team	Stakeholder
Key metrics	Velocity	ead time, cycle time, WIP
Change philosoph.	Avoid changes during the sprint.	Change can happen

Let us start with the Cadence. The functional confirmation of D365 F&S has a higher throughput than the development of code extensions. Beside the code extensions we also have the ISV solutions and the MS monthly updates.

This brings me to the next point. Release mythology: a code release is different from replicating a setup in a target environment.

Then we have the Roles. There are several Scrum Roles, where customers usually are not accustomed to, and the theory is that Kanban has no roles. To my opinion, this is not the case there is a stakeholder, wo defines what he needs and signs of when his wishes are realized. To a certain extend this resembles the Product owner role.

The change philosophy of KANBAN is more flexible. In case you have not started on a card, you can always put it back in the backlog. We see this happen when the Implementation gets disrupted by day to day topics or customer changes scope or priority.
The impact of using KANBAN at a customer implementation has an additional dimension. This dimension is the division between Development and Operations. In general, we can say the Partner is development and the customer is operations. This helps to determine who is responsible for what.

The next diagram narrows it even more clear on extensions to be delivered during implementation. The Partner (DEV) delivers a combined build into the asset library, it is up to the Customer (OPS) to continue from that point onwards.

OPS	DEV	DEV	OPS	OPS
Customer	Partner		Customer	Customer
Backlog	DEV	TEST	Acceptance	Production
	Code changes	Code changes	LCS asset	LCS asset
		Manual setup	Manual setup	Manual Setup

Additions to Kanban

Beside the concept of KANBAN and DEVOPS, Kaya also uses the phrase requested today, delivered tomorrow.

• When a Developer is finished, the solution is automatically available next day in TEST
• When a Tester is finished, he has in general a release candidate feature for the Customer
• The customer can almost do cherry picking on the release candidate features. The ones the customer select will be available next day in the combined build in the LCS asset library.

Are we in control?

So, are we in control? Yes! But be careful there is still a risk. This is so called work in Progress (WIP). The WIP on Acceptance is extremely important, all acceptance task must be completed before going to production. Who is responsible? The Customer (stakeholder) is responsible, but of course supported by the partner (which task should be completed first, dependencies, risks etc.). How can the Customer control it? Simply throttle on the requests that are picked from the backlog.

For other DEVOPS tips have a look at https://kaya-consulting.com/category/lcs/

Can you sell me Regression Suite Automation Tool (RSAT)?

With the current fast appearance of new versions of D365 FO (10 times a year). Testing can become more time consuming. So, you must decide:

1. Skip updates, take for example only the even or odd platform update numbers
2. Do not test and have confidence at MS. But please be aware at MS they are also mere humans and humans make mistakes.
3. Increase your Test team so more tests can be performed in a shorter time.
4. Start using the Regression Suite Automation Tool.

In the past we chose option 1. We did still release every month, but the release of MS was not included the releases with ISV and tailored solutions for the customer.

How we use it

When we start talking about RSAT with the customer, they always reply, that there is no budget. This is not correct, there are always costs on testing. Only the current test cost is hidden within the organization of that customer in the form of internal tests and/or issues arising in production environments. What we see is that the customer has a budget for his IT projects. The hours that the customer/key user is testing is not part of this budget.

Then we see another point, the current task recordings are most of the time outdated (see blog Why should your Business Process Modeler (BPM) be up to date?). By using RSAT you are automatically being forced to bring your BPM library up to date.
RSAT can become even more powerful by using it like a provisioning tool. Below is an example as we use it in a release pipeline. The first example is when we restore a database on a tier 1 box (MS hosted, or customer hosted)

The current RSAT solution needs to be installed on an actual machine instead of running as a cloud service. To achieve this we installed it on a cloud hosted Devbox and turned on auto shutdown to manage costs. Therefor you see steps both starting and stopping said environment in the train.
The current RSAT on a devops only runs on a real machine where it is installed. So, we installed it on cloud hosted Devbox, to keep the Azure cost low we start that VM on demand. In the Parameters of the RSAT provisioning we select the RSAT settings file pointing towards the environment we just restored the database of.

The tasks we use RSAT for are:

• Enable users
• Start batches
• Update exchange settings
• Any other integration options
  

But RSAT is also reusable when you deploy a new release with your latest tailored customer code. The reason I use RSAT this way is, testing on a Tier 1 is not representing the world, I want to test the Package in the Operational area and not in the Development area.

And the last case is for updates of Microsoft. It is similar like above , you only run the RSAT task on the MS release candidate environment.
So now that the business case has been explained, let us have a short look at the building blocks.
For starting and stopping the VM we use Azure CLI

Agent Pools

The steps we used need different Agent pools. I always prefer to use only the azure Pipelines. But those have a limitation. In case the step needs programs or files that are located on the VM, the VM also need to become an agent pool. Below list shows the additional agent pools we need at a customer implementation

And when you realize you need a building pool on an existing cloud hosted or even running on a local VM, you can still install it. Look for information on https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops
And now to wrap up; RSAT does not need a Sales story. If you know how to use it, It will start selling itself.

For other DEVOPS tips have a look at https://kaya-consulting.com/category/lcs/