Tag Archives: azure

GitHub reliability and scalability: a consequence of AI and too much for free

GitHub’s CTO Vlad Fedorov has posted apologetically about the 17th August outage which, he said, “lasted 7 hours and 47 minutes. It disrupted github.com, authentication, GitHub Actions, APIs, pull requests, issues, and Copilot, affecting developers and organizations around the world.”

As if that is not bad enough, Fedorov also apologised for a failure of Actions on August 6th; and GitHub users will be aware of other incidents over the last few months that have tarnished the company’s reputation as well as being costly for its customers.

GitHub is owned by Microsoft and there is a narrative that says: under Microsoft GitHub has lost its best engineers and is now suffering from that as well as issues with the Azure platform.

There may be something in that, but there are more obvious reasons for GitHub’s difficulties. Specifically, Fedorov states that “since April, monthly commits have grown from 1.4 billion to 2.9 billion.”

That is a rapid growth in demand for a platform that was already under stress. Agentic AI appears to be a large part of the reason. Pull requests and requests are no longer generated mainly by humans, but are now programmatic and therefore created more quickly. In addition, AI-driven development means that it is no longer necessary to be a developer in order to create software projects; more people are creating more software with AI assistance.

GitHub has long had an amazing free offer. Take a look: free plans include unlimited public or private repositories as well as 2,000 CI/CD minutes every month, or unlimited for public repositories. For just $4.00 per month those limits increase. Competitors such as GitLab have reduced their free offer but GitHub has not.

GitHub’s generous free plan

GitHub is Microsoft and can afford it, right? True but free (or too cheap) has a hidden cost, if scaling problems impact paying customers, as has happened.

A way of managing this is to run isolated platforms within GitHub such that an issue affecting some users does not automatically impact all users. It appears that GitHub’s architecture prevents this at the moment.

Part of the answer, as Fedorov discusses, is to improve and expand GitHub’s scalability. A post in April describes this work and includes the comment that “we started working on path to multi cloud,” meaning that Microsoft is willing to pay competitor cloud companies in order to fix these issues.

It seems to me though that Microsoft/GitHub also need to restrain agentic AI by means of quotas and higher prices, in the interests of maintaining the stability of the platform. If that also means reducing the free offer that is understandable; better to have a reliable platform than a free one. Docker, for example, went through this kind of transition, though in that case it was a matter of financial necessity.

In the meantime Cursor (now part of SpaceX) has introduced Origin, now in public beta as a “git forge for storing and sharing code”, which Vicent Martí described in technical detail as Git at any scale. Marti was formerly at Planetscale and before that, GitHub.

The technology is fascinating, but I suspect Cursor also has in mind not to replicate GitHub’s mistake in underpricing its platform.

Getting started with Azure Artifact Signing

Signing applications is a requirement for deploying a desktop or mobile application and can be fiddly. Microsoft’s Azure Artifact Signing always looked to me like a good solution in that the pricing is reasonable at $9.99 per month for up to 5,000 signings, and it is run by Microsoft so one would think that the certificates will be satisfactory for Windows users.

This week I got started with the service. Microsoft has a handy quickstart guide and the first thing to note is that availability is limited to USA and Canada for individual developers, or USA, Canada, EU and UK for organizations. Luckily I work for a small UK limited company so qualify as an organization.

I also noted that a pay as you go Azure subscription is required; if you have a sponsored subscription that will not work.

The first step is to create an Artifact Signing Account. Billing starts from when this account is created, even if you are unsuccessful in obtaining a signing certificate. Should this be the case, make sure to delete the account.

Next, check the roles assigned to the user that will operate the code signing. There are two: Artifact Signing Identity Verifier and Artifact Signing Certificate Profile Signer. I assigned them both to my Entra ID user.

Now there are two steps to obtaining a certificate profile. The first is Identity Validation. In my case this is for an organization; you will need a DUNS number, two valid email addresses, a functioning web site, and the first and last name of the individual that will use the service. The organization address should match the one recorded at Companies House, even if day to day work is done elsewhere.

The organization identity verification automatically progresses to individual identity verification. In my case this was done by a company called AU10TIX and I was able to use a UK driving licence.

Next I hit what looked like a roadblock. I received an email asking me to verify my email address; however clicking the link got me a web page stating “the request is blocked” and “service unavailable”. Thinking that perhaps the service was unavailable, I tried several times and on different networks. I am not the first to experience this. I was on the point of raising a support case when I got a further email advising that validation was complete.

Once identity validation is complete you can create a certificate profile. The type I need is called Public Trust. I had several mysterious failures with uninformative errors; I suspect this might be to do with non-unique profile names but no idea really. Again I was on the point of contacting support when I tried a new profile name and it worked.

The quickstart guide ends at this point and the next document you need is the one on setting up signing integrations. You will need a recent Windows SDK including SignTool.exe, the .NET 8.0 runtime, and the Azure Artifact Signing client tools, if like me you want to sign a Windows executable.

Then you need to create a JSON file traditionally called metadata.json, and a command to sign the file. The command will be lengthy so fire up Notepad or similar and put it in a .cmd file or similar. You need to discover the location of SignTool and the location of Azure.CodeSigning.Dlib.dll and note that there are different x64 and x86 versions.

The command includes a timestamp without which the code signing is useless as the certificate has a lifetime of just three days.

I carefully assembled the command and I am happy to report that it worked first time.

Output from code signing command

So far so good. The whole process is somewhat intricate but I am hoping this will prove a good solution for Windows.

A mild case of Azure bill shock: is this the most over-priced service on Microsoft’s cloud?

I have been experimenting with accessing Azure storage from remote PCs and tried out the option to use SFTP which was introduced last year. It works though there are limitations, like no support for SSH commands after connecting, no resume support for uploads, and no support for Azure AD authentication – this last is a bit of an issue since fine-grained permissions can only be done with local users, specific to the blob storage.

I actually thought I had turned this off after my experiment but I did not. So I had SFTP enabled on a test storage account, doing nothing. I spotted it of course when I got a large (for my usage) bill. Simply having SFTP enabled on a storage account costs around $220 per month.

To be fair to Microsoft, the cost is documented and there is a notice in the portal, in the details for the storage account, that enabling SFTP incurs a charge, though it does not say how much.

The cost for enabling SFTP

The price is remarkable though, especially given that it seems that the SFTP support is a bit of a hack. Perhaps Microsoft actually runs up a dedicated VM for this in the background, who knows?

“The cost is astronomical considering the service, it’s like $7.20 a day to use and roughly $220 a Month. It’s WAY cheaper to use a VM. This service is like 3x too much,” said a comment from another sufferer.

My advice is not to do this. My further advice is to track closely the actual spend on any new services you run up since is it the only reliable way to avoid this kind of problem.

.NET P/Invoke on Azure App Service for Linux

I have an online bridge game in development (yes, still!) and it is written in ASP.NET Core with C#. One of the things that interests bridge players is called double-dummy analysis; this is where you look at what would be the best play in a game if you knew where all the cards were, whereas when actually playing bridge you only see your own cards and, during play, another hand called Dummy, so half the cards are hidden.

Double-dummy analysis is a solved problem and bridge programmers benefit from an open source library called DDS (Double Dummy Solver) written primarily by Bo Hagland and Soren Hein. This is a C++ DLL that can also be compiled for Linux and MacOS.

I wanted to integrate DDS into the bridge game in order to give players information at the end of a game including whether they were in the optimum contract and whether they beat the optimum score. I started by doing a new C# wrapper for DDS though borrowing from the work here. My version is 64-bit and wraps a few more functions. I compiled the native DLL for Windows and Linux using OpenMP for concurrency, which considerably improves performance (Boost is another option but I did not find much difference).

Note: the usual caveats about P/Invoke apply here. During one of my tests I actually crashed the container running the app. The ASP.NET developers do a lot of work to make the platform reliable, and doing P/Invoke may introduce instability.

I added my wrapper into the ASP.NET application and it worked fine on my development machine. I deployed it to App Service and the P/Invoke calls did not work. Fixing this required a bit of a deep dive into Azure App Service for Linux.

I am deploying the native code .so library into the same directory as the compiled .NET code for the rest of the application. The error I got was:

Cannot open shared object file: No such file or directory

I raised the topic on Stack Overflow.

One of the things that puzzled me was that the unit tests, which include the P/Invoke code, ran OK in Azure Pipelines, which I use for deployment. But not when deployed.

The first point is that you get the “No such file” error not only when the file itself is not present (it was) but also when a dependency is missing. So step one is to SSH into the container running the ASP.NET app, which you can do with the Development Tools in the Azure portal. Note that with Azure App Service for Linux the app always runs in a container.

image

This gives you root permissions in the container though not to the host operating system. Navigate to the directory with the troublesome library and type:

ldd libdds.so

(or the name of your library). This will tell you if any dependencies are missing or other issues. I noticed two things. One is that it was missing the dependency libgomp.so.1 which is the OpenMP library. Second, ldd reported that my library required at least GLIBC 2.29 where the available version was 2.28.

How could I fix the GLIBC version? This is determined by the version of Linux and you can use

ldd – version

to check the version you have. In my case it said I had Debian with GLIBC 2.28:

image

I did some more research. If you really want to know about Azure App Service for Linux, there are a few key documents.

The basics here: Operating system functionality – Azure App Service | Microsoft Learn

The FAQ here: App Service on Linux FAQ | Microsoft Learn

Here you will learn details like why you cannot use a file-based database like SQLite in Azure App Service for Linux:

“The file system of your application is a mounted network share. This enables scale out scenarios where your code needs to be executed across multiple hosts. Unfortunately this blocks the use of file-based database providers like SQLite since it’s not possible to acquire exclusive locks on the database file.”

But I digress. To go deeper still, check this post by Jim Cheshire:

Things You Should Know: Web Apps and Linux – Microsoft Tech Community

which has lots of critical information, like why a custom container on App Service must respond to ping.

So after reading through all this and greatly improving my understanding of how App Service for Linux works, I got to the heart of my problem. When you deploy a .NET Core application to App Service for Linux, it will by default use a container from the Microsoft Artifact Registry that matches the version of .NET you are using. If you check this page you will see that the current version for ASP.NET Core 6.0 is tagged mcr.microsoft.com/dotnet/aspnet:6.0

image

If you examine this container you will find that it runs Debian Buster which uses GLIBC 2.28. It is a matter of slight concern since Debian Buster is shown on the Debian releases wiki as having an approximate end of life August 2022, though the LTS project extends that to June 2024.

Still, now I knew how to fix my problem. Either use a custom container image, or upgrade to .NET 7, or recompile libdds.so to run on Debian Buster.

I decided that the easiest short-term solution was to recompile. I downloaded Buster and recompiled the library.

What about libgomp.so.1? This was kind-of fixable by using SSH to run:

apt-get update

apt-get install libgomp1

This is not great though since Azure could replace the container at any time, and always if you do something like scale the plan up or down, to change the specification of the VM. I tried copying the buster version of libgomp.so.1 to the application directory. It works, but I also needed to add a linker option to enable DDS to use a library in the same directory:

 -Wl,-rpath='${ORIGIN}'

as explained here.

I think a better solution is to move to deploying a custom container to App Service, which is an option:

image

Care is needed though as there is a bit of special sauce in the official container images if you want features like SSH in the portal to work properly. It also means revisiting my deployment scripts, so the above hack was an easier and quicker workaround for me.

Point-in-time restore: a handy built-in feature in Azure SQL

I am working on a project that is hosted in Azure and I made a mistake, running a SQL script that was dependent on another SQL script that I had forgotten to run. It messed up the foreign keys and I would have to restore a backup … but my most recent backup was from the day before. Annoying.

But wait. Looking the Azure portal I saw this:

image

This is a plain Azure SQL instance with no extras, but look, you can restore the database from 6 minutes ago.

I did it; it restored to a second database. I deleted the bad one, renamed the restored one, ran my scripts in the right order, and all was well.

I recommend you do not run scripts in the wrong order … but if you do, or make some other error, this is a handy feature of Azure SQL which I was not aware of before.

Developing software for playing bridge

I am a duplicate bridge player in my spare time and enjoyed playing in my local club once or twice a week. That was before COVID-19 and then, in March this year, lockdown. Bridge clubs were no longer able to meet. There are more important things in the world; but bridge is both a lot of fun and a welcome distraction from weightier matters, and my thoughts soon turned to what we could do to continue playing in these new circumstances.

The answer was to play online; but while there are plenty of ways to play bridge online, the existing systems were not designed with the idea of being a way for bridge clubs to meet in a new context. If anything, the reverse is true: online bridge site were designed for people who could not easily get to a club or wanted to play at any time with whoever else happened to be available. Clubs like my own, by contrast, wanted to replicate their face-to-face meetings with an online equivalent. A further complication back in March was that the biggest online bridge site, called Bridgebase, was immediately overloaded and declared that it was unwilling to allow new people to qualify as directors, people allowed to run online bridge sessions.

My immediate instinct was to build a new site for playing bridge. I was not quite starting from scratch. Back in the early days of Windows 8, I started work on a bridge game for Microsoft’s new and as it turned out ill-fated platform. I had got some way with it; I had created a bridge engine that understood about cards and hands and tricks and shuffling and scoring and all the various elements that go into playing bridge. It was written in C# and what is now UWP XAML. It is designed of course for a solo player. Here is the bidding screen:

image

and the play screen:

image

This is how it looks on Windows 10; it looked a bit better on Windows 8 though it would not win any prizes for design. My software could play bridge though; the reason I never finished it was that I never cracked getting the AI working. But for human to human play that did not matter. A weekend or two coding, I thought, and I could have a website up and running so our club could play bridge online. I made an immediate start, registering the domain name YourBridgeClubOnline.co.uk.

Well, three months later and here we are.

image

image

It is, I have to say, still under development. But it works and we have been able to play bridge again, as a club.

What took you so long? Ha! Much of my old bridge engine code remains untouched and has proved useful; it all runs fine on .NET Core. Even the (useless) AI has been handy, as I can test the mechanics of play without involving others. But I had, of course, wildly underestimated the problem of converting a game for solo play on Windows, to a multi-player web application. There is much to think about:

The UI. I am not a designer (I am sure you can tell) but spent ages puzzling over how to get a workable user interface in the browser for everything from tablets to desktops. Not smartphones yet but it is coming. I decided early on to take a view on compatibility. No Internet Explorer. JavaScript fetch API is required. When time is against you, it is easier to say, just use another browser, than to waste too much time supporting old browsers.

Messaging – both the API kind, and the chat kind. I am using C#, ASP.NET Core and SignalR. In general it works well. SignalR uses WebSockets as first preference, but falls back to Server Sent Events or long polling where necessary. In my first experiments I did my own polling and switching to SignalR was a great relief.

Registration and login. I am using the stuff that comes in the box, ASP.NET Core Identity. It has saved me a ton of work. It’s a bit annoying and not too well documented. I don’t really like using GUIDs for the primary key, for example, and I believe there is way to avoid it, but it isn’t top priority when you are going for Minimum Viable Product.

JavaScript. I’ve written tons of it and I don’t even like the language. I have a new respect for it though. The thing is, it is very fast and there is nothing you cannot do. The worst thing is the friction of doing some debugging in the browser, and some in Visual Studio. I am thinking of switching to VS Code for development since it works nicely with ASP.NET Core and is better for JavaScript than Visual Studio.

Scoring. My Windows software could score a hand of bridge. But duplicate is different; you have to compare the scores with others who played the same hands and work out the percentages, then export the results to standard formats for display on club websites and submission to the English Bridge Union. It was more work than I had expected and I am not done yet; the system only understands Pairs at the moment, not Teams (a different way of scoring).

Directing. Someone has to manage an online bridge session, settle any arguments, and fix errors like cards played by accident. It all needs coding and there was nothing like it in the Windows version.

Movements. Imagine you have 28 people playing bridge (or 14 pairs). They need to all play the same hands, but never play the same hand twice, and it has to be so arranged that each pair plays against other pairs in a defined sequence so it is balanced and fair. We call this the movement. Online, you have a bit more flexibility because you don’t need to share physical cards: everyone can play the same hand at the same time if you like. It is still quite fiddly though, and I did not do any of this in the old Windows version. I saved some time by writing an import function to enable re-use of movements made for EBUScore, a widely used scoring and bridge session management application. There is more to do though.

Claims. This is where, half way through the hand, a player says, “There’s no point in playing on, I’m obviously going to win all the remaining tricks.” A trick is a sequence of four cards played one from each hand, which is won by one of the pairs. This statement is called a claim, and has to be agreed by the other players. Getting this working was more difficult than I had expected – because built into my bridge engine was the idea that you could score by counting the tricks each side had won. But claimed tricks are never played. With hindsight, I should have allowed for this from the beginning.

Database. Every detail of play has to be stored on the server. I am using Dapper and SQL Server currently, though it is possible that PostgreSQL would work just as well. I started with Entity Framework Core, still there as it is used by ASP.NET Core Identity, but I am happier with Dapper.

Things that worked well

Three months is longer than I had thought it would take to get to a playable system, but I suppose as a spare time project it is not too bad. It would not be possible without the likes of ASP.NET Core and Dapper and SignalR doing so much for you. C# is a delight for coding. I am also using an Azure App Service for all this test and development and that has worked well. I am deploying to a Linux container of course; but the nice thing about App Service is that it will scale to a considerable extent without the hassle of Kubernetes. If the project succeeds and needs to scale up, there is an Azure SignalR service ready and waiting. I was nevertheless interested to see that AWS now offers .NET Core on Elastic Beanstalk, complete with some nice Visual Studio integration. Trying it there would be an interesting experiment, though I’m not sure AWS is so savvy about SignalR.

Open Source?

Could this have been done quicker by making it open source and seeking collaborators early on? Will it become open source? I need help for sure, though I also feel the code needs some cleaning up before it is fit to share more widely. You will recall though that I had started out thinking that it would be a small matter to convert my solo bridge game to an online multiplayer web application. I figured it would be better to get something working and then ask for help. But I am open to offers! Note: this is not a commercial project.

Rewarding

Most of the software projects I have been involved in have been business applications. Bridge is a lot more fun. I do see software development as a creative act. I recall starting work on the bridge game back in 2011 (I think); starting a new blank project in Visual Studio and thinking, hmm, I had better write a class to represent a pack of cards. From that beginning I ended up with an application that could play bridge, after a fashion, and now one that multiple people can play concurrently. It is rewarding and I will not regret the time spent on it, irrespective of how much actual use it gets.

Annoying Azure capacity problems in UK West region

I have a test setup of Windows Virtual Desktop (WVD) and was experimenting with adding an additional VM. At least, I tried to. My WVD virtual network is in the UK West region. And when I try to create a VM I get the message: Your subscription doesn’t support virtual machine creation in UK West. Choose a different location.

image

This was annoying because my WVD virtual network is in UK West, so no, another region would not do. If you click Learn More you get this page which says that if you get this message and still want to deploy a VM in the region, you have to raise a support case.

I am guessing but I presume this is a capacity problem and that Microsoft is discouraging VM creation in the region. The problem for the customer is that such things are opaque; there is nowhere you can see which Azure regions are running close to capacity.

Microsoft earnings: strong quarter, but Xbox revenue dives

Microsoft has announced its quarterly financial statements, reporting revenue of $33.7 billion, up 12% on the same period last year.

The company stated that Azure revenue is up 64% year on year. Azure has overtaken the other two segments and is now the biggest, by a small amount. In addition, Azure gross margin has improved by 6% year on year.

Office 365 revenue is up 31% year on year.

Gaming was a black spot, declining 10% year on year – though Xbox Live monthly active users is at a record 65 million. The main problem is a 48% decline in the volume of Xbox consoles sold.

Quarter ending June 30th 2019 vs quarter ending June 30th 2018, $millions

Segment Revenue Change Operating income Change
Productivity and Business Processes 11047 +1379 4344 +878
Intelligent Cloud 11391 +1785 4502 +601
More Personal Computing 11279 +468 3559 +547

The segments break down as:

Productivity and Business Processes: Office, Office 365, Dynamics 365 and on-premises Dynamics, LinkedIn

Intelligent Cloud: Server products, Azure cloud services

More Personal Computing: Consumer including Windows, Xbox; Bing search; Surface hardware

A post that can save you money: scheduling Azure Virtual Machines for start/stop

I have written recently about Windows Virtual Desktop, the ability to set up a virtual desktop environment on Azure at a relatively low cost, provided your users have Microsoft 365 accounts. My test setup is minimal but I have been watching the cost which is currently working out at £5.39 per day. This excludes the cost of Microsoft 365; it is purely for the infrastructure including VPN gateway, storage and VM. Bandwidth is a variable cost but almost negligible on my usage. Of that cost, the VM is around 75%. So if I could shut down the VM when not in use the savings are substantial.

It turns out this is pretty easy on Azure though it requires some plumbing. VMs do have a built-in option to shutdown on a schedule, but not to start up. To get start/stop, you need an Automation Account.

image

With the automation account created, select it, hit Start/Stop VM, then click “Learn more about and enable the solution”.  You get this dialog.

image

Here we learn that to save money, we have to spend it, on three new services: Automation, Log Analytics, and Monitor. It is not too bad though as there is a free tier for these services that may be all I need. Hit Create.

image

In this window you have to configure three sections. Nothing challenging, but note that in Configuration you set the Target Resource Group Names. No pick list here, you have to type in the names. Or use a wildcard, which is unlikely to be a good idea since by default it will start and stop ALL your VMs. The schedule is not very smart, just a daily on and off, but see below. Once done, click Create to add the solution.

All done, but what about weekends, for example. This is easily fixed if you create your own schedules. Just go into your automation account and click Schedules under Shared Resources. The wizard-created schedules are listed, and you can modify them or create new ones. It looks as if you might need 5 schedules, one per weekday, recur every week, to make your VMs not run at weekends. There is no Monday-Friday option.

More documentation here. Note that automation can also run PowerShell scripts which will be even more flexible.

Scheduling cloud resources to shut down when not in use must be one of the most effective ways to reduce IT spend.

Update: here is the outcome of my efforts:

image

The Management resource group has the runbook that performs the start/stop action. The cost of this is small. Overall cost has gone down by about £2.00 or about 40% in my case. I appreciate this is a very small test deployment, but it would support maybe 4 or 5 users without any problem and my experience shows that you can indeed make a large saving by scheduling VMs to stop when not in use.

Microsoft is making lots of money. Anything else notable in its first quarter financials?

Microsoft has released its statements for the first quarter in its financial year, ending 30th September. Here is the segment breakdown. Everything has moved in the right direction.

Quarter ending September 30th 2018 vs quarter ending September 30th 2017, $millions

Segment Revenue Change Operating income Change
Productivity and Business Processes 9771 +1533 3881 +875
Intelligent Cloud 8567 +1645 2931 +794
More Personal Computing 10746 +1368 3143 +578

The segments break down as:

Productivity and Business Processes: Office, Office 365, Dynamics 365 and on-premises Dynamics, LinkedIn

Intelligent Cloud: Server products, Azure cloud services

More Personal Computing: Consumer including Windows, Xbox; Bing search; Surface hardware

Any points of interest? In his earnings call statement, CEO Satya Nadella talked Teams, the Office 365 conferencing and collaboration solution:

“Teams is now the hub for teamwork for 329,000 organizations, including 87 of the Fortune 100. And, we are adding automated translation
support for meetings, shift scheduling for firstline workers, and new industry-specific offerings including healthcare and small business.”

He also mentioned Power Apps and Flow, interesting to me because they are the most successful so far of the company’s efforts to come up with a low-code development platform:

“Power BI, Power Apps and Flow are driving momentum with customers and have made us a leader in no-code app building and business analytics in the cloud.”

He also mentioned the pending GitHub acquisition, which he says is “an opportunity to bring our tools and services to new audiences while enabling GitHub to grow and retain its developer-first ethos.”

Note that despite the cloud growth, Windows remains the biggest single segment in terms of revenue.

Determining how much of Microsoft’s business is “cloud” is tricky. The figures in the productivity segment lump together Office 365 and on-premises products, while Office 365 itself is in part a subscription to desktop Office, so not pure cloud. Equally, the “intelligent cloud” segment includes on-premises server licenses. No doubt this fuzzing of what is and is not cloud in the figures is deliberate.