Tag Archives: azure

cloud computing, database, microsoft, software development, tech

Inside Azure Cosmos DB: Microsoft’s preferred database manager for its own high-scale applications

June 27, 2018 Tim Anderson 1 Comment

At Microsoft’s Build event in May this year I interviewed Dharma Shukla, Technical Fellow for the Azure Data group, about Cosmos DB. I enjoyed the interview but have not made use of the material until now, so even though Build was some time back I wanted to share some of his remarks.

Cosmos DB is Microsoft’s cloud-hosted NoSQL database. It began life as DocumentDB, and was re-launched as Cosmos DB at Build 2017. There are several things I did not appreciate at the time. One was how much use Microsoft itself makes of Cosmos DB, including for Azure Active Directory, the identity provider behind Office 365. Another was how low Cosmos DB sits in the overall Azure cloud system. It is a foundational piece, as Shukla explains below.

There were several Cosmos DB announcements at Build. What’s new?

“Multi-master is one of the capabilities that we announced yesterday. It allows developers to scale writes all around the world. Until yesterday Cosmos DB allowed you to scale writes in a single region but reads all around the world. Now we allow developers to scale reads and writes homogeneously all round the world. This is a huge deal for apps like IoT, connected cars, sensors, wearables. The amount of writes are far more than the amount of reads.

“The second thing is that now you get single-digit millisecond write latencies at the 99 percentile not just in one region.

“And the third piece is that what falls out of this high availability. The window of failover, the time it takes to failover from one region when a disaster happens, to the other, has shrunk significantly.

“It’s the only system I know of that has married the high consistency models that we have exposed with multi-master capability as well. It had to reach a certain level of maturity, testing it with first-party Microsoft applications at scale and then with a select set of external customers. That’s why it took us a long time.

“We also announced the ability to have your Cosmos Db database in your own VNet (virtual network). It’s a huge deal for enterprises where they want to make sure that no data leaks out of that VNet. To do it for a global distributed database is specially hard because you have to close all the transitive networking dependencies.”

Technical Fellow Dharma Shukla

Does Cosmos DB work on Azure Stack?

“We are in the process of going to Azure Stack. Azure Stack is one of the top customer asks. A lot of customers want a hybrid Cosmos DB on Azure Stack as well as in Azure and then have Active – Active. One of the design considerations for multi master is for edge devices. Right now Azure has about 50 regions. Azure’s going to expand to let’s say 200 regions. So a customer’s single Cosmos DB table spanning all these regions is one level of scalability. But the architecture is such that if you directly attach lots of Azure Stack devices, or you have sensors and edge devices, they can also pretend to be replicas. They can also pretend to be an Azure region. So you can attach billions of endpoints to your table. Some of those endpoints could be Azure regions, some of them could be instances of Azure Stack, or IoT hub, or edge devices. This kind of scalability is core to the system.”

Have customers asked for any additional APIs into Cosmos DB?

“There is a list of APIs, HBase, richer SQL, there are a number of such API requests. The good news is that the system has been built in a way that adding new APIs is relatively easy addition. So depending on the demand we continue to add APIs.”

Can you tell me anything about how you’ve implemented Cosmos DB? I know you use Service Fabric. Do you use other Azure services?

“We have dedicated clusters of compute machines. Cosmos DB is a Ring 0 service. So it’s there any time Azure opens a new region, Cosmos DB clusters have provision by default. Just like compute, storage, Cosmos DB is also one of the Ring 0 services which is the bottommost. Azure Active Directory for example depends on Cosmos DB. So Cosmos DB cannot take a dependency on Active Directory.

“The dependency that we have is our own clusters and machines, on which we put Service Fabric. For deployment of Cosmos DB code itself, we use Service Fabric. For some of the load balancing aspects we use Service Fabric. The partition management, global distribution, replication, is our own. So Cosmos DB is layered on top of Service Fabric, it is a Service Fabric application. But then it takes over. Once the Cosmos DB bits are laid out on the machine then its replication and partition management and distribution pieces take over. So that is the layering.

“Other than that there is no dependency on Azure. And that is why one of the salient aspects of this is that you can take the system and host it easily in places like Azure Stack. The dependencies are very small.

“We don’t use Azure Storage because of that dependency. So we store the data locally and then replicate it. And all of that data is also encrypted at rest.”

So when you say it is not currently in Azure Stack, it’s there underneath, but you haven’t surfaced it?

“It is in a defunct mode. We have to do a lot of work to light it up. When we light up it on such on-prem or private cloud devices, we want to enable this active to active pathway. So you are replicating your data and that is getting synchronized with the cloud and Azure Stack is one of the sockets.”

Microsoft itself is using Cosmos DB. How far back does this go? Azure AD is quite old now. Was it always on Cosmos DB / DocumentDB?

“Over the years Office 365, Xbox, Skype, Bing, and more and more of Azure services, have started moving. Now it has almost become ubiquitous. Because it’s at the bottom of the stack, taking a dependency on it is very easy.

“Azure Active Directory consists of a set of microservices. So they progressively have moved to Cosmos DB. Same situation with Dynamics, and our slew of such applications. Skype is by and large on Cosmos DB now. There are still some fragments of the past. Xbox and the Microsoft Store and others are running on it.”

Do you think your customers are good at making the right choices over which database technology to use? I do pick up some uncertainty about this.

“We are working on making sure that we provide that clarity. Postgres and MySQL and MariaDB and SQL Server, Azure SQL and elastic pools, managed instances, there is a whole slew of relational offerings. Then we have Cosmos DB and then lots of analytical offerings as well.

“If you are a relational app, and if you are using a relational database, and you are migrating from on-prem to Azure, then we recommend the relational family. It comes with this fundamental scale caveat which is that up to 4TB. Most of those customers are settled because they have designed the app around those sorts of scalability limitations.

“A subset of those customers, and a whole bunch of brand new customers, are willing to re-write the app. They know that that they want to come to cloud for scale. So then we pitch Cosmos DB.

“Then there are customers who want to do massive scale offline analytical processing. So there is, Databricks, Spark, HD Insight, and that set of services.

“We realise there are grey lines between these offerings. We’re tightening up the guidance, it’s valid feedback.”

Any numbers to flesh out the idea that this is a fast-growing service for Microsoft?

“I can tell you that the number of new clusters we provision every week is far more than the total number of clusters we had in the first month. The growth is staggering.”

cloud computing, microsoft, mobile, tech

Strong financial results from Microsoft as it aims for breadth of services

February 1, 2018 Tim Anderson

Microsoft reported a big quarter (in terms of revenue) for the three months ending December 31st, with revenue of $28,918 million.

What’s notable? Mainly the big jump in Microsoft’s recent success stories: year on year Office 365 up by 41%, Azure up by 98%, Dynamics 365 up by 67%.

Windows is flat/weak as you would expect, and Surface hardware is standing still. Xbox grew a bit following the launch of Xbox One X.

LinkedIn is growing: revenue of $1.3 billion and “sessions growth of over 20%” in the quarter. In the earnings webcast, Microsoft’s Amy Hood said that the LinkedIn acquisition has both performed better, and seems more strategic, now than it did at the time.

Hood also made reference to the company’s ability to up-sell cloud users to higher-margin services. “Office 365 commercial revenue increased 41 percent from installed base growth across all customer segments, and ARPU [Average Revenue per User] expansion from continued customer migration to higher value offers in the E3 and E5 workloads.”

This point is key and is the answer (from the provider’s point of view) to the lower margins implicit in moving from software to services. When Microsoft sells a licence for you to use Windows or Office, the margin is huge because reproducing the software, or providing it for download, costs almost nothing; whereas with a subscription there is significant cost to providing the service. However the subscription has advantages which offset this, in particular the continuing interaction with the customer that both provides data, which the customer as well as the provider can mine (subject to appropriate privacy controls), and gives opportunity for the provider to extend the relationship into new or upgraded services.

CEO Satya Nadella fielded a good question about Microsoft losing out to Sony in gaming and to Alexa and Google Home in voice devices. On gaming, Nadella referred to the PC alongside Xbox as a strategic asset. “PC gaming is a growth market,” he said, as well as software such as Minecraft now on mobile devices, giving the company a broad reach. He also remarked on Azure as a gaming back end.

As for Cortana in the home (or absence from), Nadella said that the focus is on the server-side cognitive services. He also talked about voice input and control of Office 365. The key point though was that Microsoft wants to work both with its own and other voice assistant devices so it can win on services even when competitor devices are in use. “One-turn dialogs on one speaker in one home, that’s just not our vision,” he said.

Nadella made another key point in the webcast, in answer to a question about how Azure Stack (a packaged version of Azure for installation on-premises) will impact Azure. “Computing is becoming more distributed, not less distributed,” he said. IoT and sensors play a large part in this. Everything goes to the cloud but computing on the edge (the new buzzword for local processing) is important for efficiency.

It is easy to see ways in which Microsoft could stumble. The PC will decline as the number of users who need a desktop or laptop computer diminishes. Microsoft’s failure in mobile could prove costly as competitors use synergy with their own applications and cloud services to steer customers away. There are opportunities such as home automation and payments which seem closed to the company now.

Then again, strong results such as these show how the company can succeed by continuing to migrate its business users to cloud services. It remains deeply embedded in business computing.

Here is my chart summarising Microsoft’s performance:

Quarter ending December 31st 2017 vs quarter ending December 31st 2016, $millions

Segment	Revenue	Change	Operating income	Change
Productivity and Business Processes	8953	+1774	3337	+284
Intelligent Cloud	7795	+1037	2832	+541
More Personal Computing	12170	+281	2510	-51

The segments break down as:

Productivity and Business Processes: Office, Office 365, Dynamics 365 and on-premises Dynamics, LinkedIn

Intelligent Cloud: Server products, Azure cloud services

More Personal Computing: Consumer including Windows, Xbox; Bing search; Surface hardware

cloud computing, microsoft, tech

Microsoft Ignite: where next for Microsoft’s cloud? The Facebook of business?

September 25, 2017 Tim Anderson

Microsoft has futuristic domes as part of its Envision event, running alongside Ignite here in Orlando. Ignite is the company’s main technical event of the year, focusing mainly on IT Pros but embracing pretty much the whole spectrum of Microsoft’s products and services (maybe not much Xbox!). With the decline of the PC and retreat from mobile, and a server guy at the helm, the company’s focus has shifted towards cloud and enterprise, making Ignite all the more important.

This year sees around 25-30,000 attendees according to a quick estimate from one of the PRs here; a little bigger than last year’s event in Atlanta.

Microsoft will present itself as an innovative company doing great things in the cloud but the truth is more complex, much though I respect the extent to which the business has been transformed. This is a company with a huge amount of legacy technology, designed for a previous era, and its challenge has been, and still is, how to make that a springboard for moving to a new way of working as opposed to a selling opportunity for cloud-born competitors, primarily Amazon Web Services (AWS) and Google, but also the likes of Salesforce and Dropbox.

If there is one product that has saved Microsoft, it is probably Exchange, always a solid email server and basic collaboration tool. Hosted Exchange is the heart of Office 365 (and BPOS before it), making it an easy sell to numerous businesses already equipped with Office and Outlook. Email servers are horrible things to manage, so hosted has great appeal, and it has driven huge uptake. A side-effect is that it has kept customers using Office and to some extent Windows. A further side-effect is that it has migrated businesses onto Azure Active Directory, the directory behind Exchange Online.

Alongside Office 365, the Azure cloud has matured into a credible competitor to AWS. There are still shortcomings (a few of which you can expect to be addressed by announcements here at Ignite), but it works, providing the company with the opportunity to upsell customers from users of cloud infrastructure to consumers of cloud services, such as Azure IoT, a suite of tools for gathering and analysing data.

The weakness of Microsoft’s cloud efforts has been the moving parts between hosted services and Windows PCs, and legacy pieces that do not work as you would expect. OneDrive has been a persistent annoyance, with issues over reliable document sync and limitations over things like the number of documents in a folder and the total length of a path. And where are my Exchange Public Folders, or any shared folders, in Outlook for IoS and Android? And why does a PC installation of Office now and again collapse with activation or other issues, so that the only solution is removal and reinstall?

At Ignite we will not hear of such things. Instead, Microsoft will be presenting its vision of AI-informed business collaboration. Think “Facebook of business”, powered by the “Microsoft graph”, the sum of data held on each user and their files and activity, now combined with LinkedIn. The possibilities for better-informed business activity, and systems that know what you need before you ask, are enticing. Open questions are how well it will work, and old issues of privacy and surveillance.

Such things also can only work if businesses do in fact commit more of their data to Microsoft’s cloud. The business case for this is by no means as simple as the company would have us think.

.net, azure, cloud computing, microsoft, Uncategorized

Try RegBot

March 24, 2017 onlyconnect

You can try RegBot here:

Or with the following Skype link:

Or on Facebook Messenger.

azure, cloud computing, microsoft

From Windows Embedded to cloud: Microsoft announces the Connected Vehicle Platform

January 6, 2017 onlyconnect

Microsoft has announced the Connected Vehicle Platform, at the CES event under way in Las Vegas.

The company is not new to in-car systems, but its track record is disappointing. It used to be all about Windows Embedded, using Windows CE to make a vehicle into a smart device.

Ford was Microsoft’s biggest partner. It built Ford SYNC on the platform and in 2012 announced five years of partnership and 5 million SYNC-enabled vehicles.

However in 2014 Ford announced SYNC 3 with no mention of Microsoft – because SYNC 3 uses Blackberry’s QNX.

What went wrong? There’s a 2014 analysis from Bill Howard that offers a few clues. The bit that chimes with me is that Microsoft was too slow in updating the system. The overall Windows story over the last 10 years is convoluted to say the least, with many changes to the platform and disruptive (in a bad way) strategy shifts. The same factor is a large part of why Windows Phone failed.

It is not clear at this stage whether or not Microsoft’s Connected Vehicle Platform partners (which include Renault-Nissan and BMW) will use Windows Embedded in their solutions; but what is notable is that Microsoft’s release makes no mention of it. The company has shifted to a cloud strategy, and is primarily offering Azure services rather than mandating how manufacturers choose to consume them. The detail of the announcement identifies five key areas:

Telematics and Predictive services
Marketing (“Customer insights and engagement”)
Productivity (Office 365, Skype)
Connected ADAS (Advanced Driver Assistance Systems), ie. the car helping you to drive
Advanced Navigation

Cortana also gets a mention. We may think of Cortana as a virtual assistant, but what this means is a user interface to intelligent services.

There is big competition for all this of course, with Google, Amazon and Apple also in this space. There is also politics involved. If you read Howard’s analysis linked above, note that he mentions how the auto companies dislike restrictions such as Google insisting that you can’t have Google Search unless you also use Google Maps (I have no idea if this is still the case). There is a tension here. In-car systems are an important value-add for customers and critical to marketing vehicles, but the auto companies do not want their vehicles to become just another channel for big data-gathering companies like Google and Amazon.

Another point of interest is how smartphones interact with your car. If you want a simple and integrated experience, you can just dock your phone and use it for navigation, communication and entertainment – three key areas for in-car systems. On the other hand, a docked phone will not have the built-in screen and control of vehicle features that an embedded system can offer.

microsoft, windows

Microsoft financials April-June 2016: on track but continued drift away from consumers

July 20, 2016 onlyconnect 3 Comments

Microsoft has announced its latest financials, and I have made a quick table summarising the year-on-year comparison for the quarter. See the end of this post for what the confusing segment categories represent.

Quarter ending June 30th 2016 vs quarter ending June 30th 2015, $millions

Segment	Revenue	Change	Operating income	Change
Productivity and Business Processes	6969	+308	3000	-167
Intelligent Cloud	6711	+415	2190	-443
More Personal Computing	8897	-346	964	+359
Corporate and Other	-1963	-1943	-3074	+5384

A few observations.

Office 365 is Microsoft’s current big success. According to the company’s press release, Office 365 revenue grew 54%, which is huge. However, on-premise sales declined which meant that overall revenue growth in “Office commercial products and cloud services” was only 5%. Still, that’s a successful transition.

The picture was similar in consumer Office, with Office 365 consumer increasing by 23.1% while overall revenue grew by only 19%.

Dynamics CRM is moving to the cloud. Microsoft says that Dynamics CRM online grew by more than 2.5 times, while overall revenue grew only 6%. The maths may be deceptive, if CRM online grew from a small base, but it is a clear trend. Not to be confused with Dynamics 365, which is ERP/Business process management, though Nadella is also bullish on the latter.

Azure revenue grew 102%. Microsoft’s cloud results are not quite as sparkling as those from Amazon Web Services, but still impressive.

Enterprise Mobility is growing. This is a suite of tools built around InTune, Microsoft’s Mobile Device Management solution.

Surface is doing OK. Revenue up 9% thanks to Surface Pro 4 and Surface Book.

Windows news is mixed. “Windows OEM non-Pro revenue grew 27% and OEM Pro 2%” says the release, which given the weak PC market is decent. Windows 10 is at 350 million active devices, which Nadella said in the earnings webcast is the fastest ever adoption rate for a new version Windows; hardly surprising given the free upgrade offer and high-pressure upgrade marketing.

Xbox news is mixed. Gaming revenue is down 9%. Xbox Live revenue grew 4% but Xbox console revenue is down.

Windows Phone dives towards oblivion. Revenue is down 71%, from a base that was already tiny.

Microsoft cares less and less about consumers. “We will deliver more value and innovation” in Windows, says Nadella, “particularly for enterprise customers.” I also note the remark in the press release that “Search advertising revenue excluding traffic acquisition costs grew 16% (up 17% in constant currency) with continued benefit from Windows 10 usage,” suggesting that part of the Windows 10 consumer strategy is to use it as a vehicle for advertising; this is known in the business as “adware” and does not encourage me; it will push canny users towards Mac or Linux. In the earnings call, Nadella said that 40% of search advertising revenue is from Windows 10 devices. “The Cortana search box has over 100 million monthly active users with 8 billion questions asked to date,” said Nadella.

A reminder of Microsoft’s segments:

Productivity and Business Processes: Office, both commercial and consumer, including retail sales, volume licenses, Office 365, Exchange, SharePoint, Skype for Business, Skype consumer, OneDrive, Outlook.com. Microsoft Dynamics including Dynamics CRM, Dynamics ERP, both online and on-premises sales.

Intelligent Cloud: Server products not mentioned above, including Windows server, SQL Server, Visual Studio, System Center, as well as Microsoft Azure.

More Personal Computing: What a daft name, more than what? Still, this includes Windows in all its non-server forms, Windows Phone both hardware and licenses, Surface hardware, gaming including Xbox, Xbox Live, and search advertising.

azure, cloud computing, Notes from the field

The case of the disappearing Azure AD application registration

July 5, 2016 onlyconnect

Some time ago I wrote a simple web application which runs on Microsoft Azure and uses Azure Active Directory for authentication. The application is used constantly and has proved reliable; however yesterday it stopped working. A quick debug session showed that the problem was an Azure AD permissions error.

In order to use Azure AD, applications have to be registered in the Azure management portal. I use the old portal for this; I am not sure that the functionality exists in the new portal yet. There is a nice how-to here.

One of the elements in the registration is a key which has a maximum lifetime of 2 years:

My application was deployed about two years ago so I went to the portal to see if it had expired.

What I found surprised me. The application was not listed at all. It had disappeared.

Instead of simply obtaining a new key and updating my application config, I had to create a new application registration and update several keys in the config, which was an annoyance.

There is a wider point here, in the whole category of dealing with “things that expire”. Some time ago, Microsoft suffered an extended Azure outage because of an expired certificate. It is a shame that Microsoft insists on a maximum 2 year lifetime for this key but does not provide a check box for “alert me when this key is about to expire”, how difficult would that be?

Problems like this also mean that things which “just work” may not continue to do so. Of course a well organised enterprise setup can deal with this type of problem, but imagine, for example, the case of a small business with an application running on Azure where the developers have gone out of business, perhaps, or are no longer available. In fact the only code I needed to change was in web.config, but I can imagine it could take some time to figure out what to do and what to change.

Tim Anderson's IT Writing