Adding Excel Services Capabilities to a SharePoint 2016 Farm

Published January 5, 2016

Excel Services has been deprecated in SharePoint 2016, but the important features that it provided have been moved to Office Online Server 2016. This article describes the necessary steps.

It’s been fairly well documented that Excel Services will not be a part of SharePoint 2016, and that the bulk of its capabilities are being replaced by Office Online Server (OOS), which is the new name for the Office Web Apps server. The Excel Services features are not all available by default, and certainly not through the standard setup procedures of old. This post will walk through the process of restoring the bulk of the Excel Services capabilities to a SharePoint 2016 farm. SharePoint 2016 is currently in beta, and is scheduled to ship in Q2 2016, but the impact of these changes can be seen and tested now.

This is the first in a series of “how-to” articles that will outline how to get all of the SharePoint-based BI components up and running. The white paper, Deploying SQL Server 2016 PowerPivot and PowerView in SharePoint 2016, published by Microsoft in December 2015, goes into great detail on these topics, and my articles are meant to be a sort of “quick start” guide. In addition, these articles are based on pre-release versions of SharePoint 2016, OOS, and SQL Server 2016, so some of these steps may change, or will become unnecessary by final release time.

Configure the Office Online Server Farm

Starting with SharePoint 2013, it became necessary to run the Office Web Apps server independently from SharePoint. In fact, multiple Office Web Apps servers could be joined together to form a farm. This is also the case with OOS. Given that OOS is the only way for SharePoint to render Excel content in a browser, our first step is to set up a (single server) OOS farm.

The procedure for setting up OOS is fairly straightforward, and is outlined in the following TechNet article, so I won’t repeat it here:

https://technet.microsoft.com/en-ca/library/mt170644%28v=exchg.160%29.aspx?f=255&MSPPError=-2147217396

This article correctly lays out all of the prerequisites, and how to install them, but it’s not as complete as the Office Web Apps version of the same article for SharePoint 2013. In particular, it doesn’t discuss how to create a farm that uses http instead of https, which is important, and simpler for testing purposes. Luckily, you can use the same procedure for OOS 2016 as was used with Office Web Apps 2013, using the –AllowHTTP switch in the PowerShell creation commandlet.

Once completed, you can navigate to ServerAddress/hosting/discovery to determine if the procedure worked. In my case, the address was http://oos2016test/hosting/discovery.

Configure the SharePoint Farm

Once the OOS farm is in place, the SharePoint farm needs to be configured in order to use it. The procedure is well outlined in the aforementioned white paper, and is also identical to the procedure used for SharePoint 2013, so it is not worth repeating those steps here. Once connected, and an IISReset has been performed on the SharePoint front end server(s), all Office files should render successfully in a browser. The ability to work with Office files in a browser should now be the same as it was in SharePoint 2013 through Office Web Apps server. At this point, it will be necessary to enable or configure features previously available through Excel Services, and some additional setup is required.

The Excel web part in SharePoint is the most visible Excel Services feature, and is one of the most common ways that people interact with Excel Services in SharePoint. While Excel Services is gone from SharePoint 2016, the web part remains, and it will work with OOS, but using it does involve some configuration. It uses the SOAP-based Excel Services web services, and the new OOS server also exposes these web services. You’ll need to add the web services capability if you want to use the web part, or if you have any solutions that depend on Excel Services web services.

Again, you’ll need to use PowerShell to do the configuration, only this time, on a SharePoint server. In the following PowerShell script, you should replace OOSServer with the Fully Qualified Domain Name of the OOS farm or server.

$Farm = Get-SPFarm
$Farm.Properties.Add("WopiLegacySoapSupport", "http://OOSServer/x/_vti_bin/ExcelServiceInternal.asmx");
$Farm.Update()

within earlier setups of Excel Services, I advised my customers to immediately go to the Excel Services configuration in Central Admin and change some of the defaults because they weren’t terribly practical. Microsoft carried some of these defaults over to OOS, but there is no longer a user interface to configure them – so again, you need to do all of this with PowerShell. You can find all of the configurable settings by opening up a PowerShell prompt on the OOS server, and entering “Get-OfficeWebAppsFarm”. Figure 1 shows all of the Excel configuration parameters begin with “Excel”.

Figure 1: Excel Online configuration options

I think a few of these parameters are impractical, so I recommend changing them. The PowerShell to do this follows the same pattern in every case:

Set-OfficeWebAppsFarm –ParameterName ParameterValue

Here is a table with some of my recommended changes to OOS:

Increase the Maximum Workbook Size

By default, the maximum size of a workbook that can be opened in a browser is 10 MB. It’s pretty common to find larger workbooks, so I recommend increasing this limit. This increase will impact performance so keep that in mind, but by default, I like to set it to 250 MB.

Set-OfficeWebAppsFarm -ExcelWorkbookSizeMax 250

Turn Off Warning on Data Refresh

As with Excel Services, by default OOS will warn the user through a dialog box whenever it accesses external data. This gets to be particularly excessive because OOS considers PowerPivot content to be an external data source, and there is no way for users to say “stop warning me”. I recommend turning this option off.

Set-OfficeWebAppsFarm -ExcelWarnOnDataRefresh:$false

If You Aren’t Using Kerberos, Turn On EffectiveUserName

When the EffectiveUserName option is used, the server is able to pass the user’s identity to an Analysis Services server, allowing all queries to run under the permission of that user. It’s a way to delegate permissions without using Kerberos. If you’re not using Kerberos, you’ll likely want to turn this on.

Set-OfficeWebAppsFarm -ExcelUseEffectiveUserName:$true

Getting EffectiveUserName working properly requires another step. The EffectiveUserName feature requires the connecting service
(the one doing the impersonation) to have server admin-level access. In prior versions of SharePoint, this would be the service account that the Excel Services application used. With the move to OOS, there is no Excel Services, and the Excel Online service uses the Network Service account. The solution to this is to add the OOS server(s) to SSAS as administrators. To do this:

Open SQL Server Management Studio 2016 (SSMS).
Connect to the SSAS server.
Right-click on the server node and select properties.
In the resulting dialog box, select Security, and then press the add button (see figure 2).
Make sure that “Entire Directory” is selected in “From this location” – if it is not, select the Locations button and select it (see figure 2).

Figure 2: Selecting SSAS Administrators from Active Directory

Next, we need to add the name of the OOS server(s) to which we want to grant admin access. However, if we just enter the name in the object name box, it will fail, because the computer objects are not scanned by default. We must first select the “Object Types” button (figure 3) and select Computers and then click OK.

Figure 3: Adding Computer Accounts to the SSAS Administrator List

Now we can add our OOS server names to the administrators list, and the EffectiveUserName feature will work. Note that you must use SSMS 2016 to do this – prior versions do not allow computer accounts to be used.

Enabling Data Connections

The EffectiveUserName feature described above allows OOS to impersonate users to an SSAS server, but this isn’t available for other data sources. In past versions of SharePoint, Windows authentication could be provided through unattended access accounts, Secure Storage Service accounts, or Kerberos. The first two allow for proxy accounts, while Kerberos provides full identity delegation. With the move from Excel Services to OOS, the proxy account options are both gone, and we are left with Kerberos alone for Windows authentication. Therefore, in order to connect to SQL server data sources from a web-enabled workbook, it is necessary to either enable Kerberos Constrained Delegation (KCD), or switch connections to use SQL Server authentication. KCD is also required in order to use a SharePoint workbook as a data source in OOS (see below).

If you plan to use KCD, there is an additional step required. SharePoint uses Claims authentication, but Kerberos implies Windows authentication. In prior versions of SharePoint, the Claims to Windows
Token Service (C2WTS) on the Excel Services server(s) would be used to translate the Claims based identity into a Windows token. Now that the Excel engine lives in OOS, it’s necessary to configure the C2WTS on the OOS server as this is not done by default. The C2WTS is a part of the Windows Identity Foundation, and if it has not been installed, it can be added as a feature from the Add Roles and Features wizard on the server (figure 4). Also, make sure you install Microsoft Identity Extensions if you require ADFS support.

Figure 4: Installing Windows Identity Foundation in order to add C2WTS to the OOS server.

Once installed, you must configure C2WTS
to allow the Network Service account to use it, and you must configure C2WTS to start automatically. To do so, edit the C:\Program Files\Windows Identity Foundation\v3.5\c2wtshost.exe.config file, and remove the comment tags (<!– and –>) from the NT AUTHORITY\Network Service line. Once you have done so and saved the file, run the following from a PowerShell prompt:

Set-Service -Name C2WTS -startuptype "automatic"

Start-Service -Name C2WTS

Configuring KCD itself is outside the scope of this article, but once enabled, it should be possible to delegate user identities from the OOS server back to SQL Server.

If you decide to not use Excel embedded connections and instead use data connections (ODC files) from a data connection library, you will need to configure a server to server trust relationship between the SharePoint WFE server(s) and the OOS server(s). You must also establish this relationship if you want to use the PowerPivot for SharePoint IT Management Dashboard.

You can find the procedure for establishing S2S trust in this TechNet article. Note that you also need to set up a User Profile service in the SharePoint farm before you perform this procedure.

Wrapping Up

The move to Office Online Server brings a consolidation of technology and a few new features, but it’s significantly more involved to set up. The good news is that you don’t have to enable anything that you don’t need. If you are an established SharePoint shop that currently leverages Excel Services to any degree, you will want to carefully plan and test any new environment before making the leap. Here there be dragons.

If your customers use PowerPivot enabled Excel workbooks, setting up OOS will allow them to open these workbooks in a browser, but not interact with them. For that, you need to introduce an Analysis Services PowerPivot mode server into the environment. I will discuss that topic in my next article.

28 Comments

What’s Happening to Excel Services in SharePoint 2016

Published September 26, 2015

Excel Services is dead – long live Excel Online.

With the release of the first public preview of SharePoint Online, and the release of this TechNet article, the news rapidly went out that Excel Services was dead. And while this is technically true, there is very little cause for concern. Unfortunately, this has led to quite a bit of misunderstanding, some of which can be cleared up by reading more of the article than the parts in bold, and others by digging around a bit. The bottom line here is, there is nothing to worry about.

The first point to make is that everything that you know and love about Excel Services lives on. While Microsoft IS removing Excel Services, what’s it’s doing is reducing both complexity and confusion by moving the relevant capabilities of Excel Services to the Office Online Server. When Excel Services was first introduced in SharePoint 2007, it allowed (among other things) for Excel workbooks to be rendered and interacted with in a browser without the need to use and Excel client. When Office Web App server was introduced in SharePoint 2010, it also allowed for workbook rendering in a browser, along with all of the other Office file formats. It also allowed browser editing of those files. What it didn’t do was allow for data connections and interactions with pivot tables, etc. For this, we still needed Excel Services. In fact, in SharePoint 2013, this required an additional configuration step of turning off XLSX file rendering for OWA.

Moving the necessary capabilities to Office Online Server (this is the name for Office Web App Server) is simply a matter of consolidation and clarification, and I welcome it. However, the new server name is confusing for people, and it leads to the second big misconception, which is that Microsoft is forcing us to use Excel in the cloud to make all this work. Let me make is clear – that is not the case.

When the deprecation article was published that talked about this move to the Office Online Server, the Office Online Server was not yet available in preview (it has been released since). This fact, combined with the name “Office Online Server” led many to the conclusion was that the Office Online services would be required for browser rendering of Office documents. The important word to note in the name is “server” as opposed to “services”. The server can in fact be installed in your domain, and in fact, must be for those data connections and interactivity capabilities. I’m not sure that it’s even possible to bind an on-prem SharePoint farm to Office Online Services, but I could see that as being an interesting scenario

While this fact was not always clear, Microsoft has in fact been doubling down lately on their on-prem BI strategy – Look no further than the BI investments in SQL Server 2016. SharePoint remains the primary deployment platform for these BI investments moving forward.

10 Comments

Working with Excel Files in Power BI V2

Published August 11, 2015

In the beginning, Excel was at the center of Power BI. In the “V1” version of Power BI, the entire product was all about enabling analytics in the cloud, and Excel/Excel services was the delivery vehicle for those analytics. The authoring tools were all Excel add-ins, and the service revolved around updating data models in Excel, and allowing you to work with data models greater than 10 MB. Power BI “V2” changed that focus completely, completely removing that dependency on Office 365 and Excel. However, all of the Excel goodness that was within the “V1” product is still there, it just may have moved around a bit, and it works a little differently now.

In the early days of Power BI V1, I wrote up a post about the limitations of the product, specifically the file size limitations. Reviewing this article recently, I was actually struck by how little has changed. The maximum data model size remains 250 MB. In fact, 250 MB is the maximum size of any data model in Power BI, whether or not it originated in Excel. What has changed, rather drastically is the means of enabling this 250 MB limit for an Excel file, which we’ll get to below.

Let’s have a look at some of the major differences between Power BI “V2” and Power BI “V1” as it pertains to Excel.

More than just pretty face

In V1, Excel was almost always used as a means of presenting data. It was possible to use Excel as a data source, but the presentation of that data would inevitably be through another Excel file.

In V2, Excel is far more commonly used as a data source. The Power BI Designer, or the web interface can connect to Excel files, and then import the data into data models stored in the service. Native Power BI visualizations are then used for data presentation. These Excel files can be local, in OneDrive, or in OneDrive for Business. If the files are in OneDrive, or OneDrive for Business, the data models can be automatically refreshed when the source workbooks change. However, we can continue to work with Excel as a presentation mechanism.

If the Excel file is stored in OneDrive for Business (as opposed to OneDrive), you can connect the Power BI interface to the workbook in place. Once connected, the workbook can be viewed and interacted with through Excel Services right within the Power BI interface. To do this, from the dashboard, select “Get Data”, choose Files, select the OneDrive for Business option, select the desired workbook, and finally, select the “Connect” button”. You will then be presented with two options.

Importing data from Excel vs connecting to a workbook in place

Note that these two options will ONLY appear if the repository for the workbook is OneDrive for Business. Selecting “Import” will import the data contained in the file into a service based data model. This is the operation that will occur for all other repositories, and it uses Excel as a data source. Selecting “Connect” however connects to the workbook in place, and it adds the workbook as a report to the Power BI user interface.

Excel Workbook in the Power BI section

The workbook appears in the Reports section with a small Excel icon beside it. To view the workbook in Excel Services, click on the ellipsis to the right of the name, and select View. The workbook should load in a new window in full fidelity, and allow interaction.

No artifacts are created in the Datasets or Dashboards sections, the workbook is a self-contained unit. This is important because personal sharing can only be done through Dashboards. Therefore, Excel Services based reports can only be shared through Office 365 Groups.

Office 365 Groups

One of the biggest changes that Power BI “V1” users will notice (and need to deal with) is the fact that in order to work with Power BI in the same manner, Excel workbooks MUST be stored in a OneDrive for Business repository. With “V1”, a workbook could be stored within any SharePoint Online repository, and enabled for use with Power BI. Therefore, all “V1” users will need to move these workbooks into OneDrive repositories before the “V1” service is deprecate (Dec 31, 2015).

Every Office 365 user gets a OneDrive repository by default, and these repositories work just fine for personal use, but most current users that are using SharePoint Online will have workbooks in shared libraries in a collaboration environment. Office 365 Groups also each have their own OneDrive for Business repository, and all group members have access to that repository. Power BI V2 fully supports Groups, so this is the logical place to store all of the “V1” workbooks for collaboration purposes.

Connecting to a workbook in a Groups OneDrive is identical to the process above, but first, you need to navigate to the Group’s context in the Power BI UI. This is done by clicking on the My Workspace button (and not entirely obvious).

Groups Selector

Simply select the group and you will be working in that Group’s context.

Increased File Size

One of the biggest benefits of the original Power BI “V1” was the ability to work with workbooks that contained data models larger than 10 MB. The way that this was done was by “enabling” the workbook for Power BI as I outlined in this article. The enablement mechanism no longer exists, but the benefits are still there. The process of connecting a workbook outlined above intrinsically enables the workbook for Power BI and increases the maximum model size from 10 MB to 250 MB. In addition, the workbook can also be opened and interacted with directly from OneDrive. It is also possible to share that workbook with others in your organization, but in order to open those large workbooks in a browser, those users will need a Power BI licence.

Workbook Refresh

The ability to refresh workbooks stored in the cloud from data stored on-premises was, at the time of its introduction, the most important feature of Power BI. Each workbook would be enabled for refresh through an administrative interface, and if the data source matched a registered data source, the service would call an on-premises Data Management Gateway, which would facilitate the refresh of the workbook. This worked, but was somewhat difficult to get set up. Power BI “V2” has simplified the process tremendously. This does however mean that the process has changed.

Most refreshes are performed on a Dataset, but as mentioned above, an Excel report is self-contained, so the refresh options are available from the report itself. Simply click on the ellipsis to the right of the report and select “Schedule Refresh”.

Report refresh options

The first time this is done, you will need to enter the Data Source credentials. Once entered these will be used for subsequent refreshes. This will also need to be completed before the report can be refreshed on demand. You can also schedule the refresh time here.

If the data source is a supported cloud source, no further steps will be necessary. If it is on-premises, then it is necessary to install the Power BI Personal Gateway. This does not require administrative permissions to run (although it’s best if you do), nor does it require any special permission on the service side. The Personal Gateway is meant to be just that – personal and easy to use.

Hopefully this covers most of the major differences of working with Excel workbooks in V2 of the service. It looks like a big change, but most things are still possible, and some things significantly enhanced.

1 Comment

Where Did Power View Go in Excel 2016?

Published July 29, 2015

If you’ve been using the Excel 2016 Preview or just Excel 2016 (depending on when you read this), you may have noticed that there is no longer an option to insert a Power View report into o workbook. The reason is that it has been removed from the default ribbon in Excel 2016. It used to be on the Insert tab in the Reports Section, right beside Power Map.

Power View in Excel 2013

However, opening the Insert tab in Excel 2016 reveals it to be missing.

Power View Missing in Excel 2016

Did Microsoft remove Power View from Excel? What’s going on? Power View is still very much a part of Excel; the only change is that now it is no longer a default ribbon option. The good news is that it’s simple enough to add it back in. To do so, we need to edit the ribbon. Click on File-Options, and then select Advanced Options. The ribbon editor will appear. We can add Power View to any tab that we would like, or even create a new one, but here we’re just going to add it back to the Insert menu. To do so, expand the Insert menu. Each command must be added to a group, so we need to click the “New Group” button. Next, because I don’t think anyone will want their group named “New Group”, we want to rename it. In this case, we’ll rename it to “Reports”, the way that it used to be.

Adding a new group to the Insert tab

Next, we need to add Power View into the group. The easiest way to do this is to select “Commands Not in the Ribbon” from the “Choose commands from” dropdown. It’s a long list of items to choose from, and you’ll be tempted to look under “P” for Power View. You will be disappointed. The correct command is actually to be found in the “I”s, and it is “Insert a Power View Report”. Select that option, and click the “Add” button.

Once this is complete, Power View should once again appear in the Insert tab, in the Reports section.

I have no idea why Power View has been removed from the ribbon by default. It may just be temporary given that we’re not yet at release, but it could signal some other change. In any event, if you work with both Power View and Excel 2016, you can continue to do so.

31 Comments

Using Excel With External Data – What’s the Right Tool?

Published May 30, 2015

Excel has been used with external data for… well, as long as I’ve been using Excel. So why would anyone bother to write a blog post about this given that the capability is so mature? In recent years, Excel has adopted a number of new, and frankly better mechanisms for working with external data, while retaining the old. Given that there are now multiple tools in Excel for working with external data, it’s not always clear as to which one is the best, and unfortunately there is no single tool that wins over all, although I believe that that will be the case soon.

The answer, as always is, “it depends”. When it depends, the important thing is to understand the strengths and weaknesses of each approach. With that said, let’s have a look at all of the options.

ODC Connections

ODC (Office Data Connections) are the traditional method of accessing data in Excel. You can create or reuse an ODC connection from the Data tab in the Excel ribbon.

When using an ODC connection, you establish a connection with a data source, form some sort of query and import the resultant data directly into the Excel workbook. From there, the data can be manipulated and shaped in order to support whatever the end user is trying to do. The one exception to this behaviour is the connection to SQL Server Analysis Services (SSAS). When a connection is made to SSAS, only the connection is created. No data is returned until an analysis is performed (through a pivot table, chart etc), and then only the query results are retrieved.

When the workbook using an ODC connection is saved, the data is saved within it. In the case of an SSAS connected workbook, the results of the last analysis are saved along with it. For small amounts of data, this is just fine, but any large analysis is bound to quickly run into the data limits in Excel which is 1,048,576 rows by 16,384 columns in Excel 2013. In addition such a file is very large and extremely cumbersome to work with, but even as such, Excel has been the primary tool of choice for business analysts for years.

Data loaded into the workbook can be refreshed on demand, but it can also be altered, shaped, mashed up, and as is too often the case, grow stale. Workbooks such as these have become known as “spreadmarts” and are the scourge of IT and business alike. With these spreadmarts, we have multiple versions of the same data being proliferated, and it becomes harder to discern which data is most accurate/current, not to mention the governance implications.

SharePoint has provided a way to mitigate some of the concerns with these connections. SharePoint itself supports ODC connections, and therefore users can access these workbooks stored within SharePoint and it also allows them to refresh data from the source either on demand or on open. A single point of storage along with a measure of oversight and browser access helps to restore a modicum of sanity to an out of control spreadmart environment, but the core issues remain.

In order to help with the core issues, Microsoft introduced PowerPivot in 2009.

PowerPivot Connections

Created in PowerPivot

PowerPivot was originally (and still is) an add-in to Excel 2010, and is a built in add-in to Excel 2013. PowerPivot allows for the analysis of massive amounts of data within Excel, limited only by the memory available to the user’s machine (assuming a 64 bit version). It does this by highly compressing data in memory using columnar compression. The end result is that literally hundreds of millions of rows of data can be analyzed efficiently from within Excel.

You can see that compression at work by comparing the same data imported into an Excel workbook directly, and into a PowerPivot model with a workbook. The following two files contain election data, and represent the maximum number of rows that Excel can handle directly (1,048,576) and 25 columns.

Getting data into the model was originally (and still can be) a completely separate process from bringing it into Excel. PowerPivot has its own data import mechanism, accessed from the Power Pivot window itself. First, click on the PowerPivot tab in Excel and then click manage. If you don’t have a PowerPivot tab, you will need to enable the add-in. If you don’t have the add-in, you have an earlier version of Excel – you’ll need to download it.

Once the PowerPivot window opens, the “Get External Data” option is on the ribbon.

Once the appropriate data source is selected and configured, data will be loaded directly into the data model – there is no option to import that data into a worksheet. Once the data is in, pivot tables and pivot charts can be added to the workbook that connect to the data model much like when creating an ODC connection to Analysis Services. In fact, it’s pretty much exactly like connecting to Analysis services, except that the AS process is running on the workstation.

Created in Excel

PowerPivot, and more importantly the tabular data model was included in Excel 2013. With that addition, Microsoft added a few features to make the process of getting data into the data model a little easier for users that were a little less tech savvy, and may be uncomfortable working with a separate PowerPivot window. That’s actually part of the thinking in leaving the PowerPivot add-on turned off by default.

When a user creates an ODC connection as outlined above, there are a couple of new options in Excel 2013. First, the “Select Table” dialog has a new checkbox – “Enable selection of multiple tables”.

When this option is selected, more than one table from the data source can be selected simultaneously, but more importantly, the data will automatically be sent to the data model in addition to any other import destinations.

Even if the multiple selection option wasn’t chosen, the next dialog in the import process, “Import Data” also has a new check box – “Add this data to the Data Model”.

Its purpose is pretty self-explanatory. It should be noted that if you choose this option, and also choose “Only Create Connection”, the data will ONLY be added to the model, nowhere else in the workbook. This is functionally equivalent to doing the import from the PowerPivot window, without enabling the add-in.

Power Query Connections

When Power BI was originally announced, Power Query was also announced and included as a component. This was very much a marketing distinction, as Power Query exists in its own right, and does not require a Power BI license to use. It is available as an add-on to both Excel 2010 and 2013, and will be included with Excel 2016.

Power Query brings some Extract, Transform and Load (ETL) muscle to the Excel data acquisition story. Data can be not only imported and filtered, but also transformed with Power Query and its powerful M language. Power Query brings many features to the table, but this article is focused on its use as a data acquisition tool.

To use Power Query, it must first be downloaded and installed. Once installed, it is available from the Power Query tab (Excel 2010 and 2013).

Or from the data tab, New Query (Excel 2016)

Once the desired data source is selected, the query can be edited, or loaded into either the workbook, the data model, or both simultaneously. To load without editing the query, the load option at the bottom of the import dialog is selected.

Selecting “Load To” will allow you to select the destination for the data – the workbook, the model or both. Selecting Load will import the data to the default destination, which is by default the workbook. Given the fact that the workbook is an inefficient destination for data, I always recommend that you change their default settings for Power Query.

To do so, select Options from the Power Query tab (2010 and 2013) or the New Query button (2016), click the Data Load section, and then specify your default settings.

Data Refresh Options

In almost every case when external data is analyzed, it will need to be refreshed on a periodic basis. Within the Excel Client, this is simple enough – click on the data tab, and then the Refresh All button, or refresh a specific connection. This works no matter what method was used to import the data in the first place. Excel data connections can also be configured to refresh automatically every time the workbook is opened, or on a periodic basis in the background.

However, workbooks can also be used in a browser through Office Web Apps and Excel Services (SharePoint and Office 365) or as a data source for Power BI dashboards. In these cases the workbooks need to be refreshed automatically in order that the consuming users will see the most up to data when the workbooks are opened. The tricky part is that not all of the connection types listed above are supported by all of the servers or services. Let’s dive in to what works with what.

SharePoint with Excel Services

Excel Services first shipped with SharePoint 2007, is a part of 2007, 2010, and will be included with 2016. From the beginning, Excel Services allowed browser users to view and interact with Excel workbooks, including workbooks that were connected to back end data. The connection type supported by Excel Services is ODC, and ODC only.

Excel Services has no mechanism for maintaining data refresh. However, the data connection refresh options are supported which means that the workbook can be automatically refreshed when opened, or on a scheduled basis (every xxx minutes in the background). Unfortunately, this can come with a significant performance penalty, and once refreshed it is only in memory. The workbook in the library is not updated. The data in the workbook can only be changed by editing the workbook in the client, refreshing it, and re-saving it

Workbooks with embedded data models (PowerPivot) can be opened in the browser, but any attempt to interact with the model (selecting a filter, slicer, etc) will result in an error unless PowerPivot for SharePoint has been configured.

SharePoint with Excel Services and PowerPivot for SharePoint

PowerPivot for SharePoint is a combination of a SharePoint Service application and Analysis Services SharePoint mode. When installed, it allows workbooks that have embedded PowerPivot data models to be interacted with through a browser. The way that it works is that when such a workbook is initially interacted, the embedded model is automatically “promoted” to the Analysis Services instance, and a connection is made with it, thus allowing the consuming user to work with it in the same manner as with a SSAS connected workbook,

The PowerPivot for SharePoint service application runs on a SharePoint server and allows for individual workbooks to be automatically refreshed on a scheduled basis. The schedule can be no more granular than once per day, but the actual data within the model on disk is updated, along with any Excel visualizations connected to it.

When the refresh process runs, it is the functional equivalent of editing the file in the client, selecting refresh all, and saving it back to the library. However, there is one significant difference. The Excel client will refresh all connection types, but the PowerPivot for SharePoint process does not understand Power Query connections. It can only handle those created through the Excel or PowerPivot interfaces.

Power Pivot for SharePoint ships on SQL Server media, and this limitation is still true as of SQL Server 2014. At the Ignite 2015 conference in Chicago, one of the promised enhancements was Power Query support in the SharePoint 2016 timeframe.

Office 365

Office 365, or more precisely, SharePoint Online supports Excel workbooks with ODC connections and PowerPivot embedded models in a browser. These workbooks can even be refreshed if the data source is online (SQL Azure), but they cannot be refreshed automatically. In addition, only ODC and PowerPivot connections are supported for manual refresh. Power Query connections require Power BI for Office 365. In addition, Office 365 imposes a 30 MB model size limit – beyond that, the Excel client must be used. In short, the Office 365 data refresh options are very limited.

Power BI for Office 365

Power BI for Office brings the ability to automatically refresh workbooks with embedded data models. Data sources can be on premises or in the cloud. On premises refresh is achieved through the use of the Data Management Gateway. It also raises Office 365’s model size limit from 30 MB to 250 MB. With Power BI for Office 365 both manual and automatic refreshes can be performed for both PowerPivot and Power Query connections, however Power Pivot connections are currently restricted to SQL Server and Oracle only.

The automatic refresh of ODC connections is not supported. A workbook must contain a data model in order to be enabled for Power BI.

Power BI Dashboards

Power BI Dashboards is a new service, allowing users to design dashboards without necessarily having Office 365 or even Excel. It is currently in preview form, so anything said here is subject to change. It is fundamentally based on the data model and it works with Excel files as a data source currently, and it is promised to use Excel as a report source as well. The service has the ability to automatically refresh the underlying Excel files on a periodic basis more frequent than daily.

In order for a workbook to be refreshed by Power BI, it must (at present) be stored in a OneDrive or OneDrive for Business container. It also must utilize either a PowerPivot, or a Power Query connection. At present, the data source must also be cloud based (ie SQL Azure) but on premises connectivity has been promised.

SQL Server Analysis Services

Another consideration, while not a platform for workbooks is SQL Server Analysis Services (SSAS). Excel can be used to design and build a data model, and that data model can at any time be imported into SSAS. As of version 2014, SSAS fully supports all connection types for import – ODC, PowerPivot and Power Query. Once a data model has been imported into SSAS, it can be refreshed on a schedule as often as desired, and you can connect to it with Excel, and share it in SharePoint. You can also connect to it in Power BI Dashboards through the SSAS connector. From both a flexibility and power standpoint, this is the best option, but it does require additional resources and complexity.

Refresh Compatibility Summary

For convenience, the table below summarizes the refresh options for the different connection types.

	ODC	PowerPivot	Power Query
Excel Client	M	M	M
SharePoint/Excel Services	M
SharePoint/Excel Services/PP4SP	M	A
SQL Server Analysis Services Import	A	A	A
Office 365	M	M
Office 365 with Power BI		A*	A
Power BI Dashboards		A	A

M – Manual refresh

A – Both Manual and Automatic Refresh

* only limited data sources

The Right Tool

I started out above by saying that the selection of import tool would depend on circumstances, and that is certainly true. However, based on the capabilities and the restrictions of each, I believe that a few rules of thumb can be derived. As always, these will change over time as technology evolves.

Always use the internal Data Model (PowerPivot) when importing data for analysis.
Power Query is the future – use it wherever possible

All of Microsoft’s energies around ETL and data import are going into Power Query. Power Query is core to Power BI, and announcements at the Ignite Conference indicate that Power Query is being added to both SQL Server Integration Services and to SQL Server Reporting Services. Keep in mind that we have been discussing only the data retrieval side of Power Query – it has a full set of ETL capabilities as well, which should also be considered.
PowerPivot or ODC Connections must be used on premises

PowerPivot for SharePoint does not support Power Query for refresh. This means that you MUST use PowerPivot connections for workbooks with embedded models. If you are already using SSAS, use an ODC connection within Excel.
Power Query or PowerPivot must be used for cloud BI.

PowerPivot connections will work for a few limited cases, but more Power Query support is being added constantly. Where possible, invest in Power Query
If on-premises, consider importing your models into SSAS

SSAS already supports Power Query. If, instead of using PowerPivot for SharePoint, Analysts build their models using Excel and Power Query, they can be “promoted” into SSAS. All that is then required is to connect a new workbook to the SSAS server with an ODC connection for end users. The Power Query workbooks can be used in the cloud, and the SSAS connector in Power BI Dashboard can directly use the SSAS models created.
Choose wisely. Changing the connection type often requires rebuilding the data model, which in many cases is no small feat.

In summary, when importing data into Excel, the preferred destination is the tabular model, and to import data into that model, Power Query is the preferred choice. The only exception to this is on premises deployments. In these environments, consideration should be given to connecting to a SSAS server, and failing that, PowerPivot imports are the best option.

21 Comments

129 search results for "excel"