On Streaming and Formats in App-V

On Streaming and Formats in App-V

Tim Mangan
TMurgent Technologies, LLP
September 22, 2014
Related Product:  App-V 5.0 SP2 HF5 and prior

When we created the original application virtualization product, SoftGrid, streaming was a very important part of the implementation.  Over the years, the streaming aspect has become less important to implementations, while remaining important to the decision makers that choose application virtualization with App-V.  What some may not realize is how App-V 5 changed how streaming is done from the previously generation product.  This document identifies the differences and suggests that in most cases, streaming should not play such a large role in the decision to virtualize.

Historical Roots

Softricity, which created the original version of App-V called SoftGrid, was born out of another company called SoftwareWow.  SoftwareWow focused on Application Streaming for delivery of the applications, but did not virtualize the applications.  I joined SoftwareWow in 1999 due to my background as an expert in developing network protocols.

When we made the change to Softricity to build SoftGrid, streaming remained as one of the core properties of the product value.  We led discussions with customers based on three tenants: Centralized Delivery, Streaming, and Virtualization:

  • Back then, most software was manually installed by an IT Admin.  We didn’t have SCCM or use other Electronic Delivery Systems, with the exception of some home-grown scripting.  Install once and deliver globally was a great value proposition to the enterprise.
  • Because of our streaming background, delivery via streaming was important to us.  And we found that customers reacted extremely well to the streaming.  If they remembered nothing else of what the sales person told them, they remembered the demo.  The user doesn’t have the app, a shortcut appears, they launch the app and just the portions needed stream over and the app launches quickly.  Most competitors also implemented and featured streaming, with the exception of ThinStall (later known as ThinApp), which focused solely on the virtualization feature.
  • Solving application conflict, especially “DLL hell”, and allowing multiple versions of the same application simultaneously.

What is Streaming?

But what is “streaming”?  When viewed from the standpoint of the receiver (the virtualization client software), streaming is acting on the contents of the source (the virtual package) before receiving the entire contents.  In the case of App-V the entire contents refers to the entire App-V package file.

The concept of dynamic application delivery is more appealing when the dynamic delivery is fast.  While we can always suggest faster disks, faster networks, and faster processors, streaming is quite appealing as it allows for a more immediate consumption of the application.  Of course, in reality we usually aim to get the entire package pre-cached by pre-deploying the package before the user even asks for it.

This streaming is implemented differently in the original implementation (SoftGrid and App-V prior to 5.0) and in 5.0.  But in both cases, the streaming is implemented with an understanding that the Windows file system supports the consuming of files based on 4k file boundaries.

Streaming Prior to 5.0

Streaming with SoftGrid, and App-V prior to version 5.0, was implemented based on in two ways.

First of these was the development of a streaming source file format.  Internally known as the “Jigsaw format”, the base file format of the SFT package file included a division of internal files based on 4k file blocks, to mimic how Windows pages executable files from the local disk when running applications.  This file format, along with the file system drivers implemented at the client, allowed for the client to request individual blocks on demand.  For efficiency, the format actually packaged multiple of these blocks into a delivery chunk, and the chunk could be individually compressed for improved storage and delivery.  Originally there were 8 blocks (32kb before compression) in a chunk by default, and later 16 block chunks were used.

Second, the Jigsaw format allowed for customization of the layout of chunks within the file to optimize delivery.  We focused on optimizing the initial launch of the application, defining a “Feature Block” that would contain only those blocks needed to start the application.  This allowed for the client to make a single efficient request for the data, and because this feature block was stored contiguously on the disk, it made reading of the block when stored on rotating media more efficient also.

As a side note, we originally implemented two “feature blocks”, the first (FB1) containing the blocks to start the application, and the second (FB2) which held the remainder of the contents, which normally would be requested individually by the client on demand.  But our plans called for implementing additional feature blocks eventually.  We believed that we could monitor package use of FB2 at the client in order to define additional feature blocks that contained contents needed for a particular feature (such as the “mail merge” feature in Word).  So while never implemented, the idea was that we could have multiple of these feature blocks defined in the file format, with the final block being used as FB2 was currently used.

The package metadata included a listing of file offsets against the overall Sft file.  So if the application requested the bytes from the range 0 to 4k of DataFile2, this would be known to the client as part of Chunk N+1, which would be downloaded and decompressed if not present at the client.

This streaming could have been implemented using complete files, which is how Citrix Streaming was implemented.

SoftGrid and App-V before 5.0 also used an external XML file to control the virtual application and streaming process.  The OSD was actually a very early attempt at logically defining a software package for publishing purposes.  Originally proposed by Microsoft and Marimba in the mid-1990s, the proposal to the IETF went nowhere and was then abandoned until we picked it up and extended it for application virtualization when we built SoftGrid.  As we will see in App-V 5, the idea of a common file format to define a software package is of interest today by many technologies.

Streaming in 5.0

App-V 5.0 uses a different format to define the package.  See http://blogs.technet.com/b/gladiatormsft/archive/2014/09/20/app-v-5-on-app-v-package-modernization-with-the-opc-open-package-container-or-one-package-container-to-rule-them-all.aspx for a blog post introducing the format used.

This format is rooted in the Open Packaging Conventions (OPC) of ISO 29500-2 (http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=51459), which is the basis of formats for Microsoft Open Office (http://msdn.microsoft.com/en-us/magazine/cc163372.aspx), AppX, the package format for “Modern” applications (http://msdn.microsoft.com/en-us/library/windows/apps/hh464929(v=VS.85).aspx), and App-V (not yet formally documented).  In addition to sharing a base of ISO 29500-2 with these other formats, App-V leverages some of the XML formats used in AppX.

OPC defines a flexible container format to encapsulate a “package” that contains multiple items.  For the Open Office format, the container can represent a Word or Excel document, for example.  For AppX, the container would represent an application.  For App-V, the container represents a package of virtual applications.

OPC allows for use of the Zip format for the overall container.  Zip was originally defined by PKWare, but they made the format public in 1989 (http://www.pkware.com/documents/casestudies/APPNOTE.TXT).  OPC specifies a subset of the ZIP format for usage, and imposes a number of restrictions on options that specifically may not be used.

The ZIP format consists of a header (located near the end of the file), a central directory, and a number of records.  The central directory provides the list of files, and for each file the offsets compressed into which records.  Each record is up to 64KB (pre-compressed) of an individual file. In addition to the compressed data, the record contains the full file name, attributes, and extended attributes.  Although App-V could have added its own extended attributes as part of the record,

OPC mandates the 64-bit extensions to zip, and that only the “deflate” protocol may be used for the compression. There are also a number of other restrictions, such as allowable file-names, and a prohibition to encrypting files (you could include a file that was externally encrypted, but if you use the ZIP encryption you need to get a license from PKWare).

OPC also defines, but does not require, one specially named folder in the archive (_rels).  This folder contains xml files defining relationships.  App-V does not make use of this optional part of the OPC.

Content Types

OPC defines some particular files, which generally are optional.  One required file is the “[ContentTypes].xml” file.  The ContentTypes file defines how every file should be interpreted based on the file extension or filename.  This xml file consists of “Default” elements that map each of the file extensions found in the file to a file type and subtype, and “Override” elements that map specific files.

<?xml version=”1.0″ encoding=”UTF-8″?><Types xmlns=”http://schemas.openxmlformats.org/package/2006/content-types”><Default Extension=”dat” ContentType=”appv/vfs-file” />
<Default Extension=”dll” ContentType=”appv/vfs-file” />
<Default Extension=”HxS” ContentType=”appv/vfs-file” />
<Default Extension=”hxc” ContentType=”appv/vfs-file” />
<Default Extension=”hxt” ContentType=”appv/vfs-file” />
<Default Extension=”hxk” ContentType=”appv/vfs-file” />
<Default Extension=”xml” ContentType=”appv/vfs-file” />
<Default Extension=”rll” ContentType=”appv/vfs-file” />
<Default Extension=”XLL” ContentType=”appv/vfs-file” />
<Default Extension=”XLAM” ContentType=”appv/vfs-file” />
<Default Extension=”xltx” ContentType=”appv/vfs-file” />
<Default Extension=”IDX_DLL” ContentType=”appv/vfs-file” />
<Default Extension=”XLS” ContentType=”appv/vfs-file” />
<Default Extension=”XLSX” ContentType=”appv/vfs-file” />
<Default Extension=”CHM” ContentType=”appv/vfs-file” />
<Default Extension=”DOC” ContentType=”appv/vfs-file” />
<Default Extension=”txt” ContentType=”appv/vfs-file” />
<Default Extension=”xsn” ContentType=”appv/vfs-file” />
<Default Extension=”EXE” ContentType=”appv/vfs-file” />
<Default Extension=”GRA” ContentType=”appv/vfs-file” />
<Default Extension=”FLT” ContentType=”appv/vfs-file” />
<Default Extension=”EPS” ContentType=”appv/vfs-file” />
<Default Extension=”GIF” ContentType=”appv/vfs-file” />
<Default Extension=”JPG” ContentType=”appv/vfs-file” />
<Default Extension=”PNG” ContentType=”appv/vfs-file” />
<Default Extension=”WPG” ContentType=”appv/vfs-file” />
<Default Extension=”ADM” ContentType=”appv/vfs-file” />
<Default Extension=”odc” ContentType=”appv/vfs-file” />
<Default Extension=”INI” ContentType=”appv/vfs-file” />
<Default Extension=”ICO” ContentType=”appv/vfs-file” />
<Default Extension=”TTF” ContentType=”appv/vfs-file” />
<Default Extension=”CNT” ContentType=”appv/vfs-file” />
<Default Extension=”HLP” ContentType=”appv/vfs-file” />
<Default Extension=”manifest” ContentType=”appv/vfs-file” />
<Default Extension=”HTM” ContentType=”appv/vfs-file” />
<Default Extension=”PPT” ContentType=”appv/vfs-file” />
<Default Extension=”XLA” ContentType=”appv/vfs-file” />
<Default Extension=”CNV” ContentType=”appv/vfs-file” />
<Default Extension=”DIC” ContentType=”appv/vfs-file” />
<Default Extension=”MSG” ContentType=”appv/vfs-file” />
<Default Extension=”trx_dll” ContentType=”appv/vfs-file” />
<Default Extension=”msi” ContentType=”appv/vfs-file” />
<Default Extension=”ONE” ContentType=”appv/vfs-file” />
<Default Extension=”onepkg” ContentType=”appv/vfs-file” />
<Default Extension=”H” ContentType=”appv/vfs-file” />
<Default Extension=”ECF” ContentType=”appv/vfs-file” />
<Default Extension=”FAE” ContentType=”appv/vfs-file” />
<Default Extension=”CFG” ContentType=”appv/vfs-file” />
<Default Extension=”HOL” ContentType=”appv/vfs-file” />
<Default Extension=”SAM” ContentType=”appv/vfs-file” />
<Default Extension=”OFT” ContentType=”appv/vfs-file” />
<Default Extension=”POTX” ContentType=”appv/vfs-file” />
<Default Extension=”ACL” ContentType=”appv/vfs-file” />
<Default Extension=”LNG” ContentType=”appv/vfs-file” />
<Default Extension=”LEX” ContentType=”appv/vfs-file” />
<Default Extension=”dub” ContentType=”appv/vfs-file” />
<Default Extension=”SCM” ContentType=”appv/vfs-file” />
<Default Extension=”CSS” ContentType=”appv/vfs-file” />
<Default Extension=”OPG” ContentType=”appv/vfs-file” />
<Default Extension=”dotx” ContentType=”appv/vfs-file” />
<Default Extension=”ITS” ContentType=”appv/vfs-file” />
<Default Extension=”xsl” ContentType=”appv/vfs-file” />
<Default Extension=”tlb” ContentType=”appv/vfs-file” />
<Default Extension=”rdlc” ContentType=”appv/vfs-file” />
<Default Extension=”xap” ContentType=”appv/vfs-file” />
<Default Extension=”config” ContentType=”appv/vfs-file” />
<Default Extension=”xrm-ms” ContentType=”appv/vfs-file” />
<Default Extension=”mui” ContentType=”appv/vfs-file” />
<Default Extension=”TTC” ContentType=”appv/vfs-file” />
<Default Extension=”OCX” ContentType=”appv/vfs-file” />
<Default Extension=”ORP” ContentType=”appv/vfs-file” />
<Default Extension=”PSP” ContentType=”appv/vfs-file” />
<Default Extension=”chr” ContentType=”appv/vfs-file” />
<Default Extension=”SHP” ContentType=”appv/vfs-file” />
<Default Extension=”INF” ContentType=”appv/vfs-file” />
<Default Extension=”ELM” ContentType=”appv/vfs-file” />
<Default Extension=”OLB” ContentType=”appv/vfs-file” />
<Default Extension=”man” ContentType=”appv/vfs-file” />
<Default Extension=”CAB” ContentType=”appv/vfs-file” />
<Default Extension=”eftx” ContentType=”appv/vfs-file” />
<Default Extension=”thmx” ContentType=”appv/vfs-file” />
<Default Extension=”ODF” ContentType=”appv/vfs-file” />
<Default Extension=”pri” ContentType=”appv/vfs-file” />
<Default Extension=”propdesc” ContentType=”appv/vfs-file” />
<Default Extension=”gpd” ContentType=”appv/vfs-file” />
<Default Extension=”cat” ContentType=”appv/vfs-file” />
<Default Extension=”MOF” ContentType=”appv/vfs-file” />
<Default Extension=”WAV” ContentType=”appv/vfs-file” />
<Default Extension=”kic” ContentType=”appv/vfs-file” />
<Default Extension=”CPL” ContentType=”appv/vfs-file” />
<Default Extension=”msp” ContentType=”appv/vfs-file” />
<Default Extension=”WMF” ContentType=”appv/vfs-file” />
<Default Extension=”MID” ContentType=”appv/vfs-file” />
<Default Extension=”BMP” ContentType=”appv/vfs-file” />
<Default Extension=”BDR” ContentType=”appv/vfs-file” />
<Default Extension=”ICM” ContentType=”appv/vfs-file” />
<Default Extension=”PUB” ContentType=”appv/vfs-file” />
<Default Extension=”POC” ContentType=”appv/vfs-file” />
<Default Extension=”DPV” ContentType=”appv/vfs-file” />


<Override PartName=”/AppxManifest.xml” ContentType=”application/vnd.ms-appx.manifest+xml” />
<Override PartName=”/AppxBlockMap.xml” ContentType=”application/vnd.ms-appx.blockmap+xml” />


</Types>

 

Although I am unsure if the [ContentTypes].xml file is actually used by the App-V client, the sequencer does product such a file as part of the package, which would be necessary to claim compliance with the format.  As shown in the example (from the virtual Microsoft Office 2013 package), there are only two “override” elements:

  • AppXManifest mapped to “application/vnd.ms-appx.manifest+xml”.
  • AppXBlockMap mapped to “application/vnd.ms-appx.blockmap+xml”.

All of the remaining file extensions contained in the package are mapped to a general “appv/vfs-file” file type and subtype.   In this case, vfs means that it is a file in the package and does not specify PVAD versus VFS.  This default types includes the xml extension which covers the remaining xml metadata files located above the “root” folder.

AppXBlockMap

The AppXBlockMap file contains information for each file in the zip archive which appears to be more about file verification.  Each file element in this xml has the uncompressed file size, and logical file header size, plus a sub-elements for each of the zip records.  The sub-element indicates the uncompressed size of the block, and a calculated hash for the block. Given that this information is largely discoverable via the central directory with the exception of the calculated hash, it seems reasonable to assume that this is used as an extra level of verification used in AppX and borrowed by App-V.

AppXManifest

The AppXManifest is an extended version of what AppX uses to define applications.  App-V adds to the syntax defined by AppX to add virtualization specific information.  A subset of the AppXManifest file is produced external to the package by the sequencer in the form of the DeploymentConfig.xml and UserConfig.xml files.  It is part of the metadata required by the client, but not part of the streaming format.

StreamMap

The StreamMap.XML file is what defines the feature blocks of App-V 5.  In App-V 5, there are three feature blocks, the first two of which are roughly equivalent to FB1 of the old system.

  • Primary Block:  This block includes all file (parts) of the archive that are necessary to add the package to the client during “add-appvclientpackage”.  Certain required files (those above the “root” folder) are assumed and not listed but are automatically brought in by the client.
  • Publishing Block: This block includes file parts needed for publishing.  This will include portions of each application exe (even if not launched, the client needs the file header brought down), the icon files (for the shortcuts to look right), and any file parts launched during streaming configuration.

The files listed may contain sub-elements listing the blocks involved (the entire file is needed when no block elements are recorded).  These block elements consist of a starting block number, and the number of blocks needed.  The block number refers to which compressed block of the file (1 through N).

The remaining files and file parts are part of the third (unnamed) feature block and are streamed on demand.

If during streaming configuration in the sequencer, you select the checkbox to “Force application(s) to be fully downloaded before launching” then the StreamMap will consists of only a PrimaryBlock with an indication for full loading.

FileSystemMetaData

According to the Zip format, the central directory may contain records for both folders and files.  OPC, and App-V, only record files in the central directory.  The file names provide the relative path information, allowing the client to be able to extrapolate the folder objects needed in the Cache.

Unless there is a folder in your App-V package that contains no files.  While I am unsure that OPC prohibits the inclusion of directory objects in the zip central directory (the specification is a little fuzzy about that), it is not something that the Open Office Format would expect.  So the designers of the App-V file format chose to create a special XML file to hold this and other special file-system data.

The FileSystemMetaData file contains two types of XML elements.  The first lists the path for all “empty” folders.  This is why you don’t see any empty folders if you just open the AppV file using a standard zip utility (which is unaware of the special purpose of the FileSystemMetaData file).

The second element type is for defining “opaque” folders.  These are folders (whether empty or not) that are marked OverrideLocal in the virtual file-system editor.  This setting could have been added as a special attribute to a folder object in the zip archive central directory, if folders had been added.

A potentially missing element in this file would be folder “deletion markers”.  Just as registry keys and registry items can be deleted while sequencing and recorded, potentially folders and files deleted while sequencing could have been recorded.  App-V 4.* had deletion markers for both the registry and the file systems, while App-V 5 supports only registry system deletion markers.  I am unsure why this was skipped, but honestly I can’t (currently) think of an application where I require that feature.

Summary

Streaming is alive and well in App-V 5, but do you really care?  I say that streaming is important, but configuration of the streaming in a package is often irrelevant.

Most App-V implementations do not require streaming configuration.  It only affects the first time use of the application package by the user.  The addition of “background streaming” in recent years makes real-time “fault streaming” a rare event.

For most App-V environments, you either want to configuring the streaming by launching nothing (because you are using the App-V Server for publishing in a semi-persistent environment like XenApp), or use the checkbox (because you are using SCCM with BITS, aka “download and run” for delivery).

Streaming is important in non-persistent VDI, although you typically enable Shared Content Store Mode, so streaming configuration is irrelevant (everything is fault-streamed without caching).

References

Other Authorities

http://blogs.technet.com/b/gladiatormsft/archive/2014/09/20/app-v-5-on-app-v-package-modernization-with-the-opc-open-package-container-or-one-package-container-to-rule-them-all.aspx………………………… 3

http://msdn.microsoft.com/en-us/library/windows/apps/hh464929(v=VS.85).aspx……………………… 3

http://msdn.microsoft.com/en-us/magazine/cc163372.aspx…………………………………………………….. 3

http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=51459………… 3

http://www.pkware.com/documents/casestudies/APPNOTE.TXT……………………………………………… 3

 

Updated solution to Paint.Net plugins in a connection group

Maybe Steve Thomas isn’t as crazy as he looks? I thought he went a little nuts when he wrote this blog post and suggested that you should always sequence using VFS. I mean, that is dead set against everything Microsoft has ever said. I did not disagree with the idea of VFS’ing things, but I have never said to VFS all of your packages. But now I’m not so sure he is all that crazy.

In February, I posted a solution to Paint.Net plugins delivered via App-V Connection Groups (http://www.tmurgent.com/TMBlog/?p=1913). It was ugly, but instructional for solving similar problems. In our second Eastern Europe class last month, a student (let’s call him ‘C’ since I don’t have permission to use his name) finished the lab early and tried one of the things that I had said wouldn’t work. With Success!

It seems that Microsoft made a change in the Hotfixes that makes Connection Groups easier, and perhaps we didn’t really understand the small print that came with the hotfix all that well.

Ok, first of all there probably is not a great reason to use Connection Groups with Paint.Net plugins. Just put them in the main package and be done with it. But paint.net demonstrates a different sub-style of locating plugins than is normally used by an app. The primary types are:

  • Registry Key Style Browser apps, for example (Internet Explorer and others) use a registry style integration. In this style, the application supports additional plug-ins in the form of dlls. The Plug-in typically may be located anywhere on the disk, but is registered by adding a registry entry under a known app-specific key. Normally, the value of the registry item is the full path to the dll file. At app startup, the app reads all of the registry items under that key and loads the dlls into its memory space. Registry style plug-ins generally work well with App-V and Connection Groups.
  • Folder Style In this style, the application supports additional plug-ins in the form of dlls also. The Plug-in, however, must be located in a known specific folder. The application might use a hard-coded path, or a registry based string to know the location of the folder, or it might use a relative path from the current working directory, or it might do what Paint.Net does. In this style, the app looks for dlls in this folder and loads them all into its memory space. Thanks to PVAD and VFS, these styles may or may not be troublesome for App-V Connection Groups.

In the Folder Style, installing the main app to the PVAD generally caused a problem, because while you can specify the same PVAD folder for the plug-in, the PVAD folders do not overlay. Instead, the PVAD is actually moved at the client based on GUIDS, and the different packages in the connection group will have different GUIDs, such that the plug-ins are never seen.

In the folder style, installing the main app to the VFS, and plug-ins to the same VFS area seems to work when the application locates the folder using a hard-coded path, or a registry string based location. When the application uses the current working directory of the application, by default this will be rewritten as the GUID path of the primary program, but we can manipulate the current working directory to the VFS location by manually manipulating the DynamicConfiguration.xml file, and this will work. Technically, this working directory doesn’t exist until the virtual environment is open, but you can specify a fictional working directory when you launch an application so Windows doesn’t care that it isn’t present until the app starts running.

But then there is the folder sub-style that Paint.Net uses. We can’t be sure exactly what the code does, but it appears to locate the location of the executable’s exe file (probably using a programming technique referred to as reflection). And in App-V, whether or not the app is VFS’d, this location will always result in a path under the App-V cache GUIDs of the package containing the EXE. Even when the primary package is VFS’d, this meant “AppV\Guid\Guid\Root\VFS\…”. So the plug-in would not normally be seen.

My previous trick was to think like the client. This involved sequencing the main app in the PVAD, and sequence the plug-in by first installing Paint.Net exactly like it would appear at the client (under AppV\Guid\Guid\Root), and then VFS install the plug-in.

What ‘C’ found, when sequencing using 5.0 SP2 with Hotfix 4 and testing on 5.0 SP2 client with Hotfix 5, was that he could VFS install Paint.net and then also VFS the plug-ins — and it worked. I know that this did not work in the original 5.0, and I am 95% sure that it did not work in 5.0 SP2 (since I posted this after the release of SP2). So it probably came in with one of the hot-fixes. So what changed?

Well, it appears that a further merging of the VFS has been added in the hotfixes, probably in Client Hotfix 5. In the past, the VFS paths only merged when the program tried to read from the overlayed location. So previously, with both packaged VFS’d, only if the app requested C:\Program Files\Paint.Net\Effects would the merged files be seen. Since the app requested “C:\ProgramData\AppV\Guid\Guid\Root\VFS\ProgramFilesX86\Paint.Net\Effects”, previously the app would only see the files of the package using those GUIDs. But now, it seems that a request to this folder is redirected to appear as the C:\Program Files\… counterpart where merging, not only with other packages in the Connection Group, but potentially with local (native) files when the parent folders are marked merge with local.

This change, along with the change to let you choose to allow the application to update data and setting files (but not executable components) located in the VFS, make it much more reasonable to sequence to the VFS.

But should you regularly sequence to the PVAD or VFS?

Microsoft Sequencing “Best Practices” say PVAD first, unless you have a problem.

Microsoft PFE MCS Steven Thomas (known as @MadVirtualizer on Twitter and “The Gladiator” when he blogs on TechNet), recently blogged that he is changing his mind and that perhaps you should always install to the VFS due to a rash of issues that he has resolved by changing a PVAD install to a VFS one. I am guessing that Steve’s post should have referenced that he assumed that you are using 5.0 SP2 HF5 (or above).

I am not sure yet whether this is a good “standard practice”. I do know, from my research papers on App-V 5.0 SP2 performance that there is a small, yet measurable, performance penalty to opening files in the VFS. I also know that I had issues with non-PVAD installs of certain apps in the past (at least before the fixes), and issues with other apps unless I VFS’d them. With App-V 4.x, I argued that VFS’d installs were generally OK and I usually defaulted to the practice unless it caused a problem. But in App-V 4.x I never found the reputed performance hit for VFS that I see in 5.0.

So I am not (yet) confident that VFS first is the wisest default, but whether you PVAD or VFS, I would recommend always starting with one, and change the practice on a per-app basis only if you encounter a problem. But stay tuned, we are sure to address this topic again as we learn more.

Errror 000012 on Add-AppvClientPackage

A customer was using XenApp with PVS. The same PVS image was used on machines on a farm. They added App-V 5 to the PVS image and decided to use the PowerShell interface to add applications virtually at boot time. One of the servers refused every app, while everything else worked on the ‘identical’ servers.

Investigation showed that the Add cmdlet was failing with Error 0000012.

No other messages in the event logs, even turning on the debug logs.

In further investigation, the customer had configured the App-V client to place the package store on a secondary partition. On boot, while Windows was wiped out, this partition remained. The folder referenced existed on both good and bad machines, but was empty on the bad machines.

Looking at permissions, since the standard Windows error 12 is an Access error, the security settings and standard attributes of the good and bad server folders were Identical.

Solution: The App-V client protects the package cache. As such, it expects to create and set special attributes on this folder and everything under it. Possibly someone manually created this folder, causing the problem. The solution was just to delete the folder referenced in the registry config. The next time an app was added, the App-V client recreated the folder correctly, and everything worked.

The long term solution was to add the deletion of this permanent cashed to the bootup script. They also could reconfigure the client to the ‘standard’ location, but reducing writes to the PVS partition is probably preferred.

App-V 5 Error C8007005 Plus Removing App-V Client after Hotfixes

Here is the scenario.

I had my Windows 8 x64 desktop with App-V 5.0 SP2 installed and had added Hotfix 5. All was well.

I upgraded to Windows 8.1. Current packages continued working fine. Decided to add another package. I received error C8007005 on Add-AppVClientPackage. After researching and trying some things, I eventually tried an old package that worked elsewhere and got the same error. I even removed an existing, working, package and found that adding it generated the same error.

In a situation like this, you just want to remove the App-V client and start over. So I did the following:

  • Ensured that no virtual apps were running (e.g: explorer)
  • Ran a “Repair” operation on all published apps
  • UnPublished and removed all apps.
  • Uninstalled the App-V client (using Programs and Features)
  • Rebooted
  • Followed Nick Kallen’s list of additional stuff to remove
  • Rebooted again, just for fun

I then attempted to install 5.0 SP2 Client again. The Installer refused because a newer version was present already . Confused, I checked Programs and Features again and made sure it wasn’t present. It wasn’t. Eventually, I realized that the component to be removed is the hotfix. So in “Programs and Features” I selected the “View Installed Updates” link on the left side of the dialog. And this is what I found:

Removing the hotfix allowed the installs (SP2 and HF5) to happen, and everything worked out dandy.

Sequencing AutoCAD 2015 in App-V 5

The stars had aligned! I had finished all of the testing for The Deployment Performance Series research, published the first of the 8 papers, shipped my server down to North Carolina for our next training class. And great summer weather was set for a great week of golf. Then my knee acted up so I needed to give it a couple days of rest. Feeling rather pissed off, I decided to work on something horrible. Which is good for you.

I first sequenced the 2000i version of AutoCAD back in 2001. Autodesk was one of the early ISVs to really get in and support the concept of virtualizing applications. They didn’t have as many versions back then, I recall just one version of AutoCAD plus 3DS Max. But it was still big and hairy and it took more than a few passes to get it right.

Over the years, the software designers at AutoDesk are seemingly always using some cutting edge developer stuff inside their annual releases. And sometimes we struggle with how to make that work for a couple weeks, but it always does. Until a couple of years ago. The first Autodesk version I ran into with a seemingly unsolvable problem was AutoCAD Architecture 2012. Then some versions of the 2013 series, like Revit and Creation Suite seemed to have the same kind of launch issue. This was against a 4.6 sequencer, but testing against the new 5.0 showed the same symptom. The app would launch and immediately spit out an application specific error that it was not installed properly, and all of the procmons and xperfs could not help. I worked a trouble ticket with Microsoft for one customer for about 9 months before we gave up in frustration. This unanswered forum post remains.

So on a whim I pulled out App-V 5.0 SP2 with Hotfix 4 and AutoCAD 2015 and went to work in a pissed off mood. And I got it working. Here’s how.

Step 1.  The first thing you do with any Autodesk product is deal with the dependencies. There are some dependencies that are incompatible with App-V, and others that are inconvenient. These must be identified and the installer configured to not install them. You then treat them as dependencies to be installed on the sequencer prior to sequencing, and pushed out to any client machines natively that are going to get the working package.

To configure the installer, which will be a setup.exe, you modify the setup.ini file. Make a backup copy of it and then edit the setup.ini file using your favorite text editor like Notepad.

The ini file is organized in sections. A section starts with a line contained in square brackets, such as [ACAD]. You start in the [SETUP] section at the top of the file. There are several lines that set variables like “EXE_SEQUENCE”. Depending on how you start the installer, one of these is used, and it identifies which of the remaining sections will be run. I simply ignore these and make all fixes in the sections themselves.

#============================= Install Execute & UI Sequence =============================
EXE_SEQUENCE=ACAD;NLM;CADM;SNAP;AUTODESK_RECAP;ACADAPPMGR;ACADFEATUREDAPPS;PERFORMANCETOOL;GLUE_PLUGIN_AUT;ADSYNC;ADAPPLICATIONMANAGER
INSTALL_SEQUENCE=NLM;CADM;SNAP;ACAD;AUTODESK_RECAP;ACADAPPMGR;ACADFEATUREDAPPS;PERFORMANCETOOL;GLUE_PLUGIN_AUT;ADSYNC;ADAPPLICATIONMANAGER
UI_SEQUENCE=LaunchDlg;BeginDeploymentDlg;LicenseDlg;ProductInfoDlg;ProductSelectionDlg2;ACAD;CADM;SNAP;ADSYNC;ADAPPLICATIONMANAGER;FullProgressDlg;InstallCompleteDlg;DeploymentCompleteDlg;DeploymentFailedDlg

Inside a section, such as the [ACAD] section, will be a variable called “PREREQUISITE”. This variable defines other sections that are to be installed before installing the software named in the EXE_PATH variable in the section. I make a copy of the PREREQUISITE line and comment out the old copy (lines starting with a “#” are comments). I then just remove items from the list that I want to treat as dependencies.
[ACAD]
PLATFORM=NATIVE
PRODUCT_NAME=Autodesk® AutoCAD® 2015
#PREREQUISITE=ACAD_VISTA_BLOCK;IE;SSE2;DOTNET45;DOTNET45LANG;VCREDIST2008SP1X86;VCREDIST2008SP1X64;VCREDIST2010SP1X86;VCREDIST2010SP1X64;VCREDIST2012X86UPD4;VCREDIST2012X64UPD4;DIRECTX;MSXML6;WMF95X64;CM;CMILB;SNAP_PRE
PREREQUISITE=CM;CMILB;SNAP_PRE
POSTREQUISITE=ACAD_PSPACK;ACADSKETCHUPIMPORT
EXE_PATH=%platform%\acad\acad.msi
EXE_PARAM=

If you want to know what any of the dependencies are, you locate that section and look at its lines. Some, like IE just check to see if you have internet explorer and bomb out if it isn’t present. Others, like DOTNET45, install things. Typically the installer doesn’t first check to see if it is already present, so I prefer to remove the entry as a prerequisite as shown above.

Only DOTNET45, DOTNET45LANG, and DIRECTX are incompatible with the current sequencer. WMF and MSXML6 were App-V 5 prerequisites and will already be present. Those VCRuntimes are things I still prefer to keep out of my packages anyway. By looking at the prerequisite’s section, I can find the version of the installer they would have used (typically under a folder called 3rdParty). Especially with the VCRuntimes, it is important to use the version that they provide. Autodesk loves to use names, like “VC++ 2008 SP1″ that isn’t exactly the same version you’ll find on the internet with that name. In some cases over the years I have found them using VC Runtime builds that apparently never appeared on Microsoft’s public download sites. I maintain a list of VC Runtime versions here to aid in identifying versions.

You must walk through the entire file editing these prerequisites (hint: use the find function of notepad on “PREREQUISITE”) and remove the item from each. You can even be pedantic and remove the offending section completely also. Save off the edited file.

You can get bonus points for figuring out how to set your license information in the file, and removing features you don’t want, but I like to handle that through the GUI when sequencing anyway.

Step 2.  Install the prerequisites on the sequencer. My preference is to order the installs the same way that AutoDesk would have. And then reboot (if you used the command line arguments you found in the appropriate section you instructed the installer to suppress reboot notifications). And then take a snapshot. You probably won’t get the package right the first time, even with these notes to help.

Step 3.  Start the Sequencer, but not a Sequence yet.

In the Tools–>Options menu of the sequencer remove the LOCAL APPDATA exclusion. Autodesk likes to put some important stuff here that needs to be in the package. If you want to be a perfectionist, add AppData/Local/Microsoft to the list.

Step 4.  Start Sequencing. Note that if you run the installer off of a share (or even a local path) that has the path to the setup.exe longer than around 100 characters, it will fail (there actually is a section that checks for that in the setup.ini).

Take care of the licensing, select your features and go. At the finish, it prompts you to reboot. You need to immediately perform the reboot, then immediately log back in. I found that if you wait to long the sequencer doesn’t work right (at least on this package). The sequencer will pick back up and tell you to continue installing. And while you might think that you are done, you are not.

Step 5.  Wait.

The main installer will check for updates. There isn’t any right now, but Autodesk usually produces updates almost monthly. If any appear, you probably want them now. Click the Finish button for the installer when done.

Step 6.  If you included the “360″ feature, wait more. Maybe 15 minutes.

This uses a different updater and will automatically download the update in the background (there is one already) and then notify you above the icon tray. Install it.

Reboot again.

Step 7.  Fix FlexNet.

Autodesk uses Flexera’s FlexNet service for licensing. To assure the ISV that users won’t tamper with the licensing service, Flexera hardens the service security in a way that is incompatible with App-V and the client won’t be able to start the virtual service for you. Here is how I fix it.

Start a command prompt using “Run As Administrator”. Yeah, I know that you’re already an admin, but you have to elevate anyway. Then type in the following command. Scratch that, copy and paste it!
sc sdset "FlexNet Licencing Service 64" D:(A;;CCDCLCSWRPWPDTLOCRGASDRCWDWO;;;WD)

If you want to know what that is, get some scotch and google DACL. NOTE: If you have more than one virtual app running at the same time with Flexnet licensing, you’ll need to remove the service from the package and run it externally.

Step 8.  Kill the autoupdaters.

Use the tray icon for “app manager”, go to settings and uncheck box to check for updates on logon.

Afterwards, use the windows services manager and change AutoDesk Application Manager service to Manual or Disabled. (Hint: Disabled is better).

Step 9.  Avoid the AppPath bug.

My friend Rory Dan Gough (@packageologist) recently noticed that AppPaths don’t work under App-V they way they should. When a program declares an AppPath it is supposed to be used to prepend the identified paths in front of the path variable, but he noticed it was replacing the path variable. AutoCAD 2015 has two executables declaring an AppPath to prepend a common autodesk folder.

Use RegEdit and locate them under HKLM\Software\Microsoft\Windows\CurrentVersion\AppPath. You are looking for two keys named something like”AdAppMgr” which will conveniently enough be at the top of the alphabetized list. Make a copy of the path listed under the key, and delete both keys.

Then use an elevated cmd prompt to set the path variable like this:
set PATH="C:\Program Files x86\Common Files\Autodesk Shared\AppManager\R1";%PATH%

Step 10.  Avoid the TRUSTED CODE problem.

Autodesk engineers invented a new way to check that their program doesn’t launch something untoward. It breaks things at the client complaining that the file being run isn’t in the trusted path. So I edited the variable for it here.

Regedit and change
HKCU\Software\Autodesk\AutoCAD\R20.0\ACAD-E001:409\Profiles\< >\\Variables\TRUSTEDPATHS to a value of C:\Program Files\Autodesk\AutoCAD 2015/support/en-US

Step 11.   Don’t run AutoCAD! If you need to configure anything, figure out a different way to get it set.

Step 12.  Fix Stupidity.

Advance through the sequence (again, Don’t launch!) and get to the sequence editor. On the Files tab, locate the following folder:
Root\VFS\appdata\roaming\autodesk\AutoCAD 2015\r20.0\enu\

Right click on the folder and set to “Merge with Local”. Do the same for its parents up the tree.

What’s up with that? It seems that the runtime will add a couple of DLL files into that folder the first time the user opens a drawing. Somehow it manages to do this outside of the virtual environment, so the package folder hides it and you get prompted with an error that it can’t find the dll. Who the h3ll puts dlls in the user’s appdata roaming folder? What were they thinking over there?

Step 13.  Finishing off.

Clean up your shortcuts and FTAs as desired. I went to the advanced box and checked all three options. Did I need to? I don’t know, but it worked. You should probably try it without.

Here is a summary of the package contents, taken from AppV_Manage:

What’s New in App-V Manage 3.2

AppV_Manage is a free tool from TMurgent used to test newly sequenced App-V 5 application packages. Versions 3.1 and 3.2 have added integrations to round out this phase of the packaging experience.

  1. Attribution: nokhoog_buchachonAutomation  When Citrix announced the end of life for Citrix Streaming (See What’s a Citrix Streaming Customer to do now?) there was a need for a tool to help customers convert their packages to App-V. Last week at their annual customer show Synergy, Citrix’s David Wagner and Mohit Dwahan showed off a new tool they are releasing soon to help. [UPDATE: Citrix has released this tool: See blog. ] While repackaging directly into App-V from the original installer might be a better choice, if you have a large inventory Citrix Streaming packages you could leverage this tool and test to see what works. Their idea is to use the tool with a PowerShell script and the PowerShell interface to the App-V Sequencer to automate the creation process.

    Citrix asked me if AppV_Manage could be used with an automation script as part of converting Citrix Streaming packages to App-V packages. What they wanted was a command line that could be run via PSExec from another machine and return an error code indicating if the package was good or not. It is a great idea, and could also be used in automating App-V 4 package conversion, or even as part of a packaging service.

    Adding in a command line to accept a package, and to automatically add and publish was the trivial part. I also wanted to launch the apps. All of these steps might create App-V errors which I already track, so detecting issues short of app launch failures from the App-V end was easy. But I wanted more.

    So after launching all of the shortcuts, all on-screen text elements (dialog box titles or text boxes) are scanned and passed through a set of filters. The first filter is an exclusion filter. If a text item contains an exclusion string, the item is ignored. The second filter is a error filter. If a remaining text item contains this string, we declare a failure.

    These filters are configurable via registry. The installer pre-populates a set of strings for the filters based on some limited testing that I performed. Hopefully I will get some feedback and we can improve the filters over time.

    No amount of automation can possibly detect all errors. The way that apps express errors could be more numerous than the kinds of bugs that lurk in them. But this is a quick start. Although AppV_Manage will exit to return a code to a potentially waiting script, it does leave the virtual apps running. This allows for a human to now step in, connect to the VM, and test the app further.

    Details on the command line syntax and filters may be found in the online documentation here: Automated Testing.

  2. Attribution:  thanunkorn

    Antivirus/AntiMalware  We need to sequence on a machine without Antivirus/Antimalware software enabled. Because the .AppV file is an archive format with an extension not generally recognized by AV software, certain kinds of bad things can be contained in the AppV file and not detected immediately.

    This doesn’t mean that the malware can be run on a machine with a good AV product running; it is only a question of when the detection occurs. I wanted to be able to find all (or at least as many as practical) kinds of malware as early as possible and as simply as possible.

    The best way to do this is a full file system scan with every knob that the vendor provides enabled. But even on a clean VM with just one package, this can take a long time to complete.

    Publishing the package to a client machine will help, as it eliminates the unknown archive layer. But, unfortunately, streaming is implemented in a way that the bits are not scanned (by any of the AV vendors I have yet tested) on the way down to the cache. Instead, they are scanned when accessed.

    Some AV vendors support command line scanning, which allows us to request a scan of just a specific file, or sometimes folder. Some do not.

    AppV_Manage will now detect the presence of a number of AV products. The results of this detection is displayed on the Tool Config tab. If you hover the mouse over the AV Vendor result field, a tooltip will tell you what kind of support is available.

    Meanwhile, on the Publishing tab, there is a new button called “AV SCAN”, located just to the right of the Mount button. After you add and publish, you should mount the package and then click the “AV SCAN” button. Based upon the AV vendor support, the button will do the best it can to get your AV vendor to detect any malware without triggering a full disk scan.

    For some of these vendors, the scan is complete and yet very fast. Because I assume this VM is already clean before the package is added, we’ll skip the scanning of memory first when supported.

    For more information on this, see the online documentation AV Scan.

  3. What’s in Your Package?  The Publishing tab also has an “analyze” package button, added in 3.1. This analysis determines and displays all of the various “extensions” contained within the package. Unfortunately the sequencer only tells us about a few of these, like shortcuts and file type associations (but not even all of those!). This display helps you to understand how the application integrates with the OS, the user, and with other applications.

  4. ACE Integration  When it is necessary to edit the dynamic Configuration XML files, you need a good tool. Notepad is easy, because it is always on the OS, but the client parsing of the XML is extremely picky; if you use notepad you will destroy your file in an unrecoverable way because you will not be able to find the error.

    The folks over at VirtualEngine have created a good, and free, editor for these configuration files called ACE that is specifically for editing these files in a nice easy GUI.

    In 3.1 I modified AppV_Manage to detect if ACE is installed, and to add a new “edit configuration file” right click option that calls ACE. You need only to install the latest version of ACE on the client system and it will be detected. You find this on the Publishing tab of AppV_Manage. Select your package, locate the detected configuration file in the details pane, and right click for the menu. The 3.2 release fixes an issue when your package name or path includes a space.

App-V 5 SP2 HotFix 4 is more than a hotfix

Here is my explanation: Someone at Microsoft has been taking lessons on how to name product releases from Citrix. There really can be no other explanation for App-V 5.0 Service Pack 2 Hotfix 4. It is full of features. I mean, why not call it release 5.1 (or 5.3)? Or maybe Service pack 3? Surely they know how to build a full installer at Microsoft, right? You mean I have to install the client, then the hotfix, and then reboot? Really???? Sorry user, we’ll have to reboot you now. Sure, existing users would have to reboot if it were a service pack, but new users require too many reboots between the pre-requisites, the client, and hotfix. I can hear the screams of my pal Nicke from here without the phone, and he’s in Sweden and I’m in Boston.

Because there is so much in this release, you should probably fully test before deploying. But at least there is a lot of good stuff in the hotfix which you request via web (KB 2956585). Since MDOP 2014 just came out yesterday without the changes, perhaps the dev team just missed the window to get it into the MDOP release. Anyway, when you receive the zip file, ignore the x64 in the name and unpack it from any machine, you will find an update exe for each of the clients (RDS and WD), and a full installer for the sequencer.

  1. Virtual Application Deployment Speedups. With all of the good things that the version 5 rewrite brought us with openness, it turned out a lot of that openness lead to slower deployment speeds. A few of the bigger problems were addressed in SP2, but at the time Microsoft indicated that the performance of deploying virtual apps needed more work. This is especially true in semi-persistent scenarios like VDI and RDS, where apps must be deployed and streamed possibly every time the user logs in. The dev team has worked hard on this, making improvements here and there, but especially in Shared Content Store Mode.

    These performance changes do not require server side updates (as there are none) and should not require repackaging.

    In addition to the software changes, they also provide some performance guidance; information about how certain kinds of apps can affect this performance.

    In addition to the release information, over the next few weeks, I will be publishing a series of research papers that I have written quantifying the amount of impact different app features have on deployment and launch under different scenarios, where the impacts are felt, and options for dealing with it. The papers are pretty much complete now and I just need time to re-test against the final hotfix release before publishing.
  2. Ability to write files in the VFS. It was a huge step back from the old release when we lost the ability to write to files located in the VFS area. In developing 5.0, Microsoft under-estimated the creative stupidity of application developers and just assumed that they would only update files in reasonable locations. They quickly learned that we deal with a lot of strange apps, not only a bunch of apps developed in-house at enterprises 10 years ago, but also the latest releases from some major software vendors. This issue has been a blocking item for some customers that wanted to migrate to 5.0. I’m glad to see it fixed.
    Keep in mind that there are still file types that may not up written to! App-V still protects files of certain extensions, so an “autoupdate” of the vendor’s exe is not supported.

    This change is both sequencer and client oriented. It adds a checkbox to the advanced tab on the sequencer. I have not checked, but it might be possible to add the new syntax to a DeploymentConfig file of an older package on the updated client.

  3. Support for executables on network shares The App-V Client now correctly supports shortcuts to executables on a network share. It now performs user impersonation when a virtual application uses resources located on a network share. Previously, the only way to get this to work was to give the client computer account permission to the share.
  4. Package Branching is back! Branching is the ability to open up an existing package, make modifications, and save it as an entirely new package rather than as an update to the existing package. While the update method is preferred when it is a replacement for everyone, branching allows for different versions of the package to be deployed in parallel. This oversight drove a few customers to buy AVE just to have a way to branch again.
  5. Package conversion enhancements. On this one, I’ll admit that while I prefer to start over rather than convert, Microsoft has made a number of fixes and improvements to significantly increase the rate of successful conversions of App-V 4.* packages. The conversion rates are improved again (as they were in SP2 sequencer). The sequencer also now detects and warns against certain hard coded paths that it detects and cannot fix.

Plus the hotfix includes the mysterious Hotfix 3 which resolves the following:

  • Client crashes issue in certain situations with Config Manager (now resolved).
  • PowerShell windows that appear on taskbar momentarily during publishing refresh (now hidden).
  • Registry changes in in the DeploymentConfig file were ignored (now working).

Hotfix 3 was a web-available KB (2956985) for a short while only to disappear and become available after opening a support call with Microsoft, which you wouldn’t know to do because they pulled the entire KB article.

App-V hotfixes traditionally also contain all previous hotfixes, so there is no need to apply hotfix 1 and 2 if you want to apply this hotfix.

The hotfix will require a client reboot. A quick test indicates that existing apps are working prior to reboot, but additional operations like app publishing fails until the reboot is completed.

Image courtesy of pakorn.

My Favorite BriForum moment, 10 years later

BriForum 10th Anniversary This year marks the 10th Anniversary of BriForums, a rather unique conference that I have had the privilege of speaking at many times. In fact, I’ve lost count of how many I have spoken at since I have spoken at every one and for many of those years there has been both one in Europe and one in the US. This year I will be speaking both at the London show next month and the Boston show this summer.

My favorite moment at BriForum? Well it isn’t the fire and subsequent Bomb evacuation at the 2nd Washington show, which while quite memorable, doesn’t compare to the first show.

The very first BriForum was held in a movie theater in Silver City (near Washington DC). Most of the speakers barely knew each other, mostly from on-line presence and randomly running into each other at other shows like the Citrix iForum (what Synergy used to be called). Some of us had met at dinners held by posters at the ThinList, but by and large we didn’t know each other that well.

Which made it such an exciting show to hear sessions led by these awesome people. And all of us were in awe of each other, while hoping that we were good enough to not get heckled off stage when it was our turn.

So there I was, standing on the stage of a movie theater giving my presentation when I got interrupted. You know the crappy music they play along with advertisements to go buy some popcorn and soda before the movie starts? Well that started up in my room about 30 minutes into the session.

Not knowing what to do, I exited out into the lobby to find Gabe making popcorn at the popcorn machine. No, he didn’t turn on the music, but he was the guy who could figure out how to get it off. I returned to the room and it stopped. But that’s not the end of the story.

When the videos came out, I realized that the entire time I was on microphone. Everything thing I said; the conversation with Gabe and all, came out on the recorded audio.

UPDATE: Brian posted a link to the video here. The music hit at just after the 34 minute mark.

Fortunately neither Gabe nor I used any bad language!