{"id":2357,"date":"2015-09-30T13:32:33","date_gmt":"2015-09-30T17:32:33","guid":{"rendered":"https:\/\/www.tmurgent.com\/TmBlog\/?p=2357"},"modified":"2015-09-30T15:28:21","modified_gmt":"2015-09-30T19:28:21","slug":"the-case-of-the-misnamed-object","status":"publish","type":"post","link":"https:\/\/www.tmurgent.com\/TmBlog\/?p=2357","title":{"rendered":"The case of the misnamed object"},"content":{"rendered":"<p><img decoding=\"async\" src=\"https:\/\/www.tmurgent.com\/TmBlog\/wp-content\/uploads\/2015\/Objects\/AppVSpokenHere.png\" alt=\"\" align=\"left\" \/>I suddenly seem to be getting asked\u00a0by a number of software vendors to help them support customers that are using App-V.\u00a0 This is fantastic!\u00a0 Vendors that want to support App-V?\u00a0 We have dreamed of this for 15 years now.\u00a0 I am thinking that four is perhaps\u00a0more than just a blip,\u00a0maybe\u00a0it is\u00a0a trend.\u00a0 I certainly hope so.<\/p>\n<h2>Case Introduction<\/h2>\n<p>This week&#8217;s vendor (I can&#8217;t mention them by name) approached me because of a problem at a customer site.\u00a0 Unlike other ISVs to approach me recently, this one wasn&#8217;t having their software packaged for App-V, but still had a problem.\u00a0 The customer was natively installing their software on the system.\u00a0 But on systems with App-V installed, their software didn&#8217;t work.<\/p>\n<p>The software consisted of a back-end windows service and an Internet Explorer plug-in.\u00a0 If the user was published any App-V packages that included IE plug-ins (or probably Browser Helper Objects or ActiveX components), the app broke.\u00a0 Through trial and error they had found that if they could get the customer to enable COM visibility and disable Object renaming in the sequencer for ALL of those packages, the problem went away.\u00a0 The customer wasn&#8217;t satisfied with that solution and the vendor wanted to know what was going on.<\/p>\n<p>I immediately suspected that the issue had nothing to do with COM visibility and everything to do with Named Objects.\u00a0 I was right.\u00a0 So what are Named Objects and just what is this checkbox <img decoding=\"async\" src=\"https:\/\/www.tmurgent.com\/TmBlog\/wp-content\/uploads\/2015\/Objects\/AllowBox.png\" alt=\"\" \/> in the sequencer for anyway?<\/p>\n<h2>Sidebar on Named Objects<\/h2>\n<p>The kernel of the Microsoft operating system manages a variety of resources for the user mode processes.\u00a0 It does this for efficiency and\/or because multiple processes might share the resource.\u00a0 Often, the resource (not resource type) is accessed by a name, and that it what we call &#8220;Named Objects&#8221;.<\/p>\n<p>The most common form of a named object might be a file handle, where the name is the full path to the file.\u00a0 The process wants to open a file, but it is the Windows System cache that actually opens the file and manages what portions are currently in virtual memory.\u00a0 While file handles are named objects that you are familiar with, it turns out that App-V deals with files via the Virtual File System and Copy-on-Write subsystems, so this isn&#8217;t what we are talking about when we talk about named objects in an App-V world.\u00a0 Microsoft doesn&#8217;t really document what App-V does with Named Objects, so I guess it is on me to explain what I think it does.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.tmurgent.com\/TmBlog\/wp-content\/uploads\/2015\/Objects\/ObjectTypes10.png\" alt=\"\" width=\"400\" align=\"right\" \/>The image on the right is a list of the types of objects supported by the Windows 10 operating system (it is taken from I home-grown tool that I wrote that extracts the information from an unpublished kernel interface).<\/p>\n<p>Long ago, a lot of software applications were written for a single user desktop OS and didn&#8217;t take into account that multiple users might want to run the app on the same OS, and maybe at the same time. So when we built SoftGrid (the original name for App-V), we decided to intercept the calls to create these named objects and change the name (an operation called &#8220;remapping&#8221;). In App-V 5.x, if a virtualized process creates or opens one of these named objects, the software remaps by changing the names uniquely for the virtual environment it is running under.\u00a0 App-V enhances the name by prepending the characters &#8220;sg&#8221; followed by a long unique identifier for the virtual environment (a 16 character hash) and an underscore.<\/p>\n<p>Certain kinds of named objects, such as named pipes,\u00a0tend to be used more with inter-process communication and signaling,\u00a0and when that occurs\u00a0it is important that both processes use the same name.\u00a0As long as the named object is only used by processes inside\u00a0an App-V\u00a0virtual environment, the app works normally when the object is renamed.<\/p>\n<p>But the operating system includes all sorts of named objects created by the kernel or windows services that are part OS, so their end of these objects is never virtualized.\u00a0Over the years App-V has developed a list of these and provides a registry exclusion list that exempts certain names from remapping by the App-V client, which makes those object work.<\/p>\n<p>By the way, a remapping process also occurs for isolated COM objects in a package, although in that case a different dynamic CLSID Guid is created instead (there is also a COM CLSID exclusion list in the registry).\u00a0 Together, along with the VFS file system spoofing, we achieve a completely isolated virtual environment that internally works just like the native installation but can&#8217;t get messed with by outside processes.\u00a0 There is also something referred to as &#8220;side-by-side privatization&#8221; which fits in this basic category (but not for discussion today).<\/p>\n<h2>Why we use App-V<\/h2>\n<p>While we don&#8217;t see many applications that cause &#8220;<em>dll hell&#8221;<\/em> being deployed these days (most that are still in use are over 10 years old and were likely developed in-house and will need to be retired rather than upgraded), we still see a lot of applications not fully implemented for\u00a0simultaneous use on the same system.\u00a0 Most apps today, understand the concept of different users and properly implement HKLM\/HKCU and use the user&#8217;s profile fairly correctly.\u00a0 But many still fail if you put them in RDS because they don&#8217;t make use of session space.\u00a0 Still many more fail, typically with an occasional unexplained hang or crash,\u00a0if you run multiple copies within the same user session (on RDS or desktops).\u00a0 And often we want to allow the user to run two different versions of the same application, which pretty much never works without application virtualization.\u00a0 This is why we use App-V;<strong> to make the software more predictable by limiting the dependencies on other stuff sitting around and\/or running at the same time<\/strong>.<\/p>\n<p>But sometimes we want integration outside of the bubble.\u00a0 The rewrite in App-V 5 was precisely to create more opportunities for integration.\u00a0 Giving us more integration points (&#8220;extensions&#8221; in App-V terminology) with other parts of the OS, other applications, and the user.\u00a0 With that integration, we also have some granular controls over that integration.\u00a0 90% of the time we don&#8217;t need to think about those controls, but sometimes we do.<\/p>\n<h2>Back to the Problem<\/h2>\n<p>To protect the innocent, I&#8217;ll use a different software application from someone other than the software vendor that I was working with.\u00a0 I will use the excellent web sniffing tool from Telerik called Fiddler.<\/p>\n<p>Fiddler, like the ISV software I was dealing with, has both an external process and a plug-in to Internet explorer.\u00a0 With Fiddler, the external process is a standard user process that can be started from the start menu rather than a Windows service, but that doesn&#8217;t matter for our purposes.<\/p>\n<p>Fiddler, like the\u00a0ISV software, is also completely virtualizable with App-V.\u00a0 Both work like a champ if you create and deploy an App-V package containing the software.\u00a0 Both also work great is you create a Connection Group with the package and with the other Internet Explorer add-ons.<\/p>\n<p>The problem with the ISV app is seen if you <span style=\"text-decoration: underline;\">natively<\/span> deploy their app and use App-V for other Internet Explorer plugins.\u00a0 Both for Fiddler and the ISV,\u00a0the external process can end up running outside of the virtual environment for Internet Explorer.<\/p>\n<ul>\n<li>With the ISV software, the service auto-starts and is pre-existing and is obviously outside of the virtual environment.<\/li>\n<li>With Fiddler, we can start the fiddler process by using a toolbar button within iexplore, in which case fiddler will be started in the virtual environment and work just fine.<\/li>\n<li>But\u00a0if you\u00a0start the Fiddler process from the start menu, and\u00a0start iexplore.exe\u00a0in a bubble, you have a mixed situation and a potential for a problem similar to the issue experienced by the ISV app.\u00a0As it turns out, Fiddler does not do this but I will still use it to demonstrate the technique to solve the problem the ISV had.\u00a0(Fiddler uses an\u00a0IE exposed COM object. Remember that IE is native so the in-process COM\u00a0dll is available for Fiddler to access).\u00a0So let&#8217;s Fiddler\u00a0did use a shared named object and see how we resolve the issue.<\/li>\n<\/ul>\n<p>For the vast majority of software applications, one part being sucked into an App-V bubble is no problem.\u00a0 When those two processes communicate with each other using something like DDE or OLE, it works great.\u00a0 When they need to use COM, sometimes we want to integrate the COM (publishing the objects outside of the bubble and disabling the GUID remapping.\u00a0 If they communicate using a named object, there can be problems.<\/p>\n<p>We can see this behavior by running the virtual application in the runtime analyzer of AppV_Manage (a <a href=\"https:\/\/www.tmurgent.com\/appv\/index.php\/en\/resources\/AppV_Manage\/221-appv-manage-introduction\">free tool available from our website<\/a>). The image below is of a virtualized\u00a0Internet Explorer Plug-in\u00a0package showing the various processes that run.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.tmurgent.com\/TmBlog\/wp-content\/uploads\/2015\/Objects\/Timeline55.png\" alt=\"\" width=\"650\" \/><\/p>\n<p>You will notice that there are multiple iexplore.exe processes, and a rundll32.exe process colored blue. The blue color lines on the <em>Timeline<\/em> indicate that the\u00a0process is\u00a0part of the virtual environment (#55). There are two iexplore.exe processes because multi-tabbed browsers usually use one process for the overall frame and then individual processes for each tab (we only had one tab open).\u00a0 The Mavinject32 processes you see were short lived (under\u00a025 ms each) and are part of the App-V client that injects some client dlls into virtualized processes. The Fiddler process is colored in green in the image, indicating that it is not virtualized.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.tmurgent.com\/TmBlog\/wp-content\/uploads\/2015\/Objects\/Processes55.png\" alt=\"\" width=\"650\" \/><\/p>\n<p>We can see the full command line of each of the processes in the\u00a0<em>Processes<\/em> section of the display (shown above).\u00a0 Below that on the display (not shown here)is the <em>Images Loaded<\/em> section.\u00a0If we open up the process entry in the <em>Images Loaded<\/em> section of the tool we can see the various dlls loaded by the virtual process (useful if you want to be sure which copy of a dll is in use, or if any <a href=\"https:\/\/www.tmurgent.com\/TmBlog\/?p=2350\">Native Images<\/a> are in use).<\/p>\n<p>COM remapping can be seen in the <em>v.COM<\/em> section of the tool display:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.tmurgent.com\/TmBlog\/wp-content\/uploads\/2015\/Objects\/COMRemap.png\" alt=\"\" width=\"650\" \/><\/p>\n<p>And Object remapping can be seen in the <em>v.Objects<\/em> section:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.tmurgent.com\/TmBlog\/wp-content\/uploads\/2015\/Objects\/ObjRenaming2.png\" alt=\"\" width=\"650\" \/><\/p>\n<p>(Note that the last line above indicates &#8220;not remapped&#8221; because it was already remapped.\u00a0 The original request created the Mutex and returned a handle to the process, and the process would have been unaware of the renaming.\u00a0 The process later used that Mutex handle, which was already tied to the remapped name).<\/p>\n<p>Using the Manage Add-ons feature of IE itself, we can verify that both plug-ins are loaded (the &#8220;<i>App-V Usage<\/i>&#8221; that we intended to virtualize and &#8220;<i>Fiddler<\/i>&#8221; that came along for the ride):<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.tmurgent.com\/TmBlog\/wp-content\/uploads\/2015\/Objects\/ie3.png\" alt=\"\" width=\"650\" \/><\/p>\n<h2>What Happened to the ISV<\/h2>\n<p>What happened to the ISV was that the external windows service created a named pipe, and if they had called the\u00a0pipe &#8220;Fred&#8221; (a fictitious name), it would appear as a named object of type &#8220;Device&#8221; and named\u00a0&#8220;\\Device\\NamedPipe\\Fred&#8221;.<\/p>\n<p>In the analyzer trace <em>v.Objects<\/em> secton, we could see\u00a0that the\u00a0internet explorer process\u00a0was attempting to open as Fred, but it was being renamed.<\/p>\n<p>Because the service was not virtualized, we could not see any entries for object names\u00a0in the AppV_Manage Runtime Analyzer.\u00a0 But we can using the Microsoft SysInternals tool ProcessExplorer.\u00a0 You need to start ProcessExplorer using the RunAs Administrator option, then use the View menu to enable the lower pane view to Handles.\u00a0 Click on the process in question.\u00a0 The image below shows a number of named pipes used by an App-V client process:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.tmurgent.com\/TmBlog\/wp-content\/uploads\/2015\/Objects\/ProcExp.png\" alt=\"\" width=\"650\" \/><\/p>\n<p>Armed with this information, we could clearly see the name mismatch.\u00a0 In this ISV&#8217;s case we could also have seen the mismatch just using ProcessExplorer alone, because the plugin kept the\u00a0renamed named pipe handle open.\u00a0 Usually, an app would have immediately closed the handle and thrown an error, making it nearly impossible to see the renamed object in ProcessExplorer (it only shows currently running processes and currently open handles).\u00a0 So it&#8217;s nice that the Runtime Analyzer provides a history of the actions.<\/p>\n<h2>Possible solutions for the ISV:<\/h2>\n<p>There are four main solutions for this vendor to support their customers that use App-V:<\/p>\n<ol>\n<li>Recommend that the customers virtualize their software in a package and use a Connection Group with any other package that uses IE.\u00a0 App-V will virtualize the service along with IE and the problem is solved.<\/li>\n<li>Request that the customers disable object renaming in all IE plugin packages.\u00a0 Nobody likes the idea of opening back up those packages, so this option is pretty much dead-on-arrival.\u00a0 And I am not a fan of just disabling all of the object renaming without understanding the impact.\u00a0 Issues created this way will be the hardest to diagnose (which is why you used App-V to begin with).<\/li>\n<li>Request that the customers add a new entry in the system registry that adds an additional exclusion to the object exclusion list (HKLM\\Software\\Microsoft\\AppV\\Subsystems\\ObjExclusions).\u00a0 As we can see in the standard list in App-V 5.1, it turns out that the value name for the entry doesn&#8217;t really need to be the next number in the list after all, just unique, so I am recommending the ISV suggest using their already unique named object for the value name and value. This change can be pushed out via any method, including group policy preferences, or as an add-package script in one or more\u00a0of those IE addon packages.<img decoding=\"async\" src=\"https:\/\/www.tmurgent.com\/TmBlog\/wp-content\/uploads\/2015\/Objects\/ObjExclusions.png\" alt=\"\" \/><\/li>\n<li>Request that Microsoft add the exclusion to the standard exclusion list in the next client release.\u00a0 We don&#8217;t yet know what the process is to make this request, but I&#8217;m working on it.<\/li>\n<\/ol>\n<h2>For Your Action<\/h2>\n<p>So the next time you find that you can fix a problem by checking the <img decoding=\"async\" src=\"https:\/\/www.tmurgent.com\/TmBlog\/wp-content\/uploads\/2015\/Objects\/AllowBox.png\" alt=\"\" \/> checkbox, consider finding the actual problem named object and excluding it.<\/p>\n<p>If you are putting the software into a package, starting with App-V 5.1 you can also consider <a href=\"https:\/\/www.tmurgent.com\/appv\/index.php\/en\/?catid=0&amp;id=349\">editing the internal AppXManifest<\/a> file and adding the exclusion to only that package.\u00a0 The checkbox simply adds a wildcard exclusion of &#8220;*&#8221; to this file.\u00a0 You could check the checkbox, export and edit the file to change the &#8220;*&#8221; to the name you want excluded. The client uses regex against these names, so standard wildcards may be used. Save the file and import it back in.\u00a0 Anything in the list is considered an addition to the client registry stored exclusion list.<\/p>\n<p>PS: It took me five times as long to write this up than to diagnose the ISV&#8217;s problem!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I suddenly seem to be getting asked\u00a0by a number of software vendors to help them support customers that are using App-V.\u00a0 This is fantastic!\u00a0 Vendors that want to support App-V?\u00a0 We have dreamed of this for 15 years now.\u00a0 I am thinking that four is perhaps\u00a0more than just a blip,\u00a0maybe\u00a0it is\u00a0a trend.\u00a0 I certainly hope&hellip; <a class=\"more-link\" href=\"https:\/\/www.tmurgent.com\/TmBlog\/?p=2357\">Continue reading <span class=\"screen-reader-text\">The case of the misnamed object<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[47,48,50,1],"tags":[4,13],"class_list":["post-2357","post","type-post","status-publish","format-standard","hentry","category-appv5","category-sequencing","category-tools","category-uncategorized","tag-app-v","tag-sequencing","entry"],"_links":{"self":[{"href":"https:\/\/www.tmurgent.com\/TmBlog\/index.php?rest_route=\/wp\/v2\/posts\/2357","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.tmurgent.com\/TmBlog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tmurgent.com\/TmBlog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tmurgent.com\/TmBlog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tmurgent.com\/TmBlog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2357"}],"version-history":[{"count":19,"href":"https:\/\/www.tmurgent.com\/TmBlog\/index.php?rest_route=\/wp\/v2\/posts\/2357\/revisions"}],"predecessor-version":[{"id":2376,"href":"https:\/\/www.tmurgent.com\/TmBlog\/index.php?rest_route=\/wp\/v2\/posts\/2357\/revisions\/2376"}],"wp:attachment":[{"href":"https:\/\/www.tmurgent.com\/TmBlog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2357"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tmurgent.com\/TmBlog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2357"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tmurgent.com\/TmBlog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2357"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}