Nice answer from Chris Painter, adding for reference: how to register DLL's properly in wix 3.9. And one with WiX-focus: Registering COM EXE with WIX.
Self-Registration considered harmful
The proper way to register a COM file is to extract the COM registry information from the file and add to the appropriate family of COM tables in the MSI. Most MSI tools have features to support this COM extraction, see separate section towards the end of the answer for details.
This MSI SDK article lists several variations on the general issues with self registration described below, as well as describing some further details - particularly with regards to per-user registration of COM data, and run-from-source scenarios.
Extracted COM data will ensure reliable installation of your COM server as well as support for advanced MSI features such as "advertisement", "rollback", resiliency and "elevated privileges". You can read more about these advanced MSI benefits in this summary that has become somewhat popular on serverfault.com: corporate benefits of MSI.
It is also possible to use the built-in SelfReg table in Windows installer to register the file using regsvr32.exe as part of the installation process (or even invoked as a custom action), but this is considered bad practice for a number of reasons:
- Rollback: Windows Installer is unable to properly handle rollback unless the COM data is extracted and embedded in the MSI. The result is that a failed setup may not clean up its COM footprint correctly and the MSI does not put the machine back in the original state properly. The rollback of COM data really does work like "auto-magic" tracking every change in the registry whether it be addition, modification or deletion and is reliable when done right.
- Security: The self registration process of a COM server may in certain cases perform unorthodox tasks such as modifying the system's network settings or perform other crazy maneuvers that have nothing to do with COM and are hard to identify and debug. I have personally seen, in disbelief I might add, COM registration change system-wide network settings without any warning, and for no obvious reason. It might have been just an optimization for an application, but this is rarely acceptable when it changes the whole system affecting all other software. Though an EXE file run in admin mode can do the same and be equally faulty, self-registration may go under the radar and be less obvious as a security issue. This is a core reason why large corporations and Microsoft best practices insist on not allowing self-registration as it may interfere with business critical systems.
- Chained dependencies: Some COM files may need to be registered in a specific order on the system to register successfully. In other words file B can't register until file A has been registered. I have honestly never seen this in real life, but it is technically possible, and I have seen dependencies on language dlls (resource only dlls) cause COM extraction to fail. Though slightly different, it is still a dependency issue. MSI does not allow specification of the registration order (probably due to the database origin of MSI, rows are unordered). If you extract the registry data properly on the build computer and put it into the MSI, these chained dependencies will not cause an application error.
- Permission problems: Windows Installer has advanced features to elevate the privilege level of the user installing the MSI to allow all information to be registered without permission problems (no messing about with temporary admin rights). If you use the SelfReg table you are more likely to run into registration problems caused by permission or privilege peculiarities on the local system (in my experience this is particularly evident for self-repair operations). Permission problems like these occur more and more as new versions of Windows steadily put new obstacles in place for the successful deployment of software (UAC prompts, self-repair lockdown, impersonation changes etc...).
- Resiliency: If another application destroys your COM registry entries, the COM data embedded in your MSI will reinstall the COM component with all associated registry entries via self-repair if proper COM extraction is used to make the package. This means that your application should always be able to launch with its COM servers properly registered. However, this can also trigger the dreaded repetitive sequence of self repair cycles that many experienced computer users have seen (here is a simpler and shorter explanation). In other words COM extraction can be riddled with problems as well, but just using self-registration would leave your application broken, and also prone to security errors being triggered if you run repair, modify or self-repair of your product (the self registration operation may run without elevated rights and hence fail to complete if the repair is run as a restricted user). This means the errors are impossible to fix for most normal users. It is all they know how to do if the product isn't working.
- Advertisement: Advertised products are available to the user via shortcuts and registry entries, but not presently installed on the machine. An "on demand" installation can be invoked in a handful of ways - referred to as advertised entry points (recommended Symantec article), one of which is the invocation of an advertised COM server. No install will be triggered unless the file is properly advertised in the registry and a crucial trigger of "self repair" is hence missing if you use self-registration.
Installation Tool Support for COM Registration
The extraction of COM data and entry into MSI tables is a fairly involved task, and most tools on the market such as Installshield, Advanced Installer, and Wise (Wise is now off-market, unfortunately) have automated solutions for this.
In Installshield you simply enable a component flag called "Extract COM data on build", and Wise has a similar flag on the component level. WiX can extract the COM registry data using a tool called heat.exe and the generated WiX-code can be inserted into your source WiX file (there may be new features for this by now that I am not aware of). I am not aware of any features in Visual Studio that will extract the COM data automatically, but it looks like Chris Painter provides a possibility in his answer.
Check out RegSpy2 if Heat doesn't work for you (Phil Wilson - the author of "The Definitive Guide to Windows Installer" wrote RegSpy and someone extended it to RegSpy2). Also check this: Register ActiveX exe server using WiX (my answer towards the bottom for regspy.exe command line use).
Erroneous COM data inserted into an MSI - particularly for repackaged applications in corporate environments - is one of the leading causes of "unexpected cyclical self-repair". Please see this long article for explanation of this issue: How can I determine what causes repeated Windows Installer self-repair? (bullet point 3 in section "Some typical self-repair problem scenarios" describes this issue).
Several other installation tools exist with similar extraction features: What installation product to use? InstallShield, WiX, Wise, Advanced Installer, etc
