Software-based hardware emulation can include capabilities of platforms such as found on (DIGITAL, Compaq, HP) Alpha (via PALcode), where the software provides the appearance of another platform, or platforms such as IBM z-series, Intel IA-32e and AMD AMD64 (and yes, even considered for VAX), where assists can present to allow easier virtualization of the platform. And yes, there are conceptually ways to add these assists for Alpha. Yes, via PALcode. And Intel adds these to Itanium with its Vanderpool Technology (as the project was reportedly known whilst under development) or Virtualization Technology (VT-x, and VT-i) extensions (as officially known; once a released product).
But we're here to discuss more "traditional" software emulation of hardware, where a system of one architecture runs software which interpretes and executes the instructions of another. Interpreting the instructions is, comparatively, the easy part. Emulating the actions in I/O space is rather more involved, where the same instructions are used to read and write reality. You might think that you needn't worry about that part as you're probably not writing an emulator. But you do need to consider what I/O hardware is emulated, and particularly if you require specific devices or capabilities.
There are various names for the emulator and for the emulation, including a virtual machine, or an emulated machine.
The first part of using an emulator is installing the emulator on the platform. How that happens depends on the particular emulator, but it generally follows the norms for installing a software package onto the particular platform. This is the platform that's running the emulator itself, and not the platform being emulated.
Next, you need to load the operating system into the emulator. With various platforms, this is known as the guest operating system. The process used to load the guest operating system is dependant on exactly how the emulator is constructed, and how you access the emulation from the underlying platform, and what hardware the emulator presents to the emulated platform. Most typically, you have an interface that looks like the console for the emulated platform, and the emulator presents an installation device such as (in the case of a VAX) a TK50 cartridge tape or (more likely) a CD-ROM. Assuming the emulator "passes through" its CD or DVD device from the underlying physical hardware, you can load the distribution disk for the guest operating system into the CD drive, and follow the installation instructions for the operating system. OpenVMS VAX, or ULTRIX VAX, or otherwise. Using the console provided by the emulator.
The guest operating system does not need nor need to know that it is a guest for a properly-constructed emulation. There are some advantages to having the guest "know" this or to cooperate with an emulator or a virtual machine, such as a way for the guest to flag the idle loop within the guest's scheduling logic. Otherwise, the underlying operating system spends resources executing the idle loop.
The key piece you need to remember — from the perspective of the operating system running as the guest within the emulator — is that the software emulator looks and operates like real hardware. For the sake of argument, it is real hardware. This means that — once you gain access to the console within the emulator — the installation and operations are those of the target platform. It's a VAX, if you're using a VAX emulator.
Some of the available emulators that have been encountered by OpenVMS users include SRI CHARON-VAX, CHARON-ALPHA, and Trailing Edge SIMH. Others are available.
Beyond the intended usage, probably the most interesting part of virtualization is its applicability for a rootkit; as something to be exploited for nefarious purposes.
A rootkit is a piece of software — and usually nefarious — that currently typically hides its presence from the user through modifications to the operating system. Virtualization makes finding traces of the rootkit even more difficult; your whole environment could be virtualized. (Shades of The Matrix, yes.)
Detecting and digging a virtualized rootkit out of a platform using Intel VT-x (or of most any other hardware-assisted virtualization) is going to be, um, interesting. One of the central goals of Intel VT-x was and is to avoid presenting evidence of its existence to the guest operating system. And arguably, this is a design error in VT-x, as a fully non-privileged path that includes a virtualization activity flag would require far more effort to block its visibility to the guest operating system. But a goal of virtualization is complete transparency, so this omission is understandable. Though unfortunate.
Since the only way to the real hardware is through the virtualization which is itself controlled by the rootkit, the rootkit can quite effectively defend itself from and deflect attention away, whether from a classic virus scanner or otherwise. From anything operating within the guest.
You can't trust what you read in from the disk, because the disk can be emulated. Go ahead, try and read the (real) bootblock and its master boot record (MBR) structures from the guest operating system.
Existing virtualization-aware rootkits include Bluepill and SubVirt on Windows on Intel, and Vitriol on Maco OS/X on Intel Core.
But rootkits and related security consideration are fodder for another discussion.