Virtual-reality technology is giving surgeons the equivalent

            of x-ray vision, helping them to remove tumors more effectively,

         to minimize surgical wounds and to avoid damaging critical tissues


by: W. Eric l. Grimson, Ron Kikinis,

Ferenc A. Jolesz, & Peter Mcl Black

T he scene is an operating room. .A young woman is about to undergo surgery to remove a brain tumor that is causing almost daily seizures. Elimination of this mass, which has become life-threatening, should be curative, but the operation is very perilous. The tumor is pressing against the motor cortex, a strip of tissue that controls voluntary movements. The tumor and cortex look alike to the unaided eye If some of the tumor is left behind it will return. Yet if part of the motor cortex is mistakenly excised, the woman could quite probably be paralyzed for life.

The Golden Rule - Do Unto Others


BRAIN TUMOR (shown in green on the head shot above) in a woman about to undergo neurosurgery was made visible (from his vantage point) by a display that merged live video footage with a three-dimensional computer-generated model of the patient’s brain. When surgery is carried out in a special. Magnetic imaging (MRI) device (shown above in second photo), such models, which are constructed before surgery, can then be augmented by new scans that reveal whether tissues have sifted from .their original positions. The 3-D models and real timer scans therefore help physicians to pursue the safest routes to their surgical targets and, in the case of tumors, to remove growths more completely.

The neurosurgeon has agreed to perform the operation only because he has access to modern, extraordinary tools designed to greatly improve his chances of success. In a corner of the room, he is using one of those innovations now. He is looking at a monitor displaying a three-dimensional computer-generated replica of the patient’s head—a model constructed earlier from pictures produced by noninvasive magnetic resonance imaging (MRI).

He rotates the model to obtain a view similar to the one he will see when the operation is under way. Then, with a few clicks of a computer mouse, he strips away skin, fat and hone to expose the brain. There the tumor and other significant features (such as blood vessels and the motor cortex) are highlighted in colors. Noting that the tumor is not only touching the motor cortex but is also close to key blood vessels, he makes a plan for reaching and removing the tumor in a way that will both eliminate all traces of the growth and minimize the risk of bleeding and paralysis.

His strategy set, he turns to the patient. She lies not on a standard operating table but on a platform incorporated into an advanced MRI system that will provide images of the woman’s brain during surgery. A typical MRI machine consists of one large hollow cylinder, into which a patient is fully inserted for scanning. In this newer device, the cylinder—a magnet, really—has been cut in two, and the resulting doughnut-shaped sections have been moved apart. The operating table spans the “doughnut holes.” This arrangement leaves room for the surgeon to stand in the gap between the magnets and to reach the patient.

The surgeon steps into the gap and pulls down a screen displaying the same model he viewed earlier, but now it is fused with a live video image of the patient; the composite picture is in perfect register with the patient’s head as seen from the surgeon’s vantage. It is as if he has now developed “x-ray vision,” for he can locate internal structures before he ever picks up a scalpel. He takes a pen and, guided by the enhanced live video on the screen, marks on the patient’s partly shaved scalp the positions of the tumor and other selected structures. He also sketches the shape of the small window he will form in the skull to gain access to the tumor. Then he looks directly at the patient and begins cutting. As he works, he frequently checks his exact position and trajectory by gently inserting a traceable, sterile probe into the depth of the cut. A quick look at the monitor then tells him the pointer’s position relative to the otherwise invisible structures below the small area of exposed surface. The pointer, therefore, helps him to determine whether he is on course or not and likely to maintain a safe distance from the motor cortex and blood vessels he wants to avoid.

He also asks periodically for fresh MRI scans at the site of the probe. A few seconds later those images appear, superimposed on the pre-surgical model. They enable him to compare the pre-operative and current positions of the tumor and other tissues of interest, so as to discover any displacements or deformations that must be taken into account. The images, moreover, help him to locate tumor remnants that might otherwise have been left behind.

The Golden Rule - Do Unto Others


SERIAL MRI SCANS are the raw material from which three-dimensional models of a patient’s internal anatomy are constructed. To create the models, a computer combines the individual slices onto a cohesive whole.

Guided by such information, the surgeon clears out the full tumor without disrupting the motor cortex or blood vessels. Days later the patient walks out of the hospital, tired from surgery but ready to begin her life anew.



A lthough this story may sound like science fiction, here in June of 1999, it is science fact for real. Patients can be treated in just this way today. Indeed, our surgical team at Brigham and Women’s Hospital in Boston has already performed such sophisticated image-guided neurosurgery in the “double-doughnut,” or open-magnet MRI machine, on roughly 300 individuals over the past three years.

The system in service at Brigham and Women’s grew out of extensive collabo- ration. The Image-Guided Therapy Program at the hospital and General Electric Medical Systems developed the open magnet, and both groups cooperated with the Artificial Intelligence Laboratory at the Massachusetts Institute of Technology to devise the advanced image-guidance system. Michael Leventon and David Gering of MIT integrated the modeling and imaging techniques.

Advanced image-guidance systems developed by others are also in use at several centers in the U.S. and Europe and are aiding surgery for a range of problems. Together the various approaches are effecting a revolution in surgery. By enhancing the surgeon’s view, image-guided surgery is enabling doctors to treat many patients more effectively. When tumors are the focus, the imagery facilitates identification of tumor boundaries and of the safest, least damaging paths to the growths (processes known as localization and targeting). Most im- portant, it improves the physician’s ability to remove curable tumors completely and to excise more of cancers that are too diffuse or invasive to be eliminated fully (so as to ease symptoms longer or better). The technology helps the surgeon to spare functionally critical tissue during other kinds of operations. Moreover, it can make surgical procedures shorter (which minimizes anesthesia and loss of blood) and, in some cases, permits operations that would have been considered too risky just a few short years ago.

Image-guided surgery can simplify difficult procedures, but the technology behind it is complex—and fascinating.(to say the least) To demonstrate just how the images are prepared and presented, we will now step behind the scenes to reveal the central techniques involved. The methods applied in neurosurgery at Brigham and Women’s will provide our main example. The first essential step is constructing a 3-D representation of the surface and internal anatomy of the body part, or volume, being treated. This model, which will enable the surgeon to see in- ternal structures that would otherwise he hidden, must be made of course without dissecting the patient.

                            Building an Anatomical Model

The Golden Rule - Do Unto Others

N on-invasive imaging holds the key to such modeling.

The Golden Rule - Do Unto Others

Most readers have seen x-ray images of their own body. When the rays pass through bones and organs, they are absorbed to some extent. As a result, some areas show up darker than others on the detecting film. Unfortunately, the resulting picture is a flat, two-dimensional projection of a 3-D structure and provides little information about tissue other than bone.

Computed tomography (CT) and MRI, in contrast, can produce a stack of virtual slices, as if the body part of interest has been cut into hundreds of thin sections, each of which has been imaged individually in series. Both techniques also store the scans in a computer and can combine the slices into a three-dimensional model, in which every point is defined by its horizontal and vertical coordinates on a slice and by the number of the slice. Of the two, we favor magnetic resonance imaging, mainly because it demonstrates anatomy better and is more sensitive to diseased tissue. It also spares patients from exposure to ionizing radiation, relying instead on measurements of the body’s responses to magnetic fields.

As the patient lies in the bore of the cylindrical magnet, the MRI machine produces a constant magnetic field. In essence, this field causes certain protons (positively charged subatomic particles) to spin like tops. If a second field is applied briefly (as a pulse), the tops will tilt in a new position as they spin. When the pulse is gone, the tops will pop back to their original orientation, giving up a detectable amount of energy as they go.

Different tissues emit different amounts of energy in response to the pulse. More energy is recorded as more brightness, or intensity, in the MRI scan. After the two-dimensional slices are combined the full-dimensional product must be “segmented”—each small voxel (or volume element, the three-dimensional equivalent of a pixel) must be labeled by tissue type and combined with like voxels into identifiable structures. In neurosurgical cases, normal tissue might be labeled as fat, bone, blood vessel, skin, ventricle (fluid-filled cavity), cerebrospinal fluid, gray matter or white matter. In theory, a computer can be programmed to assign the labels by rote—according to each voxel’s brightness. Indeed, certain structures, such as the skull or the ventricles, are often obvious, both to the computer and to naive observers, and can have their labels locked in immediately by the computer. Tissue boundaries that are not terribly distinct may, however, be difficult to distinguish on the basis of simple readings. The computer may have trouble, say, separating gray matter in the cerebra! cortex from the underlying white matter or resolving the edges of a tumor and the normal tissue around it. To cope with that challenge, our group and others have now invented new algorithms for interpreting ambiguous signals.

One begins with a manual step. For each kind of tissue, a technician selects a few voxels that clearly belong to that tissue and records their intensities. In this way, every tissue type is assigned a range of intensities , Then the computer examines the brightness of all other voxels and group them with the ranges they approach most closely. To ensure that the assignments are correct, another algorithm is applied. This one, developed by William M (Sandy) Wells of our team, attempts to correct for variations in the pulsed magnetic fields emitted by the magnet. If the pulses were fully predictable and uniform over the entire imaging area, the readings for specific tissues would always be predictable as well. Unfortunately, that aspect of MRI technology is not perfect. Hence, one part of the scanned area might receive a different amount of energy than another part. In consequence, some voxels might be misleadingly bright or dark and might be classified incorrectly.

Wells’s program starts by generating a list indicating the intensities that would represent each tissue type of the MRI pulse were uniform everywhere Then it compares the intensity of every voxel to this list and, wherever possible, assigns a tissue label. If the intensity falls outside the range predicted for any tissues but close to one particular range, the voxel is tentatively assigned to the corresponding tissue type. Next, the program estimates the error in the magnetic field by calculating the difference between the actual and the predicted intensities. it then adjusts the in- tensities and begins the assignment and correction processes again. These steps are repeated until each voxel is assigned a single, definitive label.

At times, tissues cannot be distinguished by intensity alone. For instance white matter in the brain and muscles in the neck might have a similar molecular composition and thus yield the same range of values in MRI scans. In that case, confusion can often be resolved by an automatic program created by Tina Kapur and Simon Warfield of our group. This software predicts the general positions of different structures based on a computerized atlas of anatomy. It could note, for example that although the intensities of a span of tissue in the brain match those for both white matter and muscle the brain does not contain muscle in the region of interest; therefore, the tissue must be white matter. Finally, technicians often review segmented scans on a monitor to he sure that the final tissue assignments make anatomical sense. Once each voxel has been labeled with its tissue type, other programs delineate individual tissues with distinct markings, usually colors. Starting with a single point, the computer will paint with one color all bordering voxels having the same tissue assignment, then repeat the process until all connected muscles are affected by stimulation to specific spots in the cortex. We can also record functional information in the virtual model of the patient’s head, by keying any markings to the muscles that are affected.

The Golden Rule - Do Unto Others


3-D RECONSTRUCTIONS of one man’s brain (above) and of the base of another patients spinal column (smaller photos) have been segmented; and of the base of another patients spinal column (also small photo) have been segmented: selected structures have been high-lighted in different colors. As is shown, such models can be rotated. In addition, components can be faded back or removed. Among the features delineated in the brain are blood vessels (red), ventricles (blue), and a cyst in the tumor (yellowish). The views of the spinal column highlight bone (white) muscle (pink), the rectum (brown), spaces between vertebrae (yellow), the sac around neural tissue (blue), a tumor (green) and areas suspected of belonging to the tumor (dark green).


                               SUPPLYING EXTRA DETAIL

T HE SEGMENTED MODEL IS VERY USEFUL for displaying features that would go unseen by the surgeon’s unaided eye, such as the location and shape of a tumor.

 Standard MRI and segmentation technology cannot, however, provide certain other anatomical and physiological information that might be needed. A second bag of algorithmic tricks incorporates that extra data. In one frequent problem, the MRI scans that form the basis for building the virtual model of a patient’s head do not delineate blood vessels crisply. To achieve greater clarity, the surgical team might produce images known as MR angiograms. These angiograms are made by scanning the patient once again, but this time adjusting the MRI fields to highlight flowing blood.

The MR angiograms that result must then be aligned with the original set of MRI scans and merged into the model. Our automated registration process accomplishes- this merger by overlaying the two data sets and then searching for the best way to translate and rotate one set relative to the other. It finds the optima1 arrangement by exploiting a mathematical concept known as mutual information. Essentially, the computer aligns regions according to the amount of information they display: areas with much information (such as those exhibiting a lot of texture) are matched with a similarly information-rich regions and areas with little information (such as those of uniform intensity) are matched with information-poor regions.

As we noted earlier, the surgeon also needs to monitor the locations of regions that have critical functions—often the motor cortex. Imaging alone is insufficient, because the tissue properties of the motor cortex are indistinguishable from those in other kinds of cortex, which means that the signal intensities on MRI scans are alike. Likewise the motor cortex looks no different from cortex serving other functions, so it can not be distinguished by direct viewing.

We have two noninvasive ways to address this problem. When the body uses muscle, blood flow increases in the cortical region controlling that movement. “Functional” magnetic resonance imaging can detect those increases and thereby pinpoint the cortical areas responsible for each muscle. In addition, a device called a transcranial magnetic stimulator can be used. A pair of electromagnets induces small electric currents in focused areas of brain cortex. This stimulation is painless and harmless. By attaching electrical pickups to the patient’s skin, we can identify which muscles are affected by stimulation to specific spots in the cortex. We can also record functional information in the virtual model of our patient’s head, keying any markings to the muscles that are affected.



S EE THROUGH MODELS are a major asset or planning surgical procedures. But they are most useful when aligned with the patient on the table. That way, the internal anatomy can be “seen” from the surgeon’s point of view during the operation. This alignment spares the physician from having to transform the models mentally, perhaps incorrectly.

The classical tool for registering pre-surgical imagery with the actual patient is the stereotactic frame—a box-like structure that is screwed into the patient’s skull. If the frame is worn by the patient during preoperative testing as well as surgery, landmarks on the frame can enable the physician to correlate preoperative images with the brain itself during the operation. The frames, however, are painful and cumbersome for the patient and a hindrance to the surgeon. We therefore sought a gentler more elegant alternative, which Steven White of our group provided.

White’s system involves shining laser light on the scalp and face of the patient, whose head is clamped in a fixed position throughout the registration procedure and the surgery that follows. Light from the laser generator passes through a lens, causing the beam to spread into a line. If the line of light were hitting the surface of the operating table instead of the patient, it would remain flat and straight. When it falls across the patient’s face, it deforms in a way that matches the contours of the face, much as would occur if a piece of string were dropped across the face at that point. A camera captures the line, and a linked computer (the same one storing the segmented model of the patient’s head) records the deformations from the flat line. Then the line is moved farther up on the face in set increments, with each line cap-, tured and recorded. In the end, the series of lines describes the surface topology of e the patient’s head in its exact position. With this laser “cast” of the face stored in the computer, we invoke another algorithm devised at M.I.T. to rotate the virtual head until the face exactly matches the contours of the laser lines. But we still have another problem. The image displayed on the screen during the operation has to show the patient from the doctor’s vantage, not from that of the camera used to photograph the lasers. A second registration maneuver (involving calculating the doctor’s position relative to the position of the laser system) makes this adjustment fairly easily. Having completed the alignment step and ensured that all internal structures have moved in synchrony with the surface of the face, we can finally r insert the virtual head into a live video of the patient, yielding our “x-ray vision” visualization.

The Golden Rule - Do Unto Others


“MASK” OF LASER LIGHT (red lines) traces the contours of a patient’s face as she lies on the operating table. To precisely register a computer model of a individual’s brain with the true brain, a computer rotates and translates the model until the contours of the virtual face and of the laser mask snap into perfect alignment.



T HOUGH EXTRAORDINARY, this presentation is still rather passive. Beyond localization and targeting, a central goal of image-guided surgery is enabling surgeons to monitor the coordinates of the scalpel at all times. We accomplish this aim by providing a probe topped with infrared light-emitting diodes. As the surgeon touches the sterile tip of the probe against a bit of tissue, three cameras, set at known distances from one another, track the light, which emerges from the diodes at a fixed distance from the tissue-contacting end of the probe. By standard triangulation procedures, much like those used in surveying, computers linked to the cameras can calculate the exact position of the probe in the body and display it on the anatomical model. In our case, the monitor augments this three-dimensional display with images indicating the probe’s position in three separate cross-sectional views.

Of course, the models we build reflect the anatomy of the patient before the operation begins. Once the surgeon moves or removes tissue, the original representations promptly become old news. We depend on the open magnet to help us meet this final challenge: providing up-to-date images as surgery progresses. Because surgery occurs with in the bore of the magnet, new scans of a patient can be taken at any time and registered to previously acquired imagery and to the surgeon’s point of view. Updating is particularly important when the anatomy is flexible and can alter position from moment to moment.

The Golden Rule - Do Unto Others

DURING SURGERY a trackable probe can reveal the trajectory and exact position of the scalpel. Often the computer will display the three-dimensional model as well as two-dimensional cross section showing views from the side, top and front of the brain. The yellow arrowhead in the 3-D model and the crosshairs in the other images denote the tip of the probe. The tumor is green; the blood vessels, red.

The interactive and updating capabilities of our system have proved highly beneficial. They allow for frequent checks of the surgeon’s progress. And they very easily answer such questions as, “Where are the tumor margins?” “How close am I to the motor cortex?” “In which direction should I move now?” “Have I really extracted as much of the tumor as possible, or is there more to remove?”

For now, June 1999, the new images we produce during surgery are two- dimensional and are shown either next to the preoperative model or superimposed on it, depending on the surgeon’s preference. The ultimate goal, however, is to segment, colorize and combine the new images and to do so rapidly, so that the three-dimensional model, with all its helpful, easy-to-see information, can be updated directly throughout surgery. Such prompt updating is entirely feasible. In fact, we are currently perfecting our methods and expect to add that capability to our system within a year or so. (Starting the new century with this will be just fine.)


Already, however, the technologies we have described—and procedures much like ours—are increasingly being used for neurosurgery and beyond, including to support complex operations on bones, the nasal sinuses, the kidneys, the liver, the spine and other tissues. But their potential applications are not limited to surgery.

For cancer care in particular, the new techniques are finding many roles. In radiation therapy, model-building methods and real-time imaging are being applied to ensure that beams of radiation converge at the desired anatomical site, delivering high doses in tumors but low doses in surrounding tissues. To assess better the effectiveness of radiation and drug therapies, physicians review three-dimensional displays that highlight such features as the size and positioning of tumors before and after treatment.

The imaging capabilities might even have value for detecting nascent tumors. Our group is evaluating the ability of three-dimensional representations built from MRI scans to discern cancerous changes in a breast before they can be identified clearly in mammograms.

With respect to future surgical applications of computer-assisted visualizations, technical challenges remain. Most notably, investigators still have difficulty making useful representations of highly flexible tissues. Models made of abdominal organs, in particular, can quickly become inaccurate when the patient breathes or contracts certain muscles. Algorithms able to predict tissue deformations are being developed to help resolve that problem.

Despite its current limitations, image-guided surgery is having a powerful influence on medicine today. With the computer as a valuable assistant to the physician, surgeries of the future are likely to be less invasive, shorter, less risky and more successful.

LYNDA TOLVE, now 32, had a seizure-inducing brain tumor removed in 1996, thanks to a very preliminary version of image-guided surgery introduced at Bringham and Women’s Hospital of Boston. All physicians elsewhere had already turned her away, on the grounds that they could not excise the tumor fully without great risk of cutting into the adjacent motor cortex and paralyzing her.. The next year Tolve married her fiancé, Danile McCaferty, shown also above. She remains seizure-free.


The Authors

W. Eric L. Grimson, Ron Kikinis, Ferenc A. Jolesz and Pedter McL. Black bring a range of expertise to image-guided surgery. Grimson is Bernard Gordon Professor of Medical Engineering, professor of computer science and engineering, and associate director of the Artificial Intelligence Laboratory at the Massachusetts Institute of Technology. Kikinis is director of the Surgical Planning Laboratory at Brigham and Women’s Hospital and associate professor of radiology at Harvard Medical School . Jolesz is B.Leonard Holman Professor of Radiology at Harvard Medical School and director of the Magnetic Resonance Imaging Division and of the Image-Guided Therapy Program at Brigham and Women’s. Black is Franc D. Ingraham Professor of Neurosurgery at Harvard Medical School and neurosurgeon in chief at Brigham and Women’s and Children’s hospitals.

bar_blbk.jpg - 5566 Bytes

Return to the words of wisdom, the health index..

Return to the words of wisdom index..

Return to the main menu..

D.U.O Project
Church of the Science of God
La Jolla, California 92038-3131

Church of the Science of GOD, 1993
Web Designed by WebDiva