Operational versus theoretical definition

As noted above, in order to decide whether or not specific Cochrane reviews fell within the scope of the CAM Field, we needed to develop an operational definition of CAM. In contrast to a theoretical definition, which characterizes the fundamental nature of a construct, an operational definition tests whether a specific instance is or is not a member of the construct through a series of criteria or tests. A theoretical construct may have many different possible operational definitions. For example, in a randomized trial of depression medication, the theoretical construct of the disease being treated is ‘depression’, but the operational definition may be, for example, score of 20 or more on the Beck Depression Inventory or answering ‘yes’ to the question ‘are you depressed’. Without specific operational criteria the theoretical definition is of limited practical use.

The definition of CAM has been much debated in recent years as CAM has increasingly become a focus of public and academic attention. While there have been many theoretical definitions of CAM^6,7 there are no comprehensive operational definitions of what should be considered CAM.

The need for an operational definition of CAM to construct a set of CAM-related Cochrane reviews

Because there is no accepted operational definition of CAM, we found that each time we tried to classify Cochrane reviews as CAM or not CAM we would end up with overlapping but slightly different sets of reviews. Because of this difficulty, we decided that we had to develop and implement an operational definition in order to build the Cochrane CAM Field topics list. This is not to say that the operational definition we developed is the ‘right’ one, only that we needed to create explicit and transparent criteria in order to clarify the scope of the CAM field and build our list of CAM-related Cochrane reviews. Otherwise we would have no objective, reproducible and systematic criteria for including or excluding individual Cochrane reviews from our scope and we would continue to have the problem of including different sets of Cochrane reviews as CAM each time we populated our list.

The need for an operational definition of CAM to facilitate and harmonize research

Standardization of definitions of CAM also means that different groups working independently on database development can collaborate more effectively and build on each others’ work. For example, not only the Cochrane CAM Field, but also the UK National Health Service is developing a database of CAM studies; if we use the same definitions, then we can collaborate more easily, avoid duplication of effort, and promote harmonization of processes.

Standardization is also useful in promoting and comparing research in different CAM areas. For example, defining CAM operational criteria allows survey results on CAM prevalence to be comparable across time and across investigators. Depending on how CAM is defined, prevalence surveys on CAM use vary from 10% to 100%.⁸ It seems like an enormous waste of research resources to not allow for the results of these different studies to be put side by side. If operational criteria are consistent, then studies can be directly compared. If operational criteria are transparent, then even if there is inconsistency between different sets of criteria there is some basis for comparison.⁹

As another example, standardized operational criteria allow examination of evidence for benefits or adverse effects of CAM. That is, often people claim that there is no evidence for CAM therapies. However, this statement is not interpretable if there is no clear agreement on what is CAM. What if one were to wish to compare the evidence base for CAM versus the evidence base for conventional medicine? If there is no clear definition of what CAM is, how could we do this? Indeed, it is misleading to even use the word CAM, if different people would define it in different ways.

Using a theoretical definition of CAM for clues to develop an operational definition

To begin to develop an operational definition, we started by looking at the theoretical definitions of CAM. The widely-accepted theoretical definition of CAM (see Box 1) was arrived at by the Office of Alternative Medicine (OAM) expert panel at the Conference on CAM Research methodology in April 1995. (The Office of Alternative Medicine later became the US National Center for Complementary and Alternative Medicine (NCCAM).)

This definition is not an operational definition because it does not tell you whether or not, for example, acupuncture or relaxation therapy or omega-3 supplementation are CAM therapies. You could think about whether or not an individual therapy meets the definition above, but the definition does not provide concrete tests to tell you ‘yes’ or ‘no’ about any specific therapy. However, the theoretical definition does provide a major criterion we needed to consider in developing our operational criteria. We needed to take into account whether the therapies are founded upon the theories of disease and healing that are fundamental to the dominant health system in the culture in which the Cochrane CAM field is situated – the conventional Western medical model as it is practiced in the early twenty-first century. That is, therapies such as acupuncture or homeopathy that are based on non-allopathic theories of disease and healing are clearly CAM. One path to developing an operational definition of CAM could therefore involve identifying and listing therapies that rely upon non-allopathic models of health.

The major limitation to considering this criterion, and solely this criterion, in developing an operationalization of CAM is that the dominant medical model changes over time. As we have seen above, despite a common understanding that acupuncture is indeed ‘CAM’, many CAM therapies, including acupuncture, have experienced increased inclusion in Western medical contexts during the past several years. If one were to take the view that whether or not something is CAM depends on the evidence accepted by the dominant system, then defining what is CAM would require a periodic re-evaluation as the evidence changes over time, and the politically dominant health system incorporates the therapy not only in practice but also in principle. For example, what if acupuncture becomes explicable in terms of allopathic mechanisms and is therefore completely accepted within conventional medicine? What if acupuncture is proven highly effective for some indications and thus becomes an accepted treatment even though the allopathic model does not provide a plausible mechanism of action? What if the allopathic model itself evolves to include the concept of Qi? In each of these cases we would be forced to ask whether acupuncture is no longer CAM, despite its origins outside conventional medicine, and if it is no longer CAM then what was the defining moment when it lost its CAM status. Furthermore, it is possible to imagine places or times in which multiple medical paradigms might coexist without any single model achieving clear social or political dominance. For now, we considered these questions but decided to concentrate on defining the parameters of CAM from our current historical and geographical perspective of a dominant allopathic model. However, we felt that we needed to go beyond the non-allopathic criterion in operationalizing CAM.

Criteria considered in developing CAM operational definition

In expanding and refining our CAM operational definition, we considered several criteria. First we considered whether the historical notion of the therapy was CAM or conventional. As suggested above in the theoretical model of CAM, if the therapy was based upon the theories of a medical system outside the Western allopathic medical model, then (from the current perspective of the US and Europe, anyway) it would be labeled alternative medicine, or CAM. We therefore included therapies such as Chinese and Japanese traditional medicine, Ayurvedic medicine, homeopathy, chiropractic, naturopathy and Reiki, among others, as CAM, and we did so regardless of their current level of acceptance by the allopathic system.

For therapies that did not clearly originate outside the theories or beliefs of the allopathic medical system, we next considered whether the use of the therapy for a particular condition is currently considered to be a standard treatment within the dominant medical system. If something is currently a standard, accepted therapy, then it is not likely to be widely considered as CAM. Indicators that a therapy is accepted include government licensing of practitioners, coverage by health insurance, statements of approval by government agencies, and recommendation as part of a practice guideline. For example, the US Federal Drug Administration (FDA) has approved a device for administering transcranial magnetic stimulation (TMS) in treatment resistant depression (http://www.medinewsdirect.com/?p=593). TMS for depression may therefore be questionable as a CAM therapy. While following FDA guidelines would only be helpful for a limited number of therapies and would be exclusive to the US, we considered this to be one source of information [not definitive] on whether or not a specific therapy is CAM. Using the criterion of acceptability to the dominant health system meant that we would have to consider treatments and conditions together, since a treatment might be standard for one condition but alternative for another medical condition. For example, TMS is not approved by the FDA as a treatment for Parkinson Disease. As we stated above, in cases where we had decided that the therapy was CAM based upon its origins outside the allopathic system, we did not change our assessment if the therapy was accepted by insurance or practice guidelines. For example, some health insurance plans in the US and Europe cover acupuncture or chiropractic treatment, but we nevertheless considered these therapies to be CAM.

Third, for therapies that did not clearly originate outside of the allopathic system, we also considered the setting in which the therapy is delivered. Therapies that are self-care or delivered by alternative practitioners are more likely to be widely considered as CAM, while therapies that are delivered exclusively by conventionally credentialed medical personnel or exclusively within hospital settings are much less likely to be considered as CAM. Thus, nutritional therapies, including special diets and dietary supplements, are likely to be widely considered as CAM, while surgery is not considered CAM. Some therapies would be classified differently according to this consideration versus that of the previous criterion. For example, a self-delivered nutritional therapy that is supported by FDA recommendations might be considered as CAM by some persons and as non-CAM by others.

Finally, we should mention that we did not consider evidence of efficacy (or lack of evidence) as a test for identifying a CAM therapy. This is because there are many therapies that are not currently accepted as efficacious, but not all of them would be necessarily considered CAM. For example, a new synthetic chemotherapy agent would not be considered CAM, even if it has not been proven to be efficacious, while an herbal therapy for cancer would generally be considered CAM, even where it had trial evidence of efficacy. As with CAM therapies, many conventional therapies also do not have convincing evidence of benefit. An earlier assessment of reviews from the Cochrane Collaboration found that less than 25% of Cochrane reviews on conventional biomedical interventions resulted in significant evidence of benefit.¹⁰ Therefore, lack of proven efficacy is not an appropriate test for CAM.

With these criteria in mind, and a beginning set of CAM therapies, we looked for sources that would give us additional guidance in developing a list of specific therapies for our operational definition.

Sources considered in developing the operational definition of CAM

We examined several existing sources in developing the CAM operational definition. First, we looked at two sources within the US National Library of Medicine’s PubMed database: 1) the MeSH (Medical Subject Headings) definition of complementary therapies and 2) the Complementary Medicine subset search strategy (also called CAM on PubMed). The MeSH definition of complementary therapies is “Therapeutic practices which are not currently considered an integral part of conventional allopathic medical practice.” This is a theoretical rather than an operational definition, and therefore was of limited use for our purposes. The CAM on PubMed search strategy conducts a complex search within MEDLINE, with the terms in the search strategy representing different CAM therapies and CAM journals.¹¹ It is a form of an operational definition. However, the subset search strategy is not precise as to therapies/conditions pairings, and it is not sensitive or specific enough.

We next considered conducting a Delphi survey, which is a structured series of surveys consisting of rounds of opinion collection and feedback, to arrive at an operational definition of CAM. However, we decided against it, primarily because of time and resource limitations. We did consult recent research utilizing Delphi surveys, and confirm that we did not omit any CAM therapies identified by this strategy from our own operational definition.⁹

Classification tree structures present in existing electronic resources of CAM

At this point, we decided to take a step back and look at borrowing from existing structures to develop our own classification tree. We examined the structures of the following three classification systems that already exist online in CAM: the UK NHS Evidence-CAM specialist collection (NHS Evidence-CAM collection), the US National Library of Medicine (NLM) and its Medical Subject Headings (or MeSH), and the categories of CAM delineated by the US National Institutes of Health Center for Complementary and Alternative Medicine (NCCAM).

We began by looking at the NHS Evidence-CAM collection, which is an extensive online library of evidence, education, and patient information on CAM. The portal for the NHS Evidence-CAM collection lists the following categories of CAM therapies alphabetically: “acupuncture, aromatherapy, chiropractic, dietary and nutritional therapies, herbal medicine, homeopathy, hypnosis, massage, meditation, osteopathy, reflexology, yoga, other therapies or medical systems”. Clicking on “other therapies or medical systems” brings up a list of 15 additional heterogeneous therapies. We decided to use all of the major categories from the NHS Evidence-CAM collection as major categories in our CAM Field scheme. Therefore the two schemes would be compatible, leaving open the possibility of future collaboration. However, we decided to look further and see whether we could construct a tree structure based on a more conceptual rationale and without relying on putting multiple therapies under the category of “other”, thinking that a more specific structure might be easier to navigate.

We then looked at NLM’s MeSH heading structure for the MeSH term Complementary Therapies. This list of MeSH headings is organized alphabetically, but there are many differences between the Complementary Therapies MeSH tree and the NHS Evidence-CAM collection structure described above. As one example, in the MeSH headings acupressure is listed as its own major heading while in the NHS Evidence-CAM collection acupressure was listed under the category ‘other’. The MeSH placement of acupressure might make it easier to locate quickly, and the MeSH structure does not include an ‘other’ category. However, there are some anomalies in the MeSH structure. For example, some of the major headings are of relatively minor therapies. Anthroposophy, which derives from the philosophy of Rudolph Steiner, has only 134 citations in all of Medline/PubMed and probably should not be a major heading. Also, the MeSH headings are not always organized in an intuitively obvious or logical fashion. For example, the alternative medical systems of Homeopathy and Naturopathy are major headings but non-western alternative medical systems (e.g., Ayurvedic Medicine) are grouped together as “traditional medicine”. We thought that perhaps we could build on aspects of the MeSH structure but look elsewhere for an overarching organizational scheme and thus avoid some of the limitations we saw here.

The last structure we looked at was that developed at the US National Institute of Health’s NCCAM. NCCAM groups CAM practices into four domains, or types of therapies, recognizing there can be some overlap. In addition, NCCAM has a category of CAM whole medical systems, which cut across all domains.

The great appeal of this way of organizing CAM is that there are overarching principles by which the therapies can be organized. A possible problem with this system is that someone who does not understand the language and principles of CAM might have some difficulty in knowing what category a specific therapy would fall within. In some cases, even someone who is familiar with CAM would have several valid options for where a therapy might be classified.

Classification suggestions from the CAM literature

Finally, we reviewed the CAM literature to identify additional proposals for how to organize and classify CAM therapies. Suggestions for organizational schema included the following: epistemological perspective and type of healing method,¹³ a taxonomy of five sectors of CAM,¹⁴ four paradigms of health and disease,¹⁵ and modes of therapeutic action.¹⁶ While each of these classification schema had merit and each provided valuable insights into the parameters of CAM and the underpinnings of CAM practices, we did not feel that any one of them was sufficiently well-known or widely used to adopt for the Cochrane Field topics list.