Understanding the effects or outcome of psychotherapy is essential for both research and clinical practice, yet the on-going discrepancy between both fields impedes a mutual dialogue (Castonguay, Barkham, Lutz & McAleavey, 2013; Kazdin, 2008). As outcome research has a huge impact on clinical practice through health care policy (APA, 2006; McLeod, 2016), the question of how to define and determine the effects of therapy remains a central issue. Evidence-Based Practice emphasises ‘the integration of the best available research with clinical expertise in the context of patient characteristics, culture, and preferences’ (APA, 2006, p. 273; emphasis added), yet in terms of policy this is often limited to the “best available evidence”, which in turn is predominantly reduced to highly standardized efficacy studies. Unsurprisingly, given the complexity of clinical practice in general and treatment outcome in particular, this tendency is heavily criticized in both the academic literature and the clinical field (Gaudiano & Miller, 2013; Hill & Lambert, 2004; Westen, Novotny & Thompson-Brenner, 2004). To say the least, further conceptual and methodological developments are required in order to curtail the divide between research and clinical practice (Kazdin, 2008; McLeod, 2011).
In studying the effects of therapy, outcome has traditionally been separated from the process of treatment (Hill & Lambert, 2004), resulting in an arbitrary differentiation. This misconception has ‘clouded the thinking regarding research design’, especially in outcome research, as ‘to some extent process research is outcome research, and outcome research is equivalent to process investigation’ (Kiesler, 1966, p. 126). A unidirectional relationship between process and outcome is furthermore questioned (Stiles & Shapiro, 1989): ‘process measurements taken from on-going therapy may be the result [outcome] rather than the cause [process] of clients’ improvement’ (Stiles, Shapiro & Harper, 1994, p. 42). Long-term follow-up studies have shown the value of investigating treatment outcome over a longer period of time, looking at treatment effects in a process-oriented manner, yet to date such long-term outcome studies remain limited (Ekroll & Rønnestad, 2016).
The impact and consequences of a demarcated or narrowed conception of outcome are most apparent in the so-called gold standard research design, the randomized controlled trial (RCT). High levels of standardization (e.g., strict in- and exclusion criteria, treatment protocols; Westen et al., 2004), an over-reliance on symptom-oriented outcome measures (Braakmann, 2015; Levitt, Stanley, Frankel & Raina, 2005), and the selection of one primary outcome measure to determine treatment outcome (De Los Reyes, Kundey & Wang, 2011), clearly simplify the understanding of treatment effects. Moreover, changes in outcome scores may not represent or correspond to an individual’s functioning in daily life, nor can it give insight into how an individual interpreted the questions asked. Standardized outcome measures are therefore considered “arbitrary metrics” requiring real-life contextualization (Blanton & Jaccard, 2006; Hill, Chui & Bauman, 2013; Kazdin, 2006).
Ergo, a growing interest in qualitative and mixed methods research emerges, as they allow for the contextualization of research findings and in-depth exploration of complex phenomena (Binder et al., 2016; Kazdin, 2008; Klein & Elliott, 2006; Midgley, Ansaldo & Target, 2014). In this regard, the perspective of the patient undergoing treatment is increasingly considered an essential source of information (Binder, Holgorsen & Nielsen, 2009; McLeod, 2016; Strupp & Hadley, 1977; Valkonen, Hanninen & Lindfors, 2011). This first-person perspective stands for the subjective experience, personal view or opinion of patients and must be considered distinct from a researcher or third-person perspective represented in the pre-defined questions and answers of standard outcome scales (Englebert, Follet & Valentiny, 2017; Englebert et al., 2018; Hill et al., 2013).
Opposed to standard outcome research, qualitative inquiry of patient perspectives has revealed important dimensions of change other than symptomatic changes (McLeod, 2011), for instance pointing at existential and interpersonal domains (cf. Binder et al., 2009; Nilsson, Svensson, Sandell & Clinton, 2007). Important individual differences in the experience of outcome have also been observed depending on patients’ personal background (Kühnlein, 1999; Valkonen et al., 2011) and the type of therapy (Nilsson et al., 2007). Moreover, it seems that the effects of psychotherapy must be understood in relation to the broader life span of patients (Midgley, Target & Smith, 2006) and other life domains (Binder et al., 2009). Together these studies show the relevance of investigating the patients’ perspective on psychotherapy and outcome, the importance of a broader conception of treatment effects, and the value of understanding outcome within an idiosyncratic context.
The current study wishes to contribute to these lines of research within the particular context of inpatient psychotherapy, a setting that challenges the field with an even higher level of complexity due to its particular characteristics. The different therapy groups in inpatient treatment can create multiple therapeutic climates (Tschuschke & Dies, 1994), and various elements outside the therapy sessions (e.g., living together on a ward) can facilitate change Leszcz, Yalom, & Norden (1985). Hence, inpatient care cannot be considered equivalent to the sum of its parts. Additionally, inpatient populations are typically more complex, as they often suffer from more chronic mental difficulties and/or are situated within more complex social contexts compared to patients in outpatient psychotherapy. Due to these specific characteristics, the setting does not lend itself well for controlled research and has mostly been investigated by means of basic pre-post measurements (Kösters, Burlingame, Nachtigall & Strauss, 2006). Consequently, little is known about the particular effects of inpatient treatment and change processes over time. Using multiple methods and inquiring about patients’ perspectives, can therefore offer a major contribution to the investigation of this specific and complex setting. In line with current ideas on practice-based evidence (Castonguay et al., 2013), we further believe naturalistic bottom-up inquiry offers important deepening and real-life knowledge, benefitting both research and clinical practice. Ultimately, practice-based research has the potential to circumvent the difficult translation of the context of research into the context of clinical practice; at the same time addressing the sometimes-pressing influence of health care policy (cf. managed care) on the organisation of health care (Gaudiano & Miller, 2013).
In the context of inpatient treatment, the current study aims at understanding long-term outcome within the broader, personal life context of patients. Following a process-oriented understanding of outcome, we investigate which changes former inpatients have experienced during and after inpatient psychotherapy (RQ1) and which factors (in or outside treatment) they believe have influenced these changes (RQ2). Finally, we examine how the perspective of former inpatients complements findings of long-term outcome based on quantitative measurement (RQ3). To answer these objectives, a mixed methods design is deployed, incorporating quantitative long-term outcome findings and qualitative inquiry of patients’ perspectives on changes over time. In an attempt to facilitate the implementation of findings into clinical practice, the present study takes place in a naturalistic setting.
The study relies on naturalistic data gathered in an inpatient psychotherapeutic centre in Antwerp, Flanders (Belgium). The therapy centre offers intense, long-term inpatient psychotherapy to individuals with chronic mental difficulties, predominantly in the range of anxiety and mood disorders, and often in the context of attachment issues and personality disorders (cluster C). The overall approach of the treatment centre requires a certain level of stability, autonomy and self-reliance of patients in order to engage in the intense psychotherapeutic programme. For this reason, the centre must be considered distinct from a classical psychiatric ward; people suffering from acute psychological problems are referred.
After two initial intake interviews, a person engages in an exploratory treatment period of four weeks during which all forms of therapy are explored; only a minority is referred after this phase, when treatment is deemed to be ill-suited. During treatment, each patient receives an individualized programme consisting of four to five forms of therapy, provided in six to nine sessions of individual or group therapy per week. Therapy is based on different frameworks and provided by trained and specialized therapists in cognitive behavioural, systemic and psychoanalytic therapy. Additionally, creative therapy is provided in the form of dance, movement, drama, visual and music therapy. Regular meetings with a psychiatrist are held regarding drug treatment; a personal mentor is the patient’s contact throughout the treatment process. Finally, considerable free time is available for sports and other creative activities, all taking place in a quiet and natural surrounding environment.
Between September 2009 and August 2010, all individuals entering the treatment centre were asked to participate in an internal outcome study. At consecutive time points (i.e., at intake, every two months of treatment, post treatment, one-year and five-year follow-up) outcome was evaluated using the Outcome Questionnaire-45 (OQ-45, Lambert et al., 1996; de Jong et al., 2007). Forty-three participants entered the study; nine dropped out of the study (i.e., no post and follow-up data available). The remaining group (n = 34) provided data with variable response rate: 27 respondents at the end of treatment and 29 at one-year follow-up. The qualitative strand of this study focuses on the second follow-up inquiry, conducted five to six years after treatment termination. For this study, 27 participants (with nearly complete data) were contacted, 24 responded to our call for participation and 22 were willing to take part in the follow-up interview.
The research sample consisted of 19 women and 3 men, ranging in age from 31–61 (M = 47.95; SD = 8.94); all were of Caucasian European decent. Nine participants completed secondary education, 12 had degrees in higher education; seven of the participants were married, three were living together with their partners, and 11 participants indicated they were single (of which six were divorced) at the time of the interview (one participant did not mention her marital status). Eleven of the participants were not working at the time, five of them were retired (often due to medical reasons), were registered as disabled, and one was unemployed at that time. The rest of the participants were employed, of which one was self-employed; three did voluntary work.
Patients were diagnosed at the start of treatment according to DSM-IV-TR (American Psychiatric Association, 2000). The recorded Axis I and Axis II diagnoses were: depressive disorder (n = 14), somatization disorder (n = 5), relational problems (n = 4), anxiety disorder (n = 1), PTSD (n = 2), adjustment disorder (n = 5), personality disorder NOS (n = 12), dependent PD (n = 4), avoidant PD (n = 4), obsessive compulsive PD (n = 3) and borderline PD (n = 1). The majority (n = 18) stayed for the maximum period of 12 months in treatment, one person ended treatment after six months, two people after nine months and one person stayed for a period of ten months. Table 1 gives a summary of the research sample’s treatment before and after inpatient treatment (based on patients’ self-report). Appendix 1 offers a more detailed idiosyncratic description of the participants (in line with Thurin & Thurin, 2007), comprising a brief clinical description, diagnoses and the change in outcome scores per patient.
|Treatment history||Before inpatient treatment||After inpatient treatment||At five-year follow-up|
Both the ethical committee of health institution Emmaus and the Ethical Committee of the University Hospital of Ghent University (Belgium) (registration number: B670201524637) approved the project. Every participant signed a written informed consent, for both the initial quantitative study and the follow-up inquiry.
Before the follow-up interview, participants completed a form asking for information about gender, age, occupation, medication use and psychotherapeutic treatment (before and after treatment).
The Outcome Questionnaire (OQ-45; Lambert et al., 1996; de Jong et al., 2007). The OQ-45 is one of the most frequently used self-report instruments for measuring patient clinical distress and well-being. The measure consists of three dimensions: subjective (symptomatic) distress (intra-psychic functioning), interpersonal relations and social role. Forty-five items are filled out on a 5-point Likert scale (0 = not at all; 4 = always). Sub-scores are calculated for the three sub-categories, the sum of which gives a total-score, ranging from 0 to 180; higher scores reflect greater distress. The Dutch version of the OQ-45 shows good internal consistency (a = 0.96; total score), good test-retest reliability after two to three weeks (r = 0.82; total score for patient population) and sufficient concurrent validity (de Jong et al., 2007).
Semi-structured interview. The first author administered an adjusted version of the semi-structured Client Change Interview (CCI; Elliott, Slatick & Urman, 2001). The interview guide was constructed to evoke participants’ experiences of the period before the inpatient treatment and the reasons for consulting the therapy centre, the changes they believe occurred during and after this treatment period, and what they believe influenced these changes. Interviews were audiotaped and transcripts were analysed.
An explanatory sequential mixed-method study (Hesse-Biber, 2010) was conducted. The quantitative outcome scores and the narratives of former inpatients were analysed separately and integrated at the phase of interpretation, allowing for a complementation and deeper understanding of both the quantitative and qualitative outcome findings. This design can be summarised as “quan → QUAL”: a qualitative strand was added secondary to an existing quantitative study yet becoming the most important focus of the explanatory study (cf. Hesse-Biber, 2010).
The majority of the interviews were conducted at people’s homes, a few were held in the therapy centre and one was administered at the faculty of Psychology and Educational sciences (Ghent University). Interviews lasted 100 minutes on average. An initial in-depth analysis was conducted on a sub-selection of 13 rich cases (displaying diversity in experiences), followed by an analysis of the remaining interviews (n = 11) using the initial themes as a compass. Minor adjustments were needed in order to encompass all narratives in a single final model of long-term outcome.
Participants were classified in terms of reliable change and clinical significance based on four categories: clinical significant change (CS), reliable change (RC), no RC and deterioration (Jacobson & Truax, 1991). The outcome scores of the inpatient population were compared to Dutch norms (de Jong, Nugter, Lambert & Burlingame, 2008). In order to reach reliable change for the OQ-45 total score, a person must show a decrease in scores equal to or larger than 14. The cut-off between the clinical and nonclinical population for the Dutch OQ-45 is set at 55 (based on the test-retest reliability of 0.96; de Jong et al., 2007). We present all available quantitative outcome data (i.e., starting from the entire sample participating in the initial study) measured at the start of treatment (n = 43), at treatment ending (n = 27), at one-year (n = 29) and five-year follow-up (n = 22). To control for potential effects of drop-out, we compared the averages score on the OQ-45 (at the start of treatment) in the drop-out group (n = 9) and the remaining group (n = 34) by means of an independent samples t-test. Dependent samples t-tests were used to verify whether the outcome scores for the remaining group differed statistically significant at the end of treatment, one-year follow up and five-year follow up compared to the start of treatment.
A data-driven Thematic Analysis (Boyatzis, 1998; Braun & Clarke, 2006) based on principles of Grounded Theory (Glaser & Strauss, 1967), was conducted. This form of inquiry enabled the exploration of the phenomenon of long-term outcome in the participants’ terminology and to identify themes in the data in a bottom-up manner. In our understanding of the first-person perspective, we adhere to a contextual idea: people give meaning to their experiences yet are influenced by the broader social context (Boyatzis, 1998; Braun & Clarke, 2006). The analysis focused on the macro-process of psychotherapy, i.e., a wide angle in contrast to the analyses of micro-processes (e.g., specific therapy effects). Moreover, the study aimed at investigating the subjective experience of several different participants (i.e., between-case variation) (Denzin & Lincoln, 2005).
Both authors coded the interviews separately; after each phase of analysis discussions were held until agreement was reached through a consensual process (cf. Hill, Thompson & Williams, 1997). The process of inquiry contained the following steps that were moved along in an iterative manner:
In function of the reliability of the study, we tried to remain transparent about the entire process (Stiles, 1993) and acknowledge the influence of the perspective and background of the researchers. The first author conducted the interviews, making her personally involved with the participants and consequently also with the therapy centre. By avoiding preparatory knowledge (on the treatment setting and the quantitative outcome results), the influence of this personal involvement was countered as much as possible. The researchers’ personal interest in idiosyncratic experiences clearly navigated the focus of the study and analysis. By making this explicit, consequences of implicit guiding assumptions were controlled as much as possible (Creswell & Miller, 2000). In function of the validity of the study we worked in a systematic manner to form conclusions and interpretations (Stiles, 1993). Triangulation among researchers, several interviews (divided into two subsamples), and quantitative and qualitative indications of outcome were applied to gain different perspectives on the issue. The ultimate themes were formed by asking critical questions regarding codes and categories (Mortelmans, 2011). We attempted to stay open for any information coming from the narratives throughout the entire process (cf. non-framing interview format).
We first present the quantitative long-term outcome data from the start of therapy to five-year follow up and look into patients’ perspectives on change and explanatory factors thereafter. Interpretations of the separate findings will be described in the result section; broader integrative conclusions and implications will be presented in the discussion.
Table 2 summarizes the change in average total scores on the OQ-45 over time, the standard deviations (SD) and range in scores (i.e., minimum and maximum score) as well as their meaning in comparison to a clinical and normal population. Table 3 shows the number of patients in the clinical and functional range, as well as CS change, RC, no change and deterioration.
|Average total score (OQ-45)||Research sample||Clinical population||Normal population|
|Start therapy||97||16.5 (60–144)||High||80–95||Very high||95–100|
|End therapy||67||22.2 (36–115)||Below average||20–40||High||80–95|
|One-year FU||73||22.7 (24–124)||Average (men)
Below average (women)
|Five-year FU||76.7||19.6 (30–116)||Average||40–60||Very high||95–100|
|OQ-45 total score||Start therapy||End therapy||One-year FU||Five-year FU|
|Change from start treatment|
The average total outcome score decreased from start to end of treatment, and slightly increased again during the follow-up period. The decrease in scores showed to be statistically significant at all time points (for n = 34) with an average decrease of 30 points (95% CI: 21.01–39.1; p < 0.01) from start to end of treatment, 23.2 points (95% CI: 14.38–32.1; p < 0.01) from start to one-year follow up and 18.7 points (95% CI: 7.68–29.69; p < 0.01) from start to five-year follow up. The average distress level in the drop-out group (M = 100.56; SD = 12.053) and remaining group (M = 96.68; SD = 17.728) did not differ significantly (M = –3.879; SD = 6.287; 95% CI: –16.577–8.818, p = 0.274); the decrease in scores thus does not seem influenced by the participants who dropped out from the study. Despite the significant diminution in severity of distress over time, the average score remains high in comparison to a normal population. In terms of clinical significance, the changes in scores indicate the majority of participants have improved reliably, and this until five-year follow-up, yet only a minority reaches and remains the level of clinical significant change. The large range in scores further suggests a strong variety in the level of functioning among participants at all measurement points.
Figure 1 presents the model of long-term outcome that comprises both a taxonomy of change and explanatory factors. In order to indicate variation in the prevalence of the categories and factors we rely on different phrasings: “(nearly) all” participants, indicates 90–100%, “most of” or “the majority” stands for 80–90%, “many” or “often” indicates 60–80%, “some(times)” or “others” means about 30–50%, and “a few” ranges around 10–20% of the participants.
Table 4 presents the taxonomy of the experienced changes during and after inpatient treatment, differentiated in five categories that represent different levels of existential change: I. Reconnection to others and (the meaning of) life, II. A revelation, III. An altered self, IV. Life changes, and V. Altered expectations and ideas about recovery and treatment. In the conceptual model of long-term outcome presented in Figure 1, these changes are depicted along a timeline, indicating changes occurred at different moments in time. Without making causality claims, former patients described how the initial changes (i.e., I & II) have helped them engage in treatment and facilitated later changes (i.e., III, IV & V). In the resulting model, the experienced changes are positioned closely to each other, as they must be understood as strongly interrelated, i.e. changes coincided and influenced each another.
|Core and subcategories|
Foremost, nearly all former patients experienced a feeling of belonging at the inpatient treatment centre. They described how they could recognise themselves and their difficulties in fellow patients, were accepted for who they are and felt worthy. Furthermore, patients gained a new perspective on life, were filled with renewed energy and enthusiasm for life, and gained hope and courage to go on and overcome difficulties. Together these changes were interpreted as an experience of reconnection, which is in stark contrast to the disconnection, i.e., exclusion, loneliness, failure and hopelessness, people experienced in society before entering inpatient treatment. For many, this feeling of reconnection was not only an important condition for engaging in therapy, but also meant a lasting change as such.
‘What I remember very well is the aspect of being heard, being respected and treated as normal. I always had the feeling there was something wrong with me, that I was not okay, or it was not good enough, I had to try harder or they did not want me; I did not belong. At [the inpatient therapy centre] that all vanished. Now I feel at home anywhere I go.’
Secondly, for nearly all participants, the treatment period lead to a revelation. First of all, people mentioned “eye-openers” or insights regarding themselves and their difficulties as one of the most important changes.
‘The biggest change for me was that, you think you know what your problem is, but you notice it can be something completely different. (…) At a certain point, you notice it goes much deeper.’
Also, people described how the various creative forms of therapy offered them alternative ways, or even a first opportunity, to express emotions and thoughts; many continued to use these new ways after the treatment period (in the form of therapy, taking specific courses or hobbies).
‘To the outside world I have always been the “ever-smiling person”, I did not know anger and swallowed my frustration. Here [in the inpatient therapy centre] it changed a lot. I have learned to give words to my feelings, by first painting them on a canvas or by making something in clay.’
At a later stage in treatment and to a lesser extent after treatment, patients experienced changes in their personality. On the one hand, many participants had become milder for themselves and others, showing a process of mollification. Patients stated they have learned to acknowledge their own boundaries and accept difficulties that happened in the past, by taking them seriously.
‘I have become softer, or that’s what I am being told now by others. At my job, I now indicate my limits beforehand, as if I allow myself a private life now. (…) And I have learned it’s okay to do ‘nothing’ for a while.’
On the other hand, many of the participants indicated they have become more independent, stronger and gained self-esteem, reflecting a feeling of empowerment.
‘I have learned to stand up for myself, to choose for myself. I have been able to take more distance from my parents, with the feeling: “it’s okay; it’s okay to think about yourself, to take yourself seriously”.’
Experienced changes at the level of personality coincided with important concrete changes in dealing with or handling things in life, increasingly taking place during and after treatment.1
‘Since then [the inpatient treatment] I have always gone home [from work] on time and never took my computer with me. At five o’clock I could say: “it’s time to go home, I’m out.” Before, coming home meant turning on the computer and I could keep on working.’
Apart from changes in employment, also housing was often reconsidered and changes in family constitution and intimate relations (e.g., new partner or divorce) occurred. For some of the participants these changes were life-altering, as the period of inpatient psychotherapy truly meant a turning point in life.
‘My entire world has changed by coming here [the inpatient therapy centre]. I opened up here, my social life started and subsequently I met my husband. After years of invalidity and depending on my parents, my husband and I now live together and have a little toddler running around. I still feel tired, but I have people that rely on me now.’
Nearly all of the participants disclosed they have been dealing with severe difficulties throughout their lives and none of them stated they were “cured”. However, during and after the treatment period a change in ideas and expectations about recovery and treatment occurred. The initial idea that after a couple of months all problems would be solved, changed for most patients to perceiving recovery as an on-going and fluctuating process. Apart from the many valuable changes, and independent from the situation at the time of the interview, many participants also experienced a breakdown or recurrent difficulties.
For most of the participants, seeing recovery as an on-going and rather fluctuating process functioned as a resignation, leading to adjusted (i.e., more realistic) expectations and understanding of their own process of recovery, subsequently influencing concrete changes in life and coinciding with changes in personality (III & IV). A few participants, however, seemed to hold on to a more fixed idea about recovery or had an ideal image of treatment, eventually resulting in a disappointment, in themselves or in psychotherapy, as these (sometimes high) expectations were not met.
‘I believed all my problems would be solved: I would easily find work and build new social contacts, I would have a better relationship with my parents. But I expected too much. My image about the world and future had become too idealistic.’
Figure 1 depicts four broad factors in patients’ narratives that have explanatory value with respect to the experienced changes and their interrelation: A. The person’s unique starting point and positioning in treatment, B. Central elements of the therapy centre (in relation to the person), C. The possibility to congregate and evolve, and D. The relation to the exterior world. These factors must be understood as evolving throughout and after the treatment period and are thus also depicted along the timeline in Figure 1.
Important differences were observed among former patients regarding when or where in the course of their lives and health care trajectory they had entered inpatient treatment, making up the person’s unique starting point. Whether the inpatient treatment followed previous psychotherapeutic experiences or not, consequently influenced the process of inpatient therapy, the extent to which insights were gained and changes in personality and life could occur.
‘At that moment [in inpatient treatment] I was really ready for it, and I really wanted to do something about it. A bit opposed to the years before, I think. When I was hospitalized in psychiatry they mainly had to keep me alive. This [the inpatient therapy centre], however, has been my final hospitalization.’
Secondly, former patients described different ways of positioning oneself in treatment that could be summarized as: dependent-passive, assertive-active, overactive, and independent – keeping a distance. These four positions are neither mutually exclusive nor fixed yet present a predominant way of engaging in treatment; Table 5 provides examples of every position. Whether someone displayed a high need of guidance, engaged actively in treatment, was preoccupied with gaining instant effects, or had difficulties opening up in treatment (e.g., due to trust issues), influenced the treatment process differently and subsequently also the experienced changes.
|Position in treatment||Example excerpt|
high need of guidance, support and initiative from therapists
|‘It was hard when the initiative had to come from me. Maybe it’s my personality; I often need a lead, as I often tend to lean on others. And maybe it was the plan of the therapists that I would take initiative and stand up for myself. But having to create things myself in therapy was hard.’|
highly engaged in treatment process, taking own initiative
|‘Being here [in the inpatient therapy centre] was a unique opportunity I had to grasp with my both hands. I chose for myself, I wanted to become a happier person, so I engaged in everything they offered me. At a certain point, I even asked for an extra form of therapy.’|
preoccupied with “working hard” and time-efficiency of treatment
|‘I had prepared myself for [the inpatient] treatment. I had made drawings with all my characteristics on it, I wanted to know all of it, I did not want to do nothing there [in the inpatient therapy centre]; I wanted to work on myself.’
‘When I was drawing in therapy I thought: “I have to be at my department, I have to lead my team, my colleagues!”’
Independent-keeping a distance:
avoiding participation or inhibited to fully engage
|‘I’ve realized, I’m not very good at therapy… Personal conversations with people I don’t know so well are hard, it takes a while before I trust a person. I’m very secretive. I constantly censor myself and consider ‘what can I share and what not’ and “shouldn’t I be solving this on my own”.’|
A reciprocal arrow between the person and therapy centre in Figure 1 signifies the importance of the interaction between both factors. Specifically regarding the interaction of the patient and the overall approach of the therapy centre, people who described a more dependent-passive and independent-keeping a distance position stated to have encountered the most difficulties in displaying autonomy and self-reliance. Patients with an assertive-active position seemed to engage more easily in a process of self-exploration, while people in an over-active position encountered difficulties with the dimension of taking time for oneself (e.g., due to a feeling of not working hard enough); see Table 5.
Accordingly, the extent to which the treatment was experienced as well adjusted to personal needs differed from participant to participant, yet all mentioned at least one therapy in their treatment package they had found helpful (cf. adjusted therapy programme). The connection with the therapist (and by extension the personal mentor and psychiatrist) and the group appeared crucial for changes to occur.
‘It was hard for me to open up, and most of them [the therapists] let me be, but there was one therapist who saw through me. He said: “you are here long enough, you can make it a little bit harder on yourself”, and I thought “he is right”. The fact that he did not tolerate my attempts to hide [was important]’
The person, the treatment and their interaction must not be understood as fixed. The possibility to evolve towards a coming together of both (i.e., harmony and accordance) appeared essential to understand why changes came about or not. For instance, many described how they have evolved in the position of being in treatment (e.g., from passive to active; overcoming uncertainty and distrust), or how therapy was altered to reach a patient’s (changing) needs (i.e., via therapists’ interventions or by changing a therapy programme). A rift in the relation or evolution, on the other hand, could lead to the undermining of further treatment and change.
‘After nine months, they [the staff] told me: “we cannot continue to work with you, we are stuck.” And I knew it myself. They wanted me to participate actively but my reaction was to retreat, to build a wall around me. And they were right, no one could get in.’
Participants described their experiences explicitly in comparison and in relation to society or reality outside the therapy centre. To many, the period in the inpatient treatment centre offered a healing experience and buffer against hostile encounters (e.g., discrimination, stigmatization, high demands and pressure), although some also suffered from this contrast with the outside world.2
‘I miss the warmth of the therapy centre; people listened to you, it was okay to say “I’m not feeling very well today”. In society, this is just not tolerated, if you state you’re not doing okay you are pushed aside; you always need to be strong.’
Furthermore, the personal context (i.e., family, social and professional context) had an important influence on the treatment and change process. For some, this context was a reason for being hospitalized and distance as such was salutary; to others, the context was left aside too much during the treatment period. The extent to which, for instance, family, social life and further career planning had been integrated into the treatment period seemed an important facilitator of change and transition to daily life after ending treatment. For some patients, this was however beyond their possibilities at this point in their treatment trajectory (cf. unique starting point; A). Further, family at home sometimes meant an important reason to prematurely end therapy.
‘My daughter asked me to come home again, as I had promised my treatment would maximally take one year. But actually, my treatment wasn’t entirely finished then. Now I regret that [premature treatment termination].’
Long-term outcome findings show improved well-being for the majority of former patients and this until five to six years after ending inpatient treatment. Yet, in order to obtain a deeper understanding and contextualization of these findings, long-term outcome scores were complemented by the patients’ own perspective.
The conceptual model based on the qualitative analysis of patients’ narratives indicates long-term outcome can be understood as an ongoing and fluctuating process of predominantly existential changes. These changes are furthermore strongly interrelated and were described as coinciding and influencing each other over time (cf. Binder et al., 2009; Ekroll & Rønnestad, 2016). In line with previous (mostly outpatient) qualitative research, participants in this study mentioned changes in the domains of the self (in terms of personality), interpersonal relations (here more broadly described as the exterior world), and specific life conditions (Klein & Elliott, 2006; Midgley et al., 2006; Nilsson et al., 2007). Concrete life changes seem to correspond to behavioural changes mentioned in other studies, while alternative ways of expressing emotions and thoughts may be seen as different ways of coping (Ekroll & Rønnestad, 2016; Midgley et al., 2006; Nilsson et al., 2007). Accordingly, we observed a decrease in treatment intensity (i.e., medication, counselling, and hospitalization) following the inpatient treatment, while creative forms of therapy and other activities increased. Central to our findings is that none of the participants considered themselves as “cured” during or after treatment, yet nearly all emphasized the importance of the inpatient treatment for obtaining fundamental changes. This seems to correspond to the observation that only a minority of patients reached clinically significant change in outcome scores, although the majority showed reliable improvement compared to their initial level of functioning. Interestingly, participants in this study evaluated their process of recovery in light of their own personal life story while rejecting the normative standards of society. The inpatient treatment centre offered a healthier and healing experience that was strongly contrasted with the outside world. Whilst previous qualitative studies found changes in terms of accepting own limits and weaknesses (Kühnlein, 2011; Nilsson et al., 2007), unique for this study seems to be patients’ different perspective on recovery and treatment.
This finding touches upon the ethical question of how recovery, functioning and dysfunctioning are conceptualized. In accordance with the altered perspective of the participants in this study, a broader view on recovery can be noticed in the organisation of (mental) health care: a dichotomous understanding of “health” versus “illness” is increasingly replaced by a definition of “positive health” (Delespaul et al., 2016; Huber et al., 2011; Slade & Longden, 2015). The latter emphasizes the possibilities and abilities, given psychological, physical and social challenges and is considered more in line with clinical practice (Delespaul et al., 2016). For most of the participants, this different appraisal of recovery offered resignation, while fixated ideals were met with disappointment. Conceptualizations of dysfunctioning and recovery also provide the (often implicit) assumptions underlying the measurements in psychotherapy and outcome research (Imel & Wampold, 2008; Rieken & Gelo, 2015; Slife, 2004). From a positive perspective on recovery, the individual process and evaluation becomes central and comparison to others is only possible to a certain extent (Slade & Longden, 2015). The research field faces the challenge of adapting to the shifting perspective on outcome and the growing body of idiosyncratic research (quantitative and qualitative) is clearly promising in that respect. The current study aimed at contributing to these developments in the context of inpatient psychotherapy.
Our findings highlight the multiplicity of long-term outcome, substantiating the relevance of a broader conception and measurement of treatment effects. The complex interplay of the patient, treatment, external context and evolution over time further indicates that experienced changes cannot be explained by single or fixed factors. In this respect, our results support a dynamic understanding of the patient-therapist relationship as suggested by Clarkin and Levy (2004), although “the therapist” requires a broader interpretation in our study (i.e., elements of the treatment centre). Despite its residential nature, the influence of the context outside therapy must also be considered in this dynamic patient-treatment relationship (cf. McLeod, 2016). Regarding the patient, our study points at the importance of a patient’s starting point in treatment (influenced by life experiences and previous health care), which relates to the concept of patient characteristics that is broadly considered the single most important factor for explaining treatment effects (for an overview, see Bohart & Wade, 2013). While the patients’ perspective is often explored after treatment, it is arguable that a more in-depth understanding of the patient’s starting position can enhance knowledge on tailoring psychotherapy to individual needs (Norcross & Wampold, 2011). Qualitative analysis of patient’s experience before treatment could mean a valuable contribution here. Further, whereas most research assesses patients’ expectations statically, our study shows the potential relevance of evaluating treatment expectations as changing over time (Greenberg, Constantino & Bruce, 2006). Longitudinal prospective research would enable to look at individual trajectories of change in inpatient treatment (Bohart & Wade, 2013). The different positions in treatment that were found in this study (i.e., dependent, assertive, over-active and avoidant), seem to relate to several characteristics described in psychotherapy literature, like a patient’s “state of readiness”, level of engagement and possibility to establish good therapeutic alliance (Bohart & Wade, 2013). Further research could focus on how patients’ positioning in treatment evolves and can be facilitated during inpatient care (cf. individualized treatment programmes). The extent to which these different positions can be observed elsewhere requires replication in other settings. Likewise, our findings suggest a feeling of reconnection might be essential for patients to engage in treatment (cf. Ekroll & Rønnestad, 2016). The question can be raised whether this feeling is unique for this setting, specific for inpatient long-term treatment, or can also be observed in outpatient care and across different groups of patients.
The longitudinal design of this study enabled to outline the sustainability of changes over time; the idiosyncratic perspective of former inpatients allowed understanding these changes beyond symptomatic evolutions. Both can be considered important strengths of this study. High levels of complexity and diversity characterize the field of inpatient psychotherapy, and likewise, the treatment centre in this study cannot be considered identical to other inpatient settings (cf. Kösters et al., 2006). Nonetheless, several elements of residential care (e.g., living together on a ward; different forms of group and creative therapy) can be considered shared among inpatient facilities and therefore support transferability to a certain extent (e.g., Leszcz et al., 1985). Still, further research on inpatient settings is warranted to deepen the understanding of this particular therapy context. In the present study, we aimed to contribute to this inquiry by incorporating all available outcome scores and all 22 interviews, resulting in an overall-experience and general model of inpatient treatment. Investigating patients’ experiences of more specific aspects of inpatient treatment (e.g., the role of creative or group therapy), during or shortly after ending treatment, could complement our findings.
The participants in this study resemble a homogenous and local (Caucasian, Flemish) group of former patients. Studies that focus on specific (e.g., cultural, ethnic) or more diverse groups of patients (e.g., gender) could therefore offer an important contribution. Our participants also formed a sub-group of patients that were willing to participate in the study. This could imply that more dissatisfied people were not heard and that therefore relevant knowledge remains unexposed. Although we have incorporated negative experiences in our analyses (cf. Midgley et al., 2006; Nilsson et al., 2007), qualitative research focusing on negative outcomes remains scarce whereas it could offer an important contribution to the understanding of (inpatient) psychotherapy (e.g., Werbart, von Below, Brun & Gunnarsdottir, 2015). In line with this, effects of social desirable answers cannot be fully excluded (Thurin & Thurin, 2007).
Many external influences could have shaped former patients’ perspectives in the period of five to six years after ending treatment. Due to its naturalistic character, our study is limited in terms of standardization and control. For instance, the research sample is characterized by diverse clinical problems (e.g., diagnoses) unlike what is typical in standard outcome studies (Westen et al., 2004). The model of long-term outcome must thus be considered preliminary and we wish not to make strong causal claims regarding the effectiveness of treatment. Rather, the strength of a naturalistic research design is the clinically representative study of a particular context and population of patients, resulting into clinically valuable knowledge (Leichsenring, 2004). Finally, following Strupp and Hadley (1977), we agree that the chosen first-person perspective is only one perspective on outcome (e.g., compared to the therapist or societal perspective), and likewise only sheds light on certain aspects, while leaving others untouched. Nevertheless, studying the patients’ perspective offers major contributions to the clinical meaningfulness of research findings and understanding of psychotherapy outcome.
This study offers support to the idea that, instead of relying on a unified and delineated definition and measurement of outcome or recovery, research and practice can greatly benefit from acknowledging the inherent complexity, multiplicity and diversity of psychotherapy outcome that require the incorporation of multiple methods and perspectives.
1Note that the difference between “an altered self” and “life changes” is specifically situated on the level of the experience and feeling of “being different” (e.g., different mind-set, attitude and character) versus the concrete changes or actions a person undertook resulting from or coinciding with changes in personality (cf. categories are strongly interrelated).
2The term “reality” was taken from patients’ own description. A distinction between the “outside world” (broadly termed as “the society”) and the “inside world” of the treatment centre was strongly made. This reality or outside world was mostly seen as “cold” and in stark contrast with the safe and caring environment of the treatment centre.
We want to thank the managing board and staff of the therapy centre for all practical assistance during this study and we are grateful to the patients for their participation.
We like to acknowledge Ufuoma Angelica Norman, Goedele Hermans and Hilma-mae Onokpite for their careful reading of the final draft of this manuscript.
Melissa De Smet is funded by the FWO.
The authors have no competing interests to declare.
APA Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285. Retrieved from: http://www.ovid.com/site/index.jsp. DOI: https://doi.org/10.1037/0003-066X.61.4.271
Binder, P. E., Holgersen, H., & Nielsen, G. H. (2009). What is a “good outcome” in psychotherapy? A qualitative exploration of former patients’ point of view. Psychotherapy Research (iFirst), 1–10. DOI: https://doi.org/10.1080/10503300903376338
Binder, P. E., Schanche, E., Holgersen, H., Nielsen, G. H., Hjeltnes, A., Stige, S. H., Multu, C., et al. (2016). Why do we need qualitative research on psychological treatments? The case for discovery, reflexivity, critique, receptivity, and evocation. Scandinavian Psychologist, 3, 1–22. DOI: https://doi.org/10.15714/scandpsychol.3.e8
Blanton, H., & Jaccard, J. (2006). Arbitrary metrics in psychology. The American Psychologist, 61(1), 27–41. DOI: https://doi.org/10.1037/0003-066X.61.1.62
Bohart, C. A., & Wade, A. M. (2013). The client in psychotherapy. In: Lambert, M. J. (ed.), Bergin and Garfield’s handbook of psychotherapy and behaviour change (6th Ed.), 219–257. Hoboken, New Jersey: John Wiley & Sons, Inc.
Braakmann, D. (2015). Historical paths in psychotherapy research. In: Gelo, O. C. G., Pritz, A., & Rieken, B. (eds.), Psychotherapy research: Foundations, process and outcome, 39–66. Vienna, Austria: Springer.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77–101. DOI: https://doi.org/10.1191/1478088706qp063oa
Castonguay, L., Barkham, M., Lutz, W., & McAleavey, A. (2013). Practice-oriented research: Approaches and applications. In: Lambert, M. J. (ed.), Bergin and Garfield’s handbook of psychotherapy and behaviour change (6th Ed.), 85–133. Hoboken, New Jersey: John Wiley & Sons, Inc.
Clarkin, J. F., & Levy, K. N. (2004). The influence of client variables on psychotherapy. In: Lambert, M. J. (ed.), Bergin and Garfield’s handbook of psychotherapy and behaviour change (5th Ed.), 194–226. Hoboken, New Jersey: John Wiley & Sons, Inc.
Creswell, J. W., & Miller, D. L. (2000). Determining validity in qualitative inquiry. Theory into practice, 39(3), 124–130. Retrieved from: https://people.ucsc.edu. DOI: https://doi.org/10.1207/s15430421tip3903_2
de Jong, K., Nugter, M. A., Polak, M. G., Wagenborg, J. A. E., Spinhoven, P., & Heiser, W. J. (2007). The Outcome Questionnaire (OQ-45) in a Dutch population: A cross-cultural validation. Clinical Psychology & Psychotherapy, 14, 288–301. DOI: https://doi.org/10.1002/cpp.529
De Los Reyes, A., Kundey, S. M. A., & Wang, M. (2011). The end of the primary outcome measure: A research agenda for constructing its replacement. Clinical Psychology Review, 31, 829–838. DOI: https://doi.org/10.1016/j.cpr.2011.03.011
Ekroll, V. B., & Rønnestad, M. H. (2016). Processes and changes experienced by clients during and after naturalistic good-outcome therapies conducted by experienced psychotherapists. Psychotherapy Research, 3307, 1–19. DOI: https://doi.org/10.1080/10503307.2015.1119326
Elliott, R., Slatick, E., & Urman, M. (2001). Qualitative change process research on psychotherapy: Alternative strategies. In: Frommer, J., & Rennie, D. L. (eds.), Qualitative psychotherapy research: Methods and methodology, 69–111. Lengerich, Germany: Pabst Science.
Englebert, J., Follet, V., & Valentiny, C. (2017). Anorexia Nervosa and first-person perspective: Altruism, family system and body experience. Psychopathology, 51, 24–30. DOI: https://doi.org/10.1159/000485629
Englebert, J., Stanhellini, G., Valentiny, C., Fuchs, T., & Sass, L. (2018). Hyper-reflexivité et perspective en première personne: Un apport décisif de la psychopathologie phénoménologique contemporaine à la comprehension de la schizophrénie. L’évolution psychiatrique, 83, 77–85. DOI: https://doi.org/10.1016/j.evopsy.2017.07.004
Gaudiano, B. A., & Miller, I. W. (2013). The evidence-based practice of psychotherapy: Facing the challenges that lie ahead. Clinical Psychology Review, 33, 813–824. Retrieved from: http://www.womha.com. DOI: https://doi.org/10.1016/j.cpr.2013.04.004
Giorgi, A., & Giorgi, B. M. (2003). The descriptive phenomenological psychological method. In: Camic, P. M., Rhodes, J. E., & Yardley, L. (eds.), Qualitative research in psychology: Expending perspectives in methods and design, 243–274. Washington, DC: American Psychological Association. DOI: https://doi.org/10.1037/10595-013
Greenberg, R. P., Constantino, M. J., & Bruce, N. (2006). Are patient expectations still relevant for psychotherapy process and outcome? Clinical psychology review, 26(6), 657–678. DOI: https://doi.org/10.1016/j.cpr.2005.03.002
Hill, C. E., Chui, H., & Baumann, E. (2013). Revisiting and reenvisioning the outcome problem in psychotherapy: An argument to include individualized and qualitative measurement. Psychotherapy, 50(1), 68–76. DOI: https://doi.org/10.1037/a0030571
Hill, C. E., & Lambert, M. J. (2004). Methodological issues in studying psychotherapy processes and outcomes. In: Lambert, M. J. (ed.), Bergin and Garfield’s handbook of psychotherapy and behaviour change (5th Ed.), 84–135. Hoboken, New Jersey: John Wiley & Sons, Inc.
Hill, C. E., Thompson, B. J., & Williams, E. N. (1997). A guide to conducting consensual qualitative research. The counselling psychologist, 25(4), 517–572. DOI: https://doi.org/10.1177/0011000097254001
Huber, M., Knottnernus, J. A., Green, L., van der Horst, V., Jadad, A. R., Kromhout, D., Smid, H., et al. (2011). How should we define health? British Medical Journal, 1–3. DOI: https://doi.org/10.1136/bmj.d4163
Imel, Z., & Wampold, B. (2008). The importance of treatment and the science of common factors in psychotherapy. In: Brown, S. D., & Lent, R. W. (eds.), Handbook of counselling psychology (4th Ed.), 249–262. Hoboken, New Jersey: John Wiley & Sons, Inc.
Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 12–19. DOI: https://doi.org/10.1037//0022-006X.59.1.12
Kazdin, A. E. (2006). Arbitrary metrics: Implications for identifying evidence-based treatments. The American Psychologist, 61(1), 42–71. DOI: https://doi.org/10.1037/0003-066X.61.1.42
Kazdin, A. E. (2008). Evidence-based treatment and practice – New opportunities to bridge clinical research and practice, enhance the knowledge base, and improve patient care. American Psychologist, 63(3), 146–159. DOI: https://doi.org/10.1037/0003-066x.63.3.146
Kiesler, D. J. (1966). Some Myths of Psychotherapy Research and the Search for A Paradigm. Psychotherapy Research, 5(2), 110–136. DOI: https://doi.org/10.1080/10503309512331331186
Klein, M. J., & Elliott, R. (2006). Client accounts of personal change in process experiential psychotherapy: A methodologically pluralistic approach. Psychotherapy Research, 16, 91–105. DOI: https://doi.org/10.1080/10503300500090993
Kösters, M., Burlingame, G. M., Nachtigall, C., & Strauss, B. (2006). A meta-analytic review of the effectiveness of inpatient group psychotherapy. Group Dynamics: Theory, Research, and Practice, 10(2), 146–163. DOI: https://doi.org/10.1037/1089-2622.214.171.124
Kühnlein, I. (1999). Psychotherapy as a Process of Transformation: Analysis of Post-therapeutic Autobiographic Narrations. Psychotherapy Research, 9(3), 274–287. DOI: https://doi.org/10.1080/10503309912331332761
Lambert, M. J., Hansen, N. B., Umphress, V., Lunnen, K., Okiishi, J., Burlingame, G. M., & Reisinger, C. W. (1996). Administration and scoring manual for the OQ-45.2. East Setauket, NY: American Professional Credentialing Services.
Leichsenring, F. (2004). Randomized controlled versus naturalistic studies: A new research agenda. Bulletin of the Menninger Clinic, 68(2), 137–151. DOI: https://doi.org/10.1521/bumc.126.96.36.199952
Leszcz, M., Yalom, I. D., & Norden, M. (1985). The value of inpatient group psychotherapy: Patients’ perceptions. International Journal of Group Psychotherapy, 35(3), 411–433. DOI: https://doi.org/10.1080/00207284.1985.11491425
Levitt, H. M., Stanley, C. M., Frankel, Z., & Raina, K. (2005). An evaluation of outcome measures used in humanistic psychotherapy research: Using thermometers to weigh oranges. The Humanistic Psychologist, 33, 113–130. DOI: https://doi.org/10.1207/s15473333thp3302_3
Midgley, N., Ansaldo, F., & Target, M. (2014). The meaningful assessment of therapy outcomes: Incorporating a qualitative study into a RCT evaluating the treatment of adolescent depression. Psychotherapy, 51(1), 128–137. DOI: https://doi.org/10.1037/a0034179
Midgley, N., Target, M., & Smith, J. (2006). The outcome of child psychoanalysis from the patient’s point of view: A qualitative analysis of a long-term follow-up study. Psychology and Psychotherapy, 79, 257–269. DOI: https://doi.org/10.1348/147608305X52694
Nilsson, T., Svensson, M., Sandell, R., & Clinton, D. (2007). Patients’ experiences of change in cognitive-behavioural therapy and psychodynamic therapy: A qualitative comparative study. Psychotherapy Research, 20, 37–41. DOI: https://doi.org/10.1080/10503300601139988
Norcross, J. C., & Wampold, B. E. (2011). What works for whom: Tailoring psychotherapy to the person. Journal of Clinical Psychology, 67(2), 127–132. DOI: https://doi.org/10.1002/jclp.20764
Rieken, B., & Gelo, O. C. G. (2015). The philosophy of psychotherapy science: Mainstream and alternative views. In: Gelo, O. C. G., Pritz, A., & Rieken, B. (eds.), Psychotherapy research: Foundations, process and outcome, 67–92. Vienna, Austria: Springer.
Slade, P. M., & Longden, E. (2015). Empirical evidence about recovery and mental health. Bio Med Central Psychiatry, 1–14. DOI: https://doi.org/10.1186/s12888-015-0678-4
Slife, B. D. (2004). Theoretical challenges to therapy practice and research: The constraint of naturalism. In: Lambert, M. J. (ed.), Bergin and Garfield’s handbook of psychotherapy and behaviour change (5th Ed.), 44–83. Hoboken, New Jersey: John Wiley & Sons, Inc.
Stiles, W. B. (1993). Quality Control in Qualitative Research. Clinical psychology Review, 13, 593–618. DOI: https://doi.org/10.1016/0272-7358(93)90048-Q
Stiles, W. B., & Shapiro, D. A. (1989). Abuse of the drug metaphor in psychotherapy process-outcome research. Clinical Psychology Review, 9(4), 521–543. DOI: https://doi.org/10.1016/0272-7358(89)90007-X
Stiles, W. B., Shapiro, D. A., & Harper, H. (1994). Finding the way from process to outcome: Blind alleys and unmarked trails. In: Russell, R. L. (ed.), Reassessing psychotherapy research, 65–113. New York, NY: Guilford Press.
Strupp, H. H., & Hadley, S. W. (1977). A tripartite model of mental health and therapeutic outcome: With specific reference to negative effects in psychotherapy. American Psychologist, 32, 187–196. DOI: https://doi.org/10.1037/0003-066X.32.3.187
Tschuschke, V., & Dies, R. R. (1994). Intensive analysis of therapeutic factors and outcome in long-term inpatient groups. International Journal of Group Psychotherapy, 44(2), 185–208. DOI: https://doi.org/10.1080/00207284.1994.11490742
Valkonen, J., Hanninen, V., & Lindfors, O. (2011). Outcomes of psychotherapy from the perspective of the users. Psychotherapy Research: Journal of the Society for Psychotherapy Research, 21(2), 227–240. DOI: https://doi.org/10.1080/10503307.2010.548346
Werbart, A., von Below, C., Brun, J., & Gunnarsdottir, H. (2015). “Spinning one’s wheels”: Nonimproved patients view their psychotherapy. Psychotherapy Research, 25(5), 546–64. DOI: https://doi.org/10.1080/10503307.2014.989291
Westen, D., Novotny, C. M., & Thompson-Brenner, H. (2004). The Empirical Status of Empirically Supported Psychotherapies: Assumptions, Findings, and Reporting in Controlled Clinical Trials. Psychological Bulletin, 130(4), 631–663. DOI: https://doi.org/10.1037/0033-2909.130.4.631