June 24, 2008, Stine Lindahl Jacobsen, PhD Student, Aalborg University. (stinelj@email.dk)
The Choice and Use of Assessment Tools in Research
It is an ongoing challenge to choose the right assessment tool or test for your study. Will the test show what you are looking for and does it apply to the specific design or method of the study? Has it gone through necessary validity and reliability test? Are you authorized to use it? Is it available in the native language of your population and is it the right age range? Are there specific clinical issues you need to be aware of in applying it? All these questions and many more needs to be answered before you choose your final assessment tool or test.
To demonstrate specific issues in choosing assessment tools or tests I will shortly present my study.
1) Study of parenting competences
As music therapist working at a family care center in Denmark for a period of 2 years problematic issues becomes clear. When a parent is suspected to neglect his/her child, social services are notified and their task is to examine interaction between parent and their child in order to help the family in the best possible way. The task includes finding resources, potentials, and weaknesses in the parent together with finding the parents appropriate and inappropriate interaction pattern. In short social services investigate the parenting competences of the parents.
This task is very time consuming. It is very complex to examine parental competences because there are many factors to consider. To observe the family without influencing interaction between parent and child is very difficult. To make parents feel secure so they can react in their normal way when they are observed is difficult. To get a nuanced picture in a short period of time is very difficult, and time is an important factor considering the neglected children’s special needs. It is even a bigger challenge to try and develop or increase parenting competences and my overall motivation and focus in this study is how music therapy can contribute to these difficult tasks.
The study therefore aims to investigate how music therapy assessment can inform on parenting competencies in cases of child neglect. As part of this investigation the study seeks to compare results from parents with neglected children with result from with parents with non-neglected children. It also wishes to examine whether music therapy treatment can increase or develop parenting competences over time and whether the assessment tool can measure this or not.
Main questions
- How can music therapy assessment inform about parenting competences in cases of emotionally neglected children?
- How can the results of this assessment be compared with results from a normative sample of parents with non-neglected children?
- Can music therapy treatment with a parent and his/her emotionally neglected child improve or develop better parenting competences over time and can the assessment tool measure this development or not?
- What is the value of the music therapy treatment for the emotionally neglected child? (This question and the needed tests to measure this will not be included in this paper)
Design
The design of the study is both a within subjects and between groups design. The study consist of 3 groups; parents and their neglected children receiving both assessment and treatment (G1), parents and their neglected children receiving only assessment (G2), and parents and their non-neglected children receiving only assessment (G3). The study seeks to both investigate the assessment tool and evaluate treatment outcome. Evaluation of outcome is of course also a part of the investigation of the assessment tool.
Families with neglected children recruited from a family care center will randomly be assigned to either G1 or G2. G1 is the experimental group, G2 is the control group, and G1 and G2 will both be receiving treatment as usual at the family care center. G3 is a normative sample and in investigating the assessment tool results from G3 will be compared with result from both G1 and G2.
As part of the assessment families with neglected children will fill out a selection of standardized tests.
| G1 5 families with neglected children |
G2 12 families with neglected children |
G3 50+ families with non-neglected childrenG3 50+ families with non-neglected children |
|
|---|---|---|---|
| Music therapy assessment (Pre) | X | X | X |
| Music therapy treatment | X | ||
| Standardized test (Pre) | X | X | |
| Music therapy assessment (Post) | X | X | |
| Standardized test (Post) | X | X |
Analysis
Every music therapy assessment session will be videoed and data is collected through observation of the interaction between parent and child. The families do different exercises and free improvisations together and their behavior and interaction are analyzed through both quantitative and qualitative methods. The quantitative analysis consists of frequency and duration of specific predefined events, while the qualitative analysis consists of descriptions and evaluations of the parents’ response to the child in relation to predefined categories based on literature.
2) Choosing assessment tools and tests
In my study on music therapy in assessment and treatment of parenting competencies I am developing an assessment tool and this increase the demands of validity and reliability of any tests I wish to compare my result with. In comparing the music therapy assessment results from parents and their neglected children and from a normative sample some questions on the tool’s validity will be answered. It will be evident whether the assessment tool can measure/show/inform on different parenting competencies in the two different groups of parents, however this doesn’t say anything about whether the results are accurate compared to other tests of parenting competences and whether the tool can show development over time. For this other detailed standardized tests or assessment tools on parenting competences are needed.
Parenting competences are complex to measure because it consists of many factors like empathy skills, social skills, stress management, emotional skills, autonomy skills etc. and the comparison test should therefore be as complex as a personality test with multiple axes and continuums in order to know whether the result from the music therapy assessment are accurate or not. Comparing observed behavior with self reported tests can give some problems because the parents might not answer truthfully or might not really know their true actions. A test filled out together with a psychologist, a test that needs to be analyzed by a psychologist is therefore more desirable than a “simple” self report test with a “simple” number outcome.
Of course a similar assessment test that consists of observed behavior would be even more desirable, but this sets high demands of time recourses.
In evaluating treatment outcome it is also necessary to find a test that has test/re-test reliability in order to measure changes over time.
So, I am looking for a comparison test on parenting competences that is both rich on detail and has test/re-test reliability. This type of test is difficult to find – especially in the native language of my population. I therefore have to consider applying multiple tests to the population trying to get all I need from more than one test. This however immediately sets demands of recourses and can become very time consuming.
In the following I will list different assessment tools or tests considered for my study trying to explain considerations for choosing or rejecting them.
Tracking down tests and assessment tools can in my experience be done in multiple ways. Almost all psychological publishing firms have a list of their available tests. You can also simply search the internet for tests or you can ask clinicians working with the population of your study. It can be difficult to know everything you need to know about the test from these simple lists or from information on the internet. Not many abstracts of the tests will tell you whether it has been tested for test/re-test reliability, and sometimes you need to contact the authors of the test tool in order to get the information you need. You can of course choose to buy the test to get the information you need but these tests can be quite expensive and sometimes you do not have the authorization to buy them at all.
In my study however I am so fortunate to have a psychologist who has offered to do tests I myself am not allowed to do. This is very helpful and I am very grateful for this.
In evaluating the tests you should try to find reviews of the test and not only rely on the author’s own evaluation of the assessment tool or test. More specifically you want to know about any psychometric validations together with any descriptions on clinical applicability.
3) Parents preference test (PPT)
The Parents’ Preference Test is a picture-based, multiple-choice test with images representing everyday family activities. The parent chooses his or hers preferred parenting style by looking at different family situation illustrated by pictures. The results of the test give a description of the overall preferred parenting style according to four dimensions; Energy (a continuum of active vs. passive), attention (a continuum of focus on self vs. focus on child), Insight/empathy (a continuum of thought vs. feeling), and regulation (a continuum of strict rules vs. situation-based rules).
Theoretically the test is based on Theodore Millons dimensional personality model. However the fourth dimension pain-pleasure being replaced with the regulation dimension on the rationale that parents do not operate according to pain or pleasure but according to the needs and desires of child and of the surrounding context (Westh 2006).
The test is Danish and developed by the experienced psychologist Finn Westh. 831 persons have participated in the trail and construction of the pictures. The test has been validated and revised 3 times with a total of 608 participants. According to the main psychological publisher in Denmark (Dansk Psykologisk Forlag) PPT has a high psychometric standard and is the only one of its kind.
Westh (2006) made multiple validations of the PPT. The face and content validity of the test have relied on multiple interviews with 50 persons on the content and constellation of the pictures together with ongoing revision of the pictures. This showed in the end a concordance of 80% stability – which means that the 80 % agree on how to understand the pictures. PPT has been compared to three other psychometric tests; WAIS-R, Rorschach, and NEO PI-R. The correlations between Rorschach and NEO PI-R are very modest (0,05-0,20) and this contributes to the construct validity of the test; the test really do seem to measure specific skills in the parent.
Rosenberg (2003) made a psychometric validity and reliability test of the PPT. The validation was based on Rasch models supplied by factor analysis. Rosenberg states trough thorough statistical tests with results from 142 participants that the PPT has a strong construct validity – it really measures what it wants to measure. However the test has been validated for test / retest reliability and because of the pictures the parents remember what they answered previously. Therefore this test cannot be used in a pre/ post design study.
I have chosen this test because it is very detailed in the description of the parent and I need to compare my own result with this type of data in order to know whether the results of the study are accurate or not. It also has a minimum risk of the parents portraying themselves in an unrealistic positive light because of the nonverbal nature of the pictures. The pictures are not value bound which enables the parent to choose their style of parenting according to how they think they ought to. The pictures are also described as having an anxiety-reducing effect.
According to Westh using the test results for diagnostic as well as therapeutic purposes, however, requires insight and experience. The psychologist who will be doing PPT is experienced in applying it and in working clinically with the fragile parents of neglected children.
4) Epstein Parenting Competencies Inventory (EPCI)
Epstein Parenting Competencies Inventory is a test made by the American Dr. Robert Epstein who is very experienced in both conducting such test and in working with the population. It is a self reporting questionnaire with 100 items where the parents have to answer statements according to a 5 point Likert Scale – agree vs. disagree being the two opposites. The results are divided into ten different skills area which are;
- Autonomy & Independence
- Behavior Management
- Education & Learning
- Healthy Lifestyle
- Life Skills
- Love & Affection
- Relationship Skills (towards partner)
- Religion & Spirituality
- Safety
- Stress Management
The parent can score up to 100 % in each area. Scores under 85% gives reasons to be concerned and scores below 65% seems to indicate the norm line. Especially skill area 1, 2, 6 and 10 seems relevant to the current study, but the length of the questionnaire might be too much for the fragile population of the current study. Area skill 8 and 7 might not apply to the population because most of them are single parents and in Denmark religion and spirituality hasn’t the same cultural meaning as in the States.
I contacted the author and he states that the test has been validated, but that it hasn’t been published yet. The test hasn’t however been tested for test retest reliability. It isn’t possible to do parts of the questionnaire and even though he kindly offered to do an online EPCI test of a Danish translated version I am forced to thankfully decline this offer. Even though EPCI seems highly relevant the lack of reviews and published validations together with possible clinical application problems result in a rejection of the test for the study.
5) The Parenting Stress Index (PSI)
The Parenting Stress Index (PSI) is a clinical and research self-report instrument (101 items) designed as a screening and diagnostic assessment technique to identify parent and child systems which are under stress, and in which deviant development of the child is likely to take place, or where dysfunctional parenting is likely to occur. It is a 5-point Likert-type continuum ranging from strongly agree to strongly disagree and has a fifth grade reading level. The PSI yields a total score, three domain scores, and 15 subscales. The domains measured are stresses related to child characteristics, parental characteristics, and situation and demographic factors. The age range for the child of the parent is 0-12.
The PSI has been empirically validated to predict observed parenting behavior and children’s current and future behavioral and emotional adjustment, not only in a variety of U.S. populations but also in a variety of international populations. The trans-cultural research has involved populations as diverse as Chinese, Portuguese, French Canadian, Italian, and Korean. These studies demonstrated comparable statistical characteristics to those reported in the PSI Manual, suggesting that the PSI is a robust diagnostic measure that maintains its validity with diverse non-English-speaking cultures. This ability to effectively survive translation and demonstrate its usefulness as a diagnostic tool with non-English-speaking populations suggests that it is likely to maintain its validity with a variety of different populations.
The following is based on a review of the PCRI done by Heinze and Grisso (1996). The normative sample consisted of 2633 parents and the majority of the children were under 5 years old. Alpha reliability coefficients measuring the internal consistency of the subscales, each domain and the total score were high (0.70-0.95). The temporal stability of the test was conducted in multiple test-retest reliability studies ranging from 0.55 to 0.96. The PSI showed significant correlations with multiple tests measuring the same construct.
Heinze and Grisso seem fairly pleased with the validity, generalizability, and reliability of the PSI. The test seems to be internally consistent and temporally stable. Applying this test to the current study however raises the question on how stress is related to parenting competences. The test can inform about the parents stress levels in different situations and areas but how is this linked with competences. A comparison with this test could give interpretive problems and therefore the test is rejected for this study.
6) Parent-Child Relationship Inventory (PCRI)
This self-report inventory with 78 items seeks to inform on how parents view the task of parenting and how they feel about their children. It is a 4-point Likert-type response format ranging from strongly agree to strongly disagree. It is designed for use with mothers or fathers of 3- to 15-year-old children and gives quantified description of the parent-child relationship. It identifies specific areas in which problems may occur. It covers seven distinct scales: Parental Support, Satisfaction with Parenting, Involvement, Communication, Limit Setting, Autonomy, and Role Orientation.
In addition, two validity scales are designed to alert on the possibility that the parent is responding inconsistently or portraying the parent-child relationship in an unrealistically positive light. The validity scales consists of a Social Desirability scale and a scale intended to measure the tendency to give inconsistent responses. The test requires a fourth-grade reading level and the test offers computer based interpretation of the data. PCRI items are appropriate for either parent and separate norms are provided for mothers and fathers. High scores indicate positive parenting characteristics and low scores indicate poor parenting skills.
The following is based on a review of the PCRI done by Heinze and Grisso (1996). The normative sample consisted of 1139 parents from separate places in the States. Based on the normative data highly educated parents tended to score higher on the PCRI than less educated parents. Young parents also tended to score lower on the PCRI. The coefficient alpha values for the subscales ranged from 0.70 to 0.88 and the one week retest reliability of 22 subjects ranged from 0.68 to 0.93 for various scales. The five month test retest reliability of 82 parents for the subscales ranges from 0.44 to 0.79. The PCRI showed significant correlations with subscales from the Personality Inventory for Children that where consistent and in the right direction.
Heinze and Grisso find the tests reliability and validity promising. The results suggest good internal consistency and temporal stability. However there is need of more research on the validity scales’ ability to detect defensive responders together with research on the clinical application of the test.
This test seems highly relevant and has a fairly good validity and reliability. I will have to search for newer reviews or studies of the PCRI but the real question is the language of the test. The PCRI doesn’t exist in Danish so a translation is needed. This complicates everything because any translation in theory should be tested as thoroughly as the original test. However this test seems to be the most adequate and applicable and I will most likely choose the test having to spend much time on translating it and involving other is this as well.
Outro
As the motivation for this study explains it is a time consuming and difficult task to evaluate parenting competences. I am therefore not surprised that detailed tests with high validity and reliability on this subject are hard to come by. My quest for these comparison tests and assessment tools have once again demonstrated to me that this particular area of investigating parenting competences is in need of an assessment tool that can be detailed in information and be used before and after treatment. A test that builds on the actual behavior and interaction between parent and child is very hard to come by even though it has another very valuable type of information to offer. My motivation for doing this study has therefore increased after this search for applicable comparison tests.
References
Heinze, M & Grisso, T(1996) Review of Instruments Assessing Parenting Competencies used in Child Custody Evaluations. Behavioral Sciences and the Law, Vol. 14, 293-313.
Rosenberg, Carsten (2003) Psykometrisk validering af PPT [Psycchometric validation of PPT]. In: I dialog med familien [In dialogue with the family]. Dansk Psykologisk Forlag, Denmark.
Westh, Finn (2006 ) I dialog med familien [In dialogue with the family]. Dansk Psykologisk Forlag, Denmark.
© 2008. Nordic Journal of Music Therapy. All right reserved. This page was last updated by Rune Rolvsjord June 24, 2008.