Observer-Reported Communication Ability (ORCA) Measure Scoring

Scoring of the ORCA measure requires a SAS macro or a scoring feature available through Pattern Health. The ORCA measure cannot be scored by hand.

For Use in Research

Scoring instructions and an accompanying SAS macro (including an R translation) are available for download below. Alternatively, Pattern’s scoring feature is available for licensing. Contact Us for more information about scoring using Pattern’s platform.

Enter your access code below to download scoring instructions and materials. If you do not have an access code, please Contact Us.

For Use in Healthcare

A free scoring feature is available through Pattern’s platform. Contact Us for more information.

Rationale for Scoring Approach

The scoring approach for the ORCA measure is different from many other clinical outcome assessments by design, and reflects the unique complexities of communication ability. The ORCA measure is scored using an IRT-based model to reflect the caregiver’s observations of an individual’s communication ability over the past 30 days via reported mastery of specific behaviors within 23 concepts reflecting expressive, receptive, and pragmatic types of communication. Appendix #1 provides a list of each of the 23 communication concepts and the representative items and item numbers from the ORCA measure.

For the three concepts that use only a single ORCA item (i.e., “Number of words”, “number of symbols”, and “turns in conversation”), they have five ordinal response categories with responses to more words, more symbols, or more turns in conversation, respectively, representing higher communication ability. For the other twenty concepts, the ORCA items associated with the concepts have three response options of “No or only once”, “Sometimes”, and “Yes, almost all the time”. In the far right column of the table in Appendix #1, we indicate a hierarchical ordering of communication ability level to differentiate which behaviors represent lower ability (level 0) versus higher ability (higher level numbers) within each concept. Each concept can have different ability levels; for example, “Seek attention” has 6 levels (0 to 5) and “Refuse object” has 4 levels (0 to 3). In addition, ability levels are not equivalent across concepts (e.g., a level 3 for the concept of “Greeting” does not equal a level 3 for the concept of “Making Choices”). The ability levels assigned to items within a concept were evidence-based and developed directly from concept elicitation interviews with caregivers and communication experts. Levels were additionally confirmed by a group of nine SLPs, including two investigators from Duke University.

Based on the caregiver’s answers to ORCA items, the individual receives an ability level “score” for each concept (e.g. “seeking attention”, “refusal”) based on the highest leveled behavior within the concept that the individual had ‘mastered’. Mastery meant that individuals with AS could perform the behavior frequently and consistently over the past 30 days, and was indicated by caregivers choosing the “Yes, almost all the time” response option.

Samejima’s graded item response theory (IRT) model was used to adjust for the differing ability levels among the concepts (and levels within a concept) by fitting item parameters at the concept level. IRT PRO (version 4.2) was used to estimate the parameters using marginal maximum likelihood estimation.

The sample used to calibrate the ORCA concepts were caregivers of individuals with Angelman syndrome. Based on responses from 249 caregivers, investigators estimated the IRT model parameters for the 23 concepts with the parameters provided in Appendix #2. The “a” parameter indicates how well the concept discriminates among individuals with different levels of communication ability. The “b” parameters indicate the levels of communication ability that are assessed by each concept. In other words, the IRT “b” parameters adjust for the “difficulty” of each ORCA concept, with more “difficult” concepts (larger “b” parameters) representing higher-level communication skills. The investigators also used the IRT model to estimate the ORCA measure total score for each individual. Based on a caregiver’s response to the 23 communication concepts included in the ORCA measure, an IRT score based on the expected a posteriori (EAP) algorithm was estimated. The EAP scores were standardized to a mean of 0 and standard deviation of 1 in the calibration sample of individuals with AS. The EAP scores were then transformed to a T-score metric with a mean of 50 and standard deviation of 10.

Interpreting ORCA Scores

The ORCA measure currently produces a single score that is an estimate of an individual’s overall level of communication ability. The ORCA T-scores were standardized based on the original calibration sample of individuals with AS to have a mean of 50 and standard deviation of 10. Higher ORCA T-scores reflect greater communication ability; the mastery of expressive, receptive, and pragmatic types of communication and higher vocabularies for verbal words and symbols on assistive devices. The ORCA T-score range is from 25.8 to 83.8.

The investigators have looked into creating sub-scores of communication by expressive, receptive, and pragmatic forms of communication, but the empirical evidence (from the original calibration sample) supports that there is an overall general factor that accounts for a majority of variance observed in the item response data. If scores are produced for each form (expressive, receptive, pragmatic), they are highly correlated. Options for scoring at the expressive, receptive, and pragmatic levels will be explored as more data is collected.