The process of certification, as outlined below, was developed by both the National Commission for Certifying Agencies (NCCA) and the International Organization for Standards (ISO) 17024. This is the standard applying to the accreditation of agencies certifying individuals in various fields. These accrediting agencies and the educational and psychometric specialists in the certification industry have validated these methods with decades of research. Therefore, the IBFN will use this process in the development of the certification examinations of the Board.
Step One: The Job Task Analysis (JTA)
The JTA is developed using a group of Subject Matter Experts (SMEs) who outline the content of the job and research supporting documentation to provide both broad categories and specific information that the practitioner in the field may need to know. This information includes knowledge or cognitive information such as neuroanatomy and physiology, metabolic information and developmental information. Additionally, a delineation of skills necessary for examination and treatment of patients is included. Then, combining these two types of competency provides the abilities of the practitioner in the actual use of knowledge and skills to diagnose and design treatment plans, evaluate outcomes and provide patient teaching.
From the content outlines, a survey is developed and administered to all practitioners in the field, in as far as possible. Each item on the survey receives two ratings: importance of the material and frequency the material is actually used in practice. From the survey data, a blueprint of the knowledge, skills and abilities of the “job” of the practitioner and the “tasks” needed to perform the job is defined. Certain mathematical decision rules determine the precise categories that will be covered on the examination. Try this as a rewrite and see if that is what you mean: Since an examination is be limited in length, all items cannot be covered on each exam, thus the need for decision rules.
A secondary addition to the survey method is an actual review in terms of a brief (one (1) or two (2) day) time and motion study of one or more clinics with practitioners in the field. This provides additional data in terms of ancillary personnel, tasks that can be delegated, patient/family teaching, actual time for various identified tasks, etc. The psychometrician takes both the survey data and the time and motion data and devises the final test blueprint.
The summary test blueprint is provided to all educational bodies providing education leading to the certification examination and published online for both candidates and the public.
Step Two: Test Item Development
Because of the complexity of the jobs represented by the credentials offered by the IBFN, a methodology of cases with items related to those cases has been chosen. This type of “testlet” examination requires the higher order thinking of the practitioner. Simple multiple choice questions require basic knowledge of a fact or an association of an idea to an answer. The case method requires the tester to synthesize the information provided in the case to actually determine diagnosis(es), treatment(s), patient/family education, prognoses and outcomes. It requires a higher level of critical thinking than is found in the typical multiple choice examination.
If the JTA and subsequent test blueprint indicate the need, both a written or cognitive test and a performance or skills test will be included. The test development committee consists of a group of SMEs who will write cases, item stems (basic portion of the test item), correct answers and distractors. Each item and correct answer will be referenced in the literature supporting the education in the profession. Each distractor will also be referenced to assure that it is, indeed, incorrect or “less correct” than the identified correct answer. The IBFN does not seek to “trick” the candidate in an unusual or ambiguous wording of items.
All of the cases and the items are categorized according to the test blueprint. Any additional items that overfill an indicated category may be retained for a later test or a different test form. Once the blueprint requirements have been met, the test is assembled and a proof reader who has worked with international populations will review the test. This assists in preventing items that have ambiguious meanings in various cultures. This is critical since the IBFN credentials professionals globally.
The items on the performance examination must also meet the requirements set by the test blueprint. These may be demonstrated in the standard healthcare Objective Structured Clinical Examination (OSCE) format or another similar mechanism that tests examination and treatment skills. Additionally, video simulations or videos of actual examination findings may be used to assure that the candidates know and can rate or grade the findings of an examination. Two additional performance examination potential categories are patient/family education/communication and report writing. The test development committee will work with the test development consultants to determine the best methods and the best items for any performance examination.
Once the examination items are developed, the test development committee or a subset of that committee arranges the examination. The cognitive cases may be randomly assigned in order, but the items applying to each case must remain associated with the case and not randomly assigned. Any non-case related items may be randomly ordered. The performance examination may have subsections that may be administered in any order but the ordering of the OSCE must be in a standard format.
Step Three: Pilot Testing
After the examination has been developed and proof read, it is administered to a group of SMEs for pilot testing. The psychometrician, after analyzing the pilot test data, will make recommendations regarding the rewording or removal of any specific items. The test administrator will make these changes and reintegrated the corrected items into the test.
Step Four: Test Administration
The test is then administered to registered candidates. The IBFN plans to administer the cognitive/written examination via Internet technology using testing centers worldwide.
Step Five: Initial Psychometric Analysis
The psychometrician receives the securely delivered data and conducts the comprehensive analysis. Analysis includes item difficulty, test difficulty and point biserial values as well as other statistics such as mean, median, standard deviation and standard error.
Step Six: Standard Setting
Standard setting or “cut score” setting is the methodology used to determine the actual raw score where candidates scoring above that score pass and candidates scoring below that score fail. Because credential testing is criterion referenced testing, an additional SME group gathers in a face to face meeting, preferably in person, not via webinar. This group takes the examination. Then the group undergoes specific training in terms of the definition of the minimally qualified candidate and the unqualified candidate. The psychometrician also provides training in understanding the basis statistics provided to the panel from the actual examination administration. The individual panelists are also provided with their own item by item performance. The panelists rate each item on the examination. If there is disparity among the panelist on any items, the facilitator and psychometrician work to assist the panelists in arriving at consensus.
This data is collected and given to the psychometrician. Any feedback from the panel regarding individual items is passed along to the next test development committee for integration into the following examination.
Step Seven: Final Psychometric Analysis
The psychometrician takes the data from the standard setting panel and statistically determines the standard setting score. Any items that were determined during initial psychometric analysis and by the standard setting panel as inefficient items may be discarded from the final score for the candidate. The psychometrician provides the Executive Director with the candidate numbers and score reports by blueprint category for each candidate. The Executive Director receives the summary statistics, the pass/fail rates and the entire analysis. The test analysis report is the final responsibility of the psychometrician for any given examination.
Step Eight: Score Reporting
The Executive Director submits the psychometrician’s report to the IBFN Board of Directors for approval. Once approved, the candidates are notified regarding examination performance including strengths and weaknesses as demonstrated on the examination. Those passing are provided information on their Fellow status, the proper use of the trademark, continuing education requirements and recertification requirements and timelines. Those who were unsuccessful are provided with remedial education recommendations and the policies and timelines for re-examination and any possible appeal mechanisms.
Step Nine: Reporting to the Public
Once the new Fellows have been identified, the Executive Director adds their names to the list of Fellows published on the IBFN website. The ED also publishes on the website the pass/fail rate and the summary report of the particular examination.
If changes in the field have not been significant, a JTA may be omitted for the next reiteration of the test development cycle. However, a new JTA is performed every five years and intermittently as needed. Each year, the other steps are performed as twenty (20) percent (%) of the exam is replaced on an annual basis to discourage cheating and memorization of the examination.