How to Do Operational ResearchGeorge Schmid, M.D., M.Sc. World Health Organization/Centers for Disease Control and Prevention [email protected] • • • • Informal Practical—if you want to hear about modeling (a legitimate, important part of OR), run out the door now The session is for you, and us You will forget 90% of what you hear today. So, we've given you materials on four CD-ROMs: – Designing HIV/AIDS Intervention Studies (and other materials) (Pop Council); – Training course materials in operations research (Pop Council) – Framework for Operations and Implementation Research in Health and Disease Control Programmes (and other materials) (Global Fund and WHO) – Other materials (note the FHI guide to qualitative research) Format Doing Operations and Implementation Research Research Techniques • Systematic data collection • • • Qualitative techniques Quantitative techniques Surveys, experiments, focus groups…. The OR Process • • • • • Problem identification Solution generation Solution testing Results dissemination Results utilization OR Requires Collaboration • Managers: responsible for decision parameters and desired outcome • Researchers: responsible for recommending and implementing research techniques . Taking My Problem Into Operational Research . 2.Q1. BMJ 2005 . 4. 3.What Proportion of Individuals Potentially Helped by Bed Nets are Using Them? 1. 2% 7% 14% 24% Morel CM et al. Taking My Problem Into Operational Research . You Must Recognize a Problem • Wanting to know • Wanting to act • Having the ability to act .First. 2. 5. Who Typically Identifies Problems for OR? (more than one answer possible) 1. 3. 4.Q2. The patient The programme staff The programme manager The District manager National staff . and initiative .Zambia Example • Problem: TB patients are not being tested for HIV • Observation: "TB Corners" where TB patients are seen are in open hallways • Intervention: Provide counselling with privacy Some issues simply need some common sense. Identifying Problems • The role of anyone who works in the programme! • But. the role of the manager • Identifying problems requires a team approach – Managers must be present in their programmes – Managers must have good relationships with their staff . primarily. .You've Identified a Potential Problem • Patients may not be taking their ART • People may not be using condoms • Patients may not be returning for follow-up visits • Your health care workers seem to have high rates of absenteeism • …. 4. 2. 3. Problem identification Solution generation Solution testing Results dissemination Results utilization Diagnostic step Intervention step . 5.The OR Process 1. others more formal .There are Lots of Ways to Approach OR • Some are simple. Simple? Complex? • Attendance at the ART clinic is low – Patients do not have transport • Injection drug users are not using clean needles – Clean needles are not available . to meet the objectives of what you need to do • But. . others more formal • Flexibility needed and desired.There are Lots of Ways to Approach OR • Some are simple. let's talk about more formal approaches now…. 11. 6. Monitor project Pre-test Quality control Stakeholder discussions Dissemination plan Disseminate results Document changes Monitor changes Consider further improvement WHO (TDR)/GF (Jane Kengeya-Kayondo. 9. 13. 10. 7.The Menu of the WHO/GF OR Book 1. 4. George Shakarishvili. Research team Determine Issues Develop proposal Ethical clearance Funding Budget Capacity building 8. 12. 5. 16. 2. 14. 3. 15. Serge Xueref) and Bill Brieger and Amy Ellis of Johns Hopkins School of Public Health . Three Phases to OR 1. Follow Through . Implementation 3. Planning 2. behavioural science…? – Statistics? – Economics (It is difficult to retroactively do the economics part of a study)? • • • • Structure Assign tasks. Organize the Research Group • What disciplines do you need? – Epidemiology. with deadlines Meet regularly Give credit!!!!!! – Who had the research idea? – Builds teamwork among departments/persons and avoids hard feelings – Pays off in the future .Planning 1. Determine the research questions (and objectives) • Be clear. be specific • Research question—What do you want to know? – Observation • "Our health care workers seem to have high rates of absenteeism" – Research question • "What is the rate of absenteeism among our nurses in the Kiev city hospital. and the reasons for it?" • Objective – Not: To study adherence – But: To determine rates of adherence among attendees at the XX hospital and reasons for adherence and nonadherence .Planning 2. How the results will be used . How the study will be conducted 3. What the research is about and why it is important 2. Develop a research proposal/protocol • A document that details: 1.Planning 3. Planning 3. quality assurance) Data collection instruments Plans for use of data Budget . data management. Develop a research proposal/protocol—"standard parts" • • – – Research objectives and questions Background Literature review Local context • • • • • Research team Methods (study design. study population. sampling details. how data will be collected. and should be done . Develop a research protocol • Does every "formal" OR project need a research protocol? – Yes – Makes you clearly state what you will do (and helps to improve your thinking) – Gives you the plan that you will follow. so everyone knows what will be done.Planning 3. Yes 2. No . Does Every OR Proposal need to go through Ethical Review? 1.Q3. Obtain ethical clearance .Planning 4. Give the patient too much information . Do not contain the contact information for the local ethics board 2. Cannot be understood by potential participants 3. All risks are not revealed 5. Do not mention the word "research" 4.Q4. What is the most common thing wrong with consent forms? 1. . "a proven track record" . Identify funding • A variety of sources • Requests for small amounts of money are almost always preferred • Follow closely the guidelines of funding agencies • Ensure projects are well-written • Having money from one agency for a project enhances the ability to get further funds from any agency.Planning 5. i.e. Establish a budget • Should be comprehensive. even if not a large budget—ensure you have costed everything. including dissemination costs • A good budget is a sign of careful planning • ? Include a financial officer .Planning 6. TA) – Long term training (building capacity) – Help staff gain skills to advance . Technical capability/capacity building • Ensure that staff know their roles and responsibilities • Ensure that staff have the training and capability to perform their jobs – Short term training (courses.Planning 7. Implementation 8. that includes: – All activities to be implemented. you cannot trust the results . Quality Assurance of the project • There must be a protocol! • There almost always is a work plan. before and during the study – Starting date. completion date. intermediate deadlines – Persons who are responsible for each activity – Milestones for each activity • An advisory group that meets regularly may assist in maintaining quality and the work plan • Monitoring should be part of regular team meetings • A quality assurance plan should be part of the protocol If you do not perform the study exactly as you said you would. Pre-test all research materials • Valid? (Do data collection instruments capture the desired information reliably?) • Reliable? (Do data collection instruments capture the desired information consistently?) .Implementation 9. to check understanding during study. develop a series of 5 or so questions—patient must answer 4 or 5 to enter study) . Pre-test all research materials • Questionnaires – Is the wording clear? Do people understand what you want? Do the questions ask what they are supposed to? – Test in 5-20 persons • Focus groups – Are the guides clear? – Hold one or two focus groups.Implementation 9. • Consent forms – Do people understand them? Language level? (have your child read it) (sometimes. likely) • Data entry into computers preferably as study is ongoing so errors by data collectors can be quickly caught and fixed • Double-entry of data? .Dissemination 10. supervisors review all data collection instruments daily for completeness and accuracy. have regular meetings of the data collectors (daily. Establish and maintain data management and QA • Forms should be easy to use • Observe data collectors regularly. to identify key findings and interpretations . to identify reasons for unusual findings • Subsequently.. Explore with stakeholders interpretation of results • Review preliminary results with the team early. within a week after data entry is complete. community leaders or project advisory group.Dissemination 11. e.g.. e.g. review data with the team and stakeholders. the information locally – Local meetings – Meetings with decision makers • Audience #2 ("external"). Those who can adapt the information for their own context. Develop dissemination plan and disseminate the results • Audience #1 ("internal"). Those who can benefit from. and act on. or use it for further research – National or international meetings – Journals • Create the right documents/strategy/medium for the right audience .Follow-through 12 & 13. interviewers.. Potential participants 2. All must be . The members of the research team developing the materials 5. Data management staff 4. focus group leaders 3. Who Need Not Be Part of the Pre-Test Evaluation of Materials? 1. Staff who use the materials.Q5. e.g. why not? . Document changes as a result of the research • Were the results implemented? If not.Dissemination 14. Monitor changes in the revised program • If the findings were valid. and appropriately implemented.Dissemination 15. but gets into the area of Translation Research…. then program changes should occur when the results are implemented • More of a monitoring (and evaluation) function than OR. . Consider ways of further improving the program .Dissemination 16. 3.Q6. 5. Implementation Sci 2007 . Internet/e-mail Meetings/conferences Colleagues Journal articles Short summaries Dobbins M et al. 2. 4. Which Format of Information Sharing Do Decision-makers Like Least (Industrialized Country) 1. 2. 6. Who Typically Acts on Identified Problems? 1. 5.Q7. 4. The patient The programme staff The programme manager The District manager The National staff Too few people act . 3. Q1. 2% 7% 14% 24% Morel CM et al.What Proportion of Individuals Potentially Helped by Bed Nets are Using Them? 1. 4. 3. BMJ 2005 . 2. 2. The patient The programme staff The programme manager The District manager National staff . Who Typically Identifies Problems for OR? (more than one answer possible) 1. 4. 3. 5.Q2. No . Does Every OR Proposal need to go through Ethical Review? 1. Yes 2.Q3. Do not contain the contact information for the local ethics board 2. Do not mention the word "research" 4. Give the patient too much information . What is the most common thing wrong with consent forms? 1. Cannot be understood by potential participants 3.Q4. All risks are not revealed 5. . focus group leaders 3. interviewers. Potential participants 2.g. Staff who use the materials.Q5. Data management staff 4. Who Need Not Be Part of the Pre-Test Evaluation of Materials? 1. e. The members of the research team developing the materials 5. All must be . Implementation Sci 2007 . 4.Q6. Which Format of Information Sharing Do Decision-makers Like Least (Industrialized Country) 1. 3. Internet/e-mail Meetings/conferences Colleagues Journal articles Short summaries Dobbins M et al. 2. 5. Who Typically Acts on Identified Problems? 1.Q7. 5. 4. 6. 2. 3. The patient The programme staff The programme manager The District manager The National staff Too few people act .