Modül 5 – YouthImpact+ Tool

Module 5

Menu index & Legend

Overview of Session 1

Session 1
Unit of learning outcome
Description
Resources and materials
Role of the trainer

Title: Basic Procedures in Data Collection

Format: online

Duration: 45 min

Specifics: learners will use their computers

At the end of the session the learner will have a know what data collection is, what it is used for and the basics of its methodology by practise. He also tries the new the new information on practical exercises.

This section first refers to the term of data collection itself, explains what it is used for and how to determine the objective of data collection.
After the first exercise, the learner is acknowledged with choosing the appropriate data collection method, how to chose respondents, learns how to create questionnaires, how to evaluate acquired data etc.

Data Collection & Analysis by UNICEF: https://www.youtube.com/watch?v=HFGVJJMDo4I

This Session as well as the whole Module 5 works independently, but if a trainer is available, he/she can present the module in detail, discuss with the learner(s) its content and help with exercises.

2. Introduction

Do you need to evaluate your efforts so far?

To do that you may first need some data – and also collect them. Data collection is a simple tool that you can use to make sense of your work, or to collect relevant data that you can build on in your work going forward and use as a negotiation tool.

This guide is designed for people who are keen to undertake data collection and not be put off by the apparent complexities.

The entire guide also contains various exercises on the topic. The exercises are indicated by the following icon:

3. The Tool (1)

HOW THE TOOL WORKS?

3. The Tool (2)

How does this Module relate to the Tool?

Source: The Yout Impact+ Tool,
https://www.youthimpactplus.eu/index.html

3. The Tool (3)

Where to apply the data you collect in the Tool?

Source: Exercise Workshop Guide v1.1 2018,
https://erasmusplus.org.uk/impact-and-evaluation.html

4. Initial Self-Assessment

What is your knowledge about data collection now?

Before you proceed, please take the initial test here:

https://forms.gle/HYgFrZNvpj2UEQSXA

The test contains 15 questions and only one answer is always correct.

Keep your results (e.g. write them down) so that you will be able to compare them with the same test at the end of Module 5.

We firmly hope you will find out how you have got better!

5.1.1. About data collection in general

Data collection is a process of individual and sequential steps.
It involves:

Determination of the objective of data collection
Choice of data collection method
Data collection
Processing of acquired data
Evaluation of processed data

5.1.2. Determination of the objective of data collection

The purpose of the data collection is determined with regard to discusses various whom the data will be presented:

Is the goal to confirm/refute that my work makes sense?
Is the result for me only, or will I present it to my supervisor/donor/stakeholders?
Do I want to use the data collection results as a reference?
You can learn more about the basics of data collection in this great video by UNICEF:

Exercise 1 (page 1)

Since you have chosen to take this course, it is likely that you are currently dealing with a problem that data collection could help you with.

In one sentence, try to write down the problem statement and define a data collection objective that would contribute to solving the problem.

Exercise 1 (page 2)

Now check if your goal meets the following and try to correct it if necessary:

The objective is specific and not vague
The completed target can be further worked with if necessary
It is relevant for further work

Data collection objectives can be achieved in real conditions

5.1.3. Choice of data collection method (1)

For routine work, it is usually not necessary to conduct data collection according to the rules of scientific research on a relevant sample of the target group.
In fact, it is a form of storytelling. You can choose between two methods: – quantitative and qualitative.
Quantitative is represented mostly by questionnaires and qualitative by in-depth interviews.
There are of course other methods of data collection, which you can read more about in the theoretical part I and II.

5.1.3. Choice of data collection method (2)

Which data collection method should you choose? The optimal choice is a combination of both – in-depth interviews, from which you will learn much more than from questionnaires.

The data collected is then used to construct a questionnaire where you can verify it and get comprehensive, conclusive data that is very easy to evaluate and interpret.

5.1.4. How to select respondents for in-depth interviews?

Choose at least 5 respondents for each target group (in case someone is not available, or cannot, or responds in one word).
You can tell that you have a well-executed interview by the fact that the answers start to repeat themselves.
Write down the respondents in a clear spreadsheet where you can keep a running tally of everything important.

5.1.5. What to ask?

Write down everything you or the data collector are interested in beforehand.
Organize the questions into thematic sections.
Put the simpler ones at the beginning and the more complex and demographic questions at the end of the interview.

Exercise 2 (page 1)

Try to prepare a set of questions for the goal you have already set, from which you can draw some conclusions after answering them.

Exercise 2 (page 2)

Check if the question sets do not contain:

Vague terms (use precise terms where possible, or explain the term properly and clearly).
Suggestive questions (instead of asking “why do you think my lecture was beneficial”, ask “what exactly do you find beneficial about the lecture”).
What-if questions (instead of asking “where would you go to the theatre”, ask “which theatre did you visit last time”).

5.1.6. Implementation of the in-depth interview

By phone vs. in person
Always try an in-depth interview in person first to confirm the clarity of the questions.
Always indicate how long the interview will take (up to 20 minutes is recommended for a telephone interview, up to an hour for a face-to-face interview).
It is not necessary to follow the order of questions in the interview, but it is good to go through all of them. If a question no longer makes sense for someone, feel free to cut it out.

Exercise 3 (page 1)

Rewrite a selected set of interview questions into a questionnaire form that the interviewee is able to answer unambiguously without needing an explanation of how to do so.

(You can use a scoring scale, single choice, multiple choice, verbal evaluation, etc.)

Exercise 3 (page 2)

Make sure that the questions:

…are unambiguous (especially watch out for questions like “What age category do you fall into?” 0-18, 18-65, or 65 and over – in this case, what should the 18 and 65 year old respondent indicate?)

…have a clear explanation of whether the grading scale is points, with 1 being the least, or whether it is a school-like grading scale, with 1 being the best grade

…are formulated so that all answer alternatives are taken into account (e.g. for the question “I considered the lecture: a) beneficial, b) not beneficial” it is good to add the option c) “I cannot say” – the respondent could be absent or for some other reason does not want to evaluate

5.1.7. Questionnaire survey

1.Distribution of the questionnaire in person in paper form – high return rate, respondent answers without time delay.

2.Distribution can also be by e-mail – however, the return rate is usually low.

-You can increase it by calling the respondent in advance and asking them to fill it out, or by filling it out with them right away.
-You can promise the respondents some certain benefits for completing the questionnaire.

3.Using on-line applications such as Mentimeter or Kahoot! can help bring higher return rate and more fun to evaluation.

5.1.8. Questionnaire creation (1)

The questionnaire can be created simply on paper, but also online.
There are many programs for creating a questionnaire – for example, Google Form. It is free and you only need to have a Google account.
Here’s a link to a video tutorial for creating a Google Account Questionnaire for beginners: https://www.youtube.com/watch?v=wNMdjVxAEYY&list=PLv9Pf9aNgemuvrcJiahvWKq9io27ZmCGd

Exercise 4

Try creating your own online questionnaire and inserting the questions you have prepared.

5.1.8. Questionnaire creation (2)

When designing a questionnaire, keep in mind that not all questions apply to everyone – learn to send such respondents one question further by branching.
Before you send the questionnaire, test it with your friends to see if it is easy to understand (pilot it).

5.1.9. Data collection and processing

Distribute the questionnaire and collect the completed ones back.
If you collected the questionnaire on paper, a transcript of the respondents’ answers is required. In most cases, it is sufficient to record the responses in a structured way in Excel and then create graphs of the responses.
The online questionnaire has the advantage of automatically collecting, storing and creating clear graphs from the data.

5.1.10. Evaluation of processed data

Read out the information from the graphs for which the data collection was carried out.
Make sure you can answer the original data collection objective in a relevant way.
Recommendation: suggest next steps, i.e. what you want to do with the newly acquired information.

5.1.11. Revision

1)Which procedure belongs to quantitative methods?

a)Focus group

b)Questionnaire survey

c)In-depth interview

2)What is the minimum number of respondents per focus group
for an interview?

a)5

b)10

c)50

Overview of Session 2

Session 2
Unit of learning outcome
Description
Resources and materials
Role of the trainer

Title: Example
of a data collection process

Format: online

Duration: 15 min

Specifics: learners will use their computers

At the end of the session the learner will be acknowledged with an example of a data collection process from the beginning to the end.

This session first shows on a real example how the researcher set a goal, selected an appropriate data collection method, performed qualitative data collection, quantitative data collection, processed the acquired data, evaluated them and formulated recommendation.
The session includes one practical exercise.

Designing a Questionnaire or Survey by Dr Nic’s Maths and Stats https://www.youtube.com/watch?v=FkX-t0Pgzzs

5.2. Example of a data collection process

The following section is an illustrative simplified example of the step-by-step data collection process.

5.2.1. Setting the goal

Put yourself in the shoes of a trainer of youth who is introducing modern teaching methods and would like to check whether he is heading in the right direction. For this verification he decided to collect data from his current students.

First, it sets the goal of data collection:

“Getting feedback from the participants on the training, on the basis of which a proposal for its improvement will be made.”

5.2.2. Selection of an appropriate data collection method

The trainer then decided to approach 5 participants for an in-depth interview.

His questions were prepared in advance as follows:

Area 1 – questions focusing on what should be retained in teaching.
Area 2 – questions focusing on what should be deleted.
Area 3 – suggestions of what should be in the lesson but has not been yet.

5.2.3. Qualitative data collection

The interviews were conducted in person and the answers of the respondents were recorded and then transcribed by the lecturer. In particular, the following was repeated in the responses:

It makes sense to the respondents that the lessons should be longer in order to get into sufficient depth of the topics discussed.
Respondents rated the following didactic techniques as particularly beneficial:

1) Diamond model

2) Post-it method

3) Discussion with an outside collaborator

5.2.4. Quantitative data collection

From the responses obtained, the lecturer then constructed a questionnaire to obtain conclusive data on a larger sample for appropriate interpretation.

The questionnaire was built in Google Form:

drive.google.com/drive. -> + Add -> Google Forms

Question 2 was designed as a multiple choice question.

To complete the form, click on the “Submit” box and choose a way to send it to your respondees.

How to send out your form?

If you share Google forms with a:

Personal account: Any responder with the link can open the form.

Workspace account: You can limit your audience to your organization or make the form accessible to anyone. Learn more.

After you create a form, you can send it to people using email or social media or even embed it into a webpage.

5.2.5. Processing of acquired data

Google automatically received a total of 21 responses from respondents and created the following graphs:

Exercise 5 (page 1)

Try to interpret the results of the processed data and suggest recommendations on what the lecturer should do next.

Exercise 5 (page 2)

Make sure that:

From the interpretation, the purpose of the data collection can be verified.
In the interpretation you take into account maximum, average and minimum answers.
The interpretation of the result is unambiguous and clear.
The recommendation is in line with the objective of data collection.

5.2.6. Evaluation of processed data

From the charts above, it can be seen that the majority of respondents consider the length of teaching to be insufficient.

The Diamond Model, the Post-It Method and the Discussion with an outside collaborator are considered by respondents to be the most beneficial teaching methods. Brainstorming came in a close second.

The other methods were less frequently rated as beneficial by respondents, but no one found any of the methods to be unhelpful.

5.2.7. Recommendations

The instructor should push for extending the class and continue using the Diamond Model, the Post-It Method, and the Discussion with an outside collaborator. Alternatively, then stop giving lectures and the Spider Method unless absolutely necessary for understanding the content of the classes.

5.2.8. Revision

1.Which question is correctly worded?

a) Do you find the lessons beneficial and fun?

b) On a scale of 1 to 10, with 10 being the most beneficial, indicate how beneficial you rate the lecture.

c) Indicate on a five-point scale how much you rate the lectures as fun.

2.What is good to remember when evaluating data?

a) Whether the conclusion is consistent with the objective of the data collection

b) Whether the results are clearly formulated

c) Both of the above.

Overview of Session 3

Session 3
Unit of learning outcome
Description
Resources and materials
Role of the trainer

Title: Advanced Data Acquisition:
Theoretical Background

Format: online

Duration: 15 min

Specifics: learners will use their computers

At the end of the session the learner will have a more complete picture of acquiring data for evaluation and what to consider when choosing the right method.

A more theoretical part about choosing an appropriate research method, considering e.g. the purpose, subject, scope and type of evaluation, evaluation criteria and evaluation questions.

n/a

5.3.1. How to choose an appropriate research method? (1)

Research methods must fit appropriately into the design and plan of the evaluation. To make the right choice, consider whether the methods are relevant with respect to:

Purpose
Subject
Scope and type of evaluation
Evaluation criteria
Evaluation questions

Will these methods provide you with the information you need to answer the evaluation questions?

5.3.1. How to choose an appropriate research method? (2)

Data sources you plan to use to gather information

Will it be appropriate to provide information about the groups that will participate in the evaluation research?

Characteristics of respondents

Do the methods take into account the size of the group, its perceptual abilities, communication skills, health status, etc.?

5.3.1. How to choose an appropriate research method? (3)

Circumstances of data collection

Will all the necessary data and respondents be available at any given time? Will the chosen method be appropriate for the data collection site?

Resources you have access to?

Does the method require the availability of qualified or independent researchers and other resources (organisational, technical, financial and time)? Will you be able to use these methods independently? Do your resources allow their use?

5.3.1. How to choose an appropriate research method? (4)

After answering the above questions, you move on to choosing the appropriate research method. You can choose from quantitative and qualitative research methods. Each has its advantages and disadvantages, and not both can be applied everywhere.

The following two sections take you through the characteristics of qualitative and quantitative research methods in more detail.

5.3.2. Revision

1. Which procedure belongs to qualitative methods?

a)Analysis

b)Questionnaire survey

c)In-depth interview

2. In particular, how does quantitative data collection differ from qualitative data collection?

a)Quantitative data collection is conducted on a smaller number of participants than qualitative data collection.

b)Qualitative data collection is conducted on a higher number of participants than quantitative data collection.

c)Qualitative data collection, as opposed to quantitative, goes more in depth into the issue under study.

Overview of Session 4

Session 4
Unit of learning outcome
Description
Resources and materials
Role of the trainer

Title: Qualitative Research Methods

Format: online

Duration: 45 min

Specifics: learners will use their computers

At the end of the session the learner will have a more complete picture of acquiring data for evaluation, more specifically, about Qualitative Research method and its uses.

Theoretical Part I – Qualitative Methods of Data Collection. This session discusses: Secondary data analysis, individual in-depth interviews, focus group interviews, observations and case studies.
The session also includes two practical exercises.

Benefits and Limitations of Structured, Semi-Structured, and Unstructured Interviews by Bernard Bull
https://www.youtube.com/watch?v=LDiEX8UHnYw

5.4. Qualitative research methods

Qualitative research methods include:

Secondary data analysis
Individual in-depth interviews
Focus Group Interviews
Observations
Case studies

5.4.1. Secondary data analysis

Secondary data analysis uses data that has been collected by someone else for some other purpose.
Data was not collected to answer specific research questions and instead was collected for other purposes.
Thus, the same dataset can be a primary dataset for one researcher and a secondary dataset for another.

5.4.2. Individual in-depth interview

The individual interview takes the form of a direct conversation between interviewer and respondent, which usually takes place according to a specific script.
Interviewing allows you to obtain extensive, insightful and detailed information, to learn about the interviewee‘s views, experiences, interpretations and motives for behaviour, to explore the facts from the interviewee‘s perspective and to better understand their views.
To structure or not to structure your interview? Learn how Bernard Bull perceives this issue:

5.4.3. Group Interview – Focus Group

A Focus Group is a conversation of about 6–8 people led by a moderator who assigns the group topics for discussion and directs its course.
Participants are selected according to the specific assumptions set by the researcher and their knowledge of the topic under study.
A deeper understanding of the problem is thus conveyed. It is possible to find out how people think about an issue, what opinions they have, what influences their opinion, etc.
The degree of this influence, or the number of people who hold a particular opinion, cannot be derived from this type of research, nor is that its purpose.

Exercise 6 (page 1)

Can you identify the main difference between an individual in-depth interview and a group interview?

Exercise 6 (page 2)

The main differences between an individual in-depth interview (IHR) and a group interview (FG)

The FG is a moderated discussion of a mix of interest groups that interact and their responses go more broadly.
The IHR provides highly detailed information.
As the name implies – the difference is in the number of participants. IHR – one participant, FG – 6–8 participants.

5.4.4. Observations

This method is based on careful observation and listening to the objects and situations under study.
Observation can be participatory, semi-participatory or non-participatory, depending on the level of involvement of the researcher, who may function as an active participant in the events they observe or as an external, non-participant observer.
Observations can be conducted in an overt, semi-overt or covert manner, i.e. the participants in the event know that they are being observed, or only selected people (e.g. the trainer and/or training organiser) know about it, or only the observer knows about it.

5.4.5. Case study

It is an in-depth analysis of the issue under study using information from various sources collected by different methods.
Findings can be presented in narrative form.
The “case” analysed can be a person, a group of persons, a specific activity, a project or a group of projects.

Exercise 7 (page 1)

For what purpose do you think the case study application is suitable? Try to think of at least 3.

Exercise 7 (page 2)

Purpose of using case studies

Thoroughly know and understand the phenomenon along with its context, causes and consequences.
Illustrate a specific problem using a realistic example with a detailed description.
Generate hypotheses for further research.
To present and analyse best/worst practices and show what is worth doing and what should not be done.

5.4.6. Revision

1. The aim of the Focus Group is:

a) Find out the level of influence, or the number of people who hold a particular opinion.

b) Determine the objective of subsequent data collection.

c) To understand the problem under study in more depth.

2. The use of case studies is NOT intended to:

a) Thoroughly know and understand the phenomenon along with its context, causes and consequences.

b) Illustrate a specific problem using a realistic example with a detailed description.

c) Gain extensive, insightful and detailed information, learn about opinions, experiences, interpretations and motives for behaviour.

Overview of Session 5

Session 5
Unit of learning outcome
Description
Resources and materials
Role of the trainer

Title: Quantitative Research Methods

Format: online

Duration: 45 min

Specifics: learners will use their computers

At the end of the session the learner will have a more complete picture of acquiring data for evaluation, more specifically, about Quantitative Research method and its uses.

Theoretical Part II – Quantitative Methods of Data Collection. More specifically, this session discusses various kinds of surveys and their uses.
The session also includes one practical exercise.

Writing Good Survey Questions – Statistics Help by Dr Nic’s Maths and Stats
https://www.youtube.com/watch?v=n34OnLnKzIg

5.5. Quantitative research methods

They represent a standardized method of measurement, which allows the collection and calculation of quantitative data in a uniform manner, as well as their statistical analysis.

Thus, these are methods that are carried out on a larger number of respondents.

They are divided as follows:

Survey conducted without the interviewer present
Interviews conducted with the support of an interviewer

5.5.1. Survey conducted without the interviewer present

Its forms are as follows:

a survey in the form of a printed questionnaire filled in by the respondent,
computer-based online interview/survey,
central survey (simultaneous survey of all respondents).

In self-administered surveys, respondents read and mark their answers on the questionnaire themselves.

5.5.2. Interviews conducted with the support of an interviewer

The interviews are conducted by trained interviewers who read the questions from the questionnaire to the respondents and write down the answers obtained. The following techniques exist for this type of research:

PAPI – responses recorded on a printed questionnaire (Pen and Paper Interviewing),
CAPI – answers recorded on a computer (Computer Assisted Personal Interviewing),
CATI – Computer Assisted Telephone Interviewing.

5.5.3. Personal interviews using paper and computer questionnaires

Both of these techniques are field-based and are implemented based on direct contact between the respondent and the interviewer using a paper version of the questionnaire (PAPI) or an electronic version displayed on a laptop or tablet (CAPI).
Interviewers read the questions contained in the questionnaire and then mark the answers provided by the respondent.

5.5.4. Computer-assisted telephone interview (CATI)

The interviewer reads the questions displayed on the computer screen over the phone and marks the answers received in an electronic questionnaire on the computer.
This form is suitable for exploring experienced views and attitudes using questions that do not require extended reflection time due to the short duration of the interview (10-15 minutes maximum) and the specific way in which the information is transmitted and received (which does not allow reading the questions several times at one’s own pace).

Exercise 8 (page 1)

List the advantages and disadvantages of quantitative research methods:

Advantages and Disadvantages of Quantitative Research Methods

Advantages

fast and straightforward data collection
easy generalisation of results
the data collected is accurate, numerical and easily verifiable
the results are relatively independent of the researcher

Disadvantages

research does not take into account local specificities
doesn’t come with anything new
can only confirm or refute already established theories

5.5.5. Revision

1. In the CATI survey:

a) Responses are recorded into a printed questionnaire

b) Answers are recorded on a computer

c) There are realised computer-assisted telephone interviews

2. What is the correct data collection process?

a) Data collection – evaluation – processing

b) Determination of data collection objective – selection of appropriate data collection method – data collection – processing of responses – evaluation of processed data

c) Determining the vision – Determining the data collection objective – Determining the sub-objectives – Selecting the appropriate data collection method – Data collection – Data verification – Data collection evaluation

Overview of Session 6

Session 6
Unit of learning outcome
Description
Resources and materials
Role of the trainer

Title: Self-Assessment

Format: online

Duration: 15 min

Specifics: learners will use their computers

At the end of this session the learner will assess what he/she has learned form the previous 5 sessions.

This session includes a multiple-choice test comprising altogether of 15 questions.

n/a

This Session as well as the whole Module 5 works independently, but if a trainer is available, he/she can discuss with the learner(s) the correct answers to the questions of the test.

Final Self-Assessment (15 questions)

The last section of this guide is dedicated to checking that you have got the gist of the data collection right.

The test contains again 15 questions and only one answer is always correct.

If you kept your answers from the initial test, you can compare them with your results from the final test and find out about your development.

You can the link to the test here: https://forms.gle/ccRijkzmofVvUjdP7

Referencing

Bartosiewicz-Niziolek, M., Nalecz, S., Penza-Gabler, Z. and Pintera, E. (2020) Příručka pro evaluaci zaměstnávání mladých lidí. Available at: http://youth-impact.eu/brochures/employment_toolkit/EMPLOYMENT_TOOLKIT_CZ_web.pdf (Accessed: 25 January 2023)
Hendl, J. (2005) Kvalitativní výzkum – základní metody a aplikace. Prague: Portál.
Kalivodová, K. and Brabec, M. (2022) Obsahová strategie III: Jak na persony. Available at: https://www.seduo.cz/obsahova-strategie-iii-jak-na-persony (Accessed: 19 January 2023)
Travis, D. and Hodgson, P. (2019) Think like a UX researcher: How to Observe Users, Influence Design, and Shape Business Strategy. CRC Press.