Experiments in consumer finance

The use of experiments in consumer finance research grows out of behavioural psychology and experimental economics. Consumer finance experiments reflect the concerns of both those disciplines, especially regarding how people make choices and whether financial behaviours fit with standard economic theory.

An important precursor to consumer finance experiments was the development of modern game theory, the study of strategic decision-making, by John von Neumann and Oskar Morgenstern in the 1930s. Game theory lent itself more readily to experimentation than theoretical microeconomics or macroeconomics because it was concerned with how people make choices under specific scenarios.

Experimental researchers try to model the different ways in which people make decisions. These models can help with things like structuring financial literacy programs, identifying social sectors that might be at risk of getting into debt, and providing people with better information so that they can make better choices. Experimental methods are also the cornerstone of evaluations, which provide a way to rigorously assess the success of a particular program or product.

What is Experiments

  • Quantitative
  • Conduct experiments under controlled conditions
  • Data is collected in a lab

Experiments involve testing hypothesis under controlled conditions through the manipulation of key variables. They can take the form of lab experiments, which are carried out under controlled conditions, or field experiments, which take place in a context that is largely natural.

In consumer finance research, lab experiments are often concerned with testing the psychological components of consumer finance, especially risk-taking behaviour and how people make choices. Field experiments generally involve running interventions or treatments on people as they live their daily lives. It is more difficult to control experimental conditions in field experiments, but they can provide a more realistic picture of behaviour than lab experiments.

Lab experiments

In lab experiments, researcher(s) first devise a hypothesis, then they design an experiment to test it. They identify at least one independent variable (an event, e.g., opening a bank account) and one dependent variable (measurable effects, e.g. saving more money).

Lab experiments take place in a location that is under the control of the researchers. This is normally a laboratory or room in the researcher's institution or workplace, but lab experiments can be carried out anywhere the researchers can gain a reasonable degree of control over the environment.

For example, a café or park would probably not be suitable for a lab experiment, but an empty room in a school, house, or public building might be sufficient. This is important because it means that researchers are able to travel to undertake lab experiments, just as with field experiments and natural experiments. Experimental research is therefore not always limited to recruiting participants who they can pay to travel to the lab.

Field experiments

Field experiments involve real-life testing of a hypothesis or intervention. Like lab experiments, field experiments are about testing and measuring behaviour. But unlike lab experiments, which take place under controlled conditions, field experiments are carried out under everyday circumstances.

In the field of socio-economic development, a particular kind of field experiment is used widely: the "randomized controlled trial" (RCT). RCTs are valuable because they can reliably test the effectiveness of consumer finance initiatives, especially in microfinance.

A standard method to carry out an RCT is to plan an intervention in a particular place, such as lending money to women in a particular town, and set up a control group in another town. At the beginning of the evaluation, both groups will be surveyed. The test group will be given the intervention, but the control group will not. At the end of the intervention, both groups are surveyed again. The results are analyzed to show the effects of receiving or not receiving the intervention.

For example, a field experiment in Malawi, conducted by Xavier Giné and Dean Yang, tested whether the provision of insurance induces farmers to take out loans. The researchers selected a sample of 800 farmers and offered them credit to buy high-yielding seed. Half of the farmers were required to purchase insurance to receive this credit. They found that farmers were less likely to take up credit if they were offered insurance with their loan. These kinds of experiments are valuable because they involve people making real-life decisions.

A variation on the field experiment is the natural experiment. These resemble field experiments, but there is no intervention. Researchers simply measure the things that people are already doing. One natural experiment in India used loan repayment data to investigate the optimal structure of a microfinance loan. They comparing people who had repaid individual loans in full (the "treatment" group) with people who had an ongoing individual liability loan, but who would eventually convert to group liability due to a change in policy in the lending institution (the "control" group). By watching how customers changed their borrowing practices as their loan type transitioned, they were able to infer which loan structure worked the best.

Strengths

Permit control of variables

Because experiments take place under controlled conditions, normally in the researcher's institution or place of work, it is usually possible to limit the number of variables that impact the experiment. This is particularly true for lab experiments. It can be difficult to control for variables in field experiments since they take place in "natural" settings.

This means it is possible to make accurate measurements and identify "cause and effect" relationships can be identified (e.g., that opening a bank account leads to greater savings). These kinds of observations are generally not possible with non-experimental methods because the effect (in this case, increased savings) could be caused by any number of factors.

However, it should be noted that this ability is dependent upon solid research design and execution. It is often difficult to isolate variables and limit external influences in a study.

Experiments are replicable

Experiments are replicable when they operate under controlled conditions, use limited variables, and allocate participants randomly to test groups and control groups. Again, this is more true for lab experiments than for field experiments, since real-life conditions can change rapidly and make repeat experiments impossible.

This means that is possible for other researchers to confirm or challenge a study's results. It also means that studies can be readily compared across different groups.

These features contribute to the development of general models and theories (such as the effects of asymmetric information on decision-making), the behaviour of specific populations (such as stock market use by elderly investors), and dynamic inconsistency (how people's preferences change over time).

Nevertheless, researchers need to be aware that, while it is relatively straightforward to replicate experiments, the results can differ significantly depending upon sample choices. This can limit the "external validity" of an experiment, the generalisation of the results to other situations and other people.

RCTs demonstrate impact

An advantage of randomised controlled trials (RCTs, a kind of field experiment) is that they can be used to demonstrate whether an intervention does or does not have the hoped-for impact. This is why they are used extensively for the purpose of evaluating programs carried out with people in the area of socio-economic development.

In fact, David Roodman, in his book Due Diligence: An Impertinent Inquiry Into Microfinance, argues that randomised trials are the best way to test whether microfinance programs work. Monitoring and evaluation is a large and important field with an entire methodology of its own, and people who are interested in developing their skills in this area have a wide range of materials and courses to choose from.

Limitations

Design flaws can invalidate experiments

The effectiveness of experiments is dependent on a solid research design. Confounding effects or confounding variables are variables that the experimenter failed to control and which compromise the validity of the experiment. This is true for all kinds of experiments, whether they take place in a laboratory or in the field.

While design flaws are a problem in all research, qualitative methods tend to be more forgiving when problems arise. For example, if analysis of a set of interviews show that a crucial question has been omitted, researchers may be able to make further inquiries to fill the gap in their knowledge by asking participants directly for clarification. This is not generally possible with experiments; for reasons of validity, the entire experiment may need to be run again if an error or omission occurs.

Results may not reflect real life behaviours

The artificial nature of experiments can produce results that are unlikely to occur in real life. This can even be true of field experiments, even though they take place in real-life settings. It is not the case for natural experiments since they are by definition the study of real-life behaviours.

One critical reason why experimental behaviour may not reflect real-life behaviour is that experimental subjects may be conscious that they are being watched, and this may make them more likely to follow moral norms or make more rational decisions than in real life. This is known as the "Hawthorne Effect."

The Hawthorne Effect changes depending upon whether your participants are interacting with each other or not. Say that each of your participants completes an experiment alone, entering responses anonymously on a computer. Their answers may be affected by what they think the researchers are looking for, but they are unlikely to be directly affected by peers since their fellow participants are not observing their choices.

In contrast, experiments that require participants to interact with each other give rise to a number of methodological problems that have been closely observed and are well understood. In cases where participants are anonymously interacting with other participants, it is often observed that decisions can be anti-social: people will often act for their own benefit, not for the benefit of the group.

In cases where interactions between participants are not anonymous, the same experiment can produce very different results, as people are often more likely to cooperate and behave generously when they have to interact directly. Overall, experimenters find that repeated interactions cause subjects to eventually start cooperating, and that their cooperation increases mutual benefit. This is also we observe in the real world.

The degree to which the experiment does or does not reflect real life behaviours can be mitigated to some extent by a careful design that considers these kinds of influences.

It can be difficult to form a representative sample

Sometimes it is difficult to recruit participants that are representative of a sample of the population under scrutiny. This is equally true for lab experiments and field experiments.

This is partly because the resources required to run lab experiments are often limited, but also because it can be difficult to persuade people to take part in such studies, especially when they are required to travel to the site of the experiment.

One way of lessening recruitment issues is to offer monetary awards, but these can interfere with representativeness. Some behavioural economics experiments use financial incentives to attract participants, such as through playing games with real money. Students are often used because $1 usually means more to a student than to a someone on a stable income, and so experiments can be run for less money. However, students sometimes access the experiment several times to earn more money, even though this is against the rules. This undermines the fundamental assumption of the experiment and reduces its representativeness.

Another common method of recruiting subjects is to give study credit to college students who take part. While this is a great way to gain the required number of participants, it does not solve the problem of representativeness. This is particularly the case for experiments carried out with so-called WEIRD subjects (Western, Educated, and from Industrialized, Rich, and Democratic countries). However, using a control group can eliminate this bias. If participants are distributed randomly between each group, the differences between results from each group will have nothing to do with them being students.

While moving a lab to the field can help offset some of these concerns, anyone wishing to perform a lab experiment is advised to read up on the many ways that biases can be introduced in real life settings.

Case Study 1 — Understanding Risk Preferences and Time Preferences

Risk and time are important topics in consumer finance. Whether people are risk-takers, risk-averse, or loss-averse impacts all kinds of decisions, including taking loans, buying insurance, and making investments. Consideration of time frames is just as important: financial planning involves thinking ahead, and how people perceive time is crucial for making good plans.

Less discussed is the fact that risk and time are intertwined. For researchers wanting to understand financial behaviour, it is crucial to be aware of how time can affect people's judgement of risk.

Most studies of financial decision making over time, including prospect theory, claim that people are so biased towards the present that they will make decisions that are counter-productive in the longer term.

For example, say you are offered a choice between $100 now or $120 one week from now. The latter choice is generally the most rational, but many people choose to take the money now rather than wait. Why might this be the case? One reason why people make this decision might be that their assessment of their current needs and desires outweighs their assessment of their future needs and desires. This is called "present bias."

The most famous experiment of this kind was the Stanford marshmallow experiment into delayed gratification in the late 1960s and early 1970s. In this experiment, researchers gave children marshmallows and told them that they could eat it straight away, but if they waited for 15 minutes they would receive two marshmallows. The researchers then left the room. A minority ate their marshmallow immediately. In follow-up studies, the researchers found that children who waited had better life outcomes.

However, the economists James Andreoni and Charles Sprenger ran a series of lab experiments that contests this finding. They present their results in an article called "Risk Preferences are Not Time Preferences" (2012). In this article, they give an alternate explanation for why people might choose to take a smaller benefit now rather than a larger benefit in the near future.

Methodology

Andreoni and Sprenger ran experiments with 80 undergraduate students at the University of California, San Diego. Students participated in four experiments, which took an hour each. The researchers used a method called "convex time budgets" (CTBs) in which participants allocated a budget of tokens towards receiving an early payment (in 7 days' time) and a later payment (in 28 or 56 days' time).

The researchers varied the probability of payments being delivered and the interest paid on later payments. Participants had to choose between money sooner (to be delivered to them in a week) or money later (to be delivered to them in either 28 or 56 days). Students also received a basic participation payment, of which half was delivered in the first payment and half in the second payment. Andreoni and Sprenger explain:

For all payments involving uncertainty, a ten-sided die was rolled immediately after all decisions were made to determine whether the payments would be sent. Hence, p1 and p2 were immediately known, independent, and subjects were told that different random numbers would determine their sooner and later payments. (2012, p.9)

An important part of research design was to minimise uncertainty based on confounding variables, such as whether a payment would accidentally go missing. To achieve this, experimental participants were chosen from among students living on campus who had 24-hour access to locked, personal mailboxes in their dorms.

The researchers took care to explain the process of payment delivery thoroughly so that students would be confident that they would receive their payments. In fact, a companion survey showed that students had 100% confidence that their payments would be delivered. So, it is reasonable to assume that confounding variables were limited, and the decisions that the students made during the experiment were not affected by extraneous influences.

A major advantage of this design is that it allowed the researchers to test whether students made decisions based on risk or time: that is, were they failing to delay gratification, or were they taking risk into account?

Findings

Andreoni and Sprenger note that, according to discounted expected utility (DEU) models, participants should allocate their money according to relative risk, distributing the payment between the two delivery times.

However, they found that their participants only behaved in this way under conditions of uncertainty. For example, when two options have the same degree of uncertainty (for example, a 50% chance of Payment 1 being delivered and a 50% chance of Payment 2 being delivered), then participants would allocate their payment between these two events.

When conditions were certain, participants behaved differently. In fact, "85 percent of subjects violate common ratio predictions and do so in more than 80 percent of opportunities" (p.16-17). There was little consistency in how participants allocated the delivery of money under conditions of certainty, and they did not seem to prefer sooner payments.

Instead, it appears that people were responding to changing levels of risk. Andreoni and Sprenger point out that, for most of us, the present is certain because it is already happening, while the future is risky because it is difficult to say what will happen. They explain,

Allais (1953, p. 530) argued that when two options are far from certain, individuals act effectively as expected utility maximizers, while when one option is certain and another is uncertain a "disproportionate preference" for certainty prevails. This intuition may help to explain the frequent experimental finding of present-biased preferences when using monetary rewards (Frederick, Loewenstein, and O'Donoghue 2002). That is, perhaps certainty, not intrinsic temptation, may be leading present payments to be disproportionately preferred. (p.26)

Hence people may not be biased towards the present at all, but instead risk averse.

Applications

This case study has valuable implications for experimental design. In experiments, if you don't control for the fact that the future looks riskier than the present, then people will make decisions that appear to be "present-biased" when they are actually making a risk-averse decision.

For example, say you are one of the children in the famous Stanford marshmallow experiment. How do you know that the researchers will really give you another marshmallow? Similarly, if you invest money, how do you know that it will pay off? The future is uncertain and anything could happen: a financial crisis may wipe out your investment or the marshmallow supply. People who are risk-averse may decide that it is better to take an immediate reward than to depend upon a bigger reward in the future.

Experiments like these have also been valuable in testing the validity of economic models. They have clear real-life implications in consumer finance, such as for understanding how people will be affected by time considerations, the risk of receiving or not receiving a payment, and the effect of interest rates.

Ethics

Many of the ethical issues that arise in lab experiments are the same as in all research involving human subjects. Psychological harm is the most common type of potential harm in non-medical research, that is, creating situations that lead to embarrassment or anxiety. Researchers can help to reduce harm by providing sufficient details of the study and giving participants the option to skip questions they are not comfortable with or to leave the study altogether.

Issues can also arise from the objectification of research participants, that is, treating them as merely research material rather than as human beings. Maintenance of privacy and confidentiality is another issue, and steps need to be taken to protect privacy during all phases of the research, including data collection, analysis, data storage, and the publication of the results.

Experiments also involve some considerations that are generally not present in other kinds of research. Experiments with human subjects depend upon isolating a variable that is tested under laboratory conditions. Participants are often not told exactly what the study is trying to test, because they may change their behaviour to fit in with the experiment. This lack of information makes it difficult for participants to give informed consent.

Deception can also harm experimental research in a more general sense since it can erode trust and make people unwilling to volunteer for the study. Moreover, some researchers claim that it can alter participants' behaviour in future studies, thus compromising results for other researchers.

Another issue with experiments on human subjects is that they often depend on students to participate. Apart from the fact that students are not usually a representative sample of the population at large, there are also issues of coercion to consider. In cases where students are required to participate in experiments as part of their course evaluation or are offered extra credit, their choice to participate or not has essentially been removed. Moreover, paying students to participate can be problematic, given that poverty can drive people to accept options that they would otherwise reject.

For more information on this study, see:

Case Study 2 — Micro-finance games: Group lending versus individual lending in Peru

Lab experiments do not have to be carried out in the headquarters of a company or organisation. They can be carried out in settings that resemble the "field," so long as the researchers are able to control the experimental conditions to a satisfactory degree.

In the mid-2000s, a group of researchers working for the Financial Access Initiative and Innovations for Poverty Action carried out ten microfinance games in an experimental economics laboratory in urban Peru. The resulting article by Xavier Giné, Pamela Jakiela, Dean Karlan, and Jonathan Morduch describes the experimental process and results.

Many microfinance agencies only engage in group lending because it significantly lowers the risk of making loans within low-income communities. In group lending, individual borrowers guarantee each other's loans. Rates of repayment are generally high, at around 95%.

And yet the fact that group lending regularly out-performs individual lending is puzzling because group lending comes with problems of its own. For example, group lending is vulnerable to free riding because it is potentially easier for an individual to default against their group (who will cover for them) than against a bank. Whereas an individual who defaults runs the risk that the bank will not lend to them again, with group liability it is easier to maintain access to loans.

Does joint liability really encourage such "moral hazards"? To find out, the researchers set up a series of experiments that explored the impact of individual and group lending mechanisms on investment decisions. The purpose of these experiments was to show how liability affects whether people made risky or safe investments.

Methodology

The team set up a makeshift experimental economics lab in an empty room in a marketplace in urban Lima, Peru. They chose the location to attract participants whose profiles resembled those of microfinance customers.

The researchers recruited participants using two methods: employing delegates from the local association of micro-entrepreneurs to invite vendors to specific game sessions, and allowing participants to bring friends to subsequent experimental sessions.

Over seven months, the team ran ten experimental games an average of 29 times each. The games consisting of multiple rounds of borrowing and repayment. The researchers observe that playing a sequence of games with the same individuals allowed them to control for individual's risk preferences and assess the impact of each lending mechanism on risk-taking and loan repayment.

The researchers changed the variables in each of the ten games in order to assess the effects of different circumstances that mimic the actual conditions of microfinance programs. These included individual versus joint liability, dynamic incentives or no incentives, and the amount that players were allowed to communicate or to observe each other.

In each round of the games, experimental subjects explained the rules in Spanish. They were given "loans" of 100 points and were asked to invest their points into one of two projects: either a safe project with a certain return of 200 points, or a risky project that paid 600 points with a probability of one half. They were given game sheets on which to mark their choices. If a borrower's project succeeds they would have to repay their loan, but if their project failed they would not be able to repay. At the end of each session, participants were paid a fee for showing up and another fee for every treatment they had taken part in.

The researchers also conducted a census of the vendors in the market, which allowed them to compare their experimental group with the general market demographic and work out whether they were representative of this broader population.

Findings

The researchers found that subjects were more likely to make riskier investments when they had joint liability because, in the event that their investment failed, their debt would be looked after by the other members of the group. They observe:

Risk-taking broadly conforms to theoretical predictions, with dynamic incentives strongly reducing risk- taking even without group-based mechanisms. Group lending increases risk-taking, especially for risk-averse borrowers, but this is moderated when borrowers form their own groups. Group contracts benefit borrowers by creating implicit insurance against investment losses, but the costs are borne by other borrowers, especially the most risk averse. (Giné et al. 2010, p.1)

However, cutting off defaulting borrowers from future loans greatly reduced risk-taking behaviour.

Based on their observations and the work of other researchers, the authors suggest that joint liability is not always necessary to maintain high repayment rates. They state:

Given large enough incentives to avoid default, borrowers will choose safe projects and repay their loans. (Giné et al. 2010, p.33)

Hence it is not possible to conclude that joint liability is better than the individual liability or vice versa. Rather, how each kind of loan structure affects repayment and risk-taking depends upon how the contracts are structured.

Applications

Microfinance experiments have clear implications for policy, commercial operations, and the design of development programs. In fact, field experiments such as these have influenced microfinance institutions (MFIs), which are increasingly shifting towards individual liability loans with time-based incentive structures.

The researchers also point out that the question of whether contract structure inhibits risk-taking is important for policy development. Evidence suggests that most microfinance loans have a limited effect on the growth of businesses. If this is the case, perhaps contract structure could be altered in such a way that it encourages a level of risk-taking that is suitable for setting up a successful business.

Field experiments can be fruitfully combined with other methods to broaden their findings and applicability. Whereas the lab experiments in Peru demonstrate the effects of the collective action on individual behaviour, ethnographic studies could describe the mechanisms by which collectives operate.

For example, anthropologist David Stoll's research on debt in a Mayan town in Guatemala unravelled the puzzle of how an entire town became heavily indebted to lending institutions and to each other (see Case Study 2 in Ethnography in Consumer Finance). Similarly, Caroline Schuster, in her ethnographic research on microfinance in Paraguay, describes how joint liability loans, in which the entire group is responsible for paying back their debt, uses social relations as collateral in the absence of other viable forms of guarantee.

Ethics

Because this experiment is essentially a lab experiment that takes place of the field, the ethical issues it raises are largely the same as in the first case study in this section. However, the fact that it takes place in the field does raise some additional ethical issues.

To recruit participants, the researchers used a technique known as "snowballing"; that is, asking participants to bring along their friends to participate in the study. The problem with this method is that it makes it impossible for the researchers to know whether the new participants have been coerced into coming along or if they have agreed to participate of their own free will. Snowballing can also lead to bias in the research results. Chances are that the friends that participants recruit will be similar to them, and this may limit the representativeness of the sample.

More about Experiments

Examples in consumer finance research