Identifying magical thinking during a developer interview

The first thing to test during your technical interviews is the person’s ability to solve or at least diagnose problems using simple rational “step by step” reasoning instead of random trial and error coupled with almost-superstitious hypothesis, aka “magical thinking”.

Magical thinking or superstitious thinking is the belief that unrelated events are causally connected despite the absence of any plausible causal link between them.

Let’s take this piece of javascript code as an example:

Copy 
1const rex = /[0-9]{2}.[0-9]{2}.[0-9]{2}-[0-9]{3}.[0-9]{2}/g;
2
3rex.test("94.07.27-337.03"); // true
4rex.test("94.07.27-337.03"); // false
5rex.test("94.07.27-337.03"); // true
6rex.test("94.07.27-337.03"); // false
7// The result alternates between true and false indefinitely.

For some mysterious reason, between each run of the rex.test function, the result is different. This apparently doesn’t make any sense as you call the function with exactly the same input, so it should consistently give the same output.

With this test, we will try to measure the level at which the interviewee is prone to fall into magical thinking. To do that, we’ll have to specifically ask the person why the code behaves like this and not to “fix it”.

Also, I want to clarify that this is not a manifesto about “clean code”. As you will see, there are both dirty and clean ways to go about fixing this problem, but it is not the point, and the quality of the solutions themselves doesn’t really matter.
This is not about seeing if the person is fixing things “the right way” or “quick and dirty”, but simply to verify the quality of the reasoning used when fixing things.

If you try to ask the question ”Why does it behave like this ?”, from my experience, there are two main “debugging” paths people will follow to answer you:

First one will probably involve one of the following:

• Digging into the regex to see what’s wrong with it. If you reason about it, it can not even be the source of the problem since it works correctly half of the time, and the regex declaration is static. You will probably see a lot of time wasting on this. You can in that case note that the person did not stop and think “can the regex be the cause?” in this case, because the answer is no, obviously. So, this is a sign of rushing into hypothesis and trial and error without reasoning first.

• Fix it by removing the “g” flag. So first, as the “g” flag tells the regex to search globally or not and what happens when you remove it is that you prevent the behaviour of the regex that creates the “bug”, but also prevent the regex from matching multiple occurrences inside a longer string. This fixes the problem almost by chance, and even if it changes absolutely nothing in the final result, if it works, the person most likely has no idea why, and if it doesn’t, still has no idea why. Let’s be real, you wouldn’t want to trust someone to fix a bug in production using this kind of reasoning, would you ? You would want to know what was wrong, and exactly why. On top of that, you asked for an explanation and not a fix.

  Copy 
  1const rex = /[0-9]{2}.[0-9]{2}.[0-9]{2}-[0-9]{3}.[0-9]{2}/;
  2
  3rex.test("94.07.27-337.03"); // true
  4rex.test("94.07.27-337.03"); // true
  5rex.test("94.07.27-337.03"); // true

• Another one is to simply use a different function that will workaround this behaviour without knowing why the first method doesn’t work in the first place:

  Copy 
  1const rex = /[0-9]{2}.[0-9]{2}.[0-9]{2}-[0-9]{3}.[0-9]{2}/g;
  2
  3!!"94.07.27-337.03".match(rex); // true
  4!!"94.07.27-337.03".match(rex); // true
  5!!"94.07.27-337.03".match(rex); // true

  As with the previous “fix”, although this one is one of the actual fixes you would want to apply for this particular problem, if the person fixed the problem this way but still can’t explain why the first method doesn’t work, she has not replied to the initial question.

There are two main problem with this type of thinking:

First is that, for more complex problems, it will lead to an exponential time to resolution compared to a more logical, step by step approach. If the problem is isolated and very simple, you have a good chance of being able to fix it by trial and error fairly quickly, but if you are dealing with a much more complex problem, you might never fix it using trial and error, or at least take way too long.

Second, even if you manage to solve the problem, you haven’t learned anything from it. It might occur again in the future in a different form, and as you haven’t understood the fundamental reason creating this problem in the first place, you will struggle to fix it once again.

Learn the fundamentals. Band-Aid remedies never last.

The problem with this way of thinking is not only that it is highly inefficient, chaotic and impossible to build knowledge upon, but also that, if you think like this, you will be much less efficient at creating decent complex architectures when needed and taking quality decisions. If you want to build a complex software, website or algorithm, I’m pretty sure trial and error is not a good strategy 😁

The second “debugging method” someone would use to solve this problem could look as follows:

1) Acknowledging that the code probably “remembers” something between two runs of rex.test that triggers the change in behaviour.

2) Note by the rex.test “dot” notation that the “rex” variable is an object, so it can have properties inside of it, like the “test” function, but it could store other things too that you have no idea about.

3) At this point, you can provide a good enough answer to the question: “Each time I run the rex.test function, something is stored inside the rex object and that something triggers a change in behaviour on the next run”.

If you asked the person to explain the reasoning out-loud as it happens and you hear something along the lines of step 3, I would consider it to be a good enough answer to validate that person is using the correct way of thinking and move on to the next question.

But you can decide to keep going, and in that case this inevitably will follow:

4) Trying to figure what is this “something” by inspecting the object, and then finding out about the “lastIndex” property.

5) Some people might even reason as far as how the “lastIndex” property works, but honestly at that point checking the documentation of the regex object is probably the way to go. (Actually, from step 3, this is probably what has to be done in a non-interview situation 😉)

This is the reasoning you want people to apply when going about coding problems. It will be exponentially more efficient when problems grow more complex, it will allow your team to understand the fundamental principles that lead to the problem in the first place and build upon that knowledge.

On top of that, you will have a clear explanation of what happens and how to avoid it in the future when you debrief.

This is a very simple problem, I’m pretty sure no one will fall for this ! 🤦‍♂️

Unless this somehow becomes a over-used coding interview test and every dev knows about it, I can testify that even developers with a good amount of experience will fail this test.

From my experience while running interviews and also just asking around to developers I know, upwards of 70% fail this test.

You have to take into account that it is quite rare to have to “figure out” some mysterious behaviour, as you usually read the docs for objects you are using and the behaviour is thus “expected”. Also with the stress of a coding interview, you have a tendency to fall back to this kind of “magical thinking”.

Everybody has some amount of magical thinking, no matter how rational that person is. You might not notice it but we all use analogous, and sometimes even magical thinking all the time in different situations in everyday life.

We absolutely need analogous thinking to function as you can not spend all your time reasoning from the fundamentals every time you want to do something. But a professional should avoid the use of this kind of reasoning in his field of expertise, especially when trying to fix difficult problems.

I will write other nice short little tests you can use during your coding interviews in this series of posts, but it is already a very good sign if someone bulldozes through this in no time.

You can see that person is cool headed, has a tendency to seek the root cause of problems and can explain reasoning steps well.

This last point will surely help when communicating and fixing problems with other members of your team. Being able to dissect and communicate in an easy “step by step” way situations, problems and solutions is a very powerful skill to have when working in a team.

So, what’s the fix? 🤔

I am writing the solutions here in case you want to use this test in interviews so you will be able to verify the answers of your candidate. But as I said earlier, the solutions are not really what we are after with this test.

You have multiple ways of fixing this problem but if the person figured out the cause it will be fixed in no time at that point anyways:

1) Reset the “lastIndex” property with rex.lastIndex = 0 between each run (very dirty, wouldn’t do that).

Copy 
1const rex = /[0-9]{2}.[0-9]{2}.[0-9]{2}-[0-9]{3}.[0-9]{2}/g;
2
3rex.test("94.07.27-337.03"); // true
4rex.lastIndex = 0;
5rex.test("94.07.27-337.03"); // true
6rex.lastIndex = 0;
7rex.test("94.07.27-337.03"); // true

2) Use .match on the string instead of .test on the regex object (the ”!!” is because .match returns an array with results. If the array is empty, this “transforms” it to “false”, if it’s not, “true”).

Copy 
1const rex = /[0-9]{2}.[0-9]{2}.[0-9]{2}-[0-9]{3}.[0-9]{2}/g;
2
3!!"94.07.27-337.03".match(rex); // true
4!!"94.07.27-337.03".match(rex); // true
5!!"94.07.27-337.03".match(rex); // true

3) Reassign the regex object between each run, by wrapping the testing logic where you declare the regex and run the .test function in a separate function for example. The regex object will be a new one on each run so lastIndex will always be at its initial value.

Copy 
1const testRegex = (str) => {
2    const rex = /[0-9]{2}.[0-9]{2}.[0-9]{2}-[0-9]{3}.[0-9]{2}/g;
3    return rex.test(str);
4};
5
6testRegex("94.07.27-337.03"); // true
7testRegex("94.07.27-337.03"); // true
8testRegex("94.07.27-337.03"); // true

You should not stop there:

Even if the person fails this test, it doesn’t mean that you should stop the interview at this point.

People will fail interview tests from stress and misunderstanding amongst other things, and testing the person on different points will paint a more accurate picture for you, that has more to do with knowing where to place this person on a “seniority scale” rather than just being a binary “succeed/fail” process.

Also, I encourage you to use tests that make sense for the role the person is interviewed for. Change the language, change the test, make it more contextual. Using riddles-like problems and typical “coding interview” binary-tree academical tests is a big no-no for me. This problem is well fitted for most javascript developers, whether they’re going to be front-end or back-end; but I doubt it would be well fitted for a data scientist for example. Although, if you understood the target behind this kind of test (magical thinking) I think you can come up with your own tests that are well fitted for your particular situation pretty quickly.

If you want to learn more tips and tests for your technical interviews, here is the complete list of the articles in this series of posts:

1. Identifying magical thinking during a developer interview (this article)
2. Checking understanding of programming languages building blocks
3. Difficult but simple solutions versus easy but complex ones