We had an ads system (picture something like Google Ads) in an e-commerce site. And to make it simple, let’s say that we had 3 key endpoints:

• One to retrieve ads
• Other to count impressions (a pixel request placed by the retrieved ad)
• The last one to count clicks and do the proper redirect

To count impressions, that endpoint should get an encrypted parameter, decrypt it, verify the (internal, decrypted) parameters with a signature, and then use those values impressions counting.

While the code that generated those links was deployed in servers that were readily updatable, the code that read those links (impressions counter) was deployed in an old monolithic system. Those servers were only deployable every 2 weeks.

• Link generator –> in servers that could update
• Link/impression reader –> in servers that couldn’t update

When this code was released, both servers (clusters indeed, but let’s call them servers for simplification) were running JDK 1.5.

In a given moment the server which was part of the readily updatable ones, got updated to JDK 1.6, and at that point.. we lost something…

In the morning we were alerted by the abnormality of having 0 (zero) impressions. Fall that could not be attributed to anything organic, but a code issue.

So, what was happening?

HashMap implementation changes

Remember that I said we were creating those links with a big encrypted parameter? That had a signature mechanism that was analyzed by the impression reader server code, and if the signature (simply done by hashing) wasn’t correct, the impression was discarded.
So the code for the impression counter link was something like:

for (String key: parametersMap.keySet()) {
    appendParameter(key, parametersMap.get(key),paramsBuilder); // Appends parameter
    stringToHash += parametersMap.get(key); // append value for the string to hash to get a signature
}

// calculate the hash and add some salt
stringToHash += secret;

signature = calulateHash(stringToHash);

appendParameter("signature", signature, paramsBuilder); // Appends parameter

On the reader/impression counter side:

for (String key: parametersMap.keySet()) { // before this, this code put the parameters into a Map
    stringToHash += parametersMap.get(key);
}

// calculate the hash and add some salt
stringToHash += secret;

signatureToVerify = calulateHash(stringToHash);

Can you spot the error there? It happens that in JDK 1.6, there was a slight change in the hashing code in HashMap, which made another slight change in the order elements were placed in the internal hashtable (an array).

So at the point where the impression counter server did .keySet, it provided those keys/values in an original order (by JDK 1.5) while the new code that generated that, was delivering those values in a different order (by JDK 1.6).

Obviously the bug is that relying on that order from the HashMap to append and retrieve values was a violation of the HashMap contract (and Set for instance), where it is expressed that you shouldn’t rely on the HashMap keys order.

This class makes no guarantees as to the order of the map; in particular, it does not guarantee that the order will remain constant over time.

Extracted from the HashMap class Javadoc

Developer’s quick solution to save the day

As the server that counted those impressions wasn’t available for an update (next code upload was scheduled for the next 2 weeks!), we had to do something on the ‘pixel generator’ with that link.

As the counter was expecting those parameters in a given order, we decided to use the old HashMap code, so we could get the same order to verify the signature.

The parametersMap object ended up being an instance of the HashMap15 class, for the sake of letting it sort those elements as the impression counter endpoint expected them to be.

Cleanup and conclusion

In a next release we did sort those keys before relying on the order and ditched that temporary class. (It wasn’t an elegant solution to keep).

So that’s the story of why we should know about data structures, contracts and reading docs… or maybe about minor bugs that could go unnoticed until big consequences happen. Have you ever had to resort to this kind of tricks to save the day and continue operations?

A Tale of Two Cities Book Cover Photo by Lisa from Pexels

Algorithms and Data Structures, why?

So let’s approach this question describing when knowledge of algorithms and data structures kick in -in unexpected ways sometimes- into our jobs.

One of the clearest examples is the usage of indexes in databases. It happens that certain queries start being very slow in systems where load and size have increased. Even optimizing the query (tables order, using less data, etc) would not be enough and the need of creating an index arises. Why indexes? Well, basic understanding of data structures make it easy to understand how data access speed is influenced by the data structures enabling it. The most common index in databases are B-tree indexes, which enable searches, access, etc in logarithmmic time. So we have it, algorithms and data structures come along again.

There is something called, “The Law of leaky abstractions” or “leaky abstractions” as is. Simply said, we could trust that the database itself will take the query and run it in efficiently. As we said before, this happens to be wrong in the long run. Sooner than later, the developer (or DBA?) will have to get his hands dirty and make that query perform better, regardless of semantics.

How does this all DB SQL Query thing relates with algorithmic tests? Well, those tests will reveal if the developer is aware of performance impact on many decisions during development. Unnecessary loops, or extra storage used may show disregard on knowing how things work, or that some knowledge is missing.

In some interviews we could find that developers rely too much on a given language libraries (data structures, strings, etc), to the extent that they think the library does magic and they wouldn’t explain “how they would implement it if they had to”. This kind of shallow knowledge gives us developers that, for example, don’t know how sorting works or why allocating memory for arrays in each write access is not good.

Solutions made by aware developers will always have those performance concerns on the radar, which makes those kind of solutions desirable. Clients know that, and in their search for great developers, most of them will want to be sure that the candidate has what it takes.

That’s the reason why at Hand-Picked we do both automated algorithm tests and live coding interviews. We are interested in knowing that our developers will make the cut and have low chances of obvious performance blunders.

I would like to mention some notes about performance in general, there is a saying

Premature optimization is the root of all evil

Credited to Donald Knuth, in a longer sentence

which basically states that acting on performance when it isn’t important, does more harm than good. Will you spend time trying out an idea for your customers to use, or rather seeing if you’ll need a Big Data database to store analytics? Above all, even though algorithmic knowledge is valuable, it is more valuable for the developer to have criteria when taking decisions. So there is also other phrase about priorities.

Make it work, Make it nice, Make it fast

This formulation of this statement has been attributed to Kent Beck

We can see that making it work (correct, do what it is intended) and ‘nice’ (which means, making maintainable code in some way) come first. What will you make fast if it doesn’t work at all? Rhetoric question, as it may happen that not making something fast enough, may make the thing NOT WORK AT ALL for a given volume, users quantity or data amount. So always pay attention, it may happen that performance could be of utter importance.

These two phrases and that counter-argument may look contradictory but they are put together here just to show how good criteria must always be on top.

Conclusion

As performance is an important point in Software Development, Hand-Picked verifies the algorithmic knowledge of its candidates through an Automated Coding Platform and a Live Coding interview. This way we know we are dealing with a candidate that has the proper tools to solve performance problems that may arise during their engagements. But we don’t exaggerate it, solving CS Olympics level problems isn’t a requirement to enter the Hand-Picked network, just a good level of criteria when coding.

Cover Photo by Alvaro Reyes on Unsplash