Reef Technologies

A fully remote Python software house

Productization of a financial analyzer

For a customer from USA, we optimized, extended and refactored an application performing analysis of financial situation of large companies. When we started working together, the product was a script reading and writing csv files, with the computation time on a large data set exceeding 35 hours. After we finished, all data was ingested and written to a database, with the calculations on the same data set taking about 40 minutes (on the same hardware).

 

Jay Newton, Director of Software Development, said:

Jay NewtonWorking with Reef Technologies was a great decision. We were concerned about the technical debt that we had and low-quality code, so we needed world-class professionals to make it up. We are extremely happy and satisfied with the results, since we managed to meet our really tight deadline. We will definitely keep up this partnership! Thanks!

„what a stupid question”

For me, it is much easier to work with someone, if I assume that the person on the other side is competent to do the job, but sometimes does not have a critical piece of information. This information is often obvious to me, but missing for some reason, to the other person.

When working on a globally distributed project I found that every time, when someone from my team was appalled by a „stupid question” from the remote office, it was the missing information scenario. Sometimes it was the other way around and we were the ones with the missing information, while the remote office thought it should be obvious for us. In such case, the question didn’t seem to make sense. What I learned to do, was to provide the best answer I could without understanding what is going on, but immediately after that, ask the remote office an easy question: why are they asking about that? Turns out that every time they had a good reason.

Often our willingness to answer the question instantly changed to a top priority.

Quite often after we understood the background of the question, our answer changed 180°.

 

Performance optimization

For a customer from Lyon (France), we investigated a server performance issue. Website seemed to be slow, despite a new dedicated server with 40 CPU cores, 256GB of RAM and a 500Mbps network.

The matter was urgent, a slow website was turning away paying customers, while an expensive marketing campaign was taking place. Money was being lost, we needed to move quickly.

The root cause was a large packet loss of SYN packets sent to a third-party vendor API. The most effective solution for this was to use persistent HTTPS connections, with many requests handled over the same socket. The change in the code was performed by software developers customer has already hired, based on our analysis.

Damien Gensanne said:

Damien GensanneI had a real complex and specific problem to fix (Network + Python + API problem), 4 programmers tried during one month and all failed! But Reef Technologies found it just in 2 days and explained exactly what is the problem and how to fix it. Real professionals and the best workers I’ve met in 2 years running my business!

Parallelization in digging ditches and software development

There is a saying: 9 women can’t make a baby in a month.

This carries some meaning in software development, but it is difficult to relate – building software is very different from making babies. I heard a different version of this, called a „mathematical paradox”:

If you give a man a shovel and a goal of digging a cubic meter of dirt, he can do it in 6 hours. If you get 6 men and 6 shovels, can they do it in one hour?

Seems like easy math, right?

As it turns out that the answer is no – they would accidentally kill each other when swinging their shovels.

 

If you are building a service which needs to scrape information from 10 sources, it is fairly easy to parallelize, you can split it into multiple developers. That is because you are actually digging 10 separate holes in the dirt (around the same area).

However if you are working on something very dense, using many software engineers / construction workers can actually slow down the process.

 

Dear reader, it is now time to take 10 seconds to reflect on how this applies to your project. You know which one I am talking about. Yes, that one 🙂

10 seconds, starting now.

Jira hell

Imagine you want to file a bug in Jira. You click „new issue” and a form with a few tabs opens. Count the fields – there are more than 200 in total. You are forced to fill this thing properly by the organization.

Welcome to jira hell.

This sometimes happens, because the person in control over Jira configuration adds fields without fully understanding the consequences. Sometimes there are multiple administrators, which can lead to duplicate fields. In one project I found three fields for the same thing. My friend found dozens of fields which had an empty value in all issues – someone added those because he thought they might be needed. We don’t know who.

I’ve seen one field which had a red, bold text just under it, saying „you must fill this field!” or something like that. It sounded serious, but my team never filled it and, strangely, there were no consequences. That struck me as odd. If it is so important, not using the field must have some negative impact, right?

I decided to find out who added this field. That took a while, but the person said that it was needed in a past for analysis which has been done over the course of 6 months and was completed (terminated!) a year ago. Field or the red notice were not removed. Also, this analysis did not involve my team (there was no appropriate value to select from the combo box for what we did). As far as I know, until this day, every other team in the project is carefully filling this field for every issue that they report.

Think twice before adding a field to Jira. It is a serious decision with a long-term impact on many people.

Why does IT need HR?

Apparently the list of things HR deals with is sizeable:

  1. Payroll
  2. Insurances
  3. Absences
  4. Holidays
  5. Pensions
  6. Tax-things
  7. Reporting to authorities
  8. Occupational safety
  9. Work time management
  10. Contracts
  11. NDA’s
  12. Non-compliants
  13. Works councils
  14. Union collective agreements
  15. Benefits (non-monetary „salary”)
  16. Total compensation
  17. Bonuses
  18. Travel policies & reimbursments
  19. Merit money
  20. Travel partnerships
  21. Keys and id cards
  22. Soul doctoring
  23. Recruitment
  24. Hiring
  25. Policies, processes, practicies
  26. Mandatory trainings
  27. Development of managers
  28. Induction
  29. Employee branding
  30. Events coordination
  31. Employee/union negociation
  32. Learning & development
  33. Training
  34. Employee performance
  35. Employee development
  36. Feedback
  37. Merges & acquisitions
  38. Due dilligence
  39. Gender equality
  40. Social media (for employee branding)
  41. Well-being
  42. Expert/distant workers
  43. International project policies
  44. Help with organizational changes
  45. Employment planning & budgeting
  46. Tracking headcounts and FTEs
  47. Senior management contracts
  48. Leadership support
  49. Culture
  50. Talent
  51. HR IT systems
  52. Metrics, analytics & evidence-based decision making
  53. Employee surveys
  54. Tracking employee contact information
  55. Org charts

 

Could software developers, scrum masters, product owners, managers or CEOs do all of that? Maybe they could…

 

But why would you make them?

On hiring software developers through recruitment agencies

From what I’ve seen, the entire effort of a recruitment agency is:

  1. Gather requirements for the given position from the customer, put it in a Word document (takes maybe one hour)
  2. Send it to a recruitment portal, paste an excerpt on LinkedIn (takes a few minutes)
  3. Pretend to process the CVs that people send, for example by calling each person and asking how much money they want. 3 questions tops, so that it does not filter too many candidates (takes 15-20 minutes per candidate)
  4. Forward the CV to the customer (few minutes)

For doing the above they collect between 10% and 25% of the yearly pay of the filled position, at least where I come from. Could the client have that done the same by using internal resources? How do you think?

Attentive reader will observe that if the candidate demands the exact top of the budget the commission will be maximized for the recruitment agency.

Think about that next time an agent asks you about the budget.

consistent multi-client filesystem interface for low-cost cloud

A new low-cost could service (1/4th of AWS price) could potentially become an extremely cost effective storage asset for multiple bussinesses if it features an interface which is easy to integrate with their existing infrastructure.

Together with an open-source filesystem interface development team, we have designed many variants of a solution which would provide consistency and reliability in multiple endpoints using the same storage bucket.

The analysis has concluded that current feature setof the cloud in question does not allow for implementation of such a filesystem with acceptable performance characteristics. Using an additional service for locking would increase price too much and would potentially compromise security.

Additional features were requested from the could operator to enable implementation of a high-performance design.

X2 e-radio transmitter

X2 was designed as an advanced alternative to linux-hosted radio streaming programs available at the time.

The transmitter had full control over fade-in and fade-out, as well as on filtering the sound samples, which allowed for configuration flexibility so much desired by our radio operators. Thanks to this technology, they could separately configure the mixing of music-to-music and music-to-jingle, achieving unparalleled results in terms of end-user audio quality. X2 had also been one of the first transmitters to use a Compressor-Expander algorithm which carefully leveled the volume of music, delivering a smooth experience to the listener.

In 2005-2008 the X2 transmitter was licensed to the largest e-radio station in the world where it continuously delivered all input streams for 16 e-radio channels.

During all the years of production usage, X2 had no user-perceptible downtime, achieving 100% availability (including the period of partial power loss in the datacenter).

CPU checker for flash-based advertisements

Websites and desktop application vendors that sell flash-based advertisements often have issues with performance by the ads. This happens because the ad buyer uses vendors to prepare the content that looks stunning. Testing for CPU consumption is not a standard practice. If the performance of application is poor (due to a CPU-hungry ad), the reputation of the website/application suffers.

In 2010 we designed, reviewed and deployed a system which used a common desktop computer along with a task queue, API, web panel and email notification system. This system was used extensively by the Ad Traffic team of our customer to verify CPU consumption of the content before it was accepted for emission. The system operated continuously for many years.

In result, performance complaints for the affected services have vanished and the reputation of the customer company was gradually recovered.