Chaos Engineering: INEC and National Tech
Does tech in Nigeria really peak at the point where I get a foreign job, dreads and Apple devices?
Perhaps one of the funniest things I’ve heard this election is an INEC official saying “Nigeria is a Tech driver” on TV. There is absolutely nothing driving about how this election was conducted. four years of preparation for just one day and what you get is a grand display of royal incompetence.
Let’s take this from the top, shall we?
My polling unit was the Amphitheater, Unilag, I was there at 8 am. The INEC officials didn’t show up till 11 am, but that’s not even half the problem. They kept struggling with the BVAS device, and we got tired of waiting and decided to ask, they said “we have network issues”, so we offered our mobile hotspots and Wifis (4G or 5G - make your choice sirs!). A few minutes later it became, “we have BVAS configuration issues!”. I didn’t get to vote until 1 pm, and that was after one of their “tech personnel” showed up at the polling unit to rectify the issue.
From this point, it went downhill nationwide.
INEC came to tell us about the issue, they said they can’t “scale up the IReV”.
Lol.
Then we discovered, they built this on AWS. I’ll talk about the issues with this later, for now, let’s just follow the narrative that we can’t autoscale an AWS server..LOL
With that budget, AWS will assign on-call SREs and send you merch afterward, but let’s believe they couldn’t scale an AWS server.
Well, I must actually even commend them for considering AWS, because I’ve been around the never-changing stuck-in-the-past mindset behind some old-time engineering, they could have even pulled a CPanel on us on shared hosting, but it’s fine sha.
Let’s continue the AWS narrative. Using AWS isn’t the problem here if that’s all we have, but if you’re building for an entire nation (almost 100M registered voters), you don’t have an excuse to fail, you can already estimate how much traffic would hit your servers months before the day, it’s more or less a fixed value. There’s a way to build expecting that level of traffic.
You should get really powerful high-tier servers and be able to autoscale based on traffic, preferably, dedicated instances or dedicated hosts. How many instances are you running behind your load balancers?
The fundamental issue is how much time they had to prepare for this. There’s something called performance testing. Did you carry out load testing? did you stress test your systems? this is all the tech team should have been doing from October or November. It matters because you’re building for a whole nation. If you had four years and your servers lagged or crashed on election day, it’s because you didn’t prepare for the load.
Let’s now talk about why we’re running our national elections on AWS and most likely uploading results to S3.
Again, if you had four years, why was the budget not used to build a private solution? to be honest, AWS is only as secure as the IAM users and permissions granted to them. Not saying it doesn’t work but a country’s presidential election is not an e-commerce MVP.
How concerned were the builders about security? are your applications sitting behind reverse-proxy servers and firewalls? or is it because the riggers aren’t that intelligent to go the way of tech? They decided to do manual thuggery and rigging.
These are just the basics!
We’ve not talked about the actual business logic behind the whole BVAS thing because it does not yet make enough sense.
Fine, you’d say it was built for accreditation and whatnot, did it manage to do that well?
I mean if you built-in cameras on your device and took my picture to verify me on election day why not just go ahead and add a fingerprint scanner and make the voting biometric? what is the purpose of NIN? why is an 11-year-old able to vote? what was the actual problem BVAS was supposed to solve? if the device still needed an internet connection to transmit the results why could we not vote without using papers? can you snatch ballots that are immediately recorded and collated? why do Nigerians have to go through the excruciating experience called the “national collation center”?
BVAS offered no identity and no clear solution to any of the problems associated with the elections. It’s all chaos! honestly, Microsoft excel would have done a better job IMO.
What are the policies guiding the use of the BVAS? if they have to upload results on sheets of paper, how do you reconcile situations like this?
Was this thought of ?? what was the plan for edge cases like this?
Did you build your all-important result upload algorithm to compress photos?? wow!
Is it a social media DP (display picture)?
My final question. if this entire system have been put in the hands of the nation’s vibrant tech community who by the way are some of the smartest in-demand tech talents in the world, surely it would have been better handled right?
I mean what is this?? I can’t verify its authenticity but still, Upwork? why is INEC recruiting from here?
Edit: They definitely did. INEC upwork. I mean if they did this for a website update, what’s the probability they didn’t do this with IReV?
I personally believe the Nigerian tech bros and sis (the actual ones) have a lot more than tech conferences and articles to offer if allowed. They already built highly available applications handling massive loads daily. They already built Moniepoint, Paystack, etc. We keep talking about how the odds do not favour an electronic voting process but if anything, the grand failure of this BVAS sham, just proves it can work if handled properly, or is that place also a part of the must-not-touch national cake? INEC failed. This is on them. All of the chaos.
The Hard Parts
By the way, I’m very much interested in knowing how you would approach the system design for an electronic voting system for the 100M registered voters. I’d really like to see this fly. What system components would you use and how would you use them?
I’m probably thinking I’d try this out too. It’s all for fun, no sweats. If you can make a system design diagram that would be nice too, it would look like this for example:
you can tag me when you post your design on Twitter @el_chibs I’d really like to see it. Thanks.
I have a question though, wouldn't using a cloud infrastructure to setup the servers (if done well) have been the better option to go about things. ?
When you put it this way, one can say that INEC failed on all grounds, considering the resources and time they had to make this one of best elections in the history of Nigeria. It was all just big talks and nothing!