I got hacked
I got hacked and it was all my fault.
Writing is something I enjoy, and I want to get better at it. In high school and university, writing wasn't my strong suit. I avoided classes that were based on term papers. I loathed having a minimal page count in these reports. I should be rewarded for word and page efficiency, sigh. In the workplace, slide show decks, documentation, and email correspondence also were tasks that I didn't look forward to completing. My grammar, spelling, and communication suffered for it. Creating my own content is a goal to improve my writing.
When I created this website, I made a conscious decision to only use cloud-native technologies. All software is running within a container. All configurations and configuration management is driven from a git repository. And, of course, a few scripts to help make everything play well together. I want to play with the latest technologies for my side projects. Why not on my website and articles, too.
I completed the website set up in a few days and I was very proud of myself. Once all of the hardware and software was in place, it was time to start creating content and writing articles. I spent a few days mapping out the style and direction of the articles I wanted to write. I crafted a few rough drafts and scribbled outlines to help me organize my thoughts. Before I was hacked, I had about ~20 articles partially written, somewhere between 40 to 80 percent complete. There were 3 articles fully published and available for public viewing.
And then boom!
H A C K E D !!!
My website wouldn't load. What could be wrong? Everything was working fine for the last 20 articles. Why now? Why me?
This is my first production issue, my first personal outage. My rationale/ engineering side took over, let's get to the bottom of this. Time to follow my training and troubleshoot the incident. What is causing the issue? Then, let's do whatever it takes to make the website operational again. Even if it has no content, an empty site or friendly error page is better than a 503 error.
I logged into my webserver and read the logs. All sorts of error messages about missing database tables. I was puzzled... why wouldn't the database tables be there. The tables were there 20 articles ago.
I log into the database and sure enough, the tables listed in the log file are not there. What happened? If my tables aren't there, then what tables are available? I entered the command to list all of the tables, and there was only one non-system table. The non-system table had a funny name, just random letters, numbers, and characters. What could this be? I query the table and it only had one row, with a few columns. The columns all had random names, just like the table. The values of the columns made a sentence. In summary, if I wanted to see my data again, I would have to transfer a good amount of money to the hackers' bitcoin wallet. And then send an email, as a receipt, to request the restoration of the data.
What did I lose
I laughed! I'm getting ransomed, and this website isn't my primary business. Furthermore, my website isn't popular enough nor opinionated enough to draw such an attack. The joke is on them. My writing isn't worth THAT much. I ain't paying!
Sadly, my ego did take a hit. How did this happen? And what about all the time spent (and lost) in writing up 20 80% complete articles.
I didn't have a backup for my writing. I am using ghost.io CMS blogging software to help me manage this website and the articles. It has features to assist with editing, publishing, and restoring the articles from previous iterations. The software's content versioning system was my backup. And unfortunately, this excellent feature doesn't apply when the database is corrupted.
How I fixed it
First things first! The database has been corrupted. The required tables are no longer there. I need to get the database and web site working again. I created a new database and reconfigured the CMS software to use the new database. Then I configured the site's look and feel. These steps were pretty easy and straight forward. Since I had all of the configurations saved in my private git repository. I just had to make small updates and run my scripts.
Alright! We're back up and running. However, the content is lost. I want my content, but I'm not paying the ransom. I don't want to embolden the malicious hackers. Instead, I'd rather spend the ransom on a database backup solution.
Fortunately, this incident happened early in my website's life. I didn't lose that much. It gave me a new idea and topic to write about. So thank you, hackers! I'm having the last laugh. Ha!
How the hack worked
So how did I get hacked? I thought I had a secure setup. It's cloud-native. I'm using nginx to manage the HTTP traffic and route it to the proper docker container. Only the app servers are being opened to the public internet. And most certainly, not the database, that's locked down. Furthermore, I'm running a firewall. Only Jack Bauer and Chloe could break-in.
One of the lost articles is on the infrastructure setup for the website. Don't worry, I do plan on recreating it. Hopefully, I can remember all of the details. Stay tuned!
I did some research and discovered that my firewall, ufw, and mapping ports from a docker container don't play nicely with each other. Both use IP Tables under the hood. And since they don't know about each other, they overwrite each other. The firewall is set to block all ports except HTTP, HTTPS via IP Tables. Then the containers are run, their internal ports are mapped externally via IP tables. So all of the running containers have their ports publicly exposed to the internet.
Time to test the theory. From my laptop in my local coffee shop, I use the MySQL client to connect to my domain using the default root user, password, and port. Sure enough, a connection is established. I'm in!
That's how the hackers did it. They didn't do anything clever or especially malicious.
I had a huge configuration error and paid for it.
I modified my docker setup to no longer map the ports externally. As a replacement, I'm using the docker compose networks to allow the containers to communicate with each other and not the public internet.
Problem Solved! Lesson Learned!
What did I learn
Don't use the default root user, password and port. Even if you have a firewall running. Even if the website is just a hobby. There are malicious hackers running scripts that blindly try to connect to all possible domains and IP addresses using the default root user, passwords, and ports to many open source database software and other software. While it was quicker to set up the database, I saved time by cutting security measures. A firewall is a strong protective measure, it shouldn't be the only protective measure. Changing/ disabling the default user, password, and port is another protective measure. Don't cut simple security measures for time.
Furthermore, going forward, I'm treating my content like my configurations. By applying the same configuration management ideals to my writing, I should be able to quickly recover from any future incident. All articles are written in markdown and committed to the private git repository. Spellcheck and Grammarly are applied. My editor can peer review my writing, provide feedback, and I can update the article accordingly. Then the article can be published to the website.
Boom! Now I have continuous integration and continuous delivery workflow for writing articles.
There is also an argument for static site generators. Since the articles are simple and straight forward, there is no need for a database. And with no database, there is no default user, password, port to configure. Additionally, static site generators fit into a CI/CD workflow very nicely, too. If I have any further issues, I'll look to migrating to hugo.
The more you know!
