Building BlogDB
I read a conversation about blogs recently. One person put forth that blogs are no longer created or maintained and this is a shame. Others shared their current favorite blogs, proclaiming they never left and RSS still works. Someone else said the real problem is there is no where to find these blogs.
Once I had considered this I decided I would build BlogDB as an open-source project to show-case blogs, categorize them by tags, rate them as recommended/not-recommended, and discuss them. Then one would be able to follow blogs and get a feed of the posts, or follow users and get a feed of their followed blogs' posts.
Planning
The first thing I did was begin to design the interface. I drew out the pages and described the functionality. Sometimes the interface was too complex to go straight to the page, and first had to be drawn and redrawn on the whiteboard before it was put to the page.
From the drawings, next I needed to make the database model. I wrote the schema file first. A fair bit of it around user accounts is repetitive of how I often write an initial schema. Then I went through the drawings and for each item in the interface considered what I would need in the model. Once I had finished that, I added DBIC generation support and then generated the BlogDB::DB schema and then considered which additional methods any of the models would need and made notes on postits that I put over the interface drawings.
Once I had done this, I turned to the web interface. I go between Catalyst and Mojolicious fairly often, and the deciding factor is often if I'm going to make use of Minion, a job queue. In this project, I'm going to need to have an RSS reader, a website crawler, and something to get an image capture of the blog's homepage. For that, I'll want Minion, so I choose Mojolicious. I wrote the cpanfile and got my environment configured, then wrote the config and dispatch table and then boilerplate of the controllers. I write a script to start it and the basics of the webapp are outlined now.
One thing I really apprechiate during development is dropping the DB and bringing it back. I've been using docker for this, and I add the toolchain for it and then wrap up with a .gitignore for all the devel directories and my config file.
Coding
With all of this planning out of the way, it's time to get coding. I'll focus first on user registration and login. I tweaked some of the router for this when I realized I still wanted $person to exist in the template for pages that don't require a user to be logged in.
Next I want to work on the forgot password functionality. This is a good oppertunity to break my template down some more, I'll make a layout.tx and change register.html.tx and then to write the forgot password workflow. Now this functionality is working and that is exciting.
Now I need to add a way of creating tags. I want tags to be able to be suggested and then approved, so I do that. It's mostly working, however I need more UI buttons. So now I add additional UI & test the buttons. The tags interface is good enough that I can now move on to the blog-adding!
When I got to it I found I hadn't thought it out well enough. I spent a bit of time thinking about it, and drew out the workflows for adding a blog.
Once I had done that, I changed the routes.
Next I needed to update the database to support this new pending_blog table. I changed some columns around and regeneerated the DBIC classes, afterwards I made the controller for submitting a blog and tested it out.
While adding the blog importing functionality I noticed the dispatch table had started to clobber itself. It's certainly now time to add testing to this to ensure I don't undo previous work. What I want from testing is for each test to spin up its own copy of the database, and any functionality tested can be tested without depending on other test files, or shared state between them. I make one test that roughly folows this pattern and will become the basis for making a BlogDB::Web::Test module later.
Later has come and now the BlogDB::Web::Test exists, the initial test was cleaned up to use it and the db model modified to support connection credentials under test mode.
Now I have added the very basics of getting a new blog added. There is a small error, so a DB tweak fixes it.
I want to have screenshots of the websites be taken, and I want to download them into BlogDB and then serve them as static assets. I find, after a bit of trial and error, this backend api server that will take screenshots for me. I then write an api client that I can use for it, and publish it on CPAN. Now I go back and add the functionality to BlogDB.
Now I need to get back to working on the blog adding. I've updated the DB to support tags on pending blogs, and I've referred back to my sketch for how the edit blog page should be. Now I've added tags on the interface & controller, and made it possible to view a blog.
I have run into a problem of time. I believe it is important that I put this into production sooner rather than later, and I think some about what features I absolutely must have to put online and I make a plan in the notebook. Then I start commiting code. I make a lot of commits. A lot.
I so very much want to push this to production now, but I can’t. The next two steps on my list is that each thing must now be fully tested, and I must also change the HTML so that it sucks less. I have not done good security at all, all handlers are essentially open for anyone. That’s okay, because I was pushing it off until the tests were being written — then I could add the authorization properly and test at the same time.
Those tests took three days to write, and have ended up being these 15 commits.
Once I had completed the tests, I had this other problem. The website is really ugly. My weakest skill in this project is likely the HTML/CSS and general design. I have little intuition for it. I poked around some template locations and found a blog index and blog article type template and decided to take a go at forming a new UI. I made the following commits to make a new template, and make templates configurable so default (the first testing template) and simple (my fancy new one) can sit side by side and be changed by a configuration variable.
The template work turns out being less commits however, takes plenty of time.
Now it is wrapped. There are going to be bugs, there are tons of missing features and there is much work to be done. However, this needs to be deployed and those things need to be fixed as updates.
Deploying
I write a script to bring a Debian server up with docker and dex installed and then I build the docker containers. It's pretty smooth at this point, git clone, cd BlogDB, dex build mojo, dex build blogdb, and finally dex blogdb start.
I run into some issues and correct them. The site is now online, and I've added some blogs.
Closing
This is a good start.
There is much to be improved. There are plenty of features in the planning stage that haven’t yet been implemented. The production deployment needs to be re-done and shouldn’t use Docker. There is no method of updating code or the database in place. There is no method of backing up and restoring the databases.
The web application has been created, the database has been created, a full testing suite has been written against the endpoints and there is automation around spinning development databases up and down. BlogDB is in fact online, and this is good.