How Can We Develop For Tomorrow’s Needs?

This guest post is by James Schorr, who has been in IT for over 14 years and been developing software for over 11 years. He is the owner of an IT consulting company, Enspiren IT Consulting, LLC.  He lives with his lovely wife, Tara, and their children in Kansas City, Missouri. James spends a lot of time writing code in many languages and a passion for Ruby on Rails in particular. He loves to play chess online at FICS (his handle is kerudzo) and to take his family on nature hikes. His professional profile is on LinkedIn.

The average developer is often forced to get code out the door as quickly as possible, primarily due to unrealistic deadlines and budgets. As a result, the quality and future expandability of software is greatly harmed. Software is now used in medical machinery, our vehicles, power plants, stock markets, aircraft, weapons, etc… As software becomes more and more critical in our lives, the need to think long-term is becoming increasingly critical.

Obviously, the quickest way is almost always not the best way. I hope to give some practical steps to those involved in software development that will help in the development of stable, long-lasting software. A proper strategy session involving the below steps can help save a lot of wasted time and money.

Quality, future-resilient software is tough to define, but reveals itself when it does what it’s supposed to without unpleasant surprises, handles unpredictable user input and system issues in gracious, non-devastating ways, and, in general, makes the user’s life easier. The tough part is that user’s needs and systems change. How do we engineer for tomorrow’s needs?

The keys to successfully developing long-term software are:

Establishing the Purpose: What is the point of the software? Do the needs that it are anticipated to be met look as though they will be the same core needs in the foreseeable future? In other words, will the main needs be met by this software and can we easily build out from there? If not, we need to keep the anticipated future needs in mind as we “scope” out the architecture of the project and provide “space” for them.

Choosing the “Stack”: (what technologies, languages, etc… will be used). The stack should be chosen carefully, based upon:

• proven stability. For example, it may be “cool” but unwise to write the software in the latest-and-greatest language. I’ve seen instances where a language/framework is chosen strictly due to its current popularity. This is typically a recipe for disaster, as those who go (and enjoy) that route typically move on to the next greatest thing, leaving code behind for non-fad-following developers to handle.
• current in-house knowledge. For instance, maybe our developers love and know Ruby, should we really force them to write an app in VB? Or perhaps it is a Microsoft-shop, are time and funds available to facilitate the learning-on-the-fly of non-MS technologies? I don’t believe that it is ever appropriate to write mission critical software using a language/framework that is unfamiliar to the developers. There are times, however, where the software is so mission-critical and matches a language’s abilities to the point where it makes sense to pull in new talent. It can be argued that software can be written in almost any language, that the language itself doesn’t matter much. But sometimes it really does, both in terms of expressiveness and developer satisfaction (note: I still contend that a happy developer is a good developer, or at least becoming one).
• infrastructure requirements. Do we have the hardware and network necessary to decently support the software and its anticipated usage? Disk space, memory requirements, OS, network speed, etc… All of these matter, a lot. It’s best to always plan for 2-3x the anticipated usage. For instance, for a web app, if we anticipate 1k users, let’s build for 2-3k users, with built-in monitoring of the resources being used and a plan of how to scale up quickly when we hit a “soft” threshold.

Planning:

• Architectural Drawing: I’m a big fan of having at least the “skeleton” of the project drawn out, particularly on a white-board (I’m a bit old-school, I know). It doesn’t have to be a fancy diagram or complicated UML diagram, just a simple drawing; the more understandable, the better. This high-level overview provides guidance when we’re deep into code, as we can look up and see if we’re on track (as it’s all too easy to go down a code “rabbit trail” if we’re not careful). It is counterproductive, however, to draw out every little detail, as this will stifle creativity and overwhelm us while we’re writing code (we just won’t look at the diagram then).
• Establish Deadlines: we do need to know the deadlines. It’s best, in my opinion, to have several small deadlines with a semi-flexible final deadline. This helps us keep on track and measure our progress little by little. As we hit the small deadlines, our confidence builds, which then improves our productivity and, in general, our code quality.
• Using Available Expertise Wisely: does it make sense to assign Bob, the awesome Python programmer, to doing CSS and Bill, the great designer, to slinging code? Obviously not (I have seen some managers try this, though), we may lose both team members or end up with Google copy-and-paste code and animated GIFs in our Project. Cross-training is a nice and potentially valuable concept, but it should be done outside of a software project with its accompanying deadlines. Future minor features might provide a better opportunity ground for cross-training. If Bob’s swamped, maybe we need to find him some decent help.
• Determine the Deployment Strategy(for both during and after the Project):
  • code should be checked into our version control system prior to any deployments.
  • maybe we should only deploy code after business hours after alerting such-and-such a group. If our project has any possibility of negatively impacting others, notification is not only kind, but often necessary, especially for large changes.
  • a rollback strategy must always be in place. This strategy must be easily understandable with simple steps, so that little, if any interpretation by support staff, needs to be done in the “heat of the moment” support calls. Even if our developers are top-notch, until code gets into Production, we cannot be 100% sure that it will not need to be rolled back. This is why major companies often have to release an update quickly after a major release. Some things just can’t be easily discovered until they’re released into “the wild”.

Building with Expansion in Mind:

• One of the wonderful aspects of developing software is also its most dangerous aspect: flexibility. A feature or component can often be written in different ways. There typically are only one or two best ways, though. It can be very difficult to determine, unless one steps back from the project and thinks it through. Well-known software principles help a great deal with this, but come up short if they are not “placed up against” the anticipated needs of the future (in other words, if we don’t understand where we’re going, our code will still be awful even if we follow DRY, OOP, GOF, etc…). As much as possible, this needs to be done not by the developer but by someone outside the code, so to speak, perhaps a technical team lead, etc…
• When adding core features, we need to at least take a few moments to think through possible future implications of what we’re doing. For example, our Component A is currently parsing JSON from website B using C credentials. Component D depends upon Component A’s data. Wouldn’t it make sense to have these in an encrypted setting field somewhere to make it easy to change in the future? If Component A’s data was slightly different, would Component B “blow up”? Maybe we can abstract all of this a bit?
• Avoiding Spaghetti-Code: proper design and a commitment to sticking to the design in the future helps to prevent our code from such entanglement. In other words, we need to commit to never, ever quickly throwing code in to the project, as this leads to “spaghetti-code”. Of course, there may be the exceedingly rare occasions, where we may need to do such a stop-gap measure due to an emergency, but we must then learn from our mistake and commit to re-engineering that portion of the code properly.

Data Safety:

• As we depend more and more upon data, it’s becoming increasingly important that we do our best to have automated backups, which are then checked frequently by a person. This cannot be emphasized heavily enough. All too often, properly designed backups can stop working without anyone noticing until it is too late.
• If encryption is used:
  • the encryption keys need to be stored off-site in at least 2 secure places. Imagine if we lost our server(s), our office burned down, our VPS provider goes offline, etc…- even if we had backups, could we get to the raw data if needed? No one wants to start over from scratch.
  • Does the encryption depend upon a certain cipher? If so, what is the game plan for when that cipher is cracked someday? How easy will it be for us to move to a new cipher?
• Does our data depend upon a specific version? For instance, maybe database X version Y can open the data but no other versions can. Do we have a backup of that version to access the data if needed? Better yet, this reveals a key flaw in our design. Our data should not be heavily dependent upon any software version.
• Would our data be understandable if a new developer 10 years from now is assigned to work with it? For instance, if a column for a user’s API Key is called usrscr_ak12, we may understand it, but it’s not future-proof (a better term may be “future-resilient”, since nothing is truly future-proof). Such obfuscation attempts provide little security, as if someone can get that far (to the data), we’ve lost the security “battle” anyhow. Data should be clearly understood by those who can access it.
• Can our data be exported easily when the software that we’re lovingly developing now someday gets decommissioned? All software will eventually get replaced by something better. How easily can our data be decoupled from our application?

Pin-pointing Possible “Dominoes” in our project and code-base (e.g. if A happens, does it affect B, which then affects C, etc…, these can be like dominoes). Amazon’s recent AWS issues in 2011 revealed the criticality of this step. The more time that we spend anticipating what can go wrong, the more we can establish quick steps to both prevent such issues and to mitigate possible damage. At the bare minimum, the possible “dominoes” and recommended quick steps need to be written down somewhere. This can greatly help to expedite future troubleshooting.

• Our Software: We must try to anticipate, as much as possible, what the interdependencies are in our project and its surrounding infrastructure. These dependencies should be in written form and re-reviewed as further functionality is added to the software in the future (e.g. ITIL Change Management).
• Dependent Software: What software or systems will depend upon our software? When our system goes down, will other software be slamming our system asking for a response?
• Dependent Systems: if we saturate our network, is our software designed to “back-off” and retry after an appropriate, randomized delay?

Obviously, none of the above can be done overnight. If even some of the above is done, however, the chance of our software having a longer-lasting, positive impact will be greater. I recommend that the start of each project have at least a 3-5 days dedicated to going through these steps. Gathering input from the teams of people who are responsible for various components (e.g. clients/end users, network, sysadmins, developers of other dependent software, etc…) will be invaluable. The payoff will be great.

I hope you found this article valuable. Feel free to ask questions and give feedback in the comments section of this post. Also, do check out James’ other article: “Do You Enjoy Your Code Quality?” on RubyLearning. Thanks!

Do You Enjoy Your Code Quality?

This guest post is by James Schorr, who has been in IT for over 14 years and been developing software for over 11 years. He is the owner of an IT consulting company, Enspiren IT Consulting, LLC.  He lives with his lovely wife, Tara, and their children in Kansas City, Missouri. James spends a lot of time writing code in many languages and a passion for Ruby on Rails in particular. He loves to play chess online at FICS (his handle is kerudzo) and to take his family on nature hikes. His professional profile is on LinkedIn.

There’s something innately satisfying about programming; or, at least, there should be. It’s a creative act in and of itself that has incredible power and potential to affect change. Craftsmanship, in and of itself, typically attracts those who like to think outside the box and those who like to create. In this way, it really is more of an art than anything else and can produce a great deal of satisfaction and delight.

Compare the carpenter that hand-crafts a beautiful cabinet to a factory worker who pushes buttons to cause a machine to form identical cabinets over and over again. While both can experience a measure of satisfaction in their work, only the carpenter enjoys long-term satisfaction. The goal of this article is to enable you to improve code quality and, thus, transform the mundane into the beautiful. No matter where you’re at on the spectrum, beginner to advanced, there is always room for improvement. As the code quality improves, your ability to delight in it and enjoy what you’re doing does as well.

I have a number of applications in Production and have learned a lot over the years about the code quality, systems, and programming in general. I’ve also been regularly called on to carefully check a tremendous amount of others’ source code in my consulting business and earlier jobs. Here are some practical, real-world ways to improve your code quality and development habits in general. I have learned the hard way on a few of these. Some may only apply to the non-employee developer, but almost all are pretty universal:

Pre-Development

• Gather the requirements and “stories” from the client. I like the idea of stories better because it’s more in line with how the non-developer thinks. In other words, “I want A to happen as a B when I do C”. Make sure that those you’re meeting with are told to come prepared with as much information as possible, especially from the end users of the planned application.
• Question every need and distill them down by asking “Why?” In general, the requirements of users are based on what they know the possibilities are and what they’ve used before. You, however, may quickly see that a need is better met a different way, or even cause issues if fulfilled.
• Clarify what is necessary and “nice-to-have”, as this has a direct impact on the timeline and budget.
• Refuse to reproduce lousy software. In other words, turn down work, if necessary, that will need you to reproduce some other system that wasn’t designed well unless you’re given the freedom to do it right. It’s just not worth it, both for you or the users. How do you recognize lousy software? Look at the UI, speed, security, and stability; would you want to use it?
• Reject unrealistic timelines nicely, “Yes, I could do this in 3 months, but you won’t get the quality that you deserve. To do this right, we need 5 months.” Stick to your deadlines if possible. Ask yourself, “Do I want to work 17-20 hours a day to hit an unrealistic deadline?” Been there, done that, lesson learned.
• Accept the least amount of features for functionality for the first version. It’s far more important to have a stable foundation than a ton of features. Slate other features for future releases after the foundation is laid properly. You can expect future needs (and should), but only include the minimums for the first version or iteration.
• Realize that just because we “can” doesn’t mean that we “should”. Anything’s possible, but not everything’s advisable. This is a tough one, especially in the pre-development excitement, brainstorming phase. For instance, yes we could pull in a Google map for every row, but should we? How long will the requests take as the data set size increases?
• If you don’t know the answer to “Why” for a feature and “What” it will impact, don’t even start answering “How”.
• Ensure that you’re using a great version control system (I’m partial to GIT) and a hosted code repository if necessary. Backups are essential as well.
• Block off large amounts of undivided time to work on the project. For instance, it’s not a good day to write code if you have to leave every few minutes.

Development

• Spend as much time as possible understanding the real-world and data needs of the client before writing any code. This is absolutely critical to make sure that your application is designed from the ground as a stable, relevant application. Don’t move forward until you think that you have a pretty good grasp of the needs; get help if you need to.
• Out-engineer user-error as much as possible. In other words, never trust that the user will do what you expect, especially when entering data. For instance, about 4 years ago, I was reviewing a proposed payroll-entry system software for a Fortune-500 company. The software trusted the user to enter the number of their dependents properly for tax deduction purposes. When I questioned the developer, “What if someone enters a negative number?”, his face grew pale. We found that an employee would end up with a raise, instead of a deduction! Ouch.
• Don’t reinvent the wheel. For instance, it makes no sense to develop your own login system for Ruby on Rails when there are great ones already out there that have run through a gauntlet in the real-world.
• Don’t implement anything unless you understand, as much as time allows, what you’re “plugging in”. Read the issues reported, read through the source code, and familiarize yourself with how it works. Don’t choose something merely because 3 people recommended it in some forum. I’ve seen this often: a developer quickly slaps in a 3rd party open source app or gem and a few months later, there’s an issue that they do not know how to resolve. Usually it’s because they thought they’d found a quick fix, without thinking about any future issues. Perhaps the worst example that I’ve seen is hot-linking to Javascript code on a website somewhere, in blind trust that the link will stay live.
• Be open to including other languages and technologies where appropriate. For instance, just because your app is written in Ruby, if a Perl or Python is better suited for a certain backend feature, it may be good to use it. Even if you don’t integrate it, reading through it can help you see a better way to solve an issue at hand.
• Don’t do in code what your database can already do, as this will cut down the speed quite a bit. For instance, why use Date in Ruby in a SQL query on a MySQL-backed application when MySQL’s CURRENT_DATE() will work fine (and faster)? In other words, know the capabilities of the technologies that you’re using.
• Avoid including “experimental” or “cutting-edge” features in your project if possible. While fun, the risk of issues just isn’t worth it unless your project is just a learning exercise.
• Communicate what you need to keep the client in the loop. That being said, too much communication about every little snag and solution will only frustrate them.

Post-Development

• Review your code for speed, stability, security, and usability.
• Have someone else review it or discuss it with them.
• Have non-technical people do real-world testing on your product. You’d be surprised at how many things you think are intuitive and easy that really aren’t to an average user.
• Never, ever, ever stop learning, even if it’s unrelated to your current development language of choice.
• Don’t be afraid to ask for help. Even the best developers can gain a lot of insight and knowledge from reading others’ code and interacting with others.
• Revisit old code and see what you would’ve done differently. Often, you’ll find yourself encouraged as you see how better your code is now than then.

Enjoying Your Development:

• How do you work best? In quiet, with music, with lots of lighting, dim lighting, etc…? A happy developer is much more likely to produce quality. Spend some time setting up your work environment the way that you’re most comfortable with. It’s amazing how mood-based programming can be. Improve your mood and you’ll improve your code.
• What tool(s) do you like to use? As much as possible, insist on those tools, “Yes, I can write this app in TextEdit or Notepad, but I can do it better with this $79 tool X.”
• Give yourself time to think and rest. There are some days where I just can’t write code well; other days where it’s just flowing. This is due to how your brain functions. You need sleep and a change of pace and scenery now and then. Schedule breaks if you must and realize that some days are going to be tough. On tough days, learn something new.
• Walk away for a while. It’s easy to get “tunnel vision” and think that you’re close to solving a problem and to think that more effort will solve it. While true at times, I find that it’s best to walk away from the issue at hand for a while and do something completely different. For example, I tried once for 7 hours to power-through a complicated area of code and just couldn’t get it right. I finally stopped working on code that day, came back the next day to it, and solved it in 16 minutes. You would be surprised at the ideas or solutions that will spring into your mind as you are thinking about or doing other things.
• Set realistic deadlines. For instance, before telling your client or boss that this will be done next Friday, add a week. You never know what issues you might hit and will find yourself, more often than not, very grateful for the extra time to get it done right. If you end early, you’re a hero!
• Remember, you’re a person, not a robot. In other words, you deserve time to eat, sleep, get away, etc… When necessary, remind others nicely.
• Know your limits and enforce them. In other words, “No, I can’t and won’t recreate Gmail.”

As the work quality improves, it will stand out and you will be called upon more and more to help others, do other projects, etc… As your level of expertise grows, help others by tactfully pointing out improvements, respecting them, and encouraging them. In doing so, you will find yourself enjoying the quality of your work.

I hope you found this article valuable. Feel free to ask questions and give feedback in the comments section of this post. Thanks!