Category Archives: Programming

The Use and Abuse of the DevOps Role

In the world of cloud computing, Agile development, and the race to do more with less, the role and use of DevOps has grown exponentially. At this point it is practically a buzz word; a point of bragging rights that you don’t need developer-only developers or operations-only operations people. The article How DevOps is Killing the Developer by Jeff Knupp does a great job detailing the background, use, and drawbacks of DevOps. This post is an extension to that.

In a constrained workplace, especially in that of a startup or any small team with big goals, DevOps can be a necessity. In that environment, Developers aren’t the only ones taking on additional roles either. Everyone in the company has to step up to fill additional roles.

The more important part, and the reason for my post, is that DevOps shouldn’t be over-used when the resources are available. Developers don’t work at their best, most productive, and most innovative when they are also worrying about QAing and deploying code, maintaining servers and dealing with production issues.

As anyone who knows how I work (I would hope), I’m not afraid to ssh into production systems and update IPTables or restart Tomcat. And I enjoy being able to do that. It’s a good change of pace and it is good to experience how your code is functioning and is being used in the real world. Staying too detached from production can insulate you from issues or inefficiencies in your code. There is no better way to get a bug fixed than to put a developer on the Nagios alarms for it.

As a Developer, having a wide range of skills along the development stack is a tremendous asset. But it is an asset that should be used sparingly, as it takes developers away from their most skilled and enjoyed roles. Used effectively, however, the DevOps role can be greatly beneficial to a company, whether it is out of necessity or to mix things up.

There is an additional potential use case for a full-time DevOps role when there are also developer-only developers and operations-only operations people. In this case I think it makes the most sense as an individual or team that is part of the Operations team but focused on development of tools and systems for use by or in support of the Operations team. This could be monitoring, maintenance, or deployment management tools. It is likely not software specifically for primary business objectives, but rather in support of operations and management of the platform.


The Benefits of Having the Source Code for Libraries

As a programmer, you can never, and should never, write every single line of code that your code needs[1]. It turns out that you end up writing very little code for what your programs actually do. Lets start with a simple example. Every CS student learns how to implement a linked list, but how many times does a real life programmer actually write their own linked list implementation from scratch? Never! I’m not trying to debate whether or not CS students should have to learn how to implement a linked list, which BTW I think they should. My point is that all common programming languages provide libraries for common programming tasks, like linked lists, hash maps, sorting algorithms, math functions, and so on, so you don’t have to write them from scratch.

There are also more necessary uses of libraries that offer more than just saving time and effort. This is the case when you want to interact with any external component. GUI libraries are a good example. If you want to write software for the Android platform you must use their libraries in order to display things on the screen. There are hundreds, maybe thousands, of classes written by Google that provides access to the Android platform, primarily for creating and manipulating graphical widgets.

But how do you know how to use libraries? The typical approach to convey to you what a library does and how to use it is through API documentation and code examples. This helps most of the time, but sometimes it is a struggle to use libraries correctly. Sometimes you don’t know exactly what the API classes, methods, functions or whatever expect in and sometimes you don’t know exactly what will come out, even with what would be considered high quality documentation. Sometimes it is just complicated. Then there is poor quality documentation where it becomes a guessing game.

Where does API documentation and code examples come from? Code. Why then is code generally considered a non-traditional or even unacceptable method of learning how libraries work and how to use them in your code? First, many people are very protective of their code and don’t like to let other people see it. Maybe they think they have something special and don’t want anyone stealing their ideas. Maybe they are self-conscious about their work and are afraid of criticism or that other people will discover their poor coding abilities. Second, libraries are usually complicated and without an understanding of how the code is designed it can be difficult to make use it. It is also usually more time consuming to dig through and learn someone else’s code rather than reading their API documentation or examples. But sometimes it is worth it.

The first hurdle, actually getting people to let you read their source code, isn’t something I can appropriately address in this post. But I can say that this hurdle is a non-issue with FLOSS (free (as in libre) and open source software). It is for the very point of this post that I avoid non-free software libraries like the plague.

The second hurdle, the difficulty in reading library code, is usually easier to get around. Reading code is the most descriptive way to truly understand how a library works and how to use it. With a little time and practice, it’s really not that hard.

An interesting way I often use library code is for optimizations. As a simple example, let’s say you are using a sort method. Often times libraries will offer many ways of passing data into basically the same method. For a sort method, the library may accept either a List object or an array. If I pass in a List, does the library convert the List to an array, perform the sort on the array, then turn that array back into a list? If I can work with an array in my code just as easily as a list then I can skip the extra conversion by just working with an array. The information that says the List method has to convert to and from an array could be provided in other documentation, like Javadocs, but if it wasn’t then the only way to know is to look at the code to see what it does.

I often look at JDK library code. Sometimes it is for optimizations. Sometimes I can better understand its use if I look at the code. Sometimes it helps decide if I should extend a library class, extend one of its base classes, or something else. Sometimes I am just curious. But I know that I am more effective at coding and my code is better when I have access to library source code.

Reading other peoples code can also help you be a better programmer. You can discover new and better ways of doing things and you also get better at reading other peoples code which has benefits beyond reading library code.

So even though you may never write a linked list implementation, you may some day need to read the code of an implementation you are using and should know what it’s doing and why.

[1] This entire post is with exception of extremely uncommon situations, such as embedded systems or low-level system control.


Where to Simulate Behavior

In this post I am going to describe how behavior does not have to be simulated on the surface; it can be simulated beneath the surface through the interaction of lower level components to produce the desired behavior.

This is most evident in the real world. The real world is built on top of real things, which are themselves built from smaller things, which are built on even smaller things, and so on, possibly to infinity, but more likely ending with strings. For instance, you are made of organs, which are made of cells, which are made of chemicals, which are made of atoms, and so on. And each of these exhibit a behavior which can be simulated in software if its behavior is sufficiently known and software engineering techniques exist to simulate them. While it may not be the case that behavior at all levels can be perfectly simulated, lets assume for my arguments that some day it can be mastered.

Following the example of simulating you, we could attempt to write software that behaves exactly as you do. But we can also write software that behaves exactly as all your organs do, connect them exactly as they are connected in your body, and we will have created something that collectively behaves exactly as you do. We could also write software that behaves exactly as all the types of cells in your body do. Then we could put these cells together exactly as they are put together in the organs in your body. Then connect these exactly as they are connected in your body, and we will have created something that collectively behaves exactly as you do.

This is quite an interesting revelation for me with important implications on how behavior can be simulated. Instead of attacking a complex problem head on and recreate it in software at the top level, if you know details on how the subsystems work and can sufficiently recreate them in software, you can write software to simulate the behavior of lower level components and then link them as the real lower level components are linked to produce the desired top-level behavior.

This introduces a new problem for designing software when this approach is taken. The problem is how to connect and combine the lower level components. This may be more complex than writing software to simulate the top-level behavior. A prime example is in simulating the human brain. People have very good models of neurons and can create software the behaves very close to the real thing. But the wiring of neurons in the brain is so complex nobody has connected digital neurons in such a way that the human brain is simulated through simulating and connecting digital neurons.

But the brain is not a nearly random mass of neurons. While the brain is nearly completely made of neurons, it has been found that neurons form specific components that perform specific functions that are then combined to form higher-level components and so on. Based on my basic understanding, neurons form cortical columns that perform specific functions such as recognize patterns and make predictions. Then these cortical columns form levels of cortical hierarchy that generalize information information going up the hierarchy and localize information going down the hierarchy. Then these levels of cortical hierarchies form functional systems of the brain such as the vision system, auditory system, etc. And finally these are combined to form what we experience as consciousness. (note that when I have been talking about the brain, I have been focusing on the neocortex, which is the brain of our brain where real intelligence occurs).

Based on this structure of the brain, we can attempt to write software to simulate any of these components of the brain such as the cortical columns or levels of the cortical hierarchy. But again, with the brain, it is so complex and so little is known, this is very hard to do. And even when moving up to higher level components it is exceptionally difficult to be able to put these components together to form what we call the brain.

I am still hanging on to the idea that any behavior at any level can at least some day be simulated sufficiently in software, but this is not the case today. But there is another option for simulating behavior when the complexity at the highest level is beyond the capability for direct simulation. If lower-level components are sufficiently known, they can be simulated and combined to form the top level behavior. But there is a trade off, it can be just as hard, if not harder, to combine simulated components than it is to simulate at a higher level.

The real point I am trying to make is that real-world behavior is driven by a hierarchy of components, each of which exhibits behavior that when sufficiently known can some day be sufficiently simulated in software. When attempting to simulate something like the human brain, there are several layers of the hierarchy that can be simulated, not just the top-level of the brain or specific lower-level components such as neurons. There are several layers above and below the actual neurons that exhibit behavior that might be simulatable in software. And when attempting to solve a complex simulation problem like simulating the brain, don’t lock yourself into the obvious paths. Look for and try to understand the hierarchy of components and find the level that make sense for the problem at hand. Some lower-level components might be unnecessary to simulate while some higher-level components might be too complex to simulate. Know that options exist and explore them.

A few final closing questions to clear up some words I’ve used.

What is behavior?

Behavior is observable information. It can be observed via direct human observation or measured via instrumentation. There may be more going on under the hood of the behavior, but I don’t think it matters. If we are trying to simulate the intelligent behavior of the human brain, why should it matter if or how our artificial brain thinks to itself? If for all externally observable and measurable reasons it behaves like a brain, I don’t think it matters what happens on the inside if our only goal is to simulate the intelligent behavior of the human brain. However if you are trying to simulate every neuron and every synapse then it does matter how it works on the inside. This could be the case if you are trying to understand the human brain and/or apply knowledge learned from the simulated brain to the actual human brain.

The point is that when only the top-level behavior is of concern, there is no need to simulate, in software, at any lower level unless a higher-level behavior is too complex or there is something to be learned by lower-level behavior. Because once you simulate, in software, at a particular level, the lower levels become irrelevant and do not exist in the simulation.

What does it mean to simulate?

I’m really using the word simulate in two ways. I have mostly used it to describe the process of creating software that is supposed to act like whatever you are trying to make it act like. In this case the software is directly simulating the desired behavior. This may or may not be the top-level behavior. It is just behavior that is being simulated in software.

The other way I use the word simulate is to describe the end simulation, regardless of whether it is being directly driven by software or whether it is being driven by the interaction of lower-level components. I have heard this called an n-order simulation, or when n > 1, emergent behavior.


Java is FREE (as in Freedom)

Sun finally announced that they are releasing Java under the GPL! That’s right, the language you love to hate is FREE, and not just free as in beer, but free as in Freedom. The Free/Libre Open Source Software community has been pushing this for years now, about a year ago we started getting signs that it might happen, and a few weeks ago it finally did. This is a very good thing for a many reasons. Not only is Sun contributing a massive amount of code to the FLOSS community, which helps everyone, but it is a major testament to the power and value of FLOSS. It also means that GNU/Linux distributions can directly distribute Sun’s Java. And most importantly, developers from around the world will be able to contribute to, and improve, the Java environment.

To check out the new OpenJDK project, check out this.


Update: Sun to open-source Java

SAN FRANCISCO — Sun Microsystems Inc. at its JavaOne conference today announced that it would open-source Java but added that before it does so, company officials have to be certain the move won’t lead to diverging paths in the code.

read more | digg story


The End Of Java?

There has been a lot of talk lately about the future of Sun Microsystems. My question is, if Sun were to fall off the face of the earth, then what would happen to Java? From my view there are three paths for Java if this were to happen: Java dies with Sun, Java is sold to some company to suffer the wrath or success of this company, or the Open Source community welcomes it with open arms, with the help of Sun, and breathes life into the language.

The first is probably the least likely. It is doubtful that Sun would simply abandon Java unless it held no weight in the programming community, which is not likely considering how accustomed so many developers are to the language.

The second I think would be the most likely considering the state of business in the world today. My guess is Sun will get the best price it can get for the language in hopes of bleeding as much money as possible from its assets. I’m not sure if this would be a last ditch effort to breath life back into the company, for it to take another path, or they would use the money to pay off debts or split it up among employees/shareholders.

Now this brings on another question, who in their right mind would purchase the rights to the Java language? Microsoft certainly would not, considering they already have a Java replacement, C#. Possibly IBM, or a startup with the right backing and ideas to revive the language. Would you if you had the money? It would be an interesting acquisition for a billionaire.

Last but not least, the Open Source solution. This I think will be the most beneficial to the language. There is actually already a compiler/runtime/classpath in the works as part of the GCC project called gcj. So there is the possibility that the Open Source community will beat Sun to the punch and provide its own implementation of the language.

But if you were to ask me, I would suggest to Sun Microsystems that they even beat the Open Source community to the punch and open the source to Java. This I think would be a major step forward for Sun, and could help secure their position in the market, possibly avoiding this whole mess.