Tuesday, October 29, 2013

Automating software installs

This is my favourite software development topic: automating installs.

I'm a lazy programmer, in the sense that Larry Wall talks about when he says laziness is one the three great virtues of a programmer: that means I hate doing things manually over and over again. I'd rather spend an hour writing a script to download and install a utility, then doing it manually many times on multiple machines. Automating the installation of tools also has some other benefits: (a) it ensures the consistent replication of environments and (b) it serves as documentation of exactly what was done for future reference.

Nowadays I make a lot of use of virtual machines (mostly using Hyper-V recently) for my development on Windows. This allows me to easily delete VMs and start over as often as I like. It also means I can keep my development environments separate: e.g. one VM for Java/Eclipse, one for an older version of Visual Studio, one for playing with Ubuntu, etc.

When you install a lot of development environments over and over, you need to be aware of the different tools available to help automate things. In the Linux world, automated/unattended installs are the norm (using things like apt-get), but in Windows it takes a bit more effort to automate the installation of applications.

Here's a few ideas for installing utilities/tools from the command-line (i.e. from scripts):
  • Chocolatey - very cool tool to install lots of popular applications with a simple command like: cinst notepadplusplus
  • Ninite - Pick the applications you want to install and it generates a single installer to package them all.
  • DISM.exe - a Windows tool to Enable/Disable Windows Features from the command-line. Use this to enable things like IIS or Hyper-V from a script.
  • WebPICmd.exe - Microsoft Web Platform Installer - install certain products like SQL Server Express from the command-line.
  • In some cases it's easy enough to just write your own customer silent installer for a product. A lot of applications use InstallShield or other well-known installers and a lot of them behave the same way, so it's usually possible to figure out which command-line parameters to pass to the installer to do an unattend/silent install.
In addition to the above tools, you obviously need to be comfortable writing scripts of some sort. In the old days, I used to use Perl for all of my scripting needs. In recent years I've done more plain Batch files and some PowerShell. PowerShell is great because it's so ubiquitous on Windows systems, but it does have a steep learning curve in my opinion. I'm currently trying Python as my primary scripting language instead of plain old Batch files.

Using Chocolatey:

As an example of using Chocolatey on Windows, let's install the first two tools every developer needs on their machine: cURL and the 7-Zip Command Line tool for extracting archives from the command-line:

First, install Chocolatey itself (requires .NET Framework 4.0 to be installed) and then we use cinst to install curl and 7za:
C:\> powershell -NoProfile -ExecutionPolicy unrestricted -Command "iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))" && SET PATH=%PATH%;%systemdrive%\chocolatey\bin
C:\> cinst 7zip.commandline
C:\> cinst curl
And that's it!

Using Web Platform Installer (webpicmd):

Let's install webpicmd with Chocolatey and then we'll install IIS Express using webpicmd:
C:\> cinst webpicmd
Now list all packages matching "IIS":
C:\> webpicmd /List /ListOption:Available | findstr "IIS"
From the above we see that we're looking for the package called just "IISExpress", so install it:
C:\> webpicmd /Install /Products:IISExpress /AcceptEula
Using PowerShell to install utilities:

Now, for contrast, let's say you've got a clean Windows 7 machine and Chocolatey won't work yet (because the .NET framework ins't installed), then it's easy enough to roll your own automated installs of 7-Zip and cURL using PowerShell. Once we have 7za and curl, this bootstraps everything: next we can easily script the download and install .NET Framework and Chocolatey...

To download 7-Zip command-line version (7za) and extract it using PowerShell:
function Unzip-File
    param ([string]$zipFile, [string]$destFolder)
    New-Item -ItemType Directory -Force -Path $destFolder
    $shell = New-Object -Com Shell.Application
    $zip = $shell.NameSpace($zipFile)
    $shell.NameSpace($destFolder).CopyHere($zip.Items(), 16)

Write-Host "Installing 7-Zip Command Line (7za)..."
$dest = "$env:ProgramFiles\7-Zip"
$destFile = "$env:ProgramFiles\7-Zip\7za.exe"
if (!(Test-Path $destFile))
    Write-Host "Downloading 7za..."
    $wc = New-Object Net.WebClient
    $url = 'http://downloads.sourceforge.net/sevenzip/7za920.zip'
    $file = "$env:TEMP\7za920.zip"
    Write-Host "Unzipping to: $dest"
    Unzip-File $file $dest
    Remove-Item $file
    Write-Host "7za already installed."
Next, let's download cURL and copy it into the Windows directory (because it's a single file and then it's already in the PATH for easy use):
function Unzip-File
    param ([string]$zipFile, [string]$destFolder)
    $7z = "$env:ProgramFiles\7-Zip\7za.exe"
    if (Test-Path $7z)
        $info = New-Object Diagnostics.ProcessStartInfo
        $info.FileName = $7z
        $info.Arguments = "x -y -o""$destFolder"" ""$zipFile"""
        $info.Verb = "runas"
        $proc = [Diagnostics.Process]::Start($info)
        $shell = New-Object -Com Shell.Application
        $zip = $shell.NameSpace($zipFile)
        $shell.NameSpace($destFolder).CopyHere($zip.Items(), 16)

Write-Host "Installing cURL..."
$wc = New-Object Net.WebClient
$url = "http://www.paehl.com/open_source/?download=curl_732_0_ssl.zip"
$file = "$env:TEMP\curl.zip"
Write-Host "Unzipping to: $env:windir"
Unzip-File $file $env:windir
Remove-Item $file
Now download the .NET Framework 4.5.1 using cURL (of course you can download it with PowerShell as well, but I like to use cURL):
C:\temp> curl -L http://go.microsoft.com/fwlink/?LinkId=322116 -o c:\temp\NDP451-KB2858728-x86-x64-AllOS-ENU.exe
C:\temp> NDP451-KB2858728-x86-x64-AllOS-ENU /q

Wednesday, October 23, 2013

Parenting: guilt when your child follows in your footsteps

Tonight my daughter, Emily (5 in a few months) called me and told me she can't sleep because she can't stop thinking about something even when she tries to think about something else. She asked: "Pappa, how can you stop your brain from thinking about something?" I replied: "It's very difficult to stop your brain from thinking! I know what you mean. I also find it difficult to stop thinking about things sometimes."

I was a bit scared to ask what it was that she was trying not to think about.

So tried to explain to her a technique I've been using for many years to fall asleep. I do it almost every night. I think I may have read about it (at least in some form or another) from a book about lucid dreaming by Stephen LaBerge when I was probably around 16 years old. It works almost every time, except when I'm really struggling to sleep, such as when I wake up in the middle of the night and start worrying about random stuff (in that kind of situation the only way to get rid of the anxiety is to get out of bed and watch tv or read for a while until I feel really tired). The idea is to think about (focus on) one part of my body at a time, making sure my body is totally relaxed. I start focusing on my left foot: is my foot totally relaxed? OK, move on: is my left leg totally relaxed and flat on the bed? Next leg. Then my lower body. Etc.

Emily seemed unsure whether she'd be able to do that, but she said she'd try.

Then I dared to ask: "What is it that you're trying not to think about?"

And it was what I thought it might be. She suddenly visibly tried not to cry and said: "I think about how I'm playing outside the classroom by myself. And then I look around and there is no one to play with."

My heart sank, because my wife and I have been talking about exactly this every day for the last few days. My wife recently discovered that our daughter is not as popular at school as we had assumed for some reason: my wife went with Emily to a classmate's birthday party a few days ago and realized that all the other kids were playing together and apparently didn't even seem to know Emily. I kept telling my wife she was worried about it for no reason, because as long as Emily doesn't see it as a problem, then there is no problem, because she's still young and lots of time to make friends.

However, the problem is: Emily has obviously been noticing that she hasn't got friends at school and it clearly bothers her a lot.

I told Emily: "Don't worry, once other kids get to know you, you'll make friends." She said: "The other kids do know me."  I said: "OK, I'll tell you a trick: look for another kid who is also playing by themselves, even if it's a boy, then go to them and ask if you can play with them. Ask them lots of questions, like what games they like to play, what shows they like to watch, what food they like, and then see if they want to play with you. If not, don't worry, just keep doing that. And one day, you'll have lots of friends!"

The topic has been so active on my mind that just this morning I asked my son's (3) teacher Patricia, when dropping  him off at preschool, if he's making friends. She said he's making lots of friends, and he's one of a band of about 5 boys who are apparently always playing together. So at least he's doing well socially.

The reason this has been a fear for us: my wife and I haven't really been the best examples. Not to mention whatever influence our introverted genes might have had on the kids. And living in a new country where we have no friends or family certainly doesn't help either.

In my childhood days I never had any friends at school until I was 13. It never bothered me the least bit. I liked being by myself. But it is really sad when the story seems to repeat itself with Emily, and I have no idea what we as the parents can do about it.

Monday, October 21, 2013

Do not design for reuse - design for loose coupling

I came across this cool line today:
"It can be better to copy a little code than to pull in a big library for one function."-- Go at Google by Rob Pike
Nowadays we keep hearing about all the virtues of code reuse, so much that it's almost becoming unheard of to question it. People may look at you funny when you suggest that some DTO class should be a separate class even though it happens to contain all the same properties as an existing model class.

It can be difficult to determine where to draw the line, but remember: just because two classes happen to look the same doesn't mean that it is necessarily the same thing. Sometimes copying & pasting is better than reuse e.g. using inheritance.

Reuse means coupling (or a dependency) when there is no need for that dependency.

I've always like this OOAD 101 blog post about coupling and cohesion:
"DO NOT design for reuse. Design for loose coupling, high cohesion. Do that and reuse will happen on its own."
Also see: Is code reuse a lie?

Wednesday, October 16, 2013

Mac OSX Keyboard Shortcuts

If you're a Windows user and new to Mac OSX, like me, here are some useful keyboard shortcuts and other tips:

Key Action
+space Quick search for an app or file to open
+tab 'Alt Tab' to another application
+` To 'Alt Tab' between windows of the same app (e.g. another Chrome window)
+control+shift+4 Select an area and take screenshot to clipboard
+shift+4 then space Click on a window to save that window to a file on the Desktop
⌘+option+esc Force quit applications

If you're using Remote Desktop Connection to connect to a remote Windows machine:

Key Action
+` To 'Alt Tab' between windows within the same app e.g. to another dialog window
fn+option+F1 Windows key (Start menu)
fn+option+F1+R Run dialog

You can configure your keyboard shortcuts in the Preferences of the Remote Desktop Connection application:

Also see:

Monday, October 14, 2013

Free private Git repository on TFS

You know GitHub of course, but it's not free for private repositories. Another nice option is BitBucket by Atlassian, which supports Git and Mercurial.

In this quick post I'm going to show you an option you may like especially if you're doing work in Visual Studio (like .NET/C#).  Not many people seem to know that Microsoft has a free Git hosting option called Team Foundation Service. This is basically a free online/hosted version of the TFS that we all know and love. (A bit of sarcasm there. Maybe.) The online TFS supports up to 5 users in your project for free.

Go to tfs.visualstudio.com, log in with your "Live ID" (like Hotmail/MSN/Outlook email address), and choose your own subdomain under .visualstudio.com. Create your new Git repository, 'test' in the example below.

When you're just starting with your new repository, initialize the new repo:
git init
Let's create our first file in the new repo:
echo '# Welcome!' > README.md
Now see what's going on in our repo:
git status
This shows us that we have the new README.md file as an "untracked file". Let's add the new file:
git add .
If you want, do a git status again, and note that the file is now added and new. 
Next, commit the change:
git commit -m "Initial commit"
Add the remote server called 'origin':
git remote add origin https://myname.visualstudio.com/DefaultCollection/_git/test
Now upload your newly created repository:
git push -u origin --all
(Note that your username and email address defined in your global git config is what will be displayed when you commit changes to this Git repo. This may perhaps be different from your TFS login, which is not necessarily what you want. So just make sure to check your config if you're using multiple Git accounts.)

Now add your friends to your private repo, and then they can clone the repository:
git clone https://myname.visualstudio.com/DefaultCollection/_git/test
Also see:

* SourceTree (Very good GUI for Git by Atlassian)
* GitHub for Windows client (also useful to get the PoshGit shell which is nice for command-line work)
* To create a new Git repository from Visual Studio, see: Publish your code into Team Foundation Service

Wednesday, October 02, 2013

Good development team practices

Some good software development practices:

Automate everything.

It often takes much longer to automate a small task than just doing it manually. I don't know about you, but I hate doing the same mundane task over and over. I admit, it's sometimes possible to take it too far as well. Yesterday I easily spent at least 30 minutes creating a single line cURL command for downloading JDK 1.4 from the Oracle Archive Download site.

Automation also serves as documentation. When you write the steps for installing some utility in a script, then you don't need to explain to another developer later what needs to be installed. Just read the script to see what's needed.

Remember that there are lots of existing tools to help automating things, like NuGet and Chocolatey.

Consider choosing a standard scripting tool like PowerShell or Python for everyone in the team to use. You can do a lot with just plain old batch files too.

At a very minimum, please, please don't do your builds manually! For builds/compiling your code, you obviously need to use something Jenkins, TeamCity or TFS. Even in very small teams you should do automated builds / continuous integration.

Test everything.

If you're not already doing some kind of automated tests, you absolutely have to start. There is no excuse. Automated testing is not something you wait for your QA team to do - it's a development task.

Unit tests are possible even with legacy code. At least start writing any new code in a testable way and build your tests up over time.

Integration tests should be possible for most projects. In .NET with WPF, start using Microsoft's Coded UI tests. It's a very easy way to automate your WPF UI. If you're using ASP.NET, then you'll want to use WatiN.

Just don't get too religious about testing and code coverage and forget what you're actually trying to accomplish. Sometimes we spend ages unit testing something of pretty low value, while the actual problem areas aren't covered well. There's no point in having 100% code coverage when you don't have enough tests to prove that you've met the acceptance criteria.

Follow good check-in/commit etiquette.

Commit often in small chunks. Pull / Get Latest often. This makes it easier for yourself because you reduce the number of conflicts to deal with.

Use good comments for your commit messages, not just something vague like "fix".

When merging conflicts, be considerate. Don't just throw away someone else's changes because you're too lazy to carefully merge your changes. If you can't tell what's going on and a merge isn't feasible, be considerate and take the server version then carefully put your bits back again.

Keep learning.

Encourage individuals to stay up to date with the continuous changes in technology. Follow famous developers on Twitter and Google+. If you don't know where to start, just follow one famous person you know and go from there. For .NET, follow Scott Hanselman on Twitter and/or Google+.

As a team, make time to learn together by: (a) trying new things, (b) doing Patterns & Practices meetings, (c) doing informal "show and tell" presentations. Encourage the team to work on side-projects when you have a spare moment now and then.

Maximize savings / Eliminate waste.

Remember why we're working here. We're not here to have fun, to learn, to write unit tests or even to write code. We're here to solve business problems. Bottom line: to make the business money. (But yes, of course we want to have fun and learn while solving the real business problems.)

Before you write that cool ORM (or CSV parser or some other way to reinvent the wheel), ask yourself: are we in the business of writing ORMs (or CSV parsers etc)? If a perfectly good open source or commercial solution already exists, then don't reinvent the wheel. I love reinventing the wheel myself, I know how much fun it can be to write libraries that you imagine might be usable by someone else, but use your time wisely.

The same goes for other things that costs the company money. Don't waste printer ink & paper! Why are you still printing articles for reading? (or even worse, printing source code!?) Learn to read on a screen. Or buy a Kindle if you claim you don't like reading a computer monitor or tablet. Use Pocket or Instapaper to save articles for later reading on a suitable device.

Always look for improvements.

I often hear a colleague say something like "But  that's not Agile/Scrum!", then my response is "We call it Agile because the process itself is agile." Don't treat your processes/methodologies like a religion. Change things that don't work and do more of what works well. Have regular retrospectives and follow up on improvement actions identified. An exception would be when you're starting with a new process like Scrum: first follow the rules strictly (do "Scrum by the book"), then only tweak your process at the end of each sprint according to issues raised in the retrospective.

Use the team's strengths.

Avoid specialization, but know your team's strengths (and weaknesses). Consider team members' interests. People who work on something they find interesting will generally be more happy and more productive. Consider what's the best use of everyone's time. For example, does it make sense for the most senior team member to perform the team's admin (sprint reports, keeping documentation up to date for CMMI, etc)?  Does it make sense for the most junior team member to spend a week figuring out how to install the new TFS server when another member could do it in an hour? It's important to let everyone work in as many areas of the project as possible to distribute the knowledge as much as possible, but there's a balance to keep.

Be pragmatic.

Don't be scared to go outside of your comfort zone to use the right tool for the job. For example, if you just spent half the day searching for a Markdown to PDF converter, and the only good solution you could find is a Ruby & Linux-based solution, then don't disqualify it just because your team only has experience using Windows.

Tuesday, October 01, 2013

Programming syllabus for the absolute beginner

A friend wants to start learning programming. In fact, she's considering getting a Computer Science degree and changing careers! I want to teach her everything I know: principles of object oriented design, dependency injection, high cohesion & loose coupling, unit testing, code contracts, MVC, jQuery, Python, NoSQL, Android, and .... Aargh, where do I even start?!

I need to come up with a learning plan for the absolute beginner. I want to get to the point where we could use Learn Python the Hard Way to work through the usual programming concepts and hopefully eventually build something interesting like a website or a mobile app. But I want to start at the very basics.

I'm thinking of starting with the following topics:
  • Introduction to HTML (design a page using a few tags and some minimal CSS)
  • Numeric expressions (operator precedence etc)
  • Hello world! (string expressions)
  • Binary math (bitwise operators, bytes, and, or, xor, not, shifts etc)
  • Boolean algebra / if-then-else
Some links I want to look at for learning Python:
  • Tutorial – Bit Banging and Boolean Math without the Math
  • http://lgm.fri.uni-lj.si/PA/PYTHON/PythonProgrammingfortheAbsoluteBeginner.pdf
  • https://wiki.python.org/moin/BeginnersGuide/NonProgrammers
  • http://learnpythonthehardway.org/book/ex1.html
  • http://www.ucs.cam.ac.uk/docs/course-notes/unix-courses/PythonAB